serving
30 articles about serving in AI news
Alibaba's RecGPT-V3 Boosts GMV 3.97%, Cuts Serving Cost 52.4% on Taobao
Alibaba's RecGPT-V3, a stateful hybrid-modal recommender with continual memory, boosts GMV by 3.97% and cuts serving costs by 52.4% on Taobao.
OpenReward Launches: A Minimalist Service for Scaling RL Environment Serving
OpenReward, a new product from Ross Taylor, launches as a focused service for serving reinforcement learning environments at scale. It aims to solve infrastructure bottlenecks for RL training pipelines.
Helium: A New Framework for Efficient LLM Serving in Agentic Workflows
Researchers introduce Helium, a workflow-aware LLM serving framework that treats agentic workflows as query plans. It uses proactive caching and cache-aware scheduling to reduce redundancy, achieving up to 1.56x speedup over current systems.
Brittlebench Framework Quantifies LLM Robustness, Finds Semantics-Preserving Perturbations Degrade Performance Up to 12%
Researchers introduce Brittlebench, a framework to measure LLM sensitivity to prompt variations. Applying semantics-preserving perturbations to standard benchmarks degrades model performance by up to 12% and alters model rankings in 63% of cases.
Bluente's Open-Source MCP Server Adds Format-Preserving Document Translation to Claude and Cursor
Bluente's new open-source MCP server brings professional document translation with format preservation directly into AI coding workflows. Developers can now translate PDFs, DOCX, and other documents across 120+ languages without leaving Claude Desktop or Cursor.
FreeToken Runs 284B MoE Locally on a Gaming Desktop
FreeToken claims 284B MoE serving on a gaming desktop via unified PC inference. No benchmarks disclosed; feasibility depends on MoE sparsity and offload strategy.
OpenAI Cuts GPT-5.6 Luna Price 80% to $0.20/M Tokens
OpenAI cut GPT-5.6 Luna prices 80% to $0.20/M input tokens, citing Sol-optimized kernels that cut serving costs 20%. Luna now undercuts Gemini Flash-Lite and Claude Haiku.
Function-Aware Fill-in-the-Middle Boosts SWE-Bench by +5.4 on 14B Models
Function-aware FIM mid-training boosts SWE-Bench by +2.8 to +5.4 on 7B-14B models, preserving general abilities. Six checkpoints and 400K dataset open-sourced.
Claude Code Rate Limits Just Doubled: How to Use the New Capacity Starting Today
Claude Code's doubled rate limits and removed peak-hour throttling on Pro, Max, Team, and Enterprise plans let you stop conserving Opus quota and run parallel agent sessions without limit anxiety.
MiniMax M3: Sparse Attention, 1M Context, Multimodal via Together
MiniMax M3 uses sparse attention for 1M context and multimodality, with Together AI serving fast inference.
Inference shift opens door for AI chip startups to challenge Nvidia
Inference shift from training to serving creates opportunities for AI chip startups. Nvidia's $20B Groq acquihire validates disaggregated compute strategies.
TACO Framework Cuts Agent Token Overhead 10% via Self-Evolving Compression
Researchers introduced TACO, a framework that enables terminal agents to automatically discover and refine context compression rules from their own interaction trajectories. This approach cuts token overhead by approximately 10% on benchmarks like TerminalBench and SWE-Bench Lite while preserving task accuracy.
OpenMedKit Adds GLiNER for On-Device PII Detection on iPhone
OpenMedKit is adding the GLiNER zero-shot named entity recognition framework to its toolkit, expanding its on-device, privacy-preserving PII detection capabilities for healthcare data on iPhones.
A Practical Guide to Building Real-Time Recommendation Systems
This article provides a practical overview of building real-time recommendation systems, covering core components like data ingestion, feature stores, and model serving. It matters because real-time personalization is becoming a baseline expectation in digital commerce.
HUOZIIME: A Research Framework for On-Device LLM-Powered Input Methods
A new research paper introduces HUOZIIME, a personalized on-device input method powered by a lightweight LLM. It uses a hierarchical memory mechanism to capture user-specific input history, enabling privacy-preserving, real-time text generation tailored to individual writing styles.
New Research Proposes DITaR Method to Defend Sequential Recommenders
Researchers propose DITaR, a dual-view method to detect and rectify harmful fake orders embedded in user sequences. It aims to protect recommendation integrity while preserving useful data, showing superior performance in experiments. This addresses a critical vulnerability in e-commerce and retail AI systems.
Indian Factory Workers Wear Head Cams to Gather Embodied AI Training Data
To overcome the high cost of robot fleet data collection, companies are deploying head cameras on human factory workers. This first-person video captures the sequencing, posture, and micro-adjustments of real work, serving as a proxy for expensive robotic action data.
Claude for Word Beta Launches, Integrates AI Assistant into Microsoft 365
Anthropic has released a beta version of 'Claude for Word,' a sidebar integration that allows users to draft, edit, and revise documents directly within Microsoft Word while preserving formatting.
ReRec: A New Reinforcement Fine-Tuning Framework for Complex LLM-Based
A new paper introduces ReRec, a reinforcement fine-tuning framework designed to enhance LLMs' reasoning capabilities for complex recommendation tasks. It uses specialized reward shaping and curriculum learning to improve performance while preserving the model's general abilities. This addresses a key weakness in using off-the-shelf LLMs for sophisticated personalization.
FedUTR: A New Federated Recommendation Method Using Text to Combat Data Sparsity
Researchers propose FedUTR, a federated recommendation system that augments sparse user interaction data with universal textual item representations. It achieves up to 59% performance improvements over state-of-the-art methods, offering a path to better privacy-preserving personalization where user data is limited.
GR4AD: Kuaishou's Production-Ready Generative Recommender for Ads Delivers 4.2% Revenue Lift
Researchers from Kuaishou present GR4AD, a generative recommendation system designed for high-throughput ad serving. It introduces innovations in tokenization (UA-SID), decoding (LazyAR), and optimization (RSPO) to balance performance with cost. Online A/B tests on 400M users show a 4.2% ad revenue improvement.
Inference Beauty Today Announces Global Platform Expansion, Powering Personalized Beauty Discovery for 100+ Retailers and Brands
Inference Beauty Today has expanded its AI-powered personalized beauty discovery platform globally, now serving over 100 retailers and brands across five markets. This signals the maturation of specialized, third-party AI recommendation engines in the beauty and personal care sector.
Meta's Adaptive Ranking Model: A Technical Breakthrough for Efficient LLM-Scale Inference
Meta has developed a novel Adaptive Ranking Model (ARM) architecture designed to drastically reduce the computational cost of serving large-scale ranking models for ads. This represents a core infrastructure breakthrough for deploying LLM-scale models in production at massive scale.
Apple's On-Device Reranking Model for Private Visual Search: A Technical Breakdown
Analysis of Apple's Enhanced Visual Search system that uses multimodal features, geo-signals, and index debiasing to identify landmarks entirely on-device. This represents a significant advancement in privacy-preserving AI for visual recognition.
VISTA: A Novel Two-Stage Framework for Scaling Sequential Recommenders to Lifelong User Histories
Researchers propose VISTA, a two-stage modeling framework that decomposes target attention to scale sequential recommendation to a million-item user history while keeping inference costs fixed. It has been deployed on a platform serving billions.
QuatRoPE: New Positional Embedding Enables Linear-Scale 3D Spatial Reasoning in LLMs, Outperforming Quadratic Methods
Researchers propose QuatRoPE, a novel positional embedding method that encodes 3D object relations with linear input scaling. Paired with IGRE, it improves spatial reasoning in LLMs while preserving their original language capabilities.
RSES CLI: Hand Off Coding Sessions Between Claude Code, Codex, and OpenCode in One Command
RSES CLI lets you seamlessly transfer active coding sessions between Claude Code, Codex CLI, and OpenCode with a single command, preserving context and task history.
Why Authenticity Will Be a Luxury in Hollywood’s AI Era
The Times argues that in an AI-saturated media landscape, genuine human creativity and authentic storytelling will become scarce, high-value commodities. This mirrors a core challenge for luxury brands: preserving brand soul and heritage in an age of synthetic content.
Structured Distillation for Personalized Agent Memory: 11x Compression with Minimal Recall Loss
New research introduces structured distillation to compress AI agent conversation history by 11x (371→38 tokens/exchange) while preserving 96% retrieval effectiveness. This enables storing thousands of exchanges in a single prompt while maintaining verbatim source access.
The Diversity Dilemma: New Research Challenges Assumptions About AI Alignment
A groundbreaking study reveals that moral reasoning in AI alignment may not require diversity-preserving algorithms as previously assumed. Researchers found reward-maximizing methods perform equally well, challenging conventional wisdom about how to align language models with human values.