latency
30 articles about latency in AI news
Codex Update Cuts GUI Workflow Latency 42%
Codex app update cuts GUI workflow latency 42%, enabling near-human-speed interface operation for autonomous app building and debugging.
Google Virgo Fabric: 100K-Accelerator AI Network Cuts Latency
Google unveiled Virgo, a data center fabric for AI clusters of 100,000+ accelerators, using a flatter two-layer topology to reduce latency and improve bisection bandwidth for synchronized training workloads.
OpenCLAW-P2P v6.0 Cuts Paper Lookup Latency to <50ms
OpenCLAW-P2P v6.0 introduces a multi-layer persistence architecture and live reference verification, reducing paper retrieval latency from >3s to <50ms and operating with 14 autonomous agents that scored 50+ papers.
EventChat Study: LLM-Driven Conversational Recommenders Show Promise but Face Cost & Latency Hurdles for SMEs
A new study details the real-world implementation and user evaluation of an LLM-driven conversational recommender system (CRS) for an SME. Results show 85.5% recommendation accuracy but highlight critical business viability challenges: a median cost of $0.04 per interaction and 5.7s latency.
HexaCercle Demonstrates Multi-Robot Hand Control System with 3ms Latency, 0.001° Precision
HexaCercle has demonstrated a wireless system enabling one operator to control multiple dexterous robotic hands with 1:1 movement replication. The system achieves 3ms data transmission latency and 0.001° collection precision.
OmniForcing Enables Real-Time Joint Audio-Visual Generation at 25 FPS with 0.7s Latency
Researchers introduced OmniForcing, a method that distills a bidirectional LTX-2 model into a causal streaming generator for joint audio-visual synthesis. It achieves ~25 FPS with 0.7s latency, a 35× speedup over offline diffusion models while maintaining multi-modal fidelity.
Quantized Inference Breakthrough for Next-Gen Recommender Systems: OneRec-V2 Achieves 49% Latency Reduction with FP8
New research shows FP8 quantization can dramatically speed up modern generative recommender systems like OneRec-V2, achieving 49% lower latency and 92% higher throughput with no quality loss. This breakthrough bridges the gap between LLM optimization techniques and industrial recommendation workloads.
Text-to-Video Model Achieves Sub-100ms Prompt-to-Output Latency
An AI researcher reports a text-to-video model generating outputs in under 100 milliseconds. This represents a 300x speed improvement over current models that typically take 30+ seconds.
Codex 'Ultra-Fast' Mode Spotted in Leaked Screenshot
Leaked screenshot suggests OpenAI is adding an ultra-fast latency mode to Codex. No release date or pricing confirmed.
Two-Tower vs Vector DB + LLM: Which Wins for RecSys at Scale?
Two-tower models offer sub-10ms latency for cold-start; vector DB + LLM provides richer semantics. Hybrid architectures reduce churn by 15-20%.
MiniMax M2.7 Hits 400 TPS on SambaNova Hardware
MiniMax M2.7 reaches 400 TPS on SambaNova hardware, making latency imperceptible. Details on model size and batch size undisclosed.
AllenAI's MolmoAct2: 720-Hour Bimanual Dataset, Beats GPT-5 on Robotics
AllenAI released MolmoAct2, an open robotics model with a 720-hour bimanual dataset, beating GPT-5 and Gemini Robotics on success rate (89.4% vs 82.1%) with 40% lower latency.
How a Custom Multimodal Transformer Beat a Fine-Tuned LLM for Attribute
LeBonCoin's ML team built a custom late-fusion transformer that uses pre-computed visual embeddings and character n-gram text vectors to predict ad attributes. It outperformed a fine-tuned VLM while running on CPU with sub-200ms latency, offering calibrated probabilities and 15-minute retraining cycles.
Continuous Semantic Caching
Researchers propose a theory-grounded semantic caching system that treats user queries as points in a continuous embedding space, using dynamic ε-net discretization and kernel ridge regression to cut inference costs and latency without switching overhead.
PayPal Cuts LLM Inference Cost 50% with EAGLE3 Speculative Decoding on H100
PayPal engineers applied EAGLE3 speculative decoding to their fine-tuned 8B-parameter commerce agent, achieving up to 49% higher throughput and 33% lower latency. This allowed a single H100 GPU to match the performance of two H100s running NVIDIA NIM, cutting inference hardware cost by 50%.
CS3: A New Framework to Boost Two-Tower Recommenders Without Slowing Them Down
Researchers propose CS3, a plug-and-play framework that strengthens the ubiquitous two-tower recommendation architecture. It uses three novel mechanisms to improve model alignment and knowledge transfer, delivering significant revenue gains in a live ad system while maintaining millisecond latency.
Xiaomi's OneVL Uses Latent CoT to Beat Explicit CoT in Autonomous Driving
Xiaomi's Embodied Intelligence Team released OneVL, a vision-language model using latent Chain-of-Thought reasoning. It achieves state-of-the-art results on four autonomous driving benchmarks without the latency penalty of explicit reasoning steps.
IOWN Forum Pushes All-Photonic WAN for AI Neocloud Interconnects
The IOWN Global Forum is focusing its optical networking tech on datacenter interconnects, aiming to let GPU 'neoclouds' and financial firms use cheaper, remote facilities without latency penalties for AI workloads.
PERA Fine-Tuning Method Adds Polynomial Terms to LoRA, Boosts Performance
Researchers propose PERA, a new fine-tuning method that expands LoRA's linear structure with polynomial terms. It shows consistent performance gains across benchmarks without increasing rank or inference latency.
Seven Voice AI Architectures That Actually Work in Production
An engineer shares seven voice agent architectures that have survived production, detailing their components, latency improvements, and failure modes. This is a practical guide for building real-time, interruptible, and scalable voice AI.
Kuaishou's Dual-Rerank: A New Industrial Framework for High-Stakes
Researchers from Kuaishou introduce Dual-Rerank, a framework designed for industrial-scale generative reranking. It addresses the dual dilemma of structural trade-offs (AR vs. NAR models) and optimization gaps (SL vs. RL) through Sequential Knowledge Distillation and List-wise Decoupled Reranking Optimization. A/B tests on production traffic show significant improvements in user satisfaction and watch time with reduced latency.
Building a Memory Layer for a Voice AI Agent: A Developer's Blueprint
A developer shares a technical case study on building a voice-first journal app, focusing on the critical memory layer. The article details using Redis Agent Memory Server for working/long-term memory and key latency optimizations like streaming APIs and parallel fetches to meet voice's strict responsiveness demands.
VMLOPS's 'Basics' Repository Hits 98k Stars as AI Engineers Seek Foundational Systems Knowledge
A viral GitHub repository aggregating foundational resources for distributed systems, latency, and security has reached 98,000 stars. It addresses a widespread gap in formal AI and ML engineering education, where critical production skills are often learned reactively during outages.
Google Launches Gemini API 'Flex' & 'Turbo' Tiers, Cuts Standard Pricing by 50%
Google has added 'Flex' and 'Turbo' service tiers to its Gemini API, with Flex offering a 50% reduction in cost compared to Standard. This move provides developers with more granular control over cost versus latency for their AI applications.
GRank: A New Target-Aware, Index-Free Retrieval Paradigm for Billion-Scale Recommender Systems
A new paper introduces GRank, a structured-index-free retrieval framework that unifies target-aware candidate generation with fine-grained ranking. It significantly outperforms tree- and graph-based methods on recall and latency, and is already deployed at massive scale.
CARLA-Air Unifies CARLA and AirSim Simulators in Single Unreal Engine Process for Embodied AI
CARLA-Air merges the CARLA autonomous driving and AirSim drone simulators into one Unreal Engine process, enabling zero-latency air-ground sensor synchronization with 18 sensor types for embodied AI training.
Sam3 + MLX Enables Local, Multi-Object Video Tracking Without Cloud APIs
A developer has combined Meta's Segment Anything 3 (Sam3) with Apple's MLX framework to enable local, on-device object tracking in videos. This bypasses cloud API costs and latency for computer vision tasks.
Mistral AI Releases Voxtral TTS: 4B-Parameter Open-Weight Model Clones Voices from 3-Second Audio in 9 Languages
Mistral AI has launched Voxtral TTS, its first open-weight text-to-speech model. The 4B-parameter model clones voices from three seconds of reference audio across nine languages, with a latency of 70ms, and scored higher on naturalness than ElevenLabs Flash v2.5 in human tests.
Google Announces Gemini 3.1 Flash Live: A New Real-Time AI Model
Google has announced Gemini 3.1 Flash Live, a new model variant focused on real-time, low-latency AI interactions. The announcement came via a developer tweet, indicating a potential push for faster, more responsive AI applications.
Momentum-Consistency Fine-Tuning (MCFT) Achieves 3.30% Gain in 5-Shot 3D Vision Tasks Without Adapters
Researchers propose MCFT, an adapter-free fine-tuning method for 3D point cloud models that selectively updates encoder parameters with momentum constraints. It outperforms prior methods by 3.30% in 5-shot settings and maintains original inference latency.