speculative decoding
30 articles about speculative decoding in AI news
DeepSeek DSpark: Speculative Decoding Unifies Parallel Gen, Adaptive Verification
DeepSeek released DSpark, a speculative decoding framework unifying parallel generation with adaptive verification. No benchmarks disclosed yet; the approach targets inference latency and throughput.
JetSpec hits 1,000 t/s on Qwen-8B with speculative decoding
JetSpec achieves 1,000 t/s on Qwen-8B with a B200 GPU, claiming superiority over prior speculative decoding methods, but lacks independent verification.
NVIDIA NeMo RL Speculative Decoding: 1.8× Rollout Speed at 8B
NVIDIA's NeMo RL speculative decoding achieves 1.8× rollout speedup at 8B and projects 2.5× at 235B, cutting RL training time by over half.
PayPal Cuts LLM Inference Cost 50% with EAGLE3 Speculative Decoding on H100
PayPal engineers applied EAGLE3 speculative decoding to their fine-tuned 8B-parameter commerce agent, achieving up to 49% higher throughput and 33% lower latency. This allowed a single H100 GPU to match the performance of two H100s running NVIDIA NIM, cutting inference hardware cost by 50%.
DFlash Brings Speculative Decoding to Apple Silicon via MLX
DFlash, a new open-source project, implements speculative decoding for large language models on Apple Silicon using the MLX framework, reportedly delivering up to 2.5x speedup on an M5 Max.
NVIDIA's Kimi-K2.5 Eagle Head: Supercharging Moonshot's Reasoning with Speculative Decoding
NVIDIA has released the Kimi-K2.5 Eagle head on Hugging Face, implementing Eagle-3 speculative decoding to dramatically accelerate inference for Moonshot's reasoning models. This breakthrough promises blazing-fast performance while maintaining accuracy.
mlx-vlm v0.5.0 Adds Continuous Batching, Distributed Inference for Apple Silicon
mlx-vlm v0.5.0 adds continuous batching, speculative decoding, and distributed inference for Apple Silicon. The release supports Qwen3.5, Kimi K2.5, Gemma 4 video, and new models with 21 contributors.
Paper Details Full-Stack MFM Acceleration: Quant, Spec Decode, HW Co-Design
A research paper details a full-stack approach for accelerating multimodal foundation models, combining hierarchy-aware mixed-precision quantization, structural pruning, speculative decoding, model cascading, and a specialized hardware accelerator. Demonstrated on medical and code generation tasks.
Claude Code's Secret Efficiency Hack
Claude Code leverages speculative decoding to reduce LLM energy use by 100x. Learn how this built-in optimization makes your coding faster and cheaper.
Nebius AI's LK Losses: A Breakthrough in Making Large Language Models Faster and More Efficient
Nebius AI has introduced LK Losses, a novel training objective that directly optimizes acceptance rates in speculative decoding. This approach achieves 8-10% efficiency gains over traditional methods, potentially revolutionizing how large language models are deployed.
Prefill-as-a-Service Paper Claims to Decouple LLM Inference Bottleneck
A research paper proposes a 'Prefill-as-a-Service' architecture to separate the heavy prefill computation from the lighter decoding phase in LLM inference. This could enable new deployment models where resource-constrained devices handle only the decoding step.
Dflash with Continuous Batch Inference Teased for Draft Models
A developer teased the upcoming release of 'Dflash' with continuous batch inference, targeting current text-only draft models used in speculative execution to speed up LLM inference.
SWE-Pruner Pro Saves 39% Tokens by Reading LLM Hidden States
SWE-Pruner Pro saves up to 39% tokens on coder LLMs by reading keep-or-prune signals from hidden states, maintaining task quality without external heuristics.
7 AI Agent Cost Optimization Strategies That Cut LLM Bills by Up to 90%
The source outlines seven cost optimization strategies for AI agents, including prompt compression and model routing, that can reduce LLM bills by up to 90%. This matters for retail and luxury brands deploying AI at scale where inference costs can become prohibitive.
Hugging Face Papers: 35B Agent Matches Trillion-Parameter Performance
Hugging Face Daily Papers featured eight AI papers, including Orca (world model), Dockerless (62% SWE-bench), and a 35B agent matching trillion-parameter performance.
OpenAI Cuts Inference Costs by Half on Some Models
OpenAI cut inference costs by 50%+ on some models for logged-out ChatGPT users, per The Information. The move reduces operational expenses.
NVIDIA Blackwell Cuts DeepSeek V4 Token Costs 5x in One Month
NVIDIA claims Blackwell inference stack cut DeepSeek V4 token costs 5x in one month, per a newly published report shared by @rohanpaul_ai.
Grouped Query Experts cuts long-context attention cost 44%
GQE speeds long-context attention prefill 1.7–1.8× by routing tokens to 9 of 16 query heads, matching baseline accuracy at 56.04.
DeepSeek Raises $7B, Ends No-Funding Pledge, Doubles Staff
DeepSeek raised $7B, abandoning its no-funding pledge, to double headcount and launch a coding agent team competing with Claude Code.
llada.cpp Cuts LLaDA-8B Latency 17-42x on Mobile NPU
llada.cpp, the first NPU-aware dLLM inference framework, cuts LLaDA-8B latency 17-42x on smartphones, enabling real-time on-device generation.
OpenRouter Fusion API Claims Fable-Level IQ at Half the Cost
OpenRouter's Fusion API routes queries across providers to match Fable-level intelligence at half the cost, per company claims. No third-party benchmarks disclosed.
MiniMax M3: Sparse Attention, 1M Context, Multimodal via Together
MiniMax M3 uses sparse attention for 1M context and multimodality, with Together AI serving fast inference.
Median Coding Agent Hits 96k Input Tokens, Rewriting Inference Economics
SemiAnalysis found median coding agent uses 96k input tokens from 432k requests, shifting inference cost focus from output to context.
Compute Shortage to Split AI Market: Rich Get Agents, Poor Get Chatbots
Mollick warns compute shortage makes agents expensive while chatbots cheapen, splitting AI market by company resources.
Qwen 3.6 27B Hits 34 tok/s on M5 Max MacBook Pro
Qwen 3.6 27B hits 34 tok/s on M5 Max MacBook Pro with 90% acceptance rate, per @rohanpaul_ai. Shows viable local LLM inference on Apple Silicon.
LASAR Cuts Latent Reasoning Steps in Half for GenRec at 20x Speedup Over CoT
LASAR nearly halves latent reasoning steps and achieves 20x speedup over explicit CoT in generative recommendation, outperforming baselines on three datasets.
Google Gemma 4: 3x Faster Inference with MTP Drafters
Google's Gemma 4 claims up to 3x faster inference via MTP drafters, but released no benchmark numbers or architectural details.
RedParrot: Semantic Caching Speeds Up NL-to-DSL for Business Analytics by
Xiaohongshu researchers propose RedParrot, a framework that caches normalized structural patterns of natural language queries to bypass expensive LLM pipelines, achieving 3.6x speedup and 8.26% accuracy improvement on enterprise datasets.
Sam Altman: AI inference costs dropped 1000x from o1 to GPT-5.4
Sam Altman stated AI inference costs for solving a fixed hard problem dropped ~1000x from o1 to GPT-5.4 in ~16 months, crediting cross-layer engineering optimizations, not a single breakthrough.
TACO Framework Cuts Agent Token Overhead 10% via Self-Evolving Compression
Researchers introduced TACO, a framework that enables terminal agents to automatically discover and refine context compression rules from their own interaction trajectories. This approach cuts token overhead by approximately 10% on benchmarks like TerminalBench and SWE-Bench Lite while preserving task accuracy.