inference cost
30 articles about inference cost in AI news
OpenAI Cuts Inference Costs by Half on Some Models
OpenAI cut inference costs by 50%+ on some models for logged-out ChatGPT users, per The Information. The move reduces operational expenses.
Distilled Agentic Workflow Runs at 100x Lower Inference Cost
A new paper shows agentic workflow distillation achieving 100x lower inference cost, but lacks benchmark details.
Switchcraft Router Cuts Agentic AI Inference Cost 84%, Matches Top Model
Switchcraft, a DistilBERT-based model router for agentic tool calling, achieves 82.9% accuracy while cutting inference cost by 84%, saving over $3,600 per million queries.
AI Inference Costs Drop 5-10x Yearly: @kimmonismus Challenges Forbes
@kimmonismus claims AI inference costs drop 5-10x yearly, challenging Forbes' static compute cost narrative. This deflation rate implies rapid TCO reduction for enterprise deployments.
Sam Altman: AI inference costs dropped 1000x from o1 to GPT-5.4
Sam Altman stated AI inference costs for solving a fixed hard problem dropped ~1000x from o1 to GPT-5.4 in ~16 months, crediting cross-layer engineering optimizations, not a single breakthrough.
Image Prompt Packaging Cuts Multimodal Inference Costs Up to 91%
A new method called Image Prompt Packaging (IPPg) embeds structured text directly into images, reducing token-based inference costs by 35.8–91% across GPT-4.1, GPT-4o, and Claude 3.5 Sonnet. Performance outcomes are highly model-dependent, with GPT-4.1 showing simultaneous accuracy and cost gains on some tasks.
HyEvo Framework Automates Hybrid LLM-Code Workflows, Cuts Inference Cost 19x vs. SOTA
Researchers propose HyEvo, an automated framework that generates agentic workflows combining LLM nodes for reasoning with deterministic code nodes for execution. It reduces inference cost by up to 19x and latency by 16x while outperforming existing methods on reasoning benchmarks.
PayPal Cuts LLM Inference Cost 50% with EAGLE3 Speculative Decoding on H100
PayPal engineers applied EAGLE3 speculative decoding to their fine-tuned 8B-parameter commerce agent, achieving up to 49% higher throughput and 33% lower latency. This allowed a single H100 GPU to match the performance of two H100s running NVIDIA NIM, cutting inference hardware cost by 50%.
Thinking Tokens Drive Hidden Inference Costs in Agentic Pipelines
Thinking tokens from OpenAI, Anthropic, and Google models are priced at output rates, silently inflating costs 5x–10x in agentic pipelines. Google's 80% price cut threat exposes a structural asymmetry between startups and tech giants.
Microsoft Ditches Unlimited Copilot Tokens, Taps DeepSeek V4 for Cost Cuts
Microsoft switched Copilot Cowork to usage-based pricing, adopting DeepSeek V4 to cut inference costs by ~40%. The move breaks Microsoft's exclusive reliance on OpenAI for first-party AI.
Sleep Phase Cuts Transformer Costs by Consolidating Memory
Paper proposes sleep phase to consolidate context into fixed-size memory, reducing inference cost while improving long-horizon task performance on GSM-Infinite.
Median Coding Agent Hits 96k Input Tokens, Rewriting Inference Economics
SemiAnalysis found median coding agent uses 96k input tokens from 432k requests, shifting inference cost focus from output to context.
NVIDIA Vera Rubin NVL72 Cuts Agentic AI Cost 10x vs Blackwell
NVIDIA Vera Rubin NVL72 cuts agentic AI inference cost 10x vs Blackwell, per Huang at Dell event. 5,000 enterprises already on Dell factories.
GitHub Launches 'Caveman' Tool, Claims 75% AI Cost Reduction
GitHub has released a new tool named 'Caveman' designed to reduce AI inference costs by up to 75% for developers. The announcement, made via a developer's tweet, suggests a focus on optimizing resource usage for AI-powered applications.
OpenAI's Sora Integration: A Billion-User Gamble with Astronomical Costs
OpenAI is integrating its Sora video generation model directly into ChatGPT, potentially pushing weekly users past 1 billion. This ambitious move comes with staggering projected inference costs exceeding $225 billion by 2030, as video generation demands significantly more computational resources than text or images.
Plano AI Proxy Promises 50% Cost Reduction by Intelligently Routing LLM Queries
Plano, an open-source AI proxy powered by the 1.5B parameter Arch-Router model, automatically directs prompts to optimal LLMs based on complexity, potentially halving inference costs while adding orchestration and safety layers.
Google's 'Deep-Thinking Ratio' Breakthrough: Smarter AI Reasoning at Half the Cost
Google researchers have developed a 'Deep-Thinking Ratio' metric that identifies when AI models are genuinely reasoning versus just generating longer text. This breakthrough improves accuracy while cutting inference costs by approximately 50% through early halting of unpromising computations.
SpaceXAI Ships Grok 4.5, Blackwell-Trained Coding Model
SpaceXAI released Grok 4.5, a coding-focused model trained on Blackwell GPUs, now available in Cursor and Vercel. Inference cost claims lack independent benchmarks.
Mira Murati's Thinking Machines beats frontier models by 29.8% with Bridgewater's expert judgments
Thinking Machines beat frontier models by 29.8% fewer errors using Bridgewater's expert judgments, at 13.8x lower inference cost.
Visual-SDPO: Self-Distillation Fixes Code-Generated Visual Defects by +10 Points
Visual-SDPO uses visual-feedback self-distillation to improve code-generated visual artifacts by >10 points on ChartMimic, Design2Code, and AeSlides, with no added inference cost.
Continuous Semantic Caching
Researchers propose a theory-grounded semantic caching system that treats user queries as points in a continuous embedding space, using dynamic ε-net discretization and kernel ridge regression to cut inference costs and latency without switching overhead.
LLMAR: A Tuning-Free LLM Framework for Recommendation in Sparse
Researchers propose LLMAR, a tuning-free recommendation framework that uses LLM reasoning to infer user 'latent motives' from sparse text-rich data. It outperforms state-of-the-art models in sparse industrial scenarios while keeping inference costs low, offering a practical alternative to costly fine-tuning.
VISTA: A Novel Two-Stage Framework for Scaling Sequential Recommenders to Lifelong User Histories
Researchers propose VISTA, a two-stage modeling framework that decomposes target attention to scale sequential recommendation to a million-item user history while keeping inference costs fixed. It has been deployed on a platform serving billions.
Prompt Compression in Production Task Orchestration: A Pre-Registered Randomized Trial
A new arXiv study shows that aggressive prompt compression can increase total AI inference costs by causing longer outputs, while moderate compression (50% retention) reduces costs by 28%. The findings challenge the 'compress more' heuristic for production AI systems.
DOVA Framework Introduces Deliberation-First Orchestration for Multi-Agent Research Automation
Researchers propose DOVA, a multi-agent platform that uses explicit meta-reasoning before tool invocation, achieving 40-60% inference cost reduction on simple tasks while maintaining deep reasoning capacity for complex research automation.
The Agent.md Paradox: Why Documentation Can Hurt AI Coding Performance
New research reveals that while human-written documentation provides modest benefits (+4%) for AI coding agents, LLM-generated documentation actually harms performance (-2%). Both approaches significantly increase inference costs by over 20%, creating a surprising efficiency trade-off.
IonRouter Emerges as Cost-Efficient Challenger to OpenAI's Inference Dominance
YC-backed Cumulus Labs launches IonRouter, a high-throughput inference API that promises to slash AI deployment costs by optimizing for Nvidia's Grace Hopper architecture. The service offers OpenAI-compatible endpoints while enabling teams to run open-source or fine-tuned models without cold starts.
Why Cheaper LLMs Can Cost More: The Hidden Economics of AI Inference in 2026
A Medium article outlines a practical framework for balancing performance, cost, and operational risk in real-world LLM deployment, arguing that focusing solely on model cost can lead to higher total expenses.
The Hidden Cost of Mixture-of-Experts: New Research Reveals Why MoE Models Struggle at Inference
A groundbreaking paper introduces the 'qs inequality,' revealing how Mixture-of-Experts architectures suffer a 'double penalty' during inference that can make them 4.5x slower than dense models. The research shows training efficiency doesn't translate to inference performance, especially with long contexts.
DeepSeek, Zhipu AI Build Custom Inference Chips to Cut GPU Dependency
DeepSeek and Zhipu AI are developing custom inference chips to cut GPU costs. China's domestic chip budget share hit 46% in July 2026.