inference
30 articles about inference in AI news
Redis Creator antirez Writes MiniMax H3 Mac Inference Engine
Redis creator antirez wrote a MiniMax H3 Mac inference engine, announced on X. Open weights enabled the port, highlighting community-driven hardware support.
Altman: Inference Demand Will Fund Frontier Training
Altman says inference volume, not margins, will fund OpenAI's training. He expects fungible intelligence and compute scale as moat.
Radix Ark: The Hidden Inference Startup Powering xAI
Radix Ark, behind SGLang, powers xAI and Chinese lab inference. SemiAnalysis flags team as top inference engineers; financials undisclosed.
LMCache Splits KV Cache From Inference, 14x Faster TTFT on H200s
LMCache separates KV cache management from inference into a dedicated process, achieving 14x faster TTFT on H200s with Qwen3-235B at 50 concurrent users.
AMD-Cerebras Disaggregated Inference: 5× T/s/W, Prompt vs. Decode Split
AMD and Cerebras launched a disaggregated inference platform splitting prompt processing on Helios from decode on WSE, claiming up to 5× T/s/W.
China Blank-Checks Huawei for Inference Chip Push
China is giving Huawei a blank check for inference chips, per a tweet by Natolambert on February 3, 2026, shifting focus from training to inference.
LLMs Learn to Switch Reasoning Effort at Inference Time
@rasbt explains how LLMs switch reasoning effort using inference-time methods and training, potentially cutting token usage by 30–50% on simple queries.
Hugging Face weekly papers: Monotonic inference policy overtakes training optimization
Hugging Face's top papers July 6-12 include a paper arguing monotonic inference policies are the true LLM RL objective, and Vidu S1 for real-time interactive video generation.
DeepSeek, Zhipu AI Build Custom Inference Chips to Cut GPU Dependency
DeepSeek and Zhipu AI are developing custom inference chips to cut GPU costs. China's domestic chip budget share hit 46% in July 2026.
ZML releases free LLM inference server supporting Nvidia
ZML released LLMD, a free inference server for LLMs supporting Nvidia, AMD, Google TPU, Apple Metal, and Intel Arc, aiming to reduce AI costs and break vendor lock-in.
Alibaba's MIPI fixes LLM training-inference mismatch with direct RL
Alibaba's MIPI directly optimizes inference policy, fixing the mismatch in LLM post-training via the MIPU framework.
OpenAI Cuts Inference Costs by Half on Some Models
OpenAI cut inference costs by 50%+ on some models for logged-out ChatGPT users, per The Information. The move reduces operational expenses.
Etched Hits $5B Valuation, $1B in Orders for AI Inference Chip
Etched hits $5B valuation with $1B in orders for TSMC-made inference chips, raising $500M from top investors. The startup targets Nvidia's dominance.
OpenAI, Broadcom Unveil Jalapeño ASIC for LLM Inference
OpenAI and Broadcom unveiled Jalapeño, a custom ASIC for LLM inference, targeting volume deployment by late 2026. No performance metrics were disclosed.
Miami Startup Claims 12M-Token LLM Inference at $8 vs. $2,600 on Claude
Miami startup claims 12M-token LLM inference for $8 vs. $2,600 on Claude Opus 4.6. No paper or benchmarks released yet.
AWS Beats Cloud Rivals to NVIDIA Blackwell with EC2 G7 — 4.6x AI Inference Gain Over G6
AWS launched EC2 G7 instances on June 19, 2026, becoming the first major cloud to offer NVIDIA RTX PRO 4500 Blackwell Server Edition GPUs. The instances claim 4.6x AI inference performance over G6, backed by 700 Gbps EFA networking and 32 GB GDDR7 per GPU. The move arrives the same week AWS confirme
Vultr Picks HPE, Nvidia GB300 for Inference Shift at HPE Discover 2026
Vultr selects HPE and Nvidia GB300 systems for inference, as enterprise demand shifts from training to production workloads.
ByteDance Builds In-House AI CPUs for TikTok-Scale Agent Inference
ByteDance builds custom AI CPUs for inference at TikTok scale, targeting scarce server supply. The move signals agent workload shift from training to inference hardware.
Median Coding Agent Hits 96k Input Tokens, Rewriting Inference Economics
SemiAnalysis found median coding agent uses 96k input tokens from 432k requests, shifting inference cost focus from output to context.
Distilled Agentic Workflow Runs at 100x Lower Inference Cost
A new paper shows agentic workflow distillation achieving 100x lower inference cost, but lacks benchmark details.
Cerebras Challenges Nvidia Inference Monopoly with Wafer-Scale Edge
Cerebras is challenging Nvidia's inference dominance with wafer-scale chips, as inference workloads surpass training in AI compute spend.
Perplexity Claims 3x Blackwell Inference Throughput for 70B Models
Perplexity AI claims 3x inference throughput for 70B models on Nvidia Blackwell GPUs via FP4 and custom scheduling. The gain exceeds Nvidia's own 2x marketing claim.
Switchcraft Router Cuts Agentic AI Inference Cost 84%, Matches Top Model
Switchcraft, a DistilBERT-based model router for agentic tool calling, achieves 82.9% accuracy while cutting inference cost by 84%, saving over $3,600 per million queries.
mlx-vlm v0.5.0 Adds Continuous Batching, Distributed Inference for Apple Silicon
mlx-vlm v0.5.0 adds continuous batching, speculative decoding, and distributed inference for Apple Silicon. The release supports Qwen3.5, Kimi K2.5, Gemma 4 video, and new models with 21 contributors.
Google Gemma 4: 3x Faster Inference with MTP Drafters
Google's Gemma 4 claims up to 3x faster inference via MTP drafters, but released no benchmark numbers or architectural details.
Inference shift opens door for AI chip startups to challenge Nvidia
Inference shift from training to serving creates opportunities for AI chip startups. Nvidia's $20B Groq acquihire validates disaggregated compute strategies.
AI Inference Costs Drop 5-10x Yearly: @kimmonismus Challenges Forbes
@kimmonismus claims AI inference costs drop 5-10x yearly, challenging Forbes' static compute cost narrative. This deflation rate implies rapid TCO reduction for enterprise deployments.
Google Splits TPU Line: 8t for Training, 8i for Inference
At Cloud Next 2026, Google introduced two new AI chips — TPU 8t for training and TPU 8i for inference — splitting its custom silicon for the first time. OpenAI, Anthropic, and Meta are buying multi-gigawatt TPU capacity, signaling a crack in NVIDIA's 81% market share.
DeepSeek-V4 Ported to MLX for Apple Silicon Inference
A developer has ported DeepSeek-V4 to Apple's MLX framework, allowing the large language model to run on Apple Silicon Macs. Early results show functional inference with room for optimization.
PayPal Cuts LLM Inference Cost 50% with EAGLE3 Speculative Decoding on H100
PayPal engineers applied EAGLE3 speculative decoding to their fine-tuned 8B-parameter commerce agent, achieving up to 49% higher throughput and 33% lower latency. This allowed a single H100 GPU to match the performance of two H100s running NVIDIA NIM, cutting inference hardware cost by 50%.