llm inference
30 articles about llm inference in AI news
OpenAI, Broadcom Unveil Jalapeño ASIC for LLM Inference
OpenAI and Broadcom unveiled Jalapeño, a custom ASIC for LLM inference, targeting volume deployment by late 2026. No performance metrics were disclosed.
Miami Startup Claims 12M-Token LLM Inference at $8 vs. $2,600 on Claude
Miami startup claims 12M-token LLM inference for $8 vs. $2,600 on Claude Opus 4.6. No paper or benchmarks released yet.
Prefill-as-a-Service Paper Claims to Decouple LLM Inference Bottleneck
A research paper proposes a 'Prefill-as-a-Service' architecture to separate the heavy prefill computation from the lighter decoding phase in LLM inference. This could enable new deployment models where resource-constrained devices handle only the decoding step.
arXiv Survey Maps KV Cache Optimization Landscape: 5 Strategies for Million-Token LLM Inference
A comprehensive arXiv review categorizes five principal KV cache optimization techniques—eviction, compression, hybrid memory, novel attention, and combinations—to address the linear memory scaling bottleneck in long-context LLM inference. The analysis finds no single dominant solution, with optimal strategy depending on context length, hardware, and workload.
ZML releases free LLM inference server supporting Nvidia
ZML released LLMD, a free inference server for LLMs supporting Nvidia, AMD, Google TPU, Apple Metal, and Intel Arc, aiming to reduce AI costs and break vendor lock-in.
PayPal Cuts LLM Inference Cost 50% with EAGLE3 Speculative Decoding on H100
PayPal engineers applied EAGLE3 speculative decoding to their fine-tuned 8B-parameter commerce agent, achieving up to 49% higher throughput and 33% lower latency. This allowed a single H100 GPU to match the performance of two H100s running NVIDIA NIM, cutting inference hardware cost by 50%.
Fractal Emphasizes LLM Inference Efficiency as Generative AI Moves to Production
AI consultancy Fractal highlights the critical shift from generative AI experimentation to production deployment, where inference efficiency—cost, latency, and scalability—becomes the primary business constraint. This marks a maturation phase where operational metrics trump model novelty.
FaithSteer-BENCH Reveals Systematic Failure Modes in LLM Inference-Time Steering Methods
Researchers introduce FaithSteer-BENCH, a stress-testing benchmark that exposes systematic failures in LLM steering methods under deployment constraints. The benchmark reveals illusory controllability, capability degradation, and brittleness across multiple models and steering approaches.
Ollama Now Supports Apple MLX Backend for Local LLM Inference on macOS
Ollama, the popular framework for running large language models locally, has added support for Apple's MLX framework as a backend. This enables more efficient execution of models like Llama 3.2 and Mistral on Apple Silicon Macs.
NUS CIMERA Chip Cuts LLM Memory Wall with Compute-in-Interconnect
NUS researchers propose CIMERA, an LLM inference accelerator integrating compute-in-interconnect and memory to mitigate the memory wall, detailed in arXiv:2607.13649 (July 2026).
Dflash with Continuous Batch Inference Teased for Draft Models
A developer teased the upcoming release of 'Dflash' with continuous batch inference, targeting current text-only draft models used in speculative execution to speed up LLM inference.
Ollama vs. vLLM vs. llama.cpp
A technical benchmark compares three popular open-source LLM inference servers—Ollama, vLLM, and llama.cpp—under concurrent load. Ollama, despite its ease of use and massive adoption, collapsed at 5 concurrent users, highlighting a critical gap between developer-friendly tools and production-ready systems.
We Hosted a 35B LLM on an NVIDIA DGX Spark — A Technical Post-Mortem
A detailed, practical guide to deploying the Qwen3.5–35B model on NVIDIA's GB10 Blackwell hardware. The article serves as a crucial case study on the real-world challenges and solutions for on-premise LLM inference.
DualPath Architecture Shatters KV-Cache Bottleneck, Doubling LLM Throughput for AI Agents
Researchers have developed DualPath, a novel architecture that eliminates the KV-cache storage bottleneck in agentic LLM inference. By implementing dual-path loading with RDMA transfers, the system achieves nearly 2× throughput improvements for both offline and online scenarios.
llada.cpp Cuts LLaDA-8B Latency 17-42x on Mobile NPU
llada.cpp, the first NPU-aware dLLM inference framework, cuts LLaDA-8B latency 17-42x on smartphones, enabling real-time on-device generation.
Alibaba + Nanjing Univ Claim 9.36X Faster Million-Token Prefill vs FlashAttention-2
Alibaba + Nanjing Univ claim 9.36X faster million-token prefill vs FlashAttention-2, targeting the key bottleneck in long-context LLM inference.
Qwen 3.6 27B Hits 34 tok/s on M5 Max MacBook Pro
Qwen 3.6 27B hits 34 tok/s on M5 Max MacBook Pro with 90% acceptance rate, per @rohanpaul_ai. Shows viable local LLM inference on Apple Silicon.
MLX-LM v0.9.0 Adds Better Batching, Supports Gemma 4 on Apple Silicon
Apple's MLX-LM framework released version 0.9.0 with enhanced server batching and support for Google's Gemma 4 model, improving local LLM inference efficiency on Apple Silicon. This update addresses a key performance bottleneck for developers running models locally on Mac hardware.
Alibaba's CoPaw: The Open-Source Framework Democratizing Complex AI Agent Development
Alibaba has open-sourced CoPaw, a high-performance personal agent workstation designed to help developers build and scale sophisticated multi-channel AI workflows with persistent memory. This framework addresses the growing complexity of moving beyond simple LLM inference to autonomous agentic systems.
LLMs Learn to Switch Reasoning Effort at Inference Time
@rasbt explains how LLMs switch reasoning effort using inference-time methods and training, potentially cutting token usage by 30–50% on simple queries.
Alibaba's MIPI fixes LLM training-inference mismatch with direct RL
Alibaba's MIPI directly optimizes inference policy, fixing the mismatch in LLM post-training via the MIPU framework.
HyEvo Framework Automates Hybrid LLM-Code Workflows, Cuts Inference Cost 19x vs. SOTA
Researchers propose HyEvo, an automated framework that generates agentic workflows combining LLM nodes for reasoning with deterministic code nodes for execution. It reduces inference cost by up to 19x and latency by 16x while outperforming existing methods on reasoning benchmarks.
Meta's Adaptive Ranking Model: A Technical Breakthrough for Efficient LLM-Scale Inference
Meta has developed a novel Adaptive Ranking Model (ARM) architecture designed to drastically reduce the computational cost of serving large-scale ranking models for ads. This represents a core infrastructure breakthrough for deploying LLM-scale models in production at massive scale.
Why Cheaper LLMs Can Cost More: The Hidden Economics of AI Inference in 2026
A Medium article outlines a practical framework for balancing performance, cost, and operational risk in real-world LLM deployment, arguing that focusing solely on model cost can lead to higher total expenses.
TTQ: A New Framework for On-the-Fly Quantization of LLMs at Inference Time
Researchers propose TTQ, a test-time quantization method that compresses large language models dynamically during inference. It uses efficient online calibration to adapt to any prompt, aiming to solve domain-shift issues and accelerate inference without retraining.
LLM Waterfall Pattern: 429 Failover Beats Retries & Circuit Breakers
The LLM waterfall pattern cascades requests across providers on 429 errors, outperforming retries and circuit breakers for zero-downtime AI inference.
7 AI Agent Cost Optimization Strategies That Cut LLM Bills by Up to 90%
The source outlines seven cost optimization strategies for AI agents, including prompt compression and model routing, that can reduce LLM bills by up to 90%. This matters for retail and luxury brands deploying AI at scale where inference costs can become prohibitive.
Hugging Face weekly papers: Monotonic inference policy overtakes training optimization
Hugging Face's top papers July 6-12 include a paper arguing monotonic inference policies are the true LLM RL objective, and Vidu S1 for real-time interactive video generation.
BayesBench: LLMs Match Bayesian Posteriors But Fail Downstream Prediction
BayesBench tests 7 LLMs on multi-turn Bayesian reasoning. Scaling improves latent inference but not prediction, exposing a critical gap for agentic deployment.
Vibe Training: SLM Replaces LLM-as-a-Judge, 8x Faster, 50% Fewer Errors
Plurai introduces 'vibe training,' using adversarial agent swarms to distill a small language model (SLM) for evaluating and guarding production AI agents. The SLM outperforms standard LLM-as-a-judge setups with ~8x faster inference and ~50% fewer evaluation errors.