edge inference
30 articles about edge inference in AI news
Cerebras Challenges Nvidia Inference Monopoly with Wafer-Scale Edge
Cerebras is challenging Nvidia's inference dominance with wafer-scale chips, as inference workloads surpass training in AI compute spend.
Google's TensorFlow 2.21 Revolutionizes Edge AI with Unified LiteRT Framework
Google has launched TensorFlow 2.21, marking LiteRT's transition to a production-ready universal on-device inference framework. This major update delivers faster GPU performance, new NPU acceleration, and seamless PyTorch edge deployment, effectively replacing TensorFlow Lite for mobile and edge applications.
Why Your GPU's Memory Ceiling Predicts Your Cloud Inference Costs
Towards AI explains how GPU VRAM constraints, not compute, dictate LLM inference costs locally and in the cloud. The article details memory math, 2026 hardware shortages, and pricing trends, urging teams to manage context and quantization.
China Blank-Checks Huawei for Inference Chip Push
China is giving Huawei a blank check for inference chips, per a tweet by Natolambert on February 3, 2026, shifting focus from training to inference.
OpenAI, Broadcom Unveil Jalapeño ASIC for LLM Inference
OpenAI and Broadcom unveiled Jalapeño, a custom ASIC for LLM inference, targeting volume deployment by late 2026. No performance metrics were disclosed.
Vultr Picks HPE, Nvidia GB300 for Inference Shift at HPE Discover 2026
Vultr selects HPE and Nvidia GB300 systems for inference, as enterprise demand shifts from training to production workloads.
Pruning LLMs for Edge Triples Bias, Perplexity Hides Damage
Pruning LLMs for edge deployment amplifies bias up to 83.7% while perplexity barely changes, revealing a paradox that undermines standard evaluation practices.
Switchcraft Router Cuts Agentic AI Inference Cost 84%, Matches Top Model
Switchcraft, a DistilBERT-based model router for agentic tool calling, achieves 82.9% accuracy while cutting inference cost by 84%, saving over $3,600 per million queries.
mlx-vlm v0.5.0 Adds Continuous Batching, Distributed Inference for Apple Silicon
mlx-vlm v0.5.0 adds continuous batching, speculative decoding, and distributed inference for Apple Silicon. The release supports Qwen3.5, Kimi K2.5, Gemma 4 video, and new models with 21 contributors.
Google Gemma 4: 3x Faster Inference with MTP Drafters
Google's Gemma 4 claims up to 3x faster inference via MTP drafters, but released no benchmark numbers or architectural details.
DeepSeek-V4 Ported to MLX for Apple Silicon Inference
A developer has ported DeepSeek-V4 to Apple's MLX framework, allowing the large language model to run on Apple Silicon Macs. Early results show functional inference with room for optimization.
PayPal Cuts LLM Inference Cost 50% with EAGLE3 Speculative Decoding on H100
PayPal engineers applied EAGLE3 speculative decoding to their fine-tuned 8B-parameter commerce agent, achieving up to 49% higher throughput and 33% lower latency. This allowed a single H100 GPU to match the performance of two H100s running NVIDIA NIM, cutting inference hardware cost by 50%.
Prefill-as-a-Service Paper Claims to Decouple LLM Inference Bottleneck
A research paper proposes a 'Prefill-as-a-Service' architecture to separate the heavy prefill computation from the lighter decoding phase in LLM inference. This could enable new deployment models where resource-constrained devices handle only the decoding step.
Andrej Karpathy's LLM-Wiki Framework Solves AI Amnesia with Persistent Knowledge
Andrej Karpathy published a two-page framework called LLM-Wiki that transforms how AI systems handle accumulated knowledge. Instead of retrieving from raw documents each time, the AI compiles sources into its own structured wiki that persists across sessions.
Google Launches AI Edge Eloquent: Free, Offline-First Dictation App on iOS
Google has quietly launched AI Edge Eloquent, a free, subscription-less dictation app for iOS. It uses a Gemma-based speech recognition model to process audio locally, removing filler words and self-corrections to produce cleaner text.
Zero-Shot Cross-Domain Knowledge Distillation: A YouTube-to-Music Case Study
Google researchers detail a case study transferring knowledge from YouTube's massive video recommender to a smaller music app, using zero-shot cross-domain distillation to boost ranking models without training a dedicated teacher. This offers a practical blueprint for improving low-traffic AI systems.
Inference Beauty Today Announces Global Platform Expansion, Powering Personalized Beauty Discovery for 100+ Retailers and Brands
Inference Beauty Today has expanded its AI-powered personalized beauty discovery platform globally, now serving over 100 retailers and brands across five markets. This signals the maturation of specialized, third-party AI recommendation engines in the beauty and personal care sector.
mlx-vlm v0.4.2 Adds SAM3, DOTS-MOCR Models and Critical Fixes for Vision-Language Inference on Apple Silicon
mlx-vlm v0.4.2 released with support for Meta's SAM3 segmentation model and DOTS-MOCR document OCR, plus fixes for Qwen3.5, LFM2-VL, and Magistral models. Enables efficient vision-language inference on Apple Silicon via MLX framework.
Federated RAG: A New Architecture for Secure, Multi-Silo Knowledge Retrieval
Researchers propose a secure Federated Retrieval-Augmented Generation (RAG) system using Flower and confidential compute. It enables LLMs to query knowledge across private data silos without centralizing sensitive documents, addressing a major barrier for enterprise AI.
Fractal Emphasizes LLM Inference Efficiency as Generative AI Moves to Production
AI consultancy Fractal highlights the critical shift from generative AI experimentation to production deployment, where inference efficiency—cost, latency, and scalability—becomes the primary business constraint. This marks a maturation phase where operational metrics trump model novelty.
arXiv Survey Maps KV Cache Optimization Landscape: 5 Strategies for Million-Token LLM Inference
A comprehensive arXiv review categorizes five principal KV cache optimization techniques—eviction, compression, hybrid memory, novel attention, and combinations—to address the linear memory scaling bottleneck in long-context LLM inference. The analysis finds no single dominant solution, with optimal strategy depending on context length, hardware, and workload.
TTQ: A New Framework for On-the-Fly Quantization of LLMs at Inference Time
Researchers propose TTQ, a test-time quantization method that compresses large language models dynamically during inference. It uses efficient online calibration to adapt to any prompt, aiming to solve domain-shift issues and accelerate inference without retraining.
ReXInTheWild Benchmark Reveals VLMs Struggle with Medical Photos: Gemini-3 Leads at 78%, MedGemma Trails at 37%
Researchers introduced ReXInTheWild, a benchmark of 955 clinician-verified questions based on 484 real medical photographs. Leading multimodal models show wide performance gaps, with Gemini-3 scoring 78% accuracy while the specialized MedGemma model achieved only 37%.
Edge Computing in Retail 2026: Examples, Benefits, and a Guide
Shopify outlines the strategic shift toward edge computing in retail, detailing its benefits—real-time personalization, inventory management, and enhanced in-store experiences—and providing a practical implementation guide for 2026.
IonRouter Emerges as Cost-Efficient Challenger to OpenAI's Inference Dominance
YC-backed Cumulus Labs launches IonRouter, a high-throughput inference API that promises to slash AI deployment costs by optimizing for Nvidia's Grace Hopper architecture. The service offers OpenAI-compatible endpoints while enabling teams to run open-source or fine-tuned models without cold starts.
Understanding the Interplay between LLMs' Utilisation of Parametric and Contextual Knowledge: A keynote at ECIR 2025
A keynote at ECIR 2025 will present research on how Large Language Models (LLMs) balance their internal, parametric knowledge with external, contextual information. This is critical for deploying reliable AI in knowledge-intensive tasks where models must correctly use provided context, not just their training data.
Reinforcement Learning Ushers in New Era of Autonomous Knowledge Agents
Researchers are developing knowledge agents powered by reinforcement learning that can autonomously gather, process, and apply information. These systems represent a significant evolution beyond traditional language models toward more independent problem-solving capabilities.
EpisTwin: A Neuro-Symbolic Framework for Personal AI Using Knowledge Graphs
Researchers propose EpisTwin, a neuro-symbolic architecture that builds a Personal Knowledge Graph from fragmented user data to enable complex, verifiable reasoning. It addresses limitations of standard RAG by capturing semantic topology and temporal dependencies.
Alibaba's Qwen3-Coder-Next: The 80B Parameter Coding Agent That Only Uses 3B at Inference
Alibaba has unveiled Qwen3-Coder-Next, an 80B parameter coding agent that activates just 3B parameters during inference. It achieves competitive performance on SWE-Bench and Terminal-Bench while supporting a 256K context window.
Edge AI Breakthrough: Qwen3.5 2B Runs Locally on iPhone 17 Pro, Redefining On-Device Intelligence
Alibaba's Qwen3.5 2B model now runs locally on iPhone 17 Pro devices, marking a significant breakthrough in edge AI. This development enables sophisticated language processing without cloud dependency, potentially transforming mobile AI applications and user privacy paradigms.