inference infrastructure
30 articles about inference infrastructure in AI news
Meta Expands Broadcom Partnership for Next-Gen AI Infrastructure
Meta is expanding its partnership with semiconductor giant Broadcom to co-develop its next-generation AI infrastructure. This move signals a continued, long-term commitment to custom silicon for AI training and inference.
OpenAI Cuts Inference Costs by Half on Some Models
OpenAI cut inference costs by 50%+ on some models for logged-out ChatGPT users, per The Information. The move reduces operational expenses.
Upscale AI Raises $190M for AI Networking Infrastructure
Upscale AI raised $190M to expand AI networking infrastructure, addressing the bottleneck of 100K+ GPU clusters.
AWS Beats Cloud Rivals to NVIDIA Blackwell with EC2 G7 — 4.6x AI Inference Gain Over G6
AWS launched EC2 G7 instances on June 19, 2026, becoming the first major cloud to offer NVIDIA RTX PRO 4500 Blackwell Server Edition GPUs. The instances claim 4.6x AI inference performance over G6, backed by 700 Gbps EFA networking and 32 GB GDDR7 per GPU. The move arrives the same week AWS confirme
Vultr Picks HPE, Nvidia GB300 for Inference Shift at HPE Discover 2026
Vultr selects HPE and Nvidia GB300 systems for inference, as enterprise demand shifts from training to production workloads.
Cerebras Challenges Nvidia Inference Monopoly with Wafer-Scale Edge
Cerebras is challenging Nvidia's inference dominance with wafer-scale chips, as inference workloads surpass training in AI compute spend.
Perplexity Claims 3x Blackwell Inference Throughput for 70B Models
Perplexity AI claims 3x inference throughput for 70B models on Nvidia Blackwell GPUs via FP4 and custom scheduling. The gain exceeds Nvidia's own 2x marketing claim.
mlx-vlm v0.5.0 Adds Continuous Batching, Distributed Inference for Apple Silicon
mlx-vlm v0.5.0 adds continuous batching, speculative decoding, and distributed inference for Apple Silicon. The release supports Qwen3.5, Kimi K2.5, Gemma 4 video, and new models with 21 contributors.
GUC, Wiwynn Partner on Silicon-to-System AI Infrastructure for Hyperscalers
GUC and Wiwynn partner on silicon-to-system AI infrastructure, integrating SoC design, optical I/O, and liquid cooling for hyperscalers.
OpenAI Claims 10GW AI Infrastructure Capacity Ahead of 2029 Target
OpenAI claims 10GW AI infrastructure capacity secured, adding 3GW in 90 days, ahead of 2029 target.
AI Inference Costs Drop 5-10x Yearly: @kimmonismus Challenges Forbes
@kimmonismus claims AI inference costs drop 5-10x yearly, challenging Forbes' static compute cost narrative. This deflation rate implies rapid TCO reduction for enterprise deployments.
PayPal Cuts LLM Inference Cost 50% with EAGLE3 Speculative Decoding on H100
PayPal engineers applied EAGLE3 speculative decoding to their fine-tuned 8B-parameter commerce agent, achieving up to 49% higher throughput and 33% lower latency. This allowed a single H100 GPU to match the performance of two H100s running NVIDIA NIM, cutting inference hardware cost by 50%.
Sam Altman: AI inference costs dropped 1000x from o1 to GPT-5.4
Sam Altman stated AI inference costs for solving a fixed hard problem dropped ~1000x from o1 to GPT-5.4 in ~16 months, crediting cross-layer engineering optimizations, not a single breakthrough.
Bull Delivers HPC Infrastructure to Power Mimer AI Factory
Bull, a subsidiary of Atos, has supplied the core HPC infrastructure for Mimer's new AI factory. This facility is dedicated to training and developing large language models for the European market.
CoreWeave & Google Raise $6.7B in Junk Bonds for AI Infrastructure
Google and GPU cloud provider CoreWeave have jointly raised $6.7 billion through a junk bond offering, with Google taking $5.7 billion. The capital is earmarked for a significant build-out of AI data center infrastructure.
Prefill-as-a-Service Paper Claims to Decouple LLM Inference Bottleneck
A research paper proposes a 'Prefill-as-a-Service' architecture to separate the heavy prefill computation from the lighter decoding phase in LLM inference. This could enable new deployment models where resource-constrained devices handle only the decoding step.
Meta to Cut 8,000 Jobs in May, Redirecting Capital to AI Infrastructure
Meta is reportedly planning to lay off 8,000 employees in May, the first round of major cuts this year. The move signals a capital shift from general operations to concentrated investment in AI infrastructure like chips and data centers.
DOE Seeks Input on AI Infrastructure for Federal Lands
The U.S. Department of Energy has published a Request for Information (RFI) to solicit input on developing AI and high-performance computing infrastructure on DOE-owned lands. This marks a significant step in the federal government's strategy to directly address the national AI compute shortage.
Nvidia: Cost Per Token Is the Only AI Infrastructure Metric That Matters
Nvidia asserts that total cost of ownership for AI infrastructure must be measured in cost per delivered token, not raw compute metrics. This shift is critical for scaling profitable agentic AI applications.
Intel & Google Announce Multiyear AI & Cloud Infrastructure Partnership
Intel and Google have announced a multiyear strategic collaboration to advance AI and cloud infrastructure, focusing on optimizing Google Cloud for Intel's Xeon processors, Gaudi AI accelerators, and future chips.
Cursor AI Claims 1.84x Faster MoE Inference on NVIDIA Blackwell GPUs
Cursor AI announced a rebuilt inference engine for Mixture-of-Experts models on NVIDIA's new Blackwell GPUs, resulting in a claimed 1.84x speedup and improved output accuracy.
McKinsey: AI Infrastructure Value Creation Outpaces Business Capture
McKinsey's latest analysis indicates the pace of value creation from AI infrastructure is exceeding the rate at which most businesses are capturing it, highlighting a growing implementation deficit.
Nvidia Claims MLPerf Inference v6.0 Records with 288-GPU Blackwell Ultra Systems, Highlights 2.7x Software Gains
MLCommons released MLPerf Inference v6.0 results, introducing multimodal and video model tests. Nvidia set records using 288-GPU Blackwell Ultra systems and achieved a 2.7x performance jump on DeepSeek-R1 via software optimizations alone.
Google's AI Infrastructure Strategy: What Retail Leaders Should Watch in 2026
Google's evolving AI infrastructure and compute strategy, including data center investments and model compression techniques, will directly impact how retail brands deploy and scale AI applications by 2026. The company's focus on efficiency and real-time capabilities signals a shift toward more accessible, powerful retail AI tools.
Inference Beauty Today Announces Global Platform Expansion, Powering Personalized Beauty Discovery for 100+ Retailers and Brands
Inference Beauty Today has expanded its AI-powered personalized beauty discovery platform globally, now serving over 100 retailers and brands across five markets. This signals the maturation of specialized, third-party AI recommendation engines in the beauty and personal care sector.
Oracle Cuts 20% of Workforce to Fund AI Infrastructure Push, Shifting from Labor to Compute
Oracle is laying off 20% of its workforce to redirect capital toward massive AI infrastructure investments. The move signals a strategic pivot from traditional workforce costs to data center and compute spending.
Meta's Adaptive Ranking Model: A Technical Breakthrough for Efficient LLM-Scale Inference
Meta has developed a novel Adaptive Ranking Model (ARM) architecture designed to drastically reduce the computational cost of serving large-scale ranking models for ads. This represents a core infrastructure breakthrough for deploying LLM-scale models in production at massive scale.
Data Center Construction Boom Drives Electrician Salaries to $260k, Fueled by AI Infrastructure Demand
Mike Rowe reports data center electricians earning $260,000/year without degrees as 25.3 GW of capacity is under construction in the Americas, with 89% pre-committed. The AI infrastructure buildout is creating a high-wage, skilled trades bottleneck.
Fractal Emphasizes LLM Inference Efficiency as Generative AI Moves to Production
AI consultancy Fractal highlights the critical shift from generative AI experimentation to production deployment, where inference efficiency—cost, latency, and scalability—becomes the primary business constraint. This marks a maturation phase where operational metrics trump model novelty.
arXiv Survey Maps KV Cache Optimization Landscape: 5 Strategies for Million-Token LLM Inference
A comprehensive arXiv review categorizes five principal KV cache optimization techniques—eviction, compression, hybrid memory, novel attention, and combinations—to address the linear memory scaling bottleneck in long-context LLM inference. The analysis finds no single dominant solution, with optimal strategy depending on context length, hardware, and workload.