gpu memory
30 articles about gpu memory in AI news
Google's TurboQuant Compresses LLM KV Cache 6x with Zero Accuracy Loss, Cutting GPU Memory by 80%
Google researchers introduced TurboQuant, a method that compresses LLM KV cache from 32-bit to 3-bit precision without accuracy degradation. This reduces GPU memory consumption by over 80% and speeds up inference 8x on H100 GPUs.
How a GPU Memory Leak Nearly Cost an AI Team a Major Client During a Live Demo
A detailed post-mortem of a critical AI inference failure during a client demo reveals how silent GPU memory leaks, inadequate health checks, and missing circuit breakers can bring down a production pipeline. The author shares the architectural fixes implemented to prevent recurrence.
Flash-KMeans Achieves 200x Speedup Over FAISS by Targeting GPU Memory Bottlenecks
Flash-KMeans is an IO-aware GPU implementation of exact k-means that runs 30x faster than cuML and 200x faster than FAISS. At million-scale datasets, it completes iterations in milliseconds, enabling dynamic re-indexing and real-time quantization.
DeepSeek-V4 Hits 500K Context with 90% Less KV Cache via FlashMemory
DeepSeek-V4 achieves 500K context with 90% less KV cache via FlashMemory's lookahead sparse attention, keeping only 13.5% of cache in GPU memory without retraining.
Nvidia Vera Rubin Shifts AI Strategy Beyond Raw GPU Speed
Nvidia's Vera Rubin architecture pivots from raw GPU FLOPS to system-level AI infrastructure, targeting memory bandwidth and interconnect bottlenecks that constrain large-scale model training.
Supermicro's 7,000-lb Double-Wide Rack Targets B300 Memory Squeeze
Supermicro's 7,000-pound double-wide rack targets B300 memory squeeze with denser GPU packing and liquid cooling, announced at Computex 2026.
Apple M7 Ultra Chip Reportedly Supports 1.5TB Unified Memory
Apple's M7 Ultra chip reportedly supports 1.5TB unified memory, doubling the M3 Ultra and matching eight Nvidia B200 GPUs, but DRAM supply constraints threaten pricing.
Google's TurboQuant Cuts LLM KV Cache Memory by 6x, Enables 3-Bit Storage Without Accuracy Loss
Google released TurboQuant, a novel two-stage quantization algorithm that compresses the KV cache in long-context LLMs. It reduces memory by 6x, achieves 3-bit storage with no accuracy drop, and speeds up attention scoring by up to 8x on H100 GPUs.
98× Faster LLM Routing Without a Dedicated GPU: Technical Breakthrough for vLLM Semantic Router
New research presents a three-stage optimization pipeline for the vLLM Semantic Router, achieving 98× speedup and enabling long-context classification on shared GPUs. This solves critical memory and latency bottlenecks for system-level LLM routing.
Flash-KMeans Revolutionizes GPU Clustering with 200x Speedup Over FAISS
New Flash-KMeans algorithm achieves dramatic speed improvements in GPU-based clustering through innovative IO-aware FlashAssign kernels that eliminate memory bottlenecks and atomic contention, potentially transforming large-scale data analysis.
OpenAI Loses GPT-3 GPU Builder Scott Gray; 13 Leaders Out in 2026
Scott Gray, OpenAI's founding GPU engineer, left in 2026—the 13th senior departure. The exodus spans every function, signaling systemic churn beyond the C-suite.
SemiAnalysis: Can TileRT Software Match Cerebras on NVIDIA GPUs?
SemiAnalysis is testing TileRT InferenceX, software claiming batch-1 ultra-high interactivity on NVIDIA GPUs, targeting Cerebras, Groq LPU, and SambaNova. No benchmarks disclosed yet.
China's DFSX SuperNode Doubles GB200 Memory Bandwidth on 14nm
China's DFSX SuperNode claims 2x GB200 memory bandwidth using 14nm vertical towers, but lacks benchmarks or ship dates, warranting skepticism.
Colibri Runs 744B-Parameter Model on 25GB RAM, No GPU
Colibri claims to run a 744B-parameter model on 25GB RAM without GPU, but lacks evidence. If true, it could democratize large-model inference.
Nvidia Vera Rubin Rack Costs $7.8M; Memory Drives Price
Nvidia's Vera Rubin rack costs $7.8M with memory as key cost driver; Kyber rack delayed to 2028.
NHN Cloud Tops Korean TOP500 with FactoryX GPU Clusters
NHN Cloud tops Korean TOP500 with FactoryX GPU clusters delivering 1.2 exaflops, marking first domestic cloud provider to lead the list.
mlx-vlm v0.6.2 Adds Gemma 4 QAT Support for Local GPUs
mlx-vlm v0.6.2 adds launch-day support for Google DeepMind's Gemma 4 QAT checkpoints, enabling local inference on consumer GPUs and edge devices with video input for the 12B model.
Cerebras Hits 981 Tokens/sec on 1T-Parameter Kimi K2.6, Claims 6.7× GPU Cloud Speedup
Cerebras reported 981 tokens/sec on the 1T-parameter Kimi K2.6 model, a 6.7× speedup over the next GPU cloud, validated by an independent third party.
vLLM Optimizations Cut Voice AI Latency by 40% on 6-GPU Cluster
vLLM optimizations on a 6-GPU cluster reduced voice AI latency by 40% for a Qwen-based system, enabling 500 concurrent sessions per node without hardware upgrades.
MLX CUDA Backend Passes All Tests, Closing Apple GPU Gap
MLX CUDA backend passes all tests, enabling NVIDIA GPU support. Milestone bridges Apple Silicon and CUDA ecosystems for ML workloads.
Roundhill Memory ETF (DRAM) Surges 90% in 36 Days, Fastest ETF Ever
Roundhill Memory ETF surged 90% since April 2, hitting $6.5B assets in 36 days—fastest ETF ever—driven by AI demand for DRAM.
OpenAI's MRC Protocol Sprays Packets Across 100+ Paths to Fix GPU Stragglers
OpenAI open-sourced MRC, a networking protocol that sprays packets across hundreds of paths to reduce GPU idle time from congestion and failures, contributed to OCP.
RoundPipe: Full Fine-Tune 32B Models on a Single 24GB GPU
RoundPipe fine-tunes 32B models on a single 24GB GPU with 1.5-2.2× speedups via round-robin pipeline dispatch.
Open-Weight 1T Model Inference Margins Hit 88% on Rented GPUs
Renting a 128 GPU cluster to serve a 1T open model yields ~88% margin on tokens sold at $0.002/1K, exposing a structural arbitrage over proprietary APIs.
SemiAnalysis: NVIDIA's Customer Data Drives Disaggregated Inference, LPU Surpasses GPU
SemiAnalysis states NVIDIA's direct customer feedback is leading the industry toward disaggregated inference architectures. In this model, specialized LPUs can outperform GPUs for specific pipeline tasks.
Cisco Reveals Scale-Across GPU Networking Needs 14x DCI Bandwidth
Cisco's chief architect detailed the massive bandwidth requirements for connecting AI clusters via 'scale-across' GPU networking, which needs 14x the capacity of traditional data center interconnects. This shift is creating a multi-billion dollar market for 800G coherent pluggables and deep-buffered switches.
Gur Singh Claims 7 M4 MacBooks Match A100, Calls Cloud GPU Training a 'Scam'
Developer Gur Singh posted that seven M4 MacBooks (2.9 TFLOPS each) match an NVIDIA A100's performance, calling cloud GPU training a 'scam' and advocating for distributed, consumer-hardware approaches.
Claude MCP GPU Debugging: AI Agent Identifies PyTorch Bottleneck in Kernel
A developer used an AI agent powered by Claude Code and the Model Context Protocol (MCP) to diagnose a severe GPU performance bottleneck. The agent analyzed system kernel traces, pinpointing excessive CPU context switches as the culprit, demonstrating a practical application of agentic AI for complex technical debugging.
Nvidia to Ship 1.19 Exabytes of HBM in 2026, Apple iPhone Memory 2x Larger
An analysis projects Nvidia will ship ~1.19 exabytes of HBM memory in 2026 for AI infrastructure, while Apple will ship ~2.4 exabytes of LPDDR5 for iPhones, putting AI's massive hardware scale in consumer market perspective.
A Practical Guide to Fine-Tuning an LLM on RunPod H100 GPUs with QLoRA
The source is a technical tutorial on using QLoRA for parameter-efficient fine-tuning of an LLM, leveraging RunPod's cloud H100 GPUs. It focuses on the practical setup and execution steps for engineers.