quantization
30 articles about quantization in AI news
KV Cache Quantization Silently Breaks Safety Alignment, Paper Shows
KV cache quantization silently breaks LLM safety alignment, with Mistral-7B losing 15.2% refusals at 1.03x perplexity. PCR diagnostic recovers up to 97% alignment in 35 GPU-minutes.
Product Quantization: The Hidden Engine Behind Scalable Vector Search
The article explains Product Quantization (PQ), a method for compressing high-dimensional vectors to enable fast and memory-efficient similarity search. This is a foundational technology for scalable AI applications like semantic search and recommendation engines.
TTQ: A New Framework for On-the-Fly Quantization of LLMs at Inference Time
Researchers propose TTQ, a test-time quantization method that compresses large language models dynamically during inference. It uses efficient online calibration to adapt to any prompt, aiming to solve domain-shift issues and accelerate inference without retraining.
Efficient Fine-Tuning of Vision-Language Models with LoRA & Quantization
A technical guide details methods for fine-tuning large VLMs like GPT-4V and LLaVA using Low-Rank Adaptation (LoRA) and quantization. This reduces computational cost and memory footprint, making custom VLM training more accessible.
The Quantization Paradox: How Compressing Multimodal AI Impacts Reliability
New research reveals that compressing multimodal AI models through quantization significantly reduces their reliability, making them more likely to produce confidently wrong answers. The study identifies methods to mitigate these effects while maintaining efficiency gains.
Paper Details Full-Stack MFM Acceleration: Quant, Spec Decode, HW Co-Design
A research paper details a full-stack approach for accelerating multimodal foundation models, combining hierarchy-aware mixed-precision quantization, structural pruning, speculative decoding, model cascading, and a specialized hardware accelerator. Demonstrated on medical and code generation tasks.
Fine-Tuning an LLM on a 4GB GPU: A Practical Guide for Resource-Constrained Engineers
A Medium article provides a practical, constraint-driven guide for fine-tuning LLMs on a 4GB GPU, covering model selection, quantization, and parameter-efficient methods. This makes bespoke AI model development more accessible without high-end cloud infrastructure.
TurboQuant Ported to Apple MLX, Claims 75% Memory Reduction with Minimal Performance Loss
Developer Prince Canuma has successfully ported the TurboQuant quantization method to Apple's MLX framework, reporting a 75% reduction in memory usage with nearly no performance degradation for on-device AI models.
Google's TurboQuant Cuts LLM KV Cache Memory by 6x, Enables 3-Bit Storage Without Accuracy Loss
Google released TurboQuant, a novel two-stage quantization algorithm that compresses the KV cache in long-context LLMs. It reduces memory by 6x, achieves 3-bit storage with no accuracy drop, and speeds up attention scoring by up to 8x on H100 GPUs.
Flash-KMeans Achieves 200x Speedup Over FAISS by Targeting GPU Memory Bottlenecks
Flash-KMeans is an IO-aware GPU implementation of exact k-means that runs 30x faster than cuML and 200x faster than FAISS. At million-scale datasets, it completes iterations in milliseconds, enabling dynamic re-indexing and real-time quantization.
Quantized Inference Breakthrough for Next-Gen Recommender Systems: OneRec-V2 Achieves 49% Latency Reduction with FP8
New research shows FP8 quantization can dramatically speed up modern generative recommender systems like OneRec-V2, achieving 49% lower latency and 92% higher throughput with no quality loss. This breakthrough bridges the gap between LLM optimization techniques and industrial recommendation workloads.
LeCun's Team Uncovers Hidden Transformer Flaws: How Architectural Artifacts Sabotage AI Efficiency
NYU researchers led by Yann LeCun reveal that Transformer language models contain systematic artifacts—massive activations and attention sinks—that degrade efficiency. These phenomena, stemming from architectural choices rather than fundamental properties, directly impact quantization, pruning, and memory management.
LittleBit-2: How Geometric Alignment Unlocks Ultra-Efficient AI Below 1-Bit
Researchers have developed LittleBit-2, a framework that achieves state-of-the-art performance in sub-1-bit LLM compression by solving latent geometry misalignment. The method uses internal latent rotation and joint iterative quantization to align model parameters with binary representations without inference overhead.
AutoQRA: The Breakthrough That Makes AI Fine-Tuning 4x More Efficient
Researchers have developed AutoQRA, a novel framework that jointly optimizes quantization precision and LoRA adapters for large language models. This breakthrough enables near-full-precision performance with dramatically reduced memory requirements, potentially revolutionizing how organizations fine-tune AI models on limited hardware.
Anthropic Builds In-House Chip Team for Claude Co-Design
Anthropic is building an in-house chip team to co-design hardware with Claude, aiming for faster, cheaper inference. The company maintains a multi-chip strategy with AWS, Google, Nvidia, and AMD.
1-Bit Kimi K3 Quant Cuts 2.8T Model to 590GB, Keeps 78.7% Quality
Atomic Labs' 1-bit quant shrinks 2.8T Kimi K3 to 590GB (-62%), keeping 78.7% quality and 1M context. Runs on 4x B200s.
M4 Max Mac Studio Tops GB10 in Local AI Decode Throughput
M4 Max Mac Studio beats GB10 and Strix Halo in local AI decode throughput but memory bandwidth caps large model performance. Tom's Hardware tested llama.cpp across three platforms.
Moonshot AI Releases 1.56T-Parameter Kimi K3, Requires 2x B200 Nodes
Moonshot AI released Kimi K3, a 1.56T parameter MoE model at 1561 GB, requiring 2x B200 nodes. No benchmarks disclosed.
Alibaba Qwen3.8: 2.4T Parameter Open-Weight Model Incoming
Alibaba's Qwen3.8, a 2.4T parameter open-weight model, was announced. It would be the largest open-weight model ever, but lacks benchmark details.
Moonshot AI's Kimi K3: 2.8T params, 1M token window, $3/M input
Moonshot AI released Kimi K3, a 2.8T-parameter mixture-of-experts model with 1M token context window and $3/M input pricing, claiming autonomous chip design and research capabilities.
SambaNova Hits 850 t/s on MiniMax M2.7 via Hybrid H200-RDU Inferencing
SambaNovaAI achieved 850 t/s on MiniMax M2.7 by pairing H200 GPUs for prefill with SN50 RDUs for decode at RAISE Paris.
Colibri Runs 744B-Parameter Model on 25GB RAM, No GPU
Colibri claims to run a 744B-parameter model on 25GB RAM without GPU, but lacks evidence. If true, it could democratize large-model inference.
NVIDIA Drops 30B Nemotron Audex Audio Model with MoE
NVIDIA released Nemotron Audex 30B-A3B, a 30B-parameter MoE audio model unifying ASR, understanding, and TTS with 3B active parameters.
NVIDIA Releases FP4 Quantized Kimi-K2.7-Code with 1T Parameters
NVIDIA released FP4 quantized Kimi-K2.7-Code on Hugging Face, a 1T-parameter model for Blackwell GPUs with claimed accuracy retention.
PrismML Shrinks Qwen 3.6 to iPhone 17 Pro, Apple Eyes Deal
PrismML compressed Alibaba's 36B-parameter Qwen 3.6 to run on an iPhone 17 Pro, drawing Apple's interest for on-device AI without cloud latency.
OpenAI Cuts Inference Costs by Half on Some Models
OpenAI cut inference costs by 50%+ on some models for logged-out ChatGPT users, per The Information. The move reduces operational expenses.
NVIDIA Blackwell Cuts DeepSeek V4 Token Costs 5x in One Month
NVIDIA claims Blackwell inference stack cut DeepSeek V4 token costs 5x in one month, per a newly published report shared by @rohanpaul_ai.
JetSpec hits 1,000 t/s on Qwen-8B with speculative decoding
JetSpec achieves 1,000 t/s on Qwen-8B with a B200 GPU, claiming superiority over prior speculative decoding methods, but lacks independent verification.
Qualcomm Launches AI Data Center Program With Hyperscaler Customer
Qualcomm launched an AI data center program with a major hyperscaler customer, targeting inference workloads. Financial terms and partner identity undisclosed.
Pareto LoRA Boosts Image Quality 44.9% vs Vanilla LoRA on Emu2
Pareto LoRA reformulates multimodal instruction tuning as bi-objective optimization, achieving up to 44.9% image quality gains on Emu2 while maintaining text performance.