distillation
30 articles about distillation in AI news
Relay-OPD: On-Policy Distillation Fixes Prefix Failure in LLMs
Relay-OPD introduces on-policy distillation where a teacher briefly takes over student LLM generation to fix prefix failure, reducing error compounding without full teacher compute.
Meta Bans Claude Code, Codex to Block Distillation as Internal Spend Hits
Meta banned Claude Code and Codex internally to block distillation. Internal memo warns of 'serious escalations'; MetaCode build underway amid billions in AI spend.
Visual-SDPO: Self-Distillation Fixes Code-Generated Visual Defects by +10 Points
Visual-SDPO uses visual-feedback self-distillation to improve code-generated visual artifacts by >10 points on ChartMimic, Design2Code, and AeSlides, with no added inference cost.
Tsinghua Researchers Diagnose On-Policy Distillation Failures, Propose Fixes
Researchers from Tsinghua University have pinpointed two necessary conditions for successful on-policy distillation: compatible thinking patterns and novel teacher capabilities. They propose two recovery methods to salvage failing distillation runs.
RLSD Unifies Self-Distillation & Verifiable Rewards to Fix RL Leakage
Researchers propose RLSD, a method merging on-policy self-distillation with verifiable rewards to fix information leakage and training instability in language model reinforcement learning.
DRKL: Diversity-Aware Reverse KL Divergence Fixes Overconfidence in LLM Distillation
A new paper proposes Diversity-aware Reverse KL (DRKL), a fix for the overconfidence and reduced diversity caused by the popular Reverse KL divergence in LLM distillation. DRKL consistently outperforms existing objectives across multiple benchmarks.
Zero-Shot Cross-Domain Knowledge Distillation: A YouTube-to-Music Case Study
Google researchers detail a case study transferring knowledge from YouTube's massive video recommender to a smaller music app, using zero-shot cross-domain distillation to boost ranking models without training a dedicated teacher. This offers a practical blueprint for improving low-traffic AI systems.
Apple Reportedly Gains Full Internal Access to Google's Gemini for On-Device Model Distillation
A report claims Apple's AI deal with Google includes full internal model access, enabling distillation of Gemini's reasoning into smaller, on-device models. This would allow Apple to build specialized, efficient AI without relying solely on cloud APIs.
Aligning Language Models from User Interactions: A Self-Distillation Method for Continuous Learning
Researchers propose a method to align LLMs using raw, multi-turn user conversations. By applying self-distillation on follow-up messages, models improve without explicit feedback, enabling personalization and continual adaptation from deployment data.
Structured Distillation for Personalized Agent Memory: 11x Compression with Minimal Recall Loss
New research introduces structured distillation to compress AI agent conversation history by 11x (371→38 tokens/exchange) while preserving 96% retrieval effectiveness. This enables storing thousands of exchanges in a single prompt while maintaining verbatim source access.
Anthropic's Distillation Allegations Reveal AI's Uncharted Legal Frontier
Anthropic's claims that Chinese AI firms used thousands of fake accounts to extract capabilities from Claude models highlight the legal grey area of model distillation. The incident coincides with Anthropic relaxing its safety policies amid Pentagon pressure.
OpenAI, Anthropic, Google Form Alliance to Block Chinese Model Distillation
OpenAI, Anthropic, and Google are collaborating through the Frontier Model Forum to share intelligence and prevent Chinese firms from distilling their advanced AI models. This formalizes defensive measures in the US-China AI race.
Nvidia, Meta, Mistral Warn US Against Broad Open-Weight AI Restrictions
Nvidia, Meta, Mistral sign letter urging US against broad open-weight AI restrictions, arguing distillation is legitimate. Comes as White House weighs response to Chinese AI model distillation.
Claude Code Steganography Flagged Chinese Users; Anthropic Rolls Back
Anthropic's Claude Code 2.1.91 used steganography to detect Chinese users. After Reddit exposure, Anthropic rolled back the feature, calling it an experiment against model distillation.
Microsoft Unveils MAI-Thinking-1: 35B Active, 1T Parameters, 97% on AIME 2025
Microsoft's MAI-Thinking-1 hits 97% on AIME 2025 with 35B active params in a 1T MoE model, trained on 30T human tokens without distillation.
Distilled Agentic Workflow Runs at 100x Lower Inference Cost
A new paper shows agentic workflow distillation achieving 100x lower inference cost, but lacks benchmark details.
SDAR: Self-Distilled RL Stabilizes Multi-Turn LLM Agents, +9.4% on ALFWorld
SDAR gates self-distillation within GRPO to stabilize multi-turn LLM agent training, yielding +9.4% on ALFWorld and gains on WebShop and Search-QA across Qwen2.5 and Qwen3 models.
Subliminal Transfer Study Shows AI Agents Inherit Unsafe Behaviors Despite
New research demonstrates unsafe behavioral traits in AI agents can transfer subliminally through model distillation, with students inheriting deletion biases despite rigorous keyword filtering. This exposes a critical security flaw in agent training pipelines.
Embedding Matching Distills Genomic Models 200x, Matches mRNA-Bench Performance
A new distillation framework transfers mRNA representations from a large genomic foundation model to a specialized model 200x smaller. It uses embedding-level distillation, outperforming logit-based methods and competing with larger models on mRNA-bench.
Kuaishou's Dual-Rerank: A New Industrial Framework for High-Stakes
Researchers from Kuaishou introduce Dual-Rerank, a framework designed for industrial-scale generative reranking. It addresses the dual dilemma of structural trade-offs (AR vs. NAR models) and optimization gaps (SL vs. RL) through Sequential Knowledge Distillation and List-wise Decoupled Reranking Optimization. A/B tests on production traffic show significant improvements in user satisfaction and watch time with reduced latency.
DIET: A New Framework for Continually Distilling Streaming Datasets in Recommender Systems
Researchers propose DIET, a framework for streaming dataset distillation in recommender systems. It maintains a compact, evolving dataset (1-2% of original size) that preserves training-critical signals, reducing model iteration costs by up to 60x while maintaining performance trends.
PSAD: A New Framework for Efficient Personalized Reranking in Recommender Systems
Researchers propose PSAD, a novel reranking framework using semi-autoregressive generation and online knowledge distillation to balance ranking quality with low-latency inference. It addresses key deployment challenges for generative reranking models in production systems.
SymTorch Bridges the Gap Between Black Box AI and Human Understanding
Researchers introduce SymTorch, a framework that automatically converts neural network components into interpretable mathematical equations. This symbolic distillation approach could make AI systems more transparent while potentially accelerating inference, with early tests showing 8.3% throughput improvements in language models.
The AI Espionage Era: How Chinese Firms Launched Industrial-Scale Attacks on Claude
Anthropic reveals three massive AI model distillation campaigns by Chinese competitors who used 24,000 fake accounts to extract Claude's capabilities through 16 million exchanges. This industrial-scale intellectual property theft highlights growing tensions in the global AI race.
The AI Espionage Frontier: Anthropic Exposes Systematic Claude Data Extraction by Chinese AI Labs
Anthropic has revealed that Chinese AI companies DeepSeek, Moonshot, and MiniMax allegedly used 24,000 fake accounts to execute 16 million queries against Claude's API, systematically extracting its capabilities through model distillation techniques. This sophisticated operation bypassed access restrictions and targeted Claude's reasoning, programming, and tool usage functions.
Anthropic Exposes Massive AI Model Theft Operation Targeting Claude
Anthropic has uncovered sophisticated 'distillation' campaigns by Chinese AI firms DeepSeek, Moonshot, and MiniMax, who allegedly used thousands of fraudulent accounts to copy Claude's capabilities. The operation generated over 16 million exchanges to replicate Claude's reasoning and coding strengths.
Moonshot AI Releases 1.56T-Parameter Kimi K3, Requires 2x B200 Nodes
Moonshot AI released Kimi K3, a 1.56T parameter MoE model at 1561 GB, requiring 2x B200 nodes. No benchmarks disclosed.
CAS ZhiJing Beats GPT-5.5 on Social Cognition with FLARE Training
CAS ICT releases ZhiJing social intelligence system, Zing model beats GPT-5.5 on social cognition via FLARE training.
AlayaRenderer-Flash Hits 31.54 FPS, Enables Playable Generative Worlds
AlayaRenderer-Flash accelerates real-time scene synthesis from 0.56 to 31.54 FPS, a 56x speedup enabling playable generative worlds.
Trump Weighs Restrictions on US Firms Using Chinese AI Models
Trump admin weighs restrictions on US firms using Chinese AI models, per Axios. Businesses already adopting cheaper Chinese alternatives, creating policy tension.