mixture of experts
30 articles about mixture of experts in AI news
The Hidden Cost of Mixture-of-Experts: New Research Reveals Why MoE Models Struggle at Inference
A groundbreaking paper introduces the 'qs inequality,' revealing how Mixture-of-Experts architectures suffer a 'double penalty' during inference that can make them 4.5x slower than dense models. The research shows training efficiency doesn't translate to inference performance, especially with long contexts.
NVIDIA Nemotron 3 Super: 120B Hybrid Mamba-Transformer MoE with 1M Context
NVIDIA has released Nemotron 3 Super, a 120B parameter open hybrid Mamba-Transformer Mixture of Experts model with 12B active parameters and 1M token context length. The company claims it delivers up to 7.5x higher throughput than similar open models.
Alibaba's Qwen3.5: The Efficiency Breakthrough That Could Democratize Multimodal AI
Alibaba has open-sourced Qwen3.5, a multimodal AI model that combines linear attention with sparse Mixture of Experts architecture to deliver high performance without exorbitant computational costs, potentially making advanced AI more accessible.
Soofi S 30B-A3B: German open model tops English, German benchmarks
German consortium releases Soofi S 30B-A3B, an open MoE model beating OLMo 3 and Apertus 70B on English and German benchmarks while activating only 3.2B of 31.6B parameters.
Grouped Query Experts cuts long-context attention cost 44%
GQE speeds long-context attention prefill 1.7–1.8× by routing tokens to 9 of 16 query heads, matching baseline accuracy at 56.04.
dMoE Cuts Active Experts from 69.5 to 14.6, Retains 99.11% Performance
dMoE reduces active experts from 69.5 to 14.6 in diffusion LLMs, retaining 99.11% performance while cutting memory 80% and speeding inference 1.66×.
Microsoft Unveils MAI-Thinking-1: 35B Active, 1T Parameters, 97% on AIME 2025
Microsoft's MAI-Thinking-1 hits 97% on AIME 2025 with 35B active params in a 1T MoE model, trained on 30T human tokens without distillation.
Nvidia Claims MLPerf Inference v6.0 Records with 288-GPU Blackwell Ultra Systems, Highlights 2.7x Software Gains
MLCommons released MLPerf Inference v6.0 results, introducing multimodal and video model tests. Nvidia set records using 288-GPU Blackwell Ultra systems and achieved a 2.7x performance jump on DeepSeek-R1 via software optimizations alone.
Google's TurboQuant AI Research Report Sparks Sell-Off in Micron, Samsung, and SK Hynix Memory Stocks
Google's TurboQuant research blog publication triggered immediate market reaction, with shares of major memory manufacturers dropping 2-4% as investors anticipate AI-driven efficiency gains reducing future memory demand.
Google's Gemma 4 Emerges: The Next Generation of Open AI Models
Google has announced the upcoming release of Gemma 4, the next iteration of its open-source AI model family. This development signals Google's continued commitment to accessible AI technology and intensified competition in the open model space.
Brain-OF: The First Unified AI Model That Reads Multiple Brain Signals Simultaneously
Researchers have developed Brain-OF, the first omnifunctional foundation model that jointly processes fMRI, EEG, and MEG brain signals. This unified approach overcomes previous single-modality limitations by integrating complementary spatiotemporal data through innovative architecture and pretraining techniques.
Moonshot AI's Kimi K3: 2.8T params, 1M token window, $3/M input
Moonshot AI released Kimi K3, a 2.8T-parameter mixture-of-experts model with 1M token context window and $3/M input pricing, claiming autonomous chip design and research capabilities.
Alibaba Qwen3.6-35B-A3B: 3B-Active Sparse MoE Hits 73.4% on SWE-Bench
Alibaba released Qwen3.6-35B-A3B, a sparse mixture-of-experts model with 35B total but only 3B active parameters. It shows significant gains over its predecessor, scoring 73.4% on SWE-bench Verified and beating Claude 3.5 Sonnet on several vision tasks.
MiniMax M2.7 Tops Open LLM Leaderboard with 230B Parameter Sparse Model
MiniMax announced its M2.7 model has taken the top spot on the Hugging Face Open LLM Leaderboard. The model uses a sparse mixture-of-experts architecture with 230B total parameters but only activates 10B per token.
Cursor AI Claims 1.84x Faster MoE Inference on NVIDIA Blackwell GPUs
Cursor AI announced a rebuilt inference engine for Mixture-of-Experts models on NVIDIA's new Blackwell GPUs, resulting in a claimed 1.84x speedup and improved output accuracy.
Google Releases Gemma 4 Family Under Apache 2.0, Featuring 2B to 31B Models with MoE and Multimodal Capabilities
Google has released the Gemma 4 family of open-weight models, derived from Gemini 3 technology. The four models, ranging from 2B to 31B parameters and including a Mixture-of-Experts variant, are available under a permissive Apache 2.0 license and feature multimodal processing.
Kimi 2.5's 1T Parameter MoE Model Runs on 96GB Mac Hardware via SSD Streaming
Developers have demonstrated that Kimi 2.5's 1 trillion parameter Mixture-of-Experts model can run on Mac hardware with just 96GB RAM by streaming expert weights from SSD, with only 32B parameters active per token.
Step-3.5-Flash: 196B Open-Source MoE Model Activates Only 11B Parameters, Outperforms Kimi K2.5 and Claude Opus 4.5 on Key Benchmarks
Shanghai-based StepFun's Step-3.5-Flash, a 196B parameter sparse mixture-of-experts model that activates only 11B parameters per token, achieves top scores on AIME 2025 (97.3) and LiveCodeBench-V6 (86.4) while costing 18.9x less to run than Kimi K2.5.
NVIDIA Releases Nemotron-Cascade 2: A 30B MoE Model with 3B Active Parameters
NVIDIA has open-sourced Nemotron-Cascade 2, a 30B parameter Mixture-of-Experts model that activates only 3B parameters per token. It claims 'gold medal performance' on IMO and IOI 2025 benchmarks.
Beyond Homogenization: How Expert Divergence Learning Unlocks MoE's True Potential
Researchers have developed Expert Divergence Learning, a novel pre-training strategy that combats expert homogenization in Mixture-of-Experts language models. By encouraging functional specialization through domain-aware routing, the method improves performance across benchmarks with minimal computational overhead.
Video of Massive AI Training Lab in China Sparks Debate on Automation's Scale
A social media post showcasing a vast Chinese AI training lab has reignited discussions about job displacement, underscoring the tangible infrastructure powering the current AI surge.
Ethan Mollick Declares End of 'RAG Era' as Dominant Paradigm for AI Agents
AI researcher Ethan Mollick declared that the 'RAG era' for supplying context to AI agents has ended, marking a significant architectural shift in how advanced AI systems process information.
AI Bridges the Gap Between Data and Discovery: New Framework Aligns Scientific Observations with Decades of Literature
Researchers have developed a novel AI framework that aligns X-ray spectra with scientific literature using contrastive learning. This multimodal approach improves physical variable estimation by 16-18% and identifies high-priority astronomical targets, demonstrating how AI can accelerate scientific discovery by connecting data with domain knowledge.
GigaWorld-Policy-0.5 Hits 85ms on RTX 4090 for Robot Control
GigaWorld-Policy-0.5 runs robot control at 85ms on an RTX 4090, using a Mixture-of-Transformers architecture for real-time local deployment.
AWS Beats Cloud Rivals to NVIDIA Blackwell with EC2 G7 — 4.6x AI Inference Gain Over G6
AWS launched EC2 G7 instances on June 19, 2026, becoming the first major cloud to offer NVIDIA RTX PRO 4500 Blackwell Server Edition GPUs. The instances claim 4.6x AI inference performance over G6, backed by 700 Gbps EFA networking and 32 GB GDDR7 per GPU. The move arrives the same week AWS confirme
Mistral Medium Model Launch Teased by European AI Company
Mistral AI teased an upcoming model called Mistral Medium on X, signaling continued expansion of its model lineup. The announcement comes amid growing competition in the open-weight LLM space.
AI Chip Capacity Crisis: 10GW Left Through 2030, Prices Up Double Digits
The AI accelerator market has only 10 gigawatts of capacity left for contract through 2030, with 100GW already under contract. Prices are rising double digits as one competitor has stopped taking orders entirely.
LoopCTR: A New 'Loop Scaling' Paradigm for Efficient
A new research paper introduces LoopCTR, a method for scaling Transformer-based CTR models by recursively reusing shared layers during training. This 'train-multi-loop, infer-zero-loop' approach achieves state-of-the-art performance with lower deployment costs, directly addressing a core industrial constraint in recommendation systems.
Anthropic CEO Dario Amodei: China Will Match Mythos AI Within a Year
Anthropic CEO Dario Amodei stated China will replicate the capabilities of Anthropic's advanced 'Mythos' AI project within 12 months. He also sees no near-term slowdown in AI progress.
Anthropic's Opus 4.7 Shows Sustained Gains on Economically Critical Tasks
Ethan Mollick highlights that Anthropic's latest Claude Opus 4.7 model shows measurable performance gains on economically important tasks, continuing a rapid two-month release cycle with no signs of plateau.