benchmark
30 articles about benchmark in AI news
Soofi S 30B-A3B: German open model tops English, German benchmarks
German consortium releases Soofi S 30B-A3B, an open MoE model beating OLMo 3 and Apertus 70B on English and German benchmarks while activating only 3.2B of 31.6B parameters.
InternVLA-A1.5 Unifies Vision, Foresight, Action — SOTA on All Six Sim Benchmarks
InternVLA-A1.5 unifies vision-language understanding, latent foresight, and action into one robot policy, achieving SOTA on all six simulation benchmarks.
Claude Code Tops JetBrains' New Kotlin Benchmark with 85.7% Resolution
Claude Code with Opus 4.7 xhigh tops JetBrains' Kotlin Benchmark at 85.7%. Configure your CLAUDE.md with Kotlin conventions and use `--model opus-4.7-xhigh` to match this performance.
LLMForge: 7 Models Score 0.89 on CAD Benchmark; VLMs Fix Cylinders
LLMForge scores 0.89 on 97-design CAD benchmark. VLM critic achieves 100% watertight meshes but cylinders remain a failure mode.
Ant Group's 1.1B LingBot-Vision Beats Meta's 7B DINOv3 on 12 Benchmarks
Ant Group's 1.1B LingBot-Vision tops Meta's 7B DINOv3 on 12 spatial benchmarks, with 40% fewer FLOPs.
DARPA AIQ Program Shifts From Benchmarks to Measuring AI Capabilities
DARPA AIQ program, one year in, shifts from benchmarks to a science of AI capability, per program lead @patrickshafto.
Zhipu GLM-5.2 beats Anthropic's Mythos on bug-hunt benchmark
Zhipu AI's GLM-5.2 beat Anthropic's Claude Opus 4.8 on a cybersecurity bug-hunting benchmark, then matched it with extra instructions, marking another 'DeepSeek moment'.
GPT-5.6 Sol, Terra, Luna: Benchmark Performance Depends on Which Test You Use
OpenAI released GPT-5.6 as three tiers—Sol, Terra, Luna—on June 27, 2026. Sol tops Terminal-Bench 2.1 but trails competitors on other benchmarks. The release shifts focus to tiered pricing and efficiency, but access remains restricted.
SciCode: Epoch AI Launches Benchmark Measuring AI Research Ability
Epoch AI launched SciCode benchmark testing LLMs on real research coding tasks. Top models score below 30%, exposing gap between coding benchmarks and scientific ability.
Epoch AI's CursorBench Benchmarks AI Code Editing at Scale
Epoch AI launched CursorBench, a 500-task benchmark for AI code editors. It reveals a 15% accuracy gap vs. humans and 3x latency variance.
MirrorCode Benchmark Costs $2,600 Per Run, Challenges AI Coding Limits
Epoch AI and METR launched MirrorCode, a $2,600-per-run coding benchmark. Claude Opus 4.7 leads with 56% solve rate.
Zhipu GLM-5.2 tops global coding benchmarks, sparks 'DeepSeek moment'
Zhipu AI's GLM-5.2 ranks top-3 globally on a coding benchmark, with US engineers calling it a daily driver superior to GPT-5.5.
OpenAI GPT-5.5-Cyber Beats Anthropic Mythos on Security Benchmarks
OpenAI's GPT-5.5-Cyber beats Anthropic's Mythos on security benchmarks. Updated Codex plugin auto-patches after scanning 30M commits.
OpenAI shows small doses of beneficial-trait RL improve 44 of 53 safety benchmarks — and the gains generalize
OpenAI researchers Jagadeesh, Saab, Singhal et al. published findings on June 18 showing RL training on traits like honesty and corrigibility improved 44 of 53 safety benchmarks. Gains generalized across domains not used in training, and the model resisted harmful fine-tuning better than the baselin
Estonian Institute: Claude Tops Russian Propaganda Benchmark, Mistral Trails
Estonian Language Institute benchmark tests 60 AI models vs Russian propaganda. Claude tops, Mistral trails with 36.67% misinformation rate.
Visual-Seeker: Active Visual Reasoning Beats Proprietary MLLMs on 5 Benchmarks
Visual-Seeker achieves SOTA on five multimodal search benchmarks, surpassing proprietary models by actively harvesting visual evidence during search.
NVIDIA Blackwell Ultra Leads First Agentic AI Benchmark, 20x Agents/MW vs Hopper
NVIDIA Blackwell Ultra NVL72 leads the first AgentPerf benchmark for agentic AI, delivering 20x more agents per megawatt than Hopper.
MiniMax M3 Exceeds Human Gold-Medal on Math Benchmarks via MaxProof
MiniMax's M3 exceeded human gold-medal on math benchmarks via MaxProof, but no scores or details were disclosed.
WorldBench: Top MLLM Scores 64% on Visually Diverse Benchmark
WorldBench, a new multimodal benchmark, tests 15 MLLMs on visually diverse images. Top model scores 64.0%, exposing fundamental gaps in visual understanding.
MacArena: 421-Task macOS Benchmark Reveals 26% CUA Ranking Inversion
MacArena benchmark of 421 macOS tasks reveals 26% performance gap for top models on native tasks, suggesting current CUAs overfit to Linux distributions.
SMAC-Talk: StarCraft Benchmark Tests LLM Agents Against Deceptive Allies
SMAC-Talk extends StarCraft Multi-Agent Challenge with natural language communication, testing LLM agents against deceptive allies. Qwen3.5 models benchmarked; no model exceeds 72% win rate.
New 474-Game Benchmark Reveals LLMs Collapse on Counterfactual Reasoning
New 474-game benchmark reveals LLMs fail on counterfactual reasoning, with larger drops than contextual perturbations. Highlights metacognitive gaps in agentic AI.
SpatialBench: New Benchmark Tests Foundation Models on 3D Tasks
SpatialBench, a new benchmark from ropedia_ai, evaluates spatial foundation models across 7 tasks and 5 datasets, testing depth estimation, surface normal prediction, and 3D object detection.
NVIDIA Vera CPU Benchmarks: 1.55x Faster Than Intel Xeon in Phoronix Tests
NVIDIA Vera CPU benchmarks show 1.55x performance over Intel Xeon 6980P and 10% over AMD EPYC 9575F, with 1.2 TB/s memory bandwidth.
Microsoft SkillOpt Trains Agent Skills in Text Space, Beats 52/52 Benchmarks
Microsoft's SkillOpt trains agent skills in text space, achieving best or tied-best results in all 52 settings across 6 benchmarks and 7 models.
HAVEN Benchmark Exposes MLLM Gap Between Fluency and Video Understanding
HAVEN benchmark tests MLLMs on hierarchical video understanding across frame, shot, and video levels. Results show top models lack grounded multimodal reasoning despite fluent text generation.
ByteDance Lance 3B MoE Beats 7B Models on Multimodal Benchmarks
ByteDance released Lance, a 3B multimodal MoE model that beats 7B+ models on benchmarks through multi-task synergy and specialized pathways.
MorphoHELM Benchmark Finds Classic CV Beats Deep Learning on Cell Painting
MorphoHELM benchmark from Microsoft evaluates 20+ methods for Cell Painting, finding no deep learning model beats classic CV when batch effects are controlled.
New Paper Coins 'Curation Debt' — Benchmarks Measure Data Leakage, Not Capability
New paper coins 'curation debt' — benchmarks like MMLU measure data leakage, not capability. Proposes adversarial dynamic benchmarks.
Glean benchmark: Off-the-shelf MCP costs 30% more tokens than indexed context
Glean benchmark: off-the-shelf MCP in Claude Cowork loses 2.5x more tasks and uses 30% more tokens than indexed context.