benchmark improvement
30 articles about benchmark improvement in AI news
Cursor Launches Composer 2 with $0.50/M Input Token Pricing, Claims Major Benchmark Gains
Cursor has released Composer 2, a coding AI model priced at $0.50 per million input tokens and $2.50 per million output tokens. The company reports significant benchmark improvements over previous versions across CursorBench, Terminal-Bench 2.0, and SWE-bench Multilingual.
MiniMax M2.7 Achieves 30% Internal Benchmark Gain via Self-Improvement Loops, Ties Gemini 3.1 on MLE Bench Lite
MiniMax had its M2.7 model run 100+ autonomous development cycles—analyzing failures, modifying code, and evaluating changes—resulting in a 30% performance improvement. The model now handles 30-50% of the research workflow and tied Gemini 3.1 in ML competition trials.
GPT-5.5 Pro Leapfrogs on Epoch Benchmark; Base Model Beats Prior Pro
A tweet from @kimmonismus reveals GPT-5.5 Pro shows significant Epoch benchmark gains, and the non-Pro GPT-5.5 surpasses GPT-5.4 Pro, suggesting major efficiency improvements at OpenAI.
Memento-Skills Agent System Achieves 116.2% Relative Improvement on Humanity's Last Exam Without LLM Updates
Memento-Skills is a generalist agent system that autonomously constructs and adapts task-specific agents through experience. It enables continual learning without updating LLM parameters, achieving 26.2% and 116.2% relative improvements on GAIA and Humanity's Last Exam benchmarks.
The Hidden Contamination Crisis: How Semantic Duplicates Are Skewing AI Benchmark Results
New research reveals that LLM training data contains widespread 'soft contamination' through semantic duplicates of benchmark test data, artificially inflating performance metrics and raising questions about genuine AI capability improvements.
Benchmark lets image models answer in pixels, not text
New 'Show, Don't Tell' benchmark tests spatial cognition via pixel-level outputs. GPT Image 2 solves 37% of cases missed by GPT-5.4, highlighting a gap in text-based spatial reasoning.
KeyFrame-Compass Benchmark Targets Keyframe Video Generation Gaps
KeyFrame-Compass is the first benchmark for keyframe-conditioned video generation, with 386 samples and six metrics.
gdb: Benchmarks Saturate Too Fast for Reliable AI Progress Tracking
@gdb notes benchmarks saturate quickly. This undermines AI progress tracking and may force shift to dynamic evaluations.
LLMForge: 7 Models Score 0.89 on CAD Benchmark; VLMs Fix Cylinders
LLMForge scores 0.89 on 97-design CAD benchmark. VLM critic achieves 100% watertight meshes but cylinders remain a failure mode.
Ant Group's 1.1B LingBot-Vision Beats Meta's 7B DINOv3 on 12 Benchmarks
Ant Group's 1.1B LingBot-Vision tops Meta's 7B DINOv3 on 12 spatial benchmarks, with 40% fewer FLOPs.
MirrorCode Benchmark Costs $2,600 Per Run, Challenges AI Coding Limits
Epoch AI and METR launched MirrorCode, a $2,600-per-run coding benchmark. Claude Opus 4.7 leads with 56% solve rate.
OpenAI shows small doses of beneficial-trait RL improve 44 of 53 safety benchmarks — and the gains generalize
OpenAI researchers Jagadeesh, Saab, Singhal et al. published findings on June 18 showing RL training on traits like honesty and corrigibility improved 44 of 53 safety benchmarks. Gains generalized across domains not used in training, and the model resisted harmful fine-tuning better than the baselin
SpatialBench: New Benchmark Tests Foundation Models on 3D Tasks
SpatialBench, a new benchmark from ropedia_ai, evaluates spatial foundation models across 7 tasks and 5 datasets, testing depth estimation, surface normal prediction, and 3D object detection.
NVIDIA Vera CPU Benchmarks: 1.55x Faster Than Intel Xeon in Phoronix Tests
NVIDIA Vera CPU benchmarks show 1.55x performance over Intel Xeon 6980P and 10% over AMD EPYC 9575F, with 1.2 TB/s memory bandwidth.
Karpathy Joins Anthropic to Lead Recursive Self-Improvement Team
Andrej Karpathy joins Anthropic to lead a new recursive self-improvement team using Claude to accelerate pretraining, per @kimmonismus. The move signals a bet on synthetic data loops over brute-force scaling.
HAVEN Benchmark Exposes MLLM Gap Between Fluency and Video Understanding
HAVEN benchmark tests MLLMs on hierarchical video understanding across frame, shot, and video levels. Results show top models lack grounded multimodal reasoning despite fluent text generation.
Glean benchmark: Off-the-shelf MCP costs 30% more tokens than indexed context
Glean benchmark: off-the-shelf MCP in Claude Cowork loses 2.5x more tasks and uses 30% more tokens than indexed context.
Agentick Benchmark: GPT-5 Mini Tops at 0.309, No Agent Paradigm Dominates
Agentick benchmark evaluates RL, LLM, VLM, and hybrid agents on 37 tasks. GPT-5 mini leads at 0.309 ONS, but no paradigm dominates. ASCII beats natural language.
Simple Graph Heuristic Beats Generative Recommenders on 10 of 14 Benchmarks
A no-training graph heuristic beats generative recommenders on 10 of 14 benchmarks, exposing shortcut-solvable datasets. Relative NDCG@10 gains hit 44% on Amazon CDs.
MIT's RLM Handles 10M+ Tokens, Outperforms RAG on Long-Context Benchmarks
MIT researchers introduced Recursive Language Models (RLMs), which treat long documents as an external environment and use code to search, slice, and filter data, achieving 58.00 on a hard long-context benchmark versus 0.04 for standard models.
ThermoQA Benchmark Reveals LLM Reasoning Gaps: Claude Opus Leads at 94.1%
Researchers released ThermoQA, a 293-question benchmark testing thermodynamic reasoning. Claude Opus 4.6 scored 94.1% overall, but models showed significant degradation on complex cycle analysis versus simple property lookups.
Personalized LLM Benchmarks: Individual Rankings Diverge from Aggregate (ρ=0.04)
A new study of 115 Chatbot Arena users finds personalized LLM rankings diverge dramatically from aggregate benchmarks, with an average Bradley-Terry correlation of only ρ=0.04. This challenges the validity of one-size-fits-all model evaluations.
SocialGrid Benchmark Shows LLMs Fail at Deception, Score Below 60% on Planning
Researchers introduced SocialGrid, a multi-agent benchmark inspired by Among Us. It shows state-of-the-art LLMs fail at deception detection and task planning, scoring below 60% accuracy.
GPT-5.5 Limited Rollout Begins, Frontend Improvements Noted
OpenAI has started a limited rollout of GPT-5.5 to select users, with early reports highlighting significant frontend quality improvements. This suggests an incremental update focused on user experience rather than core model capabilities.
MLX-Benchmark Suite Launches as First Comprehensive LLM Eval for Apple Silicon
The MLX-Benchmark Suite has been released as the first comprehensive evaluation framework for Large Language Models running on Apple's MLX framework. It provides standardized metrics for models optimized for Apple Silicon hardware.
RiskWebWorld: A New Benchmark Exposes the Limits of AI for E-commerce Risk
Researchers introduced RiskWebWorld, a realistic benchmark for testing GUI agents on 1,513 authentic e-commerce risk management tasks. It reveals a major capability gap, showing even the best models fail over 50% of the time, highlighting the immaturity of AI for high-stakes operational automation.
OpenAI Quietly Phasing Out MRCR Benchmark in Claude Evaluations
An OpenAI engineer confirmed the company is phasing out the MRCR benchmark from Claude's system card, citing its poor correlation with real-world performance and high evaluation cost. This reflects a broader industry move toward more practical, cost-effective evaluation methods.
HORIZON Benchmark Diagnoses Long-Horizon Failures in GPT-5 and Claude Agents
A new benchmark called HORIZON systematically analyzes where and why LLM agents like GPT-5 and Claude fail on long-horizon tasks. The study collected over 3100 agent trajectories and provides a scalable method for failure attribution, offering practical guidance for building more reliable agents.
LLM Evaluation Beyond Benchmarks
The source critiques traditional LLM benchmarks as inadequate for assessing performance in live applications. It proposes a shift toward creating continuous test suites that mirror actual user interactions and business logic to ensure reliability and safety.
Mythos AI Model Reportedly 'Destroys' Benchmarks in Early Leak
A viral tweet claims the unreleased Mythos AI model 'destroys every other model' based on leaked benchmarks. No official confirmation or technical details are available.