coding benchmarks
30 articles about coding benchmarks in AI news
The Jagged Frontier: What AI Coding Benchmarks Reveal and Conceal
New analysis of AI coding benchmarks like METR shows they capture real ability but miss key 'jagged' limitations. While performance correlates highly across tests and improves exponentially, crucial gaps in reasoning and reliability remain hard to measure.
Claude Opus 4.8 Now Beats Gemini Pro 5 in Coding Benchmarks — What It
Claude Opus 4.8 beats Gemini Pro 5 by 11 points on Fable 5. Claude Code users should run `claude code --model opus-4.8` for complex coding tasks.
Agent Psychometrics: New Framework Predicts Task-Level Success in Agentic Coding Benchmarks with 0.81 AUC
A new research paper introduces a framework using Item Response Theory and task features to predict success on individual agentic coding tasks, achieving 0.81 AUC. This enables benchmark designers to calibrate difficulty without expensive evaluations.
Zhipu GLM-5.2 tops global coding benchmarks, sparks 'DeepSeek moment'
Zhipu AI's GLM-5.2 ranks top-3 globally on a coding benchmark, with US engineers calling it a daily driver superior to GPT-5.5.
Moonshot AI Ships Trillion-Parameter Open Model, Matches Claude Opus on Coding
Moonshot AI released a trillion-parameter open-source model that reportedly matches Anthropic's Claude Opus on most coding benchmarks. This follows the same day Anthropic committed $25B to AWS for compute, highlighting divergent AI scaling strategies.
SciCode: Epoch AI Launches Benchmark Measuring AI Research Ability
Epoch AI launched SciCode benchmark testing LLMs on real research coding tasks. Top models score below 30%, exposing gap between coding benchmarks and scientific ability.
DeepSeek DSpark: Speculative Decoding Unifies Parallel Gen, Adaptive Verification
DeepSeek released DSpark, a speculative decoding framework unifying parallel generation with adaptive verification. No benchmarks disclosed yet; the approach targets inference latency and throughput.
Meta Muse Spark 1.1 Debuts in AI Coding Battle; Zuck Post Hits 12M Views
Meta released Muse Spark 1.1 for agentic coding tasks. Zuckerberg's post got 12M views in 12 hours; no benchmarks disclosed.
SpaceXAI Ships Grok 4.5, Blackwell-Trained Coding Model
SpaceXAI released Grok 4.5, a coding-focused model trained on Blackwell GPUs, now available in Cursor and Vercel. Inference cost claims lack independent benchmarks.
Claude Opus 4.7 Launches with 3.75MP Vision, Agentic Coding, and New Tokenizer
Anthropic launched Claude Opus 4.7 today with 3x higher vision resolution (3.75MP), self-verifying coding outputs, and stricter instruction following. The update targets enterprise agentic workflows and knowledge work benchmarks.
Beyond Unit Tests: How AI Critics Learn from Sparse Human Feedback to Revolutionize Coding Assistants
Researchers have developed a novel method to train AI critics using sparse, real-world human feedback rather than just unit tests. This approach bridges the gap between academic benchmarks and practical coding assistance, improving performance by 15.9% on SWE-bench through better trajectory selection and early stopping.
Open-Source Course Shows Harness, Not Model, Lifts Coding Agent 25 Places
Open-source course shows harness engineering, not model swap, moved a coding agent from ~30th to top 5 on Terminal-Bench. Course builds Decode from scratch.
Kimi K3 Tops US Models in Front-End Coding at Smaller Scale
Moonshot AI's K3 tops US models in front-end coding at 89.2% on SWE-bench while being smaller and cheaper to train.
Emergent Raises at $1.5B Valuation for Vibe Coding IDE
Emergent raised at $1.5B valuation for vibe coding IDE. Round details undisclosed, signaling sustained investor appetite for AI developer tools.
PadCaptioner: 3B video caption model beats 7B rivals with parallel decoding
PadCaptioner, a 3B model, beats 7B rivals in dense video captioning via lossless parallel autoregressive decoding, challenging scaling orthodoxy.
Databricks Tests Coding Agents on Its Own Codebase
Databricks benchmarked coding agents on its own polyglot codebase. GLM-5.2 matched top closed models, a minimal harness halved costs, and cheaper-per-token models cost more per task.
MirrorCode Benchmark Costs $2,600 Per Run, Challenges AI Coding Limits
Epoch AI and METR launched MirrorCode, a $2,600-per-run coding benchmark. Claude Opus 4.7 leads with 56% solve rate.
JetSpec hits 1,000 t/s on Qwen-8B with speculative decoding
JetSpec achieves 1,000 t/s on Qwen-8B with a B200 GPU, claiming superiority over prior speculative decoding methods, but lacks independent verification.
Chinese Lab's Free MoE Model Matches GPT-5.5 on Agentic Coding
A Chinese lab released an Apache-2.0 open-weights MoE model matching GPT-5.5 on agentic coding. This free model challenges proprietary AI's lead with sparse MoE architecture.
MiniMax M3 Sparse Attention: 15.6x Decoding Speedup at 1M Tokens
MiniMax M3 sparse attention achieves 9.7x prefilling and 15.6x decoding speedup at 1M tokens, reversing M2's full-attention stance.
No Rigorous Productivity Tests Exist for Post-2025 Autonomous Coding Tools
No productivity studies exist for autonomous coding tools launched December 2025. All research predates the Claude Code/Codex revolution, creating a major knowledge gap.
Jensen Huang Wants Zero Coding at NVIDIA — 'Purpose vs Task'
Jensen Huang wants zero coding by NVIDIA engineers, framing it as a task to minimize. The bet is AI-generated code will match human output for performance-critical software.
Qwen 3.7-Max Agentic Coding Demo Shows Frontier-Level UI Replication
Qwen 3.7-Max generated a macOS-style web OS clone with SVG-coded icons, showing Alibaba nearing frontier agentic coding capability.
Composer 2.5 Scores 62 on Coding Index at $0.07 vs. $4-5 for Rivals
Composer 2.5 scores 62 on coding index at $0.07/task vs $4-5 for rivals scoring 65-66. 60x cost savings with near-parity performance.
NanoGPT-Bench: A New Eval for Coding Agents Doing AI Research
IntologyAI released NanoGPT-Bench, an internal eval for coding agents on an AI R&D problem. No results or task specifics have been disclosed.
Gemini Flash Rumored at 92% of GPT-5.5 Coding, 15-20x Cheaper
Unconfirmed rumor claims Gemini Flash achieves 92% of GPT-5.5 coding performance at 15-20x lower cost. Source is a single X post; no official confirmation.
Snapdragon X2 Elite Beats Intel Arrow Lake for AI Coding Agents
Snapdragon X2 Elite beat Intel Arrow Lake for Windows AI coding agents. CPU bottleneck, not inference speed, limited performance per @mweinbach.
NVIDIA NeMo RL Speculative Decoding: 1.8× Rollout Speed at 8B
NVIDIA's NeMo RL speculative decoding achieves 1.8× rollout speedup at 8B and projects 2.5× at 235B, cutting RL training time by over half.
GPT-5.5 Tops Benchmarks, Costs 2x API Price, Still Hallucinates
OpenAI launched GPT-5.5, an agentic model that tops Terminal-Bench 2.0 at 82.7% and surpasses Claude Opus 4.7 and Gemini 3.1 Pro on coding and math. However, independent testing shows higher hallucination rates and effective API costs 20% above GPT-5.4 despite doubled token prices.
PayPal Cuts LLM Inference Cost 50% with EAGLE3 Speculative Decoding on H100
PayPal engineers applied EAGLE3 speculative decoding to their fine-tuned 8B-parameter commerce agent, achieving up to 49% higher throughput and 33% lower latency. This allowed a single H100 GPU to match the performance of two H100s running NVIDIA NIM, cutting inference hardware cost by 50%.