Anthropic co-founder @gdb noted that 'benchmarks get saturated very quickly these days.' The statement highlights a growing structural problem in AI evaluation: models now hit ceiling performance on popular benchmarks within months, making it difficult to distinguish genuine progress from noise.
Key facts
- MMLU saturated within 1 year of GPT-4 release.
- Top scores above 90% on MMLU as of 2025.
- SWE-Bench and GPQA are newer, harder benchmarks.
- Rapid saturation erodes signal-to-noise in comparisons.
- Dynamic evaluations may replace static benchmarks.
In a brief post on X, Anthropic co-founder Dario Amodei — posting as @gdb — stated that 'benchmarks get saturated very quickly these days.' While the post lacks specific examples or data, it echoes a well-documented trend in the AI research community.
Current benchmarks like MMLU, HellaSwag, and GSM8K have seen top scores plateau within a year of release. For instance, MMLU's human baseline of 89.8% was surpassed by GPT-4 in March 2023; by 2025, multiple models claim scores above 90%, compressing the meaningful range. [According to prior reporting from Epoch AI], the rate of benchmark saturation has accelerated since 2022, driven by both improved training methods and data contamination.
This saturation erodes the signal-to-noise ratio in model comparisons. A 0.5% improvement on a saturated benchmark may reflect overfitting to the test set rather than genuine capability gain. The field has responded with harder benchmarks like SWE-Bench, which tests real-world software engineering tasks, and GPQA, which requires graduate-level reasoning. Yet even these newer benchmarks are showing signs of compression as frontier labs iterate rapidly.
The unique take here is that saturation is not just a measurement problem — it is a structural challenge for the entire AI development cycle. When benchmarks saturate quickly, companies lose a key tool for communicating progress to investors, regulators, and the public. It also makes it harder to detect safety-relevant regressions between model versions. The industry may need to shift from static benchmarks to dynamic, adversarial evaluations that update regularly — similar to how cybersecurity teams use penetration testing rather than fixed test suites.
Amodei's post does not propose a solution, but his platform suggests Anthropic is aware of the issue internally. The company has previously published research on measuring situational awareness and honesty in models, which could inform new evaluation paradigms.
Key Takeaways
- @gdb notes benchmarks saturate quickly.
- This undermines AI progress tracking and may force shift to dynamic evaluations.
What to watch
Watch for Anthropic's next safety paper or blog post — if it introduces a new evaluation methodology or benchmark, that would signal the company is moving from diagnosis to action. Also track whether the industry coalesces around dynamic benchmarks like the Chatbot Arena Elo system, which resists saturation by continuously adding new models and tasks.
