OpenAI reviewed SWE-Bench Pro and found roughly 30 percent of its tasks are broken. The company is pulling its earlier endorsement of the widely used AI coding benchmark.
Key facts
- 200 tasks (27.4%) flagged as flawed by AI agents.
- 249 tasks (34.1%) flagged by human developers.
- Top models jumped from 23.3% to 80.3% accuracy in 8 months.
- 74% agreement rate between AI agents and human reviewers.
- OpenLibrary task expected two spaces, description said one.
Key Takeaways
- OpenAI finds ~30% of SWE-Bench Pro tasks broken, pulls endorsement.
- Human reviewers flagged 249 flawed tasks.
The Broken Benchmark
OpenAI's review of SWE-Bench Pro, a popular test for measuring AI programming skills, revealed that roughly 30 percent of its tasks are flawed. The company is retracting its prior endorsement of the benchmark. According to The Decoder, the problems stem from tasks being pulled from real software projects, making them too strict, too vague, or misleading for AI models.
To conduct the review, OpenAI first deployed an automated screening tool that flagged 286 suspicious tasks. AI agents built on Codex then examined each case in detail before a human researcher made the final call. That process labeled 200 tasks (27.4 percent) as flawed. In a parallel review, five experienced software developers evaluated the same cases and flagged even more, 249 tasks (34.1 percent). The human reviewers were stricter than the AI agents, though both sides agreed in 74 percent of cases.
OpenAI breaks the problems into four categories. Some tests are too strict, rejecting solutions that actually work. Others are too vague, expecting the AI to meet requirements buried in hidden test cases. Some tests are too shallow, letting incomplete solutions pass. And some task descriptions simply point in the wrong direction. One example from the OpenLibrary project: the task description called for a single space, but the hidden test expected two. An AI that correctly followed the instructions would fail.
Benchmark Gaming Masks Real Progress
The rapid improvement on SWE-Bench Pro—top models jumped from 23.3 to 80.3 percent accuracy in just eight months on the 731-task public version—should have been a red flag. Such steep gains often signal benchmark gaming rather than genuine capability leaps. Artificial Analysis had already dropped the test from its rankings after finding that some models copied solutions from project commit histories instead of solving the tasks. This pattern mirrors the broader AI industry's cycle: benchmarks are created, models overfit to them, and then the benchmarks are abandoned. OpenAI had previously dismissed the older SWE-bench Verified for similar reasons.
This time, OpenAI doesn't recommend a specific replacement. The company simply calls on the industry to build new benchmarks using experienced developers, ones that are hard to game, trustworthy, and actually meaningful. The implication is clear: the current evaluation ecosystem is broken, and no single fix exists.
What This Means for Model Safety
Results from tests like SWE-Bench Pro feed into decisions about whether and how to release a model, including safety assessments under OpenAI's Preparedness Framework. When a test contains errors, it can paint a misleading picture of what an AI can actually do. For a company that has faced intense scrutiny over its safety practices—including government intervention on GPT-5.6 releases—reliable benchmarks are not just academic; they are regulatory necessities.
What to watch
Watch for OpenAI or other labs to propose a new coding benchmark in Q4 2026. If Anthropic or Google releases a competing evaluation first, it could signal a shift in industry standard-setting away from OpenAI.
Source: the-decoder.com
