
Claude 4.5 Sonnet Shows 58% Accuracy on SWE-Bench with 15.2% Variance, Study Finds Consistency Amplifies Both Success and Failure
New research on LLM agent consistency reveals Claude 4.5 Sonnet achieves 58% accuracy with low variance (15.2%) on SWE-bench, but 71% of its failures come from consistently wrong interpretations. The study shows consistency amplifies outcomes rather than guaranteeing correctness.
























