StateM hits 95.3% on Terminal-Bench 2.1 via harness scaling, per Hugging Face's weekly roundup. The result highlights how evaluation infrastructure—not model weights—can drive agent performance.
Key facts
- StateM: 95.3% accuracy on Terminal-Bench 2.1
- Method: harness scaling, not model weights
- RA-Bench: AI-generated video detection in crises
- SemaPLC: PLC code generation with verification gates
- 10+ papers listed in weekly Hugging Face roundup
StateM's 95.3% accuracy on Terminal-Bench 2.1, reported in Hugging Face's weekly paper roundup, is notable not for the model itself but for the method: harness scaling According to @HuggingPapers. This technique improves the evaluation harness—the scaffolding that connects an agent to a terminal environment—rather than the underlying policy. The implication: agent benchmarks often measure harness engineering as much as agent intelligence.
Harness scaling is a known lever. Prior work on SWE-Bench showed that better environment setup, retry logic, and tool definitions can shift scores by 10-20 points without changing the model. StateM's result, if replicated, suggests Terminal-Bench 2.1 is similarly sensitive to harness design. The roundup does not disclose StateM's architecture, training compute, or the specific harness changes, leaving the result difficult to contextualize [Per the tweet].
Key Takeaways
- StateM hits 95.3% on Terminal-Bench 2.1 via harness scaling, per Hugging Face roundup.
- Highlights evaluation infrastructure's impact on agent scores.
Benchmark Proliferation
Beyond StateM, the roundup lists several new benchmarks targeting agent and video domains. RA-Bench evaluates AI-generated video detection in real-world crises, addressing deepfake verification under chaotic conditions. SemComp-Bench measures semantic task completion in video generation, moving beyond pixel-level metrics. S^2VOPD boosts visual reasoning with self-supervised distillation, while EnvHarness reimagines static environments for agent learning—directly relevant to the harness-scaling theme.
These benchmarks share a pattern: they aim to fix evaluation blind spots. Terminal-Bench 2.1 itself was designed to test terminal command execution, a core agent skill. The proliferation suggests the field is converging on evaluation as a bottleneck for agent progress.
Verification and Embodied Learning
SemaPLC grounds PLC code generation with verification gates, ensuring generated code passes formal checks before deployment. This is a practical step for industrial automation, where unverified code is a safety risk. Zetta enables closed-loop embodied learning for physical intelligence, while HarnessEval-W brings agentified evaluation to world models—another harness-centric approach.
The trend is clear: harness engineering and verification are becoming first-class research topics. As agents move into production, the gap between model capability and reliable deployment narrows through better evaluation and control.
No training details, model sizes, or compute figures were provided for any of these papers in the roundup. The field should treat these results as preliminary until full papers and code are available.
What to watch
Watch for StateM's paper and code release—if the harness changes are open-sourced, expect replication efforts across other agent benchmarks. Also track whether Terminal-Bench 2.1 updates its leaderboard to control for harness variability, and whether RA-Bench's crisis video detection becomes a standard for deepfake verification.
