Moghe and Chin's agent harness on DeepSeek V3.2 scores 67.25% pass@2 on ARC-AGI-1. It costs $0.62 per task with zero ARC-specific fine-tuning.
Key facts
- Reflective Orchestrator: 67.25% pass@2 on ARC-AGI-1 public set
- Cost: $0.62 per task, no ARC-specific fine-tuning
- Baseline pipeline: 57.50% pass@2 at $0.25 per task
- Think tool ablation: removes 5.75 pp from pass@2
- Pipeline is generation-bound, not selection-bound (95% ceiling captured by selection)
A new arXiv preprint from researchers Kabir Moghe and Peter Chin demonstrates that an open-weight model in non-thinking mode can achieve competitive ARC-AGI-1 performance purely through agent architecture, without ARC-specific fine-tuning or heavy test-time compute. The work explores a third regime distinct from the two dominant approaches — either massive test-time compute over frontier models (evolutionary search, exhaustive sampling) or benchmark-specific fine-tuning of small models on ARC data.
The Explorer-Definer Pipeline
The baseline architecture, called the Explorer-Definer Pipeline, separates pattern discovery from executable transformation synthesis into two explicit stages. A PatternExplorer agent identifies visual patterns in the ARC-AGI-1 training pairs, then a TransformationDefiner agent synthesizes a Python program implementing the transformation. On the public 400-task evaluation set, this pipeline achieves 57.50% pass@2 at $0.25 per task [per the arXiv preprint]. This represents a ~42-point lift over the 15.50% one-shot baseline, achieved with no ARC-specific training and no heavy test-time compute.
The paper identifies the "think tool" — a component that instructs the agent to reason step-by-step before generating code — as a critical element. An ablation removing the think tool reduces pass@2 by 5.75 percentage points.
The Reflective Orchestrator
The Reflective Orchestrator augments the pipeline with autonomous re-exploration: when the TransformationDefiner's program fails on training pairs, the orchestrator generates new transformations and retests hypotheses. This lifts performance to 67.25% pass@2 at $0.62 per task [per the arXiv preprint]. The orchestrator adds +9.81 pp in unbiased pass@1, confirming the paper's diagnostic that the pipeline is generation-bound, not selection-bound.
The diagnostic analysis reveals that selection via training-pair accuracy captures ~95% of the candidate ceiling, meaning the bottleneck is generating diverse candidate solutions, not ranking them. The orchestrator directly implements this prediction through adaptive re-exploration and validates it empirically — the unbiased pass@1 lift matches the selection-mediated pass@2 lift, confirming that broader generation, not better ranking, drives improvement.
Why This Matters
The result is notable not for topping leaderboards — several systems exceed 80% pass@2 — but for the cost-performance tradeoff and the architectural insight. At $0.62 per task, the Reflective Orchestrator costs roughly 1-2 orders of magnitude less than systems relying on heavy test-time compute over frontier models. The paper also provides a falsifiable diagnostic framework: the generation-bound vs. selection-bound analysis offers a principled way to identify where future improvements should focus, rather than relying on brute-force compute scaling.
DeepSeek, the Chinese AI lab behind the V3.2 model used in the experiments, has been rapidly expanding its footprint. The company recently raised $7.4B at a $50B valuation in its first external funding round, and is developing custom inference ASICs to cut GPU dependency [per prior Gentic reporting]. The ARC-AGI-1 result further demonstrates DeepSeek's models can serve as cost-effective reasoning engines when paired with the right agentic scaffolding.
Key Takeaways
- Moghe & Chin achieve 67.25% pass@2 on ARC-AGI-1 using DeepSeek V3.2 in non-thinking mode at $0.62/task, with no fine-tuning.
- The work demonstrates agent architecture alone can lift a 15.50% baseline by ~52 points.
What to watch
Watch for whether the DeepSeek team or others apply the generation-bound diagnostic to scale candidate generation via larger models or more diverse prompts, potentially pushing pass@2 past 75% at sub-$1 per task. Also track if DeepSeek integrates this agent harness into its upcoming coding agent product.
Source: arxiv.org
