Prime Agent, an open-source RLM harness, scores 95.5% on ARC-AGI-3 via persistent IPython REPL state. The result, posted by @HuggingPapers, suggests episodic training loops are leaving reasoning performance on the table.
Key facts
- 95.5% score on ARC-AGI-3 benchmark
- Persistent IPython REPL maintains state across rollouts
- Open-source RLM harness for coding
- Continual harness state enables self-improvement
- No training compute or ablation data disclosed
Prime Agent is an open-source, self-improving reinforcement learning from machine feedback (RLM) harness designed for long-horizon coding and evaluation tasks. According to @HuggingPapers, the system uses a persistent IPython REPL and continual harness state to reach 95.5% on ARC-AGI-3, a benchmark designed to test abstract reasoning and generalization beyond training distributions.
The core architectural bet is the persistent REPL. Standard agentic training loops reset interpreter state between episodes, forcing the model to re-derive context and pay a state-reset penalty on every step. Prime Agent keeps a live IPython session and a running harness state across the entire rollout, allowing the model to accumulate variables, functions, and reasoning artifacts without serialization overhead. This mirrors how a human engineer actually works — a long-lived notebook session rather than a fresh shell per task.
Key Takeaways
- Prime Agent, an open-source RLM harness, reports 95.5% on ARC-AGI-3 using persistent IPython REPL state.
- The claim challenges episodic training norms but lacks reproducibility details.
Why persistent state changes the compute tradeoff
The 95.5% ARC-AGI-3 figure, if reproducible, would represent a step change over episodic baselines that typically land in the 70-80% range on similar abstract reasoning suites. The mechanism is plausible: ARC-style puzzles often require composing multiple transformations across steps, and a persistent REPL lets the agent cache intermediate representations and revisit them without re-computation. This effectively extends the model's working memory beyond the context window by externalizing state into the interpreter.
Self-improvement in this context means the agent can modify its own evaluation harness. The continual harness state allows the model to add test cases, refine scoring functions, or patch its own tooling mid-rollout — a form of recursive self-modification that most training pipelines explicitly forbid. This is both the most interesting and the most dangerous design decision, as it trades safety constraints for raw benchmark performance.
The source post is thin on implementation details — no architecture diagram, no training compute figures, no ablation of the persistent REPL against a reset baseline. The 95.5% number is reported without a confidence interval or a breakdown across ARC-AGI-3's difficulty tiers. The tweet links to a project page that may contain more, but the public post alone is not enough to fully evaluate the claim.
The benchmark question
ARC-AGI-3 is designed to resist memorization, with novel puzzle distributions generated per evaluation. A 95.5% score would imply near-human performance on abstract reasoning tasks that current frontier models, including GPT-4-class systems, solve at roughly 30-50% accuracy. If Prime Agent's approach generalizes beyond the benchmark, it suggests that persistent external state — not just larger context windows — is the missing ingredient for long-horizon reasoning.
Skepticism is warranted until the code is audited. The claim of "self-improving" could mean the harness tunes its own prompts or hyperparameters, or it could mean the agent is effectively overfitting to the evaluation distribution. The persistent REPL also raises a reproducibility concern: if the harness state is not seeded deterministically, results may vary run-to-run. The source does not disclose whether the 95.5% is a single run, a median, or a best-of-N selection.
What to watch
Watch for the full code release and a peer-reviewed technical report. Key metrics: whether the 95.5% reproduces across random seeds, an ablation that resets the REPL per episode, and any comparison against GPT-4-class baselines on ARC-AGI-3. A benchmark submission or formal evaluation would settle whether the persistent-state advantage is real or an artifact of the harness design.
