Two harness settings tripled GPT-5.6 Sol's ARC-AGI-3 score — and the open-source Duck pushed the community leaderboard to 55.89%

On July 29, OpenAI published a controlled experiment: two harness settings — retaining the model's private reasoning between actions and compacting old context instead of truncating it — took GPT-5.6 Sol from 13.3% to 38.3% RHAE on the ARC-AGI-3 public set, with no change to the model and roughly 6× fewer output tokens. Four days after the September 30 milestone, the Kaggle community leaderboard reached 55.89% RHAE, atop an open-source single-GPU harness running a 27B open-weight model.

What shipped

The OpenAI experiment. OpenAI says it investigated after GPT-5.6 Sol posted unexpectedly weak results on ARC-AGI-3 — 7.8% overall, versus 0.4% for GPT-5.5 (remio.ai). Rebuilding the evaluation through the Responses API with two changes — retaining the model's private chain-of-thought between actions (via previous_response_id chaining) and replacing rolling truncation with compaction, which summarizes older context instead of deleting it — raised the public-set score from 13.3% to 38.3% while using roughly 6× fewer output tokens (OpenAI, OpenAI Devs on X). The model weights never changed.

The mechanism matters. The official harness discarded all private reasoning after every game action, so the model saw its moves and brief notes but not the plans and hypotheses behind them; rolling truncation then dropped the oldest messages once the conversation passed roughly 175,000 characters (remio.ai). The agent re-derived its interpretation of the game from scratch every turn. With reasoning retained, OpenAI reports the model spent less time re-thinking before each action and used more coherent strategies over long runs — which is why the "more memory" configuration was cheaper: it stopped paying output tokens to reconstruct conclusions it had already reached. On one game where no frontier model on the relevant leaderboard had completed beyond level 1, the revised harness let GPT-5.6 Sol finish all six levels (remio.ai). OpenAI estimates the average human tester scored 48% on the same tasks, from official gameplay logs — not ARC Prize's formal baseline (remio.ai).

The leaderboard jump. The Kaggle community leaderboard for ARC-AGI-3 — the interactive-reasoning benchmark where agents infer the rules and goal of novel games, represented as sequences of 64×64 grids, with no instructions (Tufa Labs) — reached 55.89% RHAE on October 4, 2026, nearly double the 27.9% high-water mark set four days earlier by the Milestone 2 winner (TechTimes). Tufa Labs sits at 55.89%, followed by Yi-Chia Chen at 48.59%, with a cluster in the 33–38% band — figures independently mirrored by the CLIST standings tracker (CLIST). The $700,000 grand prize for a 100% score remains unclaimed, and the final submission deadline is November 2, 2026 (ARC Prize, TechTimes).

The Duck. The leaderboard's top entry is The Duck, Tufa Labs' open-source harness that won Milestone 1 ($25,000, June 30) and is, per TechTimes, "the single most influential open-source submission of the competition to date" (TechTimes, ARC Prize). It runs Qwen 3.6 27B FP8 — an open-weight Alibaba model — locally on Kaggle's single-GPU allocation (Tufa Labs). Its design is deliberately minimal, built on three choices:

• Python REPL interface. Game state is encoded as Python variables; the model perceives and acts through a live REPL, treating every game as an interactive coding problem.
• Triple-channel perception. The agent sees a rendered image, a raw ASCII grid, and a segmentation-zoom tool for inspecting specific regions, and picks whichever representation fits the moment.
• Infinite play via eviction. The harness continuously pops the oldest messages, keeping only the system prompt and recent history, so exploration can run indefinitely on a fixed context budget.

The counterintuitive finding: hand-crafted specialized tools hurt performance. "The gains came from multimodality and better base models, not hand-built tools," Tufa Labs wrote (TechTimes, ARC Prize). The harness's own mean score on the 25 public games was 1.6002 ± 0.4475 across 20 tries per game. Compared against an "executable world models" agent on GPT-5.4, it solved a similar set of games at roughly an order of magnitude lower cost per game — evidence, Tufa argues, that which games are solvable tracks model capability while the cost is mostly dictated by the harness (Tufa Labs). The full framework, including a Kaggle notebook that reproduces a competition run, is on GitHub (Tufalabs/duck-harness).

The pattern holds down the leaderboard: Milestone 2's winner, Daniel Franzen at 27.9%, kept the Duck harness architecture and swapped the serving stack (SGLang instead of vLLM) with the model "largely unchanged" (TechTimes, ARC Prize on X). Base models have stayed mostly in the Qwen-family 27B-class range; the scaffolding around them is what moved.

Why it matters

The OpenAI result inverts the usual cost–capability tradeoff: preserving more state made the agent both better and cheaper — 38.3% instead of 13.3% at 6× fewer output tokens. For anyone shipping long-horizon agents, the lesson is that your wrapper's memory policy is a first-class performance lever, not plumbing. Retained reasoning and compaction are both available today through the Responses API; the Duck's eviction-based "infinite play" is a general recipe for fixed-context agents that must explore indefinitely.

The second lesson is about evaluation. ARC-AGI-3's community leaderboard permits custom harnesses, and that is where the movement is happening; the ARC Prize verified leaderboard uses a standardized harness, where TechTimes reports GPT-6 Astra at 62.7% and Claude Opus 5 at 30.2% (TechTimes). No agentic benchmark score is cleanly interpretable as a model-capability signal without specifying the harness — as one analysis of the competition put it, "the settings in your wrapper are part of your benchmark score, and you probably never chose them deliberately" (TechTimes). The ARC Prize Foundation's own technical report concedes the point: "Better harnesses lead to further task automation abilities... While not necessarily representing progress towards AGI, [harness innovation] remains economically valuable" (TechTimes).

Caveats

• The OpenAI numbers are vendor-reported and not independently reproduced; the 13.3% → 38.3% comparison is on the public set only, and OpenAI published no factorial ablation isolating retained reasoning from compaction (OpenAI). The 48% human figure is OpenAI's estimate from gameplay logs, not ARC Prize's formal baseline.
• The 55.89% leaderboard figure is the community leaderboard, which permits custom harnesses; it is not the private holdout set that determines prize eligibility, and public scores do not directly predict private performance (TechTimes).
• The Duck's mean public score and cost comparison are Tufa Labs' own measurements, though the harness itself is open source and reproducible.
• TechTimes reports The Duck was released under an MIT license, but the GitHub repo currently carries no LICENSE file — worth checking before you build on it.
• A separate project called Schema reported raising a Claude/GPT baseline from 42.83% to 98.98% on the 25 public games via harness changes alone; those figures are not independently verified and the public games can be studied in advance (TechTimes).

Sources

• OpenAI — How two settings tripled our scores on the ARC-AGI-3 benchmark
• OpenAI Devs on X — retained reasoning and compaction, 13.3% to 38.3%, ~6x fewer output tokens
• remio.ai — OpenAI Says Two API Settings Tripled GPT-5.6 Sol's ARC-AGI-3 Score
• TechTimes — ARC-AGI-3 Scaffolding Beats Model Upgrades
• Tufa Labs — Duck Harness: Winning Solution for ARC-AGI-3 Milestone 1
• GitHub — Tufalabs/duck-harness
• ARC Prize — ARC-AGI-3 Milestone Prize #1
• ARC Prize — ARC-AGI-3 Competition page
• ARC Prize on X — Milestone 2 results
• CLIST — ARC Prize 2026 ARC-AGI-3 standings