Read the failure composition before choosing harness evolution or weight training: a 9B LoRA adapter substitutes for the whole evolution line
New preprint: process failures (loops, blocked calls) are the harness's job, content failures (poor delivered plans) are the weights' — and the boundary between what trains in and what does not is size-dependent.
A new arXiv preprint argues that the choice between evolving an agent's harness and fine-tuning its weights can be read off the failure composition of its trajectories. Process failures — loops, blocked calls, exhausted step budgets — are the harness's job; content failures, a delivered plan that is simply poor, are what weight training is for.
What shipped
"Harness Evolution Hits a Ceiling: When Weight Training Should Begin" (arXiv:2610.11655) was posted 8 October 2026 by Yuan Tian and Bing Hu (independent researchers) with Hao Wang (UC Berkeley), Binghang Lu (Purdue) and Fang Wu (Stanford). It is a 21-page preprint with no code or data release.
The setup is standard for long-horizon agents: you can improve the harness — "the code and text around a frozen model: system prompts, memory tools, checkpoints, output gates, the tools it may call" — or train the weights. The paper's contribution is a diagnostic that tells you which lever to pull, and when to stop pulling the other.
The harness loop is not new: an LLM proposes edits to its own harness and a judge verifies them, in the style of AHE (Lin et al., 2026). The evolved components split into compliance-enforcing (an exit gate, checkpoints, hard blocks) and capability-granting (a state ledger, a distilled lessons file, invented tools). What is new is the failure-composition lens. The authors sort every failed trajectory into four classes by the first failure signal that fires:
- F1: unbacked assertions in the plan
- F2: the harness's own interventions getting in the way
- F3: loops and exhausted step budgets
- F4: a delivered plan that is simply poor
A deterministic checker labels trajectories from process signals and the scorer, not an LLM judge. The paper's hypothesis: harness evolution repairs the process classes (F2, F3), the behaviour it instils can be trained into the weights, and content failures (F4) are what weight training is for.
On DeepPlanning — a travel- and shopping-planning benchmark with deterministic programmatic scoring, attributed to the Qwen team at Alibaba in the co-r-e.com write-up, where GPT-5.2-high manages only 44.6% average case accuracy — the loop lifts Qwen3.5-4B's held-out score from 0.16 to 0.30 and Qwen3.5-9B's from 0.32 to 0.44 (48 development tasks per decision, 72 held-out tasks the loop never saw). The mechanism shows up in the composition. Held-out delivery at 4B rises from 55% to 90%; loop failures (F3) fall from 64% to 37% of trajectories at 4B and from 39% to 17% at 9B. Content failures (F4) stay at 25% of trajectories at 9B and rise at 4B (10% to 28%). The harness is not fixing content; those failures are left for the weights.
The second half of the paper tests which of these gains train into the weights. LoRA adapters trained on evolved-harness trajectories add +0.13 on held-out tasks for both sizes under the original harness. The size dependence is the interesting part: at 4B the adapter stacks with the harness — combined they more than double the held-out score — while at 9B the adapter alone matches the full evolution line, cutting content failures from a quarter of trajectories to one in twenty. A placebo adapter trained on answer-shuffled trajectories falls 0.16–0.23 below the base model, and full fine-tuning on a larger trajectory set adds only about +0.04 over LoRA.
The transfer run draws the boundary more sharply. On WebArena-Lite, the loop gains +0.09 on 117 unseen tasks, but the gain "lives in what the model sees, two pages of history instead of one", and adapters trained on those trajectories do not add to it. Gains that change what the model writes train in; gains that change what it sees do not.
The paper also reports seed measurements across nine evolution lines on eight models from six families: the harness lever paid (+0.05 to +0.14) on five lines and nothing on three.
Why it matters
The headline is a diagnose-then-intervene rule applied twice: read the failure composition to choose between harness and weights, then read what the accepted edits changed to decide which gains to train in. It is a concrete decision procedure for the most common agent-engineering question.
The more interesting finding is the boundary between what trains in and what does not. At 9B, a LoRA adapter trained on evolved-harness trajectories substitutes for the whole evolution line — at that size the two levers substitute for each other. At 4B they do not, and you need both. On WebArena-Lite the harness gain is purely observational (more history), and adapter training does not recover it. If that holds up, it tells you where to spend compute: train in behavioural changes, keep observational changes in the runtime.
The paper also diagnoses why harness evolution hits a ceiling: the evidence package the proposal model reads "leads with gate and checkpoint metrics and never aggregates the scorer's failure composition: the loop optimises what it can see, and what it can see does not include the cause of F4". If you run a self-evolving harness, the fix is direct — feed the failure composition into the proposal model.
Finally, the measurement protocol is worth stealing even if you never touch harnesses: four-rollout means, a per-rollout standard deviation of 0.023, a stated noise floor (differences below 0.035 are noise), a fresh anchor after every accepted change, and comparisons drawn within one night. On noisy long-horizon tasks, that is the difference between a real signal and a rerun artifact.
Caveats
This is an unreviewed preprint with no code or data release, and none of the numbers have been independently replicated. Everything above is the authors' own reporting.
The paper does not report matched-budget test-time scaling comparisons in its abstract or contributions. That matters because prior work — Wang et al., "Rethinking the Evaluation of Harness Evolution for Agents" (arXiv:2607.12227) — argues that many harness-evolution gains are just repeated sampling, and found harness evolution fails to beat test-time scaling on Terminal-Bench 2.1. The paper's held-out tasks and fresh anchors address the data-leakage half of that criticism, but not the sampling-budget half.
There is also a protocol gap: llm-stats.com lists Qwen3.5-9B at 0.180 on DeepPlanning against the paper's 0.32 base score. The paper runs its own harness on its own 72-task held-out split, so its scores are not comparable to the public leaderboard.
Some reported deltas also sit close to the paper's own noise floor: full fine-tuning adds about +0.04 over LoRA, against a stated 0.035 floor.