Richland Technology
Richland Technology
← Back to Blog
llmagentsevaluationresearchself-improvement

Agent evolution's real bottleneck is noisy selection, not weak proposals — a 10M-call study finds selectors at coin-flip accuracy and headline gains that are mostly grader shortcuts

Seven methods, six models, 1,030 runs: selection picks the better artifact at coin-flip accuracy, and a single added sentence reproduces a median 90% of reported gains.

Richland AI Desk·21 November 2026·5 min read
Agent evolution's real bottleneck is noisy selection, not weak proposals — a 10M-call study finds selectors at coin-flip accuracy and headline gains that are mostly grader shortcuts

A 10-million-call study of agent evolution argues that self-improving agents stall not because proposers fail to write better prompts, skills, or memories, but because noisy evaluations cannot reliably tell a genuine improvement from a lucky rollout. It also shows that many headline "self-improving" gains are grader shortcuts, not real progress.

What shipped

The paper, "The Resolution Hypothesis: Agent Evolution's Key Is Identifying Improvements" (arXiv:2611.23068), is a preprint from ShanghaiTech, ByteDance, BIGAI, and SUSTech. The authors ran what they describe as the largest reproducibility study of agent evolution to date: seven evolution methods across six models, 1,030 independent evolution runs, more than 2.1M test trajectories, and over 10M LLM calls.

The seven methods split into two families: text-artifact methods (GEPA for prompts, SkillOpt-Lite and Trace2Skill for skills, ReasoningBank for memory playbooks) and harness methods (HarnessLens, Meta-Harness, AHE for agent scaffolding code). The six models are GPT-5.4 nano, GPT-5.6 Luna, GPT-5.6 Terra, DeepSeek-V4-Flash, Kimi-K3, and GLM-5.2. The variance study runs on ALFWorld, SpreadsheetBench, OfficeQA, and Terminal-Bench v2.1.

The Resolution Hypothesis is tested at the two places evaluation scores are used: choosing candidate artifacts, and ranking whole methods.

At the candidate level, selection is at or below chance. In head-to-head comparisons, execution-based selection (one rollout per artifact) picks the better of two candidate skills 49.7% of the time — statistically indistinguishable from the 50% coin flip. Native SkillOpt-Lite and GEPA selection (two visible rollouts) land at 48.6%, and three LLM-judge setups reading the full trajectory score 45.4%–49.0%, below chance. The consequence is that regressions — candidates worse than the incumbent — make up 30.5% of the candidate pool but 62.1% of what actually gets accepted. Selection enriches for harmful artifacts.

The authors stress this is not a proposal problem. A pool of 100 candidate skills sampled once already contains genuine cross-task winners: Quality@100 reaches 61.8% versus 49.2% at K=1, and exceeds an equal-quality null pool by 3.3–3.4 percentage points. The skills are there; the selector just cannot tell them apart.

At the method level, published rankings often rest on luck. Across 171 benchmark–model results in the main tables of nine agent-evolution papers, the new method's median lead over its strongest competitor is 3.4 percentage points (IQR 1.2–6.6). None of the 171 reports a variance estimate, and only six average more than one evolution run. A single "evolve once, evaluate once" run has a typical standard deviation of 1.7–4.1 points depending on benchmark. For over 60% of same-model, same-benchmark configuration pairs, the gap between two methods is smaller than one single-run standard deviation.

A one-run ranking simulation finds that a single run identifies the truly best of four text-artifact methods only 33% of the time on Terminal-Bench (random would be 25%), and 51–73% on the other three benchmarks. To detect the median 3.4-point margin at 80% power, the authors estimate you need 5 runs on SpreadsheetBench, 10 on OfficeQA, 19 on ALFWorld, and 23 on Terminal-Bench — and 181 runs at the lower-quartile margin on Terminal-Bench.

The sharpest finding is the grader-shortcut audit. Across 27 model–benchmark pairs with reported gains above 3 points, adding a single hand-picked sentence to the unevolved prompt recovers a median 90% of the gain. After the authors repaired the graders, the median gain over 12 benchmarks fell from 17.6 to 0.8 percentage points — and re-running evolution under the repaired graders did not restore it.

Independent reviewers reproduced four of these grader bugs against the methods' public code:

  • SpreadsheetBench: SkillOpt-Lite's openpyxl grader reads formulas with data_only=True, which returns empty cells. With LibreOffice recalculation, the GPT-5.4 nano baseline jumps from 33.5% to 58.0% and the reported gain collapses from +31.1 to +6.7 points.
  • OfficeQA: exact-match grading; the GPT-5.6 Luna gain goes from +21.1 to −0.1 after repair.
  • LiveMathematicianBench: a "stronger result" meta-option appears in 85 of 177 questions in SkillOpt-Lite's split and is correct in all 85.
  • AIME 2025: GEPA's int(prediction.answer) parser; the baseline rises from 32.0% to 85.8% with LaTeX-stripped equivalence.

Why it matters

For anyone building self-improving agents, the actionable takeaway is that a few-point gain from a single evolution run is not evidence. The paper recommends reporting both evaluation noise (σ̂) and run-to-run variance (τ̂) alongside point scores, so a claimed margin can be checked against the evaluation's resolution. It also argues for evaluating proposers and selectors separately on fixed candidate pools, rather than only as a coupled loop.

The grader audit is the most directly usable part: before trusting a skill the loop wrote, read the grader it was scored against. The authors' "one-rule test" — does a single suspicious sentence reproduce the gain? — is cheap to run and caught a median 90% of reported gains here. The four verified grader bugs are concrete enough to reproduce locally today.

The paper also points at a design direction: lower-noise intermediate process checks. In preliminary analysis, some process-check rates had 13×–57× lower evaluation noise than terminal pass/fail verdicts. The authors frame the conclusion as "developing higher-resolution feedback may be as important as developing more powerful evolution algorithms."

Caveats

This is a preprint, not peer-reviewed, and the core candidate-level numbers (49.7% selection accuracy, the ranking simulation, the in-loop re-evaluation) depend on trajectory data the authors say will be released "upon the acceptance of this paper." As of now, no code or trajectories are public, so those figures cannot be independently recomputed. The grader-shortcut findings were independently reproduced against the methods' own public code, but the selection-accuracy and ranking results remain vendor-reported.

The paper's own scope limits are worth noting: the candidate-level evidence covers Terminal-Bench v2.1 skills only, pools are n=100, trajectory re-evaluation covers just two methods on two models, and only weight-frozen evolution was studied — whether harness/program evolution faces the same constraint was not tested experimentally. Independent reviewers also flagged that the 62.1% false-acceptance figure carries a very wide credible interval (±42.9 points, per the paper's Table 2), and that the semantic graders used (DeepSeek-V4-Flash, GPT-5.6 Luna) are two of the models under evaluation.

Sources

Start a Conversation

Ready to build something like this?

Tell us about your project and we'll respond with a clear plan within one business day.

Tell Us About Your Project

Takes 3 minutes. We read every submission personally.