TRACE: agent-eval score changes are measurements, not verdicts — 15–36% of identical reruns flip outcomes, and two frontier judges disagree 57% of the time
A single-author Prime Intellect preprint adds paired-probe testing, counterfactual rescoring, and rerun controls — renaming tools alone drops a scripted agent's score by 0.250 with no behavioral change.
TRACE, a diagnostic protocol from AI researcher Radhika Gaonkar (Prime Intellect), turns an agent-evaluation score change from a verdict into a testable explanation — and its numbers are a warning for anyone who reads verifier scores as capability. In the paper's controlled suite, renaming a scripted agent's tools drops its score by 0.250 while the agent performs exactly the same operations; identical reruns flip 15–36% of task outcomes; and two frontier LLM judges disagree with each other on 57% of the records they both graded.
What shipped
The motivating observation is simple and uncomfortable: verifier scores are now simultaneously benchmark metrics and training rewards for LLM agents, so "a change in score is routinely read as a change in capability. It may instead reflect a change in the evaluation." TRACE (the paper is a preprint on arXiv, arXiv:2610.11678, posted October 8, 2026) is a four-step protocol for deciding which of those readings is true.
The mechanics: pick a task, an agent, and a verifier, then apply a mutation — a targeted change to one part of the evaluation. Run the paired test holding the base task, policy, and seed fixed; compare scores; inspect trajectories to see whether the agent's behavior actually changed; and when it did not, run counterfactual rescoring: correct the suspected scoring defect and rescore the same trajectory without rerunning the agent. That final step separates whole-system sensitivity (score moved) from evaluator sensitivity (the verifier changed its judgment on the same record) — only the latter is direct evidence about the verifier.
The paper defines three score-level diagnostics. The verifier robustness gap (VRG) is original minus mutated score. The mutated success rate (MSR) and mutation transfer rate (MTR) ask whether runs still pass the verifier after the mutation (pass threshold τ = 0.65), with MTR restricted to pairs whose original run passed. A proxy–true divergence (PTD) compares a visible reward proxy against a simulator-defined quality signal on the 12 tasks that have proxy channels.
Study 1: known failures in a controlled suite. The environment is ResearchOpsEnv, 25 synthetic research-operations tasks (correct a learning rate, prevent data leakage, retrieve documents, edit configs, run simulated experiments, submit reports), with a composite verifier that combines report-target terms, tool-name coverage, constraint and efficiency checks, and a hack-risk penalty. Three scripted policies run it: careful (applies the intended fix), overfit (targets scoring cues), and shortcut (manipulates proxies). The archive holds 1,575 records: 25 tasks × seven conditions × three policies, each under three seed labels.
Renaming tools while preserving dispatch and arguments produces careful's largest mean gap, 0.250 — with all other probe gaps at most 0.023 in either direction. The paired trajectories are identical: same observations, same operations. Restoring the original names at scoring time closes the entire gap (0.250 → 0.000), which pins the defect on a verifier matching tool names rather than checking operations. In a leaderboard, that drop reads as a regression that never happened; as an RL reward, it would penalize correct behavior. The same mutation also exposes a genuine behavioral failure in a second scripted policy — evidence that the protocol can distinguish a verifier defect from a real capability gap.
Study 2: public τ²-bench, real LLM agents. The external check runs four LLM agents on frozen public τ²-bench tasks — Sierra's dual-control benchmark where agent and simulated user act with distinct tools in a shared world, and success is graded as pass¹, average task success over single runs. An initial 30-task study produced mixed reward changes whose one clear effect did not replicate. A follow-up on 88 new tasks with repeated runs per condition found that renaming tools or reformatting tool outputs left reward unchanged to within ±0.10 for seven of eight agent–change pairs — meaning that for real agents, meaning-preserving evaluation edits mostly do not move scores. The eighth case matters: tool names deliberately engineered to mislead lowered every agent's reward by 0.20–0.44. That is the positive control working: a change known to matter is detected, so a null result under TRACE is interpretable, not just a broken pipeline.
The noise floor. Identical reruns flip 15–36% of task outcomes, so a single-run comparison cannot separate a presentation effect from run-to-run variation. The judge audit raises the same problem at the scoring layer: two frontier LLM judges each give consistent verdicts when a fixed trajectory is presented differently, yet they disagree with each other on 57% of the same records — in large part, the paper reports, because one grades procedure while the other grades outcome.
Why it matters
Three findings survive the caveats and change how agent evals should be built and read.
First, there is a cheap, standard control that most eval harnesses skip: identical reruns. If your rerun flip rate is in the 15–36% band the paper measures, single-run score deltas smaller than that band should not be reported as capability changes. The paper's numbers are a direct argument for budgeting reruns.
Second, the 0.250 tool-renaming result is not an exotic failure: tool-name coverage is a component of the paper's own composite verifier, and the mechanism — a verifier matching tool names rather than checking operations — is precisely what TRACE is built to surface. When verifier scores feed RL, as the paper notes they now do (citing the DeepSeek-AI et al., 2025, and Da et al., 2025, training-reward line), a name-matching verifier penalizes correct behavior; the plausible downstream effect is that training gets steered toward preserving names and formats. TRACE's counterfactual-rescoring step is the diagnostic that tells you a reward was wrong — before you spend a training run optimizing against it.
Third, the non-replication result is itself a finding about methodology. A 30-task pilot showed a "clear" effect that vanished at 88 tasks with repeated runs. The smaller the benchmark, the more of a score delta is noise — and the paper's follow-up design (repeat runs per condition plus a positive control) is the minimal skeleton of an eval infrastructure that can tell the difference.
Caveats
This is a single-author preprint, v1, posted on October 8, 2026 — not peer-reviewed and unreplicated. Study 1 uses scripted policies with prescribed behavior, which the paper itself calls "controlled examples, not models of what a learned agent would discover." Each probe family is implemented by one fixed template, so results characterize "these specific transformations, not the full space of changes in each family." The τ²-bench study is modest in scale — 30 tasks in the pilot, 88 in the follow-up — and the abstract names neither the agent systems nor the judge models, so treat those results as specific to whatever setups the paper used. The arXiv listing does not link a public repository, and the paper's references to "artifacts" (which report pass thresholds 0.55 and 0.75 as well as 0.65) could not be verified from the paper page. The headline numbers — 0.250, ±0.10, 0.20–0.44, 15–36%, 57% — are the author's reported values, not independent measurements.