Parallel agent teams beat Best-of-N and one big team: the mean–variance trade-off of test-time compute, made concrete
With a verifier, a fixed agent budget is a trade-off between raising the mean of each sample (collaboration) and buying variance to select from (sampling). Teams of ten balance both.
A new preprint from Duke, Modal Labs, Princeton, and NYU argues that when a verifier can score every attempt, the best way to spend test-time compute is many small collaborating teams sampled in parallel, keeping the best result — not Best-of-N over independent agents, and not one giant team. Across 12 long-horizon tasks with a 27B model, teams of ten beat parallel agents on 10 of 12 tasks at a 100-agent budget, and beat a single team of 100 on all four tasks where such large teams were run.
What shipped
The 29-page preprint, "Test-Time Compute Scales Better with Parallel Agent Teams" (arXiv:2611.09798, submitted 6 Nov 2026), is by Junlin Wang, Haozhe Chen, Ang Li, Rahul Chalamala, and Zhuang Liu. It is not peer-reviewed.
The setup is deliberately simple. Every agent runs in the Codex harness in its own sandbox and improves a submission for 12 hours per task (six for Browser Exploit). Teams are flat: each agent keeps its own sandbox and context but shares a forum, with no roles and no manager. A team's score is the best verified submission of any member. Given a budget of B agents, that means N = B/k teams of size k: k=1 is parallel agents, k=B is one big team, and the paper's default is teams of ten.
The main model is Qwen3.8-27B, with DeepSeek V4.1 Flash for replication and GPT-5.6-sol as the frontier comparison point. The 12 tasks span six benchmarks and nine domains, from GPU kernel optimization (KernelBench) and formal proof (Carleson Formalization) to exploit development (ExploitBench) and structural design (TopOpt).
The headline results, all at a 100-agent budget:
- Teams of ten beat parallel agents on 10 of 12 tasks. The two exceptions are Kimi-Linear Decode and Active Learning (the latter by a small margin).
- Teams of ten beat one team of 100 on all four tasks where big-team runs were done.
- At equal dollar cost, 27B teams beat GPT-5.6-sol on 4 of 8 tasks that had at least five frontier-model runs.
Averaged over the 12 tasks, teams of ten score about 10% higher than parallel agents at a budget of 10, and 27% higher at a budget of 100. Ten teams of ten beat one team of 100 by 8% to 30%. The largest single win is Carleson Formalization, where the best of 100 independent agents scores 19.0 versus 54.8 for the best team (scored as the fraction of Lean proof placeholders closed).
Why it matters
The paper's real contribution is making the mean–variance trade-off of Best-of-N concrete. The expected best of N samples equals the mean score of one sample plus a variance gain that grows with N. The paper's worked example: five runs scoring 2, 4, 5, 7, 9 give an expected best of 5.4 with one sample (the mean), 7.1 with two, and 9 with five.
That decomposition explains why both extremes waste compute:
- Parallel agents maximize the variance gain but each sample has a low mean. They saturate fast — 50 agents already capture 81% to 100% of the gain that 100 agents reach.
- One big team has a high mean but is a single sample, so it throws away the variance gain entirely. Worse, it degrades as it grows: on all four tasks tested, the score dropped going from 64 to 100 agents, and on two tasks one team of 100 scored below the best of 100 independent agents.
Parallel small teams sit in the middle: collaboration raises the mean of each sample, and sampling many teams restores the variance to select from. The rule of thumb for anyone running agent systems with a verifier: spend a fixed agent budget on several small collaborating teams (around ten) rather than one big team or all-independent agents.
Caveats
This is a self-reported preprint with no independent replication, and several limits matter before you adopt the pattern:
- The code isn't available. The linked repository (github.com/modal-projects/parallel-agent-teams-scaling) returns a 404, so the method can't currently be reproduced from the paper's own artifacts.
- The big-team comparison is thin. Only four tasks had single-team runs, with just 1–5 runs per team size, so the "big team degrades from 64 to 100 agents" result rests on limited data.
- The cost comparison is accounting-dependent. Qwen is priced by GPU time and GPT-5.6-sol by tokens, and the paper notes the 4-of-8 "equal dollar cost" claim shifts with the pricing model.
- Comparisons are at equal agent counts and fixed per-agent session time, not equal tokens or equal compute — the paper flags this itself.
- The statistical model is an approximation. It assumes roughly independent, normal scores and predicts the direction of the 100-agent team gain on 9 of 12 tasks, failing where failures create long tails or scores pile up near a ceiling.
- The whole method presumes an automatic verifier that can score every attempt. Without one, "keep the best team" is undefined.