More compute from the same model is not more evidence: DIA's disagreement-triggered heterogeneous arbitration beats full CoT in 12 of 14 settings
A new preprint shows test-time scaling's independence assumption fails inside a single model — disagreement-triggered, heterogeneous arbitration is the fix.
A new arXiv preprint argues that test-time scaling's working assumption — that extra compute from the same model yields independent evidence — fails inside a single reasoning model. A disagreement-triggered panel of heterogeneous models beats full chain-of-thought in 12 of 14 settings while activating on only ~17% of examples.
What shipped
Test-time scaling methods — chain-of-thought prompting, repeated sampling with majority vote, adaptive early exit, multi-agent reasoning — convert extra inference-time compute into accuracy gains. The authors (Ming Shen of Shanghai Jiao Tong University and the Shenzhen Loop Area Institute, and Tianjun Yao of SLAI) argue these methods share an implicit assumption: that additional computation from the same model provides independent evidence. They show the assumption fails within a single reasoning model, in three measurable ways.
Generation bias under repeated sampling. When majority-vote self-consistency (SC@20) produces a wrong answer, the erroneous samples don't spread across alternatives — they collapse onto a single wrong answer. The paper measures C_q, the fraction of incorrect samples assigned to the most frequent wrong answer, and finds that on many failed questions all incorrect samples converge on one answer (C_q = 1). Concentration rises with model scale and is highest in reasoning-oriented models. The takeaway: more sampling from the same model reinforces an existing error mode instead of introducing independent evidence.
Initial-answer persistence. A model asked to answer directly, then to reason, often keeps its initial answer — including when it's wrong. "Error Retained," the fraction of initially incorrect answers for which full CoT returns the same incorrect answer, reaches 61.9% on MuSR for Qwen3-8B and 58.7% for Qwen3-Next-80B-A3B-Instruct. Reasoning can elaborate an existing error without supplying the independent evidence needed to overturn it.
Longer-reasoning preference bias. Asked to choose between its own intermediate-stage and final-stage answers, a model picks the later, longer answer in over 95% of disagreements — and stays confident even when that choice is wrong. The authors tie this to previously documented length and verbosity biases in LLM-based evaluation.
The fix is DIA (Disagreement-triggered Independent Arbitration). When intermediate-stage and final-stage predictions agree, the answer is returned directly. When they disagree — a per-question signal of reasoning instability — a small panel of heterogeneous models solves the question independently, without seeing either candidate, and their answers are aggregated directly. The design encodes three principles: acquire extra evidence selectively (only on disagreement), from complementary sources (heterogeneous models), combined by direct aggregation rather than a judge.
Across seven benchmarks and two base models (Qwen3-4B and Qwen3-8B), DIA outperforms full chain-of-thought in 12 of 14 settings, including a 59.0% → 66.0% gain on MuSiQue. It activates on only ~17% of examples and uses up to 47.2% fewer tokens than a fully-connected multi-agent baseline that invokes every agent on every question. A baseline that routes independent agent outputs through a centralized judge underperforms direct answer aggregation (60.67% vs 66.00% on MuSiQue with Qwen3-4B). Ablations show both independent solving and solver heterogeneity are necessary for the gains.
Why it matters
For anyone building reasoning systems, the headline is the falsified assumption, not the specific numbers. Self-consistency, best-of-n, and majority voting price extra samples as if each were an independent vote. This paper shows that within a single model, errors are correlated across samples, across reasoning depth, and in self-evaluation — so the marginal value of another sample from the same model is lower than the independence assumption implies, and the value of heterogeneous evidence is higher.
Three concrete implications:
-
Disagreement is a cheap instability signal. DIA's trigger — disagreement between an intermediate and a final prediction — activates on only ~17% of examples. You don't need an ensemble on every question; you need a signal for when the model is unstable, and stage disagreement is a free byproduct of reasoning.
-
Don't let the model judge its own candidates. The >95% later-stage preference means a model asked to pick between its own answers systematically favors the longer one. A judge-based baseline — independent agent outputs routed through a centralized judge — underperforms direct aggregation (60.67% vs 66.00%), which the authors attribute to the aggregation protocol.
-
Heterogeneity is the point. The ablations show independent solving alone isn't enough; the panel has to be heterogeneous. If you're going to spend tokens on arbitration, spend them on models with different error tendencies, not more samples of the same one.
Caveats
This is a preprint (arXiv:2610.23213v1), not peer-reviewed, and no code release is mentioned. All numbers are author-reported with no independent replication yet. The base models are small — Qwen3-4B and Qwen3-8B — and the 12-of-14 win is against a single baseline (full chain-of-thought), with the headline gain on MuSiQue (59.0% → 66.0%). Treat the headline numbers as a strong signal that the independence assumption is false, not as a recipe to copy verbatim.