acceptodds
Under review as a conference paper at ICLR 2027

More Compute Is Not More Evidence: Mitigating Model-Specific Error Bias at Test Time

Abstract

Test-time scaling methods implicitly assume that additional computation from the same model provides independent evidence. We show that this assumption fails within a single reasoning model: errors are correlated across repeated samples, across reasoning depth, and in the model’s own self-evaluation, so more compute from the same model is not more evidence. We introduce **DIA (Disagreement-triggered Independent Arbitration)**, which treats disagreement between an intermediate-stage and the final-stage prediction as a per-question signal of reasoning instability and resolves it by asking a small panel of heterogeneous models to solve the question independently, without seeing either candidate, and aggregating their answers directly. Across seven benchmarks and two base models, DIA outperforms full chain-of-thought in **12 of 14 settings**, including a **7.00% gain on MuSiQue**, while activating on only ** 17% of examples** and using **up to 47.2% fewer tokens** than a multi-agent baseline. Ablations show that both independent solving and solver heterogeneity are necessary for these gains.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-agent system, language agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_D97wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors argue that scaling test-time compute has three main types of error bias: (1) repeated sampling can reproduce a wrong majority answer, (2) longer reasoning can preserve an initially incorrect prediction, and (3) self-evaluation systematically favors later and more verbose thinking. Based on these, this paper presents DIA, a pipeline that uses disagreement as the trigger to choose when to obtain external evidence.

Critical strengths
  1. The writing is clear and Figure 1 does a great job of illustrating the main three model-spefic error biases, though they were mainly built on top of existing work.

  2. The main contribution of this work is, in my opinion, putting together the main challenges and their solutions for scaling test-time compute in a clean and organized way. The method itself is simple and intuitive.

Critical weaknesses
  1. Since scaling test-time compute is the main theme, I would expect compute-matched baselines like self-consistency [1], Best-of-N [2], simple test-time scaling (s1 [3]). It is also important to match the compute budget.

  2. Adaptive reasoning should also include Adaptive Self-Consistency [4], which adaptively decides how many samples to use for each query based on a simple beta distribution.

  3. The margin is small across datasets and, on average, only outperforms the best baseline by less than 1.5%.

References:

Questions for authors
  1. I'm not sure "More compute from the same model is therefore not more evidence" is an accurate description. I understand the points about "model-specific error bias persists across generation, reasoning, and self-evaluation", but this only tells us "more compute does not resolve model-specific error bias" not "more compute =/= more evidence". What does evidence mean here?

  2. Also, more compute does, in general, improve the overall accuracy, so the expected error goes down with increased compute. How would the authors frame that? In lines 156 and 157, the Takeaway says: "Increasing the sampling budget of the same model may repeatedly reinforce an existing error mode rather than introduce independent evidence." But on average, test-time scaling methods like self-consistency [1], Best-of-N [2], simple test-time scaling (s1 [3]) does reduce errors.

  3. There's also work that tries to do finer refinement and identify error steps to enable better scaling [4], I am curious about what the main distinction is at a conceptual level, and what the performance comparison will look like.

References:

Reviewer_UJpHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

DIA requests three different models to solve a question independently if the base model’s intermediate and final answers differ. These models do not view the candidate answers; DIA then normalizes and combines their outputs. The goal is to reduce bias from self-assessment and exposure to existing candidates. Over seven benchmark subsets and two Qwen3 base models, the paper reports improvements over CoT in 12/14 settings. It also explores how results vary with different trace presentations, homogeneous solvers, larger base models, and a bigger solver panel.

Critical strengths
  1. I like the simplicity of using disagreement to decide when to seek independent answers. Keeping the solvers unaware of the existing candidates makes the procedure straightforward to understand. Comparing full traces with length-matched summaries also illustrates how presentation impacts judgment.
  2. The experiments cover various task types and involve larger base models. The finding that expanding the solver panel does not help in these experiments is also useful, since it shows a limit to the benefit of adding more solvers.
Critical weaknesses
  1. I am primarily concerned about whether CoT and DIA employ identical base-model trajectories. On 198 GPQA items, the 4B model gains about 17 correct answers while activating arbitration on approximately four items; for the 8B model, the counts are four and two, respectively. If untriggered items have the same aF, the net gain cannot surpass the number of triggered items. Separate generation of CoT outputs or a reporting error might account for the discrepancy, but separate generation would mix base-answer variation with arbitration effects. A paired comparison using identical cached, completed trajectories would clarify this: CoT would yield aF, while DIA would add arbitration only when its disagreement rule activates. If the experiments already share these trajectories, per-item records should elucidate the two tables. This unresolved issue is the primary reason for my current recommendation (Tables 2 and 5).
  2. Replacing the heterogeneous panel with three Qwen8B samples changes solver capability as well as diversity. I would like to see random activation and alternative uncertainty triggers using the same solver pool, matched at actual cost, along with a cost-matched comparison to invoking only the strongest solver. These controls would help determine what the disagreement rule contributes. The small evaluation subsets also require paired uncertainty estimates (Table 4).
Questions for authors
  1. Are CoT and DIA using precisely identical completed base-model trajectories? If not, could you conduct a paired evaluation with a shared cache, where CoT returns aF and DIA uses the same aE and aF, adding arbitration only when they diverge? The outputs should match on untriggered items. Please provide paired confidence intervals for the gains and the numbers of answers corrected or made incorrect, along with per-item GPQA aE, aF, activation flags, and final outputs to reconcile Tables 2 and 5.
  2. At equivalent actual costs, does disagreement-based activation surpass random activation or other uncertainty triggers using the same panel? How does it compare with only invoking the strongest solver?
  3. What thresholds are applied to extract intermediate answers, and how are equivalent answers recognized? What occurs when all three solvers disagree or produce invalid outputs?
Reviewer_2Vr3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates model-specific error biases in test-time scaling for large language models. The authors identify three related phenomena: correlated errors under repeated sampling, persistence of initially incorrect predictions during extended reasoning, and a preference for longer reasoning traces during self-evaluation.

Motivated by these observations, the paper proposes Disagreement-triggered Independent Arbitration (DIA). The method detects disagreement between intermediate-stage and final-stage predictions and selectively invokes a heterogeneous panel of external models to solve the question independently. Their answers are then aggregated through answer-level voting. Experiments across seven benchmarks and multiple base-model scales demonstrate improvements over standard chain-of-thought reasoning and several adaptive or multi-agent inference baselines.

Critical strengths
  • Interesting empirical observations. The paper provides useful evidence of systematic error patterns in test-time reasoning, including wrong-answer concentration, initial-answer persistence, and self-evaluation bias. These observations highlight important limitations of simply increasing computation within the same reasoning model.

  • Simple and practical inference framework. DIA combines disagreement-based routing with independent external solving, avoiding unnecessary multi-agent communication and centralized candidate evaluation. The framework is straightforward to implement and does not require additional training.

  • Broad empirical evaluation. The paper evaluates the proposed approach across multiple reasoning domains and model scales. The results show consistent improvements over standard chain-of-thought reasoning in most evaluated settings, and the ablations provide useful evidence regarding the effects of independent solving and heterogeneous model selection.

Critical weaknesses
  • Limited methodological novelty beyond selective majority voting. My primary concern is that the proposed method is essentially a variant of ensemble-based majority voting. DIA uses disagreement between intermediate and final predictions as a routing signal and, when triggered, invokes a heterogeneous set of models whose answers are aggregated through voting. While this is a reasonable engineering design, its core components—selective routing, independent model inference, heterogeneous ensembles, and majority voting—are already well-established. Candidate-blind solving also closely resembles standard independent ensemble inference. The paper does not sufficiently establish what fundamentally new reasoning or aggregation mechanism is introduced beyond combining these existing components. Consequently, the algorithmic contribution appears relatively limited despite the interesting motivating observations.

  • Insufficient comparison with standard majority voting under matched computational budgets. Although the paper compares DIA against multi-agent baselines, these baselines involve additional communication or centralized judging and therefore are not the most direct comparisons for the proposed approach. A more appropriate baseline would be independent majority voting using the same heterogeneous solver pool, evaluated under matched token or computational budgets. Without such comparisons, it is difficult to determine whether the improvements arise from the proposed arbitration mechanism or simply from selectively invoking additional models.

  • Accuracy improvements come with substantially increased token consumption. DIA improves accuracy over standard CoT, but its average token consumption is approximately 2.6–2.7 times that of CoT. Although it is more token-efficient than the evaluated multi-agent baselines, the additional computational cost remains substantial relative to single-model inference. Furthermore, the improvements over MAS-Exchange are relatively modest, particularly for Qwen3-8B. The paper would benefit from a more comprehensive accuracy–cost analysis, including comparisons under matched token budgets and, ideally, actual inference costs across heterogeneous models.

  • The contribution of model heterogeneity is not fully isolated. The homogeneous-panel ablation replaces the heterogeneous external solvers with repeated samples from Qwen3-8B. However, this comparison changes both model diversity and individual solver capability. Since the heterogeneous panel includes substantially different and potentially stronger models, the observed improvement cannot be attributed solely to reduced error correlation. More carefully controlled experiments are needed to distinguish the effects of solver diversity from solver strength.

  • The disagreement-based trigger does not fully address the identified error persistence. The paper emphasizes that model-specific errors can persist across reasoning stages. However, DIA activates external arbitration only when the intermediate and final predictions disagree. When both stages produce the same incorrect answer, the method does not intervene. This raises questions about whether reasoning-stage disagreement is an adequate signal for detecting the systematic error modes emphasized in the motivation. Comparisons with alternative uncertainty or instability signals would strengthen the justification for this design.

  • Limited theoretical insight into the proposed algorithm. The truth-margin analysis provides an intuitive explanation of how voting accuracy depends on solver correctness and wrong-answer concentration. However, the main results largely follow from standard concentration arguments under independent voting assumptions. The analysis does not directly establish the optimality or superiority of disagreement-based routing, heterogeneous solver selection, or the specific aggregation procedure used in DIA. Its contribution therefore appears primarily explanatory rather than offering new theoretical insights into test-time reasoning.

Questions for authors

Please refer to the Weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.