acceptodds
Under review as a conference paper at ICLR 2027

Reasoning Concentrates Errors, and Self-Consistency Never Notices

Abstract

Self-consistency assumes that independent samples disagree when a model is unsure, so agreement is evidence of correctness. Holding weights fixed and toggling only a reasoning mode, over five benchmarks and 74,944 samples, we show that reasoning concentrates a model’s errors: the probability that two independently drawn wrong answers coincide rises in all ten dataset–scale comparisons (p = 0.00098), and in nine of nine after restricting both arms to the problems each gets wrong. Where the answer space is unbounded, reasoning cuts the distinct answers produced to 0.43–0.65 of the non-reasoning count; where it is bounded, both arms hold an identical option set and reasoning concentrates mass on it instead, which no positional prior can explain at fixed weights. The aggregate cost is smaller than the mechanism predicts, because reasoning also shrinks the set of problems where answer diversity can decide anything, in ten of ten cells and by 2.7×; normalized for available headroom, both arms convert a quarter of it in domain. Confidence weighting does not recover what is left. Across 280 method– dataset–model combinations on eight models and five benchmarks, not one beats plain majority voting after correction; weighted voting agrees with it on 98.5% of problem–method pairs and is right 56.3% of the time on the rest; and a signal’s direction can invert within fixed weights, with answer log-probability predicting correctness when reasoning is off and error when it is on. A learned six-signal combination gains nothing out of domain. Confidence signals should be evaluated on decisions, not on discrimination.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
self-consistency, uncertainty quantification, confidence estimation, large language models, reasoning models, selective prediction, calibration, test-time compute, chain-of-thought

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_csCmOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how explicit reasoning affects self-consistency and confidence estimation in language models. Its main controlled experiment compares Qwen3 models with reasoning enabled and disabled while keeping the model weights fixed. The paper introduces a wrong-answer collision measure and reports that enabling reasoning consistently increases the concentration of incorrect answers across the evaluated datasets and model scales.

The paper then studies the downstream implications of this observation for self-consistency. It analyzes accuracy gains from repeated sampling, introduces a "danger zone" to characterize problems where voting can potentially change the outcome, evaluates confidence-weighted voting across multiple confidence signals, and further studies output length, answer-token probability, signal transfer, and calibration across models and datasets.

The main empirical finding is that reasoning makes incorrect answers more concentrated. At the same time, reasoning also reduces the number of problems on which voting can materially affect the outcome, and these effects largely cancel in aggregate self-consistency accuracy. The paper additionally reports that confidence-weighted voting does not significantly outperform majority voting under its experimental setting.

Critical strengths

1. The same-weight reasoning on/off comparison is a useful experimental control.

The strongest part of the paper is the comparison between Qwen3 thinking and non-thinking modes under identical weights. This removes many confounds that would arise from comparing independently trained reasoning and non-reasoning models. The observation that wrong-answer collision increases consistently across the paired settings is therefore interesting and deserves attention.

2. The paper is unusually transparent about failed hypotheses and statistical corrections.

The authors explicitly report analyses that did not support their original expectations, including the headroom issue in the self-consistency gain comparison, and they distinguish confirmatory from supporting evidence. The paper also reports multiple-testing corrections, matched analyses, answer rates, and pre-registered predictions. This level of transparency is valuable.

Critical weaknesses

1. The paper lacks a sufficiently coherent and stable central scientific claim.

The paper contains a large number of analyses, but I found it difficult to identify a single research question that is consistently carried from the motivation to the conclusions. The initial observation that reasoning increases wrong-answer concentration is clear. However, the expected downstream consequence is repeatedly revised when subsequent analyses do not support it.

For example, the apparent threefold difference in self-consistency gain is later shown to be an artifact of unequal accuracy headroom. After normalization, the two conditions convert essentially the same fraction of available headroom. The paper then introduces the "danger zone" to explain this result, followed by a rescue-rate analysis that the authors themselves describe as supporting rather than confirmatory. The narrative then branches further into confidence-weighted voting, output length, hedging, answer-token probability, cross-family transfer, and calibration.

I appreciate that the authors transparently revise conclusions when analyses fail. However, transparency about failed hypotheses is not the same as having a coherent scientific contribution. In its current form, the paper reads more like a sequence of exploratory analyses, where unsuccessful hypotheses motivate new metrics and new secondary findings, than a focused investigation of one well-supported central claim. After reading the full paper, I find the contribution difficult to characterize beyond the initial observation that reasoning increases wrong-answer concentration.

2. The strength of the title and broader conclusions exceeds what the controlled evidence establishes.

The title suggests a strong connection between reasoning-induced error concentration and a failure of self-consistency. However, the paper itself shows that after correcting for available accuracy headroom, the aggregate benefit of self-consistency is essentially the same in-domain for reasoning and non-reasoning models. The rescue-rate analysis, which more directly tests whether concentrated errors reduce the ability of voting to recover the correct answer, is underpowered and explicitly reported as non-confirmatory.

Moreover, the strongest causal evidence for the effect of reasoning comes from only two same-weight Qwen3 pairs. The additional Olmo and Gemma comparisons do not provide the same identification because model family, training, and other factors differ. Thus, the large number of generations and downstream analyses should not be confused with broad controlled evidence for the general claim.

I also noticed concrete inaccuracies in the bibliography. For example, "Reasoning about Uncertainty: Do Reasoning Models Know When They Don't Know?" is cited as an EACL 2025 work although the official publication is Findings of EACL 2026, and OLMo 3 is cited as a 2026 work although its paper was released in 2025. These citation errors are not my primary reason for rejection, but they further reduce my confidence in the care with which the recent literature was checked.

Questions for authors

1. What do the authors consider to be the single central scientific claim of this paper?

The initial result that reasoning increases wrong-answer concentration is clear, but several of the expected downstream conclusions are subsequently weakened or replaced after additional analyses. In particular, the raw self-consistency gain difference disappears after headroom normalization, and the rescue-rate evidence is not confirmatory. I would like the authors to clarify which claim they consider the primary contribution and which results provide direct evidence for that claim.

2. What direct evidence supports the "self-consistency never notices" part of the framing?

After headroom normalization, the aggregate self-consistency improvement is essentially identical in-domain between the reasoning and non-reasoning conditions. The more direct rescue-rate analysis is underpowered. If the intended claim is not that self-consistency performs worse, please clarify precisely what "never notices" means operationally and which experiment directly tests it.

3. Can the authors provide broader same-weight controlled evidence beyond the two Qwen3 pairs?

The central causal interpretation relies on changing reasoning behavior while holding weights fixed, which is a strong design. However, only two model pairs satisfy this condition. The remaining cross-family comparisons cannot isolate reasoning from training and model-family differences. Evidence from additional controlled reasoning on/off models would materially affect my confidence in the generality of the central claim.

4 comments
AuthorsOpenReview ↗

Response to Reviewer csCm (1/2)

Thank you for a demanding review; it improved the paper. We address each point.

W1 / Q1 (the central claim). The central claim is: toggling reasoning within fixed Qwen3 weights changes the structure of a model's errors, concentrating them on fewer distinct wrong answers, and this leaves self-consistency's accuracy unchanged while breaking its use as a confidence signal. Three results test it directly. The structural change is measured by the collision index, which rises in all ten dataset–scale comparisons and in nine of nine after restricting both arms to the problems each gets wrong. The accuracy invariance is measured by headroom-normalized self-consistency gain, 0.250 against 0.249 in domain. The confidence failure is measured across 280 method–dataset–model combinations, where no weighting beats plain majority voting after correction. The raw threefold difference in self-consistency gain does vanish under headroom normalization, and that null is part of the claim, not a casualty of it: reasoning concentrates errors, which should hurt voting, and simultaneously shrinks the set of problems where voting can change anything by 2.72×, which helps it. The two effects are opposite in sign and comparable in size, so aggregate accuracy barely moves. We accept that the paper's ordering obscured this, and we have restructured Section 1 to state the claim first and present the cancellation as predicted structure rather than as a correction discovered late.

Q2 ("never notices," operationally). It is a claim about the confidence signal, not about accuracy, and Section 1 now defines it: weighted voting agrees with plain majority voting on 98.5% of problem–method pairs and, on the remaining 1.5% where it dissents, is right 56.3% of the time. Two experiments test it directly: the 280-combination comparison, in which no confidence weighting beats majority voting after correction, and the within-weights inversion of answer log-probability at 8B, which predicts correctness with reasoning off and error with it on. As stated at submission, we do not claim a measurable self-consistency accuracy penalty at this scale, which is why headroom normalization does not undercut the claim.

W2 (title and broader conclusions). We agree, and we have changed the title to "Reasoning Concentrates Errors Within Fixed Weights, and Self-Consistency Never Notices", and scoped the abstract, the claim paragraph in Section 1 and the conclusions to the two Qwen3 pairs on which the controlled evidence rests. The second toggle family we ran during the discussion, Granite 4.2, is inconclusive (Appendix J), so we no longer imply generality beyond Qwen3. We kept "never notices" and now define it in Section 1, as above. As at submission, we do not claim a measurable self-consistency accuracy penalty at this scale. On the rescue rate: we agree it is not confirmatory, and the revision says so. Section 5 reports it as consistent with Section 4 rather than as confirmation (three of the four cells with at least ten qualifying problems in both arms, p = 0.31), and the 13-of-16 pairing count is now qualified because those pairings share models.

(Continued in part 2.)

AuthorsOpenReview ↗

Response to Reviewer csCm (2/2)

Q3 (same-weight evidence beyond the two Qwen3 pairs). We ran the experiment you describe. Granite 4.2 8B serves thinking and non-thinking modes from one checkpoint through the same chat-template switch Qwen3 uses; a test in the released code checks that the thinking template leaves the reasoning block open and the non-thinking template closes it before generation, so this is a second identified control rather than a cross-family comparison. On GPQA-Diamond, 34 problems so far have all eight samples in both Granite arms. On the 12 where both arms err at least twice, collision is 0.590 with reasoning enabled against 0.582 without (+0.008; 95% CI [−0.212, +0.251]); over each arm's own qualifying problems it is 0.649 against 0.610 (+0.039; n = 14/18). This does not resolve the question in either direction: on the same 34 problems Qwen3-8B, whose effect is clear on the full benchmark, is equally unresolved (+0.014; CI [−0.204, +0.242]). A confound specific to Granite also limits the test: on these 34 problems, 34.6% of its thinking traces reach the 32,768-token limit, against 1.5% without thinking, and dropping them leaves too few qualifying problems. Generation continues; as fixed before any result was computed, we will post the result on all completed blocks on 17 November, whatever it shows. On a 126-problem MATH-500 subset, Granite is too accurate for the index to be measured. We also want to state plainly why our Olmo and Gemma arms cannot stand in for this, since you correctly discount them: among non-reasoning models alone, collision on MATH-500 ranges from 0.297 to 0.441 across families, a spread of 0.144, larger than the within-weights effect at 8B (0.379 to 0.509, a difference of 0.130). Between-family variation exceeds the quantity of interest, which is why the answer has to be a second toggle family; Appendix B of the revision quantifies this.

Bibliography. Thank you for checking it; both errors you name were real. Reasoning about Uncertainty is now cited as Findings of EACL 2026, and Olmo 3 as a 2025 arXiv report with the correct author attribution. Prompted by your note, we audited the full bibliography and found three further errors of the same kind: two EMNLP 2025 papers (Tan et al.; Vashurin et al.) were cited as arXiv preprints, and Devic et al. carried the wrong year (2026 for a 2025 preprint). We also completed four entries that were missing a venue or arXiv identifier, and in a final pass restored capitalization lost in five titles and repaired one malformed author list. All are corrected in the revised PDF.

I keep my score at 1. The rebuttal improves framing and transparency but not the evidential base: the central claim still rests on two Qwen3 weight pairs, Granite is inconclusive by the authors’ own report, and “never notices” describes confidence-weighted voting rather than self-consistency accuracy. The paper remains a sequence of exploratory analyses rather than a coherent, well-supported contribution.

AuthorsOpenReview ↗

Response to Reviewer csCm

Thank you for reading the rebuttal and for replying. Three brief points of fact, for you and the AC.

On "never notices." We agree the claim is not about self-consistency's accuracy, and we do not make one. It is about the confidence self-consistency itself produces: the agreement fraction among its k samples. Within fixed weights that signal is less informative with reasoning on, in four of four weight-pair and split comparisons (agreement AUROC 0.918 against 0.952 in domain and 0.805 against 0.869 out of domain at 8B; 0.939 against 0.956 and 0.830 against 0.893 at 32B), and the reasoning arm sits further below its own guessing ceiling in all four (Section 6, Table 11). The effect is small, and Section 6 states its limits: the non-thinking arm wins on three of five datasets individually, and no single dataset's interval separates. Confidence-weighted voting is a second test of the same claim, not the only one.

On the evidential base. We agree it is two Qwen3 weight pairs, and the revision now says exactly that. Within that scope the support is not a single analysis: the collision index rises in ten of ten cells and in nine of nine on the matched subset, and the danger zone contracts in ten of ten with every cell's interval excluding 1×. Generality beyond Qwen3 is not established: the second Granite look, scheduled before any Granite result was computed, is posted as a separate comment (a slightly negative estimate whose interval includes zero, under a large truncation confound).

On "exploratory." Sixteen numeric predictions were recorded in writing before the analyses that test them (Table 24), including the collision threshold; five did not hold, and all five are reported. We accept that the original ordering made the paper read as a sequence of analyses; the restructured Section 1 states the claim first and presents the three tests against it.

We would be grateful if you would weigh these points in your final assessment.

Reviewer_NXgrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper finds that the widely-used self-consistency voting strategy could become less effective or even counterproductive when applied to reasoning-mode language models. Across five benchmarks and 74,944 samples, the authors demonstrate that toggling reasoning mode on concentrates a model’s errors, meaning the probability that two independently drawn wrong answers coincide rises in all ten dataset–scale comparisons. Reasoning mode cuts the number of distinct answers produced to 0.43–0.65 of the non-reasoning count. Empirically, across 280 method–dataset–model combinations, not one confidence-weighted voting method beats plain majority voting after statistical correction. The paper also finds that reasoning simultaneously shrinks the set of vulnerable questions by ~2.72× (the model gets more right) while concentrating errors within the remaining vulnerable set, and these opposing effects roughly cancel that explains why reasoning models appear to benefit similarly from self-consistency in aggregate despite fundamentally different error distributions.

Critical strengths

S-1. The finding that no confidence-weighted voting method beats plain majority voting across 280 method–dataset–model combinations has immediate practical implications, which indicates investment on sophisticated aggregation strategies is largely wasted for reasoning models.

Critical weaknesses

W-1. While the study includes eight models across three families, the controlled within-weights comparison is confined to a single model family. This limits the generalizability of the causal claims to the broader population of reasoning models.

Questions for authors

Suggestions for future work:

(1) The analysis is restricted to self-consistency, where all votes come from the same model. Your finding is that reasoning concentrates a model's errors. Would you expect cross-model consensus, where errors are plausibly less correlated, to be less affected by this concentration? Did you consider comparing it against same-model majority voting, ideally at matched sampling cost, and do you expect your conclusion that no signal beats plain majority voting to hold in that setting?

(2) All confidence signals studied operate at the answer level (e.g., answer log-probability). Process reward models score individual reasoning steps and could, in principle, detect the shared flawed reasoning path that produces a concentrated wrong answer, which answer-level signals cannot see. Do you expect your result to extend to step-level verifiers such as PRMs? If not, which of your findings would you expect to change?

5 comments
AuthorsOpenReview ↗

Response to Reviewer NXgr (1/2)

Thank you for the constructive questions. We answer each, and the weakness first.

W-1 (one controlled family). We agree, and we ran the experiment it calls for: a second family with the same within-weights toggle. Granite 4.2 8B serves thinking and non-thinking modes from one checkpoint through the same enable_thinking switch Qwen3 uses, and a test in the released code checks that the two modes render structurally different prompts. On GPQA-Diamond, 34 problems so far have all eight samples in both Granite arms. On the 12 where both arms err at least twice, collision is 0.590 with reasoning enabled against 0.582 without (+0.008; 95% CI [−0.212, +0.251]); over each arm's own qualifying problems it is 0.649 against 0.610 (+0.039; n = 14/18). This does not resolve the question in either direction: on the same 34 problems Qwen3-8B, whose effect is clear on the full benchmark, is equally unresolved (+0.014; CI [−0.204, +0.242]). A confound specific to Granite also limits the test: on these 34 problems, 34.6% of its thinking traces reach the 32,768-token limit, against 1.5% without thinking, and dropping them leaves too few qualifying problems. Generation continues; as fixed before any result was computed, we will post the result on all completed blocks on 17 November, whatever it shows. On a 126-problem MATH-500 subset Granite is too accurate for the index to be measured (two problems qualify in each arm), so the evidence rests on GPQA-Diamond. We decided before the run to report the result either way; it is inconclusive at this size, and the revised paper says so (Appendix J) and scopes its claims to Qwen3 rather than claiming a second family.

Q1 (cross-model consensus). Our object of study is self-consistency, which is within-model by construction, so our design does not speak to cross-model voting directly. But the released records let us answer the question as posed, including at matched sampling cost, with no additional generation: every arm was evaluated on identical problem identifiers across 1,171 problems. Drawing two samples from each of our four reasoning checkpoints (Qwen3-8B and Qwen3-32B in thinking mode, Olmo-3-7B-Think and Olmo-3-32B-Think) against eight from a single checkpoint, so that both spend eight generations per problem, your conjecture is confirmed in direction and bounded in size. Cross-model errors are less concentrated: collision falls in five of five datasets, by 0.055 on average, roughly a third of the +0.159 that toggling reasoning adds within fixed weights. Cross-model consensus mitigates the concentration; it does not remove it. On accuracy at matched cost, cross-model plurality never beats the strongest single model: it ties on MATH-500 and AMC and trails by 0.027 to 0.033 on AIME, GPQA-Diamond and MMLU-Pro, while beating the average single model (0.870 against 0.853), so it is a hedge against model selection rather than a gain over selecting well. The comparison is now in the revised paper (Appendix I), with its script in the supplement. On whether no signal beating majority holds in that setting, we state a prediction rather than a result, because we have not run the weighting comparison across models: we expect it to hold, and more strongly, since the residual cross-model collision is 0.448 and our finding is that answer-level signals fail precisely when wrong answers agree. We will run it if you consider it decisive. One caveat: our four reasoning checkpoints come from two families, so this measures whether the effect survives aggregation across families, not how it scales with the number of independent voters.

(Continued in part 2.)

AuthorsOpenReview ↗

Response to Reviewer NXgr (2/2)

Q2 (process reward models). Our negative result concerns outcome-level aggregation: all seven methods we evaluate score or reweight final answers, and the six signals used as vote weights, one per family, are scalars over a trace or over its answer distribution. Step-level verification is a class of method our result does not rule out, and our result sharpens where such a verifier would have to act. Reasoning shrinks the danger zone by 2.72× (0.369 to 0.136), but inside what remains the plurality is wrong by construction, and the collision result explains why no answer-level statistic recovers it: the errors agree, so the answer distribution carries no usable signal. A step-level verifier has access to exactly what the final answer destroys, namely where the reasoning diverged. As for which findings would change: the collision result and the danger-zone contraction would not, since both are properties of the error distribution measured without any confidence signal. The 280-combination null would, since it is a statement about answer-level signals. The rescue-rate regime is where we would expect a PRM to show its value. Our released code includes a reward-model harness (src/iclr27/reward_model.py), but we have not run it, and the revision removes a sentence in Section 3 that implied we had. We would rather state a falsifiable prediction: on danger-zone problems, a step-level verifier should outperform every outcome-level weight we evaluate, and its margin should grow with the collision index.

AuthorsOpenReview ↗

Follow-up to Reviewer NXgr

Thank you again for your questions. Before the discussion closes on 18 November, we wanted to check whether our response addresses them.

  • Q1 (cross-model consensus at matched cost): now Appendix I. Pooling across four reasoning checkpoints from two families lowers collision by 0.055 on average, about a third of what toggling reasoning adds within fixed weights, and cross-model plurality never beats the strongest single model.
  • Q2 (process reward models): we state a falsifiable prediction rather than a result, and say plainly that we have not run a reward model.
  • W-1 (one controlled family): the Granite 4.2 result on all 102 committed problems is posted as a separate comment: a slightly negative estimate whose interval includes zero, under a large truncation confound. The claims are scoped to the two Qwen3 pairs.

We remain glad to clarify anything before the deadline.

Reviewer_YmguOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

When a model runs in "thinking" mode, its mistakes pile up on the same few wrong answers instead of spreading out. The authors measure this with a within-problem concentration index and find it in all 10 model×benchmark cells. The sharpest result is the "danger zone": problems the model would have answered correctly without thinking, and gets wrong with thinking, at 2.7× the baseline error rate — and majority voting cannot rescue them, because the repeated wrong answer wins the vote.

Critical strengths

The paper is well written and the comparison is clear. The authors found that thinking mode piles up on the same few wrong answers.

The authors report their own failures in full — 280 aggregation variants were tested to exploit the phenomenon and essentially none worked, and that is stated plainly. The positive claims are believable precisely because the negative ones are not hidden.

Critical weaknesses

Everything rests on Qwen3-8B and 32B. The 10/10 sign test looks strong, but the cells share weights and items, so they are not 10 independent trials. If another vendor's reasoning model shows dispersed errors, the whole story inverts. It seems that the authors found a problem in Qwen but never shown to matter.

Questions for authors

Does the effect replicate, even weakly, on any non-Qwen reasoning model?

If truncated thinking traces are dropped entirely rather than reweighted, how much of the gap remains?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Ymgu (1/2)

Thank you for a careful and fair review. We take your points in order.

The ten cells are not ten independent trials. You are right, and we have corrected it. The cells are five datasets crossed with two scales; no two share both, but every pair shares one, so the exact sign test's exchangeability assumption fails and p = 0.00098 overstated the evidence. Clustering by dataset gives five units and 5/5, p = 0.031; clustering by weight-pair gives two and 2/2, p = 0.25. The revised paper reports p = 0.031 as the headline and states the clustering explicitly. The revision also makes the relationship to the per-cell evidence explicit: the cluster-bootstrap intervals over problems (Appendix B) do not treat cells as independent, but they are wide by construction and several include zero, so the inference is the consistency of sign across five datasets and two independent weight-pairs, which the clustered test now measures correctly. We applied the same qualification to the rescue-rate count in Section 5, whose model pairings also share models.

Generality and Q1 (a non-Qwen reasoning model). We agree this is the binding limitation, and we ran the experiment you ask for. Granite 4.2 8B serves thinking and non-thinking modes from a single checkpoint through the same enable_thinking switch Qwen3 uses; a test in the released code checks that the two modes render structurally different prompts. We decided before the run to report the result whichever way it came out. On GPQA-Diamond, 34 problems so far have all eight samples in both Granite arms. On the 12 where both arms err at least twice, collision is 0.590 with reasoning enabled against 0.582 without (+0.008; 95% CI [−0.212, +0.251]); over each arm's own qualifying problems it is 0.649 against 0.610 (+0.039; n = 14/18). This does not resolve the question in either direction: on the same 34 problems Qwen3-8B, whose effect is clear on the full benchmark, is equally unresolved (+0.014; CI [−0.204, +0.242]). A confound specific to Granite also limits the test: on these 34 problems, 34.6% of its thinking traces reach the 32,768-token limit, against 1.5% without thinking, and dropping them leaves too few qualifying problems. Generation continues; as fixed before any result was computed, we will post the result on all completed blocks on 17 November, whatever it shows. On a 126-problem MATH-500 subset, Granite is too accurate for the index to be measured (two problems qualify in each arm). Accordingly, the revised paper scopes the title, abstract and conclusions to the two Qwen3 pairs, and reports Granite in full in a new Appendix J.

The cross-family arms already in the paper cannot answer this question in either direction. Among non-reasoning models alone, collision on MATH-500 ranges from 0.297 to 0.441 across families, a spread larger than the within-weights effect at 8B (0.379 to 0.509, a difference of 0.130). That is why the answer has to be a second toggle family.

(Continued in part 2.)

AuthorsOpenReview ↗

Response to Reviewer Ymgu (2/2)

"Never shown to matter." We agree the paper shows no aggregate accuracy cost, and Section 1 says so. The cost is offset because reasoning also shrinks the danger zone, the set of problems with fewer than half the usable samples correct, by 2.72× (0.369 to 0.136, ten of ten cells). We flag this because the summary reads the danger zone as problems reasoning gets wrong at 2.7× the baseline rate; it is the reverse, and we have made the definition explicit where it is first used. What concentration changes is what agreement and confidence mean. Confidence weighting cannot exploit the difference: across 280 method–dataset–model combinations none beats plain majority voting after correction, and weighted voting returns the plurality answer on 98.5% of problem–method pairs. And a confidence signal can reverse within fixed weights: at 8B, higher answer-token log-probability predicts a correct answer with reasoning off and a wrong one with reasoning on (selector weights +0.129 and −0.176; at 32B it does not invert but its magnitude collapses 4.7-fold, from −0.379 to −0.080). A signal validated on one mode of a model can point the wrong way on the other, at identical weights. That is the consequence we claim.

Q2 (dropping truncated traces). We dropped every truncated generation (finish_reason == "length") entirely rather than reweighting, and recomputed the collision index. The mean gap across the ten comparisons is +0.159 with all samples and +0.160 after the drop, and the sign is unchanged in every cell. The per-cell shifts are small and bidirectional (four gaps grow, six shrink), which is not the signature of a truncation artifact: were truncation manufacturing the concentration, removing it would shrink the gaps systematically. The premise also does not hold in most cells: the non-reasoning arm truncates more often in 7 of 10. In the two cells where the reasoning arm truncates substantially more (on AIME, 0.0917 against 0.0681 at 8B and 0.0792 against 0.0458 at 32B), the gap after dropping is +0.130 and +0.094, against +0.122 and +0.097 before. A new table in Appendix F reports every cell, and scripts/truncation_check.py in the supplement reproduces it.

AuthorsOpenReview ↗

Follow-up to Reviewer Ymgu

Thank you again for your review. With the discussion closing on 18 November, we wanted to check whether our response addresses your concerns, and to summarize where each now stands.

  • Independence of the ten cells: conceded and corrected. The headline is now the exact sign test clustered by dataset (p = 0.031), with clustering by weight-pair (p = 0.25) also stated.
  • Truncated traces (Q2): dropping every truncated generation leaves the mean gap at +0.160 against +0.159 with all samples, and the sign is unchanged in all ten cells (new table in Appendix F; the script is in the supplement).
  • A non-Qwen reasoning model (Q1): we ran Granite 4.2 as a second toggle family. The result on all 102 committed problems is posted as a separate comment: a slightly negative estimate whose interval includes zero, under a large truncation confound. The paper's claims are now scoped to the two Qwen3 pairs.
  • "Never shown to matter": the concrete consequence is the within-weights inversion of answer log-probability at 8B, set out in our reply.

If anything remains unclear, we are glad to answer before the deadline.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Revisions

We have uploaded a revised PDF and supplement. In summary:

  1. Title scoped to "Reasoning Concentrates Errors Within Fixed Weights, and Self-Consistency Never Notices"; the abstract, Section 1's claim, Section 3's description of the other four models and the conclusions are scoped to the two Qwen3 pairs. (csCm W2)
  2. New: a second toggle family (Granite 4.2 8B), in a Section 4 paragraph and a new Appendix J: inconclusive at the scale we could run, with its truncation and answer-rate caveats stated. The GPQA analysis schedule and criterion were fixed before any result was computed; a second pre-declared look follows on 17 November. (all reviewers)
  3. Sign-test p-value corrected from 0.00098 to 0.031, with the clustering stated; the rescue-rate count is qualified the same way. (Ymgu)
  4. Section 1 restructured to state the claim first; "never notices" defined operationally. (csCm Q1, Q2)
  5. Truncation: a new table in Appendix F recomputes every within-weights cell with truncated traces dropped. (Ymgu Q2)
  6. Between-family spread versus the within-weights effect, quantified in Appendix B. (csCm, Ymgu)
  7. Cross-model consensus at matched sampling cost, in a new Appendix I. (NXgr Q1)
  8. Danger zone defined at its first use, in the figure caption. (Ymgu)
  9. Related work: Zhang et al. (2026) added to Section 2. (csCm)
  10. Bibliography: five entries corrected, four completed, and capitalization and one author list repaired. (csCm)
  11. Supplement: the Granite configuration, per-sample records, analysis scripts and toggle test, plus the truncation and cross-model scripts; README claims scoped to Qwen3.
  12. Corrections from our own audit of the revision against the released code. Table 8's per-cell danger-zone values now match the released script (seven values move by 0.001; the means, the 2.72× ratio and every lower bound above 1× are unchanged). Table 5's intervals are regenerated with a fixed seed so they reproduce exactly, and its caption now says three, not four, matched intervals exclude zero. Section 3 now describes the grader accurately (symbolic equivalence on the mathematics sets). Appendix A now states that the 300 MMLU-Pro questions, the first 300 of its test split, all come from its business category. Section 3 no longer mentions an external reward-model baseline, which was never run.
AuthorsOpenReview ↗

Granite 4.2: second pre-declared look (all committed blocks)

As promised in our replies, here is the Granite 4.2 8B result on all six committed blocks of GPQA-Diamond: 102 problems with all eight samples in both arms. All blocks finished on 16 November, so we report now rather than on 17 November. The statistic is the one fixed in advance: the matched-subset collision difference with a paired bootstrap 95% interval.

Granite. On the 32 problems where both arms err at least twice, collision is 0.592 with reasoning enabled and 0.658 without (−0.066; 95% CI [−0.200, +0.070]); per arm, 0.577 against 0.666 (−0.089; [−0.221, +0.058]; n = 39/47). The estimate is slightly negative and the interval includes zero.

Qwen3 on the same 102 problems. Both pairs point in the expected direction: +0.098 ([−0.007, +0.193], n = 46) at 8B and +0.118 ([−0.003, +0.232], n = 32) at 32B, in line with their matched values on the full benchmark (+0.129 and +0.097).

Truncation. Granite's thinking mode reaches the 32,768-token limit on 29.8% of its traces, against 2.3% without thinking and 0.1–0.3% for Qwen3's thinking mode. With truncated traces dropped, Granite's difference becomes +0.189 ([+0.012, +0.390]), but on only 14 problems, a subset selected by which problems Granite finishes, so we do not rest anything on it. At this budget, the run cannot separate an absence of the effect from the truncation.

The revision already confines the claim to the two Qwen3 pairs, and the first look (+0.008 on 34 problems) is reported beside this one. A final version will report both looks and the truncation analysis in Appendix J, and describe Granite in the abstract accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.