- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies how explicit reasoning affects self-consistency and confidence estimation in language models. Its main controlled experiment compares Qwen3 models with reasoning enabled and disabled while keeping the model weights fixed. The paper introduces a wrong-answer collision measure and reports that enabling reasoning consistently increases the concentration of incorrect answers across the evaluated datasets and model scales.
The paper then studies the downstream implications of this observation for self-consistency. It analyzes accuracy gains from repeated sampling, introduces a "danger zone" to characterize problems where voting can potentially change the outcome, evaluates confidence-weighted voting across multiple confidence signals, and further studies output length, answer-token probability, signal transfer, and calibration across models and datasets.
The main empirical finding is that reasoning makes incorrect answers more concentrated. At the same time, reasoning also reduces the number of problems on which voting can materially affect the outcome, and these effects largely cancel in aggregate self-consistency accuracy. The paper additionally reports that confidence-weighted voting does not significantly outperform majority voting under its experimental setting.
Critical strengths
1. The same-weight reasoning on/off comparison is a useful experimental control.
The strongest part of the paper is the comparison between Qwen3 thinking and non-thinking modes under identical weights. This removes many confounds that would arise from comparing independently trained reasoning and non-reasoning models. The observation that wrong-answer collision increases consistently across the paired settings is therefore interesting and deserves attention.
2. The paper is unusually transparent about failed hypotheses and statistical corrections.
The authors explicitly report analyses that did not support their original expectations, including the headroom issue in the self-consistency gain comparison, and they distinguish confirmatory from supporting evidence. The paper also reports multiple-testing corrections, matched analyses, answer rates, and pre-registered predictions. This level of transparency is valuable.
Critical weaknesses
1. The paper lacks a sufficiently coherent and stable central scientific claim.
The paper contains a large number of analyses, but I found it difficult to identify a single research question that is consistently carried from the motivation to the conclusions. The initial observation that reasoning increases wrong-answer concentration is clear. However, the expected downstream consequence is repeatedly revised when subsequent analyses do not support it.
For example, the apparent threefold difference in self-consistency gain is later shown to be an artifact of unequal accuracy headroom. After normalization, the two conditions convert essentially the same fraction of available headroom. The paper then introduces the "danger zone" to explain this result, followed by a rescue-rate analysis that the authors themselves describe as supporting rather than confirmatory. The narrative then branches further into confidence-weighted voting, output length, hedging, answer-token probability, cross-family transfer, and calibration.
I appreciate that the authors transparently revise conclusions when analyses fail. However, transparency about failed hypotheses is not the same as having a coherent scientific contribution. In its current form, the paper reads more like a sequence of exploratory analyses, where unsuccessful hypotheses motivate new metrics and new secondary findings, than a focused investigation of one well-supported central claim. After reading the full paper, I find the contribution difficult to characterize beyond the initial observation that reasoning increases wrong-answer concentration.
2. The strength of the title and broader conclusions exceeds what the controlled evidence establishes.
The title suggests a strong connection between reasoning-induced error concentration and a failure of self-consistency. However, the paper itself shows that after correcting for available accuracy headroom, the aggregate benefit of self-consistency is essentially the same in-domain for reasoning and non-reasoning models. The rescue-rate analysis, which more directly tests whether concentrated errors reduce the ability of voting to recover the correct answer, is underpowered and explicitly reported as non-confirmatory.
Moreover, the strongest causal evidence for the effect of reasoning comes from only two same-weight Qwen3 pairs. The additional Olmo and Gemma comparisons do not provide the same identification because model family, training, and other factors differ. Thus, the large number of generations and downstream analyses should not be confused with broad controlled evidence for the general claim.
I also noticed concrete inaccuracies in the bibliography. For example, "Reasoning about Uncertainty: Do Reasoning Models Know When They Don't Know?" is cited as an EACL 2025 work although the official publication is Findings of EACL 2026, and OLMo 3 is cited as a 2026 work although its paper was released in 2025. These citation errors are not my primary reason for rejection, but they further reduce my confidence in the care with which the recent literature was checked.
Questions for authors
1. What do the authors consider to be the single central scientific claim of this paper?
The initial result that reasoning increases wrong-answer concentration is clear, but several of the expected downstream conclusions are subsequently weakened or replaced after additional analyses. In particular, the raw self-consistency gain difference disappears after headroom normalization, and the rescue-rate evidence is not confirmatory. I would like the authors to clarify which claim they consider the primary contribution and which results provide direct evidence for that claim.
2. What direct evidence supports the "self-consistency never notices" part of the framing?
After headroom normalization, the aggregate self-consistency improvement is essentially identical in-domain between the reasoning and non-reasoning conditions. The more direct rescue-rate analysis is underpowered. If the intended claim is not that self-consistency performs worse, please clarify precisely what "never notices" means operationally and which experiment directly tests it.
3. Can the authors provide broader same-weight controlled evidence beyond the two Qwen3 pairs?
The central causal interpretation relies on changing reasoning behavior while holding weights fixed, which is a strong design. However, only two model pairs satisfy this condition. The remaining cross-family comparisons cannot isolate reasoning from training and model-family differences. Evidence from additional controlled reasoning on/off models would materially affect my confidence in the generality of the central claim.
4 comments
Response to Reviewer csCm (2/2)
Q3 (same-weight evidence beyond the two Qwen3 pairs). We ran the experiment you describe. Granite 4.2 8B serves thinking and non-thinking modes from one checkpoint through the same chat-template switch Qwen3 uses; a test in the released code checks that the thinking template leaves the reasoning block open and the non-thinking template closes it before generation, so this is a second identified control rather than a cross-family comparison. On GPQA-Diamond, 34 problems so far have all eight samples in both Granite arms. On the 12 where both arms err at least twice, collision is 0.590 with reasoning enabled against 0.582 without (+0.008; 95% CI [−0.212, +0.251]); over each arm's own qualifying problems it is 0.649 against 0.610 (+0.039; n = 14/18). This does not resolve the question in either direction: on the same 34 problems Qwen3-8B, whose effect is clear on the full benchmark, is equally unresolved (+0.014; CI [−0.204, +0.242]). A confound specific to Granite also limits the test: on these 34 problems, 34.6% of its thinking traces reach the 32,768-token limit, against 1.5% without thinking, and dropping them leaves too few qualifying problems. Generation continues; as fixed before any result was computed, we will post the result on all completed blocks on 17 November, whatever it shows. On a 126-problem MATH-500 subset, Granite is too accurate for the index to be measured. We also want to state plainly why our Olmo and Gemma arms cannot stand in for this, since you correctly discount them: among non-reasoning models alone, collision on MATH-500 ranges from 0.297 to 0.441 across families, a spread of 0.144, larger than the within-weights effect at 8B (0.379 to 0.509, a difference of 0.130). Between-family variation exceeds the quantity of interest, which is why the answer has to be a second toggle family; Appendix B of the revision quantifies this.
Bibliography. Thank you for checking it; both errors you name were real. Reasoning about Uncertainty is now cited as Findings of EACL 2026, and Olmo 3 as a 2025 arXiv report with the correct author attribution. Prompted by your note, we audited the full bibliography and found three further errors of the same kind: two EMNLP 2025 papers (Tan et al.; Vashurin et al.) were cited as arXiv preprints, and Devic et al. carried the wrong year (2026 for a 2025 preprint). We also completed four entries that were missing a venue or arXiv identifier, and in a final pass restored capitalization lost in five titles and repaired one malformed author list. All are corrected in the revised PDF.
I keep my score at 1. The rebuttal improves framing and transparency but not the evidential base: the central claim still rests on two Qwen3 weight pairs, Granite is inconclusive by the authors’ own report, and “never notices” describes confidence-weighted voting rather than self-consistency accuracy. The paper remains a sequence of exploratory analyses rather than a coherent, well-supported contribution.
Response to Reviewer csCm
Thank you for reading the rebuttal and for replying. Three brief points of fact, for you and the AC.
On "never notices." We agree the claim is not about self-consistency's accuracy, and we do not make one. It is about the confidence self-consistency itself produces: the agreement fraction among its k samples. Within fixed weights that signal is less informative with reasoning on, in four of four weight-pair and split comparisons (agreement AUROC 0.918 against 0.952 in domain and 0.805 against 0.869 out of domain at 8B; 0.939 against 0.956 and 0.830 against 0.893 at 32B), and the reasoning arm sits further below its own guessing ceiling in all four (Section 6, Table 11). The effect is small, and Section 6 states its limits: the non-thinking arm wins on three of five datasets individually, and no single dataset's interval separates. Confidence-weighted voting is a second test of the same claim, not the only one.
On the evidential base. We agree it is two Qwen3 weight pairs, and the revision now says exactly that. Within that scope the support is not a single analysis: the collision index rises in ten of ten cells and in nine of nine on the matched subset, and the danger zone contracts in ten of ten with every cell's interval excluding 1×. Generality beyond Qwen3 is not established: the second Granite look, scheduled before any Granite result was computed, is posted as a separate comment (a slightly negative estimate whose interval includes zero, under a large truncation confound).
On "exploratory." Sixteen numeric predictions were recorded in writing before the analyses that test them (Table 24), including the collision threshold; five did not hold, and all five are reported. We accept that the original ordering made the paper read as a sequence of analyses; the restructured Section 1 states the claim first and presents the three tests against it.
We would be grateful if you would weigh these points in your final assessment.
Response to Reviewer csCm (1/2)
Thank you for a demanding review; it improved the paper. We address each point.
W1 / Q1 (the central claim). The central claim is: toggling reasoning within fixed Qwen3 weights changes the structure of a model's errors, concentrating them on fewer distinct wrong answers, and this leaves self-consistency's accuracy unchanged while breaking its use as a confidence signal. Three results test it directly. The structural change is measured by the collision index, which rises in all ten dataset–scale comparisons and in nine of nine after restricting both arms to the problems each gets wrong. The accuracy invariance is measured by headroom-normalized self-consistency gain, 0.250 against 0.249 in domain. The confidence failure is measured across 280 method–dataset–model combinations, where no weighting beats plain majority voting after correction. The raw threefold difference in self-consistency gain does vanish under headroom normalization, and that null is part of the claim, not a casualty of it: reasoning concentrates errors, which should hurt voting, and simultaneously shrinks the set of problems where voting can change anything by 2.72×, which helps it. The two effects are opposite in sign and comparable in size, so aggregate accuracy barely moves. We accept that the paper's ordering obscured this, and we have restructured Section 1 to state the claim first and present the cancellation as predicted structure rather than as a correction discovered late.
Q2 ("never notices," operationally). It is a claim about the confidence signal, not about accuracy, and Section 1 now defines it: weighted voting agrees with plain majority voting on 98.5% of problem–method pairs and, on the remaining 1.5% where it dissents, is right 56.3% of the time. Two experiments test it directly: the 280-combination comparison, in which no confidence weighting beats majority voting after correction, and the within-weights inversion of answer log-probability at 8B, which predicts correctness with reasoning off and error with it on. As stated at submission, we do not claim a measurable self-consistency accuracy penalty at this scale, which is why headroom normalization does not undercut the claim.
W2 (title and broader conclusions). We agree, and we have changed the title to "Reasoning Concentrates Errors Within Fixed Weights, and Self-Consistency Never Notices", and scoped the abstract, the claim paragraph in Section 1 and the conclusions to the two Qwen3 pairs on which the controlled evidence rests. The second toggle family we ran during the discussion, Granite 4.2, is inconclusive (Appendix J), so we no longer imply generality beyond Qwen3. We kept "never notices" and now define it in Section 1, as above. As at submission, we do not claim a measurable self-consistency accuracy penalty at this scale. On the rescue rate: we agree it is not confirmatory, and the revision says so. Section 5 reports it as consistent with Section 4 rather than as confirmation (three of the four cells with at least ten qualifying problems in both arms, p = 0.31), and the 13-of-16 pairing count is now qualified because those pairings share models.
(Continued in part 2.)