- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies reasoning in 3 models and 5 datasets. The key issue is that they observe that models propose an answer that turns out to be their final answer, before spending considerable tokens on checking -- even though this doesn't change the final outcome. They are particularly interested in the link between the model's internal confidence and the initiation of backtracking behavior following an answer proposal. They use activation steering to show verbal confidence derived vectors, and commitment vectors (from late proposal marker vs early proposal marker) can reduce answer proposals, reduce backtracking token probabilities and cause early termination (shorten reasoning traces whilst preserving accuracy). Their conclusions are primarily in relation to the role of confidence in backtracking, answer proposals rather than regarding potential deployment benefits.
Critical strengths
S1: This is an important question -- whether confidence signals drive backtracking in LLMs -- often assumed but to my knowledge not clearly demonstrated before. S2: The methodology used -- the intermittent use of verbal confidence prompts to extract confidence vectors -- and activation steering are carefully applied. S3: The intervention findings i.e. the effects on answer proposal number, and early termination are convincing
Critical weaknesses
W1: Whilst the authors provide convincing evidence that intervention with confidence (and commitment vectors) can cause substantial alterations in answer proposals etc, I'm not convinced this shows that the model in it's natural state uses internal confidence signals to backtrack. Several analyses would help i) there is little demonstration apart from the mild correlation of verbal confidence with final vs early answer proposal, that confidence predicts these behaviors. This would provide much needed supportive evidence for their conclusions ii) Activation steering could be complemented by other techniques to show convincingly that internal confidence signals in its natural state are used by the model to trigger backtracking: a) activation patching b) activation swap experiments -- these try to eliminate limitations of activation patch experiments to isolate the contribution of the confidence-specific part of a representation. iii) the authors could take the confidence direction isolated by steering and look at the correlation between the projection of an activation state onto this direction and how frequently backtracking is triggered. W2: I'm not sure how to interpret the results of the commitment vector steering. This is because the final answer proposal marker representation likely contains lots of things that differentiate it from the early answer proposal marker. Can the authors perform additional analyses to narrow down what is driving the observed effects. W3: The evidence that these interventions like continued steering change backtracking behavior is indirect -- and it is not clear why proxies like the number of answer proposals and early termination were relied on. W4: (Minor) I found the related work section lacking a section on the link between internal confidence signals and behavior (e.g. in non reasoning LLMs -- e.g. Kumaran et al., Nature Machine Intelligence 2026; Ackerman 2026 https://arxiv.org/abs/2509.21545)
Questions for authors
Q1: Can the authors provide more evidence to back up their case that internal confidence signals in the natural operation of the model? Q2: What is driving the effects of the commitment vector Q3: Can more specific evidence be provided as to the link between the steering manipulations and backtracking behavior. Q4: Can the authors add relevant evidence on the link between confidence and behavior and also clarify if there paper is the first to show that a unique answer proposal is typically provided despite repeated answer proposals before?
We thank the reviewer for the detailed feedback and address their points below. Changes in the revised manuscript are in red.
[W1, Q1] Naturalness of confidence
We note that while we motivate our work with the hypothesis that models use internal confidence to commit, our main contributions demonstrate a causal steering relationship with answer commitment behavior. We agree that this does not guarantee that the linear confidence direction is naturally being used by the model to the extent that steering applies it (Mishra et al., 2026 https://arxiv.org/abs/2604.09839). This reflects a broader open question on the “naturalness” vs “causality” of steering. We add this to our limitations, and we scaled down any claims that suggested vconf steering is synonymous with the natural mechanism.
Following the reviewer’s suggestion (iii), we compute Spearman correlations between the hidden state projection onto vconf or vcommit and (a) binary answer commitment on the test set (ρmax=0.87) or (b) pbt. We report results in Tables 28 and 29 and discuss in section 4.2.
Olmo 3 32B and Qwen 3 32B, projections onto vconf have ρ=0.49,0.64 respectively, which exceed that of verbalized confidence on the test set (0.33,0.26). This disparity is largest on incorrect rollouts, where verbalized confidence is uncorrelated (ρ=0.09,-0.03) while vconf projection remains at ρ=0.36,0.58. Interestingly, on Distill 32B, the projection has no correlation with answer commitment (ρ=-0.00), yet all three models have strong vconf projection correlations with pbt (ρ∈[−0.68,−0.46]). This suggests that Distill 32B tracks and steer local backtracking behavior which then influence answer commitment (consistent with results in sections 4 and 5), but cannot track answer commitment labels themselves. One hypothesis for this difference is the presence of RL post training. Olmo 3 32B and Qwen 3 32B undergo RL, while Distill 32B uses SFT. It is possible that RL training strengthens the link between latent confidence and answer commitment. Studying how post training strategies affect internal confidence would be an exciting future direction. We leave alternative methodologies for proving naturalness, like activation patching, to future work as well.
[W2, Q2] Interpreting the commitment direction
We add logit lens results for vcommit in Table 27 and discuss them in section 4.2 in the revised manuscript. Overall, the decoded top tokens suggest that vcommit operates at the syntactic level. The negative direction decodes backtracking tokens in English and Chinese, such as “However” and “But”. Olmo 3 32B’s negative direction also decodes affirmatory adverbs like “Therefore” and “Thus”, and its positive direction decodes “</”, the first token of </think>. This suggests that steering with Olmo 3 32B’s vcommit directly promotes the reasoning termination tag and suppresses adverbs leading to any additional reasoning, which is consistent with the construction of vcommit from final and mid-trace proposal markers. This contrasts vconf, which promotes concepts semantically related to confidence.
[W3, Q3] Linking metrics to backtracking behavior
Since backtracking behavior is a qualitative characterization of the CoT, any metric used, i.e. next token probability of a backtracking token, additional answer proposal counts, early termination, response length, even qualitative inspection of language itself, is arguably a noisy proxy with its own weaknesses. For example, next token probability of a backtracking token (section 4.1) doesn’t consider the later generated tokens, but early termination of thinking does (section 5.1). The number of answer proposals (section 5.2) measures how many times the model backtracks from an answer proposal, but it does not check the language used by the model. Qualitatively checking the prose (section 5.3) does. We argue that in the absence of a single, universally accepted metric to quantify backtracking, the consistency of all the different "proxy" metrics supports our conclusion that confidence steering modulates backtracking behavior.
[W4, Q4] Additional references and novelty of Table 1
Thanks for suggesting these references. We have added them to the manuscript. We also add a mention to Chen et al. (2025, https://arxiv.org/abs/2412.21187) at the end of section 3.1, who count the number of answer proposals, but not the number of unique solutions. To our knowledge, we are the first to explicitly count the number of unique solutions. However, due to the sheer volume of existing literature on reasoning redundancy, we provide this result primarily to frame the rest of our paper rather than as a substantially novel finding.