acceptodds
Under review as a conference paper at ICLR 2027

Hidden Doubt, Missed Correction: A Causal Pathway in Chain-of-Thought Reasoning

Abstract

The errors made by large language models (LLMs) in chain-of-thought (CoT) reasoning are often decodable by linear probes from their activations. Yet the error signals in the representations do not always cause the model to self-correct. We study the gap between error awareness and error correction in LLMs. The Jacobian lens (J-lens) is an interpretability tool that maps activations to tokens the model is poised to verbalize. The mapping uses an averaged Jacobian from the activation to the final layer, which aggregates downstream paths through the model's computation graph across layers and subsequent token positions. Using the J-lens, we find that erroneous CoT steps exhibit stronger representations of doubt-related tokens, such as "But" and "However". Yet these tokens receive little probability mass in the next-token distribution. To verify whether the hidden doubt causally triggers self-correction, we steer activations along the corresponding J-lens directions, which makes the model voice the doubt and revise the reasoning path, improving final-answer accuracy. We also steer along the probe direction to amplify the error signals and find that this amplification can support self-correction by eliciting doubt. The two intervention approaches support hidden doubt as a possible mechanistic link between error awareness and self-correction. We further compare against non-doubt token interventions and direct text insertion, supporting that the effect is specific to eliciting doubt.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, chain-of-thought reasoning, self-correction, Jacobian lens

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qocYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The authors aim to address the gap between error awareness and self-correction in LLM chain-of-thought reasoning. They use the Jacobian lens to identify doubt-related token representations in erroneous reasoning steps and apply activation steering along these directions and probe-derived error directions. Experiments on three LLMs and three reasoning datasets show that these interventions can elicit doubt-related tokens and improve recovery from erroneous steps.

Critical strengths
  1. The paper is clearly written, with well-defined concepts and easy-to-follow explanations.
  2. The paper presents a coherent research narrative, examining the pathway from error awareness to hidden doubt and self-correction.
Critical weaknesses
  1. Insufficient motivation. The authors claim that prior work either detects error awareness without correction or relies on external control. To my knowledge, several studies have already explored using internal error signals to facilitate self-correction, making this motivation less convincing.
  2. Underexplored concept of hidden doubt. While hidden doubt is an interesting idea, the paper does not sufficiently establish its existence as a distinct internal representation or how to evaluate it. Instead, the analysis largely focuses on the well-studied relationship between error awareness and self-correction.
  3. Limited novelty and new insights. Most findings are closely related to prior work. Amplifying error-related signals to induce reflection is potentially interesting, but the results are inconsistent and insufficiently explored. The method also largely follows the existing J-lens framework without substantial modifications.
Questions for authors

Regarding Weaknesses 2 and 3: Can the authors provide evidence that hidden doubt is a distinct internal representation, rather than merely a manifestation of error-related representations? In particular, can the two be disentangled through additional analysis or controlled interventions?

Regarding Weaknesses 1 and 3: Could the authors further reconsider and clarify the novelty of this work in light of existing studies on error awareness, reflection, and activation steering? In particular, what new insights does this paper offer beyond prior research?

Reviewer_pXfeOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies the problem of error detection and correction in LLMs. The authors use J-lens as a mechanistic interpretability tool to locate doubt-related tokens in the thinking traces and show that they are more prevalent in erroneous traces, but they don’t receive enough probability mass to be sampled. By doing interventions, the authors amplify these tokens, leading to improved accuracy. They study also using signal from a linear classifier to amplify the doubt signal, as well as explicitly altering the thinking traces to add the doubt tokens. In both situations, the performance improves as a result of error correction. 3 models from different families (Llama, Gemma, Qwen) and 3 datasets (2 synthetic, and GSM8k) are used.

Critical strengths

The paper addresses an important problem in LLMs, the ability to detect errors and self-correct. The study has a good coverage of models with 3 different families represented (llama, gemma, qwen). These are all from open-sourced families, which is normal as the study requires access to internal activations for analysis and intervention. The proposed causal intervention tools seem appropriate to me.

Critical weaknesses

Although the method seems sound, the choice of dataset and the results reported are confusing and potentially misleading to me.

The dataset used to motivate the whole approach (stats included in Table 5 appendix) is severely imbalanced, with very low counts for erroneous steps, making the AUROC curves in Fig 2 difficult to interpret. The sample size in the test sets are tiny (as low as 27 for Qwen on Boolean dataset). Figure 2 should report confidence intervals as well. Overall, additional metrics should be provided or consider other datasets.

The different models are probed at different layers, which is fine. However, for the same model, the authors change the layer at which they extract the probe throughout the paper, depending on the analysis they do. E.g. for Gemma, the initial linear probe is done at L28 for Gemma in Fig 1, but then J-lens readout is done on L26 (in Fig 3) and linear probe interventions are done at L26, L29, and L32. How are these layers chosen in the first place? The fact that the probe needs to shift for every different analysis weakens the claims and looks like a test-set selection bias.

Other confusing statements related to datasets: L296-300 on intervention evaluation. What test sets were used more exactly? The number of erroneous steps reported in Table 6 (which is mentioned in the evaluation paragraph L296-300) includes all the erroneous steps, both train and test.

What split is reported in Table 9? When adding up the number of verbalised and non-verbalised cases, this is larger than the test size reported in Table 5.

How were the doubt tokens chosen? I understand the part with J-space selection-rate difference and checking the top-4 tokens, but feels like a subjective and tokenizer-specific selection. Moreover, in appendix C3, more doubt-related phrases are included without a clear justification.

The experiment in section 4.4 is important, but the current setup is not rigorous in my opinion. The intervention layers and gains for doubt tokens were carefully selected, but the authors kept the same layers and gains for the non-doubt tokens. It doesn’t seem like a fair analysis. Shouldn’t the same selection rule apply to non-doubt tokens to properly study the effect of amplifying them?

Questions for authors

See Weaknesses section above.

Reviewer_9CDROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies LLM reasoning mistakes during CoT, where they show that a simple linear probe can detect the mistake with 90%+ accuracy. The authors make the following arguments:

  • Error representations reflect the model's doubt
  • This doubt is drowned out and the model does not act on it naturally, although there is evidence that "but" and "however" tokens increase in rank
  • Amplifying this doubt can lead the model towards self-correction
Critical strengths
  • This paper resolves an existing puzzle by explaining why subtracting error vectors failed, and showed instead that amplifying them triggers self correction
Critical weaknesses
  • The main challenge I see with this is that we don't know when to intervene if we do not know where the errors are. In this paper, interventions happen at a pre-identified step (right before the first error). Intervening at a correct step can drop accuracy (see, e.g., Llama in Table 3), so running this intervention can be tricky in real scenarios
  • The verbalization rates in Table 3 are a bit concerning: this says that for Llama, 100% of of correct steps/incorrect steps are getting "but" and "however". This seems almost equivalent to hardcoding these tokens.
Questions for authors
  • What happens to overall accuracy if you do the intervention for every step during generation from scratch without knowing beforehand which steps are erroneous? How would you do this?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.