- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies when language models commit to their final answers during chain-of-thought reasoning. The authors propose a truncation-based framework to measure answer confidence at different reasoning steps and identify a "commitment boundary" where confidence increases sharply. They further train attention probes to predict answer-formation stages and use them for early exiting. Experiments are conducted across three models and four reasoning benchmarks.
Critical strengths
The paper studies an interesting problem: understanding when models form and stabilize their answers during reasoning. The analysis across different models and tasks, together with the early-exit application, provides potentially useful empirical observations.
Critical weaknesses
W1. Limited novelty and insufficient discussion of related work.
Several closely related works are missing from the discussion. "Answer Convergence as a Signal for Early Stopping in Reasoning" (EMNLP 2025) already studies answer stabilization through CoT truncation and learned stopping predictors. "Thought Calibration" (EMNLP 2025) also uses hidden-state probes to predict answer consistency and guide early stopping. Additionally, "Causal Sufficiency and Necessity Improves Chain-of-Thought Reasoning" (NeurIPS 2025) provides a related framework for step-level causal analysis.
The authors should better distinguish their contributions from these existing approaches and clarify the novelty of the proposed framework.
W2. Concerns about causal validity and methodological soundness.
I have several concerns about the proposed framework.
First, the confidence definition is inconsistent. Equation (2) defines p_i using argmax, which does not return a probability. Appendix B instead defines p_i = P(a_0 | X_i), where a_0 is the first token of the full-CoT answer. These are not generally equivalent, even under greedy decoding when intermediate answers differ from the final answer. This ambiguity directly affects the proposed stage labels and commitment boundary.
Second, the causal interpretation is questionable. Truncating CoT and forcing an answer measures the effect of a prefix under a specific decoding protocol, but does not necessarily identify the causal importance of individual reasoning steps. A confidence increase could simply reflect the verbalization of an already available answer rather than a meaningful reasoning step.
Third, the commitment boundary does not necessarily indicate answer stability. The authors define it using the largest confidence increase, but confidence changes can be positive or negative. The largest positive jump does not guarantee that the answer remains stable afterward. Therefore, the claim that all meaningful reasoning precedes this boundary is not sufficiently justified.
Finally, there are additional experimental concerns. The PRE and POST perturbation experiments use different amounts of context, making their causal comparison questionable. The filtering criteria may introduce selection bias. Moreover, the claimed linear probe contains nonlinear components, and some results in the main text and appendix appear inconsistent (e.g., Table 1 vs. Table 7).
Overall, I find the causal conclusions stronger than what the experiments establish.
Questions for authors
-
What exactly is p_i? How do the authors reconcile Equation (2) with Appendix B?
-
How does the proposed framework distinguish genuinely important reasoning steps from steps that simply make an existing answer easier to decode?
-
Does the largest confidence increase reliably indicate final-answer stability? How frequently do answers change or confidence decrease after the identified boundary?
-
How do the filtering criteria affect the results? Why is the nonlinear attention probe described as linear? What explains the differences between Table 1 and Table 7?