- Rating
- 3
- Confidence
- 3(+1)
- Informativeness
- 3(+1)
- Presentation
- 3(+1)
- Soundness
- 2
Read review
Paper summary
This paper studies “premature commitment” in large language models’ Chain-of-Thought (CoT), asking whether a model’s internal state can effectively lock in the final answer before the explicit reasoning trace has finished. The authors propose Causal Lookahead Probing (CLP), which causally measures commitment depth by transplanting the last-layer residual stream from another problem at a particular reasoning step and testing whether the final answer follows the transplanted source state. In parallel, they construct a prescribed trajectory, CD_P, from an explicit reference program, representing the maximum degree of answer determination that is theoretically justified at each step. Under a licensed-dependence assumption, they prove that a faithful executor should satisfy CD_M(k) <= CD_P(k). Experiments span Dyck-2, finite-group problems, finite-state automata, entity tracking, and nine model checkpoints. CLP closely recovers the known trajectories of compiled reference programs. Most base, instruction-tuned, and control models remain within the prescribed bound, whereas two reasoning-distilled models violate it substantially on several formal tasks. On A5/S5, commitment occurs up to about 2.21 times earlier than the prescribed schedule allows. The authors argue that, in these cases, the emitted CoT often behaves more like a transcript of an already-formed decision than the computation that actually produced it.
Critical strengths
-
The paper provides a genuinely useful normative reference for “premature” commitment. Prior work can often show that an answer is decodable or behaviorally recoverable early, but this alone does not establish that the timing is anomalous. The prescribed trajectory CD_P turns this qualitative concern into a task-dependent quantitative bound. I view this framing as the strongest conceptual contribution of the paper.
-
The empirical control suite is unusually thorough. In particular, the random and orthogonalized-state controls convincingly separate generic disruption from source-directed redirection: arbitrary perturbations frequently change the answer, but almost never redirect it to the source answer, whereas a genuine source-state transplant does so at a very high rate. The operation-aligned re-analysis, matched-accuracy comparison, layer-patching controls, and seed/bootstrap checks further address several obvious alternative explanations.
Critical weaknesses
-
Presentation and version-consistency issues. There is already a clear figure–text inconsistency in the main paper. Figure 4 contains five panels (Dyck-2, A4/S4, A5/S5, FSA, and Entity), whereas the caption refers to A5/S5 as the “right” panel, although the rightmost panel is Entity. This suggests that the figure and its description may not have been fully synchronized. Similar issues appear more extensively in the appendix. Section C.6 reports the same-answer sensitivity analysis over only 35 model-task cells, while the main study contains 9 models × 5 tasks = 45 cells, with no explanation of which ten cells were omitted. Section K.5 also states that four instances were manually audited while describing one instance from every exact-program task plus one entity-tracking instance, which does not appear to match that count. In addition, Figures 9 and 10 are not fully aligned with their captions or surrounding descriptions. Each issue may individually be editorial, but their repeated occurrence across both the main text and appendix makes it difficult to verify that all figures, counts, and analyses correspond to the same final experimental version.
-
Experimental scope and generality. The experimental setup limits the generality of the main conclusion. The strongest certified results are obtained on highly structured state-tracking tasks for which the authors can explicitly specify a reference program. This is a well-motivated setting for validating CLP and the prescribed trajectory, but it leaves open whether the same phenomenon extends to broader forms of reasoning. The only natural-language task, entity tracking, relies on an approximate prescribed trajectory and is therefore treated as exploratory evidence. In addition, although the two reasoning-distilled checkpoints use different backbones, both are distilled from traces generated by the same DeepSeek-R1 teacher. The current experiments therefore do not cleanly distinguish a general effect of reasoning distillation from a teacher-specific or trace-distribution-specific effect. Finally, the headline CLP results are primarily based on interventions on the last-layer residual stream. The cumulative layer-patching ablations are helpful, but the main conclusions still depend to some extent on this intervention choice.
Questions for authors
-
The main experimental design contains 45 model-task cells (9 models × 5 tasks), and Section I.11 explicitly reports statistics over all 45 cells. Why does Section C.6 state that the same-answer sensitivity analysis is conducted over only 35 cells? Which ten cells were excluded, what was the exclusion criterion, and do the reported maximum changes in U and k* remain the same if all 45 cells are included?
-
Several figures appear inconsistent with their captions or surrounding descriptions. For example, Figure 9 is a stacked-bar plot although the caption refers to a “rising curve,” and Figure 10 visually reports prompt-end probe accuracy across models while the text describes an overlay of probe trajectories with CD_M and CD_P on R1-DISTILL. Could the authors clarify whether these are purely manuscript-editing errors and confirm that the numerical results, released result files, and final plotting scripts all correspond to the same experimental snapshot?
-
Both reasoning-distilled checkpoints use different backbones but are distilled from traces generated by the same DeepSeek-R1 teacher, and the strongest certified evidence comes from structured state-tracking tasks with explicit reference programs. What evidence supports attributing the observed premature commitment to reasoning distillation more generally, rather than to DeepSeek-R1-specific trace properties or to this particular task family? If available, results from a different reasoning teacher or from an additional non-state-tracking task would substantially strengthen this claim.
6 comments
Response to Reviewer SLLu (Part 2 of 2)
W2, Q3. One teacher, one task family, one layer.
A second teacher (new Table 4 in §6.4; intervals in Table 19, Section I). We added OpenThinker3-7B (Guha et al., 2026), a supervised fine-tune of Qwen2.5-7B-Instruct on 1.2M reasoning traces generated by QwQ-32B, a different reasoning teacher that was itself trained with outcome-based reinforcement (Qwen Team, 2025). Its own parent is the control, and both run under the main protocol:
| Task | Model | [95% CI] | crossing | |
|---|---|---|---|---|
| Qwen2.5-7B (inst) | [, ] | |||
| OpenThinker3-7B | [, ] | |||
| Qwen2.5-7B (inst) | [, ] | |||
| OpenThinker3-7B | [, ] | |||
| FSA | Qwen2.5-7B (inst) | [, ] | ||
| FSA | OpenThinker3-7B | [, ] |
The parent stays within the bound on every formal task and the child crosses it on every formal task. The evidence for attributing the effect to reasoning distillation is therefore three students from three different parents under two teachers, each violating the bound while its own parent does not. The violation is smaller for OpenThinker3-7B than for the two DeepSeek-R1 students, so its size varies with teacher and backbone; its presence does not. The same table adds Qwen2.5-Math-7B-Instruct, the base of R1-Distill-Qwen after chain-of-thought SFT and GRPO reinforcement: it stays within the bound on every formal task (), so on the backbone where both routes are measured, distilling a reasoning teacher's traces moves commitment ahead of the evidence and this reinforcement stage at the student does not.
Task family. The certificate requires a reference program, which is what makes the bound a bound rather than a heuristic. We therefore claim certified violations only on the three formal tasks and report entity tracking as an extension with an approximate trajectory (Table 2). Extending the certificate needs tasks whose intermediate state can be written down, not a change to CLP. A concrete next case is PrOntoQA (Saparov & He, 2023), where every example is generated from a first-order ontology with a unique gold proof and each chain sentence corresponds to one proof step.
Last-layer intervention. The cumulative layer patching of Section I.3 addresses this choice. On at , widening the patch from the last layer to all layers raises R1-Distill's source-target rate monotonically from to and the 8B parent's from to . The ordering of the two models is the same on every patch set, so the last-layer reading is the conservative one.
Summary of revisions. Figures 4, 9, 10 and 12: captions. Section C.6: the check on all 45 cells. Section K.5: audit description. Sections I.7 and I.8: text. §6.4, §7 and Section I: Tables 4 and 19, second teacher and direct RL. Section Q: the limitation on distillation sources.
We hope this resolves the consistency concerns, and we would be glad to clarify anything further.
Dear Reviewer SLLu,
Thank you again for the close reading that caught the inconsistencies between figures, counts and text. Each is corrected in the revised PDF, and the second-teacher experiment you suggested is now in Table 4. We would be glad to know whether the corrections resolve your concern about consistency, and whether the new checkpoints address the question of a single teacher. We are happy to clarify anything further in the remaining discussion period.
Best regards,
The Authors
Response to Reviewer SLLu (Part 1 of 2)
Thank you for the close reading! Every inconsistency you list is real, and each is corrected in the revised PDF (changes in green). None of them changes a reported result. We answer W1 together with Q1 and Q2, then W2 with Q3, where we add the experiment you suggest.
W1, Q1, Q2. Figure, count and caption consistency.
Q1, the ten missing cells. They were the two Qwen checkpoints (Qwen2.5-Math and R1-Distill-Qwen) on the five tasks; the check had been run before that family was added, and there was no exclusion criterion. On all 45 cells the conclusions are the same: the range of U has median 0 and maximum 0.030, the range of k∗ has median 0.0035 and maximum 0.0116, and on 43 of 45 cells the range is narrower than that cell's own bootstrap interval for k∗ (the two exceptions exceed it by less than 0.002).
Q2, same snapshot. These were manuscript-editing errors: in each case the figure and the released file were correct and the text had not followed. We regenerated every table and figure from the released result files with the single script named in the Reproducibility Statement and checked each caption and count against its output.