- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks whether LLMs re-verify an incorrect intermediate value planted into their own previously correct execution trace of a short arithmetic program. It controls the “verification depth” k: errors at k=0 can be caught by comparing against an earlier line, while errors at k >= 1 require recomputation. Across 14 models, strong models usually catch k=0 errors, but one-operation errors propagate at 88–100%, even though the models compute the same sum correctly when asked directly. Showing the operands, changing the stated authorship, adding verification instructions, enabling reasoning, and activation steering rarely restore checking. Fine-tuning on about 600 paired examples teaches selective rechecking on two of three models, with partial generalization to unseen depths. The authors conclude that models do not spontaneously re-verify values that already appear settled in context.
Critical strengths
[S1] The protocol is clean. Outcomes are graded by program execution, and fixing trace length and operand distance lets verification depth isolate the cost of recomputation from how far back the model must look.
[S2] The k=0 vs. k=1 dissociation is sharp and consistent across 14 models and 5 developers. The direct-probe control (“What is a + b?”) separates the ability to verify from the default use of that ability.
[S3] The fine-tuning result is informative: a few hundred examples induce selective checking on Qwen2.5-32B (3/720 false checks) that partially generalizes to an unseen depth.
[S4] Reporting: pre-registered run order, prefill vs. user-turn calibration, refusals reported as unmeasured, and hyperparameters fixed before the first run.
Critical weaknesses
[W1] The format confounds the core claim.
Every condition keeps the instruction to write “nothing else” (plus “do not repeat lines” under the user-turn method), and no condition gives scratch space before continuing. Propagation may therefore reflect format compliance rather than a failure to check. The reasoning runs (GPT-5.1, R1-Distill) are never combined with a verification instruction, and the instruction arms never allow reasoning, so the most natural fix, room to think plus a reasonable instruction, is untested. The abstract’s claim that the failure “persists even when given verification instructions” is overstated for this reason.
[W2] Practical significance is asserted rather than shown.
Deliberately corrupting a committed value without any cue is an artificial scenario. The relevant real settings are the model’s own slips inside its reasoning, unflagged tool or agent outputs, and adversarial injection, but none of these are tested. The paper explicitly never perturbs the thinking channel, and the natural-slip analysis (A.3) covers only small open models with reasoning off and no instructions. If a cheap prompt plus reasoning fixes the behavior, the contribution reduces to an observation about a default prior/behavior.
[W3] Key protocol details are missing or ambiguous.
he paper does not say where the perturbed trace sits relative to R1-Distill’s <think> block, which method the GPT-5.1 reasoning run uses (inferable only indirectly), or what the exact token boundary is at the end of the prefix. Prefill and user-turn methods are mixed across models even though the method changes readable-error rates (e.g., Haiku 4.5: 0.04 vs. 0.36), which weakens cross-model comparisons.
[W4] The phenomenon looks like an expected, shallow default behavior rather than a meaningful limitation.
- Continuing from a self-consistent prefix without re-deriving committed values is exactly what ordinary pretraining and SFT data teach, since such data almost never revisit values that have already been written down. The paper’s own discussion names this as a candidate explanation (“completed computations in a prefix are usually trustworthy”).
- The paper’s own evidence suggests the behavior is easy to change and is not a capability gap:
- direct probes recover the correct value 96–100% of the time;
- a worked-example prompt cuts GPT-5.5’s one-operation propagation to 2%, even under the restrictive format;
- about 600 examples and 114 LoRA updates remove it on Qwen2.5-32B.
- Under a setup that permits it, namely a reasonable verification instruction plus room/space to reason before continuing, the behavior is likely to largely disappear in modern LLMs.
- As a result, the work reads as a careful documentation of a default prior rather than as identifying a problem that is really important to solve or that needs a dedicated fix. The title and abstract frame it as a general inability, which overstates its significance.
Questions for authors
No questions at this time. Major concerns are stated in the weaknesses section above.
2 comments
Thanks for the additional efforts and rebuttal. I think this is a honest empirical study. However, I tend to keep my score due to the following limitations of this work:
- Limited novelty and value of the studied problem: The observed failure mode is quite predictable to me given the training data distribution used by LLMs. It is a good observation and confirmation, but lacks scientific novelty, as the issue can be mitigated by wide-used "tricks" including prompting and reasoning.
- Limited practical evidence: The evaluated tasks are simple, far from realistic tasks that people are using everyday.
- Limited effectiveness of the method: Fine-tuning does not automatically generalize to diverse positions. The evaluation set is also small.
- Limited explanation to the mechanism: probing/steering results do not explain the issue completely, cannot distinguish different mechanisms. Overall, I suggest authors focus on more valuable issues and make at least one dimension of the paper very strong.
Response to Reviewer NotY
Thank you for pointing out that we had not tested a verification instruction with reasoning enabled. We ran that test and revised the claim.
W1: Format and prompting. Changing the format alone helps one model and not the other. We allowed scratch work and changes to earlier lines, with no verification instruction and reasoning off, and left the examples in the prompt unchanged. Haiku 4.5 still propagates depth-one errors at 1.00 (180 continuations, 60 programs), while GPT-5.5 falls to 0.30 (60 programs).
The worked-example instruction with reasoning works even under the original format (table in the general response). Other combinations give partial results. The targeted instruction with reasoning leaves Haiku at 0.17 and 0.15 at depths one and five, with a depth-one Wilson upper bound of 0.24. We therefore describe the effect as a large reduction, which falls short of elimination in some conditions.
On GPT-5.5, reasoning alone gives 0.43 and 0.80, whereas permission to revise plus reasoning, with no instruction to check, gives 0.03 at both depths (60 programs). On GPT-5.1, permission to revise plus the worked example already gives 0.12 and 0.18 with reasoning off (60 programs); we did not test format alone on that model. The abstract now states that the original instruction result used reasoning off and the original format.
W2/W4: Why this matters and what we tested. Models usually continue without checking, and a suitable prompt can change that. The paper's contribution is to measure how this failure depends on the work needed to find the error, separately from whether the model can fix the error when asked. The general response reports a cheap fix as well as an expensive one. The cheap fix carries over only partly to small open models: with “Wait” appended, propagation remains 0.51 on Qwen2.5-7B and 0.56 on Llama-3.1-8B (72 and 144 continuations; no correct-trace controls), and some attempted corrections calculate the wrong answer or fail to finish.
For naturally occurring errors, the original analysis covers five open models with reasoning off. Of 1,396 computed errors that would change the answer, 10 end with the correct output (Appendix A.3). We added a check of errors that R1-Distill-7B states in its own reasoning, and all 14 that we found reach the final trace. The general response lists what remains untested and, with revised Section 5.7, describes how the fine-tuned check depends on error position.
W3: Protocol. Revised Appendix B gives the missing details. The partial trace ends after the complete planted line. The Anthropic API requires trailing whitespace to be removed, and we put it back before scoring. Open-weight models and Haiku/Sonnet 4.5 receive the trace as an assistant prefill; the other chat models receive it in the user turn.
These two ways of supplying the trace give different propagation rates for readable errors, which limits cross-model comparisons; Figure 3 marks the affected bars with hatching. On the instruction traces, Haiku propagates computed errors at 1.00 at both depths under both methods (180 continuations per condition), against 0.01 and 0.03 for the readable error. The larger readable-error difference in the paper comes from a different test, the copy-line test, and the revision now distinguishes the two.
R1-Distill with reasoning on receives the trace before the template opens its think block; we score the trace that follows the block and inspect the thinking separately. With reasoning off, it runs as raw completion with no chat template or think tags, and its correct starting traces were generated with thinking on. The GPT-5.1 reasoning comparison uses the Responses API with effort none versus medium, four samples per program and no temperature setting.
The “85%” figure concerns visible summaries only: 112 of 132 non-empty GPT-5.1 reasoning summaries at depth five contain no visible recalculation. Another 108 of 240 summaries are empty and provide no evidence either way. Even a non-empty summary may omit hidden computation; these summaries cannot establish whether the model internally recalculated. Appendix B and the new tables give the decoding settings and outcomes.