- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This work shows that LLM may still fail even though all the previous steps are provided correctly. Based on controlled tasks, they find that these failures primarily reflect choosing the wrong computation to perform next rather than executing the right one incorrectly. The experiment also shows that identifying the next step or the remaining computation can mitigate this issue.
Critical strengths
The definition of the task and the experiment is carefully controlled.
The proposed method is effective to mitigate the transition errors.
Critical weaknesses
-
The main concern for this work is its generalization to real-world complex reasoning problem. The experiments are conducted on non-standard benchmarks. Even though the findings provide some insights, they may not be practically useful.
-
The chosen tasks are simple and the models evaluated are either old or small. It is necessary to evaluate on strong LLMs to validate the claims.
-
Some statements are difficult to follow due to overly-long sentences, dense terminology or lack of logical connections. For example, in lines 476-478, “A model’s own steps carry the next operation along with each result; a record of results carries the results alone, and the model enters the remaining work at the wrong step while holding the right value.” would be easier to understand if you convert it to “A model's own reasoning steps may encode both the computed result and the next operation, whereas a record containing only intermediate results may leave the next operation unspecified. As a result, a model may resume computation at the wrong step despite having the correct intermediate value.”
Questions for authors
- Is it possible to validate the claims on widely-used benchmarks such as gsm8k?
- In Figure 1, for fresh problem and continue from a record, should we regard them as two different problems instead?
2 comments
Response (Part2)
W2 — Strong LLMs. The failure persists in strong 2026 models, and reasoning models avoid it only by recomputing the completed steps. Our submission covered four model families, including the 2026 Qwen3.5 releases [L266–268, L290–291]. We now add Qwen3.5-27B and Qwen3.5-122B-A10B (thinking off and on), Llama-3.3-70B, and the reasoning models gpt-oss-20b and gpt-oss-120b (Section 4.4, Figure 2b, Table 11). All record conditions share the 192 programs, the correct record, and the demonstrations; only one appended sentence varies.
| Model | own trajectory | one step | record | + value | + next node |
|---|---|---|---|---|---|
| Llama-3.3-70B | 0.932 | 0.995 | 0.630 | 0.599 | 0.943 |
| Qwen3.5-27B, no thinking | 0.984 | 1.000 | 0.865 | 0.844 | 0.990 |
| Qwen3.5-122B-A10B, no thinking | 0.995 | 0.995 | 0.844 | 0.802 | 0.995 |
| gpt-oss-20b, low effort | 1.000 | 1.000 | 0.844 | 0.542 | 0.927 |
| gpt-oss-20b, medium effort | 1.000 | 1.000 | 0.974 | 0.771 | 0.995 |
| gpt-oss-120b, medium effort | 0.995 | 1.000 | 0.958 | 0.938 | 1.000 |
| Qwen3.5-27B, thinking | 1.000 | 1.000 | 1.000 | 0.995 | 1.000 |
| Qwen3.5-122B-A10B, thinking | 0.990 | 1.000 | 1.000 | 0.995 | 1.000 |
Without explicit reasoning, Llama-3.3-70B, Qwen3.5-27B, and Qwen3.5-122B reach only 0.630, 0.865, and 0.844 from a correct record, although each executes the next step alone at 0.995–1.000. One sentence naming the next node restores 0.943, 0.990, and 0.995 (+0.312 [0.245, 0.380], +0.125 [0.078, 0.177], +0.151 [0.104, 0.203]); restating the value does not. Reasoning models reach high accuracy by recomputing what the record leaves out: at medium effort, gpt-oss re-tests the last completed node's branch condition in 51–57% of record prompts, and both Qwen3.5 models with thinking re-execute the whole program in every record prompt, spending more tokens than when solving from scratch (4,102 vs 2,325; 3,378 vs 2,339). At lower effort the loss returns (gpt-oss-20b: 0.844), and naming the next node again restores it (0.927).
W3 — Long sentences. We adopted the reviewer's suggested rewrite. Section 6 now reads: "A model's own reasoning steps may encode both the computed result and the next operation, whereas a record containing only intermediate results may leave the next operation unspecified. As a result, a model may continue at the wrong step despite holding the correct intermediate value." We shortened long sentences throughout, and new Tables 1 and 2 list the prompt conditions and the experimental setup.
Q2 — Figure 1: two different problems? They are the same remaining computation by construction [L089–091, L185–186]: the same current value, the same remaining operations or program nodes, and the same answer. The fresh problem states that remainder directly; the continuation presents the full problem with a correct record of the completed steps. The fresh problem certifies that the remaining work is within the model's ability [L184–186], and the cause of the gap is isolated by changing one thing inside the record prompt [L303–310; new Figure 2]: adding the successor to each record line cuts first-step transition errors from 0.24–0.89 to at most 0.009, and appending "The next node is Y." lifts accuracy from 0.45/0.51/0.38 to 0.78/0.82/0.98, whereas restating the value does not.
Response (Part1)
We thank the reviewer for the comments and for the concrete writing suggestion, which we adopted. We are glad that the reviewer found that "the definition of the task and the experiment is carefully controlled" and that the proposed method "is effective to mitigate the transition errors". Reviewer LrVd likewise describes the comparisons as "carefully matched" to "separate continuation difficulty from execution ability". We ran both requested evaluations, and the revised PDF shows all changes in blue. Bracketed numbers such as [L266] are line numbers of the original submission.
W1 / Q1 — Standard benchmarks such as GSM8K. Yes, and they show the boundary our account predicts. We added GSM8K and the code benchmark CRUXEval (Section 4.5, Table 3, Appendix H), both with prefixes that are correct by construction. Accuracy for the models without explicit reasoning:
On CRUXEval, the next line depends on loop and branch state that the record does not spell out. Continuing from the correct execution record is 5.6–17.2 points less accurate than predicting from scratch for six of the seven models, 17 points for Qwen3.5-122B (−0.172 [−0.225, −0.123]), and only 175–227 of 285 continuations start at the next line to execute. On GSM8K, the problem text names each quantity to compute next, so the record suffices, for all ten model configurations including the reasoning models (differences −0.005 to +0.029). A correct record is enough when the next step can be read off the problem, and fails when it must be inferred from the state. Records of completed steps are how reasoning is verified, truncated, relayed, or reused in practice [L090–092], so this boundary matters for real systems. The controlled tasks remain our core evidence because they make every step verifiable; as Reviewer iDiv notes, this "wouldn't be possible in a setting where only the final answer can be verified."