- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies whether continued access to previous failed reasoning contributes to accuracy loss after a model has already processed it. The authors introduce an attention intervention that preserves the prompt and its initial processing but prevents generated tokens from directly attending to the failed attempt. They compare this intervention with normal attention, earlier blocking, and removing the failed history. Their experiments on selected Game24 and MATH problems show that blocking continued access recovers much of the accuracy loss. This finding is suggesting that later access to failed reasoning contributes to contextual drag.
Critical strengths
None
Critical weaknesses
The main issue that leads me to recommend rejection is the limited contribution. The fact that failed reasoning in the context can reduce accuracy has already been studied. This paper adds a controlled experiment showing that blocking continued attention to a failed attempt can recover much of the accuracy loss, even after the model has processed it. I understand that blocking is used to separate initial exposure from continued access, but I am not convinced that this additional insight is sufficiently substantial. The paper also does not establish a practical advantage over simply removing the failed history, which achieves higher accuracy as shown in the paper. Finally, the evaluation is small and based on selected questions with only two seeds per question. Although the observed effect could be substantial, this provides limited evidence of how broadly the finding holds. Altogether, I do not think the contribution and supporting evidence meet the bar for ICLR.
Questions for authors
I have no specific clarification questions.
3 comments
I thank authors for their response and the revision they made.
W3: I appreciate the authors adding more samples to their evaluation. This makes their results more reliable, but it does not resolve my concern about generalizability. All the tasks they tested are mathematical, while their main claim is about reasoning more broadly. I think they should either evaluate other types of reasoning tasks or narrow their claim to the mathematical settings they tested. In the latter case, my concern about the limited scope of the contribution would remain.
W2: Thank you for the clarification. I agree that comparing Full and LateBlock isolates the effect of continued access after the model has already read the failed draft. My concern is about the practical choice when the draft can be omitted before the retry. Clean had higher accuracy in both evaluations, although neither difference was statistically conclusive. If the draft needs to be retained for auditing, it can still be saved in a log without being included in the retry prompt. I therefore see the experimental value of LateBlock, but I do not see a practical advantage over removal in that setting.
W1: I understand how this paper builds on prior work. The new experiments strengthen that finding on Game24, but as I mentioned earlier, I remain unconvinced about the breadth of the contribution. On that point, my assessment remains unchanged.
We thank the reviewer for clarifying the concerns about generalization and draft removal.
W3 and W1 — Evidence beyond mathematics. We added ReClor, a natural-language argument reasoning benchmark, and scoped our empirical claim to the evaluated retry settings on Game24, MATH, and ReClor. We screened 476 ReClor validation questions in a fixed order and retained all 34 for which one Qwen3-8B Thinking attempt produced a completed reasoning trace and a scorable wrong answer. With two paired retry seeds per question, Clean, Full, and LateBlock scored 22/68 (32.35%), 4/68 (5.88%), and 24/68 (35.29%), respectively. Unparsable final answers counted as incorrect.
Holding the prompt, prefill, and first sampled token fixed, LateBlock improved accuracy over Full by 29.41 percentage points (95% question-bootstrap CI [+16.18, +42.65]). Of the 68 paired retries, 22 went from incorrect to correct and two went in the opposite direction. Across seeds, accuracy improved on 15 questions, declined on one, and tied on 18. This shows that continued direct access to a failed draft can impair retry accuracy in the evaluated non-mathematical setting. The ReClor evidence is limited to one model and questions selected after a scorable initial failure. See §3.6 and Table 6 (p. 8), §6 (p. 10), and Appendix J (p. 19).
W2 — Omitting the draft before retrying. We agree that the draft can be kept in a separate audit log and omitted from the retry prompt. Clean had a higher point estimate in both Game24 cohorts. On ReClor, the LateBlock–Clean difference was +2.94 percentage points (95% CI [−10.29, +17.65]). These results leave the relative accuracy of late blocking and omission uncertain. The Full–LateBlock contrast identifies the effect of restricting direct access after the model has already processed the failed draft. See §4.1 (p. 9).
We thank the reviewer for engaging with the distinction between initial exposure and continued access. The comparison with deletion prompted us to clarify what the same-text intervention identifies and to report how LateBlock compares with Clean in both Game24 cohorts.
W1 — What the study adds to contextual drag
Cheng et al. showed that failed history can impair reasoning despite error feedback. Cheng et al., Contextual Drag
Deletion changes both initial exposure and later access. Full and LateBlock preserve the same prompt, prefill, and error feedback and vary direct access only after the first generated token. On 82 frozen Game24 questions disjoint from the original 23, accuracy increased from 55/164 under Full to 95/164 under LateBlock, a gain of 24.39 percentage points (95% CI [13.41, 35.37]). This locates a causal contribution during generation, after the model has processed the draft and feedback.
In a 2 × 2 control on 50 questions drawn from the 102 eligible Game24 questions, late blocking improved accuracy by 29.0 points for same-problem failed drafts and 3.0 points for token-length-matched failed drafts from other problems under identical feedback. The difference was +26.0 percentage points (95% CI [+5.0, +46.0]). Both intervals above use question-level bootstrap resampling with paired seeds kept together.
Revised manuscript: §3.4 and Table 3, §3.5 and Table 4 (p. 7), §4.1 (p. 8), and Appendix H (p. 19).
W2 — Comparison with removing the draft
Clean is the relevant baseline when the draft can be omitted before the retry. It scored higher than LateBlock in both cohorts, 40/46 versus 36/46 on the original 23 and 104/164 versus 95/164 on the confirmation 82. The LateBlock − Clean differences were −8.7 pp (95% CI [−23.9, +4.3]) and −5.49 pp (95% CI [−15.24, +4.27]), respectively. Both intervals span zero, and an equivalence claim would require a prespecified margin. Full–LateBlock isolates the accuracy effect of a separate decision relevant to workflows that retain a failed draft for audit or reproducibility. After the model has read the draft, the workflow can retain the record and error verdict while restricting direct access during the retry.
Revised manuscript: §3.2 (p. 5), Table 2 (p. 6), and §3.4 and Table 3 (p. 7).
W3 — Sample size and decoding seeds
The two seeds share one fixed failed draft per question, and the confidence interval resamples questions with both seeds together. Across 164 paired comparisons, LateBlock produced 62 wrong-to-correct and 22 correct-to-wrong transitions. These pairs come from 82 screened questions frozen before retry evaluation and disjoint from the original 23.
Revised manuscript: §3.4 and Table 3 (p. 7), §6 (p. 10), and Appendix F (pp. 17–18).