acceptodds
Under review as a conference paper at ICLR 2027

After Error Feedback: Continued Access to Failed Reasoning Contributes to Contextual Drag

Abstract

Iterative LLM workflows increasingly rely on previous attempts and feedback to improve subsequent solutions. However, when earlier attempts are incorrect, retaining them may paradoxically impair later reasoning, a phenomenon known as contextual drag. While prior work has documented this effect, it remains unclear whether the interference arises from the initial exposure to the failed attempt or from the model's continued access to it during subsequent reasoning. We introduce a same-text attention intervention that isolates this mechanism by preserving the full prompt, token positions, prefill computation, and first-token logits, while blocking only later generated queries from attending to the failed draft. Across controlled evaluations, Qwen3-8B on selected Game24 problems shows that accuracy drops from 87.0% in clean context to 39.1% with full access to a failed draft, but recovers to 78.3% when continued access is removed, yielding a +39.1 percentage-point effect (95% question-cluster bootstrap interval: +23.9 to +54.3). Additional models and two MATH settings show consistent trends. In contrast, blocking verified correct history reduces accuracy, indicating that the effect depends on the retained history and its feedback. These results identify continued direct access after initial processing as a causal contributor to contextual drag.

Primary area:
foundation or frontier models, including LLMs
Keywords:
contextual drag, large language models, reasoning reliability, inference-time intervention, attention control, self-refinement

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yFaxOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies whether continued access to previous failed reasoning contributes to accuracy loss after a model has already processed it. The authors introduce an attention intervention that preserves the prompt and its initial processing but prevents generated tokens from directly attending to the failed attempt. They compare this intervention with normal attention, earlier blocking, and removing the failed history. Their experiments on selected Game24 and MATH problems show that blocking continued access recovers much of the accuracy loss. This finding is suggesting that later access to failed reasoning contributes to contextual drag.

Critical strengths

None

Critical weaknesses

The main issue that leads me to recommend rejection is the limited contribution. The fact that failed reasoning in the context can reduce accuracy has already been studied. This paper adds a controlled experiment showing that blocking continued attention to a failed attempt can recover much of the accuracy loss, even after the model has processed it. I understand that blocking is used to separate initial exposure from continued access, but I am not convinced that this additional insight is sufficiently substantial. The paper also does not establish a practical advantage over simply removing the failed history, which achieves higher accuracy as shown in the paper. Finally, the evaluation is small and based on selected questions with only two seeds per question. Although the observed effect could be substantial, this provides limited evidence of how broadly the finding holds. Altogether, I do not think the contribution and supporting evidence meet the bar for ICLR.

Questions for authors

I have no specific clarification questions.

3 comments

We thank the reviewer for engaging with the distinction between initial exposure and continued access. The comparison with deletion prompted us to clarify what the same-text intervention identifies and to report how LateBlock compares with Clean in both Game24 cohorts.

W1 — What the study adds to contextual drag

“The fact that failed reasoning in the context can reduce accuracy has already been studied.”

Cheng et al. showed that failed history can impair reasoning despite error feedback. Cheng et al., Contextual Drag

Deletion changes both initial exposure and later access. Full and LateBlock preserve the same prompt, prefill, and error feedback and vary direct access only after the first generated token. On 82 frozen Game24 questions disjoint from the original 23, accuracy increased from 55/164 under Full to 95/164 under LateBlock, a gain of 24.39 percentage points (95% CI [13.41, 35.37]). This locates a causal contribution during generation, after the model has processed the draft and feedback.

In a 2 × 2 control on 50 questions drawn from the 102 eligible Game24 questions, late blocking improved accuracy by 29.0 points for same-problem failed drafts and 3.0 points for token-length-matched failed drafts from other problems under identical feedback. The difference was +26.0 percentage points (95% CI [+5.0, +46.0]). Both intervals above use question-level bootstrap resampling with paired seeds kept together.

Revised manuscript: §3.4 and Table 3, §3.5 and Table 4 (p. 7), §4.1 (p. 8), and Appendix H (p. 19).

W2 — Comparison with removing the draft

“The paper also does not establish a practical advantage over simply removing the failed history, which achieves higher accuracy as shown in the paper.”

Clean is the relevant baseline when the draft can be omitted before the retry. It scored higher than LateBlock in both cohorts, 40/46 versus 36/46 on the original 23 and 104/164 versus 95/164 on the confirmation 82. The LateBlock − Clean differences were −8.7 pp (95% CI [−23.9, +4.3]) and −5.49 pp (95% CI [−15.24, +4.27]), respectively. Both intervals span zero, and an equivalence claim would require a prespecified margin. Full–LateBlock isolates the accuracy effect of a separate decision relevant to workflows that retain a failed draft for audit or reproducibility. After the model has read the draft, the workflow can retain the record and error verdict while restricting direct access during the retry.

Revised manuscript: §3.2 (p. 5), Table 2 (p. 6), and §3.4 and Table 3 (p. 7).

W3 — Sample size and decoding seeds

“Finally, the evaluation is small and based on selected questions with only two seeds per question.”

The two seeds share one fixed failed draft per question, and the confidence interval resamples questions with both seeds together. Across 164 paired comparisons, LateBlock produced 62 wrong-to-correct and 22 correct-to-wrong transitions. These pairs come from 82 screened questions frozen before retry evaluation and disjoint from the original 23.

Revised manuscript: §3.4 and Table 3 (p. 7), §6 (p. 10), and Appendix F (pp. 17–18).

I thank authors for their response and the revision they made.

W3: I appreciate the authors adding more samples to their evaluation. This makes their results more reliable, but it does not resolve my concern about generalizability. All the tasks they tested are mathematical, while their main claim is about reasoning more broadly. I think they should either evaluate other types of reasoning tasks or narrow their claim to the mathematical settings they tested. In the latter case, my concern about the limited scope of the contribution would remain.

W2: Thank you for the clarification. I agree that comparing Full and LateBlock isolates the effect of continued access after the model has already read the failed draft. My concern is about the practical choice when the draft can be omitted before the retry. Clean had higher accuracy in both evaluations, although neither difference was statistically conclusive. If the draft needs to be retained for auditing, it can still be saved in a log without being included in the retry prompt. I therefore see the experimental value of LateBlock, but I do not see a practical advantage over removal in that setting.

W1: I understand how this paper builds on prior work. The new experiments strengthen that finding on Game24, but as I mentioned earlier, I remain unconvinced about the breadth of the contribution. On that point, my assessment remains unchanged.

We thank the reviewer for clarifying the concerns about generalization and draft removal.

W3 and W1 — Evidence beyond mathematics. We added ReClor, a natural-language argument reasoning benchmark, and scoped our empirical claim to the evaluated retry settings on Game24, MATH, and ReClor. We screened 476 ReClor validation questions in a fixed order and retained all 34 for which one Qwen3-8B Thinking attempt produced a completed reasoning trace and a scorable wrong answer. With two paired retry seeds per question, Clean, Full, and LateBlock scored 22/68 (32.35%), 4/68 (5.88%), and 24/68 (35.29%), respectively. Unparsable final answers counted as incorrect.

Holding the prompt, prefill, and first sampled token fixed, LateBlock improved accuracy over Full by 29.41 percentage points (95% question-bootstrap CI [+16.18, +42.65]). Of the 68 paired retries, 22 went from incorrect to correct and two went in the opposite direction. Across seeds, accuracy improved on 15 questions, declined on one, and tied on 18. This shows that continued direct access to a failed draft can impair retry accuracy in the evaluated non-mathematical setting. The ReClor evidence is limited to one model and questions selected after a scorable initial failure. See §3.6 and Table 6 (p. 8), §6 (p. 10), and Appendix J (p. 19).

W2 — Omitting the draft before retrying. We agree that the draft can be kept in a separate audit log and omitted from the retry prompt. Clean had a higher point estimate in both Game24 cohorts. On ReClor, the LateBlock–Clean difference was +2.94 percentage points (95% CI [−10.29, +17.65]). These results leave the relative accuracy of late blocking and omission uncertain. The Full–LateBlock contrast identifies the effect of restricting direct access after the model has already processed the failed draft. See §4.1 (p. 9).

Reviewer_jnBMOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies whether continued access to a failed solution harms subsequent reasoning after explicit error feedback. LateBlock preserves the prompt, prefill computation, and first generated token, then blocks direct attention to the failed draft during further generation. On selected Game24 problems, Qwen3-8B accuracy rises from 39.1% to 78.3%. Additional models and MATH tasks show similar directional effects, while blocking correct history can hurt performance.

Critical strengths
  • Preserving the same prompt, prefill, and first token provides a clean test of continued access after initial exposure.

  • The direction agrees across models and tasks, while the correct-history control shows that blocking can also harm performance.

Critical weaknesses
  • The study uses 23 selected Game24 problems and 16 MATH problems, with two seeds and one failed draft per problem. Generality beyond these examples remains unclear.

  • Correct and failed histories differ in both content and feedback labels. Without a matched irrelevant-history control, the effect's specificity to failed reasoning remains unresolved.

Questions for authors
  • Does the effect persist on independently sampled problems without mixed-outcome screening?

  • As an additional control, what happens when equally long irrelevant history is blocked under the same feedback label?

  • How sensitive is the effect to the choice of failed draft for each problem?

3 comments

We thank the reviewer for highlighting how the shared prompt, prefill, and first token isolate the effect of continued access after initial exposure. The suggested control gave us a direct way to test whether the blocking effect differs between same-problem failures and equally long drafts from other problems under identical feedback.

W1 and Q1 — Sample size and screening

W1. “The study uses 23 selected Game24 problems and 16 MATH problems, with two seeds and one failed draft per problem.”

Q1. “Does the effect persist on independently sampled problems without mixed-outcome screening?”

The confirmation estimates the effect for retries after a verified failure on screened Game24 questions. A question qualified with at least two verified successes and two complete substantive failures in eight attempts. We froze the 20/82 development and confirmation split before retry evaluation. On 82 frozen Game24 questions disjoint from the original 23, accuracy increased from 55/164 under Full to 95/164 under LateBlock, a gain of 24.39 percentage points (95% CI [13.41, 35.37]). The question bootstrap kept both paired seeds together. This confirms the access effect in a new cohort within the mixed-outcome setting.

Revised manuscript: §2.3 (p. 4), §3.4 and Table 3 (p. 7), §6 (p. 10), and Appendix F (pp. 17–18).

W2 and Q2 — Draft content, feedback, and the matched control

W2. “Correct and failed histories differ in both content and feedback labels.”

Q2. “As an additional control, what happens when equally long irrelevant history is blocked under the same feedback label?”

The original correct-history control changes the draft and its truthful feedback together. Blocking those drafts reduced accuracy from 16/16 to 13/16. To test the effect under identical incorrect-answer feedback, we crossed draft type (same-problem or different-problem failed draft) with access (Full or LateBlock). We froze donor assignments before collecting intervention outcomes. We selected failed drafts with scorable final expressions from other problems that shared at most one input operand with the target. Matching prioritized nonoverlapping operands among drafts close in token length, with a fixed tie-break. In 25 of the 50 pairs, we inserted whitespace into the shorter draft to align the token counts and span positions while preserving both final expressions. All tokens outside the draft span were identical.

For the matched-draft control, we randomly sampled 50 Game24 questions from the 102 eligible questions before running the experiment. Each question was evaluated in four conditions with two paired seeds, yielding 400 generations. The sample included 42 confirmation and 8 development questions.

Failed draftFullLateBlockLateBlock − Full, pp [95% CI]
Same problem39/10068/100+29.0 [+13.0, +44.0]
Matched draft from another problem70/10073/100+3.0 [−8.0, +14.0]

Full accuracy was 31 points higher with donor drafts, and the gap narrowed to 5 points under LateBlock. The prespecified interaction across all 50 targets was +26.0 pp (95% CI [+5.0, +46.0]), showing a larger blocking benefit for same-problem failed drafts in this matched comparison. The donor effect alone remains uncertain, with an interval spanning a loss and a gain. We counted unresolved outputs as incorrect and resampled questions with both seeds and all four conditions together. The prespecified sensitivity rule kept 46 questions with no shared operands and at most four padding tokens. Their interaction was +29.35 pp (95% CI [+7.61, +51.09]).

Revised manuscript: §2.3 (p. 4), §3.5 and Table 4 (p. 7), Appendix G and Table 14 (p. 18), and Appendix H (p. 19). The number of padded pairs comes from the run records.

Q3 — Choice of failed draft

“How sensitive is the effect to the choice of failed draft for each problem?”

Both seeds use the same failed draft for each problem, so they sample decoding variability conditional on that draft. Estimating sensitivity within a problem would require multiple failed drafts for that problem. Across problems, the access effect reproduced on 82 new questions with their own fixed drafts.

Revised manuscript: §2.3 (p. 4), §6 (p. 10), and Appendix F (p. 17).

Reviewer_jnBMOpenReview ↗

Response by Reviewer

Thank you for addressing my concern. I will raise my score.

We sincerely thank the reviewer for the constructive feedback and for reconsidering our work. Your comments have helped us strengthen the experimental controls and clarify our findings. We greatly appreciate your thoughtful engagement and will continue to improve the manuscript.

Reviewer_KFumOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigated a problem that arises in iterative LLM workflows, where retaining a failed attempt and its error feedback can lower accuracy on the next attempt. This effect is called "contextual drag". It was unclear whether the harm comes from the initial reading of the failed attempt or from the model's continued access to it during later generation. Prompt-level fixes such as deleting or rewriting the history cannot separate these two causes.

The authors propose a same-text attention intervention called LateBlock. The full prompt, token positions, prefill computation, and first-token logits stay unchanged. After the first token is sampled, generated queries are blocked from attending to the failed draft. The authors also test PathBlock, which starts blocking earlier, at the prompt suffix after the draft.

The contributions within paper include: A new concept of Contextual Drag is proposed that is separated into initial exposure and continued access. In the proposed method, the intervention isolates continued access without changing the prompt. In experiments, on selected Game24 problems, Qwen3-8B recovers from 39.1% to 78.3% accuracy with late blocking, with the consistent direction that appears in other models and MATH settings. Another finding is blocking verified correct history lowers accuracy, so the effect depends on the retained history.

Critical strengths
  1. The paper investigated an interesting problem that current iterative LLM systems such as self-refinement and reflective agents routinely keep failed attempts, so the harm from doing so matters in practice. The paper frames the question well by separating two possible causes, initial exposure and later access, and it explains why prompt-level fixes like deleting or rewriting history cannot tell them apart. That makes the question precise and testable.

  2. In experiments, the causal factors are clearly isolated. I.e., Full and LateBlock share the same tokens, positions, and first-token logits, so only later access differs. This lets the authors attribute the recovery to continued access rather than to the prompt.

  3. The authors honestly report the uncertainty. The bootstrap interval (+23.9 to +54.3) shows the effect is not a single lucky number, even if the sample is small.

  4. The experimental results are mostly consistent across different settings. E.g., Qwen3-32B, Gemma 4 E4B, and two MATH groups show the same pattern, which supports robustness.

  5. It is a useful finding that Blocking correct history lowers accuracy, which shows the effect depends on the retained content and is not a generic "remove history" rule.

Critical weaknesses
  1. Overall the investigated problem is about a relatively small and engineering trick. The proposed solution is mainly a diagnostic tool rather than a new technology about learning.

  2. In experiments, a very small set of selected samples, Only 23 Game24 questions, are used, and they were chosen for mixed baseline outcomes, so generalization is limited.

  3. Also in experiments, the cross-model and MATH evidences are weak. Other models vary in architecture and implementation, and MATH has unresolved answers and condition-dependent termination, so these results are required.

  4. The method assumes externally verified incorrect drafts and known token spans, which real systems may not have, and the authors do not show a deployable mitigation.So this leading to limited practical value.

Questions for authors

n/a

1 comment

We thank the reviewer for recognizing the relevance of this question to iterative LLM systems that retain failed attempts, and for highlighting how the design separates initial exposure from later access. The comments on external verification and known draft spans helped us clarify the conditions under which the result informs history management.

W1 — Scientific contribution

“The proposed solution is mainly a diagnostic tool rather than a new technology about learning.”

The contribution is to identify whether later access affects retry accuracy once the model has already processed the failed draft and feedback. Deletion changes initial exposure and later access together. Full and LateBlock use the same prompt and prefill, and the first sampled token matched in all 164 confirmation pairs. Blocking begins at the query that predicts token two. The contrast measures the total effect of changing access, including changes in attention normalization and the subsequent generated trajectory. With failed drafts on the eight questions used in the correct-history comparison, exact reuse of the failed final expression fell from 9/16 under Full to 2/16 under LateBlock.

Revised manuscript: §2 (pp. 2–3), §§4.1–4.3 (pp. 8–9), and Appendix F (p. 17).

W2 — Generality of the selected Game24 sample

“they were chosen for mixed baseline outcomes, so generalization is limited.”

On 82 frozen Game24 questions disjoint from the original 23, accuracy increased from 55/164 under Full to 95/164 under LateBlock, a gain of 24.39 percentage points (95% CI [13.41, 35.37]). We froze the cohort before retry evaluation and resampled questions with both seeds together. The cohort confirms the effect within the mixed-outcome screening setting.

Revised manuscript: §3.4 and Table 3 (p. 7), and §6 (p. 10).

W3 — Cross-model and MATH evidence

“Other models vary in architecture and implementation, and MATH has unresolved answers and condition-dependent termination, so these results are required.”

Qwen3-32B and Gemma 4 E4B show the same direction on the shared 15-question Game24 subset. Each contrast holds the model and inference implementation fixed. On MATH, we report scored success at a fixed budget alongside scoring coverage and termination by condition, including more length stops under LateBlock on MATH-train. We use those panels as a directional cross-task check.

Revised manuscript: §3.3 (pp. 5–6), Table 2 (p. 6), §3.6 and Table 5 (p. 8), and Appendix B and Table 7 (p. 14).

W4 — Verified failures and known draft spans

“The method assumes externally verified incorrect drafts and known token spans”

Those inputs define the intervention. For workflows that already record verified failures, the result shows that supplying an error verdict and allowing continued direct access to the failed draft are separate decisions. Automatic verification, span selection, and deployment effectiveness are outside the evaluation.

Revised manuscript: §2 (p. 2), §2.2 (p. 3), and §6 (p. 10).

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your reviews and contributions. As we are now halfway through the discussion period, I would like to encourage you to carefully read the authors' responses and engage in further discussion where appropriate. Please also indicate whether the rebuttal has addressed your concerns or changed your assessment.

Thank you for your time and effort!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.