acceptodds
Under review as a conference paper at ICLR 2027

Contextual Drag: How Errors in the Context Affect LLM Reasoning

Abstract

Many self-improvement pipelines for large language models (LLMs) reuse prior attempts as context, relying on verification or critique to extract useful reasoning from flawed attempts. We study contextual drag: failed attempts in context bias subsequent generations toward structurally similar errors, often persisting even when models recognize them as wrong. We focus on self-drag, where a model conditions on its own incorrect drafts, mirroring iterative self-refinement. Across 10 open-weight reasoning models and 8 reasoning tasks, a single incorrect draft drops average pass@1 accuracy by around 25 points on problems where direct reasoning produces both successes and failures. For GPT-OSS-20B, which has severe single-step self-drag, iterative refinement compounds this effect into self-deterioration. Explicit error signals, from either external labels or the model’s own correct verification, do not fully eliminate the effect, suggesting that correct recognition is not sufficient to reset the draft's influence. A Game of 24 case study using tree edit distance shows that subsequent generations are structurally biased toward the in-context draft, not just less accurate. Finally, filtering the draft before reuse reduces contextual drag on every model we test, by a model-dependent amount and without training or external labels; applied recursively, it turns GPT-OSS-20B’s self-deterioration on competition math into steady improvement. These results suggest that robust self-improvement requires more than reflecting on mistakes; it requires mechanisms for context reset, independent attempts, and selective reuse of prior reasoning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Failure Mode Analysis, Contextual Denoising, Anchoring Bias

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_7me9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies “contextual drag,” showing that providing an LLM with a previous incorrect reasoning attempt can reduce its ability to solve a problem. Across a variety of models and tasks, the authors find that this effect can persist even when the model is explicitly told that the previous attempt is wrong or correctly identifies it as incorrect. They also show that filtering previous attempts before reusing them can reduce contextual drag and improve performance during iterative refinement.

Critical strengths

The paper presents a fairly comprehensive set of experiments across a broad selection of models and reasoning tasks, which strengthens the empirical evidence for contextual drag. The authors investigate several possible confounding factors, including prompt position, removal of verification, matched prompts, thinking budgets, mixed-correctness drafts, and majority voting. Additional code-generation and cross-model experiments broaden the relevance of the findings. The structural similarity analysis and iterative refinement experiments further help characterize how incorrect drafts influence subsequent reasoning. I also appreciate that the authors report cases where models remain stable or improve under iterative refinement, as well as unsuccessful training interventions, providing a more balanced assessment of the phenomenon.

Critical weaknesses
  • My main concern is about the scope of novelty. Previous work like Huang et al. (ICLR 2024) demonstrates the limitations of LLM self-correction. This current submission goes beyond by showing that incorrect drafts can continue to harm reasoning even after being flagged as wrong. However, if I am correct, the authors only verify whether the model correctly labels a draft as incorrect, without examining whether it can identify and explain the specific reasoning error. It remains unclear whether the model continues to follow an error it understands or simply cannot locate the mistake. While the structural similarity analysis provides additional evidence, the distinction between contextual drag and known self-correction or anchoring failures is not convincingly established for me. Showing that the effect persists after accurate error localization could strengthen the novelty claim.

  • FILTER first generates a fresh plan without seeing the erroneous draft, then filters the draft and produces a solution. Does the reported improvement come partly or primarily from independent planning rather than filtering itself? The authors acknowledge that they do not separate these contributions. A planning-only baseline may be necessary to determine whether retaining information from the incorrect draft provides any additional benefit.

  • REVISE and FILTER frequently perform worse than DIRECT despite requiring additional model calls. While they reduce the damage caused by erroneous drafts, this does not necessarily demonstrate that reusing those drafts is worthwhile. The paper reports some independent-sampling comparisons, but lacks compute-matched comparisons between the complete mitigation pipelines and fresh independent attempts. Even though recursive FILTER improves over the initial baseline on some models, it is still unclear to me whether comparable or better improvements could be achieved by simply generating fresh solutions using the same inference budget.

  • The main experiments select only problems for which 16 clean-slate generations produce both correct and incorrect answers. While this isolates problems where improvement is possible, the reported average accuracy drop may not reflect degradation across the full benchmark. Furthermore, DIRECT accuracy is calculated using the same samples employed for problem selection, rather than independent samples. Some model–task subsets contain only three to five problems, and different models are evaluated on different subsets, making comparisons less reliable. Reporting full-benchmark results, independent DIRECT estimates, and uncertainty measures would strengthen the evaluation.

Questions for authors

See the weaknesses above.

3 comments

Dear Reviewer, thank you for your time and insightful feedback! Your comments on error localization and the planning-only baseline helped us understand more precisely when contextual drag occurs. We are glad that you find the experiments comprehensive. Following your suggestions, we ran several new analyses:

  • W1 (novelty compared to existing works on LLM self-correction): Thank you for the suggestion to audit error localization. With LLM-assistance we ran this audit and found three results that go beyond known self-correction failures. (i) The drag persists after successful detection: 36% of the generations that correctly reject the draft do so without locating the error, and these fall 29.2 pp below DIRECT while rebuilding the draft's reasoning, which detection-based accounts of self-correction do not predict. (ii) Explicit external labels do not remove the drag either. (iii) The draft carries over its intermediate reasoning, not only its answer. We also find that an accurate diagnosis largely restores DIRECT performance, and we will state this boundary precisely in the paper.
  • W2 (planning-only baseline): On wrong drafts, FILTER's gain comes from draft-blind planning. We find that keeping the filtered draft pays off only when the draft is correct.
  • W3 (compute-matched comparison): When drafts are known to be wrong, fresh samples with voting are better. In iterative refinement, recursive FILTER is 1.7 pp ahead of voting after one round and 2.9 pp behind by round 4.
  • W4 (selection and uncertainty): We report independent DIRECT estimates, per-cell confidence intervals and full-benchmark runs, and find a drop of 25.7 pp against fresh DIRECT and of 15.2 pp on full benchmarks.

Weakness 1: Novelty beyond known self-correction failures and error localization

The distinction from known self-correction and anchoring failures is not established: the paper checks whether the model labels a draft as incorrect, not whether it identifies the specific error.

Thank you for recognizing that our work shows how incorrect drafts continue to harm reasoning even after being flagged as wrong, and for the insightful suggestion to check whether the model locates the specific error. Following your suggestions, we used Claude Haiku 5.5 to evaluate 1,800 1F-conditioned generations in which the model labels its draft as incorrect across the 8 models and 7 tasks (except for Game of 24) studied in the paper. In particular, we prompted Claude Haiku 5.5 with the problem, a reference correct answer and the draft generated by the evaluated model, and marked whether each generation identifies a specific, decisive error in the draft.

On the same generations, we also measured how much of the draft's reasoning reappears in the model's own solution. The judge listed up to 10 steps of the draft and marked each as reused or not, without knowing the condition and with final answers masked. We compare against independent DIRECT samples of the same problems.

The "incorrect" verdict comes withShareAccuracy minus DIRECT (pp)Gives the draft's answer (1F / DIRECT)Draft steps reused (1F / DIRECT)
An accurate diagnosis64%+11.29.4% / 18.7%59.5% / 57.6%
No accurate diagnosis36%−29.242.5% / 20.4%68.1% / 52.1%

(a) Accurate diagnosis largely removes the drag. When the model locates the error, it scores above DIRECT, repeats the draft's answer less often, and reuses the draft's steps at 59.5%, close to the DIRECT rate of 57.6%. This agrees with the finding of Tyen et al. [2] that models can fix an error once its location is known.

(b) A verdict without a diagnosis does not. In 36% of self-detected generations, the model calls the draft incorrect without accurately locating the error. These generations do not fall back to DIRECT behavior. They score 29.2 pp lower, give the draft's answer in 42.5% of cases (20.4% for DIRECT), and rebuild their solution from the draft's steps (+16.0 pp reuse over DIRECT). The accuracy drop and the repeated answer hold on every task.

(c) For some models, the reset after an accurate diagnosis is not complete. For Gemma4-E4B, Nemotron-7B, Qwen3-8B and Qwen3-32B, the solution written after an accurate diagnosis still reuses 4.8 to 8.7 pp more of the draft's steps than DIRECT does. For GPT-OSS-120B, GPT-OSS-20B, Gemma4-31B and Nemotron-32B, reuse falls to the DIRECT level.

Since the groups are defined by an outcome of the same generation, generations that locate the error may also be stronger in other ways. We will report the split descriptively.

Our distinct novelty claims. Based on these results, we believe contextual drag differs from known self-correction and anchoring failures in three ways:

  1. Detection is not a reset. Tyen et al. [2] show that models can correct an error once its location is known, and our audit agrees: with an accurate diagnosis, accuracy is 11.2 pp above DIRECT. However, when the model correctly rejects its draft but cannot locate the error, it does not fall back to DIRECT behavior. It is dragged toward the draft, costing 29.2 pp in performance, giving the draft's answer in 42.5% of cases instead of 20.4%, and reusing 16.0 pp more of its steps. A weaker independent sample alone would not return to the draft's own answer this often.
  2. Explicit labels are not a reset either. When the prompt states that the draft is wrong (Section 3.1), accuracy stays below DIRECT in 65 of 80 model-task pairs. Such a label gives the model a verdict but no diagnosis, and refinement pipelines often rely on such coarse signals without locating the exact error.
  3. The draft carries over its reasoning, not only its answer. Unlike anchoring on a salient value, the draft's intermediate steps reappear in the new solution (table below).

Over all 1F generations, the judge marked which of up to 10 steps of the draft reappear in the model's own solution, with final answers masked. We also counted the 8-word sequences and intermediate quantities that the solution shares with the draft, compared with matched DIRECT samples:

TaskDraft steps reused (1F / DIRECT)8-gram overlap gap (pp)Intermediate-quantity gap (pp)
AIME/HMMT (4 tasks)68.7% / 49.9%+12.2+25.8
GPQA71.3% / 51.5%+6.7+25.1
MMLU-Redux74.4% / 50.6%+5.3+26.3
CRUXEval-I55.6% / 48.0%+2.5+21.3
All68.2% / 50.0%+9.1+25.2

The reuse is not only a consequence of reaching the same answer. Among responses that end on the draft's wrong answer, 1F copies the draft's steps verbatim in 13.4% of cases, against 5.0% for DIRECT samples that reach the same answer. On Game of 24, we find the same structural carry-over with tree edit distance (Section 2.3).

In the revision, we will add this audit to Section 3.2 and state the claim in Section 3 and the abstract precisely: a correct verdict or an external label does not reset the draft's influence, while an accurate diagnosis of the error largely does.


Weakness 2: Do FILTER's gains come from independent planning rather than filtering?

A planning-only baseline is needed to tell whether keeping parts of the incorrect draft adds any benefit.

Following your suggestion, we ran a planning-only baseline that gives FILTER's draft-blind plan to the same solver prompt without any draft (three models, six tasks, same problems as 1F, pass@1 %):

DIRECT1FPlan-onlyFILTER on 1F
55.830.453.041.1

On wrong drafts, FILTER's gain over 1F comes from draft-blind planning, not from the filtered draft: planning alone is 2.8 pp below DIRECT, and keeping the filtered wrong draft lowers accuracy by a further 11.9 pp. We find that keeping draft content pays off only when the draft is correct: with one correct draft in context, FILTER reaches 83.4, 30.4 pp above plan-only. Filtering is therefore useful when a pipeline does not know whether its drafts are correct. We will report the plan-only results in Section 4.1.2 and remove any wording suggesting that FILTER benefits from incorrect drafts.

Weakness 3: Do the mitigations outperform fresh attempts at matched compute?

It is unclear whether fresh attempts with the same budget would do as well or better.

We compared the full pipelines with majority voting over fresh DIRECT samples at equal token budgets, counting every call and every prompt and generated token.

(a) When the draft is known to be wrong, fresh samples with voting are better. In the single-step 1F setting, FILTER and REVISE are 16.4 and 17.6 pp below voting at equal tokens. We agree that reusing a draft known to be wrong is not worthwhile.

(b) In iterative refinement, recursive FILTER is competitive with voting but does not beat it. Here each round reuses the previous attempt regardless of its correctness. Since FILTER writes its strategy once per problem, each round costs 1.2 DIRECT samples on average (GPT-OSS-20B, 120 math problems, pass@1 %):

RoundBudget (DIRECT samples)Naive refinementRecursive FILTERMajority vote
12.459.066.364.6
23.655.367.368.5
46.051.868.371.2
810.749.669.772.9

Recursive FILTER is 1.7 pp ahead of voting after one round, and 2.9 and 3.2 pp behind at rounds 4 and 8. Without context control, naive refinement falls 23.3 pp behind voting by round 8, and the filtering step closes 20.1 pp of this gap. Our main contribution is identifying contextual drag. We view FILTER as a proof of concept that controlling what stays in context makes reusing past attempts work. We do not propose it as a more efficient use of compute than fresh sampling. In the revision, we will add these comparisons to Section 4.1, state that fresh samples with voting are the better use of compute when a pipeline is free to discard its attempts, and replace "Simply discarding the draft is not a practical alternative" in Section 4.1.2 with this scoped statement.


Weakness 4: Are the results robust to subset selection and evaluation uncertainty?

The drop may not reflect full benchmarks, DIRECT reuses the selection samples, some subsets are very small, and uncertainty is not reported.

(a) Independent DIRECT. We drew 16 fresh DIRECT samples per problem that were never used for selection. Fresh DIRECT averages 57.9%, compared with 56.6% for the selection samples. Against fresh DIRECT, the 1F drop is 25.7 pp, compared with 24.4 pp in the paper.

(b) Uncertainty, small and different subsets. With a problem-level bootstrap interval per model-task cell, the drop is significant in 67 of 80 cells and negative in 77. Dropping the 5 cells with fewer than 6 problems gives 25.9 pp. Restricting each task to problems shared by at least 5 models gives a drop of the same sign on every task, within 1 pp of the per-task average on GPQA, MMLU-Redux and CRUXEval-I, which share the most problems.

(c) Full benchmarks. In runs where every model sees all problems of a benchmark with the same drafts (4 tasks, 8 models), 1F lowers accuracy by 15.2 pp over all problems, significant in 30 of 32 cells. The drop is smaller mainly because full benchmarks include problems the model never solves. On problems the model solves in all 8 held-out DIRECT samples (93% held-out accuracy), a single wrong draft still lowers accuracy by 17.7 pp. These runs use drafts from other models, as in our cross-drag setting.

In the revision, we will report fresh DIRECT estimates and per-cell confidence intervals, and add the full-benchmark results to the appendix.


Thank you again for your constructive review! We hope these new results and the narrowed claims address your concerns, and we are happy to answer any follow-up questions.

[1] Huang et al. Large Language Models Cannot Self-Correct Reasoning Yet. ICLR 2024.

[2] Tyen et al. LLMs Cannot Find Reasoning Errors, but Can Correct Them Given the Error Location. Findings of ACL 2024.

Reviewer_7XrEOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the negative effects of iterative reasoning methods, like self-refinement. Termed “drag,” the authors show that including an incorrect reasoning draft in the prompt can reinforce negative (incorrect) reasoning. Specifically, the authors study a few research questions and find: 1) incorrect drafts negatively and structurally influence the CoT reasoning of the iterated answer; 2) both explicit or self-verification signals do not remedy degradation; 3) there are some context-management strategies, like filtering, that can reduce drag.

The overall experiments that lead to these conclusions are conclusive in verifying self-deterioration under naive iterative refinement across several model types. A case study on the Game of 24 reveals that incorrect patterns seen in the prompt influence the reasoning in the answer, and an open question is whether this mechanism is in action more generally across all the datasets. The feedback-based results and context-management methods are similarly conclusive across model classes and datasets. The authors present further experiments to verify these findings in a "cross-drag" setting. They also attempt to resolve some issues via model training.

The paper is relatively convincing in establishing the issue, tracing the propagation, and ameliorating potential drag. Given all the detailed analyses and clear research questions and findings, this is a strong submission.

Critical strengths

The experiments and results are detailed and sufficiently support the research questions, which themselves are well-motivated to study the issue of contextual drag.

The structural characterization around tree edit distance is an interesting and insightful analysis, even though it is a synthetic example.

The experiments in the cross-drag setting comprehensively address additional concerns around potential model self-biases.

Critical weaknesses

The main part that confuses me is why iterative self-refinement and error-based feedback was so successful previously. The results would suggest that self-refinement should never work: given self or external error signal, the correct approach would be to restart from scratch. However, iterative refinement keeps everything in context. So why is there this disconnect?

The analyses are focused on Game of 24, which is related to the math datasets. However, there is no error analysis on the QA side, where the reasoning mechanisms could look different.

Questions for authors

There could be more discussion in the main body around some of the major findings from the appendices.

4 comments

Dear Reviewer, thank you for your time and positive feedback! We are glad that you find the research questions well-motivated, the tree edit distance analysis insightful and the cross-drag experiments comprehensive. Following your suggestions, we ran several new analyses. Here is a short summary of our responses:

  • W1 (connection to successful self-refinement): Our results do not imply that self-refinement should never work. Whether refinement helps depends on how many reused attempts are correct and on how informative the feedback is. Coarse error signals do not remove the drag, while an accurate diagnosis of the error largely does.
  • W2 (error analysis beyond Game of 24): Following your suggestion, we analyzed the traces for the other reported tasks beyond Game of 24, and found that a wrong draft makes the model repeat both its answer and its reasoning far more often.
  • Q1 (appendix findings): We will use the additional page in the camera-ready version to move the key appendix findings into the main text.

Weakness 1: Why has self-refinement succeeded in prior work?

The results seem to suggest that self-refinement should never work and that the model should always restart from scratch, which conflicts with prior successes of iterative refinement.

Thank you for raising this question. Our results do not imply that self-refinement should never work, and we will state this more directly in the future revisions. The two sets of findings are compatible for three reasons.

(a) Refinement outcomes depend on the balance of correct and wrong attempts. A refinement loop gains from reused attempts that are correct and loses from the drag of wrong ones. Under the same naive refinement loop, the change in accuracy after 8 rounds ranges from a decline of 11.3 pp to a gain of 9.4 pp across five models. We find that the models that improve are those with the smallest single-step drag (Appendix F.5).

(b) Informative feedback removes most of the drag. Refinement methods that work usually rely on feedback that carries information about the error, such as tool or environment signals or targeted critique. For example, RefineBench [1] finds that models rarely improve through self-refinement but often succeed when told which parts of their response are wrong. Our error signals are coarse: a label that the draft is wrong, or the model's own "incorrect" verdict. To test whether this difference matters, we audited whether the model's "incorrect" verdict comes with an accurate diagnosis of the specific error. We find that accuracy is 11.2 pp above DIRECT when the verdict comes with an accurate diagnosis, and 29.2 pp below DIRECT when it does not. This agrees with Tyen et al. [2], who find that models can fix an error once told where it is. We give the details of this audit in our response to Reviewer 7me9.

(c) Restarting is a strong baseline, but it gives up correct attempts. With one correct draft in context, accuracy is 27.6 pp above DIRECT, so discarding every attempt also discards these gains. By controlling what stays in context, simple interventions like FILTER keep these gains, and recursive filtering turns the decline of naive refinement into steady improvement (Figure 3).

In the revision, we will add this discussion to Sections 2.2 and 6.

Weakness 2: Error analysis beyond Game of 24

The analyses focus on Game of 24, and there is no error analysis on the QA side, where the reasoning mechanisms could look different.

Following your suggestion, we analyzed the traces for the other reported tasks beyond Game of 24. We find that on every task, a wrong draft makes the model repeat both its answer and its reasoning far more often.

(a) More frequent reappearance of the draft's wrong answer. We measured how often the final answer repeats the draft's wrong answer, compared with independent DIRECT samples on the same problems. We find that on every task, a wrong draft makes the model repeat its answer much more often:

Task1F gives the draft's answerDIRECT gives the same answerGap (pp)
AIME/HMMT (4 tasks)46.1%8.5%+37.6
GPQA-Diamond66.9%28.8%+38.1
MMLU-Redux74.5%32.3%+42.2
CRUXEval-I49.5%10.0%+39.5
All53.6%15.0%+38.6

The DIRECT baseline is higher on GPQA-Diamond and MMLU-Redux because they are multiple-choice, but the gap is as large as on the other tasks. Across tasks, 75.2% of the wrong 1F answers are exactly the draft's answer, compared with 36.6% of the wrong DIRECT answers.

(b) More frequent reappearance of the draft's reasoning. We also checked whether the draft's reasoning reappears in the new solution. First, 1F responses share more 8-word sequences (+9.1 pp) and more intermediate quantities (+25.2 pp) with the draft than DIRECT samples do.

Second, we used Claude Haiku 5.5 as an LLM judge and ran it through headless Claude Code over the traces. Without seeing the condition or the final answers, the judge checked which of up to 10 draft steps reappear after the model rejects the draft. We find that 68.2% of the draft's steps reappear under 1F, compared with 50.0% under DIRECT.

Third, on the multiple-choice tasks, a matching option could arise without copying the reasoning. The judge therefore checked 400 pairs of 1F and DIRECT responses for the draft's decisive error. We find that 65.2% of 1F responses repeat this error, compared with 31.2% of DIRECT responses. 86.9% of the 1F responses that land on the draft's option also contain the draft's error, so the model rarely copies only the option letter.

In the revision, we will add these results to Section 2.3 and representative examples to the appendix.


Question 1: Highlighting appendix findings in the main text

There could be more discussion in the main body around some of the major findings from the appendices.

Thank you for the suggestion. The camera-ready version allows one additional page, and we will use it to move the key appendix findings into the main text. These include the model differences under iterative refinement (Appendix F.5), the correct and mixed-draft results (Appendix C.4) and the new analyses from this rebuttal.


Thank you again for your constructive review! We hope these new results address your questions, and we are happy to answer any follow-up questions.

[1] Lee et al. RefineBench: Evaluating Refinement Capability of Language Models via Checklists. ICLR 2026.

[2] Tyen et al. LLMs Cannot Find Reasoning Errors, but Can Correct Them Given the Error Location. Findings of ACL 2024.

Thank you for your response. About W2, I was more interested in insights on the reasoning itself (like the TED analysis for Game of 24). This might be hard to analyze, and so what you've done by measuring against 1F/DIRECT might be the closest proxy.

After reading and acknowledging some the other reviewer's concerns (around novelty and FILTER) as well as the rebuttal, I have no further questions maintain my ratings and positive recommendation of clear accept.

Dear Reviewer, thank you so much for your quick follow-up and for maintaining your positive recommendation! Unlike Game of 24, free-form reasoning traces / answers usually do not admit a canonical parsable structure, so a structural analysis is quite challenging. We believe that understanding how different context conditioning interferes with the structure of free-form reasoning at scale is an exciting direction for future work.

Reviewer_K1PYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies “contextual drag”: when a reasoning model is conditioned on one or two of its own incorrect drafts and asked to verify and then re-solve, its subsequent generations become less accurate and structurally closer to the erroneous draft.

On problems where clean-slate sampling yields both correct and incorrect answers, a single incorrect self-draft reduces pass@1 by roughly 25 points on average, across 10 open-weight models and 8 tasks. Tree edit distance on Game of 24 shows that conditioned solutions resemble the draft more than clean-slate solutions do.

The drag persists under explicit external error labels and on trajectories where the model itself correctly flags the draft as incorrect. Naive iterative refinement compounds the effect for GPT-OSS-20B. As a mitigation, the authors propose FILTER: generate a draft-blind strategy, extract the draft components consistent with it, then solve. FILTER reduces drag on all models and reverses GPT-OSS-20B’s deterioration under recursive refinement.

Critical strengths

Strengths

The phenomenon is practically relevant to self-refinement and multi-agent aggregation pipelines, and the paper separates error detection from context reset clearly. The empirical coverage is broad (10 models, 5 families, 8 tasks), with both self-drag and cross-drag settings. The structural analysis via tree edit distance goes beyond final-answer accuracy and offers a concrete handle on how the draft shapes the output. Several confounds are tested and addressed: prompt position, removal of the verification instruction, a matched-prompt control (§C.3), removal of the reuse instruction (§D.3), and thinking budget (§C.1). The negative result on conditional fallback training (§G) is reported honestly.

Critical weaknesses

Limited direct evidence of copying. The structural analysis is restricted to Game of 24. The most direct measure of drag, the rate at which the final answer reproduces the draft’s final answer, is not reported for the other tasks.

The practical value of FILTER is unclear. In Figure 6, FILTER remains below DIRECT on every model, so in the single-step setting simply discarding the draft performs better. The argument that discarding forfeits the gains from correct drafts (§4.1.2) would require evaluating FILTER on correct drafts (1T) and on mixed-correctness drafts, which is not done. FILTER uses three calls versus one, and recursive filtering is not compared against compute-matched majority voting. Table 6 shows that maj@16 alone improves DIRECT by +17.6 points.

Novelty. The phenomenon is closely related to existing work on anchoring, sycophancy, hallucination snowballing, and the limits of self-correction.

Questions for authors

N/A

4 comments

Dear Reviewer, thank you for your time and thoughtful feedback! We are glad that you find contextual drag practically relevant to self-refinement and multi-agent pipelines, and that you appreciate the broad coverage, the tree edit distance analysis and the confound checks. Following your suggestions, we ran several new experiments and further analyzed the existing solution traces. Here is a short summary of our responses:

  • W1 (evidence of copying): We measure the similarity between the draft in context and the final answer, and find that on all 7 measured tasks beyond Game of 24, a wrong draft makes the model repeat the draft's answer in 53.6% of cases, against 15.0% for DIRECT. The draft's intermediate steps also reappear in the new solution.
  • W2 (practical value of FILTER): We agree that DIRECT generally performs better than FILTER and REVISE in the 1F and 2F settings. However, these settings leave out the practical case: we find that context control gains more when correct drafts are in context. Following your suggestions, we evaluate FILTER on correct and mixed drafts and against compute-matched majority voting.
  • W3 (novelty compared to previous works): We clarify how contextual drag differs from anchoring, sycophancy, hallucination snowballing and the limits of self-correction.

Weakness 1: Evidence of copying beyond Game of 24

The structural analysis covers only Game of 24, and the rate at which the final answer repeats the draft's answer is not reported for other tasks.

(a) More frequent reappearance of the final answer. Following your suggestion, we measured how often the final answer repeats the draft's wrong answer, compared with independent DIRECT samples on the same problems. We find that on every task, a wrong draft makes the model repeat its answer more often (53.6% vs 15.0% across tasks):

Task1F gives the draft's answerDIRECT gives the same answerGap (pp)
AIME/HMMT (4 tasks)46.1%8.5%+37.6
GPQA66.9%28.8%+38.1
MMLU-Redux74.5%32.3%+42.2
CRUXEval-I49.5%10.0%+39.5
All53.6%15.0%+38.6

We also find that the remaining errors concentrate on the draft's answer. 75.2% of the wrong 1F answers are exactly the draft's answer, compared with 36.6% of the wrong DIRECT answers. The gap is significant in 69 of the 70 model-task pairs.

(b) More frequent reappearance of reasoning steps. We also checked whether the draft's reasoning steps reappear in the new solution. First, the solution written after the verdict shares more 8-word sequences (+9.1 pp) and more intermediate quantities (+25.2 pp) with the draft than DIRECT samples do. This holds even among responses that reach the same wrong answer.

Second, we used Claude Haiku 5.5 as an LLM judge over the traces. Without seeing the condition or the final answers, the judge checked which of up to 10 draft steps reappeared in the model's new solution. We find that 68.2% of the draft's steps reappear under 1F, compared with 50.0% under DIRECT.

We will add these results to Section 2.3.

Weakness 2: Practical value of FILTER

FILTER stays below DIRECT in Figure 6, is not tested on correct or mixed drafts, uses three calls instead of one, and is not compared with compute-matched majority voting.

Thank you for raising this point. We agree that DIRECT generally performs better than FILTER and REVISE in the 1F and 2F settings. However, the drafts in these settings are wrong by construction, so discarding them is expected to do best. In practice, pipelines such as iterative refinement keep past attempts without knowing which are incorrect, and FILTER preserves the gain when correct drafts are in context. This is also why FILTER improves beyond the DIRECT baseline in Figure 3.

(a) Larger gain from FILTER with correct drafts in context. Following your suggestion, we ran FILTER with one correct draft (1T) and with one correct plus one wrong draft (1T1F) on three models and six tasks. We find that FILTER is 27.6 pp above DIRECT with one correct draft, and 14.3 pp above with one correct and one wrong draft:

Drafts in contextDIRECTFILTERFILTER minus DIRECT
One correct draft (1T)55.883.4+27.6
One correct and one wrong draft (1T1F)55.870.1+14.3

Discarding the draft would give up this gain.

(b) Steady gain over DIRECT under iterative refinement. When each round reuses the previous attempt regardless of its correctness, recursive FILTER stays 5.7 to 9.1 pp above DIRECT from round 1 to round 8. Naive refinement instead falls from 1.6 to 11.0 pp below DIRECT (Figure 3).

(c) Comparison with compute-matched majority voting. Following your suggestion, we compare recursive FILTER with majority voting over fresh DIRECT samples at the same token budget, on 120 math problems. Since the strategy is written only once per problem, each FILTER round costs 1.2 DIRECT samples on average. We find that recursive FILTER is 1.7 pp ahead of compute-matched majority voting at round 1, and 1.2 to 3.2 pp behind at rounds 2, 4 and 8:

RoundBudget (DIRECT samples)Naive refinement (same round)Recursive FILTERMajority vote
12.459.066.364.6
23.655.367.368.5
46.051.868.371.2
810.749.669.772.9

Without context control, naive refinement falls 23.3 pp behind voting by round 8, and with a simple filtering step, we recover 20.1 pp of this gap. Our main contribution is identifying contextual drag, and we view FILTER as a proof of concept that controlling what stays in context makes reusing past attempts work. We do not propose it as a new end-to-end method. We will add the correct-draft and compute-matched results to Section 4.1.

Weakness 3: Novelty relative to anchoring, sycophancy, snowballing and self-correction

The phenomenon is closely related to existing work on anchoring, sycophancy, hallucination snowballing, and the limits of self-correction.

Thank you for pointing out these connections. We agree that these lines of work are closely related, and we will revise the related work to state the differences below. We will also avoid any claim that we are the first to show that context can bias LLMs.

(a) Difference from the limits of self-correction. Prior work locates the failure of self-correction in error detection. Models often cannot tell that their reasoning is wrong [1] or where the error is, although they can fix it once told its location [2]. We study what happens once the draft is known to be wrong. When the prompt explicitly states this (Section 3.1), accuracy still stays below DIRECT in 65 of 80 model-task pairs, by 12.4 pp on average. We also find that a correct self-verification verdict alone does not remove the drag for several model families (Section 3.2). Knowing that the draft is wrong is therefore not enough without an accurate error localization or control over the reused context.

(b) Difference from anchoring. Anchoring studies such as [3] measure how a salient number or hint shifts a short answer. We show that in long reasoning, the draft's structure and intermediate steps reappear in the new solution (Section 2.3 and our response to Weakness 1).

(c) Difference from sycophancy. Sycophancy is deference to a view that the user endorses [5], including an answer that the user suggests [4]. Our drafts carry no endorsement. The drag persists even when the prompt states that the draft is wrong and instructs the model not to copy its final answer.

(d) Difference from hallucination snowballing. Snowballing [6] describes a model staying consistent with its own earlier claims. We will cite it as the closest analogue of the iterative decline in Section 2.2. Contextual drag also occurs with drafts written by other models (Section B), so we believe self-consistency cannot fully explain it.


Thank you again for your constructive review! We hope these new results address your concerns, and we are happy to answer any follow-up questions.

[1] Huang et al. Large Language Models Cannot Self-Correct Reasoning Yet. ICLR 2024.

[2] Tyen et al. LLMs Cannot Find Reasoning Errors, but Can Correct Them Given the Error Location. Findings of ACL 2024.

[3] Huang et al. An Empirical Study of the Anchoring Effect in LLMs: Existence, Mechanism, and Potential Mitigations. arXiv:2505.15392, 2025.

[4] Turpin et al. Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. NeurIPS 2023.

[5] Sharma et al. Towards Understanding Sycophancy in Language Models. ICLR 2024.

[6] Zhang et al. How Language Model Hallucinations Can Snowball. ICML 2024.

Many thanks for your response,after reconsideration,I decide to maintain the score.

Comments on the paper

1 comment

Dear Reviewers

The author-reviewer discussion period will close soon. The authors put much efforts for their rebuttal. Please read and respond to authors if you have not done yet.

Thank you.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.