- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies “contextual drag,” showing that providing an LLM with a previous incorrect reasoning attempt can reduce its ability to solve a problem. Across a variety of models and tasks, the authors find that this effect can persist even when the model is explicitly told that the previous attempt is wrong or correctly identifies it as incorrect. They also show that filtering previous attempts before reusing them can reduce contextual drag and improve performance during iterative refinement.
Critical strengths
The paper presents a fairly comprehensive set of experiments across a broad selection of models and reasoning tasks, which strengthens the empirical evidence for contextual drag. The authors investigate several possible confounding factors, including prompt position, removal of verification, matched prompts, thinking budgets, mixed-correctness drafts, and majority voting. Additional code-generation and cross-model experiments broaden the relevance of the findings. The structural similarity analysis and iterative refinement experiments further help characterize how incorrect drafts influence subsequent reasoning. I also appreciate that the authors report cases where models remain stable or improve under iterative refinement, as well as unsuccessful training interventions, providing a more balanced assessment of the phenomenon.
Critical weaknesses
-
My main concern is about the scope of novelty. Previous work like Huang et al. (ICLR 2024) demonstrates the limitations of LLM self-correction. This current submission goes beyond by showing that incorrect drafts can continue to harm reasoning even after being flagged as wrong. However, if I am correct, the authors only verify whether the model correctly labels a draft as incorrect, without examining whether it can identify and explain the specific reasoning error. It remains unclear whether the model continues to follow an error it understands or simply cannot locate the mistake. While the structural similarity analysis provides additional evidence, the distinction between contextual drag and known self-correction or anchoring failures is not convincingly established for me. Showing that the effect persists after accurate error localization could strengthen the novelty claim.
-
FILTER first generates a fresh plan without seeing the erroneous draft, then filters the draft and produces a solution. Does the reported improvement come partly or primarily from independent planning rather than filtering itself? The authors acknowledge that they do not separate these contributions. A planning-only baseline may be necessary to determine whether retaining information from the incorrect draft provides any additional benefit.
-
REVISE and FILTER frequently perform worse than DIRECT despite requiring additional model calls. While they reduce the damage caused by erroneous drafts, this does not necessarily demonstrate that reusing those drafts is worthwhile. The paper reports some independent-sampling comparisons, but lacks compute-matched comparisons between the complete mitigation pipelines and fresh independent attempts. Even though recursive FILTER improves over the initial baseline on some models, it is still unclear to me whether comparable or better improvements could be achieved by simply generating fresh solutions using the same inference budget.
-
The main experiments select only problems for which 16 clean-slate generations produce both correct and incorrect answers. While this isolates problems where improvement is possible, the reported average accuracy drop may not reflect degradation across the full benchmark. Furthermore, DIRECT accuracy is calculated using the same samples employed for problem selection, rather than independent samples. Some model–task subsets contain only three to five problems, and different models are evaluated on different subsets, making comparisons less reliable. Reporting full-benchmark results, independent DIRECT estimates, and uncertainty measures would strengthen the evaluation.
Questions for authors
See the weaknesses above.
3 comments
Our distinct novelty claims. Based on these results, we believe contextual drag differs from known self-correction and anchoring failures in three ways:
- Detection is not a reset. Tyen et al. [2] show that models can correct an error once its location is known, and our audit agrees: with an accurate diagnosis, accuracy is 11.2 pp above DIRECT. However, when the model correctly rejects its draft but cannot locate the error, it does not fall back to DIRECT behavior. It is dragged toward the draft, costing 29.2 pp in performance, giving the draft's answer in 42.5% of cases instead of 20.4%, and reusing 16.0 pp more of its steps. A weaker independent sample alone would not return to the draft's own answer this often.
- Explicit labels are not a reset either. When the prompt states that the draft is wrong (Section 3.1), accuracy stays below DIRECT in 65 of 80 model-task pairs. Such a label gives the model a verdict but no diagnosis, and refinement pipelines often rely on such coarse signals without locating the exact error.
- The draft carries over its reasoning, not only its answer. Unlike anchoring on a salient value, the draft's intermediate steps reappear in the new solution (table below).
Over all 1F generations, the judge marked which of up to 10 steps of the draft reappear in the model's own solution, with final answers masked. We also counted the 8-word sequences and intermediate quantities that the solution shares with the draft, compared with matched DIRECT samples:
| Task | Draft steps reused (1F / DIRECT) | 8-gram overlap gap (pp) | Intermediate-quantity gap (pp) |
|---|---|---|---|
| AIME/HMMT (4 tasks) | 68.7% / 49.9% | +12.2 | +25.8 |
| GPQA | 71.3% / 51.5% | +6.7 | +25.1 |
| MMLU-Redux | 74.4% / 50.6% | +5.3 | +26.3 |
| CRUXEval-I | 55.6% / 48.0% | +2.5 | +21.3 |
| All | 68.2% / 50.0% | +9.1 | +25.2 |
The reuse is not only a consequence of reaching the same answer. Among responses that end on the draft's wrong answer, 1F copies the draft's steps verbatim in 13.4% of cases, against 5.0% for DIRECT samples that reach the same answer. On Game of 24, we find the same structural carry-over with tree edit distance (Section 2.3).
In the revision, we will add this audit to Section 3.2 and state the claim in Section 3 and the abstract precisely: a correct verdict or an external label does not reset the draft's influence, while an accurate diagnosis of the error largely does.
Weakness 2: Do FILTER's gains come from independent planning rather than filtering?
A planning-only baseline is needed to tell whether keeping parts of the incorrect draft adds any benefit.
Following your suggestion, we ran a planning-only baseline that gives FILTER's draft-blind plan to the same solver prompt without any draft (three models, six tasks, same problems as 1F, pass@1 %):
| DIRECT | 1F | Plan-only | FILTER on 1F |
|---|---|---|---|
| 55.8 | 30.4 | 53.0 | 41.1 |
On wrong drafts, FILTER's gain over 1F comes from draft-blind planning, not from the filtered draft: planning alone is 2.8 pp below DIRECT, and keeping the filtered wrong draft lowers accuracy by a further 11.9 pp. We find that keeping draft content pays off only when the draft is correct: with one correct draft in context, FILTER reaches 83.4, 30.4 pp above plan-only. Filtering is therefore useful when a pipeline does not know whether its drafts are correct. We will report the plan-only results in Section 4.1.2 and remove any wording suggesting that FILTER benefits from incorrect drafts.
Weakness 3: Do the mitigations outperform fresh attempts at matched compute?
It is unclear whether fresh attempts with the same budget would do as well or better.
We compared the full pipelines with majority voting over fresh DIRECT samples at equal token budgets, counting every call and every prompt and generated token.
(a) When the draft is known to be wrong, fresh samples with voting are better. In the single-step 1F setting, FILTER and REVISE are 16.4 and 17.6 pp below voting at equal tokens. We agree that reusing a draft known to be wrong is not worthwhile.
(b) In iterative refinement, recursive FILTER is competitive with voting but does not beat it. Here each round reuses the previous attempt regardless of its correctness. Since FILTER writes its strategy once per problem, each round costs 1.2 DIRECT samples on average (GPT-OSS-20B, 120 math problems, pass@1 %):
| Round | Budget (DIRECT samples) | Naive refinement | Recursive FILTER | Majority vote |
|---|---|---|---|---|
| 1 | 2.4 | 59.0 | 66.3 | 64.6 |
| 2 | 3.6 | 55.3 | 67.3 | 68.5 |
| 4 | 6.0 | 51.8 | 68.3 | 71.2 |
| 8 | 10.7 | 49.6 | 69.7 | 72.9 |
Recursive FILTER is 1.7 pp ahead of voting after one round, and 2.9 and 3.2 pp behind at rounds 4 and 8. Without context control, naive refinement falls 23.3 pp behind voting by round 8, and the filtering step closes 20.1 pp of this gap. Our main contribution is identifying contextual drag. We view FILTER as a proof of concept that controlling what stays in context makes reusing past attempts work. We do not propose it as a more efficient use of compute than fresh sampling. In the revision, we will add these comparisons to Section 4.1, state that fresh samples with voting are the better use of compute when a pipeline is free to discard its attempts, and replace "Simply discarding the draft is not a practical alternative" in Section 4.1.2 with this scoped statement.
Weakness 4: Are the results robust to subset selection and evaluation uncertainty?
The drop may not reflect full benchmarks, DIRECT reuses the selection samples, some subsets are very small, and uncertainty is not reported.
(a) Independent DIRECT. We drew 16 fresh DIRECT samples per problem that were never used for selection. Fresh DIRECT averages 57.9%, compared with 56.6% for the selection samples. Against fresh DIRECT, the 1F drop is 25.7 pp, compared with 24.4 pp in the paper.
(b) Uncertainty, small and different subsets. With a problem-level bootstrap interval per model-task cell, the drop is significant in 67 of 80 cells and negative in 77. Dropping the 5 cells with fewer than 6 problems gives 25.9 pp. Restricting each task to problems shared by at least 5 models gives a drop of the same sign on every task, within 1 pp of the per-task average on GPQA, MMLU-Redux and CRUXEval-I, which share the most problems.
(c) Full benchmarks. In runs where every model sees all problems of a benchmark with the same drafts (4 tasks, 8 models), 1F lowers accuracy by 15.2 pp over all problems, significant in 30 of 32 cells. The drop is smaller mainly because full benchmarks include problems the model never solves. On problems the model solves in all 8 held-out DIRECT samples (93% held-out accuracy), a single wrong draft still lowers accuracy by 17.7 pp. These runs use drafts from other models, as in our cross-drag setting.
In the revision, we will report fresh DIRECT estimates and per-cell confidence intervals, and add the full-benchmark results to the appendix.
Thank you again for your constructive review! We hope these new results and the narrowed claims address your concerns, and we are happy to answer any follow-up questions.
[1] Huang et al. Large Language Models Cannot Self-Correct Reasoning Yet. ICLR 2024.
[2] Tyen et al. LLMs Cannot Find Reasoning Errors, but Can Correct Them Given the Error Location. Findings of ACL 2024.
Dear Reviewer, thank you for your time and insightful feedback! Your comments on error localization and the planning-only baseline helped us understand more precisely when contextual drag occurs. We are glad that you find the experiments comprehensive. Following your suggestions, we ran several new analyses:
Weakness 1: Novelty beyond known self-correction failures and error localization
Thank you for recognizing that our work shows how incorrect drafts continue to harm reasoning even after being flagged as wrong, and for the insightful suggestion to check whether the model locates the specific error. Following your suggestions, we used Claude Haiku 5.5 to evaluate 1,800 1F-conditioned generations in which the model labels its draft as incorrect across the 8 models and 7 tasks (except for Game of 24) studied in the paper. In particular, we prompted Claude Haiku 5.5 with the problem, a reference correct answer and the draft generated by the evaluated model, and marked whether each generation identifies a specific, decisive error in the draft.
On the same generations, we also measured how much of the draft's reasoning reappears in the model's own solution. The judge listed up to 10 steps of the draft and marked each as reused or not, without knowing the condition and with final answers masked. We compare against independent DIRECT samples of the same problems.
(a) Accurate diagnosis largely removes the drag. When the model locates the error, it scores above DIRECT, repeats the draft's answer less often, and reuses the draft's steps at 59.5%, close to the DIRECT rate of 57.6%. This agrees with the finding of Tyen et al. [2] that models can fix an error once its location is known.
(b) A verdict without a diagnosis does not. In 36% of self-detected generations, the model calls the draft incorrect without accurately locating the error. These generations do not fall back to DIRECT behavior. They score 29.2 pp lower, give the draft's answer in 42.5% of cases (20.4% for DIRECT), and rebuild their solution from the draft's steps (+16.0 pp reuse over DIRECT). The accuracy drop and the repeated answer hold on every task.
(c) For some models, the reset after an accurate diagnosis is not complete. For Gemma4-E4B, Nemotron-7B, Qwen3-8B and Qwen3-32B, the solution written after an accurate diagnosis still reuses 4.8 to 8.7 pp more of the draft's steps than DIRECT does. For GPT-OSS-120B, GPT-OSS-20B, Gemma4-31B and Nemotron-32B, reuse falls to the DIRECT level.
Since the groups are defined by an outcome of the same generation, generations that locate the error may also be stronger in other ways. We will report the split descriptively.