- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies how additional context affects LLM reasoning on otherwise self-contained problems. It compares isolated problem solving with three settings: irrelevant long inputs, multi-turn conversations, and jointly presented independent problems. Across multiple reasoning models and benchmarks, these settings generally shorten reasoning traces and reduce accuracy. Trace analysis and controlled continuation experiments show that additional context particularly reduces reasoning after the first candidate answer, increases termination rates, and raises verbalized confidence. Prompting for more thorough reasoning has limited effects, while context-augmented fine-tuning substantially stabilizes reasoning length but leaves accuracy gaps relative to isolated problem solving.
Critical strengths
- The paper addresses a relevant question for long-context reasoning models: whether irrelevant context changes how they solve otherwise self-contained problems.
- The paper is well written and easy to follow, progressing clearly from empirical observations to behavioral analyses and mitigation experiments.
- The experiments are extensive and reasonably designed, including additional evaluations in the appendix. In particular, the controlled continuation experiments hold reasoning prefixes fixed while varying the surrounding context, helping distinguish context effects from differences in the generated reasoning.
Critical weaknesses
The technical contribution and practical impact appear limited.
Performance degradation from irrelevant context has been documented [a, b, c], including its interaction with reasoning complexity and mitigation through distractor-augmented training [a]. This paper extends these findings with controlled analyses of early termination and reduced self-verification in reasoning models. While informative, this contribution appears incremental. The mitigation experiments in Section 5 show limited improvements in accuracy robustness, even though fine-tuning substantially stabilizes reasoning length. A clearer comparison with prior work, particularly the uncited studies [a, c], is needed to establish the paper’s distinct contribution.
The mitigation evaluation lacks comparisons with established approaches.
The mitigation study would also benefit from comparison with established approaches, such as instructions to ignore irrelevant information [c] and retrieve-then-solve context reduction [b]. These comparisons would clarify whether the observed limitations are specific to the tested interventions or persist under more targeted mitigation strategies.
The effect on self-verification is not separated from changes in reasoning length.
Section 4.5 suggests that irrelevant context limits the effectiveness of self-verification. However, removing the context increases both accuracy and reasoning length in Table 5. It therefore remains unclear whether context reduces verification effectiveness at a comparable reasoning budget or primarily causes earlier termination. Comparing continuations from the same reasoning prefixes under matched additional reasoning budgets would help distinguish these explanations.
[a] Yang, Minglai, et al. "How is llm reasoning distracted by irrelevant context? an analysis using a controlled benchmark." Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.
[b] Du, Yufeng, et al. “Context Length Alone Hurts LLM Performance Despite Perfect Retrieval.” Findings of EMNLP, 2025
[c] Shi, Freda, et al. "Large language models can be easily distracted by irrelevant context." International conference on machine learning. PMLR, 2023.
Questions for authors
See weaknesses
3 comments
It therefore remains unclear whether context reduces verification effectiveness at a comparable reasoning budget or primarily causes earlier termination.
Let us clarify what we meant by effectiveness. We took reasoning traces, removed the last few tokens from them, intervened with the prompt “Wait, let me check everything again.” and compared the accuracy and thinking length for such interventions on the same traces depending on the presence of distractor tokens. By the efficiency of such a self-check, we meant efficiency per intervention (a single intervention for each context condition).
Your question raises another point - whether the accuracy gains from the wait-intervention for the same number of thinking tokens are different depending on the presence of distractor tokens. To test that, we took continuations from our interventions for Qwen3.5-27B and, for each context condition (baseline vs. long_input), cut them to the length of the shorter of the two and evaluated the accuracy gains from these equally sized continuations. In short, both context conditions show similar accuracy gains under matched budgets.
Accuracy/thinking tokens:
| Model | Long input | Wait + long input context | Wait + baseline context | Baseline |
|---|---|---|---|---|
| Qwen3.5 27B | 69.8% / 21,432 | 71.8% / 22,114 | 71.5% / 22,114 | 77.5% / 31,717 |
That suggests that spending more time on self-verification would be beneficial for the model, but it fails to spend enough effort on that when the distractor is presented in the context.
We will update the appendix with that experiment and clarify that the claim in 4.5 refers to the efficiency of a single intervention, not the efficiency per generated token.
We hope our responses have addressed the reviewer’s concerns and thank them for their consideration.
Thank you for the clarifications and additional experiments.
- [Partially Addressed] The paper’s focus on reasoning length and self-verification is now clearer. While I agree with the analysis purpose of the mitigation experiments, their practical benefits remain limited.
- [Addressed] Additional experiments address my concern about mitigation baselines.
- [Addressed] Additional experiment with matched reasoning budgets and the clarification of the self-verification claim resolve my concern.
Based on these clarifications and additional evidence, I will increase my score.
We thank the reviewer for their thoughtful review. We address your questions and concerns below:
To our knowledge, prior work did not examine how irrelevant context affects either the number of generated reasoning tokens or the prevalence of self-verification within a reasoning trace. We will revise the related-work section to make this distinction explicit.
Prior irrelevant-context studies primarily evaluated outcome- and correctness-based robustness, often using models without separate thinking mode. Shi et al. [1] measured answer accuracy and consistency across distractor variants; Jiang et al. [2] additionally measured distractor identification; and Yang et al. [3] evaluated final-answer, intermediate-step, and reasoning-path correctness. Wu et al. [4] measured answer changes and expressed uncertainty, while Du et al. [5] demonstrated accuracy degradation even when distractor tokens were masked.
[1] Shi et al. “Large Language Models Can Be Easily Distracted by Irrelevant Context.” ICML, 2023.
[2] Jiang et al. “Enhancing Robustness in Large Language Models: Prompting for Mitigating the Impact of Irrelevant Information.” 2024.
[3] Yang et al. “How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark.” 2025.
[4] Wu et al. “How Easily Do Irrelevant Inputs Skew the Responses of Large Language Models?” 2024.
[5] Du et al. “Context Length Alone Hurts LLM Performance Despite Perfect Retrieval.” 2025.
We would also be glad to incorporate any additional related work the reviewer believes should be discussed.
First, we would like to emphasize why we conducted the mitigation study and how we interpret it. Our primary contribution is the identification and investigation of the shorter-reasoning effect; we view the mitigation experiments as an additional diagnostic analysis rather than as a definitive solution. Thus, given the nature of our evaluation setup (the tasks are created in a way that makes it easy for a model to spot redundant context), we do not think that evaluating the retrieve+solve method would give any informative signal.
We extended our prompting experiment with the “ignore irrelevant information” prompt to test whether this direct instruction helps. We test 2 variants for the long_input setup:
a) Instruction first: we first say that the model will be given redundant context and then a task, and ask the model to ignore the distractor.
b) We give a distractor first, and then an instruction to ignore it and solve a task.
Accuracy/thinking tokens: