acceptodds
Under review as a conference paper at ICLR 2027

Reasoning Shift: How Context Silently Shortens LLM Reasoning

Abstract

Large language models (LLMs) exhibiting test-time scaling behavior, such as extended reasoning traces and self-verification, have demonstrated remarkable performance on complex, long-term reasoning tasks. However, the robustness of these reasoning behaviors remains underexplored. To investigate this, we conduct a systematic evaluation of multiple reasoning models across three scenarios: (1) problems augmented with lengthy, irrelevant context; (2) multi-turn conversational settings with independent tasks; and (3) problems presented as a subtask within a complex task. We observe an interesting phenomenon: reasoning models tend to produce much shorter reasoning traces (up to 74%) for the same problem under different context conditions compared to the traces produced when the problem is presented in isolation. A finer-grained analysis reveals that this compression is associated with a decrease in self-verification and uncertainty management behaviors, such as double-checking. Importantly, we show that even when additional self-checks are forced, their efficiency depends not only on the content of the reasoning traces, but also on the presence of redundant context. We hope our findings draw additional attention to both the robustness of reasoning models and the problem of context management for LLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
llm, reasoning, long context, robustness

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xAprOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how additional context affects LLM reasoning on otherwise self-contained problems. It compares isolated problem solving with three settings: irrelevant long inputs, multi-turn conversations, and jointly presented independent problems. Across multiple reasoning models and benchmarks, these settings generally shorten reasoning traces and reduce accuracy. Trace analysis and controlled continuation experiments show that additional context particularly reduces reasoning after the first candidate answer, increases termination rates, and raises verbalized confidence. Prompting for more thorough reasoning has limited effects, while context-augmented fine-tuning substantially stabilizes reasoning length but leaves accuracy gaps relative to isolated problem solving.

Critical strengths
  • The paper addresses a relevant question for long-context reasoning models: whether irrelevant context changes how they solve otherwise self-contained problems.
  • The paper is well written and easy to follow, progressing clearly from empirical observations to behavioral analyses and mitigation experiments.
  • The experiments are extensive and reasonably designed, including additional evaluations in the appendix. In particular, the controlled continuation experiments hold reasoning prefixes fixed while varying the surrounding context, helping distinguish context effects from differences in the generated reasoning.
Critical weaknesses

The technical contribution and practical impact appear limited.

Performance degradation from irrelevant context has been documented [a, b, c], including its interaction with reasoning complexity and mitigation through distractor-augmented training [a]. This paper extends these findings with controlled analyses of early termination and reduced self-verification in reasoning models. While informative, this contribution appears incremental. The mitigation experiments in Section 5 show limited improvements in accuracy robustness, even though fine-tuning substantially stabilizes reasoning length. A clearer comparison with prior work, particularly the uncited studies [a, c], is needed to establish the paper’s distinct contribution.

The mitigation evaluation lacks comparisons with established approaches.

The mitigation study would also benefit from comparison with established approaches, such as instructions to ignore irrelevant information [c] and retrieve-then-solve context reduction [b]. These comparisons would clarify whether the observed limitations are specific to the tested interventions or persist under more targeted mitigation strategies.

The effect on self-verification is not separated from changes in reasoning length.

Section 4.5 suggests that irrelevant context limits the effectiveness of self-verification. However, removing the context increases both accuracy and reasoning length in Table 5. It therefore remains unclear whether context reduces verification effectiveness at a comparable reasoning budget or primarily causes earlier termination. Comparing continuations from the same reasoning prefixes under matched additional reasoning budgets would help distinguish these explanations.


[a] Yang, Minglai, et al. "How is llm reasoning distracted by irrelevant context? an analysis using a controlled benchmark." Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

[b] Du, Yufeng, et al. “Context Length Alone Hurts LLM Performance Despite Perfect Retrieval.” Findings of EMNLP, 2025

[c] Shi, Freda, et al. "Large language models can be easily distracted by irrelevant context." International conference on machine learning. PMLR, 2023.

Questions for authors

See weaknesses

3 comments

We thank the reviewer for their thoughtful review. We address your questions and concerns below:

Performance degradation from irrelevant context has been documented [a, b, c], including its interaction with reasoning complexity and mitigation through distractor-augmented training [a]. A clearer comparison with prior work, particularly the uncited studies [a, c], is needed to establish the paper’s distinct contribution.

To our knowledge, prior work did not examine how irrelevant context affects either the number of generated reasoning tokens or the prevalence of self-verification within a reasoning trace. We will revise the related-work section to make this distinction explicit.

Prior irrelevant-context studies primarily evaluated outcome- and correctness-based robustness, often using models without separate thinking mode. Shi et al. [1] measured answer accuracy and consistency across distractor variants; Jiang et al. [2] additionally measured distractor identification; and Yang et al. [3] evaluated final-answer, intermediate-step, and reasoning-path correctness. Wu et al. [4] measured answer changes and expressed uncertainty, while Du et al. [5] demonstrated accuracy degradation even when distractor tokens were masked.

[1] Shi et al. “Large Language Models Can Be Easily Distracted by Irrelevant Context.” ICML, 2023.

[2] Jiang et al. “Enhancing Robustness in Large Language Models: Prompting for Mitigating the Impact of Irrelevant Information.” 2024.

[3] Yang et al. “How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark.” 2025.

[4] Wu et al. “How Easily Do Irrelevant Inputs Skew the Responses of Large Language Models?” 2024.

[5] Du et al. “Context Length Alone Hurts LLM Performance Despite Perfect Retrieval.” 2025.

We would also be glad to incorporate any additional related work the reviewer believes should be discussed.

The mitigation evaluation lacks comparisons with established approaches.

First, we would like to emphasize why we conducted the mitigation study and how we interpret it. Our primary contribution is the identification and investigation of the shorter-reasoning effect; we view the mitigation experiments as an additional diagnostic analysis rather than as a definitive solution. Thus, given the nature of our evaluation setup (the tasks are created in a way that makes it easy for a model to spot redundant context), we do not think that evaluating the retrieve+solve method would give any informative signal.

We extended our prompting experiment with the “ignore irrelevant information” prompt to test whether this direct instruction helps. We test 2 variants for the long_input setup:

a) Instruction first: we first say that the model will be given redundant context and then a task, and ask the model to ignore the distractor.

b) We give a distractor first, and then an instruction to ignore it and solve a task.

Accuracy/thinking tokens:

ModelBaselineDistractorDistractor, ignore (a)Distractor, ignore (b)
Qwen3.5 27B77.50% / 31,717.5769.75% / 21,432.6669.25% / 19,185.7170.50% / 19,997.39
Gemma 4 31B73.00% / 8,851.2569.50% / 6,104.6569.00% / 6,380.6568.00% / 6,496.45

It therefore remains unclear whether context reduces verification effectiveness at a comparable reasoning budget or primarily causes earlier termination.

Let us clarify what we meant by effectiveness. We took reasoning traces, removed the last few tokens from them, intervened with the prompt “Wait, let me check everything again.” and compared the accuracy and thinking length for such interventions on the same traces depending on the presence of distractor tokens. By the efficiency of such a self-check, we meant efficiency per intervention (a single intervention for each context condition).

Your question raises another point - whether the accuracy gains from the wait-intervention for the same number of thinking tokens are different depending on the presence of distractor tokens. To test that, we took continuations from our interventions for Qwen3.5-27B and, for each context condition (baseline vs. long_input), cut them to the length of the shorter of the two and evaluated the accuracy gains from these equally sized continuations. In short, both context conditions show similar accuracy gains under matched budgets.

Accuracy/thinking tokens:

ModelLong inputWait + long input contextWait + baseline contextBaseline
Qwen3.5 27B69.8% / 21,43271.8% / 22,11471.5% / 22,11477.5% / 31,717

That suggests that spending more time on self-verification would be beneficial for the model, but it fails to spend enough effort on that when the distractor is presented in the context.

We will update the appendix with that experiment and clarify that the claim in 4.5 refers to the efficiency of a single intervention, not the efficiency per generated token.

We hope our responses have addressed the reviewer’s concerns and thank them for their consideration.

Thank you for the clarifications and additional experiments.

  1. [Partially Addressed] The paper’s focus on reasoning length and self-verification is now clearer. While I agree with the analysis purpose of the mitigation experiments, their practical benefits remain limited.
  2. [Addressed] Additional experiments address my concern about mitigation baselines.
  3. [Addressed] Additional experiment with matched reasoning budgets and the clarification of the self-verification claim resolve my concern.

Based on these clarifications and additional evidence, I will increase my score.

Reviewer_yQiVOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper provides systematic analysis on how additional unrelated context affects LLM reasoning. They use subtasks, long input and multi-turn as perturbations to the context, and test several models on coding and math benchmarks. Papers show several interesting findings: including 1) reasoning is shortened when additional context is provided, and performance generally drops 2) asking for another check helps more after removing irrelevant context.

Critical strengths
  1. The paper is well motivated from the real world multi-turn conversations scenarios, where users may ask unrelated questions in one conversation.

  2. The analysis is overall solid and interesting.

Critical weaknesses
  1. In section 4.1, “we identify the position within each trace where the first candidate answer is stated.”. It’s not the final answer. So the subsection should be named “Does the model arrive at an answer faster”.

  2. There lacks systematic analysis on whether a model can recognize there’s unrelated texts in the reasoning. It would be interesting to add this result.

  3. I think the context affects LLM reasoning beyond just shortening it. Replacing “shortens” with ”affects” in the title could better cover all the findings.

Questions for authors
  1. It’s better to highlight the takeaway message for each of the analysis subsections so readers don’t get lost.
2 comments

We thank the reviewer for their review and positive feedback. We also thank the reviewer for the presentation suggestions. We will incorporate them into the next version of our paper.

Regarding the ability of the model to recognize the presence of unrelated text, our evaluation setup is specially designed to keep the actual task clear and unrelated to a distractor. In the reasoning traces that we investigated, the models either completely ignore the distractor context or just recognize it and continue to solve the task. For example, as we mention in Appendix H, Qwen3 32B only mentions the distractor a few times across all evaluations, while Qwen3.5 27B has a different reasoning pattern: it often describes the structure of the input task first, mentions the distractor, and then solves the task. We have not observed any thinking traces where a model referred to the distractor and considered it directly related to the task.

We thank the reviewer again and are happy to answer any remaining questions.

I decide to keep my score.

Reviewer_9KpLOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how irrelevant context affects the reasoning behavior of large language models, a phenomenon termed reasoning shift. Through experiments involving independent subtasks, long distracting inputs, and multi-turn interactions, the authors show that additional context can substantially shorten reasoning traces (by up to 74%) and degrade accuracy, even when models explicitly recognize the context as irrelevant. Further analyses reveal that these changes are primarily associated with reduced self-verification and premature reasoning termination. The findings are validated across multiple reasoning models and benchmarks, including IMOAnswerBench, GPQA-Diamond, and MATH-500. The authors also explore prompting and supervised fine-tuning as potential mitigation strategies, finding that prompting has limited effectiveness, while fine-tuning can partially restore reasoning length without consistently improving accuracy.

Critical strengths

None

Critical weaknesses

[W1] Limited contributions beyond empirical observations and lack of effective mitigation. The paper primarily identifies and analyzes shortened reasoning traces under irrelevant context, without establishing a clear causal explanation for the observed performance degradation. Moreover, the mitigation attempts in Section 5 remain limited: prompting fails to consistently alleviate the effect, while SFT largely restores reasoning length without fully recovering accuracy. Consequently, the work remains largely descriptive, offering limited actionable insights or effective solutions to the identified problem.

[W2] Limited novelty relative to existing studies. The central research question substantially overlaps with prior studies investigating how long or irrelevant contexts affect LLM reasoning performance. Previous work has already demonstrated performance degradation under such conditions and explored related reasoning failures and mitigation strategies. While this paper provides additional analyses of reasoning length, self-verification, and termination behaviors, these findings appear to be relatively incremental extensions of an already well-studied problem. The paper does not sufficiently clarify how its findings advance beyond existing research, raising concerns about the novelty and significance of its contributions.

Questions for authors

[Q1] Mechanistic understanding of reasoning shifts.The analysis in Section 4 primarily focuses on observable changes in reasoning length, self-verification patterns, and termination behaviors, but provides limited insight into the underlying mechanisms. How exactly does irrelevant context interfere with the model's reasoning process, even when the model can recognize that the additional information is irrelevant? For example, does the interference arise from changes in attention allocation, internal representations, or reasoning termination mechanisms? Could the authors provide deeper mechanistic analyses or controlled experiments to distinguish these possibilities? Without a clearer understanding of the underlying causes, it remains difficult to develop effective mitigation strategies beyond restoring superficial reasoning patterns.

[Q2] Generalizability to closed-source reasoning models. While the main experiments include proprietary models, the fine-grained analyses of reasoning behaviors primarily rely on open-weight models with accessible CoT traces. Do the observed changes in self-verification and reasoning termination also generalize to closed-source reasoning models? More importantly, for models whose CoT traces are inaccessible, how can such reasoning shifts be identified and analyzed through observable signals? Exploring this question would help establish whether the proposed findings generalize beyond models with accessible reasoning traces and provide a more practical understanding of reasoning shifts in real-world settings.

2 comments

We thank the reviewer for their thoughtful review. We address your questions and concerns below:

The mitigation attempts in Section 5 remain limited: prompting fails to consistently alleviate the effect, while SFT largely restores reasoning length without fully recovering accuracy.

We would like to emphasize why we conducted the mitigation study and how we interpret it. Our primary contribution is the identification and investigation of the shorter-reasoning effect; we view the mitigation experiments as an additional diagnostic analysis rather than as a definitive solution. Prompting experiment shows that even when a model is asked to use its maximum thinking effort, their reasoning shortens under the presence of a distractor. That aligns with the result of our intervention experiment from 4.5 - interventions to the same reasoning trace under different context conditions have different average lengths and accuracy gains.

Limited novelty relative to existing studies. The central research question substantially overlaps with prior studies investigating how long or irrelevant contexts affect LLM reasoning performance.

To our knowledge, prior work did not examine how irrelevant context affects either the number of generated reasoning tokens or the prevalence of self-verification within a reasoning trace. We will revise the related-work section to make this distinction explicit.

Prior irrelevant-context studies primarily evaluated outcome- and correctness-based robustness, often using models without separate thinking mode. Shi et al. [1] measured answer accuracy and consistency across distractor variants; Jiang et al. [2] additionally measured distractor identification; and Yang et al. [3] evaluated final-answer, intermediate-step, and reasoning-path correctness. Wu et al. [4] measured answer changes and expressed uncertainty, while Du et al. [5] demonstrated accuracy degradation even when distractor tokens were masked.

[1] Shi et al. “Large Language Models Can Be Easily Distracted by Irrelevant Context.” ICML, 2023.

[2] Jiang et al. “Enhancing Robustness in Large Language Models: Prompting for Mitigating the Impact of Irrelevant Information.” 2024.

[3] Yang et al. “How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark.” 2025.

[4] Wu et al. “How Easily Do Irrelevant Inputs Skew the Responses of Large Language Models?” 2024.

[5] Du et al. “Context Length Alone Hurts LLM Performance Despite Perfect Retrieval.” 2025.

We would also be glad to incorporate any additional related work the reviewer believes should be discussed.

Mechanistic understanding of reasoning shifts. How exactly does irrelevant context interfere with the model's reasoning process, even when the model can recognize that the additional information is irrelevant? For example, does the interference arise from changes in attention allocation, internal representations, or reasoning termination mechanisms?

Thank you for raising this point. In short, it is not very clear how to separate these effects, as they might be connected to each other. We have tried different techniques, such as activation patching, logit lens, investigation of attention patterns, representation norms and directions. We avoided SAE-based methods because they lack strong generalization to longer contexts. Most of the methods demonstrate a difference in representations when we add a distractor to the same thinking trace, but we believe that building a deeper and causal explanation requires a deeper effort and probably some new techniques; thus, we leave it for separate future work.

For example, for Qwen3.5-27B, by comparing the same baseline traces with and without a distractor (long inputs), we have found:

  • Particular positions in thinking traces where adding a distractor raises the probability of the </think> token by 10% or more, while the baseline trace initiates a self-check instead of terminating the thinking block.
  • Particular positions in thinking traces where adding a distractor switches the top-1 token from “Wait” to “So”, favoring moving directly toward a conclusion instead of self-verification.
  • Several later-layer heads that give up to 30% of their mass to the distractor. But patching these heads explained only up to 5–10% of the spikes in </think> probability or Wait→So changes.
  • Relative norms of residuals (baseline vs. with a distractor) tend to slightly decrease after the last layer, especially deeper in the thinking trace (-3% on average, -8% for the last third of each thinking trace).
  • Next-token entropy also slightly reduces when adding a distractor to the same trace, with a stronger reduction for later tokens in the trace.

We will extend our appendix with a description of the techniques we used and what signals we observed. Our analysis section mostly focuses on behavioural changes because resampling is a very useful tool for analysing thinking traces [1]. We will be very thankful if you have any particular experiment in mind and will try to run it.

[1] Macar, Uzay, et al. "Thought branches: Interpreting LLM reasoning requires resampling." International Conference on Learning Representations. Vol. 2026. 2026.

Do the observed changes in self-verification and reasoning termination also generalize to closed-source reasoning models? More importantly, for models whose CoT traces are inaccessible, how can such reasoning shifts be identified and analyzed through observable signals?

For most closed-source models, we can only observe the number of thinking tokens and the reasoning summary. We do not discuss methods that try to recover hidden thinking since they might violate terms of use.

However, these observable signals hint that the observed effect is not unique to open-source models. We already have a closed-source model in our evaluations, but we also additionally evaluated 2 different closed-source models on IMOAnswerBench under baseline and long-input setups. The results demonstrate that these models are also susceptible to the effect of reasoning shortening (12% for GPT-6.1 Terra and 36% for Claude Sonnet 5.5):

Accuracy/thinking tokens:

ModelBaselineLong input (64k)
GPT-6.1 Terra88.0% / 7,095.986.0% / 6,216.7
Claude Sonnet 5.5 (high)88.5% / 2,822.984.5% / 1,802.1

Regarding the high-level patterns, we can only observe the thinking summary, which provides a proxy for the effort spent by the model on different parts of the task. While we can’t try our resampling experiments, we can observe the changes in the summaries.

For example, we can observe summarized reasoning for Gemini-3-Flash (which also exhibits a significant shortening). For multiple samples, especially for GPQA-Diamond, we find that the model may identify the answer early and then spend a large proportion of its reasoning on confirmation and reconsideration. For example, for one problem, the baseline summarized reasoning contains 32 paragraphs (with headings such as “Calculating the Expectation”). The answer is first stated in the third paragraph, and the model spends the remaining paragraphs revisiting and confirming it. In the multi-turn setting, the model also reaches the answer in the third paragraph but produces only six paragraphs in total.

We hope our responses have addressed the reviewer’s concerns and thank them for their consideration.

Comments on the paper

1 comment

Dear reviewers,

We have now received the author responses to your raised questions and concerns. Please go through them. Also, have they answered your questions appropriately? Please also look other reviewers' strengths / concerns. Have they changed your perception of the paper? In all cases, please communicate with the authors. It is important that we let them know what relevant issues still exist or have been answered. Please remember we have limited time to get any further responses from them.

Regards,

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.