acceptodds
Under review as a conference paper at ICLR 2027

The Context Trade-Off: Understanding Information Use in Language Models and Agents

Abstract

Large language models face a fundamental tension at inference time: additional context provides more evidence, yet longer inputs can induce context rot—the degradation of model performance as input length grows. The same tension arises in language model agents as tool responses accumulate in their history. We ask when additional context helps or hurts, and how context management changes this trade-off. Across multiple benchmarks, evaluation formats, and model families, performance often rises and then falls as context grows. We call this pattern context saturation. Its shape and peak vary across models and tasks. To explain this pattern, we compare a model with an ideal solver: a reader that sees the same input and extracts everything it implies about the task, so its performance measures how much information the input contains. Two properties make this comparison useful. Adding background text unrelated to the answer cannot lower the ideal solver's performance, and neither can rewriting an agent's history, as long as earlier observations can still be retrieved. Under such changes, any drop in the model's performance is therefore a failure to use information that is still available, and any gain is information made easier to use. Controlled experiments across benchmarks and model families support three findings. (1) In most controlled settings, the rise reflects evidence arriving and the fall reflects evidence becoming harder to use, so the peak depends on the model. (2) Performance falls even when all needed evidence is kept. Evidence is also used jointly: a piece of evidence without its complement is lost more often, so a missing piece can matter more in longer contexts. (3) For agents, rewriting the same observations into a compact form can raise success several-fold. Rewritten histories fail when the agent reads past its target or can no longer revisit earlier observations. In these controlled settings, long context fails not for lack of information but because information becomes hard to use.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLMs, Context Rot, Long-Context, Agents, Empirical Theory

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Ci62OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks when more context helps or hurts LLMs and agents. It compares a model with an "ideal solver" that reads the same input, and defines the model's shortfall as a "utilization gap". It argues that adding unrelated background text, or rewriting an agent's history while earlier observations stay retrievable, leaves the ideal value unchanged, so any score change is a change in use. Three sets of experiments follow: (1) on RepoQA and HELMET with controlled evidence arrival, the rise-then-fall curve is approximated by evidence coverage times the score with all evidence present from the start; (2) on HotpotQA with both evidence paragraphs kept, F1 falls by 7–16 points from 2K to 120K, more with topical background and early placement, and a lone paragraph is lost 1.4–2.2 times faster than independent erasure predicts; (3) on a synthetic linked-record agent task, rewriting a 32K history (Extract) raises Qwen2.5-7B summation from 6/128 to 70/128, Recent makes the agent read past the target, and blocking rereads hurts.

Critical strengths
  1. The experiments are carefully controlled, with paired bootstrap intervals and multiple-comparison correction.
  2. The paper reports results that weaken its own story, such as the fixed-cohort retrieval curves in Appendix C.2.
Critical weaknesses
  1. The findings are already known, and the experimental paradigm is dated. The rise-then-fall pattern with more retrieved passages was reported by Jin et al. (2025a), and Du et al. (2025) showed that length alone hurts even with perfect retrieval; both are cited in the paper. Section 5's HotpotQA factorial (random vs. topical background, early vs. late evidence) repeats the distractor and position experiments of the 2023–2024 long-context literature (lost in the middle, hard negatives). This line of experiments made sense then; in late 2026 it adds little. The paper proposes no method, and its implications (e.g., "selection should keep complementary evidence together") are already the premise of multi-hop retrieval.

  2. The rise-then-fall curve in Section 4 is trivial, and its decomposition is close to true by construction. That the score first rises because evidence arrives and then falls because more text is added is what anyone would expect without an experiment. In the controlled-arrival design, evidence is masked before its assigned budget (RepoQA masks the target function in shorter prefixes, Appendix C.3), so a question without its evidence scores near zero. If arrival budgets do not depend on question difficulty, the expected score is then coverage times the full-evidence score, which is exactly the "separable prediction" in Table 4. The fit worsens where models can answer without the evidence (PopQA, TriviaQA). Moreover, under natural retrieval with question cohorts held fixed, the fall is within noise (Figure 5: −2.4 points, 95% CI [−9.5, 4.7] on LongBench-v2; −1.5, [−3.9, 0.9] on LooGLE-v2), so the "context saturation" in Figure 2 (top) partly reflects which questions remain at each budget, and the clear fall comes from the constructed designs.

  3. The theoretical framework adds little. The ideal value G is never computed. Lemma 3.2 and Proposition 3.3 state that adding text unrelated to the answer, or rewriting a history while earlier observations remain retrievable, does not remove information, which is the assumption any controlled distractor experiment already makes. Proposition 3.5 is two lines of algebra. The framework mainly relabels "the score drops while the evidence is kept" as "the utilization gap grows", and it was hard to see what it lets one conclude that the controlled designs alone do not.

  4. The agent study (Section 6) is a toy case study with intuitive conclusions. It uses 128 synthetic instances of a three-record linked list, a scripted 32K prefix, two 7B/8B models, and history policies (Extract, Progress) written by hand for this record schema. The only model-driven policy, Summary, fails as a memory writer in 195 of 256 Llama runs. That deleting 32K tokens of irrelevant records helps, that truncating the first step makes the agent lose count, and that blocking rereads hurts are all expected. Current context-management work, including AgentFold and Context-Folding cited in the paper, lets the agent decide what to keep on realistic long-horizon tasks; this study neither tests such policies nor proposes one.

Questions for authors
  1. What conclusion does the ideal-solver framework support that the controlled designs (evidence kept, background text added) would not support on their own?
  2. In the controlled-arrival design, how are arrival budgets assigned to questions? If they are independent of difficulty and questions without evidence score near zero, is the product fit not expected by construction? What does the fit show beyond that?
  3. Do the agent findings hold on realistic long-horizon benchmarks, and with policies in which the agent itself decides what to drop?
Reviewer_vF2BOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors study context saturation in language models and agents, where adding context can first improve and then reduce performance. They define the utilization gap as the difference between the ideal performance achievable from an input and a model’s realized performance. Controlled experiments suggest that the rise often reflects evidence arrival, while the decline reflects difficulty using evidence that remains available. The paper also examines how complementary evidence is used jointly and shows that rewriting agent histories, while preserving access to earlier observations, can improve task success.

Critical strengths
  1. Well-designed controlled experiments. The paper provides a detailed study of how context affects model performance. The HotpotQA experiments cover 112 conditions across evidence availability, context length, background text, and evidence placement. These experiments help show that having the right information in the context does not always mean the model can use it effectively.

  2. Detailed analysis and discussion of limitations. The paper provides useful analysis beyond the main results. The authors acknowledge that some performance drops are not statistically significant, that the proposed approximation does not work equally well across tasks, and that some agent memory methods have high failure rates. These details make the findings easier to understand and evaluate.

Critical weaknesses
  1. Limited generalization to other models and realistic agents. Although the paper tests several models in its broader experiments, the detailed evidence analysis mainly uses Qwen2.5-14B, with a smaller study on Llama-3.1-8B. The agent experiments use only two models and 128 synthetic tasks involving simple record lookup and summation. It is unclear whether the same findings would hold for stronger long-context models or agents working on more complex tasks.

  2. Some broader explanatory claims need stronger support. In Figure 5, the fixed-cohort declines for Qwen2.5-3B are not statistically significant, and similar checks are missing for other models. The claim that background changes only information use also needs better justification for topical distractors. Moreover, the RepoQA full-evidence curve uses only 26 of 100 tasks, and the proposed approximation performs worse than the baseline on HELMET TriviaQA. These limitations make the broader explanation less convincing.

Questions for authors
  1. Fixed-cohort evaluation: Given the uncertainty in the fixed-cohort declines in Figure 5, can the authors report similar analyses for other tested models? Can cross-model comparisons of peak locations also use matched question sets?

  2. RepoQA evaluation: How were the 26 tasks used for the full-evidence curve selected? Were the evidence-arrival and full-evidence curves evaluated on the same task subset? If not, does the advantage reported in Table 4 still hold when using matched tasks?

  3. Agent experiments: In the Llama Full-history summation setting, 122 of 128 runs reach the twelve-call limit, including 119 that already received all three values. Does increasing the call limit improve success? If success remains low, what failure patterns are observed after all three values have been received?

Reviewer_nBGqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

For Large Language Models (LLMs), adding additional context can provide more evidence, but longer context sometimes harms performance, which is known as “context rot.” This paper analyzes this phenomenon and how context management changes this trade-off. Through evaluation across multiple benchmarks, the paper discovers that LLMs’ performance often rises and then falls as context grows. To explain this pattern, this paper compares LLM with an “ideal solver” which extracts useful things about the task from the input. Since the ideal solver uses the same input as other LLMs, the performance drop of other LLMs can be attributed to failures of effectiveness in using the input information, and the performance increase of other LLMs can be attributed to improved effectiveness of using the input information. This paper presents several findings from various experiments. First, as the context length grows, the initial performance improvement usually corresponds to new evidence in the increased context, while eventual performance drop usually corresponds to evidence becoming harder to use. Second, LLMs’ performance could fall even when all needed evidence exists in the context, which is because the evidence becomes harder to use. Third, asking LLMs to rewrite the input context into a compact form can change the success rate a lot.

Critical strengths

1, This paper studies a meaningful problem, and the motivation is clear. It is beneficial to develop a deeper understanding of the “context rot” problem and find solutions to it. The findings could inspire practical applications.

2, The experiments are organized in a good way, where later experiments build upon previous experiments’ findings and pursue a deeper understanding of this issue. The experiments demonstrate the “context rot” problem, and the experiment designs isolate the existence of evidence from the effects of context lengths.

Critical weaknesses

1, The novel contributions might be limited. The paper’s findings might overlap a lot with [1][2]. Although this paper proposes a new theoretical framework, its explanatory power might be limited. For example, the “ideal solver” is a purely theoretical object, and based on the experiment designs, δG\delta G should be 0. And since δU=δG−δD\delta U = \delta G - \delta D, δD\delta D should simply be −δU-\delta U, so the decomposition adds little new information. The good property that any performance change reflects the use of evidence largely follows from experiment designs. (Please feel free to let me know if my understanding is incorrect.)

2, Paper writing needs substantial improvement, and it is hard to understand the detailed information. The paper mentions many concepts and notations without explaining their meanings at their first appearance. Many sentences are too compressed or lack natural transitions, so it is hard to understand their meanings. Section 3 has too many notations, while maybe some of them can be reduced, or more high-level insights can be added to more clearly explain the theories. It would be better if the authors could carefully read through the paper and make it more readable.

[1] Yufeng Du, Minyang Tian, Srikanth Ronanki, Subendhu Rongali, Sravan Babu Bodapati, Aram Galstyan, Azton Wells, Roy Schwartz, Eliu A Huerta, and Hao Peng. Context Length Alone Hurts LLM Performance Despite Perfect Retrieval. In Findings of the Association for Computational Linguistics: EMNLP 2025, pp. 23281–23298. Association for Computational Linguistics, 2025.

[2] Bowen Jin, Jinsung Yoon, Jiawei Han, and Sercan Arik. Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG. In International Conference on Learning Representations, volume 2025, pp. 37784–37822, 2025a.

Questions for authors

Clarification requests:

1, The submitted code seems to be incomplete. It only contains part of the experiment code, so reproducibility is not good. Will you publicly share the whole codebase?

Suggestions for future work:

1, Section 5.1 states that “Questions already solved lose half their accuracy.” It uses questions an LLM answers correctly and re-evaluates them with increased context. Although the LLM becomes worse on these problems, it is possible that the LLM could become better on the remaining problems. A stricter evaluation method might be to use the overall accuracy on the full dataset.

2, The experiments only use open-source models <= 14 B. It would be better if you could also do experiments on large frontier models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.