- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks when more context helps or hurts LLMs and agents. It compares a model with an "ideal solver" that reads the same input, and defines the model's shortfall as a "utilization gap". It argues that adding unrelated background text, or rewriting an agent's history while earlier observations stay retrievable, leaves the ideal value unchanged, so any score change is a change in use. Three sets of experiments follow: (1) on RepoQA and HELMET with controlled evidence arrival, the rise-then-fall curve is approximated by evidence coverage times the score with all evidence present from the start; (2) on HotpotQA with both evidence paragraphs kept, F1 falls by 7–16 points from 2K to 120K, more with topical background and early placement, and a lone paragraph is lost 1.4–2.2 times faster than independent erasure predicts; (3) on a synthetic linked-record agent task, rewriting a 32K history (Extract) raises Qwen2.5-7B summation from 6/128 to 70/128, Recent makes the agent read past the target, and blocking rereads hurts.
Critical strengths
- The experiments are carefully controlled, with paired bootstrap intervals and multiple-comparison correction.
- The paper reports results that weaken its own story, such as the fixed-cohort retrieval curves in Appendix C.2.
Critical weaknesses
-
The findings are already known, and the experimental paradigm is dated. The rise-then-fall pattern with more retrieved passages was reported by Jin et al. (2025a), and Du et al. (2025) showed that length alone hurts even with perfect retrieval; both are cited in the paper. Section 5's HotpotQA factorial (random vs. topical background, early vs. late evidence) repeats the distractor and position experiments of the 2023–2024 long-context literature (lost in the middle, hard negatives). This line of experiments made sense then; in late 2026 it adds little. The paper proposes no method, and its implications (e.g., "selection should keep complementary evidence together") are already the premise of multi-hop retrieval.
-
The rise-then-fall curve in Section 4 is trivial, and its decomposition is close to true by construction. That the score first rises because evidence arrives and then falls because more text is added is what anyone would expect without an experiment. In the controlled-arrival design, evidence is masked before its assigned budget (RepoQA masks the target function in shorter prefixes, Appendix C.3), so a question without its evidence scores near zero. If arrival budgets do not depend on question difficulty, the expected score is then coverage times the full-evidence score, which is exactly the "separable prediction" in Table 4. The fit worsens where models can answer without the evidence (PopQA, TriviaQA). Moreover, under natural retrieval with question cohorts held fixed, the fall is within noise (Figure 5: −2.4 points, 95% CI [−9.5, 4.7] on LongBench-v2; −1.5, [−3.9, 0.9] on LooGLE-v2), so the "context saturation" in Figure 2 (top) partly reflects which questions remain at each budget, and the clear fall comes from the constructed designs.
-
The theoretical framework adds little. The ideal value G is never computed. Lemma 3.2 and Proposition 3.3 state that adding text unrelated to the answer, or rewriting a history while earlier observations remain retrievable, does not remove information, which is the assumption any controlled distractor experiment already makes. Proposition 3.5 is two lines of algebra. The framework mainly relabels "the score drops while the evidence is kept" as "the utilization gap grows", and it was hard to see what it lets one conclude that the controlled designs alone do not.
-
The agent study (Section 6) is a toy case study with intuitive conclusions. It uses 128 synthetic instances of a three-record linked list, a scripted 32K prefix, two 7B/8B models, and history policies (Extract, Progress) written by hand for this record schema. The only model-driven policy, Summary, fails as a memory writer in 195 of 256 Llama runs. That deleting 32K tokens of irrelevant records helps, that truncating the first step makes the agent lose count, and that blocking rereads hurts are all expected. Current context-management work, including AgentFold and Context-Folding cited in the paper, lets the agent decide what to keep on realistic long-horizon tasks; this study neither tests such policies nor proposes one.
Questions for authors
- What conclusion does the ideal-solver framework support that the controlled designs (evidence kept, background text added) would not support on their own?
- In the controlled-arrival design, how are arrival budgets assigned to questions? If they are independent of difficulty and questions without evidence score near zero, is the product fit not expected by construction? What does the fit show beyond that?
- Do the agent findings hold on realistic long-horizon benchmarks, and with policies in which the agent itself decides what to drop?