- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies context rot in agent trajectories across nine frontier models in four task families: three passive-recall tasks and one active state-tracking task. The authors find passive recall stays near the ceiling across lengths, but state tracking collapses near zero when K is larger than 16. Analysis indicates tool-reported errors disrupt models more than self-authored ones, and stronger reasoning models snowball more from their own mistakes. Prompt mitigations fail at deep context horizons (K >= 32).
Critical strengths
-
Clear Problem Framing: Investigating performance degradation in self-generated agent trajectories is practically relevant, and separating passive recall from active state tracking offers a clear diagnostic framework.
-
Insightful Error Analysis: Section 5 provides interesting counterfactual findings showing that tool-reported errors cause more disruption than self-authored ones, and reasoning models snowball more aggressively from early mistakes.
These two are the novelty and core contribution I particularly find in the paper
Critical weaknesses
-
The task D is artificial framing; mental arithmetic is not how a real agent solves problems. The headline claims that "evolving-state maintenance is the hidden bottleneck of long-context agents" overgeneralizes what is essentially an in-context multi-step mental arithmetic stress test. it is neither a practical test nor utilizing any feature related to "agent". All of this conflates a known limitation in sequential numerical computation with a general theory of agent failure. The claim seemed to be too big to be supported by existing evidence.
-
Lacking some Novelty and Deeper Theoretical/Mechanistic Insight: The performance cliff at deep K and the failure of prompt mitigations are predictable from existing theoretical frameworks: both from an agent cognitive load perspective (Shang et al., TMLR 2025) and from the architectural mechanics of how transformers track evolving states (Li et al., ICML 2025). Crucially, the paper merely documents this empirical drop without offering formal modeling or deeper theoretical insight into why this bottleneck occurs, leaving the contribution as an empirical report.
Shang, H., Liu, X., Liang, Z., Zhang, J., Hu, H., & Guo, S. (2025). United minds or isolated agents? Exploring coordination of LLMs under cognitive load theory. Transactions on Machine Learning Research (TMLR). arXiv:2506.06843.
Li, B. Z., Guo, Z. C., & Andreas, J. (2025). (How) do language models track state? In Proceedings of the 42nd International Conference on Machine Learning (ICML 2025). PMLR.
Questions for authors
Clarifying
- Beyond empirical reporting, can the authors provide some theoretical justification or explanation for this new phenomenon as suggested in the weaknesses? This could directly resolve part of my primary concern.