- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies the problem of context eviction in LLM agents. The authors contribute a controlled measurement for context deletion where they use paired teacher-forced replays with placebo controls to trace the effects of different context-element types across decision distance and later demonstrations. Then, the authors conduct empirical analysis to provide evidence against age- or attention-based eviction of different elements.
Critical strengths
- The problem of context deletion is both important and interesting for LLM agents. The work provides at least some insights that uncover the effects of context deletion that could potentially be useful.
- In the meanwhile, the environment is carefully controlled, the experiments are cleanly designed where the enumeration of factors is comprehensive.
Critical weaknesses
- The authors claim different measurement contributions, which I think present very limited novelty. Although the authors mention that they claim no methodological novelty for paired replay and attribution, they still fail to give any refreshing empirical insights regarding the behavior of LLM agents, especially without any models at larger scales.
- It is unclear about the impacts of the work. I think every result in the paper is from the difference in log probability in a synthetic task whose six families vary lexicon around one mechanism. The eviction comparison is quite narrow, and somehow unreliable. For a paper motivated by agent context eviction, the absence of any end-to-end outcome definitely damages the significance of the paper.
- I also find the paper extremely difficult to read. On the one hand, there are multiple estimators in the paper that yield different numbers for similar values. On the other hand, the paragraphs are also broken everywhere (for example, many undefined notations in the introduction), making many concepts and numbers difficult to interpret.
Questions for authors
- How can your methods generalize to models of different sizes and families?
- How can your findings provide any insights into free-running task success?