- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies how context compression affects the retention of standing instructions in long-running LLM agents. It tests three compression thresholds, two compression methods, and three reminder frequencies on τ-bench Retail and AppWorld. The results show that agents often forget instructions unrelated to the main task, especially output-format requirements. Instructions repeated in later interactions are better retained. Aggressive compression can also hurt task success and sometimes increase token usage.
Critical strengths
The paper investigates an important problem in long-horizon agent tasks: with context compression or truncation, agents may complete tasks successfully while failing to follow earlier instructions. This issue is relevant to real-world agent systems.
Critical weaknesses
- The constraints differ in how much effort they require from the agent. Trajectory-coupled constraints (C2–C5) may be satisfied naturally during task execution, and some, such as C4, may already align with the agent's default behavior. In contrast, task-external instructions (C1 and C6) require additional behaviors. This difference in difficulty may explain the observed gap in instruction retention. Comparing constraint satisfaction rates alone is not sufficient to support the contribution 1 (Compression destroys task-external output requirements far more often than trajectorycoupled rules).
- The paper lacks a direct analysis of what happens to instructions during compression. It uses constraint satisfaction to measure instruction retention, but does not distinguish whether an instruction was removed during compression, preserved but ignored by the agent, or rewritten that affects compliance.
- The paper mainly reports empirical findings. It only evaluate one potential solution (instruction reminders), while the other recommendations (Section 5) lack experimental validation.
- The evaluations only consider two basic compression methods, (truncation and LLM summarization), without evaluating any representative context compression methods. This limits the scope and generalizability of the empirical findings.
- The paper is difficult to follow. The descriptions and categorization of C1–C7 are confusing. The main parts lack a clear narrative, while the results mainly list metrics with limited discussion. The paper would benefit from a substantial rewrite.
Questions for authors
See above