- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes graded forgetting for capacity-bounded LLM agent memory, where old memories are progressively compressed rather than deleted. It also introduces a recall-after-eviction evaluation protocol that specifically measures recall of information removed by a destructive retention policy. On a LoCoMo-based long-term memory benchmark under a 35% storage budget, graded forgetting substantially outperforms delete-oldest (60.0% vs. 2.5% accuracy) and also outperforms capacity-constrained Mem0 (66.7% vs. 26.7%).
Critical strengths
- The problem of long memory is real and challenging.
- The main empirical effect is large and easy to interpret.
Critical weaknesses
-
The comparison against destructive forgetting is currently too weak to establish the broader case for graded forgetting. The decisive baseline is delete-oldest, which is a useful baseline, but it does not establish that graded forgetting is preferable to a realistic scenario. Cited papers are not properly discussed, such as MemRefine and "Learning What Not to Forget: Long-Horizon Agent Memory from a Few Kilobytes of Learning", which directly studies learned relevance-based eviction under matched budgets and includes LoCoMo evaluation. Comparisons against these methods, or at least comparable strong content-aware policies, seem necessary to determine whether the contribution is a competitive memory-management algorithm rather than a demonstration that compression can outperform recency deletion.
-
The empirical validation is too narrow for the current claims, especially given the computational cost. The main result contains only 40 questions from one 833-day timeline, at one 35% storage budget, using one primary backbone. There is no budget sweep, no evaluation across multiple independent timelines or benchmarks, and no uncertainty estimate. This makes it difficult to know the shape or robustness of the claimed advantage.
-
Token cost. Graded forgetting uses 33,423 tokens/query versus 2,806 for deletion, approximately a 12x difference. I agree with the authors that this may be an implementation artifact rather than intrinsic to compression, but as evaluated it is a substantial practical cost. Since the paper is motivated by resource-constrained deployment, storage cannot be considered entirely independently of inference cost.
Questions for authors
See Weaknesses.