- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces MemoryLifeBench. It combines fact invalidation times from synthetic data and future evidence-use times from real conversations into a prediction dataset with right-censoring labels. It then evaluates how these predictions can guide memory retention decisions. The authors compare lifetime-threshold policies such as TTL with future-use policies such as Utility, and measure their downstream QA performance. However, the discussion and comparison with closely related work are insufficient. The link between the prediction targets and practical memory management goals is also not fully tested. TTL performs worse than FIFO and LRU, while Utility does not show a reliable advantage over LRU. These results leave both the benchmark’s added value and the effectiveness of the proposed methods unclear.
Critical strengths
-
The research problem is practically relevant. Long-running assistants need to decide which memories to retain under a storage budget. They also need to limit the impact of lost evidence on later QA.
-
The authors report that TTL performs worse than simple baselines. They also analyze evidence retention and QA results, providing useful material for studying failures in this specific design.
Critical weaknesses
-
Missing closely related work and an unclear novelty claim. Fact duration prediction, memory retention, and memory eviction all have direct precedents. However, the paper does not cite or sufficiently compare with the following studies:
- Zhang and Choi, Mitigating Temporal Misalignment by Discarding Outdated Facts, EMNLP 2023: This work already predicts fact duration and evaluates its use in QA with outdated knowledge.
- EMBER: Efficient Memory via Budgeted Evidence Retention for Long-Horizon Agents: This work studies evidence retention under a budget when future questions are unknown. It separately evaluates evidence retention, reading, and final answer quality.
- Selective Memory Retention for Long-Horizon LLM Agents: This work scores memories using features such as age, access frequency, redundancy, and downstream utility. It compares memory eviction against FIFO, LRU, and other policies.
Without a clear comparison with these studies, the paper does not sufficiently establish fact lifetime prediction, utility-based retention, or staged evaluation as its main novel contributions.
-
The upstream prediction metrics do not provide validated guidance for downstream QA. The event in the synthetic data is fact invalidation, while the event in real conversations is future use. These events have different meanings for retention decisions. After pooling them, the overall temporal prediction score is hard to interpret as a consistent measure of memory lifetime prediction. The paper shows good prediction ranking but poor downstream results. However, it does not isolate the effects of target definitions, temporal calibration, and eviction rules through controlled experiments. It therefore does not show that a high benchmark score can guide better memory management. Showing a gap between proxy metrics and downstream results alone is not enough to establish a strong research contribution.
-
The proposed policies do not show a reliable advantage over simple LRU. TTL performs worse than FIFO and LRU in the reported downstream evaluation. Utility improves evidence retention, but its QA results do not clearly establish an advantage over LRU. The main sample also overlaps with training conversations, while the independent validation/test subset is small. Moreover, switching from TTL to Utility changes both the prediction signal and the selection rule. The paper lacks a utility-only model with matched inputs and a TTL-based ranking control. It therefore does not establish the need for survival prediction. The current results neither establish a better method nor fully explain why the complex design falls behind simple policies.
Questions for authors
-
What does this paper add beyond prior work on fact duration prediction, budgeted evidence retention, and memory eviction? Why are these closely related studies not cited, and why are applicable experimental comparisons missing?
-
Why are fact invalidation time and future-use time combined into one prediction task? Could the authors define and evaluate these targets separately? Could they also test whether better predictions consistently lead to better evidence retention and downstream QA?
-
Can the proposed policies reliably outperform LRU on a test set with no training overlap and independently specified storage budgets? Please use ablations with matched inputs and capacities to separate the effects of survival supervision, utility prediction, and selection rules. If there is still no advantage, what conclusions can the paper support beyond this specific implementation?
-
Figure 1 appears to be an unedited AI-generated SVG and is visually unappealing. I suggest that the authors revise it to improve its visual quality.