acceptodds
Under review as a conference paper at ICLR 2027

MemoryLifeBench: Evaluating Survival-Based and Utility-Ranked Memory Retention in LLM Assistants

Abstract

Long-running LLM assistants must decide which memories to retain under a storage budget. Predicting when a memory-related event occurs is a plausible intermediate objective, but does accurate prediction lead to effective retention? MemoryLifeBench examines this question using 10,152 right-censored records: synthetic records measure time to factual invalidation, whereas LoCoMo and LongMemEval records measure time to observed future evidence use. A Cox-trained predictor with 213,889 trainable parameters achieves a test C-index of 0.7218, exceeding five prompted and heuristic baselines. Nevertheless, MemoryLifeBench-TTL, which converts survival predictions into expiry thresholds, underperforms FIFO and reference-recency LRU on downstream question answering. MemoryLifeBench-Utility instead retains the top- memories by a separately supervised future-use score, increasing LoCoMo evidence retention from 0.6687 to 0.8765 at matched capacity. Its downstream improvement over FIFO and the TTL policy is exploratory: 80% of the 120-question LoCoMo sample comes from training conversations, and the 24-question validation/test subset does not establish superiority. An evidence oracle also outperforms retaining all memories with ordinary retrieval. These results separate event-time discrimination, budgeted retention, and retrieval quality, and motivate evaluating the memory decision directly rather than relying on a temporal prediction metric.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
LLM Memory, Long-Term Memory, Memory Retention, Survival Analysis, Time-to-Event Modeling, Memory Utility, LLM Agents, Benchmarking, Retrieval-Augmented Generation, Conversational AI

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_2CsTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces MemoryLifeBench. It combines fact invalidation times from synthetic data and future evidence-use times from real conversations into a prediction dataset with right-censoring labels. It then evaluates how these predictions can guide memory retention decisions. The authors compare lifetime-threshold policies such as TTL with future-use policies such as Utility, and measure their downstream QA performance. However, the discussion and comparison with closely related work are insufficient. The link between the prediction targets and practical memory management goals is also not fully tested. TTL performs worse than FIFO and LRU, while Utility does not show a reliable advantage over LRU. These results leave both the benchmark’s added value and the effectiveness of the proposed methods unclear.

Critical strengths
  1. The research problem is practically relevant. Long-running assistants need to decide which memories to retain under a storage budget. They also need to limit the impact of lost evidence on later QA.

  2. The authors report that TTL performs worse than simple baselines. They also analyze evidence retention and QA results, providing useful material for studying failures in this specific design.

Critical weaknesses
  1. Missing closely related work and an unclear novelty claim. Fact duration prediction, memory retention, and memory eviction all have direct precedents. However, the paper does not cite or sufficiently compare with the following studies:

    Without a clear comparison with these studies, the paper does not sufficiently establish fact lifetime prediction, utility-based retention, or staged evaluation as its main novel contributions.

  2. The upstream prediction metrics do not provide validated guidance for downstream QA. The event in the synthetic data is fact invalidation, while the event in real conversations is future use. These events have different meanings for retention decisions. After pooling them, the overall temporal prediction score is hard to interpret as a consistent measure of memory lifetime prediction. The paper shows good prediction ranking but poor downstream results. However, it does not isolate the effects of target definitions, temporal calibration, and eviction rules through controlled experiments. It therefore does not show that a high benchmark score can guide better memory management. Showing a gap between proxy metrics and downstream results alone is not enough to establish a strong research contribution.

  3. The proposed policies do not show a reliable advantage over simple LRU. TTL performs worse than FIFO and LRU in the reported downstream evaluation. Utility improves evidence retention, but its QA results do not clearly establish an advantage over LRU. The main sample also overlaps with training conversations, while the independent validation/test subset is small. Moreover, switching from TTL to Utility changes both the prediction signal and the selection rule. The paper lacks a utility-only model with matched inputs and a TTL-based ranking control. It therefore does not establish the need for survival prediction. The current results neither establish a better method nor fully explain why the complex design falls behind simple policies.

Questions for authors
  1. What does this paper add beyond prior work on fact duration prediction, budgeted evidence retention, and memory eviction? Why are these closely related studies not cited, and why are applicable experimental comparisons missing?

  2. Why are fact invalidation time and future-use time combined into one prediction task? Could the authors define and evaluate these targets separately? Could they also test whether better predictions consistently lead to better evidence retention and downstream QA?

  3. Can the proposed policies reliably outperform LRU on a test set with no training overlap and independently specified storage budgets? Please use ablations with matched inputs and capacities to separate the effects of survival supervision, utility prediction, and selection rules. If there is still no advantage, what conclusions can the paper support beyond this specific implementation?

  4. Figure 1 appears to be an unedited AI-generated SVG and is visually unappealing. I suggest that the authors revise it to improve its visual quality.

Reviewer_WQzdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MemoryLifeBench, a benchmark for evaluating memory retention in LLM assistants. It trains a Cox-based survival model to predict memory-related event times and compares TTL-based retention with utility-based ranking and recency baselines. The results show that better event-time prediction does not necessarily lead to better retention decisions. Utility ranking improves evidence retention, but its downstream QA advantage over LRU remains unverified.

Critical strengths
  1. A useful distinction between prediction and decision-making. The paper highlights that event-time prediction, memory retention, and retrieval quality are different objectives. This provides a useful framework for diagnosing memory management failures.

  2. Multi-stage evaluation. Evaluating both evidence retention and downstream QA, along with an evidence oracle, helps distinguish information loss from retrieval failures.

Critical weaknesses
  1. The problem formulation is too narrow and mixes different objectives. The benchmark combines factual invalidation time with future evidence-use time, which have fundamentally different meanings. More broadly, LLM memory management involves compression, consolidation, and retrieval. Explicit expiration prediction addresses only a limited part of this problem. Predicting expiration from an isolated memory is also questionable for information that remains useful as historical context after its apparent expiration.

  2. Insufficient evidence for downstream benefits. Although utility ranking substantially improves evidence retention, its QA evaluation relies heavily on questions from training conversations. The held-out evaluation is too small to establish a reliable advantage over LRU, and the comparison does not isolate the effects of the utility objective from the selection strategy.

Questions for authors
  1. Can you report source-specific survival results and clarify whether the main conclusions hold when factual invalidation and future evidence use are evaluated separately?

  2. Can you provide a downstream QA evaluation on strictly held-out conversations, with no overlap with training conversations, and compare Utility against LRU under identical capacity and retrieval settings?

Reviewer_VvP7OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper asks whether an LLM assistant knows which memories to retain (under limited storage). It frames the task as a survival problem. In synthetic dialogues, the event is a stored fact becoming outdated. In records drawn from existing long-term-memory benchmarks, the event is the memory's next use as QA evidence. The authors find that predicting event time well does not necessarily produce a good retention policy. A small Cox model (MLB-Survival) ranks event times better than prompted LLMs. Yet the corresponding TTL policy, which deletes a memory once predicted survival falls to q=0.5q = 0.5, retains less QA evidence than FIFO at the same capacity. A second output layer, MLB-Utility, keeps the top NN memories with the highest predicted future utility as evidence and retains more evidence than FIFO, LRU, and MLB-TTL.

Critical strengths
  • S1: Separating measuring ranking, retention, and retrieval helps identify where a policy loses evidence.
Critical weaknesses
  • W1: Potential train-test leakage.

    • The utility head is trained on LoCoMo evidence-use labels (§4.1), and retention is evaluated on the same labels. 8 of 10 LoCoMo conversations are in the training split (§5). The 1,304-question diagnostic includes non-held-out conversations (Table 2), so the 0.669→0.8770.669 \to 0.877 gain in Table 5 might be attributed to being in-sample.
    • This seems to be supported by the evaluation on the held-out conversations where the improvement mostly disappears. MLB-Utility answers 2 of 24 questions correctly, compared with 0 for FIFO. Its F1 is below that of LRU and No-Forget (Table 6).
  • W2: MLB-TTL's loss to FIFO seems expected.

    • On conversation records, a short predicted time means that the memory will be needed soon. A TTL policy therefore deletes the memories needed first. The result seems to support this too (§6.3).
    • The authors should consider adding a control experiment that keeps the top NN memories by survival risk at matched capacity. Without this, the authors cannot isolate where the gain of MLB-Utility's come from (could be the top-NN rule instead).
    • The authors should consider adding standard survival baseline such as DeepHit (Lee et al., 2018) or a random survival forest.

Also please improve the writing by explicitly introducing the terms used (e.g., "FIFO", "TTL", etc. were not introduced properly).

References

  • Lee et al. (2018), DeepHit: A Deep Learning Approach to Survival Analysis with Competing Risks.
Questions for authors
  • Q1: How were the scalar TTL outputs of the LLM and heuristic baselines turned into risk scores for the C-index?
  • Q2: What counts as a "reference" for the LRU baseline? If references come from the QA evidence links, LRU uses the same labels as training.
  • Q3: How were the 12 questions per conversation chosen for the 120-question sample?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.