- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes LifeBench, a new benchmark to evaluate long-term memory systems using heterogeneous personal data. It has an LLM-based simulation framework, with objective and subjective agents to generate long-term memory pieces that are realistic.
Critical strengths
-
The paper addresses the limitations of the conversation-centric memory benchmark. It emphasizes on the fragmented information that is injected in real time
-
The benchmark coverage is diverse, with 9 question types.
-
The evaluation is extensive. The authors evaluate 9 systems and identify the bottleneck of current memory system.
Critical weaknesses
-
The novelty is limited and the related work is not discussed sufficiently. The paper overlooks some closely related benchmarks, like CloneMem and RealMem. CloneMem already investigates long-horizon memory from non-conversational digital traces using hierarchical generation, while RealMem evaluates evolving memory states.
-
The empirical findings are well known. The "retrieval is a major bottleneck" is discussed extensively in [1], DynamicMem also reports the memory retrieval is responsible for most errors, the indexing methods also discussed in depth in [2]. Also, I find it's hard to locate the "so-what" message from the paper. Although it introduces a new benchmark, experiments do not demonstrate which previously unevaluated capabilities the designs reveal, why should any memory system be evaluated on it, or how the LifeBench performance translates to real-world better personalized agents.
[1] LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory [2] Does memory need graphs? a unified framework and empirical analysis for long-term dialog memory
-
The realism is overstated. The paper uses interest-drift rate to claim that the benchmark is realistic. But it relies on one synthetic user's 90-day history, while mobility evaluation covers only five Beijing personas. Simply match a few aggregate statistics does not establish the benchmark's memory notes are realistic.
-
Missing baselines. The evaluation misses some simple baselines like BM25, dense retrieval, etc.
-
Human agreement is evaluated on only 150 answers from EverMemOS, with 77.3% agreement for temporal reasoning. The judge also shares the default answer model.
Questions for authors
-
Can the authors independently ablate the generation components, like hierarchical planning, objective agent, etc? How they affect realism metrics or difficulty?
-
How do the authors ensure that implicit habits, preferences, and hidden facts are uniquely inferable from observable records?
10 comments
1. Contemporaneous Work Comparison
1.3. Evidence of Independent Development: Git Commit History
To document that LifeBench was developed independently of CloneMem and RealMem, we excerpt the earliest commit of our code repository. It dates to October 12, 2025, well before the public release of these concurrent works:
commit 011cb5c79c4ad65b15f0a15889012c0227a49d1a
Date: Sun Oct 12 00:21:51 2025 +0800
first commit
The complete git log export, including all subsequent commits, is available in the anonymous repository: https://anonymous.4open.science/r/lifebench-eval-repro-038C
2. LifeBench Experiments and Conclusion
The empirical findings are already well known. “Retrieval is the main bottleneck” has been extensively discussed in [1]; DynamicMem also reports that memory retrieval causes most errors, and indexing methods have been examined in depth in [2]. Moreover, I find it difficult to identify the “so what” message of the paper. Although it introduces a new benchmark, the experiments do not show which previously unmeasured capabilities its design reveals, why any memory system should be evaluated on this benchmark, or how performance on LifeBench translates into better personalized agents in the real world.
Our experiments compare existing strong memory systems under a unified setting and show that LifeBench remains challenging even for frontier LLMs. Regarding your concern about our conclusions: our empirical findings go beyond identifying "retrieval is the bottleneck" to quantifying the relative magnitude of retrieval error and other memory-construction loss (Table 6): the retrieval error loss of the two systems is similar, about 25% each, while the main difference lies in memory-unit construction, where EverMemOS incurs more information loss. These results indicate that retrieval error still has substantial room for improvement and that systems differ markedly in their memory units.
Comparison of dynamic and batch indexing also shows clear differences between flat and graph-structured indexes, indicating that memory systems or RAG methods differ in their adaptability to incremental indexing. This evaluation does not stop at "dynamic indexing performs worse" but explains why systems such as Hindsight are more robust: their advantage does not come from an index structure designed for streaming updates, but from broader candidate coverage at the reranking stage.
Data density effects across datasets are also informative and, to our knowledge, have not been systematically characterized.
Why LifeBench Is Useful for Memory Systems.
-
It is a challenging benchmark: across the most representative memory systems and the strongest LLM backbones, LifeBench remains challenging; in particular, on temporal reasoning and hidden-information reasoning the strongest systems do not exceed 50% accuracy, and nondeclarative memory is only slightly above 50%. Nondeclarative-memory questions (covering habits, preferences, and emotions) benefit from the dense records unique to this dataset: unlike preference-focused datasets such as PersonaMem, our data captures long-term fluctuations in life habits and interests, including numerical and regularity-level changes, rather than relying on key events to assert that a habit has formed or an interest has changed. In addition, as LLM context windows grow rapidly, our extensible synthetic data, exceeding 1M tokens, poses a new challenge to memory systems.
-
The evaluation setting is closer to reality: multi-source data are integrated, evolve over time, vary in density, and are grounded in real temporal events and locations; this is also the dataset most closely aligned with real user scenarios.
-
The simulation framework can be extended to different locations and languages; the corresponding data statistics and evaluation results are to be added. We recently exploited our simulation to cover user simulation for Britain, which is included in our anonymous repository: https://anonymous.4open.science/r/lifebench-eval-repro-038C
1. Contemporaneous Work Comparison
Thank you for pointing us to the related works CloneMem and RealMem. We would like to clarify that these are contemporaneous works (ACL, July 2026) that we were unaware of at the time of our submission (September 2026), rather than works intentionally excluded from our discussion. To provide further evidence of our independent development timeline, at the end of this response, we include our initial GitHub commits, dating back to October 2025.
We will cite and incorporate a detailed comparison with CloneMem and RealMem into the final version, as outlined below:
1.1. Statistical Comparison with CloneMem and RealMem
Both CloneMem and RealMem make important contributions: CloneMem hierarchically synthesizes persona-based life trajectories as memory corpora, while RealMem supports the evaluation of dynamic questioning and memory updates. The three benchmarks are compared below in terms of data form, scale, and realism validation:
1.2. LifeBench Offers Novel Contributions Beyond Concurrent Work
Although LifeBench shares part of its research objectives with these concurrent works, it makes distinct contributions that address important research gaps beyond their scope. Specifically, LifeBench addresses the following novel and challenging aspects:
1) Long-horizon fine-grained, dense events. This setting better reflects real-world scenarios while introducing greater practical challenges: numerous interrelated events occur within a short time window, with key information scattered across multiple pieces of evidence. Memory agents must therefore distill the main thread, retain essential details, and jointly model event information and evolving user personas. Such high event density exposes substantial differences in how well different memory agents handle information overload.
2) An extensible simulation system that also targets realism. This is also crucial. We hope the simulation system can serve and extend to more application scenarios, not just memory agents. For example, it can be used to train foundation models for personal assistants, which need to learn commonsense about real temporal and location semantics; it can also be used for app-usage optimization, user health monitoring, urban mobility planning, and related tasks.
These requirements make the simulation task highly challenging, and hierarchical planning alone cannot accomplish it:
Fine-grained, dense events: To generate such events, we need to characterize the correlated influence between related events and make a user's long-term life both consistent with the habitual patterns in the profile and reflective of change and development. For example, after an injury a user cannot exercise for a period until recovery; afterwards, the injury experience may also create a psychological shadow that affects decisions. A user's work should likewise be reflected in daily life: when scheduling key events such as travel or outings, work arrangements must be taken into account. This is where the Subjective Agent comes in—it maintains memory from the human user's perspective rather than simply generating events.
Extensible and designed for realism consideration: This part is handled by the Objective Agent. Realism consideration concerns not only reasonable temporal planning (which can integrate a calendar) and realistic locations (e.g., real restaurants, requiring the memory agent to reason that "KFC is a fried-chicken restaurant"), but also more difficult levels (user interest drift and mobility patterns). The simulation framework itself is extensible: users can import an initial persona profile, calendars, and map services to synthesize data for their own target scenarios. We recently used the framework to synthesize life data for a British programmer to extend geographic coverage, and the data has been released to the anonymous repository.