acceptodds
Under review as a conference paper at ICLR 2027

LifeBench: A Benchmark for Long-Horizon Multi-Source Memory

Abstract

Modeling user memory is essential for building personalized AI agents. However, existing benchmarks mainly focus on sparse user information explicitly stated in conversational narratives. In contrast, real-world information is scattered across diverse digital sources, such as messages, calendars, and photos. Such observations form a dense and evolving data stream that only indirectly reveals underlying user preferences and habits. This mismatch limits the effective evaluation of agent memory systems. However, collecting real user data remains prohibited due to privacy concerns. This paper introduces LifeBench, a simulation framework that combines *hierarchical outline planning* for consistency control with *Objective Agents* for grounding generated data in real-world information. Evaluation shows that LifeBench better captures real-world preference shifts than existing memory benchmarks. In addition, its Objective-Agent mechanism enables digital traces that resemble human mobility patterns in real-world data without supervised training. We further synthesize a moderate-sized dataset and evaluate state-of-the-art memory systems. Results show that LifeBench more clearly exposes performance gaps among current systems. Deeper analysis reveals insights across data distributions and settings, including, e.g., *sparse vs dense data distributions*.

Primary area:
datasets and benchmarks
Keywords:
long-term memory, synthetic data, personal agents, memory systems, benchmarks, temporal reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MUQYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes LifeBench, a new benchmark to evaluate long-term memory systems using heterogeneous personal data. It has an LLM-based simulation framework, with objective and subjective agents to generate long-term memory pieces that are realistic.

Critical strengths
  1. The paper addresses the limitations of the conversation-centric memory benchmark. It emphasizes on the fragmented information that is injected in real time

  2. The benchmark coverage is diverse, with 9 question types.

  3. The evaluation is extensive. The authors evaluate 9 systems and identify the bottleneck of current memory system.

Critical weaknesses
  1. The novelty is limited and the related work is not discussed sufficiently. The paper overlooks some closely related benchmarks, like CloneMem and RealMem. CloneMem already investigates long-horizon memory from non-conversational digital traces using hierarchical generation, while RealMem evaluates evolving memory states.

  2. The empirical findings are well known. The "retrieval is a major bottleneck" is discussed extensively in [1], DynamicMem also reports the memory retrieval is responsible for most errors, the indexing methods also discussed in depth in [2]. Also, I find it's hard to locate the "so-what" message from the paper. Although it introduces a new benchmark, experiments do not demonstrate which previously unevaluated capabilities the designs reveal, why should any memory system be evaluated on it, or how the LifeBench performance translates to real-world better personalized agents.

[1] LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory [2] Does memory need graphs? a unified framework and empirical analysis for long-term dialog memory

  1. The realism is overstated. The paper uses interest-drift rate to claim that the benchmark is realistic. But it relies on one synthetic user's 90-day history, while mobility evaluation covers only five Beijing personas. Simply match a few aggregate statistics does not establish the benchmark's memory notes are realistic.

  2. Missing baselines. The evaluation misses some simple baselines like BM25, dense retrieval, etc.

  3. Human agreement is evaluated on only 150 answers from EverMemOS, with 77.3% agreement for temporal reasoning. The judge also shares the default answer model.

Questions for authors
  1. Can the authors independently ablate the generation components, like hierarchical planning, objective agent, etc? How they affect realism metrics or difficulty?

  2. How do the authors ensure that implicit habits, preferences, and hidden facts are uniquely inferable from observable records?

10 comments

1. Contemporaneous Work Comparison

The paper overlooks closely related benchmarks such as CloneMem and RealMem.

Thank you for pointing us to the related works CloneMem and RealMem. We would like to clarify that these are contemporaneous works (ACL, July 2026) that we were unaware of at the time of our submission (September 2026), rather than works intentionally excluded from our discussion. To provide further evidence of our independent development timeline, at the end of this response, we include our initial GitHub commits, dating back to October 2025.

We will cite and incorporate a detailed comparison with CloneMem and RealMem into the final version, as outlined below:

1.1. Statistical Comparison with CloneMem and RealMem

Both CloneMem and RealMem make important contributions: CloneMem hierarchically synthesizes persona-based life trajectories as memory corpora, while RealMem supports the evaluation of dynamic questioning and memory updates. The three benchmarks are compared below in terms of data form, scale, and realism validation:

BenchmarkData typeData densityNumber of questionsData scale (average per person)Number of events (average per person)Days with events (average per person)State evolutionRealism validation
CloneMemDiaries, social posts, chats, emailsSparse1,187~355k tokens848.9176.1 daysYes×
RealMemChats, project states—1,415~266K tokens——Yes×
LifeBenchDaily activities; fragmented phone recordsDense3,380~1.31M tokens5,748.6363.2 daysYes✓

1.2. LifeBench Offers Novel Contributions Beyond Concurrent Work

Although LifeBench shares part of its research objectives with these concurrent works, it makes distinct contributions that address important research gaps beyond their scope. Specifically, LifeBench addresses the following novel and challenging aspects:

1) Long-horizon fine-grained, dense events. This setting better reflects real-world scenarios while introducing greater practical challenges: numerous interrelated events occur within a short time window, with key information scattered across multiple pieces of evidence. Memory agents must therefore distill the main thread, retain essential details, and jointly model event information and evolving user personas. Such high event density exposes substantial differences in how well different memory agents handle information overload.

2) An extensible simulation system that also targets realism. This is also crucial. We hope the simulation system can serve and extend to more application scenarios, not just memory agents. For example, it can be used to train foundation models for personal assistants, which need to learn commonsense about real temporal and location semantics; it can also be used for app-usage optimization, user health monitoring, urban mobility planning, and related tasks.

These requirements make the simulation task highly challenging, and hierarchical planning alone cannot accomplish it:

  • Fine-grained, dense events: To generate such events, we need to characterize the correlated influence between related events and make a user's long-term life both consistent with the habitual patterns in the profile and reflective of change and development. For example, after an injury a user cannot exercise for a period until recovery; afterwards, the injury experience may also create a psychological shadow that affects decisions. A user's work should likewise be reflected in daily life: when scheduling key events such as travel or outings, work arrangements must be taken into account. This is where the Subjective Agent comes in—it maintains memory from the human user's perspective rather than simply generating events.

  • Extensible and designed for realism consideration: This part is handled by the Objective Agent. Realism consideration concerns not only reasonable temporal planning (which can integrate a calendar) and realistic locations (e.g., real restaurants, requiring the memory agent to reason that "KFC is a fried-chicken restaurant"), but also more difficult levels (user interest drift and mobility patterns). The simulation framework itself is extensible: users can import an initial persona profile, calendars, and map services to synthesize data for their own target scenarios. We recently used the framework to synthesize life data for a British programmer to extend geographic coverage, and the data has been released to the anonymous repository.

1. Contemporaneous Work Comparison

1.3. Evidence of Independent Development: Git Commit History

To document that LifeBench was developed independently of CloneMem and RealMem, we excerpt the earliest commit of our code repository. It dates to October 12, 2025, well before the public release of these concurrent works:

commit 011cb5c79c4ad65b15f0a15889012c0227a49d1a
Date:   Sun Oct 12 00:21:51 2025 +0800

    first commit

The complete git log export, including all subsequent commits, is available in the anonymous repository: https://anonymous.4open.science/r/lifebench-eval-repro-038C

2. LifeBench Experiments and Conclusion

The empirical findings are already well known. “Retrieval is the main bottleneck” has been extensively discussed in [1]; DynamicMem also reports that memory retrieval causes most errors, and indexing methods have been examined in depth in [2]. Moreover, I find it difficult to identify the “so what” message of the paper. Although it introduces a new benchmark, the experiments do not show which previously unmeasured capabilities its design reveals, why any memory system should be evaluated on this benchmark, or how performance on LifeBench translates into better personalized agents in the real world.

Our experiments compare existing strong memory systems under a unified setting and show that LifeBench remains challenging even for frontier LLMs. Regarding your concern about our conclusions: our empirical findings go beyond identifying "retrieval is the bottleneck" to quantifying the relative magnitude of retrieval error and other memory-construction loss (Table 6): the retrieval error loss of the two systems is similar, about 25% each, while the main difference lies in memory-unit construction, where EverMemOS incurs more information loss. These results indicate that retrieval error still has substantial room for improvement and that systems differ markedly in their memory units.

Comparison of dynamic and batch indexing also shows clear differences between flat and graph-structured indexes, indicating that memory systems or RAG methods differ in their adaptability to incremental indexing. This evaluation does not stop at "dynamic indexing performs worse" but explains why systems such as Hindsight are more robust: their advantage does not come from an index structure designed for streaming updates, but from broader candidate coverage at the reranking stage.

Data density effects across datasets are also informative and, to our knowledge, have not been systematically characterized.

Why LifeBench Is Useful for Memory Systems.

  1. It is a challenging benchmark: across the most representative memory systems and the strongest LLM backbones, LifeBench remains challenging; in particular, on temporal reasoning and hidden-information reasoning the strongest systems do not exceed 50% accuracy, and nondeclarative memory is only slightly above 50%. Nondeclarative-memory questions (covering habits, preferences, and emotions) benefit from the dense records unique to this dataset: unlike preference-focused datasets such as PersonaMem, our data captures long-term fluctuations in life habits and interests, including numerical and regularity-level changes, rather than relying on key events to assert that a habit has formed or an interest has changed. In addition, as LLM context windows grow rapidly, our extensible synthetic data, exceeding 1M tokens, poses a new challenge to memory systems.

  2. The evaluation setting is closer to reality: multi-source data are integrated, evolve over time, vary in density, and are grounded in real temporal events and locations; this is also the dataset most closely aligned with real user scenarios.

  3. The simulation framework can be extended to different locations and languages; the corresponding data statistics and evaluation results are to be added. We recently exploited our simulation to cover user simulation for Britain, which is included in our anonymous repository: https://anonymous.4open.science/r/lifebench-eval-repro-038C

Reviewer_Xs4YOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper introduces LifeBench, a realism-aware benchmark for evaluating long-term AI memory systems using synthetic, multi-source personal data and dynamic memory indexing, revealing key limitations in existing memory systems.

Critical strengths

1. Novel Benchmark Design: LifeBench addresses an important gap in existing memory benchmarks by introducing dense, multi-source personal data with long-term temporal evolution and dynamic memory indexing.

2. Comprehensive Evaluation: The paper evaluates nine representative memory systems across diverse reasoning tasks, revealing meaningful performance differences and limitations that conventional benchmarks may overlook.

3. Insightful Experimental Analysis: The extensive diagnostic experiments provide valuable insights into memory consolidation, retrieval bottlenecks, and the effects of dynamic indexing and data density on memory-system performance.

Critical weaknesses

1. Missing Core Experimental Code: The submitted materials do not include the complete evaluation code for the nine memory systems or their raw predictions. Therefore, the main experimental results in Tables 4–7 cannot be independently reproduced or verified.

2. Inconsistent Experimental Statistics: Table 3 reports a Normalized Shannon Entropy of 0.75, but directly calculating it from the submitted data using the paper's formula gives 0.6871. The authors do not provide sufficient details to explain this discrepancy.

3. Selective Exclusion in Human Evaluation: Table 18 excludes all 181 health records from the human evaluation without clearly explaining why. Therefore, the reported 95.38% Phone-record Validity does not cover all records in the sampled data.

4. Unverifiable Realism Evaluation: Tables 2 and 15 claim that LifeBench closely matches real-world human mobility patterns. However, the submitted materials lack the complete Beijing mobility evaluation data and experimental scripts, making it impossible to verify the reported JSD results.

Questions for authors

1. Synthesis Component Ablation: Could the authors provide ablation studies to clarify the individual contributions of hierarchical outline planning and the Objective Agent? How much does each component contribute to the realism and consistency of the generated data?

2. Dynamic Indexing Fairness: Appendix F.2 indicates that batch indexing may include records from after the query timestamp. How would the results in Table 7 change if both batch and dynamic indexing were restricted to information available at the query time? Could this affect the conclusions about dynamic indexing robustness?

7 comments

1. Result Reproduction and Mobility JSD Reproduction

Missing core experimental code: The submission does not include the complete evaluation code for the nine memory systems or their raw predictions.

Thank you for your careful review. We have added and released the following content, which is stored in two anonymous repositories. The data-quality evaluation repository contains the code used for the data-quality evaluation in the paper (covering interest drift, mobility-trajectory evaluation, diversity evaluation, and automatic evaluation), as well as the evaluation scripts and result data for human mobility patterns (the experimental design follows the WorldMove paper); link: https://anonymous.4open.science/r/lifebench-eval-repro-038C. We have also written detailed usage documentation to facilitate reproduction of the paper's conclusions. The complete evaluation code and raw predictions for the nine memory systems are maintained separately in another anonymous repository; link: https://anonymous.4open.science/r/LifeBench_eval-EF67/.

2. Shannon Entropy

Inconsistent experimental statistics: Table 3 reports a normalized Shannon entropy of 0.75, but applying the paper's formula directly to the submitted data yields 0.6871. The authors do not provide enough detail to explain the discrepancy.

Both values are reproducible, and the difference stems from two statistical choices: how categories are defined and the unit of aggregation. The details are presented in the below table, followed by further explanations.

Aggregation methodCategory treatmentNormalized Shannon entropy
Ten personas computed separately, then averaged (Table 3 protocol)Variants merged into nine categories; denominator log⁡9\log 90.7478
Ten personas computed separately, then averagedOriginal labels retained (10–12 categories per person); denominator log⁡Ku\log K_u0.7141
All 57,486 events pooled (the reviewer's calculation)13 original labels retained; denominator log⁡13\log 130.6871
All 57,486 events pooledVariants merged into nine categories; denominator log⁡9\log 90.7771

Category labels. The dataset was designed around, and actually uses, nine event labels: Personal Life, Family & Living Situation, Career, Health, Relationships, Education, Finance, Other, Unexpected Events. However, in the LLM's raw annotations during synthesis, Family & Living Situation appeared in several forms: Family & Living Situation, Family&Living Situation, Family &Living Situation, Family, and Home&Living Situation. Counting raw strings, the full set contains 13 labels; the extra labels come from inconsistencies in LLM annotation. Family, Family&Living Situation, and Home&Living Situation are not new semantic categories but annotation variants of the same category; only after merging do we obtain the nine designed categories.

Why merge. Without merging, the probability of one semantic category is split across several labels: on the one hand, K is inflated (the normalization denominator changes from log⁡9\log 9 to log⁡13\log 13); on the other hand, the category's probability is underestimated. Both change the entropy. After merging, K = 9, consistent with the data design, so entropy reflects the distribution over the nine life themes rather than differences in how annotation strings are written.

Why average per person rather than pooling all events. Table 3 describes "the life diversity of a single simulated persona", so we compute the normalized entropy over the nine-category distribution separately for each persona, Hu=−∑i=19puilog⁡pui/log⁡9H_u=-\sum_{i=1}^{9}p_{ui}\log p_{ui}/\log 9, and then take the arithmetic mean over the ten personas. The reviewer's 0.6871 instead pools the 57,486 events of the ten personas into one distribution (and does not merge them into the nine categories). The latter answers "the overall diversity of ten lives mixed together": it counts between-person differences as diversity, and therefore systematically exceeds the average of individual-person diversity (with the same nine-category merging, pooling gives 0.7771, while the per-person average gives 0.7478). Both protocols are valid in themselves, but they answer different questions; Table 3 is intended to answer the question about individual-person diversity.

Therefore, the difference between 0.6871 and 0.75 is not a calculation error but a difference between two protocols: "pooled events + 13 raw labels" versus "per-person + nine merged categories". This was not clearly explained in the paper's appendix, and we will make the diversity calculation explicit in the revision. The evaluation code has also been released in the anonymous repository (https://anonymous.4open.science/r/lifebench-eval-repro-038C). Thank you very much.

3. Manual Audit of Health Records

Table 18 excludes all 181 health records from human evaluation without clearly explaining why. The reported 95.38% validity rate for phone records therefore does not cover all records in the sample.

Thank you for your comments. This is an oversight in the sampling procedure. The original procedure first sampled daily event IDs and then sampled the phone records linked to those IDs. Full-day health summaries correspond to dates rather than events, so they have no corresponding daily event ID and could not be sampled. We additionally conducted a manual check of the 181 health records, with each record evaluated by two annotators.

Evaluation Sample of Health Records

UserStartEndHealth recordsReviewer 1 unreasonable rateReviewer 2 unreasonable rate
Yu XiaoweiSep. 1Oct. 316113.1% (8/61)4.9% (3/61)
Feng HaoranJan. 1Feb. 28593.4% (2/59)1.7% (1/59)
Sun YuweiMay 1Jun. 30614.9% (3/61)8.2% (5/61)

Overall, the health data are of high quality and reflect event fluctuations and the persona: they yield plausible estimates of weight change, step count, personal calorie expenditure, stress, and heart rate from activity changes and dietary/exercise schedules. Because the data involve many fields, however, a small number of implausible cases remain, mainly: (1) cross-field inconsistency, e.g., pace, exercise duration, and distance do not satisfy the expected conversion relation; (2) misaligned exercise-event extraction, i.e., when events are extracted from the life data, activities such as stretching may be included in the duration and distance of a run, or the exercise time may be extracted incorrectly; (3) step counts are underestimated.

4. Synthesis Component Ablation

Can the authors provide ablations that disentangle the contributions of hierarchical outline planning and the Objective Agent? How much does each component contribute to the realism and consistency of the generated data?

Thank you for your suggestions. We conduct ablations by contrasting three variants:

  • No-planning: Removes macro-level planning and generates each day directly from the persona profile, without reference to a long-term storyline. Both subjective and objective agents are kept.
  • w/o Subjective (or Objective-only): Retains the Objective Agent but removes the Subjective Agent, generating daily activities without maintaining continuous persona state or memory. The planning module is kept.
  • w/o Objective (or Subjective-only): Removes the Objective Agent and its enrichment, optimization, and geographic-allocation modules, generating activities solely from the Subjective Agent's persona state and memory, without external temporal or geographic constraints. The planning module is kept.

All variants and the full model (original version) are evaluated on the same 90-day data for the same persona (Feng Haoran), covering September 10 to December 8, 2025, to ensure comparability under identical initial conditions. We compare them in terms of event diversity (Normalized Shannon Entropy), interest drift, and event consistency (Automatic Evaluation).

Note that the paper assesses realism through real-world locations, mobility patterns, and interest drift. However, as removing the Objective Agent clearly compromises the realism of locations and mobility patterns, we report only interest drift, which remains a valid metric for comparison across all variants. The results are presented below.

Data GroupEventsNormalized Shannon Entropy ↑Interest Drift *Consistency ↑
Original version1,3970.73940.54465.79%
w/o Subjective (or Objective-only)1,4230.80850.53261.78%
w/o Objective (or Subjective-only)1,4610.78600.52569.90%
No-planning1,2850.69620.46978.43%

*For interest drift, values closer to the real-data reference (0.6311, OmniBehavior, NeurIPS 2026) indicate better alignment.

Discussion: The results largely align with our expectations, showing that removing different components affects data quality in distinct ways. The original version achieves the best interest-drift alignment with real data, while maintaining reasonable diversity and balancing consistency between the two single-agent variants.

Without the Subjective Agent, the system cannot maintain well the user's evolving state and memory, resulting in lower consistency (61.78%). However, diversity remains high (normalized entropy 0.8085), partly because the Objective Agent introduces rich contextual information. Conversely, without the Objective Agent, consistency is relatively higher (69.90%), but diversity decreases (0.7860 vs. 0.8085), reflecting the loss of activity enrichment and geographic allocation.

The no-planning variant presents an interesting observation: it has the lowest diversity but the highest consistency. Without long-term planning, the LLM tends to generate fewer cross-day related events, making consistency easier to maintain. In effect, the LLM takes a shortcut to achieving consistency at the expense of diversity.

Overall, these results suggest that hierarchical planning ensures long-term (event) diversity. The Subjective Agent contributes to maintaining consistency. The Objective Agent is essential for grounding real-world locations and enforcing more realistic mobility constraints, but it also enhances diversity. We will incorporate this ablation analysis and discussion into the revised paper. Thank you for the helpful suggestion.

Reviewer_UBNfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LifeBench evaluates memory systems for personal assistants using a simulated year of personal digital records. The generation pipeline combines life-event planning, subjective and objective agents, and geographic constraints to produce records from multiple sources. The benchmark contains 10 simulated users, 47,409 phone records, and 3,380 questions across nine memory task categories. The authors evaluate nine memory systems and examine how compression, retrieval, and incremental updates affect performance. A central motivation is to test whether an assistant can recover and connect information from a long, evolving personal history.

Critical strengths

The evaluation setting is well motivated. Personal assistants often have to infer habits or recover past events from scattered records, rather than from explicit statements by the user. LifeBench brings together multiple record sources and interleaves ingestion with answering, which makes it a useful setting for studying memory under ongoing updates.

I found the diagnostic experiments useful. The Gold-Raw, Gold-Unit, and Normal Retrieval comparisons help distinguish information lost during memory construction from information missed during retrieval. The candidate-pool ablation in Appendix E.7 is also informative: keeping the retrieval channels and ranking procedure fixed makes the effect of candidate coverage easier to interpret. These analyses give some insight into why a system fails beyond its aggregate score.

Critical weaknesses
  1. My main concern is the dynamic-indexing comparison (§3.5 and Appendix F.2). The batch setting uses the full year of records, while the dynamic setting only sees records available at query time. This means the two settings differ in both how they build memory and what information they can use. Later records may add useful details or change the summaries and graphs. The differences in Table 7 therefore cannot be attributed to incremental indexing alone. The claim that graph-based memory performs worse under streaming updates needs stronger evidence.

  2. I also have concerns about whether the questions can be answered from the available records. The generator knows the personas and life events, but the evaluated systems may only see part of this information. This is especially important for causal questions: the records may not explain why an event happened.

Questions for authors
  1. Can you compare incremental indexing with rebuilding from scratch at several query cutoffs, using exactly the same history prefix in both conditions? Do the conclusions about graph-based memory still hold under this comparison?

  2. How many users and time windows contribute to the reported preference-drift rates? Does the comparison remain similar after accounting for differences in daily observation counts and missing days?

  3. What information is actually provided to the evaluated systems and to Gold-Raw: phone records only, or also activity records and other generated information? For causal and inferential questions, did annotators verify answerability using only the records visible at query time?

3 comments

1. Indexing Protocol and Same-History-Prefix Experiment

Can the authors compare incremental indexing with rebuilding from scratch at multiple query cutoffs, using exactly the same history prefix in both conditions? Does the conclusion about graph-based memory still hold under this comparison?

Thank you for your suggestions. Because rebuilding the index from scratch for every date with questions would be prohibitively expensive, we select the three query dates with the most questions for the dynamic-indexing comparison: June 15 (87 questions), August 15 (51 questions), and December 31 (1,881 questions). At each time point, we build the index in two ways, dynamic update and truncated offline, and then answer that day's questions; neither condition ingests later records, so the history prefix is identical. The results are as follows:

Dynamic Update vs. Truncated Offline under the Same History Prefix (Accuracy)

System / Method2025-06-152025-08-152025-12-31
GraphRAG · Dynamic update35.63%17.65%14.99%
GraphRAG · Truncated offline62.07%62.75%30.04%
EverMemOS · Dynamic update74.71%86.27%52.26%
EverMemOS · Truncated offline96.55%86.27%58.32%

Note: Dynamic update means ingesting records incrementally day by day and maintaining the index; truncated offline means rebuilding the index from scratch at the cutoff date. GraphRAG's dynamic update is clearly lower than truncated offline; EverMemOS's dynamic update is lower than truncated offline on 2025-06-15 and 2025-12-31, and the two are tied on 2025-08-15. The conclusion in the paper remains valid. We will update this in our final version, thank you.

2. Answerability and System Inputs

I am also concerned about whether the questions can be answered from the available records. The generator knows the persona and life events, but the evaluated systems may see only a subset of that information. This is particularly important for causal questions: the records may not explain why an event occurred.

Thank you for your comments. We would like to provide the following clarifications.

Question answerability is ensured through both the QA generation process and post-generation verification. First, although question generation may consult both life events and phone data to improve question quality, the question wording and answer are refined using phone-record evidence to ensure that the answer can be derived from phone records alone. Second, each candidate question undergoes quality checking based solely on its associated phone records, including one round of LLM-based screening and one round of human review. We also use an automated script to verify that all evidence records precede the question timestamp, ensuring temporal validity. During manual screening, annotators assess answerability using only the available phone records before the question time. Therefore, all questions, including those requiring causal reasoning or inference, are designed to be answerable from temporally valid phone-record evidence.

Further evidence of question answerability is provided in Table 4, which reports an accuracy of 97.28% under the Gold (Phone) Evidence setting, where the phone-record evidence associated with each question is directly provided to the LLM. Of the 3,380 questions, only 92 are answered incorrectly. Our analysis attributes these errors mainly to three factors: (1) complex, multi-hop reasoning required to derive the answer from the context; (2) semantically correct answers expressed using different wording; and (3) model refusals or empty responses. In other words, our manual analysis confirmed that all 92 questions are answerable from the provided evidence and that these errors arise from the answer model. We will clarify this analysis in the final version. Thank you.

Distribution of Unanswered Questions under the Gold Evidence Setting (by Primary Type)

Primary typeIncorrectTotal
Multi_hop41789
Single_hop241,346
Pattern_recognition16192
Temporal5140
Causal5104
Conflict1216
Other0593

3. Interest Drift

How many users and time windows contributed to the reported preference drift rates? Are the comparisons still similar after accounting for differences in daily observation counts and missing days?

In the paper, we follow the interest-drift analysis in the OmniBehavior paper (NeurIPS 2026). OmniBehavior provides activity records for 200 anonymous, real users over 90 days, collected from Kuaishou. The paper (OmniBehavior) presents a case study analyzing the interest evolution of a single user over 90 days, comparing real-world data with LoCoMo. We adopt this analysis as a reference for our corresponding one-user, 90-day analysis of Mem-PAL and LifeBench. We also adopt the analysis prompt provided in OmniBehavior for our experiments.

Here, we added a richer experiment: for each dataset, we take five users' 90-day trajectories, compute their interest distributions, and report the mean and cross-user standard deviation. Regarding your concern about differences in daily observation counts and missing days, we note that dense daily observations (covering a person's whole day) and few missing days are precisely an advantage of our synthetic data; nevertheless, we also added comparisons that correct for these differences. The final conclusion is unchanged: our data remains the closest to real users' interest characteristics.

Interest Drift under Different Correction Schemes (Adjacent-Day Jaccard Distance)

DatasetUsersOriginal protocolTop-5 truncationExclude missing daysTop-5 + exclude missing daysMissing days
LifeBench50.540 ± 0.0260.578 ± 0.0520.540 ± 0.0260.578 ± 0.0520.6 ± 0.9 days
Mem-PAL50.777 ± 0.0300.777 ± 0.0300.731 ± 0.0290.731 ± 0.0292.6 ± 1.8 days
OmniBehavior50.620 ± 0.0440.622 ± 0.0460.620 ± 0.0440.622 ± 0.0462.0 ± 4.5 days

The columns are defined as follows: Original protocol follows the original paper and treats an adjacent-day pair with at least one empty side as drift = 1; Top-5 truncation retains only the top five interests per day to correct for differences in daily observation counts; Exclude missing days removes adjacent-day pairs with at least one empty side to correct for missing days; Top-5 + exclude missing days applies both corrections. As can be seen, after correction the values for each dataset change only slightly, and the original conclusion remains stable. The new results (0.620 for OmniBehavior) are also consistent with those reported in the OmniBehavior paper, which shows that the interest drift of real users to be 0.6311. This agreement further supports the reliability of our findings. We sincerely thank you for this valuable suggestion, which has helped strengthen our paper. We will incorporate these new results into the final version.

4. Input Scope of the Evaluated Systems and Gold-Raw

What information is actually provided to the evaluated systems and Gold-Raw: only phone records, or also activity records and other generated information? For causal and inference questions, did annotators verify answerability using only records visible at query time?

Both the evaluated systems and Gold-Raw use only phone records; neither includes life data, activity records, or other generated information. We will clarify this in our paper, thank you.

AuthorsOpenReview ↗

Appreciate your comments

We appreciate your valuable comments, which have greatly helped refine our work. We have done our best to conduct extensive experiments to address the concerns you raised. Please let us know if you have any further questions. Thank you very much.

Comments on the paper

2 comments

We appreciate the reviewer’s valuable comments, which have greatly helped refine our work. We have conducted extensive experiments to address the concerns you raised previously. Please let us know if you have any further concerns. Thank you very much.

Dear Area Chair and Reviewers,

We sincerely thank you for your valuable feedback and constructive suggestions, which have helped us substantially improve our paper. In response to your comments, we have carefully revised and strengthened the manuscript, incorporating additional experiments, methodological clarifications, and more rigorous evaluations. Changes relative to the initial manuscript are marked in blue.

Specifically, we have made the following major improvements:

  1. Additional ablation studies on the synthesis framework: We conducted new ablation experiments to better understand the contributions of individual components.

  2. Stronger LLM-as-a-Judge evaluations: We supplemented comparisons of consistency between human judges and LLM judges under different LLM backbones and various system configurations.

  3. Interest drift quantification: We enhanced the calculation of interest drift, including using more samples per dataset, eliminating the impact of missing days, and unifying the number of interests.

  4. More rigorous dynamic vs. batch indexing comparisons: We refined our experiments to ensure that the indexing strategy is the only variable that differs between the two settings, enabling a more controlled and fair comparison.

  5. Clarification of question generation and validation: We further detailed the screening and filtering procedures for synthesized questions, as well as the mechanisms used to ensure their answerability based on the available information.

  6. Additional clarifications and corrections: We carefully addressed the remaining concerns raised by the reviewers, resolving ambiguities, correcting identified issues, and improving the overall clarity and rigor of the manuscript.

  7. More Experimental Baselines: We add naive retrieval methods as comparison baselines.

We believe these revisions have substantially strengthened the empirical evidence, methodological rigor, and overall presentation of our work. We hope that the updated manuscript and our detailed responses adequately address the reviewers' concerns and provide a clearer understanding of the contributions and reliability of our approach.

We sincerely appreciate the time and effort that the Area Chair and Reviewers have devoted to evaluating our work, and we thank you again for your thoughtful feedback and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.