acceptodds
Under review as a conference paper at ICLR 2027

LifeMemBench: Benchmarking Long-Term Memory for Evolving User States through Personalized Planning

Abstract

Personalized language-model agents increasingly interact with users over extended periods, yet user preferences, constraints, goals, and circumstances evolve over time. Agents must therefore recover relevant information, resolve what remains valid, and use the resulting user state in subsequent decisions. Existing evalua- tions have studied long-term retrieval, temporal reasoning, memory updates, and preference following, but often separately from the downstream decisions they are intended to support. We introduce LifeMemBench, a benchmark for longitudinal personalization under evolving user states. LifeMemBench constructs controlled latent user-state trajectories before realizing them as chronological interactions, providing query-time supervision for both current and historical states. It jointly evaluates state recovery, personalized planning, and replanning through a common process of recovering relevant information, resolving the valid state, and acting on it. Spanning 8K–64K tokens, planning and replanning are evaluated against explicit user-specific and feasibility constraints to test whether longitudinal information af- fects downstream behavior. We further introduce CoMem, a lightweight state-aware memory system that preserves historical evidence, tracks state changes, and derives an actionable current-state view. Under a shared Qwen3.5-4B backbone, CoMem achieves 47.23% QA correctness and 46.53%/41.24% on planning/replanning, compared with the strongest competing results of 40.27% and 33.05%/29.55%. Across systems and controlled ablations, we find that stronger information recovery does not necessarily yield better personalization: successful long-term interaction requires not only remembering user information, but resolving the right state and consistently translating it into action.

Primary area:
datasets and benchmarks
Keywords:
Long-Term Memory, Personalized Language Agents, Evolving User States, Memory Benchmarking, Personalized Planning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UbyoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LifeMemBench, a benchmark for evaluating long-term memory under evolving user states through memory QA, personalized planning, and replanning. It constructs synthetic longitudinal interactions spanning 8K–64K tokens and proposes CoMem, a state-aware memory system that separates historical evidence from current user states. Experiments show that CoMem outperforms six memory baselines on QA, planning, and replanning using a shared Qwen3.5-4B backbone.

Critical strengths
  1. The paper addresses an important problem by connecting long-term memory evaluation with downstream personalized decision-making rather than focusing solely on information retrieval.
  2. The experiments compare multiple memory systems and include dynamic-state evaluations, ablations, and statistical analyses. CoMem demonstrates consistent improvements across the three main tasks.
Critical weaknesses
  1. Limited generalization. The benchmark contains only 16 synthetic user trajectories, with downstream planning evaluated exclusively in the travel domain. This limits confidence in its applicability to more diverse users and real-world scenarios.
  2. Incomplete isolation of performance gains. CoMem uses an explicit current-state snapshot that is not provided to competing methods. Although the ablation study supports its usefulness, it remains unclear how much of the improvement comes from this additional representation rather than the overall memory architecture.
Questions for authors
  1. How well do the findings generalize beyond synthetic user trajectories and travel planning?
  2. Would CoMem's improvements remain substantial if competing methods were given comparable current-state representations?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Ubyo

We sincerely thank the reviewer for the constructive feedback. We agree that data diversity and generalization are important for evaluating LifeMemBench, and we have conducted additional experiments to address these concerns.

List of Changes in the Manuscript:

Note: All revisions are highlighted in red in the updated manuscript.

1. Persona Diversity, Human-Grounded Data, and Office Scenario Extension

  • Abstract (P1); Introduction (P2)
  • Sections 3.2–3.4 (P4–5); Sections 5.2 and 5.4 (P7–8)
  • Appendices Q.1–Q.3

2. Controlled Comparison with Shared User States

  • Section 5.5 — Ablation Studies (P8–9)
  • Appendix Q.7 — Controlled Comparison with Shared User States

Q1 About limited generalization.

A1 About limited users. We have expanded LifeMemBench from 16 to 40 user profiles, including 8 profiles and corresponding dialogue datasets constructed from real participant information, covering a wider range of ages, occupations, budgets, preferences, and Big Five personality traits. The supplementary results below cover 4 original synthetic profiles, 2 newly generated synthetic profiles, and 2 profiles based on real participants. CoMem outperforms the strongest baseline for each task by 9.69, 3.37, and 7.28 percentage points on QA, Planning, and Replanning, respectively, providing preliminary support for its effectiveness across different profile sources.

Table 1. Performance comparison on user profiles from different sources. Improvements are measured against the strongest baseline for each task.

MethodQA ↑Planning ↑Replanning ↑
Mem033.13%22.50%28.19%
A-MEM36.88%20.53%26.22%
Letta46.88%18.44%31.16%
MemBuilder46.25%24.22%31.22%
MemoryBank48.44%21.56%29.00%
CoMem58.13%27.59%38.50%

About limited downstream planning. We have introduced LifeMemBench-Office, extending downstream tasks beyond travel planning to document writing, presentation creation, task prioritization, and work plan adjustment, while retaining long-term user state evolution and the QA, Planning, and Replanning evaluation framework. CoMem achieves Planning and Replanning scores of 67.50% and 60.50%, respectively, in this setting, broadening the benchmark’s coverage of long-term memory and personalized decision-making.

Q2 About incomplete isolation of performance gains.

A2 We thank the reviewer for highlighting the contribution of explicit user state representations. To distinguish this contribution from the benefits of the overall architecture, we provided five baselines with the same current user state representations extracted by CoMem’s state maintenance module, while preserving their original memory mechanisms. State augmentation improved Replanning scores for all baselines, yet CoMem still outperformed the strongest state-augmented baseline on each task by 14.98 and 6.67 percentage points on Planning and Replanning, respectively. These results suggest that CoMem’s gains extend beyond explicit state representations to the coordinated organization of interaction evidence, state change events, and currently valid user states, together with query-conditioned state derivation and evidence-aware retrieval. The full description can be found in Appendix Q.7.

AuthorsOpenReview ↗

Kind Reminder

Dear Reviewer Ubyo,

We sincerely appreciate the time and effort you have dedicated to reviewing our work. As the discussion deadline is approaching, we would greatly appreciate it if you could kindly review our rebuttal and additional experimental results.

During the rebuttal period, we carefully addressed your concerns by expanding persona diversity, incorporating real-participant profiles and human dialogues, extending LifeMemBench to workplace scenarios, and evaluating CoMem on independent memory benchmarks. We also conducted additional experiments by incorporating comparable current-state representations into competing memory systems to better isolate the contributions of CoMem.

If you have any further questions or concerns, we would be grateful for the opportunity to address them. Moreover, if you find our responses satisfactory, we would sincerely appreciate it if you could kindly consider updating your rating.

Thank you again for your valuable feedback and consideration. We look forward to your response.

Best regards,
The Authors

Reviewer_xN9TOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces LifeMemBench, a benchmark for longitudinal personalization under evolving user preferences, constraints, and circumstances. Its construction first specifies latent user-state trajectories and then generates chronological interactions, enabling evaluation of current and historical state recovery. The benchmark connects memory QA with personalized travel planning and replanning over four cumulative interaction scales involving the same 16 users. The paper also introduces CoMem, which maintains interaction evidence, attributed events, and a derived current-state snapshot. Experiments compare CoMem with six memory systems using a shared Qwen3.5-4B backbone, supplemented by component ablations, constraint-level analyses, and user-level statistical tests.

Critical strengths
  1. The benchmark addresses a meaningful problem: remembering historical information does not necessarily establish which information remains valid for current decisions. Connecting memory evaluation to planning and replanning is useful.

  2. Explicit state trajectories support controlled temporal evaluation, while the no-memory control and user-level statistical analyses are useful experimental choices.

Critical weaknesses
  1. The realism of the synthetic user trajectories is insufficiently validated. The benchmark constructs latent states and then generates dialogues to express those states. This provides convenient supervision, but does not establish that the resulting updates, contradictions, and preference expressions resemble real longitudinal interactions. The paper does not report a convincing human validation study or evaluation on real user histories. Consequently, performance may reflect adaptation to the generation pipeline rather than robust personalization.

  2. The construction creates a risk of alignment between benchmark assumptions and CoMem’s design. The benchmark organizes user evolution through explicit state-changing events, while CoMem extracts events and maintains a current-state snapshot. This alignment is reasonable, but makes it important to test whether the advantage survives alternative dialogue generators, less structured interactions, and different state representations. The current experiments do not sufficiently separate general memory improvements from compatibility with the benchmark’s construction assumptions.

  3. The effective user diversity is small. All four scales extend the same 16 synthetic users. Increasing dialogue length and task count therefore does not provide independent evidence of generalization across users. Evaluation is additionally restricted to travel planning and one generation backbone. User-level bootstrap intervals quantify uncertainty within these trajectories but cannot establish robustness to new populations or domains.

  4. The synthetic supervision requires stronger semantic validation. Evidence appearing before a query does not establish that the dialogue uniquely supports the assigned latent state, especially for implicit preferences and conditional companion requirements.

Questions for authors
  1. What human validation or independent evaluation establishes that the synthetic dialogues and latent-state labels reflect realistic user behavior?

  2. Does CoMem retain its advantage on independently generated or human-authored trajectories under matched memory-token budgets?

  3. Why are the planning results inconsistent with the stated scoring formula? Equation (32) defines the planning score as the mean of HCR, PC, and Feasibility. CoMem’s values in Table 31 imply 0.757, whereas its reported overall score is 0.465. Table 4 also labels results as 64K while reporting values different from the 64K entries in Tables 28 and 29. These discrepancies undermine confidence in the main quantitative claims.

3 comments
AuthorsOpenReview ↗

Response to Reviewer xN9T(1/2)

We sincerely thank the reviewer for the thoughtful and constructive feedback. We appreciate your recognition of LifeMemBench’s strengths and have carefully considered the concerns and suggestions raised in the review. Accordingly, we have revised the manuscript and conducted additional experiments to strengthen our findings. (We also refer the reviewer to our general response for an overview of the major revisions.)

List of Changes in the Manuscript:

Note: All revisions are highlighted in red in the updated manuscript.

1. Persona Diversity, Human-Grounded Data, and Office Scenario Extension

  • Abstract (P1); Introduction (P2)
  • Sections 3.2–3.4 (P4–5); Sections 5.2 and 5.4 (P7–8)
  • Appendices Q.1, Q.2

2. Generalization to External Benchmarks (LongMemEval and LoCoMo)

  • Section 5.2 — Overall Performance (P7)
  • Appendix Q.6 — Generalization to External Memory Benchmarks

3. Evaluation Metric Consistency and Clarification

  • Appendix H — Updated Overall Planning metric definition (Eq. 32)
  • Appendix J.4 — Clarification of Overall scores relative to HCR, PC, and Feasibility (Table 31)

4. Token Consumption and Performance–Cost Analysis

  • Section 5.5 — Ablation Studies (P8–9)
  • Appendix Q.4 — Token Consumption and Performance–Cost Trade-off Q1 About human validation study.

A1 We thank the reviewer for raising concerns about data realism. We have added user profiles and corresponding dialogue datasets based on information from 8 real participants and are collecting additional questionnaire responses to broaden participant coverage. On these participant-based data, CoMem outperforms the strongest baseline for each task by 7.56, 4.71, and 8.14 percentage points on QA, Planning, and Replanning, respectively, providing preliminary evidence of its effectiveness with profiles grounded in real user information.

Q2 About compatibility.

A2 We thank the reviewer for highlighting the potential structural alignment between LifeMemBench’s construction and CoMem’s design. To examine whether CoMem’s gains depend on this construction pipeline, we conducted additional experiments on LOCOMO and LongMemEval, neither of which uses LifeMemBench’s predefined latent user-state trajectory generation process. With CoMem, Mistral-7B’s score on LOCOMO increased from 18.70% to 32.17% (+13.47 percentage points), while the Llama-3.1 configuration’s score on LongMemEval increased from 71.00% to 73.15% (+2.15 percentage points). These results support CoMem’s effectiveness across different data construction processes and mitigate the concern that its gains arise entirely from structural alignment with LifeMemBench.

We also provided five baselines with exactly the same current user-state representations as CoMem, while preserving their original memory mechanisms. CoMem still outperformed the strongest state-augmented baseline for each task by 14.98 and 6.67 percentage points on Planning and Replanning, respectively, suggesting that explicit state information alone is insufficient to reproduce its performance. These experiments examine dependence on the construction pipeline and the contribution of explicit state representations, but do not directly test alternative dialogue generators or different state representations. We have clarified the scope of these experiments in the revised manuscript.

Q3 About limited users.

A3 We have expanded LifeMemBench from 16 to 40 user profiles. A key addition is 8 profiles and corresponding dialogue datasets constructed from information provided by real participants, grounding the benchmark in real user characteristics and strengthening its relevance to real-world scenarios. The expanded set also covers a wider range of ages, occupations, budgets, personal preferences, and Big Five personality traits. We plan to further expand the user pool in future work.

The updated results are shown below:

Table 1. Performance comparison on user profiles from different sources. Improvements are measured against the strongest baseline for each task.

MethodQA ↑Planning ↑Replanning ↑
Mem033.13%22.50%28.19%
A-MEM36.88%20.53%26.22%
Letta46.88%18.44%31.16%
MemBuilder46.25%24.22%31.22%
MemoryBank48.44%21.56%29.00%
CoMem58.13%27.59%38.50%
Improvement+9.69 pp+3.37 pp+7.28 pp
AuthorsOpenReview ↗

Response to Reviewer xN9T(2/2)

Q4 About semantic validation.

A4 We thank the reviewer for distinguishing temporal validity from semantic support. We agree that evidence preceding a query is necessary but insufficient to establish that the dialogue unambiguously supports the assigned user state. We will conduct additional human semantic validation of implicit preferences and conditional companion requirements, checking whether each annotation has sufficient dialogue evidence, correct role attribution, and fully preserved applicability conditions. Samples with ambiguous interpretations or insufficient support will be corrected or excluded from evaluation. We will also report the validation criteria, inter-annotator agreement, and representative examples to strengthen the semantic reliability of the supervision.

Q5 About token budgets.

A5 We thank the reviewer for highlighting computational costs. We measured total token consumption for each method in the evaluation on 8 additional user profiles, with results shown below. CoMem outperforms Mem0, A-MEM, and MemBuilder on QA, Planning, and Replanning while reducing total token consumption by approximately 74.9%, 87.5%, and 97.3%, respectively. Compared with the more lightweight MemoryBank, CoMem consumes approximately 79.2% more tokens but improves scores on the three tasks by 9.69, 6.03, and 9.50 percentage points, respectively. Compared with Letta, CoMem likewise achieves better task performance with higher token consumption.

Table 2. Total token consumption and task performance across memory systems on eight user profiles.

MethodTotal TokenQAPlanningReplanning
Letta857,82746.88%18.44%31.16%
MemoryBank1,349,04748.44%21.56%29.00%
CoMem2,416,96058.13%27.59%38.50%
Mem09,612,62333.13%22.50%28.19%
A-MEM19,284,30336.88%20.53%26.22%
MemBuilder88,471,61346.25%24.22%31.22%

Q6 About difference with the stated scoring formula.

A6 We sincerely thank the reviewer for identifying these inconsistencies.

After carefully reviewing our evaluation implementation, we found that Eq. (32) and (35) did not accurately describe how the Overall scores were computed. Specifically, the reported Planning and Replanning Overall scores are obtained by averaging task-level holistic quality scores directly assigned by an LLM-based evaluator. The evaluator considers multiple aspects of planning quality, while the corresponding component metrics are reported separately for diagnostic analysis. The original equations incorrectly described the Overall score as an arithmetic mean of these component metrics.

We apologize for this inconsistency and have corrected the metric definitions in the revised manuscript to accurately reflect the evaluation implementation. We will also release the evaluation code and scoring protocol to support reproducibility.

Regarding Table 4 and Tables 28–29, the apparent discrepancy arises from how results across context lengths are aggregated. Tables 28–29 report results separately for each context length, whereas Table 4 reports the task-count-weighted aggregate across all lengths. We have clarified this distinction in the revised table caption.

Thanks. The response addressed many of my concerns. However, considering the overall quality of the paper, I have decided to maintain my original score.

Reviewer_TM4xOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The work introduces LifeMemBench, a benchmark for longitudinal personalization under evolving user states, spanning from 8k to 64k. The benchmark evaluates personalized travel planning and replanning. It further proposed a lightweight memory system CoMem to refine the problem.

Critical strengths
  1. LifeMemBench uses predefined latent user-state traj. that enables precise query-time state supervision.
  2. From 8K to 64K, the bench controls the variance to make a comparable evaluation.
Critical weaknesses
  1. LifeMemBench covers only 16 users with limited area and is constructed through llm generation. It is not enough to represent the value on real application.
  2. The comparision values have minor differences which may influence by evaluation confidence.
  3. The comparison lacks of closed-formed api model performance.
Questions for authors

what about the performance on codex, cc and any other harness with the frontier models.

2 comments
AuthorsOpenReview ↗

Response to Reviewer TM4x

We appreciate your overall positive assessment of our contributions and are grateful for your suggestion. We have already updated the new revision based on your suggestions (we also recommend the reviewer check the global response).

List of Changes in the Manuscript:

Note: All revisions are highlighted in red in the updated manuscript.

1. Persona Diversity, Human-Grounded Data, and Office Scenario Extension

  • Abstract (P1); Introduction (P2)
  • Sections 3.2–3.4 (P4–5); Sections 5.2 and 5.4 (P7–8)
  • Appendices Q.1–Q.3

2. Comparison with Frontier Agents (Codex vs. Codex + CoMem)

  • Section 5.4 — Personalized Decisions (P8)
  • Appendix Q.5 — Experimental Setup, Performance Comparison, and Analysis

Q1 About limited users.

A1 We have expanded LifeMemBench from 16 to 40 user profiles. A key addition is 8 profiles and corresponding dialogue datasets constructed from information provided by real participants, grounding the benchmark in real user characteristics and strengthening its relevance to real-world scenarios. The expanded set also covers a wider range of ages, occupations, budgets, personal preferences, and Big Five personality traits. We plan to further expand the user pool in future work.

The updated results are presented below, covering eight representative profiles: four original synthetic profiles, two newly generated synthetic profiles, and two profiles based on real participants.

Table 1. Performance comparison on user profiles from different sources. Improvements are measured against the strongest baseline for each task.

MethodQA ↑Planning ↑Replanning ↑
Mem033.13%22.50%28.19%
A-MEM36.88%20.53%26.22%
Letta46.88%18.44%31.16%
MemBuilder46.25%24.22%31.22%
MemoryBank48.44%21.56%29.00%
CoMem58.13%27.59%38.50%
Improvement+9.69 pp+3.37 pp+7.28 pp

Q2 About the Reliability of Performance Differences.

A2 We thank the reviewer for raising concerns about evaluation reliability. Since model generation and evaluation may introduce variability, scores can fluctuate across repeated runs. To mitigate the influence of such fluctuations, we repeated each experiment 10 times and reported the average results. On the original 16 synthetic profiles, using Qwen3.5-4B as the generation model, CoMem outperforms the strongest baseline for each task by 6.93, 13.25, and 11.71 percentage points on QA, Planning, and Replanning, respectively, demonstrating consistent performance advantages across all three tasks.

Q3 About Comparison of Closed-formed Api.

A3 We thank the reviewer for suggesting comparisons with frontier agents such as Codex. Using Codex CLI v0.169.0-alpha.11.1 with gpt-6.1-sol, we compared Codex with and without CoMem under the same chronological task order and evaluation protocol. Integrating CoMem increased the evaluation score from 68.75% to 80.00%, an absolute gain of 11.25 percentage points. This provides preliminary evidence that CoMem’s explicit user state management can improve frontier agents’ performance on long-term personalization tasks without modifying the underlying reasoning model. We plan to extend these comparisons to additional frontier models and agents, including Claude.

AuthorsOpenReview ↗

Kind Reminder

Dear Reviewer TM4x,

We sincerely appreciate the time and effort you have dedicated to reviewing our work. As the discussion deadline is approaching, we would greatly appreciate it if you could kindly review our rebuttal and additional experimental results.

During the rebuttal period, we carefully addressed your concerns by expanding persona diversity, incorporating real-participant profiles and human dialogues, conducting repeated evaluations to examine result reliability, and introducing comparisons with Codex and Codex + CoMem.

If you have any further questions or concerns, we would be grateful for the opportunity to address them. Moreover, if you find our responses satisfactory, we would sincerely appreciate it if you could kindly consider updating your rating.

Thank you again for your valuable feedback and consideration. We look forward to your response.

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers and Summary of Revisions

We sincerely thank all three reviewers (TM4x, Ubyo, and xN9T) for their thoughtful and constructive feedback. We appreciate their recognition of the importance of evaluating long-term memory beyond information retrieval, particularly through evolving user states and downstream personalized decision-making.

We have carefully considered the reviewers’ concerns regarding data diversity, generalization, evaluation reliability, methodological fairness, and computational efficiency. In response, we have conducted extensive additional experiments and substantially revised the manuscript to strengthen our findings.

Summary of Major Revisions

1. Expanded Persona Diversity and Human-Grounded Data (Sections 3.2–3.4, 5.2, 5.4; Appendices Q.1–Q.2).

We expanded LifeMemBench from 16 to 40 user profiles, including 8 profiles and corresponding dialogue datasets based on real participant information. Additional experiments across eight representative profiles show that CoMem outperforms the strongest baselines by 9.69, 3.37, and 7.28 percentage points on QA, Planning, and Replanning, respectively.

2. Extended to Workplace Scenarios (Sections 3.3–3.4, 5.4; Appendix Q.3).

We introduced LifeMemBench-Office, extending personalized decision-making beyond travel planning to document writing, presentation creation, task prioritization, and work plan adjustment. CoMem achieves 67.50% on Planning and 60.50% on Replanning in this setting.

3. Evaluated Generalization on External Benchmarks (Section 5.2; Appendix Q.6).

We evaluated CoMem on LoCoMo and LongMemEval, which use different data construction procedures. CoMem improves performance by 13.47 and 2.15 percentage points, respectively, providing additional evidence of its generalization beyond LifeMemBench.

4. Added Controlled Comparisons with Shared User States (Section 5.5; Appendix Q.7).

We augmented five baseline memory systems with the same current user-state representations as CoMem. CoMem still outperforms the strongest state-augmented baseline by 14.98 and 6.67 percentage points on Planning and Replanning, respectively, demonstrating benefits beyond explicit state representations alone.

5. Compared with Frontier Agents (Section 5.4; Appendix Q.5).

We compared Codex with and without CoMem under the same evaluation protocol. Integrating CoMem improves the evaluation score from 68.75% to 80.00% (+11.25 pp), providing preliminary evidence of its effectiveness with frontier agents.

6. Improved Evaluation Reliability and Computational Efficiency Analysis (Sections 5.2, 5.5; Appendix Q.4).

We conducted 10 repeated runs to mitigate evaluation variability and added token consumption comparisons. CoMem achieves better performance than Mem0, A-MEM, and MemBuilder while reducing total token consumption by approximately 74.9%, 87.5%, and 97.3%, respectively.

7. Clarified Evaluation Metrics and Reproducibility (Appendices H, J.4).

We corrected inconsistencies between the Overall metric definitions and their implementation, clarified the aggregation of results across context lengths, and will release the evaluation code and scoring protocol.

Reaffirming Our Core Contributions

We would like to emphasize that LifeMemBench goes beyond traditional memory QA benchmarks by evaluating whether agents can recover evolving user states and translate them into reliable personalized decisions.

CoMem complements this benchmark with a lightweight memory architecture that coordinates interaction evidence, state-change events, and currently valid user states.

Our additional experiments further support the central finding that accurate information retrieval alone is insufficient for long-term personalization; effective memory systems must also maintain evolving user states and support downstream decision-making.

We sincerely appreciate the reviewers’ valuable feedback, which has helped us improve the quality and rigor of our work. All revisions are highlighted in red in the updated manuscript. We hope these revisions address the reviewers’ concerns and would be happy to provide further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.