- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces MemEnv, a dynamic benchmark for evaluating long-term conversational memory across six graph-defined dimensions: Summary, Update, Reasoning, Isolation, Preference, and Transfer. The benchmark instantiates graph structures, converts them into multi-session interactions, and asks graph-derived final questions. It also introduces MemEnv+, where multiple memory dimensions are mixed in the same history. Experiments across LLMs and memory agents show uneven memory abilities and trade-offs that aggregate scores can obscure.
Critical strengths
The paper addresses an important evaluation gap: long-term memory agents should not only recall facts, but also update them, isolate entities, infer preferences, and transfer prior knowledge. The dimension-wise design is useful because it exposes trade-offs hidden by a single average score. The MemEnv+ and off-policy analyses also raise relevant questions about whether existing evaluations reflect interactive deployment.
Critical weaknesses
The main concern is external validity. The benchmark is generated from predefined graph templates and a user simulator, but the paper does not establish that success transfers to realistic long-term memory use. The mapping from graph operations to conversational phenomena remains hand-designed, and stronger human validation of conversation quality, query ambiguity, and answer derivability is needed. Measurement robustness is also unclear: model rankings may depend on surface realization, paraphrasing, distractor style, simulator policy, or answer-format assumptions. Finally, the novelty is somewhat incremental relative to recent dynamic memory benchmarks, and positioning against EvolIF and AMemGym should be sharper.
Questions for authors
- How much human validation was performed on generated conversations and final queries?
- Are model rankings stable under paraphrased realizations of the same graph states?
- Can the authors provide a manually curated realistic subset to calibrate MemEnv scores against human judgment?
6 comments
Response (II)
W2&Q1: human validation
Response:
We apologize for the ambiguity in our previous human evaluation, which combined dialogue naturalness and QA validity into a single score. To clarify the interpretation of the results, we conducted additional human evaluations to assess these aspects separately. We ensure dialogue quality through two complementary mechanisms: construction-time controls and human evaluation.
- Dialogue reliability is enforced through structured construction and verification.
- The dependency graph specifies the underlying facts, relations, and evidence dependencies. When converted into dialogue states, each state explicitly defines which information is required to be exposed and which information must remain unexposed, constraining the subsequent generation of user utterances.
- Utterances synthesized from these states undergo programmatic verification of required and prohibited core information. Only utterances that pass the checks are retained. Otherwise, they are regenerated, ensuring a 100% pass rate.
- Human evaluation validates dialogue naturalness and contextual non-interference.
- We randomly sampled 100 scripted user utterances, each independently evaluated by two annotators using a five-point scale.
- Regarding “conversation quality,” we asked annotators to assess naturalness, defined as linguistic fluency, semantic clarity, and resemblance to natural human dialogue.
- The predefined user utterances used to expose key facts achieved an average score of 4.33/5, with 82% within-one-point inter-annotator agreement.
- Regarding “query ambiguity and answer derivability,” we asked annotators to assess contextual non-interference, defined as the extent to which topic-transition utterances preserve previously established key facts without introducing contradictions, modifications, or ambiguity.
- The average score was 4.78/5, with 91% within-one-point inter-annotator agreement.
- 2% of the utterances scored below 3, indicating potential interference. We re-examined these conversations and found that both cases made previously established information more specific without changing the information being tested or compromising the validity of the original QA pairs.
- Regarding “conversation quality,” we asked annotators to assess naturalness, defined as linguistic fluency, semantic clarity, and resemblance to natural human dialogue.
- We randomly sampled 100 user utterances generated during dynamic interactions, each independently evaluated by two annotators on a five-point scale assessing overall conversational quality, including naturalness and non-interference with key facts.
- The utterances achieved an average score of 4.65/5, with 97% within-one-point inter-annotator agreement. Although a small number of utterances exhibited minor conversational deviations, none interfered with the key facts required for evaluation.
- We randomly sampled 100 scripted user utterances, each independently evaluated by two annotators using a five-point scale.
- We also evaluated the reliability of our automatic evaluation metrics against human validation.
- We sampled 100 model responses for each of the four adopted metrics and provided annotators with the corresponding ground-truth answers together to assess whether scores were reasonable.
- Agreement rates were 90.0% for Recall, 94.0% for LCS, 97.0% for LLM-as-a-Judge, and 100.0% for Accuracy, yielding an overall agreement of 95.25%.
- These results indicate strong consistency between automatic scoring and human judgment, supporting the reliability of our evaluation metrics.
Response (III)
W3&Q2: measurement robustness
Response:
We would like to address the concern on measurement robustness as follows:
- Diversity is incorporated into benchmark design. We use diverse persona and topic seeds to promote diversity in conversational content (sec 4.2), and a pool of five LLMs from different model families to promote diversity in surface forms for constructing key utterances with distractors (sec 4.2).
- We examined the influence of the user simulator model. The sensitivity analysis in Appendix F.5 shows broadly consistent system rankings and dimension-level performance patterns across different user simulators.
- Following your suggestion, we conducted an additional experiment on ranking sensitivity to linguistic realization.
- We sampled 30 graphs (five per dimension) and generated three linguistic realizations of each graph using different LLMs, while keeping the underlying task states and evaluation criteria fixed.
- We evaluated Qwen3.6-27B, Qwen3.5-9B, and Gemma3-12B-IT with instance-level pairwise ranking agreement. For each graph, we checked whether each model pair’s relative ordering (win, loss, or tie) remained unchanged across linguistic realizations. The results are:
Dimension Ranking Agreement (%) Summary 24.44 Update 68.89 Reasoning 55.56 Isolation 60.00 Preference 64.44 Transfer 73.33 - The average instance-level ranking agreement is 57.78%, indicating that relative model performance exhibits sensitivity to linguistic realizations, and highlighting the importance of using multiple LLMs to generate diverse scripted user utterances.
- We evaluated the reliability of our automatic evaluation metrics against human validation.
- We sampled 100 model responses for each of the four adopted metrics and provided annotators with the corresponding ground-truth answers together to assess whether scores were reasonable.
- Agreement rates were 90.0% for Recall, 94.0% for LCS, 97.0% for LLM-as-a-Judge, and 100.0% for Accuracy, yielding an overall agreement of 95.25%.
- These results indicate strong consistency between automatic scoring and human judgment, supporting the reliability of our evaluation metrics.
W4: novelty compared to recent dynamic benchmarks
Response:
MemEnv differs from prior dynamic benchmarks in two key respects.
- MemEnv covers a broader range of memory capabilities.
- MemEnv systematically evaluates six complementary memory dimensions.
- In comparison, EvolIF primarily focuses on multi-turn instruction following, while AMemGym emphasizes retrieval and update capabilities for tracking evolving user states.
- Long-term memory encompasses a broader range of capabilities, and evaluating them under a unified protocol provides a more comprehensive assessment.
- MemEnv uses graphs to distinguish memory dimensions and manage information.
- MemEnv uses graph structures to explicitly define and distinguish memory dimensions through their underlying facts, relations, and evidence dependencies.
- EvolIF and AMemGym rely on a single state-tracking mechanism, without explicitly modeling complex relationships among graph nodes.
- These structures govern information exposure and state transitions throughout dynamic interactions, enabling controlled information management and supporting dialogue quality.
- Our experimental results further demonstrate the need for multidimensional memory evaluation.
- Current models exhibit substantial performance differences across memory dimensions (sec 5.1), and neither models nor memory agents consistently perform well across all six dimensions (sec 5.2).
- MemEnv reveals dimension-specific strengths and weaknesses that aggregate scores may obscure, helping guide the development of more comprehensive memory capabilities.
We will incorporate the additional analyses into the Appendix of the revised paper. Please let us know if you have any further questions, as we are happy to continue the discussion.
Response (I)
Thank you for your feedback on our paper. We will try to address your concerns as follows:
W1&Q3: realistic concern
Response:
References
[1]Jiang, Bowen, et al. "Personamem-v2: Towards personalized intelligence via learning implicit user personas and agentic memory." arXiv preprint arXiv:2512.06688 (2025).
[2]Hu, Yuanzhe, Yu Wang, and Julian McAuley. "Evaluating memory in llm agents via incremental multi-turn interactions." International Conference on Learning Representations. Vol. 2026. 2026.
[3]Jiang, Bowen, et al. "Know me, respond to me: Benchmarking llms for dynamic user profiling and personalized responses at scale." arXiv preprint arXiv:2504.14225 (2025).
[4]Wu, Di, et al. "Longmemeval: Benchmarking chat assistants on long-term interactive memory." arXiv preprint arXiv:2410.10813 (2024).
[5]Chatterji, Aaron, et al. "How people use chatgpt." No. w34255. National Bureau of Economic Research, 2025.