acceptodds
Under review as a conference paper at ICLR 2027

MemEnv: Evaluating Multi-dimensional Long-Term Memory via Graph-Grounded Dynamic Interaction

Abstract

Long-term memory is essential for LLMs to function as coherent and personalized assistants. Yet, current evaluations often rely on static histories that fail to capture real-world conversational dynamics or only assess narrow memory traits. We introduce MemEnv, a dynamic benchmark designed to assess multiple dimensions of long-term memory across sessions. In MemEnv, each memory dimension is grounded in a scalable, predefined graph. A user simulator leverages this structure to orchestrate conversations, seamlessly interleaving predefined utterances with open-ended dialogue before querying the model. To better mimic the complicated nature of real-world interactions, we further present MemEnv+, which integrates multiple memory dimensions into individual conversations. Extensive experiments reveal that current LLMs and memory agents still lack well-rounded memory capabilities. While memory mechanisms boost overall performance, they introduce clear trade-offs across different memory dimensions. MemEnv+ exacerbates these challenges, especially when models are required to synthesize dispersed evidence. Furthermore, in-conversation backbone-model switching reveals a gap between on-policy and off-policy settings, with off-policy achieving larger gains on information-focused tasks regardless of the preceding model's strength. These findings highlight the need for versatile memory systems that can balance different dimensions and perform reliably in dynamic environments.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, Long-term Memory, Memory Evaluation, Memory Agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_C9sxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MemEnv, a dynamic benchmark for evaluating long-term conversational memory across six graph-defined dimensions: Summary, Update, Reasoning, Isolation, Preference, and Transfer. The benchmark instantiates graph structures, converts them into multi-session interactions, and asks graph-derived final questions. It also introduces MemEnv+, where multiple memory dimensions are mixed in the same history. Experiments across LLMs and memory agents show uneven memory abilities and trade-offs that aggregate scores can obscure.

Critical strengths

The paper addresses an important evaluation gap: long-term memory agents should not only recall facts, but also update them, isolate entities, infer preferences, and transfer prior knowledge. The dimension-wise design is useful because it exposes trade-offs hidden by a single average score. The MemEnv+ and off-policy analyses also raise relevant questions about whether existing evaluations reflect interactive deployment.

Critical weaknesses

The main concern is external validity. The benchmark is generated from predefined graph templates and a user simulator, but the paper does not establish that success transfers to realistic long-term memory use. The mapping from graph operations to conversational phenomena remains hand-designed, and stronger human validation of conversation quality, query ambiguity, and answer derivability is needed. Measurement robustness is also unclear: model rankings may depend on surface realization, paraphrasing, distractor style, simulator policy, or answer-format assumptions. Finally, the novelty is somewhat incremental relative to recent dynamic memory benchmarks, and positioning against EvolIF and AMemGym should be sharper.

Questions for authors
  1. How much human validation was performed on generated conversations and final queries?
  2. Are model rankings stable under paraphrased realizations of the same graph states?
  3. Can the authors provide a manually curated realistic subset to calibrate MemEnv scores against human judgment?
6 comments
AuthorsOpenReview ↗

Response (I)

Thank you for your feedback on our paper. We will try to address your concerns as follows:

W1&Q3: realistic concern

Response:

  • The correspondence between our MemEnv and real human interaction patterns is fundamentally grounded in both our research focus and methodological design.
    • We focus on human–AI conversations in which users seek help and suggestions from LLMs, reflecting everyday uses of AI tools.
    • MemEnv is grounded in diverse user profiles and topics in daily activities. We sampled persona profiles from PersonaMem-v2[1] and collected topics from MemoryAgentBench[2], PersonaMem[3], LongMemEval[4], and OpenAI[5], as seed data for constructing daily conversations.
    • Each profile is further paired with different topics, yielding 50 persona–topic pairings. This allows MemEnv+ capture the multifaceted nature of individual users and their needs across multiple topics. For example, Adama Njeri Kamau is both a junior developer building research software (topic: computer programming) and someone who took analytical chemistry and works with lab instruments (topic: chemistry).
    • We incorporate a user simulator to support more natural and fluent conversations for on-policy evaluation.
  • We conducted human evaluations to verify the naturalness of generated conversations. See more details in responses to W2&Q1.

References

[1]Jiang, Bowen, et al. "Personamem-v2: Towards personalized intelligence via learning implicit user personas and agentic memory." arXiv preprint arXiv:2512.06688 (2025).

[2]Hu, Yuanzhe, Yu Wang, and Julian McAuley. "Evaluating memory in llm agents via incremental multi-turn interactions." International Conference on Learning Representations. Vol. 2026. 2026.

[3]Jiang, Bowen, et al. "Know me, respond to me: Benchmarking llms for dynamic user profiling and personalized responses at scale." arXiv preprint arXiv:2504.14225 (2025).

[4]Wu, Di, et al. "Longmemeval: Benchmarking chat assistants on long-term interactive memory." arXiv preprint arXiv:2410.10813 (2024).

[5]Chatterji, Aaron, et al. "How people use chatgpt." No. w34255. National Bureau of Economic Research, 2025.

AuthorsOpenReview ↗

Response (II)

W2&Q1: human validation

Response:

We apologize for the ambiguity in our previous human evaluation, which combined dialogue naturalness and QA validity into a single score. To clarify the interpretation of the results, we conducted additional human evaluations to assess these aspects separately. We ensure dialogue quality through two complementary mechanisms: construction-time controls and human evaluation.

  • Dialogue reliability is enforced through structured construction and verification.
    • The dependency graph specifies the underlying facts, relations, and evidence dependencies. When converted into dialogue states, each state explicitly defines which information is required to be exposed and which information must remain unexposed, constraining the subsequent generation of user utterances.
    • Utterances synthesized from these states undergo programmatic verification of required and prohibited core information. Only utterances that pass the checks are retained. Otherwise, they are regenerated, ensuring a 100% pass rate.
  • Human evaluation validates dialogue naturalness and contextual non-interference.
    • We randomly sampled 100 scripted user utterances, each independently evaluated by two annotators using a five-point scale.
      • Regarding “conversation quality,” we asked annotators to assess naturalness, defined as linguistic fluency, semantic clarity, and resemblance to natural human dialogue.
        • The predefined user utterances used to expose key facts achieved an average score of 4.33/5, with 82% within-one-point inter-annotator agreement.
      • Regarding “query ambiguity and answer derivability,” we asked annotators to assess contextual non-interference, defined as the extent to which topic-transition utterances preserve previously established key facts without introducing contradictions, modifications, or ambiguity.
        • The average score was 4.78/5, with 91% within-one-point inter-annotator agreement.
        • 2% of the utterances scored below 3, indicating potential interference. We re-examined these conversations and found that both cases made previously established information more specific without changing the information being tested or compromising the validity of the original QA pairs.
    • We randomly sampled 100 user utterances generated during dynamic interactions, each independently evaluated by two annotators on a five-point scale assessing overall conversational quality, including naturalness and non-interference with key facts.
      • The utterances achieved an average score of 4.65/5, with 97% within-one-point inter-annotator agreement. Although a small number of utterances exhibited minor conversational deviations, none interfered with the key facts required for evaluation.
  • We also evaluated the reliability of our automatic evaluation metrics against human validation.
    • We sampled 100 model responses for each of the four adopted metrics and provided annotators with the corresponding ground-truth answers together to assess whether scores were reasonable.
    • Agreement rates were 90.0% for Recall, 94.0% for LCS, 97.0% for LLM-as-a-Judge, and 100.0% for Accuracy, yielding an overall agreement of 95.25%.
    • These results indicate strong consistency between automatic scoring and human judgment, supporting the reliability of our evaluation metrics.
AuthorsOpenReview ↗

Response (III)

W3&Q2: measurement robustness

Response:

We would like to address the concern on measurement robustness as follows:

  • Diversity is incorporated into benchmark design. We use diverse persona and topic seeds to promote diversity in conversational content (sec 4.2), and a pool of five LLMs from different model families to promote diversity in surface forms for constructing key utterances with distractors (sec 4.2).
  • We examined the influence of the user simulator model. The sensitivity analysis in Appendix F.5 shows broadly consistent system rankings and dimension-level performance patterns across different user simulators.
  • Following your suggestion, we conducted an additional experiment on ranking sensitivity to linguistic realization.
    • We sampled 30 graphs (five per dimension) and generated three linguistic realizations of each graph using different LLMs, while keeping the underlying task states and evaluation criteria fixed.
    • We evaluated Qwen3.6-27B, Qwen3.5-9B, and Gemma3-12B-IT with instance-level pairwise ranking agreement. For each graph, we checked whether each model pair’s relative ordering (win, loss, or tie) remained unchanged across linguistic realizations. The results are:
      DimensionRanking Agreement (%)
      Summary24.44
      Update68.89
      Reasoning55.56
      Isolation60.00
      Preference64.44
      Transfer73.33
    • The average instance-level ranking agreement is 57.78%, indicating that relative model performance exhibits sensitivity to linguistic realizations, and highlighting the importance of using multiple LLMs to generate diverse scripted user utterances.
  • We evaluated the reliability of our automatic evaluation metrics against human validation.
    • We sampled 100 model responses for each of the four adopted metrics and provided annotators with the corresponding ground-truth answers together to assess whether scores were reasonable.
    • Agreement rates were 90.0% for Recall, 94.0% for LCS, 97.0% for LLM-as-a-Judge, and 100.0% for Accuracy, yielding an overall agreement of 95.25%.
    • These results indicate strong consistency between automatic scoring and human judgment, supporting the reliability of our evaluation metrics.

W4: novelty compared to recent dynamic benchmarks

Response:

MemEnv differs from prior dynamic benchmarks in two key respects.

  • MemEnv covers a broader range of memory capabilities.
    • MemEnv systematically evaluates six complementary memory dimensions.
    • In comparison, EvolIF primarily focuses on multi-turn instruction following, while AMemGym emphasizes retrieval and update capabilities for tracking evolving user states.
    • Long-term memory encompasses a broader range of capabilities, and evaluating them under a unified protocol provides a more comprehensive assessment.
  • MemEnv uses graphs to distinguish memory dimensions and manage information.
    • MemEnv uses graph structures to explicitly define and distinguish memory dimensions through their underlying facts, relations, and evidence dependencies.
    • EvolIF and AMemGym rely on a single state-tracking mechanism, without explicitly modeling complex relationships among graph nodes.
    • These structures govern information exposure and state transitions throughout dynamic interactions, enabling controlled information management and supporting dialogue quality.
  • Our experimental results further demonstrate the need for multidimensional memory evaluation.
    • Current models exhibit substantial performance differences across memory dimensions (sec 5.1), and neither models nor memory agents consistently perform well across all six dimensions (sec 5.2).
    • MemEnv reveals dimension-specific strengths and weaknesses that aggregate scores may obscure, helping guide the development of more comprehensive memory capabilities.

We will incorporate the additional analyses into the Appendix of the revised paper. Please let us know if you have any further questions, as we are happy to continue the discussion.

Reviewer_b5V6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MemEnv, a dynamic graph-grounded benchmark for evaluating long-term conversational memory in LLMs and memory agents. The authors organize memory assessment around complementary capability dimensions and instantiates each as a graph that specifies what information should be exposed and how it depends on other information. A dependency-preserving multi-session trajectory is then realized through a hybrid user simulator that interleaves predefined state-bearing utterances with adaptive dialogue, while queries and reference answers are fixed before interaction to keep evaluation traceable. The authors then introduce MemEnv+, mixing multiple memory demands within a shared history under different interleaving strategies. Experiments across proprietary and open-weight LLMs and memory agents show that stronger backbones improve aggregate scores but gains remain uneven across memory capabilities, and memory mechanisms often trade one ability for another. MemEnv+ further lowers average performance, particularly for integration-heavy memory.

Critical strengths
  1. This paper proposes MemEnv, a systematic and traceable multi-dimensional memory benchmark whose graph-grounded design separates memory structure from linguistic realization and fixes queries and answers before interaction.

  2. MemEnv+ combines multiple memory dimensions in a shared history under different interleaving strategies to test evidence integration beyond single-dimension evaluation.

Critical weaknesses
  1. The benchmark's construct validity is not fully established, since dimensions such as Reasoning and Transfer heavily involve reasoning and adaptation, making it unclear whether low scores reflect memory failure or broader capability limitations.

  2. The evaluation relies on synthetic, LLM-generated conversations and a single user simulator, so it remains unclear how well the reported rankings and dimension-level trade-offs generalize to real user interactions.

Questions for authors
  1. Could the authors provide evidence that separates memory-specific failures from reasoning or instruction-following failures, for example through reasoning-only or no-memory control conditions?

  2. How stable are the dimension-level conclusions under different user simulators, top-k budgets, or graph sizes, and would a brief sensitivity analysis change any of the main claims?

  3. Since MemEnv+ uses only three interleaving strategies, can the authors clarify whether the observed degradation is driven mainly by memory-mode coexistence itself or by particular evidence distributions?

5 comments
AuthorsOpenReview ↗

Response (I)

Thank you for your insightful review and valuable suggestions. We would like to address your concerns in detail below.

W1&Q1: separates memory-specific failures

Response:

  • We added two control settings, each evaluated on 50 samples across 4 models, to help distinguish memory-related difficulties from underlying task-capability limitations.
    • Reasoning-only provides all required evidence alongside the target question.
    • Transfer-only provides the source demonstration and target problem directly, without preceding interactions.
      ModelReasoningReasoning-onlyTransferTransfer-only
      Gemini-3.8-Flash100.0100.051.939.8
      Qwen3.5-9B48.370.021.032.8
      Qwen3.6-27B48.380.020.037.9
      Gemma3-12B-IT46.774.012.931.1
  • The results suggest that specific capabilities, such as reasoning, affect model performance.
    • Introducing memory demands generally reduces performance, as shown by the lower Reasoning and Transfer scores of the models compared with their control settings.
    • Performance can also improve, as observed for Gemini-3.8-Flash on Transfer. Additional context may help the model recognize differences between problems and adapt the demonstrated procedure more effectively.
    • This suggests a potential future direction: Under what conditions can memory and specific capabilities reinforce each other, and how does this interaction depend on model capability?
  • Memory and other capabilities are inherently intertwined.
    • Following prior work such as LoCoMo [1] and MemoryAgentBench [2], we therefore report overall performance scores that reflect their combined effects.
    • We will additionally report single-turn control scores in the appendix to help distinguish underlying task capabilities from memory-related demands.

References

[1]Maharana, Adyasha, et al. "Evaluating very long-term conversational memory of llm agents." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

[2]Hu, Yuanzhe, Yu Wang, and Julian McAuley. "Evaluating memory in llm agents via incremental multi-turn interactions." International Conference on Learning Representations. Vol. 2026. 2026.

AuthorsOpenReview ↗

Response (II)

W2: realistic concern

Response:

We provide further clarification as follows:

  • The correspondence between our MemEnv and real human interaction patterns is fundamentally grounded in both our research focus and methodological design.
    • We focus on human–AI conversations in which users seek help and suggestions from LLMs, reflecting everyday uses of AI tools.
    • MemEnv is grounded in diverse user profiles and topics in daily activities. We sampled persona profiles from PersonaMem-v2[1] and collected topics from MemoryAgentBench[2], PersonaMem[3], LongMemEval[4], and OpenAI[5], as seed data for constructing daily conversations.
    • Each profile is further paired with different topics, yielding 50 persona–topic pairings. This allows MemEnv+ capture the multifaceted nature of individual users and their needs across multiple topics. For example, Adama Njeri Kamau is both a junior developer building research software (topic: computer programming) and someone who took analytical chemistry and works with lab instruments (topic: chemistry).
    • We incorporate a user simulator to support more natural and fluent conversations for on-policy evaluation.
  • We conducted human evaluations to verify the naturalness of generated conversations.
    • We randomly sampled 100 scripted user utterances, each independently evaluated by two annotators using a five-point scale. We asked annotators to assess naturalness, defined as linguistic fluency, semantic clarity, and resemblance to natural human dialogue.
      • The predefined user utterances used to expose key facts achieved an average score of 4.33/5, with 82% within-one-point inter-annotator agreement.
    • We randomly sampled 100 user utterances generated during dynamic interactions, each independently evaluated by two annotators on a five-point scale assessing overall conversational quality, including naturalness and non-interference with key facts.
      • The utterances achieved an average score of 4.65/5, with 97% within-one-point inter-annotator agreement.

References

[1]Jiang, Bowen, et al. "Personamem-v2: Towards personalized intelligence via learning implicit user personas and agentic memory." arXiv preprint arXiv:2512.06688 (2025).

[2]Hu, Yuanzhe, Yu Wang, and Julian McAuley. "Evaluating memory in llm agents via incremental multi-turn interactions." International Conference on Learning Representations. Vol. 2026. 2026.

[3]Jiang, Bowen, et al. "Know me, respond to me: Benchmarking llms for dynamic user profiling and personalized responses at scale." arXiv preprint arXiv:2504.14225 (2025).

[4]Wu, Di, et al. "Longmemeval: Benchmarking chat assistants on long-term interactive memory." arXiv preprint arXiv:2410.10813 (2024).

[5]Chatterji, Aaron, et al. "How people use chatgpt." No. w34255. National Bureau of Economic Research, 2025.

AuthorsOpenReview ↗

Response (III)

Q2: sensitivity analysis

Response:

  • Robustness to user simulators (Appendix F.5).
    • We have evaluated with two different user simulators, Llama3.1-8B-Instruct and Gemma4-26B-IT, while keeping other settings unchanged. Tables 9–10 reveals consistent system rankings and dimension-level performance patterns across different user simulators.
  • Sensitivity to graph sizes (Appendix F.6).
    • We investigated the effect of graph size on memory performance using Qwen3.5-9B under a fixed pruning policy. The results are as follows: | Graph Size | Summary | Update | Reasoning | Isolation | Preference | Transfer | Overall | |------------|---------|--------|-----------|-----------|------------|----------|---------| | 5×3 | 29.2 | 26.0 | 50.0 | 90.0 | 36.7 | 24.1 | 42.7 | | 3×2 | 27.0 | 48.0 | 48.3 | 76.0 | 40.9 | 25.1 | 44.2 |
    • Reducing graph size produces contrasting effects across memory dimensions, particularly improving Update while degrading Isolation. These findings highlight the dimension-dependent impact of graph size on memory performance.
    • As acknowledged by the Reviewer 7YyD, MemEnv scales all dimensions to a common structural size, providing a controlled basis for comparing dimensions at matched information loads.
  • Top-k selection (Appendix E).
    • The retrieval budget is determined by the predefined structured information loads of each memory dimension. It ensures a fair comparison across methods.
    • As detailed in Appendix E, we use task-specific top-k values of 10 or 35 based on the graph-defined number of evidence units required to answer the query. These budgets provide sufficient retrieval capacity for capable memory agents to retrieve all required information.
  • Diversity Considerations.
    • Diversity is explicitly considered in the design of MemEnv, rather than introduced as a post-hoc consideration. To account for variations in user backgrounds, interests, and conversational contexts, we sample diverse persona profiles from PersonaMem-v2 [1] and collect daily-life topics from MemoryAgentBench [2], PersonaMem [3], LongMemEval [4], and OpenAI [5].
    • These sources provide heterogeneous user profiles and conversational scenarios, reducing reliance on a narrow set of user characteristics or topics.

References

[1]Jiang, Bowen, et al. "Personamem-v2: Towards personalized intelligence via learning implicit user personas and agentic memory." arXiv preprint arXiv:2512.06688 (2025).

[2]Hu, Yuanzhe, Yu Wang, and Julian McAuley. "Evaluating memory in llm agents via incremental multi-turn interactions." International Conference on Learning Representations. Vol. 2026. 2026.

[3]Jiang, Bowen, et al. "Know me, respond to me: Benchmarking llms for dynamic user profiling and personalized responses at scale." arXiv preprint arXiv:2504.14225 (2025).

[4]Wu, Di, et al. "Longmemeval: Benchmarking chat assistants on long-term interactive memory." arXiv preprint arXiv:2410.10813 (2024).

[5]Chatterji, Aaron, et al. "How people use chatgpt." No. w34255. National Bureau of Economic Research, 2025.

Comments on the paper

7 comments
AuthorsOpenReview ↗

Response (I)

We thank the reviewer for the valuable feedback. We have conducted supplementary experiments to address these concerns, as detailed below.

W1&Q1: instance count and corresponding confidence intervals

Response:

  • MemEnv contains 50 instances per dimension and 300 conversations in total, balancing evaluation cost with comprehensive coverage of memory capabilities.

    • Its average context length is approximately 59K tokens per conversation, while MemEnv+ averages over 165K tokens. We promote diversity through 10 persona profiles pairing 50 everyday conversation topics drawn from different sources[1], [2], [3], [4], [5], with predefined utterances generated by a model randomly selected from a pool of LLMs from different model families..
    • Related benchmarks also use evaluation sets of comparable scale.
      • EvolIF[6]: 150 dialogues.
      • PersonaMem[1]: over 180 simulated user–LLM interaction histories.
      • AMemGym[7]: 20 user profiles, each paired with 10 evaluation questions.
      • MT-Bench[8]: 80 two-turn questions, comprising 10 questions in each of eight categories.
  • In Appendix F.4, we transparently report confidence intervals (CIs) for the evaluation results. Although the limited sample size contributes to wider intervals, our core conclusions remain supported:

    • Under matched information loads, models exhibit distinct strengths and weaknesses across dimensions, highlighting the importance of multidimensional memory evaluation. (sec 5.1)
    • Memory agents provide selective rather than uniform gains. (sec 5.2)
    • MemEnv+ reveals additional challenges when multiple memory dimensions are combined. (sec 5.3 & Reviewer C9sx)
    • Controlled comparisons reveal differences in performance and rankings between on-policy and off-policy evaluation. (sec 5.5 & Reviewer C9sx)

References

[1]Jiang, Bowen, et al. "Personamem-v2: Towards personalized intelligence via learning implicit user personas and agentic memory." arXiv preprint arXiv:2512.06688 (2025).

[2]Hu, Yuanzhe, Yu Wang, and Julian McAuley. "Evaluating memory in llm agents via incremental multi-turn interactions." International Conference on Learning Representations. Vol. 2026. 2026.

[3]Jiang, Bowen, et al. "Know me, respond to me: Benchmarking llms for dynamic user profiling and personalized responses at scale." arXiv preprint arXiv:2504.14225 (2025).

[4]Wu, Di, et al. "Longmemeval: Benchmarking chat assistants on long-term interactive memory." arXiv preprint arXiv:2410.10813 (2024).

[5]Chatterji, Aaron, et al. "How people use chatgpt." No. w34255. National Bureau of Economic Research, 2025.

[6]Jia, Qi, et al. "One battle after another: Probing llms’ limits on multi-turn instruction following with a benchmark evolving framework." Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

[7]CHENG, Jiayang, et al. "AMemGym: Interactive Memory Benchmarking for Assistants in Long-Horizon Conversations." Paper presented at The 14th International Conference on Learning Representations (ICLR 2026), Rio de Janeiro, Brazil. 2026.

[8]Zheng, Lianmin, et al. "Judging llm-as-a-judge with mt-bench and chatbot arena." Advances in neural information processing systems 36 (2023): 46595-46623.

AuthorsOpenReview ↗

Response (II)

W1&Q1(continued)

Response:

  • We conducted further pairwise CI analyses, and the results support our findings on: (1) the limited gains in Update and Transfer across model scales; (2) the selective gains of memory agents (Table 2); (3) robustness to user simulator choice (Table 9). More details are as follows:

    • Regarding "the Update and Transfer gain little from scale":

      • We conducted additional pairwise comparisons across three Gemma and three Qwen variants, using 95% paired bootstrap CIs to assess performance differences. The results are summarized below.

        Model ComparisonUpdateTransfer
        Gemma3-12B-IT − Gemma3-4B-IT+0.041 [−0.082, +0.163]−0.081 [−0.167, +0.010]
        Gemma4-E4B-IT − Gemma3-4B-IT−0.020 [−0.140, +0.120]+0.020 [−0.103, +0.143]
        Gemma4-E4B-IT − Gemma3-12B-IT−0.061 [−0.204, +0.082]+0.094 [+0.006, +0.188]
        Qwen3.5-9B − Qwen3.5-4B−0.047 [−0.163, +0.070]−0.028 [−0.134, +0.078]
        Qwen3.6-27B − Qwen3.5-4B−0.023 [−0.136, +0.091]−0.024 [−0.098, +0.049]
        Qwen3.6-27B − Qwen3.5-9B+0.022 [−0.067, +0.111]−0.006 [−0.105, +0.092]
        • Increasing model size from 4B to 12B within Gemma3 and from 4B to 9B within Qwen3.5 yields no statistically significant improvements in Update or Transfer.
        • These findings suggest that increasing model scale does not consistently improve Update and Transfer performance, supporting our conclusion that these capabilities remain challenging across model variants.
        • Regarding "The scaling discussion also conflates model size with model generation. ", we clarify that the scaling comparisons are conducted within the same model family and generation, varying only in model size.
    • Regarding the per-dimension gains and losses of memory agents in Table 2:

      • We report 95% paired bootstrap confidence intervals for the performance differences between seven memory systems and Gemma3-12B-IT.

        SystemSummaryIsolation
        AMemGym-AWI[-9.6, -0.4][6.4, 44.7]
        AMemGym-AWE[-17.2, -6.8][18.0, 50.0]
        RAG[-13.8, -3.2][18.0, 50.0]
        RFMem[-11.2, 0.4][16.0, 48.0]
        LightMem[-22.4, -13.4][22.0, 54.0]
        MemPalace[-12.4, -0.4][8.0, 36.0]
        HippoRAG[-15.0, -4.2][-46.0, -18.0]
      • In Isolation, six of the seven memory systems show improvements with 95% CIs. In Summary, all seven systems exhibit declines.

      • These results support our conclusion that memory augmentation provides selective rather than uniform gains across memory dimensions.

    • Regarding "the simulator-robustness result in Table 9", we assess simulator robustness through ranking consistency.

      • The Spearman correlations between each assistant's six-dimensional performance profiles range from 0.71 to 0.94, indicating broadly consistent relative strengths and weaknesses across dimensions.
      • We compute 95% paired bootstrap confidence intervals for performance differences between the Gemma4 and Llama3 simulators. The results below indicate that most observed performance differences are not statistically significant.
        SystemSummaryUpdateReasoningIsolationPreferenceTransferOverall
        Qwen3.5-9B[-.062,.028][-.182,.182][-.200,.150][-.167,.062][-.079,.070][-.144,.052][-.071,.031]
        Gemma3-4B[-.038,.028][-.160,.020][-.083,.100][-.160,-.020][-.070,.034][-.093,.073][-.056,.004]
        AWI[-.076,.004][-.286,.000][-.033,.100][-.152,.152][-.107,.132][-.111,.025][-.072,.016]
        MemPalace[-.024,.056][.000,.240][-.133,.133][-.140,.140][-.173,.004][-.121,.051][-.041,.046]
      • These results support our conclusion that MemEnv provides broadly consistent system-level evaluations across different user simulators, despite variations in absolute scores.
AuthorsOpenReview ↗

Response (III)

W1&Q1(continued)

Response:

  • We conducted further pairwise CI analyses, and the results support our findings on the dimension-dependent off-policy effects(Table 5). More details are as follows:
    • Regarding "the off-policy effects in Table 5.":
      • We refine our off-policy analysis by examining history-source differences and dimension-level performance patterns, supported by paired bootstrap confidence intervals. The results are shown below.
        SourcePositionSummaryTransferOverall
        DeepSeekEarly[-2.6, 7.2][-14.0, 3.7][-5.8, 3.0]
        DeepSeekMiddle[0.2, 9.4][-5.2, 16.0][-3.3, 5.5]
        DeepSeekEnd[6.0, 15.2][0.1, 19.6][-1.7, 6.4]
        GemmaEarly[1.4, 11.0][-10.8, 7.0][-1.3, 6.9]
        GemmaMiddle[2.8, 13.4][-5.9, 12.6][-1.3, 7.1]
        GemmaEnd[12.2, 22.6][-5.5, 14.9][-0.6, 8.2]
      • History utility varies across sources and memory dimensions. Although Gemma yields slightly higher Overall scores than DeepSeek across all handoff positions, their relative performance differs across memory dimensions. Under END, Gemma achieves higher Summary and Reasoning scores, whereas DeepSeek performs better on Update and Transfer.
      • Off-policy history transfer exhibits dimension-dependent performance patterns. Summary exceeds the on-policy baseline in all six off-policy settings. Moreover, Summary and Transfer scores increase monotonically from EARLY to END under both history sources, indicating larger benefits on information-focused tasks. In contrast, Reasoning does not exceed the on-policy baseline in any setting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.