acceptodds
Under review as a conference paper at ICLR 2027

ValMem: Value-Aware Memory Management for Long-Term LLM Agents

Abstract

Long-horizon LLM agents rely on diverse memories to support multi-session interactions, but effective memory management requires more than retrieving semantically similar history. The agent must decide which memories to retain, compress, or discard, so that storage and reasoning resources are not spent on outdated or redundant content. Although retrieval-augmented memory systems have been developed for long-term memory management, existing methods often rely on single-level memory segmentation and retrieval. Reliance on single-level memory segmentation and retrieval makes it difficult to capture complex relationships among memories. As a result, useful evidence may be overlooked, while irrelevant content may be introduced. To overcome these limitations, we propose ValMem, a value-aware memory management framework that combines a Multi-Granularity Hierarchical Memory (MGHM) structure with Value-Driven Training (VDT). MGHM organizes interaction history along temporal and granularity dimensions, providing memory units ranging from raw evidence to compact abstractions. To better identify the latent value of memory, VDT assigns credit to memory operations by optimizing downstream task utility jointly with memory cost. Experiments on three long-horizon memory benchmarks demonstrate that ValMem outperforms current state-of-the-art methods on multi-session question answering and personalized response tasks, with consistent gains across factual recall, multi-hop reasoning, temporal reasoning, and preference inference.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long-Term Memory, Agent, LLM, Reinforcement Learning, Conversation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nqJzOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes ValMem, a value-aware memory management framework for long-term LLM agents. It combines Multi-Granularity Hierarchical Memory (MGHM) with Value-Driven Training (VDT) to optimize memory retrieval, compression, and retention based on downstream utility. Experiments on three benchmarks demonstrate improvements in task performance and memory efficiency.

Critical strengths
  • The paper addresses the important challenge of balancing memory utility and storage efficiency in long-term LLM agents.
  • The combination of temporal hierarchy and multiple representation granularities provides a flexible structure for memory management.
  • VDT introduces operation-specific counterfactual rewards, enabling more precise optimization of memory decisions.
  • Clear presentation. The paper is well organized and makes it easy for the reader to quickly grasp the main idea and method.
Critical weaknesses
  • The core idea of using downstream utility to guide memory retrieval and management is closely related to prior work such as MemRL [1] and Mem-T [2], which also employs reinforcement learning and utility-based memory selection. Although ValMem introduces counterfactual credit assignment and different memory operations, the paper does not sufficiently clarify its novelty relative to MemRL or provide a direct comparison.

  • Experiments only use Qwen3.5-9B, leaving the effectiveness across different model families and sizes unclear.

  • Counterfactual rollouts require substantial computation (approximately 120 hours on four A800 GPUs), potentially limiting practical adoption. And the computational overhead of maintaining cross-granularity associations and routing over increasingly large memory banks is not sufficiently analyzed.

  • Lastly, using low entropy as an indicator of memory reliability is not fully justified, since concentrated retrieval scores do not necessarily imply factual correctness.

[1] Zhang, Shengtao, et al. "Memrl: Self-evolving agents via runtime reinforcement learning on episodic memory." NeurIPS 2026.

[2] Yue, Yanwei, et al. "Mem-t: Densifying rewards for long-horizon memory agents." ICML 2026.

Questions for authors

See weaknesses above

4 comments
AuthorsOpenReview ↗

To Reviewer nqJz

We appreciate the reviewer's positive assessment of the balancing memory utility and storage, the clarity of the writing, and the reasonable engineering. For a general view of how ValMem differs from prior memory systems, please see our "Response to All Reviewers", followed by our answers to your specific questions. For the specific content, please refer to the revised version.

W1: Novelty relative to MemRL and Mem-T, which also use RL and utility-based memory selection, no clarification or direct comparison.

Thank you for raising this point. We have revised the Related Work section accordingly, and have included a comparative explanation in the revised paper (Appendix A, Table 4):

  • MemRL is non-parametric: weights stay frozen, and a runtime utility learned from environmental feedback re-ranks retrieved episodes. ValMem trains the model to decide at write time whether to keep, discard or compress each memory under an explicit storage cost. A utility-function re-ranker based on the MemRL style can be deployed on our memory repository.
  • Mem-T propagates the reward backward on the sampling memory operation tree (MoT-GRPO) through post-event credit assignment. ValMem performs controlled intervention, starting from the same memory checkpoint, using the frozen policy and exactly the same subsequent requests, to repeat-play each action in a certain memory K times (Eq. 12–15), and only allocate the advantage to the decision tokens (Eq. 17). Compression targets and the storage term are explicit parts of the action space and reward (Eq. 9, 11).

Since direct comparisons are influenced by different memory architectures, action spaces, and tasks, we have added controlled comparisons under the common baseline model (Qwen3.5-9B) and LoCoMo split in the revised paper: a Mem-T style tree-based backward propagation credit variant, and a MemRL style runtime utility value reordering variant on our memory repository (Appendix F.2, Table 13).

W2: Only Qwen3.5-9B is used.

We agree that evaluating a single backbone limits the evidence for generality across model sizes. While cross-benchmark transfer is evaluated by training solely on LoCoMo and testing zero-shot on LongMemEval and PersonaMem-v2 in the original submitted paper, the backbone was fixed. In the revised paper, we conducted a complete process (training and evaluation) for Qwen3.5-4B and Llama-3.1-8B alongside Qwen3.5-9B across all three benchmarks, and have reported the results in Appendix C.1 (Table 6), showing consistent improvements across different model series and sizes.

BackboneVariantLoCoMo F1LongMemEval F1PersonaMem-v2 Acc.
Llama-3.1-8BBaseline (w/o all)31.846.231.5
Llama-3.1-8BValMem42.558.444.2
Qwen3.5-4BBaseline (w/o all)34.649.834.1
Qwen3.5-4BValMem45.862.347.9
Qwen3.5-9BBaseline (w/o all)36.352.636.4
Qwen3.5-9BValMem48.165.850.6
AuthorsOpenReview ↗

To Reviewer nqJz (2)

W3: Training cost (about 120 h on 4 x A800) may limit adoption, overhead of cross-granularity association and routing over growing memory banks is not analyzed.

  • Training cost. The counterfactual deployment of VDT involves a one-time training cost. There is no counterfactual deployment process during the inference stage. It takes 4.2 seconds to process every 100 samples and generates 0.62K completion tokens (Section 5.5, Figure 3; Appendix F.1). The deployment cost is not significant. To conduct a more detailed analysis of the deployment cost, we added a cost-benefit curve that varies with the number of continuations K in the revised paper, and evaluated different actions while conducting budget matching comparisons with GRPO and local redeployment (Appendix B, Table 5).
MethodGPU HoursTotal TokensOverall F1
Standard GRPO (matched budget, G=16G=16)1201.50M44.18
Local/Session Rerollout (matched budget, K=16K=16)1201.50M45.85
ValMem (K=2K=2)450.56M45.92
ValMem (K=4K=4)720.90M47.35
ValMem (K=8K=8, default)1201.50M48.14
  • Memory-bank overhead. Appendix F.1 reports the end-to-end latency, but does not report the computational overhead for maintaining the cross-granularity associations and routing. We believe it is meaningful to conduct a component-based scaling analysis. In the revised paper, we have added the scalability study in Appendix F.1 (Table 12). In this study, the memory bank is gradually expanded from the baseline size to several times the baseline size, and the delay of each query is decomposed into three parts: association (GMM acceptance/rejection), routing, and retrieval (Top-N, N=5). Since the association and routing operations are only performed within a limited candidate neighborhood rather than scanning the entire bank, and redundant entries are eliminated through a value-aware discard/compression mechanism, the bank remains compact.
Memory-Bank ScaleAssociation (GMM)Entropy RoutingTop-NN (N=5N=5)Total Overhead
1×1\times (4.25 GiB4.25\text{ GiB}, baseline)18.4 ms11.2 ms6.5 ms36.1 ms
2×2\times (8.50 GiB8.50\text{ GiB})24.7 ms14.8 ms8.9 ms48.4 ms
4×4\times (17.00 GiB17.00\text{ GiB})33.5 ms19.6 ms12.3 ms65.4 ms

W4: Low entropy as a reliability indicator is not justified, concentrated scores do not imply factual correctness.

We agree that low entropy merely indicates the retrieval concentration, rather than the accuracy of the facts. The expression "reliable matching" in Section 4.1.3 is not accurate, we have changed it to "retrieval concentration". Then, we analyzed the reliability of this retrieval concentration in the revised paper. Based on entropy, the router is designed as a structural prior, used to guide the agent towards relevant neighborhood clusters, rather than a fact-checking tool. The accuracy of facts is a specific task of ValMem. Our experiments have confirmed this division of labor:

  1. Necessity of routing. After replacing entropy routing with uniform weights, LoCoMo F1 decreased by 3.52, and LongMemEval F1 decreased by 5.64 (Appendix F.4, Table 15), proving that it significantly improved the retrieval efficiency.
  2. Value-aware processing of invalid information. In our adversarial memory injection research (Appendix F.5, Table 16), VDT successfully eliminated 72.5% of outdated memories, 58.3% of conflicting memories, and 81.6% of irrelevant memories, clearly demonstrating that it is the strategy with value-awareness, rather than the router itself, that ensures the accuracy of facts.
Memory typeDiscard rate (%)Compress rate (%)Keep rate (%)Selected-memory precision (%)
Outdated72.518.39.288.4
Duplicated65.824.69.686.1
Conflicting58.320.521.282.7
Irrelevant81.612.46.090.2

Thank you for the additional experiments and clarifications. While appreciate the new backbone evaluations and controlled comparisons, my core concerns regarding the method's soundness remain unresolved, so I will maintain my current rating. Simply renaming reliable matching to retrieval concentration does not justify using inverse entropy. In practice, an outdated or conflicting memory cluster with high lexical overlap will yield a sharp, low-entropy distribution, meaning inverse entropy actively amplifies stale information. Conversely, multi-hop queries inherently span diverse topics (higher entropy) and get penalized. Beating uniform weights only shows non-uniform weighting helps, not why inverse entropy is superior to standard max/mean neighborhood similarity.

Thank you for your constructive feedback. The router only partitions candidates into neighborhoods, while factual correctness is handled by VDT's value-aware pruning, not by entropy. High-overlap stale clusters may receive sharp low entropy, but VDT then discards or compresses them; multi-hop queries may have higher entropy, but VDT preserves necessary evidence across granularities. We will emphasize this part in the paper, and we will continue to refine the writing in other sections as well.

Reviewer_RUd4OpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ValMem, which combines a multi-granularity hierarchical memory structure with value-driven training. It uses counterfactual rollouts to estimate the downstream benefits of different operations on each memory item. It then assigns operation-level advantages to the corresponding decision tokens to learn retention, deletion, and compression policies. The authors report improvements on LoCoMo, LongMemEval, and PersonaMem-v2 and provide controlled experiments on credit assignment. However, the novelty over existing memory management and local resampling methods needs clarification. The training setup for key baselines is not clear enough to establish fairness. The evaluation also does not support broad claims about general agents that interact over long horizons.

Critical strengths
  1. The motivation is clear. The method connects memory operations to later task outcomes, compares different operations on individual memory items, and assigns training signals to the relevant decisions.

  2. Beyond the main experiments, the authors compare credit assignment methods using the same backbone and data split. These experiments try to separate the effects of trajectory-level rewards, local rerollouts, operation-level rerollouts, and decision-token credit assignment.

Critical weaknesses
  1. Limited overall novelty and an unclear contribution beyond prior work. AgeMem (ACL 2026) already uses reinforcement learning to manage storage, updating, deletion, and summarization. Memory-R2 already resamples from the same intermediate memory state to improve credit assignment. MemGAS (ICLR 2026) already includes multi-granularity representations, GMM-based association, and inverse-entropy routing. The authors cite and compare with these studies, but the main additions appear to be per-memory action enumeration and decision-token advantage assignment. The paper does not clearly separate existing mechanisms from its new designs. It also does not establish whether the gains come from better credit assignment or from extra computation for counterfactual evaluation.

  2. Unclear training sources and fairness for key baselines. The main experiments do not clearly state whether AgeMem and Memory-R2 were retrained with the same backbone and LoCoMo training conversations or used checkpoints released by the original authors. Following “official settings” is not enough to ensure a fair comparison. The Memory-R2-style local rerollout control in the appendix provides some evidence, but it is not a full reproduction under a shared training setup. The paper also does not clearly state whether training compute was matched. In addition, the appendix labels AgeMem as requiring no training. This conflicts with its original method and the main text and needs clarification.

  3. The evaluation scope does not match the broad claims about long-horizon agents. The three benchmarks mainly test long-term conversational memory, cross-session QA, and personalized inference. They do not sufficiently cover multi-turn, long-horizon interactions in which actions change the environment and environmental feedback affects later decisions. The current results therefore mainly support conversational memory use cases. They do not establish that the proposed policies improve long-term task completion for general interactive agents.

Questions for authors
  1. Which mechanisms are distinct methodological contributions beyond AgeMem, Memory-R2, and MemGAS? Do the gains from action enumeration and decision-token credit assignment remain when training generation tokens, counterfactual evaluations, or GPU time are matched?

  2. Please clarify the model and checkpoint sources, training conversation splits, training steps, and compute budgets for AgeMem and Memory-R2 in the main experiments. Could the authors provide full retraining results under a shared setup and explain the incorrect training label for AgeMem in the appendix?

  3. Does the method retain its advantage on interactive agent benchmarks with action–environment feedback, changing task states, and delayed rewards? If such experiments are unavailable, should the conclusions be limited to long-term conversational memory and personalized assistants?

3 comments
AuthorsOpenReview ↗

To Reviewer RUd4

We are grateful to the reviewer for emphasizing the conceptual clarity of our research motivation, empirical results and ablation experiments. For the unique contributions of this research, please refer to the "Response to All Reviewers". For the specific content, please refer to the revised version. Subsequently, we will respond to your detailed suggestions.

W1 / Q1: Apart from AgeMem, Memory-R2, and MemGAS, the novelty is limited, it is not clearly stated which mechanisms are new, and whether the performance improvement is due to credit allocation or additional computation.

Detailed Comparison of Related Works. We agree that the paper should clearly distinguish our methodological contributions from AgeMem, Memory-R2, and MemGAS, and we have added a comprehensive comparison table in the revised paper (Appendix A, Table 4) to illustrate this:

  • vs. AgeMem: AgeMem optimizes memory tool calls (store, update, delete, summarize) via trajectory-level step-wise GRPO over flat Short-Term and Long-Term stores, where a single episode-level reward is shared across many interleaved tool calls and reasoning steps. In contrast, ValMem defines a structured (ℓ,g)(\ell, g) action space across three temporal layers and four granularities (Eq. 9), evaluates each feasible operation on individual memory items via counterfactual rollouts, and attributes the centered advantage exclusively to the decision tokens Td\mathcal{T}_d (Eq. 15, 17).
  • vs. Memory-R2: While Memory-R2 resamples local continuation trajectories from an intermediate checkpoint, each resampled trajectory can still execute stochastic multi-item edits and answer paths, leaving individual item–operation credit entangled. ValMem instead enumerates all feasible operations A(mi)\mathcal{A}(m_i) (keep, discard, and each compress target) for a single target memory mim_i while holding all other memories Mt∖{mi}M_t \setminus \{m_i\}, the continuation policy πθold\pi_{\theta_{\mathrm{old}}}, and the next hh queries fixed (Eq. 12–15), isolating the marginal utility and storage cost of that exact operation.
  • vs. MemGAS: MemGAS applies multi-granularity construction, GMM-based association, and inverse-entropy routing over a static, training-free memory pool at read time without write-time pruning or compression. ValMem repurposes multi-granularity units as controllable write-time compression targets and couples MGHM with VDT, enabling the agent to actively compress or discard low-utility memories before answer generation rather than accumulating all granularities indefinitely.

Does the gain persist under matched computation? Yes. We isolated gains from the "extra compute" of rerollouts using controlled comparisons (Appendix B, Table 5; Appendix F.2, Table 13). When matching total GPU time/tokens:

  1. Per-memory Action Enumeration. By forcing the model to explicitly choose between keep, discard, and compress for every candidate, we prevent the model from inheriting suboptimal memory states, unlike PPO-based managers.
  2. Decision-Token Credit. By assigning reward solely to tokens that represent the memory action itself (not the whole sequence), we turn the management problem into a focused classification task. Our ablation (Appendix F.2, Table 13) shows isolating this signal adds +2.28 F1, a gain that persists even when we restrict total continuations KK to match lower-performing baselines. The boost is thus not due to "more compute," but to "better signal isolation."

W2 / Q2: Unclear training source and fairness for AgeMem and Memory-R2. "official settings" is insufficient. AgeMem is labeled "No" training in the appendix.

We apologize for the confusion. We have addressed these concerns as follows:

  • Correcting the AgeMem Label. We apologize for the typo in Appendix E (Table 9) of the original submitted paper. We have corrected the training entry of AgeMem to RL.
  • Detailed Provenance. We added a provenance and compute column to Table 9 in Appendix E of the revised paper documenting the source, backbone, data split, training steps, and GPU hours for all baselines (including AgeMem, Memory-R2, MemAgent, MEM1).
    • Where a baseline was retrained, we report the exact compute budget spent on our infrastructure (e.g., Memory-R2 trained 68h, ValMem trained 120h).
    • Where a method could not be retrained due to architectural mismatch, we marked the row as "original released checkpoint/backbone" to avoid any ambiguity. Our retrained results are now consistently reported across our benchmarks, ensuring a transparent, apples-to-apples comparison.
AuthorsOpenReview ↗

To Reviewer RUd4 (2)

W3 / Q3: The benchmarks cover conversational memory, not action-environment feedback, changing states and delayed rewards.

We agree that our benchmark tests focus on dialogue memory, but during the evaluation process, the memory repository is essentially dynamic. In our setup, the agent's actions directly modify the memory repository, thereby influencing the future state of the evaluation environment. This makes memory evolution an interactive process, where the agent's own actions shape its future retrieval environment.

Although we did not change the physical environment, we believe that our design effectively addresses the core challenge of personalized long-term assistance, where the environment itself is the evolving dialogue history. Regarding your concern about action-environment feedback and delayed rewards, we further explain:

  • VDT as a Delayed-Reward Mechanism. Our VDT framework (Eq. 11–15) is explicitly designed for delayed credit assignment. The reward C(τ)C(\tau) is measured over a horizon of h=3h=3 steps, evaluating how a memory operation at time tt influences task success at time t+ht+h. The performance gain of VDT over trajectory-level GRPO (Appendix B, Table 5; Appendix F.2, Table 13) proves that our value-aware assessment successfully handles delayed feedback by assigning credit to memory decisions based on their long-term impact, rather than immediate answer accuracy.
  • Robustness to Dynamic State Perturbations. Our experiments on adversarial memory injection (Appendix F.5, Table 16) and evidence-aware reward attribution (Appendix C.2, Table 7) confirm that ValMem robustly handles conflicting, duplicated, or irrelevant data. This ability to dynamically prune "distractor" states while maintaining high-fidelity retrieval is a core requirement for interactive agents operating in state-changing environments.
  • General applicability: Although our benchmark test focuses on conversation assistants, where the dialogue history is the main dynamic state, the MGHM+VDT framework is a universal solution applicable to any state management decision-making environment with delayed consequences in state-based reasoning.

Dear Reviewer RUd4,

Thanks for your time and effort in reviewing this paper. Your comments are valuable to improve the quality of this paper. As the discussion period between authors and reviewers is about to conclude, we gently request that you read the rebuttal and check whether your concerns are addressed. Thanks again for your time and help in improving the quality of this paper. Regards from the authors.

Reviewer_2k32OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ValMem, a framework that learns to retain, compress, and delete memories for long-term conversational agents, balancing downstream task performance against storage costs. The method consists of two components:

  1. Multi-Granularity Hierarchical Memory (MGHM). This component organizes memories into three levels, namely short-term, episodic, and long-term memory, and four granularities: raw, record, topic, and abstract. It establishes associations using GMM and generates retrieval candidates through entropy-based routing.
  2. Value-Driven Training (VDT). Starting from the same memory checkpoint, this component enumerates the feasible operations for a particular memory and samples continuation trajectories under the same sequence of subsequent requests and a frozen policy. It estimates operation values based on task rewards and storage costs, then assigns relative advantages to the corresponding decision tokens.

The authors train on LoCoMo and evaluate on LoCoMo, LongMemEval, and PersonaMem-v2, reporting improvements in both task performance and storage efficiency.

Critical strengths

S1. The problem is clearly motivated. Retaining, deleting, or compressing the same memory can affect the evidence available for subsequent tasks. By comparing alternative operations from the same checkpoint, VDT provides a more targeted assessment than directly comparing complete trajectories with different memory states.

S2. The method offers some technical advances. The method goes beyond adding a storage penalty to the reward for an entire trajectory. It enumerates alternative compression targets and assigns additional advantages to the decision tokens associated with the corresponding operations.

S3. The experiments include generalization across benchmarks and analyses of the underlying mechanisms. The authors train exclusively on LoCoMo and evaluate transfer to LongMemEval and PersonaMem-v2. The appendix also compares different credit assignment strategies and analyzes operation choices for outdated, conflicting, and irrelevant memories. In addition, the evaluation considers both task performance and storage costs.

Critical weaknesses

W1. The discussion of related work is insufficient. The GMM-based associations and entropy-based routing also appear similar to aspects of MemGAS. The use of memory representations at multiple granularities is not itself new, as related designs have appeared in works such as Mem-T. Furthermore, related methods for learning memory construction and management, including Mem-α, MemRL, and Memory-R1, are not sufficiently discussed or directly compared in the experiments. The paper should clarify its differences from these methods. The relevant references are listed below for ease of consultation:

  • [ICLR26] From Single to Multi-Granularity: Toward Long-Term Memory Association and Selection of Conversational Agents. arXiv:2505.19549
  • [ICML26] Mem-T: Densifying Rewards for Long-Horizon Memory Agents. arXiv:2601.23014
  • [arXiv] Mem-α: Learning Memory Construction via Reinforcement Learning. arXiv:2509.25911
  • [arXiv] MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memory. arXiv:2601.03192
  • [ACL26 Main] Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning. arXiv:2508.19828

W2. An internal inconsistency in the reported results undermines confidence in the conclusions. In Table 1, Memory-R2 has an Overall F1 of 38.30, whereas its F1 scores for the four reported categories are 22.37, 22.62, 30.55, and 35.29. If Overall F1 is the weighted average over the samples in these four categories, it cannot exceed every category score. This discrepancy requires clarification and, if necessary, correction.

W3. The training cost requires further clarification and a comparison under matched computational budgets. The method enumerates all feasible operations for each memory and samples K continuation trajectories for each operation. Its sampling cost may therefore be substantially higher than that of standard GRPO or local resampling. Comparisons under the same rollout token budget or GPU time are needed to distinguish the contribution of finer credit assignment from that of additional sampling computation.

Questions for authors

LoCoMo contains ten conversations, each associated with a set of approximately one hundred questions. According to its evidence annotations, each conversational turn is relevant to only a few questions, or potentially none. My understanding is that (C) in Equation 11 measures average accuracy over the entire question set. However, an individual memory operation in MGHM may concern a specific turn and therefore affect only a very small subset of questions.

This raises a question about Equations 12 and 13 in the “Latent Value Assessment” section: when changing a single memory operation may affect the accuracy of only a few questions, how appropriate is average accuracy over the entire question set as the reward for assessing that operation? In particular, could this aggregation dilute the reward signal associated with the operation? Have the authors investigated more fine-grained attribution between memory operations and accuracy on the corresponding questions? Please clarify if my understanding is incorrect.

My remaining questions are covered in the weaknesses above.

3 comments
AuthorsOpenReview ↗

To Reviewer 2k32

We thank the reviewer for recognizing the importance of the problem and the depth of our analysis, including the cross-benchmark and mechanism analyses. For clarifications on the overall contribution and positioning of ValMem, please refer to the "Response to All Reviewers". For the specific content, please refer to the revised version. Subsequently, we will respond to your detailed suggestions.

W1: Related work is insufficient; GMM association and entropy routing resemble MemGAS, multi-granularity memory appears in Mem-T, and Mem-α, MemRL and Memory-R1 are not discussed or compared.

We agree that our GMM-based association and entropy-based routing are closely related to MemGAS, and that multi-granularity memory is closely related to Mem-T. We have revised the Related Work section (Section 2) and added a comprehensive comparison table in Appendix A (Table 4), briefly explaining as follows:

  • MemGAS / Mem-T. MemGAS focuses on read-time association and routing over a static memory pool, without learning write-time retention or compression. In ValMem, MGHM couples temporal layers with semantic granularities so that coarser levels serve directly as targets for write-time keep, discard, and compress actions (Eq. 9).
  • Mem-α / Memory-R1. Both train memory managers with trajectory-level outcome rewards via PPO/GRPO. Because one episode mixes many memory edits and reasoning steps, a terminal reward cannot isolate the effect of a single edit. ValMem resolves this credit ambiguity by comparing alternative actions on the same memory from a shared checkpoint and assigning credit only to decision tokens (Eq. 17).
  • MemRL. MemRL freezes the LLM and updates a runtime utility table to re-rank retrieved episodes. Instead of only re-ranking a growing pool at read time, ValMem trains the policy to prune and compress memories at write time under a storage budget.
  • Mem-T. Mem-T back-propagates outcomes over a sampled operation tree, where future states and queries diverge across branches. ValMem instead performs a controlled counterfactual intervention: from the same snapshot, it evaluates each feasible action under a frozen policy and identical future queries (Eq. 12–15) to isolate the causal utility of one edit.

Additionally, although there was no comparison of Memory-R1 in the initial submission, we compared its official upgraded version (Memory-R2). To avoid architectural confounding and provide a direct empirical comparison for the methods you raised, we also re-implemented the MemRL-style runtime utility re-ranking, the Memory-R1-style outcome-reward manager, and the Mem-T-style tree-based credit mechanism under the same backbone, MGHM structure, and action space in the revised paper. The specific experimental results are reported in Table 13 (Appendix F.2).

MethodF1BLEU-1L-JStorage (GiB)Op. Acc. (%)
MemRL-style Runtime Utility Re-ranking41.8534.1259.400.7252.80
Memory-R1-style Outcome-Reward Manager43.1235.2460.950.5857.65
Mem-T-style Tree-based Credit44.6836.5262.740.3564.12
VDT (ours)48.1439.1365.010.0978.62

W2: Table 1 is internally inconsistent: Memory-R2 Overall F1 is 38.30, but its category F1 scores are 22.37, 22.62, 30.55 and 35.29.

Thank you for pointing out this typing error. The correct Overall F1 of Memory-R2 is 28.30, 38.30 was a typo. We have corrected it in the paper and re-checked all other entries of Table 1 against the original results.

AuthorsOpenReview ↗

To Reviewer 2k32 (2)

W3: The training cost requires further clarification and a comparison under matched computational budgets.

Thank you for your valuable suggestions regarding training costs. Table 13 (Appendix F.2) in the submitted paper compares credit assignment strategies under an identical backbone, split and storage budget, but does not equalize rollout tokens or GPU time. We'll complete the discussion regarding training costs. In the revised paper, we have added comparisons under matched generated tokens and GPU hours: (i) standard GRPO with a larger group size, (ii) Local/Session Rerollout with a larger rollout count, and (iii) VDT with fewer continuations K, plus a stage-wise token and GPU-hour breakdown (total: ~120 h on 4 x A800, Appendix B, Table 5; Appendix F.1).

MethodGPU HoursTotal TokensOverall F1
Standard GRPO (matched budget, G=16G=16)1201.50M44.18
Local/Session Rerollout (matched budget, K=16K=16)1201.50M45.85
ValMem (K=2K=2)450.56M45.92
ValMem (K=4K=4)720.90M47.35
ValMem (K=8K=8, default)1201.50M48.14

Q1: Does the average accuracy over the whole question set in Eq. 11 dilute the reward of a single operation? Was finer attribution studied?

Thank you for your question. Your understanding that a single operation only affects a few questions is right, but severe dilution is avoided for these reasons:

  • The reward is not computed over all questions. C(τ)C(\tau) (Eq. 11), as used in Eqs. 12–14, is measured on the subsequent requests within the rollout horizon (3 in our setup) after the operation, not over the full conversation.
  • Unrelated questions cancel in the advantage. All alternative actions share the same checkpoint and the same subsequent requests, so questions unaffected by the operation yield the same outcome for every action and cancel in the centered advantage (Eq. 15). They only add variance, which K-sample averaging reduces.

Additionally, regarding Finer-grained attribution (Appendix C.2, Table 7). Using LoCoMo's turn-level evidence labels, we have tested an Evidence-Aware Attribution variant that scores C(τ)C(\tau) only on questions directly tied to mm. While full-dialogue averaging hurts accuracy (Pearson r=0.54r = 0.54, LoCoMo F1 45.1045.10), our 3-step horizon (r=0.78r = 0.78, F1 48.1448.14) nearly matches ground-truth evidence attribution (r=0.81r = 0.81, F1 48.3948.39) without needing human evidence labels.

Attribution SettingPearson rrLoCoMo F1
Full-dialogue averaging0.5445.10
3-step horizon (VDT, ours)0.7848.14
Evidence-Aware Attribution (ground-truth labels)0.8148.39

Dear Reviewer 2k32,

Thanks for your time and effort in reviewing this paper. Your comments are valuable to improve the quality of this paper. As the discussion period between authors and reviewers is about to conclude, we gently request that you read the rebuttal and check whether your concerns are addressed. Thanks again for your time and help in improving the quality of this paper. Regards from the authors.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to All Reviewers

We thank all reviewers for their constructive comments, acknowledging our clear motivation, well-organized presentation, flexible hierarchical structure, and targeted operation-specific credit assignment. We appreciate their acknowledgment of our empirical results on task performance and storage efficiency across long-horizon benchmarks (2k32, RUd4, nqJz), and the depth of our analyses (2k32, RUd4), including cross-benchmark transfer (2k32), credit-assignment comparisons (2k32, RUd4), ablations (RUd4), and mechanistic analyses of outdated, conflicting, and irrelevant memories (2k32).

ValMem's Contributions to Long-Term Agent Memory (2k32 W1, RUd4 W1/Q1, nqJz W1)

Retaining, compressing, and discarding past interactions across temporal and granularity dimensions is crucial for long-term agents, yet LLM systems still struggle to optimize individual memory operations by their downstream utility. Our work in ValMem, rather than just introducing another "multi-granularity + RL" paper, aims to investigate and address this key gap in value-aware memory management.

We first reformulate long-term memory maintenance via two coupled capabilities: multi-resolution memory structuring (exposing explicit decision units across temporal layers and granularities) and operation-level credit assignment (isolating the delayed causal utility of individual memory edits under storage constraints). This scope overcomes the narrow focus of prior works on static similarity routing or coarse trajectory-level RL (2k32 W1, RUd4 W1, nqJz W1).

Given such challenges, we introduce ValMem, grounded in two designed dimensions: 1) Multi-Granularity Hierarchical Memory (MGHM) and 2) Value-Driven Training (VDT), enabling value-aware consolidation and retrieval capabilities that neither static heuristic stores nor trajectory-level RL managers support.

Across three long-horizon benchmarks and three backbones (Llama-3.1-8B, Qwen3.5-4B, Qwen3.5-9B), ValMem outperforms state-of-the-art memory systems, showing up to 12.5% F1 improvement and 22%–57% storage reduction on multi-session QA and personalized response tasks.

Multi-Granularity Hierarchical Memory: Representation & Routing Phase (2k32 W1, RUd4 W1, nqJz W4)

For memory representation, our design advances MemGAS and Mem-T by coupling temporal layers (ST,EP,LT\text{ST}, \text{EP}, \text{LT}) with four granularities (raw, record, topic, abstract) as explicit write-time action targets. This yields a more compact and adaptive store than MemGAS, Mem0, and MemRL, which focus on read-time routing or re-ranking over an unpruned pool rather than write-time retention and compression (Appendix A, Table 4).

Specifically, most existing systems either (1) route over a static multi-granularity pool (MemGAS), or (2) re-rank retrieved episodes at read time with frozen weights (MemRL). Neither exposes multi-granularity abstractions as controllable write-time compression targets paired with concentration-guided retrieval, which MGHM is designed to do:

  • Representation level: Memories are organized into explicit (ℓ,g)(\ell, g) units so coarser levels (record, topic, abstract) serve directly as candidate targets for write-time keep, discard, and compress actions (Eq. 9).
  • Routing level: Cross-granularity GMM association and inverse-entropy routing act as a structural prior to locate high-concentration neighborhoods (Section 4.1.3; Appendix F.4, Table 15), leaving factual filtering of outdated, conflicting, and irrelevant items to VDT (Appendix F.5, Table 16).

Value-Driven Training: Credit Assignment Phase (2k32 W1/W3/Q1, RUd4 W1/Q1, nqJz W1/W3)

We design VDT as a controlled counterfactual intervention process on individual memory item–operation pairs. From a shared checkpoint, the agent evaluates feasible operations (keep, discard, and each compress target) under a frozen policy and identical future queries over an hh-step horizon, making credit assignment operation-specific rather than confounded by diverging trajectories or answer tokens.

By linking downstream task utility and storage cost directly to memory-decision tokens (Eq. 11–17), each update isolates the marginal contribution of a single edit, a capability missing in prior RL paradigms (Appendix A, Table 4; Appendix F.2, Table 13):

  • Trajectory-level RL (Memory-R1, Mem-α\alpha, AgeMem) assigns one episode-level reward across interleaved edits and reasoning steps, diluting the credit of individual memory edits.
  • Tree / session rerollouts (Mem-T, Memory-R2) propagate rewards over branching trees or resamples where future states and queries diverge, whereas VDT evaluates all actions from a shared snapshot and outperforms baselines under matched budgets (Appendix B, Table 5; Appendix C.2, Table 7).

Overall, ValMem aligns multi-granularity structure with counterfactual decision-token credit assignment, providing a capability prior works lack.

AuthorsOpenReview ↗

Update Notes for Paper Draft

The updated contents include:

  • Appendix A Detailed Comparison with Related Works: Comparison of memory organization, retrieval strategy, and credit assignment mechanisms. (2k32, RUd4, nqJz)
  • Appendix B Computational Budget Analysis: Matched-budget comparisons and continuation scaling analysis across KK. (2k32, RUd4, nqJz)
  • Appendix C.1 Generality Across Backbones: Performance across Llama-3.1-8B, Qwen3.5-4B, and Qwen3.5-9B on all three benchmarks. (nqJz)
  • Appendix C.2 Evidence-Aware Attribution: Comparison between 3-step rollout attribution and ground-truth evidence attribution. (2k32, RUd4)
  • Appendix E Baseline Details: Training provenance, checkpoint sources, and compute budgets for all baselines. (RUd4)
  • Appendix F.1 Computational Cost and Memory-Bank Scalability: Component-wise query latency under growing memory-bank sizes. (2k32, nqJz)
  • Appendix F.2 Controlled Comparison of Credit Assignment Strategies: Controlled comparisons with MemRL-style, Memory-R1-style, and Mem-T-style variants. (2k32, RUd4, nqJz)

We'll further add more discussions and ablation studies in the revised version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.