acceptodds
Under review as a conference paper at ICLR 2027

MemPO: Optimizing Memory Write Policies with Retrieval-Attributed Credit

Abstract

Persistent memory allows LLM agents to use information from interactions that extend far beyond the context window. Existing memory systems, however, typically rely on fixed write rules that decide what to retain, summarize, merge, or discard before knowing which information future queries will need. Our analysis shows that no single retention strategy is sufficient: the utility of a write is sparse, often revealed only many steps later, and increasingly difficult to recover from trajectory-level feedback as the write-to-query horizon grows. We introduce MEMPO, a framework for optimizing memory write policies with retrieval-attributed credit. For each query, MEMPO attributes reward among the retrieved memory items using Shapley values, then traces each item's contribution through its provenance to the write decisions that produced it. The resulting localized returns train a policy to decide how much of each incoming segment should survive in memory. By assigning credit through the retrieved set, MEMPO ties each write's learning signal to whether it later provides useful evidence, rather than to its temporal distance from the query. Across six benchmarks spanning conversational memory, belief revision, and long-document question answering, MEMPO consistently outperforms fixed-rule and learned-memory baselines. On LoCoMo, it improves average F1 from 39.5 to 44.7 over the strongest hand-designed policy.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Meta Cognition, Knowledge Transfer, AI for Science

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HrnUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MemPO learns memory-writing policies by attributing downstream query rewards to retrieved items through Shapley values and tracing these contributions back to historical writes. Experiments demonstrate improved accuracy, compact memory, and reduced gradient variance across six benchmarks.

Critical strengths

The paper addresses an important problem in learning what agents should remember. The proposed credit-assignment approach is intuitive, and the experimental study is extensive, covering multiple benchmarks, detailed ablations, and transfer evaluations.

Critical weaknesses

Major Concern 1: Mismatched supplementary materials. The supplied code and logs concern Qwen2.5 math evaluation rather than MemPO or its reported benchmarks, contrary to the reproducibility statement. The reported results therefore cannot be verified from these artifacts. Please provide the corresponding implementation and experimental records.

Major Concern 2: The memory statistics are inconsistent. Table 1 specifies a raw LoCoMo stream of 920K tokens and a MemPO store of 13K, whereas Table 4 reports a mean stream length of 9.2K. Relative to the latter, 13K is approximately 141%, rather than the claimed 1.4%. These quantities must be reconciled to validate the compression claim.

Major Concern 3: The bias bound needs clarification. Appendix B acknowledges that splitting rewards among writes can introduce bias. However, Proposition 2 only accounts for violations of the retrieval-locality assumption. Even when this assumption holds perfectly δ=0\delta=0, reward splitting can still introduce bias. The authors should clarify how the bound accounts for this additional source of bias.

Minor Concern 1: The related-work discussion in the main text is way too brief and would benefit from clearer positioning against the closest prior methods. Additionally, the manuscript uses numerical citations rather than the author–year format specified by the ICLR template, which should be checked and corrected.

Questions for authors

My main concerns revolve the weaknesses stated above. I am willing to improve my score if the provided artifacts support the reported findings and these concerns are satisfactorily resolved.

Reviewer_znd5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MemPO, a reinforcement learning framework that optimises memory writing policies with downstream query/retrieval feedback. For each incoming experience segment, the policy selects four operations. To address delayed rewards, MemPO introduces retrieval-attributed credit assignment. It applies Shapley values to distribute query rewards among retrieval memory items and traces these contributions back to the write decisions through memory provenance. Experiments across six benchmarks and four reader models show improvements over the evaluated memory baselines and alternative credit-assignment strategies. Additional analyses investigate the learned writing behaviour, efficiency etc,.

Critical strengths
  1. I think the credit assignment is attractive to some degree.

The paper tries to address the challenge in learning memory write policies: how to associate delated downstream query/retrieval rewards with earlier memory operations. The method is plausible to some degree. The controlled comparison in Table 1 and 2 shows improvements over other strategies.

  1. The experiment and analysis are extensive to some degree.

The paper evaluates MemPO across multiple benchmarks and reader models, with additional experiments on retrieval configurations, policy operations, and training strategies. What is more, the analysis provides useful observations about credit quality, retrieval competition, and learned retention behaviour, etc.

Critical weaknesses
  1. I am confused about the research gap and methodological novelty. They might not be sufficiently established relative to existing RL-based memory management and credit assignment methods.

The introduction largely motivates MemPO by comparing fixed memory writing rules with learned policies. However, learning memory management decisions from downstream feedback has already been investigated in prior work. For example, Mem-T explicitly addresses sparse and delayed rewards in long horizon memory operations through memory operation tree backpropagation. MemRL also investigates learning memory utility from downstream retrieval feedback, etc. So I think only says MemPO is different from fixed policy rules is not enough/attractive for the motivation.

In addition, the paper does not fully distinguish its central credit assignment from these closely related approaches. While Table 2 compares several strategies, it does not say how Shapley-based retrieval attribution differs from existing memory hindsight credit assignment. Therefore, it remains unclear whether the main contribution is a fundamentally different credit-assignment mechanism, a refinement of retrieval-based hindsight attribution, or a particular combination of existing techniques.

What is more, the author says that prior memory systems rely on fixed write rules that discard potentially useful information (line 45-46), is little overly board I think. Some existing systems preserve original evidence alongside structured representations such as xMemory. This weakens the motivation based on irreversible information loss.

  1. The justification of write level credit attribution and the interpretation of the empirical results might remain incomplete.

The methods firstly apply Shapley values to assign attribution to retrieved memory items and seem to distribute each item's contribution uniformly among the write operations in its provenance (Equation 4). However, I am confused that, different writes contributing to a merged memory item may have different relevance to a downstream query, therefore, equal division of item level credit does not necessarily provide accurate write level attribution.

Moreover, Appendix Table 7 shows that MemPO's advantage over the strongest fixed policy decreases from 5.3 to 3.7 F1 points as retrieval improves. This suggests that part of the benefit may be associated with the limitations of the retrieval configuration, rather than generally better preservation of future useful information.

Questions for authors
  1. Positioning/difference relative to existing memory credit assignment.

How does MemPO's retrieval attributed credit differ from existing credit assignment methods such as Mem-T, MemRL, etc.? What are their shortcomings, and how could MemPO mitigate them with Shapley values and provenance tracing? Can the authors provide a more direct methodological comparison or an appropriately adapted experimental baseline?

  1. Theoretical validity of write level attribution.

Equation 4 uniformly divides a merged memory item's Shapley contribution among all writes in its provenance. How is this justified when the original writes contribute unequally to the queried information? Would a source-level marginal-contribution or counterfactual attribution provide different training signals?

  1. Interpretation and fairness of the retrieval comparisons.

Given the decreasing advantage under stronger retrieval in Appendix Table 7, could the authors further clarify which improvements arise from learned retention and which primarily compensate for limited retrieval precision?

  1. Dataset and memory-size consistency.

I am confused about the dataset size. Appendix Table 4 says Locomo has 50 streams and 7512 questions. But LoCoMo only has 10 long conversations and nearly 1500 question (except the fifth category). And longmemeval only has 500 questions while the table says 1500 queries. Could the author explain the difference?

Additionally, Appendix Table 4 reports a mean stream length of 9.2K tokens, while Table 1 describes a fully stored LoCoMo stream as 920K tokens and reports 13K tokens for MemPO, corresponding to approximately 1.4% retention in Table 3. Could the authors clarify the aggregation units and reconcile these memory-size statistics?

Overall, I think the problem is meaningful and appreciate the experimental effort. However, my concerns concern the core framing, method justification, and empirical positioning, rather than presentation alone, and may be difficult to fully address through clarification alone.

1 comment

Thank you for the detailed response. I appreciate the clarifications, which have addressed some of my concerns. However, my main concerns regarding the research motivation and method novelty remain insufficiently addressed. Therefore, I will maintain my rating.

Reviewer_rEDTOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MemPO, a framework for learning memory write policies from downstream question-answering rewards. A policy processes a history segment by segment, choosing KEEP, ABSTRACT, MERGE, or DROP before seeing the questions. A frozen retriever and reader subsequently answer questions using the resulting memory. MemPO computes Shapley contributions over retrieved items and routes these contributions through their provenance to earlier write decisions. The paper reports results on six benchmarks with four readers, including LoCoMo F1 of 44.7 versus 39.5 for a hand-designed policy over the same operations.

Critical strengths
  1. The method exploits a useful structure in memory-based question answering. Splitting feedback among retrieved evidence and tracing it to earlier writes offers a concrete way to address delayed supervision without assigning every write the same trajectory reward.
  2. The paper includes relevant controlled comparisons. Tables 2 and 3 compare alternative credit signals and policies over the same operations, rather than relying solely on comparisons with external systems. The reported improvement from leave-one-out credit (41.1 F1) to Shapley-based credit (44.7) supports the usefulness of the attribution choice within MemPO.
Critical weaknesses
  1. The training procedure is not fully specified in the main text. Figure 2(c) defines the write return with a cost penalty, −λc(at)-\lambda c(a_t), but this term is absent from Equations (2) and (4), and Section 4 states that the policy learns to write little "from the reward alone." Whether the penalty is used in the reported experiments therefore changes how that finding should be read. In addition, Section 2.3 states that evaluating an unselected candidate operation "requires only the memory it would produce and the retrieval that memory induces," which suggests that later writes are not re-run under each candidate. Because an action at step tt can change subsequent writes and MERGE targets, such an evaluation ignores these downstream effects and need not estimate the candidate's return. Since these choices determine the learning signal, the main text does not allow the reported credit-assignment results to be fully interpreted or reproduced.
  2. The validation of write-level credit is ambiguous for merged items. Figure 3a and Table 2 evaluate each credit scheme by its Spearman correlation with a write's "true worth," obtained by deleting it from the memory and re-answering every query. However, Section 2.2 allows MERGE to combine several writes into one item, whose credit Equation (4) divides equally among them. For such items, deleting the item removes the contribution of every write that formed it, so the reference quantity does not isolate one write's contribution, and the equal split in Equation (4) is assumed rather than validated. Since this correlation (0.86 for retrieval-attributed credit versus 0.71 for leave-one-out) is a key piece of evidence that the proposed credit tracks write utility, this ambiguity limits how strongly the results support the paper's central claim.
Questions for authors
  1. (Clarification) Is the cost penalty −λc(at)-\lambda c(a_t) in Figure 2(c) used in the reported experiments? If so, please define cc and λ\lambda and reconcile the objective with Equations (2) and (4) and with the statement in Section 4 that compact memory is learned from the reward alone.
  2. (Clarification) Section 2.3 states that evaluating an unselected candidate requires only "the memory it would produce." Is the candidate's return Gt(i)G_t^{(i)} computed on the memory at step tt, or on the final memory with the candidate's effect substituted and later writes held fixed? How are the candidate's effects on subsequent writes and MERGE targets accounted for?
  3. (Clarification) Table 3 attributes 4.5 F1 to shadow memory, which Appendix A.1 describes as a copy of discarded segments used only to evaluate DROP. How exactly is the return of DROP computed from this copy, given that DROP creates no item and has no provenance under Equation (4)?
  4. (Clarification) In the "true worth" measurement of Figure 3a, how is a single write removed when its content has been merged with other writes? Does the measured quantity reflect an item's deletion utility or an individual write's contribution?
  5. (Clarification) Table 1 states that a LoCoMo stream stored in full would read 920K tokens, whereas Appendix Table 4 reports a mean stream length of 9.2K tokens. Which is correct? The claim that MemPO's store is 1.4% of the stream, and the 13K-token store size in Table 1, depend on this value.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.