acceptodds
Under review as a conference paper at ICLR 2027

Causal Memory Policy: Making Memory Utility Identifiable by Intervening on Retrieval

Abstract

Memory-augmented large language models must decide which memories to retain, and recent systems do so by estimating each memory's effect on task performance. However, these estimates rely entirely on retrieved memories. When a memory is never retrieved, store-level interventions produce identical outcomes, leaving its utility unidentified. This is a retrieval-level positivity violation, invisible to diagnostics that examine only memory operations. We introduce Causal Memory Policy (CMP), a causal framework that restores identification by intervening on retrieval itself, reserving a fixed number of context slots for memories sampled with known propensity scores. CMP estimates memory utility via self-normalized inverse propensity weighting under a balanced assignment design with fixed sampling propensities. We prove the causal factorization of memory utility through retrieval, the unbiasedness and exact variance of the estimator, and the optimal decision rule under irreversible operations. Empirically, identification fails for of required memories on LongMemEval and on LoCoMo, and the failure persists in a deployed memory system. CMP improves discrimination between required and non-required memories from to AUC. Finally, we show that identified memory utility alone is insufficient for retention decisions: per-query utility reaches AUC on the query for which it is estimated. Code is available at: https://anonymous.4open.science/r/cmp-release-D0C3/.

Primary area:
causal reasoning
Keywords:
LLM memory, causal identification, memory-augmented agents, retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LcruOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

CMP randomizes retrieval exposure to measure memories that ordinary retrieval never surfaces. Exposed and unexposed outcomes estimate their contribution. Experiments improve required-memory discrimination, while showing that exposure-pool construction and aggregation across queries remain obstacles to retention.

Critical strengths
  1. Distinguishing untested memories from genuinely unhelpful ones is useful. The support-stratified results locate where randomized exposure adds information.
  2. The paper is candid about oracle pool construction and unsuccessful cross-query transfer. These negative results help define the method's actual scope.
Critical weaknesses
  1. The store-level effect U is zero when retrieval remains unchanged; that effect is identified. It is forced-exposure utility U_R that is unidentified. The paper repeatedly conflates the two. Proposition 1 also needs a retrieval-stability assumption beyond Definition 1 to justify a single replacement.
  2. Balanced assignment fixes exposure counts, but Eq. 6 defines Z as actual retrieval, which can include deterministic top-ranked memories. Appendix E.4 confirms that some have no excluded arm. Thus inclusion positivity alone does not support Proposition 2 for every pool member; the indicator and both-arm support conditions need correction.
  3. Poor transfer of the tested aggregations does not establish that no aggregation can predict future utility. That conclusion requires assumptions about the future query distribution. The oracle exposure pool further limits what the empirical gains establish for deployment.
Questions for authors
  1. Clarification: which intervention does the implemented estimator target, and how are deterministic inclusions handled?
  2. Clarification: under what query-distribution assumptions is the general cross-query impossibility claim valid?
1 comment
AuthorsOpenReview ↗

Response to Reviewer Lcru

We thank the reviewer for the precise comments.

W1(a) (UU vs. URU_R). We agree. The revision states after Proposition 1 that U(m∣q,c)U(m\mid q,c) is identified and equals 0 when mm is not retrieved, and that the unidentified quantity is URU_R (Sec. 2.3).

W1(b) (retrieval stability). We agree. Proposition 1 now rests on Assumption 1 (score-separable retrieval, new) and Lemma 1 (new, App. C.1): adding or removing one memory changes the retrieved set by at most one element. Under this assumption, Proposition 1 is an exact identity at a fixed store. Cosine top-BB retrieval satisfies Assumption 1; LLM-mediated retrieval, as in Mem0, need not.

W2 and Q1 (indicator, both-arm support, target). We agree. Assumption 2 (the submission's Assumption 1) now requires 0<P(m∈Rt)<10<P(m\in R_t)<1, and the design satisfies it for every pool member (Sec. 3.1):

  • Unretrieved memories receive the exposure arm. The ranked slots never reach them, so their retrieval indicator equals the assignment. The target is the effect of mm relative to a pool memory drawn by the design (Prop. 2(i)).
  • Retrieved memories, including deterministic top-ranked ones, receive an exclusion arm. Each is removed from the store with probability pflipp_{\text{flip}} and replaced by the next-ranked surviving memory. The target is the ranked-replacement effect of Eq. 3, averaged over the perturbed stores (Prop. 2(ii)).

On all 936 pool memories, CMP reaches 0.661 AUC, compared with 0.541 for store-level randomization (paired 95% CI [0.05, 0.19]; Table 2).

W3(a) and Q2 (cross-query claim). The claim fails under a stationary query distribution, where the query average estimates expected future utility. It would hold only if the observed queries carried no information about future ones, an assumption the paper neither stated nor tested. We removed the claim. Corollary 1 (new, Sec. 3.3) instead bounds the sample cost of the query average. On LoCoMo, a memory required by one question in thirty needs about 7,200 draws per arm, compared with 8 for the per-query contrast. The transfer results are reported as consistent with this cost.

W3(b) (oracle pool). We built an annotation-free pool from the query and the retriever alone: the six retrieved memories and cosine ranks 7–50 (Sec. 4.2, App. H.9). It contains 88% of the required memories and 76% of those never retrieved.

  • Against evidence labels: CMP reaches 0.642 AUC and store-level randomization 0.562. The paired 95% CI is [−0.003, 0.171] in the pre-specified run and [0.009, 0.164] in an independent replicate.
  • Against exact causal effects, with no annotations in the pool or the evaluation: CMP reaches 0.708 AUC and store-level randomization 0.592 (paired 95% CI [0.048, 0.230]).

The gain is smaller than with the annotated pool, as the limitations state (App. J).

Reviewer_ZainOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies memory utility estimation when retrieval prevents stored memories from reaching the generator. CMP reserves context slots for randomized exposure and estimates utility through differences in outcomes. It also derives an eviction threshold under asymmetric losses. On the selected LongMemEval subset, reported discrimination between required and non-required memories improves from 0.542 to 0.664 AUC. A separate LoCoMo configuration reaches 0.784 AUC for query-specific estimates. The experiments use exposure pools containing annotated required memories. The paper explicitly limits its contribution to measurement and does not establish a deployable retention policy.

Critical strengths

The paper identifies a useful distinction between retrieval failure and conditional usefulness. Store perturbations can leave the generator’s context unchanged, providing no evidence about an excluded memory’s potential contribution. The support analyses and worked examples make this problem concrete. This observation matters when systems interpret small estimated effects as evidence that deletion is safe.

The experiments expose important limits of utility-based memory management. The paper reports oracle pool construction, missing exclusion arms, exposure costs, and failed transfer across queries. These observations could support a useful empirical contribution with narrower claims and corrected theory.

Critical weaknesses
  1. The exposure design does not establish Proposition 2’s stated unbiasedness and exact variance.

    Equation 3 defines utility through a specific replacement intervention: insert the target memory and displace the lowest-ranked memory. Equation 6 instead compares contexts containing the target with contexts containing different sampled alternatives. These contrasts need not estimate the same quantity.

    Consider one query, B=k=1B=k=1, and pool {a,b,m}\{a,b,m\}, ranked a>b>ma>b>m. Define the outcomes as Y({a})=Y({m})=0,Y({b})=1.Y(\{a\})=Y(\{m\})=0,\qquad Y(\{b\})=1. The default context contains aa. Equation 3 therefore gives UR(m)=Y({m})−Y({a})=0U_R(m)=Y(\{m\})-Y(\{a\})=0. Under balanced exposure, Equation 6 instead gives U^R(m)=0−0+12=−12.\widehat U_R(m)=0-\frac{0+1}{2}=-\frac12. The estimator measures replacement against a sampled alternative, rather than the ranked alternative specified in Equation 3.

    The same example contradicts the exact variance formula. With T=6T=6 and two exposures per memory, every balanced schedule gives −1/2-1/2, so the estimator’s variance is zero. Proposition 2 instead gives

    =\frac{0}{2}+\frac{1/4}{4} =\frac1{16}.$$ Fixed exposure counts do not make outcomes independent across draws. There is also an assignment mismatch. Section 3.1 fixes counts for randomized exposure, but Section 3.2 defines $Z_m^{(t)}$ using actual retrieval. A memory can enter through the deterministic ranker without randomized exposure. Appendix H.1 explicitly reports a memory with 90 inclusions and no exclusions. Thus, the stated exposure counts and propensities do not describe actual retrieval for every pool member. Assumption 1 requires only positive inclusion probability, which does not guarantee an exclusion arm. These issues affect the central measurement guarantee and the uncertainty estimates used for eviction. Resolving them requires a precise estimand, a matching exposure mechanism, and a variance derivation that accounts for the assignment design.
  2. The empirical results do not establish the claimed explanation for query-specific gains or the general failure of aggregation.

    The LoCoMo comparison changes more than the estimand. Table 6 changes kk from 2 to 6 and changes the allocation of draws across queries. Appendix E.4 explains that this removes the deterministic ranker and creates previously missing exclusion arms. Consequently, the AUC increase cannot isolate the benefit of query-specific estimation.

    Moreover, AUC against evidence annotations does not directly validate causal-effect estimates. Appendix H.1 demonstrates this distinction: an annotated required memory can have negative measured utility. Better discrimination of evidence labels therefore does not establish unbiased estimation of the stated causal effect.

    The transfer results support a narrower conclusion: the tested aggregations perform poorly under the reported protocol. They do not establish that every admissible aggregation must fail. Under a representative stationary query distribution, averaging can estimate expected future utility. A small average effect is not itself a defective estimand. The paper needs to distinguish the absence of generalization assumptions from an inherent impossibility of aggregation.

    Finally, five observations per arm do not establish adequate statistical power. The reported coverage therefore cannot justify ruling out estimation error as an explanation for weak transfer. A comparison with matched exposure designs and sampling budgets, together with uncertainty estimates, would help distinguish these explanations.

Questions for authors
  1. What exact intervention does Equation 6 estimate, including the distribution of replacement memories? How does it match Equation 3?
  2. How does Proposition 2 handle deterministic inclusions and dependence from balanced assignment? Can you reconcile its guarantees with the counterexample above?
  3. Is there a LoCoMo comparison that holds exposure design and per-query sampling budget fixed? What supports ruling out estimation error in the transfer results?
1 comment
AuthorsOpenReview ↗

Response to Reviewer Zain

We thank the reviewer for the detailed review. The counterexample identified a real error, and we adopt the reviewer's framing of aggregation.

W1(a) and Q1 (estimand of Eq. 6). The counterexample is correct: Eq. 6 and Eq. 3 are different contrasts. Proposition 2 now separates them (Sec. 3.2):

  • (i) For an unretrieved memory mm, conditional on the exposure counts, the exposure estimate is unbiased for URπ(m∣q)=Eπ[Yt∣m∈Et]−Eπ[Yt∣m∉Et]U_R^\pi(m\mid q)=\mathbb E_\pi[Y_t\mid m\in E_t]-\mathbb E_\pi[Y_t\mid m\notin E_t], where EtE_t is the exposed set. When mm is not exposed, the reserved slots hold other pool memories assigned by the balanced design π\pi. This is the replacement distribution.
  • (ii) For a retrieved memory, conditional on both arms being non-empty, the exclusion contrast is unbiased for the ranked-replacement effect of Eq. 3, averaged over the randomly perturbed stores. A removed memory is replaced by the next-ranked surviving memory (Assumption 1, Lemma 1).

The two coincide when memories contribute additively and the two replacements contribute equally in expectation. Complementary memories, as in the counterexample, violate this. On LongMemEval, the exposure estimate differs from the exact Eq. 3 effect by 0.13 in mean absolute value (Sec. 4.2); this gap includes sampling error and generator drift (App. H.10). Where the target is Eq. 3 itself, exclusion estimates of the retrieved memories track the exact effect (AUC 0.97, r = 0.81; App. H.9).

W1(b) and Q2 (dependence). The counterexample is correct and is now in the paper (Sec. 3.2). Proposition 2(iii) states the variance formula only for independent draws; under balanced assignment it is an approximation. Unbiasedness (2(i)) does not require independence. We validated the standard error by re-randomization with four new assignment seeds (251 memories). The empirical SD is 1.04× the reported SE, and 96.3% of 1,480 pairwise 95% intervals cover (Sec. 4.2, App. H.9).

W1(c) and Q2 (deterministic inclusions). CMP now uses the exposure arm only for memories the ranker does not retrieve. The ranked slots never reach these memories, so their retrieval indicator equals the assignment (Sec. 3.1; App. F describes how this applies to the main run). Memories the ranker retrieves, including deterministic top-ranked ones, are estimated from an exclusion arm. The App. H.1 memory with 90 inclusions and no exclusions is now absent on 12 of 90 draws.

W2(a) and Q3 (LoCoMo with fixed design and budget). We added the requested comparison (Sec. 4.3). On the same draws, with all BB slots randomized, per-query estimates reach 0.784 AUC. Pooling those draws across the questions of a conversation gives 0.635 (paired difference +0.149, 95% CI [0.10, 0.21], resampling conversations). The pooled estimates use about 30× more draws per memory, so the gap is not due to the sampling budget.

W2(b) (AUC against annotations). We agree. Annotations and causal effects diverge: 27% of required memories have an exact effect above 0.1, and the exact effect separates required from non-required memories at only 0.589 AUC. We therefore added validation against exact effects (Sec. 4.2, App. H.9). CMP reaches 0.757 AUC, compared with 0.567 for store-level randomization (paired 95% CI [0.11, 0.27], computed post hoc). With the annotation-free pool, the AUCs are 0.708 and 0.592 (paired 95% CI [0.048, 0.230]).

W2(c) (aggregation). We agree. The claim that no aggregation can predict future utility is removed. Under a stationary query distribution, the query average is the appropriate retention target. The transfer results are now reported as consistent with its sample cost (W2(d)).

W2(d) and Q3 (power). Corollary 1 (new, Sec. 3.3) gives this cost. Suppose mm changes the outcome by Δ\Delta on a fraction ρm\rho_m of queries. A two-sided level-α\alpha test of the query average then needs 1/ρm21/\rho_m^2 times more draws than the per-query contrast at the same within-arm SD. On LoCoMo, consider a memory required by one question in thirty (Δ=0.30\Delta=0.30, within-arm SD 0.214, α=0.05, 80% power). The test needs about 7,200 draws per arm, compared with 8 for the per-query contrast. Our five draws per arm are of the same order as the per-query requirement and far below the query-average one.

Reviewer_i3fzOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to estimate the contribution of individual memories in memory-augmented LLM systems. The authors argue that existing utility-estimation approaches can fail when a memory is never retrieved, since adding or removing that memory from the store may not change the context presented to the LLM. To address this, the paper proposes Causal Memory Policy (CMP), which introduces randomized retrieval exposure. Instead of relying entirely on the retriever's ranking, CMP reserves a fixed number of context slots for randomly selected memories with known sampling probabilities. It then uses a propensity-based estimator to estimate their contributions to downstream task performance. The paper also develops a decision rule accounting for the irreversible nature of forgetting.

Critical strengths
  • Reframing memory retention through the lens of causal mediation is probably the strongest aspect of the paper. Pinpointing that positivity failures occur at the retrieval mediator level rather than the store level explains why existing exploration methods (e.g., store-level operation-level) fail to detect or resolve non-identifiability.
  • Theorem 2 provides a sound Bayes-optimal decision rule tailored to the asymmetry of memory management.
  • The empirical analysis isolates the exact mechanism responsible for the performance gap. The paper identifies a meaningful limitation in estimating memory usefulness from retrieved evidence alone.
Critical weaknesses
  • The central novelty needs to be further clearer. Randomizing what is shown is a standard remedy for exposure bias in search and recommendation.
  • The title is misleading. It says “Causal memory policy”; however, there doesn’t seem to be a policy proposed.
  • The paper aims to address memory management problems, but it is unclear how the estimated per-query utility can contribute to practical memory management. The authors acknowledge that these estimates cannot directly support long-term retention decisions, as their usefulness does not reliably transfer to unseen queries. This raises concerns about the practical value of the proposed framework.
  • The paper defines memory utility through an intervention that inserts or removes a memory from the retrieved context using a specific ranking-based replacement rule. However, CMP estimates utility under randomized retrieval, where the other context slots may contain different memories, potentially measuring a different causal effect.
  • The proposed method requires observing a memory both included and excluded from the retrieved context, but Assumption 1 only guarantees a positive inclusion probability. Memories consistently selected by the deterministic retriever may never be excluded, leaving their utility unidentified, as demonstrated in Figure 4. The theoretical estimator assumes fixed inclusion counts, but actual retrieval can also occur outside the randomized exposure slots.
  • What is the practical applicability of this method. The method assumes annotations for memory to construct the pool, which will not be available in the real world.
  • Although CMP improves memory-utility estimation, the paper does not demonstrate an effective end-to-end memory-retention policy or significant downstream improvements over existing memory policies.
Questions for authors
  • What is the intended practical use of the estimated per-query memory utility? Given that the authors acknowledge its limitations for future retention decisions, how can this measurement benefit a real memory management system?

  • The proposed approach relies on an exposure pool containing ground-truth required memories. How could such a pool be constructed in real deployments where the required memories are unknown? What benefits can CMP provide without this oracle information?

1 comment
AuthorsOpenReview ↗

Response to Reviewer i3fz

We thank the reviewer for the careful review.

W1 (novelty). We agree that randomized exposure is a standard remedy for exposure bias. Sec. 1 and App. A now cite this literature (Joachims et al., 2017; Schnabel et al., 2016; Ma et al., 2020). Our contribution is specific to memory systems:

  1. Theorem 1 shows that randomizing whether mm is stored, with the rest of the store fixed, cannot identify UR(m∣q)U_R(m\mid q) when the ranker does not retrieve mm.
  2. This case is common. It covers 54% of required memories on LongMemEval and 67% of (question, required memory) pairs on LoCoMo (Table 1).
  3. Eviction is irreversible, so the decision rule is one-sided (Theorem 2).

W2 (title). We agree. The method is renamed Causal Memory Probing (same acronym) throughout the paper. Since titles cannot change during the discussion period, we will adopt Causal Memory Probing: Identifying Memory Utility by Intervening on Retrieval in the camera-ready version.

W3 and Q1 (practical use). The per-query utility UR(m∣q)U_R(m\mid q) states whether mm would change the answer to qq if it were retrieved. It has two uses that need no transfer across queries:

  1. Auditing a deletion before it is executed. When the pool is ranked by CMP estimates, required memories appear among the 10 eviction candidates at 0.69× the chance rate (lower is better). Store-level randomization gives 0.97× (Table 2).
  2. Diagnosing retrieval failures. On memories the ranker never retrieves, CMP ranks causally useful memories at 0.746 AUC, compared with 0.435 for store-level randomization (Sec. 4.2). A useful memory that is not retrieved indicates a retriever error.

Retention across future queries needs the query-averaged utility. Corollary 1 (new, Sec. 3.3) quantifies its cost. On LoCoMo, a memory required by one question in thirty needs about 7,200 draws per arm, compared with 8 for the per-query contrast.

W4 (estimand). We agree that the two contrasts differ. Proposition 2 now states the target of each arm:

  • (i) The exposure arm identifies the effect of mm relative to a pool memory drawn by the design.
  • (ii) The exclusion arm identifies the ranked-replacement effect of Eq. 3, averaged over the randomly perturbed stores.

Sec. 3.2 states when the two coincide, and Sec. 4.2 measures their gap (see our reply to Zain, W1(a)).

W5 (both-arm support). We agree. Assumption 2 (the submission's Assumption 1) now requires 0<P(m∈Rt)<10<P(m\in R_t)<1, and the design satisfies it for every pool member (Sec. 3.1):

  • Unretrieved memories receive the exposure arm. The ranked slots never reach them, so their retrieval indicator equals the random assignment.
  • Retrieved memories, including those the ranker always retrieves (Figure 4), receive an exclusion arm. Each is removed from the store with probability pflipp_{\text{flip}} and replaced by the next-ranked surviving memory.

On all 936 pool memories, CMP reaches 0.661 AUC, compared with 0.541 for store-level randomization (paired 95% CI [0.05, 0.19]; Table 2).

W6 and Q2 (pool without annotations). We built an annotation-free pool from the query and the retriever alone: the six retrieved memories and cosine ranks 7–50 (Sec. 4.2, App. H.9). It contains 88% of the required memories and 76% of those never retrieved.

  • Against evidence labels: CMP reaches 0.642 AUC and store-level randomization 0.562. The paired 95% CI is [−0.003, 0.171] in the pre-specified run and [0.009, 0.164] in an independent replicate.
  • Against exact causal effects: CMP reaches 0.708 AUC and store-level randomization 0.592 (paired 95% CI [0.048, 0.230]). The exact effect is the Eq. 3 effect computed by forced insertion or exclusion at temperature 0, so annotations enter neither the pool nor the evaluation.

The gain is smaller than with the annotated pool, as the limitations state (App. J).

W7 (end-to-end gains). We agree and do not claim end-to-end retention gains. At λ=10, gold loss per 100 reclaimed slots is 7.5 for CMP and 9.9 for store-level randomization; the 95% CI of the difference includes zero ([−6.8, 1.7]; Sec. 4.2). Re-answering every question after eviction under the submission's settings shows no detectable difference in answer F1 between eviction by exposure estimates and by store-level estimates (Δ=−0.023, p=0.53; App. H.11). The submission's claim that CMP halves gold loss came from an unnormalized estimate and a threshold applied at λ≈13 instead of the stated λ=10. That claim is removed (App. I.2).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for their constructive comments. Each reply addresses the reviewer's weaknesses (W) and questions (Q) in the order of the review; (a), (b), … follow the order of the points within a weakness. In the revised PDF, changed text is shown in blue. The submission's Assumption 1 (Retrieval Positivity) is now Assumption 2 and requires overlap in both directions.

The central claim is unchanged. Randomizing whether a memory is stored, with the rest of the store fixed, cannot identify its utility once retrieved, URU_R, when the retriever does not surface it (Theorem 1); randomizing retrieval can (Proposition 2). On the memories the ranker never retrieves, CMP reaches 0.659 AUC, compared with 0.469 for store-level randomization (paired 95% CI [0.06, 0.32]; Table 3).

Four concerns were shared across reviews:

  • Estimand of the exposure design: i3fz W4; Zain W1(a), Q1; Lcru W2, Q1.
  • Missing exclusion arms: i3fz W5; Zain W1(c), Q2; Lcru W2, Q1.
  • Annotated exposure pool: i3fz W6, Q2; Lcru W3(b).
  • Cross-query claim: i3fz W3, Q1; Zain W2(c), W2(d), Q3; Lcru W3(a), Q2.

While revising, we audited all reported numbers against the draw logs; App. I.2 lists the corrections. Two change the submission's content: the gold-loss claim (i3fz W7), and an observational (AIPW) comparator and the diagnostic built on it, described in Sec. 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.