acceptodds
Under review as a conference paper at ICLR 2027

What Did the Agent Learn From? Cross-Episode Memory Attribution for Self-Improving Agents

Abstract

Agentic attribution identifies which component of an LLM agent system caused an observed success or failure. This supports debugging, accountability, and targeted improvement. Existing methods are largely within-episode: they assign outcomes to agents, actions, tool calls, or prompt context in the current trace. Persistent memory agents break this assumption because the relevant cause may not appear in that trace. A memory written many episodes earlier can affect an outcome directly through retrieval or indirectly through memories written later. We introduce memory-level attribution for self-improving agents and formalise these two paths as retrieval influence and generation influence. If no allowed intervention retrieves a memory, retrieval-only data may contain no evidence that distinguishes it from an absent memory. In our deployed retrieval configuration, individually deleting the never-retrieved memories leaves every measured top-k list unchanged. Retrieval-only data therefore do not record effects passed through later memories in these cases. This limitation matters at scale. At the deployed retrieval budget, 87–92% of a 278-memory pool is never retrieved. We therefore propose a visibility-aware attribution framework that matches the attribution unit to what can be observed. MemAttr-V computes individual-memory Shapley values on the visible set, and MemAttr-B computes lineage-band Shapley values on the observed shadow. Across three environments and four agent LLMs, MemAttr-V’s correlation with measured leave-one-in utility is positive in every evaluated cell, while MemAttr-B’s mean correlation is positive in every environment within task splits. We release the memory pools, generation lineages, and measured targets as MemAttr-Bench.

Primary area:
interpretability and explainable AI
Keywords:
memory attribution, LLM agents, self-improving agents, credit assignment, identifiability, Shapley value, agentic attribution, benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zpxoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates memory-level failure attribution for self-improving LLM agents that write and reuse persistent memories across episodes. The motivation is that existing attribution work, for both single-agent and multi-agent systems, attributes outcomes within one episode, and for memory it only sees the items retrieved into the current prompt. The paper defines two cross-episode influence paths, retrieval influence and generation influence. Retrieval influence is the change in success when one memory is deleted from the pool while every memory written after it stays fixed. Generation influence is the change when the memory is deleted and all later memory writes are regenerated without it, so it also captures effects passed through the memories that descend from it. The authors then argue that retrieval-only attribution has a blind spot. On a 278-memory pool, 87–92% of memories are never retrieved at k=3, and deleting any single never-retrieved memory leaves all 7,905 measured top-3 lists unchanged. On one pool, removing a lineage root and regenerating its descendants changes held-out success by up to 0.292.

Based on this, they propose two estimators. MemAttr-V computes exact Shapley values over the 4–6 memories that retrieval surfaces. MemAttr-B groups memories into lineage bands by their source ancestor in the logged generation DAG. It computes band-level Shapley values with permutation Monte Carlo, without regenerating memories. Both are evaluated by Spearman correlation with measured leave-one-in utility on ALFWorld, ScienceWorld and τ-bench. Experiments show that both estimators correlate positively with measured leave-one-in utility, while a pruning test finds that deleting the negative-utility memory does not beat random deletion. The main contribution is framing cross-episode memory attribution, separating direct retrieval effects from effects passed through later memories, and releasing MemAttr-Bench with the memory pools and their generation lineages.

Critical strengths
  1. The paper distinguishes single-episode from cross-episode failure attribution for agent memory and gives a clear framing of two influence paths, retrieval influence and generation influence. This framing is insightful and useful. It explains why existing trace-level and context-level attribution methods cannot see a memory whose effect only reaches the outcome through later memories, and it makes explicit which counterfactual each quantity requires.
  2. The method follows a clear design principle, the attribution unit should match what the system can actually observe and intervene on. The concept of generation DAG is quite interesting. Individual memories are scored only where retrieval provides item-level evidence (MemAttr-V), and memories without such evidence are grouped into lineage bands from the logged generation DAG (MemAttr-B).
  3. The paper provide concrete measurements and artifacts. Their MemAttr-Bench releases the generation lineage on which memories were in context when each memory was written. Together with the paper's recommendation to record this lineage, this gives future work a basis for studying generation influence directly.
Critical weaknesses
  1. My main concern is that MemAttr-B does not solve the problem raised by the paper itself. The paper's main motivation is that some memories exert their influence indirectly through their descendants, but current methods cannot observe this, so it aims to use MemAttr-B to observe this indirect influence. However, MemAttr-B's approach is to remove an entire memory family from the pool, observe the change in accuracy and use Shapley values to assign credit. What this actually measures is the value of this existing set of memories, rather than the indirect influence we expect.
  2. The paper's evaluation of whether memory is useful relies entirely on task success rate, which is reasonable as the ultimate goal, but this signal has somewhat low resolution. First, at the split sizes used in the paper, each task only has a 0 or 1 outcome, and results from different seeds often end up tied, or both the full pool and the empty pool have a success rate of 0. More importantly, a memory may clearly change the agent's behavior, but the final task outcome remains unchanged, and for the attribution this paper aims to achieve, this signal is actually important.

Some other minor points. First, figure 1 contains some blurry and unnatural textures, it is recommended to make adjustments or refinements after using tools to assist with figure creation. Second, there are some mismatches between captions and figures and inconsistent use of terminology in the text, which makes the reading experience less smooth.

Questions for authors
  1. Of the 87–92% never-retrieved memories, how many are actually relevant to the 31 evaluation goals? How large is the shadow if the evaluation goals cover the task types present in the 278-memory pool?
  2. The same environment gets different MemAttr-V numbers in different places. For example, ScienceWorld is +0.27 in Table 10, +0.58 in Table 11, and [0.58, 1.0] in Table 4. Which pools and splits does each number use?
2 comments
AuthorsOpenReview ↗

Response to Reviewer zpxo

Thank you for the review. We hope to address your concerns accordingly.


W1 – Does MemAttr-B capture indirect influence?

By design, it values the existing set. As §5 states, MemAttr-B is a fixed-pool estimator: it values a group of already-written memories, not the per-memory generation influence of its source. The attribution unit follows what the system can observe and intervene on: memories without per-memory evidence under full-pool retrieval are grouped into their lineage band (a singleton test can still value them individually; our reply to Reviewer En1S, W1 & Q1 (c)), and the logged lineage lets that band be rewritten from its root, which generation influence requires (Definition 3.1). When a planted harmful memory shapes outcomes through its descendants, the band finds that group, as the positive control shows: the planted root, never retrieved, acts only through its descendants (Table R5). The singleton test measures the root's stand-alone utility; the root-kept versus root-removed rewrite measures its influence through later writes. The rewrite uses a consolidation writer that restates the root's claim, so the +0.14 holds for this controlled setting. With the multi-replay study (our reply to Reviewer 7u3B, Q3 (b)), this control replaces the submitted single regeneration (0.292, D.7), which does not reproduce; §4 will report it.

Table R5: Three quantities in the positive control (Table R1 in our reply to Reviewer En1S).

QuantityResult
Band value in the fixed pool (band Shapley, band deletion)planted lineage lowest (−0.11); deleting it: 0.22 → 0.44 (p=0.016p=0.016), about as much as deleting only its descendants (+0.24)
Root's stand-alone utility (singleton, vs. the empty pool)−0.16 (0.18 → 0.02); deleting the root from the full pool: −0.04
Influence through later writes (rewrite without vs. with the root)+0.14 (6 replays per arm, p=0.016p=0.016)

W2 (a) – Is binary success too coarse, with ties and all-zero games?

Graded targets reduce the problem: steps-discounted success on ALFWorld and the native ScienceWorld score cut ties in the ALFWorld and ScienceWorld LOI targets from 5 to 3, and MemAttr-V reaches ρ = 0.69 against the graded, independently re-executed target over 9 cells (0.75 without τ-retail, whose target is not reproducible; τ-bench stays binary). The per-cell table in the artifact lists v(full) and v(empty) for every cell, so cells in which both score 0 are visible.


W2 (b) – Can a memory change behaviour without changing the outcome?

Yes, often. We re-ran ALFWorld with full action logging (the empty pool, each memory alone and the full set; 15 tasks × 3 repetitions). In 65–77% of singleton-versus-empty pairs the outcome is the same, yet their mean normalized edit distance is 0.69–0.71, against 0.25–0.28 for same-outcome empty-pool reruns, and the gap holds on all 15 tasks (one-sided sign test over tasks, p<0.001p<0.001). So memories change behaviour well beyond what success records. §6 will report both analyses, and §3 will separate success utility from behavioural influence.


W3 (minor) – Figure 1 textures, caption–figure mismatches and terminology

We will remove the rendering artifacts in Figure 1, fix the caption–figure mismatches, and align the terms with Definitions 3.1–3.2.


Q1 (a) – How many never-retrieved memories are relevant to the 31 evaluation goals?

By task type, 238 of the 278 memories are relevant to the 31 goals, and 85–90% of them (202–215) are never retrieved at k=3k=3 (Table R6); the other 40, τ and ScienceWorld memories, are never returned by the deployed token-overlap retriever.


Q1 (b) – How large is the shadow when the goals cover every task type in the pool?

Still large: with 59 goals covering every task type, 78–80% stay unretrieved at k=3k=3 (Table R6). We will add these rows to Table 2.

Table R6: Never-retrieved fraction (k=3k=3 / k=16k=16, range over four retrievers).

Pool and goalsNever retrieved
Submitted: 278 memories, 31 goals87–92% / 54–62%
238 ALFWorld memories, 31 goals85–90% / 47–58%
Goals covering every task type in the pool (59 goals)78–80% / 34–44%

Q2 – Which pools and splits does each ScienceWorld number use?

Each number pools different cells (Table R7). Table 12 and its footnote will give the windows each cell uses (variations 100–105 for melt and find-non-living-thing, 0–7 for v0_7), and every reported ρ will state its pool and split.

Table R7: ScienceWorld MemAttr-V numbers.

NumberPoolsNote
Table 4: [0.58, 1.0]melt +0.58, find-non-living-thing +0.82, v0_7 +1.00the find-non-living-thing pool file is lost, so that cell was not re-executed; its coalition outcomes are in the artifact
Table 11: +0.58melt—
Table 10: +0.27meltthe singleton-removed value; the full-Shapley entry should read +0.58
AuthorsOpenReview ↗

Follow-up on our response

Thank you again for your time and effort in reviewing our paper. As the discussion period nears its end, we would be grateful to know whether our responses address your concerns about whether MemAttr-B captures indirect influence (W1), the coarseness of binary success (W2), the size of the shadow once relevance is controlled (Q1), and the ScienceWorld pools and splits (Q2). If any point would benefit from further clarification, we would be happy to answer it. If our responses have resolved your concerns, we would be grateful if you would consider updating your rating accordingly.

Reviewer_7u3BOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies cross-episode memory attribution in self-improving agents with persistent textual memory. The authors distinguish two counterfactual notions: retrieval influence, which removes a memory while holding subsequent memories fixed, and generation influence, which removes that memory and replays subsequent memory-writing steps. They also distinguish observed shadows, which are not retrieved when the full memory pool is available, from unreachable shadows, which cannot be retrieved under a specified set of interventions.

To accommodate different visibility conditions, the paper proposes MEMATTR-V, which computes individual-memory Shapley values over the visible memory set, and MEMATTR-B, which treats generation-lineage groups as Shapley players. Both methods assess fixed-pool contributions within the realized memory pool, rather than directly estimating the generation influence of individual memories.

The evaluation includes retrieval-coverage and deletion-invariance measurements over 278 memories; attribution experiments on small memory pools across three environments and four agent LLMs; and a lineage-rewriting experiment on a single pool. The primary ranking metric is correlation with measured leave-one-in utility, with positive results arising mainly from evaluations within the same task split. Cross-split gains are not statistically significant, and deleting individual memories with negative leave-one-in utility does not outperform random deletion.

The paper also introduces MemAttr-Bench, comprising memory pools, generation DAGs, and empirically measured attribution targets. However, the current manuscript states that the benchmark is unavailable during the review period.

Critical strengths
  1. Valuable and operationally meaningful problem formulation

The paper’s most important contribution is the distinction between “deleting a memory now” and “reconstructing subsequent memories as if that memory had never existed.” Information in persistent memory can propagate through subsequent writes, so attribution within the current execution trace cannot fully capture the role of historical experience. This distinction clarifies the counterfactuals required by different attribution questions and explains why deleting the source text does not necessarily remove the information propagated from it.

  1. Substantial empirical examination of the limits of retrieval observability

The coverage measurements examine different retrievers and retrieval budgets, memory accumulation within a single run, and controls for task diversity. Single-memory deletion experiments further confirm that, for the deterministic retrievers and task sets tested, deleting shadow memories leaves the retrieval list unchanged, whereas deleting visible memories can alter it. The authors also explicitly limit the scope of their conclusions rather than equating observed shadows with memories that are unreachable under all interventions.

Critical weaknesses
  1. The primary evaluation target does not adequately validate the attribution question addressed by the methods and structurally favors some estimators over others

The main experiments compare Shapley-based rankings with leave-one-in (LOI) utility. However, LOI measures the utility of using a single unit in isolation; it is neither the effect of deleting that unit from the full pool, nor generation influence, nor the Shapley value itself. Particularly under the memory dependencies and interactions emphasized in this paper, these quantities can legitimately yield different rankings. Therefore, low or even negative correlation between leave-one-out (LOO) and LOI does not, by itself, establish that LOO fails to estimate its own counterfactual.

Moreover, LOI corresponds precisely to the singleton-coalition layer of the Shapley calculation. The ablation in Appendix D.9 shows that removing the shared term reduces the average correlation across seven experimental settings from +0.77 to +0.57, with the correlation on τ\tau-retail dropping from +0.82 to −0.40. The cross-split source-identification comparison also shows no statistically significant advantage.

The authors already acknowledge these limitations. My concern is therefore not that they incorrectly claim to recover generation influence, but that the current validation does not establish that these fixed-pool estimators reliably answer the proposed cross-episode attribution question, or that they answer a shared counterfactual more accurately than competing estimators. This is a central concern about methodological validity.

  1. The independent contribution of lineage-based grouping is not isolated, leaving insufficient empirical support for the methodological novelty

The potential value of MEMATTR-B stems from two distinct factors: coarsening memories into a small number of players, and performing that coarsening according to generation lineage. However, in the size-matched comparison in Appendix D.3, lineage-based grouping achieves a correlation of +0.600, while the three random groupings average +0.630. This does not establish that lineage structure itself improves attribution accuracy.

Furthermore, the generation DAG records exposure rather than causal dependence, and the optional semantic filtering is not used in the main results. The only direct rewriting experiment covers a single pool with four lineages. The correlation between band Shapley values and rewriting effects is +0.32, with sign disagreements also present. This is insufficient to interpret fixed-pool group-level values as the effects of historical sources.

I recognize the value of lineage for explaining provenance and organizing replay, and I do not consider random grouping to have been demonstrated superior. Nevertheless, given the current evidence, MEMATTR-B’s ranking gains may arise primarily from generic coarsening and standard group-level Shapley attribution, rather than its cross-episode lineage design. This leaves the paper’s most distinctive methodological contribution insufficiently validated.

Questions for authors
  1. Alignment between attribution targets and estimators Beyond correlation with LOI, which existing experimental result directly validates a shared, explicitly defined attribution counterfactual, rather than merely measuring correlation between different utility definitions? Does the LOI target reuse the singleton rollouts from the Shapley coalition table, or is it measured independently? This distinction matters for assessing shared measurement noise and the independence of the evaluation target.

  2. Independent value of the lineage-based design Given that the random-grouping comparison shows no accuracy advantage and the correlation with rewriting effects is only +0.32, which existing evidence supports grouping by generation lineage, rather than computing group-level Shapley values over arbitrary interpretable groups? Should the core contribution be framed as a framework for counterfactual formulation and provenance organization, rather than an empirically validated, more accurate attribution estimator?

  3. Whether the rewriting experiment implements the full counterfactual as defined Definition 3.1 describes removing a memory and replaying all subsequent writes, whereas Appendix D.7 describes rewriting the descendants within the band associated with the removed root. Does the experiment allow changes in retrieval to affect subsequent writes in other lineages, as well as lineage membership? If rewriting is restricted to a predefined band, please clarify whether the conditions under which this procedure is equivalent to M^{-m} hold for the evaluated pool.

  4. Statistical populations and samples underlying the main results Please explicitly specify the memory pools, splits, player sets, exact or Monte Carlo variants, and inclusion criteria used in Table 4, Table 5, and Appendices D.1 and D.8. In particular, how do the favorable mean band-level results in Table 4 relate to the average correlation of +0.370 for the positive-total subset in Appendix D.1 and the nonsignificant sign test for that subset in Appendix D.8? Were these aggregation choices specified in advance, and how were dependencies among related splits and repeated groupings handled?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 7u3B (1/2): W1 & Q1, W2 & Q2, Q3

Thank you for the review. We hope to address your concerns accordingly.


W1 & Q1 (a) – Which result validates a shared, explicit counterfactual?

Table R2 re-executes the cells into a new cache and scores each estimator against two explicit counterfactuals, LOI =v({m})−v(∅)=v(\{m\})-v(\emptyset) and LOO =v(P)−v(P∖{m})=v(P)-v(P\setminus\{m\}) (IretI_{\mathrm{ret}} on the player set). Each estimator tracks the counterfactual it executes: item LOO reaches 0.53 against LOO but 0.22 against LOI, and MemAttr-V reaches 0.63 against independent LOI (8 of 9 cells positive). Removing the singleton term barely changes MemAttr-V (0.63 → 0.66); the inflation comes from shared rollouts (same-cache 0.89 vs. 0.68–0.73 across runs). The τ-retail drop in D.9 reflects an unreproducible target (test–retest −0.21): recomputed, D.9's full-Shapley and singleton-removed means are +0.82 and +0.57, or +0.81 and +0.73 without τ-retail. Across splits, MemAttr-V transfers at 0.43 on ALFWorld, matching the agreement between the two splits' own targets.

Table R2: Independent re-execution. Same players, tasks, prompts and agent LLMs; targets from a new cache. Means over 9 cells for LOI and 10 for LOO (the re-executed τ-airline LOI is fully tied; find-non-living-thing is not re-executed because its pool file is lost).

Estimatorρ vs. independent LOIρ vs. independent LOO
MemAttr-V0.630.49
MemAttr-V without the singleton term0.660.54
Item LOO (M2)0.220.53
Judge, Sonnet-4.5, given the task goals0.430.18
Judge, Sonnet-4.5, memory text and a one-line task description (submitted protocol)0.290.07

Q1 (b) – Does the LOI target reuse the singleton rollouts of the Shapley cache?

Yes for ScienceWorld and τ-bench, whose submitted target used the Shapley cache's singleton cells; no for ALFWorld, whose target came from a separate run. Table R2 removes this overlap for all cells.


W2 & Q2 (a) – What supports grouping by lineage rather than by arbitrary interpretable groups?

The evidence is the positive control (Table R1 in our reply to Reviewer En1S): a matched content grouping finds the same group, and deleting only the descendants does as well as deleting the band (+0.24 vs. +0.22), so detecting this group does not require lineage; what lineage adds is the root link that the rewrite check needs (+0.14, p=0.016p=0.016). On the natural branched pool, lineage ranks no better than size-matched random grouping, as §6 and D.3 state. So §4, §6 and D.3 will describe lineage's role as provenance and source-rooted rewriting rather than a larger deletion effect; the two influence definitions and MemAttr-B's fixed-pool formulation are unchanged. The +0.32 rests on single regenerations with a different write prompt (Q3 (b)) and will be removed. The DAG records exposure (§3, restated in §5); on these testbeds lineage is known by construction (family-restricted write-time retrieval), so the optional semantic filter is not needed; A.5, A.11 and B will state this.


Q2 (b) – Should the contribution be framed as counterfactual formulation and provenance organization rather than as a more accurate estimator?

We would keep both. The counterfactual formulation (the two influence paths, §1) and provenance organization through the logged lineage are central, and the estimators remain a contribution: MemAttr-V is now validated against independent targets (W1 & Q1 (a)). As §5 states, MemAttr-B is a fixed-pool estimator: it values a group of already-written memories, not the per-memory generation influence of its source. When a planted harmful memory shapes outcomes through its descendants, the band finds that group (Table R1 in our reply to Reviewer En1S).


Q3 (a) – Can changes in retrieval affect writes in other lineages, or lineage membership?

Not on this pool: the build restricts write-time retrieval to the family, so the DAG has no cross-family edges (0/28), and removing a root cannot change writes in other lineages or lineage membership.


Q3 (b) – Do the conditions for equivalence to M−mM^{-m} hold for the evaluated pool?

Yes, given Q3 (a), for a rewrite that re-runs write-time retrieval with the build's write prompt. The submitted D.7 run did not (a different write prompt, no re-run of write-time retrieval); the new replay does, four times per arm (Table R3). Replays of one arm differ by up to 0.29 and the pooled root effect is −0.01 (95% CI [−0.06, +0.04]); fam0's root-removed replays score 0.42–0.50 against D.7's 0.125, so the 0.292 does not reproduce. The direct evidence is the planted root of Table R1 (our reply to Reviewer En1S): +0.14 (p=0.016p=0.016; −0.14 in Table R3's null − removed convention) under a controlled writer. §1, §4, §5, D.7, the conclusion, A.5 and E.1 will use these two results instead of the 0.292, and the +0.32 correlation is removed.

AuthorsOpenReview ↗

Response to Reviewer 7u3B (2/2): Q3 (continued) and Q4

Q3 (b) – Do the conditions for equivalence to M−mM^{-m} hold for the evaluated pool (continued)

Table R3: Multi-replay regeneration on the offset200 pool (bands named as in D.7; range = max − min over the four replays).

Bandμ(null)μ(root removed)Root effect (null − removed)Range (null / removed)
fam00.4560.467−0.0110.07 / 0.18
fam10.2330.222+0.0110.07 / 0.00
fam20.3830.367+0.0170.20 / 0.20
fam30.3440.400−0.0560.29 / 0.20

Q4 (a) – Which pools, splits, players, variants and inclusion criteria underlie Tables 4–5 and D.1/D.8?

Table R4 lists them. The Qwen and Llama cells run on Ministral-written pools, so §6 will say the ALFWorld pools are evaluated with, not written by, three agent LLMs; the per-cell table (population_table.md) will become an appendix. It is in the anonymous artifact (https://anonymous.4open.science/r/MemAttr-Bench-ICLR2027-617D), with the pools, generation DAGs, raw outcomes, caches and scripts.

Table R4: Populations behind the main results.

ResultCells and poolsShapley variant
Table 4, MemAttr-V11 visible-set cells from 7 pools; 4–6 players per cellexact
Table 4, MemAttr-B, ALFWorld (+0.72)3 of the 5 primary ALFWorld band configurations; all 5 give +0.43permutation Monte Carlo
Table 4, MemAttr-B, ScienceWorld (+0.45)2 cellspermutation Monte Carlo
Table 4, MemAttr-B, τ-bench (+0.50)1 cell (τ-airline, three bands), now in the per-cell tableexact
Table 512 cells of the six primary band configurations (D.16), drawn from 4 memory poolsexact and permutation Monte Carlo
D.1 (+0.370)positive-total subset of 16 band cells (pool, split, agent and band partition) from 5 memory pools, including bestk (a second pool from the same build tasks) and a 7-band re-partition of deepchainexact
D.8sign test on that subset, p=0.727p=0.727exact
Band vs. granularity search (M4)11 paired cells; band higher in 8permutation Monte Carlo

Q4 (b) – How do Table 4's band means relate to D.1's +0.370 and the D.8 sign test?

They pool different cells and Shapley variants (Table R4): Table 4's +0.72 averages 3 of the 5 primary ALFWorld configurations with permutation Monte Carlo, while +0.370 is the exact mean over D.1's positive-total subset, whose sign test gives p=0.727p=0.727. The recommendation rests on all primary band configurations, read as descriptive agreement (D.8); the camera-ready Table 4 will report them with their positive-total subset beside it.


Q4 (c) – Were these aggregation choices specified in advance?

Partly: pool membership was fixed before the outcomes were seen (D.16), but the aggregation rules were set during analysis.


Q4 (d) – How were dependencies among related splits and repeated groupings handled?

The six primary configurations, which D.16 calls independent and will call primary, share four memory pools, one reason D.8 treats the result as descriptive rather than as a test.


Revisions: §6 adds Table R2 with the targets' reliability, and §6 and D.17 state that the submitted judge also saw a one-line task description; Appendix C gives each target's cache; the abstract's "positive in every evaluated cell" will name the submitted targets (all 11 cells of D.8) and add the independent result; the per-cell table becomes an appendix.

AuthorsOpenReview ↗

Follow-up on our response

Thank you again for your time and effort in reviewing our paper. As the discussion period nears its end, we would be grateful to know whether our responses address your concerns about the shared counterfactual and independent targets (W1 and Q1), grouping by lineage (W2 and Q2), the conditions for equivalence to M−mM^{-m} (Q3), and the pools and aggregation behind Tables 4–5 and D.1/D.8 (Q4). If any point would benefit from further clarification, we would be happy to answer it. If our responses have resolved your concerns, we would be grateful if you would consider updating your rating accordingly.

Reviewer_En1SOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies attribution in agents that write and retrieve persistent text memories across episodes. It distinguishes deleting a memory while keeping later writes fixed from removing it and regenerating subsequent memories. These interventions measure different effects. A retrieval audit of a pool containing 278 memories shows that most memories are never surfaced under the tested workload and that individually deleting those memories leaves the measured retrieval lists unchanged. The authors propose MemAttr-V, which computes Shapley values for individual memories over the visible set, and MemAttr-B, which computes group Shapley values over bands defined by logged generation lineage. Experiments on ALFWorld, ScienceWorld, and the airline and retail domains of tau-bench compare rankings with measured singleton utility, the gain from adding one memory or band to an empty pool. A separate regeneration experiment with four lineages illustrates the difference between value in a fixed pool and generation influence. The paper also reports unsuccessful tests across task splits and unsuccessful pruning tests, and proposes releasing the pools, lineage logs, and measured targets as MemAttr-Bench.

Critical strengths
  1. Separating deletion from the retrieval pool from regeneration is the most useful contribution. An earlier memory can affect the text of later memories, so removing it from a finished pool does not undo its history. The regeneration experiment with four roots gives a concrete example: success on held out tasks changes by as much as .292, and those changes do not closely track band Shapley values in the fixed pool. The paper explicitly acknowledges that the logged DAG records exposure, not proven causal dependence, and that neither proposed estimator recovers the generation influence of individual memories. These distinctions matter for anyone using attribution to decide which stored lesson to revise. A system could delete the original text and still leave its consequences in the retrieved context. Logging the memories available during later writes is a sensible engineering recommendation even without a stronger attribution estimator.

  2. The retrieval audit and the calibration experiments make the practical problem more concrete. At k=3, between 87% and 92% of the pool containing 278 memories is never retrieved across four retrievers, and deleting each shadow memory leaves the deployed retriever's measured lists unchanged. Checks on successive prefixes within the same run show that the coverage problem is not solely an artifact of mixing unrelated sources, although it remains dependent on the workload. I also found the checks on unchanged control arms useful. Repeating an unchanged configuration can produce an effect larger than the measured pruning effects on ALFWorld, whereas the noise level in the retail domain is smaller. The paper does not turn its negative pruning result into a claim that the proposed methods choose bad deletions: that experiment uses the measured singleton utility target, not either estimator. This is a worthwhile warning about confusing attribution agreement with the benefit of an edit.

Critical weaknesses
  1. The proposed methods do not resolve the problem of causal attribution across episodes that motivates them. MemAttr-B values groups of memories that have already been written; it neither regenerates descendants nor identifies which ancestor actually shaped a later write. If a group contains a useful descendant and an unretrieved root, its Shapley value can be nonzero without establishing a contribution from the root. The lineage grouping is interpretable as provenance, but its accuracy advantage is not demonstrated: on the tested pool, random partitions with matching group sizes average correlation .630 versus .600 for the lineage partition. The authors acknowledge this, and I am not claiming that random grouping is better. The issue is what the new method adds beyond applying ordinary Shapley valuation to a convenient partition. The observability argument also depends on the allowed intervention family. Appendix A.2 notes that testing a memory alone would surface it under the retriever that returns the top k items, so lack of evidence from retrieval over the full pool and single deletions is not a general inability to estimate its value in a fixed pool. I would frame the contribution as an empirical study of audits limited by retrieval and valuation of lineage groups, unless stronger evidence connects the scores to the generation influence defined in Section 3.

  2. The evaluation provides limited evidence of attribution quality beyond agreement with singleton utility on the same task split. Singleton utility is not the proposed estimand and contributes a term to each Shapley value. The ablation removing that shared term is a helpful existing check: mean correlation remains .57 versus .28 for item LOO, but the retail result changes from .82 to -.40. Band agreement is less convincing on the subset where the game has positive total value: Appendix D.1 reports mean correlation .370 with five of eight positive, and Appendix D.8 reports an inconclusive sign test. The band comparison across task splits is also inconclusive, and several correlations involve only four players or very small effects. These are not problems that more Monte Carlo permutations alone would fix, since cached coalition noise persists. The judge shown the whole pool can tie MemAttr-V when scored on the same players, and the paper does not establish superiority over that inexpensive baseline. Finally, the proposed benchmark artifacts are explicitly unavailable during review. Together, these limits make the reported high correlations insufficient evidence for a dependable new attribution tool, particularly given the rollout cost.

Questions for authors
  1. What additional claim does Shapley valuation of lineage bands support beyond the value of the texts in that band in a fixed pool? Is there an example in the existing measurements where it correctly identifies a harmful or helpful ancestor that a matched grouping not based on lineage would miss, with the effect checked by regeneration? If not, would you narrow the method claim accordingly? Please also explain why singleton interventions on the observed shadow are excluded from the comparison, since they would provide a target for individual memories in a fixed pool without a lineage log.

  2. Can you provide one matched comparison using the same player set, the same tasks, and independent outcome estimates for method scoring and evaluation? The most useful case would compare MemAttr-V with the judge asked to produce a ranking and item LOO, while separating singleton overlap from rollout noise. For MemAttr-B, please reconcile the main positive environment averages with the analysis of games with positive total value in Appendix D.1 and state which aggregate should support the recommendation. Will the memory pools, coalition measurements, and lineage logs be made available to reviewers?

5 comments
AuthorsOpenReview ↗

Response to Reviewer En1S (1/2): W1 & Q1

Thank you for the review. We hope to address your concerns accordingly.


W1 & Q1 (a) – What does band Shapley support beyond the value of the texts in a fixed pool?

As §5 states, MemAttr-B is a fixed-pool estimator: it values a group of already-written memories, not the per-memory generation influence of its source. The band is the unit your example calls for: the consequences a source leaves in later memories, with an explicit root and write history, which a source-rooted rewrite needs. When a planted harmful memory shapes outcomes through its descendants, the band finds that group (deleting it: 0.22 → 0.44; Table R1). Its value need not reflect the root, however: deleting only the descendants does as much as deleting the band (Q1 (b)).


Q1 (b) – Is there an ancestor that a matched non-lineage grouping would miss, checked by regeneration, and if not, would you narrow the claim?

Not in the existing measurements: in the original lineages no root stands out (pooled effect −0.01; Table R3 in our reply to Reviewer 7u3B), and the submitted 0.292 does not reproduce. We therefore planted one (Table R1). In it, a matched content grouping finds the same group, and deleting only the descendants does as well as deleting the band (+0.24 vs. +0.22), so detecting this group does not require lineage; what lineage adds is the root link that the rewrite check needs (+0.14, p=0.016p=0.016). On the natural branched pool, lineage ranks no better than size-matched random grouping, as §6 and D.3 state. So, for lineage, yes: §4, §6 and D.3 will describe lineage's role as provenance and source-rooted rewriting rather than a larger deletion effect; the two influence definitions and MemAttr-B's fixed-pool formulation are unchanged.

Table R1: Positive control. A plausible but wrong lesson ("washing is optional") that full-pool retrieval never returns, plus four descendants the agent wrote from it, replaces one of the four offset200 lineages; 15 tasks × 3 repetitions; full-pool success 0.22.

InterventionChange in success
Delete the root only−0.04 (retrieved context unchanged; below the 0.054 noise floor)
Show the root alone (singleton, vs. the empty pool)−0.16 (0.18 → 0.02)
Best single-memory deletion (all 20 tried)+0.16 (a memory outside the lineage); any one descendant: at most +0.09
MemAttr-V over the 8 retrieved memories (exact; coalition texts given directly; 15 tasks × 2 repetitions)the 4 descendants get the 4 lowest values, all negative; the root is not a player
Band Shapley over the four lineagesthe planted lineage is lowest (−0.11; −0.12 and −0.10 on the two splits)
Best of 12 size-matched random groups+0.07
Delete the 4 retrieved descendants, keep the root+0.24 (5 of 15 tasks improve, none worsen)
Delete the band+0.22 (6 of 15 tasks improve, none worsen; one-sided sign test over tasks, p=0.016p=0.016)
Rewrite the descendants without the root, vs. with it+0.14 (6 replays per arm; one-sided permutation test over replays, p=0.016p=0.016)

Note: the descendants and both rewrite arms use a consolidation writer (the REFINE prompt plus an instruction to keep and restate the past experience's core claim), not the original pool's PLAIN writer, so the +0.14 holds for this controlled setting. Among band coalitions, the root is retrieved only when its band is evaluated alone. The singleton test measures the root's stand-alone utility; the root-kept versus root-removed rewrite measures its influence through later writes.


W1 & Q1 (c) – Why are singleton interventions on the shadow excluded?

The blind spot is defined for full-pool retrieval with single deletions (Appendix A.2, Definition 3.2), so the paper used singletons only as the evaluation target. They do value shadow memories: measured for every memory of two band pools, 39 of 52 (shadow memory, split) pairs differ from zero by more than the 0.054 noise floor, and the planted root alone lowers success from 0.18 to 0.02. §5 and Algorithm 1 (step 6) will therefore state that a never-retrieved memory has no per-memory evidence under full-pool retrieval but can be valued individually by a singleton intervention.

AuthorsOpenReview ↗

Response to Reviewer En1S (2/2): W2 & Q2

W2 & Q2 (a) – Is there a matched comparison with independent outcomes?

Yes. Table R2 (in our reply to Reviewer 7u3B) re-executes the cells with the same players, tasks and prompts, which removes the shared rollout noise that more permutations could not. Against independent LOI, MemAttr-V reaches 0.63 (8 of 9 cells positive; 0.85 on ALFWorld, near the target's test–retest 0.96), against 0.22 for item LOO and 0.43 for the goal-aware judge, which it beats in 6 of 9 cells. Removing the singleton term barely changes MemAttr-V (0.63 → 0.66); the inflation comes from shared rollouts (same-cache 0.89 vs. 0.68–0.73 across runs). The τ-retail drop in D.9 reflects an unreproducible target (test–retest −0.21): recomputed, D.9's full-Shapley and singleton-removed means are +0.82 and +0.57, or +0.81 and +0.73 without τ-retail.


Q2 (b) – Which band aggregate should support the recommendation?

The recommendation rests on all primary band configurations, read as descriptive agreement (D.8); the camera-ready Table 4 will report them with their positive-total subset beside it. Table 4's +0.72 averages 3 of the 5 primary ALFWorld configurations (all 5: +0.43; ScienceWorld +0.45), while D.1's +0.370 is the exact mean over the positive-total subset (p=0.727p=0.727; populations in Table R4 of our reply to Reviewer 7u3B). A single 4-player cell has little power, so cells are read only in aggregate. Appendix D will add each cell's largest |φ\varphi|, ties and costs. In the positive control, the planted lineage ranks lowest on each split separately (Table R1).


Q2 (c) – Will the pools, coalition measurements and lineage logs be available?

Yes: pools, generation DAGs, raw outcomes, caches and scripts are at https://anonymous.4open.science/r/MemAttr-Bench-ICLR2027-617D (one ScienceWorld pool file is lost; its coalition outcomes are included).


Revisions: §4 adds Table R1; §5 and Algorithm 1 state the singleton scope; §6 adds Table R2 with the targets' reliability, and §6 and D.17 state that the submitted judge also saw a one-line task description; Appendix C gives each target's cache; the abstract's "positive in every evaluated cell" will name the submitted targets (all 11 cells of D.8) and add the independent result.

Thank you for the clarification and the additional experiments. The proposed distinction between provenance for rewriting and fixed-pool valuation addresses part of my concern. I also appreciate the explicit acknowledgment that singleton interventions can value memories outside the observed retrieval set. That qualification should appear alongside the main observability claim, rather than only in the appendix.

The non-replication of the submitted 0.292 effect is important. Could you explain what changed between the original experiment and the replication, and provide the replicated estimates and their uncertainty? This result was one of the concrete examples supporting the distinction between deletion and regeneration. If it no longer holds in the original setting, the corresponding numerical claim and interpretation should be corrected throughout the paper. Please also clarify whether the discrepancy came from an implementation issue, a protocol change, or variability under the same protocol.

Table R1 provides a useful positive control, but it supports a more limited conclusion than an advantage for lineage attribution. The content grouping identifies the same group, and deleting the descendants achieves a similar improvement without deleting the root. The rewrite comparison supplies evidence of influence through later writes in this constructed setting, although the consolidation instruction differs from the original writer and explicitly encourages retention of the earlier claim. I would therefore keep this result separate from evidence about naturally occurring lineages. The proposed framing of lineage as provenance for a targeted rewrite seems more appropriate.

One detail of the new MemAttr-V experiment also needs clarification: the coalition texts are supplied directly. Does this bypass the original retriever, and therefore define a different game from the submitted evaluation? If so, please distinguish its scores from those obtained under retrieval over each coalition.

These clarifications improve the scope of the contribution. My remaining concern is whether the proposed scores are reliably validated beyond the controlled example and agreement with singleton utility on the same task split. I will consider your response to W2 and Q2 before revisiting the overall recommendation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers and the AC for their valuable time and careful feedback. Below we summarize the issues raised by more than one reviewer and how we addressed them. The individual replies give the details and answer the reviewer-specific questions. Pools, lineage logs, raw outcomes, caches and scripts are available at https://anonymous.4open.science/r/MemAttr-Bench-ICLR2027-617D.

1. What does MemAttr-B, and lineage in particular, add beyond fixed-pool valuation? (En1S, 7u3B, zpxo)

We added a positive control: a planted harmful memory that full-pool retrieval never returns, plus four descendants written from it.

  • The band finds the harmful group: deleting it raises success from 0.220.22 to 0.440.44.
  • Deleting only the descendants does as well (+0.24+0.24).
  • What lineage adds is the link to the root. Rewriting the descendants without the root raises success by 0.140.14 (p=0.016p = 0.016) under a controlled writer.

We therefore describe lineage's role as provenance and source-rooted rewriting rather than a larger deletion effect. MemAttr-B remains a fixed-pool estimator, as §5 states. (En1S, Table R1)

2. Are the attribution targets independent of the estimates? (En1S, 7u3B)

We re-executed the attribution cells into a new cache with the same players, tasks and prompts. Against independent LOI, MemAttr-V reaches ρ=0.63\rho = 0.63 (8 of 9 cells positive), compared with 0.220.22 for item LOO and 0.430.43 for a goal-aware judge; each estimator tracks the counterfactual it executes (item LOO reaches 0.530.53 against independent LOO). (7u3B, Table R2)

3. Does the submitted regeneration evidence hold? (7u3B, En1S)

No. A multi-replay study with a null arm shows that the submitted single regeneration (0.2920.292, D.7) does not reproduce; the pooled effect of the original roots is −0.01-0.01. We replace it with the positive control and the multi-replay study, and remove the +0.32+0.32 correlation. (7u3B, Table R3)

4. Which populations underlie the band averages? (En1S, 7u3B)

Table 4's +0.72+0.72 averaged 3 of the 5 primary ALFWorld configurations; all 5 give +0.43+0.43. D.1's +0.370+0.370 is the mean over its positive-total subset (sign test p=0.727p = 0.727), so we read the band results as descriptive agreement (D.8). We add a per-cell population table, and the revised Table 4 will report all primary configurations. (7u3B, Table R4)

We hope these additions address the main concerns and welcome further questions during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.