- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates memory-level failure attribution for self-improving LLM agents that write and reuse persistent memories across episodes. The motivation is that existing attribution work, for both single-agent and multi-agent systems, attributes outcomes within one episode, and for memory it only sees the items retrieved into the current prompt. The paper defines two cross-episode influence paths, retrieval influence and generation influence. Retrieval influence is the change in success when one memory is deleted from the pool while every memory written after it stays fixed. Generation influence is the change when the memory is deleted and all later memory writes are regenerated without it, so it also captures effects passed through the memories that descend from it. The authors then argue that retrieval-only attribution has a blind spot. On a 278-memory pool, 87–92% of memories are never retrieved at k=3, and deleting any single never-retrieved memory leaves all 7,905 measured top-3 lists unchanged. On one pool, removing a lineage root and regenerating its descendants changes held-out success by up to 0.292.
Based on this, they propose two estimators. MemAttr-V computes exact Shapley values over the 4–6 memories that retrieval surfaces. MemAttr-B groups memories into lineage bands by their source ancestor in the logged generation DAG. It computes band-level Shapley values with permutation Monte Carlo, without regenerating memories. Both are evaluated by Spearman correlation with measured leave-one-in utility on ALFWorld, ScienceWorld and τ-bench. Experiments show that both estimators correlate positively with measured leave-one-in utility, while a pruning test finds that deleting the negative-utility memory does not beat random deletion. The main contribution is framing cross-episode memory attribution, separating direct retrieval effects from effects passed through later memories, and releasing MemAttr-Bench with the memory pools and their generation lineages.
Critical strengths
- The paper distinguishes single-episode from cross-episode failure attribution for agent memory and gives a clear framing of two influence paths, retrieval influence and generation influence. This framing is insightful and useful. It explains why existing trace-level and context-level attribution methods cannot see a memory whose effect only reaches the outcome through later memories, and it makes explicit which counterfactual each quantity requires.
- The method follows a clear design principle, the attribution unit should match what the system can actually observe and intervene on. The concept of generation DAG is quite interesting. Individual memories are scored only where retrieval provides item-level evidence (MemAttr-V), and memories without such evidence are grouped into lineage bands from the logged generation DAG (MemAttr-B).
- The paper provide concrete measurements and artifacts. Their MemAttr-Bench releases the generation lineage on which memories were in context when each memory was written. Together with the paper's recommendation to record this lineage, this gives future work a basis for studying generation influence directly.
Critical weaknesses
- My main concern is that MemAttr-B does not solve the problem raised by the paper itself. The paper's main motivation is that some memories exert their influence indirectly through their descendants, but current methods cannot observe this, so it aims to use MemAttr-B to observe this indirect influence. However, MemAttr-B's approach is to remove an entire memory family from the pool, observe the change in accuracy and use Shapley values to assign credit. What this actually measures is the value of this existing set of memories, rather than the indirect influence we expect.
- The paper's evaluation of whether memory is useful relies entirely on task success rate, which is reasonable as the ultimate goal, but this signal has somewhat low resolution. First, at the split sizes used in the paper, each task only has a 0 or 1 outcome, and results from different seeds often end up tied, or both the full pool and the empty pool have a success rate of 0. More importantly, a memory may clearly change the agent's behavior, but the final task outcome remains unchanged, and for the attribution this paper aims to achieve, this signal is actually important.
Some other minor points. First, figure 1 contains some blurry and unnatural textures, it is recommended to make adjustments or refinements after using tools to assist with figure creation. Second, there are some mismatches between captions and figures and inconsistent use of terminology in the text, which makes the reading experience less smooth.
Questions for authors
- Of the 87–92% never-retrieved memories, how many are actually relevant to the 31 evaluation goals? How large is the shadow if the evaluation goals cover the task types present in the 278-memory pool?
- The same environment gets different MemAttr-V numbers in different places. For example, ScienceWorld is +0.27 in Table 10, +0.58 in Table 11, and [0.58, 1.0] in Table 4. Which pools and splits does each number use?
2 comments
Follow-up on our response
Thank you again for your time and effort in reviewing our paper. As the discussion period nears its end, we would be grateful to know whether our responses address your concerns about whether MemAttr-B captures indirect influence (W1), the coarseness of binary success (W2), the size of the shadow once relevance is controlled (Q1), and the ScienceWorld pools and splits (Q2). If any point would benefit from further clarification, we would be happy to answer it. If our responses have resolved your concerns, we would be grateful if you would consider updating your rating accordingly.
Response to Reviewer zpxo
Thank you for the review. We hope to address your concerns accordingly.
W1 – Does MemAttr-B capture indirect influence?
By design, it values the existing set. As §5 states, MemAttr-B is a fixed-pool estimator: it values a group of already-written memories, not the per-memory generation influence of its source. The attribution unit follows what the system can observe and intervene on: memories without per-memory evidence under full-pool retrieval are grouped into their lineage band (a singleton test can still value them individually; our reply to Reviewer En1S, W1 & Q1 (c)), and the logged lineage lets that band be rewritten from its root, which generation influence requires (Definition 3.1). When a planted harmful memory shapes outcomes through its descendants, the band finds that group, as the positive control shows: the planted root, never retrieved, acts only through its descendants (Table R5). The singleton test measures the root's stand-alone utility; the root-kept versus root-removed rewrite measures its influence through later writes. The rewrite uses a consolidation writer that restates the root's claim, so the +0.14 holds for this controlled setting. With the multi-replay study (our reply to Reviewer 7u3B, Q3 (b)), this control replaces the submitted single regeneration (0.292, D.7), which does not reproduce; §4 will report it.
Table R5: Three quantities in the positive control (Table R1 in our reply to Reviewer En1S).
W2 (a) – Is binary success too coarse, with ties and all-zero games?
Graded targets reduce the problem: steps-discounted success on ALFWorld and the native ScienceWorld score cut ties in the ALFWorld and ScienceWorld LOI targets from 5 to 3, and MemAttr-V reaches ρ = 0.69 against the graded, independently re-executed target over 9 cells (0.75 without τ-retail, whose target is not reproducible; τ-bench stays binary). The per-cell table in the artifact lists v(full) and v(empty) for every cell, so cells in which both score 0 are visible.
W2 (b) – Can a memory change behaviour without changing the outcome?
Yes, often. We re-ran ALFWorld with full action logging (the empty pool, each memory alone and the full set; 15 tasks × 3 repetitions). In 65–77% of singleton-versus-empty pairs the outcome is the same, yet their mean normalized edit distance is 0.69–0.71, against 0.25–0.28 for same-outcome empty-pool reruns, and the gap holds on all 15 tasks (one-sided sign test over tasks, p<0.001). So memories change behaviour well beyond what success records. §6 will report both analyses, and §3 will separate success utility from behavioural influence.
W3 (minor) – Figure 1 textures, caption–figure mismatches and terminology
We will remove the rendering artifacts in Figure 1, fix the caption–figure mismatches, and align the terms with Definitions 3.1–3.2.
Q1 (a) – How many never-retrieved memories are relevant to the 31 evaluation goals?
By task type, 238 of the 278 memories are relevant to the 31 goals, and 85–90% of them (202–215) are never retrieved at k=3 (Table R6); the other 40, τ and ScienceWorld memories, are never returned by the deployed token-overlap retriever.
Q1 (b) – How large is the shadow when the goals cover every task type in the pool?
Still large: with 59 goals covering every task type, 78–80% stay unretrieved at k=3 (Table R6). We will add these rows to Table 2.
Table R6: Never-retrieved fraction (k=3 / k=16, range over four retrievers).
Q2 – Which pools and splits does each ScienceWorld number use?
Each number pools different cells (Table R7). Table 12 and its footnote will give the windows each cell uses (variations 100–105 for melt and find-non-living-thing, 0–7 for v0_7), and every reported ρ will state its pool and split.
Table R7: ScienceWorld MemAttr-V numbers.