- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that whether agent memory "matters" can depend on whether the information it stores is still obtainable from its original sources, and treats this source accessibility as one condition that moderates memory's marginal contribution. To measure this, the authors propose the Memory Opportunity Audit (MOA). Acquired memory and agent state are frozen, memory on/off is crossed with intact vs. restricted source access, and the change in memory's contribution between the two regimes is reported as the memory interaction value (MIV), a 2×2 interaction. In a full-text coding of a purposive sample of 19 agent-memory evaluations, none reports memory's contribution under more than one access regime.
The primary study uses 300 new M3DocVQA text/table questions. Two memories are built from the whole corpus before any evaluation question is seen: NOTES (LLM-written factual notes) and FLAT (the same text verbatim). A Qwen3-VL-32B reader with a budget of ten search/read actions answers with all documents accessible, or under one of three fixed random masks that deny half of the documents. Under strict EM, NOTES adds +3.33pp with intact access and +6.33pp under denial, an interaction of +3.00pp [+0.33, +5.74]. FLAT's interaction is +2.56pp, with an interval that includes zero.
Adjudicated human scoring on a 150-question subset points in the same direction, as does a disjoint 400-question cohort with FLAT and Mem0. A FRAMES supplier-source study shows a large positive interaction but is used only as motivating evidence, because the blocked supplier documents are defined from memory content. The scope checks on LongMemEval, WebShop and video are mixed. The authors recommend that claims spanning access regimes report all four cells and the interaction.
Critical strengths
(1) Careful design and unusual transparency. Memory is built and frozen before any evaluation question is seen, and each mask depends only on the seed, group and document ID. The authors audit that denied documents never reach the reader. The primary estimand and scorer were fixed in advance, and the human audit used external annotators (κ = 0.885). The supplement is complete enough to recompute the headline numbers from the per-cell outputs. I also appreciated that the authors demoted, rather than led with, the FRAMES supplier-source study, which was the confirmatory cohort of its own protocol. In that study, the 12 of 211 held-out questions whose reference answer appears in the delivered memory account for 80% of the strict-EM interaction (36.4% under the study's primary corrected-substring scorer). Within the tested setting, I find the main result credible.
(2) A simple, useful reporting recommendation. For claims that span access regimes, the paper asks authors to report memory's on/off contribution within each regime instead of a single ablation. This is easy to adopt, and the literature coding shows it is rarely done: 15 of 19 evaluations keep the original information accessible in their comparison condition. The paper also reports weak or uncertain effects candidly (30- and 90-day retention, video, the canonical-prefix WebShop study), which makes its claims easier to trust.
Critical weaknesses
(1) Significance. My main concern is that the core finding is close to what one would expect, and the paper does not go beyond it. In the primary study, memory is effectively a copy of the corpus that the mask never touches. FLAT is the raw text and NOTES are notes extracted from it, and both are retrieved by BM25 with the question as the query (joint_document/code/run.py). When half of the original documents are removed, the untouched copy naturally becomes more valuable. The authors give this explanation themselves (Sec. 4.1). Sec. 7 counters that the interaction is not positive by construction, in part because NOTES stores extracted facts rather than source copies. But NOTES is still derived from the same corpus and delivered identically under every mask. The support-hit split also fits this explanation: the NOTES interaction is +4.69pp when a mask denies at least one gold supporting document and +0.00pp when it denies none (Table 10).
The effect is also small: +3.00pp for a reader whose intact-access, no-memory strict-EM accuracy is 30%. In the released outputs, the per-question interaction is positive for 33 questions and negative for 15, a net effect of about 9 of 300 questions.
For this result to go beyond a demonstration, the paper would need either (a) a way to predict when the interaction is large, zero or negative, or (b) a case where single-regime evaluation leads to a consequential wrong conclusion, such as a reversed ranking between memory systems. The paper states that it does not establish a ranking reversal (App. B.1, App. C). The verdict changes in Table 60 come from an illustrative "keep if the interval excludes zero" rule.
The scope is also narrower than the title suggests. Every tested memory is derived from content that the restricted sources hold. Mem0 and WorldMM do some consolidation, but no experiment isolates its value. I am therefore not confident the conclusions extend to memories whose value comes from abstraction or consolidation, which the title appears to cover.
(2) Generality of MIV. The paper's argument against single-regime ablations applies to MIV as well. In the second cohort's development run, a four-action budget produced negative interactions (FLAT −5.33pp, Mem0 −7.33pp). Moving to ten actions shifted MIV by about +10pp (Table 18), although the collector budget and memory contents changed at the same time. The primary study uses the same ten-action cap with a different reader and memory delivery. It passed the development source-headroom check (supplement, joint_document/reader_selection.json), but its budget sensitivity is untested.
The supplement's Phase C report calls this budget dependence "the sharpest limit on how far these numbers generalise". In the paper, however, Sec. 7 bounds the result only generically, to "this reader and execution contract". It cites the four-action result only as evidence that the interaction is "not positive by construction". Neither Sec. 7 nor App. K discusses the budget-driven sign flip as a limitation.
As the paper notes, MIV also mixes two different things: memory recovering information that was lost, and memory no longer hurting once sources are restricted. For example, Mem0's C(A) on image questions is −9.21pp (Table 17). Transferability is further limited by the artificial 50% mask and by the many heterogeneous studies, several of which carry amendments, corrected scorers or withdrawn estimates (Tables 3–4). The superseded or withdrawn estimates and the scorer correction do not affect the primary estimate. The primary study's own amendments concern execution and leave the estimand, cohort and scorer unchanged (App. B.1).
Taken together, the takeaway is roughly that the interaction can be nonzero and, depending on the setup, can take either sign in point estimate. The checklist in Table 5 asks authors to report the budget and delivery format but not to vary them, so it is unclear how practitioners should use MIV.
Questions for authors
Q1. Budget sensitivity (analysis request; the answer could change my assessment of significance). Could you report MIV for the primary design at several action budgets (e.g., 4, 6, 10, 16), together with the no-memory loss L0? If the sign follows source headroom, what else besides access should evaluators vary when making cross-regime claims?
Q2. Cases where the conclusion changes (clarification). Is there a case where the access regime changes an important comparative conclusion? Examples would be a ranking reversal between memory systems with intervals excluding zero, or a published comparison whose result changes in a new cohort. The closest analysis in the paper is the post-hoc Mem0-vs-FLAT ordering in App. C (Table 20). The point estimates reverse under every scorer, but no full-cohort interval excludes zero. The reversal is confined to image questions, where FLAT's resolved intact-access lead over Mem0 (−9.21pp [−14.49, −4.52], Mem0 minus FLAT) disappears under denial, and the paper states that this does not establish a ranking reversal. A single-system case is Mem0 under 7-day LongMemEval retention (C(A) −3.49 → C(R) +12.37pp, Table 60), but those questions had been inspected before, and the 50-question human audit gives an MIV interval of [+0.00, +22.00].
Q3. Other kinds of memory and real restrictions (clarification). Is there evidence for memories whose value comes from abstraction or consolidation across sources (e.g., experiential insights, user models)? Is there evidence from actual retention or permission policies rather than simulated restrictions? How would you define source access in such cases?
8 comments
Critical Weaknesses
Critical Weaknesses
W1. Novelty and effect size.
MOA tests content-based predictions and reveals access-dependent comparison changes; Notes offsets 74% of the accuracy lost to denial.
The revision tests your option (a) through a prediction and an intervention on stored content. In the primary study, MIV is +12.97pp on the 671 of 2,697 question–mask pairs where Notes holds an answer whose support is denied, versus +0.84pp elsewhere. These 671 pairs account for about 3.2 of the overall 3.86pp increase (Figure 2C; Table 6).
The small and negative estimates in Table 6 show why source loss matters together with memory's role in the task. WebShop's current purchase is independent of the earlier receipt; pointer and video-interval memories help the reader locate source information. These cases motivate testing what stored content contributes under each access condition.
Before running the edited memories, we predicted the order Notes > half removal > full removal. We weighted group effects from separate calibration questions by each variant's group shares. We then evaluated the 265 questions with the reader, masks and 2,048-token cap fixed: replacing answer-bearing notes gave MIVs of 8.05, 3.90 and 2.14pp, in the predicted order (Eq. 3; Figure 5; Appendix A.5). The same edit reduces accuracy more under restriction. Designers can use this procedure to predict the MIV ordering of edited memories before execution and then test it with matched runs.
Notes MIV is +3.86pp on all 899 primary questions and +3.00pp on the first 300. Notes cuts the accuracy lost to denial by 74% (strict EM; Table 2):
| Source access | No memory (%) | Notes (%) | Notes gain (pp) |
|---|---|---|---|
| Available | 30.37 | 34.37 | +4.00 |
| About half the documents denied | 25.14 | 33.00 | +7.86 |
Human labels give MIV +6.44pp vs +4.89pp under strict EM on the 150 audited questions; substring EM and token F1 give +3.97pp and +4.14 points on all 899 (Table 3). Q2 above gives cases for option (b), and Q3 above reports memories built by abstraction and integration.
We will release a reusable MOA harness to add source-access policies to existing benchmarks, audit matched runs, and generate and test calibration-based forecasts.
W2. MIV mixes recovery and reduced harm, and its generality is unclear.
With memory fixed, the increase persists at all four tested budgets; both conditional gains show where memory helps or hurts.
Reporting both contributions separates benefit from reduced harm: on 152 image questions, Mem0 contributes −9.21pp with access and +2.41pp under denial (MIV +11.62pp), and Section 3.4 interprets MIV together with both contributions. The earlier four-action sign change came with a change in memory construction; with construction fixed, MIV stays positive across reader budgets (Q1 above). Section 5's procedure now asks evaluators to hold memory construction fixed, vary the reader's action budget and the reader, and report the accuracy lost without memory alongside MIV. The offline retention and permission-style evaluations are described in Q3 above. Table 1 identifies each study's reader, scorer and restriction. The study sections define the contrasts, and Table 27 records directional predictions fixed before the named executions' outputs.
Thank you again for your thoughtful questions. We would appreciate your reassessment of our contribution based on the revised manuscript and additional experiments.
Thank you for the detailed response and the revised paper. I checked the revision against my original points.
Budget sensitivity (Q1, W2): resolved. Table 9 answers my question. With memory, masks and reader fixed, Notes MIV ranges from +2.33 to +3.67pp across 4–16 actions, and L0 stays at 4.78–5.11pp. This spread is of the same order as the gap between the original and repeat runs of the 10-action configuration (+3.00 vs +2.56pp; 89% of answers match, App. G.2) and well inside the intervals, so I read Table 9 as showing no budget trend. With the 10-action acquisition snapshot fixed, a four-action reader also gives positive MIVs (Table 9b), which supports attributing the earlier negative interactions to the change in memory construction. The Section 5 procedure (hold construction fixed, vary the budget and the reader, report L0 with MIV) is what I was asking for.
Practical use of MIV (W2): resolved. Sections 2 and 3.4 now say what each quantity is for: absolute scores to compare systems, the restricted contribution to judge benefit under restriction, and MIV to describe how the gain changes. This answers my question.
Ranking changes (Q2; option (b) in W1): partly addressed. The clearest new case is Notes vs. pointer memory. The two are tied with sources available, while under targeted blocking Notes leads with an interval that excludes zero (Table 14). This follows from what a pointer stores, but it is a real case that available-only evaluation would miss. The memory-system reversals are weaker. Figure 3 and Table 28 give point estimates only. Across the 19 settings comparing Mem0 and verbatim memory, the order changes in 7, ties in 4 and stays the same in 8 (App. G.4). Only 3 of the 8 recorded ordering predictions held (App. G.3), and the LoCoMo 32B reversal shown in Figure 3 went against both of its predictions (Table 27). This is what one would expect if the two systems are close, and it does not yet show that access reverses which one is better. The policy reversal at the pre-specified final answer appears only under substring EM and rests on a −0.27pp difference with sources available (Table 26). Could you report intervals for the Mem0–verbatim differences, or soften the abstract's claim that restriction changes which memory system scores higher?
Predicting the interaction (option (a) in W1): addressed, and it confirms my reading. The stored-answer split (+12.97pp on the 671 pairs where Notes holds an answer whose support is denied, +0.84pp elsewhere, about 3.2 of the 3.86pp) and the content edits make the mechanism explicit. The forecasts got the order right, but the full-removal forecast missed by 2.54pp against your recorded 2pp bound, and the observed reduction (5.91pp) is smaller than forecast (8.86pp; Table 11). More broadly, all 24 recorded predictions about memories that store the removed content held in sign, while 1 of 3 about pointer memory and 3 of 8 about Mem0–verbatim orderings did (App. G.3). The predictable part of the paper is the part the revised abstract now calls expected; the less obvious claims are mostly the ones that were not predicted. The revision therefore clarifies, but does not remove, my significance concern.
Other kinds of memory (Q3): partly addressed. The controlled tasks in App. E.3 give positive MIVs for learned rules, integrated records, distilled procedures and profiles, and the living-person rule and seven-day retention are closer to real policies. In most of these comparisons, however, a raw copy of the inputs contributes more under restriction than the abstracted memory. The learned-rule and distilled-procedure MIVs also come partly from harm with sources available (C(A) −27.78 and −33.33pp; Tables 22 and 24). The tasks show that such memories are also access-dependent, rather than that their value comes from abstraction itself. As controlled tasks and offline rules, they are fine if the limitations say so.
Added 599 questions. Run under a protocol fixed before their outputs, the remaining 599 eligible questions give +4.28pp [+2.24, +6.32] on their own (App. A.1, Table 27). This works as a replication of the first 300 and strengthens the primary result. Since the main text presents the 899 questions as one pre-specified cohort, I would mention the two stages in Section 3.1.
Overall, the revision resolves my budget and practical-use concerns, and the added 599 questions strengthen the primary result, which I appreciate. My main concern, significance, remains: the stored-answer analysis and the prediction record show that the predictable part of the findings is the expected one, while the memory-system reversals that would go beyond it are neither resolved nor reliably predicted. I am keeping my rating at 2 and would reconsider if the Mem0–verbatim reversals are shown with intervals that exclude zero.
Questions For Authors
Thank you for your careful review and for recognizing the design, transparency and reporting recommendation. We have clarified how MOA reveals access-dependent comparisons and strengthened the evidence across readers, budgets and memory types. We would appreciate your reassessment based on the revised paper and added evidence.
We answer Q1–Q3 here and W1–W2 in a second comment.
Questions For Authors
Q1. Could you report MIV at several budgets with L0, and what besides access should evaluators vary?
On the first 300 primary questions, only the reader's action budget changes; memory, masks and reader are fixed (Table 9). Here, Verbatim denotes the raw-text memory (FLAT).
L0 stays at 4.78–5.11pp, and both memories' MIVs are positive at every budget. The earlier negative MIVs in the second question set occurred when memory construction also changed. Fixing the acquisition snapshot gives positive MIVs at both reader budgets; changing budget 4→10 shifts Mem0 MIV by −0.42pp (Table 9). Source headroom varies more across readers. Llama-3.1-70B loses less without memory (2.44 vs 4.78pp) and has a smaller MIV (+1.33 vs +2.56pp), yet Notes recovers a similar share of that loss (55% vs 54%). Cross-regime claims should therefore hold memory construction fixed, vary the reader's budget and the reader, and report L0 with MIV.
Q2. Is there a case where the access regime changes an important comparative conclusion?
Figure 3 shows the following matched comparisons (pp; point estimates):
Pointer memory ties Notes in accuracy with sources available; Notes leads by 7.67pp when the pointer's target documents are blocked. The available-access tie thus becomes an accuracy difference under targeted blocking. The two systems swap order in opposite directions on two datasets. In the policy test (971 trajectories from 683 new source groups, 23.1% of supported writes corrupted), keep-all keeps every record and random withholding removes as many as a source check rejects. The direction was set on 720 development trajectories (−1.81→+1.30pp) and confirmed on the new pools at the pre-specified final answer under substring EM; exact EM and token F1 favor keep-all in both conditions, with a larger gap under restriction (Table 26). The rule memory lowers accuracy with sources available and raises it under denial, so a keep-or-drop decision flips (Table 22). Across 35 recorded comparisons, the preferred option changes in 14 once 12–79% of queries are restricted (Table 4).
These comparisons lead to a concrete evaluation practice: choose systems by absolute performance under the intended source access, and assess memory retention by the gain over no memory in that condition. MIV shows how that gain changes with access.
Q3. Is there evidence for abstraction or consolidation memories and actual retention or permission policies, and how is source access defined there?
In controlled tasks with 72 held-out targets each, neither source alone determines a target. A learned-rule memory stores an inferred rule (10 of 24 correct), and an integrated-record memory links values across two sources. Under random masks their MIVs are +53.70 and +39.35pp (Tables 22–23); procedures distilled from experience and user profiles also give positive MIVs (Appendix E.3). For such memories, source access means access to the inputs the memory was built from, here the two example or value sources.
Seven-day retention blocks sessions older than seven days at question time, and whole-session Mem0's gain rises 14.00→21.50pp (Section 3.3). A rule fixed before outputs blocks 783 documents about living people, giving Notes/verbatim MIVs of +5.42/+4.33pp on 277 affected questions (Appendix A.3). Restrictions apply to search and direct reads, and derived memory stays usable under its own permission (Section 5).