acceptodds
Under review as a conference paper at ICLR 2027

When Does Agent Memory Matter? Source Access Conditions Its Marginal Contribution

Abstract

Agent memory is designed to preserve information beyond an agent’s immediate context, but when does that memory actually matter? We identify source accessibility (whether stored information remains obtainable from its original sources) as one condition shaping memory’s marginal contribution. Our full-text coding of a purposive sample of 19 agent-memory evaluations finds that none reports memory’s contribution across multiple source-access regimes. We introduce Memory Opportunity Audit (MOA), which freezes acquired memory and execution state and crosses downstream memory use with source access. In our cleanest document study, memory is constructed before the question and source loss is assigned independently of memory content. Factual notes contribute +3.33pp under intact access and +6.33pp under denial, yielding a +3.00pp interaction [+0.33, +5.74]. Adjudicated human scoring and a disjoint second cohort provide further evidence of access-conditioned contribution. The effect is heterogeneous rather than universal across readers, scorers, modalities, and retention regimes, with no consistent benefit for post-observation video memory. These results show that memory contribution need not be source-access invariant: evaluating memory in only one access regime can mischaracterize when that memory matters. Claims spanning access regimes should therefore report contribution within each regime.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
agent memory, LLM agents, memory evaluation, source accessibility, long-term memory, ablation studies, retrieval, evaluation methodology

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xdFXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that whether agent memory "matters" can depend on whether the information it stores is still obtainable from its original sources, and treats this source accessibility as one condition that moderates memory's marginal contribution. To measure this, the authors propose the Memory Opportunity Audit (MOA). Acquired memory and agent state are frozen, memory on/off is crossed with intact vs. restricted source access, and the change in memory's contribution between the two regimes is reported as the memory interaction value (MIV), a 2×2 interaction. In a full-text coding of a purposive sample of 19 agent-memory evaluations, none reports memory's contribution under more than one access regime.

The primary study uses 300 new M3DocVQA text/table questions. Two memories are built from the whole corpus before any evaluation question is seen: NOTES (LLM-written factual notes) and FLAT (the same text verbatim). A Qwen3-VL-32B reader with a budget of ten search/read actions answers with all documents accessible, or under one of three fixed random masks that deny half of the documents. Under strict EM, NOTES adds +3.33pp with intact access and +6.33pp under denial, an interaction of +3.00pp [+0.33, +5.74]. FLAT's interaction is +2.56pp, with an interval that includes zero.

Adjudicated human scoring on a 150-question subset points in the same direction, as does a disjoint 400-question cohort with FLAT and Mem0. A FRAMES supplier-source study shows a large positive interaction but is used only as motivating evidence, because the blocked supplier documents are defined from memory content. The scope checks on LongMemEval, WebShop and video are mixed. The authors recommend that claims spanning access regimes report all four cells and the interaction.

Critical strengths

(1) Careful design and unusual transparency. Memory is built and frozen before any evaluation question is seen, and each mask depends only on the seed, group and document ID. The authors audit that denied documents never reach the reader. The primary estimand and scorer were fixed in advance, and the human audit used external annotators (κ = 0.885). The supplement is complete enough to recompute the headline numbers from the per-cell outputs. I also appreciated that the authors demoted, rather than led with, the FRAMES supplier-source study, which was the confirmatory cohort of its own protocol. In that study, the 12 of 211 held-out questions whose reference answer appears in the delivered memory account for 80% of the strict-EM interaction (36.4% under the study's primary corrected-substring scorer). Within the tested setting, I find the main result credible.

(2) A simple, useful reporting recommendation. For claims that span access regimes, the paper asks authors to report memory's on/off contribution within each regime instead of a single ablation. This is easy to adopt, and the literature coding shows it is rarely done: 15 of 19 evaluations keep the original information accessible in their comparison condition. The paper also reports weak or uncertain effects candidly (30- and 90-day retention, video, the canonical-prefix WebShop study), which makes its claims easier to trust.

Critical weaknesses

(1) Significance. My main concern is that the core finding is close to what one would expect, and the paper does not go beyond it. In the primary study, memory is effectively a copy of the corpus that the mask never touches. FLAT is the raw text and NOTES are notes extracted from it, and both are retrieved by BM25 with the question as the query (joint_document/code/run.py). When half of the original documents are removed, the untouched copy naturally becomes more valuable. The authors give this explanation themselves (Sec. 4.1). Sec. 7 counters that the interaction is not positive by construction, in part because NOTES stores extracted facts rather than source copies. But NOTES is still derived from the same corpus and delivered identically under every mask. The support-hit split also fits this explanation: the NOTES interaction is +4.69pp when a mask denies at least one gold supporting document and +0.00pp when it denies none (Table 10).

The effect is also small: +3.00pp for a reader whose intact-access, no-memory strict-EM accuracy is 30%. In the released outputs, the per-question interaction is positive for 33 questions and negative for 15, a net effect of about 9 of 300 questions.

For this result to go beyond a demonstration, the paper would need either (a) a way to predict when the interaction is large, zero or negative, or (b) a case where single-regime evaluation leads to a consequential wrong conclusion, such as a reversed ranking between memory systems. The paper states that it does not establish a ranking reversal (App. B.1, App. C). The verdict changes in Table 60 come from an illustrative "keep if the interval excludes zero" rule.

The scope is also narrower than the title suggests. Every tested memory is derived from content that the restricted sources hold. Mem0 and WorldMM do some consolidation, but no experiment isolates its value. I am therefore not confident the conclusions extend to memories whose value comes from abstraction or consolidation, which the title appears to cover.

(2) Generality of MIV. The paper's argument against single-regime ablations applies to MIV as well. In the second cohort's development run, a four-action budget produced negative interactions (FLAT −5.33pp, Mem0 −7.33pp). Moving to ten actions shifted MIV by about +10pp (Table 18), although the collector budget and memory contents changed at the same time. The primary study uses the same ten-action cap with a different reader and memory delivery. It passed the development source-headroom check (supplement, joint_document/reader_selection.json), but its budget sensitivity is untested.

The supplement's Phase C report calls this budget dependence "the sharpest limit on how far these numbers generalise". In the paper, however, Sec. 7 bounds the result only generically, to "this reader and execution contract". It cites the four-action result only as evidence that the interaction is "not positive by construction". Neither Sec. 7 nor App. K discusses the budget-driven sign flip as a limitation.

As the paper notes, MIV also mixes two different things: memory recovering information that was lost, and memory no longer hurting once sources are restricted. For example, Mem0's C(A) on image questions is −9.21pp (Table 17). Transferability is further limited by the artificial 50% mask and by the many heterogeneous studies, several of which carry amendments, corrected scorers or withdrawn estimates (Tables 3–4). The superseded or withdrawn estimates and the scorer correction do not affect the primary estimate. The primary study's own amendments concern execution and leave the estimand, cohort and scorer unchanged (App. B.1).

Taken together, the takeaway is roughly that the interaction can be nonzero and, depending on the setup, can take either sign in point estimate. The checklist in Table 5 asks authors to report the budget and delivery format but not to vary them, so it is unclear how practitioners should use MIV.

Questions for authors

Q1. Budget sensitivity (analysis request; the answer could change my assessment of significance). Could you report MIV for the primary design at several action budgets (e.g., 4, 6, 10, 16), together with the no-memory loss L0? If the sign follows source headroom, what else besides access should evaluators vary when making cross-regime claims?

Q2. Cases where the conclusion changes (clarification). Is there a case where the access regime changes an important comparative conclusion? Examples would be a ranking reversal between memory systems with intervals excluding zero, or a published comparison whose result changes in a new cohort. The closest analysis in the paper is the post-hoc Mem0-vs-FLAT ordering in App. C (Table 20). The point estimates reverse under every scorer, but no full-cohort interval excludes zero. The reversal is confined to image questions, where FLAT's resolved intact-access lead over Mem0 (−9.21pp [−14.49, −4.52], Mem0 minus FLAT) disappears under denial, and the paper states that this does not establish a ranking reversal. A single-system case is Mem0 under 7-day LongMemEval retention (C(A) −3.49 → C(R) +12.37pp, Table 60), but those questions had been inspected before, and the 50-question human audit gives an MIV interval of [+0.00, +22.00].

Q3. Other kinds of memory and real restrictions (clarification). Is there evidence for memories whose value comes from abstraction or consolidation across sources (e.g., experiential insights, user models)? Is there evidence from actual retention or permission policies rather than simulated restrictions? How would you define source access in such cases?

8 comments
AuthorsOpenReview ↗

Questions For Authors

Thank you for your careful review and for recognizing the design, transparency and reporting recommendation. We have clarified how MOA reveals access-dependent comparisons and strengthened the evidence across readers, budgets and memory types. We would appreciate your reassessment based on the revised paper and added evidence.

We answer Q1–Q3 here and W1–W2 in a second comment.

Questions For Authors

Q1. Could you report MIV at several budgets with L0, and what besides access should evaluators vary?

MIV stays positive at all four budgets, and the no-memory loss L0 barely moves.

On the first 300 primary questions, only the reader's action budget changes; memory, masks and reader are fixed (Table 9). Here, Verbatim denotes the raw-text memory (FLAT).

Metric4 actions6 actions10 actions16 actions
Notes MIV (pp)+3.44+2.33+2.56+3.67
Verbatim MIV (pp)+3.44+3.22+2.89+2.78
L0 (pp)5.115.004.784.89

L0 stays at 4.78–5.11pp, and both memories' MIVs are positive at every budget. The earlier negative MIVs in the second question set occurred when memory construction also changed. Fixing the acquisition snapshot gives positive MIVs at both reader budgets; changing budget 4→10 shifts Mem0 MIV by −0.42pp (Table 9). Source headroom varies more across readers. Llama-3.1-70B loses less without memory (2.44 vs 4.78pp) and has a smaller MIV (+1.33 vs +2.56pp), yet Notes recovers a similar share of that loss (55% vs 54%). Cross-regime claims should therefore hold memory construction fixed, vary the reader's budget and the reader, and report L0 with MIV.

Q2. Is there a case where the access regime changes an important comparative conclusion?

The revision adds matched comparisons in which the higher-scoring option changes with access.

Figure 3 shows the following matched comparisons (pp; point estimates):

Comparison (first − second)RestrictionAvailable gap (pp)Restricted gap (pp)
Notes − pointer memory, 32Bpointer's targets0.00+7.67
Mem0 − verbatim, LongMemEval, 32Ball sessions−1.75+3.00
Mem0 − verbatim, LoCoMo, 32Ball sessions+4.05−1.56
Keep-all − random withholdingacquired sources−0.27+2.06
Rule memory − no memoryrandom masks−27.78+25.93
Graphiti − no memory7-day retention−1.75+7.25

Pointer memory ties Notes in accuracy with sources available; Notes leads by 7.67pp when the pointer's target documents are blocked. The available-access tie thus becomes an accuracy difference under targeted blocking. The two systems swap order in opposite directions on two datasets. In the policy test (971 trajectories from 683 new source groups, 23.1% of supported writes corrupted), keep-all keeps every record and random withholding removes as many as a source check rejects. The direction was set on 720 development trajectories (−1.81→+1.30pp) and confirmed on the new pools at the pre-specified final answer under substring EM; exact EM and token F1 favor keep-all in both conditions, with a larger gap under restriction (Table 26). The rule memory lowers accuracy with sources available and raises it under denial, so a keep-or-drop decision flips (Table 22). Across 35 recorded comparisons, the preferred option changes in 14 once 12–79% of queries are restricted (Table 4).

These comparisons lead to a concrete evaluation practice: choose systems by absolute performance under the intended source access, and assess memory retention by the gain over no memory in that condition. MIV shows how that gain changes with access.

Q3. Is there evidence for abstraction or consolidation memories and actual retention or permission policies, and how is source access defined there?

Abstraction and integration memories also gain under restriction; we evaluate explicit retention and permission-style rules offline.

In controlled tasks with 72 held-out targets each, neither source alone determines a target. A learned-rule memory stores an inferred rule (10 of 24 correct), and an integrated-record memory links values across two sources. Under random masks their MIVs are +53.70 and +39.35pp (Tables 22–23); procedures distilled from experience and user profiles also give positive MIVs (Appendix E.3). For such memories, source access means access to the inputs the memory was built from, here the two example or value sources.

Seven-day retention blocks sessions older than seven days at question time, and whole-session Mem0's gain rises 14.00→21.50pp (Section 3.3). A rule fixed before outputs blocks 783 documents about living people, giving Notes/verbatim MIVs of +5.42/+4.33pp on 277 affected questions (Appendix A.3). Restrictions apply to search and direct reads, and derived memory stays usable under its own permission (Section 5).

AuthorsOpenReview ↗

Critical Weaknesses

Critical Weaknesses

W1. Novelty and effect size.

MOA tests content-based predictions and reveals access-dependent comparison changes; Notes offsets 74% of the accuracy lost to denial.

The revision tests your option (a) through a prediction and an intervention on stored content. In the primary study, MIV is +12.97pp on the 671 of 2,697 question–mask pairs where Notes holds an answer whose support is denied, versus +0.84pp elsewhere. These 671 pairs account for about 3.2 of the overall 3.86pp increase (Figure 2C; Table 6).

The small and negative estimates in Table 6 show why source loss matters together with memory's role in the task. WebShop's current purchase is independent of the earlier receipt; pointer and video-interval memories help the reader locate source information. These cases motivate testing what stored content contributes under each access condition.

Before running the edited memories, we predicted the order Notes > half removal > full removal. We weighted group effects from separate calibration questions by each variant's group shares. We then evaluated the 265 questions with the reader, masks and 2,048-token cap fixed: replacing answer-bearing notes gave MIVs of 8.05, 3.90 and 2.14pp, in the predicted order (Eq. 3; Figure 5; Appendix A.5). The same edit reduces accuracy more under restriction. Designers can use this procedure to predict the MIV ordering of edited memories before execution and then test it with matched runs.

Notes MIV is +3.86pp on all 899 primary questions and +3.00pp on the first 300. Notes cuts the accuracy lost to denial by 74% (strict EM; Table 2):

Source accessNo memory (%)Notes (%)Notes gain (pp)
Available30.3734.37+4.00
About half the documents denied25.1433.00+7.86

Human labels give MIV +6.44pp vs +4.89pp under strict EM on the 150 audited questions; substring EM and token F1 give +3.97pp and +4.14 points on all 899 (Table 3). Q2 above gives cases for option (b), and Q3 above reports memories built by abstraction and integration.

We will release a reusable MOA harness to add source-access policies to existing benchmarks, audit matched runs, and generate and test calibration-based forecasts.

W2. MIV mixes recovery and reduced harm, and its generality is unclear.

With memory fixed, the increase persists at all four tested budgets; both conditional gains show where memory helps or hurts.

Reporting both contributions separates benefit from reduced harm: on 152 image questions, Mem0 contributes −9.21pp with access and +2.41pp under denial (MIV +11.62pp), and Section 3.4 interprets MIV together with both contributions. The earlier four-action sign change came with a change in memory construction; with construction fixed, MIV stays positive across reader budgets (Q1 above). Section 5's procedure now asks evaluators to hold memory construction fixed, vary the reader's action budget and the reader, and report the accuracy lost without memory alongside MIV. The offline retention and permission-style evaluations are described in Q3 above. Table 1 identifies each study's reader, scorer and restriction. The study sections define the contrasts, and Table 27 records directional predictions fixed before the named executions' outputs.

Thank you again for your thoughtful questions. We would appreciate your reassessment of our contribution based on the revised manuscript and additional experiments.

Thank you for the detailed response and the revised paper. I checked the revision against my original points.

Budget sensitivity (Q1, W2): resolved. Table 9 answers my question. With memory, masks and reader fixed, Notes MIV ranges from +2.33 to +3.67pp across 4–16 actions, and L0 stays at 4.78–5.11pp. This spread is of the same order as the gap between the original and repeat runs of the 10-action configuration (+3.00 vs +2.56pp; 89% of answers match, App. G.2) and well inside the intervals, so I read Table 9 as showing no budget trend. With the 10-action acquisition snapshot fixed, a four-action reader also gives positive MIVs (Table 9b), which supports attributing the earlier negative interactions to the change in memory construction. The Section 5 procedure (hold construction fixed, vary the budget and the reader, report L0 with MIV) is what I was asking for.

Practical use of MIV (W2): resolved. Sections 2 and 3.4 now say what each quantity is for: absolute scores to compare systems, the restricted contribution to judge benefit under restriction, and MIV to describe how the gain changes. This answers my question.

Ranking changes (Q2; option (b) in W1): partly addressed. The clearest new case is Notes vs. pointer memory. The two are tied with sources available, while under targeted blocking Notes leads with an interval that excludes zero (Table 14). This follows from what a pointer stores, but it is a real case that available-only evaluation would miss. The memory-system reversals are weaker. Figure 3 and Table 28 give point estimates only. Across the 19 settings comparing Mem0 and verbatim memory, the order changes in 7, ties in 4 and stays the same in 8 (App. G.4). Only 3 of the 8 recorded ordering predictions held (App. G.3), and the LoCoMo 32B reversal shown in Figure 3 went against both of its predictions (Table 27). This is what one would expect if the two systems are close, and it does not yet show that access reverses which one is better. The policy reversal at the pre-specified final answer appears only under substring EM and rests on a −0.27pp difference with sources available (Table 26). Could you report intervals for the Mem0–verbatim differences, or soften the abstract's claim that restriction changes which memory system scores higher?

Predicting the interaction (option (a) in W1): addressed, and it confirms my reading. The stored-answer split (+12.97pp on the 671 pairs where Notes holds an answer whose support is denied, +0.84pp elsewhere, about 3.2 of the 3.86pp) and the content edits make the mechanism explicit. The forecasts got the order right, but the full-removal forecast missed by 2.54pp against your recorded 2pp bound, and the observed reduction (5.91pp) is smaller than forecast (8.86pp; Table 11). More broadly, all 24 recorded predictions about memories that store the removed content held in sign, while 1 of 3 about pointer memory and 3 of 8 about Mem0–verbatim orderings did (App. G.3). The predictable part of the paper is the part the revised abstract now calls expected; the less obvious claims are mostly the ones that were not predicted. The revision therefore clarifies, but does not remove, my significance concern.

Other kinds of memory (Q3): partly addressed. The controlled tasks in App. E.3 give positive MIVs for learned rules, integrated records, distilled procedures and profiles, and the living-person rule and seven-day retention are closer to real policies. In most of these comparisons, however, a raw copy of the inputs contributes more under restriction than the abstracted memory. The learned-rule and distilled-procedure MIVs also come partly from harm with sources available (C(A) −27.78 and −33.33pp; Tables 22 and 24). The tasks show that such memories are also access-dependent, rather than that their value comes from abstraction itself. As controlled tasks and offline rules, they are fine if the limitations say so.

Added 599 questions. Run under a protocol fixed before their outputs, the remaining 599 eligible questions give +4.28pp [+2.24, +6.32] on their own (App. A.1, Table 27). This works as a replication of the first 300 and strengthens the primary result. Since the main text presents the 899 questions as one pre-specified cohort, I would mention the two stages in Section 3.1.

Overall, the revision resolves my budget and practical-use concerns, and the added 599 questions strengthen the primary result, which I appreciate. My main concern, significance, remains: the stored-answer analysis and the prediction record show that the predictable part of the findings is the expected one, while the memory-system reversals that would go beyond it are neither resolved nor reliably predicted. I am keeping my rating at 2 and would reconsider if the Mem0–verbatim reversals are shown with intervals that exclude zero.

Reviewer_ZXGsOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper asks whether gains measured by a memory-on/off ablation depend on access to the original information sources. MOA freezes acquired memory and the initial execution state, then crosses memory delivery with source restrictions under matched readers, tool rules, and budgets. It estimates memory contribution separately with intact and restricted access, and uses their difference, MIV, to measure the interaction between source access and memory use. MIV is a standard interaction contrast; the methodological emphasis is the experimental contract for freezing memory, isolating branches, and auditing delivered information. The most controlled experiment constructs factual NOTES and verbatim FLAT memory from the entire M3DocVQA corpus before any evaluation question is seen. The writer receives no question, answer, evidence label, or denial mask. The study evaluates 300 text/table questions in 281 source-related groups, using three fixed random masks to deny roughly half the documents independently of memory content. Each question receives identical memory across access conditions, and Qwen3-VL-32B has the same ten-action source budget. The Strict EM benefit of NOTES increases from 3.33 percentage points with intact access to 6.33 under restriction, yielding a 3.00-point interaction with a 95% interval of [0.33, 5.74]. A separate cohort of 400 non-overlapping questions tests FLAT and Mem0, although acquisition sees the question in that study. Human annotation of 150 primary-study questions rescores the same outputs. Additional studies delete notes associated with denied sources and examine FRAMES source blocking, conversational history loss and retention windows, tool tasks, and video. The paper reports positive, unresolved, and zero interactions, limiting its conclusion to access-conditioned memory contribution while distinguishing that evaluation result from information-recovery mechanisms and deployment policies.

Critical strengths
  1. The primary study builds memory before the question, assigns source denial independently of memory content, and holds the delivered memory fixed across access conditions. This reduces confounding from question-specific memory construction and memory-dependent source selection. Appendix B.1 reports all 3,600 experimental cells, delivery checks, and paired inference over source-related groups. These controls make the result interpretable as a change in the conditional contribution of the same memory and clarify the scope of a conventional memory ablation.
  2. The primary NOTES interaction is 3.00 percentage points with a 95% interval of [0.33, 5.74], and the separate 400-question cohort has positive interactions for FLAT and Mem0. On the preselected 150-question subset, two external annotators and a third adjudicator yield a NOTES interaction of 6.44 points, compared with 4.89 under Strict EM on those same questions. The authors correctly treat this as scoring sensitivity on the same outputs. They also disclose question-visible acquisition in the second cohort, answer exposure in FRAMES, and boundary results in tool and video tasks, keeping the different evidence sources distinct.
Critical weaknesses
  1. The primary experiment jointly satisfying pre-question memory construction and independent source denial uses only M3DocVQA and Qwen3-VL-32B. The multiple-reader FRAMES results change acquisition and restriction procedures, while the second 400-question cohort exposes the question during acquisition. Neither replaces reader and budget checks under the primary protocol. The second cohort also had negative interactions in its four-action development configuration, suggesting sensitivity to execution budget. These limitations are disclosed, but stability across source-retrieval capabilities, action budgets, and memory builders remains unclear.
  2. The Strict EM retention interaction after source-coupled note deletion in Table 11 is 1.67 points with interval [-0.76, 4.09]. The corresponding token-F1 and substring intervals exclude zero, but KEEP and DROP come from different execution epochs and the planned matched KEEP rerun was not completed. Deletion also backfills the memory payload with next-ranked retrieval units, changing the delivered content mixture. In the second cohort, some of the positive Mem0 interaction arises from eliminating a negative intact-access effect. The results are consistent with preservation of information from unavailable sources, without establishing it as the main mechanism behind the interaction. This acknowledged explanatory boundary limits guidance for specific retention decisions.
  3. In Appendix E.3, Tables 45 and 46, intact-only selection, loss-mixture selection, and selection from a candidate set including recall gates all choose MEM0. Their replay gain differences on that LoCoMo evaluation are therefore zero. The interval-based screening changes in Figure 2C illustrate access-dependent evaluation conclusions, but do not establish better system selection or recall decisions. The additional insight primarily concerns evaluation interpretation; practical selection value remains unestablished. This limits the weight of the contribution rather than invalidating the primary interaction estimate.
Questions for authors
  1. Could you report the four absolute outcome cells and interaction intervals for another reader and at least one different source-action budget on the same frozen primary inputs? This would materially affect my assessment of the result’s scope.
  2. For Table 11, can the planned matched KEEP rerun be completed, or can paired checks quantify execution-epoch effects? Does the retention interaction persist under matched execution, and how can the effects of removing source-coupled notes and backfilling other retrieval units be distinguished?
  3. Given the identical choices and zero gains in Tables 45 and 46, which practical recommendations have held-out support, and which currently concern reporting contributions under multiple access conditions? Can this distinction clarify the use and limits of Figure 2C’s screening results?
2 comments
AuthorsOpenReview ↗

Questions For Authors

Thank you for recognizing the value of fixed memory and access audits. Your suggestions led to checks across readers, budgets and memory builders. We reran KEEP/DROP together and report system comparisons and selection results below.

We answer Q1–Q3 here and W1–W3 in a second comment.

Questions For Authors

Q1. Can you report the four absolute cells and interaction intervals for another reader and another action budget on the frozen primary inputs?

Both memories gain more under restriction with two more readers and at all four budgets.

On the first 300 primary questions, memory and masks stay fixed. Here, Verbatim denotes the raw-text memory (FLAT). Table 9 gives all four absolute cells and the interaction interval for every row.

SettingNotes MIV (pp)Verbatim MIV (pp)No-memory loss (pp)
32B, 4 actions+3.44+3.445.11
32B, 6 actions+2.33+3.225.00
32B, 10 actions+2.56+2.894.78
32B, 16 actions+3.67+2.784.89
Llama-3.1-70B, 10 actions+1.33+2.562.44
Qwen3-235B-A22B, 10 actions+3.00+3.004.11

Llama-3.1-70B is more accurate without memory (39.33% vs 29.67%) and loses less (2.44 vs 4.78pp), yet Notes recovers a similar share of that loss (55% vs 54%). On the first 300 questions, Notes has MIV +3.00pp with the 235B reader and the 32B primary run (paired difference 0.00pp). The primary MIV is +3.00pp on the first 300 questions and +3.86pp on all 899 (+4.28pp on the added 599).

Four absolute cells for Notes (strict EM, %):

SettingAvailable: no memory (%)Available: Notes (%)Denied: no memory (%)Denied: Notes (%)MIV (pp)
32B, 4 actions29.3332.6724.2231.00+3.44
32B, 10 actions29.6734.3324.8932.11+2.56
Llama-3.1-70B, 10 actions39.3342.6736.8941.56+1.33
Qwen3-235B-A22B, 10 actions27.0033.6722.8932.56+3.00

In the second question set, the negative four-action interactions came with a change in memory construction; with the acquisition snapshot fixed, budget 4→10 changes Mem0 MIV by −0.42pp (Table 9).

Q2. Can the matched KEEP rerun be completed, and can removing source-coupled notes be separated from backfilling other units?

The rerun is complete: KEEP and both DROP versions share an execution epoch, and deletion still costs more under restriction.

One DROP leaves the other delivered units fixed; the other refills the freed slots with next-ranked units (Table 10). Values below are accuracy losses from deletion.

DROP policyAvailable loss (pp)Denied loss (pp)Denied − available (pp)
No backfill2.894.33+1.44
Backfill3.445.00+1.56

Deletion costs more under restriction both without backfill (+1.44pp) and with it (+1.56pp). A content edit targets answer-bearing notes: on 265 questions with reader, masks and the 2,048-token cap fixed, replacing answer-bearing notes lowers MIV from 8.05 to 3.90 and 2.14pp for half and full removal, in the order forecast beforehand from separate calibration questions (Eq. 3; Figure 5; Appendix A.5).

Q3. Which practical recommendations have held-out support, and which concern reporting contributions under multiple access conditions?

Held-out support: a policy reversal on new source groups and a tested access-aware rule. Reporting guidance: cost and break-even calculations.

Held-out support. In the policy test (971 trajectories from 683 new source groups; 23.1% of supported writes replaced with unsupported values), the direction was set on 720 development trajectories and confirmed on the new groups:

The gaps below are keep-all − random withholding under substring EM.

Answer stageAvailable gap (pp)Restricted gap (pp)
Final answer (pre-specified)−0.27+2.06
First answer−1.61+3.26

At the first answer all three scorers reverse; at the final answer, exact EM and token F1 favor keep-all in both conditions, more under restriction (Table 26). With access known, choosing random withholding when sources are available and keep-all under restriction improves accuracy by +1.03pp over always random and +0.14pp over always keep-all on a 50:50 mix.

Existing dialogue cohorts. With the 32B reader, Mem0 and verbatim memory swap order between readable and fully blocked sessions, in opposite directions on LongMemEval and LoCoMo (Figure 3; Appendix G.4).

Reporting guidance. Appendix D.4 applies measured contributions to a storage/retrieval cost c: with equal shares of available access and seven-day retention, available-only and mixed-access calculations choose differently for costs above 2.00 and below 5.25pp (windowed runs), or above 14.00 and below 17.75pp (whole-session Mem0). Table 4 gives the restricted-query shares (12–79%) at which 14 of 35 recorded comparisons switch. These calculations show how declared costs and access frequencies change the decision.

AuthorsOpenReview ↗

Critical Weaknesses

Critical Weaknesses

W1. Stability across retrieval, budgets and memory builders is unclear.

The contribution increase persists across the tested retrievers, writers, readers and budgets.

Replacing BM25 with BAAI/bge-large-en-v1.5 for source search and memory recall gives Notes/verbatim MIVs of +5.00/+5.33pp, and changing the Notes writer from 8B to 32B leaves Notes MIV at +4.00pp in a matched comparison (Appendix A.4). Windowed LongMemEval uses the same controls: both memories are built before the question is shown and frozen, and random half-session denial is independent of memory. Both readers give positive MIVs (Appendix D):

ReaderVerbatim MIV (pp)Mem0 MIV (pp)
8B+9.33+10.25
32B+8.92+12.33

Section 5 now asks evaluators to hold memory construction fixed while varying the reader's budget and the reader, and to report the accuracy lost without memory alongside MIV.

W2. Information preservation is not established as the main mechanism.

The matched interventions consistently link the extra contribution under restriction to retained content.

Matched deletion (Q2) costs more under restriction. On the 265 content-edit questions, replacing all answer-bearing units reduces MIV from 8.05 to 2.14pp, a 5.91pp decrease. MIV is +12.97pp on the 671 of 2,697 question–mask pairs where Notes holds an answer whose support is denied, and +0.84pp elsewhere (Figure 2C; Table 6). Section 3.4 reports both conditional contributions: for Mem0 on image questions, −9.21pp with access and +2.41pp under denial. These show the direction and size of the contribution in each condition.

W3. Practical selection value is unestablished.

On the evaluated 50:50 mix, the access-aware rule gains +1.03pp over always random withholding and +0.14pp over always keep-all.

In this offline evaluation, the rule selects among memory views prepared in advance, using the known access condition (Q3). This supplies a tested example of using access in a selection rule. The cost and break-even analysis maps measured contributions to choices under declared costs and access frequencies (Appendix D.4).

We will release a reusable harness to add benchmark access policies, audit matched runs, and generate and test calibration-based forecasts.

Thank you again for your constructive suggestions. We would welcome your feedback on the added checks and their implications for memory evaluation.

Reviewer_RH7WOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper uses an agent model to save some memories from source documents. At test time, the model (Qwen3-VL-32B in the main experiment) has access to:

  • no memory at all (no source document, no access to the memory saved by the agent) (setting (U,0)),
  • access to the source documents only (setting (A,0)),
  • access to the memory saved by the agent only (setting (U,1)),
  • access to both (setting (A,1)).

The main goal of the paper is to compare the contribution of the saved memory between those settings. This is performed by computing the difference in performance on a downstream task with and without memory activated (for instance, computing (A,1)-(A,0) is showing the additional contribution of the saved memory, given that the source documents are also available).

The main setting uses the intact source document (named setting A in the paper). Another setting is modifying the interaction with these source documents by randomly answering a denial in 50% of the cases (setting R). Comparing setting A and R is called the MOA protocol (Memory Opportunity Audit).

The model considered in the main experiment is Qwen3-VL-32B, and the main corpus is M3DocVQA (a document QA task). Other mentioned corpus are LongMemEval, WebShop (a tool agent task) and NExT-GQA (a video task).

Using the proposed memory agent (called NOTES, which is a baseline), the average additional contribution of the memory agent is +3pp (for setting A) and +6pp (for setting R). When Mem0 memory agent is used instead of NOTES, the authors show that the contribution is negative (for setting A) and around +4pp (for setting R; in Fig.2 middle); this is discussed in Sec. 6.

I was not able to understand exactly the definition of NOTES (Sec. 4 states that "NOTES stores question-independent factual extractions" and in l.817 "NOTES is an extraction baseline, not Mem0 or a new memory algorithm").

Critical strengths

None. The topic is by itself interesting, and many experiments have been done, but I don't see any critical strength.

Critical weaknesses
  • The presentation is not adapted to human comprehension. The overall methodology is understandable.
  • The overall methodological contribution is low. The main idea (difference in performance with and without agent memory) is straightforward, and the implications (Sec. 6) are not surprising.
Questions for authors
  • The abstract is unclear and does not convey the main proposed ideas:

    • The "source-access regimes" are explained in the main paper but not in the abstract,
    • What is the baseline for the +3pp performance improvements? (after reading the full paper, I understand the meaning, but it was not clear from the abstract)
    • NOTES is not introduced in the abstract, and the only definition is provided in Sec. 4 "NOTES stores question-independent factual extractions"
    • The following sentence is very difficult to read: "A disjoint second cohort shows positive access-conditioned interactions for both tested memories, while adjudicated human scoring and semantic rescoring preserve the direction of the primary interaction."
  • Some elements are defined in the main paper but never used, e.g. the placebo setting.

  • Some sentences are tautological, e.g. l.259: the +4.69pp interaction is repeated 2 times, and the +0.00pp is because the setting is exactly the original setting in this case.

  • Some elements are not defined (particularly regarding the memory agent or memory baselines).

  • The figures are slides.

  • Can you provide an example of a (Q,A) evaluated with an Human vs. the original evaluation? (sec. 4.3)

10 comments
AuthorsOpenReview ↗

Critical Weaknesses

Thank you for your detailed review. We revised the entire paper and redrew the figures as you suggested.

MOA tests access-dependent gains from fixed memory through source controls, content edits and pre-run predictions, with checks across readers, scorers and budgets. We would appreciate your reassessment based on the revised paper and added evidence.

We address W1–W2 here and your detailed questions Q1–Q6 in a second comment.

Critical Weaknesses

W1. The paper is difficult to follow.

We have revised the organization and writing throughout the manuscript.

The revised paper explains the protocol, examines one document setting in depth, and connects the findings to memory comparisons and retention decisions. We bring shared definitions together in Section 2, study settings in Table 1 and robustness checks in Table 3. The figures now follow the evaluation steps and show the comparisons directly.

W2. The contribution beyond measuring score differences is unclear.

MOA reveals access-dependent rankings and tests predictions about the role of stored content.

MOA makes source access part of two design tasks: comparing candidate systems for their intended use, and testing how edits to stored content change the contribution under source loss.

Figure 3 shows which candidate scores higher under each access condition. Each comparison holds its question set, reader and scorer fixed across conditions; the dialogue comparisons use the 32B reader.

Comparison (first − second)Available gap (pp)Restricted gap (pp)Restriction
Mem0 − Verbatim, LongMemEval−1.75+3.00All sessions blocked
Mem0 − Verbatim, LoCoMo+4.05−1.56All sessions blocked
keep-all − random withholding, final answer−0.27+2.06Acquisition sources blocked

The higher-scoring candidate changes in every row. On LongMemEval, available-source evaluation favors Verbatim, while blocked-session evaluation favors Mem0; LoCoMo gives the opposite switch. A designer selecting by the reported accuracy would therefore choose a different system for restricted access. This shows why candidate comparisons should match the source access expected in use.

The policy row uses the pre-specified final answer and substring EM; keep-all keeps all records, while random withholding removes as many records as a source check rejects (Section 3.2; Appendix G.4).

We also predicted the effect of content edits before generating new responses. On separate calibration questions, we estimated effects for groups defined by stored answers and source blocking. We weighted those effects by each edited memory's group proportions to predict the order Notes > half removal > full removal.

On 265 questions whose Notes contain the answer, we held the reader, masks and memory text cap fixed. Removing answer-bearing units and refilling their slots gave the predicted order:

Stored contentMIV (pp)
Original Notes+8.05
Half the answer-bearing units replaced+3.90
All answer-bearing units replaced+2.14

Full replacement lowers MIV by 5.91pp: the same content edit costs more accuracy under restriction. The forecast turns the observed content–access relationship into a test made before new responses are available: replacing more answer-bearing units should reduce MIV more. The edit experiment confirms this order, giving designers a way to test which stored information sustains the contribution increase under restriction (Eq. 3; Figure 5; Appendix A.5).

We will release a reusable MOA harness to add source-access policies to existing benchmarks, audit matched runs, and generate and test calibration-based forecasts.

AuthorsOpenReview ↗

Questions For Authors

Questions For Authors

Q1. The abstract is unclear and does not convey the main proposed ideas:

Q1-a. The "source-access regimes" are explained in the main paper but not in the abstract,

The abstract now explains available sources and random denial of about half the documents.

The document mask stays fixed throughout each run and applies to search and direct reads.

Q1-b. What is the baseline for the +3pp performance improvements? (after reading the full paper, I understand the meaning, but it was not clear from the abstract)

The baseline is the reader without saved memory under the same source-access condition.

Table 2 gives the comparison on all 899 questions:

Source accessNo memory (%)Notes (%)Notes gain (pp)
Available30.3734.37+4.00
About half the documents denied25.1433.00+7.86

MIV is the increase in this gain: 7.86 − 4.00 = +3.86pp. The first 300 questions give +3.00pp.

Q1-c. NOTES is not introduced in the abstract, and the only definition is provided in Sec. 4 "NOTES stores question-independent factual extractions"

The abstract defines Notes as facts extracted before questions are shown.

Q1-d. The following sentence is very difficult to read: "A disjoint second cohort shows positive access-conditioned interactions for both tested memories, while adjudicated human scoring and semantic rescoring preserve the direction of the primary interaction."

We explain the cohort, human audit and rescoring separately.

The second cohort evaluates Verbatim and Mem0 on 400 questions with an 8B reader. They share an acquisition history created with the question visible; primary-study memory is built before questions (Appendix C). Human scoring and LLM rescoring check existing primary-study responses (Q6; Appendix A.2).

Q2. Some elements are defined in the main paper but never used, e.g. the placebo setting.

FRAMES uses equal-count placebo blocking to compare memory's own sources with other documents.

The same frozen memory is evaluated on 211 held-out questions with three access conditions (8B reader, substring EM; Appendix B, Table 12):

Source accessNo memory (%)With memory (%)Gain (pp)
Available17.0620.38+3.32
Equal number of other documents blocked15.6418.96+3.32
Memory's source documents blocked8.0621.80+13.74

Blocking memory's sources raises its gain 10.43pp more than placebo blocking; requiring the reader to answer gives +9.95pp (Section 3.3).

Q3. Some sentences are tautological, e.g. l.259: the +4.69pp interaction is repeated 2 times, and the +0.00pp is because the setting is exactly the original setting in this case.

We revised repetitive sentences throughout the manuscript and consolidated repeated explanations.

We brought shared definitions and protocol descriptions together and rewrote passages that repeated numerical results. Figure 2C now presents the source-support and stored-answer groups together. The groups distinguish masks that block supporting documents from those that leave them readable; other documents may be blocked in either group.

Q4. Some elements are not defined (particularly regarding the memory agent or memory baselines).

We now explain what each baseline stores and how the reader uses it.

Qwen3-VL-8B extracts factual Notes before questions; Qwen3-VL-32B reads them to answer. Verbatim stores source text unchanged; Pointer stores document IDs, Mem0 facts, and Graphiti facts and relations. Each question reuses the same recalled text across access conditions (Section 3.1; Appendices A.1 and D).

Q5. The figures are slides.

We completely redesigned and redrew the figures.

Figure 1 follows (a) acquisition, (b) matched runs and (c) gain calculation, with each panel explained in the text. Figures 2–3 plot conditional gains and candidate differences.

Q6. Can you provide an example of a (Q,A) evaluated with an Human vs. the original evaluation? (sec. 4.3)

Table 8 gives actual cases; human scoring preserves the contribution increase.

FieldActual audited example
QuestionWho is still alive, Bertolt Brecht and John Banville?
ReferenceJohn Banville
Model outputJohn Banville is still alive; Bertolt Brecht is deceased.
Strict EM0: extra words prevent an exact match
Human judgment1: the response correctly names John Banville

On the same 150 questions (647 response pairs), Notes MIV is +4.89pp with strict EM and +6.44pp with human labels (Tables 3 and 8; Appendix A).

Thank you again for your careful review. We would appreciate your reassessment of our contribution based on the revised manuscript and additional experiments.

AuthorsOpenReview ↗

Request to Revisit the Latest Revised Manuscript

Thank you again for your detailed review and constructive feedback.

We initially revised the manuscript in response to your comments and have since uploaded several further revisions incorporating feedback from other reviewers. Throughout this process, we have substantially improved the paper's presentation, methodological clarity, and experimental evidence.

We understand that you have a busy schedule, but we would greatly appreciate it if you could review the latest revised manuscript and reconsider your assessment in light of these improvements.

Thank you for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of Revisions

Dear Area Chair and Reviewers,

We sincerely thank you for your time and constructive feedback. We appreciate the recognition of our controlled experimental design and the value of evaluating memory under different source-access conditions. In response to your comments, we have strengthened the experimental evidence and revised the explanations and figures throughout the manuscript. The main additions are summarized below:

  1. Additional 599-question replication. The additional stage yields Notes MIV +4.28pp [95% CI: +2.24, +6.32], strengthening the primary result. The revised manuscript identifies the initial 300-question and additional 599-question stages (§3.1; Appendix A.1).

  2. Statistical analysis of system comparisons. We added paired 95% intervals for the Mem0–verbatim MIV differences on LongMemEval (+4.75pp) and LoCoMo (−5.61pp), together with conversation-deletion checks. Both intervals exclude zero, supporting access-dependent changes in the performance gap (Figure 3; Tables 28–29). Reviewer xdFX has confirmed that the paired analyses support both gap shifts and resolve their concern about the reporting of Figure 3.

  3. Robustness across readers, budgets, and memory construction. We tested two additional readers and four action budgets with memory and source masks fixed. Separate checks examine different retrieval methods and Notes writers (Table 9; Appendix A.4).

  4. Matched interventions and forecasts for stored content. We completed matched KEEP/DROP runs with and without backfilling and tested edits to answer-bearing Notes. Using effects estimated on separate calibration questions, we forecast the MIVs of original, half-replacement, and full-replacement Notes before generating new reader outputs. The observed ordering matched the forecast; numerical forecasts and observed results are reported together (Tables 10–11; Figure 5).

  5. Access-aware selection and practical guidance. We evaluated an offline selection rule on new source groups and clarified how absolute scores, conditional contributions, and MIV support decisions under specified source-access policies (Table 26; §5).

  6. Clearer definitions and presentation throughout the paper. We consolidated shared definitions and controls, defined Notes in the abstract, explained its construction in §3.1, and redesigned the figures. Our follow-up responses provide key results and precise manuscript locations to make reassessment easier.

These additions strengthen MOA's central contribution: a controlled evaluation procedure that reports conditional memory contributions and MIV together, revealing how source access affects memory benefit and system comparisons. This gives researchers and practitioners a concrete way to compare systems under the access conditions expected in use. On labeled validation questions, the calibrated forecasts can also help prioritize proposed changes to stored content for evaluation under the intended access policy.

We respectfully ask you to consider the evidence summarized above and our detailed follow-up responses when reassessing the revised paper's contribution and your current rating. We welcome any remaining questions that would affect your final assessment.

Thank you again for your time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.