acceptodds
Under review as a conference paper at ICLR 2027

Do Agents Truly Forget? A Lifecycle Benchmark and Memory Receipts for Long-Term Memory

Abstract

Long-term memory is becoming a core component of intelligent agent systems, motivating growing research on how user information should be stored, consolidated, updated, and retrieved. However, once information has been consolidated into derived long-term memory, reliably revoking its influence remains difficult to define and verify. Invalidating the original record may leave semantic residue in derived memories; removing an entire memory unit may destroy other information that should remain; and appending a negative instruction keeps the invalidated content in the model context. This paper studies verifiable forgetting in long-term agent memory: can an agent behave as if it had never observed that source, while preserving information supported by the remaining history? To make this question measurable, we construct a unified benchmark suite containing 865 evaluation cases adapted from five public benchmarks. By comparing Invalidated and Never-Seen worlds, the suite distinguishes residual personalized influence from content independently regenerated by the model. We further introduce Memory Receipts, which associate each derived memory with source lineage maintained in a separate memory maintenance layer and reconstruct only the memory units affected by invalidation. This mechanism avoids rebuilding the complete user history and requires no persistent invalidation instruction in future queries, enabling traceable and verifiable memory repair with recomputation limited to affected memory units. Our work positions forgetting as a first-class operation in the lifecycle of long-term agent memory, alongside writing, updating, and retrieval.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Machine Unlearning, Memory Management, Long-Term Memory, Agent Memory

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NHWPOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes to produce a new benchmark for agent memory and evaluation. It is especially concerned with forgetting.

Critical strengths

The basic argument as to why a new benchmark is required seem reasonable enough.

The paper generally does a good job of putting together a narrative and explaining its contributions clearly.

Critical weaknesses

Very little by way of technical contribution. This is perhaps to be expected of a benchmark paper, but still, it can be hard for a paper to be highly impactful unless the benchmark becomes highly popular, which is quite rare.

The main "memory receipts" contribution is fine but not that big. I think the main issue here (which can perhaps be clarified in rebuttal) is that it's hard to tell how much the benchmark and model are paired in a way that a "win" is guaranteed, i.e., the paper produces a benchmark to highlight a very specific issue and then introduces a model to solve that specific issue (therefore easily beating other strategies). That's fine if the specific issue being studied is sufficiently important, but that's hard for me to tell.

The actual benchmark produced here is fairly small.

The paper, even compared to others in my batch, is heavily AI-based. It is hard to tell if there is value in humans reading this paper, and seems less likely that there'll be value to future AI models following up on this work given that it's so heavily produced by AI (according to the authors' own claims). The authors claim to have done little more here than verified the output.

Questions for authors

See above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer NHWP

Response to Reviewer NHWP

Thank you for recognizing our motivation. Below we clarify our perspective, trade-offs and the authors' role.

W1: Technical contribution and what readers learn.

A: In real-world personalization, invalidation is a first-class lifecycle state—users revoke preferences, update facts and delete records—yet prior metrics cannot tell whether an agent truly forgot. We formulate this problem, introduce counterfactual paired metrics (PA/APA) to expose failure modes hidden by legacy metrics, and quantify local repair trade-offs:

1. Why legacy metrics fail & counterintuitive insight (independent of Receipts). Prior evaluations equate target absence (FAA) with forgetting. Comparing Observed-Invalidated (OI) against a matched Never-Seen (NS) world (same survivors and query) shows why FAA misleads: an agent may infer a target from priors/cues without residue (shared hit B), leak OI-only residue (I), or inflate FAA via collateral loss (N or joint miss X). On Causal-Memora, Qwen3-4B Append-only scores FAA=74.92% over 2,787 pairs, yet B/I/N/X=193/506/155/1933—masking 506 residual leaks alongside 193 prior hits. Defining forgetting via APA=B/(B+I+N) plus retention gives a lasting diagnostic lens.

2. Lifecycle benchmark & executable local repair. We unify five datasets into 1,087 paired cases with source-to-unit mappings and retention/forgetting criteria. Receipts adapts provenance tracking to LLM-compiled memory: rebuild affected units from survivors, delete unsupported units and reuse others—matching Clean Rebuild across all 1,087 states under fixed partitions and a memoized compiler.

W2: Practical importance and is a "win" guaranteed?

A: No—neither our benchmark construction nor empirical results guarantee a win for Receipts. Histories, partitions and criteria are frozen before generation; receipts never enter prompts.

Internal metrics & trade-offs. Pairing legacy metrics with APA on our 1,087-case shared-writer ablation (Qwen3-4B, local Qwen3-32B judge) clarifies both our method's strength and real trade-offs:

MethodMPAFAAFAMAAPA (Ours)
Receipts44.8190.3936.33100.00
Whole-unit deletion34.5991.0127.6950.23
Clean Rebuild44.8190.3936.33100.00

Deletion looks higher on legacy FAA (91.01 vs. 90.39), yet our APA=50.23% exposes 218 OI-vs-NS divergences (98 leaks I, 120 over-deletions N) and a 10.23-point MPA loss (Receipts +10.23, 95% CI [7.29,13.44]; FAMA +8.64; LongMemEval MPA is 1.43 lower). Against KeepShared and survivor-assisted Edit (60 cases, two actors), Edit gets higher point-estimate MPA (33.61 vs. 32.79 Qwen; 40.98 vs. 36.89 Llama; CIs cross zero) but diverges from NS (APA 62.07%/42.42% vs. 100%) and drops Llama FAA (75.38 vs. 88.46).

External benchmark (GateMem). On 20 fixed GateMem episodes (347 checkpoints, 2,082 answers; 5 dev, 15 primary) with native rubrics (higher utility, lower failure better):

Method15 episodes: utility/failureAll 20: utility/failure
Receipts74.19/34.0978.31/39.78
Whole-unit deletion71.77/31.0675.30/34.81
Global Rewrite77.42/38.6481.93/44.75
Clean Rebuild75.81/33.3379.52/39.23
TF-IDF RAG-Naive62.90/57.5863.86/59.12
TF-IDF RAG-Policy48.39/39.3951.81/42.54

On the primary 15, Receipts beats RAG-Naive by +11.29 utility (95% CI [2.44,21.26]) and -23.48 failure ([14.71,32.08]); Global Rewrite has higher utility/higher failure, and deletion CIs cross zero. Receipts cuts executed-edit tokens from Clean's 309,526 to 127,805 (58.71% less, 91 vs. 205 calls; Qwen3-4B actor, Llama-3.1-8B judge; shared parsing gaps and capped outputs kept).

W3: Benchmark size.

A: Our 1,087 paired cases match or exceed LongMemEval (500), PersonaBench (582) and Memora (600), alongside MemoryAgentBench (2,071) and PrefEval (3,000), uniquely pairing OI/NS worlds across five invalidation types.

W4: AI use and the authors' contribution.

A: We apologize our AI Use Statement was overly broad and caused misunderstanding. The human authors originated the problem, paired-world formulation, Memory Receipts, experimental design and insights. AI served primarily for coding assistance (conversion/eval scripts) and polishing—never to invent histories, select cases or replace human judgment (humans audited construction and 200 responses: 98.00% inter-human, 96.00% 3-judge match).

Camera-ready: In the camera-ready, we will incorporate every improvement prompted by your review—foregrounding our counterintuitive insights, adding internal trade-offs, external GateMem results and benchmark-scale comparisons, and rewriting the AI Use Statement to state our human contributions accurately—and sincerely thank you for your valuable contribution to strengthening our work.

Reviewer_CKPMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how agents can remove information from external long-term memory while preserving other valid information. It introduces a paired evaluation comparing an agent that has seen and then invalidated a source with one that never saw it, using 1,087 cases adapted from five benchmarks. The proposed Memory Receipts track source dependencies and rebuild only affected memory units. Experiments across four actor models report high agreement with the Never-Seen reference, receipt-coverage ablations, and lower deletion costs than rebuilding all memory.

Critical strengths

The paper introduces a useful paired evaluation for forgetting in agent memory. It compares an agent that has seen and subsequently invalidated a source with one that never received it, while keeping the surviving information the same. This provides a clear reference for assessing whether prior exposure still affects the agent’s behavior after invalidation. The framework also treats retention as a separate objective, reflecting the practical need to remove outdated or revoked information while preserving useful memory.

Critical weaknesses

W1: The retention benefit of reconstruction over whole-unit deletion is not directly demonstrated.

Preserving valid information within an affected memory unit is a central motivation for the method. Table 8 compares reconstruction with whole-unit deletion, but reports only paired behavioral metrics. Tables 4–5 report retention performance, but do not include whole-unit deletion. It therefore remains unclear how much valid information reconstruction preserves over simple deletion. Reporting both forgetting and retention in the same controlled ablation would directly test this claimed advantage.

W2: The evaluation needs stronger validation and clearer reporting.

The paper does not validate the LLM judges against human-labeled responses or report agreement among judges. The human audit checks benchmark construction rather than response scoring. Table 8 also contains an apparent numerical inconsistency: if its metrics use the same paired counts, UFR = 8.74% and NDR = 5.91% imply APA ≈ 86.31%, rather than the reported 72.36%. A different aggregation procedure may explain this, but it needs to be specified. These issues limit confidence in the reported results.

Questions for authors

Q1: In the controlled ablation, how does Memory Receipts compare with whole-unit deletion in retaining valid information within affected memory units? Please report retention results alongside forgetting performance.

Q2: Have the judge labels been checked against human annotations of actual model responses? Please report the sample size, accuracy, and agreement among judges.

Q3: Can you provide the raw paired counts needed to reconcile Table 8 with the metric definitions?

1 comment
AuthorsOpenReview ↗

Response to Reviewer CKPM

Response to Reviewer CKPM

Thank you for the constructive review. We completed the full retention ablation, calibrated our judges on 200 human-labeled responses, and confirmed the Table 8 error you identified.

W1 & Q1: Does reconstruction preserve more valid information than whole-unit deletion?

A: Yes—across all 1,087 cases, local reconstruction substantially improves retention while achieving full counterfactual forgetting on our paired metric (APA). Prior metrics like target absence (FAA) cannot tell whether an agent truly forgot or masked leaks via collateral memory loss or opposite-direction shifts; pairing legacy utility metrics (MPA/FAA/FAMA) with APA=B/(B+I+N) against a matched Never-Seen (NS) world tests both jointly (shared initial writer states and NS references; Qwen3-4B BF16 answers; local Qwen3-32B BF16 judge, memora_binary_v2_compact, temp 0, non-thinking, 64-token budget):

MethodMPAFAAFAMAAPA (Ours)
Memory Receipts44.8190.3936.33100.00
Whole-unit deletion34.5991.0127.6950.23
Clean Rebuild44.8190.3936.33100.00

Rates are percentages (2,738 retention, 3,538 target criteria; FAMA averages case scores). On legacy FAA alone, deletion looks slightly higher (91.01 vs. 90.39), yet our APA drops to 50.23%—exposing 98 residual leaks (I) and 120 over-deletions (N) canceling in FAA—while losing 10.23 MPA points. Receipts matches Clean Rebuild, improving MPA by +10.23 (context-cluster 95% CI [7.29,13.44]) and FAMA by +8.64 (FAA 0.62 lower, unrounded; MPA gains on four tracks, is 1.43 lower on LongMemEval and ties on PersonaMem).

Affected-unit answer utility. On 367 cases whose declared retention supports all lie inside affected units (80 Memora, 88 PrefEval, 199 MAB), MPA rises from 131/589 (22.24%) under deletion to 271/589 (46.01%) under Receipts—a +23.77-point gain (FAA: 590/680=86.76% vs. 592/680=87.06%; FAMA: 36.56% vs. 17.35%).

Local factual retention. Across 5,307 surviving-fact checks in Memora/MAB (260 invalidation groups, 660 unit instances), Receipts/Clean express 4,116 facts (77.56%) and retain 3,061/3,193 initially expressed facts (95.87%) (132 lost; deduplicated over 3,353 patterns: 78.26% expression, 2,060/2,172=94.84% conditional retention), whereas deletion removes the unit text.

W2 & Q2: Human calibration and inter-judge agreement.

A: Reporting inter-judge agreement, upstream calibration and our response calibration separately (historical 865 cases, five controlled methods, GPT-5.5 / Haiku 4.5 / Gemini 2.5 Flash):

ActorUnique inputs3-judge pairwise agreementCohen's kappa
Qwen3-4B20,41288.27-92.50%0.718-0.806
Llama-3.1-8B21,20287.31-91.11%0.699-0.794

Human response calibration. Beyond Memora Appendix C.2 (88.3% LLM-majority/human agreement, kappa 0.86-0.90 on 100 items), each criterion here is a direct yes/no check on explicit response text. On 200 stratified actual response-criterion pairs (100 Qwen3-4B, 100 Llama-3.1-8B across six tracks), two blind annotators (3rd-annotator adjudication on 4 splits) reach 98.00% agreement (196/200, kappa=0.960). Validating our protocol, our three-judge majority vote matches human labels on 96.00% (192/200, kappa=0.920), while local Qwen3-32B (used to scale ablations) matches 87.00% (174/200, kappa=0.740).

W2 & Q3: Reconciling Table 8 with the metric definitions.

A: You are right: original Table 8 contains statistical reporting errors. We take full responsibility and thank you for catching them. Under common pooled counts with B>0, APA = 1 / (1 + UFR/(1-UFR) + NDR/(1-NDR)) exposes clear inconsistencies:

Original rowReported APAAPA implied by UFR/NDR
Whole-unit deletion72.36%86.3124%
Receipts, 80%88.73%92.0026%
Receipts, 90%94.96%95.6879%

Implied values are a diagnostic check, not replacement scores. From our 1,087-case ablation (4,348 answers, 17,416 unique judging inputs), recomputed raw paired counts (3,538/row) are:

MethodBINX
Receipts340003198
Whole-unit deletion220981203100
Clean Rebuild340003198

Deletion yields PA/APA/UFR/NDR = 93.84/50.23/30.82/35.29%; Receipts/Clean yield 100/100/0/0% under matched compiler/cache controls (utility measured in Q1).

Camera-ready: In the camera-ready, we will incorporate every fix and addition prompted by your review—replacing Table 8 with Shared-Writer Reconstruction Ablation (panel a: raw B/I/N/X and paired metrics; panel b: MPA/FAA/FAMA, removing unverified 80%/90% rows), adding the 367-case/5,307-fact retention analyses, 200-pair human calibration and exact judge/denominator specs, and releasing all data and scripts—and sincerely thank you for your valuable contribution to strengthening our work.

Reviewer_YQtmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies forgetting in long-term agent memory and introduces a benchmark of 1,087 cases across five public benchmarks. It proposes Memory Receipts, a provenance-based mechanism that reconstructs affected memory units after invalidation while leaving unrelated units unchanged. The paired evaluation is a useful addition, but the novelty of the repair mechanism and the strength of the evidence for verifiable forgetting remain open to discussion.

Critical strengths
  1. The paper makes an important distinction between removing a target fact and removing the influence of the source that introduced it. Its paired Observed–Invalidated and Never-Seen setup also accounts for cases where a model can independently regenerate the same information. This provides a more informative evaluation perspective than measuring target absence alone.
  2. Memory Receipts tracks the source records supporting each derived memory unit and reconstructs only the units affected by invalidation. The paper also states the conditions under which this procedure is equivalent to Clean Rebuild at the external-memory level. The receipt-coverage ablation further supports the importance of complete lineage tracking.
Critical weaknesses
  1. Prior work has explored long-term memory forgetting, memory governance, and agentic unlearning, including From Recall to Forgetting, GateMem, and Agentic Unlearning [1–3]. However, the paper does not fully establish how its benchmark formulation and repair mechanism advance beyond these directions and established provenance-based maintenance techniques [4].
  2. The reported paired agreement is high, but this alone does not establish successful forgetting, as both worlds may fail to produce the target behavior. The reliability of the LLM judges is also not fully established. In addition, the method assumes complete source lineage and access to original records, which may not hold in practical deletion scenarios. [1] Uddin et al. From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents. Findings of ACL 2026. [2] Ren et al. GateMem: Benchmarking Memory Governance in Multi-Principal Shared-Memory Agents. 2026. [3] Wang et al. Agentic Unlearning: When LLM Agent Meets Machine Unlearning. 2026. [4] Buneman et al. Why and Where: A Characterization of Data Provenance. ICDT 2001.
Questions for authors
  1. A more direct comparison with existing agent-memory forgetting and provenance-based maintenance approaches would help clarify the specific contribution of Memory Receipts beyond combining these ideas.
  2. Although the paper reports paired-state distributions for Qwen3-4B, a more systematic breakdown across actor models and lifecycle tracks, together with independent validation of judge reliability, would strengthen the evidence for the proposed forgetting evaluation.
  3. A controlled evaluation with incomplete receipts or unavailable source records would help characterize how robust the method remains when its main assumptions are violated.
1 comment
AuthorsOpenReview ↗

Response to Reviewer YQtm

Response to Reviewer YQtm

Thank you for your constructive review. Rates are percentages.

W1 & Q1: Contribution and direct comparisons.

A: Our core contribution is treating invalidation as a first-class lifecycle state (revoking preferences or superseding facts) and making source-level forgetting testable across five datasets. Legacy absence metrics (FAA) cannot tell true forgetting from priors or collateral memory loss; pairing Observed-Invalidated (OI) with a matched Never-Seen (NS) world (same survivors and query) isolates counterfactual forgetting via APA=B/(B+I+N).

Relation to prior work. Receipts adapts provenance maintenance to LLM-compiled memory: track writer inputs, rebuild affected units from survivors, delete unsupported units and reuse others—matching Clean Rebuild across all 1,087 states under fixed partitions and a memoized compiler. SBU unlearns parameters; PGSR maintains sequential state; AfterDeleteBench rebuilds shared summaries.

Direct mechanism comparison. KeepShared (keep shared text, delete unsupported units) vs. survivor-assisted Edit on 60 cases/two actors (same survivors, 512-token budget; Edit also reads old summaries):

ActorReceipts MPA/FAA/APAKeepShared MPA/FAA/APAEdit MPA/FAA/APA
Qwen3-4B32.79/81.54/100.0033.61/76.92/54.2933.61/82.31/62.07
Llama-3.1-8B36.89/88.46/100.0042.62/75.38/34.2940.98/75.38/42.42

Edit gets higher point-estimate MPA (95% CIs cross zero) and 48.14%/54.94% more input tokens, but diverges from NS (APA 62.07/42.42) and drops Llama FAA (75.38 vs. 88.46; mechanism check, not full SBU/PGSR; capped outputs kept).

W2: High agreement may reflect joint failure.

A: APA=B/(B+I+N) excludes joint misses X, while MPA/FAA/FAMA measure utility separately. Pairing APA with legacy metrics on our 1,087-case shared-writer ablation (Qwen3-4B BF16, local Qwen3-32B judge) shows why legacy metrics alone cannot verify forgetting:

MethodMPAFAAFAMAAPA (Ours)
Receipts44.8190.3936.33100.00
Whole-unit deletion34.5991.0127.6950.23
Clean Rebuild44.8190.3936.33100.00

Legacy FAA makes deletion look higher (91.01 vs. 90.39), yet APA=50.23% (B/I/N/X=220/98/120/3100) exposes 98 leaks (I) and 120 over-deletions (N) canceling in FAA, plus a -10.23 MPA drop (Receipts +10.23, cluster 95% CI [7.29,13.44]; FAMA +8.64; MPA/FAA pool criteria, FAMA averages cases).

Independent answer variability. Across 1,080 uncached requests (60 cases x 2 actors x 3 rounds of OI/NS-A/NS-B, fixed writer states, temp 0), OI/NS-A PA/APA is 99.49/97.14 (Qwen) and 98.46/89.83 (Llama); NS-B/NS-A is 100/100 and 97.18/82.54—separating answer variability from cache controls (100%) without isolating writer variability.

W2 & Q2: Judge reliability and track breakdown.

A: Judge validation. Historical answers have GPT-5.5, Haiku 4.5 and Gemini 2.5 Flash labels: pairwise agreement is 88.27-92.50% (kappa 0.718-0.806, 20,412 Qwen) and 87.31-91.11% (kappa 0.699-0.794, 21,202 Llama). On 200 stratified response-criterion pairs, two blind annotators (3rd-annotator adjudication on 4 splits) reach 98.00% agreement (196/200, kappa=0.960); our three-judge majority vote matches human labels on 96.00% (192/200, kappa=0.920), and local Qwen3-32B matches 87.00% (174/200, kappa=0.740).

Track breakdown. 1,087-case Qwen retention and 60-case Llama Edit paired states (10 cases/track; targets 80/10/10/10/10/10):

TrackQwen Receipts/deletion MPALlama Edit B/I/N/X
Memora46.45/36.341/11/0/68
PrefEval52.50/46.254/3/0/3
LongMemEval61.43/62.865/1/0/4
PersonaMem50.72/50.724/2/0/4
MAB-SH44.62/16.670/0/0/10
MAB-MH9.14/2.150/1/1/8

MPA gains on four tracks, drops on one and ties on one.

Q3: Incomplete receipts or unavailable source records.

A: Three missing-position seeds on 60 cases:

ConditionQwen APALlama APA
90% receipt edges70.8966.67
80% receipt edges61.9055.38
10% sources unreadable69.7450.00
20% sources unreadable69.2350.75

Counts average per case before pooling. Source loss hits 49/60 cases (all 60 kept); unreadable survivors trigger deletion (loss 10.34%/20.21%). APA degrades predictably, while MPA is non-monotonic (independent NS 90%/80% APA: 73.68/64.20 Qwen, 54.41/50.00 Llama; Llama source-loss MPA CIs cross zero; 2,880 records kept).

Camera-ready: In the camera-ready, we will incorporate every improvement prompted by your review—prior-work links, mechanism/retention/robustness comparisons, uncached controls, human calibration, and verified per-track/actor tables with exact judges and denominators—and sincerely thank you for your valuable contribution to strengthening our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.