- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes to produce a new benchmark for agent memory and evaluation. It is especially concerned with forgetting.
Critical strengths
The basic argument as to why a new benchmark is required seem reasonable enough.
The paper generally does a good job of putting together a narrative and explaining its contributions clearly.
Critical weaknesses
Very little by way of technical contribution. This is perhaps to be expected of a benchmark paper, but still, it can be hard for a paper to be highly impactful unless the benchmark becomes highly popular, which is quite rare.
The main "memory receipts" contribution is fine but not that big. I think the main issue here (which can perhaps be clarified in rebuttal) is that it's hard to tell how much the benchmark and model are paired in a way that a "win" is guaranteed, i.e., the paper produces a benchmark to highlight a very specific issue and then introduces a model to solve that specific issue (therefore easily beating other strategies). That's fine if the specific issue being studied is sufficiently important, but that's hard for me to tell.
The actual benchmark produced here is fairly small.
The paper, even compared to others in my batch, is heavily AI-based. It is hard to tell if there is value in humans reading this paper, and seems less likely that there'll be value to future AI models following up on this work given that it's so heavily produced by AI (according to the authors' own claims). The authors claim to have done little more here than verified the output.
Questions for authors
See above.
Response to Reviewer NHWP
Response to Reviewer NHWP
Thank you for recognizing our motivation. Below we clarify our perspective, trade-offs and the authors' role.
W1: Technical contribution and what readers learn.
A: In real-world personalization, invalidation is a first-class lifecycle state—users revoke preferences, update facts and delete records—yet prior metrics cannot tell whether an agent truly forgot. We formulate this problem, introduce counterfactual paired metrics (PA/APA) to expose failure modes hidden by legacy metrics, and quantify local repair trade-offs:
1. Why legacy metrics fail & counterintuitive insight (independent of Receipts). Prior evaluations equate target absence (FAA) with forgetting. Comparing Observed-Invalidated (OI) against a matched Never-Seen (NS) world (same survivors and query) shows why FAA misleads: an agent may infer a target from priors/cues without residue (shared hit B), leak OI-only residue (I), or inflate FAA via collateral loss (N or joint miss X). On Causal-Memora, Qwen3-4B Append-only scores FAA=74.92% over 2,787 pairs, yet B/I/N/X=193/506/155/1933—masking 506 residual leaks alongside 193 prior hits. Defining forgetting via APA=B/(B+I+N) plus retention gives a lasting diagnostic lens.
2. Lifecycle benchmark & executable local repair. We unify five datasets into 1,087 paired cases with source-to-unit mappings and retention/forgetting criteria. Receipts adapts provenance tracking to LLM-compiled memory: rebuild affected units from survivors, delete unsupported units and reuse others—matching Clean Rebuild across all 1,087 states under fixed partitions and a memoized compiler.
W2: Practical importance and is a "win" guaranteed?
A: No—neither our benchmark construction nor empirical results guarantee a win for Receipts. Histories, partitions and criteria are frozen before generation; receipts never enter prompts.
Internal metrics & trade-offs. Pairing legacy metrics with APA on our 1,087-case shared-writer ablation (Qwen3-4B, local Qwen3-32B judge) clarifies both our method's strength and real trade-offs:
Deletion looks higher on legacy FAA (91.01 vs. 90.39), yet our APA=50.23% exposes 218 OI-vs-NS divergences (98 leaks I, 120 over-deletions N) and a 10.23-point MPA loss (Receipts +10.23, 95% CI [7.29,13.44]; FAMA +8.64; LongMemEval MPA is 1.43 lower). Against KeepShared and survivor-assisted Edit (60 cases, two actors), Edit gets higher point-estimate MPA (33.61 vs. 32.79 Qwen; 40.98 vs. 36.89 Llama; CIs cross zero) but diverges from NS (APA 62.07%/42.42% vs. 100%) and drops Llama FAA (75.38 vs. 88.46).
External benchmark (GateMem). On 20 fixed GateMem episodes (347 checkpoints, 2,082 answers; 5 dev, 15 primary) with native rubrics (higher utility, lower failure better):
On the primary 15, Receipts beats RAG-Naive by +11.29 utility (95% CI [2.44,21.26]) and -23.48 failure ([14.71,32.08]); Global Rewrite has higher utility/higher failure, and deletion CIs cross zero. Receipts cuts executed-edit tokens from Clean's 309,526 to 127,805 (58.71% less, 91 vs. 205 calls; Qwen3-4B actor, Llama-3.1-8B judge; shared parsing gaps and capped outputs kept).
W3: Benchmark size.
A: Our 1,087 paired cases match or exceed LongMemEval (500), PersonaBench (582) and Memora (600), alongside MemoryAgentBench (2,071) and PrefEval (3,000), uniquely pairing OI/NS worlds across five invalidation types.
W4: AI use and the authors' contribution.
A: We apologize our AI Use Statement was overly broad and caused misunderstanding. The human authors originated the problem, paired-world formulation, Memory Receipts, experimental design and insights. AI served primarily for coding assistance (conversion/eval scripts) and polishing—never to invent histories, select cases or replace human judgment (humans audited construction and 200 responses: 98.00% inter-human, 96.00% 3-judge match).
Camera-ready: In the camera-ready, we will incorporate every improvement prompted by your review—foregrounding our counterintuitive insights, adding internal trade-offs, external GateMem results and benchmark-scale comparisons, and rewriting the AI Use Statement to state our human contributions accurately—and sincerely thank you for your valuable contribution to strengthening our work.