- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies LLM agents with fixed weights that rely on external persistent memory: even after an original interaction record has been deleted by address, summaries or other derived information previously generated from it may still enable recovery of a secret. The authors construct AfterDeleteBench, using paired environments in which a secret is first exposed and then deleted (forget), or never exposed (clean), to control for guessability. They separately measure exact reconstruction R, candidate write-back C, recovery in a new session after removing the audit dialogue A, and the joint event E=R∧C∧A. The paper tests flat/hierarchical memory, descendant closure under complete provenance tracking, HMAC-based exact literal write filters, different querying auditors, and JSON answer contracts/scoring parsers.
For Qwen2.5-7B-Instruct with the main hierarchical-memory and exact-source-deletion condition, one DIRECT query produces ΔRec=0.680/0.640 on 200 paired profiles in each of the IID and OOD splits, respectively, with both clean rates equal to 0. In the C1 condition, the complete secret string is absent, but its components and a public concatenation rule remain; exact recovery rates for Qwen, Mistral, and Phi-3 across six model/split cells remain 0.615–0.990 (Table 2). The authors further isolate the role of write-back through matched write/no-write controls and show that mismatches between answer format and parser can incorrectly score outputs containing the secret as zero recovery. The authors also explicitly restrict their conclusions to synthetic exact strings, fixed memory constructions, and a limited operation vocabulary.
Critical strengths
-
Paired attribution and separation of lifecycle metrics are precise. The forget/clean worlds share the model, non-secret background, memory operations, and query budget, reducing confounding due to the guessability of secrets. Separating R, C, and A is particularly important because existing memory residue may still cause A>0 even if C=0. Module B compares write-allowed with no-write under matched states; activation in Qwen's R=1 subset increases from 64/76 to 76/76 (IID, +0.158) and from 58/75 to 69/75 (OOD, +0.147), without incorrectly calling E the causal effect of write-back (Table 9, p.18).
-
Answer-contract diagnostics expose blind spots in the evaluation system itself. In Module A, Mistral/Phi-3 have official R=0 under the legacy format, but 77/104 and 78/104 raw IID responses still contain the target; after alignment, clear recoveries can be counted. Even the same Qwen memory state exhibits an apparent write-back effect under the legacy contract that is opposite to that under the aligned contract (Table 10, p.19). This empirical diagnosis of measurement distortion, together with C1 reconstruction without the complete literal string, constitutes the two most informative contributions.
Critical weaknesses
-
The main risk and protection demonstrations are still far from real semantic agent memory. The core “hierarchical memory summary” is an immutable string formed by verbatim newline concatenation in record-ID order, and all post-deletion memory is inserted directly into the model prompt (§3, Appendix I). Therefore, deleting the original record but retaining a summary containing the complete secret, which is then recovered by a DIRECT query, is to a considerable extent a consequence of the construction's redundant information. Although C1 excludes the complete string, it retains all three segments and the public REGION-MNEMONIC-NUMBER combination rule; it verifies controlled component concatenation rather than nonliteral recovery from learned semantic memory. With complete provenance, closure yielding zero leakage after reconstructing deterministic summaries is also almost true by construction. The paper honestly acknowledges these limitations, but as a benchmark for LLM agents, it lacks at least one independent real memory stack with model-generated summaries or incomplete provenance relations, making its field-level importance difficult to assess.
-
The benchmark's discriminative power and measurement validity need further evidence. Although the paper includes a clean world, it lacks an exposed-and-retained world; therefore, ΔRec does not indicate “what fraction of information that was normally recoverable remains after deletion.” This limitation is acknowledged in §3 (p.3) and Appendix J. The main metric R uses strict exact matching of the top-level JSON value, while the paper itself shows that format mismatch can fail to count substantial actual disclosure; raw responses from some historical experiments were not preserved, preventing a unified retrospective comparison of leakage. These limitations make it difficult to determine how much usable information deletion actually removed and to compare complete and partial leakage consistently. This weakens the benchmark's interpretability as a general measurement tool, without denying the information persistence found in its controlled experiments.
Questions for authors
-
Could the authors repeat the core forget/clean audit in an independently implemented agent-memory framework using model-generated/compressed summaries, incomplete provenance, and actual retrieval limits, and report complete or partial leakage of natural-language/semantic sensitive information? If the quantitative conclusions remain valid in a real memory stack, this would clearly increase my assessment of the benchmark's significance.
-
Could the authors add an exposed-and-retained reference world and, under a unified response contract, report official exact R, raw-output exact-target-anywhere, and at least one semantic/partial-leakage metric, to show how much originally available information deletion actually removes? In particular, please distinguish genuine nondisclosure from parser false negatives.
2 comments
Thank you for your thoughtful review.
We have posted a general response addressing the shared concerns, along with a detailed reply to your individual comments. We have also updated the manuscript, with revisions highlighted in blue, and the supplementary materials.
We hope these responses and revisions have addressed your concerns. If any questions or issues remain, we would be happy to provide further clarification.
If our responses have resolved the concerns that influenced your initial assessment, we would greatly appreciate it if you could reconsider and update your rating accordingly when finalizing your review.
Memory construction and realism
V12i:
Track A keeps the original controlled memory, whose summary contains the source text verbatim; it remains useful because everything that survives deletion is inspectable. Track B adds model-generated memory: Mem0 extracts memories from each interaction with a shared Qwen3.5-9B writer, and each answerer receives at most three retrieved records above a score cutoff, from pools of 14–24 stored items (Appendix L.3, L.5). Mem0 supplies extraction and retrieval; source logging, deletion, provenance and the interventions are our wrapper. This is a controlled pipeline, not a production agent or a test of native Mem0 deletion.
Across five answerers (Qwen2.5, Mistral, Phi-3, Qwen3.6, Gemma 4), forget-world exact recovery is 44–71% in Track B and 55–75% in Track A, and every clean cell is 0/104 (Table 3; Appendix M.2). Exposure-associated recovery therefore persists with memories generated by a separate model and with limited retrieval.
For provenance, deleting every memory recorded as derived from the source (P1) leaves 0/104 exact recoveries in each structured cell, whereas hiding one recorded dependency (P2) restores 35–74/104 (Appendix M.5). P2 deliberately hides a known edge: it shows how much closure depends on complete lineage, not how often lineage is incomplete in practice.
For natural-language disclosure, 48 fictional profiles each contain three specified fields. All three appear in 31–37 of 48 forget replies and at least one in every forget reply, with no clean matches (Appendix M.4). The values were present in retrieved memory, so this measures disclosure of stored facts, not semantic inference. C1 asks a different question: memory keeps only a secret's components and a public joining rule, and forget recovery is 83–104/104 (Appendix M.3). That is rule-guided composition; in the core panels, every exact success had the intact secret in the input (Appendix M.7).
What source deletion changes
V12i:
The retained world starts from the same exposed state as forget but keeps the source records. Forget minus clean measures recovery associated with exposure; retained minus forget measures how exact recoverability changes when the source is deleted rather than kept. Neither measures the amount of information erased (Section 3; Appendix L.1).
The answer depends on the model. In Track B with three retrieved records, keeping the source adds 1–7 recoveries per 104 profiles for Qwen3.6, Gemma and Phi-3, but 14–21 for Qwen2.5 and Mistral; these four contrasts remain significant after Holm correction across the ten top-3 comparisons (Appendix M.9). With a single retrieved record, all five gain 6–13. In Track A, differences involve at most four profiles, apart from a Mistral scoring artifact discussed below.
The Qwen2.5 and Mistral gains are not simply explained by the secret disappearing from the input. In 72 of the 75 profiles where they recovered it only with the source retained, it was also present in the forget-world input; when these models saw the secret but did not give it, they mostly abstained (Appendix M.7). The measured effect of deletion thus depends on answer behavior as well as memory contents, which is why the benchmark measures both. We draw no conclusion about model size.
Disclosures that exact scoring misses
V12i:
All five answerers are asked for a top-level JSON
value, and every reply is saved. Official exact scoring reads only that field; a second check asks whether the complete secret appears anywhere in the reply. For Qwen3.6 and Gemma the two agree on all 3,328 structured evaluations per model. For Qwen2.5, Mistral and Phi-3, 4, 137 and 27 of those evaluations contain the secret but fail official scoring, mostly because the answer is nested, fenced in Markdown or placed under another key (Appendix M.8). These counts span all executed conditions and are not independent profiles. They account for Mistral's negative official Track A IID difference, which reverses under raw scoring.Partial disclosure is reported separately, as private-component mentions (Appendix M.8) and through the fictional-field rubric (Appendix M.4). A negative result on every check still does not prove semantic nondisclosure.
Scope
These experiments use one direct question per world, not the historical write-back and recall stages, and three answerers receive a format instruction (Appendix L.6). They show that the benchmark separates exposure, deletion and measurement effects under controlled conditions; they do not establish production-agent behavior or semantic forgetting guarantees.