- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses CRUD operations on parametric agent memory. Existing parametric memories such as MLP Memory are write-only and cannot handle fact updates or selective forgetting. MemEdit casts each memory operation as four stages, locate, preserve, intervene, and verify: integrated gradients localize each memory to a neuron footprint, and the three write operations share one constrained rank-one rule whose output side is confined to the footprint and whose input side is projected onto the null space of stored keys, so that INSERT preserves existing memories exactly and DELETE erases via projected gradient ascent without changing other stored keys. Every write is journaled as a reversible low-rank delta and verified, and a Mixture-of-Memory-Experts adds independently editable shards when null-space capacity is exhausted.
Critical strengths
- The problem formulation and the unified write rule form a coherent conceptual contribution. Casting CRUD on parametric memory as a duality between output-side localization and input-side preservation makes Insert, Modify, and Delete instances of one rank-one rule, with exact preservation of stored keys (Propositions 1 and 2) and null-space rank as a directly observable capacity signal that triggers MoME expansion. The design elements interlock rather than merely stack.
- The empirical treatment is systematic and honest. Beyond headline metrics, the paper provides causal validation of the attribution (necessity and sufficiency tests, 0.82 correlation with an exhaustive causal ranking), deletion robustness under paraphrase, adversarial extraction, relearning, and membership inference, controller error analysis with verification and rollback, a 10K-edit stress test with long-horizon retention tracking, and multi-seed results with standard deviations. The non-certified nature of Delete and the scope of the guarantees are discussed openly.
Critical weaknesses
- There is a gap between the preservation guarantees and deployment conditions. Propositions 1 and 2 hold only for keys cached in the key bank, whereas real queries are paraphrases that are not cached. Table 14 shows that MP on held-out keys falls to 92.2% at T = 5 and 83.1% at T = 20, and paraphrase ESR drops to 87.3% (Table 12). The headline 97.3% MP is measured with key-bank-anchored probes; the paper never directly reports preservation on the natural query distribution after an edit stream, so the distance between headline numbers and deployment conditions remains unquantified.
- The core technique is a composition of established components and the downstream gains are modest. IG attribution, null-space projection (AlphaEdit), gradient-ascent unlearning, and sparse MoE all have precedents, and the paper's own Naive-Compose ablation (Table 8) attributes 4.6 ESR at 500 edits to the way they are combined. The margin over the strongest baseline AlphaEdit-M is +2.2 ESR and +2.5 MP (Table 1). On downstream LoCoMo, MemEdit improves over the non-editable MLP Memory by only 1.4 F1 (46.2 vs. 44.8), while AlphaEdit-M (44.1) falls below the no-edit baseline. The value of CRUD lies mainly in operability (updates, deletion, auditability) rather than QA accuracy, and the paper does not consistently frame its conclusions that way.
Questions for authors
- Please report preservation on held-out natural queries (not probes constructed from memory statements) after an edit sequence. How large is the key bank in deployment and how are its keys chosen? What fraction of actual queries falls within the guarantee when coverage is incomplete?
- In Table 1, AlphaEdit-M's LoCoMo F1 (44.1) is below that of the entirely non-editable MLP Memory (44.8). Does editing itself cost downstream accuracy? If so, how much of MemEdit's +1.4 F1 over MLP Memory is attributable to the CRUD capability?
- What exactly was adapted in ROME-M, MEMIT-M, and AlphaEdit-M for the memory module? Did they receive the same footprint localization, the same per-layer editing budget, and the same verification with rollback? These details are needed to judge the fairness of the comparison.
4 comments
Response to Reviewer 3HR1 (Part 2)
(Following Part 1)
Fraction within the guarantee. 84.7% of natural queries fall inside the span in the default configuration, 61.4% with memory keys alone, and 87.3% if the anchor set is pushed to 3,072 keys. MP is 97.5% on the covered fraction against 82.4% on the uncovered fraction, so coverage is the right lever, but its cost is editing capacity rather than storage: each key consumes one null-space dimension, so at the anchor set and the inserts a shard can still absorb come out of one budget, and the last configuration leaves a shard with essentially no room to write. Raising coverage further at constant capacity needs a larger key dimension or more shards, each with its own bank.
Q2: In Table 1, AlphaEdit-M's LoCoMo F1 (44.1) is below that of the entirely non-editable MLP Memory (44.8). Does editing itself cost downstream accuracy? If so, how much of MemEdit's +1.4 F1 over MLP Memory is attributable to the CRUD capability?
A4: Yes, editing costs accuracy on content it does not target, and Appendix C.19 (new) separates the two effects. Splitting LoCoMo by whether a question touches a memory the stream edited:
| Method | Touches an edited memory (38%) | Does not (62%) | Overall |
|---|---|---|---|
| MLP Memory (no edits) | 39.2 | 48.2 | 44.8 |
| AlphaEdit-M | 44.6 | 43.8 | 44.1 |
| MemEdit | 49.8 | 44.0 | 46.2 |
So AlphaEdit-M's gain on edited content is real, +5.4 over the frozen module, but its loss on untouched content is −4.4, and the aggregate lands below the no-edit baseline. MemEdit gains 10.6 on edited content and loses 4.2 on untouched content. The +1.4 overall is therefore the sum of a large gain on 38% of questions and a smaller loss on the remaining 62%, and a method is useful only when the former outweighs the latter.
Enabling operations one at a time locates the gain further:
| Configuration | LoCoMo F1 | Δ vs. MLP Memory |
|---|---|---|
| MLP Memory (no edits) | 44.8 | 0.0 |
| MemEdit, reads only | 44.8 | 0.0 |
| MemEdit, Insert only | 46.5 | +1.7 |
| MemEdit, Insert and Modify | 46.4 | +1.6 |
| MemEdit, full CRUD (default) | 46.2 | +1.4 |
Reads alone reproduce the frozen module exactly, which is a useful sanity check. Insert accounts for the entire positive effect on LoCoMo; Modify is near-neutral because LoCoMo asks few questions whose answer changed mid-stream; Delete is slightly negative by design, since it removes information some questions still ask about. On LongMemEval, whose Knowledge Updates category targets exactly the revised facts, Modify instead carries the largest single gain (+4.2 over AlphaEdit-M). We now state in the limitations that the accuracy benefit is benchmark-dependent in this specific way.
Q3: What exactly was adapted in ROME-M, MEMIT-M, and AlphaEdit-M for the memory module? Did they receive the same footprint localization, the same per-layer editing budget, and the same verification with rollback? These details are needed to judge the fairness of the comparison.
A5: Appendix C.1 (new) answers this in full; the direct answers to your three sub-questions are: no, partly, and no, and we now isolate the last of them.
Footprint localization: no. Footprint-restricted writes are a MemEdit component under test, so the baselines write to their own natural target, unrestricted rows of for ROME-M and AlphaEdit-M, and the layers MEMIT-M spreads over.
Per-layer editing budget: the same target layer and the same held-out validation sweep for selecting it, but MEMIT-M additionally spreads batched updates over layers 2 and 3, since restricting it to a single layer would disable the mechanism that distinguishes it from ROME. Each baseline was tuned on a held-out split of 100 operations over the ranges in Appendix C.1, selecting the configuration with highest ESR subject to MP above 90%.
Verification with rollback: no in the main table, where MemEdit also runs single-pass with no rollback, so the comparison is like for like on that axis. Because the component is nonetheless ours and could be given to any method, we now report what happens when it is granted to the baselines: AlphaEdit-M rises to 93.6 ESR and 95.7 MP, MEMIT-M to 91.5 and 92.6, WISE-M to 93.0 and 97.0, while MemEdit with the same safeguard reaches 95.4 and 97.9. The ordering is unchanged, and a reader can now see the magnitude of that factor rather than taking the main-table comparison on trust.
All baselines additionally share the probe and target construction of Appendix B.2, the operation labels, the backbones, the seeds, and the evaluation protocol. Appendix C.1 presents this as a table so the asymmetries are visible at a glance.
Follow-up on Additional Evaluations and Clarifications
Dear Reviewer 3HR1,
Thank you again for your constructive review and positive assessment of our work. We would appreciate hearing whether the new natural-query preservation and key-bank coverage analyses, the decomposition of downstream gains and editing costs, and the clarified baseline adaptations and rollback comparisons address your remaining concerns. We have also revised the paper’s framing to better reflect both the operability benefits and the limitations of CRUD. Please let us know if further clarification would be helpful. If these additions strengthen your assessment, we would be grateful if you would consider a higher rating.
Best regards,
The Authors
Thank you for the detailed rebuttal and additional experiments, which address several of my concerns; however, some limitations remain, and I will maintain my original assessment.
Response to Reviewer 3HR1 (Part 1)
W1: There is a gap between the preservation guarantees ...
A1: You have identified the sharpest limitation of the method, and we have both measured it and stopped leaving it for a reader to assemble from scattered appendices.
Appendix C.10 (new) measures ESR and MP on the natural query distribution after an edit stream, using the LoCoMo QA questions, which were written without reference to our probes: ESR 85.1 and MP 95.2, against 94.6 and 97.3 on the probes and 87.3 and 96.4 on held-out paraphrases. Section 3.3 now qualifies the propositions in the body, directly where they are stated, and the limitations section gives the gap with the numbers and names coverage as the cause, so the distance between the headline figures and deployment conditions is flagged where the claim is made and quantified where it is measured.
Appendix C.10 also makes the mechanism explicit rather than leaving the gap unexplained. Using the same δkey test Insert applies, we report what fraction of natural queries falls inside the protected span and split MP along that line: coverage is 84.7% in the default configuration, MP on covered queries is 97.5%, and MP on uncovered queries is 82.4%, so essentially the entire erosion lives outside the span. The residual on covered queries is attributable to Modify, the one write that imposes preservation as a penalty rather than a hard constraint; in a delete-only stream, where only the hard constraint is active, the figure is 99.98%. This also reframes Table 14's held-out column: it is not an unexplained residual but the same quantity measured under a delete-only stress schedule.
Two interventions narrow the gap and we report both, including the one that turns out to be weak. Paraphrase-averaged attribution over five probes raises natural-query ESR from 85.1 to 90.6 at roughly 4.6 times the attribution cost. Enlarging the anchor set raises coverage from 84.7% to 87.3% and natural-query MP from 95.2 to 95.6, and no further: every key in the bank consumes one null-space dimension, so coverage is bought with editing capacity and is capped by the key dimension, and the configuration above already leaves a shard with almost no room to insert. We had previously described this lever as a storage budget, which understated its cost. Neither intervention closes the gap.
W2: The core technique is a composition of established components ...
A2: We accept the framing point and have changed the paper accordingly rather than arguing the margins are larger than they are.
Framing. The Conclusion now states that the value of the CRUD interface lies as much in operability, namely reversible updates, targeted deletion, and an auditable journal, as in the accuracy it buys, and that the gains on untouched content are deliberately small by design. A new limitation, "Scope of the downstream gains", says the benefit is concentrated on questions that interrogate revised or retracted content and is close to neutral elsewhere, so benchmarks that do not ask about changed facts will understate the framework. The abstract and the contribution bullet no longer claim the highest edit success rate among all methods.
Margins, stated accurately. Adding WISE-M as a baseline reduces the MP margin we can claim from +2.5 to +0.8, since memory-based editors are stronger on preservation than the weight-editing baselines were. We report that rather than omitting the comparison: the ESR margin over the strongest editing baseline remains +2.2, and the margin over the strongest preservation baseline is +0.8 MP with +2.5 ESR.
Where the composition matters. The Naive-Compose number you cite is the 500-edit figure; the same ladder shows 20.0 ESR at 10K edits, and we now say explicitly that the gap widens with the stream, since the properties being compared are about accumulation rather than about any single write. The composition is worth most exactly where a lifelong memory system lives.
Q1: Please report preservation on held-out natural queries ...
A3: All three are in Appendix C.10 and summarized in A1 above.
Preservation on held-out natural queries after an edit sequence. MP 95.2 and ESR 85.1 on the LoCoMo QA questions after the edit stream, against 97.3 and 94.6 on the probes.
Key bank size and key selection. Each shard's bank holds 2,560 anchor keys plus one key per memory written into that shard. It is bounded by construction: a shard reaches its expansion threshold once the bank spans 3,482 directions, about 28MB at 8KB per key and under 4% of a 740MB shard, after which new memories are routed elsewhere rather than enlarging the bank. Memory keys are the layer-l∗ keys of the probes that defined them, so their selection is determined by the write stream. Anchor keys are sampled from the pretraining corpus and stratified so their kNN target distributions span the module's output entropy range; they are never deleted. Appendix B.4 documents this.