- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Block-Rollback, a lifecycle-control scheme for knowledge injected into a hierarchical parametric memory (Pouransari et al., 2026) attached to a frozen "anchor" LM. A frozen router (MiniLM embeddings plus k-means centroids fitted once) assigns each document to one memory row ("block") per level. Whenever a block's document set changes, a deterministic, gradient-isolated trainer re-trains the block from a seed that is a public function of the block's identity, so that (Eq. 3–4). An append-only op-log of insert/edit/delete/revert defines each block's document set as a fold over per-fact last writes (Eq. 5). By construction this gives the identities of Eq. 6: a deletion equals the never-inserted state, an edit equals the only-ever-corrected state, a revert equals the truncated-log state, and the state does not depend on operation order. A hash-based certificate (seed commitment, survivor-set hash, block hash, anchor digest; Eq. 8) lets a verifier holding the public log, the surviving documents and the anchor re-materialise one block and compare. A read-only router-margin gate (Eq. 9) sends low-confidence prompts to the bare anchor to preserve general capability.
Validation uses five public anchors (GPT-2-XL, Qwen-2.5 0.5B/1.5B/7B, Llama-2-7B) plus two small natively pretrained anchors, on KnowEdit (WikiRecent, ZsRE, CounterFact), TOFU, MUSE and MQuAKE (App. I). The primary metric is the maximum element-wise parameter distance pd to a "gold" block that the same trainer re-trains on the counterfactual history. pd = 0 is reported essentially everywhere (17,430 locality checks, 16 operation orderings, 1,000-operation streams); the one bf16/flash-attention exception is isolated in App. D. Same-base baselines: ROME, MEMIT and WISE (plus GRACE on GPT-2-XL) lose locality and cannot revert. NPO, SimNPO, gradient ascent, RMU and a weight editor, applied to the same memory, end at pd ≈ 5 and are behaviourally distinguishable from the gold. The certificate rejects 10,000 randomised forgeries. Per-operation wall time tracks the touched block (), and tail replay costs 8.1× more to delete the oldest record than the newest, whereas Block-Rollback's cost is the same for both.
Critical strengths
S1. A clean constructive reframing that makes lifecycle operations exact and order-independent, backed by careful engineering. The deletable knowledge moves into blocks that are pure functions of (document set, public seed), with a frozen content router and a state defined by the log. This turns edit, delete and revert from optimisation problems into re-materialisation, and Eq. 5–6 make the semantics of each operation precise (delete = never-inserted, revert = truncated log). It is a useful counterpoint to the path-dependent, approximate behaviour of locate-then-edit and optimisation-based unlearning, and the same-base comparisons (Tables 2a, 3, 18) make the contrast concrete. The determinism engineering is careful and reusable: gather-only gradients, a fresh optimiser per block, and App. D's isolation of the flash-attention/width-256 non-determinism. This is the main reason I do not recommend clear rejection.
S2. Breadth and honesty of the implementation-level evidence. The construction is exercised on seven anchors, on both editing and unlearning benchmarks, and on streams of up to 1,000 operations. Two of the anchors were pretrained from scratch with the memory pathway, which rules out a "bolted-on adapter" artefact. The single configuration where pd ≠ 0 is reported and diagnosed rather than hidden (Table 7). I therefore trust that the implementation does what Eq. 4–6 say.
Critical weaknesses
W1. The headline evidence holds by construction, the novelty over existing designs is limited, and the experiments that would establish significance are thin or not run on the deployed system.
- By construction. The gold is produced "by the same trainer from the counterfactual history" (l.607–608). The paper itself states that "every exactness result follows from this construction" (l.603), that FQ = 1 "is a corollary of pd = 0" (l.269), and that attack values equal the gold's "by construction" (Table 10). The 17,430 locality checks, FQ = 1.00 and the attack battery of Table 3c are therefore determinism tests of the implementation, not evidence about unlearning or editing. Conversely, pd to the gold can only be zero for a method that shares and with the gold, so pd ≈ 5 for NPO/RMU (Table 3b) says little about those methods. The lifecycle algebra (Eq. 6) follows immediately once the state is defined as a fold over last writes.
- Novelty. The design is SISA-style exact retraining with content-defined shards, realised as memory rows, on top of a memory architecture whose authors already note that blocks could be removed or edited (l.120–121). The paper does not discuss closely related work on exact unlearning with content-aware or PEFT shards (e.g., SAFE, Dukler et al., ICCV 2023; S3T, Chowdhury et al., ICLR 2025). It also does not discuss memory/codebook editors that are reversible by removing entries (GRACE, which is cited; MELO, Yu et al., AAAI 2024; Larimar, Das et al., ICML 2024). The delta over this work is unclear.
- Deployed-path utility. With the gate, which the paper calls the deployed read path, the system recalls only 0.77/0.79 of injected facts at 100/300 edits (l.347–348); the Rel. = 1.00 numbers come from other settings. The Block-Rollback row of the head-to-head Table 2a is taken from the unrouted, one-fact-per-block run (App. B, l.624; Table 5: 30 facts / 8 edits / 4 deletions), whereas the editors are run on 20 facts and a 10-edit stream. Its "locality" is parameter identity of untouched blocks (l.608–609), while ROME/MEMIT/WISE are scored with behavioural KnowEdit locality. The same column thus mixes different protocols and different metrics.
- Query routing appears to be oracle. In Table 15, assigning 60 facts to 15 blocks at random still yields 0.92–1.00 reliability. In Fig. 1b, reliability is 0.854 at 100% random re-routing with K = 8. If queries were dispatched by the content router of Eq. 2, random placement should push reliability towards ≈1/K. This suggests that queries are sent to the block that stores their fact, in which case the reliability and generalisation numbers do not test routing at all.
- Scale and cost. The experiments use at most a few hundred facts, K ≤ 64 and corpora of at most 680 documents. Capability is measured on MMLU n = 72, SST-2 n = 80 and WikiText n = 60, without variance. Per-operation cost is , which grows as N/K at fixed memory size; "cost does not depend on the corpus size N" (l.201–202, Table 14) holds only because K is scaled with N. The paper does not study capacity or interference as blocks fill, imbalance of a frozen k-means router under distribution shift, or the trade-off between memory size and per-operation cost.
- Retrieval. A retrieval store reaches the same reliability and generalisation, with deletion that is trivially exact and trivially verifiable (App. J, Table 17). The paper does not show anything that parametric blocks deliver beyond retrieval (e.g., portability or reasoning with injected knowledge), which is what would justify the design.
W2. The verifiability and privacy claims rest on an under-specified threat model that conflicts with the system's own design, and the scope is narrower than the framing.
- What the certificate binds. The certificate shows that a published block hash equals a recomputation from a claimed survivor set. Nothing binds it to the weights that are actually served. The 10,000 "forgeries" (Table 12) are perturbations of hash fields and log entries, which any collision-resistant hash rejects. There is no adaptive adversary, for example one who keeps the deleted content under a new identifier or as a near-duplicate document, or who serves an uncertified model. On the limits of verifying unlearning from artefacts alone, see Thudi et al. (USENIX Security 2022). The check that "the commitment predates the deletion" needs a trusted timestamp, which is not specified. Verification also needs bitwise-reproducible training on the verifier's hardware (App. D shows pd = 11.35 under default kernels), and cross-hardware verification is not tested.
- Retention and disclosure. The verifier receives "the surviving documents" (l.216), i.e., other data subjects' records in plaintext. More fundamentally, by Eq. 6 a revert$(t')$ across a deletion re-instates the deleted document, so the append-only log must keep the content after "deletion". This contradicts the right-to-be-forgotten motivation (l.038–040). The paper does not define purge semantics, or what the certificate and revert mean after a purge.
- Scope. The guarantees cover only knowledge injected into the memory; whatever the frozen anchor holds is out of scope (l.484–485). Yet the abstract opens with "Deleting … what a deployed language model knows". For MUSE-Books (Harry Potter) on Llama-2-7B, the content is plausibly in the anchor's pretraining data. In that case VerbMem falling "to the gold's value" (Table 8) does not mean the model no longer knows the content. This should be presented as the central limitation, not as a closing remark.
Minor issues (these did not drive the score):
- Table 3c reports relearning recovery of −2.78 for Block-Rollback against 0.00 for the gold (Δ = −2.78). That is incompatible with pd = 0 if both received the same relearning procedure.
- App. H says random assignment "loses 5–8 points in four of seven" runs, but Table 15 shows three of seven. Content routing is also worse than random on Qwen-0.5B seed 2 (0.90 vs 1.00).
- The retain NLL of 0.40 for content routing in §4.6 cannot be found in Table 15 (0.99–2.15) or anywhere else.
- Centroids are fitted "on retain or public text" (l.146–147). Fitting on the retain set uses knowledge of future deletions.
- The writing is rhetorical (e.g., "Purity buys what optimisation cannot"). Key protocol details are scattered or missing: epochs, learning rate, memory parameter count relative to the anchor, and how queries are routed.
Questions for authors
Clarifications (the answers could change my assessment):
- Query routing. At evaluation time, how is a query dispatched to a block in Tables 1, 2a and 15 and in Fig. 1b? Is it the content router of Eq. 2 applied to the query text, the gated router of Eq. 9, or the block id stored with the fact? Please report reliability, generalisation, portability and behavioural KnowEdit locality for the deployed gated, content-routed path, on exactly the 20-fact / 10-edit stream used for ROME/MEMIT/WISE in Table 2a. Include locality prompts that the router sends into an edited block. If deployed reliability is around 0.77, my reading of the editing comparison changes substantially.
- Retention. After delete(u), is kept anywhere (op-log, survivor store, verifier)? How is a revert across a deletion reconciled with erasure, and what exactly is published in the "public log"? Does the verifier receive other users' surviving documents in plaintext?
- Certificate binding and threat model. What ties the certified block to the model that is actually served? How would the scheme detect an operator who re-inserts the deleted content under a new identifier or as a paraphrase? Has re-materialisation been verified across different GPU models or software stacks? If not, what tolerance would a verifier use, and does a tolerance re-open forgery?
- Table 3c relearning row. If pd = 0, why do Block-Rollback (−2.78) and the gold (0.00) differ? Was the same relearning procedure applied to both?
- Router fit. In each experiment, were the k-means centroids fitted on public text or on the retain set? If on the retain set, please rerun a representative deletion experiment with centroids fitted only on public text.
- Scale. In the largest run, what are the memory parameter count (relative to the anchor), the largest , and the distribution of block occupancy? At fixed K, how do reliability and per-operation cost behave as N grows by one to two orders of magnitude?
- Positioning. How does the construction differ, conceptually and empirically, from shard-based exact unlearning with content-aware or PEFT shards (SAFE, S3T), from reversible memory editors (GRACE, MELO, Larimar), and from the concurrent Ramesh (2026)?
Suggestions (future work; not needed for my assessment):
- A task where parametric blocks beat retrieval, such as multi-hop reasoning or portability with anchor knowledge, or a setting with a tight latency or context budget.
- A discussion of how, if at all, the approach extends to knowledge already in the anchor.