acceptodds
Withdrawn from ICLR 2027

Exact, Reversible, and Verifiable Lifecycle Control for Routed Parametric Memory

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Deleting, correcting, or reverting what a deployed language model knows is today an optimisation problem on entangled weights. Unlearning objectives and model editors move parameters until a behavioural probe is satisfied. The result is approximate, depends on the order of operations, and can be verified by no one but the operator. The problem disappears when knowledge is placed rather than optimised. Block-Rollback stores injected knowledge in the content-routed memory blocks of a hierarchical-memory language model whose backbone, the anchor, stays frozen. Every block is a deterministic function of the documents it holds and a public seed. Insert, edit, delete, and revert therefore reduce to rebuilding one block from an append-only log, and each operation lands element-wise on the counterfactual model: the one that never saw the deleted fact, only ever saw the corrected fact, or stopped at the reverted step. We test this on five public anchors from GPT-2-XL to Llama-2-7B and on two anchors pretrained from scratch with the memory pathway, across the KnowEdit, TOFU, and MUSE benchmarks. More than 17000 locality checks find no violation, and every rebuilt block matches its counterfactual exactly. On the same base, gradient editors halve their locality within ten edits and cannot revert, and optimisation-based unlearners remain separable from the counterfactual by membership, extraction, and jailbreak probes. Block-Rollback matches it on every probe, including after int8 and NF4 quantisation. A third-party certificate rejects 10000 forged deletions at the cost of rebuilding one block. Per-operation cost tracks the touched block rather than the corpus or the record's age. A read-only routing gate keeps general capability at the anchor's values through 300 edits.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
parametric memory, knowledge editing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4pj9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Block-Rollback, a lifecycle-control scheme for knowledge injected into a hierarchical parametric memory (Pouransari et al., 2026) attached to a frozen "anchor" LM. A frozen router (MiniLM embeddings plus k-means centroids fitted once) assigns each document to one memory row ("block") per level. Whenever a block's document set changes, a deterministic, gradient-isolated trainer re-trains the block from a seed that is a public function of the block's identity, so that θc=T(Dc,sc)\theta_c = T(D_c, s_c) (Eq. 3–4). An append-only op-log of insert/edit/delete/revert defines each block's document set as a fold over per-fact last writes (Eq. 5). By construction this gives the identities of Eq. 6: a deletion equals the never-inserted state, an edit equals the only-ever-corrected state, a revert equals the truncated-log state, and the state does not depend on operation order. A hash-based certificate (seed commitment, survivor-set hash, block hash, anchor digest; Eq. 8) lets a verifier holding the public log, the surviving documents and the anchor re-materialise one block and compare. A read-only router-margin gate (Eq. 9) sends low-confidence prompts to the bare anchor to preserve general capability.

Validation uses five public anchors (GPT-2-XL, Qwen-2.5 0.5B/1.5B/7B, Llama-2-7B) plus two small natively pretrained anchors, on KnowEdit (WikiRecent, ZsRE, CounterFact), TOFU, MUSE and MQuAKE (App. I). The primary metric is the maximum element-wise parameter distance pd to a "gold" block that the same trainer re-trains on the counterfactual history. pd = 0 is reported essentially everywhere (17,430 locality checks, 16 operation orderings, 1,000-operation streams); the one bf16/flash-attention exception is isolated in App. D. Same-base baselines: ROME, MEMIT and WISE (plus GRACE on GPT-2-XL) lose locality and cannot revert. NPO, SimNPO, gradient ascent, RMU and a weight editor, applied to the same memory, end at pd ≈ 5 and are behaviourally distinguishable from the gold. The certificate rejects 10,000 randomised forgeries. Per-operation wall time tracks the touched block (r≈0.999r \approx 0.999), and tail replay costs 8.1× more to delete the oldest record than the newest, whereas Block-Rollback's cost is the same for both.

Critical strengths

S1. A clean constructive reframing that makes lifecycle operations exact and order-independent, backed by careful engineering. The deletable knowledge moves into blocks that are pure functions of (document set, public seed), with a frozen content router and a state defined by the log. This turns edit, delete and revert from optimisation problems into re-materialisation, and Eq. 5–6 make the semantics of each operation precise (delete = never-inserted, revert = truncated log). It is a useful counterpoint to the path-dependent, approximate behaviour of locate-then-edit and optimisation-based unlearning, and the same-base comparisons (Tables 2a, 3, 18) make the contrast concrete. The determinism engineering is careful and reusable: gather-only gradients, a fresh optimiser per block, and App. D's isolation of the flash-attention/width-256 non-determinism. This is the main reason I do not recommend clear rejection.

S2. Breadth and honesty of the implementation-level evidence. The construction is exercised on seven anchors, on both editing and unlearning benchmarks, and on streams of up to 1,000 operations. Two of the anchors were pretrained from scratch with the memory pathway, which rules out a "bolted-on adapter" artefact. The single configuration where pd ≠ 0 is reported and diagnosed rather than hidden (Table 7). I therefore trust that the implementation does what Eq. 4–6 say.

Critical weaknesses

W1. The headline evidence holds by construction, the novelty over existing designs is limited, and the experiments that would establish significance are thin or not run on the deployed system.

  • By construction. The gold is produced "by the same trainer from the counterfactual history" (l.607–608). The paper itself states that "every exactness result follows from this construction" (l.603), that FQ = 1 "is a corollary of pd = 0" (l.269), and that attack values equal the gold's "by construction" (Table 10). The 17,430 locality checks, FQ = 1.00 and the attack battery of Table 3c are therefore determinism tests of the implementation, not evidence about unlearning or editing. Conversely, pd to the gold can only be zero for a method that shares TT and scs_c with the gold, so pd ≈ 5 for NPO/RMU (Table 3b) says little about those methods. The lifecycle algebra (Eq. 6) follows immediately once the state is defined as a fold over last writes.
  • Novelty. The design is SISA-style exact retraining with content-defined shards, realised as memory rows, on top of a memory architecture whose authors already note that blocks could be removed or edited (l.120–121). The paper does not discuss closely related work on exact unlearning with content-aware or PEFT shards (e.g., SAFE, Dukler et al., ICCV 2023; S3T, Chowdhury et al., ICLR 2025). It also does not discuss memory/codebook editors that are reversible by removing entries (GRACE, which is cited; MELO, Yu et al., AAAI 2024; Larimar, Das et al., ICML 2024). The delta over this work is unclear.
  • Deployed-path utility. With the gate, which the paper calls the deployed read path, the system recalls only 0.77/0.79 of injected facts at 100/300 edits (l.347–348); the Rel. = 1.00 numbers come from other settings. The Block-Rollback row of the head-to-head Table 2a is taken from the unrouted, one-fact-per-block run (App. B, l.624; Table 5: 30 facts / 8 edits / 4 deletions), whereas the editors are run on 20 facts and a 10-edit stream. Its "locality" is parameter identity of untouched blocks (l.608–609), while ROME/MEMIT/WISE are scored with behavioural KnowEdit locality. The same column thus mixes different protocols and different metrics.
  • Query routing appears to be oracle. In Table 15, assigning 60 facts to 15 blocks at random still yields 0.92–1.00 reliability. In Fig. 1b, reliability is 0.854 at 100% random re-routing with K = 8. If queries were dispatched by the content router of Eq. 2, random placement should push reliability towards ≈1/K. This suggests that queries are sent to the block that stores their fact, in which case the reliability and generalisation numbers do not test routing at all.
  • Scale and cost. The experiments use at most a few hundred facts, K ≤ 64 and corpora of at most 680 documents. Capability is measured on MMLU n = 72, SST-2 n = 80 and WikiText n = 60, without variance. Per-operation cost is E⋅∣Dc∣E\cdot|D_c|, which grows as N/K at fixed memory size; "cost does not depend on the corpus size N" (l.201–202, Table 14) holds only because K is scaled with N. The paper does not study capacity or interference as blocks fill, imbalance of a frozen k-means router under distribution shift, or the trade-off between memory size and per-operation cost.
  • Retrieval. A retrieval store reaches the same reliability and generalisation, with deletion that is trivially exact and trivially verifiable (App. J, Table 17). The paper does not show anything that parametric blocks deliver beyond retrieval (e.g., portability or reasoning with injected knowledge), which is what would justify the design.

W2. The verifiability and privacy claims rest on an under-specified threat model that conflicts with the system's own design, and the scope is narrower than the framing.

  • What the certificate binds. The certificate shows that a published block hash equals a recomputation from a claimed survivor set. Nothing binds it to the weights that are actually served. The 10,000 "forgeries" (Table 12) are perturbations of hash fields and log entries, which any collision-resistant hash rejects. There is no adaptive adversary, for example one who keeps the deleted content under a new identifier or as a near-duplicate document, or who serves an uncertified model. On the limits of verifying unlearning from artefacts alone, see Thudi et al. (USENIX Security 2022). The check that "the commitment predates the deletion" needs a trusted timestamp, which is not specified. Verification also needs bitwise-reproducible training on the verifier's hardware (App. D shows pd = 11.35 under default kernels), and cross-hardware verification is not tested.
  • Retention and disclosure. The verifier receives "the surviving documents" (l.216), i.e., other data subjects' records in plaintext. More fundamentally, by Eq. 6 a revert$(t')$ across a deletion re-instates the deleted document, so the append-only log must keep the content after "deletion". This contradicts the right-to-be-forgotten motivation (l.038–040). The paper does not define purge semantics, or what the certificate and revert mean after a purge.
  • Scope. The guarantees cover only knowledge injected into the memory; whatever the frozen anchor holds is out of scope (l.484–485). Yet the abstract opens with "Deleting … what a deployed language model knows". For MUSE-Books (Harry Potter) on Llama-2-7B, the content is plausibly in the anchor's pretraining data. In that case VerbMem falling "to the gold's value" (Table 8) does not mean the model no longer knows the content. This should be presented as the central limitation, not as a closing remark.

Minor issues (these did not drive the score):

  1. Table 3c reports relearning recovery of −2.78 for Block-Rollback against 0.00 for the gold (Δ = −2.78). That is incompatible with pd = 0 if both received the same relearning procedure.
  2. App. H says random assignment "loses 5–8 points in four of seven" runs, but Table 15 shows three of seven. Content routing is also worse than random on Qwen-0.5B seed 2 (0.90 vs 1.00).
  3. The retain NLL of 0.40 for content routing in §4.6 cannot be found in Table 15 (0.99–2.15) or anywhere else.
  4. Centroids are fitted "on retain or public text" (l.146–147). Fitting on the retain set uses knowledge of future deletions.
  5. The writing is rhetorical (e.g., "Purity buys what optimisation cannot"). Key protocol details are scattered or missing: epochs, learning rate, memory parameter count relative to the anchor, and how queries are routed.
Questions for authors

Clarifications (the answers could change my assessment):

  1. Query routing. At evaluation time, how is a query dispatched to a block in Tables 1, 2a and 15 and in Fig. 1b? Is it the content router of Eq. 2 applied to the query text, the gated router of Eq. 9, or the block id stored with the fact? Please report reliability, generalisation, portability and behavioural KnowEdit locality for the deployed gated, content-routed path, on exactly the 20-fact / 10-edit stream used for ROME/MEMIT/WISE in Table 2a. Include locality prompts that the router sends into an edited block. If deployed reliability is around 0.77, my reading of the editing comparison changes substantially.
  2. Retention. After delete(u), is xux_u kept anywhere (op-log, survivor store, verifier)? How is a revert across a deletion reconciled with erasure, and what exactly is published in the "public log"? Does the verifier receive other users' surviving documents in plaintext?
  3. Certificate binding and threat model. What ties the certified block to the model that is actually served? How would the scheme detect an operator who re-inserts the deleted content under a new identifier or as a paraphrase? Has re-materialisation been verified across different GPU models or software stacks? If not, what tolerance would a verifier use, and does a tolerance re-open forgery?
  4. Table 3c relearning row. If pd = 0, why do Block-Rollback (−2.78) and the gold (0.00) differ? Was the same relearning procedure applied to both?
  5. Router fit. In each experiment, were the k-means centroids fitted on public text or on the retain set? If on the retain set, please rerun a representative deletion experiment with centroids fitted only on public text.
  6. Scale. In the largest run, what are the memory parameter count (relative to the anchor), the largest ∣Dc∣|D_c|, and the distribution of block occupancy? At fixed K, how do reliability and per-operation cost behave as N grows by one to two orders of magnitude?
  7. Positioning. How does the construction differ, conceptually and empirically, from shard-based exact unlearning with content-aware or PEFT shards (SAFE, S3T), from reversible memory editors (GRACE, MELO, Larimar), and from the concurrent Ramesh (2026)?

Suggestions (future work; not needed for my assessment):

  • A task where parametric blocks beat retrieval, such as multi-hop reasoning or portability with anchor knowledge, or a setting with a tight latency or context budget.
  • A discussion of how, if at all, the approach extends to knowledge already in the anchor.
Reviewer_YBAuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Block-Rollback, which inserts, edits, deletes, and reverts knowledge in a hierarchical-memory language model whose backbone, the anchor, stays frozen. A frozen MiniLM content router assigns each document to one memory block, a deterministic trainer rebuilds a block from its documents and a public seed, and an append-only op-log defines every block's document set. A hash-based certificate lets a third party audit a deletion by re-materialising one block, and a read-only margin gate returns unmatched prompts to the anchor. Experiments use seven anchors on KnowEdit, TOFU, and MUSE.

Critical strengths

S1: Placing knowledge rather than optimising it is a clear reframing. Eq. (5) and Eq. (6) make deletion, edit, and revert identities of the op-log, and Table 1b confirms order independence over 16 orderings and a 1,000-operation stream on Qwen-2.5-0.5B.

S2: The comparison with exact peers is concrete. Table 4c shows deleting the first, middle, or last record costs about 21 seconds, whereas tail replay costs 605.8, 402.7, and 74.8 seconds, and Table 4b ties wall time to touched blocks at r = 0.9999.

S3: Verification is a practical addition few unlearning papers offer. Table 4a rejects all 10,000 randomised forgeries from 18 classes, including 539 that pass the hash layer, and Table 20 repeats the guarantees on two natively pretrained anchors.

Critical weaknesses

W1: Table 2b measures capability on MMLU n=72, SST-2 n=80, and WikiText-103 n=60, and most tables report single runs without confidence intervals. Since the claim is that the gated path matches the anchor to the last digit, full benchmark splits and intervals over several router seeds would make this result easier to trust.

W2: Block width is fixed at 64 for every 7B run, and Table 7 shows width 256 under bf16 default kernels reaching pd 11.35 on News and 7.91 on Books. A sweep over width, epochs, and facts per block, reporting recall and rebuild time, would show the operating range of the construction.

W3: The editing comparison uses ROME, MEMIT, and WISE in Table 2a, with GRACE in Table 18. Adding a recent null-space constrained editor such as AlphaEdit at equal fact counts would sharpen the case that exact revert, and not only locality, separates Block-Rollback from current editors.

Questions for authors

Q1: Does the Block-Rollback row of Table 2a use the same 20 facts as the editors? Its 0/232 count matches the 30-fact run in Table 5, and an EasyEdit behavioural locality score would help.

Q2: Table 3c gives a relearning NLL drop of -2.78 for Block-Rollback against 0.00 for the gold although pd = 0. Could the authors clarify this entry?

Q3: How is a deletion audited when survivors are private? Also, Table 1 calls three model families five, and Section 1 counts four unlearning objectives while Section 4.1 lists five.

Reviewer_q4wCOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Block-Rollback, a method for exact and verifiable knowledge editing and unlearning in LLMs using content-routed parametric memory blocks, based on the hierarchical memory architecture of Pouransari et al. (2026).

The method uses a frozen pretrained LLM (anchor) with additional trainable FFN memory blocks at different levels. Documents are assigned to blocks using a frozen MiniLM-based router, where the centroids are obtained through k-means clustering. Each block is trained independently using a deterministic initialization and training procedure.

For knowledge insertion, editing, or deletion, the authors reinitialize and retrain the affected memory blocks using the updated document set. Since the same training procedure is used to construct the counterfactual retain-only gold model, the reconstructed blocks are identical to the gold by construction. The authors also maintain an operation log to support reversion and propose a hash-based certificate to allow third-party verification.

The method is evaluated on seven model anchors, including GPT-2-XL, Qwen-2.5, Llama-2-7B, and two models pretrained with hierarchical memory. Experiments use KnowEdit (WikiRecent, ZsRE, CounterFact) for editing and TOFU/MUSE for unlearning. The authors compare against ROME, MEMIT, WISE, NPO, SimNPO, RMU, and other baselines.

Critical strengths

None

Critical weaknesses
  • The paper is poorly written. Several key metrics, including reliability, generalization, and locality, important concepts like blocks etc. are not clearly defined. This makes it extremely difficult to understand. Additionally, there are statements like "Table 3a is the TOFU protocol read literally".
  • The near-perfect reliability, generalization, and locality scores in Tables 1 and 2 are difficult to believe without further investigation. The authors should establish the following:
    • How often does the frozen anchor already know the original or edited facts ?
    • why are the memory blocks always able to override conflicting knowledge ?
  • A huge issue of this approach is that it is very similar to retraining the model. Hence the knowledge capacity of the memory blocks need to be investigated. If we scale the stored facts while keeping number of blocks the same, then does the method still work ?
    • Another issue that follows from this is the deletion cost. If we increase the knowledge capacity, then deletion will be extremely expensive.
  • There are several experimental issues:
    • Table 1 primarily evaluates parameter distance pd, which is not a meaningful or useful measure in my opinion. Since Block-Rollback reinitializes and retrains the affected block using exactly the same procedure as the retain-only gold, pd = 0 follows by construction. The experiments mostly verify that deterministic retraining is deterministic. - There are limited comparisons for ZsRE and CounterFact. Although Table 1 reports reliability and generalization on these datasets, there are no comparable head-to-head evaluations against ROME, MEMIT, WISE, etc., as performed on WikiRecent in Table 2. - There are no comprehensive editing results after 300 edits. Table 2b evaluates general model capabilities after 300 edits but does not report reliability, generalization, and locality.
    • Locality evaluation is unclear and potentially inconsistent across methods. Appendix A says "Locality checks compare every block outside the affected set with its pre-operation copy after every operation". However, ROME and MEMIT do not have these blocks.
    • No experiments establishing that training parametric memory blocks on TOFU/MUSE achieves utility comparable to conventional fine-tuning. Before evaluating unlearning, the authors should demonstrate that the memory blocks can actually acquire the benchmark knowledge with comparable performance to the original training setting.
    • Most of the paper's central lifecycle guarantees follow directly from the construction. Exact deletion, editing, reversion, order independence, and parameter locality are consequences of defining each block as a deterministic function of its document set. The experiments extensively verify these properties, but devote comparatively little attention to establishing that the resulting architecture is useful at scale.
Questions for authors

Please see weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.