acceptodds
Desk rejected at ICLR 2027

Beyond Rollback: Auditing Revocable External Memory for Hardware-Design LLMs

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Revoking a designated asset from an external-memory LLM is a lifecycle question rather than a single rollback operation. A defensible revocation claim must establish authorized utility, collateral capability, selective serving removal and artifact sanitization. We operationalize these requirements for RTL generation and instantiate them in EXCISE, a detachable content-addressed cartridge attached to a frozen backbone. On a locked 128-task evaluation, attaching EXCISE raises functional recovery from 2.3% to 21.1%, while VerilogEval-v2 capability changes from 44.9% to 42.3% (p = 0.523). Disabling all routing reproduces every base output. At the selected layer, a matched 32-asset checkpoint reaches 81% direct recovery while attached and 0% after serving deletion, passes 48.1% of VerilogEval-v2 versus 44.9% for the frozen base, and preserves all 992 by-stander functional outcomes. Across three 32-asset checkpoints, direct recovery ranges from 59% to 72% with intact cartridges and from 0% to 3% after per-asset serving deletion. Between 99.8% and 100% of bystander pairs preserve function. However, a four-prompt stochastic audit finds recovery on 75% of assets with the intact memory, 50% after serving deletion, and 44% with routing disabled, showing that serving deletion does not always match complete routing shutdown during generation. Artifact audits reveal membership signals in both serialized routing metadata (AUC 0.851) and learned embeddings (AUC 0.971). Registry rebuild removes the tested metadata signal (AUC 0.491) but not the embedding signal, so sanitized release requires a fresh cartridge. Overall, external memory supports practical asset-level serving revocation, while output-level deletion and artifact sanitization remain separate obligations

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
RTL code generation, large language models, revocable external memory, hardware IP protection, memorization auditing

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_BCY6OpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies revocation of designated assets from an LLM used for RTL generation. EXCISE stores these assets in a detachable, hash-addressed memory cartridge attached to a frozen backbone. Its construction combines reference-corpus row exclusion, per-asset mutual exclusion, and gated memory injection. The paper distinguishes four evaluation requirements: authorized recovery, collateral capability, selective serving deletion, and serialized-artifact sanitization. On a locked 128-task evaluation, functional recovery increases from 3/128 to 27/128, while VerilogEval-v2 changes from 70/156 to 66/156. A separate selected-layer 32-asset checkpoint achieves 26/32 direct functional recoveries before deletion and 0/32 afterward, preserving all 992 bystander functional outcomes. Complete routing shutdown restores the frozen computation.

Critical strengths
  • The four-part audit contract is the strongest contribution. The paper supports the distinction between serving deletion and artifact sanitization with separate routing-metadata and learned-value attacks. This provides a more meaningful evaluation framework than reporting rollback or output similarity alone.
  • The paper distinguishes compilation, synthesis, and strict functional recovery rather than relying on character overlap. Cross-deletion matrices, stochastic extraction, paired capability outcomes, layer sensitivity, and removable-memory baselines provide substantial evidence for the main configuration while exposing important failure modes.
Critical weaknesses

The exact rollback property is clear, but useful authorized recovery is strongly configuration-dependent. The locked CodeV-R1 evaluation provides essentially no net functional improvement at the capability-preserving setting. This limits confidence that EXCISE offers a broadly applicable memory mechanism rather than an effective operating point for one backbone and asset construction.

Questions for authors
  • What explains the nearly null CodeV-R1 result? Can the existing experiments distinguish insertion-layer mismatch, optimization limitations, and interference with the model’s generation trajectory?
  • What is the measured cost of rebuilding from surviving assets after revocation, compared with initial cartridge training?
5 comments

W1: Recovery and selective revocation across four backbones with a shared training recipe

Thank you for highlighting the importance of cross-backbone validation. We evaluated the same 32 assets across four backbones using a shared training recipe: layer 16, 32 presentations per asset, and carriers matching each backbone’s serving format—raw prompts for CodeV-DS/CL and chat-format carriers for CodeV-QW/R1.

Cross-backbone recovery and deletion. Seed 0, greedy direct-code decoding. Recovery is out of 32 assets and VE out of 156 tasks. The primary threshold is k=2; secondary k=1 results appear in parentheses. Both k columns are ungated. Deletion results use ungated k=2. R1’s routing-off VE is 76/156 when re-evaluated at batch size 1 (submission: 75/156).

BackboneRecovery: off → k=2 (k=1)After deletion (k=2)VE: off → k=2 (k=1)
CodeV-DS (DeepSeek-Coder)1 → 22 (28)170 → 70 (66)
CodeV-CL (CodeLlama)5 → 20 (25)444 → 43 (38)
CodeV-QW (CodeQwen1.5)3 → 15 (22)375 → 75 (75)
CodeV-R1 (reasoning-distilled Qwen2.5)8 → 12 (20)876 → 76 (34)
  • Significant recovery gains across three backbones at the primary threshold. At k=2, CodeV-DS, CodeV-CL, and CodeV-QW all achieve significant gains over routing off (exact McNemar, p < 0.001), with VE within one pass of their respective frozen backbones.

  • CodeV-R1 achieves significant recovery under the shared training recipe. In the new 32-asset experiment, the secondary threshold k=1 increases recovery from 8 to 20/32 (p < 0.001). The 3% request gate, based on k=1 prompt-hit statistics, retains all 20 recoveries with VE at 71/156, compared with 76/156 under routing off. Q1 provides the accompanying format and training-budget diagnostics.

  • CodeV-CL replicates across seeds and scales to 192 assets. Three training seeds recover 20, 22, and 21 of 32 assets, with VE 43, 46, and 49 (frozen backbone: 44). At 192 assets, CodeV-CL recovers 105 versus 15 with routing off, with VE 47 versus 44 (one seed).

  • Selective deletion remains effective across backbones. At k=2, deletion reduces recovery to at or below routing-off counts on all four 32-asset evaluations. On CodeV-CL at 192 assets, recovery falls from 105 to 16, versus 15 with routing off, while all 992 bystander functional outcomes remain unchanged in the tested 32×32 deletion submatrix.

CodeV-QW and CodeV-R1 each use one training seed. Request-gated results are offline compositions of saved outputs using prompt-hit statistics.

Q1: Controlled diagnostics identify format alignment as a key factor in CodeV-R1 recovery

Thank you for asking us to distinguish these explanations. Our controlled experiments identify training–serving format alignment as a directly supported contributor to CodeV-R1 recovery. Under the shared recipe, R1 reaches 20/32 recoveries, which request gating retains with VE at 71/156 (W1).

(1) Layer, fitting, and injection-timing diagnostics.

We first examined the original raw-carrier/chat-serving setup on development assets.

  • Insertion layer. Across six layers and two training budgets on 64 development assets, the best recovery within 10 VE passes of the backbone is 18/64 versus 12/64 with routing off.

  • Optimization. In five of the six evaluated layers, teacher-forced answer NLL decreases from 0.43 to 0.03–0.04 after 3,000 updates, demonstrating effective reference-answer fitting. In six matched 1,000-versus-3,000-update comparisons where both checkpoints remain within 10 VE passes of the backbone, recovery changes by −1 to +2 assets.

  • Injection timing. We separately tested full-request, prefill-only, decoding-only, and disabled injection.

Table B1. Functional recoveries out of 32. Both configurations use development cartridges trained on 64 development assets; the stage ablation evaluates their first 32 assets.

ConfigurationFull requestPrefill onlyDecoding onlyRouting off
Layer 21, 1,000 updates, k=14645
Layer 7, 3,000 updates, k=28755

Prefill-only injection has configuration-dependent effects and retains the memory-influenced prefill state. W1’s request-level gate instead determines whether memory is enabled for the entire request.

(2) Format controls demonstrate recovery gains at a fixed training budget.

We first tested serving-format sensitivity while keeping each trained checkpoint fixed.

Table B2. Serving-format control. Each row compares raw and chat serving using the same raw-trained checkpoint, asset set, and routing threshold. Both formats were regenerated with batch size 1. Entries report routing off → cartridge-enabled recovery; k=1 for DS/CL/QW and k=2 for R1. QW and R1 use diagnostic checkpoints distinct from those in Table B3, with each raw/chat comparison performed on the same fixed checkpoint. The R1 probe evaluates 64 assets disjoint from the development set.

BackboneEvaluated assetsRaw serving (matched)Chat serving (mismatched)
CodeV-DS321 → 286 → 5
CodeV-CL325 → 254 → 10
CodeV-QW321 → 213 → 4
CodeV-R1640 → 2914 → 17

Among the 50 assets recovered by neither routing-off format, R1 recovers 20 under matched raw serving versus 5 under mismatched chat serving.

We then conducted a separate 2×2 training–serving format experiment using the shared recipe from W1: the same 32 assets, layer 16, and 32 presentations per asset. For each backbone, raw-trained and chat-trained cartridges were evaluated under both serving formats.

Table B3. Functional recoveries out of 32, one training seed. Each cell reports routing off / mismatched carriers / matched carriers. All entries are ungated; k=2 is primary and k=1 secondary.

BackboneThresholdChat serving: off / mismatch / matchRaw serving: off / mismatch / match
CodeV-QWk=23 / 6 / 151 / 2 / 13
CodeV-QWk=13 / 6 / 221 / 2 / 18
CodeV-R1k=28 / 9 / 120 / 2 / 15
CodeV-R1k=18 / 9 / 200 / 4 / 20

Matched carriers improve recovery in all eight comparisons. For R1 at k=1, matching raises recovery from 9 to 20 under chat serving and from 4 to 20 under raw serving. All four k=1 format comparisons have 95% confidence intervals above zero.

Q2: Measured rebuilding cost relative to initial cartridge training

Thank you for asking for this comparison. On a single H100, fresh rebuilding takes 3.1–19.6 minutes in the measured settings. Rebuilding after removing one asset costs 97–99% of initial training time; removing eight of 32 assets reduces this to 79%.

Each rebuild starts with a newly initialized cartridge and optimizer and trains only on surviving assets, using 32 presentations per surviving asset with the backbone frozen. No learned state from the old cartridge is reused.

Table B4. New paired training/rebuilding timings. Seconds, with training-loop time in parentheses. CodeV-DS, layer 16, seed 0, one H100. Job time includes setup, training, internal validation, serialization, and external checks; subsequent functional, VerilogEval, and membership evaluations are excluded. Each run was timed once; both 32-asset cases share one initial run.

Assets: initial → survivingTraining updatesInitial run (s)Rebuild (s)Rebuild / initial
32 → 311,024 → 992232 (187)226 (182)97%
32 → 241,024 → 768232 (187)184 (140)79%
192 → 1916,144 → 6,1121,184 (1,127)1,174 (1,118)99%
  • Rebuilding cost follows the surviving training budget. Across these runs, training takes approximately 0.18 s per update, or 6 s per surviving asset, with 44–57 s of job overhead. Multiple revocations can share one rebuild before artifact release, as demonstrated by the 32→24 case.

  • Serving revocation requires no retraining. Updating the in-memory mask, X←X∖Xs\mathcal{X} \leftarrow \mathcal{X} \setminus \mathcal{X}_s, takes a median of 0.023 ms over 200 measurements in each of the three cases, with the cartridge loaded and mask indices prepared. This measures the mask update itself, excluding setup and distribution.

Why our release protocol uses a fresh rebuild. In a separate three-seed audit of 64-asset cartridges with 32 assets revoked, fresh rebuilding reduces the embedding-attack mean AUC to 0.479 (95% CI [0.364, 0.594]), compared with 1.000 after registry rebuilding alone. This is consistent with chance under the tested attack, which covers 29 of the 32 revoked assets.

Reviewer_vsubOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the problem of revocable knowledge integration in LLMs specialized for RTL / Verilog hardware source code generation. Traditional fine-tuning encode confidential assets with shared model weights, making subsequent machine unlearning difficult. The authors argue that asset revocation is a comprehensive lifecycle issue rather than a trivial rollback. They present EXCISE, a framework that attaches a detachable, content-addressed memory cartridge (proposed by Engram) to a frozen model backbone. The paper establishes a four-part evaluation contract and uses real hardware verification pipelines along with black-box prompt extraction and white-box membership inference attacks.

Critical strengths
  1. The paper tackles an acute industry bottleneck in hardware engineering where IP protection, licensing expiration, and clean compliance boundaries are mandatory. Framing revocation as auditable control over decoupled state rather than destructive weight manipulation is conceptually sound.

  2. The paper identifies critical flaws in prior RTL evaluation and makes comprehensive experiments. And instead of only positive metrics, they report failure modes and boundary conditions that include the capacity frontier, prompt-side collision collapse etc.

  3. The author conducted an in-depth comparison of alignment techniques such as LoRA and RAG, testing not only the black-box generation but also carrying out white-box member inference attacks (MIA) on serialized cartridge files.

Critical weaknesses
  1. The core architectural primitive of EXCISE is directly adopted from the Engram module (Cheng et al., 2026), utilizing the identical multi-head bigram/trigram hash-addressed lookup and gated residual projection. The paper's primary algorithmic modifications—namely, reference-corpus row subtraction (X=rows(S)∖rows(G))(X=rows(S)∖ rows(G)), pairwise asset exclusion, and fail-closed admission—are essentially heuristic set-difference operations executed during data preprocessing, supplemented by a hard-threshold counter kk and an ablation margin loss. While these mechanisms are practically effective for isolation, they represent incremental engineering adaptations rather than novel representation learning principles or adaptive neural architectures.

  2. A critical limitation is the performance degradation observed as the number of assets increases. As the asset count scales from 32 to 100–192, VerilogEval pass rates degrade precipitously (from 70/156 down to 12–23/156) due to out-of-domain prompt collisions with active cartridge rows. This restricts EXCISE’s viability in large-scale IP libraries without further architectural partitioning.

  3. The hash-addressed cartridge relies heavily on literal local n-gram sequences. Under alpha-renamed or reframed prompts, functional recovery drops to 0/32. The system functions as a content-addressed lookup rather than acquiring transferable structural/semantic representations.

Questions for authors

see the weaknesses.

3 comments

W1: Asset-specific row ownership for selective revocation in shared learned memory

Thank you for raising this point. EXCISE builds on Engram’s hash lookup and gated readout (Cheng et al., 2026). Our contribution is a revocation-aware construction for shared learned memory attached to a frozen backbone: each admitted asset receives an explicit, separately removable routing set that governs training and serving. A four-part audit evaluates authorized utility, collateral capability, serving revocation, and artifact sanitization.

(1) Row ownership connects construction, training, and serving-time revocation.

The set operations establish routing invariants that are enforced through the training and serving masks:

ElementRouting invariant
Reference-corpus row subtractionWith k ≥ 1, contexts enumerated from the reference corpus G at construction receive zero memory injection.
Per-asset mutual exclusionRegistered row sets are disjoint: Xs∩Xs′=∅\mathcal{X}_s \cap \mathcal{X}_{s'} = \varnothing for s≠s′s \ne s'. Deleting Xs\mathcal{X}_s removes no row owned by another asset.
Fail-closed admissionEvery admitted asset has a nonempty, separately removable row set.
Hard threshold kFewer than k active-row hits produce exactly zero injection at that step, including with nonzero projection biases.

Revocation updates the active mask, X←X∖Xs\mathcal{X} \leftarrow \mathcal{X} \setminus \mathcal{X}_s, giving an inspectable removal set independent of learned gating scores. Thus, the construction defines the unit of serving revocation before training and enforces its removal through the inference mask.

(2) Matched ablations demonstrate selective deletion with bystander preservation.

Table V1. CodeV-DS, 32 assets (29 admitted by the full construction), layer 16, one training seed, k=1, greedy decoding of each asset’s own prompt. Assets and training steps are matched; active-row counts differ. Recovery is out of 32; the 992 bystander cells pair each deleted asset with every other asset. The 5–6 range reflects a reproducible Yosys crash on one fixed generated output.

ConstructionDeletion removesRecovery, intact → deleted /32Bystander functional losses /992
Full EXCISE constructionThe asset’s own rows28 → 00
Without reference subtractionThe asset’s own rows25 → 04
Without mutual exclusionAll rows used by the asset27 → 1210
Without mutual exclusionOnly its unshared rows27 → 5–64
Without subtraction or mutual exclusionAll rows used by the asset18 → 1525
Without subtraction or mutual exclusionOnly its unshared rows18 → 412
  • The full construction combines target removal with bystander preservation. It achieves 28/32 intact recoveries and zero direct recovery after deletion, with all 992 bystander functional outcomes unchanged; 984/992 outputs remain byte-identical.

  • Mutual exclusion separates target removal from shared-row interference. Without it, intact recovery remains similar (27 versus 28), but deleting all used rows causes 210/992 bystander losses, while retaining shared rows leaves 5–6 targets recoverable.

  • Reference subtraction further supports preservation. Removing it introduces 4 bystander losses. Removing both row constraints reduces intact recovery to 18/32 and produces 525/992 bystander losses when all used rows are deleted.

Only the full construction achieves both zero direct recovery after deletion and unchanged bystander functional outcomes among these six settings.

The audit also identifies the operations needed for artifact release. Complementary white-box audits distinguish serving-route removal from membership signals in serialized metadata and learned embeddings. Fresh cartridge training on surviving assets reduces the tested embedding-attack AUC from 1.000 to 0.479, near the chance level of 0.5 (udTK W4).

W2: Routing controls preserve capability up to 192 assets in a single shared cartridge

Thank you for raising this concern. Our expanded study shows that routing controls retain near-backbone VerilogEval performance up to 192 assets without additional table partitioning. The cited drop occurred at layer 2 with k=1. We evaluated 32-, 64-, 100-, and 192-asset cartridges at selected layer 16 with 32 training presentations per asset.

Capacity and routing controls (excerpt of Table U1a in our response to udTK W5(1)). CodeV-DS, layer 16, seed 0. Cells report F / VE: functional recoveries out of N assets and VerilogEval passes out of 156, respectively. The frozen backbone scores 70 on VE.

Assets (N)k=1k=2k=5k=2 + request gate (3%)
3228 / 6622 / 7011 / 7119 / 70
6453 / 041 / 5520 / 6939 / 70
10079 / 070 / 4840 / 7066 / 68
192144 / 22111 / 7044 / 69103 / 70
  • Threshold control suppresses incidental activation. Changing only the inference threshold on the same checkpoints gives VE 69–71 across all four sizes at k=5. At 100 assets, increasing k from 1 to 5 reduces median prompt-token activation from 5.8% to 0% on VerilogEval, compared with 10.7% to 2.5% on asset prompts.

  • Request gating improves the recovery–capability trade-off. The gate enables the k=2 cartridge when at least 3% of prompt tokens meet the k=2 active-hit criterion; otherwise, the entire request uses routing off. At 192 assets, only 14/156 VerilogEval prompts pass this cutoff, versus 156/192 asset prompts, achieving 103 recoveries with VE at 70.

  • The results extend across seeds and backbones. At 192 assets, the fixed 3% request gate yields 103–106 recoveries with VE 65–70 across three CodeV-DS seeds. Reusing the same cutoff on CodeV-CL gives 100 recoveries versus 15 with routing off, with VE 44/156 in both cases.

  • Selective deletion remains effective at scale. At 192 assets, ungated k=2 deletion leaves 0–2 recoveries across three DS seeds, versus 2 with routing off. The seed-0 cartridge’s tested 32×32 deletion submatrix records zero bystander functional losses across 992 evaluations.

The fixed 3% cutoff was selected on DS and reused unchanged on CL. Cross-fitting selects the cutoff on each half to maximize recovery within a proportionally scaled ten-pass VE loss budget, applies it to the other half, and pools both held-out halves. Gated results use saved branch outputs, selected offline using only prompt-hit statistics. Table U1b in our response to udTK W5(1) reports the full cross-fitted results.

Across the tested CodeV-DS sizes and seeds, the fixed gate recovers more than half of the assets while keeping VE within five passes of the frozen backbone.

W3: Prompt-wrapper robustness and broader coverage through multi-carrier registration

EXCISE combines n-gram addressing with asset-specific row ownership for inspectable, separately removable routes. The reported 0/32 concerns alpha-renamed carriers, not instruction reframing alone. Our new experiments assess prompt-wrapper robustness and coverage through registered paraphrases under unified asset-level revocation.

(1) Recovery survives instruction reframing and few-shot examples.

Using a cartridge trained only on the original direct prompts, we change the surrounding instruction or prepend two worked RTL examples while preserving the specification.

Table V2. Recovering completions out of 1,024 per cell (32 assets × 32 samples), with recovered-asset counts in parentheses. CodeV-DS, layer 16, seed 0, k=1, temperature 0.8, top-p 0.95. Ranges reflect a reproducible Yosys crash on one fixed output. This audit uses the submission checkpoint; Tables V1 and V3 use newly trained cartridges.

Prompt wrapperCartridge intactAfter self-deletionRouting off
Original direct prompt763 (29/32 assets)13 (2/32 assets)13 (2/32 assets)
Same specification, different instruction201 (22/32 assets)34–35 (8–9/32 assets)49 (10/32 assets)
Same specification after two worked examples200 (19/32 assets)78 (11/32 assets)80 (10/32 assets)

Instruction reframing and worked examples recover 22/32 and 19/32 assets, respectively, versus 10/32 with routing off, counting success in at least one of 32 samples. After deletion, completion recovery counts are at or below routing off across all three wrappers.

(2) Multi-carrier registration broadens coverage under one revocation mask.

Multiple surface forms can be registered under the same logical asset and assigned one owned-row set, Xs\mathcal{X}_s. We register the original prompt, two natural-language rewrites (A/B), and a formatting change, while holding out rewrites C/D. Both configurations receive 32 total presentations per asset: 32 of the original form for the single-carrier cartridge, or 8 of each form for the four-carrier cartridge.

Table V3. CodeV-DS, 32 assets, layer 16, one training seed, k=1, greedy decoding. Entries report recovered assets out of 32, except the subtotal. “Registered” refers to the four-carrier configuration; the single-carrier configuration registers only the original. VE is 64 with four carriers and 66 with one (frozen backbone: 70).

Prompt formSingle-carrierFour-carrierFour-carrier after self-deletionRouting off
Original (registered)281201
Rewrite A (registered)31723
Rewrite B (registered)41726
Formatting change (registered)91932
Subtotal across registered forms (/128)4465712
Rewrite C (not registered)3325
Rewrite D (not registered)2104
  • Broader registered coverage at a fixed training budget. Recovery increases from 44 to 65 of 128 asset–form pairs, with the three added forms improving from 3, 4, and 9 to 17, 17, and 19 recoveries.

  • Unified revocation across registered forms. One mask update, X←X∖Xs\mathcal{X} \leftarrow \mathcal{X} \setminus \mathcal{X}_s, removes their registered routes together, reducing recovery from 65 to 7/128 pairs, compared with 12/128 under routing off.

At a fixed budget, multi-carrier registration trades original-prompt recovery for broader registered-form coverage; held-out C/D show no gains. Each asset’s registered forms remain jointly revocable.

Reviewer_udTKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper tackles a practical problem: how to revoke or delete a private dataset after it has been used by a hardware-design LLM. In hardware design, datasets are important assets. An effective way to use and revoke these assets would be of great value.

Critical strengths
  1. Several metrics defined in this paper could be useful for this research area.

  2. By revealing extraction risks, limited cross-model gains, capability degradation, and residual membership leakage, the authors have thoroughly characterized the method's limitations.

  3. This paper tackles an important problem.

Critical weaknesses
  1. This paper is difficult to read. For example, the abstract contains too many experimental numbers rather than providing a clear overview of the paper. The main contributions are not clearly presented in the abstract. I also found many other parts of the paper difficult to follow.

  2. The threat model section is also difficult to read. The authors should explicitly describe the goals, capabilities, and knowledge of both the attacker and the defender.

  3. Figure 1 is of low quality. The text in the figure is not clearly presented, and some words appear distorted. Moreover, this figure only illustrates a simple revocation pipeline (instead of the main technique method), which is not very informative.

  4. The novelty of EXCISE should be discussed more clearly. Works on external-memory deletion and forgetting audits, such as Auditing Forgetting in Limited Memory Language Models, explore similar design ideas. In particular, the paper should emphasize its specific contributions to shared-memory design and provide comparisons with related work. The authors should more clearly state the differences between EXCISE and existing approaches and highlight its contributions.

  5. Scalability and cross-model generalization are concerns. For example, with 100 assets, VerilogEval performance decreases from 70/156 to 12/156, while CodeV-R1 recovery increases only from 36/192 to 37/192. These results raise concerns about scalability and generalizability.

  6. The advantages over removable-memory baselines should be discussed further. The best-of-32 stochastic audit still recovers 16/32 assets after deletion, compared with 14/32 when routing is fully disabled. The authors should clarify under what conditions selective deletion can approximate complete routing shutdown.

Questions for authors

see above

4 comments

W1: Clearer contributions and improved readability

Thank you for your suggestion. The abstract below foregrounds the four-part audit contract and asset-specific row ownership, retaining only representative utility and revocation results. To improve readability throughout the paper, the proposed revisions also bring the contribution list forward, organize the threat model by actor, distinguish inherited and added components in Figure 1, and standardize terminology.

Revoking an asset from an external-memory LLM is a lifecycle problem requiring separate evidence of authorized utility, collateral capability, selective serving removal, and artifact sanitization. We formulate an audit contract for these requirements in RTL generation and instantiate it with EXCISE, a detachable content-addressed cartridge attached to a frozen backbone. Built on Engram’s lookup primitive, EXCISE assigns asset-specific ownership of shared memory rows, making each admitted asset’s serving routes inspectable and removable. On a locked evaluation, EXCISE raises functional recovery from 2.3% to 21.1%, with a non-significant 2.6-percentage-point decrease on VerilogEval-v2. In a separate selected-layer experiment on 32 assets, selective deletion reduces the direct functional recovery rate from 81% to 0%, while preserving all 992 bystander functional outcomes. Multi-backbone and capacity studies up to 192 assets further characterize how token- and request-level gating govern the recovery–capability trade-off. Complete routing shutdown restores the backbone computation, while stochastic extraction characterizes differences between selective deletion and shutdown. Artifact audits also reveal membership signals in routing metadata and learned embeddings: registry rebuilding reduces the tested metadata attack to near chance, while fresh cartridge training on surviving assets does the same for the tested embedding attack. Overall, external memory supports practical asset-level serving revocation, while output-level deletion and artifact sanitization remain separate obligations.

W2: Explicit goals, capabilities, and knowledge for each actor

The revised text below explicitly states the goals, capabilities, and knowledge of the defender and the two adversaries, while preserving the original threat assumptions.

Defender. The defender aims to retain authorized utility and general RTL capability, revoke assets selectively, and sanitize released artifacts. It controls ingestion, serving masks, and cartridge rebuilding while keeping the backbone frozen. It knows the designated assets, reference corpus, and per-asset row registry. Assets not designated before ingestion and knowledge already encoded in the backbone are outside the revocation scope. Serving adversary (black-box). The adversary aims to recover revoked keys exactly or generate functionally equivalent RTL modules. It can query the model and observe outputs, but cannot inspect cartridge internals. It knows the target specification; prefix-assisted and two-shot tests additionally supply a 32-token target prefix or two fixed non-target examples, respectively. Adaptive and fine-tuning recovery attacks are outside the evaluated scope. Artifact holder (white-box). The holder aims to infer asset membership. It can inspect the current released cartridge's metadata, projections, and embeddings, including serving-disabled rows. It knows the tokenizer, hashing procedure, reference corpus, and full candidate texts in a fixed pool. Retained pre-revocation copies, projection inversion, and hardware side channels are outside the evaluated scope. We score serving attacks by exact key recovery or functional RTL recovery against a routing-off reference, and artifact attacks by membership inference in a fixed candidate pool.

W3: Clearer visualization of EXCISE’s core mechanisms

We have redrawn Figure 1 using clearer labels and improved spacing. The revised figure explicitly shows asset-specific row ownership, masked lookup and gated residual injection into the frozen backbone, and the separate operations for serving revocation and cartridge rebuilding.

Figure 1: EXCISE lifecycle and audit contract. Phase 1 assigns asset-specific exclusive rows; Phase 2 gates cartridge injection into a frozen backbone; Phase 3 separates serving deletion from fresh rebuilding on surviving assets. The cards summarize authorized utility, collateral capability, selective deletion, and artifact sanitization. E and X mark inherited Engram components and EXCISE additions.

W4: Asset-specific row ownership and revocation audits for shared learned memory

Thank you for pointing us to Auditing Forgetting in Limited Memory Language Models (Raeesi & Roed, 2026), which we will cite and discuss alongside LMLM (Zhao et al., 2025). While EXCISE and this audit both evaluate external-memory deletion against a lookup-disabled control, their memory substrates and technical challenges differ. LMLM stores explicit text fact triplets accessed through learned lookup calls, so deletion removes database records and aliases. In contrast, EXCISE operates on a shared learned embedding table and shared affine projections attached to a strictly frozen existing backbone, with RTL recovery assessed through formal equivalence. Fully disabling cartridge injection therefore restores the original backbone computation under matched inference conditions. In this shared memory, hash collisions can cause multiple assets to read and write the same rows, while shared projections couple rows globally. We contribute asset-specific row ownership for selective serving deletion with bystander preservation, together with white-box artifact audits that distinguish route removal from membership leakage in routing metadata and learned embeddings.

Design / reported evaluationEngramLMLM and its auditEXCISE
Memory substrateLearned embedding tableExplicit fact recordsLearned embedding table
Existing backbone kept frozen when introducing memory access××✓
Asset-specific ownership of shared learned rows—N/A✓
Target-selective removal—✓✓
Paired intact / deleted / lookup-disabled audit—✓✓
Membership audit of serialized memory——✓

✓: reported; ×: not satisfied by the reported design; —: not reported; N/A: not applicable. The backbone row concerns introducing memory access, not holding weights fixed during an audit.

(1) Row ownership supports selective revocation with bystander preservation.

EXCISE combines reference-corpus row subtraction, per-asset mutual exclusion, and fail-closed admission to assign each admitted asset exclusive routing rows. This new ablation uses layer-16 cartridges trained with 32 presentations per asset, distinct from the submission checkpoint reported in Table U3. Under direct prompts and greedy decoding, the full construction achieves 28/32 intact recoveries, zero recovery after selective deletion, and no bystander losses.

Construction and deletion policyIntact recovery /32Recovery after own deletion /32Bystander losses /992
Full construction2800
No mutual exclusion; delete all rows used by the asset271210
No mutual exclusion; retain shared rows275–64

Each asset is deleted separately, yielding 32 target and 992 bystander evaluations. With comparable intact recovery, removing mutual exclusion either introduces substantial bystander losses or retains target recovery. Only the full construction achieves both zero target recovery and unchanged bystander functional outcomes in this ablation.

This single-seed ablation matches assets, insertion layer, and training steps. The construction without mutual exclusion has 97 additional shared rows (5,248 versus 5,151); the 5–6 range reflects a reproducible Yosys crash on one fixed generated output. The reference-corpus subtraction ablation is reported in Table V1 (vsub W1).

(2) Artifact audits evaluate the release protocol after revocation.

Complementing the 32-asset audit in Table 17 of the submission, we conducted a new three-seed, four-state audit on 64-asset cartridges with 32 assets revoked. Registry rebuilding reduces metadata-attack mean AUC from 0.877 to 0.451; fresh cartridge training on surviving assets reduces the residual embedding-attack mean AUC from 1.000 to 0.479 (95% CI [0.364, 0.594]), consistent with chance under the tested attack. The embedding audit covers 29 of the 32 revoked assets. Together, these audits distinguish the roles of serving-mask deletion, registry rebuilding, and fresh cartridge rebuilding.

We also evaluate recovery through complete-module functional verification, with retrieval and per-asset LoRA comparisons in Table 4 of the submission.

W5: Scalability and cross-model generalization

Thank you for raising these concerns. We expanded the capacity study to evaluate all N assets at each size and tested three additional backbones.

(1) Scaling to 192 assets with routing control.

The submission’s layer-2, k=1 capacity experiment motivates this study. We evaluate new layer-16 cartridges with 32 training presentations per asset and vary routing controls on each fixed checkpoint.

Table U1a. CodeV-DS, layer 16, seed 0. Cells report F / VE: recoveries out of N assets and VerilogEval passes out of 156. Frozen-backbone VE is 70. Request-gated results are evaluated offline from saved memory-on and routing-off outputs. Once a cutoff is selected, branch selection uses only prompt-hit statistics.

Assets (N)k=1k=2k=3k=5k=2 + request gate (3%)
3228 / 6622 / 7014 / 7211 / 7119 / 70
6453 / 041 / 5526 / 6020 / 6939 / 70
10079 / 070 / 4854 / 6440 / 7066 / 68
192144 / 22111 / 7075 / 6744 / 69103 / 70
  • Threshold control restores capability on the same checkpoint. At 100 assets, increasing k from 1 to 5 raises VE from 0 to 70 while retaining 40 recoveries.

  • Request gating retains more recovery than k=5 at comparable VE. Across three 192-asset seeds, it achieves 103–106 recoveries with VE 65–70.

  • Prompt-hit diagnostics support more selective routing. At 100 assets, median prompt-token activation at k=1/2/5 is 5.8% / 0.7% / 0% on VerilogEval, versus 10.7% / 7.7% / 2.5% on the assets’ own prompts.

  • Selective revocation remains effective at 192 assets. At k=2 without request gating, individual deletion leaves 1, 0, and 2 direct recoveries across three seeds, compared with 2 under routing off. The seed-0 cartridge’s 32 × 32 deletion submatrix shows no bystander functional losses and two gains across 992 evaluations.

Table U1b. Capacity scaling up to 192 assets. CodeV-DS/CL, layer 16, 32 presentations per asset; seed 0 unless specified. F / VE: recoveries /N and VerilogEval-v2 passes /156. Frozen-backbone VE: DS 70, CL 44.

Backbone and cartridgeRouting offk=2 (always on)k=2 + fixed 3% gatek=2 + cross-fitted gate
DS, 32 assets1 / 7022 / 7019 / 7021 / 70
DS, 64 assets1 / 7041 / 5539 / 7039 / 69
DS, 100 assets1 / 7070 / 4866 / 6864 / 66
DS, 192 assets, seed 02 / 70111 / 70103 / 70110 / 69
DS, 192 assets, seed 12 / 70111 / 52105 / 68105 / 66
DS, 192 assets, seed 22 / 70114 / 22106 / 65106 / 63
CL, 192 assets15 / 44105 / 47100 / 44103 / 46

For the fixed-gate results in Table U1b, the gate enables memory at k=2 when at least 3% of prompt tokens meet the routing criterion; otherwise, the request uses the frozen backbone. We select the 3% cutoff on DS and reuse it unchanged on CL. Cross-fitting selects the cutoff on each half to maximize recovery within a proportionally scaled ten-pass VE loss budget, applies it to the other half, and pools both held-out halves.

(2) Recovery gains across four backbones.

We use the same 32 assets, layer 16, and 32 training presentations per asset. Training carriers match serving formats: raw prompts for DS/CL and chat templates for QW/R1. For all four backbones, k=2 is the primary setting and k=1 is secondary; this choice was prespecified for the additional backbones.

Seed 0, greedy direct-code decoding. Recovery is out of 32; VE is out of 156. Entries show routing off → k=2, with k=1 in parentheses; both are ungated. R1’s batch-size-1 routing-off VE is 76/156 (submission: 75/156).

Backbone (base model)Recovery, off → k=2 (k=1)VE, off → k=2 (k=1)
CodeV-DS (DeepSeek-Coder)1 → 22 (28)70 → 70 (66)
CodeV-CL (CodeLlama)5 → 20 (25)44 → 43 (38)
CodeV-QW (CodeQwen1.5)3 → 15 (22)75 → 75 (75)
CodeV-R1 (reasoning-distilled Qwen2.5)8 → 12 (20)76 → 76 (34)
  • DS, CL, and QW achieve significant recovery gains at k=2 (exact McNemar, p < 0.001), with VE within one pass of their frozen backbones.

  • CL replicates across seeds and scales to 192 assets. Three training seeds recover 20–22/32 assets, with VE 43–49 versus a backbone score of 44.

  • QW demonstrates the benefit of matching training and serving formats. Under the same chat serving, raw-trained carriers recover 6/32 assets, while chat-trained carriers recover 15/32 (+28.1 percentage points, 95% CI [6.3, 46.9]).

  • At secondary k=1, R1 recovery rises from 8 to 20/32 (p < 0.001). The 3% request gate using k=1 prompt-hit statistics retains all 20 recoveries with VE 71/156 (routing off: 76/156). See BCY6 Q1 for format and budget diagnostics.

In a matched 192-asset DS control, increasing updates from 1,024 to 6,144 raises recovery from 14 to 111 and VE from 67 to 70.

W6: Selective revocation and removable-memory comparisons

(1) Conditions for matching complete routing shutdown.

Routing off measures recovery from the frozen backbone and prompt alone. Selective deletion exactly restores backbone computation when memory injection remains zero throughout prefill and decoding, either because surviving rows never meet the routing threshold or because the request gate selects routing off for the entire request. This assumes matched inputs, fresh KV caches, numerical execution, and decoding settings; identical sampled tokens additionally require coupled random draws.

Both conditions are supported by our audits. All audited zero-injection cases are token-identical to routing off, including 82/192 CL greedy requests and 99 nonempty sampled completions in the new audit. In the offline evaluation, the 3% request gate using k=2 prompt-hit statistics selects routing off for every tested post-deletion direct prompt: 192 per DS seed and on CL, and 32 each on QW and R1. These gate checks are separate from R1’s k=1 gated utility result.

Without a request gate, the zero-injection condition must also hold during decoding, when surviving rows can become active. The stochastic audits below evaluate this ungated setting at k=1.

Table U2. Stochastic deletion audits. Four prompt families × 32 samples give 128 attempts per asset and 4,096 completions per state. Ranges reflect a reproducible Yosys crash on one fixed output. Generation/scoring settings differ across audits; each uses its own routing-off reference. The layer-16 audit uses the submission checkpoint, distinct from W5’s newly trained cartridges.

Audit metric (k=1, ungated)IntactSelf-deletedRouting off
Submission (layer 2): assets recovered /32241614
Submission (layer 2): successful completions /4,096911 (22.2%)230 (5.6%)222 (5.4%)
New (layer 16): assets recovered /323015–1614
New (layer 16): successful completions /4,0961,242 (30.3%)172–173 (4.2%)211 (5.2%)

In the new layer-16 audit, successful direct completions fall from 763/1,024 to 13/1,024 after deletion; routing off also gives 13, on the same two assets.

The stochastic audit distinguishes asset coverage from recovery frequency. All 14 assets recovered with routing off are also recovered after deletion in each audit. The three confirmed additional assets across the audits each succeed in 1–2 of 128 attempts. Because asset-level recovery counts any successful attempt, the new audit can recover 15–16 assets versus 14 while producing fewer successful completions overall: 172–173 versus 211.

Under greedy decoding across the same four prompt families, deletion recovers 6 assets at layer 2 and 5 at layer 16, versus 6 with routing off.

(2) Serving and capability advantages over removable-memory baselines.

Both baselines support removal of their external state. EXCISE combines automatic content routing, retained RTL capability, and serving without reference RTL in the prompt.

Table U3. Removable-memory baselines. Recovery and VE results reproduce Table 4 of the submission. EXCISE uses the submission’s layer-16, k=1 checkpoint, also used in the layer-16 audit in Table U2.

System dimensionEXCISEPer-asset LoRARetrieval (RAG)
Authorized recoveries /322629 (p = 0.51 versus EXCISE)14
VE /156 (frozen backbone: 70)750–77 across adapters; mean 4165
One loaded state serves all assetsYes, routed by contentNo, one adapter per assetYes
Stored state554.6 MB, fixed-size table538 MB for 32 adapters; projected 3.87 GB for 2301.5 MB for 230 records
Reference RTL placed in the promptNoNoYes

EXCISE uses one shared cartridge with automatic content routing and achieves higher VE than the mean across loaded per-asset LoRA adapters. The evaluated LoRA baseline requires adapter selection and adds approximately 16.8 MB per asset, while EXCISE uses a fixed-size table. Compared with the evaluated RAG baseline, EXCISE achieves higher recovery without placing reference RTL in the prompt.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.