- Rating
- 3(+1)
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies revocation of designated assets from an LLM used for RTL generation. EXCISE stores these assets in a detachable, hash-addressed memory cartridge attached to a frozen backbone. Its construction combines reference-corpus row exclusion, per-asset mutual exclusion, and gated memory injection. The paper distinguishes four evaluation requirements: authorized recovery, collateral capability, selective serving deletion, and serialized-artifact sanitization. On a locked 128-task evaluation, functional recovery increases from 3/128 to 27/128, while VerilogEval-v2 changes from 70/156 to 66/156. A separate selected-layer 32-asset checkpoint achieves 26/32 direct functional recoveries before deletion and 0/32 afterward, preserving all 992 bystander functional outcomes. Complete routing shutdown restores the frozen computation.
Critical strengths
- The four-part audit contract is the strongest contribution. The paper supports the distinction between serving deletion and artifact sanitization with separate routing-metadata and learned-value attacks. This provides a more meaningful evaluation framework than reporting rollback or output similarity alone.
- The paper distinguishes compilation, synthesis, and strict functional recovery rather than relying on character overlap. Cross-deletion matrices, stochastic extraction, paired capability outcomes, layer sensitivity, and removable-memory baselines provide substantial evidence for the main configuration while exposing important failure modes.
Critical weaknesses
The exact rollback property is clear, but useful authorized recovery is strongly configuration-dependent. The locked CodeV-R1 evaluation provides essentially no net functional improvement at the capability-preserving setting. This limits confidence that EXCISE offers a broadly applicable memory mechanism rather than an effective operating point for one backbone and asset construction.
Questions for authors
- What explains the nearly null CodeV-R1 result? Can the existing experiments distinguish insertion-layer mismatch, optimization limitations, and interference with the model’s generation trajectory?
- What is the measured cost of rebuilding from surviving assets after revocation, compared with initial cartridge training?
5 comments
Q1: Controlled diagnostics identify format alignment as a key factor in CodeV-R1 recovery
Thank you for asking us to distinguish these explanations. Our controlled experiments identify training–serving format alignment as a directly supported contributor to CodeV-R1 recovery. Under the shared recipe, R1 reaches 20/32 recoveries, which request gating retains with VE at 71/156 (W1).
(1) Layer, fitting, and injection-timing diagnostics.
We first examined the original raw-carrier/chat-serving setup on development assets.
-
Insertion layer. Across six layers and two training budgets on 64 development assets, the best recovery within 10 VE passes of the backbone is 18/64 versus 12/64 with routing off.
-
Optimization. In five of the six evaluated layers, teacher-forced answer NLL decreases from 0.43 to 0.03–0.04 after 3,000 updates, demonstrating effective reference-answer fitting. In six matched 1,000-versus-3,000-update comparisons where both checkpoints remain within 10 VE passes of the backbone, recovery changes by −1 to +2 assets.
-
Injection timing. We separately tested full-request, prefill-only, decoding-only, and disabled injection.
Table B1. Functional recoveries out of 32. Both configurations use development cartridges trained on 64 development assets; the stage ablation evaluates their first 32 assets.
| Configuration | Full request | Prefill only | Decoding only | Routing off |
|---|---|---|---|---|
| Layer 21, 1,000 updates, k=1 | 4 | 6 | 4 | 5 |
| Layer 7, 3,000 updates, k=2 | 8 | 7 | 5 | 5 |
Prefill-only injection has configuration-dependent effects and retains the memory-influenced prefill state. W1’s request-level gate instead determines whether memory is enabled for the entire request.
(2) Format controls demonstrate recovery gains at a fixed training budget.
We first tested serving-format sensitivity while keeping each trained checkpoint fixed.
Table B2. Serving-format control. Each row compares raw and chat serving using the same raw-trained checkpoint, asset set, and routing threshold. Both formats were regenerated with batch size 1. Entries report routing off → cartridge-enabled recovery; k=1 for DS/CL/QW and k=2 for R1. QW and R1 use diagnostic checkpoints distinct from those in Table B3, with each raw/chat comparison performed on the same fixed checkpoint. The R1 probe evaluates 64 assets disjoint from the development set.
| Backbone | Evaluated assets | Raw serving (matched) | Chat serving (mismatched) |
|---|---|---|---|
| CodeV-DS | 32 | 1 → 28 | 6 → 5 |
| CodeV-CL | 32 | 5 → 25 | 4 → 10 |
| CodeV-QW | 32 | 1 → 21 | 3 → 4 |
| CodeV-R1 | 64 | 0 → 29 | 14 → 17 |
Among the 50 assets recovered by neither routing-off format, R1 recovers 20 under matched raw serving versus 5 under mismatched chat serving.
We then conducted a separate 2×2 training–serving format experiment using the shared recipe from W1: the same 32 assets, layer 16, and 32 presentations per asset. For each backbone, raw-trained and chat-trained cartridges were evaluated under both serving formats.
Table B3. Functional recoveries out of 32, one training seed. Each cell reports routing off / mismatched carriers / matched carriers. All entries are ungated; k=2 is primary and k=1 secondary.
| Backbone | Threshold | Chat serving: off / mismatch / match | Raw serving: off / mismatch / match |
|---|---|---|---|
| CodeV-QW | k=2 | 3 / 6 / 15 | 1 / 2 / 13 |
| CodeV-QW | k=1 | 3 / 6 / 22 | 1 / 2 / 18 |
| CodeV-R1 | k=2 | 8 / 9 / 12 | 0 / 2 / 15 |
| CodeV-R1 | k=1 | 8 / 9 / 20 | 0 / 4 / 20 |
Matched carriers improve recovery in all eight comparisons. For R1 at k=1, matching raises recovery from 9 to 20 under chat serving and from 4 to 20 under raw serving. All four k=1 format comparisons have 95% confidence intervals above zero.
Q2: Measured rebuilding cost relative to initial cartridge training
Thank you for asking for this comparison. On a single H100, fresh rebuilding takes 3.1–19.6 minutes in the measured settings. Rebuilding after removing one asset costs 97–99% of initial training time; removing eight of 32 assets reduces this to 79%.
Each rebuild starts with a newly initialized cartridge and optimizer and trains only on surviving assets, using 32 presentations per surviving asset with the backbone frozen. No learned state from the old cartridge is reused.
Table B4. New paired training/rebuilding timings. Seconds, with training-loop time in parentheses. CodeV-DS, layer 16, seed 0, one H100. Job time includes setup, training, internal validation, serialization, and external checks; subsequent functional, VerilogEval, and membership evaluations are excluded. Each run was timed once; both 32-asset cases share one initial run.
| Assets: initial → surviving | Training updates | Initial run (s) | Rebuild (s) | Rebuild / initial |
|---|---|---|---|---|
| 32 → 31 | 1,024 → 992 | 232 (187) | 226 (182) | 97% |
| 32 → 24 | 1,024 → 768 | 232 (187) | 184 (140) | 79% |
| 192 → 191 | 6,144 → 6,112 | 1,184 (1,127) | 1,174 (1,118) | 99% |
-
Rebuilding cost follows the surviving training budget. Across these runs, training takes approximately 0.18 s per update, or 6 s per surviving asset, with 44–57 s of job overhead. Multiple revocations can share one rebuild before artifact release, as demonstrated by the 32→24 case.
-
Serving revocation requires no retraining. Updating the in-memory mask, , takes a median of 0.023 ms over 200 measurements in each of the three cases, with the cartridge loaded and mask indices prepared. This measures the mask update itself, excluding setup and distribution.
Why our release protocol uses a fresh rebuild. In a separate three-seed audit of 64-asset cartridges with 32 assets revoked, fresh rebuilding reduces the embedding-attack mean AUC to 0.479 (95% CI [0.364, 0.594]), compared with 1.000 after registry rebuilding alone. This is consistent with chance under the tested attack, which covers 29 of the 32 revoked assets.
W1: Recovery and selective revocation across four backbones with a shared training recipe
Thank you for highlighting the importance of cross-backbone validation. We evaluated the same 32 assets across four backbones using a shared training recipe: layer 16, 32 presentations per asset, and carriers matching each backbone’s serving format—raw prompts for CodeV-DS/CL and chat-format carriers for CodeV-QW/R1.
Cross-backbone recovery and deletion. Seed 0, greedy direct-code decoding. Recovery is out of 32 assets and VE out of 156 tasks. The primary threshold is k=2; secondary k=1 results appear in parentheses. Both k columns are ungated. Deletion results use ungated k=2. R1’s routing-off VE is 76/156 when re-evaluated at batch size 1 (submission: 75/156).
Significant recovery gains across three backbones at the primary threshold. At k=2, CodeV-DS, CodeV-CL, and CodeV-QW all achieve significant gains over routing off (exact McNemar, p < 0.001), with VE within one pass of their respective frozen backbones.
CodeV-R1 achieves significant recovery under the shared training recipe. In the new 32-asset experiment, the secondary threshold k=1 increases recovery from 8 to 20/32 (p < 0.001). The 3% request gate, based on k=1 prompt-hit statistics, retains all 20 recoveries with VE at 71/156, compared with 76/156 under routing off. Q1 provides the accompanying format and training-budget diagnostics.
CodeV-CL replicates across seeds and scales to 192 assets. Three training seeds recover 20, 22, and 21 of 32 assets, with VE 43, 46, and 49 (frozen backbone: 44). At 192 assets, CodeV-CL recovers 105 versus 15 with routing off, with VE 47 versus 44 (one seed).
Selective deletion remains effective across backbones. At k=2, deletion reduces recovery to at or below routing-off counts on all four 32-asset evaluations. On CodeV-CL at 192 assets, recovery falls from 105 to 16, versus 15 with routing off, while all 992 bystander functional outcomes remain unchanged in the tested 32×32 deletion submatrix.
CodeV-QW and CodeV-R1 each use one training seed. Request-gated results are offline compositions of saved outputs using prompt-hit statistics.