- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Localized Recall-Capped Unlearning (LRCU), a parameter-efficient unlearning method for diffusion language models. This method selects three consecutive Transformer blocks through activation patching and updates only the parameters of these layers. The training objective combines capped forget cross-entropy, retain loss, and Min-SNR-based masking-level weighting, where the capped loss stops gradient ascent after the cross-entropy of the forgotten token exceeds a threshold. The paper also proposes a theoretical bound on the verbatim span recall probability. The authors evaluate the methods on LLaDA and Dream using WMDP, TOFU, and RWKU, and supplement this with comparisons of different forgetting objective functions, ablation experiments, computational cost analysis, and relearning attack.
Critical strengths
- The LRCU method is simple, requiring no reference model and updating only a small number of Transformer blocks, thus reducing training costs. The comparison of the objective function under a fixed band of updated blocks also provides some support for the effectiveness of the capped forget loss.
- The paper evaluates the method on two diffusion language models and multiple unlearning benchmarks, and provides component ablation, runtime analysis, and autoregressive control. These experiments help to understand the practical effects and limitations of the method.
Critical weaknesses
- In Tables 1–2, LRCU uses different learning rates, optimizers, masking weights, and retain losses compared to the baselines. Therefore, performance differences may stem from the training protocol rather than the method itself. In particular, some baselines lack retain loss but are directly used for forgetting-utility comparisons, resulting in a lack of fair control over the main experimental results.
- The core design of LRCU significantly overlaps with existing work. RMU already uses local parameter updates for WMDP unlearning, while Cha et al. (ICLR 2025) have investigated bounded forgetting objectives and efficient parameter updates. This paper does not directly compare with these methods; therefore, its new contributions and practical advantages remain unclear.
Questions for authors
- Can LRCU be compared with full-model baselines again under the same retain loss, optimizer, and parameter tuning protocol? Do the zero scores on TOFU still appear for GA, NPO, and SimNPO?
- How does LRCU perform compared to the methods of RMU and Cha et al. (ICLR 2025)? Can WMDP-bio, WMDP-cyber, and bio-adjacent MMLU be reported simultaneously?
- The performance difference between capped loss and WGA on LLaDA in Table 3 is small. Can multi-seed results be provided to confirm whether this improvement outweighs experimental fluctuations?
- After only five relearning updates, WMDP-bio accuracy almost recovered to its original level. How did the authors distinguish between true knowledge removal and temporary recall suppression?
- The band selected based on causal analysis is not optimal. Could similar performance be achieved using a fixed early-layer band?
- Tables 1, 3, 4, and 7 all report LRCU at blocks 5–7 on LLaDA WMDP-bio with c=1, but H ranges from 0.641 to 0.675. What differs between these runs?
- How are WMDP and MMLU accuracies computed for the diffusion models, e.g., from a single forward pass over the masked answer letter or from a likelihood estimate of each option? Which forget data are used for WMDP-bio and WMDP-cyber?
Minor: the number of MMLU subjects differs across experiments; Table 5 reports A100 while Appendix E reports H100; and the bound in Proposition 1 is relatively weak and does not cover greedy decoding.
8 comments
Response 2: New Baselines and Seeds
[W2, Q2] RMU and Cha et al.
We added the missing comparisons under the same protocol: RMU, IHL (Cha et al.), SHA256 and Cheng et al. (2026). The three closest methods differ as follows:
- RMU steers the hidden states of forget inputs toward a random direction and updates fixed layers 5–7.
- IHL bounds a probability margin for AR models and updates LoRA weights.
- LRCU caps the per-token CE at masked positions and updates a band selected for each DLM and task.
The RMU row uses our band, so it differs from LRCU in the forget objective, which is the comparison of Table 3. RMU's fixed layers 5–7 with our cap is our fixed-early-band control, as Section 2 now states. Among these methods, LRCU has the highest mean H on both models, and IHL is within noise on LLaDA.
| Method | LLaDA H | LLaDA Bio MMLU | Dream H | Dream Bio MMLU |
|---|---|---|---|---|
| LRCU (cap + band) | 0.657 ± 0.013 | 0.525 | 0.596 ± 0.027 | 0.562 |
| IHL (band) | 0.652 ± 0.018 | 0.500 | 0.512 ± 0.098 | 0.440 |
| RMU (band) | 0.583 ± 0.008 | 0.451 | 0.450 ± 0.038 | 0.307 |
| SHA256 (blocks 0–5 MLP) | 0.619 ± 0.027 | 0.354 | 0.505 ± 0.132 | 0.331 |
| Cheng et al. (two stages, full model) | 0.473 ± 0.013 | 0.651 | 0.430 ± 0.016 | 0.733 |
| Cheng et al. (stage 1, band) | 0.648 ± 0.021 | 0.512 | 0.524 ± 0.085 | 0.681 |
We also ran the new baselines on WMDP-cyber under the same protocol:
| Method | LLaDA cyber acc. ↓ | LLaDA H | LLaDA Bio MMLU | Dream cyber acc. ↓ | Dream H | Dream Bio MMLU |
|---|---|---|---|---|---|---|
| LRCU (cap + band) | 0.266 ± 0.015 | 0.646 ± 0.017 | 0.633 | 0.273 ± 0.009 | 0.706 ± 0.006 | 0.745 |
| IHL (band) | 0.250 ± 0.010 | 0.578 ± 0.129 | 0.521 | 0.258 ± 0.008 | 0.683 ± 0.009 | 0.726 |
| RMU (band) | 0.353 ± 0.003 | 0.632 ± 0.003 | 0.645 | 0.402 ± 0.031 | 0.642 ± 0.018 | 0.740 |
| SHA256 (blocks 0–5 MLP) | 0.256 ± 0.007 | 0.650 ± 0.005 | 0.666 | 0.268 ± 0.036 | 0.619 ± 0.114 | 0.573 |
| Cheng et al. (stage 1, band) | 0.368 ± 0.096 | 0.524 ± 0.144 | 0.517 | 0.247 ± 0.005 | 0.661 ± 0.047 | 0.706 |
On Dream, LRCU has the best H and beats IHL, RMU and SHA256 on all three seeds. On LLaDA it is on par with SHA256 while updating fewer parameters (8.2% vs 11%), and above the rest. On both models accuracy falls to chance while Bio MMLU stays near the base value (0.664 and 0.742). These rows, with Bio MMLU, are now in Table 1, and Related Work cites the methods.
[Q3, Q6] Seeds and the four H values
The four values (Tables 1, 3, 4 and 7) came from different runs: Tables 1, 4 and 7 are single-seed runs and Table 3 is a three-seed mean.
| Model | Table 1 | Table 3 | Table 4 | Three seeds (H200) |
|---|---|---|---|---|
| LLaDA | 0.641 | 0.668 | 0.675 | 0.659 ± 0.008 |
| Dream | 0.649 | 0.634 | 0.624 | 0.618 ± 0.024 |
On cap vs WGA (Q3), the cap now has a higher mean H on both models (LLaDA 0.657 ± 0.013 vs 0.634 ± 0.025, Dream 0.596 ± 0.027 vs 0.554 ± 0.067) and wins on all three seeds on LLaDA. Tables 1, 3 and 4 now report three-seed mean ± std.
Response 3: Relearning and Fixed Early Band
[Q4] Removal or suppression
We agree that fast recovery shows limited resistance, and we extended the relearning test to more methods and budgets:
| Method | After unlearning | lr 1e-5, 10 updates | lr 1e-5, 40 updates | lr 5e-5, 80 updates | Unseen questions |
|---|---|---|---|---|---|
| LRCU | 0.274 ± 0.021 | 0.290 ± 0.038 | 0.628 ± 0.030 | 0.622 ± 0.018 | 0.651 ± 0.012 |
| IHL | 0.267 ± 0.008 | 0.338 ± 0.133 | 0.521 ± 0.150 | 0.609 ± 0.018 | 0.509 ± 0.228* |
| WGA | 0.266 ± 0.008 | 0.279 ± 0.019 | 0.504 ± 0.145 | 0.620 ± 0.012 | 0.647 ± 0.010 |
| RMU | 0.329 ± 0.019 | 0.474 ± 0.022 | 0.637 ± 0.006 | 0.628 ± 0.006 | 0.654 ± 0.016 |
| NPO | 0.251 ± 0.005 | 0.275 ± 0.028 | 0.391 ± 0.233* | 0.472 ± 0.198* | 0.387 ± 0.245* |
| SimNPO | 0.264 ± 0.026 | 0.331 ± 0.055 | 0.640 ± 0.028 | 0.495 ± 0.221* | 0.516 ± 0.235* |
WMDP-bio accuracy on LLaDA (base 0.657) after full-model relearning on the forget corpus. Unseen questions: the held-out half after 40 updates on the other half of the WMDP-bio questions. *: the attack also breaks general ability in some seeds, so the low value is not resistance.
The held-out questions recover as well, so we now describe LRCU as lowering recall under the tested conditions, not as removing knowledge. This holds for all tested methods. Within that scope the effect is stable across three prompt formats (0.273 to 0.280). We have added this to Appendix H and to the limitations.
[Q5] Fixed early band
To test a fixed early band, we kept the capped loss fixed and changed only the band.
| Band selection | LLaDA band | LLaDA H | Dream band | Dream H |
|---|---|---|---|---|
| Causal (ours) | 5–7 | 0.659 ± 0.008 | 10–12 | 0.618 ± 0.024 |
| Fixed early layers (RMU-style) | 5–7 | 0.659 ± 0.008 | 5–7 | 0.582 ± 0.089 |
| Gradient saliency | 6–8 | 0.656 ± 0.006 | 25–27 | 0.488 ± 0.033 |
| Random (mean of 5 bands) | various | 0.555 | various | 0.563 |
| Deep band | 19–21 | 0.457 ± 0.013 | 20–22 | 0.534 ± 0.014 |
| Simplified: fixed 5–7, uniform weights | 5–7 | 0.661 ± 0.003 | 5–7 | 0.638 ± 0.008 |
| No cap (GA in our band) | 5–7 | 0.380 ± 0.022 | 10–12 | 0.359 ± 0.002 |
Every row uses the capped loss and the paper training setting, except the last.
On LLaDA the fixed early band is our selected band, and neighbors 4–6 and 7–9 are within noise (0.662 ± 0.003, 0.656 ± 0.011). So a fixed early band is a good default for WMDP-bio.
One caution: this default is known only with hindsight. The causal analysis is what points to the early region, and it is a cheap, forward-only step that also avoids poor ones. For TOFU it selects blocks 23–25 (Fig. 3), so a fixed early band may not transfer. This table is in Appendix J.
Response 1: Fair Control in the Main Experiment
We thank the reviewer for the comments. We ran new experiments to address them. Unless stated otherwise, they use WMDP-bio with three seeds (mean ± std) on H200 GPUs, and H is the harmonic mean of 1 − forget and MMLU.
[W1, Q1] Fair control in the main experiment
For a fair control, we reran every method with the same CE retain loss, optimizer, schedule, mask weighting and tuning budget, in a band scope and a full-model scope.
H under the unified protocol. Band is blocks 5–7 (LLaDA) and 10–12 (Dream). † SHA256 in its original scope (MLPs of blocks 0–5). For reference, LRCU under the paper setting gives 0.659 ± 0.008 (LLaDA) and 0.618 ± 0.024 (Dream).
On LLaDA, cap + band has the highest mean H of all 21 cells; IHL (band) and Cheng et al. (stage 1, band) are within one standard deviation. On Dream, it is above every objective ported from AR models or classifiers. Only the MDU-style anchor is higher, and it gains from our band on both models. Compared with this anchor, LRCU needs no reference model, updates under 10% of the parameters and is 2.4× faster per step on LLaDA.
The TOFU zeros in Q1 came from baselines without a retain loss. With the CE retain loss, full-model NPO and SimNPO no longer collapse on WMDP-bio (MMLU 0.473 to 0.658), and we will update TOFU under the same protocol in the final version. These rows are now in Table 1.