acceptodds
Under review as a conference paper at ICLR 2027

Localized Recall-Capped Unlearning for Diffusion Language Models

Abstract

Diffusion language models (DLMs) are emerging as an alternative to autoregressive language models. Unlearning is needed to remove unwanted knowledge from trained models, yet weight-space unlearning in DLMs remains largely unexplored. Empirically, existing objectives either suppress target knowledge at the expense of general capability or preserve utility with insufficient forgetting. We identify update location and gradient control as two key factors underlying these failures. Our method, Localized Recall-Capped Unlearning (LRCU), uses causal-mediation localization to select three consecutive transformer blocks. It updates only these blocks using capped per-token cross-entropy, with each token contributing zero forgetting gradient once its cross-entropy exceeds the cap. LRCU requires no reference model and updates less than 10% of parameters, with a 1.5 average per-step speedup over the compared full-model baselines on LLaDA. Across WMDP-bio, WMDP-cyber, RWKU, and TOFU on two DLM backbones, LRCU achieves a competitive forgetting-utility trade-off compared with existing methods. These results highlight the importance of update location and gradient control for effective DLM unlearning. We will release an Open-DLU codebase upon acceptance for weight-space unlearning in DLMs, together with unlearned checkpoints to support further research.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
machine unlearning, diffusion language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NpULOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Localized Recall-Capped Unlearning (LRCU), a parameter-efficient unlearning method for diffusion language models. This method selects three consecutive Transformer blocks through activation patching and updates only the parameters of these layers. The training objective combines capped forget cross-entropy, retain loss, and Min-SNR-based masking-level weighting, where the capped loss stops gradient ascent after the cross-entropy of the forgotten token exceeds a threshold. The paper also proposes a theoretical bound on the verbatim span recall probability. The authors evaluate the methods on LLaDA and Dream using WMDP, TOFU, and RWKU, and supplement this with comparisons of different forgetting objective functions, ablation experiments, computational cost analysis, and relearning attack.

Critical strengths
  1. The LRCU method is simple, requiring no reference model and updating only a small number of Transformer blocks, thus reducing training costs. The comparison of the objective function under a fixed band of updated blocks also provides some support for the effectiveness of the capped forget loss.
  2. The paper evaluates the method on two diffusion language models and multiple unlearning benchmarks, and provides component ablation, runtime analysis, and autoregressive control. These experiments help to understand the practical effects and limitations of the method.
Critical weaknesses
  1. In Tables 1–2, LRCU uses different learning rates, optimizers, masking weights, and retain losses compared to the baselines. Therefore, performance differences may stem from the training protocol rather than the method itself. In particular, some baselines lack retain loss but are directly used for forgetting-utility comparisons, resulting in a lack of fair control over the main experimental results.
  2. The core design of LRCU significantly overlaps with existing work. RMU already uses local parameter updates for WMDP unlearning, while Cha et al. (ICLR 2025) have investigated bounded forgetting objectives and efficient parameter updates. This paper does not directly compare with these methods; therefore, its new contributions and practical advantages remain unclear.
Questions for authors
  1. Can LRCU be compared with full-model baselines again under the same retain loss, optimizer, and parameter tuning protocol? Do the zero scores on TOFU still appear for GA, NPO, and SimNPO?
  2. How does LRCU perform compared to the methods of RMU and Cha et al. (ICLR 2025)? Can WMDP-bio, WMDP-cyber, and bio-adjacent MMLU be reported simultaneously?
  3. The performance difference between capped loss and WGA on LLaDA in Table 3 is small. Can multi-seed results be provided to confirm whether this improvement outweighs experimental fluctuations?
  4. After only five relearning updates, WMDP-bio accuracy almost recovered to its original level. How did the authors distinguish between true knowledge removal and temporary recall suppression?
  5. The band selected based on causal analysis is not optimal. Could similar performance be achieved using a fixed early-layer band?
  6. Tables 1, 3, 4, and 7 all report LRCU at blocks 5–7 on LLaDA WMDP-bio with c=1, but H ranges from 0.641 to 0.675. What differs between these runs?
  7. How are WMDP and MMLU accuracies computed for the diffusion models, e.g., from a single forward pass over the masked answer letter or from a likelihood estimate of each option? Which forget data are used for WMDP-bio and WMDP-cyber?

Minor: the number of MMLU subjects differs across experiments; Table 5 reports A100 while Appendix E reports H100; and the bound in Proposition 1 is relatively weak and does not cover greedy decoding.

8 comments
AuthorsOpenReview ↗

Response 1: Fair Control in the Main Experiment

We thank the reviewer for the comments. We ran new experiments to address them. Unless stated otherwise, they use WMDP-bio with three seeds (mean ± std) on H200 GPUs, and H is the harmonic mean of 1 − forget and MMLU.

[W1, Q1] Fair control in the main experiment

For a fair control, we reran every method with the same CE retain loss, optimizer, schedule, mask weighting and tuning budget, in a band scope and a full-model scope.

MethodLLaDA bandLLaDA full modelDream bandDream full model
LRCU (cap)0.657 ± 0.0130.563 ± 0.0200.596 ± 0.0270.542 ± 0.067
GA + retain (GD)0.449 ± 0.0060.546 ± 0.0910.455 ± 0.0120.395 ± 0.001
NPO0.628 ± 0.0380.625 ± 0.0550.527 ± 0.0920.547 ± 0.008
SimNPO0.566 ± 0.0720.533 ± 0.0250.550 ± 0.0350.525 ± 0.002
WGA0.634 ± 0.0250.526 ± 0.0220.554 ± 0.0670.559 ± 0.067
IHL0.652 ± 0.0180.486 ± 0.0410.512 ± 0.0980.497 ± 0.051
RMU0.583 ± 0.008n/a0.450 ± 0.038n/a
SHA2560.449 ± 0.0020.619 ± 0.027†0.476 ± 0.0270.505 ± 0.132†
Cheng et al. (two stages)0.436 ± 0.0370.473 ± 0.0130.494 ± 0.0040.430 ± 0.016
Cheng et al. (stage 1)0.648 ± 0.0210.486 ± 0.0200.524 ± 0.0850.497 ± 0.008
MDU-style anchor0.637 ± 0.0140.592 ± 0.0170.652 ± 0.0070.630 ± 0.020

H under the unified protocol. Band is blocks 5–7 (LLaDA) and 10–12 (Dream). † SHA256 in its original scope (MLPs of blocks 0–5). For reference, LRCU under the paper setting gives 0.659 ± 0.008 (LLaDA) and 0.618 ± 0.024 (Dream).

On LLaDA, cap + band has the highest mean H of all 21 cells; IHL (band) and Cheng et al. (stage 1, band) are within one standard deviation. On Dream, it is above every objective ported from AR models or classifiers. Only the MDU-style anchor is higher, and it gains from our band on both models. Compared with this anchor, LRCU needs no reference model, updates under 10% of the parameters and is 2.4× faster per step on LLaDA.

The TOFU zeros in Q1 came from baselines without a retain loss. With the CE retain loss, full-model NPO and SimNPO no longer collapse on WMDP-bio (MMLU 0.473 to 0.658), and we will update TOFU under the same protocol in the final version. These rows are now in Table 1.

AuthorsOpenReview ↗

Response 2: New Baselines and Seeds

[W2, Q2] RMU and Cha et al.

We added the missing comparisons under the same protocol: RMU, IHL (Cha et al.), SHA256 and Cheng et al. (2026). The three closest methods differ as follows:

  • RMU steers the hidden states of forget inputs toward a random direction and updates fixed layers 5–7.
  • IHL bounds a probability margin for AR models and updates LoRA weights.
  • LRCU caps the per-token CE at masked positions and updates a band selected for each DLM and task.

The RMU row uses our band, so it differs from LRCU in the forget objective, which is the comparison of Table 3. RMU's fixed layers 5–7 with our cap is our fixed-early-band control, as Section 2 now states. Among these methods, LRCU has the highest mean H on both models, and IHL is within noise on LLaDA.

MethodLLaDA HLLaDA Bio MMLUDream HDream Bio MMLU
LRCU (cap + band)0.657 ± 0.0130.5250.596 ± 0.0270.562
IHL (band)0.652 ± 0.0180.5000.512 ± 0.0980.440
RMU (band)0.583 ± 0.0080.4510.450 ± 0.0380.307
SHA256 (blocks 0–5 MLP)0.619 ± 0.0270.3540.505 ± 0.1320.331
Cheng et al. (two stages, full model)0.473 ± 0.0130.6510.430 ± 0.0160.733
Cheng et al. (stage 1, band)0.648 ± 0.0210.5120.524 ± 0.0850.681

We also ran the new baselines on WMDP-cyber under the same protocol:

MethodLLaDA cyber acc. ↓LLaDA HLLaDA Bio MMLUDream cyber acc. ↓Dream HDream Bio MMLU
LRCU (cap + band)0.266 ± 0.0150.646 ± 0.0170.6330.273 ± 0.0090.706 ± 0.0060.745
IHL (band)0.250 ± 0.0100.578 ± 0.1290.5210.258 ± 0.0080.683 ± 0.0090.726
RMU (band)0.353 ± 0.0030.632 ± 0.0030.6450.402 ± 0.0310.642 ± 0.0180.740
SHA256 (blocks 0–5 MLP)0.256 ± 0.0070.650 ± 0.0050.6660.268 ± 0.0360.619 ± 0.1140.573
Cheng et al. (stage 1, band)0.368 ± 0.0960.524 ± 0.1440.5170.247 ± 0.0050.661 ± 0.0470.706

On Dream, LRCU has the best H and beats IHL, RMU and SHA256 on all three seeds. On LLaDA it is on par with SHA256 while updating fewer parameters (8.2% vs 11%), and above the rest. On both models accuracy falls to chance while Bio MMLU stays near the base value (0.664 and 0.742). These rows, with Bio MMLU, are now in Table 1, and Related Work cites the methods.


[Q3, Q6] Seeds and the four H values

The four values (Tables 1, 3, 4 and 7) came from different runs: Tables 1, 4 and 7 are single-seed runs and Table 3 is a three-seed mean.

ModelTable 1Table 3Table 4Three seeds (H200)
LLaDA0.6410.6680.6750.659 ± 0.008
Dream0.6490.6340.6240.618 ± 0.024

On cap vs WGA (Q3), the cap now has a higher mean H on both models (LLaDA 0.657 ± 0.013 vs 0.634 ± 0.025, Dream 0.596 ± 0.027 vs 0.554 ± 0.067) and wins on all three seeds on LLaDA. Tables 1, 3 and 4 now report three-seed mean ± std.

AuthorsOpenReview ↗

Response 3: Relearning and Fixed Early Band

[Q4] Removal or suppression

We agree that fast recovery shows limited resistance, and we extended the relearning test to more methods and budgets:

MethodAfter unlearninglr 1e-5, 10 updateslr 1e-5, 40 updateslr 5e-5, 80 updatesUnseen questions
LRCU0.274 ± 0.0210.290 ± 0.0380.628 ± 0.0300.622 ± 0.0180.651 ± 0.012
IHL0.267 ± 0.0080.338 ± 0.1330.521 ± 0.1500.609 ± 0.0180.509 ± 0.228*
WGA0.266 ± 0.0080.279 ± 0.0190.504 ± 0.1450.620 ± 0.0120.647 ± 0.010
RMU0.329 ± 0.0190.474 ± 0.0220.637 ± 0.0060.628 ± 0.0060.654 ± 0.016
NPO0.251 ± 0.0050.275 ± 0.0280.391 ± 0.233*0.472 ± 0.198*0.387 ± 0.245*
SimNPO0.264 ± 0.0260.331 ± 0.0550.640 ± 0.0280.495 ± 0.221*0.516 ± 0.235*

WMDP-bio accuracy on LLaDA (base 0.657) after full-model relearning on the forget corpus. Unseen questions: the held-out half after 40 updates on the other half of the WMDP-bio questions. *: the attack also breaks general ability in some seeds, so the low value is not resistance.

The held-out questions recover as well, so we now describe LRCU as lowering recall under the tested conditions, not as removing knowledge. This holds for all tested methods. Within that scope the effect is stable across three prompt formats (0.273 to 0.280). We have added this to Appendix H and to the limitations.


[Q5] Fixed early band

To test a fixed early band, we kept the capped loss fixed and changed only the band.

Band selectionLLaDA bandLLaDA HDream bandDream H
Causal (ours)5–70.659 ± 0.00810–120.618 ± 0.024
Fixed early layers (RMU-style)5–70.659 ± 0.0085–70.582 ± 0.089
Gradient saliency6–80.656 ± 0.00625–270.488 ± 0.033
Random (mean of 5 bands)various0.555various0.563
Deep band19–210.457 ± 0.01320–220.534 ± 0.014
Simplified: fixed 5–7, uniform weights5–70.661 ± 0.0035–70.638 ± 0.008
No cap (GA in our band)5–70.380 ± 0.02210–120.359 ± 0.002

Every row uses the capped loss and the paper training setting, except the last.

On LLaDA the fixed early band is our selected band, and neighbors 4–6 and 7–9 are within noise (0.662 ± 0.003, 0.656 ± 0.011). So a fixed early band is a good default for WMDP-bio.

One caution: this default is known only with hindsight. The causal analysis is what points to the early region, and it is a cheap, forward-only step that also avoids poor ones. For TOFU it selects blocks 23–25 (Fig. 3), so a fixed early band may not transfer. This table is in Appendix J.

Reviewer_p3GpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies parameter-space machine unlearning for Diffusion Language Models (DLMs) and proposes Localized Recall-Capped Unlearning (LRCU). The method first uses Causal-Mediation Localization to select three consecutive Transformer blocks for parameter updates, and then employs a token-wise truncated negative cross-entropy unlearning objective that stops the corresponding unlearning gradient once the cross-entropy of a target token reaches a preset threshold, while using a retain loss to protect other model capabilities. Experiments are conducted on LLaDA and Dream, covering WMDP, RWKU, and TOFU, and demonstrate a favorable forgetting–retention trade-off as well as some training-efficiency benefits.

Critical strengths
  1. The research problem is of practical significance. Knowledge unlearning for DLMs is a relatively new direction, and their bidirectional conditional prediction and iterative denoising mechanisms may impose new requirements on conventional unlearning methods.
  2. The experimental coverage is relatively comprehensive. The paper compares multiple existing unlearning objectives and provides fixed-update-region, loss-function ablation, update-region sensitivity, and relearning experiments, which help analyze the effectiveness and limitations of the method.
Critical weaknesses
  1. The methodological novelty and DLM-specific contributions need to be further clarified. Prior work [1] combines Causal Mediation Analysis, local Transformer-layer updates, and joint forget–retain optimization for LLM unlearning. Another work [2] uses a truncated loss of the form min⁡(CE,c)\min({\mathrm{CE},c}) in unlearning optimization to limit excessively large cross-entropy produced by some samples. Although [2] mainly treats clipping as a sample-level training stabilization technique, whereas LRCU designs it as a token-wise unlearning-gradient stopping mechanism in randomly masked contexts, the two have a direct technical connection. The authors are advised to discuss these prior studies more fully and to clarify which methodological designs or theoretical insights are newly contributed by this paper for the bidirectional prediction and iterative denoising mechanisms of DLMs.

  2. The effectiveness of causal localization and the fairness of experimental comparisons still need to be strengthened. In the WMDP-bio experiments on LLaDA, the 5–7 layers selected by Causal Localization rank only ninth among all 30 candidate consecutive three-layer regions, underperforming the 7–9 layers, while the 4–6 layers also achieve very close results. Therefore, whether the ACE metric can stably identify update regions more suitable for unlearning, and what additional value it provides over simple layer-selection strategies, still require more sufficient validation. In addition, in the main experiments, LRCU updates only about 8%–9% of the parameters and adopts retain CE, a learning rate of 5×10−55\times10^{-5}, and a constant learning rate after warm-up; in contrast, several full-parameter baselines use a learning rate of 2×10−52\times10^{-5} and cosine decay, and the main experiments for NPO, SimNPO, and WGA do not include a retain loss. Although the paper provides controls with fixed update layers and partially unified training conditions, it is still recommended to further control these differences in order to more accurately evaluate the respective contributions of the capped loss and localization.

  3. The robustness of the forgetting effect still warrants further investigation. The relearning experiment in Appendix H shows that, after LRCU unlearning, the accuracy of LLaDA on WMDP-bio recovers from 29.6% to 63.9% after only five full-model fine-tuning runs using the original forget corpus, approaching the original model’s 65.8%. Considering that this experiment re-supplies the original forget data and allows full-model updates, this result cannot directly indicate that the knowledge has not been forgotten, but it does suggest that the current method has limited resistance to targeted relearning. The authors are advised to further evaluate forgetting robustness under different prompts, unseen questions, and different relearning budgets, and to consider comparing with a reference model trained from scratch on retain data, so as to more fully demonstrate the actual forgetting effect.

References

[1] Agrawal, S., & Huang, K.-H. (2025). SHA256 at SemEval-2025 Task 4: Selective Amnesia – Constrained Unlearning for Large Language Models via Knowledge Isolation. SemEval 2025.

[2] Cheng, J., Liu, P., Li, Q., & Zhang, C. (2026). Machine Unlearning under Retain–Forget Entanglement. ICLR 2026.

Questions for authors

Please see my comments on weaknesses.

5 comments
AuthorsOpenReview ↗

Response 1: Novelty and DLM-Specific Contribution

We thank the reviewer for the comments and for pointing us to SHA256 and Cheng et al. We ran new experiments to address them. Unless stated otherwise, they use WMDP-bio with three seeds (mean ± std) on H200 GPUs, and H is the harmonic mean of 1 − forget and MMLU.

[W1] Novelty and DLM-specific contribution

Cheng et al. (2026) was public before our submission and we missed it; we now cite and compare both SHA256 and Cheng et al.

  • Differences from SHA256. SHA256 targets AR models and, like LRCU, uses causal mediation analysis to choose where to update; it then applies unbounded gradient ascent with an adaptive retain weight to early MLP layers. LRCU differs in what it localizes (answer recovery under masked corruption in a DLM, selected per task) and in how far it pushes each token (the cap).
  • Differences from Cheng et al. They target classifiers, update the full model in two stages, and clip the loss only to avoid large per-sample losses. LRCU uses the cap (c = 1) as the forgetting target itself at masked positions, in one stage, on three blocks selected for a DLM.

We ported both methods to DLMs under the same protocol as all other methods:

MethodLLaDA HLLaDA Bio MMLUDream HDream Bio MMLU
LRCU (cap + band)0.657 ± 0.0130.5250.596 ± 0.0270.562
SHA256 (blocks 0–5 MLP)0.619 ± 0.0270.3540.505 ± 0.1320.331
Cheng et al. (two stages, full model)0.473 ± 0.0130.6510.430 ± 0.0160.733
Cheng et al. (stage 1, band)0.648 ± 0.0210.5120.524 ± 0.0850.681

SHA256 forgets but loses adjacent knowledge. The second stage of Cheng et al. restores the forget set together with the adjacent knowledge, so the full method stays near the base model; its first stage alone is below LRCU on both models.

DLM-specific contribution. Objectives moved directly from AR models give a poor trade-off on DLMs: GD lowers MMLU to 0.437 and 0.266 (Table 1), while NPO and SimNPO leave more target knowledge. A DLM predicts each token from bidirectional context over many denoising steps, so a fact must be suppressed in every masked context the sampler can reach. We therefore cap the loss per masked token and weight by mask rate, and the forgetting holds when the decoder changes:

StepsRemaskingAnswer revealedBase forgetLRCU forget ↓Base retainLRCU retain ↑
16low-confidence0%0.8250.239 ± 0.0610.8800.843 ± 0.006
64low-confidence0%0.8540.250 ± 0.0600.8950.877 ± 0.005
128low-confidence0%0.8540.250 ± 0.0600.8950.877 ± 0.005
64random0%0.7970.235 ± 0.0470.8550.830 ± 0.007
64low-confidence25%0.8660.247 ± 0.0660.9100.903 ± 0.010
64low-confidence50%0.8930.256 ± 0.0740.9320.922 ± 0.008

TOFU recall on LLaDA (ROUGE-L, three seeds). Revealing part of the answer raises the base recall but not the recall after LRCU. Appendix L has the full table.

We have added both methods to Table 1 and to Related Work.

AuthorsOpenReview ↗

Response 2: Causal Localization and Fairness

[W2] Causal localization and fairness

To address fairness, we reran every method with the same CE retain loss, optimizer, schedule and tuning budget, in a band scope and a full-model scope.

MethodLLaDA bandLLaDA full modelDream bandDream full model
LRCU (cap)0.657 ± 0.0130.563 ± 0.0200.596 ± 0.0270.542 ± 0.067
GA + retain (GD)0.449 ± 0.0060.546 ± 0.0910.455 ± 0.0120.395 ± 0.001
NPO0.628 ± 0.0380.625 ± 0.0550.527 ± 0.0920.547 ± 0.008
SimNPO0.566 ± 0.0720.533 ± 0.0250.550 ± 0.0350.525 ± 0.002
WGA0.634 ± 0.0250.526 ± 0.0220.554 ± 0.0670.559 ± 0.067
IHL0.652 ± 0.0180.486 ± 0.0410.512 ± 0.0980.497 ± 0.051
RMU0.583 ± 0.008n/a0.450 ± 0.038n/a
SHA2560.449 ± 0.0020.619 ± 0.027†0.476 ± 0.0270.505 ± 0.132†
Cheng et al. (two stages)0.436 ± 0.0370.473 ± 0.0130.494 ± 0.0040.430 ± 0.016
Cheng et al. (stage 1)0.648 ± 0.0210.486 ± 0.0200.524 ± 0.0850.497 ± 0.008
MDU-style anchor0.637 ± 0.0140.592 ± 0.0170.652 ± 0.0070.630 ± 0.020

H under the unified protocol. Band is blocks 5–7 (LLaDA) and 10–12 (Dream). † SHA256 in its original scope (MLPs of blocks 0–5). For reference, LRCU under the paper setting gives 0.659 ± 0.008 (LLaDA) and 0.618 ± 0.024 (Dream).

On LLaDA, cap + band has the highest mean H of all 21 cells; IHL (band) and Cheng et al. (stage 1, band) are within one standard deviation. On Dream, it is above every objective ported from AR models or classifiers. Only the MDU-style anchor is higher, and it gains from our band on both models. Compared with this anchor, LRCU needs no reference model, updates under 10% of the parameters and is 2.4× faster per step on LLaDA.

For localization, we kept the capped loss fixed and changed only the band:

Band selectionLLaDA bandLLaDA HDream bandDream H
Causal (ours)5–70.659 ± 0.00810–120.618 ± 0.024
Fixed early layers (RMU-style)5–70.659 ± 0.0085–70.582 ± 0.089
Gradient saliency6–80.656 ± 0.00625–270.488 ± 0.033
Random (mean of 5 bands)various0.555various0.563
Deep band19–210.457 ± 0.01320–220.534 ± 0.014
Simplified: fixed 5–7, uniform weights5–70.661 ± 0.0035–70.638 ± 0.008
No cap (GA in our band)5–70.380 ± 0.02210–120.359 ± 0.002

Every row uses the capped loss and the paper training setting, except the last.

Blocks 5–7, 4–6 and 7–9 on LLaDA are within noise over three seeds (0.659 ± 0.008, 0.662 ± 0.003, 0.656 ± 0.011), so the single-seed rank is not meaningful. What matters is the early region, which deep bands and gradient saliency on Dream miss; a random band is lower on average. We have added these results to Tables 1 and 3 and Appendices I and J.

AuthorsOpenReview ↗

Response 3: Robustness of Forgetting

[W3] Robustness of forgetting

As suggested, we tested other prompts, unseen questions and several relearning budgets.

MethodOriginal promptInstruction templateRotated options
LRCU0.274 ± 0.0210.280 ± 0.0170.273 ± 0.010
IHL0.267 ± 0.0080.280 ± 0.0160.266 ± 0.015
WGA0.266 ± 0.0080.269 ± 0.0100.271 ± 0.009
RMU0.329 ± 0.0190.424 ± 0.0070.317 ± 0.008

WMDP-bio accuracy on LLaDA after unlearning (base 0.657).

MethodAfter unlearninglr 1e-5, 10 updateslr 1e-5, 40 updateslr 5e-5, 80 updatesUnseen questions
LRCU0.274 ± 0.0210.290 ± 0.0380.628 ± 0.0300.622 ± 0.0180.651 ± 0.012
IHL0.267 ± 0.0080.338 ± 0.1330.521 ± 0.1500.609 ± 0.0180.509 ± 0.228*
WGA0.266 ± 0.0080.279 ± 0.0190.504 ± 0.1450.620 ± 0.0120.647 ± 0.010
RMU0.329 ± 0.0190.474 ± 0.0220.637 ± 0.0060.628 ± 0.0060.654 ± 0.016
NPO0.251 ± 0.0050.275 ± 0.0280.391 ± 0.233*0.472 ± 0.198*0.387 ± 0.245*
SimNPO0.264 ± 0.0260.331 ± 0.0550.640 ± 0.0280.495 ± 0.221*0.516 ± 0.235*

WMDP-bio accuracy on LLaDA (base 0.657) after full-model relearning on the forget corpus. Unseen questions: the held-out half after 40 updates on the other half of the WMDP-bio questions. *: the attack also breaks general ability in some seeds, so the low value is not resistance.

  • Prompts. LRCU forgetting does not depend on the prompt format, while RMU rises to 0.424 under the instruction template.
  • Budgets. With a small budget LRCU stays near chance, similar to WGA and NPO, while RMU returns to 0.474. With a larger budget every method that keeps utility recovers.
  • Unseen questions. The held-out half recovers as well. We therefore describe LRCU as lowering recall under the tested conditions, not as removing knowledge, and we leave relearning resistance to future work. This holds for all tested methods.
  • Reference model. A retain-only reference is not available for WMDP-bio, because retraining an 8B DLM is not feasible. We use the base accuracy (0.657) and chance (0.25) as references.

Appendix H now reports these results. We hope these results clarify the novelty and the robustness of the method, and we would be grateful if the reviewer would take them into account in the final assessment.

Reviewer_Wqt2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LRCU, an unlearning method for masked diffusion language models (LLaDA-8B and Dream-7B). In a first stage, block-wise activation patching on forget probes selects a band of three consecutive blocks (Eq. 7, Fig. 3), and all other parameters are frozen (about 91% of the model). In a second stage, only the band is trained for 500 steps with a capped forget loss, -min(CE, c) with c = 1, which stops pushing a token down once its probability falls below e^{-1}, together with a retain cross-entropy loss and a Min-SNR style weight on the mask rate (Algorithm 1). Proposition 1 bounds verbatim span recall by e^{-c} for one class of samplers. Results are reported with H, the harmonic mean of (1 - forget score) and utility, on WMDP (Table 1), TOFU (Table 2) and RWKU (Table 6), with ablations of the cap, band and weighting (Tables 3, 4, 7), a cost comparison (Table 5) and a relearning test (Table 9).

Critical strengths
  1. Choosing where to update, and updating only a few layers. The main positive point for me is that the method gives an explicit procedure to find which layers to change and then changes only those (8 to 9% of parameters). This makes unlearning cheaper and needs no reference model: with the same loss, training the band instead of the full model cuts step time from 835 to 440 ms on one A100, mostly from the optimizer (233 to 21 ms, Table 5). Restricting updates to a small part of the model is also a natural way to limit damage to the rest of it, which is the central worry in unlearning.

  2. Good experimental results, especially on hazardous knowledge. On WMDP-bio LRCU reaches the best H on both backbones (0.641 vs 0.547 for the best baseline on LLaDA, 0.649 vs 0.572 on Dream, Table 1), with forget accuracy near chance. The capped loss is clearly the component that carries this: with the band and retain loss fixed, removing the cap drops H from 0.675 to 0.391 on LLaDA and from 0.624 to 0.361 on Dream (Table 4), and the cap also helps on an autoregressive Qwen2.5 control (0.692 vs 0.369, Table 8). Given the direction of unlearning for diffusion LMs, which is still new, these results are the main reason I lean toward acceptance.

Critical weaknesses
  1. The comparison with baselines does not isolate what makes LRCU better. In the main tables, the baselines update the full model with different settings: lr 2e-5, cosine decay, 8-bit AdamW, and no retain loss for NPO, SimNPO and WGA, while LRCU uses lr 5e-5, a retain loss and Min-SNR weighting (App. A). As a result several baselines collapse on TOFU (Table 2), while published diffusion-LM results with a retain term do not (MDU, arXiv 2605.18253: NPO on LLaDA keeps 0.726 retain recall). I would like to see the other methods given the same treatment that LRCU gets: properly tuned, with a retain loss, and fine-tuning only a selected subset of layers (or a LoRA of matched size). Table 3 does this for one band, and there the gap to WGA shrinks to 0.007 H on LLaDA (0.668 vs 0.661), though it stays at 0.048 on Dream. Where the published MDU numbers are used (RWKU, Table 6), MDU beats LRCU on both backbones (0.539 vs 0.510 and 0.514 vs 0.485). All results are single runs, and the same LLaDA configuration gives H between 0.641 and 0.675 across Tables 1, 3, 4 and 7, which is larger than several of the margins. RMU, the standard WMDP baseline, is missing, although its default layers 5 to 7 coincide with the band LRCU selects for LLaDA. This matters because it is unclear how much of the gain comes from the cap, from the layer choice, or from the stronger training setup.

  2. The method is more complicated than its results require, and the value of the layer search itself is not yet shown. LRCU combines a separate patching pass over probe items, a per-token cap, a retain loss, Min-SNR weighting, a mask-rate range, a Dream-specific logit shift and several hyperparameters (c, alpha, gamma, band width). The paper's own ablations suggest most of these are not essential: uniform weighting works as well on LLaDA (0.671 vs 0.675, Table 4), a 16-block band costs little (0.641), nearby bands do about as well (Table 7), and the causally selected band ranks 9th of 30 after training (Fig. 4). Recent work also questions whether localization informs unlearning ("Does Localization Inform Unlearning?", EMNLP 2025, arXiv 2505.16252; Hase et al. 2023 for editing), and data-driven parameter selection for unlearning already exists (SalUn, ICLR 2024; WAGLE, NeurIPS 2024). Since the layer selection is what I find most valuable in the paper, I would like it compared directly with simple alternatives (random band, fixed early layers as in RMU, gradient saliency). This matters because a practitioner will adopt the simplest version that works, and the paper should show which parts are needed.

Other issues:

  • Forgetting is shallow: five fine-tuning steps restore WMDP-bio accuracy from 0.296 to 0.639 (Table 9). Relearning-robustness work ("Unlearning or Obfuscating?", ICLR 2025; sharpness-aware unlearning, ICML 2025) is not discussed.
  • Collateral damage on nearby knowledge is large: on LLaDA WMDP-bio, biology-related MMLU falls from 0.664 to 0.476 (Table 1). The retain data is generic WikiText, which does not protect neighbouring knowledge.
  • The capped loss is close to the Inverted Hinge Loss (Cha et al., ICLR 2025, arXiv 2408.06621), which is not cited. Proposition 1 follows directly from span recall requiring each token, gives only 0.37 at c = 1, and excludes LLaDA's confidence-based remasking.
  • The cap c is never swept on the diffusion models, although Table 8 shows it matters (H 0.554 at c = 1 vs 0.692 at c = 4). The Dream TOFU starting model is weak (forget recall 0.614 vs 0.954 in MDU).
  • More broadly, the paper could better motivate why unlearning in diffusion LMs specifically matters, beyond transferring autoregressive unlearning to a new architecture.
Questions for authors
  1. [Suggestion, affects my score] How do NPO, SimNPO, WGA and the published MDU perform when given the same treatment as LRCU: tuned learning rate, retain loss, and fine-tuning only a selected subset of layers (or a matched-size LoRA)?

  2. [Suggestion, affects my score] How does the causal band compare with a random band, RMU's fixed early layers and gradient-saliency selection, all with the same capped loss? This would show whether the layer-selection procedure is needed.

  3. [Clarification] Can you report mean and standard deviation over at least 3 seeds for Tables 1 to 3, and explain why the same LLaDA configuration gives four different H values?

  4. [Clarification] Which components can be removed without loss? A simplified variant (cap plus retain loss on fixed layers, uniform weighting) would help practitioners.

  5. [Clarification] How sensitive are the diffusion-model results to the cap c, and does the forgetting survive light fine-tuning better than the baselines?

6 comments
AuthorsOpenReview ↗

Response 1: Fair Comparison and Seeds

We thank the reviewer for the detailed comments. We ran new experiments to address them. Unless stated otherwise, they use WMDP-bio with three seeds (mean ± std) on H200 GPUs, and H is the harmonic mean of 1 − forget and MMLU.

[W1, Q1, Q3] Fair comparison and seeds

As suggested, we reran every method under one protocol: the same CE retain loss, optimizer, schedule and tuning budget, in a band scope and a full-model scope. We added RMU, IHL, SHA256 and Cheng et al. (2026), and gave the MDU-style anchor the same treatment.

MethodLLaDA bandLLaDA full modelDream bandDream full model
LRCU (cap)0.657 ± 0.0130.563 ± 0.0200.596 ± 0.0270.542 ± 0.067
GA + retain (GD)0.449 ± 0.0060.546 ± 0.0910.455 ± 0.0120.395 ± 0.001
NPO0.628 ± 0.0380.625 ± 0.0550.527 ± 0.0920.547 ± 0.008
SimNPO0.566 ± 0.0720.533 ± 0.0250.550 ± 0.0350.525 ± 0.002
WGA0.634 ± 0.0250.526 ± 0.0220.554 ± 0.0670.559 ± 0.067
IHL0.652 ± 0.0180.486 ± 0.0410.512 ± 0.0980.497 ± 0.051
RMU0.583 ± 0.008n/a0.450 ± 0.038n/a
SHA2560.449 ± 0.0020.619 ± 0.027†0.476 ± 0.0270.505 ± 0.132†
Cheng et al. (two stages)0.436 ± 0.0370.473 ± 0.0130.494 ± 0.0040.430 ± 0.016
Cheng et al. (stage 1)0.648 ± 0.0210.486 ± 0.0200.524 ± 0.0850.497 ± 0.008
MDU-style anchor0.637 ± 0.0140.592 ± 0.0170.652 ± 0.0070.630 ± 0.020

H under the unified protocol. Band is blocks 5–7 (LLaDA) and 10–12 (Dream). † SHA256 in its original scope (MLPs of blocks 0–5). For reference, LRCU under the paper setting gives 0.659 ± 0.008 (LLaDA) and 0.618 ± 0.024 (Dream).

On LLaDA, cap + band has the highest mean H of all 21 cells; IHL (band) and Cheng et al. (stage 1, band) are within one standard deviation. On Dream, it is above every objective ported from AR models or classifiers. Only the MDU-style anchor is higher, and it gains from our band on both models. Our MDU-style anchor masks the entire input, while the original MDU masks only the prompt (Section 4.1). Compared with this anchor, LRCU needs no reference model, updates under 10% of the parameters and is 2.4× faster per step on LLaDA. On the reviewer's other points:

  • Cap vs WGA (Table 3). The gap is now larger than 0.007: 0.657 vs 0.634 on LLaDA, where the cap wins on all three seeds, and 0.596 vs 0.554 on Dream.
  • TOFU. The baselines that collapse on TOFU had no retain loss. With the CE retain loss, full-model NPO and SimNPO no longer collapse on WMDP-bio (MMLU 0.473 to 0.658), and we will update TOFU under the same protocol in the final version.
  • RWKU. We agree: MDU (τ = 1) has a higher H than LRCU on RWKU on both models, as the paper states (Table 6). LRCU reaches its score with a much smaller footprint: it edits three blocks (8.2–9.2% of the parameters) and needs no reference model, which makes the update less intrusive and cheaper per step than full-model training (Table 5).

On Q3, the four H values for the same setting (Tables 1, 3, 4 and 7) came from different runs: Tables 1, 4 and 7 are single-seed runs and Table 3 is a three-seed mean. Tables 1, 3 and 4 now report three-seed means on 57 MMLU subjects:

ModelTable 1Table 3Table 4Three seeds (H200)
LLaDA0.6410.6680.6750.659 ± 0.008
Dream0.6490.6340.6240.618 ± 0.024

We have added the new rows to Tables 1 and 3, with every method in both scopes in Appendix I, and now report three-seed mean ± std in Tables 1, 3 and 4.

AuthorsOpenReview ↗

Response 2: Band Selection, Simplified Version, and Cap Sensitivity

[W2, Q2, Q4] Value of band selection, and a simplified version

To test the value of band selection, we kept the capped loss fixed and changed only the band, and we replaced each component with a simpler one.

Band selectionLLaDA bandLLaDA HDream bandDream H
Causal (ours)5–70.659 ± 0.00810–120.618 ± 0.024
Fixed early layers (RMU-style)5–70.659 ± 0.0085–70.582 ± 0.089
Gradient saliency6–80.656 ± 0.00625–270.488 ± 0.033
Random (mean of 5 bands)various0.555various0.563
Deep band19–210.457 ± 0.01320–220.534 ± 0.014
16 blocks0–150.634 ± 0.0450–150.555 ± 0.091
Simplified: fixed 5–7, uniform weights5–70.661 ± 0.0035–70.638 ± 0.008
No cap (GA in our band)5–70.380 ± 0.02210–120.359 ± 0.002

Every row uses the capped loss and the paper training setting, except the last.

For band selection (Q2), deep bands and gradient saliency on Dream lose clearly, and a random band is lower on average. Blocks 5–7, 4–6 and 7–9 are within noise over three seeds (0.659 ± 0.008, 0.662 ± 0.003, 0.656 ± 0.011), so what matters is the early region, not the exact band.

We therefore see the causal analysis as a cheap, forward-only way to avoid poor regions rather than to find the single best band. For TOFU it selects a different region (blocks 23–25, Fig. 3).

For the simplified version (Q4), we replaced each component with a simpler one:

  • Fixed band with uniform weights. The version the reviewer describes works well on this task: it matches the full method on LLaDA and is higher on Dream, so we now present it as a practitioner recipe for WMDP-bio. One caution is that it is defined with hindsight. Fixed blocks 5–7 are the region that the causal analysis selected for WMDP-bio, so the recipe may be specific to this benchmark and may not replace the selection step on other tasks.
  • Wider band. The 16-block band is worse than the 3-block band in mean H and seed variance, and it trains about five times as many parameters.
  • No cap. This is the only simplification that breaks the method.

The full table is now in Appendix J.


[Q5, I4] Sensitivity to c, and durability

We swept c on both DLMs.

cLLaDA HDream H
0.50.662 ± 0.0020.588 ± 0.027
10.659 ± 0.0080.618 ± 0.024
20.652 ± 0.0190.595 ± 0.048
40.622 ± 0.0420.553 ± 0.087
60.615 ± 0.0480.452 ± 0.091
80.593 ± 0.0730.533 ± 0.105
No cap0.380 ± 0.0220.359 ± 0.002

Unlike the AR case, H is flat for c from 0.5 to 2 on LLaDA, and c = 1 is best on Dream. Without a cap both models collapse (Appendix K).

For durability, we ran relearning attacks on LLaDA. LRCU stays near chance with a small budget (0.290), like WGA and NPO, while RMU returns to 0.474. With a larger budget every method that keeps utility recovers, so we do not claim stronger durability.

AuthorsOpenReview ↗

Response 3: Relearning and Adjacent Knowledge

[I1] Shallow forgetting

Appendix H already reports this, and we now extend it to more methods and budgets.

MethodAfter unlearninglr 1e-5, 10 updateslr 1e-5, 40 updateslr 5e-5, 80 updatesUnseen questions
LRCU0.274 ± 0.0210.290 ± 0.0380.628 ± 0.0300.622 ± 0.0180.651 ± 0.012
IHL0.267 ± 0.0080.338 ± 0.1330.521 ± 0.1500.609 ± 0.0180.509 ± 0.228*
WGA0.266 ± 0.0080.279 ± 0.0190.504 ± 0.1450.620 ± 0.0120.647 ± 0.010
RMU0.329 ± 0.0190.474 ± 0.0220.637 ± 0.0060.628 ± 0.0060.654 ± 0.016
NPO0.251 ± 0.0050.275 ± 0.0280.391 ± 0.233*0.472 ± 0.198*0.387 ± 0.245*
SimNPO0.264 ± 0.0260.331 ± 0.0550.640 ± 0.0280.495 ± 0.221*0.516 ± 0.235*

WMDP-bio accuracy on LLaDA (base 0.657) after full-model relearning on the forget corpus. Unseen questions: the held-out half after 40 updates on the other half of the WMDP-bio questions. *: the attack also breaks general ability in some seeds, so the low value is not resistance.

Every method that keeps utility recovers, so this is a shared limitation, which we now state. It agrees with findings on AR models (Hu et al., 2025; Łucki et al., 2024; Deeb & Roger, 2024), which we now cite in Section 2 and Appendix H, and we name resistance to relearning (Fan et al., 2025) as future work.


[I2] Bio MMLU

The drop comes from the general WikiText retain set, which does not cover knowledge adjacent to the forget set. We state this as a limitation and leave a retain set that protects adjacent knowledge to future work.

With three seeds the drop is smaller than in the submission (0.664 → 0.580 ± 0.015 on LLaDA in the paper setting). Under the unified protocol, cap + band keeps the highest mean Bio MMLU among the nine LLaDA cells that bring forget accuracy below 0.30:

Method (LLaDA)Forget ↓Bio MMLU ↑
Base0.6580.664
LRCU (cap + band)0.264 ± 0.0080.525 ± 0.065
Cheng et al. (stage 1, band)0.282 ± 0.0190.512 ± 0.048
IHL (band)0.261 ± 0.0170.500 ± 0.049
MDU-style (band)0.274 ± 0.0150.489 ± 0.084
WGA (band)0.265 ± 0.0070.464 ± 0.070
NPO (band)0.256 ± 0.0080.460 ± 0.081
SimNPO (band)0.257 ± 0.0130.390 ± 0.107
GD (full model)0.248 ± 0.0080.376 ± 0.119
SHA256 (blocks 0–5 MLP)0.256 ± 0.0150.354 ± 0.062

Cheng et al. (stage 1) and IHL are within one standard deviation of LRCU. Table 1 now reports Bio MMLU for all methods.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Summary of Changes in the Updated Draft

We sincerely thank all reviewers for their careful and constructive feedback. We have uploaded a revised draft (new content in blue) and have run every experiment requested. The main additions are:

  • Fair comparison under one protocol (Sec. 4.2, Table 1; Appendix I). Every method now uses the same retain loss, optimizer, schedule and tuning budget, with three seeds on H200 GPUs. On WMDP-bio, LRCU has the highest mean H of all methods and update scopes on LLaDA.
  • New baselines (Sec. 2, Table 1; Appendix I). We ported RMU, IHL, SHA256 and Cheng et al. (2026) to DLMs. LRCU has the highest mean H among them on both models for WMDP-bio, and on Dream for WMDP-cyber.
  • Where the gain comes from (Sec. 4.3–4.4, Tables 3–4; Appendix J–K). With the capped loss fixed, the causal band is above random and deep bands on both models, and removing the cap lowers H sharply.
  • Efficiency (Sec. 4.4, Table 5). LRCU updates under 10% of the parameters and needs no reference model. It is up to 2.4× faster per step than the full-model baselines.
  • Robustness of forgetting (Appendix H, Appendix L). Forgetting holds across prompt templates, option orders, samplers, step counts and partial answer reveal.
  • Related work and clarity (Sec. 1, Sec. 2, Sec. 4.1). We discuss SHA256, Cheng et al. (2026) and RMU, restate the DLM-specific motivation, and specify the evaluation protocol and forget data.

We address each reviewer's points in the individual responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.