- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Masked Diffusion Unlearning (MDU), a machine forgetting method for masked diffusion language models. This method achieves forgetting of target knowledge by minimizing the KL divergence between the model's conditional prediction distribution and the frozen prediction distribution under prompt-masked conditions, and adjusts the smoothness of the target distribution using the parameter . The authors also adapt several existing unlearning methods to masked diffusion language models and evaluate them on LLaDA and Dream using TOFU, RWKU, relearning attack, and safety unlearning. Experimental results show that MDU achieves good forgetting performance in multiple settings while retaining some model capabilities. The paper also analyzes the method's behavior from the perspectives of conditional mutual information and token-level KL.
Critical strengths
- MDU uses the predictions of the frozen model under prompt-masked conditions as the unlearning target, which naturally aligns with the modeling mechanism of masked diffusion language models. The method is simple, easy to implement, and provides a flexible forgetting control mechanism through the parameter .
- The paper evaluates the LLaDA and Dream models using TOFU and RWKU, with additional relearning-attack and safety-unlearning experiments on LLaDA, and adapts several existing unlearning methods to MDLM. Extensive experimental setups provide initial support for the practical value of the method.
Critical weaknesses
- Under the main settings of LLaDA–TOFU (tau=0.5), the forget-set output of MDU exhibited garbled text and empty answers, and pseudo-perplexity also increased significantly. Therefore, the lower forgetting scores may stem from output degradation rather than genuine unlearning. Furthermore, the tau=1 used in the main mechanism analysis is inconsistent with the main experimental settings.
- The paper claims to follow the official TOFU protocol but fails to report Forget Quality, Model Utility, or retain-only references. The lack of these metrics, especially considering the degradation of some forget-set outputs, makes it difficult to distinguish between genuine unlearning and over-forgetting.
Questions for authors
- Under non-degenerate settings of (tau=0.75) or 1, does MDU still maintain its main performance advantage? How was tau chosen in each experiment?
- Could you supplement TOFU's Forget Quality and Model Utility and compare them with the retain-only model?
- What learning rate, training epochs, and retain weight were used for each method? Were the baselines tuned with the same hyperparameters?
- The paper has proposed an autoregressive analogue for MDU. Can this method achieve similar results on AR models? If so, what justifies the necessity of the MDLM-specific design?
- LLaDA's MMLU score on RWKU is significantly lower than publicly reported. Could the authors explain the evaluation protocol used and the reason for this difference?
5 comments
Official Comment by Authors (2/2)
[Q3] Hyperparameter selection and baseline tuning
As detailed in Appendix D.4 (Table 11), we searched learning rate and unlearning epochs for MDU and the compared baselines, considering forgetting and retention when selecting configurations. Learning rates ranged from 5 × 10⁻⁶ to 3 × 10⁻⁵, and epochs were searched over {1, 3, 5, 7, 9, 11, 13, 15}. The retain reconstruction coefficient was held at λ=1 for all TOFU methods using that term. The selected configurations are listed below (learning rate / unlearning epochs).
| Method | LLaDA–TOFU | Dream–TOFU | LLaDA–RWKU | Dream–RWKU |
|---|---|---|---|---|
| GA | 5e-6 / 9 | 5e-6 / 11 | 1e-5 / 1 | 5e-6 / 1 |
| GD | 1e-5 / 5 | 8e-6 / 9 | — | — |
| NPO | 1e-5 / 9 | 8e-6 / 9 | 1e-5 / 1 | 1e-5 / 3 |
| SimNPO | 1e-5 / 9 | 8e-6 / 13 | 1e-5 / 1 | 1e-5 / 3 |
| WGA | 1e-5 / 9 | 8e-6 / 9 | 1e-5 / 1 | 5e-6 / 3 |
| DPO | 1e-5 / 9 | 8e-6 / 9 | 1e-5 / 5 | 1e-5 / 3 |
| ARL | 3e-5 / 3 | 3e-5 / 3 | 1e-5 / 5 | 1e-5 / 5 |
| MDU | 1e-5 / 9 | 8e-6 / 9 | 1e-5 / 9 | 1e-5 / 3 |
[Q4] Applicability to autoregressive models
We have retained the AR formulation in Appendix C and added the AR unlearning comparison to Appendix E.5 (Table 18), evaluating TOFU-fine-tuned Llama-2-7B-Chat alongside GA, GD, and NPO.
| Method | Forget rL ↓ | Forget p ↓ | Retain rL ↑ | Retain p ↑ | RA rL ↑ | RA p ↑ | WF rL ↑ | WF p ↑ |
|---|---|---|---|---|---|---|---|---|
| GA | 0.342 | 0.203 | 0.566 | 0.331 | 0.909 | 0.044 | 0.907 | 0.032 |
| GD | 0.318 | 0.347 | 0.655 | 0.843 | 0.890 | 0.018 | 0.854 | 0.004 |
| NPO | 0.319 | 0.218 | 0.640 | 0.853 | 0.901 | 0.042 | 0.908 | 0.044 |
| MDU-AR | 0.396 | 0.497 | 0.585 | 0.846 | 0.838 | 0.028 | 0.839 | 0.058 |
The AR analogue maintains retention comparable to GD and NPO but achieves weaker forgetting. AR conditional and null predictions share a causal prefix, whereas MDU aligns predictions across partially masked response states with potentially bidirectional context. This comparison motivates a target distribution tailored to MDLM masking states.
[Q5] RWKU MMLU evaluation protocol and corrected results
We acknowledge the discrepancy between the initially reported LLaDA MMLU accuracy and the released model’s published performance. We found that the MMLU prompt format and the procedure used to extract a multiple-choice answer from generated text understated accuracy in our original evaluation. We corrected the evaluation procedure and reevaluated the base and unlearned checkpoints consistently. The corrected LLaDA results, now reported in Table 3 of the revised manuscript, are:
| Method | Previous MMLU | Reevaluated MMLU ↑ |
|---|---|---|
| Base | 0.395 | 0.662 |
| GA | 0.388 | 0.641 |
| NPO | 0.386 | 0.647 |
| SimNPO | 0.390 | 0.643 |
| WGA | 0.409 | 0.691 |
| DPO | 0.389 | 0.655 |
| ARL | 0.393 | 0.643 |
| MDU | 0.418 | 0.712 |
The same correction was applied to the Dream–RWKU evaluation (including a corrected Base result of 0.760, shown in revised Table 3):
| Method | Previous MMLU | Reevaluated MMLU ↑ |
|---|---|---|
| Base | 0.750 | 0.760 |
| GA | 0.632 | 0.701 |
| NPO | 0.621 | 0.670 |
| SimNPO | 0.685 | 0.679 |
| WGA | 0.236 | 0.478 |
| DPO | 0.642 | 0.742 |
| ARL | 0.635 | 0.711 |
| MDU | 0.658 | 0.750 |
The corrected LLaDA Base accuracy is consistent with published model performance. On both LLaDA and Dream, MDU achieves the highest reevaluated MMLU accuracy among the compared unlearning methods. This resolves the discrepancy noted in the review and updates the utility comparison accordingly.
Thank you again for the detailed questions. They helped us improve the evaluation and correct the reported utility results.
Thank you again for your careful review. Your feedback helped us improve the evaluation protocol and examine the forgetting–utility trade-off more thoroughly. We summarize the main revisions below.
-
Output quality and temperature. We added a full temperature sweep and categorized 400 forget-set responses at each setting. The analysis distinguishes coherent incorrect answers from knowledge disclosures and degenerate outputs, and clarifies why Table 4 uses τ=1.
-
Forget Quality and Model Utility. We evaluated TOFU Forget Quality (FQ) and Model Utility (MU) against a retain-only reference. MDU achieved the highest MU among the evaluated unlearning methods, while the FQ analysis also identifies a remaining distributional difference from the retain-only reference.
-
Hyperparameter and AR evaluations. We reported the selected learning rates and training epochs for all methods, evaluated the retain reconstruction weight, and added an autoregressive analogue comparison.
-
Corrected MMLU evaluation. We corrected the MMLU prompting and answer-extraction procedure and reevaluated all checkpoints consistently. MDU achieved MMLU accuracies of 0.712 on LLaDA and 0.750 on Dream, the highest among the compared unlearning methods.
As the discussion period draws to a close, we would be grateful to hear whether these revisions have addressed your main concerns.
Official Comment by Authors (1/2)
Thank you for your careful review and constructive feedback. We respond to [W#] and [Q#] below. The table, section, and appendix numbers below refer to the revised manuscript.
[W1, Q1] Output degeneration and performance at higher τ
Output categories.
We evaluated the LLaDA–TOFU temperature sweep and categorized 400 generated forget-set responses per setting to test whether the forgetting advantage survives when output degeneration is reduced.
Performance at higher temperatures.
As τ increases, corrupted and empty responses become less frequent, while responses disclosing target knowledge become more common. This confirms that temperature affects the balance between answer suppression and generation quality.
We have added the full sweep and output categorization to Appendix E.6 (Tables 19–20). At τ=0.75 and τ=1, Forget ROUGE-L is lower than the reported baselines, but Forget answer probability rises and Retain ROUGE-L declines; the result is a trade-off rather than uniform dominance.
We chose τ=0.5 for LLaDA–TOFU and τ=0.25 for Dream–TOFU and the two RWKU settings based on the model- and benchmark-specific forgetting–retention trends in Figure 5. Table 4 uses τ=1 to analyze the original prompt-masked anchor without temperature scaling.
[W2, Q2] TOFU Forget Quality, Model Utility, and retain-only reference
We trained a retain-only reference from the released LLaDA checkpoint using only retain90 training examples (excluding forget10), and evaluated Forget Quality (FQ) and Model Utility (MU) on LLaDA–TOFU forget10. For FQ, Truth Ratios were computed using the reconstruction-based answer-probability estimator in Eq. (8), and their distributions were compared with the retain-only reference using a two-sample Kolmogorov–Smirnov test. MU follows the TOFU aggregation protocol.
MDU achieves the highest MU (0.658), with Retain ROUGE-L of 0.853 close to the retain-only reference (0.872). It also achieves the lowest Forget ROUGE-L (0.098) and answer probability (0.001). Its FQ is a KS-test p-value of 8.25 × 10⁻⁵ (NPO: 2.31 × 10⁻⁴); both values are below 0.05, indicating that their Truth Ratio distributions still differ from the retain-only reference. Overall, MDU combines strong suppression of target-answer reproduction with high model utility.