- Rating
- 4
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper addresses a limitation of Masked Diffusion Unlearning (MDU) in that it can retain certain information despite being trained to forget them. MDU trains the model to match a frozen anchor with the prompt masked, with which the authors identify two problems. The anchor can still predict the forget answer from visible answer tokens, and the anchor-last KL loss has vanishing logit gradients when the model is highly confident in its predictions. The paper thus proposes Enhanced Masked Diffusion Unlearning (EMDU), which uses anchor-first KL and fully masks the answer span in the anchor. Experiments show better forgetting and retention than MDU on TOFU, with additional improvements on RWKU.
Critical strengths
-
The explanation of why KL direction matters for unlearning is convincing. Both objectives have the same minimizing distribution, but their logit gradients differ near one-hot predictions, which are common in LLaDA after TOFU fine-tuning. In Table 1, changing only the KL direction to anchor-first leaves far fewer high-confidence forget-token predictions after the same training run.
-
EMDU is better than MDU at both forgetting and answering retained questions on the two TOFU models (Table 3). Although either change alone achieves low Forget rL, combining anchor-first KL and full answer-span masking gives substantially higher Retain scores on both models (Table 2). Appendix E.1 also checks the comparison across additional MDU configurations and retain weights.
Critical weaknesses
None
Questions for authors
None
We thank the reviewer for the careful reading and the positive assessment. We are glad that the reviewer found our analysis of the KL direction convincing, and that our experiments on TOFU and RWKU support it. We also thank the reviewer for acknowledging EMDU's improvements over MDU in both forgetting and retained utility.