acceptodds
Under review as a conference paper at ICLR 2027

Machine Unlearning for Masked Diffusion Language Models

Abstract

Recent masked diffusion language models (MDLMs), such as LLaDA and Dream, have achieved performance comparable to autoregressive large language models. Unlike autoregressive models, which generate text sequentially, MDLMs generate text by iteratively denoising masked positions in parallel. During fine-tuning, MDLMs learn to recover responses from masked response states conditioned on a prompt, thereby shifting their predictions from a prompt-masked unconditional distribution toward a prompt-conditional distribution. Given this distinct generative and fine-tuning mechanism, specifying the target predictions at masked response positions is an important design consideration for MDLM unlearning. In this paper, we propose **Masked Diffusion Unlearning (MDU)**, an unlearning framework for MDLMs, by revisiting the process of learning specific knowledge in terms of diffusion. Specifically, MDU minimizes a forward KL divergence from the prompt-conditional prediction to a frozen prompt-masked unconditional anchor at every masked response position, with a temperature scaling parameter to control the privacy-utility trade-off. Our empirical results on standard benchmarks and MDLM backbones show that MDU achieves strong unlearning performance compared with the evaluated unlearning baselines.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Machine unlearning, Masked diffusion language models, Large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_n31wOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Masked Diffusion Unlearning (MDU), a machine forgetting method for masked diffusion language models. This method achieves forgetting of target knowledge by minimizing the KL divergence between the model's conditional prediction distribution and the frozen prediction distribution under prompt-masked conditions, and adjusts the smoothness of the target distribution using the parameter τ\tau. The authors also adapt several existing unlearning methods to masked diffusion language models and evaluate them on LLaDA and Dream using TOFU, RWKU, relearning attack, and safety unlearning. Experimental results show that MDU achieves good forgetting performance in multiple settings while retaining some model capabilities. The paper also analyzes the method's behavior from the perspectives of conditional mutual information and token-level KL.

Critical strengths
  1. MDU uses the predictions of the frozen model under prompt-masked conditions as the unlearning target, which naturally aligns with the modeling mechanism of masked diffusion language models. The method is simple, easy to implement, and provides a flexible forgetting control mechanism through the parameter τ\tau.
  2. The paper evaluates the LLaDA and Dream models using TOFU and RWKU, with additional relearning-attack and safety-unlearning experiments on LLaDA, and adapts several existing unlearning methods to MDLM. Extensive experimental setups provide initial support for the practical value of the method.
Critical weaknesses
  1. Under the main settings of LLaDA–TOFU (tau=0.5), the forget-set output of MDU exhibited garbled text and empty answers, and pseudo-perplexity also increased significantly. Therefore, the lower forgetting scores may stem from output degradation rather than genuine unlearning. Furthermore, the tau=1 used in the main mechanism analysis is inconsistent with the main experimental settings.
  2. The paper claims to follow the official TOFU protocol but fails to report Forget Quality, Model Utility, or retain-only references. The lack of these metrics, especially considering the degradation of some forget-set outputs, makes it difficult to distinguish between genuine unlearning and over-forgetting.
Questions for authors
  1. Under non-degenerate settings of (tau=0.75) or 1, does MDU still maintain its main performance advantage? How was tau chosen in each experiment?
  2. Could you supplement TOFU's Forget Quality and Model Utility and compare them with the retain-only model?
  3. What learning rate, training epochs, and retain weight were used for each method? Were the baselines tuned with the same hyperparameters?
  4. The paper has proposed an autoregressive analogue for MDU. Can this method achieve similar results on AR models? If so, what justifies the necessity of the MDLM-specific design?
  5. LLaDA's MMLU score on RWKU is significantly lower than publicly reported. Could the authors explain the evaluation protocol used and the reason for this difference?
5 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

Thank you for your careful review and constructive feedback. We respond to [W#] and [Q#] below. The table, section, and appendix numbers below refer to the revised manuscript.


[W1, Q1] Output degeneration and performance at higher τ

Output categories.

We evaluated the LLaDA–TOFU temperature sweep and categorized 400 generated forget-set responses per setting to test whether the forgetting advantage survives when output degeneration is reduced.

Output categoryτ=0.00τ=0.25τ=0.50τ=0.75τ=1.00
Coherent incorrect answer84139271319348
Knowledge disclosure914182429
Corrupted output239193763917
Empty response685435186
Total400400400400400

Performance at higher temperatures.

As τ increases, corrupted and empty responses become less frequent, while responses disclosing target knowledge become more common. This confirms that temperature affects the balance between answer suppression and generation quality.

We have added the full sweep and output categorization to Appendix E.6 (Tables 19–20). At τ=0.75 and τ=1, Forget ROUGE-L is lower than the reported baselines, but Forget answer probability rises and Retain ROUGE-L declines; the result is a trade-off rather than uniform dominance.

τForget rL ↓Forget p ↓Retain rL ↑Retain p ↑RA rL ↑RA p ↑WF rL ↑WF p ↑
0.000.0690.0000.8680.3810.6290.0930.8480.193
0.250.1350.0000.8570.3920.6160.1160.8420.204
0.50 (main)0.0980.0010.8530.4470.6450.1330.8420.205
0.750.0780.0400.6840.5350.6120.1550.8270.233
1.000.0340.0740.5110.4850.5680.1100.7770.187

We chose τ=0.5 for LLaDA–TOFU and τ=0.25 for Dream–TOFU and the two RWKU settings based on the model- and benchmark-specific forgetting–retention trends in Figure 5. Table 4 uses τ=1 to analyze the original prompt-masked anchor without temperature scaling.


[W2, Q2] TOFU Forget Quality, Model Utility, and retain-only reference

We trained a retain-only reference from the released LLaDA checkpoint using only retain90 training examples (excluding forget10), and evaluated Forget Quality (FQ) and Model Utility (MU) on LLaDA–TOFU forget10. For FQ, Truth Ratios were computed using the reconstruction-based answer-probability estimator in Eq. (8), and their distributions were compared with the retain-only reference using a two-sample Kolmogorov–Smirnov test. MU follows the TOFU aggregation protocol.

MethodFQ ↑MU ↑Forget rL ↓Forget p ↓Retain rL ↑Retain p ↑
Base SFT8.22×10⁻¹¹—0.8840.3800.8700.330
Retain-only1.0000.6130.4080.0160.8720.335
GA3.12×10⁻²⁰0.3030.3480.0200.3610.018
GD6.65×10⁻¹⁰0.4620.5330.0610.6760.169
NPO2.31×10⁻⁴0.4670.3720.0090.7260.138
SimNPO5.92×10⁻⁵0.5480.4850.0360.8040.273
WGA2.27×10⁻¹⁸0.5130.1220.0100.6960.304
DPO5.42×10⁻¹⁵0.5790.4790.1680.7960.401
ARL6.22×10⁻¹²0.3470.1400.0380.2620.076
MDU8.25×10⁻⁵0.6580.0980.0010.8530.447

MDU achieves the highest MU (0.658), with Retain ROUGE-L of 0.853 close to the retain-only reference (0.872). It also achieves the lowest Forget ROUGE-L (0.098) and answer probability (0.001). Its FQ is a KS-test p-value of 8.25 × 10⁻⁵ (NPO: 2.31 × 10⁻⁴); both values are below 0.05, indicating that their Truth Ratio distributions still differ from the retain-only reference. Overall, MDU combines strong suppression of target-answer reproduction with high model utility.


AuthorsOpenReview ↗

Official Comment by Authors (2/2)

[Q3] Hyperparameter selection and baseline tuning

As detailed in Appendix D.4 (Table 11), we searched learning rate and unlearning epochs for MDU and the compared baselines, considering forgetting and retention when selecting configurations. Learning rates ranged from 5 × 10⁻⁶ to 3 × 10⁻⁵, and epochs were searched over {1, 3, 5, 7, 9, 11, 13, 15}. The retain reconstruction coefficient was held at λ=1 for all TOFU methods using that term. The selected configurations are listed below (learning rate / unlearning epochs).

MethodLLaDA–TOFUDream–TOFULLaDA–RWKUDream–RWKU
GA5e-6 / 95e-6 / 111e-5 / 15e-6 / 1
GD1e-5 / 58e-6 / 9——
NPO1e-5 / 98e-6 / 91e-5 / 11e-5 / 3
SimNPO1e-5 / 98e-6 / 131e-5 / 11e-5 / 3
WGA1e-5 / 98e-6 / 91e-5 / 15e-6 / 3
DPO1e-5 / 98e-6 / 91e-5 / 51e-5 / 3
ARL3e-5 / 33e-5 / 31e-5 / 51e-5 / 5
MDU1e-5 / 98e-6 / 91e-5 / 91e-5 / 3

[Q4] Applicability to autoregressive models

We have retained the AR formulation in Appendix C and added the AR unlearning comparison to Appendix E.5 (Table 18), evaluating TOFU-fine-tuned Llama-2-7B-Chat alongside GA, GD, and NPO.

MethodForget rL ↓Forget p ↓Retain rL ↑Retain p ↑RA rL ↑RA p ↑WF rL ↑WF p ↑
GA0.3420.2030.5660.3310.9090.0440.9070.032
GD0.3180.3470.6550.8430.8900.0180.8540.004
NPO0.3190.2180.6400.8530.9010.0420.9080.044
MDU-AR0.3960.4970.5850.8460.8380.0280.8390.058

The AR analogue maintains retention comparable to GD and NPO but achieves weaker forgetting. AR conditional and null predictions share a causal prefix, whereas MDU aligns predictions across partially masked response states with potentially bidirectional context. This comparison motivates a target distribution tailored to MDLM masking states.


[Q5] RWKU MMLU evaluation protocol and corrected results

We acknowledge the discrepancy between the initially reported LLaDA MMLU accuracy and the released model’s published performance. We found that the MMLU prompt format and the procedure used to extract a multiple-choice answer from generated text understated accuracy in our original evaluation. We corrected the evaluation procedure and reevaluated the base and unlearned checkpoints consistently. The corrected LLaDA results, now reported in Table 3 of the revised manuscript, are:

MethodPrevious MMLUReevaluated MMLU ↑
Base0.3950.662
GA0.3880.641
NPO0.3860.647
SimNPO0.3900.643
WGA0.4090.691
DPO0.3890.655
ARL0.3930.643
MDU0.4180.712

The same correction was applied to the Dream–RWKU evaluation (including a corrected Base result of 0.760, shown in revised Table 3):

MethodPrevious MMLUReevaluated MMLU ↑
Base0.7500.760
GA0.6320.701
NPO0.6210.670
SimNPO0.6850.679
WGA0.2360.478
DPO0.6420.742
ARL0.6350.711
MDU0.6580.750

The corrected LLaDA Base accuracy is consistent with published model performance. On both LLaDA and Dream, MDU achieves the highest reevaluated MMLU accuracy among the compared unlearning methods. This resolves the discrepancy noted in the review and updates the utility comparison accordingly.

Thank you again for the detailed questions. They helped us improve the evaluation and correct the reported utility results.

Thank you again for your careful review. Your feedback helped us improve the evaluation protocol and examine the forgetting–utility trade-off more thoroughly. We summarize the main revisions below.

  1. Output quality and temperature. We added a full temperature sweep and categorized 400 forget-set responses at each setting. The analysis distinguishes coherent incorrect answers from knowledge disclosures and degenerate outputs, and clarifies why Table 4 uses τ=1.

  2. Forget Quality and Model Utility. We evaluated TOFU Forget Quality (FQ) and Model Utility (MU) against a retain-only reference. MDU achieved the highest MU among the evaluated unlearning methods, while the FQ analysis also identifies a remaining distributional difference from the retain-only reference.

  3. Hyperparameter and AR evaluations. We reported the selected learning rates and training epochs for all methods, evaluated the retain reconstruction weight, and added an autoregressive analogue comparison.

  4. Corrected MMLU evaluation. We corrected the MMLU prompting and answer-extraction procedure and reevaluated all checkpoints consistently. MDU achieved MMLU accuracies of 0.712 on LLaDA and 0.750 on Dream, the highest among the compared unlearning methods.

As the discussion period draws to a close, we would be grateful to hear whether these revisions have addressed your main concerns.

Reviewer_1ZrNOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a method for machine unlearning in masked diffusion models — Masked Diffusion Unlearning (MDU). As the main objective they propose minimizing the KL-divergence between conditional distribution and unconditional distribution of frozen original model. They scale the target distribution by parameter \tau to allow trade-off between forget and utility. The method is compared against existing baselines on TOFU and RWKU with LLaDA-8B and Dream-7B, and the authors additionally evaluate relearning robustness and jailbreak-safety unlearning.

Critical strengths
  1. Simple objective for unlearning. It has just a single interpretable parameter \tau.
  2. Broad comparison with baselines.
Critical weaknesses
  1. The proposed objective is fairly straightforward and closely related ideas have already been studied in prior work: Who’s Harry Potter [1], its causal-intervention follow-up [2], and ELM [3]. None of them are discussed in the paper. Moreover, the \tau = 0 case is exactly maximum-entropy unlearning [4].
  2. There is no ablation of a fixed vs. moving target for the proposed objective. The retain loss weight \lambda is not ablated either.
  3. It is not explained how to choose \tau. And it varies across different benchmarks.
  4. An AR analogue of the objective is proposed but not evaluated.

[1] Eldan and Russinovich. “Who's Harry Potter? Approximate Unlearning in LLMs”. 2023

[2] Liu et al. “Revisiting Who's Harry Potter: Towards Targeted Unlearning from a Causal Intervention Perspective”. 2024

[3] Gandikota et al. “Erasing Conceptual Knowledge from Language Models”. 2024

[4] Yuan et al. “A Closer Look at Machine Unlearning for Large Language Models”. 2024

Questions for authors
  1. Is there any rule you converged on for selecting \tau?
  2. Could you report the performance of the proposed objective on AR models and compare it with MDLM results?
3 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

Thank you for your careful review and constructive feedback. We respond to [W#] and [Q#] below. The table, section, and appendix numbers below refer to the revised manuscript. The references listed at the end are also cited in the revised manuscript.


[W1] Relationship to prior unlearning methods and Maximum Entropy

We agree that modifying model inputs or constructing alternative prediction targets is not, by itself, a novel idea. We have clarified the relationship to the relevant methods in Appendix A:

  • Who’s Harry Potter? (WHP) [1]: Constructs alternative training targets using model-derived predictions and generic replacements.
  • Causal-intervention unlearning [2]: Models target knowledge as a confounder and approaches forgetting through deconfounding.
  • ELM [3]: Uses model-derived concept information to construct distribution-matching targets.

MDU instead specifies a frozen prompt-masked reference evaluated at the same partially masked, bidirectional response state and aligns token distributions using forward KL. Its contribution is this MDLM-specific target construction and learning objective, not the general idea of altering inputs or predictions.

The relationship to Maximum Entropy [4] is exact at τ=0, when the anchor becomes uniform and the forget objective reduces to entropy maximization. For τ>0, MDU instead uses a nonuniform prompt-masked reference that retains the predictive structure available from the observed response context. Thus, Maximum Entropy arises as a limiting case of the objective, whereas the general formulation offers context-dependent reference distributions controlled by τ.


[W2] Frozen versus moving anchor and retain-weight ablations

Frozen versus moving anchor.

We have added the frozen/moving-anchor and retain-weight ablations to Appendix E.7 (Table 22), using LLaDA–TOFU. We first compare a frozen anchor with a moving, stop-gradient anchor.

SettingForget rL ↓Forget p ↓Retain rL ↑Retain p ↑RA rL ↑RA p ↑WF rL ↑WF p ↑
Frozen (MDU)0.0980.0010.8530.4470.6450.1330.8420.205
Moving (stop-grad.)0.1430.0050.8610.5010.6420.1350.8450.210

The frozen anchor produces stronger forgetting, whereas the moving anchor yields slightly higher retention. Keeping the reference fixed therefore provides a more favorable forgetting result in this comparison, with a modest cost in retain performance.

Retain reconstruction weight.

To examine the role of retain reconstruction, we evaluated λ = 0, 0.5, 1, and 2.

SettingForget rL ↓Forget p ↓Retain rL ↑Retain p ↑RA rL ↑RA p ↑WF rL ↑WF p ↑
λ=00.0210.0010.6010.1560.6110.0650.8320.154
λ=0.50.0620.0010.7110.3590.6280.1110.8340.199
λ=1 (MDU)0.0980.0010.8530.4470.6450.1330.8420.205
λ=20.2130.0050.8620.4910.6620.1510.8370.194

As λ increases, retention improves while forgetting gradually weakens. The default setting λ=1 provides a balanced trade-off between forgetting and knowledge preservation.


[W3, Q1] Temperature selection and sensitivity

We have clarified the temperature settings in Section 5.1 and the trade-off analysis in Section 5.3 (Figure 5), with the complete sweep added to Appendix E.6 (Table 19). We select τ by considering forgetting and retention rather than applying a universal numeric optimum rule. We used τ=0.5 for LLaDA–TOFU and τ=0.25 for Dream–TOFU, LLaDA–RWKU, and Dream–RWKU. Below are the full LLaDA–TOFU results.

τForget rL ↓Forget p ↓Retain rL ↑Retain p ↑RA rL ↑RA p ↑WF rL ↑WF p ↑
0.000.0690.0000.8680.3810.6290.0930.8480.193
0.250.1350.0000.8570.3920.6160.1160.8420.204
0.50 (main)0.0980.0010.8530.4470.6450.1330.8420.205
0.750.0780.0400.6840.5350.6120.1550.8270.233
1.000.0340.0740.5110.4850.5680.1100.7770.187

For LLaDA–TOFU, τ=0.5 maintains low Forget answer probability with competitive retention and unrelated-knowledge performance, although low Forget ROUGE-L is observed at other temperatures as well. The other three model–benchmark combinations use τ=0.25 based on their respective trends in Figure 5.


AuthorsOpenReview ↗

Official Comment by Authors (2/2)

[W4, Q2] Autoregressive analogue and the need for MDLM-specific design

We have expanded the AR comparison in Appendix C and added its unlearning results to Appendix E.5 (Table 18). We implemented the analogue on TOFU-fine-tuned Llama-2-7B-Chat and compared it with GA, GD, and NPO.

MethodForget rL ↓Forget p ↓Retain rL ↑Retain p ↑RA rL ↑RA p ↑WF rL ↑WF p ↑
GA0.3420.2030.5660.3310.9090.0440.9070.032
GD0.3180.3470.6550.8430.8900.0180.8540.004
NPO0.3190.2180.6400.8530.9010.0420.9080.044
MDU-AR0.3960.4970.5850.8460.8380.0280.8390.058

The AR analogue maintains Retain answer probability comparable to GD and NPO but achieves weaker forgetting. AR predictions rely on a causal prefix, whereas MDU aligns predictions across partially masked response states that can include bidirectional context. This difference motivates designing the unlearning objective around MDLM denoising states.

References

[1] Eldan and Russinovich. Who's Harry Potter? Approximate Unlearning in LLMs. 2023.
[2] Liu et al. Revisiting Who's Harry Potter: Towards Targeted Unlearning from a Causal Intervention Perspective. 2024.
[3] Gandikota et al. Erasing Conceptual Knowledge from Language Models. 2026.
[4] Yuan et al. A Closer Look at Machine Unlearning for Large Language Models. 2024.

Thank you again for the valuable suggestions. They helped us clarify the methodological contribution and add the requested comparisons.


Thank you again for your constructive review. Your suggestions helped us clarify the methodological contribution and extend the experimental analysis. We summarize the main revisions below.

  1. Relationship to prior work. We expanded Appendix A to discuss Who's Harry Potter, causal-intervention unlearning, and ELM. We also clarified that MDU reduces exactly to Maximum Entropy unlearning at τ=0 and distinguished its MDLM-specific reference construction.

  2. Anchor and retain-weight ablations. We compared frozen and moving anchors and evaluated retain reconstruction weights λ=0, 0.5, 1, and 2. The results characterize how these choices affect forgetting and retention.

  3. Temperature selection. We clarified the temperature settings for each model–benchmark combination and added the full LLaDA–TOFU sweep, showing how τ controls the forgetting–retention trade-off.

  4. Autoregressive comparison. We implemented and evaluated an AR analogue on Llama-2-7B-Chat. Its weaker forgetting performance relative to AR baselines further motivates investigating unlearning objectives tailored to MDLM denoising states.

As the discussion period draws to a close, we would appreciate hearing whether our responses and additional experiments have addressed your main concerns.

Reviewer_F9j7OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Masked Diffusion Unlearning (MDU) matches prompt-conditioned token predictions to a frozen, prompt-masked anchor at the same partially revealed response state. Temperature scaling controls target entropy, with retain-set reconstruction when available. Results on LLaDA-8B and Dream-7B show strong forgetting metrics.

Critical strengths
  1. Coherent objective for masked denoising. Holding response context fixed while changing prompt access gives a clear intervention. The frozen anchor provides a stable target, and the temperature sweep includes the maximum-entropy endpoint.

  2. Substantial reported gains with useful diagnostics. On LLaDA TOFU, the reported metrics are better than baselines. Output examples, pseudo-perplexity, relearning attacks, and safety evaluation expose limitations as well as improvements.

Critical weaknesses
  1. The information quantity does not identify knowledge removal. Conditional mutual information can vanish when visible response tokens identify the prompt, despite perfect recovery of remaining facts. The anchor also contains forget knowledge. Appendix C acknowledges these limits, which constrain the conclusion's knowledge-erasure claim.

  2. Tempered targets can be inconsistent across masks. Even for a coherent anchor, separately powering and normalizing its predictions can violate the law of total probability. Eq. 6 therefore need not describe one target joint, and decoding order can matter. Eq. 7's conceptual interpolation does not resolve this. Moreover, local matching is not trajectory matching. Training averages over corrupted gold responses, not the updated decoder's visited states. Small token KL there does not control trajectory or terminal-distribution error. Decoding-step, unmasking-order, and informative prefix/suffix tests should assess this DLM-specific gap.

Questions for authors
  1. Table 4 uses temperature exponent 1; headline results use 0.5 or 0.25. At 0.5, LLaDA produces corrupted/empty outputs and median forget pseudo-PPL, which seems to be a response-quality issue. Can you explain that? T

  2. Temperature changes both target content and entropy. An entropy-matched nonsemantic anchor and a KL-direction comparison would test the defining choices. At the headline temperatures, what fractions of outputs are coherent refusals, factual substitutions, residual disclosures, or corrupted text? Does the advantage survive matched fluency and retention?

  3. How close is MDU to retain-only retraining, and can forgotten facts still be recovered through informative bidirectional response contexts?

4 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/3)

Thank you for your careful review and constructive feedback. We respond to [W#] and [Q#] below. The table, section, and appendix numbers below refer to the revised manuscript. The references listed at the end are also cited in the revised manuscript.


[W1] Information-theoretic interpretation and knowledge removal

MDU builds on prior information-theoretic machine unlearning studies [1, 2], which approach unlearning by reducing the predictive dependence on information designated for forgetting. We adapt this perspective to MDLMs by considering the additional information that a forget prompt contributes to token predictions, conditioned on the partially masked response state. MDU aligns prompt-conditional predictions with a frozen prompt-masked reference at the same response state to reduce this prompt-dependent information.

We have clarified this information-theoretic formulation in Section 3 and Appendix C of the revised manuscript. Specifically, we derive a decomposition of MDU's expected KL objective into conditional mutual information (CMI) and a nonnegative marginal-to-anchor KL term, showing that the objective upper-bounds CMI. This gives a precise interpretation of the information dependency optimized by MDU. The prompt-masked anchor remains conditioned on the observed response context; it serves as a reference for removing the explicit prompt contribution rather than as a distribution assumed to contain no target knowledge.

To evaluate the effect of this approach during generation, we have added contextual recovery and decoding-sensitivity experiments in Section 5.3 and Appendix E.8. Their protocols and results are discussed in [W2, Q3] below.


[W2, Q3] Mask-state consistency, decoding trajectories, and contextual recovery

Equation (6) defines a token-level alignment objective at sampled masked response states; it does not assert that the temperature-scaled conditionals form a single consistent joint distribution or guarantee matching of complete generation trajectories. We have added the contextual recovery results to Section 5.3 (Table 8), and the decoding sensitivity and detailed protocols to Appendix E.8 (Tables 23–24). These experiments vary decoding schedules and test recovery from partially observed responses.

Decoding steps and unmasking order (LLaDA–TOFU).

UnmaskingStepsForget rL ↓Retain rL ↑RA rL ↑WF rL ↑
Confidence320.0970.8260.6570.876
Confidence640.0970.8540.6250.855
Confidence1280.0980.8530.6450.842
Random320.1110.7680.6210.874
Random640.1050.7790.6470.869
Random1280.1110.7840.6600.817

Forget ROUGE-L remains between 0.097 and 0.111 under both unmasking orders and all three step budgets. Confidence-based unmasking generally preserves higher Retain ROUGE-L than random unmasking, indicating that the forgetting effect is stable across these tested decoding choices even when retention varies.

Response-span recovery under prefix and suffix context. We retain the original question and mask the target response span. To compare all three context conditions on identical examples, we select 237 of 400 forget10 questions with an identifiable span and word-level context on both sides. We then measure exact-match recovery of the masked span.

ModelPrefix-only EM ↓Suffix-only EM ↓Prefix + Suffix EM ↓
Base SFT156/237 (65.82%)234/237 (98.73%)236/237 (99.58%)
Retain-only2/237 (0.84%)6/237 (2.53%)7/237 (2.95%)
MDU (τ=0.5)0/237 (0.00%)0/237 (0.00%)1/237 (0.42%)

Base SFT frequently recovers the target, particularly when suffix or bidirectional context is available. In contrast, MDU recovers at most 0.42% of target spans and remains below the retain-only reference in every condition. These results extend the evaluation beyond the masked states sampled during training to partially revealed response contexts during generation.


AuthorsOpenReview ↗

Official Comment by Authors (2/3)

[Q1] Temperature settings in mechanism analysis and output quality

Table 4 uses τ=1 to examine the original, unscaled prompt-masked anchor and its token-role-specific conditional–anchor KL. This is a mechanism diagnostic rather than an evaluation at the main experimental temperature. For the main experiments, we selected temperatures by considering the forgetting–retention trade-offs in Figure 5, using τ=0.5 for LLaDA–TOFU and τ=0.25 for the other three model–benchmark settings. We have included the complete LLaDA–TOFU sweep in Appendix E.6 (Table 19), reproduced below.

τForget rL ↓Forget p ↓Retain rL ↑Retain p ↑RA rL ↑RA p ↑WF rL ↑WF p ↑
0.000.0690.0000.8680.3810.6290.0930.8480.193
0.250.1350.0000.8570.3920.6160.1160.8420.204
0.50 (main)0.0980.0010.8530.4470.6450.1330.8420.205
0.750.0780.0400.6840.5350.6120.1550.8270.233
1.000.0340.0740.5110.4850.5680.1100.7770.187

We have added the output-category analysis to Appendix E.6 (Table 20), categorizing 400 forget-set responses per τ to assess generation quality. As τ increases, corrupted and empty outputs become less frequent, while target-knowledge disclosures become more frequent, consistent with the pseudo-PPL trend reported in Table 6.

Output categoryτ=0τ=0.25τ=0.5τ=0.75τ=1
Coherent incorrect answer84139271319348
Knowledge disclosure914182429
Corrupted output239193763917
Empty response685435186
Total400400400400400

At the main setting, 271/400 responses are coherent but incorrect, while 111/400 are corrupted or empty; no explicit coherent refusals were observed. Output degradation therefore contributes to low overlap scores but does not fully explain them. We examine this distinction further in [Q2] using retention-matched comparisons and non-degenerate responses.


AuthorsOpenReview ↗

Official Comment by Authors (3/3)

[Q2] Entropy-matched reference, KL direction, and quality-controlled comparisons

Anchor construction and KL direction.

We have added the anchor and KL-direction ablations to Section 5.3 (Table 7) and their full results to Appendix E.7 (Table 21). These LLaDA–TOFU experiments use τ=0.5. The entropy-matched control permutes the anchor probabilities across vocabulary tokens, preserving entropy while disrupting token–probability associations. The reverse-KL control keeps the original anchor and reverses the KL direction.

MethodForget rL ↓Forget p ↓Retain rL ↑Retain p ↑RA rL ↑RA p ↑WF rL ↑WF p ↑
MDU0.0980.0010.8530.4470.6450.1330.8420.205
Permuted anchor0.0750.0030.6790.1580.6080.0540.8210.152
Reverse KL0.0820.0030.8160.4130.6270.1190.8380.201

Although the permuted anchor and reverse KL both obtain lower Forget ROUGE-L than MDU, each produces a higher Forget answer probability and lower retention. In particular, permuting the anchor preserves its entropy but substantially reduces retention. This comparison supports the value of token-specific anchor structure beyond entropy alone, while the reverse-KL result shows that divergence direction also affects the forgetting–retention balance.

Retention-matched baseline comparison.

We have added the retention-matched baseline comparison to Appendix E.9 (Table 25), along with non-degenerate output rates on the same 400 forget queries.

MethodForget rL ↓Forget p ↓Retain rL ↑Retain p ↑Non-degenerate ↑
GA0.7780.3560.8240.38292.50%
GD0.6050.2860.8790.46196.50%
NPO0.4860.1190.8380.40690.50%
SimNPO0.5270.1730.8670.43895.50%
WGA0.2940.0570.8120.37183.50%
DPO0.5360.2240.8910.47297.50%
ARL0.4580.1360.8460.41988.50%
MDU0.0980.0010.8530.44772.25%

At comparable retention levels, MDU retains the lowest Forget ROUGE-L and answer probability among the evaluated methods. Its non-degenerate output rate is lower than those of the baselines, however. To check whether corrupted and empty generations account for the result, we recomputed Forget ROUGE-L over MDU's 289 non-degenerate responses and obtained 0.1118, compared with 0.098 over the full set. The forgetting result therefore persists on this subset, while response quality remains an important trade-off; the full response categories appear in [Q1].

References

[1] Xu and Strohmer. Machine Unlearning via Information Theoretic Regularization. 2025.
[2] Xu et al. Forgetting-MarI: LLM Unlearning via Marginal Information Regularization. 2025.

Thank you again for the constructive review. Your feedback helped us clarify the objective and strengthen its empirical evaluation.


Thank you again for your thoughtful review. Your comments helped us clarify the theoretical interpretation and strengthen the empirical evaluation. We summarize the main revisions below.

  1. Information-theoretic interpretation. We clarified the relationship between MDU's KL objective and conditional mutual information (CMI), deriving an upper bound in Section 3 and Appendix C and specifying the role of the prompt-masked anchor.

  2. Decoding and contextual recovery. We evaluated different denoising steps and unmasking orders. We also tested target-span recovery under prefix-only, suffix-only, and bidirectional response contexts, where MDU achieved recovery rates of 0%, 0%, and 0.42%, respectively.

  3. Temperature and output quality. We added a full temperature sweep and categorized generated responses to distinguish coherent incorrect answers, knowledge disclosures, corrupted outputs, and empty responses.

  4. Anchor ablations and controlled comparisons. We evaluated an entropy-matched permuted anchor, reverse KL, and retention-matched baselines. We also examined forgetting on non-degenerate responses to assess the influence of output degradation.

As the discussion period draws to a close, we would be grateful to hear whether these revisions have addressed your main concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.