acceptodds
Under review as a conference paper at ICLR 2027

Enhanced Masked Diffusion Unlearning

Abstract

Masked diffusion language models (MDLMs) generate text by iteratively unmasking tokens with bidirectional context. As these models reach real-world deployment, *machine unlearning* becomes essential. This process aims to remove the influence of a designated 'forget set' without retraining from scratch, so that sensitive, copyrighted, or restricted content can no longer be elicited while preserving retained capabilities. Masked Diffusion Unlearning (MDU) is the first publicly available unlearning method customized for these models: it tunes the model to match a frozen prompt-masked anchor on the forget set by minimizing the *anchor-last* divergence . We identify two failure modes of MDU. First, random answer masking can leave key forget tokens visible to the anchor even when the question is fully masked. Second, at a memorized (near-one-hot) prediction, the anchor-last logit gradient vanishes while the *anchor-first* divergence logit gradient does not. As a result, MDU training stalls on the tokens we most need to forget. Backed by theoretical and empirical analyses, we propose Enhanced Masked Diffusion Unlearning (EMDU): anchor-first KL divergence toward a frozen anchor whose entire answer span is masked. On TOFU and RWKU benchmarks using LLaDA-8B and Dream-7B, our EMDU matches or exceeds the forgetting of MDU and adapted AR baselines, while better preserving retain utility.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Masked Diffusion Language Models, Machine Unlearning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_SiD6OpenReview ↗
Rating
4
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses a limitation of Masked Diffusion Unlearning (MDU) in that it can retain certain information despite being trained to forget them. MDU trains the model to match a frozen anchor with the prompt masked, with which the authors identify two problems. The anchor can still predict the forget answer from visible answer tokens, and the anchor-last KL loss has vanishing logit gradients when the model is highly confident in its predictions. The paper thus proposes Enhanced Masked Diffusion Unlearning (EMDU), which uses anchor-first KL and fully masks the answer span in the anchor. Experiments show better forgetting and retention than MDU on TOFU, with additional improvements on RWKU.

Critical strengths
  • The explanation of why KL direction matters for unlearning is convincing. Both objectives have the same minimizing distribution, but their logit gradients differ near one-hot predictions, which are common in LLaDA after TOFU fine-tuning. In Table 1, changing only the KL direction to anchor-first leaves far fewer high-confidence forget-token predictions after the same training run.

  • EMDU is better than MDU at both forgetting and answering retained questions on the two TOFU models (Table 3). Although either change alone achieves low Forget rL, combining anchor-first KL and full answer-span masking gives substantially higher Retain scores on both models (Table 2). Appendix E.1 also checks the comparison across additional MDU configurations and retain weights.

Critical weaknesses

None

Questions for authors

None

1 comment

We thank the reviewer for the careful reading and the positive assessment. We are glad that the reviewer found our analysis of the KL direction convincing, and that our experiments on TOFU and RWKU support it. We also thank the reviewer for acknowledging EMDU's improvements over MDU in both forgetting and retained utility.

Reviewer_QhKSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EMDU, an unlearning method for masked diffusion LMs (LLaDA, Dream) that improves on the prior masked diffusion unlearning (MDU) approaches. The paper show MDU has two flaws. Its randomly masked anchor can still see forget tokens, and its anchor-last KL gradient vanishes on confidently memorized tokens, so training stalls where forgetting matters most. EMDU fixes this by switching to anchor-first KL i.e., gradient stays nonzero near memorized predictions, and by fully masking the answer span when computing the anchor. On TOFU and RWKU, EMDU achieves the strongest forgetting with the best retain utility, typically abstaining with empty outputs on forget queries, and the results hold across generation settings.

Critical strengths
  • Knowledge leakage and vanishing gradients are motivated.
  • Results in Table 1 for comparing anchor-last and anchor-first KL shows difference.
Critical weaknesses
  • Experiments/evaluations are limited and no significant difference between MDU i.e., prior method.
    • Results are on (Subset of) TOFU and RWKU datasets.
    • Other datasets such as MUSE and WMDP datasets are not experimented.
    • Forget Quality metrics used in the original TOFU dataset [1] is not used without justification in the paper.
  • Presentation of the paper can be improved. Takeaway from Table 1 is unclear and requires a lot of effort from readers to understand.

[1] https://arxiv.org/abs/2401.06121

Questions for authors
  • Are there justifications of not using the Forget Quality metrics in TOFU and membership inference attack (MIA) in RWKU datasets?
  • These sentences seems like contradicting to each other (Lines 470-473) "The methods in Tables 3 and 4 are trained and evaluated at a single fixed seed, following the practice of Lee et al. (2026), due to limited compute resources. We re-run EMDU’s chosen configurations for TOFU and RWKU with 2 additional independent unlearning seeds per backbone and report mean ± std in Tables 9 and 10 (Appendix F)."
    • Why is the manuscript stating "limited compute resources." but running additional experiments in Appendix F?
7 comments
AuthorsOpenReview ↗

Response to Reviewer QhKS (1/2)

We thank the reviewer for the detailed review, and for finding the knowledge-leakage and vanishing-gradient motivation clear and the anchor-last vs. anchor-first comparison in Table 1 informative. We address each point below.

W1: Empirical coverage.

The paper focuses on analyzing the role of the KL direction and anchor construction in MDLM unlearning. Building on MDU, the first publicly available unlearning method designed for MDLMs, we show why its objective stalls on memorized forget tokens and propose EMDU, which changes exactly these two components. We therefore kept MDU's benchmarks and evaluation protocol. Our EMDU consistently and significantly outperforms MDU in the same evaluation setup (Tables 3 and 4), well beyond the seed-to-seed variation (see Q2).

W1.1: TOFU and RWKU splits.

As in MDU, we evaluate on TOFU forget10 and on the first 10 of RWKU's 200 subjects. Forget10 is the largest of TOFU's three forget splits and so the hardest one to forget while keeping the retain set intact. Evaluating on forget10 alone is common practice. Zhang et al. (ICML 2026, cited in our paper), the only other unlearning method for MDLMs besides MDU, use only this split, as do WAGLE (Jia et al., NeurIPS 2024), SOUL (Jia et al., EMNLP 2024), ReLearn (Xu et al., ACL 2025), NGDiff (Jin et al., NAACL 2025) and PMC (Scholten et al., ICLR 2026), the last referring to it as "the most challenging split". On RWKU, LAU (Yuan et al., AAAI 2025) and Kim et al. (EMNLP 2026) likewise evaluate on the same first 10 subjects.

W1.2: Experiments on other datasets.

Following the reviewer's suggestion, we added WMDP (Table 13, Appendix G of the revised paper). We unlearn the WMDP-Bio forget corpus and report WMDP-Bio accuracy together with MMLU. EMDU attains the lowest WMDP-Bio accuracy among all methods while keeping the highest MMLU.

The WMDP forget corpus consists of raw documents without a prompt, which matters for two methods: DPO and MDU. DPO does not apply, since it needs a preferred alternative response for every forget sample, which TOFU (perturbed answers) and RWKU (counterfactual answers) provide but WMDP does not. MDU's anchor is the pre-unlearning model with the prompt masked. A document has no prompt, so the anchor receives the same input as the unlearned model, which starts from the same weights. MDU's loss is therefore zero from the start and yields no gradient. EMDU's anchor, on the other hand, masks the full answer span (FAM). Thus EMDU provides nonzero gradients and can be applied directly, which is another practical advantage of its anchor construction. To examine the KL direction aspect of MDU, we give MDU the same fully masked anchor (MDU+FAM, the anchor-last variant with FAM ablated in Table 2), so it differs from EMDU only in the KL direction. EMDU outperforms this MDU+FAM, again showing the benefit of anchor-first KL.

Unlike WMDP, which probes knowledge that the pretrained model already holds, MUSE requires a target model fine-tuned on its corpora. The released MUSE targets are autoregressive, and fine-tuning MDLMs on these corpora, as we did for TOFU, is not feasible within the discussion period. We will make our best effort to add MUSE to the final version.

W1.3: Additional TOFU metric.

Forget Quality (FQ) is a statistical measure: the p-value of a Kolmogorov-Smirnov test between the truth-ratio distributions of the unlearned model and of a model fine-tuned on the retain set only, computed on the forget questions. A high FQ means that the two distributions cannot be told apart, so FQ takes the retain-only model as the target behavior. However, the retain-only model is an unreliable reference for unlearning. It never saw the forget data, yet it still answers the forget questions with plausible guesses. These guesses use cues from the question and correlations learned from the retain set, which shares the forget set's templates and attribute distributions. Some of them may even be correct. Its behavior on the forget set is therefore unpredictable, and FQ rewards a model for imitating these guesses rather than for removing the knowledge.

EMDU often produces empty completions on forget queries instead of guessing. This goes beyond abstention: EMDU also drives the probability of the ground-truth forget answers close to zero (0.0080.008 on LLaDA and 0.0010.001 on Dream, Table 3), far below that of a model trained only to abstain (EosSFT, Table 2). Since FQ measures agreement with a model that guesses, it scores this behavior as a deviation from the target, even though such a model discloses less than one that guesses.

Several works reach the same conclusion. Yuan et al. ("A Closer Look at Machine Unlearning for Large Language Models", ICLR 2025) argue that the behavior of the retain model on the forget set cannot be predicted and often consists of hallucinations, and recommend that unlearned models instead avoid relevant content or exhibit ignorance.

AuthorsOpenReview ↗

Response to Reviewer QhKS (2/2)

W1.3: Additional TOFU metric - Continue

Yuan et al. also show that FQ can change by orders of magnitude between consecutive epochs while the generated answers stay the same, and that it can be high while the answers still leak the forget set. Cho et al. ("Reference-Specific Unlearning Metrics Can Hide the Truth: A Reality Check", 2025) find that FQ depends strongly on the choice of reference answers, and that a model whose FQ improves can still reproduce the forget answers verbatim. TOFU itself states that FQ "does not fully capture approximate unlearning", and that almost none of the models it tests reach the 0.05 significance threshold.

For these reasons, we do not use FQ to compare methods. TOFU's likelihood-based evaluation is still a useful complement to Rouge-L and answer probability, so we report the statistic on which FQ is built, the Truth Ratio (TR), in the revised paper (Figure 8, Appendix F). TR compares the likelihood that the model assigns to the correct answer with that of incorrect ones but, unlike FQ, does not rely on a retain-only model. On LLaDA, EMDU lies on the Forget/Retain Pareto front and gives by far the most balanced trade-off: the only method with a higher Forget TR (WGA) has a much lower Retain TR, and the methods with a higher Retain TR (SimNPO and NPO) forget much less. On Dream, EMDU attains the highest Forget TR of all methods while keeping a Retain TR comparable to that of methods that forget far less. On both backbones, EMDU improves over MDU on both Forget and Retain TR.

W2: Clarified takeaway of Table 1.

We agree that the takeaway of Table 1 was not stated clearly enough. Because of the page limit, its discussion in the submission was brief. Table 1 tests the theoretical insight of Section 4 on real data. Near a memorized, almost one-hot prediction, the gradient of the anchor-last KL vanishes while that of the anchor-first KL does not. Therefore, anchor-last-based unlearning is expected to struggle to escape memorized forget tokens, contrary to anchor-first-based unlearning. For each masked answer position, qyq_y is the model's probability of the ground-truth forget token yy, and a value close to 1 means that the token is memorized. Table 1 presents the share of memorized positions along the unlearning epochs for runs of LLaDA on TOFU that differ only in the KL direction. Anchor-last KL still leaves 18.8% of the forget tokens memorized (qy>0.99q_y>0.99) after 9 epochs (with a trend that flattens by epoch 6), while anchor-first KL removes almost all of them within 4 epochs, aligned with our theory. We have added this explanation to the caption of Table 1 and to the paragraph that discusses it in the revised paper.

Q1: On Forget Quality and membership inference.

As noted in our answer to W1, we followed MDU's evaluation protocol, which reports neither of the two. For Forget Quality, please see our answer to W1.3, which discusses its limitations and reports TOFU's Truth Ratio instead.

Regarding membership inference on RWKU, our ethics statement already notes that abstaining with empty completions can itself act as a membership signal, so abstention should not be read as a complete privacy guarantee. We still consider it the safer output for unlearning, for the reasons given in Section 1. An empty completion is auditable and cannot leak partial or hallucinated forget-set content. A long incoherent answer can still contain leaked tokens (MDU at τ=0\tau=0, Figure 1), and a fluent answer risks either repeating the original fact or confidently stating a plausible but wrong one (MDU at τ=1\tau=1, Figure 2). Both fluent cases are indistinguishable from a real answer to a downstream user, and a confident wrong answer is presented as fact, which is a safety concern in its own right, especially in critical applications.

Q2: Clarified seed protocol, with new MDU seed runs.

The two sentences refer to different sets of runs, and we agree that the wording was confusing. Running every method over several seeds multiplies the cost of the comparison, since each method is tuned separately on both benchmarks and backbones. At submission time we could afford this only for EMDU, so the main tables (Tables 3 and 4) report a single seed for every method, as in Lee et al. (2026), and Appendix F reports the seed variance of EMDU. "Limited compute resources" referred to the seed runs of the other methods. We have rewritten this sentence in the revised paper.

Since the submission we obtained additional compute and ran MDU over the same three seeds as EMDU. The revised paper now reports mean ±\pm std for both methods (Tables 9 and 10, Appendix F). On both benchmarks and backbones, the difference between the two methods is well beyond the seed-to-seed variation.

We hope that these answers and the revised paper address the reviewer's concerns, and we would be glad to answer any further questions. We would be grateful if the reviewer would consider raising their score.

AuthorsOpenReview ↗

Kind reminder and additional results

Dear Reviewer QhKS,

Thank you again for your review. As the discussion period ends soon, we would like to kindly check whether our response addressed your concerns. For your convenience, we summarize the main additions below, including new results in an updated revision of the paper.

WMDP (W1.2). As you suggested, we added WMDP-Bio for LLaDA and Dream in our first revision (now Table 13, Appendix G). The LLaDA results are:

MethodWMDP-Bio ↓\downarrowMMLU ↑\uparrow
Base0.6610.68
GA0.5890.55
GD0.6330.54
NPO0.5170.32
SimNPO0.5930.44
WGA0.4520.52
MDU+FAM0.4100.58
EMDU0.3970.61

EMDU attains the lowest WMDP-Bio accuracy while keeping the highest MMLU. This also holds on Dream (Table 13). As explained in our response, DPO does not apply, since WMDP provides no preferred responses. MDU's own loss is zero on prompt-less documents, whereas EMDU applies directly thanks to its fully masked anchor. We therefore report MDU+FAM, which gives MDU our fully masked anchor, so that it differs from EMDU only in the KL direction.

TOFU forget05 and forget01 (W1.1). Following your comment, we also evaluated TOFU's smaller forget splits. Below are MDU and EMDU on LLaDA (Forget ↓\downarrow; Retain, RA and WF ↑\uparrow):

SplitMethodForget rLForget ppRetain rLRetain ppRA rLRA ppWF rLWF pp
forget05Base0.8240.3480.8080.3010.6470.0440.8400.133
MDU0.1130.1160.4380.3010.6850.0600.8170.094
EMDU0.0190.0070.7810.4840.6740.0950.8650.168
forget01Base0.8370.3860.8080.3010.6470.0440.8400.133
MDU0.0040.0650.4270.2230.7540.0860.8290.081
EMDU0.0000.0000.8250.4670.6520.0880.8580.169

The updated revision reports both splits on both backbones and for all baselines (Tables 11 and 12, Appendix F). EMDU has the lowest Forget Rouge-L and the highest Retain in every setting, with RA and WF best or competitive.

To recap the other points of our response:

  • Forget Quality (W1.3, Q1): we explain why FQ is not a reliable basis for comparing methods, and report TOFU's Truth Ratio instead, on which EMDU improves over MDU on both Forget and Retain (Figure 8, Appendix F).
  • Seeds (Q2): MDU now also has three seeds, and its gap to EMDU is well beyond the seed-to-seed variation (Tables 9 and 10).
  • Table 1 (W2): its takeaway is now stated in the caption and in the text.

Please let us know if there is anything else we can address.

Reviewer_nDAKOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Enhanced Masked Diffusion Unlearning (EMDU), a method for unlearning knowledge from masked diffusion language models. It makes two main changes to the existing MDU method: using anchor-first KL instead of anchor-last KL, and fully masking the answer when creating the anchor. The authors show that anchor-last KL has very small gradients when the model is highly confident in a memorized token, while anchor-first KL can still provide a strong learning signal. Experiments on TOFU and RWKU with LLaDA and Dream show that EMDU improves forgetting while better preserving model utility than MDU and several baselines.

Critical strengths
  1. Clear analysis of the KL direction. The paper provides a theoretical explanation of why anchor-last KL can have very small gradients on highly confident predictions. This gives a clear reason for trying anchor-first KL, rather than relying only on empirical results.
  2. Good experimental coverage and useful ablations. The paper evaluates two model families on TOFU and RWKU. The ablations also study the effects of the KL direction and full answer masking separately. The results show that combining both changes can improve forgetting while preserving more utility.
Critical weaknesses

The evaluation does not fully show that the target knowledge has been removed. EMDU often produces empty completions on forget queries. This reduces direct leakage, but it does not necessarily mean that the model no longer stores the knowledge.

Questions for authors

EMDU produces many EOS-only completions on forget queries. Have you tested whether the model can still reveal the forgotten facts through indirect prompts or other extraction attacks? How do you distinguish actual knowledge removal from learning to abstain on known forget queries?

5 comments
AuthorsOpenReview ↗

Response to Reviewer nDAK

We thank the reviewer for the thoughtful review, and for finding our analysis of the KL direction clear and our ablations useful. The central question is whether EMDU's empty completions mean that the knowledge was removed, or that the model learned to abstain while still storing it. The paper already reports two pieces of evidence that tell these cases apart, the answer probability pp and the EosSFT ablation, and we have made this argument explicit in Section 5 of the revised paper.

W1: Evidence that the knowledge is removed.

Our evaluation does not rely on generation alone. Tables 2 and 3 also report the answer probability pp (Eq. 13, Appendix D.1), the probability the model assigns to the ground-truth forget answer. It scores the ground-truth answer directly: the answer is given to the model, and we measure the probability it assigns to its tokens. This makes pp independent of what the model generates or of whether it emits EOS. A model that stores the fact but declines to state it still has a high pp. EMDU's forget pp is 0.0080.008 on LLaDA and 0.0010.001 on Dream (Table 3), down from 0.3370.337 and 0.8370.837 before unlearning. The generations are consistent with this. Empty completions occur almost only on the forget set (at most 0.5% on Retain, RA and WF; Section 5), and we checked the full generation canvas for leakage rather than only the text before the first EOS. In these completions every position, starting from the first, is EOS, so nothing is generated past the point where Rouge-L stops reading.

Q1: Can the forgotten facts be extracted with indirect prompts or other attacks?

We evaluated the forget set with TOFU's official paraphrased questions (Table 8, Appendix F). The fine-tuned model still answers these paraphrases (Rouge-L 0.4390.439 on LLaDA and 0.6070.607 on Dream), whereas EMDU stays low on both backbones (Rouge-L 0.0900.090 and 0.0080.008, pp 0.0160.016 and 0.0040.004). EMDU's forgetting is therefore not tied to the wording of the forget queries.

RWKU's question-answer probes (F-L2), built by paraphrasing and restructuring knowledge about each unlearning target, point the same way: EMDU is the best method on them on LLaDA and tied for best on Dream. RWKU's adversarial-attack probes (F-L3) query each target with one of nine jailbreak-style attacks designed to elicit forgotten knowledge, such as prefix injection, affirmative suffixes, role playing and reverse queries. EMDU is the best method on F-L3 on LLaDA and second best on Dream (Table 4).

EMDU's forgetting is not tied to the decoding configuration either. The concurrent work of Zhang et al. (ICML 2026, cited in our paper), the only other unlearning method for MDLMs besides MDU, shows that unlearning in MDLMs is highly sensitive to the output length and the number of denoising steps. We therefore re-evaluated EMDU under all six length and step combinations of their protocol (Table 7, Appendix F). EMDU's Forget Rouge-L stays between 0.0250.025 and 0.0370.037 on LLaDA and at exactly 00 on Dream across all six settings, against 0.7760.776 to 0.8920.892 and 0.8210.821 to 0.9350.935 for the fine-tuned model.

Q2: How to distinguish knowledge removal from learned abstention?

Our ablation in Table 2 includes a model that learns exactly this behavior. EosSFT is trained with the standard SFT loss to answer every forget query with EOS, so it learns to abstain without removing anything. Its forget Rouge-L drops to 0.1370.137 on LLaDA and 0.1570.157 on Dream, but its answer probability stays high (0.2450.245 and 0.5530.553), because the model still knows the answers. EMDU reaches a lower Rouge-L (0.0300.030 and 0.0000.000) and a forget pp of 0.0080.008 and 0.0010.001, about 30×30\times and 500×500\times lower than EosSFT. Learned abstention lowers Rouge-L but leaves pp high, while EMDU lowers both, as expected when the knowledge itself is removed. We now make this comparison explicit in Section 5 of the revised paper.

We hope that these answers and the revised paper address the reviewer's concerns, and we would be glad to answer any further questions. We would be grateful if the reviewer would consider raising their score.

AuthorsOpenReview ↗

Kind reminder

Dear Reviewer nDAK,

Thank you again for your review. As the discussion period ends soon, we would like to kindly check whether our response addressed your concern, namely whether EMDU removes the knowledge or only learns to abstain. In brief:

  • EMDU drives the probability of the ground-truth forget answers down to 0.0080.008 on LLaDA and 0.0010.001 on Dream, whereas a model trained only to abstain (EosSFT) keeps it high (0.2450.245 and 0.5530.553).
  • EMDU's forgetting holds under TOFU's paraphrased questions, RWKU's adversarial-attack probes, and all six decoding settings of Zhang et al.
  • We made this comparison explicit in Section 5 of the revised paper.

We have also uploaded a new revision with additional experiments on WMDP-Bio and on TOFU's forget05 and forget01 splits (Appendices G and F), where EMDU again attains the strongest forgetting together with the highest MMLU and Retain, respectively.

Please let us know if there is anything else we can address.

AuthorsOpenReview ↗

Follow-up before the discussion ends

Dear Reviewer nDAK,

Thank you again for your review. The discussion period closes in less than 24 hours. If any questions about our response or the revised paper remain, please let us know, and we will reply promptly. We have put substantial effort into this work. If your concern has been addressed, we would sincerely appreciate it if you would consider raising your score.

Comments on the paper

3 comments

We thank the reviewers and the Area Chair for their time and effort in handling our submission. Below, we provide point-by-point responses to the comments raised by the reviewers. We also uploaded a revised version of the paper where the modifications are marked with blue.

AuthorsOpenReview ↗

Summary of the reviews and our responses

We thank the reviewers and the Area Chair again for their time. As the discussion period ends soon, we summarize the reviews and how we addressed them.

What the reviewers found valuable. All three reviewers highlight our analysis of the KL direction. Reviewer nDAK finds it clear and grounded in theory, Reviewer QhKS finds the knowledge-leakage and vanishing-gradient issues well motivated, and Reviewer SiD6 finds it convincing. Reviewer nDAK also notes good experimental coverage and useful ablations. Reviewer SiD6 notes that EMDU improves over MDU in both forgetting and retained utility on both TOFU models.

Reviewer SiD6 recommended clear acceptance and raised no weaknesses or questions.

Reviewer nDAK asked whether EMDU removes the knowledge or only learns to abstain. Experiments in the original submission already address this. EMDU drives the probability of the ground-truth forget answers to 0.0080.008 on LLaDA and 0.0010.001 on Dream, while a model trained only to abstain keeps it high (0.2450.245 and 0.5530.553). The forgetting also holds under paraphrased questions, RWKU's adversarial attacks and all decoding settings of Zhang et al. (ICML 2026). We pointed to these results in our response and made the comparison explicit in Section 5 of the revision.

Reviewer QhKS asked for a broader evaluation and for clarifications on Table 1 and the seed protocol. We clarified both and added:

  • WMDP-Bio on both backbones, where EMDU attains the lowest WMDP-Bio accuracy with the highest MMLU.
  • TOFU forget05 and forget01 for all methods and both backbones, where EMDU has the lowest Forget Rouge-L and the highest Retain in every setting, with RA and WF best or competitive. Its Forget pp is essentially zero throughout (at most 0.0070.007).
  • TOFU's Truth Ratio, where EMDU lies on the Forget/Retain Pareto front on both backbones and improves over MDU on both axes. We report it instead of Forget Quality, which, as explained in our response, is not a reliable basis for comparing methods.
  • MDU runs over three seeds, which show that its gap to EMDU is well beyond the seed-to-seed variation, contrary to the concern that the two methods do not differ significantly. For example, on TOFU, EMDU's Forget pp is 0.009±0.0010.009 \pm 0.001 on LLaDA and 0.001±0.0000.001 \pm 0.000 on Dream, against 0.134±0.0180.134 \pm 0.018 and 0.308±0.0150.308 \pm 0.015 for MDU, and its Retain Rouge-L is 0.794±0.0080.794 \pm 0.008 on LLaDA and 0.857±0.0180.857 \pm 0.018 on Dream, against 0.382±0.1070.382 \pm 0.107 and 0.305±0.0780.305 \pm 0.078 (Table 9).

We uploaded two revised versions of the paper, with all changes marked in blue. So far, Reviewers nDAK and QhKS have not replied to our responses and follow-ups.

We have put substantial effort into this work, and we believe that our responses and revisions address the concerns of both reviewers. We would be grateful for any further feedback before the discussion ends, and we hope that our responses will be taken into account in the final assessment.

AuthorsOpenReview ↗

Update on the reviewers' replies

Reviewer nDAK has read our response and raised the score to weak acceptance, noting that the EosSFT ablation results are helpful.

Reviewer QhKS replied less than five hours before the end of the discussion period, after our two reminders, and kept the score. Our full answer is under the reviewer's comment. In brief:

  • Retained utility. On TOFU, EMDU has the highest Retain in all six settings (forget10, forget05 and forget01 on both backbones). On RWKU, it is best on three of the four neighbor columns, RWKU's counterpart of a retain set, and best or tied for best on 10 of all 16 columns. The objection concerns two columns on Dream (N-L2 and TriviaQA), where every method ahead of EMDU forgets less (F-L3 of 0.2070.207 to 0.2480.248 against EMDU's 0.1490.149). We will state the claim in L105-107 as the best overall trade-off between forgetting and retained utility.
  • Membership inference. We answered this in our response, and the Ethics Statement already states that abstention is not a privacy guarantee. MDU and Zhang et al. (ICML 2026) do not report MIA either. The alternatives, an incoherent response or a wrong answer, are not safer. An incoherent response can still contain leaked forget tokens or be vulnerable to MIA, and a wrong answer presents misinformation as fact.
  • RKLU (NAACL 2025). Its teacher is the original model's own prediction on the full forget input, which still carries the answer, and its own forward-KL model reproduces the ground-truth forget answer. In MDLMs, our direction-only comparisons favor anchor-first on both backbones, with and without full answer-span masking (Table 2), and on WMDP on both backbones, where EMDU beats MDU+FAM on both WMDP-Bio and MMLU (0.3970.397 vs. 0.4100.410 and 0.610.61 vs. 0.580.58 on LLaDA, 0.3070.307 vs. 0.4460.446 and 0.530.53 vs. 0.360.36 on Dream). The original review listed the vanishing-gradient analysis and Table 1 among the paper's strengths.
  • Backbones. LLaDA and Dream are the standard MDLMs and the only backbones of MDU and Zhang et al. (ICML 2026). Adding new models is not feasible in the remaining hours.

We respectfully ask the Area Chair to take these answers into account.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.