acceptodds
Under review as a conference paper at ICLR 2027

Forget the Fact, Keep the Form: Distribution Redistribution for Retain-Free Unlearning in Masked Diffusion Language Models

Abstract

Masked Diffusion Language Models (MDLMs) have emerged as a promising alternative to autoregressive models, enabling parallel generation and flexible token ordering. Language-model unlearning aims to remove unwanted information while preserving the model's behavior on retained knowledge. Retain-free methods are increasingly desirable because retain data is often unavailable in practice. However, existing retain-free approaches face two fundamental challenges. Overly aggressive suppression can degrade generation fluency and coherence, whereas suppressing only ground-truth tokens may still allow the model to express the same information through paraphrases. We introduce **Di**stribution **R**edistribution **U**nlearning (**DiRU**), a retain-free framework that suppresses the model's reliance on forgotten information while preserving the structure and quality of its responses. **DiRU** uses the original model as a frozen teacher and the model being unlearned as a student. It masks query and response tokens associated with the forgotten information to construct an unconditional teacher. The contrast between the conditional and unconditional teachers identifies condition-dependent probability mass, including mass assigned to paraphrases of the targeted fact. The student is then distilled toward the unconditional teacher while this condition-dependent probability mass is suppressed. Through this redistribution process, **DiRU** reduces both direct recall and paraphrased re-expression of targeted information while maintaining overall generation quality. Experiments across diverse benchmarks and MDLM families show that **DiRU** outperforms existing retain-free methods, achieving strong forgetting while substantially improving model utility and preserving linguistic quality close to that of the original model.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Diffusion Language Model, Machine Unlearning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dRnyOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes DiRU, a retain-free unlearning method for masked diffusion language models. DiRU selects query tokens to be masked through Monte Carlo importance estimation, and compares the predicted distributions of the frozen model under the original query and the partially masked query to identify vocabulary tokens relevant to the target fact. Subsequently, DiRU constructs an attribution-based redistribution target and updates the model through KL distillation, preserving language generation capabilities while suppressing target knowledge. The paper evaluates DiRU on Dream and LLaDA models using TOFU, MUSE, and RWKU, and provides theoretical analysis and component ablation.

Critical strengths
  1. Avoiding both output degradation and paraphrase leakage is a crucial issue in retain-free unlearning. DiRU constructs the target distribution using a frozen model, requiring no additional retain data; the method is intuitive and has practical value.
  2. The paper covers three unlearning benchmarks and multiple MDLM backbones, simultaneously evaluating forgetting, utility, and linguistic quality. Qualitative case studies and component ablation also help analyze the failure modes of different methods.
Critical weaknesses
  1. There is a serious inconsistency between the method definition and experimental results in the paper. At K = 0%, the target distribution should equal the original teacher distribution, and the KL loss and gradient at student initialization should be zero, but Table 9 still reports significant forgetting. Furthermore, the default K = 25% in Table 6 is inconsistent with the results in Tables 1 and 9, making it difficult to confirm the actual configuration of the main experiments.

  2. The evaluation pipeline contains internal inconsistencies that make the headline EQ/MU/LQ results difficult to trust. For example, in Appendix E.5, nearly identical NPO and SimNPO responses receive ROUGE-L scores of 0.96 and 0.26, and DiRU's non-refusal answer receives Relevance 9, although the judge rubric assigns 0 to responses that do not refuse.

Questions for authors
  1. Why does K=0% still produce substantial forgetting in Table 9? Also, Table 1's Dream Forget01 results match the K=50% ablation, while Table 6 lists K=25% as the default. Which setting was actually used? Why does removing the suppression term (β = 0) barely change performance?

  2. Some ROUGE-L and relevance scores in Appendix E.5 do not seem to match the displayed responses. How were these scores computed? Could the authors also report the standard TOFU and MUSE metrics separately and provide a small human evaluation to check the LLM judge?

  3. Figure 2 suggests that several baselines perform reasonably well before collapsing later in training. How do the results change with early stopping or separate hyperparameter tuning? Were the training budgets matched, particularly on RWKU?

  4. How does DiRU compare with random query masking or direct distillation from a fully prompt-masked teacher, without attribution suppression? These seem like useful controls for understanding which parts of the method matter.

  5. The redistribution target seems similar to the generic-prediction target in Who's Harry Potter? (Eldan & Russinovich, 2023), which also uses a ReLU-truncated difference between predictions. What is the key difference? How does DiRU compare with FLAT (Wang et al., 2025b), which is cited but not evaluated?

  6. The paper motivates DiRU partly through squeezing and knowledge resurfacing under different denoising contexts. Does the method still prevent leakage when the question is paraphrased, the masking pattern changes, or part of the answer is already revealed?

  7. How were the forget query–response pairs constructed for MUSE and RWKU? What data did the unlearning method actually have access to in these experiments?

11 comments
AuthorsOpenReview ↗

Response to Reviewer dRny

Q1. Why does K=0% still produce substantial forgetting in Table 9? Also, Table 1's Dream Forget01 results match the K=50% ablation, while Table 6 lists K=25% as the default. Which setting was actually used? Why does removing the suppression term (β = 0) barely change performance?

We sincerely thank the reviewer for identifying these inconsistencies. We apologize for the oversight in our implementation and experimental reporting.

Regarding the K=0% setting, we found that our implementation enforces a minimum of one masked position, even when KK is set to 0%. Consequently, the K=0% configuration does not correspond to a true zero-masking setting; at least one position remains masked during the process. This implementation detail was not clearly documented in the original manuscript, which may have caused the substantial forgetting observed in Table 9 to appear counterintuitive. We acknowledge that labeling this configuration as KK=0% was misleading, and will clarify the implementation and correct the corresponding description in the revised manuscript.

Regarding the discrepancy between Tables 1 and 6, we acknowledge that the reported KK values were inconsistent. We have rechecked the experimental configurations and logs to verify the actual KK value used in each experiment. Specifically, the default configuration for Forget01 uses K=50K=50. We will correct the tables accordingly and explicitly state this default setting to ensure consistency, clarity, and reproducibility.

Regarding why removing the suppression term barely change the performance, we analyze that this is caused by the factual tokens whose attribution score is relatively small, leading to the suppression term exp[−βsti(v)]∼1exp[-\beta s_t^i(v)] \sim 1. Moreover, β=0\beta=0 still induces forget because the target distribution becomes q0,ti(v)=pθ0,ti(v∣M(x;K),yt)≤exp⁡[−sti(v)]q_{0,t}^i(v) = p_{\theta_0,t}^i(v\mid M(x;K),y_t) \leq \exp[-s_t^i(v)], which can still induce forgetting at factual tokens whose attribution score is high . Therefore, increasing β=0\beta=0 to 10/1510/15 only covers a few more tokens, which only increases slightly the performance of the unlearned model.

To support our analysis, we conduct an additional experiment with a large suppression weight, β=50,000\beta=50{,}000, and report the resulting forgetting performance below.

β\betaForget Probability ↓Forget ROUGE ↓Forget Truth Ratio ↑Forget Relevance ↑Forget Correctness ↓EQ ↑
00.410.520.537.212.910.61
100.380.450.606.412.550.64
50,0000.150.400.596.502.200.69

When β=50,000\beta=50,000, it can captures more factual tokens whose attribution scores are small, improving erasure quality (EQ) about 5%.

Q2. Some ROUGE-L and relevance scores in Appendix E.5 do not seem to match the displayed responses. How were these scores computed? Could the authors also report the standard TOFU and MUSE metrics separately and provide a small human evaluation to check the LLM judge?

We thank the reviewer for raising these concerns. We clarify the computation of our evaluation metrics, distinguish the two evaluation rubrics, and will provide separate benchmark-specific results for TOFU and MUSE. We will also conduct a small-scale human evaluation to assess the reliability of the LLM judge.

1. How the scores are computed Response: Thank you for pointing this out. ROUGE-L and Relevance are computed using two independent evaluation procedures:

  • ROUGE-L: We compute ROUGE-L recall between each model-generated response and its corresponding ground-truth answer using the rouge_score package with stemming enabled. It is a deterministic lexical-overlap metric, not an LLM-based evaluation. Therefore, semantically equivalent paraphrases may still receive relatively low ROUGE-L scores.

  • Relevance: We use Qwen3.8-27B as an LLM judge. Given the question, ground-truth answer, and model response, the judge assigns an integer score from 0 to 10 based on how directly the response addresses the specific question, independently of factual correctness. We normalize the scores to [0, 1] and average them over the corresponding evaluation split.

Thus, the reported scores are computed using the full evaluation sets rather than inferred from the displayed examples alone. We will clarify these evaluation procedures in the revised appendix to avoid potential confusion.

2. Why DiRU's answer receives a high Relevance score We would like to clarify that our operational definition of refusal is not limited to explicit refusal statements, such as "I do not know the answer" or "I cannot provide that information." It also encompasses implicit refusals, in which the model avoids disclosing the target fact while providing a meaningful and contextually relevant alternative response.

Under this definition, a successful refusal can take two forms: (1) an explicit abstention, in which the model clearly indicates that it cannot or does not know the answer; or (2) an implicit refusal, in which the model provides an alternative response that addresses the question without revealing the target information.

This distinction explains why DiRU answer in Example 1, Appendix E.5 receives a high score of 9. Although DiRU does not explicitly refuse to answer, it successfully avoids disclosing the target fact while providing a coherent response that directly addresses the user's question. In contrast, GA in the same example receives a score of 0 because its output consists of unintelligible and repetitive text. Although such an output may not reveal the target fact, it is neither a meaningful response to the question nor a recognizable form of abstention. Therefore, it does not satisfy our intended criteria for a successful refusal.

For clarity, non-disclosure alone is not sufficient to constitute a successful refusal under our evaluation protocol. The response must either express a recognizable abstention or provide a meaningful, contextually relevant alternative without revealing the target information.

Importantly, refusal and factual correctness are evaluated as distinct aspects of model behavior. A response can successfully withhold the target fact while still receiving a low Correctness score because its alternative answer does not match the ground-truth answer. This distinction is intentional: our evaluation aims to assess whether the model retains the ability to generate meaningful responses while preventing the disclosure of information designated for forgetting.

Illustrative Example: Forgetting the Target Fact "French Literature"

Consider the following forget query:

  • Question: What genre is author Basil Mahfouz Al-Kuwaiti most known for?
  • Target fact to forget: French literature.

The responses from different methods illustrate why avoiding disclosure and generating a meaningful response must be considered together:

MethodResponseFluencyRelevanceCorrectness
BaseIdentifies French literature as the author's genre.---
GAProduces unintelligible and repetitive text.000
DiRU"Basil Mahfouz Al-Kuwaiti is most known for his contributions to the Urban Fiction genre."1090
Retain-only"Basil Mahfouz Al-Kuwaiti is most renowned for his contributions to the genre of Gothic Literature."10100

This example highlights three distinct behaviors:

  1. Base discloses the target fact: It explicitly identifies French literature, indicating that the information designated for forgetting is still disclosed.
  2. GA generates meaningless output: Although it does not reveal French literature, its response is unintelligible and does not meaningfully address the question. Consequently, it receives a Relevance score of 0.
  3. DiRU provides a meaningful alternative: It identifies Urban Fiction instead of French literature. Although this alternative does not match the ground-truth fact, it is a coherent and contextually relevant response that avoids disclosing the target information. Therefore, it receives a high Relevance score of 9 and a Correctness score of 0. The key distinction is that DiRU demonstrates meaningful non-disclosure, whereas GA produces an unintelligible response. This is why merely avoiding the target fact does not automatically guarantee a high evaluation score. The response must also remain meaningful and relevant to the original question.
Reviewer_cJYaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Distribution Redistribution Unlearning (DiRU), a retain-free unlearning method for masked diffusion language models (MDLMs). DiRU masks the most important query tokens to obtain a fact-ablated teacher, and uses the log-ratio between the original and fact-ablated teachers to down-weight tokens promoted by the forget query. The student is then distilled towards the resulting target. Empirically, DiRU achieves competitive forgetting while preserving model utility and fluency much better than the baselines.

Critical strengths
  • Retain-free unlearning is a practical setting, and the method is simple and intuitive.
  • DiRU preserves the generation quality of the original model much better than the baselines, which tend to collapse into degenerate text.
Critical weaknesses
  • Missing Discussion on Closely Related Work: The idea of contrasting two predictive distributions to identify and suppress fact-related tokens has been explored in Who's Harry Potter (WHP) [1], which fine-tunes towards vbaseline−α ReLU(vreinforced−vbaseline)v_{\text{baseline}} - \alpha\,\mathrm{ReLU}(v_{\text{reinforced}} - v_{\text{baseline}}). This is closely connected to Eq. 7, but WHP is neither cited nor discussed. To me, the contributions of DiRU are applying this idea to the masked states of MDLMs, building the fact-ablated teacher via query token importance. It would also be helpful to adapt WHP to MDLMs as a baseline, to see whether the gains come from the redistribution objective or simply from suppressing fact-related tokens.
  • Information Leakage through the Response: The fact-ablated teacher only masks the query, while yty_t still comes from the original answer (Algorithm 2). For example, with the answer "Einstein was born in Ulm, Germany", masking "Einstein" in the query does not stop the teacher from predicting "Ulm" from "Einstein was born in [MASK], Germany". The attribution score of "Ulm" is then close to zero, so it will not be suppressed. This should be worse at small tt, where more response tokens are visible.
  • Gap between Theory and Unlearning: In Assumption 5.1, VfV_f and VsV_s are defined by the attribution score, not by whether a token actually carries the fact. This is related to my previous point of information leakage: a higher attribution score does not necessarily mean a true fact token. So Theorem 5.4 only says that high-score tokens are suppressed, which follows directly from Eq. 7. It does not guarantee that the fact is forgotten, since a low score does not mean the token is unrelated to the fact (see the leakage above). The claim that DiRU "provably suppresses the forgotten fact" is a bit overclaim.

[1] Eldan, Ronen, and Mark Russinovich. "Who's Harry Potter? Approximate unlearning in LLMs."

Questions for authors
  • In Table 9, β=0\beta = 0 (i.e., simply distilling towards the fact-ablated teacher) performs almost the same as β=10\beta = 10 (which is the only difference on EQ). This suggests that most of the gain comes from the fact-ablated teacher rather than from the redistribution.
  • With K=0%K = 0\%, the target is just the original teacher, so the model should stay close to Base. Why is the EQ 0.50 in Table 9?
5 comments

We appreciate the reviewer’s constructive feedback for improving our paper. We would like to address the concerns and questions raised below.

W1. Missing Discussion on Closely Related Work: The idea of contrasting two predictive distributions to identify and suppress fact-related tokens has been explored in Who's Harry Potter (WHP) [1], which fine-tunes towards vbaseline−α.RELU(vreinforced−vbaseline)v_{baseline} - \alpha.RELU(v_{reinforced}-v_{baseline}). This is closely connected to Eq. 7, but WHP is neither cited nor discussed. To me, the contributions of DiRU are applying this idea to the masked states of MDLMs, building the fact-ablated teacher via query token importance. It would also be helpful to adapt WHP to MDLMs as a baseline, to see whether the gains come from the redistribution objective or simply from suppressing fact-related tokens.

We agree that Who's Harry Potter? (WHP) is closely related and will cite and discuss it in the revised manuscript. Our attribution score sti(v)s_t^i(v) serves a similar purpose to ReLU(vreinforced−vbaseline)\mathrm{ReLU}(v_{\mathrm{reinforced}}-v_{\mathrm{baseline}}) in WHP: both identify predictive support associated with the information to be forgotten. However, the methods differ in the following two aspects.

1. WHP selects a replacement token from corrected logits, which can still favor the factual token. WHP computes

zWHP=vbaseline−αReLU˙(vreinforced−vbaseline)z_{\mathrm{WHP}} = v_{\mathrm{baseline}} - \alpha\.\mathrm{ReLU}\left(v_{\mathrm{reinforced}}-v_{\mathrm{baseline}}\right)

and selects y^=arg⁡max⁡vzWHP(v)\hat y=\arg\max_v z_{\mathrm{WHP}}(v) as the replacement label for fine-tuning. Adapted to masked diffusion states, this corresponds to

LWHP=−E(x,y0)∼Df,t,yt[1∣Mt∣∑i∈Mtlog⁡pθ,ti(y^0i∣x,yt)].\mathcal L_{\mathrm{WHP}} = -\mathbb E_{(x,y_0)\sim D_f,t,y_t}\left[\frac{1}{|M_t|}\sum_{i\in M_t}\log p_{\theta,t}^i(\hat y_0^i\mid x,y_t)\right].

However, raw logits from the baseline and reinforced models need not be directly comparable as measures of predictive support. For example, let vbaseline=[1,−1,−2]v_{\mathrm{baseline}}=[1,-1,-2] and vreinforced=[1,−2,−3]v_{\mathrm{reinforced}}=[1,-2,-3]. The ReLU-truncated difference is [0,0,0][0,0,0], so the corrected logits remain [1,−1,−2][1,-1,-2] for every α\alpha. The first token is therefore still selected as the replacement label, even though its normalized probability increases from approximately 0.8440.844 to 0.9360.936 after reinforcement. If this token carries the fact to be forgotten, training toward this replacement label continues to reinforce that fact. Furthermore, a single replacement label does not explicitly preserve the teacher's relative probabilities among structural alternatives.

2. DiRU redistributes probability over the entire vocabulary while anchoring the target to the fact-ablated teacher. At each masked response position, DiRU constructs

qβ,ti(v)=pθ0,ti(v∣M(x;K),yt)exp⁡[−βsti(v)]Zβ,tiq_{\beta,t}^i(v) = \frac{p_{\theta_0,t}^i(v\mid M(x;K),y_t)\exp[-\beta s_t^i(v)]}{Z_{\beta,t}^i}

and optimizes LDiRU\mathcal L_{\mathrm{DiRU}} to match this full distribution. The attribution penalty suppresses high-score tokens, while the fact-ablated teacher determines how probability is allocated among the remaining alternatives. In particular, structural tokens with zero attribution preserve their relative probabilities under this reference distribution. Our theoretical analysis formalizes suppression and structural preservation under its stated assumptions.

Thus, the distinction extends beyond the use of ReLU: WHP trains toward selected replacement labels, whereas DiRU trains toward a redistributed vocabulary distribution with an explicit structural reference.

We agree that adapting WHP to masked diffusion language models would provide a useful baseline and will report this comparison to assess the contribution of the redistribution objective.

WHP baseline results: We additionally evaluate both methods on TOFU Forget10 across all three DLMs under the same evaluation protocol, as reported above:

MethodDream-7BLLaDA-8BLLaDA-1.5
EQMULQAvg.EQMULQAvg.EQMULQAvg.
Base0.070.610.990.560.060.720.990.590.080.630.990.57
Retain-only0.750.580.920.750.740.650.880.760.680.670.970.77
GA0.560.060.130.250.590.140.200.310.420.310.240.32
NPO0.570.090.180.280.590.120.200.300.420.200.220.28
SimNPO0.560.110.200.290.580.100.160.280.430.200.210.28
WGA0.580.120.190.300.620.110.170.300.580.190.190.32
BST0.580.070.120.260.590.110.160.290.590.120.180.30
FLAT0.580.210.240.340.590.180.230.330.570.240.270.36
WHP0.540.360.560.490.560.300.490.450.560.410.500.49
DiRU0.600.380.990.660.610.400.990.670.640.400.990.68

[1] Eldan, Ronen, and Mark Russinovich. "Who's Harry Potter? Approximate unlearning in LLMs."

W2. Information Leakage through the Response: The fact-ablated teacher only masks the query, while yty_t still comes from the original answer (Algorithm 2). For example, with the answer "Einstein was born in Ulm, Germany", masking "Einstein" in the query does not stop the teacher from predicting "Ulm" from "Einstein was born in [MASK], Germany". The attribution score of "Ulm" is then close to zero, so it will not be suppressed. This should be worse at small tt, where more response tokens are visible.

We admit that visible response tokens can reveal the target fact even after important query tokens are masked. We also observed this issue during student training. When the target token remains predictable from the response context under both the original and masked queries, its attribution score can be close to zero, resulting in weak suppression. This concern is particularly relevant at low response masking levels, where more of the original answer remains visible.

Additional response masking in our TOFU experiments. To mitigate this leakage, we supplement query masking with additional masking in the noisy response yty_t. Specifically, we also mask response tokens that appear among the query tokens selected for masking. For example:

  • Original query: Where was Einstein born?
  • Masked query: Where was [MASK] born?
  • Noisy response: Einstein [MASK] born in [MASK], Germany.
  • After additional response masking: [MASK] [MASK] born in [MASK], Germany.

This step removes the repeated entity cue, "Einstein," from the response as well as the query, reducing one source of factual leakage.

Dependence on the masking timestep. Our analysis in the response to Reviewer 6znC, Q2 examines attribution scores across response masking timesteps while keeping query masking fixed at K=50%K=50\%. The two illustrative examples show that factual tokens can have near-zero attribution scores at lower or intermediate noise levels, with larger scores at higher noise levels. This suggests that exposing less of the original response can make the attribution signal more informative, although the effect depends on the example.

Motivated by this observation, we plan to explore replacing uniform timestep sampling with different distributions such as Beta distribution that places more probability mass on larger tt. The aim is to emphasize training states with less visible response context and potentially stronger attribution signals. This remains a direction for future experiments.

W3. Gap between Theory and Unlearning: In Assumption 5.1, Vf\mathcal{V}_f and Vs\mathcal{V}_s are defined by the attribution score, not by whether a token actually carries the fact. This is related to my previous point of information leakage: a higher attribution score does not necessarily mean a true fact token. So Theorem 5.4 only says that high-score tokens are suppressed, which follows directly from Eq. 7. It does not guarantee that the fact is forgotten, since a low score does not mean the token is unrelated to the fact (see the leakage above). The claim that DiRU "provably suppresses the forgotten fact" is a bit overclaim.

We acknowledge that a low attribution score does not imply that a token is unrelated to the target fact, as the model may recover memorized information from very limited context. In our response to W2, we describe our mitigation strategy and propose further steps to address this issue. However, we believe that a high attribution score is a strong indicator of relevance to the target fact, because masking fact-relevant query context is expected to affect factual tokens more strongly than other tokens.

Q1. In Table 9, β=0\beta=0 (i.e., simply distilling towards the fact-ablated teacher) performs almost the same as β=10\beta=10 (which is the only difference on EQ). This suggests that most of the gain comes from the fact-ablated teacher rather than from the redistribution.

We agree that the fact-ablated teacher provides a substantial part of the forgetting signal on TOFU. When β=0\beta=0, the target distribution becomes

q0,ti(v)=pθ0,ti(v∣M(x;K),yt).q_{0,t}^i(v) = p_{\theta_0,t}^i(v\mid M(x;K),y_t).

For a token with positive attribution score sti(v)>0s_t^i(v)>0, the definition of the score gives

pθ0,ti(v∣M(x;K),yt)=pθ0,ti(v∣x,yt)exp⁡[−sti(v)]≤exp⁡[−sti(v)].p_{\theta_0,t}^i(v\mid M(x;K),y_t) = p_{\theta_0,t}^i(v\mid x,y_t)\exp[-s_t^i(v)] \leq \exp[-s_t^i(v)].

Thus, high-attribution tokens already have reduced probability under the fact-ablated teacher, even without the additional suppression controlled by β\beta. The β\beta is to suppress more on factual tokens which receive small attribution score sti(v)>0s_t^i(v)>0.

Our attribution-score analysis on TOFU suggests that most factual tokens have large scores, while a remaining subset has very small scores, as illustrated by Example 2 in our response to Reviewer 6znC, Q2. For the former, the fact-ablated teacher already assigns low probability. For tokens whose scores are sufficiently small can cause the suppression exp⁡[−βsti(v)]∼1\exp[-\beta s_t^i(v)] \sim 1 . Together, these observations offer an explanation for the small performance difference between β=0\beta=0 and β=10\beta=10 on TOFU.

To further examine the effect of suppression strength, we also evaluated a much larger value, β=50,000\beta=50{,}000:

β\betaForget Probability ↓Forget ROUGE ↓Forget Truth Ratio ↑Forget Relevance ↑Forget Correctness ↓EQ ↑
00.410.520.537.212.910.61
100.380.450.606.412.550.64
50,0000.150.400.596.502.200.69

Setting β=50,000\beta=50{,}000 strengthens the suppression of factual tokens with small positive attribution scores, yielding a modest improvement in forgetting performance. This suggests that forgetting effectiveness depends strongly on the distribution of attribution scores in the forget dataset. To address this dependence, we plan to explore alternative timestep distributions, such as Beta distribution that places more probability mass on noisier timesteps near t=1t=1.

Finally, DiRU theoretically preserves structural behavior and remains stable across different values of β\beta. This explains why the MU and LQ results are identical across the two settings β=0\beta=0 and β=10\beta=10.

Q2. With K=0K=0, the target is just the original teacher, so the model should stay close to Base. Why is the EQ 0.50 in Table 9?

We sincerely thank the reviewer for identifying these inconsistencies. We apologize for the oversight in our implementation and experimental reporting.

Regarding the K=0K=0% setting, we found that our implementation enforces a minimum of one masked position, even when KK is set to 0%. Consequently, the K=0K=0% configuration does not correspond to a true zero-masking setting; at least one position remains masked during the process. This implementation detail was not clearly documented in the original manuscript, which may have caused the substantial forgetting observed in Table 9 to appear counterintuitive. We acknowledge that labeling this configuration as K=0K=0% was misleading, and will clarify the implementation and correct the corresponding description in the revised manuscript.

Regarding the discrepancy between Tables 1 and 6, we acknowledge that the reported KK values were inconsistent. We have rechecked the experimental configurations and logs to identify the actual KK value used for each experiment. We will correct the tables and explicitly state the default setting to ensure consistency and reproducibility.

Reviewer_6znCOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Distribution Redistribution Unlearning (DiRU), a retain-free unlearning framework for masked diffusion language models (MDLMs). DiRU addresses two limitations of existing methods: incomplete forgetting due to paraphrased responses and degraded generation quality caused by excessive suppression. It uses a frozen teacher model to identify fact-related tokens by comparing predictions under original and partially masked queries, then redistributes their probability mass to preserve response structure. The student model is trained through knowledge distillation without retain data. Theoretical analysis and experiments on TOFU, MUSE, and RWKU demonstrate competitive forgetting performance with improved model utility and linguistic quality compared to existing baselines.

Critical strengths
  1. Interesting and well-motivated methodological design. The paper introduces an interesting perspective on retain-free unlearning by focusing on probability redistribution rather than simply suppressing undesirable outputs. The proposed teacher–student framework uses fact-ablated queries to distinguish fact-related information from general response structure, providing a reasonable approach to balancing forgetting effectiveness and generation quality without retain data. The theoretical analysis further supports the proposed redistribution mechanism.

  2. Strong and comprehensive experimental results. The paper demonstrates consistently strong performance across multiple benchmarks (TOFU, MUSE, and RWKU) and different masked diffusion language models. DiRU achieves competitive forgetting effectiveness while substantially improving model utility and linguistic quality over existing retain-free baselines. The consistent improvements across different experimental settings provide convincing empirical support for the effectiveness of the proposed method.

Critical weaknesses
  1. DiRU preserves response structure by matching the student to a fact-ablated teacher distribution constructed from forget samples. However, since the student parameters are updated without explicit retain-data supervision, it remains unclear why this local constraint should preserve unrelated knowledge on unseen retain inputs. Although the experimental results are encouraging, the connection between structural preservation and retained knowledge preservation could be better justified.

  2. DiRU assumes that masking important query tokens sufficiently removes the target factual information from the teacher’s predictions. However, the fact-ablated teacher may still assign substantial probability to the forgotten knowledge. Moreover, the attribution mechanism identifies tokens whose probabilities increase under the original query, rather than all tokens carrying the target information. This raises concerns about whether the proposed redistribution can reliably suppress forgotten knowledge across different contexts.

Reproducibility. The paper does not appear to provide a clear commitment to releasing the implementation. Given the complexity of the proposed framework and its strong experimental results, publicly available code and evaluation scripts would be valuable for reproducibility.

Questions for authors
  1. Theorem 5.4 provides a theoretical guarantee for preserving structural tokens on forget samples. However, it is unclear whether this guarantee extends to preserving unrelated knowledge on unseen retain inputs. Could the authors clarify the connection between these two objectives, beyond the empirical utility results?

  2. How does DiRU handle cases where the fact-ablated teacher still assigns high probability to the target knowledge? Since the attribution scores depend on the probability difference between original and masked queries, could the authors clarify whether such cases may lead to incomplete forgetting? An analysis of residual factual information under different masking levels would help address this concern.

  3. Do the authors plan to release the implementation, training configurations, and evaluation scripts upon acceptance?

4 comments

We thank the reviewer for your constructive review, which have helped us improve the quality of our paper. Below, we address the concerns and questions raised in the review.

W1 & Q1. Theorem 5.4 provides a theoretical guarantee for preserving structural tokens on forget samples. However, it is unclear whether this guarantee extends to preserving unrelated knowledge on unseen retain inputs. Could the authors clarify the connection between these two objectives, beyond the empirical utility results?

Theorem 5.4 provides a theoretical guarantee for preserving structural tokens on forget samples, but it does not guarantee the preservation of unrelated knowledge on unseen retain inputs. Extending the analysis to knowledge preservation requires understanding how the model's parameter updates interact with the likelihood-score directions of the retain data.

To investigate this interaction, we conducted a small experiment examining the updates induced by LDiRU\mathcal L_{\mathrm{DiRU}} in relation to the retain-data likelihood score ∇θlog⁡pθ,ti(v∣x,yt)\nabla_\theta\log p_{\theta,t}^i(v\mid x,y_t), where (x,y0)∼Dr(x,y_0)\sim D_r. Following the TOFU setup, we use synthetic-author data as the forget dataset DfD_f and examine two retain datasets: Real Authors, which is related to the forget domain, and World Fact, which is unrelated. After 35 training steps, we measure the Fisher sensitivity of each retain dataset's predictions to the parameter updates:

World FactReal Authors
Sensitivity / Step0.0300.087

The lower sensitivity on World Fact suggests that the unlearning updates perturb its predictions less than those on Real Authors.

For the unlearned student with β=10\beta=10, we also report retention performance below. Each value is expressed as a percentage of the corresponding teacher score:

Retain datasetROUGECorrectnessRelevanceFluency
World Fact85.7%78.1%101.3%103.2%
TOFU Retain44.5%46.7%93.4%103.4%

These results suggest that structural aspects of generation, reflected in Relevance and Fluency, remain largely preserved across the evaluated retain datasets, while knowledge retention, reflected in ROUGE and Correctness, varies substantially. The findings support our view that knowledge preservation depends on how the unlearning updates affect retain-data predictions and on the relationship between the forget and retain datasets. But the theoretical guarantee of the phenomenon remains an open question, and we will explore this in future work.

W2. DiRU assumes that masking important query tokens sufficiently removes the target factual information from the teacher’s predictions. However, the fact-ablated teacher may still assign substantial probability to the forgotten knowledge. Moreover, the attribution mechanism identifies tokens whose probabilities increase under the original query, rather than all tokens carrying the target information. This raises concerns about whether the proposed redistribution can reliably suppress forgotten knowledge across different contexts.

We admit that there are cases where the fact-ablated teacher still assign substantial probability to the forgotten knowledge. We have analyzed that case in Q1. Here, we report the performance of our method DiRU under different contexts including paraphrased questions and partially revealed answers.

1. Paraphrased questions. We test the models on reworded versions of the forget-set questions and report ROUGE-L, P, fluency, relevance, and correctness.

MethodPara-ROUGE-LPara-PFluencyRelevanceCorrectness
Base0.380.049.795.879.36
Retain-only0.280.038.975.662.37
DiRU0.280.088.884.872.11

2. Partially revealed answers. We use GPT-4 to locate information targeted for forgetting within each ground-truth answer. We then randomly mask answer tokens at each ratio r∈{0,0.25,0.5,0.75}r\in\{0,0.25,0.5,0.75\} and allow the model to reconstruct the complete sequence. Each sample is evaluated eight times with independently generated masking patterns. We compute ROUGE-L over the identified positions to quantify recovery of the target information.

Methodr=0r=0r=0.25r=0.25r=0.5r=0.5r=0.75r=0.75
Base0.971.001.001.00
Retain-only0.110.280.400.48
DiRU0.120.310.440.53

Q2. How does DiRU handle cases where the fact-ablated teacher still assigns high probability to the target knowledge? Since the attribution scores depend on the probability difference between original and masked queries, could the authors clarify whether such cases may lead to incomplete forgetting? An analysis of residual factual information under different masking levels would help address this concern.

We agree that residual factual information in the fact-ablated teacher can lead to incomplete forgetting. DiRU suppresses tokens according to their attribution scores. If a target token remains highly probable under both the original and masked queries, with little difference between the two predictions, its attribution score can be close to zero and the resulting suppression weak.

To examine this behavior, we analyze two examples across response masking timesteps tt, keeping the query masking proportion fixed at K=50%K=50\%. The tables report the attribution scores st(v)s_t(v) of the target factual tokens.

Example 1: Profession of the author's father.

  • Question: What is the profession of Hsiao Yun-Hwa's father?
  • Masked question: What is the profession of [MASK] [MASK] father?
  • Ground-truth answer: Hsiao Yun-Hwa's father practices civil engineering as his profession.
Token vvs0(v)s_0(v)s0.25(v)s_{0.25}(v)s0.5(v)s_{0.5}(v)s0.75(v)s_{0.75}(v)s0.9(v)s_{0.9}(v)
civil0.0070.5214.4455.7364.560
engineering0.0020.2251.0793.0844.877

Example 2: The author's identity.

  • Question: What does Hsiao Yun-Hwa identify as in terms of gender?
  • Masked question: What does [MASK] [MASK] [MASK] as in [MASK] of gender?
  • Ground-truth answer: Hsiao Yun-Hwa is part of the LGBTQ+ community.
Token vvs0(v)s_0(v)s0.25(v)s_{0.25}(v)s0.5(v)s_{0.5}(v)s0.75(v)s_{0.75}(v)s0.9(v)s_{0.9}(v)
LGBTQ0.0000.0000.0040.0411.115
community0.0000.0000.0020.2103.979

These examples illustrate that the attribution signal depends on both the target fact and the masking timestep. In Example 1, the scores are near zero at t=0t=0 but become substantially larger at higher noise levels. In Example 2, the scores remain near zero through t=0.5t=0.5 and are still relatively small at t=0.75t=0.75, before increasing at t=0.9t=0.9.

We interpret this behavior as a consequence of overfitting during DLM training: the model can reconstruct memorized factual information from limited context, even at relatively high noise levels such as t=0.75t=0.75. Consequently, query masking may leave substantial factual information in the teacher's predictions. When query ablation produces little change in the probability of a target token, attribution-based suppression can be insufficient, and DiRU does not guarantee complete forgetting in these cases.

Q3. Do the authors plan to release the implementation, training configurations, and evaluation scripts upon acceptance?

Yes. We are committed to ensuring the reproducibility and transparency of our work. Upon acceptance, we will publicly release the complete implementation of DiRU, including the training configurations, hyperparameter settings, data preprocessing pipelines, and evaluation scripts. We will also provide detailed instructions for reproducing the main experimental results, including the baseline configurations and evaluation protocols used in our comparisons. We are pleased to share these resources to support and encourage further research on diffusion language model unlearning.

Dear Reviewer 6znC,

Thank you again for your time and constructive feedback. We have carefully considered your comments and provided additional clarifications and experimental evidence in our latest responses and revised manuscript.

As the discussion period closes on November 18, Anywhere on Earth, we would be grateful if you could review these updates. Should any questions or concerns remain, we would be happy to address them before the deadline.

We hope the additional evidence addresses your concerns and may help inform your final assessment of our submission.

Sincerely,

Reviewer_6znCOpenReview ↗

Keep weak accept

Dear authors,

Thank you for your effort. You response make me no more confused. I keep my decision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.