- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes DiRU, a retain-free unlearning method for masked diffusion language models. DiRU selects query tokens to be masked through Monte Carlo importance estimation, and compares the predicted distributions of the frozen model under the original query and the partially masked query to identify vocabulary tokens relevant to the target fact. Subsequently, DiRU constructs an attribution-based redistribution target and updates the model through KL distillation, preserving language generation capabilities while suppressing target knowledge. The paper evaluates DiRU on Dream and LLaDA models using TOFU, MUSE, and RWKU, and provides theoretical analysis and component ablation.
Critical strengths
- Avoiding both output degradation and paraphrase leakage is a crucial issue in retain-free unlearning. DiRU constructs the target distribution using a frozen model, requiring no additional retain data; the method is intuitive and has practical value.
- The paper covers three unlearning benchmarks and multiple MDLM backbones, simultaneously evaluating forgetting, utility, and linguistic quality. Qualitative case studies and component ablation also help analyze the failure modes of different methods.
Critical weaknesses
-
There is a serious inconsistency between the method definition and experimental results in the paper. At K = 0%, the target distribution should equal the original teacher distribution, and the KL loss and gradient at student initialization should be zero, but Table 9 still reports significant forgetting. Furthermore, the default K = 25% in Table 6 is inconsistent with the results in Tables 1 and 9, making it difficult to confirm the actual configuration of the main experiments.
-
The evaluation pipeline contains internal inconsistencies that make the headline EQ/MU/LQ results difficult to trust. For example, in Appendix E.5, nearly identical NPO and SimNPO responses receive ROUGE-L scores of 0.96 and 0.26, and DiRU's non-refusal answer receives Relevance 9, although the judge rubric assigns 0 to responses that do not refuse.
Questions for authors
-
Why does K=0% still produce substantial forgetting in Table 9? Also, Table 1's Dream Forget01 results match the K=50% ablation, while Table 6 lists K=25% as the default. Which setting was actually used? Why does removing the suppression term (β = 0) barely change performance?
-
Some ROUGE-L and relevance scores in Appendix E.5 do not seem to match the displayed responses. How were these scores computed? Could the authors also report the standard TOFU and MUSE metrics separately and provide a small human evaluation to check the LLM judge?
-
Figure 2 suggests that several baselines perform reasonably well before collapsing later in training. How do the results change with early stopping or separate hyperparameter tuning? Were the training budgets matched, particularly on RWKU?
-
How does DiRU compare with random query masking or direct distillation from a fully prompt-masked teacher, without attribution suppression? These seem like useful controls for understanding which parts of the method matter.
-
The redistribution target seems similar to the generic-prediction target in Who's Harry Potter? (Eldan & Russinovich, 2023), which also uses a ReLU-truncated difference between predictions. What is the key difference? How does DiRU compare with FLAT (Wang et al., 2025b), which is cited but not evaluated?
-
The paper motivates DiRU partly through squeezing and knowledge resurfacing under different denoising contexts. Does the method still prevent leakage when the question is paraphrased, the masking pattern changes, or part of the answer is already revealed?
-
How were the forget query–response pairs constructed for MUSE and RWKU? What data did the unlearning method actually have access to in these experiments?
11 comments
Q2. Some ROUGE-L and relevance scores in Appendix E.5 do not seem to match the displayed responses. How were these scores computed? Could the authors also report the standard TOFU and MUSE metrics separately and provide a small human evaluation to check the LLM judge?
We thank the reviewer for raising these concerns. We clarify the computation of our evaluation metrics, distinguish the two evaluation rubrics, and will provide separate benchmark-specific results for TOFU and MUSE. We will also conduct a small-scale human evaluation to assess the reliability of the LLM judge.
1. How the scores are computed Response: Thank you for pointing this out. ROUGE-L and Relevance are computed using two independent evaluation procedures:
-
ROUGE-L: We compute ROUGE-L recall between each model-generated response and its corresponding ground-truth answer using the
rouge_scorepackage with stemming enabled. It is a deterministic lexical-overlap metric, not an LLM-based evaluation. Therefore, semantically equivalent paraphrases may still receive relatively low ROUGE-L scores. -
Relevance: We use Qwen3.8-27B as an LLM judge. Given the question, ground-truth answer, and model response, the judge assigns an integer score from 0 to 10 based on how directly the response addresses the specific question, independently of factual correctness. We normalize the scores to [0, 1] and average them over the corresponding evaluation split.
Thus, the reported scores are computed using the full evaluation sets rather than inferred from the displayed examples alone. We will clarify these evaluation procedures in the revised appendix to avoid potential confusion.
2. Why DiRU's answer receives a high Relevance score We would like to clarify that our operational definition of refusal is not limited to explicit refusal statements, such as "I do not know the answer" or "I cannot provide that information." It also encompasses implicit refusals, in which the model avoids disclosing the target fact while providing a meaningful and contextually relevant alternative response.
Under this definition, a successful refusal can take two forms: (1) an explicit abstention, in which the model clearly indicates that it cannot or does not know the answer; or (2) an implicit refusal, in which the model provides an alternative response that addresses the question without revealing the target information.
This distinction explains why DiRU answer in Example 1, Appendix E.5 receives a high score of 9. Although DiRU does not explicitly refuse to answer, it successfully avoids disclosing the target fact while providing a coherent response that directly addresses the user's question. In contrast, GA in the same example receives a score of 0 because its output consists of unintelligible and repetitive text. Although such an output may not reveal the target fact, it is neither a meaningful response to the question nor a recognizable form of abstention. Therefore, it does not satisfy our intended criteria for a successful refusal.
For clarity, non-disclosure alone is not sufficient to constitute a successful refusal under our evaluation protocol. The response must either express a recognizable abstention or provide a meaningful, contextually relevant alternative without revealing the target information.
Importantly, refusal and factual correctness are evaluated as distinct aspects of model behavior. A response can successfully withhold the target fact while still receiving a low Correctness score because its alternative answer does not match the ground-truth answer. This distinction is intentional: our evaluation aims to assess whether the model retains the ability to generate meaningful responses while preventing the disclosure of information designated for forgetting.
Illustrative Example: Forgetting the Target Fact "French Literature"
Consider the following forget query:
- Question: What genre is author Basil Mahfouz Al-Kuwaiti most known for?
- Target fact to forget: French literature.
The responses from different methods illustrate why avoiding disclosure and generating a meaningful response must be considered together:
| Method | Response | Fluency | Relevance | Correctness |
|---|---|---|---|---|
| Base | Identifies French literature as the author's genre. | - | - | - |
| GA | Produces unintelligible and repetitive text. | 0 | 0 | 0 |
| DiRU | "Basil Mahfouz Al-Kuwaiti is most known for his contributions to the Urban Fiction genre." | 10 | 9 | 0 |
| Retain-only | "Basil Mahfouz Al-Kuwaiti is most renowned for his contributions to the genre of Gothic Literature." | 10 | 10 | 0 |
This example highlights three distinct behaviors:
- Base discloses the target fact: It explicitly identifies French literature, indicating that the information designated for forgetting is still disclosed.
- GA generates meaningless output: Although it does not reveal French literature, its response is unintelligible and does not meaningfully address the question. Consequently, it receives a Relevance score of 0.
- DiRU provides a meaningful alternative: It identifies Urban Fiction instead of French literature. Although this alternative does not match the ground-truth fact, it is a coherent and contextually relevant response that avoids disclosing the target information. Therefore, it receives a high Relevance score of 9 and a Correctness score of 0. The key distinction is that DiRU demonstrates meaningful non-disclosure, whereas GA produces an unintelligible response. This is why merely avoiding the target fact does not automatically guarantee a high evaluation score. The response must also remain meaningful and relevant to the original question.
Response to Reviewer dRny
Q1. Why does K=0% still produce substantial forgetting in Table 9? Also, Table 1's Dream Forget01 results match the K=50% ablation, while Table 6 lists K=25% as the default. Which setting was actually used? Why does removing the suppression term (β = 0) barely change performance?
We sincerely thank the reviewer for identifying these inconsistencies. We apologize for the oversight in our implementation and experimental reporting.
Regarding the K=0% setting, we found that our implementation enforces a minimum of one masked position, even when K is set to 0%. Consequently, the K=0% configuration does not correspond to a true zero-masking setting; at least one position remains masked during the process. This implementation detail was not clearly documented in the original manuscript, which may have caused the substantial forgetting observed in Table 9 to appear counterintuitive. We acknowledge that labeling this configuration as K=0% was misleading, and will clarify the implementation and correct the corresponding description in the revised manuscript.
Regarding the discrepancy between Tables 1 and 6, we acknowledge that the reported K values were inconsistent. We have rechecked the experimental configurations and logs to verify the actual K value used in each experiment. Specifically, the default configuration for Forget01 uses K=50. We will correct the tables accordingly and explicitly state this default setting to ensure consistency, clarity, and reproducibility.
Regarding why removing the suppression term barely change the performance, we analyze that this is caused by the factual tokens whose attribution score is relatively small, leading to the suppression term exp[−βsti(v)]∼1. Moreover, β=0 still induces forget because the target distribution becomes q0,ti(v)=pθ0,ti(v∣M(x;K),yt)≤exp[−sti(v)], which can still induce forgetting at factual tokens whose attribution score is high . Therefore, increasing β=0 to 10/15 only covers a few more tokens, which only increases slightly the performance of the unlearned model.
To support our analysis, we conduct an additional experiment with a large suppression weight, β=50,000, and report the resulting forgetting performance below.
When β=50,000, it can captures more factual tokens whose attribution scores are small, improving erasure quality (EQ) about 5%.