- Rating
- 3(+2)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+2)
Read review
Paper summary
This paper introduces a post-unlearning fine-tuning method designed to improve robustness against knowledge recovery through subsequent fine-tuning while preserving model plasticity. UMU can be integrated into existing unlearning methods as a plug-in enhancement. The proposed approach is evaluated across multiple unlearning methods and unlearning benchmarks, demonstrating improvements in relearning resistance and downstream learning performance.
Critical strengths
- Great presentation: the paper is clear and well-written.
- The method is flexible and well-motivated. We want to make sure that the unlearned model can still be adapted (via fine-tuning) to downstream tasks, while the undesired data/knowledge remains forgotten. The proposed method can be applied to any unlearned model.
- Several unlearning algorithms and benchmarks are considered in the experiments.
- Ablation studies clearly show how the two components (replay and selection) are complimentary.
Critical weaknesses
- Lack of direct comparison with previous robust LLM unlearning methods under matched compute. Methods such as Invariance-based LLM Unlearning (ILU), TAR, or Sharpness-Aware unlearning are mentioned but not included in the experiments for comparison. While the setting of UMU differs from theirs, both UMU and previous approaches use additional compute (in terms of GPU hours) compared to the base unlearning algorithms. Additionally, I fail to see any computation cost evaluation of UMU.
- Model plasticity is evaluated only on a single task, i.e., AGNews fine-tuning. A solid empirical validation should consider multiple datasets and a more diverse task setting, such as math, truthfulness, instruction following, and reasoning.
- For evaluating unlearning robustness against weight tampering, only benign fine-tuning (via LoRA) is considered. Additional attacks like relearning and full-parameter fine-tuning could strengthen the empirical validations.
Questions for authors
- Are there any reasons why experiments are not conducted on the WMDP benchmark?
- What is the relation between UMU and continual unlearning for LLMs? Suppose we have multiple learning and unlearning requests, can UMU work in this setting?
- The NPO algorithm is not included in the paper. However, it seems to me that NPO is an improved version of GradDiff and performs quite well on multiple unlearning benchmarks. Are there any reasons why NPO is not integrated with UMU?
6 comments
Thanks for the authors' response. My previous concerns were mostly about empirical validations. I will consider raising my score if the above changes are reflected in the manuscript.
Thank you very much for your positive assessment and support of our work. In the revised manuscript PDF, the revisions and clarifications responding to your comments are highlighted in blue for ease of reference. We sincerely appreciate your helpful feedback and the time you have devoted to reviewing our paper. Thank you again.
W1 + Q3. Additional baselines, NPO integration and GPU time
Thank you for recommending stronger baselines, NPO integration and computational-cost reporting. General Response, item (1) reports the NPO, NPO+SAM and NPO+UMU comparisons on both benchmarks and backbones. NPO+UMU lowers harmful and benign recovery in all four settings and improves AGNews NS over NPO by 1.82–3.16 points. On FaithUn with Qwen, release-state FS remains 0.0, harmful recovery falls from 8.92/9.61 to 4.72/8.31, benign recovery falls from 8.37 to 3.42, and NS rises from 85.07 to 86.89. Thus, UMU improves resistance after base unlearning has suppressed the target at release, alongside better measured learning. We will add these comparisons to Section 5.2 and document the NPO/SAM implementations and GPU-hours on the same hardware in Appendix C, including training, tuning.
W2. Adaptation beyond AGNews
Thank you for requesting learning evidence beyond AGNews. We added fine-tuning evaluations on SST-2, QQP and GSM8K using the released checkpoints; General Response, item (1) reports the complete results. All 48 additional-task comparisons improve after adding UMU. For example, on FaithUn with Qwen, adding UMU to RMU raises SST-2 from 90.94 to 94.04, QQP from 85.46 to 90.50, and GSM8K from 71.04 to 76.72. The 5.68-point GSM8K gain shows that the observed benefit also appears in mathematical reasoning. Together with improvements across both backbones and all four base methods, these results support learning gains beyond AGNews. We will add the per-task comparisons and analysis to Section 5.2, and describe the task datasets, fine-tuning settings and separation from selection in Appendix D.
W3. Targeted relearning and stronger weight attacks
Thank you for requesting stronger weight attacks. The original evaluation includes targeted relearning on 10% or 30% of the eligible forget pool and benign tuning on 1,000 Alpaca instructions, optimized anew against each released checkpoint. We also completed full-parameter attacks on TOFU Forget01 with Llama-3.2-3B; General Response, item (4) reports the results. UMU lowers harmful and benign recovery for all four base methods; for GD, harmful recovery falls from 59.03/67.25 to 39.11/44.55. We will insert the comparison figure and analyze recovery in Section 5.2, and document the trainable parameters, attack budgets, stopping rule and aggregation in Appendix D.
Q1. WMDP in the benchmark evaluation
Thank you for suggesting WMDP. We have added WMDP evaluation with Llama-3.2-3B, comparing RMU and RMU+UMU; General Response, item (5) reports the benchmark results. Harmful recovery falls from 56.84/57.92 to 38.42/40.93, while AGNews NS rises from 79.69 to 84.63. We will insert the WMDP/MUSE comparison figure and result analysis in Appendix E, cite it from Section 5.2, and define the benchmark-specific unlearning and recovery metrics in Appendix D.
Q2. Repeated requests and continual unlearning
Thank you for this question. Our study focuses on resisting knowledge recovery under subsequent fine-tuning while preserving learning ability after base unlearning. Continual unlearning concerns successive deletion requests, possibly interleaved with learning. UMU is a defense stage that can complement this process, rather than a complete continual-unlearning framework.