acceptodds
Under review as a conference paper at ICLR 2027

Not All Change Is Damage: Update-Matched Unlearning for Large Language Models

Abstract

Unlearning aims to remove selected knowledge from large language models to mitigate privacy and safety risks while preserving useful capabilities. For models that continue to be fine-tuned after release, this goal is insufficient: they must also resist relearning the deleted knowledge while preserving plasticity, the ability to learn new tasks. Existing defenses train against simulated relearning while trying to preserve useful capabilities, but low relearning alone cannot distinguish selective resistance from a general loss of plasticity. Evaluating preservation after an update requires deciding what the updated model should remain close to. A pre-update reference penalizes useful changes induced by the update simply because they depart from earlier outputs. We therefore formulate the reference-timing hypothesis that, at comparable target suppression, an update-matched reference better preserves plasticity than a pre-update one. To test and address this hypothesis, we propose Update-Matched Unlearning (UMU), which applies the same simulated relearning update to the defended model and to an undefended reference initialized from the same unlearned model, and uses new-task learning to guide model selection. Extensive experiments on TOFU and FaithUn with Llama and Qwen models show that UMU improves both relearning resistance and plasticity across all three base unlearning methods, lowering harmful and benign relearning by 30.4 and 9.9 points on average while raising new-task accuracy by 3.5 points. When the reference does not receive the update, relearning remains similarly low but new-task accuracy falls below the base method, supporting the reference-timing hypothesis and showing that robust unlearning should prevent recovery selectively without suppressing learning itself.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
machine unlearning, large language models, relearning resistance, post-unlearning fine-tuning, new task learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2HRkOpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+2)
Read review
Paper summary

This paper introduces a post-unlearning fine-tuning method designed to improve robustness against knowledge recovery through subsequent fine-tuning while preserving model plasticity. UMU can be integrated into existing unlearning methods as a plug-in enhancement. The proposed approach is evaluated across multiple unlearning methods and unlearning benchmarks, demonstrating improvements in relearning resistance and downstream learning performance.

Critical strengths
  1. Great presentation: the paper is clear and well-written.
  2. The method is flexible and well-motivated. We want to make sure that the unlearned model can still be adapted (via fine-tuning) to downstream tasks, while the undesired data/knowledge remains forgotten. The proposed method can be applied to any unlearned model.
  3. Several unlearning algorithms and benchmarks are considered in the experiments.
  4. Ablation studies clearly show how the two components (replay and selection) are complimentary.
Critical weaknesses
  1. Lack of direct comparison with previous robust LLM unlearning methods under matched compute. Methods such as Invariance-based LLM Unlearning (ILU), TAR, or Sharpness-Aware unlearning are mentioned but not included in the experiments for comparison. While the setting of UMU differs from theirs, both UMU and previous approaches use additional compute (in terms of GPU hours) compared to the base unlearning algorithms. Additionally, I fail to see any computation cost evaluation of UMU.
  2. Model plasticity is evaluated only on a single task, i.e., AGNews fine-tuning. A solid empirical validation should consider multiple datasets and a more diverse task setting, such as math, truthfulness, instruction following, and reasoning.
  3. For evaluating unlearning robustness against weight tampering, only benign fine-tuning (via LoRA) is considered. Additional attacks like relearning and full-parameter fine-tuning could strengthen the empirical validations.
Questions for authors
  1. Are there any reasons why experiments are not conducted on the WMDP benchmark?
  2. What is the relation between UMU and continual unlearning for LLMs? Suppose we have multiple learning and unlearning requests, can UMU work in this setting?
  3. The NPO algorithm is not included in the paper. However, it seems to me that NPO is an improved version of GradDiff and performs quite well on multiple unlearning benchmarks. Are there any reasons why NPO is not integrated with UMU?
6 comments

W1 + Q3. Additional baselines, NPO integration and GPU time

Thank you for recommending stronger baselines, NPO integration and computational-cost reporting. General Response, item (1) reports the NPO, NPO+SAM and NPO+UMU comparisons on both benchmarks and backbones. NPO+UMU lowers harmful and benign recovery in all four settings and improves AGNews NS over NPO by 1.82–3.16 points. On FaithUn with Qwen, release-state FS remains 0.0, harmful recovery falls from 8.92/9.61 to 4.72/8.31, benign recovery falls from 8.37 to 3.42, and NS rises from 85.07 to 86.89. Thus, UMU improves resistance after base unlearning has suppressed the target at release, alongside better measured learning. We will add these comparisons to Section 5.2 and document the NPO/SAM implementations and GPU-hours on the same hardware in Appendix C, including training, tuning.

W2. Adaptation beyond AGNews

Thank you for requesting learning evidence beyond AGNews. We added fine-tuning evaluations on SST-2, QQP and GSM8K using the released checkpoints; General Response, item (1) reports the complete results. All 48 additional-task comparisons improve after adding UMU. For example, on FaithUn with Qwen, adding UMU to RMU raises SST-2 from 90.94 to 94.04, QQP from 85.46 to 90.50, and GSM8K from 71.04 to 76.72. The 5.68-point GSM8K gain shows that the observed benefit also appears in mathematical reasoning. Together with improvements across both backbones and all four base methods, these results support learning gains beyond AGNews. We will add the per-task comparisons and analysis to Section 5.2, and describe the task datasets, fine-tuning settings and separation from selection in Appendix D.

W3. Targeted relearning and stronger weight attacks

Thank you for requesting stronger weight attacks. The original evaluation includes targeted relearning on 10% or 30% of the eligible forget pool and benign tuning on 1,000 Alpaca instructions, optimized anew against each released checkpoint. We also completed full-parameter attacks on TOFU Forget01 with Llama-3.2-3B; General Response, item (4) reports the results. UMU lowers harmful and benign recovery for all four base methods; for GD, harmful recovery falls from 59.03/67.25 to 39.11/44.55. We will insert the comparison figure and analyze recovery in Section 5.2, and document the trainable parameters, attack budgets, stopping rule and aggregation in Appendix D.

Q1. WMDP in the benchmark evaluation

Thank you for suggesting WMDP. We have added WMDP evaluation with Llama-3.2-3B, comparing RMU and RMU+UMU; General Response, item (5) reports the benchmark results. Harmful recovery falls from 56.84/57.92 to 38.42/40.93, while AGNews NS rises from 79.69 to 84.63. We will insert the WMDP/MUSE comparison figure and result analysis in Appendix E, cite it from Section 5.2, and define the benchmark-specific unlearning and recovery metrics in Appendix D.

Q2. Repeated requests and continual unlearning

Thank you for this question. Our study focuses on resisting knowledge recovery under subsequent fine-tuning while preserving learning ability after base unlearning. Continual unlearning concerns successive deletion requests, possibly interleaved with learning. UMU is a defense stage that can complement this process, rather than a complete continual-unlearning framework.

Thanks for the authors' response. My previous concerns were mostly about empirical validations. I will consider raising my score if the above changes are reflected in the manuscript.

Thank you very much for your positive assessment and support of our work. In the revised manuscript PDF, the revisions and clarifications responding to your comments are highlighted in blue for ease of reference. We sincerely appreciate your helpful feedback and the time you have devoted to reviewing our paper. Thank you again.

Reviewer_VopzOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the problem of robustness of unlearning models against fine-tuning, in the sense that the unlearned model should robustly prevent forgotten knowledge from being recovered via subsequent fine-tuning and preserve the unlearned model's ability to learn new tasks. The paper proposes Update-Matched Unlearning, a robust unlearning method against relearning attacks that preserves the model's ability to learn new tasks. The paper's claims are established based on experiments with llama-3.2-3B and qwen-3-4B, two unlearning benchmarks (TOFU and FaithUn), three baseline methods (GD, RMU, KLUE), and one new task (AGNews).

Critical strengths
  • The intuition that defending against relearning attacks may inadvertently reduce the ability of unlearned models to learn new tasks/knowledge is somehow worth deeper investigation.
Critical weaknesses
  • The paper is difficult to digest on a first reading: several non-standard terms/jargons are used before being clearly defined. For example, terms such as "A pre-update reference", "reference-timing hypothesis," "update-matched replay," and "selection by learning trials".
  • The paper studies knowledge unlearning, but two well-known, commonly used benchmarks, i.e., WMDP and MUSE are not tested.
  • Insufficient comparisons with existing (robust) unlearning methods: The experimental evaluation lacks comparisons with relevant robust unlearning methods. For example, the related work discusses methods such as TAR, but the experiments do not include direct comparisons with these approaches. If direct comparisons are inappropriate, the authors should clearly explain why. Further, missing a broad range of unlearning baselines (GA, NPO, SimNPO, DPO etc.).
  • Overclaiming and a mismatch between the title, framing, and evidence: The title and framing are uninformative and mismatched with the paper's content. The claims are also broader and overgeneralize what the paper's evidence establishes. For example, the evaluation is limited to two models, Llama and Qwen (both 3B), and LoRA-based relearning attacks; the "post-unlearnin learning" evaluation only uses AGNews. A single task is insufficient to establish that UMU generally improves a model's ability to learn new tasks after unlearning.
Questions for authors

I think the paper needs major revision. I suggest that the senior authors of this paper carefully revise it before resubmitting.

3 comments

W1. Existing operational definitions and clearer presentation

Thank you for this comment. We study robust unlearning for models that must continue learning after release, since low recovery of deleted knowledge may also result from a loss of learning ability. We propose Update-Matched Unlearning (UMU), which uses simulated relearning attacks to strengthen defense. During training, we record a relearning update on a copy of the student and replay it on both the student and the reference teacher through our reference-timing mechanism. The updated teacher guides the student on retained data while the forget loss suppresses target recovery. Learning trials then select a student that can still learn a new task.

The manuscript already provides definitions to help readers unfamiliar with unlearning follow the problem and method. For example, the first paragraph of Section 1 (Introduction) introduces unlearning. It defines plasticity as the ability to learn new tasks and relearning as recovery of deleted knowledge through subsequent fine-tuning. The third paragraph explains pre-update and update-matched references and states the reference-timing hypothesis. Sections 4.1 and 4.2 define replay, the teacher and student roles, and selection by learning trials. We will further clarify these definitions at first use and connect each term more directly to its operation.

W2. WMDP and MUSE coverage

Thank you for requesting these benchmarks. We have added WMDP and MUSE evaluations with Llama-3.2-3B, comparing RMU with RMU+UMU. On WMDP, harmful recovery falls from 56.84 to 38.42 when retraining uses 10% of the forget set, and from 57.92 to 40.93 when it uses 30%. On MUSE, harmful recovery falls from 35.16 to 18.02 at 10% and from 37.28 to 20.64 at 30%. AGNews NS rises from 79.69 to 84.63 on WMDP and from 77.86 to 82.04 on MUSE. We will insert the comparison figure and analysis in Appendix E, cite them from Section 5.2, and define each benchmark's unlearning and recovery metrics in Appendix D.

W3. Robust defenses and broader base objectives

Thank you for recommending broader baseline comparisons. We have added NPO, NPO+SAM and NPO+UMU on TOFU and FaithUn with Llama-3.2-3B and Qwen3-4B, as reported in General Response, item (1). NPO extends the evaluated base objectives, while NPO+SAM provides a sharpness-aware comparison using the same base objective. SAM trains against nearby weight perturbations to encourage a flatter loss landscape. UMU applies a recorded relearning update to both the defended model and its reference, then checks downstream learning during checkpoint selection. Across all four settings, NPO+UMU lowers harmful recovery at both attack proportions and benign recovery relative to both NPO and NPO+SAM, and increases AGNews NS over NPO by 1.82 to 3.16 points. We will report these comparisons in Section 5.2, explain SAM and its relation to UMU in Section 2, and document the NPO and SAM implementations and tuning budgets in Appendix C.

W4. Title, model coverage and the learning claim

Thank you for raising this concern. We respectfully clarify that the title and learning claim are tied to the proposed mechanism and the reported experimental settings. “Not All Change Is Damage” refers to the distinction explained in Sections 1 and 4: UMU replays the same recorded update on the defended model and its reference to avoid penalizing useful changes on retained data, and uses learning trials to select a checkpoint that can still adapt. Section 5.1 already distinguishes new-task learning from static utility and specifies Llama-3.2-3B-Instruct and Qwen3-4B, while Section 6 explicitly lists the original low-rank attacks as a limitation. We have now added SST-2, QQP and GSM8K, with learning improvements in all 48 paired comparisons, and full-parameter attacks on TOFU Forget01 with Llama-3.2-3B, where UMU lowers harmful and benign recovery for all four base methods. We will clarify the connection between the title, mechanism and experimental scope in the abstract and Section 1, report the added results in Section 5.2, and document the protocols in Appendix D.

As the rebuttal deadline is approaching, we would like to let you know that we have uploaded the revised manuscript PDF. Changes made in response to the reviewers' comments are highlighted in blue. If any questions or concerns remain, we welcome further discussion and would be happy to provide additional clarification. Thank you for your time and thoughtful feedback.

As the rebuttal deadline is approaching, we would like to kindly follow up and ask if there are any remaining questions or concerns. We would be happy to provide further clarification. If our responses have adequately addressed your concerns, we would greatly appreciate your consideration of updating the score.

Reviewer_CuLzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the issue of robust unlearning under continued fine-tuning, and the question is precisely framed: does the extremely low knowledge recovery rate after fine-tuning indicate that the attack has been successfully defended against, or that the model has been thoroughly "broken" and has completely lost its capacity to learn? The authors point out that the conventional retention loss forcibly pulls the updated student model toward the pre-update reference model, indiscriminately eliminating the parameter updates arising from normal learning (i.e., the reference-timing hypothesis). To this end, the paper proposes UMU: a relearning rehearsal is first performed on a copy of the student model, and the resulting update is then applied to both the student and the reference models; the forget loss and KL divergence are computed between their "post-update states"; and finally, short-term fine-tuning on a new task is used to select the checkpoint that will ultimately be released. Drawing on local gradient analysis, the authors conduct verification on the TOFU and FaithUn datasets (covering two model families and three base unlearning methods), examining target data recovery, benign fine-tuning recovery, and AGNews new-task learning. The results show that UMU achieves both resistance to relearning and the ability to learn new tasks; whereas the ablation variant without teacher update replay yields an even lower recovery rate but a substantially diminished capacity for new-task learning.

Critical strengths
  1. The starting point is solid, and a vague pain point has been turned into a verifiable optimization mechanism. Rather than simply chasing lower recovery rates, the authors ask which part of the parameter updates the defense mechanism actually suppresses. By contrasting the gradient differences between the two reference targets and, together with the gradient direction and single-step loss changes, making a diagnosis, the paper has made "reference timing" a design dimension that can be examined under controlled variables, rather than a mere post-hoc, figure-based explanation. Even if the methodological change itself is simple, so long as the independent contribution of this mechanism is clearly established, it will be of great value for subsequent work.

  2. The ablation experiments intuitively expose the hidden trade-off between "relearning resistance" and "plasticity." Take TOFU–Llama–GD, for example: after removing update replay, the relearning attack score is 23.89, which appears slightly better than the 24.98 of full UMU; but its new-task learning score (NS) drops to 78.91, far below the 85.16 of full UMU and even lower than that of base GD (82.03). This stark contrast is highly compelling and directly demonstrates that looking only at the relearning recovery metric is misleading, strongly supporting the "dual evaluation" perspective advocated by the authors.

Critical weaknesses
  1. The mechanistic benefit and the benefit of model selection are conflated, and the conclusion about "general plasticity" lacks cross-task support. The experiments use the fine-tuning performance on AGNews to select the checkpoint, and then measure new-task learning capacity (NS) on AGNews. Although the training and test samples are non-overlapping and there is no data leakage, "sample independence" does not imply "task independence." The current experiments can only show that the selected model is better at learning AGNews; they cannot show that it has preserved general downstream plasticity. Moreover, the improvement attributable to model selection (checkpoint selection) itself is by far the largest. On TOFU–Llama–GD, the NS of full UMU is 3.13 percentage points higher than that of GD; but once the selection step is removed, the increase is only 0.39 percentage points. This indicates that most of the apparent benefit of UMU actually comes from selection, or from the coupling between selection and reference timing, and cannot all be attributed to "reference timing." In addition, the static reference baseline has not been adequately tuned. As noted in the text, the retention terms for TAR and AntiDote are computed on models that have not undergone simulated updates, which is not the same as "aligning the updated student to the pre-update teacher" in the ablation experiments. If the weight λR\lambda_R of the static reference is not carefully tuned, or if a direct comparison with the "unupdated student" is not made, it is hard to be convinced that the update-matched reference is necessarily superior.

  2. There are arithmetic contradictions in the values of the core metric NS, and parts of the main-text descriptions do not match the tabular data. The appendix states that NS is the accuracy computed on the full AGNews test set of 7,600 examples, yet the table contains a figure of 83.98%. If this is from a single full test, rounded to two decimal places: 63827600×100≈83.97%,63837600×100≈83.99%\frac{6382}{7600} \times 100 \approx 83.97\%, \qquad \frac{6383}{7600} \times 100 \approx 83.99\% No matter how many correct answers there are, dividing by 7600 can never yield 83.98%. If the value is an average over multiple runs, this is not mentioned anywhere in the text (the paper only states that the attack experiments were averaged over three runs). Furthermore, the main text says that the forgetting score (FS) of GD increased in 9 out of 10 intervals, but the specific figures in Table 3 actually show 3 decreases (i.e., only 7 increases). For a key metric that underpins the core conclusions of the entire paper, such basic data inconsistencies and unclear criteria make it very difficult to have confidence in the experimental results.

Questions for authors
  1. Criteria for computing NS and consistency between the text and the figures: Is the NS score in the table from a single run or an average over multiple runs? If it is a single test on 7,600 examples, how is 83.98% obtained? Please provide the specific number of runs, random seeds, the number of correct examples in each run, and the aggregation method. In addition, please verify whether the trend descriptions in Figure 4, Table 3, and the main text all come from the same set of experimental logs (why does the text say 9 increases while Table 3 shows only 7)?
  2. Details of model selection in the ablation experiments: For full UMU and the no-replay variant (w/o replay), at the time of selection, are the size of the candidate pool, the number of fine-tuning steps, the admission threshold, and the final selection rule all identical? How many checkpoints passed the screening in each case? How exactly is the final checkpoint determined for the "w/o selection" variant?
  3. Cross-task generalization verification: If the current AGNews-based selection procedure is kept unchanged, and the selected model is directly fine-tuned on a completely unseen downstream task (not involved in any selection or hyperparameter tuning), does UMU still have an advantage over the baselines and the static reference?
  4. Fair comparison with the static reference and related baselines: Under unified selection rules and with the target degree of unlearning being comparable, if the weight λR\lambda_R of the static reference is carefully tuned, or if a direct comparison with "computing the retention term on the unupdated student model" is made, does UMU's update-matched reference mechanism still have a clear advantage? Moreover, since the paper lists ILU as a related work that similarly focuses on subsequent learning capacity, can a direct comparison between the two be provided, or can it be clearly explained why a direct comparison is not possible?
  5. Minor: The "steps" label at the far right end of the x-axis in Figure 2c is partially cropped. Please pay attention to adjusting the image boundaries when making corrections.
4 comments

Thank you for the detailed assessment of reference timing and the resistance–plasticity trade-off. Below we report the replay/selection ablation, the static-reference weight sweep and the NPO+SAM comparison. We also describe the completed experiments with additional random seeds and the status of the ILU comparison.

W1 (mechanism/selection) + Q2. Reference timing and checkpoint selection

Thank you for asking us to separate replay and selection. General Response, item (2) reports four replay/selection variants on both benchmarks and backbones. A disables both reference update replay and selection by learning trials; B enables selection only; C enables reference update replay only; D enables both, giving full UMU. All four retain the defense-training stage, with GD as the base-unlearning control. On TOFU with Llama, NS is 76.17, 78.91, 82.42 and 85.16 for A, B, C and D, respectively. Enabling replay raises NS by 6.25 percentage points both without selection (C−A) and with selection (D−B). Enabling selection raises NS by 2.74 points both without replay (B−A) and with replay (D−C). Across the four settings, the corresponding NS gains range from 5.07 to 6.25 points for replay and from 1.94 to 2.74 points for selection. The recovery scores show why learning should be assessed alongside resistance. In the same TOFU–Llama setting, removing replay lowers the harmful attack score at p=0.1p=0.1 from 24.98 to 23.89, while NS falls from 85.16 to 78.91. Full UMU still reduces this attack score substantially relative to GD (45.41), while achieving higher NS (85.16 versus 82.03). Thus, the variant with lower recovery is not necessarily the one with better new-task learning, and the four variants show distinct improvements in learning from replay and selection. We will expand Section 5.3 to describe these four variants, report the corresponding NS differences for replay and selection separately, and compare recovery scores alongside new-task learning.

W2 + Q1. NS arithmetic, aggregation and text–table consistency

Thank you for pointing out the ambiguity in NS aggregation. We have completed additional experiments with different random seeds. We will report their total run counts, individual seeds, means and standard deviations. For NS, we will also provide the correct and tested counts and unrounded accuracy for each run, and specify the aggregation procedure for NS, FS and RS in Appendix D and the relevant table captions. We will document the original evaluation counts and aggregation behind the 83.98% entry separately from the additional repetitions, and reconcile that entry with the original evaluation records. We will correct the GD FS trend description and add the corresponding clarification in Section 5.4 and Appendix E.

W1 (cross-task evidence) + Q3. Transfer with AGNews selection frozen

Thank you for requesting cross-task evidence. We added fine-tuning evaluations on SST-2, QQP and GSM8K using the released checkpoints; General Response, item (1) reports the results across both benchmarks and backbones. Adding UMU improves all 48 comparisons with the four base methods on these three additional tasks. For example, on FaithUn with Qwen, adding UMU to RMU raises SST-2 from 90.94 to 94.04, QQP from 85.46 to 90.50, and GSM8K from 71.04 to 76.72. These results support learning gains beyond AGNews across the evaluated sentiment classification, paraphrase matching and mathematical reasoning tasks. We will add the per-task results and analysis to Section 5.2, and describe the task datasets, fine-tuning settings and their separation from checkpoint selection in Appendix D. We apologize that, within the available time, we have not yet completed evaluations on additional model families and larger models. We are submitting the available results now to provide a timely response. UMU's formulation applies the same recorded update to the student and reference and does not assume a particular model family or parameter count. We will continue these experiments to establish whether the observed benefits extend to additional families and scales.

W1 (reference controls) + Q4. Reference-weight sweep, SAM comparison and ILU progress

Thank you for requesting a tuned static-reference comparison. General Response, item (3) reports a retention-weight sweep at λR∈{0.1,0.3,1,3}\lambda_R\in\{0.1,0.3,1,3\} on TOFU with Llama-3.2-3B and GD. UMU achieves higher AGNews NS at all four tested weights. We will add these results to Section 5.3 and Appendix E, and clarify the reference construction, selection protocol and achieved target suppression in Section 4.1 and Appendix C. General Response, item (1) also reports our completed NPO+SAM comparison. Due to time constraints, the direct comparison with ILU is not yet complete. To allow timely discussion, we are sharing the completed results first and will continue the ILU comparison. Once results are available, we will report recovery and learning together with the training, tuning and selection budgets in a follow-up response. We welcome further questions about the current comparisons or evaluation protocol.

Q5. Figure 2c boundary

Thank you for pointing out the cropped label. We will increase the export margin for Figure 2c so that the rightmost “steps” label is fully visible, and check the figure in the revised PDF. This is a layout correction; it does not change the plotted data.

Reviewer_CuLzOpenReview ↗

Feedback

Thank you for your response to my review comments. Compared with the initial submission, the rebuttal shows clear improvements in both the scope of experimental coverage and the mechanistic argumentation. I particularly appreciate the newly added multi-task learning results and the ablation of the replay/selection factors, which make the paper's main claims more convincing.

I recommend that the authors prioritize the following in the final revision:

  1. Provide the per-run records, sample counts, and aggregation method for NS, and clearly explain the origin of 83.98%.

  2. Incorporate the newly added cross-task experiments and the static-reference weight sensitivity results into the main text, rather than leaving them only in the appendix.

  3. Add the computational overhead of UMU, and further discuss the distinction from related work such as ILU.

  4. Correspondingly narrow the conclusions regarding model scale, full-parameter attacks, and cross-task generalization, so as to avoid going beyond the actual scope of the experiments.

Overall, I believe this paper has developed from a manuscript with an interesting idea but insufficient experimental support into a work with a relatively clear methodological contribution and fairly complete experimental support. I am willing to adjust my recommendation to Weak Accept, provided that the remaining issues, particularly the NS numerical inconsistency, are adequately clarified in the final revision.

Thank you very much for your positive assessment and support of our work. In the revised manuscript PDF, the revisions and clarifications responding to your comments are highlighted in blue for ease of reference. We sincerely appreciate your helpful feedback and the time you have devoted to reviewing our paper. Thank you again.

Comments on the paper

3 comments

Thank you to all reviewers for the constructive suggestions on baseline comparisons, downstream learning and robustness. We summarize these five sets of experiments and their results below.

(1) We evaluated NPO, NPO+SAM and NPO+UMU alongside GD, RMU and KLUE on TOFU Forget01 and FaithUn with Llama-3.2-3B and Qwen3-4B.

\text{Dataset / model} & \text{Method} & \text{FS}\downarrow & \text{RS}\uparrow & \text{US}\uparrow & \text{Harmful p=0.1}\downarrow & \text{Harmful p=0.3}\downarrow & \text{Benign}\downarrow & \text{AGNews NS}\uparrow & \text{SST-2 NS}\uparrow & \text{QQP NS}\uparrow & \text{GSM8K NS}\uparrow \\\\ \hline \text{TOFU Llama-3.2-3B} & \text{Default} & 90.42 & 93.54 & 91.45 & \text{--} & \text{--} & \text{--} & 86.72 & 94.95 & 90.10 & 40.94 \\\\ & \text{GD} & 19.32 & 89.36 & 92.88 & 45.41 & 51.73 & 20.98 & 82.03 & 92.43 & 85.41 & 36.69 \\\\ & \text{RMU} & 14.04 & 93.70 & 91.74 & 87.97 & 90.97 & 20.33 & 79.69 & 92.78 & 83.07 & 39.95 \\\\ & \text{KLUE} & 18.71 & 86.86 & 92.81 & 43.71 & 49.38 & 22.19 & 81.25 & 92.66 & 84.63 & 38.13 \\\\ & \text{NPO} & 18.31 & 87.28 & 89.95 & 18.15 & 17.15 & 18.47 & 83.50 & 93.35 & 86.88 & 38.89 \\\\ & \text{NPO+SAM} & 19.64 & 81.63 & 89.18 & 22.56 & 22.83 & 19.26 & 83.62 & 93.58 & 86.98 & 38.67 \\\\ & \text{GD+UMU} & 18.45 & 92.91 & 92.01 & 24.98 & 31.04 & 19.08 & 85.16 & 93.35 & 86.57 & 37.98 \\\\ & \text{RMU+UMU} & 13.69 & 93.56 & 91.44 & 52.34 & 58.22 & 15.35 & 83.98 & 94.27 & 89.40 & 41.17 \\\\ & \text{KLUE+UMU} & 17.87 & 92.54 & 91.98 & 24.04 & 29.63 & 18.95 & 84.38 & 94.84 & 89.34 & 39.12 \\\\ & \text{NPO+UMU} & 17.73 & 89.62 & 90.27 & 15.13 & 15.44 & 14.86 & 86.05 & 95.07 & 90.04 & 40.03 \\\\ \text{TOFU Qwen3-4B} & \text{Default} & 92.16 & 94.28 & 92.87 & \text{--} & \text{--} & \text{--} & 87.89 & 96.44 & 92.00 & 83.02 \\\\ & \text{GD} & 20.03 & 90.43 & 94.16 & 44.80 & 50.80 & 21.60 & 83.98 & 93.35 & 87.10 & 77.33 \\\\ & \text{RMU} & 14.55 & 94.43 & 93.13 & 89.07 & 92.10 & 20.89 & 82.03 & 92.78 & 84.60 & 76.04 \\\\ & \text{KLUE} & 19.39 & 88.13 & 94.09 & 44.39 & 50.06 & 22.87 & 83.20 & 93.23 & 86.40 & 78.92 \\\\ & \text{NPO} & 18.99 & 89.24 & 91.23 & 18.83 & 17.83 & 19.15 & 85.45 & 93.81 & 88.65 & 79.15 \\\\ & \text{NPO+SAM} & 20.32 & 83.59 & 90.46 & 23.24 & 23.51 & 19.94 & 85.57 & 94.15 & 88.75 & 79.00 \\\\ & \text{GD+UMU} & 19.12 & 93.70 & 93.36 & 24.64 & 30.48 & 19.74 & 86.72 & 94.27 & 88.10 & 81.80 \\\\ & \text{RMU+UMU} & 14.19 & 94.30 & 92.85 & 53.00 & 58.94 & 15.87 & 85.55 & 93.58 & 86.91 & 80.52 \\\\ & \text{KLUE+UMU} & 18.52 & 93.36 & 93.33 & 24.41 & 30.04 & 19.61 & 85.94 & 94.50 & 87.60 & 81.58 \\\\ & \text{NPO+UMU} & 18.41 & 91.58 & 91.55 & 15.81 & 16.12 & 15.54 & 87.55 & 95.18 & 90.30 & 82.87 \\\\ \text{FaithUn Llama-3.2-3B} & \text{Default} & 91.92 & 89.22 & 57.40 & \text{--} & \text{--} & \text{--} & 86.72 & 94.95 & 90.10 & 40.94 \\\\ & \text{GD} & 0.0 & 81.03 & 55.77 & 48.13 & 54.76 & 33.33 & 80.86 & 89.33 & 84.16 & 34.42 \\\\ & \text{RMU} & 0.0 & 77.80 & 56.31 & 52.63 & 75.53 & 14.29 & 78.12 & 87.39 & 83.09 & 33.97 \\\\ & \text{KLUE} & 0.0 & 81.68 & 56.72 & 57.14 & 62.96 & 28.33 & 80.08 & 90.02 & 85.23 & 35.71 \\\\ & \text{NPO} & 0.0 & 56.25 & 52.64 & 8.46 & 9.78 & 8.19 & 82.30 & 90.94 & 83.71 & 36.32 \\\\ & \text{NPO+SAM} & 0.0 & 50.60 & 51.87 & 11.74 & 12.86 & 9.63 & 83.89 & 91.17 & 83.81 & 36.09 \\\\ & \text{GD+UMU} & 0.0 & 87.99 & 56.44 & 11.11 & 13.58 & 9.88 & 84.77 & 92.89 & 87.70 & 38.13 \\\\ & \text{RMU+UMU} & 0.0 & 87.51 & 56.71 & 15.79 & 24.56 & 5.26 & 83.20 & 92.20 & 88.06 & 37.30 \\\\ & \text{KLUE+UMU} & 0.0 & 88.09 & 56.91 & 12.70 & 15.87 & 9.52 & 83.98 & 93.92 & 90.20 & 37.68 \\\\ & \text{NPO+UMU} & 0.0 & 61.42 & 56.42 & 4.51 & 5.25 & 5.15 & 85.46 & 95.53 & 89.67 & 39.42 \\\\ \text{FaithUn Qwen3-4B} & \text{Default} & 85.66 & 81.05 & 71.00 & \text{--} & \text{--} & \text{--} & 87.89 & 96.44 & 92.00 & 83.02 \\\\ & \text{GD} & 0.0 & 69.35 & 73.20 & 26.00 & 40.11 & 21.83 & 83.59 & 91.51 & 86.58 & 71.87 \\\\ & \text{RMU} & 0.0 & 72.92 & 66.03 & 41.77 & 46.30 & 23.62 & 81.64 & 90.94 & 85.46 & 71.04 \\\\ & \text{KLUE} & 0.0 & 69.75 & 72.32 & 32.84 & 47.22 & 22.40 & 82.81 & 91.17 & 88.19 & 75.36 \\\\ & \text{NPO} & 0.0 & 54.96 & 66.50 & 8.92 & 9.61 & 8.37 & 85.07 & 92.09 & 87.18 & 76.88 \\\\ & \text{NPO+SAM} & 0.0 & 49.31 & 65.73 & 13.33 & 15.29 & 9.16 & 85.19 & 92.20 & 87.55 & 76.42 \\\\ & \text{GD+UMU} & 0.0 & 79.29 & 71.95 & 5.56 & 9.26 & 3.70 & 86.72 & 94.27 & 91.20 & 77.18 \\\\ & \text{RMU+UMU} & 0.0 & 79.83 & 68.37 & 12.28 & 14.04 & 10.53 & 85.16 & 94.04 & 90.50 & 76.72 \\\\ & \text{KLUE+UMU} & 0.0 & 79.35 & 71.51 & 7.94 & 12.70 & 6.35 & 85.94 & 94.72 & 91.00 & 80.21 \\\\ & \text{NPO+UMU} & 0.0 & 61.18 & 71.28 & 4.72 & 8.31 & 3.42 & 86.89 & 95.76 & 90.20 & 81.20 \\\\ \end{array}$$

(2) Replay and selection ablations. We evaluated four variants on both benchmarks and backbones: A disables both replay on the reference and selection by learning trials; B enables selection only; C enables replay only; D enables both (full UMU). All four retain the defense-training stage, with GD as the base-unlearning control. On TOFU with Llama, C exceeds A and D exceeds B by 6.25 NS points, while B exceeds A and D exceeds C by 2.74 points. Removing replay lowers harmful recovery at p=0.1 from 24.98 to 23.89, but also lowers NS from 85.16 to 78.91. These comparisons distinguish the observed contributions of replay and selection and explain why recovery should be assessed alongside learning. We will add the four-variant results and interpretation to Section 5.3, and describe the training and release rules in Appendix B.

\text{Dataset / model} & \text{Variant} & \text{Replay} & \text{Selection} & \text{FS}\downarrow & \text{Harmful p=0.1}\downarrow & \text{Harmful p=0.3}\downarrow & \text{Benign}\downarrow & \text{AGNews NS}\uparrow \\\\ \hline \text{TOFU Llama-3.2-3B} & \text{GD} & \text{--} & \text{--} & 19.32 & 45.41 & 51.73 & 20.98 & 82.03 \\\\ & \text{UMU (A)} & 0 & 0 & 18.91 & 25.58 & 31.64 & 19.38 & 76.17 \\\\ & \text{No replay (B)} & 0 & 1 & 18.61 & 23.89 & 29.86 & 18.62 & 78.91 \\\\ & \text{No selection (C)} & 1 & 0 & 18.33 & 24.52 & 30.65 & 18.83 & 82.42 \\\\ & \text{Full UMU (D)} & 1 & 1 & 18.45 & 24.98 & 31.04 & 19.08 & 85.16 \\\\ \text{TOFU Qwen3-4B} & \text{GD} & \text{--} & \text{--} & 20.03 & 44.80 & 50.80 & 21.60 & 83.98 \\\\ & \text{UMU (A)} & 0 & 0 & 19.58 & 25.91 & 31.68 & 20.09 & 79.70 \\\\ & \text{No replay (B)} & 0 & 1 & 19.28 & 25.31 & 31.08 & 19.36 & 81.64 \\\\ & \text{No selection (C)} & 1 & 0 & 19.04 & 24.28 & 30.06 & 19.79 & 84.77 \\\\ & \text{Full UMU (D)} & 1 & 1 & 19.12 & 24.64 & 30.48 & 19.74 & 86.72 \\\\ \text{FaithUn Llama-3.2-3B} & \text{GD} & \text{--} & \text{--} & 0.0 & 48.13 & 54.76 & 33.33 & 80.86 \\\\ & \text{UMU (A)} & 0 & 0 & 0.0 & 11.71 & 14.18 & 10.18 & 76.57 \\\\ & \text{No replay (B)} & 0 & 1 & 0.0 & 8.89 & 12.50 & 8.08 & 78.91 \\\\ & \text{No selection (C)} & 1 & 0 & 0.0 & 10.00 & 12.50 & 9.09 & 82.42 \\\\ & \text{Full UMU (D)} & 1 & 1 & 0.0 & 11.11 & 13.58 & 9.88 & 84.77 \\\\ \text{FaithUn Qwen3-4B} & \text{GD} & \text{--} & \text{--} & 0.0 & 26.00 & 40.11 & 21.83 & 83.59 \\\\ & \text{UMU (A)} & 0 & 0 & 0.0 & 7.27 & 11.71 & 4.34 & 78.51 \\\\ & \text{No replay (B)} & 0 & 1 & 0.0 & 6.67 & 11.11 & 4.04 & 81.25 \\\\ & \text{No selection (C)} & 1 & 0 & 0.0 & 5.56 & 8.33 & 3.03 & 83.98 \\\\ & \text{Full UMU (D)} & 1 & 1 & 0.0 & 5.56 & 9.26 & 3.70 & 86.72 \\\\ \end{array}$$ (3) Retention-weight sensitivity. Static uses the checkpoint obtained after base unlearning as a reference that does not receive the student's recorded relearning update. The updated student matches this reference on retained data. UMU applies the same recorded update to the reference before matching. We compared these references at retention weights 0.1, 0.3, 1 and 3 in the TOFU/Llama-3.2-3B GD setting. UMU has higher AGNews NS at every weight, by 1.17–10.26 points. At weight 0.3, UMU also has lower harmful recovery, 30.11/37.42 versus 31.06/38.79, with NS 83.72 versus 80.32. At weight 3, Static lowers recovery further but NS falls to 74.36, compared with 84.62 for UMU. The scan addresses sensitivity to the reference weight and makes the recovery–learning trade-off explicit. We will define the Static control and weight grid in Appendix C, and add the sensitivity results and analysis to Section 5.3 and Appendix E. $$\begin{array}{rlrrrrrrr} \lambda_R & \text{Reference} & \text{FS}\downarrow & \text{RS}\uparrow & \text{US}\uparrow & \text{Harmful p=0.1}\downarrow & \text{Harmful p=0.3}\downarrow & \text{Benign}\downarrow & \text{AGNews NS}\uparrow \\\\ \hline 0 & \text{GD} & 19.32 & 89.36 & 92.88 & 45.41 & 51.73 & 20.98 & 82.03 \\\\ 0.1 & \text{Static} & 19.28 & 89.51 & 92.71 & 44.12 & 50.63 & 20.71 & 81.44 \\\\ 0.1 & \text{UMU} & 19.05 & 89.74 & 92.83 & 42.87 & 49.05 & 20.14 & 82.61 \\\\ 0.3 & \text{Static} & 18.92 & 91.26 & 92.44 & 31.06 & 38.79 & 19.55 & 80.32 \\\\ 0.3 & \text{UMU} & 18.71 & 91.58 & 92.39 & 30.11 & 37.42 & 19.21 & 83.72 \\\\ 1 & \text{Static} & 18.61 & 93.12 & 92.18 & 23.89 & 29.86 & 18.62 & 78.91 \\\\ 1 & \text{UMU} & 18.45 & 92.91 & 92.01 & 24.98 & 31.04 & 19.08 & 85.16 \\\\ 3 & \text{Static} & 17.94 & 93.88 & 91.42 & 9.74 & 11.86 & 12.63 & 74.36 \\\\ 3 & \text{UMU} & 17.68 & 93.55 & 91.16 & 11.32 & 14.05 & 13.71 & 84.62 \\\\ \end{array}$$

(4) Full-parameter attacks. We evaluated fine-tuning that updates all model weights on TOFU Forget01 with Llama-3.2-3B, using harmful retraining at p=0.1 and p=0.3 and benign instruction tuning. UMU lowers harmful and benign recovery for all four base methods. For GD, harmful recovery falls from 59.03/67.25 to 39.11/44.55, while benign recovery falls from 26.85 to 25.03. These results extend the observed resistance beyond LoRA attacks to full-parameter fine-tuning in this setting. We will add the comparison figure and analysis in Section 5.2, and the attack protocol to Appendix D.

\text{Method} & \text{Harmful p=0.1}\downarrow & \text{Harmful p=0.3}\downarrow & \text{Benign}\downarrow \\\\ \hline \text{GD} & 59.03 & 67.25 & 26.85 \\\\ \text{GD+UMU} & 39.11 & 44.55 & 25.03 \\\\ \text{RMU} & 89.73 & 92.79 & 26.02 \\\\ \text{RMU+UMU} & 62.45 & 64.58 & 21.24 \\\\ \text{KLUE} & 61.19 & 69.13 & 28.40 \\\\ \text{KLUE+UMU} & 40.54 & 45.80 & 25.29 \\\\ \text{NPO} & 29.03 & 33.64 & 23.61 \\\\ \text{NPO+UMU} & 25.42 & 29.45 & 20.14 \\\\ \end{array}$$ (5) WMDP and MUSE evaluation. We extended benchmark coverage by comparing RMU and RMU+UMU on WMDP and MUSE with Llama-3.2-3B. On WMDP, harmful recovery falls from 56.84/57.92 to 38.42/40.93 and AGNews NS rises from 79.69 to 84.63. On MUSE, harmful recovery falls from 35.16/37.28 to 18.02/20.64 and NS rises from 77.86 to 82.04. Benign recovery also falls on both benchmarks, with RS and US reductions below one point. These results address the requested benchmark coverage and pair reduced recovery with better subsequent learning on the added datasets. We will add the comparison figure and analysis to Appendix E, cite it from Section 5.2, and define the benchmark-specific metrics in Appendix D. $$\begin{array}{llrrrrrrr} \text{Dataset / model} & \text{Method} & \text{FS}\downarrow & \text{RS}\uparrow & \text{US}\uparrow & \text{Harmful p=0.1}\downarrow & \text{Harmful p=0.3}\downarrow & \text{Benign}\downarrow & \text{AGNews NS}\uparrow \\\\ \hline \text{WMDP Llama-3.2-3B} & \text{RMU} & 27.64 & 53.82 & 51.36 & 56.84 & 57.92 & 34.28 & 79.69 \\\\ & \text{RMU+UMU} & 24.71 & 52.96 & 50.48 & 38.42 & 40.93 & 22.14 & 84.63 \\\\ \text{MUSE Llama-3.2-3B} & \text{RMU} & 7.24 & 45.38 & 47.92 & 35.16 & 37.28 & 14.21 & 77.86 \\\\ & \text{RMU+UMU} & 5.63 & 44.52 & 47.04 & 18.02 & 20.64 & 9.13 & 82.04 \\\\ \end{array}$$

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.