- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Smoothing Gradient Ascent (SGA), which adapts label smoothing to LLM unlearning by incorporating normal data into the forgetting objective. The authors provide a gradient-based analysis of the smoothing mechanism and evaluate SGA on three unlearning benchmarks (TOFU, Harry Potter, and MUSE-News) across multiple LLMs. The results demonstrate that SGA can improve forgetting effectiveness and model utility under appropriately selected smoothing rates.
Critical strengths
-
Clear presentation and meaningful methodological exploration. The paper is well written and presents a reasonable adaptation of label smoothing to LLM unlearning. The gradient-based analysis provides useful insights into how smoothing may influence the forgetting process.
-
Comprehensive experimental evaluation. The authors evaluate SGA across multiple models and unlearning tasks, comparing it against several established baselines. The results demonstrate the potential effectiveness of the proposed approach.
Critical weaknesses
-
Strong sensitivity to the smoothing rate. The effectiveness of SGA depends heavily on the choice of r, with substantially different preferred values across models and tasks. For example, Tables 1 and 2 show that positive smoothing rates can be effective in some settings, whereas negative rates are preferred in others. Moreover, SGA does not consistently outperform competing methods across all evaluation metrics. These observations make its overall advantage less conclusive.
-
Unclear parameter selection and generalizability. Although the authors provide a gradient-based analysis, the paper lacks a practical criterion for selecting the sign and magnitude of r. More importantly, it remains unclear whether a smoothing rate tuned on one forget set can generalize to unseen forget sets. Since real-world unlearning requests can vary substantially, requiring extensive tuning for each new forget set would significantly limit the method’s practical applicability.
Therefore, my recommendation is Weak Reject, but I am open to reconsideration if the rebuttal provides convincing explanations.
Questions for authors
Q1. Can the authors provide a practical criterion for selecting the sign and magnitude of r without extensive hyperparameter tuning? In particular, can the gradient-based analysis in Section 4.3 and Figure 2 be used to guide this selection?
Q2. Can a smoothing rate selected on one forget set generalize to other unseen forget sets? For example, how does SGA perform when using a fixed r across different forget sets without additional tuning?
Q3. Given the variations across metrics in Table 1, can the authors clarify whether SGA consistently provides a better forgetting–utility trade-off than strong baselines such as DPO-RT, rather than outperforming them only on selected metrics or under carefully tuned settings?
5 comments
Maintain rejection
Thank you for the clarification. I acknowledge that smoothing can improve the forgetting–utility trade-off with an appropriately chosen rate. However,
- the response does not resolve my main concern:
- how to select the sign and magnitude of without relying on final unlearning metrics, or
- whether a rate selected on one forget set remains effective on unseen forget sets. The local geometric analysis provides useful insight, but does not establish such a selection criterion or demonstrate its predictive validity.
- I do not expect a single rate to work universally across models and datasets. However, evidence of transfer across forget sets within a fixed model would be important for establishing practical applicability. At present, the results support a promising empirical finding under suitable tuning, while the reliability and reusability of that tuning remain unverified.
I therefore maintain my weak reject recommendation.
In Section 4.3, we provide a criterion for estimating the range of suitable smoothing rates . Applying this criterion to TOFU, we estimate that is suitable for Llama2-7B and that is suitable for OPT-2.7B and Phi-1.5B. We tested additional values of within the recommended range for each model and observed consistently good performance across the tested values. We report these additional results in Appendix E, and they are broadly consistent with our analysis in Section 4.3.
| Llama2-7B (Recommended ) | OPT-2.7B (Recommended ) | Phi-1.5B (Recommended ) | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | FQ ↑ | MU ↑ | F-RL ↓ | R-RL ↑ | Method | FQ ↑ | MU ↑ | F-RL ↓ | R-RL ↑ | Method | FQ ↑ | MU ↑ | F-RL ↓ | R-RL ↑ |
| GA () | 0.0068 | 0.5990 | 0.4817 | 0.9204 | GA () | 0.0541 | 0.4851 | 0.4490 | 0.6440 | GA () | 0.0541 | 0.5058 | 0.4914 | 0.8012 |
| SGA () | 0.0068 | 0.6018 | 0.4777 | 0.9229 | SGA () | 0.0971 | 0.4851 | 0.3973 | 0.6148 | SGA () | 0.0971 | 0.5100 | 0.4831 | 0.8062 |
| SGA () | 0.0068 | 0.6027 | 0.4792 | 0.9163 | SGA () | 0.0971 | 0.4859 | 0.4021 | 0.6939 | SGA () | 0.0971 | 0.5093 | 0.4851 | 0.8009 |
| SGA () | 0.2657 | 0.4784 | 0.3959 | 0.6012 | SGA () | 0.0971 | 0.5120 | 0.4816 | 0.8043 | |||||
| SGA () | 0.4046 | 0.4747 | 0.3863 | 0.5890 | SGA () | 0.0971 | 0.5095 | 0.4891 | 0.7953 |
We hope these results address your concern. Thank you for taking the time to review our work and for your thoughtful feedback.
Rebuttal to Reviewer Xzrq
We thank you for your time and valuable feedback. Our responses to your comments are provided below:
Weakness 1. Strong sensitivity to the smoothing rate
Please refer to “Response to Common Concerns Raised by Reviewers: 1. Sensitivity to the Smoothing Rate.”
Weakness 2. Unclear parameter selection and generalizability
Referring to Equation (1) in Section 3.1, “Formulation,” the baselines we consider and our motivation are based on the assumption that achieving forgetting on the forget set fulfills the unlearning objective. Regarding the reviewer’s concern, SGA’s use of an external LLM to generate specifically tailored, semantically similar normal data gives it an advantage in generalizability over the baselines discussed in this paper. During training, the model simultaneously forgets the forget data and learns from highly similar normal data, where the key target may remain unchanged while the content differs. For example: “Jaime Vasquez was born on the 17th of April in the year 1962” and “Jaime Vasquez was born on the 8th of March in the year 1955.” By learning to distinguish between the forget data and the normal data, the model learns modified knowledge associated with the specific unlearning target.
The reviewer raises a meaningful question about whether a smoothing rate tuned on one forget set can generalize to new forget sets. However, this work focuses on unlearning with a given forget set. In future work, we may explore ideas from continual learning to address this issue.
Question
1. Please refer to Weakness1.
2 and 3. Appendix D provides the detailed hyperparameter configurations for each experiment. We would also like to clarify that SGA uses exactly the same experimental settings, including all shared hyperparameters, as the baselines in the corresponding experiments. The only difference is the method-specific smoothing rate r, which can be computed using the criterion in Section 4.3. Upon acceptance, we will release the complete experimental code to ensure reproducibility.
We look forward to your response and would be happy to discuss any aspect of the paper with you.