acceptodds
Under review as a conference paper at ICLR 2027

Understanding and Mitigating the Overcorrection in Machine Unlearning via Feature Learning

Abstract

Machine unlearning aims to remove the influence of particular data points from a pre-trained model in response to growing privacy and ethical concerns. While gradient-ascent (GA)-based unlearning provides a simple and widely used strategy, its optimization dynamics remain poorly understood. In this work, we identify a striking non-monotonic behavior in GA-based unlearning, which we call the overcorrection phenomenon: the forgetting quality evaluated only on the forgetting data first improves and then deteriorates as unlearning proceeds. Specifically, the Kolmogorov-Smirnov (KS) distance between the truth-ratio distributions of the unlearned model and the retrained model first decreases and then increases, indicating that the unlearned model initially approaches the retrained reference but later deviates from it. To explain this phenomenon, we provide a three-stage theoretical characterization under a two-layer convolutional neural network model. Our analysis specifies when the forgetting quality enters a favorable regime, how long it remains there, and how it eventually deteriorates. The theoretical results further reveal the key role of the learning rate, motivating a simple learning-rate adjustment strategy that mitigates overcorrection. Experiments on both synthetic and real-world datasets support our theoretical predictions and demonstrate the effectiveness of the proposed strategy.

Primary area:
learning theory
Keywords:
deep learning theory, machine unlearning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GoVzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the overcorrection phenomenon in gradient-ascent-based machine unlearning, where the forgetting quality initially improves but subsequently deteriorates as unlearning continues. To explain this behavior, the authors develop a three-stage theoretical analysis using a two-layer ReLU convolutional neural network under a signal-noise data model. The analysis characterizes when the unlearned model approaches the retrained reference, how long it remains in a favorable forgetting regime, and how its behavior eventually deviates. Based on these findings, the authors propose a loss-threshold-based learning-rate adjustment strategy to mitigate overcorrection. Experiments on synthetic data, CIFAR-10, CIFAR-100, and Tiny ImageNet demonstrate non-monotonic forgetting behavior and suggest that the proposed strategy can delay deterioration.

Critical strengths
  1. The paper investigates an important limitation of gradient-ascent-based unlearning by focusing on forgetting quality relative to retraining. The proposed three-stage analysis offers a potentially useful perspective on the dynamics of unlearning.

  2. The overcorrection phenomenon is demonstrated across multiple image classification benchmarks and different forgetting configurations. The experiments provide useful evidence that excessive gradient ascent can deteriorate forgetting quality.

Critical weaknesses
  1. There appear to be inconsistencies in the signal-noise decomposition and its initialization between Eq. (4.1) and Appendix E.2. Additionally, the gradient-ascent update normalization differs between Eq. (2.3) and the corresponding appendix derivations. These issues raise concerns about the correctness of the theoretical analysis and the claimed three-stage characterization.

  2. The theoretical analysis focuses on KL divergence, whereas the main real-world experiments evaluate KS distance without establishing a rigorous connection between the two. Moreover, the proposed mitigation strategy is essentially a learning-rate reduction rule, and comparisons with simple alternatives such as early stopping and standard learning-rate scheduling are missing. The experiments also provide limited evidence regarding retained utility and overall unlearning effectiveness.

Questions for authors
  1. Could the authors clarify the initialization of the signal-noise decomposition in Eq. (4.1) and Appendix E.2, and explain the different gradient-ascent normalization factors used in the main text and appendix? How do these differences affect the validity of Theorem 3.2?

  2. What theoretical or empirical evidence supports transferring the KL-divergence-based analysis to the KS-distance-based evaluation? Can the authors directly verify the predicted three-stage behavior using the theoretical metric on real-world datasets?

  3. How does the proposed learning-rate adjustment compare with early stopping and conventional learning-rate schedules under matched initial learning rates and computational budgets? In particular, can the authors clarify the different initial learning rates used for CIFAR-10 in Figures 2 and 4 and report the corresponding retained-data performance?

3 comments

Thanks for your constructive feedback! We address your questions and concerns as follows.

Q1. Could the authors clarify the initialization of the signal-noise decomposition in Eq. (4.1) and Appendix E.2, and explain the different gradient-ascent normalization factors used in the main text and appendix? How do these differences affect the validity of Theorem 3.2?

The initialization of the signal-noise decomposition in Eq. (4.1) and Appendix E.2 is consistent, with some intermediate reparameterization steps omitted for brevity. The unlearned model is initialized from the final weights of the base model trained on both the forgetting and retained data, i.e., wj,su,0=wj,sb,tendw_{j,s}^{u,0}=w_{j,s}^{b,t_{\mathrm{end}}}. In Eq. (4.1), the decomposition is expressed relative to wj,su,0w_{j,s}^{u,0}, so the initial coefficient increments are zero. In Appendix E.2, we equivalently absorb the fixed coefficients inherited from the base model into the unlearning coefficients, with a corresponding shift of the decomposition's reference point. Specifically, we define γj,su,t(μf)=γj,sb,tend(μf)+Δγj,su,t(μf)\gamma_{j,s}^{u,t}(\mu_f)=\gamma_{j,s}^{b,t_{\mathrm{end}}}(\mu_f)+\Delta\gamma_{j,s}^{u,t}(\mu_f), and similarly for ρj,s,ifu,t(ξf,if)\rho_{j,s,i_f}^{u,t}(\xi_{f,i_f}). Since the initial increments are zero, this gives γj,su,0(μf)=γj,sb,tend(μf)\gamma_{j,s}^{u,0}(\mu_f)=\gamma_{j,s}^{b,t_{\mathrm{end}}}(\mu_f) and ρj,s,ifu,0(ξf,if)=ρj,s,ifb,tend(ξf,if)\rho_{j,s,i_f}^{u,0}(\xi_{f,i_f})=\rho_{j,s,i_f}^{b,t_{\mathrm{end}}}(\xi_{f,i_f}), as stated in Appendix E.2. The inherited coefficients remain fixed, so this reparameterization preserves the same weight trajectory and coefficient update equations.

Regarding the gradient-ascent normalization factors, the correct denominator during unlearning is nfmn_fm, since gradient ascent is performed only on the forgetting dataset. The denominator m(nf+nr)m(n_f+n_r) appearing in the proof of Lemma E.2, including Eqs. (E.9)–(E.11), is a typographical error carried over from the base-model training derivation. Importantly, the update equations stated in Lemma E.2 are correct and consistent with Eq. (2.3) and Lemma 4.1 in the main text, all using η/(nfm)\eta/(n_fm). Thus, correcting this typographical error and clarifying the equivalent parameterizations do not change the underlying unlearning dynamics or the conclusions of Theorem 3.2.

Q2. What theoretical or empirical evidence supports transferring the KL-divergence-based analysis to the KS-distance-based evaluation?

In our setting, both empirical and theoretical evidence support the strong consistency between KL-divergence and KS-distance dynamics. We have added synthetic experiments with KS distance, and the results in Table 1 show that KL and KS exhibit highly consistent dynamics. For a more fine-grained comparison, we report their favorable intervals, defined as the steps at which the normalized metric is below 0.1, and their turning points. The favorable intervals are [2907, 2937] for KS and [2893, 2944] for KL, with corresponding turning points at steps 2922 and 2925, respectively. These results demonstrate that the favorable intervals substantially overlap and the turning points are nearly identical.

Step05001000150020002500300035004000
KL Divergence1.0010.9930.9820.9610.9190.7830.5850.9280.928
KS Distance0.9430.9350.9180.9000.8730.7750.6781.0001.000

Table 1: KL divergence and KS distance on synthetic data.

From a theoretical perspective, Lemma 4.2 shows that a small signed-margin gap ∣zu−zr∣|z^u-z^r|, where zu=yffu(xf)z^u=y_f f^u(x_f) and zr=yffr(xf)z^r=y_f f^r(x_f), implies a small KL divergence. Meanwhile, in the binary setting, the truth ratio is q(z)/(1−q(z))=exp⁡(z)q(z)/(1-q(z))=\exp(z), which is a strictly monotone transformation of the signed margin. Therefore, closeness of the signed-margin distributions is equivalent to closeness of the corresponding truth-ratio distributions, and hence a smaller KS distance. Our KL analysis thus captures the same underlying mechanism measured by KS: in the first stage, the unlearned margin approaches the retrained margin; in the second stage, they remain close; and in the third stage, they deviate, producing the same decrease-then-increase pattern. This combination of KS-based evaluation and KL-based theoretical analysis has also been adopted in [1].

Q3. Can the authors directly verify the predicted three-stage behavior using the theoretical metric on real-world datasets?

The predicted three-stage behavior can also be verified using the theoretical metric, KL divergence, on real-world datasets. We conduct additional experiments on CIFAR-100 to evaluate the evolution of KL divergence during unlearning. As shown in Table 2, the KL divergence first decreases and then increases, demonstrating that the overcorrection phenomenon persists under the theoretical metric on real-world data.

StageInitialMinimumFinal
KL1.0000.3760.802

Table 2: KL divergence at different stages of GA on CIFAR-100.

Q4. How does the proposed learning-rate adjustment compare with early stopping and conventional learning-rate schedules under matched initial learning rates and computational budgets? In particular, can the authors clarify the different initial learning rates used for CIFAR-10 in Figures 2 and 4 and report the corresponding retained-data performance?

Our proposed learning-rate adjustment strategy achieves lower KS distance and higher retain accuracy (RA) than early stopping and conventional learning-rate schedules. We conduct additional experiments on CIFAR-100, comparing our strategy with early stopping at the best-performing checkpoint (which is difficult to identify in practice), standard step decay, and cosine decay under matched initial learning rates and computational budgets. We also report the corresponding retain accuracy. As shown in Table 3, our strategy outperforms all these baselines in both KS distance and RA, demonstrating its effectiveness in improving forgetting quality while preserving retained-data performance.

MethodKS Distance ↓RA (%) ↑
GA0.76037.72
Early Stopping0.19467.29
Step Decay0.41060.11
Cosine Decay0.39460.85
Our Strategy0.18667.53

Table 3: Comparison of learning-rate adjustment strategies on CIFAR-100 in terms of KS distance and retain accuracy (RA).

We also clarify that the discrepancy in the reported initial unlearning learning rates for CIFAR-10 between Figures 2 and 4 is solely a typographical error. The actual initial learning rate used in both experiments is 5×10−55 \times 10^{-5}.

References

[1] Zhang, R., et al. Negative preference optimization: From catastrophic collapse to effective unlearning. arXiv, 2024

Dear Reviewer GoVz,

As the discussion period is approaching its deadline, we would like to briefly follow up on our rebuttal. In response to your constructive suggestions, we have clarified the intermediate reparameterization steps omitted for brevity when describing the initialization of the unlearning process. We have also provided both empirical and theoretical evidence supporting the strong consistency between KL-divergence and KS-distance dynamics in our setting. Furthermore, we have conducted additional experiments comparing our proposed learning-rate adjustment strategy with early stopping and conventional learning-rate schedules, demonstrating its superiority in improving forgetting quality while preserving retained-data performance.

We believe that these clarifications and additional results have addressed all of your concerns. Since we will no longer be able to respond after the discussion closes, we would greatly appreciate it if you could briefly let us know whether any concerns remain. We would be happy to provide any final clarification before the deadline.

Thank you again for your time and valuable feedback.

Best regards, Authors of Submission 31009

Reviewer_r3cDOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors identify and analyze an overlooked problem in machine unlearning (MU), namely overcorrection during gradient ascent (GA)-based unlearning. They investigate how long gradient ascent should be performed to achieve effective unlearning without eventually moving the unlearned model away from the retrained model.

Specifically, they show both empirically and theoretically that the Kolmogorov–Smirnov (KS) distance between the retrained and unlearned models initially decreases until a certain time (t_1), after which it remains in a favorable regime for some time before eventually increasing again. Based on their theoretical analysis, the authors propose mitigating overcorrection by reducing the learning rate once the unlearning process enters this favorable regime.

Critical strengths
  1. The paper is extremely well written. The motivation, problem formulation, and theoretical analysis are presented clearly, making the overall argument easy to follow.

  2. Each theorem and lemma is accompanied by clear intuition, and the theoretical findings can be directly connected to the experimental observations. This is particularly valuable for a theory-oriented paper.

  3. The identified problem is interesting and appears to be a relatively overlooked aspect of machine unlearning. The analysis provides useful insights into the dynamics of gradient ascent-based unlearning and could motivate future research in this direction.

Critical weaknesses
  1. Simplified theoretical setting: Although this is not a major concern, the theoretical analysis is conducted under a rather simplified setting involving a two-layer ReLU neural network, Gaussian data, and binary classification. Such assumptions are understandable for establishing initial theoretical results. However, it would be interesting to understand how these findings extend to more complex settings, particularly deep neural networks and billion-parameter models that are increasingly relevant in modern AI. Even a discussion of the potential challenges in extending the theory would strengthen the paper.

  2. Generality across loss functions: It is unclear whether the theoretical findings would continue to hold under different loss functions, even within the same simplified model and data setting. For example, what happens if logistic loss is replaced with another classification loss? Is there a particular property or family of loss functions for which the same overcorrection behavior can be established? Clarifying this would help readers understand the generality of the theoretical results.

  3. Applicability beyond gradient ascent: The analysis primarily focuses on gradient ascent-based unlearning. Does the overcorrection phenomenon, or at least the underlying theoretical intuition, extend to other unlearning approaches, such as noisy fine-tuning or Hessian-based Newton update methods? It would be valuable if the authors could discuss whether there are more general properties of the unlearning dynamics that explain when overcorrection occurs, beyond the specific GA setting.

  4. Theoretical guidance for learning-rate reduction: The authors propose reducing the learning rate once the unlearning process enters the favorable regime. However, it is not entirely clear how much the learning rate should be reduced. Since the paper provides a theoretical characterization of the favorable regime, it would be useful to investigate whether this analysis can also provide a principled estimate or bound for the learning-rate reduction. Currently, this connection between the theoretical analysis and the practical choice of learning rate appears somewhat incomplete.

  5. Connection to broader optimization and overfitting phenomena: Is the theoretical characterization of overcorrection specific to gradient ascent-based machine unlearning, or could similar dynamics arise in more general optimization or overfitting settings? In particular, can the initial improvement followed by eventual deterioration be understood through a more general feature-learning perspective? A discussion of what makes this phenomenon specific to unlearning would help clarify the broader significance of the results.

Questions for authors
  1. Learning-rate selection: Can the theoretical analysis provide guidance on how much the learning rate should be reduced once the favorable regime is reached? Is the current choice primarily empirical?

  2. Generality across loss functions: Do the theoretical results depend specifically on logistic loss, or can they be extended to a broader family of loss functions?

  3. Applicability beyond gradient ascent: Is there any theoretical intuition suggesting that similar overcorrection behavior could occur in other unlearning approaches, such as noisy fine-tuning or Newton-based methods?

  4. Practical applicability: How sensitive is the proposed mitigation strategy to identifying the favorable regime accurately? In practice, how can this regime be identified without access to a fully retrained reference model?

5 comments

Thanks for your constructive feedback! We address your questions and concerns as follows.

Q1. Can the theoretical analysis provide principled guidance or bounds on how much the learning rate should be reduced upon entering the favorable regime, or is the current choice primarily empirical?

Theorem 3.2 provides principled guidance for learning rate adjustment. Specifically, it establishes that the duration of the favorable forgetting regime scales inversely with the learning rate, suggesting that reducing the learning rate can effectively prolong this regime. While the theorem provides theoretical guidance on the direction and effect of learning rate adjustment, the specific reduction factor is selected empirically. Based on experiments across different settings on CIFAR-10, CIFAR-100, and TinyImageNet, we identify a practical range of 0.01–0.1 times the original learning rate. To evaluate the sensitivity of this choice, we conduct additional experiments on CIFAR-100. As shown in Table 1, different learning rate multipliers consistently achieve substantial improvements in KS distance compared with the original GA baseline (KS = 0.760), demonstrating that the proposed strategy is robust to the choice of learning rate multiplier.

LR Multiplier0.010.030.060.10
KS Distance0.2420.2180.1860.231

Table 1: Sensitivity analysis of KS distance to the learning rate multiplier on CIFAR-100.

Q2. Do the theoretical results and overcorrection phenomenon depend specifically on logistic loss, or can they be extended to a broader family of loss functions?

The overcorrection phenomenon is not specific to logistic loss and can extend to a broader family of loss functions, including hinge loss. We conduct additional experiments using hinge loss on CIFAR-100. As shown in Table 2, the KS distance first decreases and then increases during unlearning, demonstrating that the overcorrection phenomenon persists under hinge loss. The underlying theoretical mechanism can be understood through the evolution of signed margins on the forgetting data. In Stage 1, the unlearned margins move toward those of the retrained model, improving forgetting quality. In Stage 2, the margins remain close, corresponding to the favorable forgetting regime characterized by Lemma 4.2. In Stage 3, continued unlearning drives the margins away from those of the retrained model, causing forgetting quality to deteriorate. This three-stage mechanism is driven by the evolution of signed margins rather than the specific form of logistic loss.

StageInitialMinimumFinal
KS Distance0.9990.2000.806

Table 2: KS distance at different stages of hinge-loss-based unlearning on CIFAR-100.

Q3. Does the overcorrection phenomenon extend beyond gradient ascent-based unlearning to other methods, such as noisy fine-tuning or Newton-based methods, and what general properties of unlearning dynamics might explain its occurrence?

The overcorrection phenomenon extends beyond GA-based unlearning to other unlearning methods. We conduct additional experiments on CIFAR-100 using noisy fine-tuning [1] and CuReNU [2] (Cubic-Regularized Newton Unlearning). As shown in Table 3, the KS distance under both methods first decreases and then increases, demonstrating that the overcorrection phenomenon persists.

MethodInitial KSMin KSFinal KS
Noisy Ft0.9990.0660.332
CuReNU0.9980.2160.979

Table 3: KS distance at different stages of noisy fine-tuning and CuReNU unlearning.

This behavior reflects a general trajectory-level property of unlearning: the unlearned model initially approaches the retrained reference on the forgetting data, but continued updates eventually drive it away. Such non-monotonic dynamics are not specific to GA but can also arise under other unlearning methods.

Q4. How sensitive is the proposed mitigation strategy to identifying the favorable regime accurately? In practice, how can this regime be identified without access to a fully retrained reference model?

The proposed mitigation strategy exhibits stable performance across a range of loss-threshold choices for identifying the favorable forgetting regime. We conduct additional sensitivity analysis on CIFAR-100 by varying the loss threshold. As shown in Table 4, different threshold choices all achieve substantial improvements in KS distance compared with the original GA baseline (KS = 0.760), demonstrating the robustness of our strategy to threshold selection. Furthermore, based on our experiments across CIFAR-10, CIFAR-100, and TinyImageNet, we find that effective loss thresholds generally lie between 8 and 9.

Loss Threshold7.58.08.59.0
KS Distance0.2690.2230.1860.225

Table 4: Sensitivity analysis of KS distance to the loss threshold on CIFAR-100.

To avoid relying on manually selected thresholds, we further introduce an automatic learning rate adjustment strategy based on an online Bayesian Information Criterion (BIC) change-point detector. Motivated by our theoretical characterization, we observe that the forgetting loss evolves less stably in Stage 1 but becomes more stable in Stage 2. At each epoch, the detector compares a single linear fit with a continuous two-segment linear fit on the smoothed and normalized forgetting-loss trajectory. We reduce the learning rate when BIC consistently favors the two-segment model and the estimated breakpoint remains stable. Importantly, this approach relies only on the unlearned model and forgetting data, without requiring access to a fully retrained reference model or a dataset-specific absolute loss threshold. As shown in Table 5, this automatic strategy also substantially improves the KS distance, demonstrating its effectiveness in identifying the favorable forgetting regime and mitigating overcorrection.

MethodKS Distance
GA0.760
BIC0.188

Table 5: Comparison of GA and our online BIC-based learning rate adjustment strategy on CIFAR-100.

Q5. Can the theoretical findings be extended beyond the simplified setting to more complex models, particularly deep neural networks and billion-parameter models, and what challenges might arise in such extensions?

The overcorrection phenomenon and our feature-learning theoretical framework can be extended beyond two-layer CNNs to more complex architectures. Our original paper already includes experiments with ResNet-18, and we further conduct experiments with ResNet-34 and ViT on CIFAR-100. As shown in Table 6, the KS distance first decreases and then increases for both architectures, demonstrating that the overcorrection phenomenon persists in deeper neural networks and vision transformers. From a theoretical perspective, by adapting proof techniques similar to those in [3], our feature-learning analysis framework can be extended to characterize overcorrection in ViTs. We adopt a two-layer CNN primarily to simplify the theoretical analysis and keep the proofs concise.

ArchitectureInitial KSMin KSFinal KS
ResNet-341.0000.2040.826
ViT1.0000.1661.000

Table 6: KS distance at different stages of unlearning for ResNet-34 and ViT.

Q6. Is the overcorrection phenomenon specific to gradient ascent-based unlearning, or can similar dynamics be explained from a broader optimization, overfitting, or feature-learning perspective, and what distinguishes overcorrection in unlearning from these general phenomena?

The overcorrection phenomenon is not specific to GA-based unlearning and can be understood from a broader optimization and feature-learning perspective. Our additional experiments with noisy fine-tuning and Newton-based unlearning further support this observation. From a feature-learning perspective, the phenomenon reflects the evolution of learned representations during unlearning: the unlearned model initially approaches the retrained reference, but continued updates eventually drive it away, resulting in overcorrection. This reflects a general optimization phenomenon in which continued optimization can move a model beyond the desired target.

Importantly, overcorrection differs from conventional overfitting. While overfitting typically refers to a deterioration in generalization performance despite improved training performance, overcorrection concerns the deterioration of forgetting quality on the forgetting data itself, measured by the discrepancy between the unlearned and retrained models. In particular, the forgetting loss can continue increasing even after the unlearned model has approached the retrained reference, causing the KS distance to increase again. Thus, overcorrection highlights a distinct failure mode in machine unlearning, where excessive optimization leads to worse forgetting quality rather than better approximation of the retrained model.

References

[1] Koloskova, A., et al. Certified Unlearning for Neural Networks. In ICML 2025

[2] Bui, N., et al. How to cure Newton for unlearning neural networks? an empirical study from the Hessian perspective. In ICLR 2026

[3] Jiang J, et al. Unveil benign overfitting for transformer in vision: Training dynamics, convergence, and generalization. In NeurIPS 2024

Reviewer_PqQPOpenReview ↗
Rating
1
Confidence
2
Informativeness
2(+1)
Presentation
1
Soundness
3
Read review
Paper summary

Machine unlearning (MU) aims to remove the knowledge associated with a certain subset of the data (forget data), and preserve the knowledge associated with the remaining data. A common method in MU is Gradient Ascent (GA). The authors identify a phenomenon in GA-based unlearning, denoted as “overcorrection”. This phenomenon shows that the quality of forgetting evaluated on the forget data first improves, but then starts degrading as unlearning continues. The authors provide a three-stage theoretical analysis on a two-layer CNN model to explain this phenomenon. Additionally, they identify the role of learning rate, and provide a mitigation strategy based on that. Experiments conducted on real-world and synthetic data shows agreement with the theoretical findings.

Critical strengths
  • The paper identifies an interesting phenomenon in GA-based unlearning, which can help improve these unlearning approaches.
Critical weaknesses
  • The overcorrection phenomenon is only studied on a limited setting with a two-layer CNN, GA-based unlearning, and binary classification (as acknowledged by the authors). This heavily limits the scope of the paper, and it is not yet clear if the conclusions hold for any other scenarios.

  • While the motivation and main idea of the paper is clear to understand, the theoretical results, notations, and proofs in the paper can be hard to read and follow. Additionally, it is not fully clear how the learning rate strategy is obtained from the theory (Lemma 4.8). The authors discuss the learning rate mitigation strategy in the experiments section, but how the loss threshold and the new learning rate are chosen is not discussed. Also, the sensitivity to the loss threshold and learning rate changes is not studied or ablated. Overall, revising the writing of the paper can help the clarity of the theory.

Questions for authors

Can the authors provide details on how the best loss threshold and new learning rates are identified?

4 comments

Thanks for your feedback. We address your questions as follows.

Q1. Does the overcorrection phenomenon generalize beyond two-layer CNNs, GA-based unlearning, and binary classification?

The overcorrection phenomenon is not limited to two-layer CNNs, binary classification, or GA-based unlearning. Our original paper already includes experiments with ResNet-18 on CIFAR-100 and TinyImageNet, under forgetting ratios of 1%, 5%, and 10%, demonstrating that overcorrection also occurs in deeper neural networks and multiclass classification settings. Furthermore, we have conducted additional experiments using noisy fine-tuning [1], a non-GA-based unlearning method. As shown in Table 1, the KS distance initially decreases and then increases, exhibiting the same overcorrection phenomenon. These results provide further evidence that overcorrection generalizes beyond shallow CNNs, binary classification, and GA-based unlearning.

Step0510152025303540
KS Distance1.0000.5710.7870.0660.1720.2750.2940.3320.332

Table 1: KS distance under noisy fine-tuning on CIFAR-100.

Q2. How does Lemma 4.8 motivate the proposed learning rate adjustment strategy, and how are the loss threshold and new learning rate selected and their sensitivity evaluated?

The proposed learning rate adjustment strategy is jointly motivated by Lemma 4.8 and Theorem 3.2. Lemma 4.8 shows that when the KS distance is small (i.e., in the favorable forgetting regime), the forgetting loss of the unlearned model is close to that of the retrained model, suggesting that the forgetting loss can serve as a practical proxy for learning rate adjustment. Theorem 3.2 further establishes that the duration of the favorable forgetting regime scales inversely with the learning rate, providing theoretical guidance for reducing the learning rate to prolong this regime. Based on experiments across different settings on CIFAR-10, CIFAR-100, and TinyImageNet, we empirically select a forgetting-loss threshold between 8 and 9 and reduce the learning rate to 0.01–0.1 times its original value. To evaluate the sensitivity of these choices, we conduct additional experiments on CIFAR-100. As shown in Table 2 and 3, different loss thresholds and learning rate reduction factors consistently yield substantial improvements in KS distance compared with the original GA baseline (KS = 0.760), demonstrating the robustness of the proposed strategy to these hyperparameters.

Loss Threshold7.58.08.59.0
KS Distance0.2690.2230.1860.225

Table 2: Sensitivity analysis of KS distance to the loss threshold on CIFAR-100.

LR Multiplier0.010.030.060.10
KS Distance0.2420.2180.1860.231

Table 3: Sensitivity analysis of KS distance to the learning rate multiplier on CIFAR-100.

To further reduce reliance on manually selected thresholds, we additionally introduce an automatic learning rate adjustment strategy based on an online Bayesian Information Criterion (BIC) change-point detector. Motivated by the theoretical characterization of the forgetting dynamics, we observe that the forgetting loss evolves less stably in Stage 1 but becomes more stable in Stage 2. Accordingly, at each epoch, the detector compares a single linear fit with a continuous two-segment linear fit on the smoothed and normalized forgetting-loss trajectory. The learning rate is reduced when BIC consistently favors the two-segment model and the estimated breakpoint remains stable. Importantly, this approach relies only on the unlearned model and forgetting data, without requiring access to the retrained model or a dataset-specific absolute loss threshold. As shown in Table 4, this automatic strategy also substantially improves the KS distance, further demonstrating the practical applicability of our learning rate adjustment strategy.

MethodKS Distance
GA0.760
BIC0.188

Table 4: Comparison of GA and our online BIC-based learning rate adjustment strategy on CIFAR-100.

References

[1] Koloskova, A., et al. Certified Unlearning for Neural Networks. In ICML 2025

Dear Reviewer PqQP,

As the discussion period is approaching its deadline, we would like to briefly follow up on our rebuttal. In response to your constructive suggestions, we have provided additional experiments demonstrating that the overcorrection phenomenon persists beyond gradient ascent-based unlearning, including in noisy fine-tuning. We have also conducted sensitivity analyses of the loss threshold and learning rate for our proposed learning-rate adjustment strategy, demonstrating its effectiveness in substantially improving forgetting quality.

We believe that these clarifications and additional results have addressed all of your concerns. Since we will no longer be able to respond after the discussion closes, we would greatly appreciate it if you could briefly let us know whether these additional results have addressed your concerns. We would be happy to provide any final clarification before the deadline.

Thank you again for your time and valuable feedback.

Best regards, Authors of Submission 31009

I thank the authors for their response. I appreciate the clarification on the choice of learning rate, and also the BIC-based strategy. The response provides clarity on the loss threshold and learning rate adjustment. However, a few of my concerns regarding the experiments still stand. While the new provided experiments are helpful to further strengthen the observations, the scope is still fairly limited. The paper includes experiments with ResNet-18 on GA-based unlearning, and the rebuttal experiments are done with noisy-fine-tuning as an additional method and ResNet-34 and ViT as additional architectures. However, only CIFAR-100 is included for the new experiments. (This is understandable considering the limited time during the rebuttal, but I believe additional experiments should be added to the paper to strengthen the claims). Additionally, methods other than GA and noisy fine-tuning are not studied. Furthermore, the experiments in the rebuttal are not incorporated in the paper, so it is not clear how the revised paper will look.

Overall, while the paper makes an interesting observation and the rebuttal addresses some of my initial concerns, the current experimental setting is limited. I would suggest adding experiments under several different architectures, methods, and datasets. This can include several CNN-based and transformer-based architectures, several unlearning methods, and all the datasets currently included. I would also suggest including the BIC-based learning rate adjustment strategy in the paper. I believe these additions could strengthen the paper. Therefore, considering the current state of the manuscript, I still lean towards rejection. I will consider any discussions with the reviewers and AC for my final recommendation.

Thank you for your follow-up and constructive suggestions. Our additional experiments further demonstrate that the overcorrection phenomenon persists across different model architectures and unlearning methods. Beyond the experiments presented in our earlier rebuttal, we have evaluated ResNet-34 and ViT architectures and further conducted experiments with CuReNU [1] (Cubic-Regularized Newton Unlearning), a Newton-based unlearning method. As shown in Tables 1 and 2, the KS distance initially decreases and then increases, confirming that the overcorrection phenomenon persists across these additional settings.

MethodInitial KSMin KSFinal KS
Noisy Ft0.9990.0660.332
CuReNU0.9980.2160.979

Table 1: KS distance at different stages of noisy fine-tuning and CuReNU unlearning.

ArchitectureInitial KSMin KSFinal KS
ResNet-341.0000.2040.826
ViT1.0000.1661.000

Table 2: KS distance at different stages of unlearning for ResNet-34 and ViT.

[1] Bui, N., et al. How to cure Newton for unlearning neural networks? an empirical study from the Hessian perspective. In ICLR 2026

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.