acceptodds
Withdrawn from ICLR 2027

ISRO: Inner-Step Refinement Optimization for Retain-Free Unlearning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Retain-free unlearning involves a fundamental trade-off between effective forgetting and preserving model utility. In this work, we show that using large learning rates can achieve a more favorable trade-off, but often suffers from instability, whereas small learning rates provide stable updates yet result in less effective unlearning. It raises a need for heuristic hyperparameter tuning to achieve optimal results. To address this limitation, we propose Inner-Step Refinement Optimization (ISRO), the first unlearning-oriented optimizer that stabilizes learning rate sensitivity while optimizing the retain-forget trade-off. Extensive experiments demonstrate that ISRO integrates effectively with existing methods and consistently outperforms vanilla approaches: for the same level of forgetting, it achieves better utility with fewer epochs and lower computational cost. Our findings are supported by theoretical analysis and empirical results across image classification, image generation, and large language model unlearning tasks.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
retain-free unlearning, optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5djQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies retain-free unlearning and shows that gradient ascent (GA) is sensitive to the learning rate. Large learning rates can give a better retain-forget trade-off but are unstable, while small learning rates are stable yet ineffective. It proposes ISRO, an optimizer wrapper that takes multiple inner ascent steps along a gradient computed once per epoch until the forget loss reaches a fraction α of a noise-based reference loss, turning unlearning into a threshold objective. Experiments across diverse tasks show that ISRO variants achieve better retain performance at matched forgetting levels and remain effective at small learning rates where vanilla methods fail.

Critical strengths
  1. A simple, plug-in method. Learning-rate sensitivity of retain-free GA is a genuine obstacle, and ISRO composes with existing objectives (GA, RL, SalUn, ESD) with minimal modification. The claimed robustness is supported where it matters most. At lr = 1e-6 and 1e-7 in Tables 1 to 3, vanilla methods fail to reach any forgetting threshold while ISRO still succeeds, in all three domains. This suggests that the method widens the usable learning-rate range rather than improving a single tuned configuration.

  2. The noise-based reference loss reframes unlearning as a threshold objective. Using lossref=L(xf,noise,θ)loss_{ref} = L(x_f, noise, \theta) as a normalized, task-agnostic target of sufficient forgetting (grounded by the log⁡C\log C bound in Proposition 4.1) replaces open-ended loss maximization with a controllable stopping criterion. This idea is transferable beyond the specific optimizer and gives the α\alpha hyperparameter an interpretable meaning that the learning rate lacks.

Critical weaknesses
  1. The mechanism is not disentangled from an adaptive per-epoch step size, and the theory does not establish the claimed benefit. Inner steps reuse the stale gradient computed once per epoch. For a locally quadratic loss, KK inner steps of size η\eta along one fixed gradient equal a single step of size KηK\eta, so ISRO may largely function as an auto-tuned large learning rate. Meanwhile, Theorem 3.3 only shows that an upper bound on the trade-off ratio decreases with η\eta, under convexity and diagonal-Hessian assumptions, with c≤1c \leq 1 holding trivially; this does not prove that the actual ratio improves. Without an ablation against a per-epoch enlarged learning rate KηK\eta or a line search or scheduler, the gains cannot be attributed to the threshold-stopping design rather than to simply taking larger steps. This directly concerns the central contribution claim.
  2. The evaluation never demonstrates complete forgetting and omits retain-free benchmarks and baselines. In the class-wise experiments (Table 1), results are reported only down to accf=0.6acc_f = 0.6, meaning the model still answers 60% of the forget class correctly. The paper never shows that ISRO can reach accf≈0acc_f \approx 0 while keeping accracc_r at a meaningful level, which is the regime that defines success in full-class unlearning. This matters for two reasons. First, the motivating example in Section 2 asserts that approximately 0% forget accuracy with high retain accuracy is infeasible in the retain-free setting, but recent retain-free work contradicts this. MU-Mis [1] achieves forget accuracy near the retrain oracle with retain accuracy close to retrain on full-class CIFAR-100 and TinyImageNet, and SEMU [2] similarly reaches retrain-level unlearning accuracy with minimal retain damage. Second, both works evaluate random-data forgetting (randomly forgetting 10% to 50% of training data). Finally, SEMU is cited in related work but never compared experimentally, and MU-Mis, a retain-free method built on the same base objectives that ISRO wraps (negative gradient and random labeling), is not cited. On the LLM side, the evaluation reports only forget and retain QA probability, omitting the standard TOFU suite (Truth Ratio, ROUGE-L, Model Utility); ISRO-GA also regresses below GA at lr = 1e-5 for deeper forgetting (Table 3). Meanwhile, α\alpha is strongly task-dependent ([0.01, 0.05] for LLMs versus [0.1, 0.9] elsewhere) and cannot be validated without retain data, so the advertised hyperparameter insensitivity is partially relocated rather than removed.

[1] Cheng et al.,Remaining-data-free Machine Unlearning by Suppressing Sample Contribution

[2] Sendera et al., SEMU: Singular Value Decomposition for Efficient Machine Unlearning

Questions for authors

Clarification requests:

  1. [Mechanism attribution] Inner steps reuse the stale, once-per-epoch gradient; for a quadratic loss, KK steps of size η\eta equal one step of size KηK\eta. Can you ablate ISRO against (a) a per-epoch enlarged learning rate KηK\eta and (b) a standard learning-rate scheduler or line search? This determines whether the gains come from the threshold-stopping design or from larger steps, and is decisive for the contribution claim.
  2. [Comparison with retain-free baselines and complete forgetting] Can you (a) extend Table 1 to accf≈0acc_f \approx 0 and show whether ISRO preserves a meaningful acc_r there, and (b) compare against MU-Mis [1] and SEMU [2], both retain-free, on class-wise forgetting with FA → 0 and on random-data forgetting (for example 10% and 50% of CIFAR-10/100)? MU-Mis shares the exact base objectives that ISRO wraps (negative gradient, random labeling), so this comparison directly tests whether ISRO's optimizer-level gains persist against state-of-the-art retain-free methods.
  3. [Evaluation completeness] Can you report the full TOFU suite (Truth Ratio, ROUGE-L, Model Utility) for the LLM experiments? QA probability alone cannot establish genuine forgetting, so these results would directly change my Soundness assessment.
  4. [Table 3 regression] Why does ISRO-GA underperform GA at lr = 1e-5 for probf≤0.7prob_f \leq 0.7? What α\alphawas used per learning rate?
  5. [α\alpha selection in practice] The α\alpha ranges differ by an order of magnitude across domains ([0.01, 0.05] for LLMs versus [0.1, 0.9] elsewhere), and no retain set is available for validation in this setting. What procedure do you recommend for selecting α\alpha ?
  6. [Optimizer details] Do ISRO's inner steps use Adam moments or the raw gradient? If Adam is used, how do the momentum and second-moment estimates interact with the stale gradient? This affects both reproducibility and the interpretation of the mechanism.
  7. [Overclaim] You claimed that your method is the first unlearning-oriented optimizer, but there is another optimizer-based work, DualOptim [3], which also targets to improve unlearning stability even though it is not designed for retain-free unlearning tasks.

[1] Cheng et al.,Remaining-data-free Machine Unlearning by Suppressing Sample Contribution

[2] Sendera et al., SEMU: Singular Value Decomposition for Efficient Machine Unlearning

[3] Zhong et al., DualOptim: Enhancing Efficacy and Stability in Machine Unlearning with Dual Optimizers

4 comments
AuthorsOpenReview ↗

Rebuttal 1 - P1

We thank the reviewer for the detailed review.

[W1, Q1] The mechanism is not disentangled from an adaptive per-epoch step size, and the theory does not establish the claimed benefit. Can you ablate ISRO against (a) a per-epoch enlarged learning rate KηK\eta and (b) a standard learning-rate scheduler or line search?

Response.

W1: Is ISRO an auto-tuned large learning rate? Yes, by design, ISRO is equivalent to an adaptive per-epoch learning rate KtηK_t\eta, i.e., θt+1=θt±Ktη,∇L(θt)\theta_{t+1}=\theta_t\pm K_t\eta\\,\nabla\mathcal L(\theta_t). However, how the learning rate affects unlearning, and how to adapt the learning rate accordingly, remain largely unexplored. We are the first to theoretically characterize the learning-rate sensitivity of retain-free unlearning (Theorems 3.3–3.4; Fig. 3; Appendix D). ISRO turns this into stable optimization: it achieves consistent unlearning performance across learning rates spanning several orders of magnitude, and remains effective at small learning rates where vanilla optimizers fail to unlearn (Tables 1–3).

W1: Does the theory establish the claimed benefit? We would like to clarify our theoretical contributions as follows. Theorems 3.3 and 3.4 diagnose the learning-rate sensitivity of vanilla GA. Theorem 3.3 shows that a larger learning rate can yield a more favorable retain-forget trade-off. Theorem 3.4 explains why simply using a large learning rate is difficult: as η\eta increases, the iteration window satisfying the (ϵr,ϵf)(\epsilon_r,\epsilon_f) criterion becomes increasingly narrow. These findings motivate ISRO's threshold-based mechanism, which reuses a computed gradient direction until a target reference loss is reached, rather than taking a single update per gradient computation. Theorem 4.2 then analyzes this mechanism by establishing an upper bound on the number of inner steps required to reach the α\alpha-defined loss threshold. Thus, Theorems 3.3 and 3.4 motivate the design of ISRO, while Theorem 4.2 provides its theoretical efficiency guarantee.

W1: Theorem 3.3 only bounds the trade-off ratio; does the actual ratio improve? The actual ratio is reflected in Tables 1–3 and 5–15: at each fixed forgetting target, the methods optimized with ISRO achieve better retain performance than their vanilla counterparts, i.e., less retain degradation for the same amount of forgetting, and hence an improved kk.

Q1 (a): Per-epoch enlarged learning rate KηK\eta. Fig. 5 and Appendix F.2 (Figs. 25–30: GA, RL, SalUn, BE, BS and LoTUS, each at η∈10−6,10−5\eta\in\\{10^{-6},10^{-5}\\}) already show the value of KtK_t at each iteration, i.e., the per-iteration enlarged learning rate KtηK_t\eta that ISRO selects. KtK_t varies sharply across iterations: it is about 10310^3 in the first iteration ($\sim$1000 for α=0.1\alpha=0.1, $\sim$3000 for α=0.9\alpha=0.9) and drops to only 1–2 after the first few iterations.

Q1 (b): Line search. We compare ISRO with a stochastic Armijo line search (SLS) [1], both applied to MU-Mis (Table R1). SLS-MU-Mis fails to reach the deeper forgetting levels at η=10−5\eta=10^{-5} and any level at η=10−6\eta=10^{-6}, whereas ISRO-MU-Mis reaches all levels at both learning rates with better retain performance at most levels.

Table R1 (added to Appendix E.1, Table 7 of the revised paper). Retain performance accr\texttt{acc}_r (%) at different accf\texttt{acc}_f levels under various settings for ViT-based 30% class-wise unlearning on CIFAR-100 with MU-Mis. SLS-MU-Mis = MU-Mis optimized with SLS [1] with Armijo constant cc.

η\etaMethodα\alpha0.90.80.70.60.50.40.30.20.1
10−510^{-5}MU-Mis–87.883.959.345.536.123.716.79.42.6
10−510^{-5}SLS-MU-Mis (c=0.1c=0.1)–87.480.565.053.747.837.428.5✗✗
10−510^{-5}SLS-MU-Mis (c=10−4c=10^{-4})–89.578.267.3✗✗✗✗✗✗
10−510^{-5}ISRO-MU-Mis0.189.281.978.579.777.564.532.817.06.9
10−510^{-5}ISRO-MU-Mis0.389.280.071.563.852.742.032.021.411.0
10−510^{-5}ISRO-MU-Mis0.588.779.268.457.947.838.131.023.016.2
10−610^{-6}MU-Mis–✗✗✗✗✗✗✗✗✗
10−610^{-6}SLS-MU-Mis (c=0.1c=0.1)–✗✗✗✗✗✗✗✗✗
10−610^{-6}SLS-MU-Mis (c=10−4c=10^{-4})–✗✗✗✗✗✗✗✗✗
10−610^{-6}ISRO-MU-Mis0.189.276.363.249.938.828.820.014.06.8
10−610^{-6}ISRO-MU-Mis0.390.277.667.959.350.241.132.323.412.7
10−610^{-6}ISRO-MU-Mis0.589.482.472.862.853.741.230.220.811.4

References

AuthorsOpenReview ↗

Rebuttal 1 - P2

[W2, Q2] Complete forgetting (accf≈0\texttt{acc}_f\approx0) and comparison with retain-free baselines.

Response.

Retain-free baselines are already included. All experiments are run in the retain-free setting and compare several distinct unlearning methods across three domains (Tables 1–3; Appendix E, Tables 5–15). They fall into two groups:

  • Methods designed to be retain-free (forget data only): GA, Random Labeling (RL), Boundary Expanding (BE) and Boundary Shrink (BS) for classification, and ESD for Stable Diffusion.
  • Methods adapted to the retain-free setting by removing their retain-set loss: SalUn and LoTUS for classification, SalUn for Stable Diffusion, and NPO, SimNPO, RMU and IdkPO for LLMs.

The adaptation is stated in the note at the beginning of Section 5, and all setups are described in Appendix C.2–C.4. We also add MU-Mis [1] at our 30% scale (Table R1; Appendix E.1, Table 7 of the revised paper).

Different forgetting scales. MU-Mis evaluates full-class unlearning of a single class ("rocket", i.e. 1% of CIFAR-100). Our experiments are far more aggressive: we forget 30% and 50% of the classes (Section 5.1; Appendix Tables 5–6), i.e. 30–50 times more data. As the forget set grows, the coupling between forget and retain gradients grows as well: on CIFAR-100, γ\gamma rises from 0.283 at 3% to 0.891 at 10% of the classes (Appendix C.1, Table 4). Without retain data, retain degradation then becomes inevitable as the forget accuracy decreases, and Table 1 shows this for every method. This is why we formulate retain-free unlearning as reaching a target forgetting level ϵf\epsilon_f (Definition 2.1; practical target in Section 2), and why our experiments report accr\texttt{acc}_r at the target levels accf∈0.9,0.8,0.7,0.6\texttt{acc}_f\in\\{0.9,0.8,0.7,0.6\\} (Appendix C.5).

On the Section 2 example (paragraph "Example"). The example illustrates that, without Dr\mathcal D_r, lowering forget accuracy also lowers retain accuracy, so the ideal target of 90% retain and 0% forget accuracy cannot be guaranteed.

References

  • [1] Cheng et al., Remaining-data-free Machine Unlearning by Suppressing Sample Contribution (MU-Mis), ICLR 2026.

[W2, Q2] (cont.) MU-Mis at our forgetting scale (30% of the classes).

Response. We use the MU-Mis objective [1] as the base method and compare MU-Mis with ISRO-MU-Mis on ViT for CIFAR-100, forgetting 30% of the classes. We report accr\texttt{acc}_r at the first update where accf\texttt{acc}_f falls below each target from 0.9 to 0.1 (Table R1; Appendix E.1, Table 7 of the revised paper).

  • η=10−6\eta=10^{-6}. Vanilla MU-Mis reaches no accf\texttt{acc}_f target (accf\texttt{acc}_f stays at 88.74%), whereas ISRO-MU-Mis reaches every target down to 0.1 for every α\alpha.
  • η=10−5\eta=10^{-5}. For every α\alpha, ISRO-MU-Mis keeps a higher accr\texttt{acc}_r than MU-Mis at every target from 0.7 to 0.1.
  • Across learning rates. ISRO-MU-Mis at η=10−6\eta=10^{-6}, α=0.5\alpha=0.5 exceeds MU-Mis at its best learning rate at every target except 0.8.

References

  • [1] Cheng et al., Remaining-data-free Machine Unlearning by Suppressing Sample Contribution (MU-Mis), ICLR 2026.
AuthorsOpenReview ↗

Rebuttal 1 - P3

[Q4] Why does ISRO-GA underperform GA at η=10−5\eta=10^{-5} for probf≤0.7\texttt{prob}_f\le0.7? Which α\alpha was used?

Response.

Why. As we already discuss in Section 3.1 and illustrate in Fig. 2b, a large learning rate makes unlearning unstable and hard to control: a single step can overshoot the forgetting target. η=10−5\eta=10^{-5}, the largest learning rate in Table 3, falls in this regime. ISRO checks its stopping rule after every step of size η\eta; at this learning rate, each step is too large to stop precisely at the stricter forgetting levels (probf≤0.7\texttt{prob}_f\le0.7). ISRO's gains lie at smaller learning rates:

  • Small learning rates. At η∈10−6,5×10−7\eta\in\\{10^{-6},5\times10^{-7}\\}, vanilla GA and NPO fail to reach any forgetting threshold, while ISRO-GA and ISRO-NPO reach all of them (Table 3; Appendix Tables 10–11).
  • Best achievable performance. Over all learning rates, the best ISRO-GA result exceeds the best GA result at every probf\texttt{prob}_f level (Table 3).

Which α\alpha. We sweep α∈0.01,0.02,0.03,0.04,0.05\alpha\in\\{0.01,0.02,0.03,0.04,0.05\\} (Appendix C.4) and report the best result in Table 3. Experiments on the robustness of ISRO to α\alpha are shown in Fig. 4 and Appendix F.1 (Figs. 19–24).

[Q3] Full TOFU evaluation suite.

Response. The revised paper reports TOFU's probability metric on the forget and retain sets (probf\texttt{prob}_f, probr\texttt{prob}_r; Table 3 and Appendix Tables 8–15). We also report Forget Quality, Model Utility, and ROUGE-L (Table R2; Appendix E.3, Table 17 of the revised paper).

Table R2. SimNPO on Llama-3.2-1B-Instruct, TOFU-10%. All metrics are measured at the checkpoint where probf\texttt{prob}_f first falls below 0.70 (consistent with Appendix Tables 8–11); ✗ = level never reached.

lrMethodprobf\texttt{prob}_f ↓probr\texttt{prob}_r ↑FQ ↑MU ↑ROUGE-L (f) ↓ROUGE-L (r) ↑
10−510^{-5}SimNPO0.6530.7040.4820.8800.4240.711
10−510^{-5}ISRO-SimNPO0.6490.7040.4860.8800.4220.711
5×10−65\times10^{-6}SimNPO0.6550.6980.4790.8720.4260.705
5×10−65\times10^{-6}ISRO-SimNPO0.6520.6980.4830.8720.4240.705
10−610^{-6}SimNPO✗✗✗✗✗✗
10−610^{-6}ISRO-SimNPO0.6560.7030.4780.8790.4260.710
5×10−75\times10^{-7}SimNPO✗✗✗✗✗✗
5×10−75\times10^{-7}ISRO-SimNPO0.6710.7040.4620.8800.4360.711

At large learning rates, where both methods reach the target, ISRO-SimNPO matches or slightly improves SimNPO on all metrics. At small learning rates, vanilla SimNPO never reaches the target, whereas ISRO-SimNPO reaches it with high Forget Quality while keeping Model Utility close to the original model.

References

  • Fan et al., Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning, 2025.

[Q7] Overclaim: "first unlearning-oriented optimizer"; DualOptim.

Response. We thank the reviewer for pointing us to this work. DualOptim [1] requires a retain set, whereas ISRO is an optimizer-level method designed for the retain-free setting. We have refined this claim in the revised paper and discuss DualOptim in Appendix B.1.

References

  • [1] Zhong et al., DualOptim: Enhancing Efficacy and Stability in Machine Unlearning with Dual Optimizers, NeurIPS 2025.

[Q5] How should α\alpha be selected without a retain set? Why do the ranges differ by an order of magnitude?

Response. For tuning α\alpha, our theorems do not give specific guidance. However, empirically, Appendix G shows robustness across α∈[0.1,0.9]\alpha\in[0.1,0.9] for image classification and NSFW Stable Diffusion unlearning, but a narrower range, typically [0.01,0.05][0.01,0.05], for LLM unlearning. In practice, we therefore recommend choosing α\alpha within these ranges. We hypothesize that LLMs require a smaller α\alpha because their \texttt{loss\\_ref}, computed over a vocabulary of about 128K tokens rather than 100 classes, is much larger, so a small fraction of it already suffices for forgetting.

[Q6] Do the inner steps use Adam moments or the raw gradient?

Response. ISRO is applied after the momentum and bias-correction steps of the base optimizer (Adam/AdamW). In each epoch, Adam computes its update direction once from the gradient, including momentum and bias correction, and ISRO repeats this update in the inner loop until the threshold is reached. The moment estimates are therefore updated once per gradient computation, exactly as in vanilla Adam, and do not depend on the number of inner steps. Algorithm 2 writes the update in its plain-gradient form for readability; we have stated this explicitly in Appendix C.6 of the revised paper.

Thanks for your rebuttal. After reviewing other reviewers' comments, I think my current rating can reflect the quality of this paper. Therefore, I will keep my rating.

Reviewer_zX9MOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

In machine unlearning, there are two simultaneous objectives: increasing the loss on the forget set while preserving the model's utility on the retain set. The paper considers the more difficult setting where the retain set is unavailable and asks if it is possible to achieve better retain utility with less compute cost for the same forget quality. Firstly, the paper makes the theoretical observation that a larger learning rate can potentially yield a better retain-forget trade-off and checks it empirically. Then, the paper separately proposes the Inner-Step Refinement Optimization algorithm. Instead of maximising the forget loss as in gradient ascent, the goal is to increase the forget loss to some reference loss determined by noisy labels. The experiments suggest that the proposed algorithm lead to better retain utility and less sensitivity to learning rates.

Critical strengths
  1. The identified problem is significant. The paper considers the more difficult setting where the retain set is unavailable and asks if it is possible to achieve better retain utility with less compute cost for the same forget quality.
Critical weaknesses
  1. The main paper does not sufficiently distinguish its contributions from existing work. There is limited discussion of related works, existing theoretical results and empirical evidence to show that the proposed ISRO algorithm outperforms other methods that address the problem caused by gradient ascent. For example, it would be important to compare with [Z] which directly address the catastrophic collapse.

  2. The theoretical contributions of the paper in Sec 3 seem separate from the proposed algorithm in Sec 4. The theoretical contributions are also not clearly explained/justified.

    • In the main paper, some notation such as λ\lambda was not explained.
    • In practice, is the γ\gamma FRG gap small enough for the bound to be meaningful? How is it affected by the size/similarity of the forget set vs. the retain set?
    • The Taylor approximation used to derive the results only holds within a small radius of convergence. It is less likely to hold for large learning rates or when the loss function has rapidly changing curvature. Do these occur in practice?
    • It is well-known that hyperparameter tuning (e.g., learning rate) is important. Does the theorem/propositions suggest how to tune more efficiently and effectively?
    • What is the connection between Sec 3 and Sec 4?

[Z] Zhang, R., Lin, L., Bai, Y., & Mei, S. (2024). Negative preference optimization: From catastrophic collapse to effective unlearning. arXiv preprint arXiv:2404.05868.

Questions for authors
  1. Distinguish the contributions from existing work like [Z] in a related word section and in experiment results. Explain the choice of baseline in Sec 5.
  2. Clarify the issues with theoretical contributions raised in Weakness 2.
3 comments
AuthorsOpenReview ↗

Rebuttal 1 - P1

We thank the reviewer for the comments.

[W1] Limited discussion of related works.

Response. We already discuss related work in Appendix B, covering image classification, Stable Diffusion and LLM unlearning (Appendices B.1–B.3), and position ISRO against existing retain-free methods in Section 1.

[W1, Q1] Distinguishing contributions; comparison with NPO [Z]; choice of baselines.

Response. We note that NPO is already evaluated in the paper. NPO is an unlearning method (an objective), whereas ISRO is an unlearning optimizer that wraps any objective. The appropriate comparison is therefore vanilla NPO vs. NPO optimized with ISRO, which we report in Appendix E.3 (Tables 8–15; setup in Appendix C.4; related work in Appendix B.3). For example, at η=10−6\eta=10^{-6} and 5×10−75\times10^{-7}, vanilla NPO fails to reach any probf\texttt{prob}_f unlearning target, while ISRO-NPO reaches all of them (Appendix Tables 10–11).

The same X vs. ISRO-X protocol covers 11 distinct objectives across three domains: GA, RL, SalUn, BE, BS and LoTUS for classification; ESD and SalUn for SD; and GA, NPO, SimNPO, RMU and IdkPO for LLMs (Tables 1–3, 5–15).

References

  • [Z] Zhang et al., Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning, 2024.

[W2-a] Notation λ\lambda not explained.

Response. [λ1(t)]r[\lambda_1^{(t)}]_{r} and [λd(t)]r[\lambda_d^{(t)}]_{r} denote the largest and smallest eigenvalues of the retain Hessian, used in Eq. (5) and defined at the beginning of Appendix A. We have made this definition explicit in Section 3.1 of the revised paper.

AuthorsOpenReview ↗

Rebuttal 1 - P2

[W2-b] Is γ\gamma small enough to be meaningful? How does it depend on forget/retain size and similarity?

Response. We measured γ\gamma during retain-free GA (Appendix C.1, Table 4; 10 seeds × 3 learning rates; max over all steps):

Forget ratio3%5%8%10%
CIFAR-1000.2830.3350.5630.891
TinyImageNet0.4200.5310.5710.614

In all settings γ<1\gamma<1, so the bound guarantees k<1k<1: the retain loss increases less than the forget loss. γ\gamma grows with the forget-set size (table above) and with forget/retain similarity: shared features make forget-set updates also affect the retain loss. Hence, γ\gamma captures the intrinsic difficulty of unlearning for a forget/retain pair and upper-bounds the achievable trade-off kk (Theorem 3.3; Appendix C.1).

[W2-c] Validity of the Taylor approximation for large learning rates.

Response. Equation (4) should include the third-order remainder term O(η3)\mathcal O(\eta^3), and Proposition 3.2(a,b) should accordingly include O(Tη3)\mathcal O(T\eta^3). We have corrected them in the revised paper (Section 3.1; Appendix A.2) as Δf=η∑t=0T−1∇f(t)⊤(I+η2Hf(t))∇f(t)+O(Tη3),\Delta_f=\eta\sum_{t=0}^{T-1}{\nabla_f^{(t)}}^\top\Big(I+\tfrac{\eta}{2}H_f^{(t)}\Big)\nabla_f^{(t)}+\mathcal O(T\eta^3), with the analogous expression for Δr\Delta_r.

This correction does not change the result in Theorem 3.3(a). Factoring out the common leading term gives Δj=η,(Aj+ηBj+O(η2)),\Delta_j=\eta\\,\big(A_j+\eta B_j+\mathcal O(\eta^2)\big), so the ratio k=ΔrΔfk=\frac{\Delta_r}{\Delta_f} still admits a second-order expansion around η=0\eta=0. Therefore, the conclusion of Theorem 3.3(a) remains valid. We acknowledge that the current analysis omits the higher-order Taylor remainder terms for analytical tractability. The behavior of these remainder terms is not explicitly characterized in our theoretical framework, so our analysis is restricted to the second-order approximation. We now state this limitation explicitly in Appendix G of the revised paper and leave a rigorous treatment of the higher-order remainder terms as future work.

[W2-d] Do the theorems/propositions suggest how to tune hyperparameters more efficiently and effectively?

Response. Our theorems do not give a specific learning rate, but they show why it is hard to tune. Theorem 3.3 shows that a larger learning rate gives a better retain-forget trade-off, while Theorem 3.4 shows that it also narrows the window of iterations satisfying the (ϵr,ϵf)(\epsilon_r,\epsilon_f) criterion. ISRO alleviates this trade-off: it performs consistently across learning rates spanning several orders of magnitude (Tables 1–3), and we show that it is robust over a wide range of α\alpha in image classification and Stable Diffusion unlearning (α∈[0.1,0.9]\alpha\in[0.1,0.9]) and over α∈[0.01,0.05]\alpha\in[0.01,0.05] in LLM unlearning (Appendix G). Hence, ISRO substantially reduces the tuning effort compared with tuning the learning rate of vanilla methods.

For tuning α\alpha, our theorems do not give specific guidance; the ranges above are empirical. We hypothesize that the difference between them is driven by the properties of the loss function in each task. We agree that a principled, task-independent rule remains open and leave this limitation for future work.

[W2-e] Connection between Sec. 3 and Sec. 4.

Response. Theorems 3.3 and 3.4 diagnose the learning-rate sensitivity of vanilla GA. Theorem 3.3 shows that a larger learning rate can yield a more favorable retain-forget trade-off. Theorem 3.4 explains why simply using a large learning rate is difficult: as η\eta increases, the iteration window satisfying the (ϵr,ϵf)(\epsilon_r,\epsilon_f) criterion becomes increasingly narrow. These findings motivate ISRO's threshold-based mechanism, which reuses a computed gradient direction until a target reference loss is reached, rather than taking a single update per gradient computation. Theorem 4.2 then analyzes this mechanism by establishing an upper bound on the number of inner steps required to reach the α\alpha-defined loss threshold. Thus, Theorems 3.3 and 3.4 motivate the design of ISRO, while Theorem 4.2 provides its theoretical efficiency guarantee.

I have read the other reviews and rebuttal. Although the rebuttal is helpful, I will keep my rating as I share 4zcQ's concern about the significance/informativeness of the theoretical results.

Reviewer_4zcQOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies retain-free unlearning and claiming that gradient-ascent unlearning is sensitive to the learning rate. The authors propose ISRO, which applies each forget-set gradient repeatedly in an inner loop until the forget loss reaches a threshold controlled by α\alpha. They study the performance of the proposed method ISRO on image classification, Stable Diffusion, and LLM unlearning, claiming better retain performance at the same level of forgetting and lower learning-rate sensitivity.

Critical strengths

The paper's retain-free setting is practically relevant, and its evaluation spans classification, image generation, and language modeling.

Critical weaknesses

I have the following major concerns:

(1) In Appendix A.3, Equation 46 effectively asserts, for 0≤c≤10\leq c\leq1 and x≥0x\geq0, we have $ \frac{1+c\eta x/2}{1+\eta x/2} \leq \frac{1+c\eta/2}{1+\eta/2}. $

This is not generally true. Taking c=0.5c=0.5, η=1\eta=1, and x=0.01x=0.01, the left-hand side is approximately 0.9980.998, while the right-hand side is 0.8330.833. Therefore, the stated learning-rate-dependent trade-off bound is not established by the provided proof.

(2) Algorithm 2 reuses the same gradient across multiple inner updates (Eq. 65), where the proof applies the standard smoothness-based descent inequality (Eq. 64), which requires evaluating the gradient at the current iterate θk\theta_k. Additionally, the final use of the stopping condition has an inequality-direction problem: L(θT)≤(1−α)L(θD)\mathcal{L}(\theta^T) ≤ (1 − \alpha)\mathcal{L}(\theta^D) gives a lower bound on L(θD)−L(θT)\mathcal{L}(\theta^D)-\mathcal{L}(\theta^T), so it cannot be substituted into Eq. 70 to obtain an upper bound on KK.

(3) ISRO repeatedly applies the same gradient within its inner loop until the loss reaches a threshold. If KtK_t inner steps are performed, the overall update is θt+1=θt+Ktη∇Lf(θt)\theta_{t+1}=\theta_t+K_t\eta\nabla L_f(\theta_t). This is mathematically equivalent to using an adaptive effective learning rate of KtηK_t\eta for that batch. ISRO simply rescales the learning rate according to a loss threshold. Not novel!

(4) The authors claim that ISRO consistently outperforms other optimization methods across different unlearning tasks. However, the detailed LLM results are considerably less convincing. For instance, on TOFU-10% with learning rate 10−510^{-5}, at matched forget probability of 0.700.70.

Questions for authors

Please check the Critical Weaknesses section.

5 comments
AuthorsOpenReview ↗

Rebuttal 1 - P1

We thank the reviewer for checking the proofs in detail. Both points concern individual proof steps; as we show below, they lead to minor updates of Theorems 3.3 and 4.2, and the conclusions of both theorems remain unchanged.

[W1] Eq. 46 overlooks the range of xx.

Response. In Eq. 46 (Appendix A.3), we overlooked the range of xx. We have updated Eq. 46 and Theorem 3.3 in the revised paper (Section 3.1; Appendix A.3) as follows.

Here x=B/A=∑t∇f(t)⊤Hf(t)∇f(t)∑t∣∇f(t)∣2x = B/A = \frac{\sum_t {\nabla_f^{(t)}}^\top H_f^{(t)}\nabla_f^{(t)}}{\sum_t \\|\nabla_f^{(t)}\\|^2}, so x≥λ‾f≥0x \ge \underline{\lambda}_f \ge 0, where λ‾f\underline{\lambda}_f is the smallest eigenvalue of Hf(t)H_f^{(t)} (Appendix A). Since 0≤c≤10\le c\le1 (cc as in Theorem 3.3), h(x)=1+cηx/21+ηx/2h(x) = \frac{1+c\eta x/2}{1+\eta x/2} is non-increasing in xx. Hence Eq. 46 becomes k;≤;γ,h(λ‾f);=;γ,1+c,ηλ‾f/21+ηλ‾f/2.k \\;\le\\; \gamma\\, h(\underline{\lambda}_f) \\;=\\; \gamma\\,\frac{1+c\\,\eta\underline{\lambda}_f/2}{1+\eta\underline{\lambda}_f/2}. The conclusions of Theorem 3.3 are unchanged: the bound is non-increasing in η\eta, tends to γ\gamma as η→0\eta\to0, and tends to cγc\gamma as η→∞\eta\to\infty.

The other parts are unchanged. Proposition 3.2, Theorem 3.4, the remaining proof steps, the empirical analysis (Section 3.2) and ISRO itself do not use Eq. 46.

[W2] Theorem 4.2: stale gradient and inequality direction.

Response. We clarify both steps below.

(i) Stale gradient. Eq. (64) states that, for any θ\theta and η≤1/L\eta\le1/L, Lalt(θ−η∇Lalt(θ))≤Lalt(θ)−η∣∇Lalt(θ)∣2+Lη22∣∇Lalt(θ)∣2≤Lalt(θ)−η2∣∇Lalt(θ)∣2.\mathcal L_{\rm alt}(\theta-\eta\nabla\mathcal L_{\rm alt}(\theta))\le\mathcal L_{\rm alt}(\theta)-\eta\\|\nabla\mathcal L_{\rm alt}(\theta)\\|^2+\tfrac{L\eta^2}{2}\\|\nabla\mathcal L_{\rm alt}(\theta)\\|^2\le\mathcal L_{\rm alt}(\theta)-\tfrac{\eta}{2}\\|\nabla\mathcal L_{\rm alt}(\theta)\\|^2 . The bound depends only on the gradient at the starting point θ\theta. By Eq. (65), every inner step of epoch tt uses the same gradient ∇Lalt(θ0(t))\nabla\mathcal L_{\rm alt}(\theta^{(t)}_0), so after KtK_t inner steps θKt(t)=θ0(t)−Ktη,∇Lalt(θ0(t)).\theta^{(t)}_{K_t}=\theta^{(t)}_0-K_t\eta\\,\nabla\mathcal L_{\rm alt}(\theta^{(t)}_0). For nsteps≤1/(Lη)n_{steps}\le1/(L\eta) and Kt≤nstepsK_t\le n_{steps} (Algorithm 1), applying Eq. (64) at θ=θ0(t)\theta=\theta^{(t)}_0 with step size KtηK_t\eta gives

\mathcal L_{\rm alt}(\theta^{(t)}_{K_t}) &\le\mathcal L_{\rm alt}(\theta^{(t)}_0)-K_t\eta\\,\\|\nabla\mathcal L_{\rm alt}(\theta^{(t)}_0)\\|^2+\tfrac{L(K_t\eta)^2}{2}\\|\nabla\mathcal L_{\rm alt}(\theta^{(t)}_0)\\|^2\\\\ &=\mathcal L_{\rm alt}(\theta^{(t)}_0)-K_t\eta\Big(1-\tfrac{LK_t\eta}{2}\Big)\\|\nabla\mathcal L_{\rm alt}(\theta^{(t)}_0)\\|^2\\\\ &\le\mathcal L_{\rm alt}(\theta^{(t)}_0)-\tfrac{\eta}{2}K_t\\|\nabla\mathcal L_{\rm alt}(\theta^{(t)}_0)\\|^2 \qquad (LK_t\eta\le1)\\\\ &=\mathcal L_{\rm alt}(\theta^{(t)}_0)-\tfrac{\eta}{2}\sum_{k=0}^{K_t-1}\\|\nabla\mathcal L_{\rm alt}(\theta^{(t)}_0)\\|^2 . \end{aligned}$$ Since $\theta^{(t+1)}_0=\theta^{(t)}_{K_t}$ and $\theta^{(0)}_0=\theta^{\mathcal D}$, summing over $t=0,\dots,T-1$ yields Eq. (66): $$\mathcal L_{\rm alt}(\theta^{(T)})\le\mathcal L_{\rm alt}(\theta^{\mathcal D})-\tfrac{\eta}{2}\sum_{t=0}^{T-1}\sum_{k=0}^{K_t-1}\\|\nabla\mathcal L_{\rm alt}(\theta^{(t)}_0)\\|^2. \tag{66}$$ We have added the condition $n_{steps}\le1/(L\eta)$ to Theorem 4.2 and present the detailed proof in Appendix A.6 of the revised paper. **(ii) Inequality direction.** We have updated the final step of our proof (original Eqs. (70)–(71)) in the revised paper as follows. Since $\theta^{(T)}$ is the *first* iterate below the threshold, the preceding iterate $\theta^{(T-1)}_{K_{T-1}-1}$ (after $K-1$ inner steps) is still above the threshold. Combining this with Eq. (69) at that iterate yields $$(1-\alpha)\\,\mathcal L_{\rm alt}(\theta^{\mathcal D})<\mathcal L_{\rm alt}(\theta^{(T-1)}_{K_{T-1}-1})\le\mathcal L_{\rm alt}(\theta^{\mathcal D})-\tfrac{\eta}{2}g_{\min}(K-1),$$ and hence $K<1+\frac{2\alpha}{\eta\\,g_{\min}}\mathcal L_{\rm alt}(\theta^{\mathcal D})$. The revised Theorem 4.2 states this bound.
AuthorsOpenReview ↗

Rebuttal 1 - P2

[W3] ISRO is equivalent to an adaptive learning rate KtηK_t\eta; not novel.

Response. We respectfully disagree. As the ICLR Reviewer Guide states, submissions bring value "when they convincingly demonstrate new, relevant, impactful knowledge." The identity θt+1=θt±Ktη,∇L(θt)\theta_{t+1}=\theta_t\pm K_t\eta\\,\nabla\mathcal L(\theta_t) holds by adaptive learning rate design, but how the learning rate affects unlearning, and how to adapt the learning rate accordingly, remain largely unexplored. To the best of our knowledge, we are the first to theoretically characterize the learning-rate sensitivity of retain-free unlearning (Theorems 3.3–3.4; Fig. 3; Appendix D). ISRO turns this into stable optimization: it achieves consistent unlearning performance across learning rates spanning several orders of magnitude, and remains effective at small learning rates where vanilla optimizers fail to unlearn (Tables 1–3).

[W4] LLM results at TOFU-10%, η=10−5\eta=10^{-5}.

Response. As we already discuss in Section 3.1 and illustrate in Fig. 2b, a large learning rate makes unlearning unstable: a single step can overshoot the forgetting target. η=10−5\eta=10^{-5}, the largest learning rate in Table 3, falls in this regime. ISRO checks its stopping rule after every step of size η\eta; at this learning rate, each step is too large to stop precisely at the stricter forgetting levels (probf≤0.7\texttt{prob}_f\le0.7). ISRO's gains lie at smaller learning rates:

  • Small learning rates. At η∈10−6,5×10−7\eta\in\\{10^{-6},5\times10^{-7}\\}, vanilla GA and NPO fail to reach any forgetting threshold, while ISRO-GA and ISRO-NPO reach all of them (Table 3; Appendix Tables 10–11).
  • Best achievable performance. Over all learning rates, the best ISRO-GA result exceeds the best GA result at every probf\texttt{prob}_f level (Table 3).

We have revised the claim accordingly in the revised paper (Abstract; Section 5.2).

W1 (Theorem 3.3): Still unresolved.

Thank you for correcting Eq. (46). While this addresses my original counterexample, after examining Appendix A.1, I found a more fundamental issue in Lemma A.2.

The gradient-norm condition in Definition 3.1 does not imply the Hessian matrix inequality in Eq. (14)/(26). Specifically, Eq. (23) introduces a sufficient condition that does not follow from Eq. (22), but the subsequent proof treats it as established.

This issue leads to a counterexample to Theorem 3.3 itself. Consider

$ L_f(x,y)=\frac12(x^2+y^2),\quad L_r(x,y)=\frac12x^2+\frac1{20}(y-2)^2. $

The combined objective LD=111Lf+1011LrL_D=\frac1{11}L_f+\frac{10}{11}L_r is minimized at θD=(0,1)\theta^D=(0,1). Both Hessians are diagonal and positive definite, and the γ\gamma-FRG condition holds for every gradient-ascent iterate with γ=0.1\gamma=0.1. For η=1\eta=1 and T=4T=4, Algorithm 1 yields θ(T)=(0,16)\theta^{(T)}=(0,16), giving Δf=127.5\Delta_f=127.5 and Δr=9.75\Delta_r=9.75. Thus,

$ k=\frac{\Delta_r}{\Delta_f}\approx0.07647. $

However, with c=0.1c=0.1 and λf=1\lambda_f=1, the revised Theorem 3.3 claims k≤0.07k\leq0.07, which is violated. Since both losses are quadratic, the Taylor expansion is exact, so higher-order remainder terms cannot explain the discrepancy. Furthermore, the asymptotic claim as η→∞\eta\to\infty is not justified by discarding the O(η2)O(\eta^2) remainder in Eq. (48).

Consequently, I believe that this paper is not ready and I will keep my current rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their feedback. Below we summarize the main clarifications and additions. All changes are highlighted in blue in the revised paper.

1. Theory (4zcQ, 5djQ, zX9M).

  • Theorem 3.3 (4zcQ): we update Eq. 46 in its proof (Appendix A.3 of the revised paper). The conclusions of Theorem 3.3 and all other results are unchanged.
  • Theorem 4.2 (4zcQ): we revise its proof and update the theorem accordingly (Appendix A.6 of the revised paper).
  • Taylor remainder: Eq. (4) and Proposition 3.2 now include the remainder terms, and we state explicitly that Section 3 is a second-order analysis (Appendix G of the revised paper).

2. Novelty of the mechanism (4zcQ, 5djQ). By design, ISRO is equivalent to an adaptive learning rate KtηK_t\eta. Our contribution is the theoretical characterization of learning-rate sensitivity in retain-free unlearning (Theorems 3.3–3.4) and the efficiency guarantee of ISRO (Theorem 4.2); ISRO turns this into stable optimization across learning rates (Tables 1–3).

3. Evaluation extended (5djQ, zX9M). We apply ISRO to the MU-Mis objective on CIFAR-100 (ViT), where ISRO-MU-Mis achieves better retain performance than MU-Mis at most forgetting targets. We also compare with MU-Mis optimized by a stochastic Armijo line search (Appendix E.1, Table 7 of the revised paper), and report additional TOFU metrics: Forget Quality, Model Utility, ROUGE-L (Appendix E.3, Table 17 of the revised paper).

4. Claims clarified.

  • We soften the novelty and performance claims (Abstract; Section 5.2) and discuss MU-Mis and DualOptim (Appendix B.1).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.