- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies retain-free unlearning and shows that gradient ascent (GA) is sensitive to the learning rate. Large learning rates can give a better retain-forget trade-off but are unstable, while small learning rates are stable yet ineffective. It proposes ISRO, an optimizer wrapper that takes multiple inner ascent steps along a gradient computed once per epoch until the forget loss reaches a fraction α of a noise-based reference loss, turning unlearning into a threshold objective. Experiments across diverse tasks show that ISRO variants achieve better retain performance at matched forgetting levels and remain effective at small learning rates where vanilla methods fail.
Critical strengths
-
A simple, plug-in method. Learning-rate sensitivity of retain-free GA is a genuine obstacle, and ISRO composes with existing objectives (GA, RL, SalUn, ESD) with minimal modification. The claimed robustness is supported where it matters most. At lr = 1e-6 and 1e-7 in Tables 1 to 3, vanilla methods fail to reach any forgetting threshold while ISRO still succeeds, in all three domains. This suggests that the method widens the usable learning-rate range rather than improving a single tuned configuration.
-
The noise-based reference loss reframes unlearning as a threshold objective. Using as a normalized, task-agnostic target of sufficient forgetting (grounded by the bound in Proposition 4.1) replaces open-ended loss maximization with a controllable stopping criterion. This idea is transferable beyond the specific optimizer and gives the hyperparameter an interpretable meaning that the learning rate lacks.
Critical weaknesses
- The mechanism is not disentangled from an adaptive per-epoch step size, and the theory does not establish the claimed benefit. Inner steps reuse the stale gradient computed once per epoch. For a locally quadratic loss, inner steps of size along one fixed gradient equal a single step of size , so ISRO may largely function as an auto-tuned large learning rate. Meanwhile, Theorem 3.3 only shows that an upper bound on the trade-off ratio decreases with , under convexity and diagonal-Hessian assumptions, with holding trivially; this does not prove that the actual ratio improves. Without an ablation against a per-epoch enlarged learning rate or a line search or scheduler, the gains cannot be attributed to the threshold-stopping design rather than to simply taking larger steps. This directly concerns the central contribution claim.
- The evaluation never demonstrates complete forgetting and omits retain-free benchmarks and baselines. In the class-wise experiments (Table 1), results are reported only down to , meaning the model still answers 60% of the forget class correctly. The paper never shows that ISRO can reach while keeping at a meaningful level, which is the regime that defines success in full-class unlearning. This matters for two reasons. First, the motivating example in Section 2 asserts that approximately 0% forget accuracy with high retain accuracy is infeasible in the retain-free setting, but recent retain-free work contradicts this. MU-Mis [1] achieves forget accuracy near the retrain oracle with retain accuracy close to retrain on full-class CIFAR-100 and TinyImageNet, and SEMU [2] similarly reaches retrain-level unlearning accuracy with minimal retain damage. Second, both works evaluate random-data forgetting (randomly forgetting 10% to 50% of training data). Finally, SEMU is cited in related work but never compared experimentally, and MU-Mis, a retain-free method built on the same base objectives that ISRO wraps (negative gradient and random labeling), is not cited. On the LLM side, the evaluation reports only forget and retain QA probability, omitting the standard TOFU suite (Truth Ratio, ROUGE-L, Model Utility); ISRO-GA also regresses below GA at lr = 1e-5 for deeper forgetting (Table 3). Meanwhile, is strongly task-dependent ([0.01, 0.05] for LLMs versus [0.1, 0.9] elsewhere) and cannot be validated without retain data, so the advertised hyperparameter insensitivity is partially relocated rather than removed.
[1] Cheng et al.,Remaining-data-free Machine Unlearning by Suppressing Sample Contribution
[2] Sendera et al., SEMU: Singular Value Decomposition for Efficient Machine Unlearning
Questions for authors
Clarification requests:
- [Mechanism attribution] Inner steps reuse the stale, once-per-epoch gradient; for a quadratic loss, steps of size equal one step of size . Can you ablate ISRO against (a) a per-epoch enlarged learning rate and (b) a standard learning-rate scheduler or line search? This determines whether the gains come from the threshold-stopping design or from larger steps, and is decisive for the contribution claim.
- [Comparison with retain-free baselines and complete forgetting] Can you (a) extend Table 1 to and show whether ISRO preserves a meaningful acc_r there, and (b) compare against MU-Mis [1] and SEMU [2], both retain-free, on class-wise forgetting with FA → 0 and on random-data forgetting (for example 10% and 50% of CIFAR-10/100)? MU-Mis shares the exact base objectives that ISRO wraps (negative gradient, random labeling), so this comparison directly tests whether ISRO's optimizer-level gains persist against state-of-the-art retain-free methods.
- [Evaluation completeness] Can you report the full TOFU suite (Truth Ratio, ROUGE-L, Model Utility) for the LLM experiments? QA probability alone cannot establish genuine forgetting, so these results would directly change my Soundness assessment.
- [Table 3 regression] Why does ISRO-GA underperform GA at lr = 1e-5 for ? What was used per learning rate?
- [ selection in practice] The ranges differ by an order of magnitude across domains ([0.01, 0.05] for LLMs versus [0.1, 0.9] elsewhere), and no retain set is available for validation in this setting. What procedure do you recommend for selecting ?
- [Optimizer details] Do ISRO's inner steps use Adam moments or the raw gradient? If Adam is used, how do the momentum and second-moment estimates interact with the stale gradient? This affects both reproducibility and the interpretation of the mechanism.
- [Overclaim] You claimed that your method is the first unlearning-oriented optimizer, but there is another optimizer-based work, DualOptim [3], which also targets to improve unlearning stability even though it is not designed for retain-free unlearning tasks.
[1] Cheng et al.,Remaining-data-free Machine Unlearning by Suppressing Sample Contribution
[2] Sendera et al., SEMU: Singular Value Decomposition for Efficient Machine Unlearning
[3] Zhong et al., DualOptim: Enhancing Efficacy and Stability in Machine Unlearning with Dual Optimizers
4 comments
Rebuttal 1 - P2
[W2, Q2] Complete forgetting () and comparison with retain-free baselines.
Response.
Retain-free baselines are already included. All experiments are run in the retain-free setting and compare several distinct unlearning methods across three domains (Tables 1–3; Appendix E, Tables 5–15). They fall into two groups:
- Methods designed to be retain-free (forget data only): GA, Random Labeling (RL), Boundary Expanding (BE) and Boundary Shrink (BS) for classification, and ESD for Stable Diffusion.
- Methods adapted to the retain-free setting by removing their retain-set loss: SalUn and LoTUS for classification, SalUn for Stable Diffusion, and NPO, SimNPO, RMU and IdkPO for LLMs.
The adaptation is stated in the note at the beginning of Section 5, and all setups are described in Appendix C.2–C.4. We also add MU-Mis [1] at our 30% scale (Table R1; Appendix E.1, Table 7 of the revised paper).
Different forgetting scales. MU-Mis evaluates full-class unlearning of a single class ("rocket", i.e. 1% of CIFAR-100). Our experiments are far more aggressive: we forget 30% and 50% of the classes (Section 5.1; Appendix Tables 5–6), i.e. 30–50 times more data. As the forget set grows, the coupling between forget and retain gradients grows as well: on CIFAR-100, rises from 0.283 at 3% to 0.891 at 10% of the classes (Appendix C.1, Table 4). Without retain data, retain degradation then becomes inevitable as the forget accuracy decreases, and Table 1 shows this for every method. This is why we formulate retain-free unlearning as reaching a target forgetting level (Definition 2.1; practical target in Section 2), and why our experiments report at the target levels (Appendix C.5).
On the Section 2 example (paragraph "Example"). The example illustrates that, without , lowering forget accuracy also lowers retain accuracy, so the ideal target of 90% retain and 0% forget accuracy cannot be guaranteed.
References
- [1] Cheng et al., Remaining-data-free Machine Unlearning by Suppressing Sample Contribution (MU-Mis), ICLR 2026.
[W2, Q2] (cont.) MU-Mis at our forgetting scale (30% of the classes).
Response. We use the MU-Mis objective [1] as the base method and compare MU-Mis with ISRO-MU-Mis on ViT for CIFAR-100, forgetting 30% of the classes. We report at the first update where falls below each target from 0.9 to 0.1 (Table R1; Appendix E.1, Table 7 of the revised paper).
- . Vanilla MU-Mis reaches no target ( stays at 88.74%), whereas ISRO-MU-Mis reaches every target down to 0.1 for every .
- . For every , ISRO-MU-Mis keeps a higher than MU-Mis at every target from 0.7 to 0.1.
- Across learning rates. ISRO-MU-Mis at , exceeds MU-Mis at its best learning rate at every target except 0.8.
References
- [1] Cheng et al., Remaining-data-free Machine Unlearning by Suppressing Sample Contribution (MU-Mis), ICLR 2026.
Rebuttal 1 - P3
[Q4] Why does ISRO-GA underperform GA at for ? Which was used?
Response.
Why. As we already discuss in Section 3.1 and illustrate in Fig. 2b, a large learning rate makes unlearning unstable and hard to control: a single step can overshoot the forgetting target. , the largest learning rate in Table 3, falls in this regime. ISRO checks its stopping rule after every step of size ; at this learning rate, each step is too large to stop precisely at the stricter forgetting levels (). ISRO's gains lie at smaller learning rates:
- Small learning rates. At , vanilla GA and NPO fail to reach any forgetting threshold, while ISRO-GA and ISRO-NPO reach all of them (Table 3; Appendix Tables 10–11).
- Best achievable performance. Over all learning rates, the best ISRO-GA result exceeds the best GA result at every level (Table 3).
Which . We sweep (Appendix C.4) and report the best result in Table 3. Experiments on the robustness of ISRO to are shown in Fig. 4 and Appendix F.1 (Figs. 19–24).
[Q3] Full TOFU evaluation suite.
Response. The revised paper reports TOFU's probability metric on the forget and retain sets (, ; Table 3 and Appendix Tables 8–15). We also report Forget Quality, Model Utility, and ROUGE-L (Table R2; Appendix E.3, Table 17 of the revised paper).
Table R2. SimNPO on Llama-3.2-1B-Instruct, TOFU-10%. All metrics are measured at the checkpoint where first falls below 0.70 (consistent with Appendix Tables 8–11); ✗ = level never reached.
| lr | Method | ↓ | ↑ | FQ ↑ | MU ↑ | ROUGE-L (f) ↓ | ROUGE-L (r) ↑ |
|---|---|---|---|---|---|---|---|
| SimNPO | 0.653 | 0.704 | 0.482 | 0.880 | 0.424 | 0.711 | |
| ISRO-SimNPO | 0.649 | 0.704 | 0.486 | 0.880 | 0.422 | 0.711 | |
| SimNPO | 0.655 | 0.698 | 0.479 | 0.872 | 0.426 | 0.705 | |
| ISRO-SimNPO | 0.652 | 0.698 | 0.483 | 0.872 | 0.424 | 0.705 | |
| SimNPO | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | |
| ISRO-SimNPO | 0.656 | 0.703 | 0.478 | 0.879 | 0.426 | 0.710 | |
| SimNPO | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | |
| ISRO-SimNPO | 0.671 | 0.704 | 0.462 | 0.880 | 0.436 | 0.711 |
At large learning rates, where both methods reach the target, ISRO-SimNPO matches or slightly improves SimNPO on all metrics. At small learning rates, vanilla SimNPO never reaches the target, whereas ISRO-SimNPO reaches it with high Forget Quality while keeping Model Utility close to the original model.
References
- Fan et al., Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning, 2025.
[Q7] Overclaim: "first unlearning-oriented optimizer"; DualOptim.
Response. We thank the reviewer for pointing us to this work. DualOptim [1] requires a retain set, whereas ISRO is an optimizer-level method designed for the retain-free setting. We have refined this claim in the revised paper and discuss DualOptim in Appendix B.1.
References
- [1] Zhong et al., DualOptim: Enhancing Efficacy and Stability in Machine Unlearning with Dual Optimizers, NeurIPS 2025.
[Q5] How should be selected without a retain set? Why do the ranges differ by an order of magnitude?
Response. For tuning , our theorems do not give specific guidance. However, empirically, Appendix G shows robustness across for image classification and NSFW Stable Diffusion unlearning, but a narrower range, typically , for LLM unlearning. In practice, we therefore recommend choosing within these ranges. We hypothesize that LLMs require a smaller because their \texttt{loss\\_ref}, computed over a vocabulary of about 128K tokens rather than 100 classes, is much larger, so a small fraction of it already suffices for forgetting.
[Q6] Do the inner steps use Adam moments or the raw gradient?
Response. ISRO is applied after the momentum and bias-correction steps of the base optimizer (Adam/AdamW). In each epoch, Adam computes its update direction once from the gradient, including momentum and bias correction, and ISRO repeats this update in the inner loop until the threshold is reached. The moment estimates are therefore updated once per gradient computation, exactly as in vanilla Adam, and do not depend on the number of inner steps. Algorithm 2 writes the update in its plain-gradient form for readability; we have stated this explicitly in Appendix C.6 of the revised paper.
Thanks for your rebuttal. After reviewing other reviewers' comments, I think my current rating can reflect the quality of this paper. Therefore, I will keep my rating.
Rebuttal 1 - P1
We thank the reviewer for the detailed review.
[W1, Q1] The mechanism is not disentangled from an adaptive per-epoch step size, and the theory does not establish the claimed benefit. Can you ablate ISRO against (a) a per-epoch enlarged learning rate Kη and (b) a standard learning-rate scheduler or line search?
Response.
W1: Is ISRO an auto-tuned large learning rate? Yes, by design, ISRO is equivalent to an adaptive per-epoch learning rate Ktη, i.e., θt+1=θt±Ktη,∇L(θt). However, how the learning rate affects unlearning, and how to adapt the learning rate accordingly, remain largely unexplored. We are the first to theoretically characterize the learning-rate sensitivity of retain-free unlearning (Theorems 3.3–3.4; Fig. 3; Appendix D). ISRO turns this into stable optimization: it achieves consistent unlearning performance across learning rates spanning several orders of magnitude, and remains effective at small learning rates where vanilla optimizers fail to unlearn (Tables 1–3).
W1: Does the theory establish the claimed benefit? We would like to clarify our theoretical contributions as follows. Theorems 3.3 and 3.4 diagnose the learning-rate sensitivity of vanilla GA. Theorem 3.3 shows that a larger learning rate can yield a more favorable retain-forget trade-off. Theorem 3.4 explains why simply using a large learning rate is difficult: as η increases, the iteration window satisfying the (ϵr,ϵf) criterion becomes increasingly narrow. These findings motivate ISRO's threshold-based mechanism, which reuses a computed gradient direction until a target reference loss is reached, rather than taking a single update per gradient computation. Theorem 4.2 then analyzes this mechanism by establishing an upper bound on the number of inner steps required to reach the α-defined loss threshold. Thus, Theorems 3.3 and 3.4 motivate the design of ISRO, while Theorem 4.2 provides its theoretical efficiency guarantee.
W1: Theorem 3.3 only bounds the trade-off ratio; does the actual ratio improve? The actual ratio is reflected in Tables 1–3 and 5–15: at each fixed forgetting target, the methods optimized with ISRO achieve better retain performance than their vanilla counterparts, i.e., less retain degradation for the same amount of forgetting, and hence an improved k.
Q1 (a): Per-epoch enlarged learning rate Kη. Fig. 5 and Appendix F.2 (Figs. 25–30: GA, RL, SalUn, BE, BS and LoTUS, each at η∈10−6,10−5) already show the value of Kt at each iteration, i.e., the per-iteration enlarged learning rate Ktη that ISRO selects. Kt varies sharply across iterations: it is about 103 in the first iteration ($\sim$1000 for α=0.1, $\sim$3000 for α=0.9) and drops to only 1–2 after the first few iterations.
Q1 (b): Line search. We compare ISRO with a stochastic Armijo line search (SLS) [1], both applied to MU-Mis (Table R1). SLS-MU-Mis fails to reach the deeper forgetting levels at η=10−5 and any level at η=10−6, whereas ISRO-MU-Mis reaches all levels at both learning rates with better retain performance at most levels.
Table R1 (added to Appendix E.1, Table 7 of the revised paper). Retain performance accr (%) at different accf levels under various settings for ViT-based 30% class-wise unlearning on CIFAR-100 with MU-Mis. SLS-MU-Mis = MU-Mis optimized with SLS [1] with Armijo constant c.
References