ISRO: Inner-Step Refinement Optimization for Retain-Free Unlearning

Retain-free unlearning involves a fundamental trade-off between effective forgetting and preserving model utility. In this work, we show that using large learning rates can achieve a more favorable trade-off, but often suffers from instability, whereas small learning rates provide stable updates yet result in less effective unlearning. It raises a need for heuristic hyperparameter tuning to achieve optimal results. To address this limitation, we propose Inner-Step Refinement Optimization (ISRO), the first unlearning-oriented optimizer that stabilizes learning rate sensitivity while optimizing the retain-forget trade-off. Extensive experiments demonstrate that ISRO integrates effectively with existing methods and consistently outperforms vanilla approaches: for the same level of forgetting, it achieves better utility with fewer epochs and lower computational cost. Our findings are supported by theoretical analysis and empirical results across image classification, image generation, and large language model unlearning tasks.