Theory of Certified Unlearning for a Family of Regularization-Based Continual Learning

Integrating certified unlearning into continual learning, an emerging paradigm termed continual learning-unlearning (CLU), enables models to learn sequentially while retaining past knowledge and reliably removing requested information to protect privacy. Despite its importance, CLU remains largely unexplored. We provide the first unified $(\epsilon ,\delta )$-certified CLU framework for a broad family of regularization-based CL algorithms, with the objective of jointly minimizing CL excess risk and unlearning loss. We derive an algorithm-dependent CL excess-risk bound for nonconvex losses, showing that regularizers that better accommodate task drift, such as Hessian-based regularization, improve learning performance. We then develop two complementary certified-unlearning mechanisms. Our first-order mechanism exploits natural forgetting induced by CL updates with minimal additional cost. Our second-order method enables precise request-specific unlearning for arbitrary requests, with unlearning loss strongly depending on request order through prior-correction propagation and regularizer mismatch. Both methods show that CL algorithms with stronger regularization, such as MAS, achieve lower unlearning loss, revealing a fundamental trade-off between retaining undeleted-task knowledge and removing deleted-task influence. Empirical results reveal distinct Pareto frontiers between certified-unlearning loss and CL excess risk, showing that no single method minimizes both. Thus, method selection should prioritize lower unlearning loss for stringent privacy requirements and lower CL excess risk for better predictive performance.