Behavioral Guarantees for Proxy Based Unlearning

This paper generalizes recent proxy based unlearning methods and proves theoretical guarantees about the behavior of the resulting unlearned model: upper bounds on the Kullback-Leibler divergence from the ideal retrained. We model approximate unlearning as a constrained optimization problem and interpret the family of solutions as introducing a scaled unlearning signal in the output space. The unlearning signal arises from proxies of the posterior data distributions. Its scale is adapted to the proxies to ensure the behavioral upper bounds. This framework relies on the structure of the data distributions in order to create proxies. If need be, the solution serves as a teacher to distill the update in the weights. Our approach is experimentally validated over three forgetting scenarios as reaching the closest classifier to the ideal retrained model.