Forgetting Must Ignite: Overcoming Under-Forgetting in Machine Unlearning with Pair-Free Cauchy–Schwarz Angular Repulsion

Approximate machine unlearning seeks to remove the influence of a forget set at substantially lower cost than full retraining. Yet strong baselines such as fine-tuning often under-forget: they preserve utility while leaving forget-set representations close to their original, memorized state. We address this failure mode with CSD-Unlearning, a feature-level objective based on Cauchy–Schwarz divergence that preserves retain-set representations while selectively repelling forget-set features from the original model. We show that the resulting objective admits a geometric interpretation as a negative log-cosine divergence between kernel mean embeddings. This analysis reveals that the repulsive gradient vanishes for deterministic feature extraction when the current and reference representations coincide. A capped one-dimensional surrogate further exhibits a sharp transition between locally suppressing and amplifying nonzero displacements, providing a mechanistic explanation for the threshold-like behavior observed in several training sweeps. Across CIFAR-10, CIFAR-100, and Tiny ImageNet with ResNet-18 and vision transformers, CSD-Unlearning consistently reduces the gap to retraining when added to fine-tuning. We also characterize its regime of effectiveness: the method is most useful for under-forgetting baselines, provides limited gains once a baseline already over-forgets, and degrades in selectivity at large forget ratios or outside a narrow stability range on vision transformers.