Sequential Subspace Noise Injection Prevents Accuracy Collapse in Certified Unlearning

Certified unlearning via differential privacy provides formal guarantees but remains impractical due to the severe accuracy collapse observed in current noisy fine-tuning (NFT) approaches. We propose Block-wise Noisy Fine-Tuning, which utilizes sequential subspace noise injection to partition the parameter space into orthogonal blocks and update only one block per step. This modification redistributes the noise budget over time; by restricting perturbations to lower-dimensional subspaces, we significantly reduce per-step distortion and prevent noise from overwhelming the training signal. We extend the privacy analysis to our block-wise schedule, proving that the same $(\epsilon ,\delta )$ privacy budget is retained. Empirical results on image classification benchmarks demonstrate that our approach prevents accuracy collapse, bridging the gap between rigorous guarantees and practical utility.