GeSPU: Retain-Anchored Spectral Projection for Budgeted LLM Unlearning

Parameter-efficient machine unlearning requires purging requested information from a pretrained model while preserving retained utility within a strict deployment footprint. Existing low-rank adaptation methods decouple direction from capacity: updates are optimized in isotropic Euclidean space, while ranks are allocated uniformly or through post-hoc heuristics. Consequently, explicit negative deletion gradients inject extreme update energy that triggers catastrophic parameter collisions on retained tasks. We introduce Geometric Spectral Unlearning (GESPU), which formulates budgeted unlearning as a constrained optimization problem on a Kronecker-factored retain-curvature manifold. We prove that minimizing the resulting local surrogate admits a closed-form spectral solution via Eckart–Young singular-value truncation coupled with a discrete Multiple-Choice Knapsack problem, guaranteeing global surrogate optimality with certified descent under exact steps. Crucially, before retained loss gradients become active, the retain-metric factors act as an anisotropic filter, steering initial deletion updates away from sensitive retained pathways. A dual-timescale schedule optimizes lowdimensional core matrices between periodic subspace refreshes, exporting standard low-rank adapters with zero deployment overhead. Evaluated on TOFU and WMDP benchmarks across LLaMA-3 and Qwen2.5 models, GESPU establishes superior Pareto trade-offs over existing PEFT baselines, effectively suppressing target knowledge, preserving complex reasoning capabilities, and resisting knowledge recovery under benign relearning.