Gauge-Induced Teleportation for Parameter-Efficient LLM Unlearning

Parameter-efficient LLM unlearning has emerged as an important research direction that aims to efficiently remove specific knowledge while preserving the model's remaining general capabilities. In this paper, we first observe that existing methods typically reach the empirical Pareto frontier prematurely, with limited further improvement even when optimization is prolonged. We therefore turn to an additional degree of freedom in optimization, i.e., gauge transformation, which is largely overlooked but can induce alternative optimization trajectories under the same LoRA function. However, gauge selection itself presents a temporal dilemma: selecting a gauge too early incurs substantial long-horizon risk, whereas selecting it too late suffers from history-induced incompatibility with the accumulated optimization state. To address this issue, we first introduce an orbit-marginalized training strategy before gauge selection to improve compatibility across gauge choices. We then employ an $H$-step rollout proxy to identify a favorable gauge for subsequent standard optimization. Furthermore, we adopt a reference-relative objective stabilization mechanism to prevent excessive optimization of the forgetting objective. Extensive experiments across standard LLM unlearning benchmarks and diverse base models demonstrate that Ours consistently pushes beyond the empirical limits of existing approaches by non-negligible margins, establishing new competitive performance under prevailing evaluation metrics.