acceptodds
Under review as a conference paper at ICLR 2027

Gauge-Induced Teleportation for Parameter-Efficient LLM Unlearning

Abstract

Parameter-efficient LLM unlearning has emerged as an important research direction that aims to efficiently remove specific knowledge while preserving the model's remaining general capabilities. In this paper, we first observe that existing methods typically reach the empirical Pareto frontier prematurely, with limited further improvement even when optimization is prolonged. We therefore turn to an additional degree of freedom in optimization, i.e., gauge transformation, which is largely overlooked but can induce alternative optimization trajectories under the same LoRA function. However, gauge selection itself presents a temporal dilemma: selecting a gauge too early incurs substantial long-horizon risk, whereas selecting it too late suffers from history-induced incompatibility with the accumulated optimization state. To address this issue, we first introduce an orbit-marginalized training strategy before gauge selection to improve compatibility across gauge choices. We then employ an -step rollout proxy to identify a favorable gauge for subsequent standard optimization. Furthermore, we adopt a reference-relative objective stabilization mechanism to prevent excessive optimization of the forgetting objective. Extensive experiments across standard LLM unlearning benchmarks and diverse base models demonstrate that Ours consistently pushes beyond the empirical limits of existing approaches by non-negligible margins, establishing new competitive performance under prevailing evaluation metrics.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
LLM Unlearning, Teleportation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UaBVOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Gauge-Induced Teleportation (GIT), a parameter-efficient LLM Unlearning method based on gauge transformation. The authors first observe that existing Unlearning methods often suffer from premature pareto saturation during training, meaning that the Forget–Retain Trade-off reaches its limit early. To address this issue, the authors use the Gauge Freedom in LoRA to explore different optimization trajectories without changing the current model outputs.

Critical strengths

The experimental results seem to show that GIT is effective to some extent. The authors conduct experiments on the TOFU and MUSE Benchmarks, covering models of different sizes and several existing Unlearning methods. The results show that GIT can improve the Forget–Retain Trade-off in several settings. These results provide some evidence for the effectiveness of GIT.

Critical weaknesses

W1: The writing and organization of the paper are not clear enough, making the method difficult to understand. The overall structure of the paper could be improved, and some sections are not presented in a clear order. For example, in Section 4 (Motivation), the authors present several experimental results related to GIT in Figures 2 and 4 before formally introducing the method. In particular, Figure 4 involves specific components such as Orbit Marginalization, but these concepts are not clearly explained until Section 5. This makes it difficult for readers to understand the purpose and meaning of these experiments. In addition, Section 6 is relatively short and mainly focuses on Table 1, while important experiments, such as the Ablation Study and MUSE Benchmark, are placed in the Appendix without enough discussion in the main text. The main paper also lacks a separate Conclusion section (which is only included in Appendix F), making the paper end rather suddenly. I suggest that the authors improve the organization of the paper and move some important experiments and analyses to the main text to make the paper clearer and more complete.

W2: The motivation for GIT and its connection to LLM Unlearning are not sufficiently explained. Gauge Transformation is a general LoRA parameter transformation and optimization technique. It does not directly depend on the LLM Unlearning task. However, the paper does not clearly explain why this technique can effectively address the Forget–Retain Trade-off in Unlearning. Section 4 mainly shows that existing Unlearning methods suffer from Premature Pareto Saturation and that different Gauges can lead to different optimization trajectories and Pareto Frontiers. However, these results are not enough to show that Fixed Gauge is an important cause of Premature Pareto Saturation. The paper also does not clearly explain how Gauge Transformation improves Forget–Retain Optimization. In addition, the main components of GIT, including Orbit Marginalization and Delayed Gauge Fixing, mainly focus on general LoRA optimization problems. They seem to be applicable to other training tasks, such as standard SFT. Therefore, although the experiments show that GIT can improve the performance of some Unlearning methods, its connection to the specific challenges of Unlearning is still unclear. I suggest that the authors provide more analysis to explain how Gauge Transformation helps improve the Forget–Retain Trade-off and better support the motivation and need for the proposed method.

Questions for authors
  1. Gauge Transformation is a general LoRA optimization technique. Could the authors further explain why it is particularly useful for Unlearning, rather than other training tasks such as standard SFT?

  2. Section 4 partly links Premature Pareto Saturation to Fixed Gauge. Could the authors provide more direct evidence to support this claim? Could other optimization factors also cause this problem?

  3. In Table 4, Retain Performance improves significantly only after adding SCF. Could the authors add experiments with Baseline + SCF to better show how much improvement comes from Gauge Optimization itself?

  4. GIT needs to compute multiple Gauges at each step and also requires extra Rollouts. Could the authors report the actual computational cost?

Reviewer_rmw2OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes gauge-induced teleportation (GIT) for parameter-efficient LLM unlearning.

Critical strengths

S1: This paper conducts comprehensive experiments.

S2: This paper discloses experimental details.

Critical weaknesses

W1 (undefined notations): Many notations are only verbally described but never rigorously defined, so it would be impossible to check the correctness of their theoretical claims. Consequently, the faithfulness of their AI use statement becomes questionable: The authors claim that they used AI to check the correctness of proofs, but this seems impossible as many notations remain undefined to AI if they used AI as how they claimed.

W2 (lack of justification): Many design choices in Section 5 lack justification. It is unclear whether or not there exist better choices.

W3 (irrelevant theory): Theorem 1 and Proposition 1 are irrelevant to the motivation or justification of the proposed method. The paper also admits that these theoretical results have almost no implication about the proposed method (lines 251-256, 414-418). Therefore, it is unclear why the authors decided to present these theoretical results in the paper.

W4 (inconsistent improvement): The proposed method fails to achieve consistent improvement for various methods on different datasets. Since this method introduces a lot of hyperparameters, it is a bit hard to distinguish the performance improvement from pure hyperparameter tuning.

Questions for authors

See weaknesses.

Reviewer_ut1qOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors introduce a novel method for optimizing the trade-off between forgetting and retention in parameter-efficient unlearning. The paper proposes a novel framework, Gauge-Induced Teleportation (GIT), which consists of three components: (1) orbit marginalization, which involves making one-step updates with multiple gauges and averaging the resultant states after mapping them to a common set of coordinates; (2) H-step rollout proxy, which involves evaluating several candidate gauges by running H-step updates and selecting one; and (3) saturation-controlled forgetting, which involves applying a scaling factor to the forgetting loss based on the difference in negative log-likelihood for forget-set examples between the current model and the pre-unlearning reference model. The paper motivates its proposed framework by showing the superior coverage of the forgetting–retention Pareto frontier by the proposed method compared to the baselines. The paper also provides theoretical and/or empirical motivations for the components of the proposed framework. Empirical results using LLM unlearning benchmarks show that the proposed method improves the performance of baselines when the GIT framework is introduced.

Critical strengths
  • The paper discusses a pertinent problem in the machine learning domain, which is unlearning in LLMs. The authors expose the limitations of the existing methods in attaining a desirable trade-off between forgetting and retention.
  • The paper provides theoretical and/or empirical validation for the design choices of the proposed GIT framework.
Critical weaknesses
  • The proposed framework consists of multiple components, but it is unclear how much each component contributes to the improved performance. More comprehensive ablation studies and a detailed discussion in the main text seem necessary to better understand the contribution of each component.
  • Some important details needed for understanding the paper seem missing/deferred to the appendix. For example, the (empirical) motivation for introducing saturation-controlled forgetting, which is an important component of the GIT framework, is not provided in the main text.
  • It is unclear whether the computational overhead for the proposed method, specifically used for multiple rollouts for selecting candidate gauges, is justified by the marginal performance gains (and on some occasions degradations) shown in Table 1.
Questions for authors
  • Can the authors provide comprehensive ablation studies and discussion to better understand the individual contributions of each component of GIT?
  • Can the authors provide a more detailed explanation of why GIT does not consistently outperform its corresponding baselines across all evaluation metrics, despite orbit marginalization, selecting favorable gauges using the H-step rollout proxy, and saturation-controlled forgetting?
  • Can the authors report the additional computational overhead introduced by GIT, including orbit-marginalized training and the H-step candidate rollouts, and discuss whether the performance improvements justify this overhead compared to the baselines?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.