- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes Gauge-Induced Teleportation (GIT), a parameter-efficient LLM Unlearning method based on gauge transformation. The authors first observe that existing Unlearning methods often suffer from premature pareto saturation during training, meaning that the Forget–Retain Trade-off reaches its limit early. To address this issue, the authors use the Gauge Freedom in LoRA to explore different optimization trajectories without changing the current model outputs.
Critical strengths
The experimental results seem to show that GIT is effective to some extent. The authors conduct experiments on the TOFU and MUSE Benchmarks, covering models of different sizes and several existing Unlearning methods. The results show that GIT can improve the Forget–Retain Trade-off in several settings. These results provide some evidence for the effectiveness of GIT.
Critical weaknesses
W1: The writing and organization of the paper are not clear enough, making the method difficult to understand. The overall structure of the paper could be improved, and some sections are not presented in a clear order. For example, in Section 4 (Motivation), the authors present several experimental results related to GIT in Figures 2 and 4 before formally introducing the method. In particular, Figure 4 involves specific components such as Orbit Marginalization, but these concepts are not clearly explained until Section 5. This makes it difficult for readers to understand the purpose and meaning of these experiments. In addition, Section 6 is relatively short and mainly focuses on Table 1, while important experiments, such as the Ablation Study and MUSE Benchmark, are placed in the Appendix without enough discussion in the main text. The main paper also lacks a separate Conclusion section (which is only included in Appendix F), making the paper end rather suddenly. I suggest that the authors improve the organization of the paper and move some important experiments and analyses to the main text to make the paper clearer and more complete.
W2: The motivation for GIT and its connection to LLM Unlearning are not sufficiently explained. Gauge Transformation is a general LoRA parameter transformation and optimization technique. It does not directly depend on the LLM Unlearning task. However, the paper does not clearly explain why this technique can effectively address the Forget–Retain Trade-off in Unlearning. Section 4 mainly shows that existing Unlearning methods suffer from Premature Pareto Saturation and that different Gauges can lead to different optimization trajectories and Pareto Frontiers. However, these results are not enough to show that Fixed Gauge is an important cause of Premature Pareto Saturation. The paper also does not clearly explain how Gauge Transformation improves Forget–Retain Optimization. In addition, the main components of GIT, including Orbit Marginalization and Delayed Gauge Fixing, mainly focus on general LoRA optimization problems. They seem to be applicable to other training tasks, such as standard SFT. Therefore, although the experiments show that GIT can improve the performance of some Unlearning methods, its connection to the specific challenges of Unlearning is still unclear. I suggest that the authors provide more analysis to explain how Gauge Transformation helps improve the Forget–Retain Trade-off and better support the motivation and need for the proposed method.
Questions for authors
-
Gauge Transformation is a general LoRA optimization technique. Could the authors further explain why it is particularly useful for Unlearning, rather than other training tasks such as standard SFT?
-
Section 4 partly links Premature Pareto Saturation to Fixed Gauge. Could the authors provide more direct evidence to support this claim? Could other optimization factors also cause this problem?
-
In Table 4, Retain Performance improves significantly only after adding SCF. Could the authors add experiments with Baseline + SCF to better show how much improvement comes from Gauge Optimization itself?
-
GIT needs to compute multiple Gauges at each step and also requires extra Rollouts. Could the authors report the actual computational cost?