- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper makes a comparative and diagnostic contribution rather than introducing a new method or establishing a new performance record. It investigates how different parameter-efficient fine-tuning (PEFT) methods compare, examining six methods across language and diffusion models in terms of task performance, forgetting, and changes in pre-trained weight geometry.
The findings suggest that different PEFT methods can achieve similar task performance while exhibiting substantially different levels of forgetting. LoRA often preserves pre-trained behavior most effectively, DoRA frequently achieves somewhat higher task scores, and PiSSA tends to incur greater retention costs. The paper also finds that explicitly preserving pre-trained weight geometry does not consistently lead to better outcomes.
Critical strengths
The paper provides an effective empirical comparison of different PEFT methods, with several interesting observations regarding the relationship between adaptation performance, forgetting, and pre-trained weight geometry.
In particular, the authors show that restoring pre-trained singular values often preserves adaptation performance. They also employ selective spectral interventions to investigate which components of the weight matrices contribute to adaptation and forgetting. These experiments provide interesting insights into the behavior of different PEFT methods.
Critical weaknesses
My main reservation is the lack of a significant theoretical contribution. The paper is primarily comparative and diagnostic, without introducing a new method or establishing a new performance record.
I would support acceptance if this type of empirical and comparative contribution is considered sufficient under ICLR's standards. The experimental observations are interesting, but the overall significance of the contribution should be evaluated in that context.
Questions for authors
None.
3 comments
Thanks for your comments. I therefore maintain my scores.
Dear Reviewer adDB,
Thank you for taking the time to review our work and consider our response.
Kind regards, The Authors
Empirical contribution. We appreciate the reviewer’s positive assessment and agree that our main contribution is empirical. ICLR’s reviewer guide recognizes new empirical knowledge as a contribution.
Motivation. Many PEFT variants claim gains over LoRA, but these gains do not always stand the test of time. We ask which designs improve performance or reduce forgetting under equal tuning budgets, and which patterns of adaptation and forgetting hold across methods.
Our findings.