- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
It is known that merging several LoRA adaptors can result in worse per-task performance due to, for example, cancellation of weights. This paper broadly studies when an infeasible combined pair of LoRA adaptors can be "rescued" by merging a third adaptor. The authors provide a particular rank-one regime where rescuers can be easily found. The experiments look at several banks of LoRA adaptors and show that most bad pairs can be rescued by merging a third LoRA adaptor.
Critical strengths
None
Critical weaknesses
- I found this paper to be extremely hard to parse in terms of both its motivation and its main theorems. There are many times when terms are used before definitions are given, if a definition is given at all. This makes its very difficult to understand what is actually being proved. For example, I can't find a definition of a "opposite-type pair" or "feasibility". The related works seems to just list out papers and their main contributions without addressing how they relate to the work at hand. I also find it strange that the main body is only 6.5 pages long, yet the total paper is 60 pages. It would have been nice to use the extra space in the main body to flesh out the general motivation, definitions, and notation further.
Questions for authors
None
Thank you for pointing out where the motivation, definitions, and connections to related work could be clearer. We will draw on your feedback as we continue to refine the paper after this review cycle.