- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Balanced-KnOTS, a task-balanced subspace selection method for merging multiple LoRA adapters into a single adapter with a fixed rank. The authors argue that conventional joint SVD maximizes aggregate reconstruction energy but may fail to preserve task-specific information, especially when the number of tasks increases. To address this issue, the paper introduces task-wise projection coverage to measure the information retained for each task and proposes a mean-soft-min objective to balance average and worst-task coverage. The selected subspace is then combined with TIES to construct the merged adapter. Experiments on 20 LoRA experts trained on security-related tasks show improvements in worst-task projection coverage and downstream task performance compared with Joint KnOTS-TIES.
Critical strengths
- The paper is overall well-written and easy to follow
- Their motivation is clear and makes sense. Their method addresses their proposed challenge, supported by empirical results.
- Their experiments are comprehensive with sufficient ablation study.
Critical weaknesses
- The main concern is the diversity of their experiments. They only tested their method on one model with one set of datasets. I recommend adding more model families and benchmarks.
- Another concern is their computational cost. I think their method would be costly, and I suggest that the authors report the runtime instead of a theoretical cost analysis.
Questions for authors
- In Sec.4.3, I wonder if the searched subspace by joint SVD still favors the tasks with a higher coverage?
- Sec.5 is not well motivated by Sec.4 (lines 251-252): why do we need a dynamic router rather than a static LoRA? How do a router solve the problems in Sec.4?
- There is a trade-off when we balance the subspace: when the main task is suppressed, other tasks may benefit, but the main task's performance may drop; when the main task is well preserved, other tasks suffer. For me, it is hard to tell which option is better.
- I am interested in the comparison between the task coverage and the performance drop after merging. Not sure whether the main task will suffer a higher performance drop.