- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors propose a preprocessing step prior to any LoRA weight fusion. They "align" these LoRA adapters in order to reduce the variability due to potential transformations in the latent space, which do not change the individual adapters' outputs but may affect the results when they are fused.
Critical strengths
- The paper is well written and easy to follow. The problem is clearly stated, and the proposed solution is simple and fast to run, making it practical to use in real-world applications.
- The experimental evaluation is extensive and addresses the most important questions raised by the proposed approach, including alternative alignment strategies (e.g., permutation-based alignment), the benefit of SOA initialization before GRA (appendix), the reduction of factor-space misalignment, computational overhead, and out-of-distribution performance. In addition, many fusion methods are explored, along with several variants of the proposed alignment method (A-only, B-only, and Joint). Despite considerable variation in performance depending on the task, with no consistent winner among the alignment variants, the proposed solution generally brings improvements across the different fusion methods.
Critical weaknesses
- Despite the main experiments supporting the paper's central claim and the many promising results, I think the paper lacks some theoretical or experimental analysis of the methodological choices. Some space is devoted to relatively trivial results (such as Proposition 1), which could have been better used to explain why the proposed approach works. Since can be any invertible matrix, a deeper characterization of the optimal R transformations would have been scientifically interesting, particularly to understand why SOA initialization improves GRA compared with random initialization, as observed in Appendix E.
- Despite the overall positive results, the improvements remain marginal for some fusion methods, such as DARE-TIES. This suggests that sufficiently robust fusion methods may already mitigate part of the problem. However, given the low computational cost of the proposed approach, even marginal improvements are still worthwhile.
Questions for authors
I wonder how the proposed method would interact with ensembling techniques. For instance, if we independently trained two LoRA adapters on the same task, aligned them, and then fused them, would this improve performance otf the task? Ensembles often benefit from diversity, but I wonder whether parameter-space misalignment could be detrimental in this setting. I am not asking the authors to conduct additional experiments, but I would be interested in their intuition based on their experience with this problem.
Have the authors considered other ways to mitigate misalignment beyond what they propose (i.e. orthogonal alignment followed by optimization over general invertible transformations), or imposing some constraints/regularization on the GRA optimization? How did the authors arrive at this particular approach?
4 comments
W2.
Weakness 2: Despite the overall positive results, the improvements remain marginal for some fusion methods, such as DARE-TIES. This suggests that sufficiently robust fusion methods may already mitigate part of the problem. However, given the low computational cost of the proposed approach, even marginal improvements are still worthwhile.
Response:
We thank the reviewer and agree that the gains for DARE-TIES are modest. A plausible explanation is that its sparsification and sign-consistent merging already mitigate some destructive interference, partially overlapping with the benefits of alignment. These operations nevertheless act within the existing factor coordinates and do not explicitly align the latent bases across adapters.
Q1.
Question 1: I wonder how the proposed method would interact with ensembling techniques. For instance, if we independently trained two LoRA adapters on the same task, aligned them, and then fused them, would this improve performance otf the task? Ensembles often benefit from diversity, but I wonder whether parameter-space misalignment could be detrimental in this setting. I am not asking the authors to conduct additional experiments, but I would be interested in their intuition based on their experience with this problem.
Response: We thank the reviewer for this question. Alignment preserves each adapter's function, so it leaves an ensemble that averages their predictions unchanged and preserves their functional diversity. Parameter-space fusion remains sensitive to factor coordinates, even when the adapters are trained on the same task. Motivated by Model Soups [1] and Git Re-Basin [2], we expect alignment to help same-task fusion when coordinate mismatch impedes composition.
Q2.
Question 2: Have the authors considered other ways to mitigate misalignment beyond what they propose (i.e. orthogonal alignment followed by optimization over general invertible transformations), or imposing some constraints/regularization on the GRA optimization? How did the authors arrive at this particular approach?
Response:
We thank the reviewer for this question. Our approach emerged from exploring progressively more structured alignment strategies. We first considered directly matching the LoRA factors without constraining scale changes, leading to least-squares-based alignment formulations. However, their downstream performance was unsatisfactory, motivating us to control the scale distortion introduced by the alignment transformations. We therefore adopted iterative optimization with the orthogonality regularizer , which discourages excessive scaling while permitting non-orthogonal adjustments.
Even with this regularization, we observed substantial sensitivity to initialization: different starting points could yield markedly different downstream performance. This is consistent with the nonconvexity discussed in our response to Weakness 1, where different initializations may lead to different optimization outcomes. We then considered restricting the transformation to an orthogonal matrix, under which the alignment problem admits a closed-form optimal solution. This leads to SOA: a norm-preserving alignment that provides a principled starting point for subsequent optimization. These considerations led to our SOA–GRA procedure. The corresponding ablation studies and analyses are provided in Appendix E.
References
[1] Wortsman, Mitchell, et al. "Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time." ICML 2022.
[2] Ainsworth, Samuel K., et al. "Git Re-Basin: Merging Models modulo Permutation Symmetries." ICLR 2023.
reply to rebuttal
Thank you for the insightful answers. I am satisfied with the authors' response and will maintain my score.
Thank you for your thoughtful follow-up. We are glad that our clarifications have addressed your concerns, and we appreciate your constructive feedback.
We thank the reviewer for the thoughtful comments and constructive feedback.
W1.
Response:
We thank the reviewer for this question. SOA uses factor-matching information to initialize GRA. We compared it with random orthogonal initialization under identical optimization settings; both initializations are orthogonal, with condition number 1.
SOA starts with a lower alignment loss and retains this advantage on both benchmarks within the tested optimization budget. The GRA optimization objective is highly nonconvex. We examined one source of this nonconvex geometry by analyzing the loss along a straight-line path between the two optimized solutions. For the final transformations RSOA and Rrandom obtained under the two initializations, we evaluated R(α)=(1−α)RSOA+αRrandom at 201 equally spaced values of α∈[0,1].
Both benchmarks show a pronounced loss peak near α=0.5, far above either endpoint loss, while the minimum singular value is near zero. Many corresponding endpoint pairs have opposite determinant signs, so any continuous path between them must cross a singular matrix. Near this crossing, amplification of the inverse-factor term R−1A can produce the observed loss peaks. We also provide the proof of this singular-crossing property in the revised version (Appendix H.3). In this nonconvex landscape, SOA supplies a better initial factor match whose loss advantage persists through local refinement.