acceptodds
Desk rejected at ICLR 2027

Exploiting Symmetry in Low-Rank Decomposition for LoRA Fusion

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Low-Rank Adaptation (LoRA) is widely used to fine-tune Large Language Models across diverse downstream tasks, motivating model fusion as a way to integrate multiple task-specific LoRA adapters into a single model without access to the original training data. Existing LoRA fusion methods, ranging from naive averaging to interference-resolution approaches, overlook a key geometric property of the LoRA parameter space: LoRA adapters admit multiple reparameterizations that are functionally equivalent, a property we refer to as low-rank decomposition (LRD) symmetry. Ignoring this symmetry, however, causes direct fusion of independently trained adapters to suffer from parameter-space misalignment, leading to degraded representations and performance. To address this issue, we propose a symmetry-aware parameter alignment framework that maps LoRA adapters into a shared latent basis prior to fusion, first through a closed-form rotational alignment and then through a gradient-based refinement that relaxes orthogonality to the wider space of invertible reparameterizations. Our method is a plug-and-play enhancement compatible with existing LoRA fusion techniques. Extensive experiments on real-world benchmarks show broad improvements across diverse fusion methods. Our implementation is available on https://anonymous.4open.science/r/GRA.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
model merging, LoRA, low-rank adaptation, parameter space symmetry

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ceLYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose a preprocessing step prior to any LoRA weight fusion. They "align" these LoRA adapters in order to reduce the variability due to potential RR−1RR^{-1} transformations in the latent space, which do not change the individual adapters' outputs but may affect the results when they are fused.

Critical strengths
  • The paper is well written and easy to follow. The problem is clearly stated, and the proposed solution is simple and fast to run, making it practical to use in real-world applications.
  • The experimental evaluation is extensive and addresses the most important questions raised by the proposed approach, including alternative alignment strategies (e.g., permutation-based alignment), the benefit of SOA initialization before GRA (appendix), the reduction of factor-space misalignment, computational overhead, and out-of-distribution performance. In addition, many fusion methods are explored, along with several variants of the proposed alignment method (A-only, B-only, and Joint). Despite considerable variation in performance depending on the task, with no consistent winner among the alignment variants, the proposed solution generally brings improvements across the different fusion methods.
Critical weaknesses
  • Despite the main experiments supporting the paper's central claim and the many promising results, I think the paper lacks some theoretical or experimental analysis of the methodological choices. Some space is devoted to relatively trivial results (such as Proposition 1), which could have been better used to explain why the proposed approach works. Since RR can be any invertible matrix, a deeper characterization of the optimal R transformations would have been scientifically interesting, particularly to understand why SOA initialization improves GRA compared with random initialization, as observed in Appendix E.
  • Despite the overall positive results, the improvements remain marginal for some fusion methods, such as DARE-TIES. This suggests that sufficiently robust fusion methods may already mitigate part of the problem. However, given the low computational cost of the proposed approach, even marginal improvements are still worthwhile.
Questions for authors

I wonder how the proposed method would interact with ensembling techniques. For instance, if we independently trained two LoRA adapters on the same task, aligned them, and then fused them, would this improve performance otf the task? Ensembles often benefit from diversity, but I wonder whether parameter-space misalignment could be detrimental in this setting. I am not asking the authors to conduct additional experiments, but I would be interested in their intuition based on their experience with this problem.

Have the authors considered other ways to mitigate misalignment beyond what they propose (i.e. orthogonal alignment followed by optimization over general invertible transformations), or imposing some constraints/regularization on the GRA optimization? How did the authors arrive at this particular approach?

4 comments

We thank the reviewer for the thoughtful comments and constructive feedback.

W1.

Weakness 1: Despite the main experiments supporting the paper's central claim and the many promising results, I think the paper lacks some theoretical or experimental analysis of the methodological choices. Some space is devoted to relatively trivial results (such as Proposition 1), which could have been better used to explain why the proposed approach works. Since (R) can be any invertible matrix, a deeper characterization of the optimal R transformations would have been scientifically interesting, particularly to understand why SOA initialization improves GRA compared with random initialization, as observed in Appendix E.

Response:

We thank the reviewer for this question. SOA uses factor-matching information to initialize GRA. We compared it with random orthogonal initialization under identical optimization settings; both initializations are orthogonal, with condition number 1.

Benchmark / initializationInitial loss → Final loss
NLI / SOA0.02326 → 0.02319
NLI / Random orthogonal0.02468 → 0.02463
GLUE / SOA0.01495 → 0.01484
GLUE / Random orthogonal0.01578 → 0.01575

SOA starts with a lower alignment loss and retains this advantage on both benchmarks within the tested optimization budget. The GRA optimization objective is highly nonconvex. We examined one source of this nonconvex geometry by analyzing the loss along a straight-line path between the two optimized solutions. For the final transformations RSOAR_{\mathrm{SOA}} and RrandomR_{\mathrm{random}} obtained under the two initializations, we evaluated R(α)=(1−α)RSOA+αRrandomR(\alpha)=(1-\alpha)R_{\mathrm{SOA}}+\alpha R_{\mathrm{random}} at 201 equally spaced values of α∈[0,1]\alpha\in[0,1].

Both benchmarks show a pronounced loss peak near α=0.5\alpha=0.5, far above either endpoint loss, while the minimum singular value is near zero. Many corresponding endpoint pairs have opposite determinant signs, so any continuous path between them must cross a singular matrix. Near this crossing, amplification of the inverse-factor term R−1AR^{-1}A can produce the observed loss peaks. We also provide the proof of this singular-crossing property in the revised version (Appendix H.3). In this nonconvex landscape, SOA supplies a better initial factor match whose loss advantage persists through local refinement.

W2.

Weakness 2: Despite the overall positive results, the improvements remain marginal for some fusion methods, such as DARE-TIES. This suggests that sufficiently robust fusion methods may already mitigate part of the problem. However, given the low computational cost of the proposed approach, even marginal improvements are still worthwhile.

Response:

We thank the reviewer and agree that the gains for DARE-TIES are modest. A plausible explanation is that its sparsification and sign-consistent merging already mitigate some destructive interference, partially overlapping with the benefits of alignment. These operations nevertheless act within the existing factor coordinates and do not explicitly align the latent bases across adapters.

Q1.

Question 1: I wonder how the proposed method would interact with ensembling techniques. For instance, if we independently trained two LoRA adapters on the same task, aligned them, and then fused them, would this improve performance otf the task? Ensembles often benefit from diversity, but I wonder whether parameter-space misalignment could be detrimental in this setting. I am not asking the authors to conduct additional experiments, but I would be interested in their intuition based on their experience with this problem.

Response: We thank the reviewer for this question. Alignment preserves each adapter's function, so it leaves an ensemble that averages their predictions unchanged and preserves their functional diversity. Parameter-space fusion remains sensitive to factor coordinates, even when the adapters are trained on the same task. Motivated by Model Soups [1] and Git Re-Basin [2], we expect alignment to help same-task fusion when coordinate mismatch impedes composition.

Q2.

Question 2: Have the authors considered other ways to mitigate misalignment beyond what they propose (i.e. orthogonal alignment followed by optimization over general invertible transformations), or imposing some constraints/regularization on the GRA optimization? How did the authors arrive at this particular approach?

Response:

We thank the reviewer for this question. Our approach emerged from exploring progressively more structured alignment strategies. We first considered directly matching the LoRA factors without constraining scale changes, leading to least-squares-based alignment formulations. However, their downstream performance was unsatisfactory, motivating us to control the scale distortion introduced by the alignment transformations. We therefore adopted iterative optimization with the orthogonality regularizer λ∥I−R⊤R∥F2\lambda\|I-R^\top R\|_F^2, which discourages excessive scaling while permitting non-orthogonal adjustments.

Even with this regularization, we observed substantial sensitivity to initialization: different starting points could yield markedly different downstream performance. This is consistent with the nonconvexity discussed in our response to Weakness 1, where different initializations may lead to different optimization outcomes. We then considered restricting the transformation to an orthogonal matrix, under which the alignment problem admits a closed-form optimal solution. This leads to SOA: a norm-preserving alignment that provides a principled starting point for subsequent optimization. These considerations led to our SOA–GRA procedure. The corresponding ablation studies and analyses are provided in Appendix E.

References

[1] Wortsman, Mitchell, et al. "Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time." ICML 2022.

[2] Ainsworth, Samuel K., et al. "Git Re-Basin: Merging Models modulo Permutation Symmetries." ICLR 2023.

Reviewer_ceLYOpenReview ↗

reply to rebuttal

Thank you for the insightful answers. I am satisfied with the authors' response and will maintain my score.

Thank you for your thoughtful follow-up. We are glad that our clarifications have addressed your concerns, and we appreciate your constructive feedback.

Reviewer_MprCOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies factor-space fusion of independently trained LoRA adapters. Because the update ΔW = BA is invariant under any invertible rank-space transformation (A, B) → (R⁻¹A, BR), equivalent adapters can have different factor coordinates, making coordinate-sensitive fusion vulnerable to parameter-space mismatch. The authors propose a pre-fusion alignment module with two stages: SVD-based Orthogonal Alignment (SOA), which solves an Orthogonal Procrustes problem in closed form, and Gradient-based Refined Alignment (GRA), which starts from the SOA solution and optimizes a more general transformation with a soft near-orthogonality regularizer. A-only, B-only, and Joint variants control which factor(s) are directly matched to an anchor while reparameterizing both factors to preserve each adapter’s update when the transformation remains invertible. Experiments cover eight fusion baselines plus Git-Rebasin on Llama-3.1-8B GLUE/NLI adapter pools, Flan-T5-Large on cross-task BBH, and supplementary Qwen3-0.6B, anchor-sensitivity, layer-selection, ablation, and runtime analyses. The paper reports average improvements of 2.29 and 2.49 percentage points across alignment variants on GLUE and NLI, respectively; GRA(J) gives the strongest average gains among the variants on those two benchmarks. BBH gains are smaller and variant-dependent. The method is specifically scoped to factor-space fusion with compatible factor shapes.

Critical strengths
  1. The paper identifies an exact reparameterization symmetry of LoRA and uses it to motivate a function-preserving alignment procedure. SOA has a closed-form Procrustes solution, while GRA adds a lightweight refinement stage. This is a clear, technically grounded design, and the preprocessing step does not require retraining the source adapters or add inference-time latency after the aligned factors are cached.
  2. The paper provides extensive evaluations across multiple LoRA fusion baselines, showing average improvements of 2.29 and 2.49 percentage points on GLUE and NLI, respectively. Additional experiments on Qwen3-0.6B and BBH, along with ablation and robustness analyses, further support the effectiveness of the proposed method.
Critical weaknesses
  1. The evidence for the proposed mechanism is suggestive but not conclusive. Since the alignment objective directly minimizes factor-space distance, the reductions in Figure 3 do not independently establish that symmetry-induced misalignment is the main cause of fusion failures. A more controlled experiment would apply known invertible reparameterizations to adapters with identical weight updates and test whether alignment restores fusion performance.
  2. The out-of-domain generalization results are limited. On BBH, the average improvements are modest (+0.61, +0.40, and +0.17 for GRA(A/B/J), respectively), with occasional performance degradation. Broader evaluations are needed to establish robust generalization.
Questions for authors
  1. The anchor selection strategy may itself be sensitive to the reparameterization symmetry that the paper aims to address. The medoid is selected using Frobenius distances between LoRA factors, which can change under equivalent reparameterizations that preserve the same weight updates. It would be useful to evaluate whether anchor selection and fusion performance remain stable under such transformations.
  2. Additional evidence relevant to the central claim: Could you evaluate controlled equivalent reparameterizations by applying known random invertible transforms to the factors of the same adapters while keeping ΔW unchanged, then measuring whether SOA/GRA recovers consistent factor-space fusion behavior?
4 comments

We thank the reviewer for the thoughtful comments and constructive feedback.

W1, Q1 & Q2.

Weakness 1: The evidence for the proposed mechanism is suggestive but not conclusive. Since the alignment objective directly minimizes factor-space distance, the reductions in Figure 3 do not independently establish that symmetry-induced misalignment is the main cause of fusion failures. A more controlled experiment would apply known invertible reparameterizations to adapters with identical weight updates and test whether alignment restores fusion performance.

Question 1: The anchor selection strategy may itself be sensitive to the reparameterization symmetry that the paper aims to address. The medoid is selected using Frobenius distances between LoRA factors, which can change under equivalent reparameterizations that preserve the same weight updates. It would be useful to evaluate whether anchor selection and fusion performance remain stable under such transformations.

Question 2: Additional evidence relevant to the central claim: Could you evaluate controlled equivalent reparameterizations by applying known random invertible transforms to the factors of the same adapters while keeping ΔW unchanged, then measuring whether SOA/GRA recovers consistent factor-space fusion behavior?

Response:

We thank the reviewer for these questions. Model Soups [1] and Git Re-Basin [2] suggest that bringing models into a shared low-error basin helps weight averaging. This motivates our intuition that bringing equivalent LoRA factors closer makes them easier to fuse.

To test coordinate sensitivity in real adapter pools, we independently changed the factor coordinates of every adapter and layer in the original six-adapter NLI and GLUE pools. We use random (seed=101) orthogonal transformations and general invertible transformations with condition number 2 and numerically verified that each source update remained unchanged. The selected medoid remained unchanged in all four tested reparameterization settings. The table reports six-task mean accuracy and update deviation dupd(m)d_{\mathrm{upd}}^{(m)}, which is defined for each fusion method mm as

dupd(m)=max⁡ℓ∥ΔWm,ℓ′−ΔWm,ℓ∥F∥ΔWm,ℓ∥Fd_{\mathrm{upd}}^{(m)}=\max_{\ell}\frac{\Vert\Delta W_{m,\ell}^{\prime}-\Delta W_{m,\ell}\Vert_F}{\Vert\Delta W_{m,\ell}\Vert_F}

where ΔWm,ℓ\Delta W_{m,\ell} and ΔWm,ℓ′\Delta W'_{m,\ell} are the fused updates at layer ℓ\ell before and after reparameterization, respectively, using the same fusion method mm.

PoolTransformSimple accuracy (%)SOA(J) accuracy (%)GRA(J) accuracy (%)Simple update deviationSOA(J) update deviationGRA(J) update deviation
NLIOrthogonal47.22350.27250.4671.313.18e-060.000394
NLIGeneral, condition 247.15750.42950.3491.390.5400.540
GLUEOrthogonal52.76352.93253.1581.214.97e-060.000736
GLUEGeneral, condition 253.17253.46853.3461.300.4790.479

In all four settings, SOA(J) and GRA(J) have lower fused-update deviation than Simple, showing reduced sensitivity of the fused update to equivalent coordinates. Both methods also achieve higher accuracy than Simple on the reparameterized pools.

We also applied independent orthogonal transformations to five of six copies of the same RTE adapter, keeping one copy as a fixed reference:

SettingRTE accuracy (%)
Original adapter (reference)85.56
Reparameterized copies + Simple51.99
Reparameterized copies + GRA(J)84.84

Changing only the coordinates lowers Simple accuracy from 85.56% to 51.99%, while GRA(J) restores it to 84.84%. This shows how alignment helps recover fusion performance when the source updates are identical. We have incorporated these analyses in our revised version (Appendix H.2).

Complementing the coordinate-perturbation test, Appendix F.1 evaluates fusion accuracy under all six choices of reference adapter. Table 7 reports 48.91±0.12% for NLI/Simple with GRA(J) across these choices, showing similar accuracy across anchors. Figure 8 extends this check to additional fusion settings.

W2.

Weakness 2: The out-of-domain generalization results are limited. On BBH, the average improvements are modest (+0.61, +0.40, and +0.17 for GRA(A/B/J), respectively), with occasional performance degradation. Broader evaluations are needed to establish robust generalization.

We thank the reviewer for this comment. To examine how the modest BBH gains depend on source-adapter selection, we sampled 30 existing Flan-T5-Large adapters and evaluated all 27 BBH tasks. The table reports mean accuracy (%) for baseline fusion and fusion with GRA.

FusionBaseline+ GRA(A)+ GRA(J)
Simple19.58219.60719.607
LoRAHub27.34827.41527.538

Both methods show small positive average gains with this additional source selection. Broader out-of-domain evaluation remains future work.

References

[1] Wortsman, Mitchell, et al. "Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time." ICML 2022.

[2] Ainsworth, Samuel K., et al. "Git Re-Basin: Merging Models modulo Permutation Symmetries." ICLR 2023.

Thank you for the detailed rebuttal. I appreciate the additional evidence and will maintain my original rating of 3.

Thank you for your thoughtful follow-up. We are glad that our clarifications have addressed your concerns, and we appreciate your constructive feedback.

Reviewer_B8a1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies coordinate sensitivity in fusion methods that combine LoRA factors directly. Since an adapter update BA is unchanged by replacing its factors with BR and R⁻¹A, individually equivalent adapters can produce different results under factor-wise fusion. The proposed preprocessing chooses a medoid adapter as an anchor, solves an orthogonal Procrustes problem for A-only, B-only, or joint alignment, and optionally refines the transformation with a regularized, non-orthogonal optimization. Both factors are transformed so that each source update is preserved before fusion. Experiments cover multiple fusion rules on GLUE and NLI, out-of-domain BBH composition, an additional Qwen backbone, anchor sensitivity, and leave-target-out LoRAHub/LoraRetriever protocols. The reported gains are broadest in the in-family settings, particularly for magnitude-sensitive fusion rules, and smaller and less uniform on BBH.

Critical strengths
  1. The proposed intervention targets a precisely defined source of fusion sensitivity. Proposition 1 and Theorems 1–2 establish function preservation for each source adapter and the optimum of the orthogonal alignment objective. Sections 3.2–3.3 then separate the alignment objective from the downstream fusion rule. This is a clean, low-dimensional intervention whose effect cannot be attributed to modifying individual source models before merging.

  2. The evaluation goes beyond one fusion rule or a single reference choice. Tables 2–3 show improvements across several factor-space fusion methods; Appendix E evaluates the regularizer, least-squares alternatives, and initialization. Appendix F.1 evaluates all anchor choices, F.2 adds Qwen3-0.6B, and Table 8 tests leave-target-out composition. This breadth makes the empirical observation useful even though gains are heterogeneous and not universal.

Critical weaknesses
  1. The practical value relative to an invariant, equal-capacity fusion route remains unresolved. The method explicitly excludes pipelines operating only on ΔW (Appendix A), which is a valid scope restriction. However, all principal comparisons remain within coordinate-sensitive factor fusion. A natural alternative is to merge the source updates BᵢAᵢ and compress the result to the same deployment rank. No such matched-rank comparison is reported in the main tables or extended analyses. This does not invalidate the measured improvements, but it leaves unclear whether correcting factor coordinates is preferable to avoiding that coordinate dependence, and what accuracy/memory/preprocessing tradeoff justifies the proposed route.

  2. Evidence for small improvements and the proposed mechanism is mostly descriptive. Appendix F.1 explicitly measures anchor sensitivity rather than uncertainty from independently trained adapter pools. Some improvements in Tables 2–4 are small, and BBH's average gains are only 0.61/0.40/0.17 points for the three variants, with several regressions. Moreover, reducing the Frobenius distance optimized by the method (Section 4.3) is not by itself evidence that this proxy explains downstream improvements. The large, heterogeneous Magnitude-Prune gains in Figure 11 make this distinction consequential: a subset of tasks improves sharply while others deteriorate. The existing results support a useful heuristic more strongly than a broadly reliable geometric explanation.

Questions for authors
  1. [Evidence request] How does aligned factor fusion compare with merging ΔW updates followed by compression to the same deployment rank, using the same adapter pool and merge weights? Please include preprocessing memory/cost if that is the reason to prefer factor-space fusion. This would determine whether the improvement establishes a compelling practical alternative.

  2. [Evidence request] Can you quantify the reliability of representative gains across independently trained adapter pools, separating this from the existing anchor sweep? This is most important for the small Simple/TIES/BBH differences, rather than a request to repeat the entire grid.

  3. [Clarification] If the input adapters are first replaced by randomly reparameterized but functionally identical factors, how stable are medoid selection, aligned fusion, and downstream accuracy? Such a controlled check would help distinguish mitigation of arbitrary coordinates from favorable behavior under the original training parameterization; complete invariance is not assumed to be a claimed guarantee.

5 comments

We thank the reviewer for the thoughtful comments and constructive feedback.

W1 & Q1

Weakness 1: The practical value relative to an invariant, equal-capacity fusion route remains unresolved. The method explicitly excludes pipelines operating only on ΔW (Appendix A), which is a valid scope restriction. However, all principal comparisons remain within coordinate-sensitive factor fusion. A natural alternative is to merge the source updates BᵢAᵢ and compress the result to the same deployment rank. No such matched-rank comparison is reported in the main tables or extended analyses. This does not invalidate the measured improvements, but it leaves unclear whether correcting factor coordinates is preferable to avoiding that coordinate dependence, and what accuracy/memory/preprocessing tradeoff justifies the proposed route.

Question 1: [Evidence request] How does aligned factor fusion compare with merging ΔW updates followed by compression to the same deployment rank, using the same adapter pool and merge weights? Please include preprocessing memory/cost if that is the reason to prefer factor-space fusion. This would determine whether the improvement establishes a compelling practical alternative.

Response:

We thank the reviewer for suggesting this comparison. Related work has examined interference in direct LoRA factor merging and SVD-based alternatives [1], [2]. Following Tang et al. [1], we merge the source adapters in update space and apply truncated SVD to obtain a rank-16 adapter (ΔW + SVD16). We compare this baseline with direct and aligned factor fusion on the original six-adapter NLI and GLUE pools.

Matched-rank accuracy comparison. Scores are six-task mean accuracy in percent. All variants have a deployment rank of 16.

NLI, ΔW + SVD16: 55.27%

MethodDirect factor fusion+ SOA(J)+ GRA(J)
Simple47.0948.6149.01
Fisher50.7654.7556.82
Magnitude-Prune51.0255.5056.06

GLUE, ΔW + SVD16: 62.07%

MethodDirect factor fusion+ SOA(J)+ GRA(J)
Simple53.5753.9954.41
Fisher58.9859.3763.67
Magnitude-Prune57.9564.2165.86

ΔW + SVD16 outperforms the Simple factor-fusion variants at rank 16, while GRA(J) with Fisher or Magnitude-Prune achieves higher accuracy than this baseline on both benchmarks.

BenchmarkPreprocessing routeTime (s)Peak GPU memory (MiB)
NLISOA(J) (ours)0.358384.9
NLIGRA(J) (ours)6.364385.2
NLICompact ΔW + SVD166.721391.3
NLIFull dense ΔW + SVD16136.115887.9
GLUESOA(J) (ours)0.297201.0
GLUEGRA(J) (ours)3.143201.7
GLUECompact ΔW + SVD164.314209.3
GLUEFull dense ΔW + SVD16133.126705.9

In these measurements, SOA/GRA factor fusion uses similar peak GPU memory to Compact ΔW + SVD16 and takes less preprocessing time. We have added these experiments to Appendix H.1 of the revised manuscript. We also corrected the GRA timing reported in the original manuscript: the earlier measurement inadvertently omitted the gradient-descent stage. The corrected timings continue to support our conclusion that alignment adds only a small, one-time preprocessing cost relative to downstream evaluation.

W2 & Q2.

Weakness 2 (reliability): Evidence for small improvements and the proposed mechanism is mostly descriptive. Appendix F.1 explicitly measures anchor sensitivity rather than uncertainty from independently trained adapter pools. Some improvements in Tables 2–4 are small, and BBH's average gains are only 0.61/0.40/0.17 points for the three variants, with several regressions.

Question 2: [Evidence request] Can you quantify the reliability of representative gains across independently trained adapter pools, separating this from the existing anchor sweep? This is most important for the small Simple/TIES/BBH differences, rather than a request to repeat the entire grid.

Response:

We thank the reviewer for requesting this check. To assess training variability separately from anchor sensitivity, we independently retrained source adapters with two additional training seeds (42 and 43) for subsets of the original task pools: RTE, SICK, and SciTail for NLI, and RTE and MRPC for GLUE. The table reports mean accuracy (%) over these subsets for baseline fusion and fusion with GRA(J). We will update the table with full-pool results once the remaining source-adapter training and evaluations are complete.

SettingFusion methodTraining seed 42: baselineTraining seed 42: +GRA(J)Training seed 43: baselineTraining seed 43: +GRA(J)
NLISimple53.16953.35150.60050.207
NLIFisher56.75563.38951.65454.888
NLIMagnitude-Prune55.98058.71651.50454.322
NLITIES52.93953.34449.97750.614
GLUESimple53.05853.78756.72658.145
GLUEFisher54.09054.33557.17258.442
GLUEMagnitude-Prune53.66453.96757.82258.803
GLUETIES52.62653.83857.70058.952

NLI/TIES improves under both training seeds, while NLI/Simple improves under seed 42 and decreases under seed 43. All reported GLUE methods improve under both seeds.

Separately, we checked sensitivity to source-adapter selection on BBH. We sampled 30 existing Flan-T5-Large adapters and evaluated all 27 tasks. The table reports mean accuracy (%) for baseline fusion and fusion with GRA.

FusionBaseline+ GRA(A)+ GRA(J)
Simple19.58219.60719.607
LoRAHub27.34827.41527.538

Both methods show small positive average gains with this additional source selection.

W2 & Q3.

Weakness 2 (mechanism): Moreover, reducing the Frobenius distance optimized by the method (Section 4.3) is not by itself evidence that this proxy explains downstream improvements. The large, heterogeneous Magnitude-Prune gains in Figure 11 make this distinction consequential: a subset of tasks improves sharply while others deteriorate. The existing results support a useful heuristic more strongly than a broadly reliable geometric explanation.

Question 3: [Clarification] If the input adapters are first replaced by randomly reparameterized but functionally identical factors, how stable are medoid selection, aligned fusion, and downstream accuracy? Such a controlled check would help distinguish mitigation of arbitrary coordinates from favorable behavior under the original training parameterization; complete invariance is not assumed to be a claimed guarantee.

Response:

We thank the reviewer for suggesting this check. Model Soups [3] and Git Re-Basin [4] suggest that bringing models into a shared low-error basin helps weight averaging. Our intuition is that bringing equivalent LoRA factors closer makes them easier to fuse. To isolate the effect of factor coordinates, we changed their coordinates while keeping each adapter's learned update fixed.

We reparameterized the factors as Bi′=BiQi,Ai′=Qi−1AiB_i'=B_iQ_i,A_i'=Q_i^{-1}A_i and numerically checked that Bi′Ai′=BiAiB_i'A_i'=B_iA_i. First, we formed six copies of the same RTE adapter, kept one copy as a fixed reference, and independently applied orthogonal transformations to the other five copies:

SettingRTE accuracy (%)
Original adapter (reference)85.56
Reparameterized copies + Simple51.99
Reparameterized copies + GRA(J)84.84

Changing only the coordinates lowers Simple accuracy from 85.56% to 51.99%, while GRA(J) restores it to 84.84%. Since all six copies represent the same source update, the drop and recovery show how factor coordinates affect fusion and how alignment helps.

We then repeated the experiments on the original six-adapter NLI and GLUE pools. We independently transformed every adapter and layer using random (seed=101) orthogonal matrices or general invertible matrices with condition number 2. The selected medoid remained unchanged in all four tested reparameterization settings. The table reports six-task mean accuracy and update deviation dupd(m)d_{\mathrm{upd}}^{(m)}, which is defined for each fusion method mm as

dupd(m)=max⁡ℓ∥ΔWm,ℓ′−ΔWm,ℓ∥F∥ΔWm,ℓ∥Fd_{\mathrm{upd}}^{(m)}=\max_{\ell}\frac{\Vert\Delta W_{m,\ell}^{\prime}-\Delta W_{m,\ell}\Vert_F}{\Vert\Delta W_{m,\ell}\Vert_F}

where ΔWm,ℓ\Delta W_{m,\ell} and ΔWm,ℓ′\Delta W'_{m,\ell} are the fused updates at layer ℓ\ell before and after reparameterization, respectively, using the same fusion method mm.

PoolTransformSimple accuracy (%)SOA(J) accuracy (%)GRA(J) accuracy (%)Simple update deviationSOA(J) update deviationGRA(J) update deviation
NLIOrthogonal47.22350.27250.4671.313.18e-060.000394
NLIGeneral, condition 247.15750.42950.3491.390.5400.540
GLUEOrthogonal52.76352.93253.1581.214.97e-060.000736
GLUEGeneral, condition 253.17253.46853.3461.300.4790.479

In all four settings, SOA(J) and GRA(J) have lower fused-update deviation than Simple, showing reduced sensitivity of the fused update to equivalent coordinates. The deviations are smaller under orthogonal than general invertible transformations. On the reparameterized pools, both methods also achieve higher accuracy than Simple averaging. We have incorporated these analyses in our revised version (Appendix H.2).

For Magnitude-Prune, Section F.6 and Figure 11 show GRA(J) gains on four of six tasks in each benchmark, averaging 7.91 percentage points on GLUE and 5.04 on NLI, alongside a SciTail regression of 8.91 points. Factor-wise magnitude pruning depends on the chosen coordinates: GRA preserves every source update BiAiB_iA_i but can change which factor entries survive at the same sparsity. Thus, alignment can affect the pruning decisions as well as subsequent factor composition. Its shared factor-matching objective does not directly optimize the pruned model's accuracy on each task. These task-level results characterize the resulting tradeoffs.

References

[1] Tang, Dennis, et al. "LoRA Merging with SVD: Understanding Interference and Preserving Performance." ICML Workshop on Reliable and Responsible Foundation Models 2025.

[2] Stoica, George, et al. "Model merging with SVD to tie the KnOTS." ICLR 2025.

[3] Wortsman, Mitchell, et al. "Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time." ICML 2022.

[4] Ainsworth, Samuel K., et al. "Git Re-Basin: Merging Models modulo Permutation Symmetries." ICLR 2023.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.