acceptodds
Under review as a conference paper at ICLR 2027

Orthogonal Yet Coupled: Decoupling Geometric Components for Model Merging

30 views

Abstract

Merging pretrained models has emerged as an effective approach for consolidating diverse capabilities into a single unified model. However, prevailing merging methods typically treat each task vector as an indivisible merging unit, overlooking the heterogeneous geometric changes encoded within it. This treatment can induce cross-component coupling: when merging decisions are derived from statistics of the complete task vector, the geometric characteristics of one component may influence how another is selected, weighted, or combined, potentially degrading the quality of the merged model. To address this issue, we propose DiGA, a Disentangled Geometry-Aware model merging framework. Using the pretrained weights as a shared geometric reference, DiGA orthogonally decomposes each task vector into components corresponding to distinct geometric attributes. Rather than merging the task vectors as a whole, DiGA aggregates corresponding components independently within their respective subspaces and subsequently recombines them into a unified update. This component-wise formulation preserves the geometric identity of each component and prevents the characteristics of one component from interfering with the aggregation of another. Furthermore, DiGA can be incorporated into a broad range of existing model merging methods. Extensive experiments across diverse models, tasks, and merging methods demonstrate that DiGA improves merged-model performance and reduces capability degradation. Our repository is on https://anonymous.4open.science/r/DiGA-A878/ .

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Model Merging, Multi-task Learning, Knowledge Transfer

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5taQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DiGA decomposes task updates into two orthogonal subspaces defined by a shared QR frame of the pretrained weights. It aggregates the orientation and structural components separately, then recombines them. The motivation is that nonlinear statistics used in merging can couple these components even when the original components are orthogonal.

Critical strengths

The paper provides evidence aimed at the proposed mechanism. Figure 1 varies one component while holding the other fixed to examine how merging decisions change. Table 4 compares whole-update and separate-component processing with the same aggregation rule, improving task average from 58.24 to 58.60. Table 5 shows smaller but consistent benefits when the separation is applied to TIES, DARE, and TSVM. The appendix also evaluates all 120 expert orderings; 118 outperform OrthoMerge. These controls make the modest gains more persuasive.

Critical weaknesses

The novelty should be positioned more carefully relative to OrthoMerge, which already separates rotational and residual components for merging. DiGA offers a different shared-frame decomposition and an analysis of coupling through nonlinear aggregation statistics; this is a more specific contribution than introducing geometric separation itself. The full OFT improvement over the strongest baseline should also not be attributed entirely to separation, whose controlled gain in Table 4 is 0.36 points. The scope of the geometric guarantee needs clarification for the generic merging operators in Table 5. The proposed aggregation preserves the component subspaces through linear combinations within each branch. Entrywise masking or other nonlinear operations in existing mergers do not generally preserve those subspaces. Independent branch decisions and preservation of component identity after aggregation are distinct claims.

Questions for authors

When applying TIES, DARE, and TSVM separately to the components in Table 5, are their outputs projected back into the corresponding subspaces? If not, please specify which geometric guarantees remain valid and limit the component-preservation claim accordingly.

4 comments

Response to W1: Novelty Relative to OrthoMerge

We thank the reviewer for this suggestion. We agree that OrthoMerge has already introduced orthogonal-residual decoupling for model merging, and we do not claim geometric separation itself as a novel contribution.

Our contribution lies in identifying and addressing cross-component coupling in adaptive merging decisions. Unlike OrthoMerge, which employs orthogonal transformations and Procrustes-based decomposition, DiGA constructs a shared orthonormal reference frame from the pretrained weights, enabling a unified Frobenius-orthogonal decomposition of task updates across different fine-tuning methods.

Importantly, geometric orthogonality does not guarantee independence during adaptive aggregation. Statistics computed from complete task vectors can cause structural components to influence the merging decisions applied to orientation components, despite their orthogonality. Figure 1 explicitly demonstrates this interference mechanism. DiGA addresses it by independently aggregating the two component families, preventing cross-component interference in aggregation decisions.

We will revise our contribution statements to more clearly distinguish DiGA from OrthoMerge, emphasizing the identification and mitigation of cross-component coupling, with the shared-frame decomposition providing its geometric foundation.

Response to W2: Attribution of Performance Gains

We agree with the reviewer that the full OFT improvement over OrthoMerge should not be attributed solely to component separation.

As shown in Table 4, under an identical aggregation strategy, component-wise processing improves Task Avg. from 58.24 to 58.60 (+0.36 points), isolating its contribution. The overall improvement over OrthoMerge (57.32 →\rightarrow 58.60, +1.28 points) reflects the combined effects of component-wise processing and our aggregation design. The ablations further show that removing redundancy-aware orthogonalization, consensus calibration, or source-magnitude weighting reduces Task Avg. by 0.39, 0.33, and 3.42 points, respectively.

Importantly, Table 5 provides additional evidence for component-wise processing beyond our aggregation strategy. Applying it to TIES, DARE, and TSVM improves Task Avg. by 0.26, 0.56, and 0.37 points, respectively. Moreover, for TSVM, Component-Separate outperforms Component-Joint by 2.62 points (56.55 vs. 53.93), highlighting the importance of maintaining separate aggregation decisions.

We will clarify this distinction in the revised manuscript to avoid attributing the full performance gain to component separation alone.

Response to W3 and Question: Scope of Geometric Guarantees

We thank the reviewer for highlighting this important distinction. We agree that the strict subspace-preservation guarantee applies to DiGA's proposed aggregation operator, whereas the generic extensions in Table 5 aim to evaluate the benefits of component-wise processing rather than enforce strict geometric preservation.

For the generic extensions in Table 5, no additional projection onto the corresponding subspaces is performed after aggregation. Although the initial decomposition is strictly orthogonal, existing merging operators involving elementwise selection or spectral transformations do not generally preserve subspace membership. Consequently, their aggregated outputs are not guaranteed to remain within their respective subspaces or retain mutual orthogonality.

Importantly, strict subspace preservation is not required to mitigate cross-component coupling in aggregation decisions. By processing the two component families independently, aggregation decisions within each branch are determined solely by the corresponding components, preventing the cross-component influence arising from whole-vector aggregation statistics. Thus, Table 5 provides empirical evidence for the benefits of independent component-wise processing, rather than extending DiGA's strict geometric guarantees to arbitrary merging operators. We will clarify this distinction and explicitly specify the scope of our geometric guarantees in the revised manuscript.

Response for Question

No additional projection onto the corresponding subspaces is performed after applying TIES, DARE, or TSVM in Table 5. The exact Frobenius-orthogonal decomposition of the input task updates remains valid, and the aggregation decisions are computed independently within each component family. However, unlike DiGA's proposed aggregation operator, these generic merging operators do not necessarily preserve subspace membership or mutual orthogonality of the aggregated outputs. Therefore, the geometric guarantee of subspace preservation applies specifically to DiGA's proposed aggregation operator, while Table 5 demonstrates the empirical benefits of independent component-wise processing. We will explicitly clarify this distinction and limit the component-preservation claims accordingly.

Thank you very much for your constructive comments. We hope that our responses have addressed your concerns. Please feel free to let us know if any further clarification is needed.

Thank you for the detailed response. The distinction from OrthoMerge, the attribution of the performance gains, and the scope of the geometric guarantees are now clear. These clarifications address my earlier concerns and should be reflected in the revised manuscript. My overall assessment remains unchanged.

We sincerely thank the reviewer for the positive assessment and constructive feedback. We will incorporate the additional experimental results into the revised manuscript.

Reviewer_un1hOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DiGA to decouple the structural components in the task vectors. DiGA uses the pretrained weights to generate an initial basis of the shared structural components. Then, it rotates the basis slightly with the downstream task vectors. Several experiments across Llama and Qwen2.5-VL show the effectiveness of the proposed method.

Critical strengths
  • This paper is well-motivated. The idea of decoupling is new and interesting.
  • The decomposition is mathematically sound.
  • The writing is easy to follow.
Critical weaknesses
  • The major weakness of this paper is the limited performance. The average improvements are limited on Llama with 8B and 3B, and are also limited in the experiments with Qwen-VL. The results make the proposed complicated design seem meaningless.
  • Also, the ablation study shows that the separation machinery contributes little to the performance improvement.
  • The paper positions itself against structured merging methods (TSVM, Iso-CTS) whose standard evaluation is the CLIP 8/14/20-task suites. It is recommended to provide results on such commonly used settings.
  • What is the computational complexity of the proposed method?
Questions for authors
  • Why does the proposed method select QR rather than the commonly used SVD? Does QR have some unique advantages?
12 comments

Response to W1: Limited Performance Improvements

Table 1: Model merging performance on ViT-B-32 across 8, 12, and 16 tasks.

Method8 tasks12 tasks16 tasks
Acc.Norm.Acc.Norm.Acc.Norm.
DC-Merge64.1573.8868.3877.2066.2775.79
+DiGA65.4674.1568.8577.3267.3476.86
TSVM56.8566.0464.1472.6262.3971.56
+DiGA58.8268.1565.0073.5863.2472.42
ISO-CTS63.0172.7066.2775.0064.6474.04
+DiGA63.5773.2467.3576.0964.8774.57

Table 2: Model merging performance on ViT-B-16 across 8, 12, and 16 tasks.

Method8 tasks12 tasks16 tasks
Acc.Norm.Acc.Norm.Acc.Norm.
DC-Merge70.5578.8873.1580.5970.5578.88
+DiGA70.8979.9373.5381.5771.6478.97
TSVM61.5069.2967.9575.0366.2474.28
+DiGA63.0871.0069.0576.2167.2375.35
ISO-CTS69.0477.3671.5278.8769.8878.22
+DiGA69.5277.7571.9079.7469.6577.99

Table 3: Model merging performance on ViT-L-14 across 8, 12, and 16 tasks.

Method8 tasks12 tasks16 tasks
Acc.Norm.Acc.Norm.Acc.Norm.
DC-Merge82.6489.4483.6389.3279.5285.71
+DiGA82.6989.4883.5989.2679.5585.74
TSVM73.9680.3177.7083.0973.5479.41
+DiGA76.3182.7879.0484.4674.1180.03
ISO-CTS81.6388.3181.3486.8677.5183.67
+DiGA81.9888.6782.6787.3979.2484.35

Table 4: Model merging performance on Gemma-2-2B across five tasks: instruction following, coding, mathematics, safety, and multilingual understanding. Bold indicates the best performance in each column.

MethodInstructionCodingMathSafetyMultilingualAvg.
Gemma-2-2b27.9524.5728.2842.9930.4130.84
TA29.6930.4738.3642.2232.2834.60
TIES22.7126.8933.1343.6931.4631.58
DC-Merge29.4834.2246.4741.8232.6236.92
TSVM39.5235.2443.6745.4332.3839.25
DiGA43.2335.1545.2046.5333.8640.79

We thank the reviewer for this concern. We acknowledge that the improvements on some Llama and Qwen-VL settings are relatively modest. However, we believe that the effectiveness of DiGA should be assessed across different model families and merging scenarios, rather than solely by the magnitude of improvements in individual settings.

To further evaluate its effectiveness, we conducted additional experiments on Gemma-2-2B, merging five experts specialized in instruction following, coding, mathematics, safety, and multilingual understanding. As shown in the Table 4, DiGA achieves an average score of 40.79, outperforming the strongest baseline, TSVM (39.25), by 1.54 percentage points, and surpassing TA, TIES, and DC-Merge by 6.19, 9.21, and 3.87 points, respectively. DiGA also outperforms TSVM on four of the five tasks, with a particularly notable improvement in instruction following (+3.71 points). These results demonstrate that DiGA can achieve more substantial gains under different model families and capability combinations.

Furthermore, our additional CLIP experiments across three vision backbones (ViT-B-32, ViT-B-16, and ViT-L-14) and different numbers of tasks (8, 12, and 16) demonstrate that incorporating DiGA into existing merging methods, including DC-Merge, TSVM, and ISO-CTS, generally improves their performance. These results further support the applicability of DiGA across different architectures and merging strategies. Detailed results are provided in our response to Weakness3.

Importantly, DiGA is training-free and task-data-free, and introduces no additional parameters or computational overhead during inference. Although it incurs additional offline merging computations, these costs do not affect model deployment or inference efficiency. Therefore, even when the performance improvements are modest, DiGA offers a practical way to improve model merging without additional training or deployment costs.

Overall, the additional Gemma and CLIP experiments provide broader evidence of DiGA's effectiveness and applicability, while we acknowledge that the magnitude of performance gains varies across experimental settings.

Response to W2: the ablation study shows that the separation machinery contributes little to the performance improvement

Table A: Ablation study of DiGA on five fully fine-tuned Gemma-2-2B experts across instruction following, coding, mathematics, safety, and multilingual tasks.

MethodInstructionCodingMathSafetyMultilingualAvg.
DiGA43.2335.1545.2046.5333.8640.79
T only21.4026.4333.5146.1131.9631.88
R only38.2133.8343.9040.6832.8137.89
Whole-vector aggregation42.5835.0343.9046.4932.3240.06
w/o redundancy removal43.1934.3044.4345.1632.3539.89
w/o consensus calibration37.7718.5332.1575.2128.5338.44
w/o source magnitude42.3636.4141.5548.2032.3040.16
Symmetric orthogonalization41.7035.4144.0544.5332.3239.60

We thank the reviewer for this comment. To further assess the contribution of component separation, we conducted additional ablation experiments on five fully fine-tuned Gemma-2-2B experts covering instruction following, coding, mathematics, safety, and multilingual capabilities.

Under the same aggregation strategy, component-wise processing improves the average performance from 40.06 (whole-vector aggregation) to 40.79 (+0.73 points), compared with the +0.36-point improvement observed in the Llama-3.1-8B OFT setting (Table 4). This provides further evidence that the contribution of geometric separation extends beyond the original experimental setting.

Moreover, retaining only the orientation or structural component reduces the average performance to 31.88 and 37.89, respectively, compared with 40.79 for the full method, highlighting the complementary contributions of both components.

Together, these results provide additional empirical support for the effectiveness of component-wise processing. We will incorporate the new ablation results into the revised manuscript and clarify the contribution of geometric separation relative to the other aggregation components.

Response to W3: Results on CLIP task suites

We thank the reviewer for this valuable suggestion. To address this concern, we conduct additional experiments on three CLIP vision backbones (ViT-B-32, ViT-B-16, and ViT-L-14) across different task scales (8, 12, and 16 tasks). We evaluated the integration of DiGA with three representative merging methods, including DC-Merge, TSVM, and ISO-CTS, to further assess its effectiveness in structured model merging settings. The results are presented in Tables 1–3, where Acc. and Norm. denote average accuracy and normalized accuracy, respectively. Bold values indicate improvements over the corresponding original merging method.

Across the three backbones and task scales, integrating DiGA improves 50 out of 54 accuracy and normalized-accuracy comparisons. In particular, DiGA consistently improves TSVM across all evaluated configurations, with gains of up to 2.35 percentage points in average accuracy and 2.47 points in normalized accuracy. Improvements are also observed for DC-Merge and ISO-CTS in most settings, although a few configurations exhibit slight performance decreases.

These additional experiments provide broader evidence that DiGA is applicable to existing structured merging methods and can improve their performance across different CLIP architectures and task scales. We will incorporate these results into the revised manuscript.

Response to W4: Computational Complexity and Efficiency

We thank the reviewer for raising this question. DiGA introduces additional computational costs during offline model merging, primarily from three operations: QR decomposition, geometric component projection, and Gram–Schmidt orthogonalization.

For a weight matrix W0∈Rm×nW_0\in\mathbb{R}^{m\times n} with m≥nm\geq n and KK expert updates, the computational complexity of each operation is as follows:

  1. QR decomposition: Computing the reduced QR decomposition of the pretrained weight matrix requires O(mn2)O(mn^2) operations. This decomposition is performed only once per weight matrix and shared across all experts.
  2. Geometric decomposition: Projecting each expert update onto the orientation and structural subspaces requires O(mn2)O(mn^2) operations, resulting in O(Kmn2)O(Kmn^2) complexity across all experts.
  3. Component-wise aggregation: Gram–Schmidt orthogonalization is performed independently on the two components, requiring O(K2mn)O(K^2mn) operations in total.

Therefore, the overall computational complexity per weight matrix is O(Kmn2+K2mn).O\left(Kmn^2+K^2mn\right).

We additionally measured end-to-end merging time and sampled peak GPU memory for five LoRA experts on an NVIDIA RTX A6000. Both methods used identical pre-materialized expert weights, recomputed their method-specific decompositions without reusing pre-existing decomposition caches, and included model saving. DiGA completed in 477 s with 19.91 GiB peak sampled GPU memory, compared with 71,148 s and 28.72 GiB for OrthoMerge-C-TA[1], respectively. As an offline, training-free merging method, DiGA introduces no additional inference overhead.

[1] Orthogonal Model Merging. 2026ICML

Response to Q1: Why QR Rather Than SVD?

We thank the reviewer for this question. The choice of QR decomposition is motivated by the geometric formulation of DiGA. Unlike spectral merging methods that exploit singular values or dominant singular directions, DiGA aims to construct an orthonormal reference frame from the pretrained weights to disentangle task updates into orientation and structural components.

Both QR and SVD can provide orthonormal bases. However, DiGA requires only an orthonormal reference frame, rather than the additional spectral information provided by SVD. QR directly satisfies this requirement without computing singular values or performing spectral ordering, while generally being computationally less expensive.

Therefore, we adopt QR as a simple and efficient way to establish the geometric reference frame. QR is a natural choice aligned with the geometric requirements of our method.

Reviewer_Hpk7OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the interference among different geometric components within task vectors during model merging. The authors argue that existing methods typically treat each task vector as an indivisible unit, which induces "cross-component coupling": the aggregation statistics of one geometric component may inappropriately influence the selection or weighting of another. To address this, the paper proposes Disentangled Geometry-Aware Merging (DiGA). Utilizing the QR decomposition of pretrained weights as a shared geometric reference, DiGA orthogonally decomposes each task vector into orientation and structural components. These components are aggregated independently within their respective subspaces using Gram-Schmidt orthogonalization and magnitude calibration, and subsequently recombined. Extensive experiments on LoRA, OFT, fully fine-tuned language models, and vision-language models demonstrate that DiGA outperforms several existing merging frameworks like TIES, DARE, and TSVM.

Critical strengths

Strengths

S1. A novel and valuable perspective on model merging. Instead of merely focusing on the interference among different expert task vectors, the authors shift the attention to the intra-vector geometric coupling. This perspective is refreshing, and Figure 1 provides intuitive empirical evidence showing how the structural component can alter the merging decisions of a fixed orientation component.

S2. Clear mathematical structure and plug-and-play compatibility. The paper provides a rigorous orthogonal decomposition formulation and proves its existence, uniqueness, and cross-expert orthogonality in the appendix. Moreover, the component-wise merging paradigm is data-free and can be seamlessly integrated into existing operators like TIES, DARE, and TSVM, demonstrating good generality.

S3. Extensive experiments and thorough analysis. The evaluation covers parameter-efficient fine-tuning, full fine-tuning, and vision-language models. The inclusion of component ablations, merging update analysis, and especially the sensitivity analysis over all 120 permutations of five experts, adds significant credibility to the stability of the proposed method.

Critical weaknesses

Weaknesses

W1. The geometric definition of the orientation component lacks sufficient justification and may be inconsistent with the claimed structure-preserving property. This is my primary theoretical concern. In Section 3.1, the paper uses the QR decomposition W0=QHW_0 = QH and defines the orientation subspace using skew-symmetric matrices {QA:A⊤=−A}\{QA : A^\top = -A\}. However, this construction describes a class of tangential variations of the orthogonal reference frame QQ, which does not necessarily correspond to structure-preserving changes of the original weights W0W_0. For instance, an infinitesimal rotation Q′=Q+ϵQAQ' = Q + \epsilon QA that keeps HH invariant induces a weight change of ϵQAH\epsilon QAH, not ϵQA\epsilon QA. When H≠IH \neq I, the two are generally inconsistent. Therefore, the proposed projection does not fully capture the true structure-preserving variations. The algebraic correctness of the orthogonal decomposition does not automatically validate the semantic assignment of "orientation" and "structure" to these specific subspaces.

W2. The Gram-Schmidt aggregation strategy lacks solid theoretical motivation and may produce unstable merging directions for highly similar expert updates. Section 3.3 uses Gram-Schmidt (GS) orthogonalization to remove directional redundancy while retaining the original norms as aggregation weights. However, this design may not truly preserve the geometric information of the original updates. Consider two nearly parallel unit directions: u1=e1u_1 = e_1 and u2=cos⁡(θ)e1+sin⁡(θ)e2u_2 = \cos(\theta)e_1 + \sin(\theta)e_2. As long as the residual exceeds the numerical threshold, GS will transform the second direction into e2e_2, even if θ\theta is extremely close to zero. Subsequently, the method assigns e2e_2 the same norm weight as the original expert update. Consequently, two nearly identical expert updates could produce a merged result that significantly deviates from the original direction. The aggregation rule may lack continuity when directions become linearly dependent, potentially amplifying small residual noises.

W3. The experiments do not sufficiently prove that the performance gains primarily stem from the proposed cross-component decoupling. The core contribution of the paper is avoiding cross-component coupling, but the ablation study shows that the performance gain from introducing component separation alone is quite marginal. For example, in Table 4, the complete DiGA achieves a Task Avg. of 58.60, while whole-vector aggregation using the exact same aggregation operator achieves 58.24 (a mere 0.36% drop). In contrast, removing source-magnitude weighting causes a 3.42% drop. Similarly, in Table 5, applying component separation to TIES, DARE, and TSVM only yields marginal improvements (0.26%, 0.56%, and 0.37%). These results suggest that the new aggregation design and magnitude calibration play a dominant role, leaving it unclear to what extent the geometric decoupling itself constitutes the main contribution.

W4. Lack of comparison with stronger recent baselines and insufficient evaluation of generalization across varying numbers of experts. The paper primarily compares DiGA with earlier merging methods (e.g., TA, TIES, DARE, TSVM, OrthoMerge). However, it lacks comparison with stronger, more recent baselines that also address geometric or directional consistency in parameter-efficient fine-tuning model merging, such as WUDI-merge[1] and DC-Merge[2]. Without this comparison, the relative advantage of DiGA is hard to accurately assess. Furthermore, the generalization capability of DiGA regarding the number of experts is not fully explored, as current experiments are mostly limited to merging 3 or 5 experts. To convincingly demonstrate the scalability and robustness of the proposed geometric decoupling, it would be highly helpful to evaluate DiGA in scenarios with more experts. For instance, conducting experiments on Vision Transformers (ViT) by merging a larger number of classification tasks would provide strong evidence of its generalization ability in more complex, multi-expert settings.

Reference

[1] Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectors

[2] DC-Merge: Improving Model Merging with Directional Consistency. CVPR2026

Questions for authors

Questions for Authors

Q1. Could you strictly define the "orientation-preserving update" and explain why the skew-symmetric projection based on QQ correctly characterizes the structure-preserving changes of W0W_0? Specifically, does this definition still hold when W0=QHW_0 = QH and H≠IH \neq I? Should the inverse of HH be introduced to correct the projection?

Q2. What is the theoretical justification for retaining the original norms as weights after Gram-Schmidt orthogonalization? For highly similar or nearly linearly dependent expert updates, could this approach amplify small residual directions? Have you considered projection aggregation methods that retain residual magnitudes as an alternative?

Q3. Given that the ablation study shows marginal gains from component separation, could you design more rigorous controlled experiments to strictly disentangle the effects of geometric decomposition, component separation, and the aggregation operator? Can you provide empirical or theoretical evidence from the task loss Hessian perspective to show how geometric decoupling reduces functional interference?

Q4. Why was DiGA not compared with stronger recent baselines like DC-Merge? Additionally, could you provide experiments on ViT models merging more classification tasks (e.g., >5 experts) to demonstrate the generalization and scalability of your method regarding the number of experts?

10 comments

Response to W1 & Q1: Geometric Interpretation of the Orientation Component

We thank the reviewer for this insightful question. We would like to clarify that DiGA defines the orientation component relative to the base-induced orthonormal reference frame QQ, rather than the original weight matrix W0W_0. We acknowledge that some of our descriptions, particularly the term structure-preserving, may have suggested the latter interpretation and thus caused confusion. We will revise these statements to clarify the intended geometric meaning. As stated in Section 3.1, DiGA explicitly takes Q, rather than W0W_0, as the geometric anchor for decomposing task updates. Our intention is to characterize geometric changes relative to this orthogonal reference frame, rather than to fully characterize the structure-preserving variations of the original weights.

Specifically, the subspace TQ={QA:A⊤=−A}\mathcal{T}_Q=\{QA:A^\top=-A\} characterizes skew-symmetric tangent directions associated with in-frame reorientation of QQ. We agree that, when H≠IH\neq I, these directions do not generally correspond to structure-preserving changes of the original weights W0W_0. In particular, rotating QQ while keeping HH fixed induces a weight variation of the form QAHQAH, rather than QAQA. However, DiGA defines the orientation component relative to the reference frame QQ, through the Frobenius-orthogonal projection onto TQ\mathcal{T}_Q, rather than through the structure-preserving variations of W0W_0.

Consequently, the proposed decomposition remains exact and unique for any full-column-rank W0=QHW_0=QH, without requiring H=IH=I or introducing H−1H^{-1}. Incorporating HH would correspond to a different geometric construction, rather than a correction to our reference-frame-based decomposition.

We will revise Section 3.1 to consistently describe the orientation component as capturing in-frame reorientation relative to the orthogonal reference frame QQ. These clarifications do not affect the mathematical formulation or the proposed merging procedure.

Response to W2 & Q2: Motivation and Stability of Gram–Schmidt Aggregation

We thank the reviewer for raising this concern. The motivation for Gram–Schmidt (GS) aggregation is to reduce directional redundancy among expert updates while preserving their original contribution magnitudes. Specifically, GS constructs mutually orthogonal directions, preventing overlapping directions from being repeatedly accumulated. By weighting these orthonormal directions using the original update norms, DiGA retains the magnitude information of individual experts while avoiding cross-terms in the squared norm of the aggregated update.

Regarding numerical stability, we acknowledge that near-linearly dependent expert updates could produce small GS residuals, potentially amplifying numerical errors during normalization. To investigate this concern, we conducted additional experiments using three pairs of independently trained same-task LoRA experts, as shown in Table A, differing only in their random seeds (43 vs. 44). For each pair, we evaluated 224 weight matrices in both the orientation (T) and structural (R) branches. The input components were normalized to unit Frobenius norm before GS, and we measured the pre-normalization GS residual norm of the second expert, ρ2=∣∣r2∣∣F\rho_2=||r_2||_F.

Since the input components have unit Frobenius norm, the residual satisfies ρ2=1−cos⁡2θ\rho_2=\sqrt{1-\cos^2\theta}, where θ\theta is the angle between the two expert components. Across all 1,344 measurements, the minimum residual norm is 0.7968, with 97.32% at or above 0.90 and 93.15% at or above 0.95. These results indicate that no near-linearly dependent directions were encountered, even among independently trained experts specializing in the same task.

Furthermore, DiGA employs a numerical safeguard with a residual threshold to handle degenerate cases. In addition, evaluating all 120 permutations of five experts yields a Task Avg. of 58.03 ± 0.31, providing complementary evidence that the method is not excessively sensitive to expert ordering in the evaluated setting.

These observations provide empirical evidence that GS aggregation remains numerically well-behaved under the tested merging conditions. We will clarify the theoretical motivation and numerical stability considerations in the revised manuscript.

Table A: Distribution of pre-normalization GS residual norms across same-task expert pairs (seed=43/44).

Expert pairBranchObserved min-max[0.75, 0.80)[0.80, 0.90)[0.90, 0.95)[0.95, 0.99)[0.99, 1]
InstructionFollowingT0.8772-0.9999051245162
InstructionFollowingR0.8787-0.999806938171
MagicoderT0.7972-1.00001101251150
MagicoderR0.7968-1.00001111151150
MetaMathQAT0.8712-0.999601664153
MetaMathQAR0.8824-0.999701656161

Q: Projection Aggregation with Residual Magnitude Preservation

We thank the reviewer for this valuable suggestion. To investigate this alternative, we conducted additional experiments using a projection-based aggregation strategy that retains the residual magnitudes, under the same experimental setting as DiGA. The results are summarized below:

Table: Comparison between projection-based aggregation with residual magnitude preservation and DiGA.

MATH500HUMANEVAL+ScienceQACommonsenseQASocial-IQATasK Avg.AGIEvalARC-AvgTransfer Avg.
projection-aggregation28.20%38.25%88.17%78.02%54.32%57.39%37.74%41.52%39.63%
DiGA29.40%40.12%88.26%79.93%55.27%58.60%38.97%42.86%40.92%

DiGA outperforms the projection-based alternative across all five evaluated benchmarks, achieving improvements of 1.21 points in Task Avg. and 1.29 points in Transfer Avg. These results suggest that preserving residual magnitudes does not necessarily lead to better merging performance. In the evaluated setting, DiGA's aggregation strategy provides a more favorable balance between retaining individual expert contributions and reducing directional redundancy.

We will include this additional comparison in the revised manuscript.

Response to W3 & Q3: Contribution of Cross-Component Decoupling

Table A: Ablation study of DiGA on five fully fine-tuned Gemma-2-2B experts across instruction following, coding, mathematics, safety, and multilingual tasks

MethodInstructionCodingMathSafetyMultilingualAvg.
DiGA43.2335.1545.2046.5333.8640.79
T only21.4026.4333.5146.1131.9631.88
R only38.2133.8343.9040.6832.8137.89
Whole-vector aggregation42.5835.0343.9046.4932.3240.06
w/o redundancy removal43.1934.3044.4345.1632.3539.89
w/o consensus calibration37.7718.5332.1575.2128.5338.44
w/o source magnitude42.3636.4141.5548.2032.3040.16
Symmetric orthogonalization41.7035.4144.0544.5332.3239.60

We thank the reviewer for this concern. The overall performance gains of DiGA should not be attributed solely to cross-component decoupling, as the proposed aggregation strategy also contributes to the final performance. To further isolate the contribution of component separation, we conducted additional ablation experiments on five fully fine-tuned Gemma-2-2B experts covering instruction following, coding, mathematics, safety, and multilingual capabilities.

Under the same aggregation operator, component-wise processing improves the average performance from 40.06 to 40.79 (+0.73 points), compared with the +0.36-point improvement observed in the Llama-3.1-8B OFT setting in Table 4. These results provide additional controlled evidence that geometric decoupling contributes independently to performance improvements across different models and fine-tuning settings.

Furthermore, retaining only the orientation or structural component reduces the average performance to 31.88 and 37.89, respectively, highlighting the complementary contributions of the two geometric components.

Importantly, our central contribution is not the claim that component separation accounts for the majority of the overall performance improvement, but the identification of cross-component coupling as a distinct source of interference and the development of a component-wise framework to address it. The controlled ablations, together with the coupling analysis in Figure 1 and the improvements across existing merging operators in Table 5, provide complementary evidence for this contribution.

We will incorporate the additional ablation results and clarify the respective roles of geometric decoupling and aggregation design in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.