- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
DiGA decomposes task updates into two orthogonal subspaces defined by a shared QR frame of the pretrained weights. It aggregates the orientation and structural components separately, then recombines them. The motivation is that nonlinear statistics used in merging can couple these components even when the original components are orthogonal.
Critical strengths
The paper provides evidence aimed at the proposed mechanism. Figure 1 varies one component while holding the other fixed to examine how merging decisions change. Table 4 compares whole-update and separate-component processing with the same aggregation rule, improving task average from 58.24 to 58.60. Table 5 shows smaller but consistent benefits when the separation is applied to TIES, DARE, and TSVM. The appendix also evaluates all 120 expert orderings; 118 outperform OrthoMerge. These controls make the modest gains more persuasive.
Critical weaknesses
The novelty should be positioned more carefully relative to OrthoMerge, which already separates rotational and residual components for merging. DiGA offers a different shared-frame decomposition and an analysis of coupling through nonlinear aggregation statistics; this is a more specific contribution than introducing geometric separation itself. The full OFT improvement over the strongest baseline should also not be attributed entirely to separation, whose controlled gain in Table 4 is 0.36 points. The scope of the geometric guarantee needs clarification for the generic merging operators in Table 5. The proposed aggregation preserves the component subspaces through linear combinations within each branch. Entrywise masking or other nonlinear operations in existing mergers do not generally preserve those subspaces. Independent branch decisions and preservation of component identity after aggregation are distinct claims.
Questions for authors
When applying TIES, DARE, and TSVM separately to the components in Table 5, are their outputs projected back into the corresponding subspaces? If not, please specify which geometric guarantees remain valid and limit the component-preservation claim accordingly.
4 comments
Thank you very much for your constructive comments. We hope that our responses have addressed your concerns. Please feel free to let us know if any further clarification is needed.
Thank you for the detailed response. The distinction from OrthoMerge, the attribution of the performance gains, and the scope of the geometric guarantees are now clear. These clarifications address my earlier concerns and should be reflected in the revised manuscript. My overall assessment remains unchanged.
We sincerely thank the reviewer for the positive assessment and constructive feedback. We will incorporate the additional experimental results into the revised manuscript.
Response to W1: Novelty Relative to OrthoMerge
We thank the reviewer for this suggestion. We agree that OrthoMerge has already introduced orthogonal-residual decoupling for model merging, and we do not claim geometric separation itself as a novel contribution.
Our contribution lies in identifying and addressing cross-component coupling in adaptive merging decisions. Unlike OrthoMerge, which employs orthogonal transformations and Procrustes-based decomposition, DiGA constructs a shared orthonormal reference frame from the pretrained weights, enabling a unified Frobenius-orthogonal decomposition of task updates across different fine-tuning methods.
Importantly, geometric orthogonality does not guarantee independence during adaptive aggregation. Statistics computed from complete task vectors can cause structural components to influence the merging decisions applied to orientation components, despite their orthogonality. Figure 1 explicitly demonstrates this interference mechanism. DiGA addresses it by independently aggregating the two component families, preventing cross-component interference in aggregation decisions.
We will revise our contribution statements to more clearly distinguish DiGA from OrthoMerge, emphasizing the identification and mitigation of cross-component coupling, with the shared-frame decomposition providing its geometric foundation.
Response to W2: Attribution of Performance Gains
We agree with the reviewer that the full OFT improvement over OrthoMerge should not be attributed solely to component separation.
As shown in Table 4, under an identical aggregation strategy, component-wise processing improves Task Avg. from 58.24 to 58.60 (+0.36 points), isolating its contribution. The overall improvement over OrthoMerge (57.32 → 58.60, +1.28 points) reflects the combined effects of component-wise processing and our aggregation design. The ablations further show that removing redundancy-aware orthogonalization, consensus calibration, or source-magnitude weighting reduces Task Avg. by 0.39, 0.33, and 3.42 points, respectively.
Importantly, Table 5 provides additional evidence for component-wise processing beyond our aggregation strategy. Applying it to TIES, DARE, and TSVM improves Task Avg. by 0.26, 0.56, and 0.37 points, respectively. Moreover, for TSVM, Component-Separate outperforms Component-Joint by 2.62 points (56.55 vs. 53.93), highlighting the importance of maintaining separate aggregation decisions.
We will clarify this distinction in the revised manuscript to avoid attributing the full performance gain to component separation alone.
Response to W3 and Question: Scope of Geometric Guarantees
We thank the reviewer for highlighting this important distinction. We agree that the strict subspace-preservation guarantee applies to DiGA's proposed aggregation operator, whereas the generic extensions in Table 5 aim to evaluate the benefits of component-wise processing rather than enforce strict geometric preservation.
For the generic extensions in Table 5, no additional projection onto the corresponding subspaces is performed after aggregation. Although the initial decomposition is strictly orthogonal, existing merging operators involving elementwise selection or spectral transformations do not generally preserve subspace membership. Consequently, their aggregated outputs are not guaranteed to remain within their respective subspaces or retain mutual orthogonality.
Importantly, strict subspace preservation is not required to mitigate cross-component coupling in aggregation decisions. By processing the two component families independently, aggregation decisions within each branch are determined solely by the corresponding components, preventing the cross-component influence arising from whole-vector aggregation statistics. Thus, Table 5 provides empirical evidence for the benefits of independent component-wise processing, rather than extending DiGA's strict geometric guarantees to arbitrary merging operators. We will clarify this distinction and explicitly specify the scope of our geometric guarantees in the revised manuscript.
Response for Question
No additional projection onto the corresponding subspaces is performed after applying TIES, DARE, or TSVM in Table 5. The exact Frobenius-orthogonal decomposition of the input task updates remains valid, and the aggregation decisions are computed independently within each component family. However, unlike DiGA's proposed aggregation operator, these generic merging operators do not necessarily preserve subspace membership or mutual orthogonality of the aggregated outputs. Therefore, the geometric guarantee of subspace preservation applies specifically to DiGA's proposed aggregation operator, while Table 5 demonstrates the empirical benefits of independent component-wise processing. We will explicitly clarify this distinction and limit the component-preservation claims accordingly.