- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the respective roles of singular vectors and singular values of task update matrices in data-free model merging. Under a unified covariance-weighted merging framework, the authors decompose the task operator into three components: the support defined by the right-singular subspace, the anisotropic weights within that support, and the aggregate spectral scale of the operator. The core experiments show that, while retaining the same right-singular subspace and matching the operator trace, replacing direction-specific inverse spectral weights with a uniform scalar improves merging performance across all tested settings. Based on this finding, the authors propose SOS-Merge, which constructs the task operator using the right-singular projection and a fractional inverse spectral moment, and calibrates the exponent from checkpoint spectral information. The method is validated on three CLIP ViT backbones under 8/14/20-task settings.
Critical strengths
-
The research question is clearly formulated, and the core controlled experiments are convincing.
-
The performance gains are fairly consistent, and become more pronounced when more tasks are merged.
-
The ablations are relatively thorough, and the paper is transparent about weaker results.
-
The method is easy to understand and requires neither task samples nor additional training.
Critical weaknesses
-
Insufficient justification for exponent calibration. Equation (15) selects by matching to , but the rationale for this target is unclear. The analysis uses , whereas the final method uses . Global spectral-mass matching is not equivalent to task-wise normalization, since changing also redistributes weights across tasks and layers.
-
Spectral scale is confounded with relative regularization. The Shape–Scale comparison changes task-operator magnitudes while fixing and , thereby changing the balance between expert matching and regularization. Its performance gap therefore cannot distinguish useful spectral-scale information from a better regularization balance.
-
The contribution and performance gains need clearer attribution. ISO [1] methods already exploit isotropic spectral constructions, while ACE-Merging [2] provides covariance-based closed-form merging. Moreover, applying the same depth multiplier raises ACE-Merging to 86.38%, reducing SOS-Merge’s advantage to 0.5%. Table 16 further shows only an approximately 0.06% benefit from normalized spectral shape, without consistent gains across random subsets. The evidence supports appropriate scalar scaling more strongly than the necessity of the full spectral statistic.
-
Baseline comparisons and experimental scope are limited. The paper does compare against several recent merging methods, but it still omits some important recent baselines, particularly ESM [3] and RegMean++ [4], which also evaluate model merging on CLIP ViT benchmarks. Nor does it discuss why such comparisons are omitted or infeasible. In addition, the paper lacks experiments in NLP scenarios and does not discuss whether the method is limited to vision scenarios.
-
Limited support for classification-aware refinement. Theorem 1 gives a standard sub-Gaussian union bound with dependence, but does not justify multiplying the spectral exponent by this factor. The improvement is only 0.17%, and its robustness across independent checkpoints or task subsets is unclear. Additional evidence would help justify this refinement. The required access to task class counts should also be stated explicitly as a metadata assumption.
[1] No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces. ICML, 2025.
[2] ACE-Merging: Data-Free Model Merging with Adaptive Covariance Estimation. CVPR, 2026.
[3] Model Merging in the Essential Subspace. CVPR, 2026.
[4] RegMean++: Enhancing Effectiveness and Generalization of Regression Mean for Model Merging. TMLR, 2026.
Questions for authors
- What motivates the calibration target , and how does it compare with explicit task weighting?
- Does the advantage of Scale over Shape persist after controlling for their relative regularization strength?
- How much improvement remains when strong baselines use the same depth multiplier?
- Could the authors provide comparisons with ESM and RegMean++ under clearly stated data-access assumptions, or explain their omission? Is the method applicable beyond vision?
- Is the classification-aware refinement consistently beneficial across different checkpoints or task subsets?
9 comments
Response to DjDn-W2/Q2
W2/Q2: Spectral scale is confounded with relative regularization. The Shape–Scale comparison changes the relative strength of expert matching and regularization. Does the advantage of Scale over Shape persist after controlling for their relative regularization strength?
To examine whether the Shape–Scale difference is caused by the relative regularization strength, we conducte an additional experiment that allows each operator to independently adjust its complete regularizer. Specifically, we multiply the combined pretrained covariance and ridge regularizer by , where . This directly changes the balance between expert matching and regularization while keeping the task operators unchanged. We select for each operator using held-out validation accuracy, with the base ridge coefficient fixed at .
On ViT-B/32 with 8, 14, and 20 tasks, Shape selects for both exponents, while Full and Scale select . The results are reported below and in Tab. 22 of Appendix D.1.
| Tasks | Shape () | Shape () | Full () | Scale () | |
|---|---|---|---|---|---|
| -1.0 | 8 | 70.47 | 84.69 | 86.05 | 86.16 |
| -1.0 | 14 | 67.69 | 78.61 | 81.07 | 82.35 |
| -1.0 | 20 | 63.53 | 74.48 | 76.63 | 79.31 |
| -0.5 | 8 | 74.04 | 84.03 | 84.21 | 84.91 |
| -0.5 | 14 | 69.42 | 78.15 | 78.31 | 79.97 |
| -0.5 | 20 | 64.98 | 73.86 | 73.17 | 75.73 |
| Average | 68.36 | 78.97 | 79.91 | 81.41 |
Adjusting the complete regularizer substantially improves Shape, raising its average accuracy from 68.36% to 78.97%. This shows that the original Shape–Scale gap was partly affected by regularization strength. However, Scale remains the best-performing operator in all six settings, outperforming the adjusted Shape by 2.44 percentage points on average.
More importantly, the Full–Scale comparison directly examines the effect of spectral anisotropy. Full and Scale have identical support and trace for every task and layer, and both select . Therefore, they share the same total spectral strength and regularization setting, with direction-specific weighting being the only difference in their task operators. Under these matched conditions, Scale outperforms Full by 1.50 on average.
These results show that the advantage of Scale persists after independently adjusting regularization strength. In particular, the Full–Scale comparison supports our central finding that removing direction-specific inverse weighting improves merging performance under matched support and spectral mass.
Response to DjDn-W3/Q3
W3/Q3: The contribution and performance gains need clearer attribution. The depth multiplier contributes to SOS-Merge's performance gains, while the normalized spectral-shape component provides only a small improvement. How much improvement remains when strong baselines use the same depth multiplier?
To examine how much of the improvement comes from the depth multiplier, we applied the same depth scaling to SOS-Merge and the baselines. Across all nine settings, SOS achieves 86.92%, compared with 86.38% for ACE with the same multiplier. Without depth scaling, SOS also outperforms ACE by 0.30, reaching 86.04% versus 85.74% (Tab. 8). Thus, SOS maintains its advantage under both conditions.
We further extended the comparison to SWUDI-A and ISO-CTS + SVC on ViT-B/32. In addition, we examined whether the performance differences remain when ACE's auxiliary components are incorporated into SOS. For this comparison, we considered four cumulative configurations, denoted E1 to E4. E1 represents each method's basic merging formulation, with SOS-E1 corresponding to the original SOS-Merge. E2 adds adaptive covariance normalization, E3 introduces a collective structural prior, and E4 further adds spectral refinement. Full ACE-Merging corresponds to ACE-E4. The complete configurations and results are provided in Tab. 26 of Appendix D.4.
| Configuration | No depth scaling, mean | Matched scaling, 8 tasks | 14 tasks | 20 tasks | Mean |
|---|---|---|---|---|---|
| ACE-E3 | 81.56 | 86.38 | 82.43 | 79.98 | 82.93 |
| ACE-E4 (full ACE) | 81.56 | 86.39 | 82.04 | 79.77 | 82.73 |
| SWUDI-A | 81.50 | 86.49 | 82.75 | 80.18 | 83.13 |
| ISO-CTS + SVC | 80.70 | 86.35 | 82.50 | 78.58 | 82.48 |
| SOS-E1 (original SOS) | 81.76 | 86.39 | 83.05 | 80.18 | 83.21 |
| SOS-E3 | 82.08 | 86.92 | 83.58 | 80.77 | 83.76 |
With matched depth scaling, original SOS achieves the highest mean among the original methods, reaching 83.21% compared with 83.13% for SWUDI-A and 82.73% for full ACE. Therefore, the depth multiplier alone does not explain the improvement.
The additional configurations also help distinguish the effects of the auxiliary components. In SOS, introducing the collective structural prior at E3 increases average accuracy from 83.21% to 83.76%. For SOS, adaptive covariance normalization and spectral refinement are rarely triggered in these settings and have limited effects on performance. For ACE, the corresponding configurations achieve 82.93% at E3 and 82.73% at E4. Thus, SOS also performs better when both methods incorporate the collective structural prior, with an average advantage of 0.83 points at E3.
Finally, the limited contribution of normalized spectral shape is consistent with our main finding. Tab. 18 shows that including this component in the scalar spectral statistic improves average accuracy by only 0.06, while Fig. 3 shows that the improvement is not consistent across random task subsets. These results support our conclusion that task support and scalar spectral strength play the central roles in the merging operator. The controlled comparisons in Sec. 2 further establish the benefit of removing direction-specific weighting under matched support and trace.
Response to DjDn-W1/Q1
Thank you for your constructive feedback and thoughtful questions. We provide detailed responses and additional experimental evidence to address your concerns below.
Our calibration has two purposes. It controls the overall spectral strength and adjusts how this strength is distributed across tasks and layers.
First, F(1)/K provides a reference for the overall spectral strength. In Sec. 2, we use the reciprocal spectral response with uniform task weights wi=1/K. The total weighted trace of the task operators is exactly F(1)/K. We therefore use this quantity as the calibration target to control the aggregate strength of expert matching relative to the pretrained and ridge regularization terms as the number of tasks increases.
Second, the fractional exponent allows us to redistribute spectral strength across tasks and layers. Matching F(q) to F(1)/K constrains the global spectral mass without imposing uniform normalization on individual tasks. In contrast, explicit 1/K weighting scales all task operators uniformly and preserves their relative strengths. This distinction is important because the reciprocal response can assign excessively large weights to small singular values. For two singular values a<b, their inverse-response ratio is b/a at q=1, while a fractional exponent reduces it to (b/a)q. Consequently, the fractional exponent reduces the influence of small singular values and changes the relative strengths of task-layer operators while maintaining the reference total mass.
To examine the benefit of this redistribution, we compare the two approaches under the same global spectral mass. On ViT-B/32, we use the class-free variant and solve cF(q)=F(1)/K for each coefficient c. Explicit 1/K weighting corresponds to (q,c)=(1,1/K), while our default uses (q∗,1). The results are reported below and in Tab. 23 of Appendix D.2 of the revised paper.
The calibrated exponent improves average accuracy over explicit 1/K weighting by 0.34, demonstrating the benefit of adjusting spectral strength allocation under the same global mass. Meanwhile, the default c=1 is within 0.01 points of the best tested mean, indicating little benefit from an additional global coefficient. The layerwise scale-matched comparison in Tab. 5 further supports this finding, with calibration improving the ViT-B/32 average from 82.65% to 82.95% even when the total scalar weight in each layer is fixed.
These results support our calibration design, where F(1)/K determines the reference spectral strength and the fractional exponent adjusts its allocation across tasks and layers. We clarify this motivation in revised Sec. 3.2 and provide the additional experiments in Appendix D.2.