acceptodds
Under review as a conference paper at ICLR 2027

Revisiting Singular Structure in Data-Free Model Merging

Abstract

Recent data-free model merging methods increasingly exploit the singular structure of task updates, yet they often couple singular-vector support and singular-value weighting within a single spectral operator. We revisit this design choice and study how different components of the singular structure should actually be used for merging. Under a common covariance-weighted framework, we disentangle task support, within-support anisotropy, and task-layer scale. Our analysis shows that right singular vectors provide a useful task-relevant support, while singular values are more effective when compressed into a scalar task-layer scale than when used for direction-wise reweighting. In particular, a scale-only inverse spectral response consistently outperforms its anisotropic counterpart under matched support and total scale. Motivated by this finding, we propose SOS-Merge, which represents each task operator as a right-singular projector scaled by a fractional inverse spectral moment. We further develop a data-free calibration rule for the shared spectral exponent, achieving performance close to post-hoc grid search without validation data. Across three CLIP ViT backbones and 8-, 14-, and 20-task merging settings, SOS-Merge consistently outperforms recent data-free baselines, with its advantage becoming more pronounced as more tasks are merged.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Model Merging

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_DjDnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the respective roles of singular vectors and singular values of task update matrices in data-free model merging. Under a unified covariance-weighted merging framework, the authors decompose the task operator into three components: the support defined by the right-singular subspace, the anisotropic weights within that support, and the aggregate spectral scale of the operator. The core experiments show that, while retaining the same right-singular subspace and matching the operator trace, replacing direction-specific inverse spectral weights with a uniform scalar improves merging performance across all tested settings. Based on this finding, the authors propose SOS-Merge, which constructs the task operator using the right-singular projection and a fractional inverse spectral moment, and calibrates the exponent from checkpoint spectral information. The method is validated on three CLIP ViT backbones under 8/14/20-task settings.

Critical strengths
  1. The research question is clearly formulated, and the core controlled experiments are convincing.

  2. The performance gains are fairly consistent, and become more pronounced when more tasks are merged.

  3. The ablations are relatively thorough, and the paper is transparent about weaker results.

  4. The method is easy to understand and requires neither task samples nor additional training.

Critical weaknesses
  1. Insufficient justification for exponent calibration. Equation (15) selects qq by matching F(q)F(q) to F(1)/KF(1)/K, but the rationale for this target is unclear. The analysis uses wi=1/Kw_i=1/K, whereas the final method uses wi=1w_i=1. Global spectral-mass matching is not equivalent to task-wise normalization, since changing qq also redistributes weights across tasks and layers.

  2. Spectral scale is confounded with relative regularization. The Shape–Scale comparison changes task-operator magnitudes while fixing C0C_0 and λ\lambda, thereby changing the balance between expert matching and regularization. Its performance gap therefore cannot distinguish useful spectral-scale information from a better regularization balance.

  3. The contribution and performance gains need clearer attribution. ISO [1] methods already exploit isotropic spectral constructions, while ACE-Merging [2] provides covariance-based closed-form merging. Moreover, applying the same depth multiplier raises ACE-Merging to 86.38%, reducing SOS-Merge’s advantage to 0.5%. Table 16 further shows only an approximately 0.06% benefit from normalized spectral shape, without consistent gains across random subsets. The evidence supports appropriate scalar scaling more strongly than the necessity of the full spectral statistic.

  4. Baseline comparisons and experimental scope are limited. The paper does compare against several recent merging methods, but it still omits some important recent baselines, particularly ESM [3] and RegMean++ [4], which also evaluate model merging on CLIP ViT benchmarks. Nor does it discuss why such comparisons are omitted or infeasible. In addition, the paper lacks experiments in NLP scenarios and does not discuss whether the method is limited to vision scenarios.

  5. Limited support for classification-aware refinement. Theorem 1 gives a standard sub-Gaussian union bound with log⁡Ni\sqrt{\log N_i} dependence, but does not justify multiplying the spectral exponent by this factor. The improvement is only 0.17%, and its robustness across independent checkpoints or task subsets is unclear. Additional evidence would help justify this refinement. The required access to task class counts should also be stated explicitly as a metadata assumption.

[1] No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces. ICML, 2025.

[2] ACE-Merging: Data-Free Model Merging with Adaptive Covariance Estimation. CVPR, 2026.

[3] Model Merging in the Essential Subspace. CVPR, 2026.

[4] RegMean++: Enhancing Effectiveness and Generalization of Regression Mean for Model Merging. TMLR, 2026.

Questions for authors
  1. What motivates the calibration target F(1)/KF(1)/K, and how does it compare with explicit 1/K1/K task weighting?
  2. Does the advantage of Scale over Shape persist after controlling for their relative regularization strength?
  3. How much improvement remains when strong baselines use the same depth multiplier?
  4. Could the authors provide comparisons with ESM and RegMean++ under clearly stated data-access assumptions, or explain their omission? Is the method applicable beyond vision?
  5. Is the classification-aware refinement consistently beneficial across different checkpoints or task subsets?
9 comments
AuthorsOpenReview ↗

Response to DjDn-W1/Q1

Thank you for your constructive feedback and thoughtful questions. We provide detailed responses and additional experimental evidence to address your concerns below.

W1/Q1: Insufficient justification for exponent calibration. What motivates the calibration target F(1)/KF(1)/K, and how does it compare with explicit 1/K1/K task weighting? Global spectral-mass matching is not equivalent to task-wise normalization

Our calibration has two purposes. It controls the overall spectral strength and adjusts how this strength is distributed across tasks and layers.

First, F(1)/KF(1)/K provides a reference for the overall spectral strength. In Sec. 2, we use the reciprocal spectral response with uniform task weights wi=1/Kw_i=1/K. The total weighted trace of the task operators is exactly F(1)/KF(1)/K. We therefore use this quantity as the calibration target to control the aggregate strength of expert matching relative to the pretrained and ridge regularization terms as the number of tasks increases.

Second, the fractional exponent allows us to redistribute spectral strength across tasks and layers. Matching F(q)F(q) to F(1)/KF(1)/K constrains the global spectral mass without imposing uniform normalization on individual tasks. In contrast, explicit 1/K1/K weighting scales all task operators uniformly and preserves their relative strengths. This distinction is important because the reciprocal response can assign excessively large weights to small singular values. For two singular values a<ba<b, their inverse-response ratio is b/ab/a at q=1q=1, while a fractional exponent reduces it to (b/a)q(b/a)^q. Consequently, the fractional exponent reduces the influence of small singular values and changes the relative strengths of task-layer operators while maintaining the reference total mass.

To examine the benefit of this redistribution, we compare the two approaches under the same global spectral mass. On ViT-B/32, we use the class-free variant and solve cF(q)=F(1)/KcF(q)=F(1)/K for each coefficient cc. Explicit 1/K1/K weighting corresponds to (q,c)=(1,1/K)(q,c)=(1,1/K), while our default uses (q∗,1)(q^*,1). The results are reported below and in Tab. 23 of Appendix D.2 of the revised paper.

TasksExplicit 1/K1/Kc=0.4c=0.4c=0.6c=0.6c=0.8c=0.8c=1c=1 (ours)c=1.2c=1.2
886.15686.21586.23886.22486.24486.247
1482.35282.65082.68982.70682.70482.700
2079.31379.77579.83679.89379.90179.929
Mean82.60782.88082.92182.94182.94982.959

The calibrated exponent improves average accuracy over explicit 1/K1/K weighting by 0.34, demonstrating the benefit of adjusting spectral strength allocation under the same global mass. Meanwhile, the default c=1c=1 is within 0.01 points of the best tested mean, indicating little benefit from an additional global coefficient. The layerwise scale-matched comparison in Tab. 5 further supports this finding, with calibration improving the ViT-B/32 average from 82.65% to 82.95% even when the total scalar weight in each layer is fixed.

These results support our calibration design, where F(1)/KF(1)/K determines the reference spectral strength and the fractional exponent adjusts its allocation across tasks and layers. We clarify this motivation in revised Sec. 3.2 and provide the additional experiments in Appendix D.2.

AuthorsOpenReview ↗

Response to DjDn-W2/Q2

W2/Q2: Spectral scale is confounded with relative regularization. The Shape–Scale comparison changes the relative strength of expert matching and regularization. Does the advantage of Scale over Shape persist after controlling for their relative regularization strength?

To examine whether the Shape–Scale difference is caused by the relative regularization strength, we conducte an additional experiment that allows each operator to independently adjust its complete regularizer. Specifically, we multiply the combined pretrained covariance and ridge regularizer by cc, where c∈{0.01,0.1,1,10}c\in\{0.01,0.1,1,10\}. This directly changes the balance between expert matching and regularization while keeping the task operators unchanged. We select cc for each operator using held-out validation accuracy, with the base ridge coefficient fixed at λ=0.2\lambda=0.2.

On ViT-B/32 with 8, 14, and 20 tasks, Shape selects c=0.01c=0.01 for both exponents, while Full and Scale select c=1c=1. The results are reported below and in Tab. 22 of Appendix D.1.

ttTasksShape (c=1c=1)Shape (c=0.01c=0.01)Full (c=1c=1)Scale (c=1c=1)
-1.0870.4784.6986.0586.16
-1.01467.6978.6181.0782.35
-1.02063.5374.4876.6379.31
-0.5874.0484.0384.2184.91
-0.51469.4278.1578.3179.97
-0.52064.9873.8673.1775.73
Average68.3678.9779.9181.41

Adjusting the complete regularizer substantially improves Shape, raising its average accuracy from 68.36% to 78.97%. This shows that the original Shape–Scale gap was partly affected by regularization strength. However, Scale remains the best-performing operator in all six settings, outperforming the adjusted Shape by 2.44 percentage points on average.

More importantly, the Full–Scale comparison directly examines the effect of spectral anisotropy. Full and Scale have identical support and trace for every task and layer, and both select c=1c=1. Therefore, they share the same total spectral strength and regularization setting, with direction-specific weighting being the only difference in their task operators. Under these matched conditions, Scale outperforms Full by 1.50 on average.

These results show that the advantage of Scale persists after independently adjusting regularization strength. In particular, the Full–Scale comparison supports our central finding that removing direction-specific inverse weighting improves merging performance under matched support and spectral mass.

AuthorsOpenReview ↗

Response to DjDn-W3/Q3

W3/Q3: The contribution and performance gains need clearer attribution. The depth multiplier contributes to SOS-Merge's performance gains, while the normalized spectral-shape component provides only a small improvement. How much improvement remains when strong baselines use the same depth multiplier?

To examine how much of the improvement comes from the depth multiplier, we applied the same depth scaling to SOS-Merge and the baselines. Across all nine settings, SOS achieves 86.92%, compared with 86.38% for ACE with the same multiplier. Without depth scaling, SOS also outperforms ACE by 0.30, reaching 86.04% versus 85.74% (Tab. 8). Thus, SOS maintains its advantage under both conditions.

We further extended the comparison to SWUDI-A and ISO-CTS + SVC on ViT-B/32. In addition, we examined whether the performance differences remain when ACE's auxiliary components are incorporated into SOS. For this comparison, we considered four cumulative configurations, denoted E1 to E4. E1 represents each method's basic merging formulation, with SOS-E1 corresponding to the original SOS-Merge. E2 adds adaptive covariance normalization, E3 introduces a collective structural prior, and E4 further adds spectral refinement. Full ACE-Merging corresponds to ACE-E4. The complete configurations and results are provided in Tab. 26 of Appendix D.4.

ConfigurationNo depth scaling, meanMatched scaling, 8 tasks14 tasks20 tasksMean
ACE-E381.5686.3882.4379.9882.93
ACE-E4 (full ACE)81.5686.3982.0479.7782.73
SWUDI-A81.5086.4982.7580.1883.13
ISO-CTS + SVC80.7086.3582.5078.5882.48
SOS-E1 (original SOS)81.7686.3983.0580.1883.21
SOS-E382.0886.9283.5880.7783.76

With matched depth scaling, original SOS achieves the highest mean among the original methods, reaching 83.21% compared with 83.13% for SWUDI-A and 82.73% for full ACE. Therefore, the depth multiplier alone does not explain the improvement.

The additional configurations also help distinguish the effects of the auxiliary components. In SOS, introducing the collective structural prior at E3 increases average accuracy from 83.21% to 83.76%. For SOS, adaptive covariance normalization and spectral refinement are rarely triggered in these settings and have limited effects on performance. For ACE, the corresponding configurations achieve 82.93% at E3 and 82.73% at E4. Thus, SOS also performs better when both methods incorporate the collective structural prior, with an average advantage of 0.83 points at E3.

Finally, the limited contribution of normalized spectral shape is consistent with our main finding. Tab. 18 shows that including this component in the scalar spectral statistic improves average accuracy by only 0.06, while Fig. 3 shows that the improvement is not consistent across random task subsets. These results support our conclusion that task support and scalar spectral strength play the central roles in the merging operator. The controlled comparisons in Sec. 2 further establish the benefit of removing direction-specific weighting under matched support and trace.

Reviewer_9DeJOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how the singular structure of task updates should enter a covariance-weighted model-merging pipeline. The objective and its closed-form solution correspond to a RegMean-style merge with an additional term, C0ℓC_{0\ell}, that acts as a regularizer. The manuscript analyzes several variants of the task operator CiℓC_{i\ell}, which is built from a low-rank SVD of the per-layer task updates. The analysis separates three factors: the support (the right-singular projector PiℓP_{i\ell}), the anisotropy within that support, and the total spectral mass. It includes an interesting experiment showing that inverse spectral responses (Σ~t\tilde{\Sigma}^{t} with t<0t<0) outperform direct ones (t>0t>0) by a wide margin. It also indicates that an isotropic operator outperforms an anisotropic one. Building on these findings, the paper proposes SOS-Merge, a method built in four parts:

  1. the task operator Ciℓ=siℓPiℓC_{i\ell} = s_{i\ell} P_{i\ell}, where siℓs_{i\ell} is a fractional inverse spectral moment;
  2. a shared exponent calibrated from the checkpoint spectra;
  3. a class-count modulation of the exponent;
  4. a LiNeS-style depth multiplier applied after the closed-form solve.
Critical strengths
  • The paper asks which components of the SVD actually help in spectral and covariance-based model merging. This is a valuable question, and the controlled Full/Scale/Shape ablation addresses it directly.
  • The advantage of Scale over Full holds in all nine settings and for both exponents in Table 3, which strengthens this finding.
Critical weaknesses
  1. Clarity. The manuscript is difficult to follow, because its notation and phrasing are often more complex than necessary. For example, Table 2 evaluates t∈{−1,−0.5,0.5,1}t \in \{-1, -0.5, 0.5, 1\} but groups these values into “Strong” (∣t∣=1|t| = 1) and “Moderate” (∣t∣=0.5|t| = 0.5) pairs. This yields the interleaved order 1,−1,0.5,−0.51, -1, 0.5, -0.5. Presenting the values as a single monotone sweep over tt would make the trend easier to read.
  2. Motivation and derivation of the objective. Equations (1) and (2) come with only a one-sentence motivation (matching the outputs of the merged and expert layers) and cite no prior work. There is no background explaining where this formulation comes from or why it is appropriate. Readers familiar with the area can reconstruct the reasoning, but readers new to the field are likely to find these equations opaque.
    • Neither the equality in Eq. (1) nor the closed-form solution in Eq. (3) is derived, in the main text or in the appendix.
    • The additional pretrained-model term ∥MℓC0ℓ1/2∥F2\|M_\ell C_{0\ell}^{1/2}\|_F^2 in Eq. (2) appears to be an arbitrary choice; no justification is given for including it. I would have preferred a brief background on the closely related RegMean (Jin et al., ICLR 2023, “Dataless Knowledge Fusion by Merging Weights of Language Models”) and MaTS (Tam et al., TMLR 2024, “Merging by Matching Models in Task Parameter Subspaces”) formulations.
  3. Range of the spectral exponent. Tables 2 and 3 only test exponents with ∣t∣≤1|t| \le 1. The best-performing value, t=−1t = -1, lies on the boundary of this range, and the choice of range is not motivated in Sec. 2. It remains unclear whether the trend continues beyond this boundary, e.g., at t=−1.1t = -1.1.
  4. Checkpoint provenance. App. B.1 states that the paper follows the CLIP benchmark used by Representation Surgery, ISO-C/ISO-CTS, TSV-M and ACE-Merging. However, the fine-tuned reference accuracies are 1.2–2.3 points lower than those reported by ISO-C/ISO-CTS, TSV-M and ACE-Merging. This suggests that the paper uses different checkpoints from those works, but the paper states neither the source of its checkpoints nor the fine-tuning recipe.
  5. Scope. The evaluation is limited to vision benchmarks. It does not include the NLP settings commonly used in this literature, such as GLUE or LLM merging benchmarks.
Questions for authors
  1. Connection to Iso-C. Making the spectrum isotropic is closely related to Iso-C/Iso-CTS. Note that with t=−1t = -1, ΔiℓCiℓfull=UiℓViℓ⊤\Delta_{i \ell} C_{i \ell}^{\text{full}} = U_{i \ell} V_{i \ell}^\top. The first factor of Eq. (3), ∑iwiΔiℓCiℓ\sum_i w_i \Delta_{i\ell} C_{i\ell}, therefore becomes a sum of spectrally flattened task updates, in the same spirit as Iso-C. Why is this connection not discussed explicitly?
  2. Best-vs-best comparison. How do Full, Shape and Scale compare when each is given its own best global multiplier (or λ\lambda)?
  3. Hyperparameter selection. How were λ=0.2\lambda = 0.2, r=48r = 48, the reference B=F(1)/KB = F(1)/K and αℓ=0.5+bℓ/bmax⁡\alpha_\ell = 0.5 + b_\ell / b_{\max} chosen? Which criterion guided these choices (e.g., validation accuracy), and on which data?
23 comments
AuthorsOpenReview ↗

Response to 9DeJ-W1

Thank you for your careful review and detailed comments. We detail our response below point by point. Please kindly let us know whether you have any further concerns.

W1: Clarity. The manuscript is difficult to follow, because its notation and phrasing are often more complex ... Table 2 evaluates t=1,−1,0.5,−0.5t=1,-1,0.5,-0.5 ... Presenting the values as a single monotone sweep over would make the trend easier to read.

The column order in Tab. 2 is designed to highlight two controlled comparisons, rather than to present a sweep over tt. Specifically, we compare positive and negative spectral responses at the same exponent magnitude, while keeping the right-singular support fixed. Therefore, placing t=1t=1 next to t=−1t=-1, and t=0.5t=0.5 next to t=−0.5t=-0.5, makes these comparisons more direct and easier to follow.

This arrangement is important because our goal in Sec. 2.3 is to understand whether direct or inverse spectral weighting is more effective. As shown in Tab. 2, inverse weighting consistently performs better across all nine settings. The average accuracy increases from 63.39% to 85.29% for ∣t∣=1|t|=1, and from 68.54% to 82.57% for ∣t∣=0.5|t|=0.5. We further separate the effects of spectral anisotropy and total scale in Tab. 3.

We will revise Sec. 2.3 and Table 2 caption explicitly explain this paired comparison, and simplify the notation to make it easier to follow.

AuthorsOpenReview ↗

Response to 9DeJ-W2

W2: Motivation and derivation of the objective.

W2-part1: The connection to RegMean and MaTS is missing.

Our objective follows the layerwise output matching principle used in regression based merging, including RegMean and MaTS. We adopt this framework because merging aims to preserve the behavior of several task experts, and the main question in our work is how the matching operators can be constructed when only their checkpoints are available. We will discuss this connection explicitly in Sec. 2.1 and Related Work in the revised paper.

W2-part2: Eqs. (1)-(3) lack sufficient motivation and derivation. The pretrained-model term in Eq. (2) appears arbitrary.

The motivation and derivation of Eqs. (1)-(3) are as follow.

First, Eq. (1) follows directly from output matching. For a shared layer input xx, the output difference between the merged model and expert ii is (Mℓ−Δiℓ)x(M_\ell-\Delta_{i\ell})x. Taking its expected squared norm and using Ciℓact=E[xx⊤]C_{i\ell}^{act}=\mathbb{E}[xx^\top] gives $\mathbb{E}\lVert(M_{\ell}-\Delta_{i\ell})x\rVert_2^2 = \lVert(M_{\ell}-\Delta_{i\ell})(C_{i\ell}^{\mathrm{act}})^{1/2}\rVert_F^2 $. Since task activations are unavailable, we replace their second moments with checkpoint-derived task operators.

Second, the pretrained term in Eq. (2) is NOT arbitrary. It is designed to serve as a reference constraint. The pretrained model has zero task update Δ0ℓ=0\Delta_{0\ell}=0, so its output difference from the merged model is MℓxM_\ell x. We therefore penalize this deviation using ∥MℓC0ℓ1/2∥F2\lVert M_\ell C_{0\ell}^{1/2}\rVert_F^2, where C0ℓC_{0\ell} is constructed from the truncated input-side Gram matrix of the pretrained weights. This penalizes changes more strongly along directions with larger pretrained responses, while the ridge term provides additional regularization. Removing the pretrained term reduces average accuracy from 86.92% to 86.62% (Tab. 8).

Finally, Eq. (3) is the closed-form solution of Eq. (2). Eq. (2) is quadratic in MℓM_\ell. Setting its gradient to zero gives ∑iwi(Mℓ−Δiℓ)Ciℓ+MℓC0ℓ+λMℓ=0\sum_i w_i(M_\ell-\Delta_{i\ell})C_{i\ell}+M_\ell C_{0\ell}+\lambda M_\ell=0, which directly yields Eq. (3).

We add this explanation to Sec. 2.1, with the complete derivation in the appendix A.2 in the revised paper.

AuthorsOpenReview ↗

Response to 9DeJ-W3

W3: Range of the spectral exponent.. Tables 2 and 3 only test exponents with ∣t∣≤1|t|\le1... It remains unclear whether the trend continues beyond this boundary, e.g., at t=−1.1t=-1.1.

The original experiments used t=±1t=\pm1 and t=±0.5t=\pm0.5 to compare direct and inverse weighting at matched exponent magnitudes rather than to identify the optimal tt.

To examine the behavior beyond t=−1t=-1, we extended the sweep from t=−0.5t=-0.5 to t=−1.5t=-1.5 in steps of 0.10.1, and additionally evaluated t=−2t=-2. All other settings remain unchanged. The table below reports the average accuracy across all nine backbone–task-count settings.

The results show that Scale reaches a broad peak around t=−1t=-1. Its best accuracy is 86.46% at t=−1.1t=-1.1, only 0.07 percentage points above the originally reported result at t=−1t=-1. Thus, extending the range does not achieve a large improvement beyond the original boundary. Moreover, Scale outperforms Full throughout t∈[−1.5,−0.5]t\in[-1.5,-0.5], although the gap narrows for sharper inverse responses. At t=−2t=-2, both methods deteriorate, and Full becomes better than Scale. This indicates that increasingly strong inverse weighting is not always beneficial. These additional results support our original finding for moderate inverse responses while clarifying its limits. The extended sweep and its interpretation are included in Appendix D.1 in the revised paper.

ttShapeScaleFull
-0.574.4383.8882.57
-0.674.0684.7583.35
-0.773.6485.3984.02
-0.873.2585.8784.53
-0.972.8686.2084.97
-1.072.4886.3985.29
-1.172.1086.4685.53
-1.271.7286.4585.69
-1.371.3586.3285.78
-1.470.9986.0785.78
-1.570.6285.7285.70
-2.069.0881.4483.73
Reviewer_mWarOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper separates right-singular support, within-support anisotropy, and overall spectral scale in data-free model merging. Under matched support and trace, replacing direction-specific inverse spectral weights with a single scalar consistently improves performance. The resulting SOS-Merge achieves competitive results across nine CLIP merging settings.

Critical strengths
  1. The full scale comparison is well controlled and provides useful evidence that direction-specific inverse weighting is unnecessary in the tested framework.
  2. The method performs well against strong baselines, particularly when merging more tasks.
Critical weaknesses
  1. Some calibration rule in this paper lacks a clear rationale, e.g., why should matching F(q)F(q) to F(1)/KF(1)/K select a good exponent?
  2. All experiments use CLIP vision encoders, this limits confidence that the central finding and calibration rule transfer beyond this model family, despite their broadly applicable formulation.
Questions for authors
  1. What motivates the specific calibration target (F(1)/K), and how does it relate to preserving expert behavior? The choice appears to lack a clear theoretical justification.
  2. Since siℓs_{i\ell} may partly compensate for this imbalance, do SOS-Merge’s relative gains increase when task-update magnitudes differ more?
  3. Could the authors provide additional merging experiments on LLMs such as Qwen? Experiments on other architectures would help assess whether the findings extend beyond CLIP.
5 comments
AuthorsOpenReview ↗

Response to mWar-W1/Q1

Thank you for your careful review and helpful suggestions. We address your questions below, including additional experiments to further examine the proposed method.

W1/Q1: What motivates the specific calibration target F(1)/KF(1)/K, and how does it relate to preserving expert behavior? why should matching F(q)F(q) to F(1)/KF(1)/K select a good exponent?

Our calibration has two purposes. It controls the overall strength of expert matching and adjusts how this strength is distributed across tasks and layers.

First, F(1)/KF(1)/K provides a reference for the overall spectral strength. In Eq. (2), the task operators determine how strongly the merged model is encouraged to match each expert, while the pretrained and ridge terms constrain changes to the model. Therefore, their relative strengths affect the balance between preserving expert behavior and retaining pretrained knowledge. In Sec. 2, we use the reciprocal spectral response with uniform task weights wi=1/Kw_i=1/K, whose total weighted trace is exactly F(1)/KF(1)/K. We use this quantity as the calibration target to maintain a task-normalized reference strength as the number of experts increases.

Second, the fractional exponent adjusts the allocation of this strength. The reciprocal response can assign excessively large weights to small singular values. For two singular values a<ba<b, their inverse-response ratio is b/ab/a at q=1q=1, while a fractional exponent reduces it to (b/a)q(b/a)^q. By selecting qq to match F(q)F(q) with F(1)/KF(1)/K, we maintain the reference total strength while reducing the disproportionate influence of small singular values across task-layer operators.

Our experiments support the effectiveness of this calibration. Under matched layerwise total scalar weight, the calibrated class-free response improves average accuracy from 86.46% with the fixed reciprocal response to 86.75% (Tab. 5). Moreover, Fig. 2 shows that the calibrated exponent achieves accuracy within 0.05 percentage points of the best sampled exponent on average across all nine settings.

We further examine the calibration under the same global spectral mass by solving cF(q)=F(1)/KcF(q)=F(1)/K for different coefficients cc. The following results on ViT-B/32 are also reported in Tab. 23 of Appendix D.2 of the revised paper.

TasksExplicit 1/K1/KCalibrated (c=1c=1)Best tested cc
886.15686.24486.247
1482.35282.70482.706
2079.31379.90179.929
Mean82.60782.94982.959

The calibrated exponent improves over explicit 1/K1/K weighting by 0.34 on average, while the default c=1c=1 is within 0.01 points of the best tested mean. Together, these results show that the calibration provides an effective balance between overall matching strength and its allocation across tasks and layers, using only checkpoint spectra. We clarify this motivation in revised Sec. 3.2 and provide the additional comparisons in Appendix D.2.

AuthorsOpenReview ↗

Response to mWar-Q2

Q2: Since siℓs_{i\ell} may partly compensate for this imbalance, do SOS-Merge’s relative gains increase when task-update magnitudes differ more?

To examine how task-update magnitude imbalance affects merging performance, we conduct a controlled experiment on ViT-B/32. We rank the task updates by their original ℓ2\ell_2 norms and divide them into two groups. To increase magnitude imbalance, we scale the smaller-norm updates by 0.9 and the larger-norm updates by 1.1. We then reverse these factors to reduce the imbalance. Both methods use the same modified expert checkpoints, and we also evaluate the modified experts individually.

The following results are averaged over the 8, 14, and 20-task settings and reported in Tab. 27 of Appendix D.4.

SettingNorm CVExpertACESOSSOS − ACE
Original0.10889.9181.5683.21+1.64
Increased imbalance0.19689.6881.3383.36+2.02
Reduced imbalance0.06889.9680.4982.68+2.19

Norm CV measures the coefficient of variation of task-update norms across experts. All values are averaged over the three task-count settings. The SOS–ACE differences are calculated before rounding.

Increasing magnitude imbalance raises SOS's advantage over ACE from 1.64 to 2.02, while the single-task expert accuracy changes only slightly. However, reducing the imbalance also increases the relative gain to 2.19 because ACE's performance decreases more than SOS's.

These results show that SOS's advantage does not increase monotonically with magnitude imbalance. Although scalar spectral weights adjust the contributions of different task updates, their benefit cannot be explained by magnitude differences alone. We include this analysis in revised Appendix D.4 to further examine the role of magnitude compensation.

AuthorsOpenReview ↗

Response to mWar-W2/Q3

W2/Q3: All experiments use CLIP vision encoders. Could the authors provide additional merging experiments on LLMs such as Qwen? Experiments on other architectures would help assess whether the findings extend beyond CLIP.

To examine whether SOS-Merge extends beyond CLIP, we evaluate RoBERTa-Base/Large and Qwen2.5-1.5B/7B. These experiments cover both encoder-based and decoder-only language models, with tasks involving language understanding, mathematical reasoning, and code generation. We use the class-free SOS variant throughout, with hi=1h_i=1 and the shared exponent calibrated directly from checkpoint spectra.

Qwen2.5-1.5B and Qwen2.5-7B. We follow the Math + Coder expert-merging setting and benchmark suite of the PRISM study, which includes six general tasks and four additional generation metrics. We compare SOS with Task Arithmetic, TIES, ACE, and PRISM. Complete results are provided in Tabs. 6 and 7 of revised Sec. 4.5, with experimental details in Appendix D.5.

ModelMethod6-task meanGSM8K-CoTMATH-500HumanEvalMBPP10-metric mean
1.5BTask Arithmetic36.844.601.601.832.2023.13
TIES42.4317.205.0010.9813.6030.14
ACE47.1028.6010.0015.2423.6036.01
PRISM55.6558.4026.6034.7642.8049.65
SOS56.2762.8027.2036.5943.8050.80
7BTask Arithmetic49.8521.807.2012.8027.4036.83
TIES61.1066.2022.2026.2241.6052.28
ACE59.2056.0020.0040.8541.6051.37
PRISM67.8581.6032.2051.2263.2063.53
SOS67.9783.2038.6054.8863.6064.81

The six-task mean covers GSM8K, ARC-Challenge, MMLU, TruthfulQA, HellaSwag, and WinoGrande. The ten-metric mean additionally includes GSM8K-CoT, MATH-500, HumanEval, and MBPP, and therefore includes GSM8K under two different evaluation protocols. ACE uses c=0.2c=0.2. Bold indicates the best merging method, while the unmerged base model is included as a reference.

SOS achieves the highest six-task and ten-metric averages among the compared merging methods at both model sizes. In particular, its ten-metric mean exceeds PRISM by 1.15 at 1.5B and 1.28 at 7B. At the same time, SOS maintains performance close to the unmerged base models, reaching 50.80% versus 50.58% at 1.5B and 64.81% versus 64.88% at 7B.

RoBERTa on GLUE. We also merge eight task experts using RoBERTa-Base and RoBERTa-Large and evaluate them on the 6,562 examples in the WUDI protocol. Neither method uses depth scaling. Tabs. 28 and 29 in Appendix D.5 provide the complete task scores, normalized by the corresponding individual experts.

ModelACESOS + ACE components
RoBERTa-Base90.3790.88
RoBERTa-Large91.6892.41

SOS improves the normalized GLUE macro-average by 0.51 and 0.73 on the two backbones.

Together, these experiments extend our evaluation across different language model architectures and scales. Importantly, both settings use the scale-only spectral construction and checkpoint-based calibration without requiring task data to select the exponent during merging. The results therefore provide further evidence that the proposed design is effective beyond CLIP.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.