acceptodds
Under review as a conference paper at ICLR 2027

Spectral Components Require Distinct Coefficients in Model Merging

Abstract

Model merging combines task-specific experts into a single multi-task model without joint retraining. Cross-task orthogonalization eliminates overlap between the retained task subspaces in weight space, yet the merged model's predictions still differ from those of the task experts. We show that reweighting retained spectral components reduces this expert discrepancy while keeping their directions fixed. We find that spectral components from the same task and layer affect expert discrepancy differently and therefore require distinct coefficients. Accordingly, we jointly learn component-wise spectral coefficients in a fixed, task-structured orthogonal basis by matching expert predictions on unlabeled calibration inputs. Across three CLIP backbones and benchmarks of 8, 14, and 20 tasks, our method achieves the highest accuracy among the evaluated methods in all nine settings. On ViT-B/32, it improves normalized accuracy over the strongest baseline by 3.2, 4.0, and 6.3 percentage points with 8, 14, and 20 tasks, respectively. Experiments on MergeBench further show improvements over the evaluated merging baselines for both base and instruction-tuned Llama-3.2-3B models.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
model merging, task arithmetic

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KyfjOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies model merging after cross-task orthogonalization and argues that the remaining expert discrepancy can be reduced by learning separate scaling coefficients for individual spectral components. The proposed method fixes a task-structured orthogonal basis and jointly optimizes non-negative component-wise coefficients using expert predictions on unlabeled calibration inputs. The method is evaluated on multiple CLIP backbones with 8, 14, and 20 tasks, as well as on MergeBench with Llama-3.2-3B models. The empirical results show consistent improvements over the evaluated baselines, and the paper provides extensive ablations on coefficient granularity, optimization objectives, basis construction, and coefficient constraints.

Critical strengths
  • Broad and practically meaningful empirical improvements. The proposed method consistently improves merging performance across multiple vision backbones and task counts, suggesting that component-wise reweighting can be practically useful in a wide range of merging settings.

  • Thorough ablation studies. The paper carefully analyzes several important design choices, including coefficient granularity, optimization objectives, optimization strategies, basis construction, calibration-set size, and coefficient constraints. These experiments help clarify which parts of the proposed method are important for its empirical performance.

Critical weaknesses
  • The paper does not yet provide a sufficiently convincing mechanistic explanation for why component-wise optimization is specifically beneficial for model merging. It is plausible that a finer-grained parameterization improves the optimization objective simply because it introduces substantially more degrees of freedom. However, the current analysis does not clearly distinguish this generic increase in optimization capacity from a mechanism that is fundamentally specific to model merging or cross-task interference. Figure 4 provides useful supporting evidence by showing that different components can have different local effects on the source-task and other-task KL. However, these quantities are local gradients evaluated at γ=1\gamma=1. Components may have different gradient signs at initialization while still converging to similar scaling behavior after optimization, so this analysis alone does not establish that distinct component-wise coefficients are intrinsically necessary. Section 4.3 is also helpful, but it does not fully resolve this concern. The experiment fixes the mean coefficient within each layer-task group and allows redistribution among components. This controls for changes in the group-level mean scale, but it still leaves substantially more optimization degrees of freedom than the shared-coefficient baseline; for a group with kk components, the redistribution still introduces roughly k−1k-1 additional degrees of freedom. Therefore, this is not a parameter-count- or capacity-matched comparison. I would like to see either stronger theoretical/mechanistic justification or a comparison against an alternative parameterization with the same number of optimization degrees of freedom as the proposed component-wise scheme.
  • Important closely related baselines are missing. The paper itself identifies AdaRank and SWIM-Merging as closely related adaptive spectral merging methods, yet neither appears in the main experimental comparison.
  • Robustness under distribution shift is not evaluated. Since the proposed method introduces many more optimization variables, it has greater capacity to fit the in-domain calibration distribution. This raises a concern that improved in-domain fitting may come at the cost of increased sensitivity to calibration-deployment distribution shift. The current experiments use task-specific calibration data drawn from the corresponding validation distributions, so it remains unclear how robust the learned coefficients are under distribution shift.
  • The language-model experiments are based on a single run. Unlike the vision experiments, the language-model results are not evaluated over multiple random seeds. Since the proposed method does not outperform every baseline on every individual domain, it is difficult to assess whether the reported average improvements are statistically reliable.
Questions for authors
  • Prior work has discussed failure modes of model merging under more challenging and practically relevant conditions [1]. How robust is the proposed method in such settings? In particular, what happens when the norms of the task vectors differ substantially across tasks, or when the expert predictions used for alignment have low confidence?
  • In practical applications, task-specific input data may no longer be available at merging time. The current method assumes access to calibration inputs from each task/domain. How robust is the method when the coefficients are optimized using surrogate or proxy data instead of in-domain task-specific inputs?

[1] Kotaro Yoshida et al. “Distac: Conditioning Task Vectors via Distillation for Robust Model Merging.” International Conference on Learning Representations, 2026.

8 comments
AuthorsOpenReview ↗

Response to Reviewer Kyfj (W1)

We thank the reviewer for recognizing the broad empirical improvements across backbones and task counts and the thoroughness of our ablation studies. We respond to each point below. All changes in the revised manuscript are highlighted in blue.

W1. Mechanism versus additional degrees of freedom

We agree that Figure 4 shows local effects at γ=1\gamma=\mathbf{1} and that the redistribution in Section 4.3 is not matched in capacity, and we have revised Section 4.2 to present Figure 4 as motivation rather than proof. We address the concern with three analyses that do not rely on either.

Comparison with the same number of coefficients. Every row of Table A uses the same objective, calibration inputs, optimization, and number of coefficients as ours, and differs only in where the coefficients act. Every alternative falls below ours, by 0.8 to 2.2 points at 8 tasks and by 2.0 to 4.3 points at 20 tasks, so the gain cannot be attributed to capacity.

Table A. Absolute accuracy on ViT-B/32 with the same number of coefficients, collected from Table 4 and Table 7 of the revised manuscript

Where each coefficient acts8 tasks14 tasks20 tasks
Contiguous parameter chunks87.184.284.6
Components of the SVD of the summed update87.484.884.5
Components of the QR basis (same subspace as ours)86.083.882.3
Task-attributed polar components (ours)88.286.286.6

The gain comes from reduced interference between tasks. Because each component is attributed to a source task, we can remove the components of the other tasks when a task is evaluated (Table B). With only its own components, our method is 0.6 points above TSV-M, but with all components it is 5.8 points above, because the other tasks' components cost TSV-M 5.8 points and our method only 0.6 points. The gain therefore comes from reducing the loss caused by the other tasks' components, which arises only when task updates are combined.

Table B. Average absolute accuracy on ViT-B/32 with eight tasks. Own task only removes the components of the other seven tasks from every reweighted layer, so the task is free of their interference. All tasks keeps every component, as in the actual merged model.

Own task onlyAll tasksDrop
TSV-M88.2382.42-5.81
Ours88.8688.25-0.61

The learned coefficients remain distinct after optimization. Every coefficient starts at 1, and after optimization about one fifth are exactly zero while about 15% exceed twice the mean of their layer-task group, with at least 97% of the groups containing a zero. Randomly permuting the learned coefficients within each group keeps their values and every group mean but lowers accuracy on ViT-B/32 with 8 tasks from 88.2 to 75.0, below the 82.4 of TSV-M. The benefit therefore lies in which component receives which coefficient, which a shared coefficient cannot represent.

AuthorsOpenReview ↗

Response to Reviewer Kyfj (W2)

W2. Comparison with AdaRank and SWIM-Merging

We thank the reviewer for the suggestion and have added this comparison to the revised manuscript.

The original submission did not include AdaRank and SWIM-Merging because both perform test-time adaptation on unlabeled test inputs, so their reported numbers are not directly comparable with a method that never sees test data. However, we agree that a comparison under a common protocol is informative.

We therefore evaluate both methods under our protocol, keeping their objective and reweighting unit and replacing test inputs with the same 256 unlabeled calibration inputs per task that our method uses (Table C). All other settings, including how the entropy is computed, follow the original papers. SWIM-Merging is applied to Task Arithmetic, one of the base merges used in the original paper.

Table C. Absolute accuracy on ViT-B/32 under our calibration protocol

Method8 tasks14 tasks20 tasks
AdaRank83.578.273.2
SWIM-Merging80.572.363.2
Ours88.286.286.6

Our method outperforms both at every task count, and its margin is larger at 14 and 20 tasks than at 8 tasks.

AuthorsOpenReview ↗

Response to Reviewer Kyfj (W3)

W3. Robustness under calibration-deployment distribution shift

We agree that robustness under distribution shift was not evaluated, and we have now measured it. The coefficients are learned on clean calibration inputs and kept fixed, and only the test inputs are corrupted with the seven corruption types used by AdaMerging [1]. To isolate the effect of the number of variables, we compare coefficient units learned with the same objective and the same calibration inputs (Table D).

Table D. Absolute accuracy on ViT-B/32 with 8 tasks on clean and corrupted test inputs (average over the seven corruption types)

Method#Coeff.CleanCorrupted
TSV-M (no learning)082.4259.33
Layer-task group38486.0563.51
Parameter chunk36,86487.0564.26
Ours36,86488.2565.58

More variables do not make the method more sensitive. Our method still achieves the highest average accuracy on corrupted inputs. However, its accuracy still drops substantially on corrupted inputs, so the reviewer's concern is partly valid.

[1] Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. "AdaMerging: Adaptive Model Merging for Multi-Task Learning." International Conference on Learning Representations, 2024.

Reviewer_YAdLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the re-weighting operations for model merging. It argues that current merged models still output distinct predictions from individual experts, limiting its performance. Therefore, it further considers re-weighting from the fine-grained component level and optimizes such re-weighting factors by minimizing the divergence between the merged and individual models with unlabelled sets. Extensive experiments conducted on multiple benchmarks have demonstrated the effectiveness of the proposed method.

Critical strengths
  1. This paper is generally well organized and easy to follow.

  2. The motivation and key idea are clear and appear to be technically sound.

  3. The results appear to be competitive.

Critical weaknesses
  1. Although such an operation is not presented previously, the motivation and the proposed design are both straightforward, making it less novel to this domain.

  2. Since the proposed approach is not data-free, the selection of the unlabeled set is crucial for coefficient optimization. It's unclear if the proposed method is robust to the set selection.

  3. Figure 1 could be more clearly explained with some extra descriptions on the upper row part.

  4. Please show the exact distinction between this paper and related works.

Questions for authors

Please focus on addressing my first two concerns listed in the weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer YAdL (W1)

We thank the reviewer for finding the paper well organized and easy to follow, the motivation and key idea clear and technically sound, and the results competitive. Following the reviewer's request, we focus on the first two concerns and then address the remaining two briefly. All changes in the revised manuscript are highlighted in blue.

W1. The motivation and the design are straightforward

We agree that the method is simple. However, neither its motivation nor its design follows directly from prior work.

Prior adaptive merging methods, including Expert Merging and AdaRank, implicitly assume that components from the same task and layer can share one coefficient. Our analysis shows that this assumption does not hold. Within a layer-task group, amplifying some components reduces Expert KL on both the source task and the other tasks, whereas amplifying others increases it on both (Figure 4 and Appendix B.1), so a shared coefficient must scale both kinds in the same direction.

The design is also not a straightforward refinement, because neither of its two choices achieves the full gain on its own (Table A). Learning a coefficient for every component with entropy minimization stays 10.4 to 19.8 points below Expert KL. With Expert KL, assigning the same number of coefficients to contiguous parameter chunks, as in Expert Merging++, stays 1.1 to 2.0 points below per-component coefficients. We have added this comparison to Table 4 and revised the introduction accordingly.

Table A. Absolute accuracy on ViT-B/32 (TT is the number of tasks)

Unit and objective#Coeff.8 tasks14 tasks20 tasks
Per component, entropy minimization≈\approx 36K77.872.766.9
Per layer-task group, Expert KL48×T48\times T85.983.783.7
Per parameter chunk≈\approx 36K87.184.284.6
Per component, Expert KL (ours)≈\approx 36K88.286.286.6
AuthorsOpenReview ↗

Response to Reviewer YAdL (W2)

W2. Robustness to the selection of the unlabeled set

We agree that the calibration set matters.

All calibration-based results in Table 3 are averages over three independently sampled calibration sets. Table B reports the mean and the standard deviation of our method over these three sets. Across the nine backbone and task-count settings, the standard deviation is at most 0.19 points.

Table B. Normalized accuracy of our method over three independently sampled calibration sets (mean ± standard deviation)

Backbone8 tasks14 tasks20 tasks
ViT-B/3297.52 ± 0.1796.11 ± 0.1995.69 ± 0.05
ViT-B/1697.81 ± 0.0496.62 ± 0.0996.47 ± 0.07
ViT-L/1499.19 ± 0.0498.59 ± 0.0898.42 ± 0.03

What the method does require is that the inputs come from each task's distribution. When the inputs are replaced by images from six datasets outside the eight merged tasks, absolute accuracy falls to 82.7, because the experts' predictions on such inputs no longer reflect their tasks.

AuthorsOpenReview ↗

Response to Reviewer YAdL (W3)

W3. Description of the upper row of Figure 1

We thank the reviewer for the suggestion and have revised the caption of Figure 1. The revised caption reads as follows.

The top row shows the three stages in weight space from left to right. The task vectors ΔθA\Delta\theta_A and ΔθB\Delta\theta_B are not orthogonal, so the component of ΔθB\Delta\theta_B along ΔθA\Delta\theta_A (arrow) alters the update for task A when the two are added. Cross-task orthogonalization eliminates task-subspace overlap but leaves expert discrepancy. Spectral component reweighting reduces the remaining discrepancy. Here, Δθt\Delta\theta_t denotes the task vector for tt, Ct\mathcal{C}_t its collection of retained spectral components after orthogonalization, cjc_j an individual component, and γj\gamma_j its learned scaling coefficient.

Reviewer_Sz18OpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper identifies prediction discrepancies between merged models and task-specific experts despite cross-task orthogonalization. It proposes learning component-wise spectral coefficients in a fixed orthogonal basis to reduce these discrepancies by matching expert predictions on unlabeled calibration data.

Critical strengths
  1. The research motivation is clear; by analyzing the expert prediction biases that persist even after orthogonalization—as well as the differential impact of various spectral components on the task—the authors provide a solid experimental foundation for their method's design.

  2. The proposed method is both simple and effective; it requires no changes to the model architecture and has demonstrated strong experimental results across various tasks involving the merging of vision and language models.

Critical weaknesses
  1. This paper optimizes model merging by fixing the spectral directions and learning the coefficients for each spectral component—an approach very similar to that proposed in SWIM-Merging [1]. Although the two methods employ different spectral bases and optimization objectives, the overall methodology is quite close. The authors should clarify the specific innovations of this work compared to SWIM-Merging.

  2. While the paper cites SWIM-Merging, it does not include a direct experimental comparison. Table 6 presents experiments involving SVD-based merging, but this does not constitute the full SWIM method. It is recommended to include SWIM as a baseline and to further disentangle whether performance gains stem from the novel spectral basis construction or the "Expert KL" optimization objective.

  3. Although this method does not require labeled data, the optimization process still necessitates access to the original expert models for each task to obtain their predictions. This could pose a practical limitation. The authors should discuss the feasibility of this approach in scenarios where expert models are unavailable or when the number of tasks is large.

  4. While the paper provides comprehensive experiments on vision models, the evaluation for Large Language Models (LLMs) is limited to two 3B-parameter models. For Llama-3.2-3B-Instruct, the improvement over the strongest baseline is only 0.77 percentage points, and the method fails to achieve the best results on some tasks. The authors should discuss the applicability of this approach to larger-scale models and address why the performance gains are not more pronounced across certain tasks.

[1] Li B, Xu J, Liu W, et al. SWIM-merging: Reweighting singular values of weight matrices for efficient test-time model merging [J]. Neurocomputing

Questions for authors

See Weaknesses.

7 comments
AuthorsOpenReview ↗

Response to Reviewer Sz18 (W1)

We thank the reviewer for recognizing the clear motivation grounded in our analysis and the simplicity and effectiveness of the method on both vision and language models. We respond to each point below. All changes in the revised manuscript are highlighted in blue.

W1. Relation to SWIM-Merging and the specific contribution of this work

We agree that both methods learn one coefficient per component while keeping the spectral directions fixed. SWIM-Merging reweights the singular values of an already merged model by minimizing prediction entropy on test inputs. We jointly reweight components derived from individual tasks using Expert KL on unlabeled calibration inputs before deployment. This difference is not minor. With the components held fixed, replacing Expert KL with entropy minimization lowers accuracy by 3.4 to 6.0 points (see W2).

What the paper adds beyond the objective. SWIM-Merging's components are not attributed to source tasks, so it does not examine which tasks a given component helps or harms. Every component in our basis is attributed to a source task (Appendix B.4), which lets us measure its effect on its own task and on the other tasks (Section 4.2) and test redistribution within a layer–task group at a fixed group mean (Section 4.3). It also lets us remove the other tasks' components. On ViT-B/32 with eight tasks, this removal raises average accuracy by 5.81 percentage points for TSV-M and by only 0.61 points for our method (Table A). The learned weights therefore reduce the accuracy penalty of including other tasks' components.

Table A. Average absolute accuracy on ViT-B/32 with eight tasks. Own task only removes the components of the other seven tasks from every reweighted layer, so the task is free of their interference. All tasks keeps every component, as in the actual merged model.

Own task onlyAll tasksDrop
TSV-M88.2382.42-5.81
Ours88.8688.25- 0.61
AuthorsOpenReview ↗

Response to Reviewer Sz18 (W2)

W2. Direct comparison with SWIM-Merging, and disentangling basis from objective

We agree that the SVD-based row of Table 6 in the original submission is not the full SWIM-Merging method, since it changes only the basis and keeps our objective. The original submission did not include SWIM-Merging itself because it performs test-time adaptation on unlabeled test inputs, so its reported numbers are not directly comparable with a method that never sees test data.

We therefore evaluate SWIM-Merging under our protocol, keeping its objective and reweighting unit and replacing test inputs with the same 256 unlabeled calibration inputs per task that our method uses (Table B). All other settings, including how the entropy is computed, follow the original paper. SWIM-Merging is applied to Task Arithmetic, one of the base merges used in the original paper.

Table B. Absolute accuracy on ViT-B/32 under our calibration protocol

ComponentsObjective8 tasks14 tasks20 tasks
SWIM-MergingSVD of the summed task updateEntropy minimization80.572.363.2
Basis replacedSVD of the summed task updateExpert KL87.484.884.5
Objective replacedTask-attributed componentsEntropy minimization77.872.766.9
OursTask-attributed componentsExpert KL88.286.286.6

Under this protocol, our method exceeds SWIM-Merging by 7.7, 13.8, and 23.4 points at 8, 14, and 20 tasks.

The revised Section 5.2 separates the two factors by replacing one at a time (second and third rows of Table B).

  • Replacing only the objective, from Expert KL to entropy minimization on our components, lowers accuracy by more than 10 points at every task count.
  • Replacing only the basis, from our task-attributed components to the SVD of the summed task update under Expert KL, lowers accuracy by at most 2.1 points.

The objective therefore accounts for most of the gain.

AuthorsOpenReview ↗

Response to Reviewer Sz18 (W3)

W3. Access to the task experts

We agree that our method requires the task experts, which is the same assumption as in Expert Merging, and we have now stated it explicitly in the revised Section 3.2. In practice, the fine-tuned experts are the inputs of model merging, so they are typically available when the merge is performed. When only task vectors are released, each expert can be recovered as the pre-trained model plus its task vector. The experts are used only to compute predictions on the calibration inputs, and only the merged model is deployed.

The cost is small and grows linearly in the number of tasks. The experts are processed one at a time, each running a single forward pass over its calibration inputs, and their predictions are kept as fixed targets. Peak GPU memory therefore grows only by these cached inputs and predictions (Table C). The number of coefficients stays roughly constant, since each weight matrix retains ⌊r/T⌋\lfloor r/T \rfloor components per task and thus at most rr in total. Each step processes one mini-batch per task, so merging time grows linearly with the number of tasks, and with 20 tasks our method merges faster than the baselines that search α\alpha (Table D).

Table C. Peak GPU memory (GB) on ViT-B/32 with a single GPU

8 tasks14 tasks20 tasks
Calibration cache1.232.163.08
Optimization3.113.143.26
Total4.345.306.34

Table D. Merging time (minutes) on ViT-B/32 with a single GPU

8 tasks14 tasks20 tasks
TSV-M with α\alpha search13.330.237.6
ESM with α\alpha search13.027.945.5
Ours10.417.023.0

With 20 tasks, our method merges faster than the baselines that search α\alpha, and its margin over the strongest baseline grows with the number of tasks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.