- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies model merging after cross-task orthogonalization and argues that the remaining expert discrepancy can be reduced by learning separate scaling coefficients for individual spectral components. The proposed method fixes a task-structured orthogonal basis and jointly optimizes non-negative component-wise coefficients using expert predictions on unlabeled calibration inputs. The method is evaluated on multiple CLIP backbones with 8, 14, and 20 tasks, as well as on MergeBench with Llama-3.2-3B models. The empirical results show consistent improvements over the evaluated baselines, and the paper provides extensive ablations on coefficient granularity, optimization objectives, basis construction, and coefficient constraints.
Critical strengths
-
Broad and practically meaningful empirical improvements. The proposed method consistently improves merging performance across multiple vision backbones and task counts, suggesting that component-wise reweighting can be practically useful in a wide range of merging settings.
-
Thorough ablation studies. The paper carefully analyzes several important design choices, including coefficient granularity, optimization objectives, optimization strategies, basis construction, calibration-set size, and coefficient constraints. These experiments help clarify which parts of the proposed method are important for its empirical performance.
Critical weaknesses
- The paper does not yet provide a sufficiently convincing mechanistic explanation for why component-wise optimization is specifically beneficial for model merging. It is plausible that a finer-grained parameterization improves the optimization objective simply because it introduces substantially more degrees of freedom. However, the current analysis does not clearly distinguish this generic increase in optimization capacity from a mechanism that is fundamentally specific to model merging or cross-task interference. Figure 4 provides useful supporting evidence by showing that different components can have different local effects on the source-task and other-task KL. However, these quantities are local gradients evaluated at . Components may have different gradient signs at initialization while still converging to similar scaling behavior after optimization, so this analysis alone does not establish that distinct component-wise coefficients are intrinsically necessary. Section 4.3 is also helpful, but it does not fully resolve this concern. The experiment fixes the mean coefficient within each layer-task group and allows redistribution among components. This controls for changes in the group-level mean scale, but it still leaves substantially more optimization degrees of freedom than the shared-coefficient baseline; for a group with components, the redistribution still introduces roughly additional degrees of freedom. Therefore, this is not a parameter-count- or capacity-matched comparison. I would like to see either stronger theoretical/mechanistic justification or a comparison against an alternative parameterization with the same number of optimization degrees of freedom as the proposed component-wise scheme.
- Important closely related baselines are missing. The paper itself identifies AdaRank and SWIM-Merging as closely related adaptive spectral merging methods, yet neither appears in the main experimental comparison.
- Robustness under distribution shift is not evaluated. Since the proposed method introduces many more optimization variables, it has greater capacity to fit the in-domain calibration distribution. This raises a concern that improved in-domain fitting may come at the cost of increased sensitivity to calibration-deployment distribution shift. The current experiments use task-specific calibration data drawn from the corresponding validation distributions, so it remains unclear how robust the learned coefficients are under distribution shift.
- The language-model experiments are based on a single run. Unlike the vision experiments, the language-model results are not evaluated over multiple random seeds. Since the proposed method does not outperform every baseline on every individual domain, it is difficult to assess whether the reported average improvements are statistically reliable.
Questions for authors
- Prior work has discussed failure modes of model merging under more challenging and practically relevant conditions [1]. How robust is the proposed method in such settings? In particular, what happens when the norms of the task vectors differ substantially across tasks, or when the expert predictions used for alignment have low confidence?
- In practical applications, task-specific input data may no longer be available at merging time. The current method assumes access to calibration inputs from each task/domain. How robust is the method when the coefficients are optimized using surrogate or proxy data instead of in-domain task-specific inputs?
[1] Kotaro Yoshida et al. “Distac: Conditioning Task Vectors via Distillation for Robust Model Merging.” International Conference on Learning Representations, 2026.
8 comments
Response to Reviewer Kyfj (W2)
W2. Comparison with AdaRank and SWIM-Merging
We thank the reviewer for the suggestion and have added this comparison to the revised manuscript.
The original submission did not include AdaRank and SWIM-Merging because both perform test-time adaptation on unlabeled test inputs, so their reported numbers are not directly comparable with a method that never sees test data. However, we agree that a comparison under a common protocol is informative.
We therefore evaluate both methods under our protocol, keeping their objective and reweighting unit and replacing test inputs with the same 256 unlabeled calibration inputs per task that our method uses (Table C). All other settings, including how the entropy is computed, follow the original papers. SWIM-Merging is applied to Task Arithmetic, one of the base merges used in the original paper.
Table C. Absolute accuracy on ViT-B/32 under our calibration protocol
| Method | 8 tasks | 14 tasks | 20 tasks |
|---|---|---|---|
| AdaRank | 83.5 | 78.2 | 73.2 |
| SWIM-Merging | 80.5 | 72.3 | 63.2 |
| Ours | 88.2 | 86.2 | 86.6 |
Our method outperforms both at every task count, and its margin is larger at 14 and 20 tasks than at 8 tasks.
Response to Reviewer Kyfj (W3)
W3. Robustness under calibration-deployment distribution shift
We agree that robustness under distribution shift was not evaluated, and we have now measured it. The coefficients are learned on clean calibration inputs and kept fixed, and only the test inputs are corrupted with the seven corruption types used by AdaMerging [1]. To isolate the effect of the number of variables, we compare coefficient units learned with the same objective and the same calibration inputs (Table D).
Table D. Absolute accuracy on ViT-B/32 with 8 tasks on clean and corrupted test inputs (average over the seven corruption types)
| Method | #Coeff. | Clean | Corrupted |
|---|---|---|---|
| TSV-M (no learning) | 0 | 82.42 | 59.33 |
| Layer-task group | 384 | 86.05 | 63.51 |
| Parameter chunk | 36,864 | 87.05 | 64.26 |
| Ours | 36,864 | 88.25 | 65.58 |
More variables do not make the method more sensitive. Our method still achieves the highest average accuracy on corrupted inputs. However, its accuracy still drops substantially on corrupted inputs, so the reviewer's concern is partly valid.
[1] Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. "AdaMerging: Adaptive Model Merging for Multi-Task Learning." International Conference on Learning Representations, 2024.
Response to Reviewer Kyfj (W1)
W1. Mechanism versus additional degrees of freedom
We agree that Figure 4 shows local effects at γ=1 and that the redistribution in Section 4.3 is not matched in capacity, and we have revised Section 4.2 to present Figure 4 as motivation rather than proof. We address the concern with three analyses that do not rely on either.
Comparison with the same number of coefficients. Every row of Table A uses the same objective, calibration inputs, optimization, and number of coefficients as ours, and differs only in where the coefficients act. Every alternative falls below ours, by 0.8 to 2.2 points at 8 tasks and by 2.0 to 4.3 points at 20 tasks, so the gain cannot be attributed to capacity.
Table A. Absolute accuracy on ViT-B/32 with the same number of coefficients, collected from Table 4 and Table 7 of the revised manuscript
The gain comes from reduced interference between tasks. Because each component is attributed to a source task, we can remove the components of the other tasks when a task is evaluated (Table B). With only its own components, our method is 0.6 points above TSV-M, but with all components it is 5.8 points above, because the other tasks' components cost TSV-M 5.8 points and our method only 0.6 points. The gain therefore comes from reducing the loss caused by the other tasks' components, which arises only when task updates are combined.
Table B. Average absolute accuracy on ViT-B/32 with eight tasks. Own task only removes the components of the other seven tasks from every reweighted layer, so the task is free of their interference. All tasks keeps every component, as in the actual merged model.
The learned coefficients remain distinct after optimization. Every coefficient starts at 1, and after optimization about one fifth are exactly zero while about 15% exceed twice the mean of their layer-task group, with at least 97% of the groups containing a zero. Randomly permuting the learned coefficients within each group keeps their values and every group mean but lowers accuracy on ViT-B/32 with 8 tasks from 88.2 to 75.0, below the 82.4 of TSV-M. The benefit therefore lies in which component receives which coefficient, which a shared coefficient cannot represent.