acceptodds
Withdrawn from ICLR 2027

Relevance Meets Spectrum: Joint Calibration For Model Merging

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Singular value decomposition (SVD) has recently emerged as a key technique in several strong methods for model merging. In particular, existing SVD-based methods organize task-related directional structures and spectra in different ways, with the general goal of preserving useful information from multiple task-specific updates. However, a single merged update generally cannot preserve the full directional structures of all task-specific updates simultaneously. Motivated by this issue, we prioritize task-relevant structure within the dominant low-rank representation of the merged update, rather than seeking complete directional alignment. This prioritization also makes spectral allocation important, since the singular values determine how strongly the retained directions contribute to the merged update. Based on these considerations, we propose RSS-Merge, a model-merging method that combines relevance-oriented directional construction with order-preserving spectrum calibration. Extensive experiments show that RSS-Merge achieves the best average accuracy across all nine evaluated vision settings. On the vision-language MM-MergeBench, it obtains the highest unseen-task average and ranks second on seen tasks, only marginally behind the best result.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Model Merging, singular value decomposition, multi-task learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_p5t7OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies SVD-based model merging, focusing on how to preserve task-relevant directional information while controlling the singular-value distribution of the merged update. The authors propose RSS-Merge, which consists of two main stages: (1) relevance construction, which aggregates truncated SVD components of centered task updates and incorporates the bilaterally projected average update to preserve complementary directions; and (2) order-preserving spectrum calibration, which smooths the singular values while keeping their associated singular vectors and ordering unchanged. The paper also introduces Direction-Band Alignment (DBA) to measure how well the leading singular subspaces capture task-update information.

The method is evaluated on vision and vision-language model merging benchmarks. RSS-Merge achieves the highest average accuracy among the compared methods across nine vision settings involving three CLIP backbones and different numbers of merged tasks. On MM-MergeBench, it reports the highest average accuracy on unseen tasks and the second-highest on seen tasks. Additional ablations examine relevance construction, spectral calibration, bilateral projection, and parameter sensitivity.

Critical strengths
  1. The paper provides a clear formulation for combining directional construction and spectral calibration in model merging. In particular, the bilateral projection in Eq. (2) ensures that the residual and complementary branches occupy orthogonal left and right subspaces, allowing their singular directions to remain unchanged after combination. This makes the construction mathematically well defined and provides a reasonable basis for separating directional selection from spectral adjustment.
  2. The experimental results are strong across different model merging settings. RSS-Merge achieves the highest average accuracy among the compared methods in all nine vision settings, covering three backbones and different numbers of merged tasks. The ablations in Tables 3 and 4 also examine the contributions of relevance construction, spectrum calibration, and bilateral projection. These results provide useful empirical support for the approach, even though the theoretical justification for its task-relevance claims remains incomplete.
Critical weaknesses
  1. Eq. (1) does not guarantee that task-relevant residual directions are preserved. The authors independently truncate the SVD of each centered task update and then sum the resulting matrices. However, the leading singular directions of different task residuals can be misaligned or have opposing contributions. Their sum may therefore cancel some retained components or rotate the resulting dominant subspace away from the individual task directions. More fundamentally, SVD truncation selects directions by matrix energy rather than task sensitivity. The paper does not establish why this procedure should produce a leading subspace that is representative of all tasks, which is the central motivation of RSS-Merge.
  2. The bilateral projection in Eq. (2) imposes a stronger restriction than the stated objective requires. The authors retain only the component of the average update that is orthogonal to both the left and right singular subspaces of the residual branch. This removes not only potentially redundant components but also average-update information that overlaps with the residual subspaces. Such overlapping information may reinforce useful task directions rather than interfere with them. Eq. (4) proves that the two branches preserve their singular directions after combination, but does not prove that this separation is beneficial for task performance. In particular, the method prioritizes preserving the residual geometry without establishing that the discarded overlap is less important than the retained orthogonal component.
  3. The spectrum calibration in Eq. (5) is not sufficiently connected to the proposed relevance objective. The calibrated singular values are defined as s_i' = (1 − γ)s_i + γs̄, with γ = 0.99 in the experiments. At this value, the method nearly flattens all nonzero singular values, substantially reducing the original differences in spectral strength. Although their ordering is mathematically preserved, the gaps between singular values are reduced to only 1% of their original values. Thus, order preservation alone is a weak justification for claiming that the leading-subspace priority is maintained in a meaningful spectral sense. Moreover, applying nearly uniform strength to all retained directions may amplify weak or noisy components. The paper provides no task-dependent criterion for deciding which singular values should be increased or decreased.
  4. The proposed DBA metric in Eq. (8) is insufficient to validate the claimed task relevance of the merged update. DBA separately measures the energy of each task update captured by the leading left and right singular subspaces. However, these marginal projections do not measure whether the corresponding left-right directional pairs preserve the actual task transformation. For example, a merged update may have left and right subspaces that individually overlap strongly with a task update while pairing those directions differently, resulting in a substantially different matrix operation. DBA can therefore remain high even when the merged update poorly approximates the task-specific update. Since the paper uses DBA as its main structural evidence for the leading-subspace principle, this distinction is important and requires a more direct validation of task-relevant directional preservation.
Questions for authors
  1. In Eq. (1), the leading singular components of each centered task update are extracted independently and then summed. How does the method handle cancellation or interference between residual directions from different tasks? Is there evidence that the dominant subspace of the aggregated residual actually preserves task-relevant directions, rather than simply directions with large singular values?
  2. Eq. (2) removes the components of the average update that overlap with the left and right residual subspaces. Can the authors quantify how much update energy is discarded by this bilateral projection? More importantly, how can they establish that the discarded components are redundant or harmful rather than useful shared task information? This would help explain why strict orthogonality is preferable to retaining the overlapping components.
  3. In Eq. (5), the authors set γ = 0.99, which reduces the differences between singular values to 1% of their original magnitude. Why is such aggressive spectrum flattening necessary? Does the method still preserve meaningful directional priority when the singular values become nearly identical? A comparison across a broader range of γ values would help determine whether the improvement comes from the proposed joint calibration or primarily from near-isotropic spectral scaling.
  4. Eq. (8) measures task-update energy captured by the leading left and right singular subspaces separately. However, this does not account for how the left and right singular directions are paired in the actual merged matrix. Can the authors explain why DBA is an appropriate measure of task relevance under this limitation? Would the conclusions remain the same when evaluating the reconstruction error of task updates or the preservation of their action on task-specific activations?
Reviewer_BqyVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RSS-Merge, which jointly constructs direction relevance and singular-value spectrum allocation to merge multiple task-specific model updates into one weight update. The authors first center each task update by subtracting the mean task update, then apply truncated SVD to the resulting task-specific residuals to synthesize a relevance-related residual branch. They then project the mean update on both sides into the complement orthogonal to the left and right singular subspaces of that branch, thereby preserving singular directions after combining the two branches. Finally, with a coefficient γ, they shrink nonzero singular values toward their mean while maintaining their relative ordering. The authors emphasize giving task-relevant directions priority in the leading low-rank subspace.

The experiments report the highest mean accuracy across nine vision settings with three CLIP backbones and 8/14/20 tasks. On MM-MergeBench using LLaVA-v1.5-7B LoRA, the method obtains 40.07% average accuracy on unseen tasks and 59.20% on seen tasks, ranking first and second, respectively. The authors support the method with DBA structural metrics, direction/spectrum component ablations, one-sided/two-sided projection ablations, and hyperparameter analyses.

Critical strengths
  1. The two-sided projection structure has a clear technical meaning and experimental support. Two-sided Frobenius projection makes the centered residual branch and the complementary mean branch orthogonal in both row and column subspaces, avoiding the direction rotations introduced by direct addition. In Table 4 (p.8), two-sided projection reaches 91.96%/87.31% on the 8/14-task ViT-B/16 settings, compared with 90.75%/85.27% without projection, an increase of 1.21/2.04 percentage points, respectively.

  2. The evaluation covers different task scales and a vision-language model. Table 1 reports the leading result in all nine vision settings; the unseen-task average in Table 2 is 40.07%, slightly above DC-Merge's 39.84%. Figure 1 and the appendix's DBA analyses attempt to explain geometric changes, so the paper shows awareness of mechanisms beyond a single table of scores.

Critical weaknesses
  1. The independent benefits of the new mechanism and its distinction from related work remain insufficiently established. The ideas of centered residuals and truncated SVD are related to CART, and spectrum shaping is close to methods such as Iso-CTS. In Table 3 (p.8), Task Arithmetic on the eight-task ViT-B/32 setting improves from 69.12% to 86.30% (+17.18) with spectrum calibration alone, whereas complete RSS-Merge reaches 88.76%, adding another 2.46 points over calibrated TA; the corresponding additional gains for 14/20 tasks are 4.05/6.01. This indicates that spectrum calibration is a major driver, but there is no quantitative report of the actual proportion of residual-branch directions among the leading k singular vectors. The DBA plots also omit Iso-CTS, which the authors themselves identify as a nearest related method. In addition, the SVC work by Li et al. [1] has already discussed excessive accumulation and calibration of shared singular values in model merging; the paper does not adequately locate its differences from this work. What needs to be demonstrated is not merely better results, but that the new structure captures a mechanism not addressed by existing spectral methods.

  2. The method is sensitive to p, and several performance leads lack uncertainty information. Figure 2 (p.9) shows that as the residual retention ratio for eight-task ViT-B/32 increases from p=1% to 10%, accuracy drops from approximately 88.8% to 64%. The best value lies at the boundary of the examined range, and tests of p<1% and cross-validation on larger backbones or vision-language settings are missing. γ defaults to 0.99, but there is no complete sensitivity curve. In Table 1, ViT-L/14 on eight tasks leads Iso-CTS by only 94.73% versus 94.71%, and ViT-B/16 on twenty tasks leads DC-Merge by only 84.43% versus 84.31%. The unseen-task average in Table 2 leads by only 0.23 percentage points. These nine vision settings share backbones and have nested task sets, so they cannot be treated as nine independent replications. The paper does not provide paired confidence intervals at the test-example level, validation-split stability, or a clear baseline hyperparameter-tuning budget. This makes it difficult to determine whether the small gains are robust.

[1] Li, Y., Peng, Z., Zhang, J., Guo, J., Duan, Y. & Shi, Y.. (2026). When Shared Knowledge Hurts: Spectral Over-Accumulation in Model Merging. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:69355-69375 Available from https://proceedings.mlr.press/v306/li26ep.html.

Questions for authors
  1. Please directly report the number or energy proportion of residual-branch directions among the leading k singular directions across layers, model backbones, and task scales, as well as the relative spectral values of λΣ_res and Σ_⊥; please also include Iso-CTS in the DBA structural comparison. If the claimed direction-priority mechanism can be demonstrated to occur in typical model layers, the novelty would be more convincing.
  2. Could the authors add sensitivity experiments across multiple settings for p<1%, p=1% to 10%, and γ from 0 to close to 1, and explain whether p=1% and γ=0.99 were fixed using an independent validation set? The current substantial performance decline makes generalization of these parameters important to the acceptance assessment.
Reviewer_SX36OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RSS-Merge combines truncated centered task residuals with a bilaterally projected mean update, followed by spectrum smoothing. It achieves the highest average accuracy among the evaluated methods across nine CLIP settings and competitive results on MM-MergeBench.

Critical strengths
  1. The vision results are consistent across three backbones and three task counts, providing useful evidence of the method’s practical value.
  2. The ablations support the individual components: relevance construction improves over calibrated TA, and bilateral projection outperforms directly adding the mean update.
Critical weaknesses
  1. The method builds closely on CART and Iso-CTS. While Table 4 supports bilateral projection empirically, I am not yet convinced that the resulting construction provides a sufficiently distinct methodological contribution.
  2. Hyperparameter selection is only partially specified, and closely related baselines are missing from the main accuracy comparisons. These gaps leave the claimed advantage and practical tuning requirements insufficiently established.
Questions for authors
  1. While the paper discusses its connections to CART and Iso-CTS, the core methodological novelty remains unclear. What is the central contribution beyond combining centered residual truncation, complementary subspace construction, and spectrum smoothing? What principled rationale or theoretical guidance motivates this combination?
  2. The method involves several hyperparameters, raising concerns about the tuning required for new architectures or tasks. How were the fixed defaults for p, λ, and γ determined? Can these defaults be reused, or what additional search is required?
  3. The main accuracy comparisons omit closely related methods such as CART[1], SVC[2], and ACE-Merging[3]. Could the authors provide comparisons using the same checkpoints and evaluation protocol?

[1] Revisiting Weight Averaging for Model Merging

[2] When Shared Knowledge Hurts: Spectral Over-Accumulation in Model Merging

[3] ACE-Merging: Data-Free Model Merging with Adaptive Covariance Estimation

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.