- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies SVD-based model merging, focusing on how to preserve task-relevant directional information while controlling the singular-value distribution of the merged update. The authors propose RSS-Merge, which consists of two main stages: (1) relevance construction, which aggregates truncated SVD components of centered task updates and incorporates the bilaterally projected average update to preserve complementary directions; and (2) order-preserving spectrum calibration, which smooths the singular values while keeping their associated singular vectors and ordering unchanged. The paper also introduces Direction-Band Alignment (DBA) to measure how well the leading singular subspaces capture task-update information.
The method is evaluated on vision and vision-language model merging benchmarks. RSS-Merge achieves the highest average accuracy among the compared methods across nine vision settings involving three CLIP backbones and different numbers of merged tasks. On MM-MergeBench, it reports the highest average accuracy on unseen tasks and the second-highest on seen tasks. Additional ablations examine relevance construction, spectral calibration, bilateral projection, and parameter sensitivity.
Critical strengths
- The paper provides a clear formulation for combining directional construction and spectral calibration in model merging. In particular, the bilateral projection in Eq. (2) ensures that the residual and complementary branches occupy orthogonal left and right subspaces, allowing their singular directions to remain unchanged after combination. This makes the construction mathematically well defined and provides a reasonable basis for separating directional selection from spectral adjustment.
- The experimental results are strong across different model merging settings. RSS-Merge achieves the highest average accuracy among the compared methods in all nine vision settings, covering three backbones and different numbers of merged tasks. The ablations in Tables 3 and 4 also examine the contributions of relevance construction, spectrum calibration, and bilateral projection. These results provide useful empirical support for the approach, even though the theoretical justification for its task-relevance claims remains incomplete.
Critical weaknesses
- Eq. (1) does not guarantee that task-relevant residual directions are preserved. The authors independently truncate the SVD of each centered task update and then sum the resulting matrices. However, the leading singular directions of different task residuals can be misaligned or have opposing contributions. Their sum may therefore cancel some retained components or rotate the resulting dominant subspace away from the individual task directions. More fundamentally, SVD truncation selects directions by matrix energy rather than task sensitivity. The paper does not establish why this procedure should produce a leading subspace that is representative of all tasks, which is the central motivation of RSS-Merge.
- The bilateral projection in Eq. (2) imposes a stronger restriction than the stated objective requires. The authors retain only the component of the average update that is orthogonal to both the left and right singular subspaces of the residual branch. This removes not only potentially redundant components but also average-update information that overlaps with the residual subspaces. Such overlapping information may reinforce useful task directions rather than interfere with them. Eq. (4) proves that the two branches preserve their singular directions after combination, but does not prove that this separation is beneficial for task performance. In particular, the method prioritizes preserving the residual geometry without establishing that the discarded overlap is less important than the retained orthogonal component.
- The spectrum calibration in Eq. (5) is not sufficiently connected to the proposed relevance objective. The calibrated singular values are defined as s_i' = (1 − γ)s_i + γs̄, with γ = 0.99 in the experiments. At this value, the method nearly flattens all nonzero singular values, substantially reducing the original differences in spectral strength. Although their ordering is mathematically preserved, the gaps between singular values are reduced to only 1% of their original values. Thus, order preservation alone is a weak justification for claiming that the leading-subspace priority is maintained in a meaningful spectral sense. Moreover, applying nearly uniform strength to all retained directions may amplify weak or noisy components. The paper provides no task-dependent criterion for deciding which singular values should be increased or decreased.
- The proposed DBA metric in Eq. (8) is insufficient to validate the claimed task relevance of the merged update. DBA separately measures the energy of each task update captured by the leading left and right singular subspaces. However, these marginal projections do not measure whether the corresponding left-right directional pairs preserve the actual task transformation. For example, a merged update may have left and right subspaces that individually overlap strongly with a task update while pairing those directions differently, resulting in a substantially different matrix operation. DBA can therefore remain high even when the merged update poorly approximates the task-specific update. Since the paper uses DBA as its main structural evidence for the leading-subspace principle, this distinction is important and requires a more direct validation of task-relevant directional preservation.
Questions for authors
- In Eq. (1), the leading singular components of each centered task update are extracted independently and then summed. How does the method handle cancellation or interference between residual directions from different tasks? Is there evidence that the dominant subspace of the aggregated residual actually preserves task-relevant directions, rather than simply directions with large singular values?
- Eq. (2) removes the components of the average update that overlap with the left and right residual subspaces. Can the authors quantify how much update energy is discarded by this bilateral projection? More importantly, how can they establish that the discarded components are redundant or harmful rather than useful shared task information? This would help explain why strict orthogonality is preferable to retaining the overlapping components.
- In Eq. (5), the authors set γ = 0.99, which reduces the differences between singular values to 1% of their original magnitude. Why is such aggressive spectrum flattening necessary? Does the method still preserve meaningful directional priority when the singular values become nearly identical? A comparison across a broader range of γ values would help determine whether the improvement comes from the proposed joint calibration or primarily from near-isotropic spectral scaling.
- Eq. (8) measures task-update energy captured by the leading left and right singular subspaces separately. However, this does not account for how the left and right singular directions are paired in the actual merged matrix. Can the authors explain why DBA is an appropriate measure of task relevance under this limitation? Would the conclusions remain the same when evaluating the reconstruction error of task updates or the preservation of their action on task-specific activations?