Relevance Meets Spectrum: Joint Calibration For Model Merging

Singular value decomposition (SVD) has recently emerged as a key technique in several strong methods for model merging. In particular, existing SVD-based methods organize task-related directional structures and spectra in different ways, with the general goal of preserving useful information from multiple task-specific updates. However, a single merged update generally cannot preserve the full directional structures of all task-specific updates simultaneously. Motivated by this issue, we prioritize task-relevant structure within the dominant low-rank representation of the merged update, rather than seeking complete directional alignment. This prioritization also makes spectral allocation important, since the singular values determine how strongly the retained directions contribute to the merged update. Based on these considerations, we propose RSS-Merge, a model-merging method that combines relevance-oriented directional construction with order-preserving spectrum calibration. Extensive experiments show that RSS-Merge achieves the best average accuracy across all nine evaluated vision settings. On the vision-language MM-MergeBench, it obtains the highest unseen-task average and ranks second on seen tasks, only marginally behind the best result.