Revisiting Singular Structure in Data-Free Model Merging

Recent data-free model merging methods increasingly exploit the singular structure of task updates, yet they often couple singular-vector support and singular-value weighting within a single spectral operator. We revisit this design choice and study how different components of the singular structure should actually be used for merging. Under a common covariance-weighted framework, we disentangle task support, within-support anisotropy, and task-layer scale. Our analysis shows that right singular vectors provide a useful task-relevant support, while singular values are more effective when compressed into a scalar task-layer scale than when used for direction-wise reweighting. In particular, a scale-only inverse spectral response consistently outperforms its anisotropic counterpart under matched support and total scale. Motivated by this finding, we propose SOS-Merge, which represents each task operator as a right-singular projector scaled by a fractional inverse spectral moment. We further develop a data-free calibration rule for the shared spectral exponent, achieving performance close to post-hoc grid search without validation data. Across three CLIP ViT backbones and 8-, 14-, and 20-task merging settings, SOS-Merge consistently outperforms recent data-free baselines, with its advantage becoming more pronounced as more tasks are merged.