Task Singular Vectors go Functional: Model Merging in the Data Metric

Spectral model merging compresses task vectors and reduces interference by orthogonalizing their retained directions. Existing methods make both decisions in the Euclidean geometry of weight space, however, directions that are orthogonal in parameter space can still overlap in the activations the model actually sees. We derive a joint least-squares perspective that unifies several existing merging methods in a common objective, and use it to replace Euclidean geometry with a data-induced metric estimated from a few unlabeled samples. Functional TSV (F-TSV) utilizes this metric to choose each task’s low-rank subspace and to define cross-task orthogonality, while preserving the expert update within the chosen subspace. Our derivation predicts a square-root metric weighting, and validation sweeps across vision, GLUE, and generative merging select the same setting. Ablations reveal the main empirical effect: data-aware cross-task orthogonalization accounts for almost all of the gains in vision, whereas data-aware subspace selection contributes minimally outside GLUE. F-TSV is training-free, relies only on label-free metric, and produces a single checkpoint with no inference-time overhead, and it consistently improves over its Euclidean counterpart across all reported benchmarks.