acceptodds
Under review as a conference paper at ICLR 2027

Task Singular Vectors go Functional: Model Merging in the Data Metric

Abstract

Spectral model merging compresses task vectors and reduces interference by orthogonalizing their retained directions. Existing methods make both decisions in the Euclidean geometry of weight space, however, directions that are orthogonal in parameter space can still overlap in the activations the model actually sees. We derive a joint least-squares perspective that unifies several existing merging methods in a common objective, and use it to replace Euclidean geometry with a data-induced metric estimated from a few unlabeled samples. Functional TSV (F-TSV) utilizes this metric to choose each task’s low-rank subspace and to define cross-task orthogonality, while preserving the expert update within the chosen subspace. Our derivation predicts a square-root metric weighting, and validation sweeps across vision, GLUE, and generative merging select the same setting. Ablations reveal the main empirical effect: data-aware cross-task orthogonalization accounts for almost all of the gains in vision, whereas data-aware subspace selection contributes minimally outside GLUE. F-TSV is training-free, relies only on label-free metric, and produces a single checkpoint with no inference-time overhead, and it consistently improves over its Euclidean counterpart across all reported benchmarks.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Model Merging, Representation Alignment, Task Singular Vectors, SVD, Orthogonalization, Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_zRTGOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

F-TSV replaces the Euclidean geometry of spectral model merging with a metric estimated from unlabeled activations. The metric guides subspace selection and cross-task orthogonalization; a two-sided extension additionally estimates downstream sensitivity. Experiments cover three CLIP backbones, RoBERTa on GLUE and Llama-3.2-3B on MergeBench. The central empirical result is that pooled-metric orthogonalization contributes substantially more than metric-guided selection, particularly on vision tasks.

Critical strengths

The paper separates two operations that are often conflated in spectral merging. Across nine vision settings, the reported increments are +0.11 normalized points for selection and +4.16 for subsequent orthogonalization. Exact reductions to TSV-M provide a useful implementation control, while repeated statistics draws on ViT-B/32 with eight tasks and on GLUE support the reliability of the gains in those settings. The discussion is appropriately more restrained about generative merging.

Critical weaknesses
  1. The empirical justification for the particular spectral block remains incomplete. Section 4 explicitly states that F-TSV does not minimize the joint least-squares objective, and Section 4.1 acknowledges that Eq. 6 is weakly suboptimal for the isolated functional approximation problem. The proposed reason for using it is that preserving the selected input subspace benefits the later orthogonalization. That is a testable design claim, but comparing against ESM does not isolate it because ESM also changes the orthogonalization. Replacing only Eq. 6 with the stated exact left-projection block, while retaining the same pooled-metric orthogonalization, would directly test the justification. Proposition 2 establishes the square-root power for the orthogonalization step; it does not settle this block choice.

  2. MaTS is a consequential omission. It appears in the paper's curvature framework but not in the experiments. On GLUE, F-TSV's advantage over RegMean is 2.4 normalized points, much smaller than its advantage over TSV-M. A comparison under matched data and curvature assumptions, with label access and computational costs stated explicitly, would clarify whether the spectral construction offers an advantage beyond the choice of metric.

  3. The two-sided extension has limited practical support. Across five paired statistics draws with the coefficient pinned, its gain is 0.15±0.05 normalized points on ViT-B/32 with eight tasks and 0.13±0.76 on GLUE. The paper acknowledges this and reports some statistics-collection and matrix-merging costs, but does not isolate the incremental time and memory required to collect G alongside these gains. Possible error in the K-FAC factorization remains an undiagnosed explanation, not an established cause.

  4. MergeBench provides only modest support for the mechanism's transfer. The selected coefficient changes from 0.8 to 1.8 between the Euclidean and functional methods, and the average improvement is 0.35 points, with coding declining from 40.41 to 36.46. Selecting each method's coefficient on validation data is legitimate, but those rows do not isolate the effect of geometry. A common-coefficient diagnostic or a coefficient-response curve would clarify the attribution; it should supplement, not replace, the main selection protocol.

Questions for authors
  1. In Appendix E's ViT-B/32 eight-task sweep, validation favors r=32 only marginally: r=32–512 lie within 0.15 points. How was the default r=128 chosen?

  2. Can you provide a matched MaTS comparison on GLUE, specifying the data, label access and statistics-collection costs?

7 comments
AuthorsOpenReview ↗

Answer to W1 - Choice Eq. 6

We thank the reviewer for recognizing the clear distinction between subspace selection and orthogonalization, the rigorous experimental controls and reliability analyses, and the appropriate interpretation of the results.

We also thank the reviewer again for identifying this complementary ablation that reinforces our method. We agree, as stated in §5, that Eq. 6 does not minimize the isolated functional approximation error, still, the following experiments favor our choice. Proposition 2 fixes the power of the orthogonalization constraint but does not establish the optimality of the selected block. The comparison with ESM [1] alone cannot isolate this choice, as ESM also changes other components.

Following the reviewer's suggestion, we replaced only Eq. 6 by the exact block minimizer: U~(k)U~(k)⊤Δt\tilde U^{(k)}\tilde U^{(k)\top}\Delta_t and kept everything else fixed: statistics, conditioning, p=12p=\tfrac12, rank budget, and the pooled-metric polar orthogonalization of Eq. 8. Repeating the comparison with Euclidean orthogonalization yields a 2×2 ablation: block × orthogonalization.

The table reports normalized scores (MergeBench average), with gaps in parentheses measuring Eq. 6 minus the exact block.

Euclidean orthogonalization: Eq. 6 / exact (gap)pooled-metric orthogonalization (ours): Eq. 6 / exact (gap)
Vision (mean of 9 cells)91.20 / 90.90 (+0.30)95.36 / 95.27 (+0.09)
GLUE80.42 / 75.66 (+4.76)87.86 / 84.95 (+2.91)
MergeBench38.60 / 39.14 (−0.54)39.80 / 39.32 (+0.48)

The results support our choice of Eq. 6, under pooled-metric orthogonalization, the exact minimizer does not outperform Eq. 6 on average in any benchmark. On vision, the blocks are tied (+0.09 normalized points), with Eq. 6 ahead in 7/9 settings and differences ranging from −0.12 to +0.35. On GLUE, Eq. 6 is 2.91 points better, while on MergeBench it is better by 0.48.

Importantly, pooled-metric orthogonalization consistently improves performance regardless of the block: +4.16/+4.37 on vision and +7.44/+9.29 on GLUE (Eq. 6/exact), with gains in every vision setting. The smaller block advantage under pooled-metric orthogonalization is consistent with reduced room for improvement.

The exact block minimizes the functional reconstruction error for each expert independently, whereas Eq. 6 also preserves the selected input subspace before cross-task orthogonalization. Our results support our choice of Eq. 6 as an empirically effective block, particularly on GLUE, without establishing that preserving the selected input subspace is the sole cause of its advantage. We think §4.1 remains largely valid and needs only minor changes, along with the addition of these experimental tests (in the Appendix) that confirm our choice. In conclusion, we have seen that the theoretically optimal approximation is not necessarily the empirically preferable building block for model merging, and we think this is a useful finding in its own right.

Section 2 connects ESM to the exact block followed by Euclidean orthogonalization. However, this ablation cell is at most +0.01 above TSV-M [2] on vision (mean −0.19), whereas ESM gains +0.54 to +2.38 above it. Hence, ESM's gains cannot be attributed to the block alone; its other components (expert-side statistics, rank allocation, reweighting, spectrum flattening, and layer rescaling) also differ. We present this as a structural connection, not a complete derivation or causal attribution of ESM's gains.

Minor revisions: we will clarify in §4.1 that neither Eq. 2 nor Proposition 2 determines the block choice, retain its motivation with appropriate qualifications, and add the full ablation and analysis of the remaining performance gap to the Appendix.

AuthorsOpenReview ↗

Answer to W2 - MaTS & RegMean

We thank the reviewer for highlighting these related works. MaTS [3] is added as an additional baseline. F-TSV remains the best-performing data-aware method across all benchmarks. We implemented the method following Tam et al. (Sec. 6.2), using RegMean’s [4] objective per linear layer, with the reference code's stopping rule.

Vision absolute accuracy and normalized accuracy in parentheses:

B/32 T=8B/32 T=14B/32 T=20B/16 T=8B/16 T=14B/16 T=20L/14 T=8L/14 T=14L/14 T=20
RegMean82.5 (91.0)77.2 (86.5)75.4 (84.3)85.3 (92.0)80.6 (87.8)78.0 (84.7)91.0 (96.5)88.1 (94.1)85.6 (90.9)
MaTS83.1 (91.7)77.8 (87.1)76.1 (85.1)86.2 (93.0)81.1 (88.4)78.9 (85.7)91.3 (96.8)88.4 (94.5)86.3 (91.7)
F-TSV86.8 (96.0)83.6 (93.7)82.2 (91.8)89.1 (96.4)85.6 (93.6)85.0 (92.3)93.2 (98.9)91.5 (97.9)91.9 (97.7)

MaTS consistently improves on RegMean (+0.3 to +0.9 absolute, +0.3 to +1.0 normalized), but F-TSV outperforms MaTS by 1.9–6.2 absolute (2.1–6.7 normalized) points, with larger gains as task count increases.

GLUE under matched data, curvature and cost. In its full fine-tuning recipe, MaTS shares the curvature form of RegMean and F-TSV, Ht=Ct⊗IH_t = C_t \otimes I, so the comparison can be made exactly: each method also runs on the other's CtC_t, and within a setting all methods use the same statistics.

CtC_t computed fromExamples/task (↓)Models run (↓)Labels (↓)Covariances collection (↓)
pretrained encoder (F-TSV style)5121 (pretrained)none8.1 ± 0.2 s
each expert on its own task (RegMean style)10248 (one expert per task)none10.5 ± 0.2 s

Collection is one forward pass over the listed examples; its time is wall-clock for all 8 tasks on one GPU, mean ± std over 10 runs; each statistic stores 567 MiB per task, and none uses labels. The F-TSV setting therefore needs half the examples, a single model (the pretrained encoder, where RegMean and MaTS must run every expert) and less collection time. The merge step, timed the same way with the statistics in memory, is where F-TSV pays: 44.0 ± 0.9 s, against 2.2 ± 0.4 s for RegMean and 20.5 ± 0.4 s for MaTS on their own statistics.

CtC_t computed fromF-TSVRegMeanMaTS
pretrained encoder (F-TSV style)73.1 (87.9)57.3 (66.2)57.0 (65.9)
each expert on its own task (RegMean style)63.2 (76.5)71.6 (85.5)71.7 (85.5)

GLUE average, absolute (normalized); every method selects α on the same labeled validation split. As expected, each method is strongest in the setting it is derived for: RegMean and MaTS match every expert's layer outputs on that expert's own inputs, while F-TSV measures the update at the pretrained model's inputs. Each in its own setting, F-TSV leads RegMean and MaTS by +1.4–1.5 absolute and +2.4 normalized points, while using the cheaper setting of the first table.

  1. In F-TSV’s setting, the spectral construction is the difference. On F-TSV's statistics RegMean and MaTS fall to Task Arithmetic's level (57.1). Per-task subspace selection and decorrelation in Cˉ\bar C turn the same matrices into +21.6 normalized points over RegMean.
  2. RegMean and MaTS need expert-side statistics, i.e. a forward pass through every expert and twice the examples: the experts' Gram matrices diverge with depth, and their weighting relies on that difference. With them, they reach 85.5; F-TSV on its own statistics remains 2.4 normalized points ahead while using half the examples and only the pretrained encoder.
  3. The converse holds. On expert-side statistics, F-TSV drops to 76.5, mostly due to STS-B (Spearman 2.3), without STS-B, it follows RegMean and MaTS by 1.1 and 1.4 points, respectively, so the drop in performance, in the competitor setting, is mostly attributable to a single failure statistic, not to a generalized behavior, showing great adaptability.
AuthorsOpenReview ↗

Answer to W3 - F-TSV gains

We thank the reviewer for raising this point. We want to point reviewer zRTG to read response W4 of Reviewer Xa2q, since they partially overlap; we summarize the essential parts of our response there and then add the two items it does not cover: memory and the K-FAC explanation.

  • We agree that the gain of F-TSV$^2$ over F-TSV is small: Appendix A reports the paired numbers you quote, and the paper does not rest on F-TSV$^2$.
  • G costs more FLOPs than C on the encoders and consequently takes longer wall-clock time, for example, 1.4 min in ViT-B/32 with 8 tasks on GLUE (see the table in W4 answer of Reviewer Xa2q for a full breakdown). For LLMs, it costs at most 1.5× the FLOPs.
  • G helps most on tasks where the one-sided merge recovers poorly: +1.35 points on average on vision tasks (best in small- to mid-sized models). With a coefficient nearly the same as F-TSV, it also improves all four safety benchmarks on both LLMs, without refusing more safe prompts.
  • F-TSV therefore remains the default, and F-TSV² is an option when compute resources are not limited and the last points matter.

Memory. The runs recorded the peak GPU memory of each collection and the size of each stored statistic. Each cell is F-TSV vs F-TSV$^2$:

ModelPeak GPU memory to collect
ViT-B/32 (T = 8)1.3 / 5.9 GiB
ViT-B/16 (T = 8)2.2 / 19.4 GiB
ViT-L/14 (T = 8)5.2 / 66.5 GiB
RoBERTa-base (GLUE)1.6 / 11.4 GiB
Llama-3.2-3B (MergeBench)30.9 / 62.9 GiB

On MergeBench, F-TSV's statistics include its token-embedding metric (the biggest matrix: 128,256 × 3,072). The paired gains thus cost, on ViT-B/32 (T = 8), 1.4 minutes, a 5.9 instead of 1.3 GiB GPU peak for +0.15 ± 0.05 points, on GLUE, 1.4 minutes, 11.4 instead of 1.6 GiB for +0.13 ± 0.76. G's peak comes from keeping one forward pass's activations for all the backward passes of its probes, so it scales with the batch size and can be reduced with smaller batches.

K-FAC. Following the reviewer's suggestion, we measured it: on ViT-B/32 (T = 8) and GLUE, for the rank-one components the F-TSV$^2$ merge keeps in every layer, we computed their exact functional inner products and compared each metric with them. Against a measurement ceiling of 0.98–0.99, the K-FAC metric reproduces the exact cross-task geometry at r = 0.30 (vision) and 0.44 (GLUE), F-TSV's input-only metric at 0.24 and 0.28, the Euclidean at 0.10 for both. Raw and resampled statistics give the same r, so the loss is the factorization itself, mainly the assumed independence of inputs and downstream Jacobians (dropping only the cross-token terms keeps 0.62 and 0.77), hinting at what we described for the answer W2 to reviewer KfWN. Accordingly, K-FAC predicts that G lowers the merge's exact functional error in 58% and 74% of (task, layer) pairs, but it does in 26% and 55%, predicting a larger benefit from G than the real benefit. Probe noise is not a factor: 4× the probes leave the paired gain unchanged, more probes do not change the gain (32 probes +0.15 ± 0.04 vs 128 probes +0.15 ± 0.05 on vision; +0.04 ± 0.73 vs +0.13 ± 0.76 on GLUE). The revision will report this in Appendix A as a measured, contributing cause.

Reviewer_Xa2qOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

F-TSV addresses the limitation of parameter-space orthogonality by introducing a data-induced metric estimated from unlabeled samples to better capture functional interference between tasks. It unifies existing merging methods under a joint least-squares framework and uses this metric for task subspace selection and cross-task orthogonalization without additional training.

Critical strengths
  1. The paper is clearly written and well organized. The motivation, theoretical formulation, and overall methodology are easy to follow.

  2. The paper provides a solid theoretical foundation and comprehensive ablation studies.

Critical weaknesses
  1. The motivation of this paper overlaps to some extent with existing studies. For example, PAVE [1] and ESM [2] have already explored using data distributions or activation information to improve spectral decomposition in model merging. The authors are encouraged to clarify the differences from these works and better highlight the novelty of data-aware cross-task orthogonalization.

  2. The ablation studies show that data-aware subspace selection contributes little to vision tasks and even negatively affects generative tasks, while most improvements come from cross-task orthogonalization. The authors are encouraged to further explain the necessity of the subspace selection step and more clearly report the results of using data-aware orthogonalization alone.

  3. The improvements on LLM merging are relatively limited, with the average MergeBench score increasing only from 39.45 to 39.80. Moreover, improvements in mathematical reasoning come at the expense of coding and multilingual performance. Further analysis of these trade-offs and experiments on more LLMs would help demonstrate the effectiveness of the method.

  4. F-TSV2 further considers downstream functional changes by introducing Jacobian-based sensitivity estimation. However, its improvements over F-TSV are relatively limited and vary across tasks. It would be helpful to further analyze the scenarios where the two-sided metric is more effective, as well as the trade-off between additional computational cost and performance gains.

  5. Although the method is training-free and only requires unlabeled data for estimating statistics, the merging process still requires access to task-specific input data. In addition, the vision and GLUE experiments use labeled validation sets to select the merging coefficient. The authors are encouraged to further discuss how these data requirements may affect the practical applicability of the method.

References

[1] Purifying Task Vectors in Knowledge-Aware Subspace for Model Merging.

[2] Model Merging in the Essential Subspace.

Questions for authors

See Weaknesses.

9 comments
AuthorsOpenReview ↗

Answer to W1 - Relation to PAVE and ESM

We thank the reviewer for their positive assessment of the paper’s clarity and organization, as well as for recognizing the solid theoretical foundation and comprehensive ablation studies that support our methodology.

We clarify here the relationship and comparison between our method, ESM [4] and PAVE [5]. We agree that ESM and PAVE already use activation statistics for spectral decompositions. Our novelty lies in using a pooled data-induced metric to define and enforce cross-task orthogonality, rather than merely selecting task-relevant directions.

PAVE is presented as a plug-in rather than a standard merging method: it independently purifies each expert through covariance-weighted, near-full-rank truncation before applying an existing merger, without cross-task orthogonalization. Our whitening-power family offers a way to read it as a merger of its own: its ranking rule corresponds to the exponent p=1p = 1, which, when used throughout our framework (see Q4 table), this exponent is 0.95 normalized points below the derived value p=1/2p = 1/2 on vision (ViT-B/32, T = 8) and 5.34 below on GLUE.

ESM computes the optimal rank-kk approximation of activation shifts (Eckart–Young), but subsequently applies Euclidean polar orthogonalization with activation-derived importance weights. In contrast, F-TSV selects directions in the input space (Eq. 6), and the cross-task step orthogonalizes them, so the same statistics that select them also measure their overlap.

Thus, PAVE and ESM retain Euclidean cross-task orthogonality (v⊤v′=0v^\top v'=0), whereas F-TSV enforces pooled-metric orthogonality (v⊤Cˉv′=0v^\top\bar C v'=0, Eq. 8), separating directions by their responses to the input distribution. This also determines the exponent p=1/2p=1/2 (Proposition 2).

The following ablation isolates per-task selection (rows) from cross-task orthogonalization (columns). Scores are normalized averages across 9 vision settings / GLUE:

Per-task blockEuclidean orthogonalizationPooled-metric orthogonalization (ours)
Euclidean top-k (TSV-M)91.09 / 78.0195.28 / 85.37
Data-aware minimizer (ESM-projection type)90.90 / 75.6695.27 / 84.95
Data-aware, Eq. 6 (F-TSV)91.20 / 80.4295.36 / 87.86

Note that the ESM-style row isolates its per-task projection and should not be confused with the complete ESM algorithm, which omits the rescaling steps and has scores of 92.51 / 66.56.

The results demonstrate that changing the cross-task orthogonality metric is the dominant improvement. Applying pooled-metric orthogonalization to unchanged TSV-M selection gains +4.19 points on vision and +7.36 on GLUE, versus only +0.11 / +2.41 from changing selection alone. The ESM-style row further demonstrates that our orthogonalization complements existing data-aware decompositions.

Overall, F-TSV outperforms complete ESM across all nine vision settings (by 1.3–4.0 normalized points) and on GLUE (87.86 vs. 66.56). These results establish that data-aware selection and data-aware interference removal are distinct mechanisms, with the latter driving most of F-TSV's gains. We will emphasize this distinction in the manuscript. Further PAVE comparisons appear in our response to reviewer KfWN, Weakness W1.

AuthorsOpenReview ↗

Answer to W2 (part 1) - Subspace selection

We thank the reviewer for highlighting this crucial point. We agree that data-aware cross-task orthogonalization drives nearly all vision improvement, while selection contributes little, but we want to remark that the paper's claim is which geometry should govern each step, not whether the two steps are necessary. We clarify the distinction between rank truncation and data-aware selection, report a complete factorial ablation, examine the retained subspaces, and identify where selection helps.

We first distinguish rank truncation from data-aware selection, TSV [2], allocates k=⌊min⁡(m,n)/T⌋k=\lfloor\min(m,n)/T\rfloor directions per task to fit a shared orthogonal basis. These can be chosen using either Euclidean or our data-aware criteria:

Step 1 - Selection (per task)Step 2 - Orthogonalization (across task)
PurposeRetain kk directions per expert.Remove cross-tasks overlap.
EuclideanSelect largest singular directions of Δt\Delta_t.Orthogonalize directions in weight space.
Data-awareSelect singular directions of ΔtCt1/2\Delta_t C_t^{1/2}, preserving the expert's action on the selected input subspace.Orthogonalize directions under the pooled metric Cˉ\bar C, eliminating their data-weighted overlap.
PreservationPreserve the expert update on the selected subspace.Preserve the retained singular values s^\hat{s}.

Thus, data-aware orthogonalization alone uses Euclidean selection followed by orthogonalization under Cˉ\bar C.

Example 1: Selection. Suppose k=1k=1 and two singular directions a,ba,b produce output changes 1.0,0.61.0,0.6, with input standard deviations 0.3,1.50.3,1.5, respectively. Euclidean selection retains aa (1.0>0.61.0>0.6), whereas data-aware selection retains bb (0.6×1.5=0.90>1.0×0.3=0.300.6\times1.5=0.90> 1.0\times0.3=0.30). Importantly, Eq. 6 preserves the expert's action on the selected input subspace rather than scaling its retained content by the metric. As acknowledged in §4.1, this is not the exact minimizer of isolated functional approximation; its usefulness also depends on subsequent orthogonalization.

Example 2: Orthogonalization. Consider retained directions at 10∘10^\circ and 60∘60^\circ, with pooled input variances 1.81.8 and 0.20.2 along 30∘30^\circ and 120∘120^\circ. Their normalized data-weighted overlap is ≈0.95\approx0.95. Euclidean orthogonalization produces directions at −10∘-10^\circ and 80∘80^\circ, yet their data-weighted overlap remains ≈0.80\approx0.80. Instead, metric orthogonalization, (V⊤CˉV)−1/2V⊤(V^\top\bar C V)^{-1/2}V^\top, produces directions ≈143∘\approx143^\circ apart (Euclidean lengths 1.58,1.751.58,1.75) with zero data-weighted overlap. Thus, Euclidean orthogonality need not eliminate interference on the model's inputs.

Complete ablation

To address the reviewer's request, we report all four combinations of selection and orthogonalization, keeping the same rank budget, data statistics, and evaluation protocol.

SelectionOrthogonalizationVisionGLUEGenerative
EuclideanEuclidean (TSV-M)91.0978.0139.45
DataEuclidean91.2080.4238.60
EuclideanData (orthogonalization alone)95.2885.3736.72
DataData (F-TSV)95.3687.8639.80
Additional gain from data selection with data orthogonalization+0.08+2.49+3.08

Vision. Data-aware orthogonalization alone improves normalized accuracy from 91.09 to 95.28 (+4.19). Adding data-aware selection yields only another +0.08, with mixed effects across configurations (Fig. 2). Thus, selection is not necessary for the observed vision gains. The dominant mechanism is cross-task orthogonalization in the data metric.

GLUE. Data-aware orthogonalization alone improves normalized accuracy from 78.01 to 85.37 (+7.36). Data-aware selection increases it to 87.86 (+2.49), demonstrating clear empirical evidence from the selected subspaces.

Generative merging. Data-aware orthogonalization with Euclidean selection scores 36.72, below TSV-M's 39.45, whereas complete F-TSV reaches 39.80. Importantly, the +3.08 difference is confounded by independently selected merging coefficients: α=1.2\alpha=1.2 for Euclidean selection and α=1.8\alpha=1.8 for F-TSV. At the common coefficient α=1.2\alpha=1.2, F-TSV instead scores 36.03, compared with 36.72 for Euclidean selection. Consequently, we do not interpret the +3.08 as a benefit of data-aware selection alone.

AuthorsOpenReview ↗

Answer to W2 (part 2) - Subspace selection

What data-aware selection changes in real layers

Data-aware selection retains directions with higher pooled input variance. Across nine vision configurations, their median normalized variance is 2.1–4.0, versus 1.4–1.8 under Euclidean selection; on GLUE, the values are 0.94 vs. 0.71. The merged update's data-space norm relative to TSV-M is also different: 0.49–0.58 for F-TSV versus 0.52–0.68 with Euclidean selection and data-aware orthogonalization. F-TSV consequently selects a larger validation-tuned α\alpha in every vision and GLUE setting. Thus, the retained subspaces and merge normalization differ even when final accuracies are similar, although these differences do not independently establish better task performance.

Why does selection help more on GLUE? In vision, at most 6% of Euclidean-selected directions have below-average pooled input variance (median across layers and nine configurations). On GLUE, this fraction is 79%, dropping to 56% with data-aware selection. Euclidean selection therefore already favors data-excited directions in vision, leaving less room for improvement. On GLUE, data-aware selection changes the retained subspace more substantially, consistent with its larger gains. Nevertheless, input variance alone does not establish downstream task relevance.

Conclusion and revisions

We agree with the reviewer that data-aware orthogonalization is the primary empirical contribution. The ablation establishes that the orthogonalization step alone recovers nearly all of F-TSV's vision performance and most of its GLUE improvement. We retain data-aware selection not because it is universally necessary, but because it provides a functionally motivated criterion for choosing the required low-rank subspaces, consistent with the geometry used in orthogonalization, without requiring additional statistics collection. Its empirical contribution is substantial on GLUE (+2.49), marginal on vision (+0.08), and interlaced with coefficient selection tuning on the generative benchmark.

We will revise the manuscript to (i) include the complete ablation in §5 (Q2), (ii) explicitly distinguish the dominant contribution of orthogonalization from the benchmark-dependent contribution of selection, and (iii) add the geometric examples and subspace diagnostics to Appendix J.

Reviewer_KfWNOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes F-TSV, a data-aware extension of Task Singular Vector merging for experts fine-tuned from a shared base model. It estimates layerwise input second moments from unlabeled calibration examples and uses these statistics in two places: selecting a low-rank subspace for each task update, and orthogonalizing the retained directions across tasks under a pooled data metric. The resulting metric-polar construction motivates a square-root weighting of the covariance. An optional two-sided extension, F-TSV^2, also incorporates downstream representation sensitivity estimated through Jacobian probes. The merged update is scaled using a validation-selected coefficient.

Experiments cover CLIP encoders, RoBERTa on GLUE, and Llama-3.2-3B on MergeBench. The main improvements appear on the encoder benchmarks. The ablations attribute most of the vision gains to the pooled-metric orthogonalization stage, with a smaller contribution from data-aware subspace selection. The generative results show a modest average improvement and tradeoffs between capabilities.

Critical strengths
  1. The motivation for changing the orthogonality criterion is clear. Directions that are orthogonal in weight space can still produce correlated responses on the data. The identity E∥Δx∥2=∥ΔC1/2∥F2\mathbb{E}\|\Delta x\|^2=\|\Delta C^{1/2}\|_F^2 gives a concrete way to account for this, and the metric-polar step in Eq. (8) follows naturally from the corresponding constrained approximation problem. This is a useful extension of the TSV framework, with a clear distinction between selecting directions and deciding how they should coexist.

  2. The encoder results are substantial and reasonably broad. The evaluation includes three CLIP backbones and 8-, 14-, and 20-task settings.

  3. The paper makes a serious effort to understand the gains. The experiments separate selection from orthogonalization, examine where the input and output metrics help, and vary the conditioning rank and whitening powers.

  4. The method produces a static merged checkpoint in the original architecture, with no additional routing or inference computation. This is a practical advantage. The distinction between the main one-sided method and the more expensive sensitivity-based extension is also useful, since the former already accounts for most of the reported improvement.

Critical weaknesses
  1. The comparison with the closest data-aware spectral methods needs more care. The p=1p=1 column in the whitening-power sweep and Table 13 is labelled “PAVE”, but it still runs F-TSV’s projection and cross-task orthogonalization. PAVE instead truncates the fine-tuned weights under the input covariance, unwhitens the result, and then subtracts the base weights. Changing F-TSV’s exponent therefore does not reproduce PAVE. An actual PAVE comparison, with the aggregation rule and calibration budget specified, would make the empirical distinction much clearer.

  2. The connection to task-loss curvature needs a more precise statement. Eq. (3) uses J⊤JJ^\top J, with JJ defined through the final representation. For a general task loss, the Gauss–Newton term contains J⊤HℓJJ^\top H_{\ell}J, where HℓH_{\ell} is the loss Hessian with respect to that representation. The proposed metric has a valid interpretation through squared representation change, but the assumption connecting it to the actual task loss should be explicit. Similarly, the per-task rank constraint alone does not imply the polar merging rule. The manuscript already acknowledges that F-TSV is not an exact minimizer of Eq. (2); the description of the unification, particularly the “solution” column in Table 6, should consistently reflect that qualification.

Questions for authors

See above

4 comments
AuthorsOpenReview ↗

Answer to W1 - PAVE clarification

We thank the reviewer for recognizing the clear theoretical motivation and principled formulation of our approach, the substantial and broad experimental results, and the thorough analysis of its improvements. We also appreciate their acknowledgment of the method’s practical advantages, particularly its ability to produce a static merged model without additional inference overhead.

We agree with the suggestion, as the column labeled PAVE [1] in the whitening-power tables could be misinterpreted as an actual PAVE run. In the revision, we will make these minor style fixes: we will rename the column p=1 “PAVE-style” in both the §5 table and the full grid in the appendix, to indicate the data-aware version of it with our projection and cross-task orthogonalization. Then, as suggested, we add a PAVE comparison, the results show that F-TSV is above PAVE in each setting. We report here a more detailed breakdown with absolute accuracy and normalized accuracy in parentheses:

TAPAVE + TATSV-MPAVE + TSV-MF-TSV
ViT-B/32, 8 tasks70.4 (77.9)69.8 (77.2)83.4 (92.0)81.5 (89.9)86.8 (96.0)
ViT-B/32, 14 tasks66.8 (75.2)66.7 (75.0)79.1 (88.6)76.9 (86.2)83.6 (93.7)
ViT-B/32, 20 tasks64.0 (71.9)63.3 (71.1)76.8 (85.9)74.2 (83.1)82.2 (91.8)
RoBERTa, GLUE57.1 (66.1)56.9 (66.0)66.3 (78.0)58.4 (68.2)73.1 (87.9)

F-TSV's gain comes from the step PAVE does not have: cross-task orthogonalization in the pooled data metric, the vision gain in the table.

In what follows, we describe the implementation details and work to ensure the reproducibility of this comparison. We reproduced PAVE from the authors' released code, including its rank allocation (Alg. 1), and verified collection, rank search, and purification on identical inputs (we measured that the outputs agree to 10−510^{−5}).

  • Aggregation rule. PAVE purifies each task vector and leaves the merge to a standard method. We pass its purified vectors to Task Arithmetic (TA) and to TSV-M.
  • Calibration budget. 512 unlabeled training examples per task, forwarded through each fine-tuned expert as PAVE prescribes.
  • Selection. PAVE's rank ratio ρ is chosen on held-out validation from its manuscript's grid {7/8, 15/16, 31/32, 63/64}, and the coefficient α on validation for every column.

Across three vision backbones with 8, 14, and 20 tasks, PAVE reduces performance for every tested merger except ESM (+0.03). On GLUE, only weight averaging improves (+0.98). Smaller ρ\rho consistently worsens vision results, while all tested values underperform plain TA on GLUE.

Three things are particularly relevant (in normalized points):

  1. Purified experts preserve 99.6–100.1% of their original accuracy on vision and 99.2–102.2% on GLUE, exactly reproducing PAVE's findings in our implementation.
  2. The purification seems not to help. Validation selects the largest ρ=63/64\rho=63/64 in every cell, yet PAVE remains 0.1–0.8 points below TA and 2.2–2.8 (vision) or 9.8 (GLUE) below TSV-M. Increasing ρ\rho to 127/128127/128, 255/256255/256, and 0.99830.9983 only approaches the unpurified baseline: PAVE + TSV-M improves from 89.9 to 91.8 versus 92.0 on ViT-B/32 (8 tasks), and from 68.2 to 74.5 versus 78.0 on GLUE. PAVE + TA moves from 77.2 to 77.9 versus 77.9 on vision, and ranges from 66.0 to 67.5 versus 66.1 on GLUE. Across all seven ρ\rho values, validation favors the unpurified host in these four settings.
  3. Over four independent calibration draws, PAVE + TA and PAVE + TSV-M achieve 77.2 ± 0.1 and 89.9 ± 0.1 on ViT-B/32 (T=8), and 66.2 ± 0.3 and 66.5 ± 2.0 on GLUE; the issue is not calibration noise.

The key distinction is that PAVE removes directions weakly activated by each expert's own data, independently of other tasks. Although individual accuracy is preserved, the removed weights exceed the task vector itself (median 2.4×2.4\times on ViT-B/32 at ρ=63/64\rho=63/64). Even at ρ=7/8\rho=7/8, where every purified expert retains at least 99.2% individual accuracy, merging with TSV-M drops to 63.8 versus 92.0 without purification.

AuthorsOpenReview ↗

Answer to W2 - Task-loss curvature

We thank the reviewer for identifying these points. We follow the suggestion and state more precisely: (a) the relationship between our functional metric and task-loss curvature, (b) the optimization status of the polar merging rule, (c) the point on Table 6. These are all clarifications that do not affect the proposed method or any reported experimental results.

(a) Connection to task-loss curvature. Our formulation assumes isotropic representation curvature (Hℓ=IH_\ell=I), measuring squared first-order representation change rather than arbitrary task-loss curvature. We will explicitly state this assumption in Appendix B:

“Assumption 4 (isotropic representation curvature). HℓH_\ell: task t's loss is measured by the squared change of the final representation.”

and generalize Eq. (3) to include Hℓ=∇f2ℓtH_\ell=\nabla_f^2\ell_t, clarifying that our method specializes the isotropic assumption. Since is common knowledge that the generalized Gauss–Newton curvature involves J⊤HℓJJ^\top H_\ell J, rather than simply J⊤JJ^\top J.

This deliberate approximation provides a task-loss-agnostic, label-free sensitivity metric that can be efficiently estimated through Gaussian vector-Jacobian products. As shown in Appendix E, our normalization removes global positive scaling differences, but not anisotropy in a general HℓH_\ell. Following the reviewer's suggestion, we think that incorporating this anisotropy represents an interesting extension for future work and for building upon our methods, to relax this assumption.

(b) Rank constraint and polar merging. We make it clear that the rank constraint specifies the spectral budget and the polar rule additionally follows from the non-interference constraint and the nearest-factor criterion. Specifically, after stacking the retained factors, we seek the closest factors satisfying orthonormality in their respective geometries. For the right factor, this yield the constrained optimization problem

min⁡R∣(R−V^⊤)Cˉp∣F2subject toRCˉ2pR⊤=I.\min_R \left|(R-\hat V^\top)\bar C^p\right|_F^2 \quad\text{subject to}\quad R\bar C^{2p}R^\top=I.

whose exact solution (Proposition 1) is

R⋆=polar⁡(V^⊤Cˉp)Cˉ−p.R^\star=\operatorname{polar}(\hat V^\top\bar C^p)\bar C^{-p}.

The left-factor follows analogously. Thus, the polar merging rule is an exact solution to the factor-level constrained approximation problem, but not necessarily to the joint least-squares objective in Eq. (2). The latter motivates the geometry, while the spectral budget, non-interference constraint, and nearest-factor criterion define a tractable merging rule. We will make this distinction more explicit in §4 and Appendix C.

(c) Qualification of the unification. We agree that Table 6 may suggest that all methods exactly solve Eq. (2). We will rename the column “Solution of Eq. (2)” to “Merge rule” and clarify that unconstrained methods solve Eq. (2), whereas spectral methods use structured rules motivated by its geometry.

These minor revisions clarify the theoretical scope without altering our central contribution: a unified least-squares perspective that motivates data-aware spectral selection and cross-task orthogonalization.

AuthorsOpenReview ↗

References

[1] An B. et al., Mitigating Model Merging Conflicts via Knowledge-Aware Task Vector Purification, EMNLP 2026

AuthorsOpenReview ↗

Regarding author response

Dear Reviewer KfWN,

Thank you once again for the time and care you have devoted to reviewing our paper. Your comments have been genuinely valuable to us and have helped us improve our work.

We have posted detailed responses to each of the weaknesses and questions you raised, and we hope you will find them helpful and that they adequately address your concerns.

We fully understand that this is a particularly busy period, and we truly appreciate any time you are able to spare. Should any point remain unclear or not fully convincing, we would be more than happy to provide further clarifications. As the discussion period will close in just a couple of days, we would be very thankful if you could take a look at our responses and share your thoughts at your earliest convenience: this would give us the chance to address any remaining questions as thoroughly and carefully as they deserve.

If you feel that our responses have satisfactorily addressed your concerns, we would be sincerely grateful if you would consider updating your score to reflect this. Of course, we fully respect your judgment either way.

Thank you again for your time and thoughtful feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal Revision

We sincerely thank the reviewers for their time and constructive feedback. We have addressed every comment raised in the reviews. We believe the limited need for further discussion reflects that the submitted manuscript was already solid: the revisions clarify and strengthen it, without changing the method or its main results. Below, we summarize these revisions. All references refer to the revised manuscript, where new or changed text is marked in red.

Main revision

  • MaTS added as a baseline. MaTS now appears in Tables 1–2. On GLUE, a comparison runs F-TSV, RegMean and MaTS on each other's statistics and reports the examples per task, the models run, label use and collection time (App. S, Table 19).

  • PAVE comparison.The purified task vectors of PAVE are merged with TA and TSV-M. The aggregation rule, calibration budget and selection of ρ\rho are specified (App. S, Table 20).

  • Complete selection × orthogonalization ablation. This ablation, which includes data-aware orthogonalization alone, is now in §5 (Q2). Two examples and diagnostics of the retained subspaces are in App. J.

  • Another LLM: Gemma-2-2B on MergeBench. The Gemma-2-2B results cover all five pillars, with the per-pillar trade-offs are in App. T and Table 21.

  • Spectral-block test. The ablation that replaces Eq. 6 with exact left-projection block under both orthogonalizations is now in App. R and Table 18.

Minor revision

  • Coefficient attribution. A common-coefficient diagnostic now covers all settings (App. J), and the MergeBench validation curves are added (App. Q and Table 17).

  • Curvature assumption made explicit. Eq. 3 now includes the loss Hessian HℓH_\ell, and the isotropic case is stated as Assumption 4 (App. B).

  • Status of the polar rule clarified. The Table 6 column "Solution of Eq. 2" is renamed "Merge rule".

  • K-FAC factorization error measured on ViT-B/32 and GLUE. It is now reported in App. A.

  • The p=1p=1 column is relabelled "PAVE-style" in the §5 grid and Table 13.

  • Difference from PAVE and ESM stated in §2. Both make only the per-task step data-aware, whereas F-TSV also orthogonalizes across tasks in the data metric.

  • F-TSV is recommended by default, and F-TSV² only when maximizing performance outweighs its additional offline cost (App. A).

  • Data requirements discussed, covering unlabeled inputs, label use and coefficient selection (App. A).

We hope these revisions address the reviewers' concerns, and we remain available for any further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.