- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
F-TSV replaces the Euclidean geometry of spectral model merging with a metric estimated from unlabeled activations. The metric guides subspace selection and cross-task orthogonalization; a two-sided extension additionally estimates downstream sensitivity. Experiments cover three CLIP backbones, RoBERTa on GLUE and Llama-3.2-3B on MergeBench. The central empirical result is that pooled-metric orthogonalization contributes substantially more than metric-guided selection, particularly on vision tasks.
Critical strengths
The paper separates two operations that are often conflated in spectral merging. Across nine vision settings, the reported increments are +0.11 normalized points for selection and +4.16 for subsequent orthogonalization. Exact reductions to TSV-M provide a useful implementation control, while repeated statistics draws on ViT-B/32 with eight tasks and on GLUE support the reliability of the gains in those settings. The discussion is appropriately more restrained about generative merging.
Critical weaknesses
-
The empirical justification for the particular spectral block remains incomplete. Section 4 explicitly states that F-TSV does not minimize the joint least-squares objective, and Section 4.1 acknowledges that Eq. 6 is weakly suboptimal for the isolated functional approximation problem. The proposed reason for using it is that preserving the selected input subspace benefits the later orthogonalization. That is a testable design claim, but comparing against ESM does not isolate it because ESM also changes the orthogonalization. Replacing only Eq. 6 with the stated exact left-projection block, while retaining the same pooled-metric orthogonalization, would directly test the justification. Proposition 2 establishes the square-root power for the orthogonalization step; it does not settle this block choice.
-
MaTS is a consequential omission. It appears in the paper's curvature framework but not in the experiments. On GLUE, F-TSV's advantage over RegMean is 2.4 normalized points, much smaller than its advantage over TSV-M. A comparison under matched data and curvature assumptions, with label access and computational costs stated explicitly, would clarify whether the spectral construction offers an advantage beyond the choice of metric.
-
The two-sided extension has limited practical support. Across five paired statistics draws with the coefficient pinned, its gain is 0.15±0.05 normalized points on ViT-B/32 with eight tasks and 0.13±0.76 on GLUE. The paper acknowledges this and reports some statistics-collection and matrix-merging costs, but does not isolate the incremental time and memory required to collect G alongside these gains. Possible error in the K-FAC factorization remains an undiagnosed explanation, not an established cause.
-
MergeBench provides only modest support for the mechanism's transfer. The selected coefficient changes from 0.8 to 1.8 between the Euclidean and functional methods, and the average improvement is 0.35 points, with coding declining from 40.41 to 36.46. Selecting each method's coefficient on validation data is legitimate, but those rows do not isolate the effect of geometry. A common-coefficient diagnostic or a coefficient-response curve would clarify the attribution; it should supplement, not replace, the main selection protocol.
Questions for authors
-
In Appendix E's ViT-B/32 eight-task sweep, validation favors r=32 only marginally: r=32–512 lie within 0.15 points. How was the default r=128 chosen?
-
Can you provide a matched MaTS comparison on GLUE, specifying the data, label access and statistics-collection costs?
7 comments
Answer to W2 - MaTS & RegMean
We thank the reviewer for highlighting these related works. MaTS [3] is added as an additional baseline. F-TSV remains the best-performing data-aware method across all benchmarks. We implemented the method following Tam et al. (Sec. 6.2), using RegMean’s [4] objective per linear layer, with the reference code's stopping rule.
Vision absolute accuracy and normalized accuracy in parentheses:
| B/32 T=8 | B/32 T=14 | B/32 T=20 | B/16 T=8 | B/16 T=14 | B/16 T=20 | L/14 T=8 | L/14 T=14 | L/14 T=20 | |
|---|---|---|---|---|---|---|---|---|---|
| RegMean | 82.5 (91.0) | 77.2 (86.5) | 75.4 (84.3) | 85.3 (92.0) | 80.6 (87.8) | 78.0 (84.7) | 91.0 (96.5) | 88.1 (94.1) | 85.6 (90.9) |
| MaTS | 83.1 (91.7) | 77.8 (87.1) | 76.1 (85.1) | 86.2 (93.0) | 81.1 (88.4) | 78.9 (85.7) | 91.3 (96.8) | 88.4 (94.5) | 86.3 (91.7) |
| F-TSV | 86.8 (96.0) | 83.6 (93.7) | 82.2 (91.8) | 89.1 (96.4) | 85.6 (93.6) | 85.0 (92.3) | 93.2 (98.9) | 91.5 (97.9) | 91.9 (97.7) |
MaTS consistently improves on RegMean (+0.3 to +0.9 absolute, +0.3 to +1.0 normalized), but F-TSV outperforms MaTS by 1.9–6.2 absolute (2.1–6.7 normalized) points, with larger gains as task count increases.
GLUE under matched data, curvature and cost. In its full fine-tuning recipe, MaTS shares the curvature form of RegMean and F-TSV, , so the comparison can be made exactly: each method also runs on the other's , and within a setting all methods use the same statistics.
| computed from | Examples/task (↓) | Models run (↓) | Labels (↓) | Covariances collection (↓) |
|---|---|---|---|---|
| pretrained encoder (F-TSV style) | 512 | 1 (pretrained) | none | 8.1 ± 0.2 s |
| each expert on its own task (RegMean style) | 1024 | 8 (one expert per task) | none | 10.5 ± 0.2 s |
Collection is one forward pass over the listed examples; its time is wall-clock for all 8 tasks on one GPU, mean ± std over 10 runs; each statistic stores 567 MiB per task, and none uses labels. The F-TSV setting therefore needs half the examples, a single model (the pretrained encoder, where RegMean and MaTS must run every expert) and less collection time. The merge step, timed the same way with the statistics in memory, is where F-TSV pays: 44.0 ± 0.9 s, against 2.2 ± 0.4 s for RegMean and 20.5 ± 0.4 s for MaTS on their own statistics.
| computed from | F-TSV | RegMean | MaTS |
|---|---|---|---|
| pretrained encoder (F-TSV style) | 73.1 (87.9) | 57.3 (66.2) | 57.0 (65.9) |
| each expert on its own task (RegMean style) | 63.2 (76.5) | 71.6 (85.5) | 71.7 (85.5) |
GLUE average, absolute (normalized); every method selects α on the same labeled validation split. As expected, each method is strongest in the setting it is derived for: RegMean and MaTS match every expert's layer outputs on that expert's own inputs, while F-TSV measures the update at the pretrained model's inputs. Each in its own setting, F-TSV leads RegMean and MaTS by +1.4–1.5 absolute and +2.4 normalized points, while using the cheaper setting of the first table.
- In F-TSV’s setting, the spectral construction is the difference. On F-TSV's statistics RegMean and MaTS fall to Task Arithmetic's level (57.1). Per-task subspace selection and decorrelation in turn the same matrices into +21.6 normalized points over RegMean.
- RegMean and MaTS need expert-side statistics, i.e. a forward pass through every expert and twice the examples: the experts' Gram matrices diverge with depth, and their weighting relies on that difference. With them, they reach 85.5; F-TSV on its own statistics remains 2.4 normalized points ahead while using half the examples and only the pretrained encoder.
- The converse holds. On expert-side statistics, F-TSV drops to 76.5, mostly due to STS-B (Spearman 2.3), without STS-B, it follows RegMean and MaTS by 1.1 and 1.4 points, respectively, so the drop in performance, in the competitor setting, is mostly attributable to a single failure statistic, not to a generalized behavior, showing great adaptability.
Answer to W3 - F-TSV gains
We thank the reviewer for raising this point. We want to point reviewer zRTG to read response W4 of Reviewer Xa2q, since they partially overlap; we summarize the essential parts of our response there and then add the two items it does not cover: memory and the K-FAC explanation.
- We agree that the gain of F-TSV$^2$ over F-TSV is small: Appendix A reports the paired numbers you quote, and the paper does not rest on F-TSV$^2$.
- G costs more FLOPs than C on the encoders and consequently takes longer wall-clock time, for example, 1.4 min in ViT-B/32 with 8 tasks on GLUE (see the table in W4 answer of Reviewer Xa2q for a full breakdown). For LLMs, it costs at most 1.5× the FLOPs.
- G helps most on tasks where the one-sided merge recovers poorly: +1.35 points on average on vision tasks (best in small- to mid-sized models). With a coefficient nearly the same as F-TSV, it also improves all four safety benchmarks on both LLMs, without refusing more safe prompts.
- F-TSV therefore remains the default, and F-TSV² is an option when compute resources are not limited and the last points matter.
Memory. The runs recorded the peak GPU memory of each collection and the size of each stored statistic. Each cell is F-TSV vs F-TSV$^2$:
| Model | Peak GPU memory to collect |
|---|---|
| ViT-B/32 (T = 8) | 1.3 / 5.9 GiB |
| ViT-B/16 (T = 8) | 2.2 / 19.4 GiB |
| ViT-L/14 (T = 8) | 5.2 / 66.5 GiB |
| RoBERTa-base (GLUE) | 1.6 / 11.4 GiB |
| Llama-3.2-3B (MergeBench) | 30.9 / 62.9 GiB |
On MergeBench, F-TSV's statistics include its token-embedding metric (the biggest matrix: 128,256 × 3,072). The paired gains thus cost, on ViT-B/32 (T = 8), 1.4 minutes, a 5.9 instead of 1.3 GiB GPU peak for +0.15 ± 0.05 points, on GLUE, 1.4 minutes, 11.4 instead of 1.6 GiB for +0.13 ± 0.76. G's peak comes from keeping one forward pass's activations for all the backward passes of its probes, so it scales with the batch size and can be reduced with smaller batches.
K-FAC. Following the reviewer's suggestion, we measured it: on ViT-B/32 (T = 8) and GLUE, for the rank-one components the F-TSV$^2$ merge keeps in every layer, we computed their exact functional inner products and compared each metric with them. Against a measurement ceiling of 0.98–0.99, the K-FAC metric reproduces the exact cross-task geometry at r = 0.30 (vision) and 0.44 (GLUE), F-TSV's input-only metric at 0.24 and 0.28, the Euclidean at 0.10 for both. Raw and resampled statistics give the same r, so the loss is the factorization itself, mainly the assumed independence of inputs and downstream Jacobians (dropping only the cross-token terms keeps 0.62 and 0.77), hinting at what we described for the answer W2 to reviewer KfWN. Accordingly, K-FAC predicts that G lowers the merge's exact functional error in 58% and 74% of (task, layer) pairs, but it does in 26% and 55%, predicting a larger benefit from G than the real benefit. Probe noise is not a factor: 4× the probes leave the paired gain unchanged, more probes do not change the gain (32 probes +0.15 ± 0.04 vs 128 probes +0.15 ± 0.05 on vision; +0.04 ± 0.73 vs +0.13 ± 0.76 on GLUE). The revision will report this in Appendix A as a measured, contributing cause.
Answer to W1 - Choice Eq. 6
We thank the reviewer for recognizing the clear distinction between subspace selection and orthogonalization, the rigorous experimental controls and reliability analyses, and the appropriate interpretation of the results.
We also thank the reviewer again for identifying this complementary ablation that reinforces our method. We agree, as stated in §5, that Eq. 6 does not minimize the isolated functional approximation error, still, the following experiments favor our choice. Proposition 2 fixes the power of the orthogonalization constraint but does not establish the optimality of the selected block. The comparison with ESM [1] alone cannot isolate this choice, as ESM also changes other components.
Following the reviewer's suggestion, we replaced only Eq. 6 by the exact block minimizer: U~(k)U~(k)⊤Δt and kept everything else fixed: statistics, conditioning, p=21, rank budget, and the pooled-metric polar orthogonalization of Eq. 8. Repeating the comparison with Euclidean orthogonalization yields a 2×2 ablation: block × orthogonalization.
The table reports normalized scores (MergeBench average), with gaps in parentheses measuring Eq. 6 minus the exact block.
The results support our choice of Eq. 6, under pooled-metric orthogonalization, the exact minimizer does not outperform Eq. 6 on average in any benchmark. On vision, the blocks are tied (+0.09 normalized points), with Eq. 6 ahead in 7/9 settings and differences ranging from −0.12 to +0.35. On GLUE, Eq. 6 is 2.91 points better, while on MergeBench it is better by 0.48.
Importantly, pooled-metric orthogonalization consistently improves performance regardless of the block: +4.16/+4.37 on vision and +7.44/+9.29 on GLUE (Eq. 6/exact), with gains in every vision setting. The smaller block advantage under pooled-metric orthogonalization is consistent with reduced room for improvement.
The exact block minimizes the functional reconstruction error for each expert independently, whereas Eq. 6 also preserves the selected input subspace before cross-task orthogonalization. Our results support our choice of Eq. 6 as an empirically effective block, particularly on GLUE, without establishing that preserving the selected input subspace is the sole cause of its advantage. We think §4.1 remains largely valid and needs only minor changes, along with the addition of these experimental tests (in the Appendix) that confirm our choice. In conclusion, we have seen that the theoretically optimal approximation is not necessarily the empirically preferable building block for model merging, and we think this is a useful finding in its own right.
Section 2 connects ESM to the exact block followed by Euclidean orthogonalization. However, this ablation cell is at most +0.01 above TSV-M [2] on vision (mean −0.19), whereas ESM gains +0.54 to +2.38 above it. Hence, ESM's gains cannot be attributed to the block alone; its other components (expert-side statistics, rank allocation, reweighting, spectrum flattening, and layer rescaling) also differ. We present this as a structural connection, not a complete derivation or causal attribution of ESM's gains.
Minor revisions: we will clarify in §4.1 that neither Eq. 2 nor Proposition 2 determines the block choice, retain its motivation with appropriate qualifications, and add the full ablation and analysis of the remaining performance gap to the Appendix.