- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates why post-hoc task-vector sparsification can improve multi-task model merging. It introduces Linearized Functional Decomposition (LFD), a target-conditioned intermediate function that combines the transformed target model with an aggregated non-target model. The framework tracks merged–LFD disagreement, LFD–target disagreement, and retained single-task accuracy. A curvature-based interpolation bound decomposes the merged–LFD representation discrepancy into task-vector squared norms and signed cross-task interactions. The authors use this decomposition to analyze several sparsification rules, evaluate diagnostic associations on ViT and BERT, and provide a controlled comparison between rank truncation and TSV orthogonalization.
Critical strengths
-
A useful separation of functional disagreement and retained capability.
-
Careful treatment of signed cross-task interactions.
-
Several concrete and testable geometric results.
-
Substantial empirical documentation.
Critical weaknesses
-
The geometric bound is not sufficiently connected to actual functional differences. Theorem 3.1 bounds representation discrepancy by , but subsequent analyses mainly compare . Different masks and merge weights may change both the relevant curvature and the tightness of the bound. Even under a common , a smaller upper bound does not imply a smaller actual discrepancy.
-
Object-specific KNN classifiers complicate the interpretation of the vision diagnostics. Appendix D.1 constructs separate KNN classifiers for the merged, LFD, target, and non-target representations. Consequently, prediction disagreement reflects changes in both query representations and exemplar banks. Proposition 3.2 remains valid, but Theorem 3.1 does not directly explain changes between these decision rules.
-
The analysis does not adequately separate sparsity from magnitude reduction. Squared-norm reduction is central to the explanation, yet the paper lacks a dense scaling baseline that matches each sparsified task vector’s norm while preserving its direction. DARE does not isolate this factor because it changes both support and scaling.
-
The LFD interpretation becomes sensitive to parameter extrapolation at large merge weights. The non-target displacement scales as , reaching factors of 9 and 99 at and . Although Appendix C.1 appropriately qualifies endpoint convergence, reduced LFD–target disagreement near the endpoint may partly reflect the intermediate construction and output scaling.
-
The NLP metrics are not fully aligned with the theoretical guarantee. Proposition 3.2 concerns classification accuracy, whereas the reported aggregate includes MCC and F1, which the proposition does not cover. STS-B is also reformulated as five-class classification, making the aggregate different from conventional GLUE evaluation.
Questions for authors
-
Do changes in reliably track measured representation gaps and prediction disagreement after accounting for curvature and decision margins?
-
Do the vision findings persist under a shared frozen classifier and alternative KNN settings?
-
Can norm-matched dense scaling reproduce the effects attributed to sparsification?
-
Are the LFD conclusions robust to excluding large extrapolation factors or using alternative intermediate paths?
-
Does LFD outperform simple diagnostic baselines, and do its predictive relationships transfer to new task subsets or unseen transformations?
-
Do the NLP conclusions hold under conventional GLUE metrics, with accuracy-based guarantees distinguished from empirical associations?
4 comments
W4 & Q4: The LFD interpretation becomes sensitive to parameter extrapolation at large merge weights. The non-target displacement scales as , reaching factors of 9 and 99 at and . Although Appendix C.1 appropriately qualifies endpoint convergence, reduced LFD–target disagreement near the endpoint may partly reflect the intermediate construction and output scaling. Are the LFD conclusions robust to excluding large extrapolation factors or using alternative intermediate paths?
Thank you for highlighting the issues of extrapolation and output scaling at large merge weights. We agree that a decrease in disagreement between the intermediate and target functions near the endpoint cannot, on its own, establish reduced interference in the actual merged model. We therefore conducted further experiments excluding the region of large extrapolation. We restricted the merge weight to at most 0.5 and 0.3, respectively, so that the non-target parameter scaling factor was at most 1.0 and 0.43. Within each restricted range, we refitted the predictors using the same leave-one-target-task-out protocol and compared rank correlations for the stronger baseline (method one-hot features, , , , and their interactions) with those obtained after adding the LFD path diagnostics. The prediction target was normalized merging gain.
| Region | Stronger baseline | With LFD path diagnostics |
|---|---|---|
| <0.5 | 0.589 | 0.791 |
| <0.3 | 0.514 | 0.740 |
Even after strictly excluding points with large extrapolation, the LFD path diagnostics still provide incremental information beyond method identity and merge weight. These results support the robustness of LFD's main diagnostic conclusions to the exclusion of large extrapolation factors and indicate that their effectiveness is not driven solely by the construction and output scaling near the endpoint.
W5 & Q6: The NLP metrics are not fully aligned with the theoretical guarantee. Proposition 3.2 concerns classification accuracy, whereas the reported aggregate includes MCC and F1, which the proposition does not cover. STS-B is also reformulated as five-class classification, making the aggregate different from conventional GLUE evaluation. Do the NLP conclusions hold under conventional GLUE metrics, with accuracy-based guarantees distinguished from empirical associations?
Our NLP setup is designed to provide a controlled comparison for analyzing sparsification mechanisms. All tasks share the same frozen MLM head, and the prompt template and verbalizer are fixed within each target task, so that differences among the merged, LFD, and target models primarily reflect changes in backbone parameters, reducing the additional influence of task-specific output heads. We reformulate STS-B as five-class classification to include it in a unified analysis of discrete predictions and disagreements. This choice is justified by the experimental design, while admittedly changing the definition of the original regression task. The other seven classification tasks retain their customary performance metrics, including MCC for CoLA and accuracy/F1 for MRPC and QQP, to preserve the practical meaning of task evaluation.
Q5: Does LFD outperform simple diagnostic baselines, and do its predictive relationships transfer to new task subsets or unseen transformations?
Thank you for pointing out the limitations of the predictive baselines and generalization evidence. To address your concerns, we added the -only, method one-hot + , and -only baselines, the last of which requires no forward passes, using exactly the same data, prediction target, and leave-one-target-task-out protocol. As shown below, their Spearman correlations are 0.542, 0.658, and 0.093, respectively, all substantially lower than that of the three LFD diagnostics. More directly, adding the three diagnostics to the method + baseline increases Spearman correlation from 0.658 to 0.846 and pairwise method-ranking accuracy within the same task × seed × group from 66.56% to 83.85%. These results support the incremental predictive information provided by the diagnostics beyond the baselines using a linear merge-weight feature and method identity evaluated here.
| Predictors | Spearman | Pairwise |
|---|---|---|
| -only | 0.542 | 50.00% |
| Method-Onehot+ | 0.658 | 66.56% |
| ΔB-only | 0.093 | 56.01% |
| Three Factors | 0.856 | 81.71% |
| Three Factors+Method-Onehot+ | 0.846 | 83.85% |
[1] Representation Surgery for Multi-Task Model Merging. ICML 2024
[2] Representation Surgery in Model Merging with Probabilistic Modeling. ICML 2025
[3] MOMA: Masked Orthogonal Matrix Alignment for Zero-Additional-Parameter Model Merging. arXiv:2412.13526
Thank you for the authors’ response. This partially addresses my concern, and I have raised my score from 1 to 2. However, because the core explanatory chain of the paper remains unclosed, the KNN evaluation in the vision diagnostics remains confounded, and sparsification is still not sufficiently separated from magnitude reduction, I regret that I still maintain a negative score.
Thank you for your valuable and constructive comments. Our responses are provided below.
Thank you for this valuable insight. In Theorem 3.1, M was defined as a common upper bound across methods under the smoothness and boundedness assumptions; the same M applies to all masks and values of λ being compared. In light of your concern, we recognize the need to further estimate the tightest local upper bound for each configuration. Our preliminary estimates suggest that the methods studied have broadly comparable upper bounds. We will include the detailed estimates in the next version.
Our approach is motivated by previous studies [1,2,3], which found that the conventional evaluation protocol—using the fine-tuned model's classification head to evaluate the merged model—is inappropriate. Empirical evidence shows that merged representations may form well-separated clusters by class label while remaining misaligned with a fixed classification head. In some settings, a fixed head can therefore underestimate the performance of certain model-merging techniques, which is why we did not adopt this potentially problematic evaluation protocol. In our experiments, using a shared fixed classifier would confound the observations and prevent us from drawing valid conclusions.
Thank you for suggesting this simple comparison baseline. We found that simply rescaling the vector norms can indeed reduce the bound to a level comparable to that of methods such as MIN. However, the corresponding M increases substantially, so the merged–LFD gap does not shrink to the same level as with MIN and similar methods. We will discuss this issue in a subsequent version and work to refine our theoretical framework so that it can explain the behavior of this baseline.