acceptodds
Desk rejected at ICLR 2027

Understanding Sparsification in Multi-Task Model Merging via Linearized Functional Decomposition

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-task model merging offers an efficient alternative to multi-task training, yet post-hoc parameter sparsification remains weakly explained by training-time analyses. We introduce linearized functional decomposition (LFD), which separates target-task and non-target contributions and bounds the merged–LFD discrepancy. The resulting path diagnostics also track LFD–target consistency and retained single-task quality. We examine how transformations change task-vector norms and cross-task interactions, then compare the bound with measured path behavior. Across vision and language tasks, effective sparsification mainly narrows the merged–LFD gap, while the other diagnostics reveal changes in target consistency and capability. The diagnostics also predict relative gains for held-out tasks and show where transformations beyond sparsification help or hurt.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Model Merging, Multi-Task Learning, Task Vector, Weight Disentanglement

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RDxROpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why post-hoc task-vector sparsification can improve multi-task model merging. It introduces Linearized Functional Decomposition (LFD), a target-conditioned intermediate function that combines the transformed target model with an aggregated non-target model. The framework tracks merged–LFD disagreement, LFD–target disagreement, and retained single-task accuracy. A curvature-based interpolation bound decomposes the merged–LFD representation discrepancy into task-vector squared norms and signed cross-task interactions. The authors use this decomposition to analyze several sparsification rules, evaluate diagnostic associations on ViT and BERT, and provide a controlled comparison between rank truncation and TSV orthogonalization.

Critical strengths
  1. A useful separation of functional disagreement and retained capability.

  2. Careful treatment of signed cross-task interactions.

  3. Several concrete and testable geometric results.

  4. Substantial empirical documentation.

Critical weaknesses
  1. The geometric bound is not sufficiently connected to actual functional differences. Theorem 3.1 bounds representation discrepancy by MB^i/2M\widehat B_i/2, but subsequent analyses mainly compare B^i\widehat B_i. Different masks and merge weights may change both the relevant curvature and the tightness of the bound. Even under a common MM, a smaller upper bound does not imply a smaller actual discrepancy.

  2. Object-specific KNN classifiers complicate the interpretation of the vision diagnostics. Appendix D.1 constructs separate KNN classifiers for the merged, LFD, target, and non-target representations. Consequently, prediction disagreement reflects changes in both query representations and exemplar banks. Proposition 3.2 remains valid, but Theorem 3.1 does not directly explain changes between these decision rules.

  3. The analysis does not adequately separate sparsity from magnitude reduction. Squared-norm reduction is central to the explanation, yet the paper lacks a dense scaling baseline that matches each sparsified task vector’s norm while preserving its direction. DARE does not isolate this factor because it changes both support and scaling.

  4. The LFD interpretation becomes sensitive to parameter extrapolation at large merge weights. The non-target displacement scales as λ/(1−λ)\lambda/(1-\lambda), reaching factors of 9 and 99 at λ=0.9\lambda=0.9 and 0.990.99. Although Appendix C.1 appropriately qualifies endpoint convergence, reduced LFD–target disagreement near the endpoint may partly reflect the intermediate construction and output scaling.

  5. The NLP metrics are not fully aligned with the theoretical guarantee. Proposition 3.2 concerns classification accuracy, whereas the reported aggregate includes MCC and F1, which the proposition does not cover. STS-B is also reformulated as five-class classification, making the aggregate different from conventional GLUE evaluation.

Questions for authors
  1. Do changes in B^i\widehat B_i reliably track measured representation gaps and prediction disagreement after accounting for curvature and decision margins?

  2. Do the vision findings persist under a shared frozen classifier and alternative KNN settings?

  3. Can norm-matched dense scaling reproduce the effects attributed to sparsification?

  4. Are the LFD conclusions robust to excluding large extrapolation factors or using alternative intermediate paths?

  5. Does LFD outperform simple diagnostic baselines, and do its predictive relationships transfer to new task subsets or unseen transformations?

  6. Do the NLP conclusions hold under conventional GLUE metrics, with accuracy-based guarantees distinguished from empirical associations?

4 comments

Thank you for your valuable and constructive comments. Our responses are provided below.

W1 & Q1: The geometric bound is not sufficiently connected to actual functional differences. Theorem 3.1 bounds representation discrepancy by $MB_i/2 $, but subsequent analyses mainly compare BiB_{i}. Different masks and merge weights may change both the relevant curvature and the tightness of the bound. Even under a common MM, a smaller upper bound does not imply a smaller actual discrepancy. Do changes in B^i{\hat B}_{i} reliably track measured representation gaps and prediction disagreement after accounting for curvature and decision margins?

Thank you for this valuable insight. In Theorem 3.1, MM was defined as a common upper bound across methods under the smoothness and boundedness assumptions; the same MM applies to all masks and values of λ\lambda being compared. In light of your concern, we recognize the need to further estimate the tightest local upper bound for each configuration. Our preliminary estimates suggest that the methods studied have broadly comparable upper bounds. We will include the detailed estimates in the next version.

W2 & Q2: Object-specific KNN classifiers complicate the interpretation of the vision diagnostics. Appendix D.1 constructs separate KNN classifiers for the merged, LFD, target, and non-target representations. Consequently, prediction disagreement reflects changes in both query representations and exemplar banks. Proposition 3.2 remains valid, but Theorem 3.1 does not directly explain changes between these decision rules. Do the vision findings persist under a shared frozen classifier and alternative KNN settings?

Our approach is motivated by previous studies [1,2,3], which found that the conventional evaluation protocol—using the fine-tuned model's classification head to evaluate the merged model—is inappropriate. Empirical evidence shows that merged representations may form well-separated clusters by class label while remaining misaligned with a fixed classification head. In some settings, a fixed head can therefore underestimate the performance of certain model-merging techniques, which is why we did not adopt this potentially problematic evaluation protocol. In our experiments, using a shared fixed classifier would confound the observations and prevent us from drawing valid conclusions.

W3 & Q3: The analysis does not adequately separate sparsity from magnitude reduction. Squared-norm reduction is central to the explanation, yet the paper lacks a dense scaling baseline that matches each sparsified task vector’s norm while preserving its direction. DARE does not isolate this factor because it changes both support and scaling. Can norm-matched dense scaling reproduce the effects attributed to sparsification?

Thank you for suggesting this simple comparison baseline. We found that simply rescaling the vector norms can indeed reduce the bound to a level comparable to that of methods such as MIN. However, the corresponding MM increases substantially, so the merged–LFD gap does not shrink to the same level as with MIN and similar methods. We will discuss this issue in a subsequent version and work to refine our theoretical framework so that it can explain the behavior of this baseline.

W4 & Q4: The LFD interpretation becomes sensitive to parameter extrapolation at large merge weights. The non-target displacement scales as λ/(1−λ)\lambda/(1-\lambda), reaching factors of 9 and 99 at λ=0.9\lambda=0.9 and 0.990.99. Although Appendix C.1 appropriately qualifies endpoint convergence, reduced LFD–target disagreement near the endpoint may partly reflect the intermediate construction and output scaling. Are the LFD conclusions robust to excluding large extrapolation factors or using alternative intermediate paths?

Thank you for highlighting the issues of extrapolation and output scaling at large merge weights. We agree that a decrease in disagreement between the intermediate and target functions near the endpoint cannot, on its own, establish reduced interference in the actual merged model. We therefore conducted further experiments excluding the region of large extrapolation. We restricted the merge weight λ\lambda to at most 0.5 and 0.3, respectively, so that the non-target parameter scaling factor was at most 1.0 and 0.43. Within each restricted range, we refitted the predictors using the same leave-one-target-task-out protocol and compared rank correlations for the stronger baseline (method one-hot features, λ\lambda, λ2\lambda^2, λ3\lambda^3, and their interactions) with those obtained after adding the LFD path diagnostics. The prediction target was normalized merging gain.

RegionStronger baselineWith LFD path diagnostics
λ\lambda<0.50.5890.791
λ\lambda<0.30.5140.740

Even after strictly excluding points with large extrapolation, the LFD path diagnostics still provide incremental information beyond method identity and merge weight. These results support the robustness of LFD's main diagnostic conclusions to the exclusion of large extrapolation factors and indicate that their effectiveness is not driven solely by the construction and output scaling near the endpoint.

W5 & Q6: The NLP metrics are not fully aligned with the theoretical guarantee. Proposition 3.2 concerns classification accuracy, whereas the reported aggregate includes MCC and F1, which the proposition does not cover. STS-B is also reformulated as five-class classification, making the aggregate different from conventional GLUE evaluation. Do the NLP conclusions hold under conventional GLUE metrics, with accuracy-based guarantees distinguished from empirical associations?

Our NLP setup is designed to provide a controlled comparison for analyzing sparsification mechanisms. All tasks share the same frozen MLM head, and the prompt template and verbalizer are fixed within each target task, so that differences among the merged, LFD, and target models primarily reflect changes in backbone parameters, reducing the additional influence of task-specific output heads. We reformulate STS-B as five-class classification to include it in a unified analysis of discrete predictions and disagreements. This choice is justified by the experimental design, while admittedly changing the definition of the original regression task. The other seven classification tasks retain their customary performance metrics, including MCC for CoLA and accuracy/F1 for MRPC and QQP, to preserve the practical meaning of task evaluation.

Q5: Does LFD outperform simple diagnostic baselines, and do its predictive relationships transfer to new task subsets or unseen transformations?

Thank you for pointing out the limitations of the predictive baselines and generalization evidence. To address your concerns, we added the λ\lambda-only, method one-hot + λ\lambda, and ΔB\Delta B-only baselines, the last of which requires no forward passes, using exactly the same data, prediction target, and leave-one-target-task-out protocol. As shown below, their Spearman correlations are 0.542, 0.658, and 0.093, respectively, all substantially lower than that of the three LFD diagnostics. More directly, adding the three diagnostics to the method + λ\lambda baseline increases Spearman correlation from 0.658 to 0.846 and pairwise method-ranking accuracy within the same task × seed × λ\lambda group from 66.56% to 83.85%. These results support the incremental predictive information provided by the diagnostics beyond the baselines using a linear merge-weight feature and method identity evaluated here.

PredictorsSpearmanPairwise
λ\lambda-only0.54250.00%
Method-Onehot+λ\lambda0.65866.56%
ΔB-only0.09356.01%
Three Factors0.85681.71%
Three Factors+Method-Onehot+λ\lambda0.84683.85%

[1] Representation Surgery for Multi-Task Model Merging. ICML 2024

[2] Representation Surgery in Model Merging with Probabilistic Modeling. ICML 2025

[3] MOMA: Masked Orthogonal Matrix Alignment for Zero-Additional-Parameter Model Merging. arXiv:2412.13526

Thank you for the authors’ response. This partially addresses my concern, and I have raised my score from 1 to 2. However, because the core explanatory chain of the paper remains unclosed, the KNN evaluation in the vision diagnostics remains confounded, and sparsification is still not sufficiently separated from magnitude reduction, I regret that I still maintain a negative score.

Reviewer_tRqLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Linearized Functional Decomposition (LFD) to analyse why post-hoc task-vector sparsification (TIES with its magnitude and sign-election steps, DARE and random dropping, Breadcrumbs, TALL/CPS) helps multi-task model merging. The main theoretical result is a second-order Taylor (interpolation-error) bound on the merged-vs-LFD gap, in terms of a quantity that splits into squared task-vector norms and signed cross-task inner products.

Critical strengths
  1. Careful, internally consistent analysis with honest scoping. The algebra in Eq. 8 and Appendix C is correct as far as I checked. Examples: the RAND/DARE expectations in Lemma C.2; the TALL contraction; the ELT task-expectation identity (Eq. 25) and its crossover λ⋆\lambda^\star (Eq. 27). The authors do not stop at the orthogonal idealisation. They check the exact signed finite conditions with full-vector float64 Gram quantities for every task, run and weight (504 task–run–weight comparisons per model family, 1,008 in total; App. C.2, C.3 and Table 5). They also state limits themselves: "causal effects and evaluation-free selection remain open" (App. F). This matters because it makes the geometric claims checkable rather than heuristic.
  2. One non-trivial and falsifiable prediction. The ELT analysis predicts that sign election lowers the bound only for small λ\lambda (guaranteed for λ≤2/T=0.25\lambda\le 2/T=0.25; measured crossover ≈0.32\approx 0.32 on both model families). This matches Fig. 5a/5d, where ELT has lower ξmg−lfd\xi_{\rm mg-lfd} and slightly higher accuracy than TA at small λ\lambda and is worse at large λ\lambda. The TSV case study (Eq. 31) shows a similar sign change in λ\lambda. To me this is the most useful insight in the paper: whether a transformation helps depends on the merge weight it is used with.
Critical weaknesses
  1. The main explanatory claim is confounded with effective merge scale. The bound compares masks "for a fixed curvature bound MM" (l.216-217), but MM depends on a region that changes with both the mask and λ\lambda, is never estimated, and is assumed to cancel; only orderings of the bound are used. Pure deletion lowers the bound largely by shrinking the update, which lowering λ\lambda also does. Fig. 3 fits this reading: DARE, which restores scale, almost overlaps TA, and RAND has the lowest bound but the worst NormAccuracy, while MIN's peak accuracy is only a few points above TA's. All mask comparisons are at equal nominal λ\lambda, never at matched update norm and (apart from the TSV case study) never at each method's best λ\lambda. So "effective sparsification narrows the merged-LFD gap" may largely restate "smaller step, smaller second-order error", and the paper does not separate where coordinates are removed from how much scale is removed, which is the question it sets out to answer (l.67-73).

  2. The predictive and practical evidence is weak. The Table 2 features require the merged model and labelled target-task data, so anyone who can compute them can measure merged accuracy directly; the authors concede that evaluation-free selection remains open (App. F). There is no λ\lambda-only or method-identity baseline, so the pooled Spearman (0.856) may come largely from the shared λ\lambda grid. The scope is also narrow: only ViT-B/32 and BERT-base (no larger models or LLMs), a non-standard KNN evaluation, a single-seed TSV study, and no method or weight-selection rule derived from the diagnostics.

Questions for authors
  1. What justifies comparing B^i\hat{\mathcal B}_i across masks and λ\lambda under a common MM? Could you estimate local curvature, for example with Hessian-vector products along a−ba-b at a few values of λ\lambda, to show that MM (or an effective directional curvature) does not differ systematically between masks?
  2. For Table 2, please add baselines using (i) λ\lambda alone, (ii) method one-hot plus λ\lambda, and (iii) ΔB^i\Delta\hat{\mathcal B}_i alone, which needs no forward passes. Please also report Spearman within each fixed λ\lambda, alongside the existing fixed-weight pairwise accuracy. If the LFD diagnostics add little beyond λ\lambda and method identity, the predictive contribution should be toned down.
  3. How does ξmg−lfd\xi_{\rm mg-lfd} relate to the known gap between weight averaging and output ensembling? flfdf_{\rm lfd} is an output-level ensemble of two models, and the setup is close to cross-task linearity (Zhou et al., 2024b, which the paper cites). Please state precisely what LFD adds beyond these.
3 comments

Thank you for your valuable and constructive comments. Our responses are provided below.

W1(a) & Q1: The main explanatory claim is confounded with effective merge scale. The bound compares masks "for a fixed curvature bound MM" (l.216-217), but MM depends on a region that changes with both the mask and λ\lambda, is never estimated, and is assumed to cancel; only orderings of the bound are used. What justifies comparing B^i\mathcal{\hat B}_{i} across masks and λ\lambda under a common MM? Could you estimate local curvature, for example with Hessian-vector products along a−ba-b at a few values of λ\lambda, to show that MM (or an effective directional curvature) does not differ systematically between masks?

Thank you for this valuable insight. In Theorem 3.1, MM was defined as a common upper bound across methods under the smoothness and boundedness assumptions; the same MM applies to all masks and values of λ\lambda being compared. In light of your concern, we recognize the need to further estimate the tightest local upper bound for each configuration. Our preliminary estimates suggest that the methods studied have broadly comparable upper bounds. We will include the detailed estimates in the next version.

W1(b) & : Pure deletion lowers the bound largely by shrinking the update, which lowering λ\lambda also does. Fig. 3 fits this reading: DARE, which restores scale, almost overlaps TA, and RAND has the lowest bound but the worst NormAccuracy, while MIN's peak accuracy is only a few points above TA's. So "effective sparsification narrows the merged-LFD gap" may largely restate "smaller step, smaller second-order error", and the paper does not separate where coordinates are removed from how much scale is removed, which is the question it sets out to answer (l.67-73).

Thank you for suggesting this simple comparison baseline. We found that simply rescaling the vector norms can indeed reduce the bound to a level comparable to that of methods such as MIN. However, the corresponding MM increases substantially, so the merged–LFD gap does not shrink to the same level as with MIN and similar methods. We will discuss this issue in a subsequent version and work to refine our theoretical framework so that it can explain the behavior of this baseline.

W1(c): All mask comparisons are at equal nominal λ\lambda , never at matched update norm and (apart from the TSV case study) never at each method's best λ\lambda.

We would like to clarify that the theorem at fixed λ\lambda primarily characterizes mask effects under the given conditions by analyzing how different masks change the norm and interaction terms in the upper bound. We do not use minimization of this bound as a criterion for selecting λ\lambda. In fact, ξlfd−tgt\xi_{\rm lfd-tgt} decreases monotonically as λ\lambda increases. When effective sparsification reduces ξmg−lfd\xi_{\rm mg-lfd}, the λ\lambda associated with the best merged accuracy becomes larger than before sparsification, consistent with our observations. When describing each method's model-merging performance, we always use its own best merged accuracy, rather than its performance at a shared λ\lambda.

W2 & Q2: The predictive and practical evidence is weak. The Table 2 features require the merged model and labelled target-task data, so anyone who can compute them can measure merged accuracy directly; the authors concede that evaluation-free selection remains open (App. F). There is no λ\lambda-only or method-identity baseline, so the pooled Spearman (0.856) may come largely from the shared λ\lambda grid. The scope is also narrow: only ViT-B/32 and BERT-base (no larger models or LLMs), a non-standard KNN evaluation, a single-seed TSV study, and no method or weight-selection rule derived from the diagnostics. For Table 2, please add baselines using (i) λ\lambda alone, (ii) method one-hot plus λ\lambda, and (iii) ΔB^i\Delta\mathcal{\hat B}_{i} alone, which needs no forward passes. Please also report Spearman within each fixed λ\lambda, alongside the existing fixed-weight pairwise accuracy. If the LFD diagnostics add little beyond λ\lambda and method identity, the predictive contribution should be toned down.

Thank you for pointing out the limitations of the predictive baselines and generalization evidence. To address your concerns, we added the λ\lambda-only, method one-hot + λ\lambda, and ΔB\Delta B-only baselines, the last of which requires no forward passes, using exactly the same data, prediction target, and leave-one-target-task-out protocol. As shown below, their Spearman correlations are 0.542, 0.658, and 0.093, respectively, all substantially lower than that of the three LFD diagnostics. More directly, adding the three diagnostics to the method + λ\lambda baseline increases Spearman correlation from 0.658 to 0.846 and pairwise method-ranking accuracy within the same task × seed × λ\lambda group from 66.56% to 83.85%. These results support the incremental predictive information provided by the diagnostics beyond the baselines using a linear merge-weight feature and method identity evaluated here.

PredictorsSpearmanPairwise
λ\lambda-only0.54250.00%
Method-Onehot+λ\lambda0.65866.56%
ΔB-only0.09356.01%
Three Factors0.85681.71%
Three Factors+Method-Onehot+λ\lambda0.84683.85%

Q3: How does ξmg−lfd\xi_{\rm mg-lfd} relate to the known gap between weight averaging and output ensembling? flfdf_{\rm lfd} is an output-level ensemble of two models, and the setup is close to cross-task linearity (Zhou et al., 2024b, which the paper cites). Please state precisely what LFD adds beyond these.

The empirical study of cross-task linearity focused on merging two models and found that the merged representations were largely consistent with LFD. The extension to more than two tasks was only theoretical: it proposed f(x;θ+∑i=1Tαiτi)≈∑i=1Tαif(x;θ+τi)f(x; \theta+\sum_{i=1}^T \alpha_i\tau_i) \approx \sum_{i=1}^T \alpha_i f(x; \theta+\tau_i) (Theorem 4.2 in [1]), without empirical verification. In practice, we find that, in common multi-task merging settings involving more than eight tasks, the discrepancy between the merged representations and the sum of the TT model outputs in [1] is too large, and the corresponding bound is not sufficiently concise. We therefore propose an alternative approximation: for a fixed target task tt, we measure the gap between f(x;θ+λ∑i=1Tτ^i)f(x; \theta+\lambda\sum_{i=1}^T\hat{\tau}_i) and λf(x;θ+τ^t)+(1−λ)f(x;θ+λ1−λ∑i≠tτ^i)\lambda f(x; \theta+\hat{\tau}_t)+ (1-\lambda)f(x; \theta+\frac{\lambda}{1-\lambda} \sum_{i \neq t} \hat{\tau}_i). This yields a sufficiently concise bound and a smaller merged–LFD discrepancy.

Reviewer_tRqLOpenReview ↗

Follow-up on the rebuttal

Thank you for the additional experiments. The stronger baselines and grouped holdouts address much of my concern about the predictive evidence and support useful information beyond merge weight and method identity.

My main concern remains the explanatory link. Theorem 3.1 is valid under its assumptions, but ordering its geometric term does not establish an ordering of actual functional discrepancies. Could you provide the quantitative norm-matched comparison and curvature estimates behind the dense-rescaling observation? This would help separate coordinate selection from scale reduction. I acknowledge the paper's stated diagnostic scope and its limitations regarding causal explanation and evaluation-free selection.

Reviewer_Ks7pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a framework LFD for analyzing the effects of sparsification on multi-task model merging. The paper constructs a function-space diagnostic that interpolates between a target-task model and a merged model, aiming to separate target-task behavior from the effects of other task vectors. It introduces three indicators to characterize the functional changes induced by merging and their relationship to target-task performance.

The paper’s claim is that LFD provides useful functional diagnostics for understanding and predicting the effects of sparsification in multi-task model merging. The main question is whether these diagnostics provide sufficiently independent explanatory and predictive value beyond existing task-vector statistics and performance-based indicators.

Critical strengths
  1. A relevant diagnostic perspective on multi-task model merging. The paper focuses on understanding why sparsification affects merged-model performance, rather than proposing another merging heuristic.

  2. Evaluation across multiple merging and sparsification settings. The experiments cover ViT-B/32 and BERT, several merging and sparsification methods, and additional cases involving low-rank truncation and TSV-Merge orthogonalization.

Critical weaknesses
  1. Limited incremental value of LFD. The framework builds on linearization and task-vector interactions. Although LFD provides a target-conditioned path, its additional value over simpler task-vector statistics or performance gaps remains unclear.

  2. Limited predictive evidence. The prediction experiment uses only eight ViT tasks, and the 4,000 configurations share tasks, checkpoints, masks, and merge weights. The reported baselines mainly compare different LFD diagnostics. Comparisons with simpler task-vector statistics and uncertainty estimates would strengthen the results.

  3. Limited connection to merging performance. A smaller representation-gap bound does not directly imply higher merged accuracy. The relationship between the theoretical bound and actual performance remains unclear.

Questions for authors
  1. What additional value do LFD diagnostics provide over simpler task-vector statistics or predictors of merged accuracy?

  2. How robust are the predictive results across different tasks, checkpoints, and random seeds?

  3. How closely does the bound in Theorem 3.1 reflect the observed merged–LFD disagreement and merged accuracy? Does a smaller bound consistently lead to higher accuracy?

  4. How can LFD diagnostics guide the selection of sparsification methods or merge weights without additional evaluation?

3 comments

Thank you for your valuable and constructive comments. Our responses are provided below.

W1: Limited incremental value of LFD. The framework builds on linearization and task-vector interactions. Although LFD provides a target-conditioned path, its additional value over simpler task-vector statistics or performance gaps remains unclear.

We respectfully disagree with the assessment of LFD's incremental value. LFD enables a more precise understanding of how sparsification works: within certain interpolation regions, sparsification brings the merged model's representations closer to those produced by linearized functional decomposition (LFD), resulting in more consistent downstream task performance. As illustrated in Figure 1, we introduced LFD to connect with the only existing interpretability approach in model merging, namely NTK-based weight disentanglement. Previously, this framework could explain the merging results of models modified during training, but could not explain post-hoc techniques such as sparsification. Extending this interpretability to sparsification is precisely the central contribution of our paper.

W2 & Q1: Limited predictive evidence. The prediction experiment uses only eight ViT tasks, and the 4,000 configurations share tasks, checkpoints, masks, and merge weights. The reported baselines mainly compare different LFD diagnostics. Comparisons with simpler task-vector statistics and uncertainty estimates would strengthen the results. What additional value do LFD diagnostics provide over simpler task-vector statistics or predictors of merged accuracy

Thank you for pointing out the limitations of the predictive baselines and generalization evidence. To address your concerns, we added the λ\lambda-only, method one-hot + λ\lambda, and ΔB\Delta B-only baselines, the last of which requires no forward passes, using exactly the same data, prediction target, and leave-one-target-task-out protocol. As shown below, their Spearman correlations are 0.542, 0.658, and 0.093, respectively, all substantially lower than that of the three LFD diagnostics. More directly, adding the three diagnostics to the method + λ\lambda baseline increases Spearman correlation from 0.658 to 0.846 and pairwise method-ranking accuracy within the same task × seed × λ\lambda group from 66.56% to 83.85%. These results support the incremental predictive information provided by the diagnostics beyond the baselines using a linear merge-weight feature and method identity evaluated here.

PredictorsSpearmanPairwise
λ\lambda-only0.54250.00%
Method-Onehot+λ\lambda0.65866.56%
ΔB-only0.09356.01%
Three Factors0.85681.71%
Three Factors+Method-Onehot+λ\lambda0.84683.85%

W3 & Q3: Limited connection to merging performance. A smaller representation-gap bound does not directly imply higher merged accuracy. The relationship between the theoretical bound and actual performance remains unclear. How closely does the bound in Theorem 3.1 reflect the observed merged–LFD disagreement and merged accuracy? Does a smaller bound consistently lead to higher accuracy?

As discussed in the paper, the merged–LFD gap, ξmg−lfd\xi_{\rm mg-lfd}, cannot predict model accuracy on its own. This limitation is consistent with previous studies of linearized models [1,2]. We observe that both NTK fine-tuning during training and post-hoc sparsification of task vectors bring the merged model's behavior closer to its corresponding LFD form. However, both NTK fine-tuning and post-hoc sparsification reduce the target-task performance of the individual models before merging, thereby lowering the merged model's performance ceiling. To our knowledge, current deep learning research does not provide a quantitative estimate of this degradation. We have clarified this point in the updated manuscript.

Q2: How robust are the predictive results across different tasks, checkpoints, and random seeds?

Thank you for your attention to predictive robustness and incremental value. Because configurations share tasks, checkpoints, and masks, we added experiments that hold out entire groups to avoid treating correlated configurations as independent samples. All predictors use the same ridge regression setup, with standardization and parameter fitting performed only on the training data. The prediction target is the same as in Table 2: normalized accuracy gain relative to TA. The table below reports predictive results under different training–test splits. Each row defines a split; the last three columns specify the predictor's input features, rather than model-merging methods. The splits are applied to configuration records used by the predictor, while the set of merged tasks remains fixed. Values are Spearman correlations computed after concatenating predictions from all test folds; higher is better.

Test configurations in each foldRecords excluded from trainingλ\lambda, λ2\lambda^2, λ3\lambda^3 onlyThree LFD diagnostics onlyλ\lambda, λ2\lambda^2, λ3\lambda^3 + three LFD diagnostics
All configurations for one target taskAll records for that task0.5420.8560.866
All configurations for one training seedAll records for that seed0.5860.8730.887
All configurations for one transformation methodAll records for that method0.4690.8610.859
All configurations for one transformation familyAll records for all methods in that family0.5060.8710.863
Intersection of a specified task and a specified seedAll records for either that task or that seed0.5430.8560.867
Intersection of a specified task and a specified methodAll records for either that task or that method0.4370.8470.842

The three LFD diagnostics are changes in the two path disagreements and in NormAccuracy, all computed relative to TA. Transformation families were defined in advance as RAND/DARE, MIN/ELT/MEL/Breadcrumbs, and CPS/TALL to reduce the influence of closely related transformations remaining in the training set. For joint holdouts, the test set is the intersection of the two conditions, while the training set excludes their union. The table below also reports the incremental value of the path diagnostics over a stronger baseline. This baseline uses method one-hot features, λ\lambda, λ2\lambda^2, λ3\lambda^3, and their interactions; the two path diagnostics do not include NormAccuracy. The first three numerical columns report Spearman correlations, and the final column reports a paired interval for the improvement from adding the two path diagnostics to the stronger baseline.

Holdout splitStronger baselineStronger baseline + two path diagnosticsFurther adding NormAccuracy95% interval for the improvement from path diagnostics
Target-task holdout0.7840.9010.881[0.067, 0.168]
Training-seed holdout0.8200.9200.902[0.056, 0.150]

These results support relatively stable predictive associations across the evaluated seeds and show that the diagnostics retain incremental information for transformations excluded from fitting and under joint holdouts. The path diagnostics also provide information beyond the baseline using nonlinear functions of the merge weight and method identity evaluated here.

Q4: How can LFD diagnostics guide the selection of sparsification methods or merge weights without additional evaluation?

Thank you for raising this practical question. Using LFD diagnostics to guide the selection of sparsification methods or merge weights is a worthwhile direction for further investigation. At present, however, we cannot provide a reliable selection rule that requires no additional evaluation. Specifically, NormAccuracy requires labeled samples. Although the two path disagreements can be computed on unlabeled queries, they still require model forward passes and therefore incur additional evaluation costs. The current value of LFD lies primarily in helping us understand how sparsification affects path discrepancies and the retention of task capability. We will clarify this scope in the revised manuscript and leave evaluation-free method and weight selection for future work.

[1] Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models. NeurIPS 2023

[2] Mastering Task Arithmetic: τ\tauJp as a Key Indicator for Weight Disentanglement. NeurIPS 2024

Thank you for the response and additional experiments. I still believe the contributions are limited and maintain my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.