acceptodds
Under review as a conference paper at ICLR 2027

Predicting When Model Merging Beats Reverting to the Initialization

Abstract

Model merging is typically evaluated by whether the merged model outperforms the two fine-tuned models it combines. We show that this comparison is misleading whenever both fine-tuned models score below their shared initialization on the reported metric, as is common after self-training, continual fine-tuning, or fine-tuning for accuracy. Interpolating a single one of them back toward the initialization, by the same distance the merge moves, can then match or exceed the merge. We propose this interpolation, the matched regression-to-ancestor control, as a baseline that merging evaluations should report. We also derive a criterion that predicts which of the two will score higher from the initialization and the two fine-tuned models alone, before either candidate is evaluated. Both candidates lie in the plane spanned by the two task vectors, where a first-order approximation, which we test and whose error we bound, reduces the comparison to a single inequality with no fitted parameters. On a synthetic self-training testbed with exactly known structure, the criterion correctly predicts the better candidate on 28 of 28 lineage pairs. We further fine-tune three experts from each of six base models spanning three families and evaluate every pair on four benchmarks, for 72 cells in total. The control outperforms the merge in 97% of the cells in which the initialization outscores both experts, compared with 44% of the remaining cells. On third-party fine-tunes of Llama-3.1-8B, the control recovers a median 69% of the gain of a merge that outperforms both of its parents. The criterion does not uniformly favor the control: on the standard 8-task CLIP benchmark, which we reproduce, the best expert outscores the initialization, the criterion accordingly predicts no advantage for the control, and the merge outperforms it by 12.5 points. The control requires a single interpolation, and the criterion selects the better of the two candidates in 86% of cells, compared with 32% for the common practice of always selecting the merge. Code is available at https://anonymous.4open.science/r/merge-plane-geometry-4591.

Primary area:
generative models
Keywords:
model merging, task vectors, fine-tuning, self-training, continual learning, baseline evaluation, regression-to-ancestor control, CLIP benchmark, Llama-3.1-8B, weight averaging, model interpolation, behavior survival

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nnzeOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper compares merging two parent checkpoints against interpolating one towards the shared initialization. This matters when the initialization actually outperforms both parent checkpoints, because in such a case, simply reverting towards the initialization can get you a better model. The proposed method is to regress one parent checkpoint to the initialization by the same weight-space distance that the merge would have displaced the parent. A merge makes sense when it can outperform this regressed baseline, which is always ignored in the literature. Theory-wise, the authors show that every linear merge lies in the plane spanned by the two parent task vectors. Assuming performance varies linearly across that plane, a single inequality predicts which wins, using only the initialization, the two parents, and the task-vector geometry. Because task vectors are near-orthogonal in practice, the merge should rarely win when the base beats both parents.

Critical strengths
  • Simple and cheap method. The method costs one interpolation and two evaluations and needs no training data
  • Brings to attention a real evaluation gap. It points out that "beats both parents" is uninformative when the initialization checkpoint already beats both, and this is rarely checked in the literature.
  • Principled choice of the interpolation magnitude. Matching the displacement on weight-space distance has a real justification, since equal step lengths cancel isotropic curvature.
  • Good breadth. The evaluation is broad. It combines a synthetic testbed with known ground truth, six base models from three families, third-party checkpoints, and a reproduction of the standard CLIP benchmark.
  • Honest scope. The paper explicitly states that the confound depends on the evaluation metric, not on merging itself, and reports the cases where its own control loses.
Critical weaknesses
  • The theory part is heavy to read and to somehow oversold:
    • Proposition 2 is barycentric interpolation, Corollary 4 is the parallelogram law, Proposition 5 is three lines of rearrangement, and Proposition 11 is the triangle inequality. Nothing is wrong, but perhaps most of these propositions and corollaries don’t need to have an explicit header.
    • Terms like "master inequality," "exact", and "certified" overstate what's there.
    • The theory takes several pages that could be better used to show additional evidence.
  • Inadequate structure. The paper spends too much space on propositions and corollaries. That is not wrong by itself, but it hurts readability.
  • The theory doesn't add predictive power. The simple exposure test (does the base beat the better parent?) is statistically indistinguishable from the full inequality (0.778 vs 0.861, with a large p=0.146).
  • Assumption 1 fails where it matters, even though limitations acknowledge it. The linear-performance model holds on the synthetic testbed but not on benchmark accuracy (held-out R^2 = 0.35), yet the headline claims rest on benchmarks.
  • Partly circular result. If the initialization checkpoint already beats both parents, then the fact that reverting towards it helps is pretty much definitional.
  • Limited practical relevance. From appendices F and G, I see that every reported merging evaluation the authors check is unexposed (i.e., initialization does not outperform both parents), so no previously reported merging result is shown to have the issue of initialization beating both parents (the central issue this paper identifies).
  • Missing references. Prior works [1-4] studying how to predict model merging outcomes before merging are not cited.

References

[1] Yadav et al., What Matters for Model Merging at Scale? (TMLR 2025)

[2] Hu et al., Exploring Model Kinship for Merging Large Language Models. (Findings of EMNLP 2025)

[3] Zhou et al., Demystifying Mergeability: Interpretable Properties to Predict Model Merging Success. (ICML 2026)

[4] Rahamim et al., Will it Merge? On The Causes of Model Mergeability. (Findings of ACL 2026)

Questions for authors
  • Have you evaluated on tasks that benefit from positive transfer? I.e., where the merged model actually outperforms both parents because knowledge from both parents is complementary for the task.
  • You used the weight-space L2 distance to measure the displacement. Have you tried other quantities that "regression to initialization" needs to match w.r.t. the merge? E.g., KL divergence from the parent's outputs on held-out data.
7 comments
AuthorsOpenReview ↗

Response to Reviewer nnze (1/2)

Thank you for a detailed and helpful review. We agree with most of your points and have revised the paper. On one point (W3), our new experiment supports your view, and we say so directly.

W1 and W2. The theory is heavy and oversold. Proposition 2 is barycentric interpolation, Corollary 4 is the parallelogram law, Proposition 5 is three lines of algebra, and Proposition 11 is the triangle inequality. Terms like "master inequality", "exact" and "certified" overstate things, and the structure hurts readability. We agree. Proposition 2, Corollary 4 and Proposition 11 are now short inline remarks, and "master inequality" is now "decision inequality". We removed "exact" and "certified" where they only added emphasis, and the text now says plainly that these results are simple. Their value is that they let you compute the comparison before evaluating anything, not that they are deep. The space saved, plus the extra page allowed in the revision, now holds the new experiments.

W3. The theory does not add predictive power: the simple exposure test is statistically no different from the full inequality (0.778 vs 0.861, p=0.146p=0.146). Our new experiment supports your view. The revision now recommends the simple exposure test for practical use.

Why the simple test works. On an exposed cell, the original model beats both fine-tuned models. A short calculation shows that the full inequality must then pick the control whenever α⋆≥12\alpha^\star\ge\tfrac12 (for equal-sized task vectors, this means the cosine is at most 1/2, which is the threshold in Corollary 7). In all our data, α⋆\alpha^\star is at least 0.58. So on exposed cells the two rules always make the same choice: on all 33 grid cells and all 119 exposed CLIP cells. In other words, the exposure test is what the full rule reduces to in practice.

Where the two rules differ, the full rule is not better. They can only disagree on unexposed cells:

  • On the 39 unexposed grid cells, the full rule is right on 9 cells where the simple test is wrong, and wrong on 3 (not a significant difference).
  • On the 105 unexposed cells of the new CLIP grid, which we set up specifically to test this, the simple test does better: 24 cells against 15 (p=0.20p=0.20).
  • For merges of 3, 4 and 6 experts, the simple test is clearly better (p=0.004p=0.004, 1.1×10−41.1\times10^{-4} and 5.4×10−75.4\times10^{-7}).

We can see why. When an expert was trained on the task, the merge almost always wins. But the linear approximation underestimates how well the merge does there, which pushes the full rule toward the control.

What the theory still adds. It explains why the simple test works (the threshold is never reached at the cosines we measure). It predicts the direction of the gap well: 0.82 on the grid and 0.90 on 448 CLIP comparisons, with Spearman 0.88. It also tells you when a predicted sign can be trusted. But we no longer claim that the full rule is a better decision rule than the simple test.

W4. Assumption 1 fails where it matters: held-out R2=0.35R^2=0.35 on benchmark accuracy, yet the main claims are about benchmarks. We agree that the linear model does not predict accuracy levels well. We set up that test in advance, reported that it failed, and on benchmarks we only use the model to predict the sign of a difference. Our main benchmark claims do not depend on it. "The control beats the merge on 32 of 33 exposed grid cells and 117 of 119 exposed CLIP cells" compares models we actually evaluated. The revision now marks each claim as measured or predicted. Where we do use the model for signs, we measure how often it is right (0.82 on the grid, 0.90 on CLIP), and we report where it gets worse: merges of more experts, and the unexposed cells in W3.

W5. Partly circular: if the original model beats both parents, of course moving toward it helps. It is almost true by definition that moving a parent toward a better original model improves that parent. Section 8 says the existence of the effect is not surprising. But that is not what we test. We test whether the merge beats the moved parent when both move the same distance. The merge is a different model: it sits halfway between the two parents and does not move toward the original model at all. Being exposed does not decide this comparison, because the merge could gain from the other parent instead. Corollary 7 says exactly when that would happen. The data show the effect is real, not circular. In the 33 exposed grid cells, the merge beats both of its parents in 15, which is what merging papers count as success. In 14 of those 15, the simple control beats the merge anyway.

AuthorsOpenReview ↗

Response to Reviewer nnze (2/2)

W6. Limited practical relevance: every published result you checked is unexposed. That was fair. Two of the three papers we checked do not report the original model's score on the combined metric, so the test cannot be applied to their tables. Your point stands: we have not shown that a result already published in a paper is affected. What we add is evidence that this situation is common in the standard evaluation itself. When we merge every pair of the eight CLIP experts from the task-arithmetic benchmark and test on all eight tasks, 119 of 224 cells are exposed, all on tasks neither expert was trained on, and on 117 of them the simple control beats the merge. A results table that compares these merges only with their parents would count most of them as merging gains, even though a single interpolation achieves the same.

W7. Missing references [1-4]. Thank you. All four are now cited. They ask a related but different question: will a merge succeed compared with its parents or a multitask model, and can model properties predict this? We ask what a merge should be compared against, and we predict the result of that comparison. Yadav et al. find that merging works better with strong base models. A strong base model is also what makes the problem we study appear, so the two findings fit together.

Q1. Positive transfer: tasks where the merge beats both parents because their skills combine? Yes, and our test correctly says the control is not needed there:

  • On the standard 8-task CLIP benchmark, the merge beats every parent (0.645 vs 0.553) and its control (0.520). This setting is not exposed.
  • In the pairwise CLIP grid, none of the 56 cells where an expert was trained on the task is exposed, and the merge beats the control on 54 of them.
  • On our synthetic model, when each lineage learns new behaviours the original model lacks, the merge is on the best trade-off curve of the control in all six pairs.
  • On public Llama-3.1-8B fine-tunes, a math × tool-use merge beats both parents on GSM8K, CRUXEval and MATH-500.

Q2. Matching on another quantity, such as KL divergence from the parent's outputs? Thank you; we ran exactly this, with pass thresholds set in advance. In theory, revised Proposition 4 works for any way of measuring distance: matching the step length cancels the part of the curvature that is the same in every direction under that distance, and KL matching is, to second order, matching in the Fisher metric. In practice, on the CLIP grid, for each pair and each parent, we picked the point on the path back to the original model whose output change (KL divergence from the parent, on held-out training images) equals the merge's. Then we repeated the whole comparison on all 224 cells:

  • The weight-matched and output-matched controls pick the same winner on 209 of 224 cells.
  • The output-matched control beats the merge on 118 of 119 exposed cells and on 10 of 105 others (p=2.3×10−50p=2.3\times10^{-50}).

So the conclusion does not depend on how distance is measured. We also found that the output-matched control moves further toward the original model (median coefficient 0.94, versus 0.68 for weight distance), because the merge changes the parent's predictions more than an equal weight step toward the original does. Matching on weight distance is therefore the more cautious choice for the control, and we keep it as the main definition.

Reviewer_nnzeOpenReview ↗

Response to Author Rebuttal

Thank you for the detailed rebuttal, and for reporting the pre-set thresholds that were not met. Here is what I think is addressed, partially addressed, not addressed, and think might not be addressable in a rebuttal.


Addressed

The oversold terms (W1) are fixed: "master inequality" is now "decision inequality", the simple results are inline remarks, and the text says plainly that they are elementary. The worked example helps too. All four missing references (W7) are cited and positioned in Related Work. On positive transfer (Q1), I'm satisfied: the merge beats the control by 12.5 points on the 8-task CLIP benchmark and on most cells where a parent was trained on the task. And thanks for running the KL matching (Q2) on all 224 CLIP cells. It picks the same winner as weight matching on most of them. One thing I noticed is that the KL-matched coefficient has a median of 0.94, so this control is almost the pretrained model itself.


Partially addressed

On structure and space (W1, W2), Section 3 still takes about 2.5 of the 10 pages, and the revision actually adds a new assumption and proposition to it. The main text still has no results table, and the new experiments are squeezed into one paragraph of Section 7, with the details left to Appendix H. Tables are not mandatory, but they help the reader read comparisons.

On circularity (W5), the 15 exposed cells where the merge beats both parents, with the control winning 14 of them, is a good answer. But I don't agree that the merge "does not move toward the original model at all". When the task vectors are nearly orthogonal, averaging already pulls the merge about 30% closer to the initialization, and the matched control ends up even closer (about 0.3‖τ‖ away, against 0.7‖τ‖ for the merge). Your new paragraph in Section 3 also shows that on exposed cells the inequality always picks the control in your data, so under your own model, the control winning follows directly from exposure. For TSV-M, the control is the initialization itself in 84% of pairwise directions, so that comparison is really the pretrained model against a merge on tasks where the pretrained model already beats both experts.

On Assumption 1 (W4), separating measured from predicted claims is useful, and I agree the headline counts don't depend on the assumption. My remaining issue is that the sign agreement (0.82 and 0.90) includes the exposed cells, where the control is predicted to win anyway. It would be more informative on unexposed cells only.


Not addressed

Practical relevance (W6) is still unaddressed for me. The rebuttal agrees that no published result is shown to be affected, and the new CLIP grid doesn't change that: all 119 exposed cells are on tasks that neither expert was trained on, and none of the 56 cells where a parent was trained on the task is exposed. Merged models are normally evaluated on the tasks of the parent models, so I don't agree that this is "exactly how merged models are usually evaluated". Under the standard protocol, the confound simply doesn't show up on this benchmark. In the language-model grid, 14 of the 33 exposed cells are on a task the pair was trained on, which means the expert trained on that task is worse than the base model on it. That's not a setting where anyone would merge.


Might not be addressable

The new experiment confirms that the criterion adds no predictive power (W3). Where the two rules can differ, the inequality loses to the exposure test (15 vs 24 cells for pairs), and the paper now recommends the exposure test. Yet the criterion is still the title, contribution 1, and the abstract's "86% vs 32%", which compares it only with always merging and not with the exposure test. So the framing no longer matches what the paper recommends, and what is left is a baseline plus a check of whether the base model beats both parents.

Assumption 1 also has little support left beyond the measured sign agreement. The Section 5 sentence that justified it is gone, the condition of the proposition that replaces it (revised Proposition 2) is not met on your own backbone (0.47 against a threshold of 0.10), and held-out R² on benchmark accuracy is 0.35.


Some of my concerns are addressed, but the most fundamental ones are conceded rather than resolved, and the revised paper differs substantially from the submitted one.

Reviewer_jEo7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether improvements from merging fine-tuned models reflect useful recombination or merely recovery toward their shared initialization. It introduces a distance-matched interpolation toward the initialization and derives a rule for predicting whether merging will outperform this control. Experiments on synthetic model collapse, language models, and a vision benchmark show that the control often explains substantial merging gains, while also identifying settings where merging remains beneficial.

Critical strengths
  • Comparing a merge with a parent interpolated toward the initialization helps assess whether its gains require combining information from different fine-tunes.
  • The experiments cover controlled synthetic settings, several language-model families, third-party checkpoints, and a standard vision benchmark.
Critical weaknesses
  • A network obtained by interpolating weights need not produce that mixture. Appendix D.1 explicitly acknowledges this distinction and reports that only approximately 56% of individual observations lie within the proposed bracket. Nevertheless, Section 5 describes the failed approximation gate as a bias that the theory predicts and bounds, and claims that subsequent predictions systematically overstate the merge’s disadvantage. These conclusions do not follow from the proposition.
Questions for authors
  • What assumption connects the output-mixture quantity in Proposition 3 to the actual weight-interpolated network?
  • Please clarify the evaluation populations underlying the reported sign-agreement and model-selection results. Table 2 reports 60 constructed-grid cells, whereas the main text reports 72; which third-party results and generation budgets enter each pooled statistic?
3 comments
AuthorsOpenReview ↗

Response to Reviewer jEo7 (1/2)

Thank you for a very precise review. Both of your technical points are correct, and fixing them improved the paper.

W1. Section 5 calls the failed check "a bias the theory predicts and bounds", but Proposition 3 is about an output mixture, only about 56% of observations fall inside the bound, and the conclusions do not follow. We agree and have removed that sentence, along with the claim that every later prediction is cautious. Proposition 3 is about a mixture of the three models' outputs, and a model made by averaging weights is not such a mixture. The 56% figure in Appendix D shows that this bound does not hold point by point for real merged models. The revision describes the observed direction of the error only as an empirical fact, and moves Proposition 3 to the appendix.

Q1. What assumption connects the output-mixture quantity in Proposition 3 to the actual weight-interpolated network? In the submission there was none; the connection was only empirical. The revision adds a condition and a result that apply directly to the weight-averaged model.

The condition. The model's output scores (logits) change linearly as we move across the plane spanned by the two task vectors: zt(u,v)=(1−u−v)zt0+u zta+v ztb+et(u,v)z_t(u,v)=(1-u-v)z^0_t+u\,z^a_t+v\,z^b_t+e_t(u,v), where the error ete_t is at most εt\varepsilon_t. This is the "linearized" regime often used to explain task arithmetic.

The result (revised Proposition 2). For any fixed sequence xx with log-probability ℓx\ell_x, and any point inside the triangle, −2∑tεt  ≤  ℓx(w)−∑kwk ℓx(k)  ≤  18∑t∑kwk osc(ztk−zˉt)2+2∑tεt.-2\textstyle\sum_t\varepsilon_t\;\le\;\ell_x(w)-\sum_k w_k\,\ell_x(k)\;\le\;\tfrac18\sum_t\sum_k w_k\,\mathrm{osc}\big(z^k_t-\bar z_t\big)^2+2\textstyle\sum_t\varepsilon_t . In words: if the logits are exactly linear (ε=0\varepsilon=0), the log-probability is concave across the plane. The simple linear prediction can then only underestimate the weight-averaged model, and by no more than an amount we can compute from the three models alone. Any departure from linearity widens the bound by at most 2εt2\varepsilon_t per position. When a behaviour is made up of several sequences, one extra term appears, and the revision states that case separately.

What we found when we checked. A unit test on random models with exactly linear logits shows no violations in 10,000 trials, and the upper bound is reasonably tight (typically about 4 times the true gap). We then measured whether our synthetic model meets the condition, fixing the pass threshold in advance. The theorem holds on every sequence, as it must. But the condition does not hold: the logits are far from linear (median relative error 0.47, where our threshold was 0.10). If we ignore that error, the bound contains 70% of sequences, and every sequence outside it has an error in the predicted direction.

So the direct answer to your question is: the link holds when the logits are close to linear, and our synthetic model is not in that regime. The revision says this plainly. The one-directional error we see on that model remains an observation that the theorem does not explain, and no claim in the paper relies on it.

Q2. Which cells, third-party results and generation budgets go into each pooled statistic? Table 2 shows 60 grid cells, but the text says 72. Thank you for catching this; Table 2 was wrong. The script that builds the table used the short model name as a key. Our own Qwen2.5-7B grid row was therefore overwritten by the third-party Qwen2.5-7B row, which dropped 12 cells. The script also read the Llama third-party row from the older 256-token run that the paper had replaced. The corrected table now matches the text: 72 cells, 33 exposed, and the control wins 32 of 33 exposed and 17 of 39 other cells. The Llama-3.1-8B row (1,024 tokens) has 7 exposed cells (the control wins 4) and 2 other cells, and is kept separate from the grid. Figure 2 had the same problem and has been redrawn.

While checking every statistic for this kind of problem, we found one more. The sign-condition analysis in Appendix A had mixed an older five-model grid with both third-party sets at 256 tokens (156 comparisons in total). We reran it on the 72-cell grid only (136 comparisons; exact ties left out, as in our other sign statistics). How often the predicted sign is right, by quartile of the predicted gap: 0.65, 0.76, 0.91 and 0.97 (Spearman 0.34, p=5.4×10−5p=5.4\times10^{-5}). Abstaining when the predicted gap is small answers 42% of comparisons with 92% accuracy. If ties are counted as errors instead, the quartiles are 0.58, 0.83, 0.83 and 0.97; the revision reports both. The conclusion is unchanged.

AuthorsOpenReview ↗

Response to Reviewer jEo7 (2/2)

The revision adds this table (Appendix H):

StatisticUnitNWhat is includedToken budget
How often exposed; control vs merge (Section 7)cell72our grid, 6 base models256
Decision rules (Table 5)cell72same256
Sign agreementmodel direction136same; 144 minus 8 exact ties256
Predicted share of gain absorbedmodel direction66where a merge gain is both predicted and measured256
Sign condition and abstentionmodel direction136our grid only (was 156)256
Llama-3.1-8B third-partycell9public fine-tunes; reported separately, never pooled1,024
Llama-3.1-8B share absorbed (the 69% in the abstract)model direction6where the merge beats both parents1,024
Qwen2.5-7B third-partycell9appendix only, limited by token budget256
Pairwise CLIP (new)cell22428 expert pairs × 8 tasksn/a
3-, 4- and 6-expert CLIP (new)cell80 each10 groups × 8 tasksn/a

Thank you for sharing the additional results! My concerns have been sufficiently addressed, and I maintain my weak acceptance recommendation.

Reviewer_7wCuOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Model merginig is a popular technique to merge multiple fine-tuned models. In this work, authors argue that the current evaluation setup for merging methods is incorrect. If the pre-trained model is better at the given task than the fine-tuned models, then model merging is only reverting back to the initialization. They study in a simplified setting with one-pretrained model and two fine tuned models

Critical strengths

The premise of the paper is interesting, and the method may serve as a useful baseline but the work is not coherently written and has several limitations (discussed below).

Critical weaknesses
  1. The paper is hard to follow, in particular section 3 in it's entirety.

  2. The setting the paper studies assume that the pretrained model is better at the target than the fine-tuned. This is not case usually.

  3. Some of the variables used in section are not defined: eg. gg in assumption.

  4. it is not clear how this setup will extend or even hold true for multi model settings.

  5. The authors explaination in sec 5 on assumption 1 is not clear. Since the paper analysis is based on the validity of assumption 1, it makes the validity of the subsequent analysis questionable.

The missed gate is therefore not an unexplained defect but a censoring bias the theory predicts and bounds: Assumption 1 is accurate on the quantity it is stated about and conservative at the threshold by a computable margin, so every prediction below overstates the merge’s disadvantage.

  1. If the authors want to claim that undoing finetuning is better than most merging mehtods, then the authors should present a comparsion with the sota methods, no comparison has been presented.
Questions for authors
  1. Would you say your method is better than existing merging methods? if so, what is the best evidence you have presented in the paper?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 7wCu (1/2)

Thank you for your review. We reply to each point below. We agree with your points on readability and notation and have fixed them. For the other points, we show where the paper already covers them and add new evidence.

What the paper is about, in short. Merging papers usually call a merge a success when it beats the two models it was made from. We show that this test is misleading when both of those models are worse than the original pretrained model on the reported metric. In that case, simply moving one model back toward the original model, by the same distance the merge moves, often does just as well or better. We call this the matched control and argue that merging papers should report it. We also give a simple test, computed before any merging, that tells you when the control is needed and which of the two will win. We measured how often this situation happens on our own fine-tuned models, on public third-party models, and on the standard CLIP merging benchmark.

Q1. "Would you say your method is better than existing merging methods?" No, and the paper does not claim that. The control is a baseline, not a new merging method. It answers one question: is the merge's gain real, or could you get the same gain by just moving one model back toward the original? The answer depends on the setting, and we show both cases:

  • When both models are worse than the original on the reported metric, the control beats the merge on 32 of 33 cells of our language-model grid and on 117 of 119 cells of the CLIP grid. The same is true for TSV-M, a recent strong method: the control beats it on 113 of 119 such cells.
  • When this is not the case, merging wins, and our test says so. On the standard 8-task CLIP benchmark, the merge beats the control by 12.5 points, and TSV-M beats its control by 35.4 points. Where an expert was trained on the task, the merge beats the control on 54 of 56 cells.

W6. "No comparison [with SOTA methods] has been presented." The submission does include one: Table 3 (Appendix D) compares seven merging methods on our synthetic model (simple averaging, SLERP, TIES, DARE, DARE-TIES, Fisher merging, and tuned-coefficient merging). All seven lose to the control on 28 of 28 pairs (21 of 28 for the tuned version), and this repeats on a second, independently trained copy of the model.

We agree this should go beyond a synthetic setting. The revision adds TSV-M (Gargiulo et al., CVPR 2025), a recent state-of-the-art method, on the CLIP benchmark:

  • Our implementation scores 0.834 on the standard 8-task benchmark. The authors report 0.859 with their own experts; our simple average is similarly a little lower than theirs (0.645 vs 0.663).
  • We compared TSV-M with a control that moves the same distance TSV-M moves. On the 8-task benchmark, TSV-M wins clearly (0.834 vs 0.480).
  • On the 119 exposed cells, the control beats TSV-M on 113. On the other 105 cells, it beats TSV-M on only 11.
  • TSV-M moves each model further than that model's own distance to the original, so its matched control is usually the original model itself. On exposed cells, even this strong method does not beat the original model.

W2. "The setting assumes that the pretrained model is better at the target than the fine-tuned. This is not the case usually." The paper does not assume this. It gives a test for it (is the original model better than both fine-tuned models on the reported metric?) and studies both cases. The key point is which metric is reported. A fine-tuned model usually gets better at its own task. But merged models are tested on all the tasks in the merge, and on the other tasks a fine-tuned model is often worse than the original. That is why this case is common:

  • In our grid of standard fine-tuned instruct models, 33 of 72 cells are exposed. 14 of these are even on a task the pair was trained on.
  • In the standard CLIP benchmark, merged in pairs, 119 of 224 cells are exposed, all on tasks neither expert was trained on.

W4. "Not clear how this setup will extend or even hold true for multi model settings." The paper covers more than two models. Section 3 ("Any number of parents") gives the formulas for kk models, with proofs in Appendix A, and Section 7 tests k=3k=3 on language models. The revision adds CLIP merges of 3, 4 and 6 experts (10 random groups each):

  • The main result holds for every kk. On exposed cells the control beats the merge on 32 of 32, 21 of 22 and 8 of 8 cells. On the other cells it does so on only 4 of 48, 5 of 58 and 6 of 72.
  • Our prediction of who wins gets less accurate as kk grows (it is right 85%, 74% and 63% of the time). We report this as a limitation.
AuthorsOpenReview ↗

Response to Reviewer 7wCu (2/2)

W5. Section 5's explanation of Assumption 1 is unclear, and the quoted sentence makes the later analysis questionable. You are right about the quoted sentence. It claimed more than Proposition 3 shows, and we removed it (please see our reply to Reviewer jEo7 for what replaces it). However, our main findings do not depend on Assumption 1. "The control beats the merge on 32 of 33 exposed grid cells and 117 of 119 exposed CLIP cells" is a direct comparison of models we evaluated. The assumption is only used to predict that result in advance, and we measure how good those predictions are. The revision now marks each claim as measured or predicted.

W1 and W3. Section 3 is hard to follow; gg is not defined. We agree. gig_i is the gradient of the margin sis_i on the plane. It is now defined where it first appears, and we checked every other symbol in Sections 3 to 5 the same way. Section 3 has been rewritten as described in point 1 of the general response, including a worked example.

Thank you for providing additional results! I agree with reviewer nnze; the paper has changed substantially and require a fresh assessment.

Thank you for checking the new results and for your feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers and the Area Chair

Thank you all for the careful reviews. We have uploaded a revised PDF. In our replies, theorem and section numbers refer to the submitted version unless we write "revised". Section 3 was renumbered: submitted Propositions 5, 6 and 8 and Corollary 7 are now Propositions 3, 4 and 6 and Corollary 5. All new results are in the new Appendix H.

Three concerns came up in more than one review. We address them first.

1. The theory section was hard to read and made some results sound bigger than they are. We agree. We rewrote Section 3:

  • Only the results the rest of the paper uses keep a numbered statement.
  • Simple results (the barycentric identity, the "parallelogram" identity, the sign condition) are now short inline remarks, and we say plainly that they are simple.
  • "Master inequality" is now "decision inequality". We removed "exact" and "certified" where they only added emphasis.
  • Every symbol is defined when it first appears, including gig_i (the gradient of the margin on the plane).
  • We added a worked example: using only three accuracies from Figure 1, the rule picks the winner before either model is evaluated.

2. A sentence in Section 5 overclaimed what the theory shows. All three reviewers pointed to it, and Reviewer jEo7 explained why it is wrong: Proposition 3 is about a mixture of the models' outputs, not about the model you get by averaging weights. We removed the sentence. In its place we prove a result about the weight-averaged model itself, and we also measured whether our model meets its condition (see the reply to jEo7). We now also separate two kinds of claim in the paper:

  • Measured claims compare two models we actually evaluated. They need no assumption. The main finding is one of them: the control beats the merge on 32 of 33 exposed cells.
  • Predicted claims use the approximation in Assumption 1.

3. Is this situation common in practice? To check, we merged every pair of the eight CLIP image experts from the standard task-arithmetic benchmark and tested each merge on all eight tasks. That gives 224 results ("cells").

  • 119 of the 224 cells are exposed (the original pretrained model beats both experts). All of these are on tasks that neither expert was trained on, which is exactly how merged models are usually evaluated.
  • On those 119 cells, the simple control beats the merge on 117. On the other 105 cells, it beats the merge on only 16.
  • Where one of the experts was trained on the task, the merge beats the control on 54 of 56 cells. So the control does not count against merging in general.
  • A recent, strong merging method, TSV-M (Gargiulo et al., CVPR 2025), behaves the same way: the control beats it on 113 of the 119 exposed cells, while on the standard 8-task benchmark TSV-M beats its control by 35.4 points.

Two errors we found and fixed.

  • Table 2 was wrong. The script that builds it mixed up two rows with the same model name, so our own Qwen2.5-7B rows were replaced by a different experiment's row. The table showed 60 cells instead of 72, and it used an older setting (256 tokens) for the Llama results. The numbers in the text were correct; the table was not. It is now fixed.
  • One appendix analysis (the "sign condition") used the wrong set of cells. We reran it on the correct 72 cells. The conclusion did not change: the prediction is right more often when the predicted gap is larger, rising from 65% to 97% across the four quartiles.

The revision also adds a table that lists exactly which cells go into each reported statistic.

New experiments. We wrote down each new experiment's goal and pass/fail thresholds in our project log before running it, and we report every result, including the ones that did not pass:

  • (a) the pairwise CLIP grid above;
  • (b) merges of 3, 4 and 6 CLIP experts;
  • (c) TSV-M, compared with a control matched to its own distance;
  • (d) a control matched on how much the model's outputs change (KL divergence), instead of how far its weights move;
  • (e) a new theorem about the weight-averaged model, with tests, plus a check of whether our model meets its condition.

Three of our pre-set thresholds were not met, and the paper says so:

  • For merges of 4 and 6 experts, our prediction picks the right sign less often (0.74 and 0.63, below our 0.75 target).
  • Our full rule does not beat the simpler "exposure test" on unexposed cells (reply to nnze, W3).
  • Our synthetic model does not meet the condition of the new theorem (reply to jEo7, Q1).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.