- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper compares merging two parent checkpoints against interpolating one towards the shared initialization. This matters when the initialization actually outperforms both parent checkpoints, because in such a case, simply reverting towards the initialization can get you a better model. The proposed method is to regress one parent checkpoint to the initialization by the same weight-space distance that the merge would have displaced the parent. A merge makes sense when it can outperform this regressed baseline, which is always ignored in the literature. Theory-wise, the authors show that every linear merge lies in the plane spanned by the two parent task vectors. Assuming performance varies linearly across that plane, a single inequality predicts which wins, using only the initialization, the two parents, and the task-vector geometry. Because task vectors are near-orthogonal in practice, the merge should rarely win when the base beats both parents.
Critical strengths
- Simple and cheap method. The method costs one interpolation and two evaluations and needs no training data
- Brings to attention a real evaluation gap. It points out that "beats both parents" is uninformative when the initialization checkpoint already beats both, and this is rarely checked in the literature.
- Principled choice of the interpolation magnitude. Matching the displacement on weight-space distance has a real justification, since equal step lengths cancel isotropic curvature.
- Good breadth. The evaluation is broad. It combines a synthetic testbed with known ground truth, six base models from three families, third-party checkpoints, and a reproduction of the standard CLIP benchmark.
- Honest scope. The paper explicitly states that the confound depends on the evaluation metric, not on merging itself, and reports the cases where its own control loses.
Critical weaknesses
- The theory part is heavy to read and to somehow oversold:
- Proposition 2 is barycentric interpolation, Corollary 4 is the parallelogram law, Proposition 5 is three lines of rearrangement, and Proposition 11 is the triangle inequality. Nothing is wrong, but perhaps most of these propositions and corollaries don’t need to have an explicit header.
- Terms like "master inequality," "exact", and "certified" overstate what's there.
- The theory takes several pages that could be better used to show additional evidence.
- Inadequate structure. The paper spends too much space on propositions and corollaries. That is not wrong by itself, but it hurts readability.
- The theory doesn't add predictive power. The simple exposure test (does the base beat the better parent?) is statistically indistinguishable from the full inequality (0.778 vs 0.861, with a large p=0.146).
- Assumption 1 fails where it matters, even though limitations acknowledge it. The linear-performance model holds on the synthetic testbed but not on benchmark accuracy (held-out R^2 = 0.35), yet the headline claims rest on benchmarks.
- Partly circular result. If the initialization checkpoint already beats both parents, then the fact that reverting towards it helps is pretty much definitional.
- Limited practical relevance. From appendices F and G, I see that every reported merging evaluation the authors check is unexposed (i.e., initialization does not outperform both parents), so no previously reported merging result is shown to have the issue of initialization beating both parents (the central issue this paper identifies).
- Missing references. Prior works [1-4] studying how to predict model merging outcomes before merging are not cited.
References
[1] Yadav et al., What Matters for Model Merging at Scale? (TMLR 2025)
[2] Hu et al., Exploring Model Kinship for Merging Large Language Models. (Findings of EMNLP 2025)
[3] Zhou et al., Demystifying Mergeability: Interpretable Properties to Predict Model Merging Success. (ICML 2026)
[4] Rahamim et al., Will it Merge? On The Causes of Model Mergeability. (Findings of ACL 2026)
Questions for authors
- Have you evaluated on tasks that benefit from positive transfer? I.e., where the merged model actually outperforms both parents because knowledge from both parents is complementary for the task.
- You used the weight-space L2 distance to measure the displacement. Have you tried other quantities that "regression to initialization" needs to match w.r.t. the merge? E.g., KL divergence from the parent's outputs on held-out data.
7 comments
Response to Reviewer nnze (2/2)
W6. Limited practical relevance: every published result you checked is unexposed. That was fair. Two of the three papers we checked do not report the original model's score on the combined metric, so the test cannot be applied to their tables. Your point stands: we have not shown that a result already published in a paper is affected. What we add is evidence that this situation is common in the standard evaluation itself. When we merge every pair of the eight CLIP experts from the task-arithmetic benchmark and test on all eight tasks, 119 of 224 cells are exposed, all on tasks neither expert was trained on, and on 117 of them the simple control beats the merge. A results table that compares these merges only with their parents would count most of them as merging gains, even though a single interpolation achieves the same.
W7. Missing references [1-4]. Thank you. All four are now cited. They ask a related but different question: will a merge succeed compared with its parents or a multitask model, and can model properties predict this? We ask what a merge should be compared against, and we predict the result of that comparison. Yadav et al. find that merging works better with strong base models. A strong base model is also what makes the problem we study appear, so the two findings fit together.
Q1. Positive transfer: tasks where the merge beats both parents because their skills combine? Yes, and our test correctly says the control is not needed there:
- On the standard 8-task CLIP benchmark, the merge beats every parent (0.645 vs 0.553) and its control (0.520). This setting is not exposed.
- In the pairwise CLIP grid, none of the 56 cells where an expert was trained on the task is exposed, and the merge beats the control on 54 of them.
- On our synthetic model, when each lineage learns new behaviours the original model lacks, the merge is on the best trade-off curve of the control in all six pairs.
- On public Llama-3.1-8B fine-tunes, a math × tool-use merge beats both parents on GSM8K, CRUXEval and MATH-500.
Q2. Matching on another quantity, such as KL divergence from the parent's outputs? Thank you; we ran exactly this, with pass thresholds set in advance. In theory, revised Proposition 4 works for any way of measuring distance: matching the step length cancels the part of the curvature that is the same in every direction under that distance, and KL matching is, to second order, matching in the Fisher metric. In practice, on the CLIP grid, for each pair and each parent, we picked the point on the path back to the original model whose output change (KL divergence from the parent, on held-out training images) equals the merge's. Then we repeated the whole comparison on all 224 cells:
- The weight-matched and output-matched controls pick the same winner on 209 of 224 cells.
- The output-matched control beats the merge on 118 of 119 exposed cells and on 10 of 105 others ().
So the conclusion does not depend on how distance is measured. We also found that the output-matched control moves further toward the original model (median coefficient 0.94, versus 0.68 for weight distance), because the merge changes the parent's predictions more than an equal weight step toward the original does. Matching on weight distance is therefore the more cautious choice for the control, and we keep it as the main definition.
Response to Author Rebuttal
Thank you for the detailed rebuttal, and for reporting the pre-set thresholds that were not met. Here is what I think is addressed, partially addressed, not addressed, and think might not be addressable in a rebuttal.
Addressed
The oversold terms (W1) are fixed: "master inequality" is now "decision inequality", the simple results are inline remarks, and the text says plainly that they are elementary. The worked example helps too. All four missing references (W7) are cited and positioned in Related Work. On positive transfer (Q1), I'm satisfied: the merge beats the control by 12.5 points on the 8-task CLIP benchmark and on most cells where a parent was trained on the task. And thanks for running the KL matching (Q2) on all 224 CLIP cells. It picks the same winner as weight matching on most of them. One thing I noticed is that the KL-matched coefficient has a median of 0.94, so this control is almost the pretrained model itself.
Partially addressed
On structure and space (W1, W2), Section 3 still takes about 2.5 of the 10 pages, and the revision actually adds a new assumption and proposition to it. The main text still has no results table, and the new experiments are squeezed into one paragraph of Section 7, with the details left to Appendix H. Tables are not mandatory, but they help the reader read comparisons.
On circularity (W5), the 15 exposed cells where the merge beats both parents, with the control winning 14 of them, is a good answer. But I don't agree that the merge "does not move toward the original model at all". When the task vectors are nearly orthogonal, averaging already pulls the merge about 30% closer to the initialization, and the matched control ends up even closer (about 0.3‖τ‖ away, against 0.7‖τ‖ for the merge). Your new paragraph in Section 3 also shows that on exposed cells the inequality always picks the control in your data, so under your own model, the control winning follows directly from exposure. For TSV-M, the control is the initialization itself in 84% of pairwise directions, so that comparison is really the pretrained model against a merge on tasks where the pretrained model already beats both experts.
On Assumption 1 (W4), separating measured from predicted claims is useful, and I agree the headline counts don't depend on the assumption. My remaining issue is that the sign agreement (0.82 and 0.90) includes the exposed cells, where the control is predicted to win anyway. It would be more informative on unexposed cells only.
Not addressed
Practical relevance (W6) is still unaddressed for me. The rebuttal agrees that no published result is shown to be affected, and the new CLIP grid doesn't change that: all 119 exposed cells are on tasks that neither expert was trained on, and none of the 56 cells where a parent was trained on the task is exposed. Merged models are normally evaluated on the tasks of the parent models, so I don't agree that this is "exactly how merged models are usually evaluated". Under the standard protocol, the confound simply doesn't show up on this benchmark. In the language-model grid, 14 of the 33 exposed cells are on a task the pair was trained on, which means the expert trained on that task is worse than the base model on it. That's not a setting where anyone would merge.
Might not be addressable
The new experiment confirms that the criterion adds no predictive power (W3). Where the two rules can differ, the inequality loses to the exposure test (15 vs 24 cells for pairs), and the paper now recommends the exposure test. Yet the criterion is still the title, contribution 1, and the abstract's "86% vs 32%", which compares it only with always merging and not with the exposure test. So the framing no longer matches what the paper recommends, and what is left is a baseline plus a check of whether the base model beats both parents.
Assumption 1 also has little support left beyond the measured sign agreement. The Section 5 sentence that justified it is gone, the condition of the proposition that replaces it (revised Proposition 2) is not met on your own backbone (0.47 against a threshold of 0.10), and held-out R² on benchmark accuracy is 0.35.
Some of my concerns are addressed, but the most fundamental ones are conceded rather than resolved, and the revised paper differs substantially from the submitted one.
Response to Reviewer nnze (1/2)
Thank you for a detailed and helpful review. We agree with most of your points and have revised the paper. On one point (W3), our new experiment supports your view, and we say so directly.
W1 and W2. The theory is heavy and oversold. Proposition 2 is barycentric interpolation, Corollary 4 is the parallelogram law, Proposition 5 is three lines of algebra, and Proposition 11 is the triangle inequality. Terms like "master inequality", "exact" and "certified" overstate things, and the structure hurts readability. We agree. Proposition 2, Corollary 4 and Proposition 11 are now short inline remarks, and "master inequality" is now "decision inequality". We removed "exact" and "certified" where they only added emphasis, and the text now says plainly that these results are simple. Their value is that they let you compute the comparison before evaluating anything, not that they are deep. The space saved, plus the extra page allowed in the revision, now holds the new experiments.
W3. The theory does not add predictive power: the simple exposure test is statistically no different from the full inequality (0.778 vs 0.861, p=0.146). Our new experiment supports your view. The revision now recommends the simple exposure test for practical use.
Why the simple test works. On an exposed cell, the original model beats both fine-tuned models. A short calculation shows that the full inequality must then pick the control whenever α⋆≥21 (for equal-sized task vectors, this means the cosine is at most 1/2, which is the threshold in Corollary 7). In all our data, α⋆ is at least 0.58. So on exposed cells the two rules always make the same choice: on all 33 grid cells and all 119 exposed CLIP cells. In other words, the exposure test is what the full rule reduces to in practice.
Where the two rules differ, the full rule is not better. They can only disagree on unexposed cells:
We can see why. When an expert was trained on the task, the merge almost always wins. But the linear approximation underestimates how well the merge does there, which pushes the full rule toward the control.
What the theory still adds. It explains why the simple test works (the threshold is never reached at the cosines we measure). It predicts the direction of the gap well: 0.82 on the grid and 0.90 on 448 CLIP comparisons, with Spearman 0.88. It also tells you when a predicted sign can be trusted. But we no longer claim that the full rule is a better decision rule than the simple test.
W4. Assumption 1 fails where it matters: held-out R2=0.35 on benchmark accuracy, yet the main claims are about benchmarks. We agree that the linear model does not predict accuracy levels well. We set up that test in advance, reported that it failed, and on benchmarks we only use the model to predict the sign of a difference. Our main benchmark claims do not depend on it. "The control beats the merge on 32 of 33 exposed grid cells and 117 of 119 exposed CLIP cells" compares models we actually evaluated. The revision now marks each claim as measured or predicted. Where we do use the model for signs, we measure how often it is right (0.82 on the grid, 0.90 on CLIP), and we report where it gets worse: merges of more experts, and the unexposed cells in W3.
W5. Partly circular: if the original model beats both parents, of course moving toward it helps. It is almost true by definition that moving a parent toward a better original model improves that parent. Section 8 says the existence of the effect is not surprising. But that is not what we test. We test whether the merge beats the moved parent when both move the same distance. The merge is a different model: it sits halfway between the two parents and does not move toward the original model at all. Being exposed does not decide this comparison, because the merge could gain from the other parent instead. Corollary 7 says exactly when that would happen. The data show the effect is real, not circular. In the 33 exposed grid cells, the merge beats both of its parents in 15, which is what merging papers count as success. In 14 of those 15, the simple control beats the merge anyway.