acceptodds
Withdrawn from ICLR 2027

Which Task Loses? Understanding Asymmetrical Degradation in Model Merging

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

How many tasks can a merged model retain, and which task is most likely to suffer? We study merging as task-vector superposition and derive two-sided capacity bounds. Under a curvature assumption along the merge path, a stationary expert's loss increase lies between curvature multiples of its interferers' energy. The upper bound limits capacity in the task count ; the lower bound quantifies unavoidable damage. For two tasks, the smaller-vector task loses more whenever the squared norm ratio exceeds the curvature ratio. All 11 Llama pairs satisfying this condition under sampled, uncertified curvature ranges follow the predicted ordering. More broadly, the smaller-norm task's loss rises more in 91% of 58 full-data task-arithmetic pairs. Across 486 directional evaluations spanning four 7B families and three methods, within-merge contrasts account for 64.1% of degradation variance, which symmetric mergeability scores cannot explain; the smaller-norm task loses more in 83.5% of comparisons. Retraining links this asymmetry to exposure: norms grow with exposure, while matching exposure across tasks (three seeds) compresses their ratios and removes the ordering signal (51.8%). On four pairs, shrinking the larger vector protects the smaller task better than uniform scaling. Equalizing norms at fixed total norm instead stretches the smaller vector and harms 13 of 16 targets.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
model merging, task arithmetic, task vectors, capacity bounds, task interference, training exposure

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_pUQFOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates which task is more severely degraded when two or more fine-tuned experts are merged, rather than how much the merged model degrades on average. The theoretical analysis applies a second-order Taylor expansion along the segment from each expert to the merged model. This yields a two-sided bound on each expert’s loss increase, expressed in terms of the interferer energy AiA_i and a coherence bracket. The empirical study covers eight GLUE tasks, LoRA experts from four ~7B model families, and TA, TIES, and DARE merges, for a total of 486 directional evaluations. Finally, the authors intervene on training exposure and on coefficient allocation. These interventions show that both the task-vector norm and the partners’ degradation increase monotonically with training-set size. They also show that shrinking only the larger vector protects the smaller-norm task better than scaling both vectors uniformly, although sometimes at the expense of the larger-norm task.

Critical strengths
  1. Careful statistical treatment. Because experts are shared across pairs, the observations are not independent. The authors account for this dependence through shared-expert (dyadic) standard errors, expert-level bootstraps, and exact permutation tests over all norm assignments. This is an appropriate and commendable choice.
  2. A meaningful baseline. Comparing the task-vector norm against training-set size and optimizer step count is a valuable control. The finding that the norm carries no predictive information beyond training exposure is useful for the community.
Critical weaknesses
  1. Presentation and readability. The manuscript is difficult to follow:
    • Several paragraphs are dense with formulas, which makes them very hard to read.
    • The notation is overloaded.
    • Many paragraphs read as experimental logs, with long chains of numerical results and frequent references to the appendix.
    • The text contains unusual formatting choices, such as chained arrows to report numerical series (e.g., “11 → 27 → 69 → 302”). This style is often typical of AI-generated text.The use of AI tools is not a concern in itself, and the authors disclose it. My concern is whether the generated text has been sufficiently proofread and edited for a human reader.
  2. Limited novelty. The contribution appears incremental with respect to prior work, in particular Section 6.1 of DisTaC (Yoshida et al., 2026). That section already establishes that norm imbalance between task vectors drives the asymmetric degradation, and traces it to differences in training steps and learning rate.
  3. Limited experimental scope. The main evidence comes from GLUE, which is a relatively easy benchmark for 7B models. The standard vision merging benchmarks (e.g., ViT models on the 8/14/20-task suites) are absent. The generative MergeBench experts appear only in an appendix check (Appendix G), and there the proposed ordering rules do not transfer. This raises doubts about how general the conclusions are.
Questions for authors
  1. Could the authors carefully revise the manuscript for readability? Specifically, reducing the density of numerical results in the main text, simplifying the notation, and moving secondary statistics to tables or the appendix would make the paper substantially more accessible.
  2. The degradation analysis aggregates heterogeneous GLUE metrics (accuracy, F1, Matthews correlation, and Pearson correlation), which are not directly comparable in magnitude. For instance, a drop of a few points in accuracy is not equivalent to a drop of the same size in F1 or MCC. How does the analysis account for these differences, and how sensitive are the main conclusions to them?
  3. Beyond the role of norm imbalance, which is already documented in prior work, what do the authors consider the central novel contribution of this paper?
  4. What distinguishes the settings in which the norm-based ordering holds, such as full-data GLUE experts, from those in which it fails, such as exposure-matched experts and the MergeBench generative experts?
Reviewer_1e71OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper derives task-wise loss bounds for task-vector superposition using interferer energy, coherence, and curvature along the merge path. For two stationary experts, a sufficiently large norm ratio predicts greater loss increase for the smaller-vector task. Experiments examine this ordering across model families and merging methods, retrain experts under matched exposure, and compare coefficient interventions. The findings link norm imbalance to training exposure while showing that naive norm equalization can damage retention.

Critical strengths
  1. The question is clearly defined and scientifically useful. The mean/contrast decomposition in Section 4 demonstrates why pair-average mergeability cannot identify the more vulnerable task. Evaluating both targets is an appropriate response.

  2. The study includes substantive controls. Matched-exposure retraining with three seeds, dose-response experiments, held-group prediction, and shared-expert uncertainty estimates go well beyond a simple correlation between norm and degradation.

  3. The paper states important negative findings transparently. Norm does not outperform the exposure baseline, loses reliable ordering power after exposure matching, and fails to transfer to public generative experts in Appendix G. These qualifications make the diagnostic contribution more credible.

Critical weaknesses
  1. The theory is conditionally sound but has limited predictive force for the actual experts. Corollary 1 requires stationarity and positive pathwise curvature. Yet Section 5.1 reports nonpositive loss changes for 29 of 116 targets, and Appendix J's sampled curvature ranges are not certified. Moreover, defining effective curvature from observed loss changes makes Eq. (2) an identity rather than an independent prediction. The results support an empirical scale association more strongly than a practical capacity bound or a generally usable loser predictor.

  2. The strongest empirical rule is restricted to a narrow training regime. Most evidence comes from GLUE LoRA experts with task-specific heads and strongly unequal exposure. The norm and data ratios correlate at 0.979, and the exposure baseline performs equally well. At matched exposure the pooled metric-ordering accuracy falls to 51.8%; Appendix G reports near-chance ordering on external generative experts. The work remains meaningful as a diagnosis of exposure-induced imbalance, but its generality and incremental contribution beyond the magnitude/exposure literature discussed in Section 2 should be stated more narrowly.

  3. Selection and intervention protocols limit causal and practical conclusions. Full-data checkpoints are selected on the evaluation sets, experts are QC-filtered, and matched exposure also reduces total training to 2,490 examples. Thus, reduced damage cannot be attributed to equalization alone. In addition, the stretching and shrinking experiments use different evaluators and only four pairs in two families. The paper acknowledges these issues, but a common evaluation protocol is needed for a direct quantitative comparison of the recommended interventions.

Questions for authors
  1. Independent prediction: Which reported results test the curvature-based condition using quantities independent of the observed merge loss, and how large are the residual gradient terms in those cases?

  2. Evaluation consistency: Can the existing records quantify whether the stretching-versus-shrinking comparison changes under a common evaluator? How sensitive are the main ordering statistics to checkpoint selection and QC filtering?

  3. Scope and novelty: What conclusion remains useful beyond training-exposure metadata, given their equal predictive performance and the negative external transfer result? Please distinguish the new contribution from prior magnitude/exposure findings using the comparisons already discussed in Section 2.

Reviewer_NMUROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposed a theory that suggests during model merging, the tasks with smaller task-vector norms will suffer more degradation. This theory is validated on mainly pairwise merging settings among LLMs.

Critical strengths

None

Critical weaknesses
  1. The paper is poorly written. I find it very hard to understand.
  2. The merging coefficients are not receiving enough consideration. The theoretical analysis assumes all the merging coefficients are the same. But in practice that's not the case. My worry is that once we have more realistic merging coefficients, norm ratio would no longer be a good predictor for the asymmetric degradation. The interaction of merging coefficients and task vector norms is not fully characterized.
Questions for authors
  1. How should practitioners choose merging coefficients to avoid/minimize task degradation?
  2. Would the conclusion holds up when more models are being merged together?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.