acceptodds
Under review as a conference paper at ICLR 2027

Which Task Loses? Understanding Asymmetrical Degradation in Model Merging

Abstract

How many tasks can a merged model retain, and which task is most likely to suffer? We study merging as task-vector superposition and derive two-sided capacity bounds. Under a curvature assumption along the merge path, a stationary expert's loss increase lies between curvature multiples of its interferers' energy. The upper bound limits capacity in the task count ; the lower bound quantifies unavoidable damage. For two tasks, the smaller-vector task loses more whenever the squared norm ratio exceeds the curvature ratio. All 11 Llama pairs satisfying this condition under sampled, uncertified curvature ranges follow the predicted ordering. More broadly, the smaller-norm task's loss rises more in 91% of 58 full-data task-arithmetic pairs. Across 486 directional evaluations spanning four 7B families and three methods, within-merge contrasts account for 64.1% of degradation variance, which symmetric mergeability scores cannot explain; the smaller-norm task loses more in 83.5% of comparisons. Retraining links this asymmetry to exposure: norms grow with exposure, while matching exposure across tasks (three seeds) compresses their ratios and removes the ordering signal (51.8%). On four pairs, shrinking the larger vector protects the smaller task better than uniform scaling. Equalizing norms at fixed total norm instead stretches the smaller vector and harms 13 of 16 targets.

open until 14 Dec 2026

est. 32% chance this paper gets accepted at ICLR 2027.

Reject 68%Accept 32%

What do you think this paper will get?

All positions stay anonymous.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.