acceptodds
Under review as a conference paper at ICLR 2027

Erasing the Non-Additive Residue Is Insufficient to Repair Over-Scaled Model Merging

Abstract

Model merging by task arithmetic can fail under over-scaling, but an internal non-additivity measure need not identify the cause of that failure. We isolate the layerwise non-additive residue between a merged model and the additive reference built from its single-task paths. In two-task merges, a numerically audited factorial ledger shows that this residue is transported, regenerated after early erasure, and coupled to output non-additivity through its direction. The same audit shows that naive bfloat16 estimates of local generation are dominated by rounding error. We then erase the residue in two tested over-scaled configurations, at 1.5B and 7B. Complete erasure at the latest tested boundary does not reliably restore the audited code scores, and persistent erasure at three late boundaries across every generated position also fails to restore the 1.5B merge. In a fixed 12-item code instrument, partial persistent erasure (lambda=0.25) raises the score in all three seeds while worsening expert-reference negative log-likelihood in all three. That dose was exploratory. Complete residue removal is therefore insufficient as a repair in these configurations. Without same-coefficient single-task controls we cannot tell whether these failures are specific to merging, and the partial-dose response precludes calling the residue behaviourally irrelevant. At 1.5B the output-side interaction ratio loses method-ranking information under a shared counterfactual, while corresponding state-space measures rank four audited methods at both scales.

Then back it, or bet against it.

Related papers

Open the market on this paper to see 7 more related papers.