What Merging Misses: The Defining Direction of Joint Training

Model merging rests on a working assumption: that reweighting a set of independently trained models can stand in for one model trained jointly on all their tasks. We question this assumption and show that it fails in a precise, structural way. Across 81 LLM settings, even the best reweighting of independent endpoints loses to joint training in 80 cases. This failure is not because the endpoints are aimed in the wrong directions. Their span already captures about 82% of the joint update, but it misses a single direction outside that span, and no reweighting of the endpoints can recover it, no matter how it is tuned. This missing direction follows a simple law: its energy, measured by the missing squared norm, scales with the number of task pairs rather than the number of tasks. As a result, the gap grows quadratically as tasks accumulate, and a law fit on a few task counts can predict the gap at counts held out from the fit. The missing direction also has a definite origin. Almost all of it comes from a path-dependent term that joint training builds as the trajectory is pulled from one task toward the next. When this term is read from the training path and added back, the lost performance returns; when its reverse, or a random direction of the same length, is added instead, it does not. What sets joint training apart, then, is not where independently trained runs end up, but what their weights never record: a contribution built from the interaction between tasks as they are learned together. Recombining separately trained results, however carefully they are weighted, cannot reconstruct it.