What Can a Task Checkpoint Identify? Invariant Task-Core Fusion for Model Merging

Model merging combines independently fine-tuned experts into one model, but interference can degrade performance. Without task data, data-free methods rely on checkpoints and often use task vectors, the parameter changes induced by fine-tuning, as proxies for inaccessible task data. In a controlled linear setting, we show that different training data can produce the same final weights. A task vector therefore does not uniquely characterize its originating data, and treating checkpoint geometry as if it faithfully reflected the original task data can misguide fusion. We propose Invariant Task-Core Fusion (ITCF), which builds basis-invariant task cores from checkpoint-identifiable directions and uses core overlap to capture shared support and reconcile expert updates. Across eight vision and language configurations and more than 500 task compositions, ITCF remains competitive across model families and merge scales, including full-model merges of up to 166 experts. Core fusion is roughly two orders of magnitude faster than iterative optimization, with no additional parameters or inference cost.