Flow Merge: Flow Matching for Model Merging

Model merging is a recent area of study that attempts to find methods to merge individual models into a single checkpoint that retains the accuracies of its constituents. Most approaches to model merging rely on the Linear Mode Connectivity hypothesis, which posits that models trained under SGD lie in the same loss basin and can be merged by simply linearly interpolating through the checkpoints. In this paper, we propose that, rather than assuming a straight-line path in weight space, we learn a path in activation space using flow matching. We find that our method, FlowMerge outperforms Git Re-Basin on CIFAR-10 with MLP and ResNet-20 by 10 and 33 percentage points respectively. More broadly, our results suggest that model merging need not be confined to the linear interpolation paradigm implied by Linear Mode Connectivity, and that richer transport-based formulations may open up new possibilities for combining independently trained models.