acceptodds
Withdrawn from ICLR 2027

Flow Merge: Flow Matching for Model Merging

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Model merging is a recent area of study that attempts to find methods to merge individual models into a single checkpoint that retains the accuracies of its constituents. Most approaches to model merging rely on the Linear Mode Connectivity hypothesis, which posits that models trained under SGD lie in the same loss basin and can be merged by simply linearly interpolating through the checkpoints. In this paper, we propose that, rather than assuming a straight-line path in weight space, we learn a path in activation space using flow matching. We find that our method, FlowMerge outperforms Git Re-Basin on CIFAR-10 with MLP and ResNet-20 by 10 and 33 percentage points respectively. More broadly, our results suggest that model merging need not be confined to the linear interpolation paradigm implied by Linear Mode Connectivity, and that richer transport-based formulations may open up new possibilities for combining independently trained models.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Model merging, Flow Matching

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_42ymOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a model-merging framework, using a small calibration dataset to find a flow between the logit distributions of two neural network models.

Critical strengths

(1) The idea of using Logit-space to connect modes is new and novel in the (linear) mode connectivity literature. Current AutoNEB and FGE (Bezier curve) methods are in the weights space.

(2) The authors try to propose a new candidate method for mode connectivity (connecting independently trained models, not linear mode connectivity). This is exactly what the mode connectivity field needs.

Critical weaknesses

(1) Evaluation is somehow out of date. Existing (Linear) Mode Connectivity (LMC) works have pushed the architecture family toward transformers [1] and MoE LLMs [3]. The authors evaluate FlowMerge only on CNN/MLP on the MNIST/CIFAR datasets, which makes it hard to show real-world benefits.

(2) The authors claim that "FlowMerge is the only approach that yields a low loss barrier" (~0.5721) on Line 255. However, this is not true. LLPF can reach 0.006 training loss barrier for ResNet18 on the CIFAR-10 dataset [1]. In addition, AutoNEB [2] can also connect two independently obtained modes with a loss barrier of <0.1 for ResNet20 on CIFAR10, see Figure 5 of [2].

Minor: Text in Figure 2 is too small and hard to read.

[1] https://arxiv.org/abs/2505.02604

[2] https://arxiv.org/abs/1803.00885

[3] https://arxiv.org/abs/2509.11348

Questions for authors

(1) I am not sure whether the authors would like to identify their "FlowMerge" method as a MC method (able to connect independently trained models) or a LMC method (connect models within the same basin to obtain better models). For MC, we care about the max training loss along the path (what you called the "barrier"). For LMC, we aim to have better-generalized merged models.

(2) Does FlowMerge also work for transformer architectures? For example, Compact Convolution Transformer on the CIFAR dataset, ViT on the ImageNet dataset?

If the authors can demonstrate that their method also works for transformer architectures and has practical benefits compared with existing MC methods or LMC methods, I will raise my score. (For MC, I suggest AutoNEB or LLPF, you should aim to find a path with lower barrier. For LMC, I suggest comparing with FGE. Your aim should be to get better generalized models than FGE, not on disjoint datasets)

Minor remarks: (1) What is the "small calibration dataset" in the implementation? Is it a small part of the training partition? (2) Section 4.1 talks about "independently trained models". This goes far beyond Linear Mode Connectivity (LMC) and reaches the Mode Connectivity (MC) field. However, I would like to remind that (1) MC problems are much more complicated; (2) model merging is currently popular in the LMC context, not in the MC context.

Reviewer_XFLTOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes FlowMerge, a method for identifying low-loss paths between two models trained on the same task, via a novel flow matching problem formulation. For each model pair, the authors generate a "source" and "target" distribution formed from the output logits of each model using a predefined calibration set shared between models. The authors then learn the velocity field between the distributions, via the flow matching objective. Once this is learned, the authors improve the performance of the "source" model, by partially denoising from its logits using the flow model, and stopping at an assigned time step. The authors show that doing so improves upon model merging baselines such as Git-Rebasin and ZipIt!, whilst also achieving low loss barriers on ResNet and MLP architectures on MNIST, CIFAR-10 and CIFAR-100.

Critical strengths
  • The conceptual idea of the paper is very interesting: It is well known that linear mode connectivity is rarely found in models of standard width, with its theoretical likelihood being directly tied to the width of the models [1,2]. By designing the low-loss path finding objective as a general transport problem, the authors create a hypothesis space that allows more complex (e.g., non-linear) interpolation paths between two models to be elegantly found.
  • The authors show that (non-linear) low-loss barriers exist between arbitrary model pairs trained on the same dataset at standard width, across three different datasets (MNIST, CIFAR-10, CIFAR-100) and architecture families (MLP, ResNet).
Critical weaknesses

This paper is not a model merging paper: Model merging involves interpolating between the parameters of two or more models, in order to build a new merged model. This paper involves no such merging process. Instead, the paper defines a new flow-matching head over which the outputs of one model are further processed to obtain the final result. Additionally, authors state in line 198 that the head can be "fused" with one of the underlying models, yet they do not elaborate on what this means -- do the authors perform one step denoising and then multiply the affine head into the classification layer of the underlying model? Even if this was the case, it would still not be model merging though.

Instead, this paper tackles the related problem of uncovering low-loss paths between two models. There is no discernible reason for why the two models used must share the same architecture, given that only their output logits are used and no weights are ultimately combined between the two models. This is still an interesting problem, but it changes how the paper should be framed, and the types of works it should compare against. I highly recommend the authors adjust their framing to be true to their approach.

ZipIt! is shockingly poor: The performance of ZipIt! is extremely poor on ResNet architectures. Do the authors have any hypotheses for the performance of the method on these tasks? It is stated in lines 239-240 that ZipIt! does not have an interpolation variable, yet does Git-Rebasin also perform equally poorly when using an interpolation of 0.5 (simple averaging)? How did the authors implement this method? For instance, did the authors reset the batch norm statistics as is described in the original paper?

No compute cost comparison: The authors are missing a FLOP comparison or similar for each method compared, both for obtaining the final model and for the inference cost. This is only superficially described in lines 404-406. How much slower is FlowMerge compared to the existing merging baselines? If one of the merged baselines was finetuned / trained on the dataset for the same FLOPs / compute budget as FlowMerge, does FlowMerge still outperform it? This is a similar setting to OT-Fusion [3,4] approaches.

Other important observations:

  • No explanation of the calibrated dataset: The calibration dataset is not defined anywhere in the paper. What is it?
  • More information on the models employed needed: I presume that the models employed have different initializations. Is this correct? It is not mentioned anywhere that I can see -- apologies if it is.
  • Simple datasets and non-SoTA architectures employed: How does FlowMerge perform on transformer architectures? Or more challenging datasets such as ImageNet?
  • Extension beyond two models: Does a formulation exist beyond interpolating between two models?
  • Unknown how important -- Appendix appears cut off: The Appendix appears cut off on Page 13. Is this intentional? The page ends with an incomplete sentence.

[1] The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks, Entezari et al., ICLR 2022 [2] ZipIt!, Stoica and Bolya et al., ICLR 2024 [3] Model Fusion via Optimal Transport, Singh et al., NeurIPS 2020 [4] Transformer Fusion with Optimal Transport, Imfeld, Graldi, and Giordano et al., ICLR 2024

Questions for authors

Please see the critical weaknesses block. I believe that a response to every point raised in the section is needed to understand true the significance of the author's contributions, yet extensions beyond two models and ImageNet results can be thought of as suggestions for future work. I would be happy to adjust my review based on the author responses.

Reviewer_fQKHOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes FlowMerge, which combines two trained classifiers by learning a transport between their logit distributions using calibration inputs. At inference, a learned velocity field transforms the source model’s logits to a chosen integration time. The authors derive a least-squares solution for an affine velocity field and observe that its discretized transformation can be absorbed into the source classifier’s final layer. They also provide a cross-entropy risk bound involving endpoint risks and the flow-matching residual. Experiments evaluate MLPs, ResNet-20, and VGG-11 on MNIST and CIFAR datasets, including independently trained classifiers and complementary label-skewed training subsets.

Critical strengths
  • Operating in logit space avoids explicitly aligning hidden units or parameters between the source models.
  • Fitting the flow requires calibration inputs and model predictions, without ground-truth labels.
  • The affine formulation permits closed-form fitting and absorption into the source classifier, preserving its inference architecture.
  • The cross-entropy bound provides a conditional stability guarantee for the learned trajectory under the stated assumptions.
Critical weaknesses
  1. The methodological contribution is insufficiently established.

FlowMerge applies an established conditional flow-matching objective to paired classifier logits. The affine fitting procedure follows standard least-squares regression, while folding the resulting transformation into the classifier follows from composition of affine maps. The main unresolved question is whether learning and integrating a velocity field offers a meaningful advantage over simpler output transformations.

The experimental evaluation is limited to small-scale vision benchmarks and relatively dated architectures. It would be substantially strengthened by evaluating the method in genuine task-merging settings, where the source models are trained for different tasks or on clearly distinct domains, as well as on more modern architectures.

The comparisons do not isolate this benefit: they primarily involve parameter-space merging methods, without direct affine logit regression, residual output adapters, or ensemble distillation under matched calibration budgets. Consequently, the experiments do not establish which benefits are attributable specifically to flow matching. The formulation appears incremental, and its practical significance remains unclear.

  1. The results provide limited evidence of improvement over retaining a source model.

Outperforming a parameter-space merge that substantially degrades accuracy does not, by itself, demonstrate useful knowledge combination. In Table 1, Analytical FlowMerge, a method which is not defined in the paper, achieves 87.53% on CIFAR-10 ResNet-20, matching model A and remaining below model B’s 87.71% and the ensemble’s 88.99%. On CIFAR-100 ResNet-20, its 57.04% is below both source models and the ensemble’s 60.53%.

The complementary-data experiment is more informative, but Analytical FlowMerge reaches 46.35%, compared with 45.91% for model A and 53.04% for the ensemble. This modest gain requires further analysis of variability and which complementary predictions are recovered. The paper should distinguish preserving source performance from acquiring capabilities or ensemble benefits.

  1. Calibration and reproducibility details are insufficient.

The manuscript does not clearly specify the calibration-set size, selection procedure, overlap with source-model training data, or allocation across data-dependent methods. These omissions prevent a reliable assessment of data requirements and comparison fairness.

Reproducibility also requires fuller source-model training schedules, clear seed counts for the main results, baseline configurations, including normalization-statistic handling where applicable, and identification of the analytical fitting variant used experimentally. Calibration-budget sensitivity and runtime measurements would help establish the method’s practical value.

  1. Several claims conflict with the reported evidence, and parts of the manuscript are unfinished.

Section 4.3 reports 57.02% accuracy for deeper heads, whereas Table 3 reports a maximum of 47.20%. The discussion also characterizes depth as having little effect despite an increase from 44.42% to 47.20%. Section 4.2 alternates between CIFAR-10 and CIFAR-100, while the appendix describes a label partition over classes 0–9.

Claims about outperforming individual models and closely approaching ensembles require qualification in light of the tables. Reported cross-entropy values should also be distinguished from loss barriers relative to endpoint losses. Because the learned endpoint only approximates model B, endpoint fidelity is necessary to interpret claims about connecting the original models.

Table 2 also requires clarification: both source models perform substantially better on the oppositely numbered test split, suggesting a possible mismatch in split labels or model assignments.

The conclusion and Appendix C contain unfinished sentences. Together, these problems obstruct interpretation of the experiments rather than merely affecting readability.

  1. The paper inadequately engages with task arithmetic, disentanglement, and model linearization.

The motivation emphasizes limitations of linear mode connectivity but does not adequately position the method relative to research on composing task-specific updates from pretrained models. Relevant developments include Task Arithmetic, the analysis of weight disentanglement and tangent-space fine-tuning by Ortiz-Jimenez et al. (2023), selective fine-tuning studied by Jin et al. (2025), and curvature-based disentanglement regularization by Porrello et al. (2026).

These methods have different assumptions and are not automatically applicable baselines for independently initialized checkpoints. Nevertheless, this literature is relevant to the paper’s broad motivation of combining independently acquired knowledge. The authors should clarify whether FlowMerge supports distinct task capabilities and how its training and data requirements differ. A representative task-composition experiment, or an explicit restriction of scope and claims, is needed.

Questions for authors
  • What advantage does flow matching provide over directly fitting an affine transformation from model A’s logits to model B’s logits, or distilling the ensemble, using the same calibration inputs and comparable compute?

  • What is the exact calibration protocol? Please specify its size, source, overlap with training or test data, and allocation across baselines. Which analytical fitting variant produces the reported results? Which settings demonstrate a reliable benefit over retaining or selecting a source model? Can you measure correction of source-model errors and agreement with model B at the learned endpoint? Could you resolve the dataset and numerical inconsistencies, complete the unfinished passages, and supply the missing training and baseline details?

  • Could the authors clarify what “Analytical FlowMerge” denotes in the tables? How it differs from the learned FlowMerge implementation, and whether both variants are evaluated using the same Euler integration and calibration protocol.

  • What are the computational and memory costs of FlowMerge? Please report the time required to fit the flow head, the calibration-set size, the number of ODE/Euler evaluations at inference.

  • When does merging improve upon retaining a source model? In Table 1, Analytical FlowMerge matches model A’s CIFAR-10 ResNet-20 accuracy and falls below both source models on CIFAR-100. Could you clarify which settings demonstrate a benefit over selecting a source checkpoint, and distinguish this from outperforming unsuccessful parameter-space merges?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.