- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a model-merging framework, using a small calibration dataset to find a flow between the logit distributions of two neural network models.
Critical strengths
(1) The idea of using Logit-space to connect modes is new and novel in the (linear) mode connectivity literature. Current AutoNEB and FGE (Bezier curve) methods are in the weights space.
(2) The authors try to propose a new candidate method for mode connectivity (connecting independently trained models, not linear mode connectivity). This is exactly what the mode connectivity field needs.
Critical weaknesses
(1) Evaluation is somehow out of date. Existing (Linear) Mode Connectivity (LMC) works have pushed the architecture family toward transformers [1] and MoE LLMs [3]. The authors evaluate FlowMerge only on CNN/MLP on the MNIST/CIFAR datasets, which makes it hard to show real-world benefits.
(2) The authors claim that "FlowMerge is the only approach that yields a low loss barrier" (~0.5721) on Line 255. However, this is not true. LLPF can reach 0.006 training loss barrier for ResNet18 on the CIFAR-10 dataset [1]. In addition, AutoNEB [2] can also connect two independently obtained modes with a loss barrier of <0.1 for ResNet20 on CIFAR10, see Figure 5 of [2].
Minor: Text in Figure 2 is too small and hard to read.
[1] https://arxiv.org/abs/2505.02604
Questions for authors
(1) I am not sure whether the authors would like to identify their "FlowMerge" method as a MC method (able to connect independently trained models) or a LMC method (connect models within the same basin to obtain better models). For MC, we care about the max training loss along the path (what you called the "barrier"). For LMC, we aim to have better-generalized merged models.
(2) Does FlowMerge also work for transformer architectures? For example, Compact Convolution Transformer on the CIFAR dataset, ViT on the ImageNet dataset?
If the authors can demonstrate that their method also works for transformer architectures and has practical benefits compared with existing MC methods or LMC methods, I will raise my score. (For MC, I suggest AutoNEB or LLPF, you should aim to find a path with lower barrier. For LMC, I suggest comparing with FGE. Your aim should be to get better generalized models than FGE, not on disjoint datasets)
Minor remarks: (1) What is the "small calibration dataset" in the implementation? Is it a small part of the training partition? (2) Section 4.1 talks about "independently trained models". This goes far beyond Linear Mode Connectivity (LMC) and reaches the Mode Connectivity (MC) field. However, I would like to remind that (1) MC problems are much more complicated; (2) model merging is currently popular in the LMC context, not in the MC context.