- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates mode connectivity between independently trained diffusion and contrastive models, extending a line of work established mainly for classifiers. Building on LLPF, it combines architecture-specific layer movement, variance correction, and Adam-based refinement to construct low-training-loss trajectories toward a target model. Experiments cover DDPM on Flowers102 and NanoCLIP on Flickr30k, including baseline comparisons, ablations, and checks of interpolation loss and task performance.
Critical strengths
- The paper addresses an interesting foundational question: whether independently trained generative and contrastive models share connected low-loss regions. Evidence in these two settings meaningfully broadens the empirical study of loss geometry.
Critical weaknesses
-
The transferability and practical meaning of the method remain unclear. The paper provides schedules for two specific architectures, but does not fully specify how to derive them for related architectures. It also offers limited analysis of why maintaining low training loss produces worse test loss and generation quality along the path. This limits understanding of both how broadly the method applies and what useful model behavior the constructed paths preserve.
-
The paper does not clearly specify a consistent loss threshold and evaluation protocol for establishing connectivity. For example, NanoCLIP’s loss scale differs substantially between Figure 3 and the interpolation check in Figure 8, while the final DDPM connection reaches a higher loss than the preceding trajectory. Without a common evaluation protocol and an explicit acceptable loss increase relative to the endpoints, it is difficult to determine whether the complete paths satisfy the claimed low-loss criterion.
Questions for authors
-
Clarification of connectivity: What loss threshold defines successful connectivity? Under a consistent evaluation protocol, does each complete path remain below that threshold?
-
Which scheduling choices follow automatically from the architecture, and which require manual design? What evidence shows that the proposed schedule improves on an LLPF-style forward-order schedule when both use Adam refinement? Were complete paths evaluated in this comparison?
3 comments
Thank you for your answers and clarifications. However, without a clearly defined low-loss threshold, the complete paths cannot be evaluated against the claimed low-loss criterion. Therefore, I will keep my rating unchanged.
Thank you for your follow-up. To explain why we don't adopt a fixed numerical loss threshold, we provide the comparison with prior studies below (Draxler et al., 2018; Garipov et al., 2018; Tian et al., 2026). We also discussed this point in our second response to Reviewer UYcp and reproduce the relevant evidence here for completeness.
| Method | Model | Dataset | Endpoint Loss | Maximum Path Loss | Ratio |
|---|---|---|---|---|---|
| FGE | ResNet-158 | CIFAR-10 | 0.015 / 0.015 | 0.047 | 3.1× |
| AutoNEB | ResNet-56 | CIFAR-10 | 0.0022 | 0.0141 | 6.4× |
| AutoNEB | ResNet-56 | CIFAR-100 | 0.0428 | 0.2978 | 7.0× |
| LLPF | ResNet-18 | CIFAR-10 | 0.025 | 0.04 | 1.6× |
| LLPF | DLA | CIFAR-10 | 0.0012 | 0.0065 | 5.4× |
| Ours | DDPM | Flowers102 | 0.0261 / 0.0259 | 0.0339 | 1.3× |
| Ours | NanoCLIP | Flickr30k | 0.0236 / 0.0206 | 0.1038 | 4.7× |
If low-loss connectivity is defined in terms of the ratio of maximum path loss to endpoint loss, our constructed paths clearly meet the standard demonstrated by previous studies that reported low-loss connections. However, we do not want to impose a fixed cutoff such as 5× or 10×, as this would exclude some previously reported low-loss connections under our definition. This is why we emphasize that, in practice, training loss is reduced rather than necessarily minimized, and the differences reflect the quality of the constructed paths.
References
- Draxler et al. 2018: Essentially No Barriers in Neural Network Energy Landscape
- Garipov et al. 2018: Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
- Tian et al. 2026: Connecting independently trained modes via layer-wise connectivity
Thank you for your careful review and constructive comments. We would first like to further explain the weakness of the practical meaning of the method remaining unclear that you have raised. Our algorithm extends the scope of mode connectivity to contrastive and generative models. It enables investigation of the geometry and topology of low-loss regions, which in turn contributes to a deeper understanding of the loss landscape and neural network behavior, especially for complex non-classifier models.
All previous works (Draxler et al., 2018; Garipov et al., 2018; Tian et al., 2026) of mode connectivity did not strictly define the low-loss connection or provide a threshold for low loss. In practice, the training loss is merely reduced rather than minimized, and the differences lie only in the quality of the path. If the training loss is sufficiently large that the model’s accuracy becomes unacceptable, we consider the construction of a low-loss connection to have failed.
Architecture-derived choices. The moving sequence follows two rules: layers are introduced in the direction of data flow, and attention modules are processed individually before proceeding to subsequent layers. For standard acyclic sequential architectures, these rules allow the sequence to be generated automatically from the computational graph.
Manual choices. For complex non-sequential architectures, applying these rules requires manually identifying corresponding feature-processing stages and specifying stage granularity. Examples include grouping DDPM encoder and decoder blocks by spatial resolution and specifying the interleaving schedule between NanoCLIP’s image and text encoders.
The second ablation row compares our schedule with an LLPF-style forward-order schedule while keeping Adam refinement. Our schedule reduces average and maximum training loss on both DDPM and NanoCLIP. Due to limitations in computing resources, we did not evaluate the complete path as significant differences could already be observed within limited ticks.
References