acceptodds
Under review as a conference paper at ICLR 2027

Mode Connectivity Beyond Classifiers: Evidence from Generative and Contrastive Models

Abstract

The loss landscape of deep neural networks (DNNs) exhibits highly complex and non-convex properties. Recent studies have revealed the phenomenon of mode connectivity, demonstrating that independently trained network modes can be connected via a continuous low-loss path. However, existing research is predominantly confined to classifier-based models, leaving it an open question whether similar geometric properties exist in modern complex models. In this paper, we extend the boundaries of mode connectivity to generative and contrastive domains (specifically DDPM and NanoCLIP). Addressing the unique architecture of DDPM and CLIP, we propose an architecture-aware connection building algorithm. Extensive empirical results demonstrate for the first time that we successfully discover mode connectivity between independently trained DDPM and NanoCLIP modes. Our work provides a novel perspective for understanding the geometric properties of the loss landscapes in modern generative and contrastive models.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Mode Connectivity, Loss Landscape, Generative Models, Contrastive Learning

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_FwBtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates mode connectivity between independently trained diffusion and contrastive models, extending a line of work established mainly for classifiers. Building on LLPF, it combines architecture-specific layer movement, variance correction, and Adam-based refinement to construct low-training-loss trajectories toward a target model. Experiments cover DDPM on Flowers102 and NanoCLIP on Flickr30k, including baseline comparisons, ablations, and checks of interpolation loss and task performance.

Critical strengths
  1. The paper addresses an interesting foundational question: whether independently trained generative and contrastive models share connected low-loss regions. Evidence in these two settings meaningfully broadens the empirical study of loss geometry.
Critical weaknesses
  1. The transferability and practical meaning of the method remain unclear. The paper provides schedules for two specific architectures, but does not fully specify how to derive them for related architectures. It also offers limited analysis of why maintaining low training loss produces worse test loss and generation quality along the path. This limits understanding of both how broadly the method applies and what useful model behavior the constructed paths preserve.

  2. The paper does not clearly specify a consistent loss threshold and evaluation protocol for establishing connectivity. For example, NanoCLIP’s loss scale differs substantially between Figure 3 and the interpolation check in Figure 8, while the final DDPM connection reaches a higher loss than the preceding trajectory. Without a common evaluation protocol and an explicit acceptable loss increase relative to the endpoints, it is difficult to determine whether the complete paths satisfy the claimed low-loss criterion.

Questions for authors
  1. Clarification of connectivity: What loss threshold defines successful connectivity? Under a consistent evaluation protocol, does each complete path remain below that threshold?

  2. Which scheduling choices follow automatically from the architecture, and which require manual design? What evidence shows that the proposed schedule improves on an LLPF-style forward-order schedule when both use Adam refinement? Were complete paths evaluated in this comparison?

3 comments

Thank you for your careful review and constructive comments. We would first like to further explain the weakness of the practical meaning of the method remaining unclear that you have raised. Our algorithm extends the scope of mode connectivity to contrastive and generative models. It enables investigation of the geometry and topology of low-loss regions, which in turn contributes to a deeper understanding of the loss landscape and neural network behavior, especially for complex non-classifier models.

  • Q1: What loss threshold defines successful connectivity?

All previous works (Draxler et al., 2018; Garipov et al., 2018; Tian et al., 2026) of mode connectivity did not strictly define the low-loss connection or provide a threshold for low loss. In practice, the training loss is merely reduced rather than minimized, and the differences lie only in the quality of the path. If the training loss is sufficiently large that the model’s accuracy becomes unacceptable, we consider the construction of a low-loss connection to have failed.

  • Q2: Which scheduling choices follow automatically from the architecture, and which require manual design?

Architecture-derived choices. The moving sequence follows two rules: layers are introduced in the direction of data flow, and attention modules are processed individually before proceeding to subsequent layers. For standard acyclic sequential architectures, these rules allow the sequence to be generated automatically from the computational graph.

Manual choices. For complex non-sequential architectures, applying these rules requires manually identifying corresponding feature-processing stages and specifying stage granularity. Examples include grouping DDPM encoder and decoder blocks by spatial resolution and specifying the interleaving schedule between NanoCLIP’s image and text encoders.

  • Q3: What evidence shows that the proposed schedule improves on an LLPF-style forward-order schedule when both use Adam refinement?

The second ablation row compares our schedule with an LLPF-style forward-order schedule while keeping Adam refinement. Our schedule reduces average and maximum training loss on both DDPM and NanoCLIP. Due to limitations in computing resources, we did not evaluate the complete path as significant differences could already be observed within limited ticks.

References

  • Draxler et al. 2018: Essentially No Barriers in Neural Network Energy Landscape
  • Garipov et al. 2018: Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
  • Tian et al. 2026: Connecting independently trained modes via layer-wise connectivity

Thank you for your answers and clarifications. However, without a clearly defined low-loss threshold, the complete paths cannot be evaluated against the claimed low-loss criterion. Therefore, I will keep my rating unchanged.

Thank you for your follow-up. To explain why we don't adopt a fixed numerical loss threshold, we provide the comparison with prior studies below (Draxler et al., 2018; Garipov et al., 2018; Tian et al., 2026). We also discussed this point in our second response to Reviewer UYcp and reproduce the relevant evidence here for completeness.

MethodModelDatasetEndpoint LossMaximum Path LossRatio
FGEResNet-158CIFAR-100.015 / 0.0150.0473.1×
AutoNEBResNet-56CIFAR-100.00220.01416.4×
AutoNEBResNet-56CIFAR-1000.04280.29787.0×
LLPFResNet-18CIFAR-100.0250.041.6×
LLPFDLACIFAR-100.00120.00655.4×
OursDDPMFlowers1020.0261 / 0.02590.03391.3×
OursNanoCLIPFlickr30k0.0236 / 0.02060.10384.7×

If low-loss connectivity is defined in terms of the ratio of maximum path loss to endpoint loss, our constructed paths clearly meet the standard demonstrated by previous studies that reported low-loss connections. However, we do not want to impose a fixed cutoff such as 5× or 10×, as this would exclude some previously reported low-loss connections under our definition. This is why we emphasize that, in practice, training loss is reduced rather than necessarily minimized, and the differences reflect the quality of the constructed paths.

References

  • Draxler et al. 2018: Essentially No Barriers in Neural Network Energy Landscape
  • Garipov et al. 2018: Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
  • Tian et al. 2026: Connecting independently trained modes via layer-wise connectivity
Reviewer_UYcpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether mode connectivity extends to a generative DDPM and a contrastive NanoCLIP model. It progressively moves subsets of layers toward the target model and applies layer-wise variance correction followed by refining the intermediate model using Adam until its training loss falls below a specified threshold. The layer movement follows a dataflow-based schedule. For the DDPM U-Net, encoder and decoder blocks at matching resolutions are paired and moved from the outside in. For NanoCLIP, image and text encoder blocks are interleaved. Experiments on Flowers102 with DDPM and Flickr30k with NanoCLIP evaluate three independently trained model pairs per architecture. The resulting paths end within approximately 0.2% of the original endpoint separation in relative L2 distance while maintaining low training loss. The method also outperforms AutoNEB method in the reported comparisons. However, intermediate DDPM checkpoints exhibit worse FID than the endpoints despite having similar training loss, highlighting that low loss along a path does not necessarily preserve generative quality.

Critical strengths
  1. The paper addresses an interesting question whether mode connectivity extends beyond image classifiers to generative and contrastive models. It studies independently trained modes without relying on permutation alignment or a shared training trajectory, making the setting more challenging.
  2. The authors conduct clear ablation studies to evaluate key design choices, including layer movement order and optimizer selection. Dense interpolation between consecutive checkpoints (Appendix A.2) helps check for loss barriers that might be missed by evaluating only selected points along the path. The improvement over AutoNEB on NanoCLIP is substantial, providing convincing empirical evidence that the proposed approach is effective in this setting.
  3. The authors also report limitations and negative findings transparently, including the degradation in FID along the DDPM path despite low training loss.
Critical weaknesses
  1. The study evaluates only two models, two datasets, and three mode pairs per model, which is insufficient to support broad claims about mode connectivity in generative and contrastive models.
  2. The layer movement schedules are hand-designed for each architecture, making it unclear how the method would generalize to new architectures.
  3. The paths do not consistently satisfy the stated loss thresholds. For DDPM, the threshold is 0.03, but the maximum training loss is approximately 0.034 for all pairs. For NanoCLIP, the threshold is 0.04, while the mean loss is approximately 0.044 and the maximum reaches 0.10.
  4. The paper does not define low loss relative to the endpoint losses, which are also not reported in Table 2. This makes it difficult to assess the magnitude of the remaining loss barrier.
  5. Intermediate DDPM checkpoints show substantially higher FID than the endpoints despite having similar training loss (Figure 10). This raises concerns whether low denoising loss is sufficient to demonstrate meaningful mode connectivity in diffusion models, since the path does not consistently preserve generation quality.
  6. The ablation gains are small. For example, dataflow order improves average loss over forward order by only 0.002. The ablations also use single runs on partial paths without error bars, making the benefits difficult to establish.
  7. AutoNEB is the only baseline evaluated directly, making it difficult to assess how the proposed method compares with other mode connectivity approaches. The claim that the importance of optimizer choice is identified for the first time is also insufficiently contextualized with respect to prior work, since the AutoNEB baseline already uses AdamW.
  8. The computational comparison is not compute-matched. The proposed method takes 59 and 77 hours, compared with 23 and 41 hours for AutoNEB, and the methods run on different GPUs. This makes the efficiency comparison inconclusive and raises scalability concerns. 9)The discussion of foundation-model loss landscapes goes beyond the evidence provided by experiments on a small DDPM.
  9. Equation 3 states that two initializations have exactly equal variance, although only their expected variances are equal.
Questions for authors
  1. What are the endpoint training losses, and how was each threshold chosen?
  2. Why does the maximum training loss exceed the refinement threshold? Is it because refinement hits the cap on the number of rounds?
  3. How is the DDPM training loss measured along the path? Is it with fixed timesteps and noise, or with fresh random samples? How much of the reported variation is due to sampling noise?
  4. How does the path change if variance correction is removed? It would be interesting to see this ablation.
  5. How sensitive are the results to the step sizes, learning rate and other hyperparameters?
  6. How does your path's barrier compare with linear interpolation after permutation alignment, for example Git Re-Basin (Ainsworth et al., 2023)?
  7. Have you tried a larger or different architecture, such as a bigger diffusion model or standard CLIP?
  8. What are the practical implications of these paths for example for model merging or weight space learning? It would be fun to see if this work can be applied for weight space learning.
4 comments

Thank you for your review. We address all your questions below, and we hope our answers can further help to distinguish between mode connectivity and linear mode connectivity.

  • Q1: What are the endpoint training losses, and how was each threshold chosen?
ModelPairStart endpointEnd endpoint
DDPM10.02610.0259
DDPM20.02810.0263
DDPM30.02900.0285
NanoCLIP10.02360.0206
NanoCLIP20.03430.0227
NanoCLIP30.03050.0222

The endpoints were trained for 700 epochs for DDPM and 400 epochs for NanoCLIP. The training refinement threshold is an empirical choice to balance runtime and the quality of low-loss path, and it is not a universal definition of low loss.

  • Q2: Why does the maximum training loss exceed the refinement threshold?

Refinement terminates when either (i) the training loss falls below the target after the minimum number of steps or (ii) the maximum number of steps is reached. The caps are 500 steps for DDPM and 200 for NanoCLIP. A checkpoint can therefore remain above the target when the cap is reached.

  • Q3: How is the DDPM training loss measured along the path?

For training loss measurement, every image independently receives a freshly sampled diffusion timestep and Gaussian noise on each forward pass.

For test loss measurement, evaluation traverses the complete official Flowers102 validation split and reports a mean over all images. The validation images and preprocessing are fixed, but timesteps and Gaussian noise are freshly sampled.

  • Q4-Q5: How does the path change if variance correction is removed? And how sensitive are the results to the step sizes, learning rate and other hyperparameters?

Variance correction is inherited from Tian et al., 2026, not introduced as a new contribution in this paper. We agree that more ablation analysis of variance correction, step sizes, learning rate and other hyperparameters would be informative. However, such analysis will require a substantial number of additional experiments, which is beyond the scope of this paper. At this stage, the core issue is whether we can find mode connectivity beyond the classifiers.

  • Q6: How does your path's barrier compare with linear interpolation after permutation alignment, for example Git Re-Basin (Ainsworth et al., 2023)?

In the related work section, we explicitly pointed out that Git Re-Basin belongs to linear mode connectivity: it first aligns parameter symmetries so that a linear interpolation becomes viable. Our study asks whether a nonlinear path can be constructed without applying such endpoint permutation. We therefore did not include Git Re-Basin as a baseline and cannot report its barrier on these models.

  • Q7: Have you tried a larger or different architecture, such as a bigger diffusion model or standard CLIP?

A complete path already requires 59 hours for DDPM and 77 hours for NanoCLIP on NVIDIA RTX 5090. Conservatively estimating, the expected time to construct a complete path for standard CLIP would reach one month, which makes hyperparameter adjustment almost unfeasible and beyond our capability.

  • Q8: What are the practical implications of these paths for example for model merging or weight space learning?

Our results suggest that independently trained DDPM and NanoCLIP modes can be connected through a low-loss path. The trajectory could therefore support path-aware model merging or weight-space learning. It is worth noting that model merging or weight space learning is more directly related to linear mode connectivity, and mode connectivity mainly helps us to gain a deeper understanding of the loss landscape of neural networks.

References

  • Tian et al. 2026: Connecting independently trained modes via layer-wise connectivity
  • Ainsworth et al., 2023: Git Re-Basin: Merging Models modulo Permutation Symmetries

I thank the authors for their response and for reporting the endpoint losses and clarifying the refinement and loss-measurement procedures (Q1 to Q3).

However, the reported endpoint losses reinforce my concern about the remaining loss barrier. For DDPM, the path maximum (about 0.034) is roughly 17 to 31% above the endpoints (0.026 to 0.029). For NanoCLIP, the path mean (about 0.044) is 1.3 to 2.1x and the maximum (0.10) about 3 to 5x the endpoint losses (0.021 to 0.034), and the 0.04 threshold lies above all endpoint losses. Without a principled, endpoint-relative definition of 'low loss', it is hard to judge whether these paths show meaningful mode connectivity.

The rebuttal also does not address the weaknesses in my review. Also the requested ablations (Q4 and Q5) and the alignment-based reference barrier (Q6) were declined. I would note that Q6 asks for a reference point to put the barrier in context, not for Git Re-Basin as a competing method.

Given that most concerns remain open, I am keeping my score.

Thank you for requesting further clarification on low loss and mode connectivity. Previous studies (Draxler et al., 2018; Garipov et al., 2018; Tian et al., 2026) in this field did not provide a strict definition of a low-loss connection, yet they all claimed to demonstrate mode connectivity. The table below compares these studies with our work. We believe this comparison helps explain why, rather than specifying a fixed numerical threshold for low loss, we prefer the following criterion: if the training loss is sufficiently large that the model’s accuracy becomes unacceptable, we consider the construction of a low-loss connection to have failed.

MethodModelDatasetEndpoint LossMaximum Path LossRatio
FGEResNet-158CIFAR-100.015 / 0.0150.0473.1×
AutoNEBResNet-56CIFAR-100.00220.01416.4×
AutoNEBResNet-56CIFAR-1000.04280.29787.0×
LLPFResNet-18CIFAR-100.0250.041.6×
LLPFDLACIFAR-100.00120.00655.4×
OursDDPMFlowers1020.0261 / 0.02590.03391.3×
OursNanoCLIPFlickr30k0.0236 / 0.02060.10384.7×

Notes: AutoNEB does not report results for individual paths. Its values in the table represent the average loss across endpoint models and the average of the maximum losses across paths from repeated experiments. LLPF does not provide a complete set of numerical endpoint and maximum path losses. We therefore estimated the starting-mode loss and maximum path loss from Figure 2 of that paper.

We would also like to clarify the difference between the NanoCLIP and DDPM results. As shown in Table 4 of our paper, NanoCLIP allows up to 200 refinement rounds, whereas DDPM allows up to 500. This difference helps explain why DDPM has a lower maximum loss than NanoCLIP. It is believed that increasing the number of refinement rounds for NanoCLIP would improve the quality of its path.

We understand that Q6 requests an alignment-based reference rather than a competing baseline. However, our study concerns nonlinear connections between independently trained, unpermuted modes, whereas Git Re-Basin constructs a linear path between permutation-aligned endpoints and thus addresses a different research area. An alignment-based reference (Q6), as well as ablations of variance correction, step size, learning rate, and other hyperparameters (Q4-5), would not directly isolate the contribution of either component. The effectiveness of our two principal designs—dataflow-based layer moving and Adam-based refinement—is already examined through the controlled ablations in Figure 5 of our paper. We are happy to conduct additional ablation studies to validate the core designs. Accordingly, we have adopted Reviewer EsuM’s suggestion and are conducting an ablation study of random layer movement.

References

  • Draxler et al. 2018: Essentially No Barriers in Neural Network Energy Landscape
  • Garipov et al. 2018: Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
  • Tian et al. 2026: Connecting independently trained modes via layer-wise connectivity

I thank the authors for comparing their loss ratios with prior work. I agree that the DDPM path is low loss by previous standards, and NanoCLIP falls within the range reported before. However, by the authors own criterion of acceptable task quality, the FID degradation along the DDPM path remains a concern. In addition, since most other weaknesses and the requested ablations (Q4 to Q6) remain unaddressed, I am keeping my score.

Reviewer_yKR7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The manuscript investigates mode connectivity beyond supervised image classifiers, considering DDPM on Flowers102 and NanoCLIP on Flickr30k. Building on layer-wise connectivity methods, it combines variance-corrected parameter movement, architecture-specific layer schedules, and Adam-based refinement. Experiments on three independently trained endpoint pairs per model show trajectories with low training loss and small residual distances to the target endpoints. Additional experiments examine interpolation between checkpoints, generation quality, and local geometry near an endpoint. The empirical question is interesting. The main value is evidence that low-training-loss paths can be constructed in these two settings, together with practical adaptations that make their construction possible. I assess the paper as an empirical loss-landscape study.

Critical strengths
  1. A worthwhile empirical question. Extending connectivity studies to generative and contrastive objectives can improve our understanding of neural-network loss landscapes. An empirical contribution of this kind can be valuable independently of state-of-the-art task performance.
  2. Evidence across multiple endpoint pairs. Testing three distinct pairs per model provides stronger support than a single illustrative connection.
  3. Interpolation checks between checkpoints. Appendix A.2 evaluates twenty interpolation points between consecutive checkpoints. This meaningfully addresses the possibility that low-loss anchors conceal barriers along their connecting segments.
  4. Useful analysis of training loss versus generation quality. Section 6 and Appendix A.6 explicitly show that low denoising loss can coexist with worse FID. This distinction is scientifically useful and appropriately qualifies what training-loss connectivity establishes.
  5. Substantial improvement over the tested baseline. Figure 4 reports maximum training losses of 0.17 versus 0.03 for DDPM and 6.69 versus 0.09 for NanoCLIP, comparing AutoNEB with the proposed method. These differences are substantial evidence that the proposed construction is effective in the reported configurations. Table 2 separately reports NanoCLIP path maxima up to 0.1038 across its three pairs; the Figure 4 comparison should be interpreted within its own evaluation setting.
Critical weaknesses

1 The operational definition of low loss should be explicit Algorithm 1 approaches the target endpoint with residual distances of approximately 0.11-0.18% of the initial separation in Table 2. Appendix A.2 evaluates twenty interpolation points between consecutive checkpoints, and Appendix A.7 separately evaluates the remaining DDPM segment to the exact endpoint. Thus, the paper already provides evidence beyond a discrete sequence of low-loss anchors. Appendix A.7 reports a maximum loss of 0.038 on the final DDPM segment, compared with approximately 0.034 along the preceding optimization checkpoints. Appendix A.2 also reports that dense interpolation can raise the maximum DDPM loss by up to 9.7% relative to the checkpoint trajectory. The endpoint observation therefore warrants an explicit tolerance and consistent reporting, rather than an inference that the reported low-loss connection has failed. Please give the endpoint-inclusive maximum loss and excess barrier for every pair under a stated low-loss criterion, and distinguish this criterion from the refinement stopping thresholds in Table 4. Fixed starting-layer variances help explain a small residual distance, but a final interpolation segment may leave that variance constraint. Exact equality of the optimization checkpoint and target is therefore not required if the completed path satisfies the stated loss tolerance. 2 NanoCLIP initialization deserves explicit confirmation The manuscript explicitly states that endpoint models use different random initializations without shared early checkpoints (Section 2.3 and Section 3). The cited NanoCLIP repository describes fine-tuning DINOv2-S and SBERT/all-MiniLM-L6-v2 encoders. This makes an explicit description of reinitialization and any frozen or pretrained parameters important for reproducibility. The repository defaults do not establish that the authors used shared pretrained weights. I take the manuscript's stated independent-initialization setting at face value pending this clarification. [1] If the encoders were shared pretrained checkpoints, the contrastive result would need to be framed as connectivity following shared pretraining. Confirmation that all relevant parameters were independently initialized would resolve this concern without requiring a new scientific contribution. 3 Comparative conclusions should match the tested configurations AutoNEB uses thirteen total pivots, while the proposed procedure constructs thousands of checkpoints with repeated refinement. Tables 3-4 also report runtimes on different GPU types. These differences limit conclusions about comparative compute efficiency and the best achievable performance of AutoNEB; they do not negate the empirical existence of the constructed low-loss paths. The partial ablations provide directional evidence for the proposed choices, although they do not establish complete-path success rates for each variant. Reporting gradient-evaluation counts and the forward-order-plus-SGD comparator would strengthen attribution of the improvement. The current results support success relative to the tested baseline configuration; broader claims that prior methods cannot connect these model classes should be narrowed. 4 Reproducibility and task quality need clearer interpretation Appendices A.3-A.4 describe path optimization in useful detail. The endpoint-training configuration and evaluation protocol should also be explicit, including model size, dataset splits, resolution, contrastive-loss settings, and control of stochastic DDPM loss evaluation. The manuscript states that code and preprocessing scripts are supplied as supplementary material; those files were not available for this assessment, so I cannot conclude that they omit these details. Figure 3 shows NanoCLIP held-out Recall@1 around 0.006-0.010 alongside training accuracy near 0.98. Please specify the retrieval direction, candidate pool, and metric definitions. These values limit the interpretation of endpoint quality but do not by themselves contradict connectivity of the training-loss landscape. Similarly, worse intermediate DDPM test loss and FID qualify claims about preserving task quality. Section 6 appropriately acknowledges this distinction, and it should be maintained consistently throughout the conclusions.

Questions for authors
  1. Independent initialization. Please confirm the manuscript's stated independent random initialization for NanoCLIP and identify any pretrained or frozen components. Which encoder parameters were reinitialized for each endpoint?
  2. Complete path validation. What explicit tolerance defines a low-loss connection, and what are the endpoint-inclusive maximum losses and excess barriers for the six pairs? Please clarify the relation between the dense interpolation checks in A.2 and the final DDPM segment in A.7.
  3. Comparative scope. Please report gradient-evaluation counts and clarify how the tested baseline and partial ablations support the claimed architectural and optimizer contributions. A complete-path forward-order-plus-SGD result would be helpful.
  4. Evaluation protocol. How are NanoCLIP training accuracy and held-out Recall@1 computed, and how are DDPM loss evaluations controlled for sampling noise?
2 comments

Thank you for your careful review and constructive comments. We have added the endpoint initialization, training configuration and evaluation protocol in the latest version of the paper. We address your questions below:

  • Q1: Independent initialization

We thank the reviewer for requesting this clarification. The original NanoCLIP constructors initially load DINOv2-S and all-MiniLM-L6-v2 weights. Our endpoint-generation pipeline (generate_high_accuracy_model.py) can decide whether to retain pretrained weights. The NanoCLIP configuration explicitly uses freeze_encoder=False, so all the encoders are not frozen and the final endpoints do not share a common pretrained initialization.

  • Q2: Complete path validation

All previous works (Draxler et al., 2018; Garipov et al., 2018; Tian et al., 2026) of mode connectivity did not strictly define the low-loss connection or provide a threshold for low loss. In practice, the training loss is merely reduced rather than minimized, and the differences lie only in the quality of the path. If the training loss is sufficiently large that the model’s accuracy becomes unacceptable, we consider the construction of a low-loss connection to have failed.

A.2 checks continuity within the constructed trajectory from the start mode to the last tick checkpoint. A.7 complements A.2 by testing this terminal segment from the last checkpoint to the end mode. We present final DDPM segment only to highlight the interesting finding that endpoint parameter proximity does not imply a flat base.

  • Q3: Comparison scope
MethodDDPM gradient evaluationsNanoCLIP gradient evaluations
Ours1,122,869908,022
AutoNEB154,000154,000

Our method uses more total gradient evaluations because it performs repeated refinement over 16,400 DDPM ticks and 8,600 NanoCLIP ticks, whereas AutoNEB only optimizes 11 internal pivots. From this perspective, our method constructs a low-loss path that is more continuous and requires fewer optimizations for each point. In contrast, AutoNEB only contains a few discrete intermediate points, and the training loss is significantly higher than ours.

Due to limited computing resources, we did not complete complete paths for the forward-order schedule and SGD optimizer, but sufficient differences could already be observed within limited ticks. For example, in the NanoCLIP with SGD experiment, the loss value exceeded 1 at the 250th tick, so we only ran the experiments for 500 ticks.

  • Q4: Evaluation protocol

NanoCLIP training accuracy is an in-batch text-to-image top-1 accuracy, and the reported value is averaged across training batches. Held-out Recall@1 is also top-1 accuracy but evaluated on the complete held-out 20% Flickr30k split.

For training loss of DDPM, every image independently receives a freshly sampled diffusion timestep and Gaussian noise on each forward pass. For test loss, evaluation traverses the complete official Flowers102 validation split and reports a mean over all images. The validation images and preprocessing are fixed, but timesteps and Gaussian noise are freshly sampled.

References

  • Draxler et al. 2018: Essentially No Barriers in Neural Network Energy Landscape
  • Garipov et al. 2018: Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
  • Tian et al. 2026: Connecting independently trained modes via layer-wise connectivity

Thank you for the rebuttal. I am pleased to maintain my positive assessment.

Reviewer_EsuMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper describes an algorithm to construct a low-loss path between trained DDPM and CLIP models, in the context of (non-linear) mode connectivity. It introduces a dataflow-based layer moving strategy to a previously introduced layer-wise weight moving algorithm. It furthermore proposes to switch the optimizer used in the moving algorithm to Adam. These changes enable the computation of such low-loss paths for the considered model types, which are more complex than the models previously considered.

The dataflow-based layer moving strategy consists of defining a model-specific layer sequence, and the moving algorithm is applied to the model layer by layer according to that sequence. For the DDPM U-Net, the sequence is: first downsampling layer -> last upsampling layer -> second downsampling layer -> second upsampling layer, etc; For NanoCLIP, it involves introducing blocks from the image and text encoders in an interleaved manner.

Critical strengths
  • The proposed algorithm seems to successfully construct low-loss paths between trained DDPM and CLIP models. Furthermore, judging from Fig. 4, these paths seem to outperform (in case of NanoCLIP significantly outperform) the paths constructed by the previously proposed AutoNEB method (Draxler et al. 2018), which leads to more spiky paths.
  • The finding that low training loss does not imply high generation quality is interesting and points towards being careful with respect to which metric one defines mode connectivity.
Critical weaknesses
  • The paper seems to heavily build on the previous work Connecting Independently Trained Modes via Layer-Wise Connectivity (Tian 2026): In particular, the concepts of the variance sphere, related notation and the section introducing it are adopted very closely, and the algorithm for constructing the low-loss paths seems at least closely related. While this previous work is cited, the connections are not made clear in sections 3 and 4. It is hard for the reader to tell which parts are just depicting the previous work, and which parts are new contributions. I assume that the weighted movement and training refinment parts are from the previous work, and the dataflow-based layer moving strategy and Adam-based refinement are novel, but I would encourage the authors to clarify in the paper which parts are the novel contribution. Given my understanding is correct here, I find the changes relatively incremental compared to the previous work: it comes down to trying the algorithm on different types of models, changing the optimizer, and changing the sequence in which layers are processed.
  • The paper claims to be the first work to consider mode connectivity of generative models, and the first to consider mode connectivity beyond image classification tasks. There is however a growing literature (e.g. Juneja 2023; Imfeld 2024; Theus 2025; Tran 2025; Verma 2024; Li 2026) that considers linear mode connectivity for Transformers, i.e. generative models, and seems to be ignored in this statement. While the authors seem to distinguish linear mode connectivity (which is usually achieved via symmetry alignment or a shared prefix of the training trajectory) from general mode connectivity (which they achieve without alignment by considering non-linear paths through weight space), and the considered generative models in this paper are diffusion models instead of Transformers, it still seems misleading to claim that mode connectivity for generative models has never been considered before: Such a claim should better be qualified further, e.g. by more clearly distinguishing the permutation-alignment/linear mode connectivity literature from the literature that finds non-linear weight space paths without permutation alignment.
  • The dataflow-based layer moving strategy to my understanding means a model-specific layer sequence, and the moving algorithm is applied to the model layer by layer in that sequence. It is presented to follow from two rules: "(1) layers should be introduced in the direction of data flow according to model architectures; and (2) attention modules should be processed individually before proceeding to subsequent layers." I don't understand how the decided-upon sequence naturally follows from these rules: for instance, one could also interpret rule (1) to mean that the stages of the U-Net should be processed sequentially from left to right (forward order), instead of from outermost to innermost. In Fig. 5, the authors compare the forward order to their order and find theirs to be better, but the difference does not to be very clear-cut; it would also be interesting to see how it compares to other orders, e.g. a random permutation of layers. Since this sequence seems to be the main algorithmic contribution, I would prefer the finding that this is the "correct" sequence to be supported by more evidence.
  • The presentation and mathematical depiction could be significantly improved, and it's not entirely clear how the proposed algorithm operates:
    • The description of the algorithm in section 4.1 is not that easy to follow, and dives into the details immediately while in my opinion it would be important to first explain the general idea and the used symbols before using them. For example, at the point of the sentence "We first move the active layers of PiP_i to obtain M1M_1 toward the destination PEP_E , controlled by (α,β,γ)(\alpha, \beta, \gamma) and Ui\mathcal U_i", none of the symbols Pi,M1,α,β,γ,UiP_i, M_1, \alpha, \beta, \gamma, \mathcal U_i have been previously introduced, and it is up to the reader to figure them out from context (they are mentioned at the top of Algorithm 1, but also not further explained). It seems like the algorithm selects a subset of layers (the "active layers") and moves their weights closer to the destination PEP_E, but the description given in the text is hard to follow since this idea is not explained on a high level first. Furthermore, some key details of the algorithm are not clear: What is ΠUi\Pi_{\mathcal U_i}? What is P0PE⌢\overset{\frown}{P_0 P_E}? Alternatively, if the proposed algorithm is indeed broadly similar to the one in the previous work, its description could be scaled back with reference to the previous work, and the new contributions could take a larger part.
    • In the description of the variance sphere, in equation (3), it is unclear to me what the variance refers to. Generally, the authors seem to write Var\text{Var} for the entry-wise empirical variance compute from the entries of a kk-dimensional vector. In equation (3), they write "θ0,θ0′=Init(M)⟹Var(θ0)=Var(θ0′)\theta_0, \theta'_0 = \mathrm{Init}(\mathcal M) \Longrightarrow \mathrm{Var}(\theta_0) = \mathrm{Var}(\theta_0')". This might make sense if θ0,θ0′\theta_0, \theta_0' are random variables following the initialization distribution, but it doesn't make sense if they are fixed vectors, and Var\mathrm{Var} is the entry-wise empirical variance: In that case, Var(θ0)≈Var(θ0′)\mathrm{Var}(\theta_0) \approx \mathrm{Var}(\theta_0') might hold true but we would not expect the empirical variances computed over the samples to be exactly equal; it should be made clearer what is meant. In general, the variance sphere section seems to boil down to the calculation that Var(θk)=1dk∑i(θik−θk‾)2≈1dk∑i(θik)2=1dk∥θk∥22∝∥θk∥22\mathrm{Var}(\theta^k) = \frac{1}{d_k} \sum_i (\theta^k_i - \overline {\theta^k})^2 \approx \frac{1}{d_k} \sum_i (\theta^k_i)^2 = \frac{1}{d_k} \Vert \theta^k \Vert_2^2 \propto \Vert \theta^k \Vert_2^2 under the approximation that the mean θk‾≈0\overline {\theta^k} \approx 0.
    • The authors distinguish between parameter vectors θ\theta and points in parameter space PP. As far as I can tell, these are the same mathematical objects, but the points in parameter space are used for the geometrical discussions. The descriptions could be simplified by only taking one of those viewpoints (my feeling is that using parameter vectors θ\theta would be more standard and easier understood by readers in the context of the literature).

References:

  • Tian et al. 2026: Connecting independently trained modes via layer-wise connectivity
  • Juneja et al. 2023: Linear Mode Connectivity Reveals Generalization Strategies
  • Imfeld et al. 2024: Transformer Fusion with Optimal Transport
  • Verma et al. 2024: Merging Text Transformer Models from Different Initializations
  • Theus et al. 2025: Generalized Linear Mode Connectivity for Transformers
  • Tran et al. 2025: On Linear Mode Connectivity of Mixture-of-Experts Architectures
  • Li et al. 2026: Scaling Linear Mode Connectivity and Merging to Billion Parameter Pretrained Transformers
Questions for authors

Q1: Is the algorithm broadly the same as in the Tian et al. 2026 paper, or are there changes beyond changing the layer sequence and optimizer?

Q2: It seems like the algorithm supports optimizing not only single layers sequentially, but subsets of layers at a time (and the ablations compare sequential optimization with optimization of all layers at the same time). Have you also considered optimizing proper subsets, like two or three related layers, at the same time?

Q3: Regarding the finding of low training loss not implying high generation quality: I find this observation interesting, do you have an idea what might explain this discrepancy? Can a well-performing path with respect to the FID score also be constructed using a similar path construction algorithm?

3 comments

Thank you for highlighting these related studies. We have added the papers in the table below and improved the mathematical depiction in the latest version of the paper.

But it should be emphasized that our statement indicates that mode connectivity has not been studied in models outside of classifiers before, which is based on the definition of mode connectivity emphasized in the paper: we explicitly restrict our scope to mode connectivity between independently-trained models without further permutation. Therefore, we acknowledge that models outside of classifiers have indeed been studied in the linear mode connectivity field, but we are still the first work in the field of mode connectivity.

WorkModelsDatasetsPretrainingPermutation
Juneja et al., 2023BERT-baseMNLI, QQP, CoLAYesNo
Verma et al., 2024BERT-base (MultiBERTs)WikiText-103, GLUENoYes
Theus et al., 2025ViT, GPT-2CIFAR-10, CIFAR-100, Tiny ImageNet, Tiny Shakespeare, BookCorpusNoYes
Tran et al., 2025ViT-MoE, GPT-2-MoEMNIST, CIFAR-10, CIFAR-100, ImageNet-1k, WikiText-103, One Billion WordYesYes
Li et al., 2026ViT, Pythia, OLMoImageNet-1k, WikiText-103NoYes

We address your questions below, following the order in which they were raised:

  • Q1: Is the algorithm broadly the same as in the Tian et al. 2026 paper?

We agree that the manuscript should distinguish the inherited components from our contributions more clearly.

Inherited from Tian et al., 2026: the variance-sphere observation, variance correction and the framework of movement, correction and retraining.

New in our paper: we reorganize the algorithm into two parts, weighted movement and training refinement. Dataflow-based strategy and Adam-based refinement are the main modifications to each part. Most importantly, these designs adapt the framework to diffusion and contrastive models, and extend mode connectivity to models outside of classifiers, which is missing from the work of Tian et al., 2026.

  • Q2: Have you also considered optimizing proper subsets, like two or three related layers, at the same time?

Yes, our algorithm supports and indeed optimizes more than one layer at a time. For example, downs.0.0 of DDPM in Table 4 actually consists of 8 layers, including downs.0.0.block1.norm.g, downs.0.0.block1.proj.bias, downs.0.0.block1.proj.weight, etc. To help assess the effect of layer selection and movement order, we are conducting an ablation of randomly moving layers, but the results will be presented in the final version of the paper due to the long running time.

  • Q3: Why can low denoising loss coexist with worse FID?

Training loss and FID are two unrelated indicators, and low training loss cannot guarantee the quality of generated images. Dongwoo Kim, 2026 provides a more detailed explanation: the choice of loss mainly controls the denoising quality (MSE), while the choice of prediction space mainly controls the sample quality (FID). Currently, our path construction algorithm does not guarantee that a low-FID path exists. It requires modifying our optimization objectives and conducting targeted design.

References

  • Juneja et al. 2023: Linear Mode Connectivity Reveals Generalization Strategies
  • Verma et al. 2024: Merging Text Transformer Models from Different Initializations
  • Theus et al. 2025: Generalized Linear Mode Connectivity for Transformers
  • Tran et al. 2025: On Linear Mode Connectivity of Mixture-of-Experts Architectures
  • Li et al. 2026: Scaling Linear Mode Connectivity and Merging to Billion Parameter Pretrained Transformers
  • Tian et al. 2026: Connecting independently trained modes via layer-wise connectivity
  • Dongwoo Kim 2026: Analyzing Learning-Dynamics Across Loss and Prediction Choices for Diffusion Models

I thank the authors for their response and appreciate the inclusion of the related literature on LMC for Transformers. The authors now state that Linear Mode Connectivity has been studied beyond classifiers, but they are the first to study non-classifiers for Mode Connectivity. I find this choice of terminology confusing to potential readers, given that one would expect that "Mode Connectivity" encompasses "Linear Mode Connectivity". The distinguishing factor rather seems to be that unlike the LMC literature, this branch of arbitrary-path mode connectivity does not allow the application of function-preserving transformations to the weights while allowing for non-linear paths.

The authors' response confirms that the main contributions are: 1. two algorithmic changes, which are a specific layer processing sequence and the adoption of Adam, and 2. the application of the modified algorithm to diffusion and contrastive models, i.e. models trained with a different objective. My concern remains that the contribution is relatively small. While it might be valuable to show that a certain algorithm works on another class of models even with no or with small modifications, I am missing a coherent story about what makes these contrastive/diffusion models sufficiently different from the classifier models previously considered in terms of mode connectivity, and why this proposed change is necessary to make it work. Also, I don't feel like the layer processing sequence contribution is sufficiently evaluated (e.g. in Fig. 4 their order it shows a 0.01 advantage in training loss along the path over the forward order, but that seems to be a single pair of models, not the average over multiple pairs; also, there might be other sequences that perform just as well or better, which hasn't been evaluated).

I could not see significant changes to the mathematical depiction and the presentation of the algorithm in the revised manuscript despite the claim (the highlighted changes are relatively minimal). While the authors acknowledge in the rebuttal that the inherited components of the previous work and the differentiation of inherited and novel algorithmic components should be made more clear, I do not see such changes to the manuscript, and my concerns remain. Furthermore, the authors haven't addressed my question about how exactly the decided-upon layer sequence follows from the set-out rules (I see this more as a presentation quirk, it seems one could also just leave out these rules of the text).

Given that several concerns remain, I maintain my score.

Thank you for clarifying your remaining concerns. Indeed, the similar expressions "mode connectivity" and "linear mode connectivity" are easy for readers to connect them, but there are significant differences between the two in terms of research settings, objectives, and experimental protocols. That's why we specifically distinguish between the two in related work and claim the first to study non-classifiers for mode connectivity.

  • What makes these contrastive/diffusion models sufficiently different from the classifier models previously considered in terms of mode connectivity, and why this proposed change is necessary to make it work?

In our evaluated settings, there are two main differences between the comparison/diffusion model and the classifier model -- model architecture, as well as differences in larger parameter distances and learning objectives. Accordingly, we proposed a dataflow-based layer moving strategy to accommodate the architecture of DDPM and NanoCLIP, while Adam-based refinement provides adaptive updates to adapt to larger layer wise separations and learning objectives of contrast/diffusion model.

  • The layer processing sequence contribution is not sufficiently evaluated.

To strengthen the evaluation of the layer moving strategy, we have added a random-order comparison as shown in Figure 5 in the latest revision. Random-order movement increases average training loss by approximately 25.6% on DDPM and 2.6% on NanoCLIP over the respective evaluated tick ranges. This complements the forward-order comparison and broadens the evidence supporting our strategy. We think an exhaustive search over all possible sequences is beyond the scope of this study, and we do not claim that the proposed ordering is uniquely optimal.

  • How exactly the decided-upon layer sequence follows from the set-out rules?

We apologize for not directly addressing how the concrete layer sequences follow from the stated rules in our initial response. We provide the clarification here, as also discussed in our response to Reviewer FwBt:

Architecture-derived choices. Layers are introduced in the direction of data flow, and attention modules are processed individually before proceeding to subsequent layers. For standard acyclic sequential architectures, these rules allow the sequence to be generated automatically from the computational graph. Manual choices. For complex non-sequential architectures, applying these rules requires identifying corresponding feature-processing stages and specifying their granularity. These choices include grouping DDPM encoder and decoder blocks by spatial resolution and specifying the interleaving schedule between NanoCLIP’s image and text encoders.

Lastly, the revisions to the mathematical presentation and algorithm primarily clarify previously unexplained notation. We have also explicitly attributed the inherited variance sphere component in Section 3 in the latest version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.