- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies how time-dependent loss weighting affects the training of diffusion and flow-matching models on multimodal distributions. Focusing on high-dimensional Gaussian mixtures, the authors analyze gradient-flow dynamics at fixed noise levels. At high SNR, the network learns mode directions while the mode-probability parameter remains frozen in the analyzed asymptotic dynamics. Near speciation, directions and probabilities are learned jointly. For hierarchical mixtures, the noise level also determines whether directions of different strengths are acquired together or on different timescales. The authors then use effective SNR weighting to interpret integrated training objectives and investigate these predictions numerically.
Critical strengths
-
The paper provides an interpretable analysis of training dynamics for unbalanced and hierarchical Gaussian mixtures. Tracking direction alignment, orthogonal components, and mode probabilities makes the different aspects of structure learning explicit. I find the comparison of their learning timescales across noise regimes particularly interesting.
-
The controlled Gaussian-mixture experiments are well aligned with the theoretical questions. In particular, comparing different weightings while keeping the network and diffusion process fixed helps isolate the role of the training objective. The experiments with image and genomic data also provide useful evidence about the scope of the predicted learning patterns.
Critical weaknesses
-
The analytical setting is restrictive. The score networks in Section 3.1 are tailored to the known Gaussian-mixture score, and the hierarchical model assumes that the two learned directions occupy prescribed disjoint coordinate blocks. Moreover, Eq. (8) describes gradient flow on the population loss rather than empirical-loss optimization. Although the generic-network experiments partly address these limitations, the extent to which the analytical conclusions depend on these assumptions remains unclear.
-
There is a gap between the fixed-SNR analysis in Section 3.1 and the interpretation of integrated training objectives in Section 3.2. A shared network receives gradients from multiple noise levels, whose interactions are not characterized by the fixed-SNR results. The effective weights in Eq. (14) therefore provide a useful interpretation, but do not by themselves establish quantitative predictions for integrated training and show the optimality of such weighting schedules.
Questions for authors
-
Which conclusions of Results 1 and 2 are expected to remain valid for generic network architectures and finite-sample training?
-
Could you clarify the connection between the effective SNR weights in Eq. (14) and weighting schemes motivated by final sampling-error guarantees? For example, Theorem 1 of “Maximum Likelihood Training of Score-Based Diffusion Models” (Song et al. (2021)) bounds the reverse-SDE output’s KL error using likelihood-weighted score error. In your OU setting, this corresponds to the noise-prediction weight How would your analysis predict feature-learning behavior under this weighting? Or can you design the "optimal" weighting schedule based on your insight?
4 comments
Response to Reviewer pNkV [2/2]
Q2 (likelihood weighting). We thank the reviewer for this very interesting question, which highlights the difference between weightings motivated by the final samples and by the training dynamics. Song et al. choose the weighting of the training loss so that it bounds the KL error of the final samples: it tells which score errors matter at the end of training. Our effective weighting instead tells which features are learned first, and a weighting that accelerates a given feature need not be optimal for score-error minimization.
Our framework allows us to assess the effect on training dynamics of the KL-based weighting. In our setting becomes for small . This is exactly the same behavior as standard diffusion, hence we expect uniformly weighted DSM-like behavior: directions learned from Regime II, and the weight learned from the noise levels around speciation.
As we now state in the revised version, our results should not be read as optimality results but as insights into the noise scales relevant for each feature; principled schedule design remains future work.
We thank the reviewer again and remain available for any further question.
I thank the authors for the detailed response and revisions. The revised conclusion clarifies which predictions carry over to generic networks and the new two-noise-level example in Sec. 3.2 helps connect the fixed-SNR analysis to integrated training. I understand that the remaining modeling assumptions are needed for tractability, and the experiments support the main idea. I will maintain my positive rating.
We thank the reviewer for their suggestions and positive assessment of the revised version of the manuscript, a process which we believe has improved the clarity and impact of the paper.
Response to Reviewer pNkV [1/2]
We warmly thank the reviewer for the positive and encouraging assessment. We are glad that the reviewer found the comparison of learning timescales across noise regimes interesting, and that they see the value of the controlled experiments, in which only the weighting changes. We have built further on these experiments in the revision: the MNIST experiment now follows the same controlled protocol as the ResNet one, and the ResNet experiments are extended to larger dimensions. Below we answer each of the reviewer's questions.
W1 (restrictive setting).
W2 (fixed SNR vs. integrated objective). We have now added a new part to Sec. 3.2 in order to present an analysis of the integrated loss. We use a time-shared exact parametrization of the score. This encodes in our model two features of practical training: the loss is averaged over noise levels, and a single network, conditioned on the noise level, is shared across all of them. The integrated dynamics is then the weff-weighted superposition of the single-SNR dynamics, which yields closed-form timescales for two-time weightings. For real networks, we expect the weight-related contrast between weightings, such as the transient even split under uniform DSM, to depend on how much the parameters are shared across noise levels: it is sharp in the unshared limit and, as our solvable model shows, lifted in the fully shared one. Our ResNet results are consistent with weak sharing. The amplitude ordering of features, in contrast, appears in all our experiments. Finally, we agree with the reviewer on optimality: we have revised the tone of Sec. 3.2, and now state that the effective weights provide insights rather than prescriptions.
Q1 (what survives for generic networks and finite samples). The revised manuscript answers this question explicitly, in the revised conclusion, which compares the exact-score models with the generic networks of Fig. 3 (ResNet), Fig. 4 (U-Net on MNIST) and Fig. 5 (MDLM on HGD). In short:
On finite samples, see W1.