acceptodds
Under review as a conference paper at ICLR 2027

Weighting Schedules Govern What and When Score-Based Generative Models Learn from Multimodal Data

Abstract

Score-based generative models generate new samples by integrating a time-dependent drift that carries Gaussian noise onto the target distribution. In practice this drift is modeled by a neural network, trained on a loss integrated over time with a weighting schedule . Along the backward dynamics, and for multi-modal distributions, trajectories commit to modes of the target within a narrow time window, the speciation time. In this work, focusing on high-dimensional data, we decompose the integrated loss into its single-time contributions and analyze each at fixed signal-to-noise ratio : we show that sets the rate at which each feature of a multimodal target—the mode directions and their relative weights—is acquired during training. Crucially, at high all mode directions are acquired together, on a single timescale insensitive to their amplitudes, while the relative weights are not learned at all. Only near the speciation time, where becomes of order one, do all features become learnable, each on its own timescale: the weights are acquired jointly with the directions, and the directions at rates set by their relative amplitudes. For models trained on time-integrated objectives, the learning dynamics is then governed by how much of the weighting effectively sits near the speciation time, which provides insights on design choices. These results follow from an exact high-dimensional analysis of the training dynamics of unbalanced and hierarchical Gaussian mixtures. Numerical experiments on image and human genome haplotype generation recover the predicted hierarchy of learning timescales in more complex settings.

Primary area:
generative models
Keywords:
Score-Based Generative Models, Multimodal Data, Weighting Scheme

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_pNkVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how time-dependent loss weighting affects the training of diffusion and flow-matching models on multimodal distributions. Focusing on high-dimensional Gaussian mixtures, the authors analyze gradient-flow dynamics at fixed noise levels. At high SNR, the network learns mode directions while the mode-probability parameter remains frozen in the analyzed asymptotic dynamics. Near speciation, directions and probabilities are learned jointly. For hierarchical mixtures, the noise level also determines whether directions of different strengths are acquired together or on different timescales. The authors then use effective SNR weighting to interpret integrated training objectives and investigate these predictions numerically.

Critical strengths
  1. The paper provides an interpretable analysis of training dynamics for unbalanced and hierarchical Gaussian mixtures. Tracking direction alignment, orthogonal components, and mode probabilities makes the different aspects of structure learning explicit. I find the comparison of their learning timescales across noise regimes particularly interesting.

  2. The controlled Gaussian-mixture experiments are well aligned with the theoretical questions. In particular, comparing different weightings while keeping the network and diffusion process fixed helps isolate the role of the training objective. The experiments with image and genomic data also provide useful evidence about the scope of the predicted learning patterns.

Critical weaknesses
  1. The analytical setting is restrictive. The score networks in Section 3.1 are tailored to the known Gaussian-mixture score, and the hierarchical model assumes that the two learned directions occupy prescribed disjoint coordinate blocks. Moreover, Eq. (8) describes gradient flow on the population loss rather than empirical-loss optimization. Although the generic-network experiments partly address these limitations, the extent to which the analytical conclusions depend on these assumptions remains unclear.

  2. There is a gap between the fixed-SNR analysis in Section 3.1 and the interpretation of integrated training objectives in Section 3.2. A shared network receives gradients from multiple noise levels, whose interactions are not characterized by the fixed-SNR results. The effective weights in Eq. (14) therefore provide a useful interpretation, but do not by themselves establish quantitative predictions for integrated training and show the optimality of such weighting schedules.

Questions for authors
  1. Which conclusions of Results 1 and 2 are expected to remain valid for generic network architectures and finite-sample training?

  2. Could you clarify the connection between the effective SNR weights in Eq. (14) and weighting schemes motivated by final sampling-error guarantees? For example, Theorem 1 of “Maximum Likelihood Training of Score-Based Diffusion Models” (Song et al. (2021)) bounds the reverse-SDE output’s KL error using likelihood-weighted score error. In your OU setting, this corresponds to the noise-prediction weight wKL(t)=21−e−2t.w_{\mathrm{KL}}(t)=\frac{2}{1-e^{-2t}}. How would your analysis predict feature-learning behavior under this weighting? Or can you design the "optimal" weighting schedule based on your insight?

4 comments
AuthorsOpenReview ↗

Response to Reviewer pNkV [1/2]

We warmly thank the reviewer for the positive and encouraging assessment. We are glad that the reviewer found the comparison of learning timescales across noise regimes interesting, and that they see the value of the controlled experiments, in which only the weighting changes. We have built further on these experiments in the revision: the MNIST experiment now follows the same controlled protocol as the ResNet one, and the ResNet experiments are extended to larger dimensions. Below we answer each of the reviewer's questions.

W1 (restrictive setting).

  • Tailored architecture. Score-mirroring architectures are the standard setting in this literature (Cui et al. 2024, 2025; Shah et al. 2023), as they make the training dynamics analytically tractable. In fact, for Gaussian mixture data models, no parametrization of the score is currently known that remains analytically tractable in high dimensions.
  • Disjoint blocks. We used disjoint blocks as a shortcut to avoid mode collapse, and as a technical requirement of the tanh⁡\tanh construction of the exact score. Note, however, that the ResNet and U-Net experiments impose no such structure and still display the hierarchy.
  • Population gradient flow. Sample complexity is an interesting and difficult question, but it lies beyond the scope of this work. Our results are a necessary first step: they lay the groundwork for analyzing training with the empirical loss. In practice, all our experiments, including the exact-score ones, are trained on sampled minibatches rather than by gradient flow on the population loss, and this does not seem to affect the predicted phenomenology. The MNIST and HGD experiments are moreover strongly multi-epoch, and the sequential acquisition of features persists there.

W2 (fixed SNR vs. integrated objective). We have now added a new part to Sec. 3.2 in order to present an analysis of the integrated loss. We use a time-shared exact parametrization of the score. This encodes in our model two features of practical training: the loss is averaged over noise levels, and a single network, conditioned on the noise level, is shared across all of them. The integrated dynamics is then the weffw_{\rm eff}-weighted superposition of the single-SNR dynamics, which yields closed-form timescales for two-time weightings. For real networks, we expect the weight-related contrast between weightings, such as the transient even split under uniform DSM, to depend on how much the parameters are shared across noise levels: it is sharp in the unshared limit and, as our solvable model shows, lifted in the fully shared one. Our ResNet results are consistent with weak sharing. The amplitude ordering of features, in contrast, appears in all our experiments. Finally, we agree with the reviewer on optimality: we have revised the tone of Sec. 3.2, and now state that the effective weights provide insights rather than prescriptions.

Q1 (what survives for generic networks and finite samples). The revised manuscript answers this question explicitly, in the revised conclusion, which compares the exact-score models with the generic networks of Fig. 3 (ResNet), Fig. 4 (U-Net on MNIST) and Fig. 5 (MDLM on HGD). In short:

  • Holds in all generic networks: features of decreasing amplitude are acquired sequentially, in the order of their amplitudes (Result 2b; Figs. 3b, 4b, 5).
  • Depends on parameter sharing across noise levels: the frozen weight under DSM and the quantitative separation of onset times with the amplitude (Results 1–2; Fig. 3), which we now discuss together with the parameter-sharing limits of Sec. 3.2.
  • Not observed: a clear DSM/FM contrast on MNIST, even under the controlled protocol (Fig. 4a), and the quantitative collapse beyond the two leading haplotype directions (Fig. 5).

On finite samples, see W1.

AuthorsOpenReview ↗

Response to Reviewer pNkV [2/2]

Q2 (likelihood weighting). We thank the reviewer for this very interesting question, which highlights the difference between weightings motivated by the final samples and by the training dynamics. Song et al. choose the weighting of the training loss so that it bounds the KL error of the final samples: it tells which score errors matter at the end of training. Our effective weighting instead tells which features are learned first, and a weighting that accelerates a given feature need not be optimal for score-error minimization.

Our framework allows us to assess the effect on training dynamics of the KL-based weighting. In our setting wKL(t)=2/(1−e−2t)w_{\rm KL}(t)=2/(1-e^{-2t}) becomes weffKL=d/(Λ(d−σ2Λ))≃1/Λw_{\rm eff}^{\rm KL}=d/(\Lambda(d-\sigma^2\Lambda))\simeq1/\Lambda for small Λ\Lambda. This is exactly the same behavior as standard diffusion, hence we expect uniformly weighted DSM-like behavior: directions learned from Regime II, and the weight learned from the noise levels around speciation.

As we now state in the revised version, our results should not be read as optimality results but as insights into the noise scales relevant for each feature; principled schedule design remains future work.

We thank the reviewer again and remain available for any further question.

I thank the authors for the detailed response and revisions. The revised conclusion clarifies which predictions carry over to generic networks and the new two-noise-level example in Sec. 3.2 helps connect the fixed-SNR analysis to integrated training. I understand that the remaining modeling assumptions are needed for tractability, and the experiments support the main idea. I will maintain my positive rating.

We thank the reviewer for their suggestions and positive assessment of the revised version of the manuscript, a process which we believe has improved the clarity and impact of the paper.

Reviewer_LriQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The paper studies the effect of the weighting schedule in the training of diffusion models and flow matching, which governs which noise levels are emphasized during training. The paper analyzes this problem for tractable data distributions given by two and four-mode Gaussian mixtures in the high-dimensional regime, using a shallow neural network whose structure mimics that of the true score function. When training a score network at a fixe time, it is shown that when the signal-to-noise ratio (SNR) is O(d)O(d), the data structure is fully resolved, and the model is only able to learn the locations of each mode (but not the relative weights); in contrast, for SNR=O(1)SNR = O(1), the network jointly learns the locations and weights of each mode. For the time-integrated loss, this motivates choosing a weighting schedule which emphasizes the time window in which the network is able to learn both the locations and weights of each mode. Experiments are presented to illustrate the theory.

Critical strengths

Understanding feature emergence in generative models is an important problem. While most existing works study this problem from the sampling perspective, this work considers the effect of design choices in the training procedure on feature emergence, and in turn offers some practical insights for how to choose the weighting schedule.

Critical weaknesses

The analysis is carried out for a rather simplified and specific family of distributions; while I understand that such assumptions are needed to make the theory tractable, it limits the scope of the implications of the paper. Part of the analysis also assumes that the norm of the weight vector of the neural network is fixed, which does not hold in practice.

I also found the presentation to be very difficult to follow. For instance, the theoretical results are stated very informally and are not suitably interpreted, and the experimental results are not clearly tied back to the theory. The results seem to suggest a principled way of choosing the weighting schedule, but this is not clearly explained.

Questions for authors

For both D1\mathcal{D}_1 and D2\mathcal{D}_2, each mode is assumed to have the same (isotropic) covariance. Can the results be generalized to the case where each mode has a different variance, or a common anisotropic local covariance matrix?

3 comments
AuthorsOpenReview ↗

Response to Reviewer LriQ

We thank the reviewer for the report and for acknowledging the novelty of studying feature emergence from the training rather than the sampling side. We understand that some presentation choices made the paper harder to read than it should be, and we list the concrete changes made in the revised manuscript under W2.

W1 (scope; fixed-norm assumption). Gaussian mixtures are the standard setting for this question in the statistical physics analysis of diffusion (Biroli & Mézard, 2023; Biroli et al., 2024; Raya & Ambrogioni, 2023), in the learning theory of score-based models (Cui et al., 2024, 2025; Aranguri & Insulla, 2024; Shah et al., 2023), and in statistics, where mixture estimation is the textbook latent-variable problem. We chose this setting because it makes each feature (direction, weight, amplitude hierarchy) an explicit order parameter whose timescale can be computed exactly. Going beyond this setting is an open problem. In fact, beyond Gaussian mixtures, no data model or parametrization of the score is currently known in which both the data distribution and a learned score remain analytically tractable in high dimension. The revision states this scope early and presents Sec. 4 as explicit tests of its limits.

The fixed-norm constraint is not used to derive Results 1–2, which are exact. We agree that it does not hold in practice: we use it only as a heuristic that yields a closed-form timescale λ(γ)\lambda(\gamma). This closed form is then tested against dynamics in which the norm is not constrained: the exact-score runs of Fig. 2 train the unconstrained model by SGD, and rescaling time by λ(γ)\lambda(\gamma) collapses them.

W2 (presentation). We aimed for a balance between mathematical rigor and interpretability in the main text, especially in the statements of Results 1–2, and we take the reviewer's comment as a sign that we did not reach it. We use the extra page of the revision for a more detailed presentation. We would gladly welcome further suggestions from the reviewer if another arrangement reads better; so far, we have made the following changes:

  1. Results 1–2 are expanded into full theorem statements, self-contained and with all assumptions (initialization, exact closure of the dynamics at any dd, timescale and limit of each summary statistic), each followed by an interpretation paragraph.
  2. Each experiment in Sec. 4 ends with an explicit prediction / observation / verdict.
  3. Sec. 3.2 now states the qualitative guiding principle the reviewer asked for. A feature is learned at a rate set by the training signal the weighting places where that feature's information lies. For mode weights this is Λ=O(1)\Lambda=O(1); for mode directions it is Regime II. Shifting effective weight towards Λ≈1\Lambda\approx1 therefore accelerates weight learning.

Q1 (other covariances). Both extensions can be (partially) carried out within our framework; we sketch them below. However, our view is that they do not change the main messages of the paper, which is why we chose not to add them to the revised version. We would be happy to include these discussions as an appendix if the reviewer finds them useful, and we leave this choice to the reviewer.

  • Shared anisotropic covariance Σ\Sigma. The exact score is −St−1x+αSt−1μtanh⁡(αμTSt−1x+b∗)-S_t^{-1}\mathbf{x}+\alpha S_t^{-1}\boldsymbol{\mu}\tanh(\alpha\boldsymbol{\mu}^TS_t^{-1}\mathbf{x}+b^*) with St=α2Σ+β2IS_t=\alpha^2\Sigma+\beta^2 I. Our architecture carries over with the scalar skip connection replaced by a matrix, and the relevant SNR becomes α2μTSt−1μ\alpha^2\boldsymbol{\mu}^TS_t^{-1}\boldsymbol{\mu}. For a spectrum of order one, the anisotropy drops out at the speciation scale, where α2=O(1/d)\alpha^2=O(1/d), so Results 1b–2b are unchanged at leading order. In Regime II the frozen-bias mechanism only needs the tanh⁡\tanh to saturate, so it persists. What changes is the Gaussian part, now learned eigendirection by eigendirection as in the linear theory of Wang et al. (2026). Covariance eigenvalues of order dd, comparable to the mode separation, are a genuinely different regime, out of the scope of the present theory.
  • Mode-dependent variances. The tanh⁡\tanh form is no longer exact: the score becomes a softmax-gated mixture of two linear experts, with a gate quadratic in x\mathbf{x}. We nevertheless expect the speciation / Regime II picture to be unchanged. In particular, in Regime II the softmax saturates to a hardmax, so the frozen-bias mechanism persists, which we checked numerically. Per-mode width parameters then receive gradient only from their own mode and should be learned at rates proportional to ω\omega and 1−ω1-\omega, the minority mode last.

We hope these changes and clarifications address the reviewer's concerns, and we would be grateful if the reviewer would take them into account in the final assessment. We are happy to clarify any remaining point.

Thank you for your reply. I agree that the mixture family you study is a natural choice for diffusion models/flow matching, and moving beyond Gaussian mixtures would require substantially new mathematical tools. The changes in presentation do make the results easier to understand. I also think that my last question has been adequately answered, and it doesn't seem like generalizing all of the results to allow a common local covariance is worth the effort it would require. I have raised my score and recommend borderline acceptance.

We thank the reviewer for thoroughly going through the changes proposed in the revised version, endorsing the various choices made with respect to their comments and raising the overall score of the manuscript.

Reviewer_XnnLOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper asks which features of a multimodal target a score model learns during training, and when, depending on which noise levels the training objective emphasizes. The authors work in the stochastic-interpolant framework and use the signal-to-noise ratio (SNR) as a common noise coordinate. They define the speciation time as the time at which the SNR equals one.

The theory looks at training at a single noise level, for two Gaussian mixtures. D1 is bimodal with unbalanced mode weights. D2 has four modes along two orthogonal directions. The score model has the exact functional form of the target score, with trainable coefficients. Under population gradient flow, as d goes to infinity, the dynamics reduce to four summary statistics. In both regimes, only the skip coefficient moves during the first stage of training (times of order one), and the model learns the best unimodal Gaussian. The multimodal structure is learned later, on times of order d, and differently in each regime.

For D1 (Result 1), in Regime II (SNR of order d) the model learns the direction but never the mode weight, because the bias gets no gradient. At speciation (SNR of order one) it learns direction and weight together. For D2 (Result 2), in Regime II the model learns both directions at the same time. At speciation it learns them hierarchically, the dominant block first.

A fixed-norm ansatz gives a closed-form growth rate, which the paper uses throughout to rescale training time.

Section 3.2 rewrites the integrated loss in terms of the SNR and derives the effective weighting of Diffusion and of Flow Matching.

Section 4 tests these predictions with a ResNet on the Gaussian mixtures, a U-Net on unbalanced and tinted MNIST, and masked discrete diffusion on haplotypes.

Critical strengths
  • The question (which features the trained score resolves, as a function of the weighting) is the right complement to the speciation literature, which assumes the exact score.

  • The single-time analysis is exact and complete: the reduction of the high-dimensional gradient flow to a few summary statistics is carried out in full, the separation between a fast skip coefficient and slow structural variables is clearly identified.

  • The frozen-bias mechanism at high SNR (the tanh becomes a sign, so the bias receives no gradient) is simple and convincing, and is confirmed by the frozen-weight curves of Fig. 1.

  • The reduction of D2, at speciation, to two decoupled copies of D1 at rescaled SNRs is elegant and gives a clean explanation of hierarchical learning.

  • The Diffusion/Flow Matching comparison on the Gaussian mixtures is well controlled

Critical weaknesses

Theory

  • The theory only covers training at a single noise level. All results assume the model is trained at one fixed noise level, while real pipelines train on all noise levels at once. The paper itself says the single-level dynamics "do not simply superpose", and in the toy model one set of parameters cannot be optimal at two noise levels at once.

  • The comparison between Diffusion and Flow Matching require on the premise that the effective weighting depends on writing the loss as noise prediction. Does the analysis change when it is written as data prediction? Futhermore, the paper counts nominal weight instead training signal: Flow Matching puts most of its weight at very low SNR, where predicting the noise is trivial and contributes little. The step from "Flow Matching favours low SNR" to "Flow Matching training is dominated by speciation" relies on the claim that the useful signal peaks at speciation. Is this true?

  • Some timescale claims are stronger than what is derived. “Training deep in Regime II of Biroli et al. (2024) recovers every mode direction on a common timescale but leaves the relative weights untouched, however long one trains.”, is true only in the infinite-dimensional limit; at finite dimension the effect is a strong delay, and it is weak early in training.

Experimental Validation

The predicted difference between Diffusion and Flow Matching is shown convincingly in only one experiment:

  • The MNIST comparison is inconclusive, and its two pipelines differ in more than the weighting (different network outputs)
  • The Gaussian-mixture ResNet experiment (Fig. 3a) is the only clear confirmation, at a single small dimension (d = 64)
  • The four-mode experiment (Fig. 3b) shows only a difference of degree between the two weightings, without quantification

Related work

Please consider discussing the relationship with the related work Shah et al, Learning Mixtures of Gaussians Using the DDPM Objective https://proceedings.neurips.cc/paper_files/paper/2023/file/3ec077b4af90f2556b517b556e186f64-Paper-Conference.pdf

Minor/Presentation

Please consider adding to figures 1 and 2 some vertical bars to differentiate the two τ\tau regimes, and consider either changing the markers for the various quantities or add a legend/ arrows with the quantities m,c and mode weight

Questions for authors

Can the integrated objective be analyzed for the toy parametrization, even for a two-point weighting? What replaces the closed-form rate?

How sensitive are the collapses to the convention fixing the speciation SNR?

Under uniform weighting in Fig. 3a the ResNet does eventually learn the mode weight. Is that because the weighting still places non-negligible mass near speciation, or because a generic network can learn the weight at high SNR by a mechanism the toy bias cannot?

3 comments
AuthorsOpenReview ↗

Response to Reviewer XnnL [1/2]

We thank the reviewer for a very precise, encouraging and challenging report. In particular, the questions on the integrated objective spurred new analytical results, which we believe strengthen the revised manuscript. We answer each weakness and question below.

W1 (single noise level). We understand the concern, and hope to address it with the new discussion in Sec. 3.2. Nevertheless, we would like to stress that we see the single-noise-level analysis as the elementary building block of the integrated problem rather than as a limitation to be removed. It is also the setting of the closest theoretical works: Shah et al. (2023) train at one fixed noise level per stage, and Cui et al. (2024) at a fixed noising time. Our new analysis (Q1) makes the building-block picture precise. With a time-shared exact parametrization, the integrated dynamics is the weighted superposition of the single-SNR dynamics, so the integrated timescales are assembled from the fixed-SNR ones.

W2a (noise vs. data prediction). It is true that the shape of the effective weights depends on whether the noise or data residual is used: "Diffusion is scale-free" is a statement in the noise-prediction convention, and does not hold for data prediction. The comparison between pipelines, however, does not depend on it: the ratio weffFM/weffDM≃d/Λw^{\rm FM}_{\rm eff}/w^{\rm DM}_{\rm eff}\simeq\sqrt{d/\Lambda} is the same for every residual, and all comparative statements of Sec. 3.2 rest on it. The revision states the convention and this invariance explicitly (App. A.3).

W2b (does the useful signal peak at speciation?). We agree that nominal weight is not training signal. For the mode weight, the answer is yes: in Regime II the tanh⁡\tanh saturates and the bias receives an exponentially small gradient (the frozen-bias mechanism of Result 1), so the weight-learning signal is concentrated around Λ=O(1)\Lambda=O(1). For the mode directions it is not: their signal is dominated by Regime II. We therefore restrict the claim to the mode weight, and replace "FM training is dominated by speciation" by "FM places more training signal than Diffusion on the speciation window, where the mode-weight information lies".

W3 (timescale claims). The reviewer is correct. Since the reduction of the gradient flow to the summary statistics (c,m,q,b)(c,m,q,b) is exact at every dd, we integrated it at finite dd. At fixed tt in Regime II, the weight is learned on a timescale exponentially large in dd, while directions are learned on O(d)O(d). The revision now states, after Result 1, that at fixed tt in Regime II "the weight is learned only on a timescale exponentially large in dd, and not at all as d→∞d\to\infty" (derivation in App. A.2).

W4 (experiments). We agree with the diagnosis. The revised Sec. 4 states which predictions of the GMM theory are expected to hold quantitatively, and discusses the observed discrepancies.

  • MNIST: we reran the experiment with a protocol mirroring the ResNet one (fixed architecture, SGD with momentum, only the weighting changes). This better isolates the role of the weighting, but the DSM/FM contrast remains unclear. We now discuss this in light of the parameter-sharing analysis of Sec. 3.2: a U-Net shares its parameters strongly across noise levels, which, we argue, should blur the contrast.
  • Dimension: we ran additional ResNet experiments on the unbalanced mixture at d∈{128,256}d\in\{128,256\}, shown in a new appendix figure (Fig. 7), and recover the same phenomenology; the even-split plateau under DSM even lengthens with dd, while the FM curves barely depend on it. This is consistent with the theory, for which d=64d=64 is if anything the least favorable case: the FM/DM ratio of effective weights at speciation grows as d\sqrt d and the Regime-II bias timescale exponentially.
  • Fig. 3b: the message of the figure is that DSM and FM display the same phenomenology. The statement that one weighting "separates more cleanly", which referred to the smoothness of the curves, has been removed: the text now states that both display the same ordering, differing in degree rather than in kind.

W5 (Shah et al., 2023). We thank the reviewer for pointing us to this closely related work, which we now cite and discuss in the related works and in Sec. 3 as a complementary result. Shah et al. prove that gradient descent on the denoising score matching objective recovers the centers of an equally weighted Gaussian mixture, relating the high- and low-noise dynamics to power iteration and EM, whereas we characterize the timescales on which each feature, including the mixing weights, is learned as a function of the SNR and of the weighting.

W6. We added explicit legends for mm, cc and the weight in Figs. 1–2, and the location of each regime is now marked in the captions of both figures.

AuthorsOpenReview ↗

Response to Reviewer XnnL [2/2]

Q1 (integrated objective, two-time weighting). Yes, and we thank the reviewer for this question, which led to the new analysis of Sec. 3.2 (detailed derivations in the new App. A.4). In our single-time parametrization the optimum w∗=αμ/Γ\mathbf{w}^*=\alpha\boldsymbol{\mu}/\Gamma depends on tt, so, as the reviewer notes, one set of parameters cannot be optimal at two noise levels. We therefore move the tt-dependence into the architecture,

sθ(x,t)=1Γt[c x+αt wtanh⁡(αtΓtwTx+b)],s_\theta(\mathbf{x},t)=\frac{1}{\Gamma_t}\Big[c\,\mathbf{x}+\alpha_t\,\mathbf{w}\tanh\Big(\frac{\alpha_t}{\Gamma_t}\mathbf{w}^T\mathbf{x}+b\Big)\Big],

whose optimum c=−1c=-1, w=μ\mathbf{w}=\boldsymbol{\mu}, b=b∗b=b^* is the same at every noise level. At each tt this is our original model up to a rescaling of (c,w)(c,\mathbf{w}), so the exact single-time reduction carries over, and the single-time loss depends on tt only through the SNR Λ\Lambda. The integrated dynamics is then the weffw_{\rm eff}-weighted superposition of the single-SNR dynamics. For a two-time weighting, with mass π1\pi_1 in Regime II (Λ1=O(d)\Lambda_1=O(d)) and π2\pi_2 at speciation (Λ2=O(1)\Lambda_2=O(1)), the closed-form rate is replaced by two clocks, with ϵj=αj2βj2/Γj2\epsilon_j=\alpha_j^2\beta_j^2/\Gamma_j^2:

  • direction: driven by the Regime-II mass, on τdir≃d/π1ϵ1\tau_{\rm dir}\simeq d/\pi_1\epsilon_1, blind to the mode amplitudes;
  • weight: once the direction is aligned, relaxes on τw≃[π2 ϵ2 K(Λ2)]−1\tau_{\rm w}\simeq[\pi_2\,\epsilon_2\,K(\Lambda_2)]^{-1}, set by the speciation mass alone, where K(Λ)K(\Lambda) is the curvature in bb of the single-SNR loss at its optimum.

A dominant Regime-II mass therefore reproduces the single-time phenomenology: direction first, weight at an even split, then weight on the speciation clock. Sec. 3.2 also discusses how these conclusions depend on the degree of parameter sharing across noise levels (see the general comment).

Q2 (speciation convention). Only weakly, for two reasons. First, the exact-score runs (Fig. 2) are performed at t=12log⁡dt=\tfrac12\log d, where Λ=1\Lambda=1 up to O(1/d)O(1/d), and the rate is evaluated at that SNR, so no convention enters. Second, in the other experiments a collapse depends only on ratios of rates, since a common factor shifts all curves together in log⁡τ\log\tau. Changing the convention rescales all γi2\gamma_i^2 by the same O(1)O(1) factor, and since λ(γ)=4γ2−6γ4+…\lambda(\gamma)=4\gamma^2-6\gamma^4+\dots, the ratios are unchanged at leading order.

Q3 (weight learned under uniform weighting). We believe it is mainly the first mechanism the reviewer suggests: at finite dd, uniform DSM places a sizeable share of its signal near and below speciation. As in the two-time analysis above, this suffices for the weight to be learned, on the timescale set by the speciation mass, of order dd rather than exponentially large in dd. We cannot rule out the second mechanism for a generic network, but we do not believe it is needed to explain the observation.

We hope these answers address the reviewer's questions, and we are happy to provide further details during the discussion period.

I thank the authors for their clear and complete rebuttal! As I have no further (major) doubts or questions, I have decided to raise my score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General comment

We thank the three reviewers for careful and constructive reports. They value studying feature emergence from the training rather than the sampling side (LriQ, XnnL), and agree on the strengths of the analytical part: an exact high-dimensional reduction of the training dynamics (XnnL, pNkV), the frozen-bias mechanism at high SNR (XnnL), the reduction of the hierarchical mixture to two copies of the unbalanced one at rescaled SNRs (XnnL), and the controlled Gaussian-mixture experiments, in which only the weighting changes (XnnL, pNkV). They also raise three shared concerns:

  1. From fixed-SNR training to the integrated loss. Our analytical results focus on training at a single noise level, which we see as the elementary building block for understanding full training pipelines. Of course, in practice one trains a shared network on all noise levels, and the reviewers ask how the two connect analytically. We have therefore generalized our analysis.
  2. Experimental support and calibration of claims. Some experiments support the predicted phenomenology less clearly than others, and our claims need more precise scoping.
  3. Presentation. The main results are stated too informally and are not interpreted enough.

We have worked on a revised version of the manuscript in order to address the first point analytically, the second by quantifying and re-scoping the experimental analysis, and the third by rewriting the statements of our main results. In summary:

  • The time-integrated loss can be analyzed in the GMM setup. We introduce an alternative parametrization of the exact score, whose optimum is the same at every noise level and whose loss at noise level tt depends on tt only through the SNR Λ\Lambda. This encodes in our model two features of practical training: the loss is averaged over noise levels, and a single network, conditioned on the noise level, is shared across all of them. In this setting, the dynamics of the summary statistics under any weighting is the weighted superposition of the single-SNR dynamics. For a two-time weighting, this gives closed-form timescales and a phenomenology inherited from the single-SNR analysis: the direction is learned from the Regime-II mass while the mode weight is learned on a clock set by the speciation mass alone, so a dominant Regime-II mass produces the "direction first, weight at an even split, then weight" sequence.

  • Two limits of parameter sharing. Real networks sit somewhere between two idealized limits. If each noise level has its own parameters, the fixed-SNR results apply noise level by noise level. If all parameters are shared, as in our solvable model, the bias learned around speciation is used at every noise level and the Regime-II freeze is lifted: under any weighting, the weight is then learned alongside the direction. We therefore expect the contrast between weightings in weight learning to be sharper the less parameters are shared. The even-split plateau of our ResNet runs under DSM is consistent with weak sharing, while its absence on MNIST may reflect stronger sharing in the U-Net. The amplitude ordering of features, by contrast, is observed in all our experiments.

Important changes to the manuscript (marked in teal in the revised PDF):

  • Sec. 3.2 now discusses the integrated loss: the two-point analysis (with the derivation of the timescales in a new appendix) and the two limits of parameter sharing that bracket real networks.
  • To improve clarity, Results 1–2 are expanded into self-contained theorem statements, bringing into the main text the notation previously confined to the appendix.
  • The experimental analysis of Sec. 4 is restructured as prediction / observation / verdict for each experiment, stating where the GMM theory succeeds (sequential acquisition of features of decreasing amplitude, in all pipelines) and where it does not (the DSM/FM contrast on MNIST; collapse beyond the leading HGD directions).
  • We reran the unbalanced MNIST experiment to mirror the ResNet protocol exactly: the architecture is fixed, only the weighting changes, and training uses SGD with momentum. The results are unchanged, but the experiment now better isolates the role of the weighting.
  • We add ResNet GMM runs at d∈{128,256}d\in\{128,256\}, which recover the same phenomenology (see our answer to XnnL, W4).

We believe these changes address the reviewers' main concerns while keeping the analytical core intact. We answer each reviewer individually below, and remain open to further discussion and changes to the manuscript, in particular to improve clarity.

Dear Reviewers, The authors have now submitted their responses to the reviews. There is only a limited amount of time left for you to review their replies and provide any further comments.

Your feedback is crucial to ensure that the review process proceeds smoothly. We would greatly appreciate it if you could share your reactions before the discussion period closes.

Thank you very much for your work. Best regards.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.