- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how prediction targets, timestep sampling, and model capacity affect diffusion generation. It studies a simplified Gaussian setting with a linear model that shares a bottleneck across time. The main idea is that limited capacity forces the model to select certain covariance directions; prediction targets and timestep weights influence this selection and the behavior of omitted directions. It derives optimal targets, bottleneck sizes, and sampling distributions under specific assumptions, plus a worst-case error result near x-prediction. Synthetic experiments and CelebA-HQ experiments with JiT reproduce familiar trends: x-prediction, nonuniform sampling, and intermediate bottlenecks are helpful.
Critical strengths
The strength is how it connects what the model learns to what it actually generates. The spectral analysis makes clear why a lower training loss doesn’t necessarily mean better generation, and shows how prediction targets, timestep sampling, and limited capacity interact within one tractable framework.
Critical weaknesses
-
Some findings have been studied in prior work. [1] already derive dimension-dependent optimal prediction targets. [2] provide Gaussian velocity and covariance formulas matching key parts of this analysis. [3] connect covariance spectra to endpoint error and inference schedule optimization. The paper needs to explain what additional theoretical insight its shared-bottleneck constraint provides.
-
Weak evaluation. the experiments mostly reproduce familiar trends without directly testing the new mechanism. The datasets benchmarked on are overall simple. There is no clear measurement of learned-subspace alignment or direct test of the proposed optimal sampling results, leaving the main theoretical contributions weakly validated.
-
Restrictive assumptions. Gaussian data and a linear predictor make covariance structure sufficient to describe the entire problem. Real diffusion models learn nonlinear, multimodal distributions. The paper offers little evidence that its conclusions survive beyond this simplified setting, making it unclear whether the proposed mechanism explains the real-data results.
[1] Revisiting Diffusion Model Predictions Through Dimensionality [2] An Analytical Theory of Spectral Bias in the Learning Dynamics of Diffusion Models [3] Spectral Analysis of Diffusion Models with Application to Schedule Design
Questions for authors
none
We sincerely thank the reviewer for the careful reading and constructive feedback. Below, we address each point.
We thank the reviewer for pointing out these connections. We agree that the cited works [1, 2, 3] also study Gaussian data and linear settings theoretically, and thus share some aspects of the results. However, we would like to emphasize that our distinct theoretical insights lie not in deriving these baseline properties, but in characterizing how prediction target, timestep sampling and model capacity jointly shape generation quality.
In particular, as we discussed in lines 111-118, [1] compares prediction targets using training loss, whereas we compare them by characterizing the Wasserstein distance of the generated distributions. This seemingly subtle difference yields fundamentally distinct insights: in a time-independent linear setting as in [1], a lower training loss can correspond to a larger endpoint Wasserstein error, and this error can even diverge (see Appendix A). Thus, comparing training losses alone across different targets does not reliably indicate which prediction target is superior.
In addition, while [2] focuses on how the eigenvalues govern the training speed along each direction during optimization, ours characterizes which directions are selected at the training loss minimizer. Furthermore, [3] provides theoretical insights into inference-time noise schedule design that complement our analysis of timestep sampling during training. We have cited these works and added the analysis regarding [1] to Appendix A of the revised manuscript.
We agree with the reviewer that our experiments do not sufficiently validate the theoretical findings. To explore whether our findings also arise beyond theoretical settings, we trained nonlinear models on Fourier data while jointly varying the three design choices rather than conducting separate sweeps. Following Gagneux et al. (2026), we constructed each data sample from a weighted sum of 16 Fourier modes. This setting provides a controlled spectral structure in which individual Fourier modes can be assigned different coefficient variances, allowing us to directly examine which spectral components are prioritized by the trained model. We compared relative reductions in prediction loss across modes with different coefficient variances and reported the results in Figure 3.
For x-prediction (α=1) with a bottleneck width of 4, the model substantially reduced prediction loss in the higher-variance components, with little reduction in the lower-variance components. Which components the trained model prioritized was jointly affected by the three design choices. We have added these results and training details to Appendix F.2 of the revised manuscript.
Furthermore, we conducted experiments to test the optimal density proposed in Eq. (21). We trained the model on the synthetic setting of Appendix F.1 under four different timestep distributions, including the distribution in Eq. (21). We used α=0.99, spectrum λi=i−1, dimension D=d=10, and width m=5, and evaluated the generated distributions at the same excess loss of η=10−3.
Although the sampling distribution in Eq. (21) exhibits generation error comparable to the other choices rather than showing a clear empirical advantage, this result is not inconsistent with Theorem 6, since the theorem states that this density minimizes the worst-case generation error over all predictors with excess loss at most η, whereas practically trained predictors may not correspond to the worst case.
We agree that assumptions do not capture all aspects of realistic settings. However, by providing tractable analysis and isolating specific mechanisms in diffusion models, Gaussian data and linear settings serve as standard theoretical testbeds [1, 2, 3, GMM+26]. Such theoretical analyses can offer insight into the behavior of more complex nonlinear models. In this regard, we conducted additional experiments showing that several qualitative trends predicted by our theory appear to transfer to nonlinear settings, as detailed in our response to [W2].
Still, extending our theoretical analysis to nonlinear predictors, for example through random feature models, remains an exciting direction for future work.
[GMM+26] Gagneux, Martin, Gribonval, & Massias. Training Flow Matching: The Role of Weighting and Parameterization. ICLR Workshop on DGM, 2026.