- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper develops a statistical theory of frequency recovery in diffusion models from the perspective of the Discrete Fourier Transform (DFT). For a prespecified Fourier cutoff, the authors establish score approximation and estimation guarantees for over-parameterized parallel ReLU networks under cutoff-dependent norm constraints. The error bound separates a statistical term, whose sample-size exponent depends on dimension and distributional smoothness parameters, from other terms that reflect cross-cutoff dependence and spectral energy above the cutoff. The authors further derive a KL recovery bound for the low-frequency projection of the generated distribution, with a statistical exponent approaching the corresponding minimax exponent as the neural network depth diverges.
Critical strengths
-
Substantial theoretical contribution. The paper has an intensive set of theoretical results that somewhat cover the "end-to-end" pipeline of diffusion training and sampling, including score approximation, statistical estimation of such score, and generation/sampling. Score estimation guarantees are provided that explicitly take into account the NN architecture and norm assumptions. A KL generation bound is then given using these previous score error bounds. Although the step of upper bounding generation error by score-matching error itself is standard and has appeared numerous times in diffusion model literature, I think that the accompanying asymptotic minimax lower bound for the expected KL to be interesting and significant. Overall the analysis given in the paper is substantial and worthwhile.
-
Clear and interesting interpretation. The DFT is an interesting conceptual framework for studying generation; the interesting part here is the connection between this and diffusion models that the authors were able to make. The decomposition into various terms (i.e. Theorem 5.1) is very informative and anchors the narrative of the paper. Throughout the writing is clear and communicates the high-level ideas very well, even though the mathematical content is much more complicated.
Critical weaknesses
- The cutoff has to be specified beforehand. The results of the paper are conditional on a pre-specified Fourier cutoff. Different cutoffs may have different, say, asymptotically minimax-optimal neural network estimators. Is there a way to learn the "right" cutoff from the data, or incorporate the cutoff as a design choice in the minimax expected KL formulation?
Questions for authors
I have no further questions or concerns.
2 comments
Dear Reviewer,
We sincerely thank you again for your positive assessment and constructive feedback.
We would like to share two updates to our revised manuscript. First, to improve clarity and facilitate verification, we have added Appendix A (Roadmap of the Main Proofs) and Figure 3, which summarize the key proof steps and illustrate their dependencies and connections to our main theoretical results.
Second, the additional experiments previously presented in Appendices D.5 and D.6 have been moved to Appendices E.5 and E.6, respectively. In particular, Table 9, which reports the empirical cutoff-selection results discussed in our rebuttal, is now located in Appendix E.6.
Thank you again for your time and consideration.
We sincerely appreciate the reviewer’s comments and suggestions. We respond below to the main weakness.
A1 We thank the reviewer for this insightful suggestion. We agree that data-driven selection of the optimal frequency cutoff is an interesting and important direction. For a given error tolerance ϵ and fixed sample size n, the optimal cutoff can be interpreted as the largest frequency cutoff up to which the error can be controlled within the tolerance, while learning frequencies beyond this cutoff incurs an error that is lower bounded above ϵ. Our current study has direct implications for the upper-bound side of this characterization. However, establishing a corresponding lower bound that identifies when the error tolerance can no longer be achieved requires substantial additional theoretical analysis. Therefore, we leave it for future work.
Nevertheless, our CIFAR-10 experiments provide an empirical illustration of cutoff selection under a fixed error tolerance. Specifically, in Figure 2(d), we use the illustrative threshold pFID⋆=6.5 and select the largest frequency cutoff for which all lower cutoffs also satisfy this threshold. As further reported in Appendix D.6 (Table 9), across three training seeds, the selected cutoffs are 6, 7, and 8 for sample sizes n=5k, 10k, and 20k, respectively, and range from 10 to 12 for n=40k. These results illustrate that the empirically recoverable frequency range under a common error tolerance expands with increasing sample size.