acceptodds
Under review as a conference paper at ICLR 2027

A Statistical Theory of Frequency Recovery in Overparameterized Diffusion Models

Abstract

Diffusion models have achieved broad success in image, video, and sequence generation, yet they often learn global layouts and temporal trends more faithfully than fine local details. Existing theory establishes diffusion models as powerful distribution estimators, but largely concerns full distribution estimation rates, leaving unclear how data properties and network capacity govern accuracy at different resolutions. We address this gap by developing a statistical theory of frequency recovery in overparameterized diffusion models, using the discrete Fourier transform to describe data at different resolutions. For a prescribed frequency cutoff, we establish approximation and estimation guarantees for the score function associated with the low-frequency component of the data distribution. We further prove a distribution recovery bound separating a statistical error, whose rate depends on an effective dimension determined by the number of retained frequencies, from a remainder reflecting dependence between low and high frequencies and the spectral energy of the latter. Crucially, our guarantees use weight matrix norm constraints tailored to the frequency cutoff and, at fixed depth and width, remain unchanged as the number of available parallel networks grows. Synthetic and CIFAR-10 experiments illustrate how sample size and weight decay affect frequency recovery.

Primary area:
learning theory
Keywords:
Diffusion models, statistical estimation, network approximation theory, frequency recovery

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_XWV6OpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper develops a statistical theory of frequency recovery in diffusion models from the perspective of the Discrete Fourier Transform (DFT). For a prespecified Fourier cutoff, the authors establish score approximation and estimation guarantees for over-parameterized parallel ReLU networks under cutoff-dependent norm constraints. The error bound separates a statistical term, whose sample-size exponent depends on dimension and distributional smoothness parameters, from other terms that reflect cross-cutoff dependence and spectral energy above the cutoff. The authors further derive a KL recovery bound for the low-frequency projection of the generated distribution, with a statistical exponent approaching the corresponding minimax exponent as the neural network depth diverges.

Critical strengths
  1. Substantial theoretical contribution. The paper has an intensive set of theoretical results that somewhat cover the "end-to-end" pipeline of diffusion training and sampling, including score approximation, statistical estimation of such score, and generation/sampling. Score estimation guarantees are provided that explicitly take into account the NN architecture and norm assumptions. A KL generation bound is then given using these previous score error bounds. Although the step of upper bounding generation error by score-matching error itself is standard and has appeared numerous times in diffusion model literature, I think that the accompanying asymptotic minimax lower bound for the expected KL to be interesting and significant. Overall the analysis given in the paper is substantial and worthwhile.

  2. Clear and interesting interpretation. The DFT is an interesting conceptual framework for studying generation; the interesting part here is the connection between this and diffusion models that the authors were able to make. The decomposition into various terms (i.e. Theorem 5.1) is very informative and anchors the narrative of the paper. Throughout the writing is clear and communicates the high-level ideas very well, even though the mathematical content is much more complicated.

Critical weaknesses
  1. The cutoff has to be specified beforehand. The results of the paper are conditional on a pre-specified Fourier cutoff. Different cutoffs may have different, say, asymptotically minimax-optimal neural network estimators. Is there a way to learn the "right" cutoff from the data, or incorporate the cutoff as a design choice in the minimax expected KL formulation?
Questions for authors

I have no further questions or concerns.

2 comments

We sincerely appreciate the reviewer’s comments and suggestions. We respond below to the main weakness.

W1 The cutoff has to be specified beforehand. The results of the paper are conditional on a pre-specified Fourier cutoff. Different cutoffs may have different, say, asymptotically minimax-optimal neural network estimators. Is there a way to learn the "right" cutoff from the data, or incorporate the cutoff as a design choice in the minimax expected KL formulation?

A1 We thank the reviewer for this insightful suggestion. We agree that data-driven selection of the optimal frequency cutoff is an interesting and important direction. For a given error tolerance ϵ\epsilon and fixed sample size nn, the optimal cutoff can be interpreted as the largest frequency cutoff up to which the error can be controlled within the tolerance, while learning frequencies beyond this cutoff incurs an error that is lower bounded above ϵ\epsilon. Our current study has direct implications for the upper-bound side of this characterization. However, establishing a corresponding lower bound that identifies when the error tolerance can no longer be achieved requires substantial additional theoretical analysis. Therefore, we leave it for future work.

Nevertheless, our CIFAR-10 experiments provide an empirical illustration of cutoff selection under a fixed error tolerance. Specifically, in Figure 2(d), we use the illustrative threshold pFID⋆=6.5\mathrm{pFID}_\star=6.5 and select the largest frequency cutoff for which all lower cutoffs also satisfy this threshold. As further reported in Appendix D.6 (Table 9), across three training seeds, the selected cutoffs are 66, 77, and 88 for sample sizes n=5kn=5k, 10k10k, and 20k20k, respectively, and range from 1010 to 1212 for n=40kn=40k. These results illustrate that the empirically recoverable frequency range under a common error tolerance expands with increasing sample size.

Dear Reviewer,

We sincerely thank you again for your positive assessment and constructive feedback.

We would like to share two updates to our revised manuscript. First, to improve clarity and facilitate verification, we have added Appendix A (Roadmap of the Main Proofs) and Figure 3, which summarize the key proof steps and illustrate their dependencies and connections to our main theoretical results.

Second, the additional experiments previously presented in Appendices D.5 and D.6 have been moved to Appendices E.5 and E.6, respectively. In particular, Table 9, which reports the empirical cutoff-selection results discussed in our rebuttal, is now located in Appendix E.6.

Thank you again for your time and consideration.

Reviewer_shLROpenReview ↗
Rating
4(+1)
Confidence
1(−2)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies frequency recovery in diffusion models trained on full-dimensional data. For each Fourier cutoff, the authors construct a norm-constrained parallel ReLU score estimator, derive denoising score matching risk bounds, and obtain projected forward-KL guarantees. The statistical exponent depends on the retained dimension, with residuals for cross-frequency dependence and discarded spectral energy. Synthetic and CIFAR-10 experiments examine recovery across cutoffs, sample sizes, and regularization strengths.

Critical strengths
  1. Extending exact-subspace analyses to full-dimensional distributions is useful. Proposition E.10 connects the mismatch between full-score training and marginal-score recovery to dependence between retained and discarded coordinates. Lemma C.1 explicitly establishes a full-distribution guarantee; Proposition 5.4 states only the projected version.

  2. Norm control makes the covering bounds independent of the available branch count at fixed branch depth and width (Lemmas E.8–E.9). Appendix C handles early stopping and supplies a projected minimax lower bound. The writing is generally clear, with detailed proofs and experimental protocols in the appendices.

Critical weaknesses
  1. Lemma C.1 bounds full-distribution KL and obtains the projected guarantee by data processing. Thus, the same-order bound in Proposition 5.4 also controls full KL and every projected marginal of the same estimator. Its cutoff dependence comes from estimator tuning in (B.3)–(B.4); it does not compare recovery across frequencies of one fixed model, as plotted in Figures 1(b,c) and 2(c,d). The main text should clarify this distinction.

  2. The optimal-cutoff orders after Corollary 5.3 need a uniform derivation. Equation (B.29) contains constants and logarithmic powers that grow with deff(w)d_{\mathrm{eff}}(w); Appendix B.3 explicitly acknowledges the lack of uniformity over growing cutoffs (p. 31, lines 1669–1670). Optimizing the abbreviated fixed-cutoff rate does not control these terms. The stated orders should follow from a bound retaining that dependence, or be identified as heuristic balances.

  3. Theorem 5.1 gives L=CLlog⁡3(dn)L=C_L\log^3(dn), while Theorem 4.2 allows CLC_L to depend on T=log⁡nT=\log n. For β=1\beta=1, the appendix gives a sufficient construction with depth O((log⁡n)387)O((\log n)^{387}) ((A.28), Appendix B.3). The theorem should display this dependence. The claim that deeper networks give faster rates (Section 5.1) also overlooks the L2L^2 prefactor: over the prescribed range L≥2log⁡nL\ge2\log n, n−γLn^{-\gamma_L} already matches the minimax sample exponent up to a constant factor, and L2n−γLL^2n^{-\gamma_L} increases with LL.

Questions for authors

Please refer to weaknesses part.

4 comments

We sincerely appreciate the reviewer’s comments and suggestions. We respond below to the main weaknesses and questions.

W1: Lemma C.1 bounds full-distribution KL and obtains the projected guarantee by data processing. Thus, the same-order bound in Proposition 5.4 also controls full KL and every projected marginal of the same estimator. Its cutoff dependence comes from estimator tuning in (B.3)–(B.4); it does not compare recovery across frequencies of one fixed model, as plotted in Figures 1(b,c) and 2(c,d). The main text should clarify this distinction.

A1 We thank the reviewer for this helpful observation. Our theoretical analysis characterizes frequency-dependent recovery guarantees under cutoff-specific norm constraints within a shared architecture, while our experiments examine recovery across different frequency cutoffs using models trained with fixed regularization settings. These two perspectives provide complementary insights into frequency recovery.

We have revised the Introduction (line 079-081) and Section 4 (line 202-204) to clarify the distinction between shared architecture and cutoff-specific network parameters, and Section 6 (line 377) to specify that each trained model is evaluated across multiple frequency cutoffs without retraining.

W2: The optimal-cutoff orders after Corollary 5.3 need a uniform derivation. Equation (B.29) contains constants and logarithmic powers that grow with deff(w)d _{\mathrm{eff}}(w); Appendix B.3 explicitly acknowledges the lack of uniformity over growing cutoffs (p. 31, lines 1669–1670). Optimizing the abbreviated fixed-cutoff rate does not control these terms. The stated orders should follow from a bound retaining that dependence, or be identified as heuristic balances.

A2 We thank the reviewer for pointing this out. We agree that the stated optimal-cutoff orders are heuristic balances rather than rigorously established asymptotic rates. As noted in Appendix B.3, the constants and logarithmic factors in Eq. (B.29) depend on deff(w)d_{\mathrm{eff}}(w), and therefore cannot be ignored when the cutoff grows with nn.

The stated orders are obtained by balancing the leading statistical term with the spectral remainder, without accounting for this additional cutoff dependence. We have revised the discussion after Corollary 5.3 to explicitly identify these orders as heuristic and remove any claim of rigorous optimality (line 340-344). This clarification does not alter the fixed-cutoff error bounds in Theorem 5.1 and Corollary 5.3.

W3 Theorem 5.1 gives L=CLlog⁡3(dn)L=C_L\log^3(dn), while Theorem 4.2 allows CLC_L to depend on T=log⁡nT=\log n. For β=1\beta=1, the appendix gives a sufficient construction with depth O((log⁡n)387)O((\log n)^{387}) ((A.28), Appendix B.3). The theorem should display this dependence. The claim that deeper networks give faster rates (Section 5.1) also overlooks the L2L^2 prefactor: over the prescribed range L≥2log⁡nL\geq 2\log n, n−γLn^{-\gamma_L} already has the minimax sample exponent up to a constant factor, and L2n−γLL^2n^{-\gamma_L} increases with LL.

A3 We thank the reviewer for identifying these two issues.

Depth specification. We agree that Theorem 5.1 should explicitly account for the dependence on T=log⁡nT=\log n. We replaced its depth specification by

C_{\mathrm{depth}}[\log(dn)]^{p_\beta} \right\rceil, \qquad p_\beta= 64(\lceil\beta\rceil+2)(1+\beta^{-1})+3,$$ where $C_{\mathrm{depth}}$ is sufficiently large and depends only on the fixed problem parameters, not on $n$. For $\beta=1$, this specifies a sufficient depth proportional to $(\log n)^{387}$ for fixed problem parameters. We emphasize that this logarithmic dependence is a conservative upper bound arising from our proof technique and has not been optimized; in particular, we do not claim that such a large exponent is necessary. With this depth choice, the statistical term remains $\widetilde O(n^{-2\beta/(2\beta+d_{\mathrm{eff}}(w))})$ for fixed parameters and cutoff, but with larger logarithmic factors than suggested by the original depth specification. **Interpretation of depth.** We agree that the discussion in Section 5.1 should account for the $L^2$ prefactor. Writing $$r_w=\frac{2\beta}{2\beta+d_{\mathrm{eff}}(w)}, \qquad \gamma_L(w)=r_w\frac{L-1}{L+1},$$ the statistical bound contains the term $L^2n^{-\gamma_L(w)}$. Indeed, for fixed $n$ and $w$, this term increases with $L$ over the prescribed depth range $L\geq 2\log n$, despite the increase in $\gamma_L(w)$. However, our intended interpretation concerns the convergence rate with respect to the sample size $n$, rather than the monotonicity of the bound in $L$ for a fixed $n$. The exponent $\gamma_L(w)$ increases with $L$ and approaches $r_w$, indicating an improvement in the polynomial convergence exponent. Specifically, as clarified in our revised depth specification for Theorem 5.1, we choose $$L(n)=\left\lceil C_{\mathrm{depth}}[\log(dn)]^{p_\beta}\right\rceil, \qquad p_\beta=64(\lceil\beta\rceil+2)(1+\beta^{-1})+3,$$ where $C_{\mathrm{depth}}$ is sufficiently large, depends only on the fixed problem parameters and is independent of $n$. Under this choice, $$\begin{aligned} L(n)^2n^{-\gamma_{L(n)}(w)} &=n^{-r_w}L(n)^2 \exp\left(\frac{2r_w\log n}{L(n)+1}\right)=\widetilde O(n^{-r_w}), \end{aligned}$$ for fixed problem parameters and cutoff, since the depth-dependent prefactor contributes only logarithmic factors. We have revised Theorem 5.1 to make the sufficient depth dependence explicit (line 299-302) and clarified the role of increasing depth with sample size in the Introduction (line 094) and Section 5.1 (line 313-321), while accounting for the $L^2$ prefactor in the statistical bound.

Thank you very much for the authors' response. Due to the length and complexity of the proofs in the paper, I have not checked every theorem and proof for absolute correctness; however, based on the current revisions, I have raised my score to 4 while lowering my confidence score to 1.

We sincerely thank the reviewer for the thoughtful assessment and for updating the score. We understand the concern regarding the length and complexity of the proofs.

To improve clarity and facilitate verification, we have added Appendix A (Roadmap of the Main Proofs) and Figure 3, which provide a structured overview of the theoretical arguments. Specifically, Appendix A summarizes five key steps, covering the low-dimensional score representation, network approximation, statistical error control via an oracle inequality, controlling the discrepancy between the score targets, and distribution recovery. Figure 3 further illustrates the logical dependencies between these steps, the main theoretical results, and their detailed proofs in the subsequent appendices.

We hope these additions make the proof structure and key technical arguments easier to follow and verify.

Reviewer_Yh5GOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Existing theory establishes diffusion models as powerful distribution estimators, but largely concerns full distribution estimation rates, leaving unclear how data properties and network capacity govern accuracy at different resolutions. The paper addresses this gap by developing a statistical theory of frequency recovery in overparameterized diffusion models, using the discrete Fourier transform to describe data at different resolutions.

Critical strengths

The paper clearly distinguishes the projected full-distribution score from the score of the low-frequency marginal and accounts for their discrepancy through cross-frequency dependence. Its main technical strength is a complexity analysis that is independent of the number of parallel branches under fixed architectural and norm constraints. The distribution recovery analysis also explicitly addresses initialization and early-stopping errors, while synthetic experiments with analytically available scores enable direct evaluation of the retained-score risk.

Critical weaknesses
  1. The theory uses cutoff-specific constraints. Different frequency cutoffs require different norm budgets, so the results probably not directly show that a single trained model achieves the stated guarantees across all frequency ranges.

  2. The experiments provide limited validation of the theoretical claims. The synthetic data have independent Fourier coordinates, so they probably not test the cross-frequency dependence term. The CIFAR-10 experiments use only one training run per condition, leaving uncertainty about reproducibility across random seeds.

My familiarity with this specific research area is very limited. These concerns should therefore be considered alongside the assessments of reviewers with stronger expertise, particularly regarding the theoretical contributions.

Questions for authors

Please address the questions corresponding to the weaknesses listed above.

5 comments

We sincerely appreciate the reviewer’s comments and suggestions. We respond below to the main weaknesses and questions.

W1 The theory uses cutoff-specific constraints. Different frequency cutoffs require different norm budgets, so the results probably not directly show that a single trained model achieves the stated guarantees across all frequency ranges.

A1 We thank the reviewer for the comment. We would like to clarify the distinction between a shared network architecture for approximating scores of different frequencies and a trained model under a chosen norm budget. Our approximation theory shows the existence of a shared network architecture for representing scores, whereas our statistical learning theory is established under a pre-fixed frequency cut-off. We do not claim that a single trained model simultaneously achieves the cutoff-specific guarantees across all frequency ranges.

Specifically, Theorem 4.2 establishes that, within a sufficiently overparameterized architecture shared across all cutoffs up to wmax⁡w_ {\max}, for each prescribed cutoff ww, there exists a choice of weight parameters such that the resulting network approximates the corresponding low-frequency score function. The architecture parameters (M,L,W)(M,L,W) are chosen to accommodate all considered cutoffs, while the norm budgets (Bw,Bout,w)(B_w,B_{\mathrm{out},w}) and the weight parameters may depend on ww. Theorem 5.1 further provides statistical guarantees for the empirical minimizer over the corresponding cutoff-specific constrained network family.

Therefore, our theory characterizes frequency-dependent approximation and estimation guarantees within a shared architecture, rather than simultaneous attainment of all cutoff-specific rates by one fitted model. We have revised the Introduction (lines 079–081) and Section 4 (lines 202–204) to make this distinction more explicit and avoid potential confusion.

W2 The experiments provide limited validation of the theoretical claims. The synthetic data have independent Fourier coordinates, so they probably not test the cross-frequency dependence term. The CIFAR-10 experiments use only one training run per condition, leaving uncertainty about reproducibility across random seeds.

A2. Controlled cross-frequency dependence.

We thank the reviewer for pointing out that the original synthetic distribution has independent Fourier coordinates, so the cross-frequency dependence term in Theorem 5.1 vanishes. To address this limitation, we extend the original d=15d=15 distribution by introducing dependence between the cosine coordinates at frequencies 2 and 6:

\qquad \rho\in\\\{0,0.5,0.8\\\}.$$ This construction preserves every coordinate marginal and the entire spectral-energy profile while introducing cross-frequency dependence. At cutoff $w=3$, frequency 2 is retained while frequency 6 is discarded. We evaluate the cross-frequency mutual information $I(P_{\mathrm{data}}^{(w)},P_{\mathrm{data}}^{(>w)})$ appearing in Theorem 5.1, Eq. (5.2). **Table 1. Controlled cross-frequency dependence at cutoff $w=3$.** | Coupling $\rho$ | $I(P_{\mathrm{data}}^{(3)},P_{\mathrm{data}}^{(>3)})$ | Spectral tail at $w$=3 | | --- | ---: | ---: | | 0 | 0.000000 | 4.255673 | | 0.5 | 0.019236 | 4.255673 | | 0.8 | 0.051229 | 4.255673 | The cross-frequency mutual information increases with coupling strength, while the spectral tail remains unchanged. This confirms that the construction introduces nonzero dependence across the frequency cutoff, directly examining the dependence term in Theorem 5.1. **Training experiments with dependent Fourier coordinates.** We further fix $\rho=0.8$ and conduct training experiments using the sample-size and regularization configurations in Figure 1. We reconstruct the simulation protocol in Appendix D.2, using the same network architecture and parameter grids, with the same training updates per trajectory. For each configuration, we use three independent data seeds (17, 29, 43). Checkpoints follow the primary selection criterion in Appendix D.2. We evaluate the retained marginal-score risk $R_w$ defined in Eq. (5.1) on independent test data. The following results are reported as mean ± sample standard deviation across the three data seeds, with all frequency cutoffs $w=0,\ldots,7$ included.

Table 2. Sample-size comparison on the dependent distribution (ρ=0.8\rho=0.8, λ=0.01\lambda=0.01).

Cutoff wwn=128n=128n=256n=256n=512n=512n=1024n=1024n=2048n=2048n=4096n=4096
05.231 ± 2.5521.328 ± 0.3840.130 ± 0.0580.042 ± 0.0050.030 ± 0.0010.023 ± 0.001
127.019 ± 6.8316.317 ± 3.6230.487 ± 0.3140.140 ± 0.0150.104 ± 0.0070.086 ± 0.004
260.850 ± 6.28824.700 ± 8.7910.964 ± 0.6940.328 ± 0.0130.273 ± 0.0020.244 ± 0.003
397.017 ± 5.83344.713 ± 21.1134.617 ± 6.3280.626 ± 0.0100.545 ± 0.0040.506 ± 0.004
4132.930 ± 3.19167.159 ± 28.0445.333 ± 6.6351.030 ± 0.0100.922 ± 0.0110.872 ± 0.010
5163.798 ± 3.83387.395 ± 32.5459.637 ± 10.7591.536 ± 0.0141.397 ± 0.0111.340 ± 0.007
6189.294 ± 10.280105.929 ± 37.53013.781 ± 14.3282.223 ± 0.0082.060 ± 0.0121.923 ± 0.115
7220.104 ± 14.566124.914 ± 45.52917.608 ± 18.5092.917 ± 0.0042.723 ± 0.0102.567 ± 0.111

Entries report the retained marginal-score risk RwR_w (Eq. (5.1)), as mean ± sample standard deviation across three independent data seeds. These results correspond to the sample-size and frequency-cutoff comparisons in Figure 1(a–c).

Table 3. Weight-decay comparison on the dependent distribution (ρ=0.8\rho=0.8, n=2048n=2048).

Cutoff wwλ=0\lambda=0λ=10−4\lambda=10^{-4}λ=10−3\lambda=10^{-3}λ=10−2\lambda=10^{-2}λ=10−1\lambda=10^{-1}
09.620 ± 0.5209.163 ± 0.3871.517 ± 1.1520.030 ± 0.0010.051 ± 0.004
130.268 ± 1.19928.545 ± 0.89114.921 ± 1.2720.104 ± 0.0070.179 ± 0.003
249.110 ± 1.43646.865 ± 0.77930.150 ± 1.1340.273 ± 0.0020.446 ± 0.003
367.404 ± 1.55864.442 ± 1.17143.840 ± 1.9880.545 ± 0.0040.851 ± 0.005
484.449 ± 1.60180.863 ± 1.43757.944 ± 3.4830.922 ± 0.0111.382 ± 0.007
5101.460 ± 2.08097.342 ± 1.82871.228 ± 4.3911.397 ± 0.0112.034 ± 0.007
6118.318 ± 1.676113.750 ± 1.34486.087 ± 4.8702.060 ± 0.0122.890 ± 0.005
7135.211 ± 2.393130.249 ± 2.44899.465 ± 5.7452.723 ± 0.0103.743 ± 0.004

Entries report mean ± sample standard deviation across three independent data seeds. These results correspond to the regularization comparison in Figure 1(d), with λ=0.01\lambda=0.01 shared with Table 2.

Sample-size and regularization effects. The results show that increasing sample size consistently reduces the mean retained-score risk across all evaluated cutoffs, including w=2,…,5w=2,\ldots,5, where cross-frequency dependence is nonzero. At n=2048n=2048, the intermediate regularization strength λ=0.01\lambda=0.01 achieves the lowest mean risk across all cutoffs among the tested coefficients, while stronger regularization (λ=0.1\lambda=0.1) worsens recovery. These trends are consistent with the qualitative findings of the original synthetic experiments.

Table 4. Largest empirically learnable cutoffs at R⋆=1R_\star=1, with λ=0.01\lambda=0.01.

Training size nnSeed 17Seed 29Seed 43
128NoneNoneNone
256NoneNoneNone
512123
1,024333
2,048444
4,096444

Following Figure 1( c ), we consider the frequency band up to ww empirically learnable if Rv≤R⋆R_v\leq R_\star for all evaluated cutoffs v≤wv\leq w. "None" indicates that no cutoff satisfies this criterion.

The empirically learnable frequency range expands with sample size, reaching w=3w=3 at n=1024n=1024 and w=4w=4 at n=2048n=2048 across all three seeds. Although the largest learnable cutoff remains at w=4w=4 for n=4096n=4096, the retained-score risk continues to decrease. Importantly, the expanded learnable frequency range includes cutoffs where cross-frequency dependence is nonzero.

Overall, these additional experiments extend the original independent-coordinate simulations to a setting with nonzero cross-frequency dependence. The results show that the main sample-size, regularization, and frequency-recovery trends observed in the original simulations persist in this dependent setting.

(b) CIFAR-10 reproducibility across training seeds. We agree that the original single-seed experiments do not establish reproducibility across training runs. We therefore repeated the CIFAR-10 experiments in Figure 2 using two additional training seeds (712, 713), alongside the original seed (711), covering both the sample-size and weight-decay comparisons. We followed the original training and evaluation protocol, keeping the training subsets, validation data, architecture, and EMA checkpoint-selection criterion fixed while varying initialization and training randomness. Each selected model generates 10,000 images, evaluated against the same 10,000 CIFAR-10 test images. We report mean ± sample standard deviation across the three training seeds.

Table 5. CIFAR-10 sample-size comparison (λ=0\lambda=0).

Training size nnpFID2\mathrm{pFID}_2pFID4\mathrm{pFID}_4pFID8\mathrm{pFID}_8FID
5k3.253 ± 0.0424.752 ± 0.0297.889 ± 0.07415.012 ± 0.179
10k3.130 ± 0.0334.567 ± 0.0096.939 ± 0.07310.723 ± 0.093
20k3.069 ± 0.0074.401 ± 0.0196.377 ± 0.0128.027 ± 0.031
40k2.905 ± 0.0664.229 ± 0.0205.976 ± 0.0646.916 ± 0.086

The reported cutoffs correspond to Figure 2(b). FID equals pFID16\mathrm{pFID}_{16}. Values are mean ± sample standard deviation across three training seeds.

Table 6. CIFAR-10 frequency-cutoff comparison (λ=0\lambda=0).

Cutoff wwn=5kn=5kn=10kn=10kn=20kn=20kn=40kn=40k
11.582 ± 0.0091.502 ± 0.0071.476 ± 0.0261.418 ± 0.043
23.253 ± 0.0423.130 ± 0.0333.069 ± 0.0072.905 ± 0.066
34.083 ± 0.0353.882 ± 0.0173.730 ± 0.0283.567 ± 0.033
44.752 ± 0.0294.567 ± 0.0094.401 ± 0.0194.229 ± 0.020
55.334 ± 0.0345.099 ± 0.0384.909 ± 0.0224.761 ± 0.052
66.096 ± 0.0445.700 ± 0.0595.471 ± 0.0325.267 ± 0.028
77.019 ± 0.0526.362 ± 0.0625.950 ± 0.0195.670 ± 0.049
87.889 ± 0.0746.939 ± 0.0736.377 ± 0.0125.976 ± 0.064
98.858 ± 0.1017.437 ± 0.0626.654 ± 0.0236.167 ± 0.065
109.771 ± 0.0847.958 ± 0.0816.864 ± 0.0276.309 ± 0.060
1110.797 ± 0.1148.480 ± 0.1117.053 ± 0.0106.448 ± 0.046
1211.641 ± 0.1228.891 ± 0.1147.234 ± 0.0076.546 ± 0.056
1312.377 ± 0.1179.314 ± 0.1217.393 ± 0.0226.606 ± 0.058
1413.136 ± 0.1559.725 ± 0.1047.551 ± 0.0306.694 ± 0.045
1514.009 ± 0.16010.122 ± 0.1187.741 ± 0.0346.765 ± 0.057
1615.012 ± 0.17910.723 ± 0.0938.027 ± 0.0316.916 ± 0.086

Entries report pFIDw\mathrm{pFID} _w (Eq. (6.1)) as mean ± sample standard deviation across three independent training seeds. The results correspond to Figure 2(c,d) and cover all evaluated frequency cutoffs w=1,…,16w=1,\ldots,16. At w=16w=16, pFID16\mathrm{pFID} _{16} equals ordinary FID.

Table 7. CIFAR-10 weight-decay comparison (n=20000n=20000).

Cutoff wwλ=0\lambda=0λ=10−7\lambda=10^{-7}λ=10−6\lambda=10^{-6}λ=10−5\lambda=10^{-5}
11.476 ± 0.0261.517 ± 0.0182.352 ± 0.2686.905 ± 1.083
23.069 ± 0.0073.108 ± 0.0374.263 ± 0.44010.325 ± 1.008
33.730 ± 0.0283.871 ± 0.0345.092 ± 0.2319.358 ± 0.551
44.401 ± 0.0194.637 ± 0.0456.173 ± 0.25610.588 ± 0.683
54.909 ± 0.0225.295 ± 0.0507.201 ± 0.39712.914 ± 0.701
65.471 ± 0.0326.008 ± 0.0298.369 ± 0.47015.569 ± 0.609
75.950 ± 0.0196.625 ± 0.0559.749 ± 0.46618.715 ± 0.690
86.377 ± 0.0127.112 ± 0.06210.883 ± 0.35521.797 ± 0.765
96.654 ± 0.0237.543 ± 0.07912.201 ± 0.33424.892 ± 0.888
106.864 ± 0.0277.944 ± 0.09213.436 ± 0.33027.883 ± 0.837
117.053 ± 0.0108.363 ± 0.08314.778 ± 0.25330.676 ± 0.810
127.234 ± 0.0078.655 ± 0.07415.867 ± 0.27933.200 ± 0.837
137.393 ± 0.0228.950 ± 0.07616.811 ± 0.28235.332 ± 0.877
147.551 ± 0.0309.309 ± 0.08017.772 ± 0.34437.362 ± 0.897
157.741 ± 0.0349.657 ± 0.08518.704 ± 0.30139.720 ± 0.986
168.027 ± 0.03110.191 ± 0.08520.214 ± 0.34642.593 ± 0.919

Entries report pFIDw\mathrm{pFID}_w (Eq. (6.1)) as mean ± sample standard deviation across three independent training seeds. The results correspond to Figure 2(e).

Reproducibility of the main experimental findings. The results in Tables 5 and 6 consistently show that increasing the training sample size improves frequency-resolved generation quality. For all cutoffs, pFID decreases with increasing sample size. Similarly, in Table 7 at n=20000n=20000, increasing the tested weight-decay coefficient worsens pFID across all cutoffs. These results support the reproducibility of the main empirical conclusions in Figure 2 across different training seeds.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

Dear Reviewers, Area Chair, Senior Area Chair, and Program Chairs,

We would like to express our sincere appreciation for your careful evaluation of our work and for the constructive discussion throughout the review process.

In response to the feedback raised during the review and discussion period, we have revised the manuscript to sharpen the theoretical statements, clarify the scope of our guarantees, improve the accessibility of the proofs, and strengthen the experimental validation. The main revisions are summarized below.

  1. Clarified the scope of the frequency-dependent theoretical guarantees.
    We now explicitly distinguish between a shared network architecture and a single fitted model. Our approximation theory uses one sufficiently overparameterized architecture, specified by common network dimensions (M,L,W)(M,L,W), while the network weights and norm budgets may depend on the prescribed frequency cutoff. Accordingly, the statistical guarantees are cutoff-specific and do not claim that a single fitted model simultaneously attains all cutoff-specific theoretical rates. We also clarified in the experimental section that each trained model is evaluated across multiple frequency cutoffs without retraining.
    Revised in: the Introduction (lines 079–081), Section 4 (lines 202–204), and Section 6 (line 377).

  2. Corrected and sharpened the theoretical statements on cutoff selection and network depth.
    We revised the discussion following Corollary 5.3 to make clear that the proposed cutoff scales are heuristic balances, rather than rigorously established optimal asymptotic orders, because the constants and logarithmic factors in the fixed-cutoff bounds themselves depend on the effective dimension. We also corrected the depth specification in Theorem 5.1 to explicitly account for its dependence on T=log⁡nT=\log n, and revised the Introduction and the corresponding discussion in Section 5.1 to clarify the role of increasing depth with sample size while accounting for the L2L^2 prefactor in the statistical bound. The revised discussion now distinguishes improvement of the polynomial convergence exponent from monotonic improvement of the finite-sample bound as network depth increases.
    Revised in: Theorem 5.1 (lines 299–302), Section 5.1 (lines 313–321), the discussion following Corollary 5.3 (lines 340–344), and the Introduction (line 094).

  3. Added a roadmap of the main theoretical arguments.
    To make the lengthy proofs easier to follow and verify, we added a new Appendix A (Roadmap of the Main Proofs) together with Figure 3. The roadmap organizes the analysis into five main steps, from low-dimensional score representation and network approximation to statistical error analysis, controlling the discrepancy between the score targets, and distribution recovery. The new figure illustrates how these steps and intermediate results connect Theorems 4.2 and 5.1 to Proposition 5.4. Added in: Appendix A and Figure 3.

  4. Added controlled experiments with nonzero cross-frequency dependence.
    The original synthetic experiments used independent Fourier coordinates, for which the cross-frequency mutual-information term in Theorem 5.1 vanishes. To directly examine this term, we added a set of experiments using a controlled dependent distribution that introduces coupling between retained and discarded Fourier coordinates while preserving the coordinate marginals and spectral-tail energy. We explicitly verify that the cross-frequency mutual information becomes nonzero as the coupling strength increases. We then repeat the sample-size and regularization experiments under this dependent distribution using multiple independent data seeds. The results show that the main sample-size, regularization, and frequency-recovery trends observed in the original simulations persist in the presence of nonzero cross-frequency dependence.
    Added in: Appendix E.5.

  5. Strengthened the CIFAR-10 reproducibility study and empirical cutoff analysis.
    We repeated the experiments using two additional independent training seeds, giving three independent training runs for each configuration, and report mean and sample standard deviation for the sample-size, frequency-cutoff, and weight-decay comparisons. We also compute the empirically recovered frequency cutoff separately for each training seed. Across all three runs, increasing the training sample size consistently improves frequency-resolved generation quality and expands the range of frequencies satisfying the common pFID criterion. These additional experiments strengthen the reproducibility evidence for the main empirical conclusions and provide the seed-wise empirical cutoff-selection results discussed in our rebuttal.
    Added in: Appendix E.6.

We sincerely thank you again for your careful evaluation and constructive feedback, which helped us improve the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.