- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies the approximation error incurred when exact DDPM reverse conditionals are replaced by their KL-optimal, moment-matched Gaussian distributions. It derives an exact derivative identity for the averaged one-step Gaussianization error and obtains a bulk error bound under a finite-second-moment assumption. With the first-step noise and terminal signal level fixed, this bound is uniquely minimized by geometrically increasing noise-to-signal odds, equivalently a linear log-SNR schedule. Experiments compare matched-endpoint schedules on seven unconditional image datasets and class-conditional CIFAR-10 under variational and hybrid training objectives.
Critical strengths
-
The exact characterization of Gaussianization error is an interesting theoretical contribution. Theorem 1 isolates the residual error after matching both the posterior mean and covariance and expresses its derivative as a nonnegative quantity involving third derivatives of the smoothed log-density. The proof addresses the extension from bounded inputs to finite-second-moment inputs. Although the resulting linear log-SNR schedule is already known, the analysis provides a useful perspective on why it arises from a finite-step Gaussian approximation criterion.
-
The matched-endpoint experimental design controls important confounding factors. Within each baseline/geometric pair, the endpoints and training settings are held fixed, and the reported variational NLL bound improves in all 30 comparisons across unconditional and conditional settings. The paper also explicitly reports the limitations of the sample-quality results: FID improvements occur mainly under variational training, while hybrid training often favors the baselines. This balanced reporting makes the empirical contribution easier to assess.
Critical weaknesses
- The connection between the optimized bound and the practical model error remains insufficiently validated. The theory concerns unrestricted moment-matched Gaussian kernels, whereas the experiments learn diagonal Gaussian kernels. Restricting the covariance introduces an additional approximation error that can remain even when the true reverse conditional is Gaussian. This error is distinct from both the analyzed Gaussianization error and finite-capacity or optimization errors. The paper acknowledges the distinction, but does not establish which component accounts for the observed improvements.
The upper bound is also quantitatively loose at the experimental dimensions. For 32-by-32 RGB images and 4000 steps, substituting the stated endpoints into the optimized bulk bound gives approximately 8.6e7 nats for the linear-endpoint setting and 3.9e8 nats for the cosine-endpoint setting. These values do not invalidate the bound, but they leave its usefulness as a surrogate for actual Gaussianization error uncertain. A controlled non-Gaussian example with computable reverse posteriors would help connect the theoretical optimization to the quantity it aims to reduce.
- The experiments provide limited evidence about robustness across training randomness and step counts. The tables report point estimates without specifying the number of independent training seeds or uncertainty, although some NLL-bound improvements are small; for example, the variational SVHN linear/g-lin comparison changes from 1.715 to 1.710 bits/dim. All experiments also use 4000 steps, leaving the dependence on the number of transitions untested. Since the schedule's functional form is already established in prior work, evidence that the proposed criterion gives a reliable practical benefit is important to the paper's overall contribution.
Questions for authors
The following clarification and validation questions could affect my assessment:
-
How well does the optimized upper bound track the actual accumulated Gaussianization error? Can the authors provide a controlled non-Gaussian example with exact or accurately computable posterior moments and errors, comparing schedules at fixed endpoints?
-
Can the additional error from the diagonal-covariance restriction be quantified or separated from Gaussianization and model-fitting errors? What evidence supports attributing the observed likelihood-bound improvements to the mechanism analyzed in the theory?
-
How many independent training runs underlie the reported results, and what is the uncertainty in representative matched comparisons? Is there evidence at a smaller step count indicating whether the observed benefit extends beyond the 4000-step setting?
For the discussion in Appendix A.4, I recommend considering the following related papers on information-theoretic perspectives on diffusion:
6 comments
Response to Q2 of Reviewer 58yt
Q2. Can the additional error from the diagonal-covariance restriction be quantified or separated from Gaussianization and model-fitting errors?
Thanks for your question! We used diagonal covariances to follow the DDPM experimental setup of Nichol and Dhariwal (2021). We wanted a controlled experiment that followed an existing setup, changing only the noise schedule, so that we could isolate the contribution of the schedule. Although our schedule is derived for the full-covariance setting, it improves the NLL bound even with diagonal covariances, suggesting that its benefits are robust to this restriction.
In general, we can decompose the error as follows. For a bulk reverse conditional , its moment-matched Gaussian , and any nondegenerate Gaussian approximation (for example, a Gaussian with diagonal covariance), moment matching gives
.
The first term is the Gaussianization error analyzed in our theory, averaged over the noisy input. The second term is the additional mismatch from restricting the covariance and learning the reverse kernel.
Motivated by your question, we ran a comparison with learned full bulk covariance. We tested grayscale versions of MNIST, Fashion-MNIST, and CIFAR-10. For each dataset, we trained cosine and endpoint-matched geometric-cosine schedules with diagonal and full bulk covariance using three training seeds per condition (36 runs in total). These smaller experiments use the same MLP backbone architecture, 250 diffusion steps, and 50,000 updates of direct VLB training. The full-covariance model adds off-diagonal outputs; the backbone and mean-head architectures are unchanged. The first-step decoder remains diagonal in both cases.
We evaluated the final exponential moving average (EMA) checkpoints on all 10,000 test images. The table reports the variational NLL bound in bits per grayscale pixel, as mean sample standard deviation across the three training seeds.
| Dataset | Bulk covariance | Cosine | Geometric-cosine |
|---|---|---|---|
| MNIST | Diagonal | 4.3817 ± 0.0241 | 4.3158 ± 0.0164 |
| MNIST | Full | 4.3748 ± 0.0194 | 4.2824 ± 0.0031 |
| Fashion-MNIST | Diagonal | 5.4903 ± 0.0149 | 5.3892 ± 0.0088 |
| Fashion-MNIST | Full | 5.4722 ± 0.0144 | 5.3588 ± 0.0183 |
| CIFAR-10 | Diagonal | 6.8195 ± 0.0013 | 6.8132 ± 0.0008 |
| CIFAR-10 | Full | 6.8076 ± 0.0014 | 6.7979 ± 0.0006 |
Geometric-cosine improved the NLL bound in all 18 matched seed comparisons, including all nine with full bulk covariance. Thus the improvement persists when the diagonal restriction on bulk transitions is removed. These experiments do not separately quantify how much of the learned models' NLL-bound improvement comes from each term in the decomposition.
We will add this decomposition and the covariance experiments to the paper.
Response of Q3 of Reviewer 58yt
Q3. How many independent training runs underlie the reported results, and what is the uncertainty? Is there evidence at a smaller step count?
Thank you for this question. Each result in the submitted paper is based on one independent training run per configuration. We agree that repeated runs would strengthen the evidence. As detailed in our response to Reviewer 1 (gZg3), Question 3, training and evaluating the 60 models in the paper already took about 3,900 H100 GPU-hours. Given this cost and our limited computational resources, we prioritized comparisons across datasets, objectives, and baseline schedules over additional training seeds. We will clarify the number of training runs per configuration in the paper.
As reported in our response to Question 2, our additional covariance experiments use three independent training seeds per configuration and report the mean and sample standard deviation. These results provide some evidence that the variation across training seeds is relatively small.
Finally, to test a smaller step count, we trained Cosine and endpoint-matched Geometric Cosine models on full-resolution CIFAR-10 with diffusion steps and 250,000 training updates under the hybrid objective. Both models used the same architecture and training settings apart from the noise schedule. We evaluated their final EMA checkpoints on 10,000 test images for the NLL bound and 50,000 generated samples for FID, using the full 1,000-step sampler:
| Schedule | NLL bound (bits/dim) (lower is better) | FID (lower is better) |
|---|---|---|
| Cosine | 3.3795 | 3.908 |
| Geometric Cosine | 3.1234 | 10.069 |
Geometric Cosine lowers the NLL bound by 0.2560 bits/dim, showing that the likelihood benefit extends to this smaller-step setting. Cosine achieves a lower FID, consistent with the hybrid-training results at . This comparison uses one training seed per schedule. These results provide some evidence that the benefit of the geometric schedule persists with fewer diffusion steps and fewer training updates.
We will include these additional results in the revised paper. If you think a broader comparison at across additional datasets would is important to strengthen the paper, we would be happy to include it in the camera-ready version.
Response to Q1 of Reviewer 58yt
Thank you for your careful review and helpful suggestions!
Q1. How well does the optimized upper bound track the actual accumulated Gaussianization error?
Thank you for this question! We agree that the upper bound can be loose. One source of looseness is its cubic dependence on dimension. For example, for d independent copies of a scalar distribution, both the reverse posterior and its moment-matched Gaussian factorize, so the actual Gaussianization error is exactly d times the scalar error. Our general bound does not exploit this structure and instead has a factor Cd=16d(d+2)(d+4). This illustrates room for improving the dimension dependence, although dimension is not the only source of looseness. More broadly, our intuition is that a sharper bound could depend on the intrinsic dimension of the data distribution, which can be much smaller than the ambient dimension d. However, note that if such a refinement changes only a multiplicative constant, it would tighten the bound without changing the optimal schedule.
Although the bound can overestimate the size of the error, it captures an important part of how the error depends on the step size. Let λ be the starting log-SNR level and s>0 the gap to the next level, λ−s. Write K(λ,s) for the averaged KL divergence from the true reverse conditional to its best Gaussian approximation for this step. Our one-step upper bound is CdΨ(es−1), where Ψ(r)=r2/2−r+log(1+r) and Cd is the dimension-dependent constant defined above. Under our finite-second-moment assumption, at a fixed starting level with positive pre-existing Gaussian noise, we have, as s→0,
=c(\lambda)s^3+o(s^3), \qquad C_d\Psi(e^s-1) =\frac{C_d}{3}s^3+O(s^4).$$ Here $c(\lambda)$ is a finite coefficient that depends on the data distribution and the starting noise level. This expansion follows from our exact derivative identity and the Gaussian smoothing already present, and applies to both discrete and continuous data. Thus both the actual error and the bound have a cubic leading term for non-Gaussian data: doubling a sufficiently small log-SNR gap makes each approximately eight times larger. Our bound captures this dependence but replaces the varying coefficient $c(\lambda)$ with a conservative uniform constant. This helps explain why avoiding large log-SNR gaps can help. To examine this directly, we used the $7\times7$ image-patch model of [Zach et al., *Product of Gaussian Mixture Diffusion Models* (JMIV, 2024)](https://link.springer.com/article/10.1007/s10851-024-01180-3), with the [authors' released parameters](https://github.com/VLOGroup/PoGMDM). This is a synthetic Gaussian-mixture model that the authors fitted to grayscale BSDS500 image patches to capture statistical structure in natural images. We chose it because it connects the experiment to real image data while allowing us to accurately compute the actual Gaussianization error by numerical integration, without training a reverse model. We used $T=100$ steps and compared each baseline with its matched geometric schedule, fixing the first-step noise and terminal signal level within each pair. The following are the accumulated bulk errors, summed over steps 2 through $T$, and their theoretical bounds, in nats per patch: | Schedule | Actual bulk KL | Bulk upper bound | | --- | --- | --- | | Linear | $0.423622$ | $7.20\times10^6$ | | Geometric Linear | $0.016459$ | $4.32\times10^5$ | | Cosine | $0.205934$ | $9.95\times10^{11}$ | | Geometric Cosine | $0.029823$ | $9.33\times10^5$ | The bounds remain much larger than the actual errors, but the geometric schedules reduce the actual bulk KL by $96.1\\%$ and $85.5\\%$, respectively. The first-step error is identical within each pair, so the sum including the first step also improves. Figure 1 in the supplementary file `rebuttal_figure.pdf` helps explain the improvement. The first four bulk transitions, $t=2,\ldots,5$, account for $98.4\\%$ of Linear's actual bulk error and $94.9\\%$ of Cosine's. These are transitions after the fixed first step, in a relatively clean region where the baseline takes large log-SNR steps. Geometric spacing assigns more steps to this region and substantially reduces its contribution. Each rectangle's width is one log-SNR gap, and its height is the actual one-step error divided by that gap. Its area is the actual one-step error, and the total area is the accumulated bulk error. We will add these controlled examples and clarify that the bound provides a distribution-free schedule-design criterion, rather than a numerically tight estimate or a guarantee of minimizing the actual error for each distribution. The examples are exploratory illustrations of its usefulness and limitations.