acceptodds
Under review as a conference paper at ICLR 2027

Optimizing Noise Schedules in Denoising Diffusion Probabilistic Models

Abstract

We study how to choose noise schedules in denoising diffusion probabilistic models by analyzing the error that can remain even with the best Gaussian reverse kernels. We derive an exact identity for the averaged one-step KL error and use it to obtain an explicit upper bound for every step after the first, assuming only a finite second moment of the data distribution. With the first-step noise and terminal signal level fixed, we minimize the resulting bound in closed form. The unique optimal schedule has geometrically growing noise-to-signal odds, or equivalently, linearly decreasing log-SNR. Across seven image datasets and class-conditional CIFAR-10, the resulting schedules improve the variational bound on negative log-likelihood compared with linear and cosine baselines, while also improving sample-quality metrics such as FID in many cases.

Primary area:
generative models
Keywords:
diffusion models, noise schedule, DDPM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_58ytOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the approximation error incurred when exact DDPM reverse conditionals are replaced by their KL-optimal, moment-matched Gaussian distributions. It derives an exact derivative identity for the averaged one-step Gaussianization error and obtains a bulk error bound under a finite-second-moment assumption. With the first-step noise and terminal signal level fixed, this bound is uniquely minimized by geometrically increasing noise-to-signal odds, equivalently a linear log-SNR schedule. Experiments compare matched-endpoint schedules on seven unconditional image datasets and class-conditional CIFAR-10 under variational and hybrid training objectives.

Critical strengths
  1. The exact characterization of Gaussianization error is an interesting theoretical contribution. Theorem 1 isolates the residual error after matching both the posterior mean and covariance and expresses its derivative as a nonnegative quantity involving third derivatives of the smoothed log-density. The proof addresses the extension from bounded inputs to finite-second-moment inputs. Although the resulting linear log-SNR schedule is already known, the analysis provides a useful perspective on why it arises from a finite-step Gaussian approximation criterion.

  2. The matched-endpoint experimental design controls important confounding factors. Within each baseline/geometric pair, the endpoints and training settings are held fixed, and the reported variational NLL bound improves in all 30 comparisons across unconditional and conditional settings. The paper also explicitly reports the limitations of the sample-quality results: FID improvements occur mainly under variational training, while hybrid training often favors the baselines. This balanced reporting makes the empirical contribution easier to assess.

Critical weaknesses
  1. The connection between the optimized bound and the practical model error remains insufficiently validated. The theory concerns unrestricted moment-matched Gaussian kernels, whereas the experiments learn diagonal Gaussian kernels. Restricting the covariance introduces an additional approximation error that can remain even when the true reverse conditional is Gaussian. This error is distinct from both the analyzed Gaussianization error and finite-capacity or optimization errors. The paper acknowledges the distinction, but does not establish which component accounts for the observed improvements.

The upper bound is also quantitatively loose at the experimental dimensions. For 32-by-32 RGB images and 4000 steps, substituting the stated endpoints into the optimized bulk bound gives approximately 8.6e7 nats for the linear-endpoint setting and 3.9e8 nats for the cosine-endpoint setting. These values do not invalidate the bound, but they leave its usefulness as a surrogate for actual Gaussianization error uncertain. A controlled non-Gaussian example with computable reverse posteriors would help connect the theoretical optimization to the quantity it aims to reduce.

  1. The experiments provide limited evidence about robustness across training randomness and step counts. The tables report point estimates without specifying the number of independent training seeds or uncertainty, although some NLL-bound improvements are small; for example, the variational SVHN linear/g-lin comparison changes from 1.715 to 1.710 bits/dim. All experiments also use 4000 steps, leaving the dependence on the number of transitions untested. Since the schedule's functional form is already established in prior work, evidence that the proposed criterion gives a reliable practical benefit is important to the paper's overall contribution.
Questions for authors

The following clarification and validation questions could affect my assessment:

  1. How well does the optimized upper bound track the actual accumulated Gaussianization error? Can the authors provide a controlled non-Gaussian example with exact or accurately computable posterior moments and errors, comparing schedules at fixed endpoints?

  2. Can the additional error from the diagonal-covariance restriction be quantified or separated from Gaussianization and model-fitting errors? What evidence supports attributing the observed likelihood-bound improvements to the mechanism analyzed in the theory?

  3. How many independent training runs underlie the reported results, and what is the uncertainty in representative matched comparisons? Is there evidence at a smaller step count indicating whether the observed benefit extends beyond the 4000-step setting?

For the discussion in Appendix A.4, I recommend considering the following related papers on information-theoretic perspectives on diffusion:

6 comments
AuthorsOpenReview ↗

Response to Q1 of Reviewer 58yt

Thank you for your careful review and helpful suggestions!

Q1. How well does the optimized upper bound track the actual accumulated Gaussianization error?

Thank you for this question! We agree that the upper bound can be loose. One source of looseness is its cubic dependence on dimension. For example, for dd independent copies of a scalar distribution, both the reverse posterior and its moment-matched Gaussian factorize, so the actual Gaussianization error is exactly dd times the scalar error. Our general bound does not exploit this structure and instead has a factor Cd=16d(d+2)(d+4)C_d=16d(d+2)(d+4). This illustrates room for improving the dimension dependence, although dimension is not the only source of looseness. More broadly, our intuition is that a sharper bound could depend on the intrinsic dimension of the data distribution, which can be much smaller than the ambient dimension dd. However, note that if such a refinement changes only a multiplicative constant, it would tighten the bound without changing the optimal schedule.

Although the bound can overestimate the size of the error, it captures an important part of how the error depends on the step size. Let λ\lambda be the starting log-SNR level and s>0s > 0 the gap to the next level, λ−s\lambda-s. Write K(λ,s)\mathcal K(\lambda,s) for the averaged KL divergence from the true reverse conditional to its best Gaussian approximation for this step. Our one-step upper bound is CdΨ(es−1)C_d\Psi(e^s-1), where Ψ(r)=r2/2−r+log⁡(1+r)\Psi(r)=r^2/2-r+\log(1+r) and CdC_d is the dimension-dependent constant defined above. Under our finite-second-moment assumption, at a fixed starting level with positive pre-existing Gaussian noise, we have, as s→0s\to0,

=c(\lambda)s^3+o(s^3), \qquad C_d\Psi(e^s-1) =\frac{C_d}{3}s^3+O(s^4).$$ Here $c(\lambda)$ is a finite coefficient that depends on the data distribution and the starting noise level. This expansion follows from our exact derivative identity and the Gaussian smoothing already present, and applies to both discrete and continuous data. Thus both the actual error and the bound have a cubic leading term for non-Gaussian data: doubling a sufficiently small log-SNR gap makes each approximately eight times larger. Our bound captures this dependence but replaces the varying coefficient $c(\lambda)$ with a conservative uniform constant. This helps explain why avoiding large log-SNR gaps can help. To examine this directly, we used the $7\times7$ image-patch model of [Zach et al., *Product of Gaussian Mixture Diffusion Models* (JMIV, 2024)](https://link.springer.com/article/10.1007/s10851-024-01180-3), with the [authors' released parameters](https://github.com/VLOGroup/PoGMDM). This is a synthetic Gaussian-mixture model that the authors fitted to grayscale BSDS500 image patches to capture statistical structure in natural images. We chose it because it connects the experiment to real image data while allowing us to accurately compute the actual Gaussianization error by numerical integration, without training a reverse model. We used $T=100$ steps and compared each baseline with its matched geometric schedule, fixing the first-step noise and terminal signal level within each pair. The following are the accumulated bulk errors, summed over steps 2 through $T$, and their theoretical bounds, in nats per patch: | Schedule | Actual bulk KL | Bulk upper bound | | --- | --- | --- | | Linear | $0.423622$ | $7.20\times10^6$ | | Geometric Linear | $0.016459$ | $4.32\times10^5$ | | Cosine | $0.205934$ | $9.95\times10^{11}$ | | Geometric Cosine | $0.029823$ | $9.33\times10^5$ | The bounds remain much larger than the actual errors, but the geometric schedules reduce the actual bulk KL by $96.1\\%$ and $85.5\\%$, respectively. The first-step error is identical within each pair, so the sum including the first step also improves. Figure 1 in the supplementary file `rebuttal_figure.pdf` helps explain the improvement. The first four bulk transitions, $t=2,\ldots,5$, account for $98.4\\%$ of Linear's actual bulk error and $94.9\\%$ of Cosine's. These are transitions after the fixed first step, in a relatively clean region where the baseline takes large log-SNR steps. Geometric spacing assigns more steps to this region and substantially reduces its contribution. Each rectangle's width is one log-SNR gap, and its height is the actual one-step error divided by that gap. Its area is the actual one-step error, and the total area is the accumulated bulk error. We will add these controlled examples and clarify that the bound provides a distribution-free schedule-design criterion, rather than a numerically tight estimate or a guarantee of minimizing the actual error for each distribution. The examples are exploratory illustrations of its usefulness and limitations.
AuthorsOpenReview ↗

Response to Q2 of Reviewer 58yt

Q2. Can the additional error from the diagonal-covariance restriction be quantified or separated from Gaussianization and model-fitting errors?

Thanks for your question! We used diagonal covariances to follow the DDPM experimental setup of Nichol and Dhariwal (2021). We wanted a controlled experiment that followed an existing setup, changing only the noise schedule, so that we could isolate the contribution of the schedule. Although our schedule is derived for the full-covariance setting, it improves the NLL bound even with diagonal covariances, suggesting that its benefits are robust to this restriction.

In general, we can decompose the error as follows. For a bulk reverse conditional qq, its moment-matched Gaussian g⋆g^\star, and any nondegenerate Gaussian approximation gg (for example, a Gaussian with diagonal covariance), moment matching gives

KL(q∥g)=KL(q∥g⋆)+KL(g⋆∥g)\mathrm{KL}(q\Vert g)=\mathrm{KL}(q\Vert g^\star)+\mathrm{KL}(g^\star\Vert g).

The first term is the Gaussianization error analyzed in our theory, averaged over the noisy input. The second term is the additional mismatch from restricting the covariance and learning the reverse kernel.

Motivated by your question, we ran a comparison with learned full bulk covariance. We tested 8×88\times8 grayscale versions of MNIST, Fashion-MNIST, and CIFAR-10. For each dataset, we trained cosine and endpoint-matched geometric-cosine schedules with diagonal and full bulk covariance using three training seeds per condition (36 runs in total). These smaller experiments use the same MLP backbone architecture, 250 diffusion steps, and 50,000 updates of direct VLB training. The full-covariance model adds off-diagonal outputs; the backbone and mean-head architectures are unchanged. The first-step decoder remains diagonal in both cases.

We evaluated the final exponential moving average (EMA) checkpoints on all 10,000 test images. The table reports the variational NLL bound in bits per grayscale pixel, as mean ±\pm sample standard deviation across the three training seeds.

DatasetBulk covarianceCosineGeometric-cosine
MNISTDiagonal4.3817 ± 0.02414.3158 ± 0.0164
MNISTFull4.3748 ± 0.01944.2824 ± 0.0031
Fashion-MNISTDiagonal5.4903 ± 0.01495.3892 ± 0.0088
Fashion-MNISTFull5.4722 ± 0.01445.3588 ± 0.0183
CIFAR-10Diagonal6.8195 ± 0.00136.8132 ± 0.0008
CIFAR-10Full6.8076 ± 0.00146.7979 ± 0.0006

Geometric-cosine improved the NLL bound in all 18 matched seed comparisons, including all nine with full bulk covariance. Thus the improvement persists when the diagonal restriction on bulk transitions is removed. These experiments do not separately quantify how much of the learned models' NLL-bound improvement comes from each term in the decomposition.

We will add this decomposition and the covariance experiments to the paper.

AuthorsOpenReview ↗

Response of Q3 of Reviewer 58yt

Q3. How many independent training runs underlie the reported results, and what is the uncertainty? Is there evidence at a smaller step count?

Thank you for this question. Each result in the submitted paper is based on one independent training run per configuration. We agree that repeated runs would strengthen the evidence. As detailed in our response to Reviewer 1 (gZg3), Question 3, training and evaluating the 60 models in the paper already took about 3,900 H100 GPU-hours. Given this cost and our limited computational resources, we prioritized comparisons across datasets, objectives, and baseline schedules over additional training seeds. We will clarify the number of training runs per configuration in the paper.

As reported in our response to Question 2, our additional covariance experiments use three independent training seeds per configuration and report the mean and sample standard deviation. These results provide some evidence that the variation across training seeds is relatively small.

Finally, to test a smaller step count, we trained Cosine and endpoint-matched Geometric Cosine models on full-resolution CIFAR-10 with T=1000T=1000 diffusion steps and 250,000 training updates under the hybrid objective. Both models used the same architecture and training settings apart from the noise schedule. We evaluated their final EMA checkpoints on 10,000 test images for the NLL bound and 50,000 generated samples for FID, using the full 1,000-step sampler:

ScheduleNLL bound (bits/dim) (lower is better)FID (lower is better)
Cosine3.37953.908
Geometric Cosine3.123410.069

Geometric Cosine lowers the NLL bound by 0.2560 bits/dim, showing that the likelihood benefit extends to this smaller-step setting. Cosine achieves a lower FID, consistent with the hybrid-training results at T=4000T=4000. This comparison uses one training seed per schedule. These results provide some evidence that the benefit of the geometric schedule persists with fewer diffusion steps and fewer training updates.

We will include these additional results in the revised paper. If you think a broader comparison at T=1000T=1000 across additional datasets would is important to strengthen the paper, we would be happy to include it in the camera-ready version.

Reviewer_NzLdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper derives an upper bound on DDPM’s Gaussian reverse-kernel error and shows that, with fixed endpoints, the bound is minimized by a schedule with linearly decreasing log-SNR.

  • Across seven image datasets and class-conditional CIFAR-10, this proposed approach improves the variational NLL bound over linear and cosine baselines, and improves FID in many cases.

Critical strengths
  • A principled derivation of the schedule. By modeling each step through a Gaussian-smoothed random variable, the paper derives an exact identity for the averaged KL mismatch and an explicit upper bound under only a finite-second-moment assumption. Minimizing that bound yields a unique schedule with linearly decreasing log-SNR. This gives a clear theoretical basis for a practical schedule, rather than relying only on empirical tuning.

  • Consistent likelihood results in controlled comparisons. The authors match schedule endpoints and training settings when comparing against linear and cosine baselines, and report lower variational NLL bounds across seven datasets and class-conditional CIFAR-10. Although sample-quality results are mixed, the breadth and consistency of the likelihood improvements support the practical relevance of the analysis.

Critical weaknesses
  • Limited conceptual novelty relative to prior schedule analyses. Aghapour et al., Entropy-Controlled Time-Discretization Bounds and Loss-Adaptive Schedules for Diffusion Models (TMLR 2026), initially circulated as Entropy-Based Dimension-Free Convergence and Loss-Adaptive Schedules for Diffusion Models, also derive schedule results from reverse-SDE error bounds. The manuscript itself notes that a linear log-SNR schedule can be obtained from their entropy-based analysis. Although this paper studies discrete DDPM Gaussianization error, it should clarify what substantive insight this adds beyond deriving a familiar schedule from a different error bound.

  • No comparison to a directly relevant, published training-schedule method. Hang et al., Improved Noise Schedule for Diffusion Training (ICCV 2025), directly study training-time noise schedules and report improvements over cosine on ImageNet-256 and ImageNet-512. While the methods optimize different criteria, a head-to-head comparison or a clearer discussion of why one is not appropriate would better establish the proposed schedule’s empirical value.

  • The practical quality gains are mixed and evaluated in a limited range of settings. The variational NLL bound improves across the reported matched comparisons, but under hybrid training, geometric-cosine has worse FID than cosine on all seven unconditional datasets. The experiments also use U-Nets on 32×32 and 64×64 image benchmarks, leaving the benefits for higher-resolution generation and other model setups unclear.

Questions for authors
  • Aghapour et al. also derive a linear log-SNR schedule by minimizing an error bound. Could you clarify the key theoretical distinction and the additional insight provided by your analysis?

  • Do the NLL improvements persist when the schedule endpoints—particularly the terminal SNR—are tuned, rather than inherited from the linear and cosine baselines?

  • If possible, could you compare your method with the training schedule proposed by Hang et al. (ICCV 2025) under a matched training budget? This would help clarify how your method compares with recent schedule-design approaches.

7 comments
AuthorsOpenReview ↗

Response to Q1 of Reviewer NzLd

Thank you for your careful review and helpful suggestions!

Q1. Aghapour et al. also derive a linear log-SNR schedule by minimizing an error bound. Could you clarify the key theoretical distinction and the additional insight provided by your analysis?

Although our analysis leads to the same linear log-SNR schedule, recovering this schedule is only one part of our theoretical contribution. We ask how much error is introduced by using Gaussian reverse transitions at finite step sizes, even when their mean and covariance are chosen optimally. Our exact characterization and bounds quantify this error and explain how it depends on the step sizes. These results have value independently of the particular schedule that minimizes the resulting bound.

Below, we explain what Aghapour et al. do, how our approach differs, and how our assumptions and theoretical guarantees compare.

What Aghapour et al. do. To isolate discretization error, they consider an exact denoiser that outputs the conditional mean of X0X_0 given the noisy image at any noise level. They then analyze a specific discretization of the reverse SDE that holds the denoiser output fixed between evaluations. Their geometric schedule comes from minimizing an entropy-based upper bound on the resulting discretization error.

What we do. We approach schedule design from a different perspective: How much error remains even with the best Gaussian approximation at each step? We optimize both the mean and full covariance to minimize the KL divergence from the true reverse conditional to its Gaussian approximation. This question comes from the usual justification for Gaussian reverse kernels (see, e.g., Sohl-Dickstein et al., 2015; Ho et al., 2020): the true reverse transitions become approximately Gaussian when the step sizes are sufficiently small. In practice, however, we use finite step sizes, and our main contribution is to characterize and bound the error that remains from this approximation. As a concrete example, if the data distribution is Gaussian, the true reverse transitions are already Gaussian, so the Gaussianization error we study is exactly zero. On the other hand, the discretization error studied by Aghapour et al. is not zero even for Gaussian data.

We derive an exact characterization of this error and its derivative, and use it to obtain a cubic small-step bound. These results quantify the accuracy of the Gaussian approximation independently of the resulting schedule formula. Minimizing our bound also recovers the linear log-SNR schedule.

Comparing the assumptions and results. The Shannon-entropy bound of Aghapour et al. uses the following assumptions (Assumptions 1–2 in their paper):

  1. The data distribution has a finite second moment: E∣X0∣2<∞\mathbb{E}\\|X_0\\|^2 < \infty.

  2. The data distribution has countable discrete support C\mathcal C, with probability mass function pp.

  3. The Shannon entropy H:=E[−log⁡p(X0)]H:=\mathbb{E}[-\log p(X_0)] is finite, and the information content −log⁡p(X0)-\log p(X_0) satisfies a sub-exponential concentration condition. Specifically, there are constants ν2>0\nu^2 > 0 and b∈(0,2]b\in(0,2] such that

    \le \exp(\nu^2\lambda^2) \qquad\text{for all }|\lambda|\le 1/b.$$ This controls fluctuations in the information content around its mean, beyond requiring the mean to be finite.

They also give an alternative to the third condition using finite Rényi entropy of order 1/21/2, namely H1/2:=2log⁡∑x∈Cp(x)<∞H_{1/2}:=2\log\sum_{x\in\mathcal C}\sqrt{p(x)} < \infty (Remark 5), with the bound expressed in terms of H1/2H_{1/2}.

Our bulk bound requires only the finite second moment assumption, and applies to discrete and continuous data. Our bound also has a better dependence on the number of time steps. Writing TT for the number of steps, with the noise endpoints fixed, our accumulated bulk Gaussianization bound is O(T−2)O(T^{-2}), whereas their discretization bound is O(T−1)O(T^{-1}). One tradeoff is that their bound has the advantage of no explicit dependence on ambient dimension, whereas our constant depends on dimension.

We will revise the related-work discussion to make these distinctions explicit.

AuthorsOpenReview ↗

Response to Q2 of Reviewer NzLd

Q2. Do the NLL improvements persist when the schedule endpoints—particularly the terminal SNR—are tuned, rather than inherited from the linear and cosine baselines?

Thank you for this question. We ran some additional experiments, whose results suggest that the likelihood benefit is not specific to the endpoints inherited from the baselines. To check this, we conducted an endpoint study using short training runs on CIFAR-10. We then trained models for the full training budget using the selected endpoints and transferred these endpoints to three other datasets without further tuning. The details are below.

Here β1\beta_1 is the first-step noise variance and αˉT\bar\alpha_T is the terminal signal retention. We tested 28 endpoint pairs on CIFAR-10, using hybrid-training runs of 50,000 updates, with 45,000 training images and 5,000 validation images. We used a coarse grid covering the endpoint scales of the standard Linear and Cosine schedules. We also tested larger terminal signal levels, up to αˉT=10−2\bar\alpha_T=10^{-2}, to examine whether the likelihood benefit persists beyond the terminal SNRs used by these baselines. The search comprised β1∈10−6,3×10−6,10−5,3×10−5\beta_1\in\\{10^{-6},3\times10^{-6},10^{-5},3\times10^{-5}\\} crossed with αˉT∈10−10,10−8,10−6,10−4\bar\alpha_T\in\\{10^{-10},10^{-8},10^{-6},10^{-4}\\}, and β1∈3×10−6,10−5,3×10−5\beta_1\in\\{3\times10^{-6},10^{-5},3\times10^{-5}\\} crossed with αˉT∈3×10−4,10−3,3×10−3,10−2\bar\alpha_T\in\\{3\times10^{-4},10^{-3},3\times10^{-3},10^{-2}\\}. We selected the pair with the lowest validation NLL bound: β1=10−5\beta_1=10^{-5} and αˉT=0.003\bar\alpha_T=0.003.

These endpoints substantially change the terminal SNR, which is αˉT/(1−αˉT)\bar\alpha_T/(1-\bar\alpha_T). The selected value is approximately 0.0030090.003009, compared with 1.52×10−101.52\times10^{-10} for standard Cosine; over seven orders of magnitude larger.

We then tested these selected endpoints on four datasets without any further tuning. The resulting NLL bounds on held-out data, in bits/dimension, are shown below:

DatasetObjectiveStandard CosineGeometric Cosine endpointsGeometric Selected endpoints
MNISTVLB0.31560.31280.3089
MNISTHybrid0.34630.28650.3274
Fashion-MNISTVLB0.81650.77330.7722
Fashion-MNISTHybrid0.82790.76120.7777
CIFAR-10VLB2.98132.92482.9194
CIFAR-10Hybrid3.20692.92132.9282
ImageNet-64VLB3.35713.31023.3059
ImageNet-64Hybrid3.36683.31593.3034

All schedules use 4,000 diffusion steps. We used 500,000 training updates for the 32×3232\times32 datasets (MNIST, Fashion-MNIST, and CIFAR-10) and 200,000 for ImageNet-64. We evaluate the NLL bound on 10,000 held-out samples per dataset using the final exponential moving average (EMA) checkpoint.

The selected endpoints do not consistently improve upon the inherited endpoints, but they preserve the improvement over standard Cosine across all eight dataset/objective comparisons. Thus, substantially different terminal SNRs can retain the likelihood benefit, including when the endpoints are transferred to other datasets without further tuning.

We would be happy to add this endpoint study to the paper if you think it would be useful.

AuthorsOpenReview ↗

Response to Q3 of Reviewer NzLd

Q3. Could you compare your method with the training schedule proposed by Hang et al. (ICCV 2025) under a matched training budget?

Thanks for pointing us to this paper! We adapted Hang et al.'s Laplace schedule to our DDPM training setup and compared it on CIFAR-10 with our geometric schedule matched to the endpoints of the resulting discrete chain. In this experiment, our schedule compares favorably to the Laplace schedule. The details are below.

Let λ(u)\lambda(u) denote the log signal-to-noise ratio at continuous diffusion time uu. Hang et al.'s Laplace schedule is

\begin{cases} \mu-b\log(2u), & 0 < u < 1/2,\\\\ \mu+b\log\bigl(2(1-u)\bigr), & 1/2\le u < 1, \end{cases}$$ with $\mu=0$ and $b=0.5$. To obtain a $T=4000$-step DDPM schedule, we set $\bar\alpha(u)=1/(1+e^{-\lambda(u)})$, with $\bar\alpha(0)=1$ and $\bar\alpha(1)=0$, and discretized it on the uniform grid $u=i/T$: $$\beta_i=\min\\!\left\\{1-\frac{\bar\alpha(i/T)}{\bar\alpha((i-1)/T)},\\;0.999\right\\}, \qquad i=1,\ldots,T.$$ We then matched our geometric schedule to the endpoints of the discrete Laplace schedule. We used the same unconditional CIFAR-10 hybrid-training setup and training budget as the cosine/geometric-cosine comparison in our paper. The results for one training seed per schedule are: | Schedule | NLL bound (bits/dim) | FID | | --- | --- | --- | | Laplace | 5.627 | 38.175 | | Geometric-Laplace | **4.941** | **8.439** | Training and evaluation took about two days with the two models run in parallel: roughly 39 hours of training and 9 hours of evaluation per model. Given the computational cost, we focused on this controlled CIFAR-10 comparison during the rebuttal period. We are happy to include this comparison in the revised paper. We would also be happy to extend the comparison to additional datasets and training objectives in the camera-ready version, if you think this would be useful!
Reviewer_gZg3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary
  • This paper studies the theoretical design of noise schedules for DDPMs by analyzing the irreducible approximation error introduced by Gaussian reverse transitions.

    • The authors first decompose the pathwise KL divergence between the true reverse process and an idealized Gaussian reverse chain into a terminal mismatch term and a sum of per-step Gaussianization errors.
    • They then derive an exact differential identity for the expected KL divergence between the true reverse conditional distribution and its moment-matched Gaussian approximation.
      • Based on this identity, they establish a distribution-independent upper bound on the Gaussianization error for each bulk step under only a finite-second-moment assumption.
      • By fixing the first-step noise level and the terminal signal retention, they formulate and solve a schedule optimization problem in closed form, showing that the unique minimizer of the resulting bulk upper bound has geometrically increasing noise-to-signal odds, equivalently corresponding to a linearly decreasing log-SNR schedule.
  • The theoretical analysis is supported by experiments on seven image datasets and class-conditional CIFAR-10.

    • The proposed schedules consistently improve the variational upper bound on negative log-likelihood (NLL) over matched linear and cosine baselines.
    • However, improvements in sample quality are less consistent, particularly under the hybrid training objective.
  • Overall, the paper presents an interesting theoretical perspective on noise schedule design, but I have several concerns regarding the tightness and practical relevance of the derived bounds, the connection between the idealized analysis and learned diffusion models, the novelty of the resulting schedule, and the robustness of the empirical improvements.

Critical strengths
  • The paper identifies the mismatch between the true reverse conditional distribution and its Gaussian approximation as an important source of error in discrete-time diffusion models.
    • The pathwise KL decomposition in §2\S2 provides a clear framework for isolating this source of error. I find this perspective valuable because it complements existing analyses of score estimation and reverse-SDE discretization error.
  • The paper is well organized, and the theoretical results, proofs, and implementation details are presented systematically. The derivations are generally easy to follow.
  • Theorem 4 establishes a unique minimizer for the proposed fixed-endpoint bulk upper-bound objective. The resulting geometric-odds schedule is simple, requires no additional learned parameters, and can be incorporated into existing DDPM implementations with minimal modifications.
Critical weaknesses
  • The paper derives an exact identity and an upper bound for the per-step Gaussianization error, but it does not empirically evaluate the actual Gaussianization error or compare it with the derived bound.
    • Consequently, the theoretical analysis provides an appealing qualitative principle, but its quantitative explanatory power and practical relevance remain unclear.
    • For example, experiments on tractable synthetic distributions could help assess how tight the bound is and whether its minimization reliably predicts reductions in the actual Gaussianization error.
  • The empirical results reveal a substantial discrepancy between likelihood-bound improvements and FID under the hybrid training objective.
    • In particular, the geometric-cosine schedule consistently produces worse FID than the cosine baseline across all seven unconditional image datasets, despite improving the variational NLL bound in every case.
    • Since the hybrid objective is a commonly used training formulation in practice, this finding limits the claim that the proposed schedule provides broadly beneficial improvements for diffusion-based image generation.
    • A more thorough investigation of the discrepancy between likelihood and perceptual quality, potentially accounting for the training objective and learned reverse-kernel approximation error, would strengthen the paper.
Questions for authors
  1. Why does geometric-cosine consistently underperform the cosine baseline in FID under hybrid training? Both schedules improve the variational NLL bound, yet their sample-quality results differ substantially. Can the authors provide an explanation, supported by additional analysis or experiments, of how the hybrid objective interacts with the proposed schedule? In particular, could the discrepancy be related to the relative weighting of different timesteps or to differences between likelihood optimization and perceptual quality?

  2. Can the authors quantitatively connect the theoretical Gaussianization error to the observed improvements in the variational NLL bound? The experiments use learned reverse kernels with diagonal covariance matrices, whereas the theoretical analysis considers the KL-optimal Gaussian approximation without restricting the covariance structure. Could the authors provide an error decomposition or controlled experiments that distinguish the intrinsic Gaussianization error from the additional errors introduced by learning the reverse kernels and restricting their covariance matrices to be diagonal?

  3. Could the authors report the hardware specifications and computational costs of the experiments? In particular, GPU models, training time, and total computational cost would help readers assess the computational requirements and reproducibility of the experimental results.

  4. How does the theoretical contribution differ from the geometric log-SNR schedule derived by Aghapour et al. (2026)?[1] Both approaches lead to closely related schedule forms, but optimize different error criteria. Could the authors clarify the precise differences in the quantities being optimized, the assumptions required, and the theoretical guarantees obtained by the two approaches? A more explicit comparison would help establish the novelty and complementary nature of the present analysis.

[1] Entropy-Based Dimension-Free Convergence and Loss-Adaptive Schedules for Diffusion Models.

7 comments
AuthorsOpenReview ↗

Response to Q1 of Reviewer gZg3

Thank you for your careful review and helpful suggestions!

Q1. Why does geometric-cosine consistently underperform the cosine baseline in FID under hybrid training?

This is a good question. We believe one important part of the explanation is that changing the noise schedule also changes how the hybrid objective weights different noise levels. The MSE term gives equal weight to each timestep, but the corresponding noise levels differ between schedules. Cosine concentrates more timesteps at intermediate noise levels, whereas Geometric Cosine spreads them evenly over log-SNR.

For example, in our 4,000-step setup, 61.5% of Cosine's timesteps have SNR between 0.1 and 10, compared with only 13.5% for Geometric Cosine. Conversely, Geometric Cosine allocates 59.5% of its timesteps to SNR below 0.1, compared with 19.7% for Cosine. Since timesteps are sampled uniformly, these fractions also describe how the unweighted MSE objective allocates its weight. Our hypothesis is that Cosine's greater emphasis on intermediate noise levels, where the input contains both signal and substantial noise, helps the model learn features relevant to perceptual quality.

To test this explanation, we ran an additional CIFAR-10 experiment that keeps the Geometric Cosine schedule but reweights its MSE term to approximately match Cosine's allocation of weight across noise levels. We normalized the weights to have mean one and kept the variational-bound term and all other training settings unchanged. Timesteps were still sampled uniformly; only their MSE weights changed.

We trained all three models for 500,000 updates with 4,000 diffusion steps, using one training seed per condition, and evaluated their final exponential moving average (EMA) checkpoints:

Schedule and MSE weightingNLL (bits/dim)FID
Cosine, original weighting3.2155.366
Geometric Cosine, original weighting2.9229.868
Geometric Cosine, cosine-like weighting3.1494.139

Reweighting improves Geometric Cosine's FID from 9.868 to 4.139, bringing it below Cosine's 5.366. Its NLL bound worsens relative to the original Geometric Cosine model, but remains better than Cosine's. These results support the explanation that the allocation of denoising-loss weight across noise levels contributes to the FID gap under hybrid training.

This also suggests a promising heuristic to combine the two approaches: use the geometric noise schedule together with cosine-like weighting of the denoising loss. In this experiment, that combination improves both FID and the NLL bound relative to Cosine.

We will add this experiment and the discussion of how loss weighting affects the FID results to the paper.

AuthorsOpenReview ↗

Response to Q2 of Reviewer gZg3

Q2. Can the authors quantitatively connect the theoretical Gaussianization error to the observed improvements in the variational NLL bound?

Thanks for your question! We used diagonal covariances to follow the DDPM experimental setup of Nichol and Dhariwal (2021). We wanted a controlled experiment that followed an existing setup, changing only the noise schedule, so that we could isolate the contribution of the schedule. Although our schedule is derived for the full-covariance setting, it improves the NLL bound even with diagonal covariances, suggesting that its benefits are robust to this restriction.

In general, we can decompose the error as follows. For a bulk reverse conditional qq, its moment-matched Gaussian g⋆g^\star, and any nondegenerate Gaussian approximation gg (for example, a Gaussian with diagonal covariance), moment matching gives

KL(q∥g)=KL(q∥g⋆)+KL(g⋆∥g)\mathrm{KL}(q\Vert g)=\mathrm{KL}(q\Vert g^\star)+\mathrm{KL}(g^\star\Vert g).

The first term is the Gaussianization error analyzed in our theory, averaged over the noisy input. The second term is the additional mismatch from restricting the covariance and learning the reverse kernel.

Motivated by your question, we ran a comparison with learned full bulk covariance. We tested 8×88\times8 grayscale versions of MNIST, Fashion-MNIST, and CIFAR-10. For each dataset, we trained cosine and endpoint-matched geometric-cosine schedules with diagonal and full bulk covariance using three training seeds per condition (36 runs in total). These smaller experiments use the same MLP backbone architecture, 250 diffusion steps, and 50,000 updates of direct VLB training. The full-covariance model adds off-diagonal outputs; the backbone and mean-head architectures are unchanged. The first-step decoder remains diagonal in both cases.

We evaluated the final EMA checkpoints on all 10,000 test images. The table reports the variational NLL bound in bits per grayscale pixel, as mean ±\pm sample standard deviation across the three training seeds.

DatasetBulk covarianceCosineGeometric-cosine
MNISTDiagonal4.3817 ± 0.02414.3158 ± 0.0164
MNISTFull4.3748 ± 0.01944.2824 ± 0.0031
Fashion-MNISTDiagonal5.4903 ± 0.01495.3892 ± 0.0088
Fashion-MNISTFull5.4722 ± 0.01445.3588 ± 0.0183
CIFAR-10Diagonal6.8195 ± 0.00136.8132 ± 0.0008
CIFAR-10Full6.8076 ± 0.00146.7979 ± 0.0006

Geometric-cosine improved the NLL bound in all 18 matched seed comparisons, including all nine with full bulk covariance. Thus the improvement persists when the diagonal restriction on bulk transitions is removed.

We will add this decomposition and the covariance experiments to the paper.

AuthorsOpenReview ↗

Response to Q3 of Reviewer gZg3

Q3. Could the authors report the hardware specifications and computational costs of the experiments?

Yes! We will add these specifications to the paper. Each model was trained on one NVIDIA H100 GPU with 80 GB of GPU memory, using eight CPU cores and 32–64 GB of host memory. Each evaluation job also used one H100 GPU. The table below gives the recorded training and evaluation times for the 60 models reported in the paper.

DatasetModelsUpdates per modelTraining hours/modelEvaluation hours/model
SVHN8500,00017.7–18.116.4–16.5
CIFAR-108500,00047.0–58.616.4–16.6
CIFAR-1008500,00017.7–20.016.4–16.5
Conditional CIFAR-104500,00048.3–49.316.4–16.5
CelebA-648200,00024.7–25.256.7–57.0
LSUN-Bedroom-648200,00024.5–25.248.0–48.6
LSUN-Church-648200,00024.6–25.647.9–48.1
ImageNet-648200,00024.7–25.856.7–67.2

Across these 60 models, training used approximately 1700 GPU-hours, and evaluation used approximately 2,200 GPU-hours, for a combined total of about 3,900 GPU-hours.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.