acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Second-Order Solvers for Fast Stochastic Diffusion Sampling

Abstract

Diffusion models rely on numerical solvers requiring time-discretization, which has a large influence on the tradeoff between sampling cost and quality. However, the computational difficulty of the reverse process varies along the sampling trajectory and across data distributions, making the choice of discretization important. We adapt proportional-integral (PI) step-size control to diffusion, using our diffusion noise-normalised error estimator. Unlike existing adaptive methods in diffusion that respond only to the current error, the PI solver also incorporates the previous error, yielding smoother step adaptation. We further show that these per-sample trajectories exhibit shared structure and can be aggregated into a fixed schedule that retains much of the benefit of adaptive sampling. We evaluate both approaches on natural-image and language datasets, in terms of quality, measured by FID at a matched number of neural network evaluations (NFE), comparing them with widely used stochastic solvers and schedules. For images, our average discretization outperforms the commonly used EDM schedule in terms of sample quality when used with the stochastic Heun sampler, and with the EDM-churn sampler at low NFE. Additionally, our PI adaptive solver obtains better FID than most stochastic and adaptive baselines, although it does not beat the EDM-churn sampler at low NFE. Moreover, we find our solver outperforms both the EDM and the entropy schedule on language diffusion at low-to-medium NFE in terms of perplexity, with the drawback of lower token entropy. Lastly, we find that the benefit of per-sample adaptivity is problem-dependent. It is highly beneficial in 1D toy examples, while only marginal for image and language data, where the average schedule sometimes even outperforms the per-sample adaptive solver. Code is available at https://anonymous.4open.science/r/adaptive-generative-diffusion-solvers-4EE5/

Primary area:
generative models
Keywords:
Diffusion Models, Diffusion Sampling, Adaptive Solvers, Stochastic Differential Equations, Numerical Methods, Time-Step Scheduling, PI Control

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_x4orOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a proportional-integral (PI) adaptive solver for stochastic diffusion sampling, using current and previous local errors to adjust step sizes. It introduces a diffusion noise-normalized error estimator and a method for averaging adaptive trajectories into reusable static sampling schedules. Experiments on Gaussian mixtures, FFHQ, ImageNet-64, and LM1B show improvements over several baselines in sampling quality at matched NFE. The results also suggest that the benefits of per-sample adaptivity are task-dependent, with static schedules sometimes performing comparably or better.

Critical strengths

1. Practical and reusable data-driven sampling schedules. A notable strength is the conversion of per-sample adaptive trajectories into reusable static schedules. These schedules can be integrated with existing samplers without modifying or retraining the underlying diffusion model. The improvements observed with stochastic Heun and EDM-churn suggest that the learned allocation of sampling steps captures useful information about the diffusion dynamics. This makes the approach practically relevant beyond the proposed adaptive solver itself.

2. Evaluation across multiple modalities with informative ablations. The paper evaluates the approach on synthetic distributions, image generation, and language diffusion. The ablations help isolate the contributions of PI control and noise-normalized error estimation. Importantly, the authors also acknowledge settings where per-sample adaptivity provides limited benefits. These results offer useful empirical insights into when adaptive sampling or static schedules may be preferable.

Critical weaknesses

W1. Limited Comparison with Existing Optimized Sampling Schedules. The paper demonstrates promising improvements over the handcrafted EDM schedule and includes comparisons with the entropy-based schedule. However, the experimental evaluation does not cover other relevant data-driven scheduling methods already discussed by the authors in the Related Work section, such as Align Your Steps (Sabour et al., 2024) and Adaptive Time-Stepping Schedules (Chen et al., 2024). Although these methods may target different solver settings, additional comparisons where applicable, or a more detailed discussion of their differences and applicability, would help better establish the effectiveness of the proposed PI Static schedule relative to existing approaches.

W2. Limited evidence for practical efficiency gains over strong baselines. The adaptive PI solver does not consistently outperform the strongest existing sampler, EDM-churn. In particular, at 49 NFE, its FID is substantially worse on both FFHQ and ImageNet-64. Some of the strongest results instead come from combining the proposed static schedule with EDM-churn. Furthermore, efficiency is evaluated primarily through NFE, without reporting wall-clock sampling time or the computational cost of constructing the static schedules. These omissions make it difficult to assess the practical efficiency advantage of the proposed approach, particularly when per-sample adaptive execution introduces additional implementation overhead.

Questions for authors

Q1. Comparison with Existing Schedules. Could the authors provide comparisons with other optimized scheduling methods discussed in the Related Work, such as Align Your Steps and Adaptive Time-Stepping Schedules, where applicable, or clarify why these methods are not directly comparable?

Q2. Practical Efficiency. Could the authors report wall-clock sampling time in addition to NFE, and clarify the computational overhead of constructing the PI Static schedule?

Q3. Adaptive vs. Static Sampling. Why does the PI adaptive solver perform substantially worse than EDM-churn at low NFE, while the PI Static schedule combined with EDM-churn achieves better results? Could the authors further explain this difference?

2 comments
AuthorsOpenReview ↗

Rebuttal by the Authors

Thank you for the review and the provided feedback. Our responses follow.

Q1 Comparison with Existing Schedules

We have looked at implementing the Align Your Steps method and Adaptive Time-Stepping Schedules. Neither method supplies an implementation that works for the EDM pipeline (Align Your Steps does not supply an implementation at all). Therefore, both methods would need to be implemented and tuned for the EDM pipeline, which would introduce additional confounds and make these hard to use as a proper baseline.

Q2 Practical Efficiency

The difference in wall-clock time between the PI solver and other methods is fairly small. We ran an experiment comparing total wall-clock time required for generating 10 batches. The overhead created by the more complex sampling algorithm is about 1-2% compared to simpler algorithms such as Euler-Maruyama or EDM. Compared to the Gotta Go Fast sampler, our algorithm is about 2-5% faster in terms of seconds/sample at matched NFE. The difference between method likely shrinks with larger models, as larger models require more compute per sample, while the compute required by our algorithm remains the same. For a full table of the results, refer to section 6.4 of the updated manuscript.

The time it takes to compute a static schedule from sampled PI trajectories depends on the number sampled. In this paper, we used 50000 sampled trajectories to compute the static schedule at matched NFE. Aggregating these sampled trajectories into a single static schedule takes about 5 seconds, and only has to be done once before the schedule is usable. Sampling trajectories is the computationally expensive part. Sampling 50000 images takes about 18 hours on ImageNet-64 and 6 hours on FFHQ at 75NFE. However, this large sample size is likely not required to obtain good schedules due to how homogeneous the individual per-sample schedules are.

Q3 Adaptive vs Static Sampling

EDM-churn with PI static performs substantially better than PI adaptive at 49 NFE because these use different sampling mechanisms. EDM-churn does not directly sample from an SDE. Instead, it samples from the probability flow ODE, increasing the noise level at step tit_i to γσ(ti)\gamma \sigma(t_i), then denoising this in a predictor and corrector steps to ti+1t_{i+1}. Instead, PI adaptive uses stochastic Heun’s method on the reverse SDE. This adds a noise increment Δwi\Delta w_i to both the predictor and corrector step. This different mechanism of adding noise in the two samplers likely explains the large difference in performance between these two samplers. Even though this difference in samplers exists, we still show that our PI Static schedule can improve performance over the EDM schedule in a sampler using a different sampling method.

Thank you for the additional experiments and clarifications. The new wall-clock results and the explanation of the differences between adaptive and static sampling have addressed most of my concerns. While I understand the challenges of directly comparing different scheduling approaches, the relative advantages of PI Static over existing methods remain insufficiently established. Therefore, I will maintain my original rating.

Reviewer_oR75OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper adapts proportional-integral (PI) step-size control to stochastic diffusion sampling using a diffusion-noise-normalized error estimator. It also aggregates per-sample adaptive time grids into reusable static schedules. Experiments on synthetic, image, and language tasks examine sampling quality at matched NFE and the benefits of per-sample adaptivity.

Critical strengths

Noise-aware error normalization is a simple, effective adaptation supported by component ablations. Besides, aggregating adaptive grids into static schedules offers a practical way to reuse their allocation patterns. Experiments cover multiple modalities and report uncertainty for the main image comparisons.

Critical weaknesses
  1. The paper repeatedly describes stochastic Heun as a second-order solver without distinguishing deterministic and convergence orders. Neither a convergence analysis nor an empirical order test is provided for the implemented adaptive scheme. Using two network evaluations per step does not establish second-order strong convergence.
  2. The PI controller is inherited from Ilie et al. (2015), while the main additions appear to be diffusion-noise normalization and aggregation into static schedules.
  3. Every proposed step is accepted before its estimated error affects the next step size. Consequently, a large initial error cannot be corrected by subsequent step-size reduction.
  4. No end-to-end timing comparison is provided, and the costs of hyperparameter tuning and static-schedule construction are not quantified.
  5. The final ODE stage uses a predetermined schedule, but its contribution is not systematically separated from adaptive SDE stepping. Image results cover only three NFE budgets, despite substantial performance changes across them. Actual NFE distributions for adaptive methods are not reported, and comparisons with compatible optimized schedules remain limited. The language results also involve a quality–diversity trade-off that requires more than perplexity alone to assess.
  6. The data-endpoint notation is inconsistent, several abbreviations are undefined, and the explanations following Eqs. (3) and (4) contain sentence fragments.
Questions for authors
  1. What precisely does "second-order" denote? Can you provide the relevant assumptions, supporting analysis, or an empirical convergence-order test for the actual implementation, including adaptive stepping and disabled rejection?
  2. How does the Euler–Heun difference relate to the true local error? Please clarify its distinction from Ilie’s Milstein step-doubling estimator.
  3. What happens when the first accepted step has a large estimated error? Can you report sensitivity to the initial step size, numerical failure rates, and a matched-NFE comparison with step rejection enabled?
  4. Can you report measured latency across batch sizes, alongside actual NFE distributions?
  5. How much of the improvement comes from the final ODE stage?
  6. Does "identical noise" mean shared Gaussian draws or consistent increments from the same Brownian path?
  7. For language generation, could you assess perplexity at comparable diversity or supplement unigram entropy with sequence-level diversity metrics?
4 comments
AuthorsOpenReview ↗

Rebuttal by the Authors

Thank you for the detailed review. Our responses follow. Note that this is sent in two messages, as the full message exceeds the OpenReview character limit

Q1 What precisely does "second-order" denote? Can you provide the relevant assumptions, supporting analysis, or an empirical convergence-order test for the actual implementation, including adaptive stepping and disabled rejection?

With second-order, we do not mean convergence order. Instead, we mean it as a stochastic Runge-Kutta method of deterministic order 2, i.e., we take an Euler-Maruyama step (first order) using x^i+1=x^i+f(x^i,ti)hi+g(ti)Δwi\mathbf{\hat{x}}_{i+1} = \mathbf{\hat{x}}_i + \mathbf{f}(\mathbf{\hat{x}}_i, t_i)h_i + g(t_i)\Delta \mathbf{w}_i And then take a corrector step (second-order, a stochastic Heun’s step) using the first order step: x^i+1’’=x^i+[f(x^i,ti)+f(x^i+1′,ti+1)]hi2+12[g(ti)+g(ti+1)]Δwi\mathbf{\hat{x}}_{i+1}’’ = \mathbf{\hat{x}}_i + \left[\mathbf{f}(\mathbf{\hat{x}}_i, t_i) + \mathbf{f}(\mathbf{\hat{x}}_{i+1}', t_{i+1})\right]\frac{h_i}{2} + \frac{1}{2}\left[g(t_i) + g(t_{i+1})\right]\Delta \mathbf{w}_i In the noise-free limit, this reduces to Heun’s method, which has deterministic order 2. We make no claim about the strong or weak order of the stochastic scheme, with or without adaptive stepping, and we have not run an empirical order test. We have clarified this in the introduction and background, and we state the absence of a convergence analysis in the limitations section.

Q2 How does the Euler–Heun difference relate to the true local error? Please clarify its distinction from Ilie’s Milstein step-doubling estimator.

We use a notion known as the local extrapolated error as an approximation of the true local error, which compares a less and more accurate estimate of the next state. Ilie et al (2015) also do this, although their method differs from ours. They compute one step using a Milstein method at the base step size hh, and then compute the same step by taking two smaller steps of step h/2h/2 along the same Brownian path. We instead take the approach of an embedded method, computing the local extrapolated error as the difference between the more accurate higher-order method (stochastic Heun’s), and the lower-order method (Euler’s method). Both methods aim to approximate the same value, namely the true local error.

Q3 What happens when the first accepted step has a large estimated error? Can you report sensitivity to the initial step size, numerical failure rates, and a matched-NFE comparison with step rejection enabled?

First step size The first accepted step having a large estimated error is generally not a problem. However, we generally observe that when the first step is too large, sample quality degrades very quickly. The influence of the initial step-size itself is primarily in the NFE-quality trade-off, as generally larger initial step-size leads to less required NFE and does not cause large differences in terms of quality, until it is too large and quality degrades noticeably.

Numerical Failures We did not observe numerical failures.

Step reject enabled We are currently running an experiment with step-rejection enabled, and will report these in the camera-ready version. Currently available results on FFHQ show an FID of 155.48 (compared to 83.95 without step rejection) at 49 NFE and 3.80 (compared to 3.32) at 75 NFE. These preliminary results imply step-rejection has a negative effect on the NFE-FID trade-off.

AuthorsOpenReview ↗

Rebuttal by the Authors

Q4 Can you report measured latency across batch sizes, alongside actual NFE distributions?

We have provided plots of the NFE distributions in the updated version of the manuscript in Appendix E. The distributions are generally very concentrated at the target NFE, with a bit of spread below and above the target NFE. The only one with quite some variance is 49 NFE of ImageNet-64, with quite a large amount of samples being sampled at 51 NFE.

We also ran experiments to determine the batch latency. The difference in wall-clock time between the PI solver and other methods is fairly small. We ran an experiment comparing total wall-clock time required for generating 10 batches. The overhead created by the more complex sampling algorithm is about 1-2% compared to simpler algorithms such as Euler-Maruyama or EDM. Compared to the Gotta Go Fast sampler, our algorithm is about 2-5% faster in terms of seconds/sample at matched NFE. The difference between method likely shrinks with larger models, as larger models require more compute per sample, while the compute required by our algorithm remains the same. For a full table of the results, refer to section 6.4 of the updated manuscript.

The time it takes to compute a static schedule from sampled PI trajectories depends on the number sampled. In this paper, we used 50000 sampled trajectories to compute the static schedule at matched NFE. Aggregating these sampled trajectories into a single static schedule takes about 5 seconds, and only has to be done once before the schedule is usable. Sampling trajectories is the computationally expensive part. Sampling 50000 images takes about 18 hours on ImageNet-64 and 6 hours on FFHQ at 75NFE. However, this large sample size is likely not required to obtain good schedules due to how homogeneous the individual per-sample schedules are.

Q5 How much of the improvement comes from the final ODE stage?

Please note that to make the experiments more fair, instead of only letting our solver take steps on the ODE for the last few steps, we also had every other solver take an equal amount of steps on the ODE. Therefore, the ODE steps themselves should not have a big influence on the final improvements of our solver over the others. However, we have now run an additional experiment, running our solver without the ODE tail on FFHQ at 49, 75, and 99 NFE. This leads to FIDs of 84.10, 4.04, and 3.34 for 49, 75, and 99 NFE, respectively. At 49NFE, the difference is minimal compared to using an ODE tail (83.95 vs 84.10). However, at 75 and 99 NFE, the difference is more substantial (3.32 vs 4.04 and 3.14 vs 3.34). Thus, taking a few steps on the ODE of the trajectory does improve performance.

Q6 Does "identical noise" mean shared Gaussian draws or consistent increments from the same Brownian path?

With identical noise, we mean shared Gaussian draws between the predictor and corrector steps.

Q7 For language generation, could you assess perplexity at comparable diversity or supplement unigram entropy with sequence-level diversity metrics?

We have not executed the language diffusion experiments at a shared diversity. The difference in diversity, although present, is quite smal (about 2%). Additionally, it is hard to fully match diversity using our sampler as we do not have hyperparameters that directly influence the diversity.

Thanks for the thorough response. The author has addressed most of my questions, but some results are still pending. I understand that time is limited, so would it be possible to receive a preliminary report on the results?

AuthorsOpenReview ↗

Rebuttal by the Authors

The preliminary results on disabling step rejection are now finalized. We have added these to the manuscript as an additional row in Table 3, and discuss these results in Section 6.4: PI Solver Ablations. As reported earlier, with step rejection, the PI solver obtains an FID of 155.48 (compared to 83.95 without step rejection) at 49 NFE and 3.80 (compared to 3.32) at 75 NFE. Now we also report that at 99 NFE, the solver obtains an FID of 2.71 (compared to 3.11 without step rejection). Note that we still count the NFE of a rejected step, as the computation happened even though the step did not get accepted. The experiment shows that step rejection is negative for performance at low NFE, while improving performance at 99 NFE. This can likely be explained by step rejection wasting steps, with an already limited NFE budget. At a lower budget, the solver cannot afford to waste any steps. However, at higher NFE, more steps are available, thus the wasted effort of a rejected step is less relevant compared to the obtained improvement by more accurate sampling.

Reviewer_Qu2GOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

A classical challenge in diffusion sampling is dynamically and adaptively determining step sizes so that difficult sampling regions (i.e., where the next diffusion state is harder to predict) are allocated more NFEs. This paper applies the proportional-integral (PI) control method proposed by Ilie et al. (2015) to adaptive second-order sampling in diffusion models, making the step sizes sample-dependent. Additionally, the authors propose a static second-order PI control method that averages adaptive timesteps across a batch. Experiments demonstrate that the adaptive PI controller yields superior performance on certain tasks, whereas the static PI controller performs better on others.

Critical strengths

The paper, for the first time, adopts the PI stepsize control method from Ilie et al. (2015) for second order sampling in diffusion models, which is new in the field of diffusion sampling. In addition, the authors propose the static PI control method.

Critical weaknesses

(1) As I mentioned earlier, the PI stepsize control method is from Ilie et al. (2015). The authors basically applied it in the field of diffusion sampling. As a result, the contribution is NOT significant.

(2) The experimental results indicate that the adaptive PI controller yields superior performance on certain tasks, whereas the static PI controller performs better on others. So the method is not elegant. Researchers need to test those different schemes to find out which one performs better.

(3) In my opinion, the performance gain for image generation in Table 1 is not significant.

Questions for authors

(1) In Table 1, why does the entropy method not evaluate for ImageNet 64*64? I think it is crucial to include the results.

  1. The authors should evaluate their method on a large-scale diffusion model (e.g., Stable Diffusion) to test its scalability.
2 comments
AuthorsOpenReview ↗

Rebuttal by the Authors

Thank you for your review. We address your questions below.

Q1 In Table 1, why does the entropy method not evaluate for ImageNet 64*64? I think it is crucial to include the results.

We ran the additional entropy baseline on ImageNet-64x64, and report the results in the updated manuscript (Section 6.2).

Q2 The authors should evaluate their method on a large-scale diffusion model (e.g., Stable Diffusion) to test its scalability.

We have not yet systematically run experiments on Stable Diffusion, although informal results on a small set of prompts using classifier-free guidance show that the sampler runs and produces visually reasonable images. However, we lack the quantitative metrics and therefore do not report this in the paper. We leave application to Stable Diffusion to future work, and have addressed this in the limitations of the updated manuscript.

I still think the contribution is not significant and the performance gain is marginal. Also it is not clear if the method is scalable or not. Therefore, I will keep my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.