acceptodds
Under review as a conference paper at ICLR 2027

Beyond the PF-ODE: Learn the Reverse Dynamics for Few-Step Diffusion Sampling

Abstract

Diffusion solvers accelerate sampling by numerically approximating the reverse process with only a few function evaluations. This reverse process is commonly formulated as either a deterministic probability-flow ODE (PF-ODE) or a stochastic reverse SDE. Reverse SDEs can dissipate distribution mismatch accumulated during sampling, but stronger stochasticity also amplifies the effect of score error. PF-ODEs avoid this additional stochasticity, yet preserve inherited distribution mismatch under exact scores. We show that this creates a step-dependent trade-off between distribution correction and score reliability, suggesting that the preferable reverse dynamics need not remain fixed throughout sampling. Motivated by this observation, we propose RD-Solver, which jointly learns the reverse dynamics and their discretization. Specifically, RD-Solver learns an interval-wise stochasticity coefficient together with multistep predictor–corrector weights and calibrated score-evaluation times. We optimize these parameters with a frozen network using an endpoint loss that upper-bounds the squared 2-Wasserstein distance to a high-accuracy teacher in the supervision space. Experiments across diffusion and flow models demonstrate improved few-step generation, achieving FID scores of 2.71 on CIFAR-10 and 4.62 on ImageNet-64 with five function evaluations (5 NFEs).

Primary area:
generative models
Keywords:
Generative Models, Fast Sampling, Diffusion Solver

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_JUPAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates and proposes a new method for diffusion sampling, generalizing the existing stochastic (SDE) and deterministic (PF-ODE) samplers. Their method jointly learns the drift and noise variance parameters that parameterize the solver. The solver is tested on several benchmarks and outperforms existing, fixed and learnable, approaches in the majority of settings.

Critical strengths
  1. Impact
  • The proposed method shows strong performance on several benchmarks, indicating a clear advancement over existing approaches.
  1. Clarity of writing
  • Outside of the complexity of the content, the paper is clearly presented, with very helpful use of stylization and well-executed figures and tables.
Critical weaknesses

None, but there are minor weaknesses detailed in the questions.

Questions for authors

Clarifications:

  • Section 4 and Figure 2 nicely illustrate the role of γ\gamma. I would be interested to see what values of γ\gamma you observed in the runs reported in Section 5. Of particular interest would be whether γ\gamma follows the trade-off that the theory predicts, i.e., more stochasticity where distribution mismatch is large and the score is reliable.
  • Class conditioning dramatically improves generative performance for diffusion models and potentially affects different sampling methods differently. You only report 2 class-conditional benchmarks, even though CIFAR-10 and ImageNet64 also offer labels. Do you have additional results on those?
  • FID can have significant run-to-run variation, which is especially relevant for small FID values (<3) and allows for seed selection. It appears that all FID numbers are individual runs, as opposed to, e.g., averages of 3. Please clarify how you handled seed noise.

Future work:

  • The pixel models use an Inception loss and are then evaluated with FID, in the same feature space used for the loss. This is a problematic confound that distorts the results. Do the results change when decoupling these, e.g., using FD_DINOv2?
  • FID and FD_DINOv2 often yield qualitatively different results, even without the aforementioned confounding, and FID has several known problems [1]. Showing that the RD-Solver beats existing approaches on most benchmarks with FDD as well would improve the evaluation.

[1] Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models, Stein et al., 2023

1 comment
Reviewer_JUPAOpenReview ↗

Reply to rebuttal

Thank you for the detailed and thorough response. The clarifications on the learned stochasticity coefficients, the class-conditional setting, and the seed variance fully answer my questions. I think the new analyses in Appendices C.3–C.5 improve the paper.

My one remaining suggestion concerns the evaluation metric. The FD-DINOv2 results in the rebuttal are, in my view, a very valuable addition. I would encourage the authors to report FD-DINOv2 in as many places in the paper as possible, alongside FID. FID has well-documented limitations [Stein et al., 2023], and FD-DINOv2 tracks perceptual quality more closely. I believe it should be the standard metric for evaluating generative performance and that FID is still around mostly for historical reasons. This is of course especially helpful when feature spaces are used for a loss. On CIFAR-10, the improvement over iPNDM was inflated under FID, as they are noticeably smaller under FD-DINOv2. The gains remain clear, but reporting both metrics gives readers a more faithful picture of the method’s advantage. It would also make the paper a stronger reference for future work.

If not already planned, I would add the remaining rebuttal results, namely the class-conditional CIFAR-10 results, the seed study, and the DINOv2-loss ablation, to the appendix.

Overall, I remain positive about this submission and maintain my score.

Reviewer_ZJ2sOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RD-Solver, a learned few-step sampler that interpolates between the PF-ODE and the reverse SDE. It learns a per-interval stochasticity coefficient γi\gamma_i, multistep predictor–corrector weights and calibrated evaluation times. The method is motivated by a KL analysis. The parameters are trained with an loss against a teacher, which upper-bounds W22W_2^2. It reports state-of-the-art FID across pixel, latent, and text-to-image models and datasets.

Critical strengths
  • The method is well designed and motivated by the KL decomposition.
  • The writing is careful about its own limits
  • The experiments are broad with both pixel and latent space ablations
  • Training is very cheap compated to baselines.
Critical weaknesses
  1. The loss is a regression to a deterministic target. So it may directly penalize the variance γ\gamma injects. The objective should push γ\gamma toward zero unless noise helps the conditional mean (Eq. A118). γi\gamma_i are initialized at 1e−41e-4 but the final values are never reported. Have you looked at the final γi\gamma_i values? How much they differ from each other? Do they stay tiny?

  2. When γi=0\gamma_i = 0, both calibrated times reduce to ti+1t_{i+1}, so this ablation also disables calibration. The gain attributed to learning the dynamics (6.87 vs. 17.63 at 3 NFEs) is therefore entangled with calibration and κi\kappa_i.

  3. Pixel-space models are trained with an Inception-feature loss and evaluated with FID, also using the same Inception features. Part of the gain may come from optimizing directly in the metric space. Why not using a pixel-space loss (or a metric outside Inception space)? How does it perform?

  4. The CLIP scores reported in Table 2 seems particularly high (especially at 5 and 7 NFEs). What would explain such a big improvement? Those scores are not compared to the teacher CLIP scores (nor FID)

Questions for authors
  1. What are the final learned γ_i values? How much do they vary across intervals and NFE budgets? Do they remain close to zero?

  2. Have you tried a pixel-space loss for the pixel models? How does it compare with the Inception loss?

  3. In Table 2, the FID gap with baselines shrinks as NFE increases, while the CLIP gap grows (+1.6 at 3 NFEs, +4.1 at 7 NFEs). How do you explain this? Did you notice differences in the generated images? What is the teacher’s CLIP score?

Reviewer_sTFKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents RD-Solver, a learning-based diffusion solver that jointly optimizes reverse dynamics and numerical discretization for few-step generation. To this end, the authors first analyze the trade-off between distribution mismatch correction and score estimation error, motivating the use of learnable stochasticity across sampling intervals. Specifically, RD-Solver combines interval-wise stochasticity with learnable multistep predictor–corrector weights and score-evaluation time calibration. Moreover. with the pretrained model kept frozen, these parameters are optimized using an endpoint matching objective that upper-bounds the squared 2-Wasserstein distance to a high-accuracy teacher distribution in the supervision space. Importantly, experiments on CIFAR-10, ImageNet-64, and text-to-image generation with Stable Diffusion and FLUX demonstrate improved generation quality over existing solvers, particularly at low NFEs.

Critical strengths

The paper puts forward an interesting algorithm to few-step diffusion sampling by learning the reverse dynamics together with their numerical discretization, rather than relying on a fixed PF-ODE. To be specific, the analysis of distribution mismatch and score estimation error provides a useful motivation for introducing adaptive stochasticity during sampling. Another strength is the extensive experimental evaluation across multiple datasets, pretrained models, and sampling budgets, where RD-Solver achieves strong improvements over existing methods, especially at low NFEs. The ablation studies also demonstrate the effectiveness of the main components, including learnable stochasticity, time calibration, and predictor–corrector updates, while keeping the pretrained model frozen.

Critical weaknesses
  1. I am not fully convinced that the theoretical analysis explains the improvements observed in few-step sampling. The KL dissipation analysis is derived in continuous time, while the main experiments focus on coarse discretization with only 3–5 NFEs. As the authors acknowledge, these theoretical results do not guarantee error reduction in such settings. Although Table 3 shows the benefits of learnable stochasticity, it remains unclear whether the improvements come from adapting stochasticity across intervals or simply introducing stochasticity into the solver. Comparisons with fixed nonzero stochasticity coefficients would help clarify this distinction. I would also like to see the learned coefficient schedules and some evidence that they actually reduce distribution mismatch during sampling.

  2. I also have concerns about the endpoint matching objective. Although the proposed loss provides an upper bound on the Wasserstein distance, this bound may be loose, and reducing the paired endpoint error does not necessarily imply better distribution matching. In particular, the teacher is deterministic, whereas the student introduces additional noise. The regression loss therefore penalizes stochastic variations, even though such variations may help correct distribution mismatch. This deserves further explanation, given that learning stochasticity is one of the main contributions. Regarding the experiments, most benchmarks are evaluated at 3–9 NFEs, whereas the Stable Diffusion experiments on MS-COCO only consider 8–20 NFEs. Although this difference partly comes from counting both conditional and unconditional evaluations, it makes it difficult to assess the method's performance under very limited sampling budgets. I suggest that the authors include additional results at lower NFEs, such as 4–6, to provide a more complete evaluation.

Questions for authors
  1. Could the authors compare RD-Solver with fixed nonzero stochasticity coefficients to clarify whether the improvements come from learning interval-wise stochasticity rather than simply introducing stochasticity?

  2. The KL dissipation analysis is based on continuous-time dynamics. Could the authors provide empirical evidence showing that the learned reverse dynamics actually reduce distribution mismatch under extremely limited sampling budgets, such as 3–5 NFEs?

  3. Since the teacher is deterministic while the student introduces stochastic noise, could the authors explain how the endpoint matching loss encourages distribution matching without unnecessarily penalizing stochastic variations?

  4. While the paper evaluates RD-Solver at 3–9 NFEs on several benchmarks, the Stable Diffusion experiments on MS-COCO only consider 8–20 NFEs. Could the authors explain this choice and provide results at lower sampling budgets, such as 4–6 NFEs?

  5. Since RD-Solver is a learning-based method, could the authors provide a more detailed comparison of training costs, including total FLOPs and wall-clock time, with other learned solvers such as AMED and LD3 under comparable hardware settings? It would also be helpful to clarify whether the reported costs include teacher sample generation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.