acceptodds
Under review as a conference paper at ICLR 2027

Target Fidelity Is Not Numerical Fidelity Signed Bias-Discretization Alignment at Equal NFE

Abstract

At equal numbers of score evaluations (NFE), Heun can converge faster to a fixed learned probability-flow ODE while Euler remains closer to the target. We prove this separation for a smooth Gaussian VP construction: numerical errors have exact orders one and two, yet Euler has strictly smaller target error at every sufficiently large even budget. The result persists in a one-dimensional neighborhood of smooth nonlinear fields, with a uniform budget threshold. The target-error gap decays as and the relative advantage vanishes. Signed alignment between continuous-flow bias and Euler's leading defect determines the eventual coupled ranking. In DSM-trained GMMs, the joint ranking persists beyond the earliest checkpoint: at 2400 and 6000 updates, respectively, 6/15 and 11/15 fixed networks retain the opposite coupled target ranking throughout a common 128-2048 NFE normal-order diagnostic window, while Heun has smaller numerical error in every comparison. Oracle-assisted alignment estimates do not outperform simple target-gap extrapolation. A controlled 32-dimensional study provides additional coupled evidence; An unpaired kernel-embedding test finds no statistically certified MMD reversal. These results establish a locally robust distributional existence theorem and a controlled learned-score mechanism; Image-scale relevance remains open.

Primary area:
generative models
Keywords:
Score-based diffusion models, probability-flow ODEs, numerical discretization, score approximation error, error cancellation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_UF3hOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates if higher numerical accuracy necessarily leads to better target fidelity in diffusion sampling. The authors show that Euler can achieve better target fidelity than Heun despite having larger numerical discretization error. They establish theoretical results for Gaussian VP models and conduct experiments on synthetic Gaussian mixtures, including learned-score experiments in 2D and 32D.

Critical strengths
  1. The paper studies an interesting distinction between numerical accuracy and sample quality.
  2. The experiments consider different training checkpoints and NFE budgets and illustrate the behavior under different settings. The visualization in figure 1 is nice and clear.
Critical weaknesses
  1. The experiments are conducted on synthetic Gaussian mixture models, but the behavior remains unclear on realistic diffusion models and image generation.
Questions for authors

Following weakness.

  1. I am curious about how this insight could be applied to practical diffusion models. In particular, since the experiments use synthetic distributions where the ground-truth score is analytically available, how would a similar experiment be designed for real-world datasets where the true score is unknown? How could the target fidelity be evaluated in such settings?
  2. Regarding experiment section 5.5, given that the unpaired MMD evaluation does not show statistically significant reversals, how might this insight translate into improved generation quality or inform sampler selection in real-world diffusion models?
Reviewer_n8JUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how the numerical accuracy of diffusion solver schemes affects sampling accuracy when the score is imperfect. Focusing on a highly simplified scenario, it shows that Euler outperforms Heun given the same NFEs despite Heun’s higher numerical accuracy. Synthetic experiments are conducted to support the theoretical results.

Critical strengths

The paper studies how the choice of the discretization scheme affects sampling quality, which is an important question of broad interest to the diffusion modeling community.

Critical weaknesses
  1. The main theoretical result is restricted to a highly simplified one-dimensional SDE with a constant drift β\beta. More critically, it relies on a specially constructed score error (Eq. 6), and the nonlinear extension remains artificial. The paper does not establish why errors produced by realistic score learning should satisfy these conditions. Consequently, the practical implications of the theoretical results remain unclear.
  2. The paper is extremely poorly written and difficult to follow. For example,
  • Key quantities are not always clearly defined when first introduced. For instance, what is RMS in Eq. (2)?
  • The update rules of Euler and Heun are not presented, making it inaccessible to readers.
  • The sections are organized confusingly. For example, it appears that Sections 5.1–5.2 discuss extensions and checkpoint comparisons before Section 5.3 introduces the base training setup.
  1. Experiments on synthetic data again fail to provide practical implications of the results.
Questions for authors
  1. Can the theory be extended to cover the errors produced by score learning in practice?
  2. Does the main finding hold on real-world datasets?
Reviewer_CbRLOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how numerical integration error interacts with error in a learned score function when sampling via a probability-flow ODE. The authors construct a one-dimensional example in which, at the same number of function evaluations, Heun’s method solves the approximate ODE more accurately, while Euler’s method yields a smaller 2-Wasserstein distance to the target distribution. They prove that this result is locally robust to smooth nonlinear perturbations and examine the corresponding mechanism in experiments with Gaussian mixtures.

Critical strengths

I believe this is a potentially interesting idea that could yield practical recommendations for selecting a solver.

Critical weaknesses

(W1) In my view, the practical significance of this particular counterexample in Theorem 1 is unclear. The authors also admit that the results do not constitute a practical solver selector (see l.434)

(W2) Theorem 1 compares the output distributions using the 2-Wasserstein distance to the target distribution. In the experiments with trained score networks, the main target-error measure is the root mean square distance between endpoints obtained from the same initial random variable. This is the cost of a particular coupling, which has not been shown to be optimal in these experiments. Therefore, as far as I understand, the observed ordering of these errors does not establish an analogous ordering under the 2-Wasserstein distance for trained networks. Consequently, the experimental evidence that the theorem's distributional reversal occurs with learned score functions remains limited.

Questions for authors

(Q1) Is there a condition on the error in a learned score function under which one can expect a reversed ranking with a non-negligible target-error gap at finite NFE? If not, what conclusion about sampling with learned models follows from Theorem 1 beyond the existence of a specifically constructed counterexample?

(Q2) Can the authors demonstrate, for at least one trained network, a reversed ranking in W2W_2 distance to the target distribution while accounting for uncertainty in the distributional estimates?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.