acceptodds
Under review as a conference paper at ICLR 2027

Same Marginals, Different Errors: Gauge Control for Probability-Flow ODEs

Abstract

Probability-flow ODEs do not uniquely determine how individual samples move. Different velocity fields can generate exactly the same continuous-time marginals. We show that this freedom has a direct numerical consequence: two equivalent velocity fields can incur substantially different discretization errors under the same solver. Based on this observation, we introduce GaugeFix, which modifies only the marginal-preserving component of the velocity field while keeping the probability path, numerical solver, and time grid fixed. For Gaussian probability paths, we characterize exactly when the leading covariance error can be cancelled and quantify the unavoidable residual when complete cancellation is impossible. Beyond Gaussian distributions, we derive exact finite-time Wasserstein errors for rotationally symmetric and multi-block paths, and show that GaugeFix improves the distributional convergence order of an order- Runge–Kutta method to at least , and to for symmetric methods. Experiments confirm these predictions and show that the effect persists with learned scores and frozen pretrained diffusion models. On CIFAR-10, GaugeFix improves eight-NFE Euler FID by while adding no denoiser evaluations and only a few percent sampling overhead.

Primary area:
generative models
Keywords:
Probability-Flow ODEs, Gauge Freedom, Diffusion Models, Numerical Integration, Wasserstein Distance

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tJV6OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The manuscript proposed to use the gauge symmetry in flow-based models to correct leading-order numerical discretization error when deploying a learned model. The authors investigated idealized Gaussian to Gaussian transport models, and one large numerical experiment on CIFAR-10 dataset. Additional details are presented below in Strengths and Weaknesses sections.

Critical strengths
  • [S1] The manuscript proposes an interesting idea of using gauge freedom to compensate higher-order numerical errors.

  • [S2] Using gauge freedom-inspired corrector for CIFAR-10 dataset is an interesting attempt of post-training fine-tuning or distillation for diffusion or flow-based models.

Critical weaknesses
  • [W1. Limited generalizability.] The basic idea of the proposed method is to inject an additional velocity to transport samples to "make up" the leading-order numerical errors. The reviewer finds this idea interesting, but found the manuscript incomplete to recognize the difficulty of such an approach when the transport lacks of a certain invariant structure.

    • For example, suppose the initial distribution is Gaussian but the radial expansion rate depends on the angle, i.e., x˙=c(θ)x\dot{x} = c\left ( \theta \right ) x for x∈R2x \in \mathbb{R}^2. The terminal distribution can be a non-Gaussian one, which is much more relevant setting for generative models. The reviewer has a hard time rationalizing how the proposed method can be achieved globally.
    • The blockwise construction does not solve this problem
    • A relevant problem is if there's any odd transport left in the system, e.g., even for anisotropic Gaussian transport in R3\mathbb{R}^3 with three distinct growth rates, the method cannot be used.
    • The authors also pointed out it's not always possible to use the added field for making correction, even under the extremely restrictive isotropic case: in Proposition 2, "If a(t)<0a(t)<0, no real rotational gauge can cancel the term."
  • [W2. Lack of comparison with existing training or distillation methods.] The reviewer finds the idea interesting but as the manuscript currently stands, it lacks discussions why this method is more advantageous than existing methods for improving inference quality of a trained diffusion model.

    • The most straightforward idea is to use a higher-order ODE solver. Higher order methods were presented in the manuscript, but accuracy v. computational cost tradeoff is not.
    • The underlying problem is some form of curvature or nonlinearity of the ODE solver leads to numerical error. However, built in the diffusion model is the choice of noise schedule, which is tightly connected to a nonlinear transformation of time, in either the typical zero-drift Wiener process (Variance Exploding) or Ornstein-Uhlenbeck process (Variance Preserving); see "Using Ornstein-Uhlenbeck Process to understand Denoising Diffusion Probabilistic Model and its Noise Schedules". For DDPM, the "velocity" is not estimated but instead the conditional mean displacement between discrete times, and a suitable choice of noise scheduling could lead to significantly reduce "discretization error" of the discrete-time ssytem.
    • [W2.3] Using a curved trajectory to "add up" higher order error is very contradictory to existing movements of attempting to learn straight-line flows, such as Rectified Flow. The distinction of the proposed method and the pursue of sraight-line flow models is a deep discrepancy in the modeling philosophy: the proposed model seeks literal accuracy of the numerical methods for integrating a prescribed, but learned, velocity field, at every point of time, while the straight-line flows recognize the there exist multiple intermediate trajectories connecting the initial and terminal distributions. During inference time, accurately simulate a poorly learned, curved (in space and in time) trajectory, may not be as effective as a less accurate simulation of a more straight trajectory. Note that in the idealized where a suitable Monge transport map exists, the endpoint transport can be represented through straight-line displacement interpolation, potentially eliminating trajectory discretization errors.
  • [W3. Disconnection between the idealized Gaussian transport and the CIFAR-10 numerical results.] The reviewer finds a large gap between the theoretical exhibition and the numerical experiments on CIFAR-10.

    • This ties to [W2.3], that the reviewers found the concept of a theoretical transport velocity that satisfies the Liouville equation, that transports the data distribution and the prior distribution, and a learned transport velocity, mingled in this manuscript. The learned velocity does not have to satisfy the boundary conditions but still satisfies the Liouville equation, see for example, Liouville Flow Importance Sampler, in which the gauge freedom and Helmholtz-Hodge decomposition was extensively discussed. Accurately simulating a learned but inaccurate transport velocity does not ensure good sampling property.
    • At a more nuanced level, the gauge freedom can be considered as a vast null space of the Liouville operator, see Vuong et al. 2026. The study established that the learned velocity field may have already had some component in the null kernel space. Using the motivating example of the manuscript, let us consider the true transport as the isotropic and radially expanding Gaussian, let c=1c=1. Let us assume a learned model somehow picked up a spiral signal, which leads to the coupled ODE \dot{y} = y + \varepsilon x$$ or equivalently in a vector form $\dot{\mathbf{x}} = c \mathbf{x} + \varepsilon \mathbf{J}\cdot \mathbf{x}$. Now, applying the proposed method to this *learned* velocity field leads to $c \mathbf{x} + \varepsilon \mathbf{J}\cdot \mathbf{x} + \mathbf{J}\cdot \left[ c \mathbf{x} + \varepsilon J\cdot \mathbf{x}\right]$. While the part $\varepsilon \mathbf{J} \cdot \mathbf{x} + c \mathbf{J}\cdot \mathbf{x}$ lives in the null space, the residual term $\varepsilon J^2 \mathbf{x}=- \varepsilon \mathbf{x}$ does not ensure the marginal distribution. - Noted, in CIFAR-10 the authors operates on the score function, but a similar argument can be established. With this illustrative example in mind, the reviewer has a hard time processing the CIFAR-10 numerical results. In particular, the manuscript does not establish that the proposed correction remains within the Liouville null space when constructed from a learned score, even if the learned score already generates the correct marginal evolution. As such, one cannot rule out the hypothesiis that the reported improvements are due to changes to the underlying continuous-time probability distribution, instead of the cancellation of numerical discretization errors.
    • The reported improvements are significant only for high-FID models; For the much better Heun baseline, FID improves only from 3.050 to 2.996. Moreover, the five-run projected-FID evaluation reported in Appendix E does not establish a statistically significant improvement for Heun, as its 95% interval includes zero. This raises questions about the practical significance of the correction when applied to an already accurate sampler. FID is extremely sensitive to number of samples and training checkpoints: in a sense, an additional "training" was performed on the baseline, pre-trained model, by fitting the correction coefficients. The cost should be considered in assessing the overall accuracy–efficiency tradeoff.
Questions for authors
  • [Q1] Outside the highly symmetric analytic cases, what principle determines the gauge family? In particular, why should the chosen 18-dimensional channel/spatial operator bank contain useful solver-error-reducing directions, and how would the authors construct such a family for a new architecture or data distribution?

    • [Q2] The non-Gaussian order-lifting theorem assumes invariance under the full continuous action eαJe^{\alpha J} for every α\alpha, not merely a discrete rotational symmetry. Can the authors clarify this restriction in the high-level claims and explain whether any analogous result survives under only finite or approximate symmetries?

    • [Q3] Appendix D acknowledged that the learned score correction need not exactly preserve the marginal distributions. Although the authors presented empirical diagnostics, they involve projected features and finite resolutions. How did the authors determined that the observed FID improvement is due to reduced discretization error or a change in the fundamental transport?

    • [Q4] How do the authors view the practical regime in which GaugeFix is preferable to simply using a stronger solver, and is there evidence that the method remains useful once the base discretization is already competitive?

    • [Q5] How were development, validation, and final evaluation separated across the reported experiments? In particular, were the operator bank, schedule, candidate-search procedure, and advancement criteria fixed before inspecting the corresponding evaluation metrics, and how many configurations were considered before the final reported settings were selected?

Reviewer_T5qHOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed GaugeFix, a method for reducing the numerical error of ODE integration that keeps the numerical solver, time grid and probability path unchanged. GaugeFix does this by pairing up marginal preserving directions in the base of the velocity field into two dimensional rotation blocks. For rotation invariant probability paths they show how to construct such a gauge-coupling and show that they can reduce the integration error by an order of one. They show their approach improves the ODE integration for symmetric toy problems and cifar-10 for Euler integration with 8 ODE steps.

Critical strengths

The idea of GaugeFixing by modifying the vector field to couple dimensions is an elegant idea and could be a helpful addition -- especially for the Geometric ML community.

Critical weaknesses

While GaugeFix is a nice idea, but the experiments are not convincing me at all.

  • The higher order convergence breaks down for any shown real setting -- as soon as an actually trained model is used. Scaling of the integration error with step-size is also not shown anymore.
  • The same holds true with \textit{free} having better performance -- according to the presented metrics -- as soon as anything but Euler-8 is used.
  • The presentation of the experiments could be improved a lot. Figure 1 and Fig 2b are the only plots in the main part that are immediately understandable. The others don't really contain more information than a small table would give as well. The ablation study in table 2 is also hardly enlightening. I would be nice to see how the proposed GaugeFix sizes up to any other methods. For me as the reader the paper also has not made a clear cut case why "keeping the [...] numerical solver, and time grid fixed" is a constraint that is necessary/ helpful. I'm sure there are cases why we'd want that, but they are not shown in the paper. To me the idea is very promising, but I would really change the manuscript significantly, in order for it to have a proper impact.
Questions for authors
  • Could you show some scaling of the error with stepsize for the experiments -- especially the Cifar10 ones.
  • Why you did not use a setting with maybe rotation equivariant MLPs for the scores. I can imagine that GaugeFix would be an extremely helpful addition there, but you haven't shown that in the paper.
Reviewer_xDscOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how the choice of velocity field affects the numerical accuracy of probability-flow ODEs. It proposes GaugeFix, which exploits marginal-preserving perturbations to reduce distributional discretization error while keeping the solver and time grid fixed. For Gaussian scaling paths, the paper characterizes when time-independent linear gauges can cancel the leading covariance error and derives residual errors when complete cancellation is impossible. For linear flows with rotationally invariant distributions, it establishes exact finite-time Wasserstein error formulas and distributional convergence improvements of at least one order, or two orders for symmetric RK methods under the stated assumptions. Experiments cover analytic paths, nonlinear toy distributions, and pretrained diffusion models. The strongest reported practical result is a 14.6% FID improvement for eight-NFE Euler sampling on CIFAR-10, without additional denoiser evaluations.

Critical strengths
  1. An interesting perspective on numerical sampling. The paper identifies the choice of a marginal-equivalent velocity field as a way to improve distributional accuracy. The distinction between trajectory error and distributional error is insightful, and the introductory radial-versus-spiral example makes the underlying mechanism easy to understand.
  2. Experiments that address both utility and interpretation. The analytic experiments agree with the predicted cancellation and convergence behavior, while the pretrained CIFAR-10 Euler results demonstrate a practical benefit from a small adapter with no additional denoiser calls. The paper also distinguishes target quality from same-field integration diagnostics and explicitly discusses how approximate scores can break exact marginal preservation. Reporting unsuccessful or inconclusive transfer settings in the appendix adds useful context to the positive results.
Critical weaknesses
  1. While the theory provides useful insights under specific structural assumptions, it is less clear how these insights guide the correction used in pretrained diffusion models. A more intuitive explanation of this transition would help readers understand both the motivation behind the practical design and the extent to which the theory explains the observed improvements.
  2. The strongest improvement is obtained with Euler sampling on CIFAR-10. The gains are smaller or inconclusive for other solvers and transfer settings, and some results show slight degradation. These findings leave it unclear how broadly useful the method is and under what conditions a practitioner should expect an improvement.
Questions for authors

See the Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.