Same Marginals, Different Errors: Gauge Control for Probability-Flow ODEs

Probability-flow ODEs do not uniquely determine how individual samples move. Different velocity fields can generate exactly the same continuous-time marginals. We show that this freedom has a direct numerical consequence: two equivalent velocity fields can incur substantially different discretization errors under the same solver. Based on this observation, we introduce GaugeFix, which modifies only the marginal-preserving component of the velocity field while keeping the probability path, numerical solver, and time grid fixed. For Gaussian probability paths, we characterize exactly when the leading covariance error can be cancelled and quantify the unavoidable residual when complete cancellation is impossible. Beyond Gaussian distributions, we derive exact finite-time Wasserstein errors for rotationally symmetric and multi-block paths, and show that GaugeFix improves the distributional convergence order of an order-$r$ Runge–Kutta method to at least $r+1$, and to $r+2$ for symmetric methods. Experiments confirm these predictions and show that the effect persists with learned scores and frozen pretrained diffusion models. On CIFAR-10, GaugeFix improves eight-NFE Euler FID by $14.6\%$ while adding no denoiser evaluations and only a few percent sampling overhead.