acceptodds
Withdrawn from ICLR 2027

Training as a flow of distributions

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Analyses of neural-network training usually examine one aspect at a time, the dynamics of the optimizer, the energy of the loss landscape, or the geometry of the underlying space (representational, functional, or parameter), each in isolation. To unify all in a single apparatus, we use the Wasserstein gradient flow, the steepest descent of a free energy in Wasserstein geometry, which coincides with the Fokker-Planck equation and is realized in discrete time by the Jordan-Kinderlehrer-Otto (JKO) proximal scheme. This scheme unifies the energy, geometry, and learning dynamics in one object, making their interaction measurable. The measurements that this framework relies on and employs are high-dimensional and lack a ground truth to validate them, and they might deviate from the correct measurement without any indication of failure. We therefore validate each measurement on small, analytically tractable models, in which the answer is known in closed form. Employing generative classifiers versus discriminative models, and focusing on measurements introduced and known in the related literature, such as calibration, we compare these models and validate these results: (i) a predictor's calibration is controlled by the transport distance of its parameter distribution to the posterior, tracked along the flow down to a finite-sample floor; (ii) the geometry of the landscape sets the rate at which calibration is reached, governed by the slowest mode, (iii) the crossover of discriminate over generative classifiers occurs as a result of misspecification and in the case of the correct assumption, the crossover does not happen.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
JKO scheme, Fokker-Planck equation, free energy, flow of distributions

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_o3tCOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies training as a Wasserstein gradient flow in a controlled, analytically tractable setting (Gaussian likelihood + Gaussian prior), where the posterior is available in closed form. The authors build a measurement pipeline (energy, geometry, dynamics) and validate it against exact references. They report three main empirical findings: (1) calibration error is bounded by and tracks the transport distance to the posterior; (2) the convergence rate of this calibration is set by the smallest Hessian eigenvalue; and (3) the generative-vs-discriminative crossover occurs only under misspecification. The paper positions itself as a foundational first step before scaling to real networks.

Critical strengths

The controlled setting is a sensible choice for validating measurement tools, and the use of closed-form posteriors as ground truth is methodologically clean.

Critical weaknesses

Line 032: "Modern studies of training artificial systems tend to look at one thing at a time." This is not correct. Modern DNNs can be trained with multiple combined losses. Line 037: " These are usually studied with separate tools, and their interactions, where the interactions of energy, geometry, and dynamics have not been addressed." This is not correct either.

The theories are all well-established. The paper largely empirically confirms known results in a Gaussian toy setting.

All results are in the Gaussian/quadratic regime.

The paper's claimed contributions are overstated, and the motivation is not clear.

Questions for authors

Fig. 1: Is this landscape a real landscape of a network, or from toy data, or AI-generated?

1 comment

We thank the reviewer for their time and for acknowledging that using closed-form posteriors in a controlled setting is a methodologically clean way to validate measurement tools.

On Motivation and "One Aspect at a Time" (Lines 032–037): The main goal of our research program is to study learning dynamics through the simultaneous interaction of dynamics, geometry, and energy—for example, to ask how landscape geometry and free-energy dissipation interact during early memorization or generalization loss. The Line 032 "Modern studies of training artificial systems tend to look at one thing at a time," points to this general aim of the whole project and the lack of such a coherent framework of study. What we meant was that the energy (loss surface), the dynamics (optimizer trajectory), and the geometry (metric/preconditioner) are typically analyzed with separate theoretical tools, whereas the Wasserstein gradient flow / JKO framework couples all three into a single variational object. Though it is true that different studies have examined training modern DNNs with multiple loss terms, it is not completely compatible with the main goal of our study. Similarly, our restriction to the Gaussian/quadratic regime in this paper is an intentional first step to verify the measurements against exact analytical ground truth before moving to non-convex networks where no ground truth exists.

On the Ethics Flag and AI Use Statement: We strongly but respectfully reject the characterization of our work as "unprepared AI slop."

  1. Originality and Transparent Disclosure: All scientific ideas, study designs, code implementations, and experiments were conducted by the authors. In full compliance with ICLR policy, we explicitly stated in our AI Use Statement (Lines 481–485) that generative AI was used for polishing text and for rendering the conceptual schematic in Figure 1 based on a human-drawn draft by the authors. This directly answers the reviewer's question regarding Figure 1. In the spirit of full transparency, we are also happy to broaden our AI Use Statement to note that LLM tools were used as a conversational sounding board during early project brainstorming, in addition to language polishing and rendering Fig. 1.
  2. Specificity of the Critique: In the Ethics Flag details, the reviewer states that "The theorem is unnecessarily too technical without providing any motivation and proper notation." We respectfully note that there is no theorem in our paper, so we are unsure which part of the manuscript this comment refers to. Where the manuscript did fall short—and for which we take full human responsibility—was in late-stage editing oversights (such as leaving references to our next-stage Sinkhorn/kNN estimators in the Introduction and leaving LaTeX template instructions on page 10). While we accept these unintentional editing mistakes and are apologetic about them, the research and experiments are genuine, author-developed work.
Reviewer_ECPgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper uses Wasserstein gradient flows to study how distributions over model parameters evolve during training. In tractable Gaussian settings, it uses a Gaussian JKO scheme and ensemble Langevin sampling to examine transport distance, free energy, calibration, and convergence rates. It also compares generative and discriminative classifiers under correct specification and misspecification. The proposed contribution is a framework for checking these measurements against known references before applying them to more complex models.

Critical strengths

Using models with known posteriors is a sensible way to check measurements that may become unreliable in more complex settings. I also find the exact-posterior comparison useful: it helps distinguish the effect of posterior averaging from any benefit specific to the flow. Including MAP and temperature-scaled logistic regression makes the predictive comparison more informative. The paper appropriately limits its conclusions to the controlled setting.

Critical weaknesses
  1. The calibration claim needs a precise statement. Appendix A.1 calls the posterior predictive calibrated by construction, without specifying the probability law. At ρ=π\rho=\pi, the bound ECE(ρ)≤cW2(ρ,π)\mathrm{ECE}(\rho)\leq cW_2(\rho,\pi) requires zero calibration error, which is not automatic for a finite-data Bayesian predictor evaluated under fixed generating parameters. For the stated generative model, averaging p(y∣x,θ)p(y\mid x,\theta) under p(θ∣D)p(\theta\mid D) also generally differs from conditioning the posterior predictive joint distribution on xx. The ensemble is a valid predictor, but its calibration needs justification. The assertion that ECE is Lipschitz needs assumptions for the binned statistic used here; fitting cc to the observations in Figure 2 does not establish a guarantee. Nor can the empirical ECE plateau be attributed solely to ensemble size without assessing test-sample estimation error, a known issue for binned ECE (Roelofs et al., 2022). Test-set sizes for Figures 2 and 6 are not reported. Appendix A.3 also calls the advantage over temperature scaling statistically resolved, but gives a paired confidence interval only for flow versus exact draws. That contrast does not establish the claimed advantage over the tuned baseline. The temperature-scaled ECE is also reported as 0.030 on p. 15 and 0.026 on p. 16; these values need to be reconciled.

  2. Several validations advertised as contributions are not reported. The contribution statement on p. 3, lines 127–129 claims non-Gaussian transport tests and trajectory-level agreement between JKO and the explicit sampler; I could not find either. Nor are the Sinkhorn and nearest-neighbor entropy estimators discussed in the introduction checked against known references. Gaussian moment estimates are reasonable here, but their accuracy should be quantified. Figure 3 shows fitted rates of approximately 25, 10, and 6 against λmin⁡\lambda_{\min} values of 20, 8, and 5, while describing λmin⁡\lambda_{\min} as the late-time rate. The exact Gaussian trajectory depends on the initial mean and covariance, so this discrepancy need not indicate an implementation error. Without the initialization, exact fitting window, and treatment of the sampling floor, however, the comparison is difficult to interpret. Fixed-step Langevin bias also needs to be distinguished from finite-ensemble error (Wibisono, 2018, Example 2). The exact-posterior control and Figure 4’s controlled illustration of misspecification are useful. However, the paper’s main proposed advance is a validated measurement framework, and these comparisons do not supply the missing estimator and trajectory checks. Together with the omitted JKO covariance update, this leaves that contribution insufficiently supported.

Questions for authors
  1. Is the calibration claim intended under a Bayesian joint predictive law or under the fixed data-generating parameters used in each experiment? How does that choice justify calibration of the particular ensemble predictor in Appendix A.3?

  2. What test-set sizes were used for Figures 2 and 6, and how was test-sample estimation error assessed when interpreting the ECE plateau? Which temperature-scaled ECE value is intended (0.030 on p. 15 or 0.026 on p. 16), and what paired confidence interval supports the claimed advantage in Appendix A.3?

  3. Are the non-Gaussian transport tests and direct JKO-versus-explicit trajectory comparisons already available? Their numerical errors and implementation settings would help determine whether the claimed validation is complete.

  4. For Figure 3, what initialization, fitting window, and treatment of the sampling floor were used? Is an exact Gaussian-flow reference available for the same initialization, to clarify whether the fitted rate ratio of about 1.2 reflects transient dynamics or the numerical approximation?

  • Rebecca Roelofs, Nicholas Cain, Jonathon Shlens, and Michael C. Mozer. Mitigating Bias in Calibration Error Estimation. AISTATS, 2022.

  • Andre Wibisono. Sampling as optimization in the space of measures: The Langevin dynamics as a composite optimization problem. COLT, 2018.

1 comment

We sincerely thank the reviewer for their careful reading and constructive, precise technical questions.

  1. The data is generated under fixed generating parameters for the mentioned experiment. What makes the ensemble calculation different is the choice of the predictors. The main point of this figure is to show the essential difference between the posterior-based computation, which is driven by different processes, and the discriminative logistic method. The generated data for all ensembles are equal, and the differences come from the computation of the posterior, where the flow is based on the JKO computation; exact draw is the closed form of the posterior based on the Gaussian distribution of the data and particles, and the MAP point keeps only the single most probable parameter, the posterior mode, which, for a Gaussian, equals the mean μ∗\mu^*.

    We agree with the reviewer's theoretical point regarding Appendix A.1: under fixed data-generating parameters and a finite training set, the Bayesian posterior predictive does not have zero calibration error. Thus, by the triangle inequality and Lipschitz continuity of the predictive map, the formal bound at finite sample size is ECE(ρ)≤ECE(π)+c W2(ρ,π)ECE(\rho)\leq ECE(\pi)+c W_2(\rho,\pi).

  2. The test sample for both Fig. 2 and Fig. 6 is fixed at 500 points per class (1500 total). The correct temperature-scaled ECE value is 0.030; thanks for the careful reading and pointing this out. The inconsistency is due to the different random seed and multiple tests, and the value in the paragraph "The temperature-scaled baseline" was not updated. The confidence interval for Appendix A.3 is set to [−0.001,+0.002][−0.001, +0.002].

  3. All transport distances are computed based on the closed-form Gaussian and the Sinkhorn, and non-Gaussian benchmarks have been wrongly omitted from the manuscript and should not have been listed as completed contributions in this paper.

  4. The time window tt is fixed at 1 for the upper bound. The time step is considered as η∝1λmax(A)\eta\propto\frac{1}{\lambda_{max}(A)} and for kk iteration t=kηt=k\eta. The iterations kk for each geometry differ and are computed as below:

    • (20,20,20)→λmax≈20.1,η≈2.5⋅10−3,k≈402(20,20,20) \rightarrow \lambda_{max} \approx 20.1, \eta \approx 2.5·10⁻³, k \approx 402 steps
    • (8,20,60)→λmax≈60.1,η≈8.3⋅10−4,k≈1203(8,20,60) \rightarrow \lambda_{max} ≈ 60.1, \eta \approx 8.3·10⁻⁴, k \approx 1203 steps
    • (5,15,120)→λmax≈120.1,η≈4.2⋅10−4,k≈2403(5,15,120)\rightarrow \lambda_{max} ≈ 120.1, \eta \approx 4.2·10⁻⁴, k \approx 2403 steps

    The particles are sampled from N(0,4I)N(0, 4I) and the start is the same for all three. What varies is the geometry of the landscape, by changing the Hessian of the potential VV. The exact Gaussian- flow reference for the same initialization is computed via an implicit engine, evolving (mk,Σk)(m_k, \Sigma_k) by a closed-form update. The rate is approximately equal to the diffused particles; the exact rate =1.3=1.3.

Reviewer_8jaTOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

See Clarity issues in Critical weaknesses.

Critical strengths

See Clarity issues in Critical weaknesses.

Critical weaknesses

Clarity

This paper is tedious to read and very obscure on many aspects. So far, I understand that it attempts to unify the "energy", the "geometry", and the "dynamic" of the training of a model through the flow of distributions.

A first major issue is the clarity of the motivations and contributions. The introduction goes too much into detail before the reader even understand in which setup we are. For instance: "distributional (generative) formulation is more sample-effective [...] distributional form of the model." I must say: I do not understand clearly any statement in this sentence. And I do not understand why it is relevant here.

A second major issue is the correctness of the in-text claims. For instance, in the introduction: "The obstacles and limitations do not come from theory, rather it is mostly related to the available instruments." Such very bold claims must be avoided when there is no unified "deep learning theory" with a broad consensus about the approach. Besides, if a theory works, but is (almost) never applicable, then the "obstacles and limitations" come surely from the theory itself.

A third major issue is the exaggerated complexity of the mathematical objects. The paper uses heavy machinery for a simple model. Wasserstein gradient flows, JKO, Fokker–Planck and log-Sobolev constants are used to study an Ornstein–Uhlenbeck process with a quadratic potential.

To summarize these points, the main issues are:

  • unnecessary technical details and jargon, especially in the Abstract and the Introduction;
  • too many vague terms;
  • bold unsupported claims.
Questions for authors

None

1 comment

We thank the reviewer for their time and feedback.

While we recognize that condensing the mathematical apparatus into this baseline study made parts of the presentation dense, our primary goal, as outlined in the Abstract and Discussion, is to build a unified, explainable measurement framework for learning in artificial systems across three components: energy, dynamics, and geometry.

In statistical physics, the Fokker–Planck equation connects these three components to describe the evolution of a stochastic process toward its stationary distribution. In mathematics, the Jordan–Kinderlehrer–Otto (JKO) proximal scheme formulates this evolution in discrete time as the steepest descent (gradient flow) of a free-energy functional in Wasserstein-2 space. Because these mathematical foundations are well-proven in the continuous/population setting, our statement in the Introduction, "The obstacles and limitations do not come from theory; rather it is mostly related to the available instruments", refers specifically to the challenge of empirical measurement: in high-dimensional settings, exact W2W_2 distances and differential entropies are computationally intractable and must be replaced by finite-sample estimators that can degrade silently without ground truth.

Therefore, before applying approximate estimators to complex deep networks, our goal in this first study was to establish a verified baseline in a controlled regime where every quantity is known in closed form. Even though in this simplified Gaussian setting concepts such as the log-Sobolev constant reduce to the smallest eigenvalue λmin(A)\lambda_{min}(A) of the quadratic potential, introducing the full Wasserstein gradient flow apparatus is essential because it defines the exact quantities and scaling laws we aim to track in the subsequent phases of this research program.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.