acceptodds
Under review as a conference paper at ICLR 2027

Learning Invariant Measures with Physics Supervision

Abstract

Many fluid and plasma applications require samples of statistically stationary turbulence to estimate mean observables, fluctuations, and spatial correlations. However, obtaining these samples typically involves long-time evolution, either during sampling or when constructing training datasets for generative models. We introduce the Invariant Measure Generator (IMG), which learns a direct sampler of long-time stationary statistics using supervision from short physical evolutions. Its self-consistent conditional flow-matching objective uses evolved model samples as targets, providing tractable supervision for high-dimensional fields without requiring stationary training samples. Our analysis connects the self-consistent objective to distributional invariance. Experiments on Lorenz-63, the Kuramoto-Sivashinsky equation, and the Hasegawa-Wakatani (HW) system demonstrate accurate stationary statistics, including spectra and spatial increments in two-dimensional turbulence. In limited-data settings, the same objective improves data-trained samplers. Using an existing frozen neural operator to construct its targets improves probability density function (PDF), spectral, and increment statistics on HW without additional stationary training samples or numerical-solver calls during sampler training.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Invariant Measure, PDE Simulation, Generative Model, Neural PDE Solver

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZC88OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes the Invariant Measure Generator (IMG), a flow-matching model that learns to sample the long-time stationary distribution (invariant measure) of chaotic/turbulent systems without stationary training data. At each update, the current model generates samples, these are evolved by the physical solver for a random short time δt ∈ (0, ΔT), and the evolved states are used as fixed targets for conditional flow matching. A theorem bounds W2(q_θ, P_t q_θ) by the flow-matching loss for fixed targets. In a few-data setting, the same loss is added to standard data-trained CFM, optionally using a frozen neural operator instead of the solver (IMG-neural). Experiments on Lorenz–63, Kuramoto–Sivashinsky (KS), and 2D Hasegawa–Wakatani (HW, 131k dimensions) show IMG outperforming physics-supervised baselines (FNO, U-Net, PI-DeepONet, WGS) on PDF, spectrum, and increment statistics, and improving data-trained CFM in the few-data setting.

Critical strengths
  1. Simple, scalable idea. Training a generator on its own short-time evolved samples avoids estimating high-dimensional distribution distances and works at 131k dimensions, far beyond prior physics-supervised samplers (~100 dimensions). This is the main reason the paper is of interest.
  2. Thorough and honest evaluation. The paper uses multiple physically meaningful metrics (spectra, increments, flatness), tests stability under further evolution, and openly reports limitations (drift under evolution, weak discriminative power of SW2, no demonstrated speedup over DNS). The KS few-data gains also hold across 3 training seeds.
Critical weaknesses
  1. "Short-time supervision" hides long total evolution, and cost is not competitive. IMG is essentially a fixed-point iteration q ← P_δt q; over 20,000 updates the model population is effectively evolved for thousands of physical time units, far more than the DNS burn-in (t = 400). The full HW run takes 52.7 GPU hours, while DNS burn-in takes about 212 s and 41 fields about 423 s. The paper itself states that no speedup over DNS is established, so the main motivation (avoiding long-time evolution) is not supported in practice.
  2. "Data-free" is overstated. DNS validation trajectories are used for checkpoint selection and hyperparameter search (including ΔT and λ), so stationary DNS data still shapes the final model.
  3. Weak theory. Theorem 1 holds only for frozen targets; the gradient identity is the standard CFM result, and convergence of the actual iterative procedure is not analyzed. The constant C_t grows exponentially with t for chaotic systems (κ > 0), so the bound is nearly vacuous at long horizons. Lipschitz constants are assumed uniform in θ, which is unrealistic. Measure selection is unresolved: trivial invariant measures (e.g., the zero field for KS/HW) also satisfy self-consistency, and avoiding collapse is shown only empirically.
  4. Learned samples are not actually invariant. Under further HW evolution, IMG's spectrum, generalized enstrophy, and particle flux drift noticeably more than the DNS control (Tables 14 and 16), which directly weakens the claim of learning the invariant measure.
  5. Missing or weak baselines. Several physics-supervised baselines (PI-DeepONet, WGS) produce near-zero-amplitude fields (Fig. 12), so beating them says little. The most natural baseline, plain DNS or an evolved ensemble at the same compute budget, is missing; the particle-pool variant (essentially ensemble DNS) appears only as an ablation. In the few-data setting, data CFM is given only 15 KS snapshots or 10 short HW windows; a fair comparison would give it as much DNS data as IMG's training compute could produce.
  6. Limited statistical evidence. Main tables use one training seed, and error bars reflect sampling only. HW evaluation uses only 64 fields per repeat, and the authors show SW2 cannot distinguish the stronger models from DNS-to-DNS noise. Lorenz–63 uses a single run, and IMG's density is visibly more diffuse than DNS.
Questions for authors

N/A

6 comments
AuthorsOpenReview ↗

Official Response to Reviewer ZC88 (1/5)

We thank the reviewer for the detailed and careful comments. We are encouraged that the reviewer finds our method simple and scalable, and the evaluation is thorough. Below, we address the concerns of the reviewer point-by-point.

Weakness 1: "Short-time supervision" hides long total evolution, and cost is not competitive. IMG is essentially a fixed-point iteration q ← P_δt q; over 20,000 updates the model population is effectively evolved for thousands of physical time units, far more than the DNS burn-in (t = 400). The full HW run takes 52.7 GPU hours, while DNS burn-in takes about 212 s and 41 fields about 423 s. The paper itself states that no speedup over DNS is established, so the main motivation (avoiding long-time evolution) is not supported in practice.

We thank the reviewer for the comment. We agree that the training of IMG reported in the main table is expensive, involving many solver calls which surpasses the cost of a DNS run.

However, we would like to clarify that

  1. IMG training is not identical to evolving a state population.

  2. our IMG has a lower inference time than DNS, so the high training cost can be amortized. This is relevant because several applications need repetitive sampling.

  3. our contribution remains valid under fair comparisons where the total training budget (DNS data generation + model training) is matched.

  4. there is room for further accelerating IMG training.

These points are elaborated as follows:

(1) We agree that IMG uses the distributional invariance condition q=Ptqq=P_t q to construct self-consistent training targets, but its actual updates are not identical to applying the physical evolution operator PtP_t on the distribution qq. First, we use a random δt\delta t within each batch. Second, the neural network parameterization and CFM loss induce a preconditioner on the network updates, which changes the distributional update dynamics (details in Appendix A.2 in the revised manuscript). This may improve convergence relative to direct evolution. We show such possibility in an example of a slowly-mixing double well (see Appendix E.12 in the revised manuscript), where IMG converges faster than directly evolving the same ensemble.

(2) We clarify that IMG is useful even if its training is more expensive than DNS. We aim to learn a generator of invariant measure states, which is substantially different from direct numerical simulation (DNS). Once trained, IMG can synthesize samples from the invariant measure with a lower per-sample wall-clock inference cost than DNS. We report the inference time per second per field on an A100 GPU in the table below.

Method \ Problem settingKSHW
IMG (ours)0.460.50
DNS75.41211.17

Therefore, the training cost can be amortized when many samples are required. In HW, the original ~50 hour training time can be amortized if about 900 samples are needed. An example of potential application is perturbation–response studies where many states from the invariant measure are perturbed and evolved to probe the system response as in [1,2].

Furthermore, IMG can be conditioned on physical parameters, amortizing its training cost across multiple physical settings. Once IMG is trained, it can generate invariant measure samples faster than DNS in new physical conditions.

This is of practical use in high-fidelity tokamak plasma simulations [3], where the heat and particle flux induced by saturated microturbulence (the invariant measure of HW equation in our experiment is an instance) needs to be computed repetitively with different background physical parameters.

We conduct a new experiment to demonstrate that IMG can approximate invariant measure statistics conditioned on different physical parameters (density-gradient drive κ\kappa and electron adiabaticity α\alpha) in the HW equation. The results are summarized in the table below, with each cell showing predicted particle flux / reference particle flux:

α\alpha \ κ\kappa0.512
0.50.2065 / 0.18171.0431 / 0.99515.5057 / 5.1105
10.1345 / 0.11010.7477 / 0.69314.0259 / 3.9369
20.0761 / 0.05380.4885 / 0.43472.8422 / 2.6805

Despite visible error, the trend of the particle flux's dependency on α\alpha and κ\kappa is captured. This demonstrates that IMG can be trained to learn invariant measures conditioned on physical parameters, and therefore obtain efficiency gains over DNS after training.

More details are included in Section 5.3 in the edited manuscript.

AuthorsOpenReview ↗

Official Response to Reviewer ZC88 (2/5)

(response to weakness 1, continued)

(3) We clarify that our training signal is beneficial under fair comparisons with a data-driven invariant measure sampler. Thus, our contribution of turning dynamical evolution into training signals for invariant measure learning holds whether or not our training is faster than one DNS.

In a new HW experiment, under the same solver budget and matched training wall-clock time, we compare IMG (not trained with DNS data), data CFM (first generate data with DNS and then train CFM on it), and mixtures of IMG and data CFM. The details are shown in Section 5.3 in the revised manuscript, and we summarize the results below:

Solver budget ratioSpectrum L1 ↓PDF-TV ↓Flatness rel-L1 ↓Increment PDF-TV ↓Flux MAE ↓A100 GPU hours
IMGDNS 0% IMG 100%0.415130.066270.079180.047030.0885817.86
Data CFM + IMG 25%DNS 25% IMG 75%0.271430.050480.094020.053370.0529217.90
Data CFM + IMG 50%DNS 50% IMG 50%0.263100.051220.098610.056190.1021417.90
Data CFM + IMG 75%DNS 75% IMG 25%0.295990.056420.091160.055300.1022717.88
Data CFMDNS 100% IMG 0%0.333450.059050.088770.064830.1192317.90

The results demonstrate that comibning invariance supervision is better than pure data supervision even under controlled compute budget.

Besides, IMG can utilize the dynamics information hidden in trajectory data, which previous data-driven generative samplers are not able to use, to supervise the learning of the invariant measure.

In the original manuscript, this ability has shown benefits in the few-data HW experiment (section 5.2, with a new 3-seed result summarized below), where IMG-neural outperforms data-driven CFM. There, neural operators train on the same limited dataset, and then executes dynamical evolution in IMG. Thereby, IMG-neural does not require extra solver calls than data CFM.

MethodKSKSKSHWHWHWHW
Coarse SW₂ ↓Spectrum L₁ ↓PDF-TV ↓Spectrum L₁ ↓PDF-TV ↓Increment PDF-TV ↓Flatness L₁ ↓
Data CFM 3-seed mean ± train std0.1041 ± 0.00520.1624 ± 0.01210.0744 ± 0.00120.2096 ± 0.06250.0426 ± 0.01050.0382 ± 0.00990.0621 ± 0.0123
Data CFM + IMG-neural (ours) 3-seed mean ± train std0.0714 ± 0.00130.0987 ± 0.01210.0370 ± 0.00100.1228 ± 0.00490.0306 ± 0.00540.0284 ± 0.00660.0736 ± 0.0220

We have added these results into the revised manuscript, and we edited the introduction to differentiate learned samplers and DNS.

(4) Additionally, we improved the training efficiency by engineering optimizations in the training pipeline. Specifically, we reuse samples in training steps and adopted a replay buffer mechanism with details in Appendix E.13.

MethodSpectrum rel. L1 ↓PDF-TV ↓Increment PDF-TV ↓Flatness rel. L1 ↓Training wall-clock time (h)
IMG0.06300.01160.00940.027352.71
IMG-RB0.06520.01340.00800.02863.73

These results suggest that training IMG has the potential to be further accelerated. The training may be faster than a single DNS rollout especially in more computationally expensive systems or slowly-mixing systems (see the double-well result in Appendix E.12).

[1] T. Matsumoto et al. “Correlation function and linear response function of homogeneous isotropic turbulence in the Eulerian and Lagrangian coordinates.” Journal of Fluid Mechanics, 919, A9, 2021.

[2] R. Nies, et al. “Saturation of magnetized plasma turbulence by propagating zonal flows.” Physical Review Research, 8, 013295, 2026.

[3] A. M. Dimits et al. “Comparisons and physics basis of tokamak transport models and turbulence simulations.” Physics of Plasmas, 7, 969–983, 2000.

Weakness 2: "Data-free" is overstated. DNS validation trajectories are used for checkpoint selection and hyperparameter search (including ΔT and λ), so stationary DNS data still shapes the final model.

We thank the reviewer for this comment. We agree that we use the validation data for hyperparameter search and checkpoint selection, but we argue that our central claim is not weakened by the term.

We would like to clarify that our claim is that IMG can learn a stationary-state sampler without using stationary samples as training targets, and we stated this scope in the original manuscript p. 6, line 313. The validation data is also used for the physics-supervised baselines as they follow the same checkpoint selection protocal. The training signal, which is vital to model training, is still free of invariant measure samples.

AuthorsOpenReview ↗

Official Response to Reviewer ZC88 (3/5)

(reponse to weakness 2, continued)

We also clarify that the λ\lambda search concerns the separate few-data setting, where λ\lambda weights the self-consistency term relative to the data-based CFM objective. This setting explicitly uses stationary training samples and is not presented as data-free.

To avoid potential misunderstandings in the term "data-free", we have replaced "data-free" with more accurate wording throughout the manuscript.

Weakness 3: Weak theory. Theorem 1 holds only for frozen targets; the gradient identity is the standard CFM result, and convergence of the actual iterative procedure is not analyzed. The constant C_t grows exponentially with t for chaotic systems (κ > 0), so the bound is nearly vacuous at long horizons. Lipschitz constants are assumed uniform in θ, which is unrealistic. Measure selection is unresolved: trivial invariant measures (e.g., the zero field for KS/HW) also satisfy self-consistency, and avoiding collapse is shown only empirically.

We thank the reviewer for raising these questions, but we respectfully disagree that this makes the theory weak. We established an upper bound of the target distributional invariance residual (which we aim to minimize) to be the marginal CFM loss. Our claim was intuitively that, if the training loss is minimized, we minimize the upper bound of the target. Under our assumptions, we prove this result and provide justifications for our method design.

Next, we clarify the questions on the convergence, the constant CtC_t and measure selection.

(1) We did not claim that the distributional residual will strictly decrease during training. We agree that studying the distributional convergence behavior of IMG is an important direction for future work.

(2) For the exponential growth in Ct, we provide an additional proof. If the dynamical system satisfies the mixing condition in Appendix B.2 (which holds for certain chaotic dynamical systems including the stochstic L63 in our experiments), we show that a finite-time residual controls the residual at all later times, and thus stops the exponential growing bound (see Appendix B.2 in the revised manuscript). Besides, we would like to clarify that the dependence of the error constant on physical evolution time is common, which appears in multiple previous works, including DeepONet (Theorem 3) and Markov Neural Operator (Theorem 1, Appendix B.2) [1-2]. We note that our original says that, when the marginal flow matching loss vanishes, the distributional invariance residual also vanishes.

The Lipshitz constant being uniform in theta (or equivalently, limiting the theory to a model class that has a uniform Lipshitz constant) is another regularity condition adpoted in classical generative model works including WGAN (Theorem 3, Section 3), NeuralODE (Theorem 1), and consistency models (Theorem 1) [3-5].

(3) We agree that measure selection remains an open problem. However, we clarify that the supervision comes from the invariant measure alone cannot distinguish the physical measure from other invariant measures by definition. Similarly, WGS cannot guarantee to recover the physical measure in such systems either.

In the few-data-assisted scenario, though, IMG has more information and may be able to select the physical measure. We believe that the theoretical analysis in this case is an interesting future direction.

Furthermore, we would like to point out that invariant measures other than the physical measure are not always trivial or undesired. In the modified HW system, our observations suggest that our IMG can find candidate distributions of states that are slowly evolving. These states are not observed in our DNS reference (see Appendix E.12). This result is relevant because identifying almost invariant structures can reveal organization of state space that is difficult to uncover through direct numerical simulation [6], which is an important potential application of IMG.

[1] Li et al., Learning Chaotic Dynamics in Dissipative Systems, NeurIPS, 2022.

[2] Lu et al., Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators, Nature Machine Intelligence, 2021.

[3] Arjovsky et al., Wasserstein Generative Adversarial Networks, ICML, 2017.

[4] Chen et al., Neural Ordinary Differential Equations, NeurIPS, 2018.

[5] Song et al., Consistency Models, ICML, 2023.

[6] Froyland et al., Estimating Long-Term Behavior of Flows without Trajectory Integration: The Infinitesimal Generator Approach, SIAM Journal on Numerical Analysis, 2013.

Reviewer_ubd2OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposes the Invariant Measure Generator (IMG) a flow matching sampler for steady states of chaotic dyanmical systems. At each update, sample from the current generator are advnaced by the true physical dynamics for a random short duration of steps and then used as frozen targets for conditional flow matching. Exepriments cover the standard Lorents 63 and Kuramoto-Sivahinksy but also notably the Hasegawa–Wakatani equations which has 130,000 state coordinates. There are two variants of the model, one without data from the steady state and one where small amount of data form the steady state are known and a self-consistency term is added.

Critical strengths
  1. The method is simple and, as the authors demonstrates, scale to a trubulent model which has 2 or 3 order of magnitude more dimensions than the usual examples in the literature (to my knowledege as least).

  2. In case where few data from the steady state are known, adding the self-consistency terms markedly improves all metrics the result.

  3. The evaluation is tough and self-critical with reference-to-reference for calibration of the Wasserstein distance and invariance test under physical evolution and many others. The paper is very well written.

  4. The theorem seems correct and and the the stop gradient identity is clean.

Critical weaknesses
  1. It seems to me that the main weakness is that the method does not avoid long-time evolution, it just distributes it over training. For the high dimensional example the paper use 4 times 10^7 solver steps which should be enough to produce a large number of independent runs of the systems. If one has access to the solver the results should be compared head to head with direct simulation, rather than other physics-informed samplers.

  2. In appendix E7 where the invariance measure is checked, it seems that the measured quantities changes much more than the direct simulation control and suggest that the learned measure is not invariant.

  3. The theory is a standard Gronwall inequality and the constant in front will be growing exponentially in time and in the Lipschitz bound.

  4. No code is provided.

Questions for authors
  1. Can you report a direct comparison with direct simulation with same solver budget?

  2. Have you tried your approach on slowly mixing systems? AI Claude suggested me to use Hasegawa–Wakatani with zonal flows or Lorentz 96 with multiscale fast slow coupling for high dimensional examples but even standard low-dimensional models would be a good test.

6 comments
AuthorsOpenReview ↗

Official Response to Reviewer ubd2 (1/4)

We thank the reviewer for the thorough and constructive comments. We are pleased that the reviewer finds our method simple and scalable. We also appreciate that the results in the few-data experiment, the thoughness of the empirical evaluation, and the correctness of the theorem is acknowledged. Next, we address the reviewer's concern point-by-point:

Weakness 1: It seems to me that the main weakness is that the method does not avoid long-time evolution, it just distributes it over training. For the high dimensional example the paper use 4 times 10^7 solver steps which should be enough to produce a large number of independent runs of the systems. If one has access to the solver the results should be compared head to head with direct simulation, rather than other physics-informed samplers.

We thank the reviewer for raising this concern. We agree that the training time of IMG is costly for the experiments reported in the original manuscript.

However, we would like to clarify that

  1. our IMG has a lower inference time than DNS, so the high training cost can be amortized. This is relevant in applications that need repetitive sampling.

  2. our contribution remains valid under fair comparisons to data-driven samplers where the training budget is matched.

  3. there is room for further accelerating IMG training.

These points are elaborated as follows:

(1) We clarify that IMG is useful even if its training is more expensive than DNS. We aim to learn a generator of invariant measure states, which is substantially different from direct numerical simulation (DNS). Once trained, our IMG can synthesize more samples from the invariant measure with a few network forward passes. IMG thus has a lower per-sample wall-clock inference cost than DNS. We report the inference time per second per field on an A100 GPU in the table below.

Method \ Problem settingKSHW
IMG (ours)0.460.50
DNS75.41211.17

Therefore, the training cost can be amortized when many samples are required. In HW, the original ~50 hour training time can be amortized if about 900 samples are needed. An example of potential application is perturbation-response studies where many states from the invariant measure are perturbed and evolved to probe the system response as in [1,2].

Furthermore, IMG can be conditioned on physical parameters, amortizing its training cost across multiple physical settings. Once IMG is trained, it can generate invariant measure samples faster than DNS in new physical conditions.

This is of more practical use. For example, high-fidelity tokamak plasma simulations [3], where the heat and particle flux induced by saturated microturbulence (the saturated turbulence of HW equation in our experiment is an instance) needs to be repetitively computed ∼103−105\sim 10^3-10^5 times on different background physical parameters.

We conduct a new experiment to demonstrate that IMG can approximate invariant measure statistics conditioned on different physical parameters (density-gradient drive κ\kappa and electron adiabaticity α\alpha) in the HW equation.

In the table, each cell shows predicted particle flux / reference particle flux.

α\alpha \ κ\kappa0.512
0.50.2065 / 0.18171.0431 / 0.99515.5057 / 5.1105
10.1345 / 0.11010.7477 / 0.69314.0259 / 3.9369
20.0761 / 0.05380.4885 / 0.43472.8422 / 2.6805

Despite some visible error, the trend of the particle flux's dependency on α\alpha and κ\kappa is captured. This demonstrates that IMG can be trained to learn invariant measures conditioned on physical parameters, and therefore obtain efficiency gains over DNS after training.

More details are included in Section 5.3 in the edited manuscript.

AuthorsOpenReview ↗

Official Response to Reviewer ubd2 (2/4)

(response to weakness 1, continued)

(2) We clarify that our training signal is beneficial under fair comparisons with a data-driven invariant measure sampler. Thus, our contribution of turning dynamical evolution into training signals for invariant measure learning holds whether or not our training is faster than one DNS.

In a new HW experiment, under the same solver budget and matched training wall-clock time, we compare IMG (not trained with DNS data), data CFM (first generate data with DNS and then train CFM on it), and mixtures of IMG and data CFM. The details are shown in Section 5.3 in the revised manuscript, and we summarize the results below:

Solver budget ratioSpectrum L1 ↓PDF-TV ↓Flatness rel-L1 ↓Increment PDF-TV ↓Flux MAE ↓A100 GPU hours
IMGDNS 0% IMG 100%0.415130.066270.079180.047030.0885817.86
Data CFM + IMG 25%DNS 25% IMG 75%0.271430.050480.094020.053370.0529217.90
Data CFM + IMG 50%DNS 50% IMG 50%0.263100.051220.098610.056190.1021417.90
Data CFM + IMG 75%DNS 75% IMG 25%0.295990.056420.091160.055300.1022717.88
Data CFMDNS 100% IMG 0%0.333450.059050.088770.064830.1192317.90

The results demonstrate that combining invariance supervision is better than pure data supervision even under controlled compute budget.

Besides, IMG can utilize the dynamics information hidden in trajectory data, which previous data-driven generative samplers are not able to use, to supervise the learning of the invariant measure. In the original manuscript, this ability has shown benefits in the few-data HW experiment (section 5.2, with a new 3-seed result summarized below), where IMG-neural outperforms data-driven CFM. There, neural operators train on the same limited dataset, and then executes dynamical evolution in IMG. Thereby, IMG-neural does not require extra solver calls than data CFM.

MethodKSKSKSHWHWHWHW
Coarse SW₂ ↓Spectrum L₁ ↓PDF-TV ↓Spectrum L₁ ↓PDF-TV ↓Increment PDF-TV ↓Flatness L₁ ↓
Data CFM 3-seed mean ± train std0.1041 ± 0.00520.1624 ± 0.01210.0744 ± 0.00120.2096 ± 0.06250.0426 ± 0.01050.0382 ± 0.00990.0621 ± 0.0123
Data CFM + IMG-neural (ours) 3-seed mean ± train std0.0714 ± 0.00130.0987 ± 0.01210.0370 ± 0.00100.1228 ± 0.00490.0306 ± 0.00540.0284 ± 0.00660.0736 ± 0.0220

We have added these results into the revised manuscript, and we edited the introduction to differentiate learned samplers from DNS.

(3) Lastly, we improved the training efficiency by engineering optimizations in the training pipeline. Specifically, we reuse samples in training steps and adopted a replay buffer mechanism with details in Appendix E.13.

MethodSpectrum rel. L1 ↓PDF-TV ↓Increment PDF-TV ↓Flatness rel. L1 ↓Training wall-clock time (h)
IMG0.06300.01160.00940.027352.71
IMG-RB0.06520.01340.00800.02863.73

These results suggest that IMG training has the potential to be further accelerated.

It may be faster than a single DNS rollout, especially in more computationally expensive systems or slowly-mixing systems. We also did experiments in a slowly-mixing double-well Langevin system, where preliminary results suggest that IMG trainng may be faster than DNS alone (Appendix E.12 in the revised submission).

[1] T. Matsumoto et al. “Correlation function and linear response function of homogeneous isotropic turbulence in the Eulerian and Lagrangian coordinates.” Journal of Fluid Mechanics, 919, A9, 2021.

[2] R. Nies, et al. “Saturation of magnetized plasma turbulence by propagating zonal flows.” Physical Review Research, 8, 013295, 2026.

[3] A. M. Dimits et al. “Comparisons and physics basis of tokamak transport models and turbulence simulations.” Physics of Plasmas, 7, 969–983, 2000.

AuthorsOpenReview ↗

Official Response to Reviewer ubd2 (3/4)

Weakness 2: In appendix E7 where the invariance measure is checked, it seems that the measured quantities changes much more than the direct simulation control and suggest that the learned measure is not invariant.

We thank the reviewer for this comment. We agree that the reported result indicates IMG exhibits higher statistical change in further dynamical evolution than DNS, but we clarify the implications of these results and our central claim as follows:

Our empirical claim is IMG can learn an approximation to the target invariant measure which reproduces important stationary statistics. This claim is directly supported by the main-table comparison with DNS statistics covering fluctuation amplitudes, spatial organization, and multiscale intermittency.

In addition to those, original Appendix E.7 further tests how well these statistics persist under physical evolution. We clarify that the original results of IMG are from a checkpoint of an earlier run. We now applied the same evaluation procedure to the checkpoint used in the main table. We evaluate more baselines to give a more informative comparison between our method and learned invaraint measure approximators. The drift in the statistics of the baseline, the main-table IMG checkpoint and the original results are shown below:

MethodRelative L1L^1 spectral change ↓Generalized-enstrophy change (%) ↓Mean particle-flux change (σΓ,DNS\sigma_{\Gamma,\mathrm{DNS}}) ↓
DNS control0.03550.5270.403
IMG (ours) main-table checkpoint0.08130.2030.207
IMG (ours) Original Appendix E.7 checkpoint0.10793.6720.543
FNO0.38634.1442.953
U-Net1.464466.0147.165
PI-DeepONet17.117133.2347.246
WGS1.965226.1946.839

The results show that both checkpoints of IMG fluctuate smaller than baselines on the three reported metrics, closer to the level of DNS. Notably, on the generalized enstrophy and Mean particle-flux change, the main-table checkpoint realizes a smaller fluctuation than DNS control.

The results are updated in the revised manuscript Appendix E.8.

Weakness 3: The theory is a standard Gronwall inequality and the constant in front will be growing exponentially in time and in the Lipschitz bound.

We agree that proof of our theory uses Gronwall inequality. We clarify that our original theorem shows, under the stated assumptions, the invariance residual vanishes when the marginal CFM loss vanishes.

Regarding the exponentially growing constant CtC_t as the physical evolution time t grows, we clarify that is a common behavior in scientific machine learning methods. Examples include Markov Neural Operator (Theorem 1, Appendix B.2) [1], DeepONet (Theorem 3) [2], etc.

Moreover, we further tightened this bound in a newly added theorem in Appendix B.2. We prove that for a system satisfying the mixing assumptions in Appendix B.2 (which holds for certain chaotic dynamical systems including the stochstic Lorenz-63 in our experiments), the invariance residual at an arbitrary time can be uniformly controlled by a finite time residual.

[1] Li et al., Learning Chaotic Dynamics in Dissipative Systems, NeurIPS, 2022.

[2] Lu et al., Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators, Nature Machine Intelligence, 2021.

Weakness 4: No code is provided.

A: We will release the code upon acceptance of the submission. We have added this commitment in the Reproducibility statement in the revised manuscript.

Question 1: Can you report a direct comparison with direct simulation with same solver budget?

We would like to clarify that our IMG is a learned sampler which is substantially different from DNS, because IMG can amortize the training cost in many inference calls (please see our response to Weakness 1).

For the completeness of the evaluation, we provide the performance of DNS under the same solver budget (simulation of 5×1055\times 10^5 time units), where DNS demonstrates better performance.

KS EquationHW Equations
MethodCoarse SW₂ ↓Spectrum rel. L₁ ↓PDF TV ↓Spectrum rel. L₁ ↓PDF TV ↓Increment PDF-TV ↓Flatness rel. L₁ ↓
IMG (ours)0.050000.038000.0176000.06300.0120000.009400.027000
DNS0.045670.021550.0057130.03680.0082550.005330.009556
Reviewer_uStMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

A common problem in fluid dynamics is the characterization of the steady state e.g. through distributions of summary statistics. For steady state situations, these summary statistics are invariant distributions under the push-forward operation P implied by the time evolution. Other methods find such distributions by either exploiting long-range rollouts or training generative models on samples of long-range rollouts.

Here, a technique is proposed to directly target the invariant measures by learning via a self-consistency L = D(Pq_theta, q_theta). Conditional Flow Matching (CFM) is used to approximate the distributional distance in a tractable way

Critical strengths

The paper identifies a well-motivated method for distributions that are invariant under a push-forward operator implicitly defined by a PDE kernel. This differs from current state of the art methods and opens up a new direction.

The paper is nicely written and easy to follow and should be accessible to people outside of the core field.

Critical weaknesses

The paper correctly identifies the weakness of the method re: "measure selection". Just because a distribution is invariant under push-forward may not imply that these are the invariant distributions that occur during long roll-outs.

The compute cost of the training is significant (50 H800h) and seems to exceed the cost of running roll-outs until burn in to directly access invariance measures. To my understanding this only makes the method applicable for situations where achieving burn-in is very slow, and it's a limitation that is not highlighted in the paper.

Questions for authors

What is the relevance of CFM choice? The core idea could also be implemented using other types of distributional distance estimators such as MMD or similar. There is no baseline reported for this.

What is the comparison of compute cost between sampling via burn-in of autoregressive models vs spending the compute on training the proposed model. If the former is cheap, it's not clear the proposed method will be much better.

3 comments
AuthorsOpenReview ↗

Official Response to Reviewer uStM (1/3)

We thank the reviewer for the constructive feedback. We are glad that the reviewer recognizes our work's motivation, novelty, presentation, and potential impact of opening a new direction. Below, we address each of the reviewer's concern:

Weakness 1: The paper correctly identifies the weakness of the method re: "measure selection". Just because a distribution is invariant under push-forward may not imply that these are the invariant distributions that occur during long roll-outs.

We agree that measure selection remains an open problem. Instead, we empirically demonstrated that our method can learn invariant distributions that accurately approximates the statistics of long DNS rollouts (i.e., the physical measure) in KS and HW, as we discussed in the manuscript.

Meanwhile, we clarify that measure selection is a common problem if the training signal solely comes from the distributional invariance condition. For systems that have multiple invariant measures, methods like WGS face the same measure selection problem. Nevertheless, we note that IMG can utilize data from the physical measure to assist training, and with such additional information it may be able to select the physical measure. Thus, theoretical guarantees for IMG to identify the physical measure under data assistance is an interesting direction.

Moreover, non-physical invariant measures are not always trivial or "wrong". Our results on the mHW system suggest that our IMG finds candidate distributions of slow-varying states, with spatial patterns not observed in our DNS reference (see Appendix E.12). This is relevant because identifying almost invariant structures is crucial in the investigation of slow dynamics, but they are difficult to obtain through direct numerical simulation [1], which is an important potential application of IMG.

[1] Froyland et al., Estimating Long-Term Behavior of Flows without Trajectory Integration: The Infinitesimal Generator Approach, SIAM Journal on Numerical Analysis, 2013.

Weakness 2: The compute cost of the training is significant (50 H800h) and seems to exceed the cost of running roll-outs until burn in to directly access invariance measures. To my understanding this only makes the method applicable for situations where achieving burn-in is very slow, and it's a limitation that is not highlighted in the paper.

We thank the reviewer for raising this concern. We agree that the training time of IMG surpasses one DNS burn-in for the experiments reported in the original manuscript.

However, we would like to clarify that:

  1. Our IMG has a lower inference time than DNS, so the high training cost can be amortized. This is relevant because several applications need repetitive sampling.
  2. Our contribution remains valid under fair comparisons where training budget is matched.
  3. There is room for further accelerating IMG training.

These points are elaborated as follows:

(1) We clarify that IMG is useful even if its training is more expensive than DNS. We aim to learn a generator of invariant measure states, which is substantially different from direct numerical simulation (DNS). Once trained, IMG can synthesize samples from the invariant measure with a lower per-sample wall-clock inference cost than DNS. We report the inference time per second per field on an A100 GPU in the table below.

Method \ Problem settingKSHW
IMG (ours)0.460.50
DNS75.41211.17

Therefore, the training cost can be amortized when many samples are required. In HW, the original ~50 hour training time can be amortized if about 900 samples are needed. An example of potential application is perturbation-response studies where many states from the invariant measure are perturbed and evolved to probe the system response as in [1,2].

Furthermore, IMG can be conditioned on physical parameters, amortizing its training cost across multiple physical settings. Once IMG is trained, it can generate invariant measure samples faster than DNS in new physical conditions.

This is of more practical use. For example, high-fidelity tokamak plasma simulations [3], where the heat and particle flux induced by saturated microturbulence (the invariant measure of HW equation in our experiment is an instance) needs to be computed repetitively with different background physical parameters.

We conduct a new experiment to demonstrate that IMG can approximate invariant measure statistics conditioned on two physical parameters (density-gradient drive κ\kappa and electron adiabaticity α\alpha) in the HW equation. The results are shown below, with each cell showing predicted particle flux / reference particle flux.

α\alpha \ κ\kappa0.512
0.50.2065 / 0.18171.0431 / 0.99515.5057 / 5.1105
10.1345 / 0.11010.7477 / 0.69314.0259 / 3.9369
20.0761 / 0.05380.4885 / 0.43472.8422 / 2.6805
AuthorsOpenReview ↗

Official Response to Reviewer uStM (2/3)

(response to weakness 2, continued)

Despite visible error, the trend of the particle flux's dependency on α\alpha and κ\kappa is captured. This demonstrates that IMG can be trained to learn invariant measures conditioned on physical parameters, and therefore obtain efficiency gains over DNS after training.

More details are included in Section 5.3 in the edited manuscript.

(2) We clarify that our training signal is beneficial under fair comparisons with a data-driven invariant measure sampler. Thus, our contribution of turning dynamical evolution into training signals for high-dimensional invariant measure learning holds, even when our training is slower than one DNS.

In a new HW experiment where the model is conditioned on physical parameters, under the same solver budget and matched training wall-clock time, we compare IMG (not trained with DNS data), data CFM (first generate data with DNS and then train CFM on it), and mixtures of IMG and data CFM. The details are shown in Section 5.3 in the revised manuscript, and we summarize the results below:

Solver budget ratioSpectrum L1 ↓PDF-TV ↓Flatness rel-L1 ↓Increment PDF-TV ↓Flux MAE ↓A100 GPU hours
IMGDNS 0% IMG 100%0.415130.066270.079180.047030.0885817.86
Data CFM + IMG 25%DNS 25% IMG 75%0.271430.050480.094020.053370.0529217.90
Data CFM + IMG 50%DNS 50% IMG 50%0.263100.051220.098610.056190.1021417.90
Data CFM + IMG 75%DNS 75% IMG 25%0.295990.056420.091160.055300.1022717.88
Data CFMDNS 100% IMG 0%0.333450.059050.088770.064830.1192317.90

The results demonstrate that comibning invariance supervision is better than pure data supervision under controlled compute budget.

Besides, IMG can utilize the dynamics information hidden in trajectory data, which previous data-driven generative samplers are not able to use, to supervise the learning of the invariant measure.

In the original manuscript, this ability has shown benefits in the few-data HW experiment (section 5.2, with a new 3-seed result summarized below), where IMG-neural outperforms data-driven CFM. There, neural operators train on the same limited dataset, and then executes dynamical evolution in IMG. Thereby, IMG-neural does not require extra solver calls than data CFM.

KSKSKSHWHWHWHW
Method \ MetricCoarse SW₂ ↓Spectrum L₁ ↓PDF-TV ↓Spectrum L₁ ↓PDF-TV ↓Increment PDF-TV ↓Flatness L₁ ↓
Data CFM 3-seed mean ± train std0.1041 ± 0.00520.1624 ± 0.01210.0744 ± 0.00120.2096 ± 0.06250.0426 ± 0.01050.0382 ± 0.00990.0621 ± 0.0123
Data CFM + IMG-neural (ours) 3-seed mean ± train std0.0714 ± 0.00130.0987 ± 0.01210.0370 ± 0.00100.1228 ± 0.00490.0306 ± 0.00540.0284 ± 0.00660.0736 ± 0.0220

We have added these results into the revised manuscript, and we edited the introduction to differentiate learned samplers from DNS.

(3) Finally, we show that better training efficiency can be achieved by engineering optimizations in the training pipeline. Specifically, we reuse samples in training steps and adopt a replay buffer mechanism with details in Appendix E.13.

MethodSpectrum rel. L1 ↓PDF-TV ↓Increment PDF-TV ↓Flatness rel. L1 ↓Training wall-clock time (h)
IMG0.06300.01160.00940.027352.71
IMG-RB0.06520.01340.00800.02863.73

These results suggest that IMG training has the potential to be further accelerated.

It may be faster than a single DNS rollout, especially in more computationally expensive systems or slowly-mixing systems. We also did experiments in a slowly-mixing double-well Langevin system, where preliminary results suggest that IMG trainng may be faster than DNS alone (Appendix E.12 in the revised submission).

[1] T. Matsumoto et al., Correlation function and linear response function of homogeneous isotropic turbulence in the Eulerian and Lagrangian coordinates, Journal of Fluid Mechanics, 2021.

[2] R. Nies, et al., Saturation of magnetized plasma turbulence by propagating zonal flows, Physical Review Research, 2026.

[3] A. M. Dimits et al., Comparisons and physics basis of tokamak transport models and turbulence simulations, Physics of Plasmas, 2000.

AuthorsOpenReview ↗

Official Response to Reviewer uStM (3/3)

Question 1: What is the relevance of CFM choice? The core idea could also be implemented using other types of distributional distance estimators such as MMD or similar. There is no baseline reported for this.

We thank the reviewer for raising this insightful question.

We actually did think about this question, and tested distributional discrepancies including MMD or a W2W_2 distance approximated with Sinkhorn OT as training loss. The experiment has been added into Appendix E.14 in the revised manuscript, with key results summarized below:

MethodKSKSKSHWHWHWHW
Coarse SW₂ ↓Spectrum L₁ ↓PDF-TV ↓Spectrum L₁ ↓PDF-TV ↓Increment PDF-TV ↓Flatness L₁ ↓
IMG (ours)0.0500.03800.01760.063000.012000.009400.02700
IMG MMD0.4130.16450.18290.197390.203800.086480.33186
IMG Sinkhorn OT0.4820.10570.21030.158630.163610.052150.13922

Our observation was that, despite extensive hyperparameter tuning (e.g., the kernel of MMD), their performance is constantly worse than CFM in our experiments.

We speculate that this is because MMD and W2W_2 are hard to compute in high-dimensional systems. On the contrary, CFM loss is more scalable, which is why we designed our IMG this way. To resolve the problem of MMD and Sinkhorn OT, computing distributional discrepencies in more compact representation space (for example, image representation models like DINO or PDE foundation model like DPOT) may be helpful. We believe this is an interesting future direction.

Question 2: What is the comparison of compute cost between sampling via burn-in of autoregressive models vs spending the compute on training the proposed model. If the former is cheap, it's not clear the proposed method will be much better.

We report the training time and inference time of our method, compared with DNS and other autoregressive (AR) neural operators in the table below.

IMG (ours) inferenceDNS (AR numerical simulation) inferenceFNO (AR neural operator) inferenceUNet (AR neural operator) inferenceIMG (ours) training
KS0.46 seconds75.41 seconds0.13 seconds1.21 seconds5.02 hours
HW0.50 seconds211.17 seconds1.93 seconds5.32 seconds52.71 hours (has been decreased to 3.73 h, see our response to W2)

We would clarify that:

  1. Even if the training time of IMG is higher than one DNS, it can still be useful (please see our response to W2, point 1 and 2)
  2. There is still room for reducing the training time of IMG by further engineering optimization (e.g., from ~53h to ~4h in HW while retaining comparable errors).

Please see our response to W2, point 3 for details.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response by the Authors

We thank the reviewers for their constructive feedback in the review comments. We are glad that the reviewers find our method to be well-motivated, opens a new direction (Reviewer uStM), and simple and scalable (Reviewer ubd2 and ZC88). The reviewers also acknowledge that the paper is well-written (uStM and ubd2), and that the evaluation is honest and thorough (Reviewer ubd2 and ZC88).

Based on the reviewers' comments, we have done new experiments, made clarifications, and updated our manuscript accordingly. The improvements are summarized as follows:

  1. In response to the shared concern that the training of IMG is longer than DNS (Reviewer uStM, ubd2, and ZC88), we clarify that the high training cost can be amortized, and our method is beneficial under fair comparison with data-driven samplers. This is relevant because several applications need repetitive sampling from invariant measures. A new experiment demonstrates IMG has such applications by showing it learns different invariant measures conditioned on physical parameters.

    We did another experiment under matched training budget, where IMG shows improvements over data-driven samplers. This further validates the contribution of our method, in addition to the original solver-budget-fair comparison. Additionally, we showed that IMG training has the potential to be further accelerated while having comparable performance.

    Please see our response to Reviewer uStM weakness 2, Reviewer ubd2 weakness 1, and Reviewer ZC88 weakness 1. The new experiments are added to the revised manuscript Section 5.3 and Appendix E.13.

  2. The reviewers raised questions about the Constant CtC_t in the error bound (Reviewer ubd2 and ZC88) and measure selection (Reviewer uStM and ZC88). We clarify that it is common for the constant in the error bound to grow exponentially with time. We also added a theorem in Appendix B.2 showing that, under a mixing condition, the distributional invariance residual at a fixed mixing time controls the residual at all later times.

    For the measure selection problem, we clarify that the invariance supervision alone cannot select the physical measure by definition, and we point out that identifying invariant measures not produced by DNS may also be of interest in some studies.

    Please see our response to Reviewer uStM weakness 1, Reviewer ubd2 weakness 3, Reviewer ZC88 weakness 3. The new theory is incorporated into Appendix B.2.

  3. Reviewer ubd2 and ZC88 are concerned that our IMG-generated samples are not statistically invariant under further physical evolution, as reported in our original Appendix. We evaluated baseline methods on this test, and compared to learning-based methods. Our IMG exhibits the least statistical drift on most metrics. We also found that the original statistical invariance results of IMG were from an early run. After updating it to the checkpoint in the main table, we observe that IMG's drift of statistics reaches the level of DNS reference on 2 out of 3 metrics concerned by Reviewer ZC88.

    See our response to Reviewer ubd2 weakness 2 and Reviewer ZC88 weakness 4 for details. The results are updated in the revised manuscript Appendix E.8.

  4. We also performed an additional three-training-seed run on the main table results and the Lorenz 63 results following Reviewer ZC88's suggestions.

    See our response to Reviewer ZC88 weakness 6 for details. These results are updated in the revised manuscript Section 5.1 and 5.2.

  5. We try to use different distribution discrepancies as the training loss including MMD and Sinkhorn OT, following Reviewer uStM's question. Our CFM-style loss turns out to be better in the experiments. We associate this with the scalability of the CFM loss.

    Please see our response to question 1 of Reviewer uStM. The results are incorporated into Appendix E.14 in the updated manuscript.

  6. We also tested IMG on slowly mixing systems, including the modified HW (mHW) system and a double well system, as Reviewer ubd2 suggested. In the mHW system, IMG finds candidate slowly-varying manifolds that are were not produced in our DNS runs. In the double-well problem, IMG recovers the asymmetric probability density better than DNS under the same solver budget. These results imply IMG's potential application in dynamical system studies and possible acceleration over DNS.

    Please see our response to question 2 of Reviewer ubd2. We added these results in the revised manuscript Appendix E.12.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.