acceptodds
Under review as a conference paper at ICLR 2027

Reliability of Probabilistic Emulation of Physical Systems

Abstract

Two dominant approaches have emerged for generating probabilistic forecasts of physical systems: generative models, such as diffusion or flow matching; and ensembles of deterministic models with stochasticity injected, trained using the continuous ranked probability score (CRPS) loss. While both approaches have demonstrated strong predictive accuracy, the reliability of their uncertainties has not been systematically assessed. We address this gap by developing a framework to evaluate both approaches across diverse 2D spatiotemporal physical systems, under matched model size and computational budget. We assess the reliability of probabilistic emulation by inspecting the empirical coverage of predictive intervals, while also considering accuracy and computational efficiency metrics. CRPS-trained ensembles typically achieve more reliable uncertainties on both single-step prediction and autoregressive rollouts, demonstrating better coverage than the standard alternative of training generative models in a latent space. Moreover, the CRPS approach offers significantly faster inference. When generative models are trained in ambient rather than a compressed latent space, which is often infeasible for high-dimensional problems, they exhibit comparable coverage to CRPS-trained ensembles, though with substantially larger inference latency. In contrast, when CRPS-trained ensembles are trained in latent space they do not show a marked degradation in coverage with respect to ambient space. Both generative models and CRPS-trained ensembles demonstrate good predictive accuracy. To facilitate future research and application, we release AutoCast, a modular framework implementing both generative models and CRPS-trained ensembles, alongside AutoSim, a flexible dataset generation package for rapid prototyping.

Primary area:
datasets and benchmarks
Keywords:
UQ calibration, generative modelling, CRPS training, physical systems, PDEs, benchmarking

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_m1Q9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper systematically compares two classes of methods for predicting the evolution of 2D dynamical systems, focusing on predictive accuracy and the reliability of their uncertainty estimates.

Critical strengths
  • Good presentation and clear structure
  • Important and relatively underexplored problem
  • Careful experimental protocol, ablations, and robustness checks
  • Fair and balanced presentation of results
Critical weaknesses
  • The uncertainty setup is conceptually unclear. The simulators are mostly deterministic, so the true conditional distribution is essentially a point mass, while standard diffusion or flow matching does not directly represent epistemic uncertainty over the learned model.
  • Relatedly, this may simply be a problem class where generative models are not especially needed. Their advantage should be more relevant when the conditional target is genuinely stochastic or multimodal.
  • The main comparison may reflect latent compression rather than a fundamental difference between model classes.
  • There is no real-world evaluation. A weather or climate dataset would be a natural test case.
  • The title is broader than the evidence supports.
  • Flow matching involves many architectural and training choices, and the paper does not explore enough of this design space to support broad conclusions about the model class.
  • Overall, the contribution feels more like a careful empirical comparison and benchmarking framework than a sufficiently novel contribution for ICLR.
Questions for authors
  • Why are these model classes and losses appropriate for the uncertainty considered here, especially flow matching for epistemic uncertainty?
  • Why should the conclusions from these mostly deterministic simulations transfer to real-world settings?
2 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

We thank the reviewer for their thoughtful review. We greatly appreciate the opportunity to clarify the scope of the paper. Below we address each of the points raised. Where weaknesses and questions overlap, we answer them together.

W1, W2 and Q1 (Uncertainty in deterministic systems and choice of model classes)

Even though the simulators are deterministic (Section 3.2), there is still uncertainty for the emulator to learn arising from approximating the dynamics from finite training data and finite model capacity. Neither the CRPS-trained ensembles nor the generative models considered place a posterior over model parameters or decompose epistemic and aleatoric uncertainty; both learn the overall predictive distribution. This follows current practice in weather forecasting [1, 2, 3] and in deterministic PDE emulation [4, 5], where learned spread is interpreted as forecast uncertainty. Our paper evaluates whether this uncertainty is reliable. We find that the answer depends on how the models are deployed: ambient-space flow matching achieves coverage comparable to CRPS-trained ensembles, while latent-space flow matching (FM) generally does not (Appendix F.1).

The paper also includes a conventional approximate Bayesian baseline, MC dropout trained with a regularized MSE loss [6], which was less competitive than both main approaches (Appendix F.17). Stochastic PDEs are a natural extension, which we have added to the discussion of future work.

W3 (Latent compression versus model class)

We agree that latent-space training plays a key role. Ambient FM achieves coverage comparable to CRPS-trained ensembles (Appendix F.1), while stronger compression further degrades FM performance on CNS (Appendix F.2). With the same autoencoder, latent CRPS shows limited coverage degradation on AD, CNS and GS, although accuracy worsens on CNS and GS and the GPE run collapsed. These findings motivate latent representations that more directly target the forecast distribution (Section 5). Our main comparison uses latent FM and ambient CRPS, with latent training motivated by the computational cost of generative emulation [4]. We clarified in the introduction that the crossed ambient/latent ablation is integral to interpreting this comparison.

W4 and Q2 (Real-world evaluation and transfer)

We agree that our claims should reflect the scope of the evaluation, and that we do not establish that the conclusions transfer to real-world systems. We chose controlled, smaller-scale simulations to compare the two approaches across multiple systems through ablations and robustness checks.

The 64×6464 \times 64 resolution made this breadth of experiments feasible within our compute budget. Scaling to higher resolutions and real-world data is a natural next step, which this study helps target by identifying design choices that affect reliability. AutoCast is interoperable with The Well, facilitating follow-up evaluation on larger datasets.

The effect of latent compression on reliability is likely to be relevant to real-world applications where latent spaces are commonly used, particularly when autoencoders are trained for reconstruction rather than forecasting. Whether the effect persists at that scale nevertheless requires further experiments.

We have qualified the claims in the Abstract and Conclusions to refer explicitly to "the four evaluated systems" or "our experimental setting", and have revised the limitations in Section 5 to read:

Establishing whether the observed trends persist in larger physical systems requires further experiments.

References

  • [1] GenCast: Diffusion-based ensemble forecasting for medium-range weather, Price et al.
  • [2] AIFS-CRPS: Ensemble forecasting using a model trained with a loss function based on the Continuous Ranked Probability Score, Lang et al.
  • [3] FourCastNet 3: A geometric approach to probabilistic machine-learning weather forecasting at scale, Bonev et al.
  • [4] Lost in Latent Space: An Empirical Study of Latent Diffusion Models for Physics Emulation, Rozet et al.
  • [5] Probabilistic Retrofitting of Learned Simulators, Diaconu et al.
  • [6] Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning, Gal and Ghahramani.
AuthorsOpenReview ↗

Official Comment by Authors (2/2)

W5 (Title)

We would respectfully like to keep the current title. It names the question the paper studies, the reliability of probabilistic emulation of physical systems, rather than asserting a conclusion. The evidence addresses this question broadly:

  • four physical systems with distinct dynamics: transport, buoyancy-driven flow, reaction–diffusion pattern formation and a driven quantum fluid;
  • the two most prominent probabilistic approaches in the field;
  • further variants, including denoising diffusion, U-Net and FNO backbones, MC dropout and post-hoc conformal calibration.

The scope of the study is stated in the opening sentences of the abstract. As described in response to W4, we have further qualified our claims so that the conclusions are explicitly tied to the evaluated systems and experimental setting.

We would nonetheless be happy to consider a specific alternative if the reviewer feels the current title remains misleading given these clarifications.

W6 (Flow matching design space)

We agree that FM involves many design choices and that we have not explored them exhaustively. Our goal was not to determine the best achievable FM model, but to compare the two approaches under matched backbones, parameter counts and training budgets. We used configurations that follow established latent physics-emulation practice [4].

Within that scope, the paper examines the FM choices we expected to matter most for uncertainty reliability:

  • latent versus ambient-space training (Appendix F.1);
  • autoencoder compression (Appendix F.2);
  • the number of ODE sampling steps (Appendix F.8);
  • EMA weights (Appendix F.9);
  • FM versus denoising diffusion with a matched backbone (Appendix F.10);
  • training seed and training data (Appendices F.13 and F.14).

Motivated by the reviewer's comment, and by a related request from Reviewer ixhz, we have expanded the comparison to include FM with a U-Net backbone. These complement the CRPS-trained U-Net ensemble already reported in Appendix F.11. We have added these results to Appendix F.11.

W7 (Contribution)

We acknowledge that the paper does not propose a new method. Its contribution is, to our knowledge, the first controlled comparison of the uncertainty reliability of the two dominant probabilistic forecasting paradigms for physical systems under matched model size and compute. Reviewer BKjx similarly notes that the value of the work lies in examining whether established probabilistic approaches provide reliable uncertainty estimates.

We believe such evidence is important because these models are increasingly used as replacements for simulators in settings where their uncertainty informs decisions. The study yields a concrete and somewhat unexpected finding: the calibration gap is driven largely by latent-space training rather than by the paradigm itself. This finding motivates a specific direction for future work, namely autoencoders trained with a distributional objective so that latent representations preserve calibration (Section 5). Together with the released AutoCast and AutoSim frameworks, we believe this makes the paper a useful contribution to the ICLR community working on probabilistic emulation.

References

  • [4] Lost in Latent Space: An Empirical Study of Latent Diffusion Models for Physics Emulation, Rozet et al.
Reviewer_ixhzOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents an empirical study comparing flow models and CRPS-trained ensembles for probabilistic emulation of physical systems, with a particular focus on the reliability of uncertainty quantification. The authors evaluate the two approaches on four 2D spatiotemporal physical simulation datasets. The evaluation considers both single-step prediction and long-horizon autoregressive rollouts, measuring predictive accuracy, empirical coverage of prediction intervals, and computational efficiency. The main experiments compare latent-space Flow Matching (FM) with ambient-space CRPS-trained ensembles using similarly sized ViT backbones and matched training budgets. The experimental results suggest that CRPS-trained ensembles provide more reliable uncertainty estimates than latent-space FM models, particularly for long-horizon predictions, while achieving lower inference latency. The paper also introduces AutoSim and AutoCast, two software frameworks for generating physical simulation datasets and benchmarking probabilistic emulators.

Critical strengths
  • The paper is well-organized and easy to follow. The motivation, evaluation methodology, and experimental findings are clearly presented, making the study accessible to readers interested in probabilistic physical system emulation.
  • The paper provides a useful empirical comparison between flow models and CRPS-trained ensembles for uncertainty quantification in physical system forecasting. In particular, evaluating empirical coverage rather than relying solely on predictive accuracy or spread-based metrics provides insights into the reliability of probabilistic predictions.
  • The authors conduct experiments across multiple physical systems and evaluate both single-step prediction and long-horizon autoregressive rollouts. The additional ablation studies examining latent versus ambient space training, ODE sampling steps, and alternative model configurations provide useful insights.
Critical weaknesses
  • The evaluations are limited to relatively small-scale datasets, with a low spatial resolution of 64×6464 \times 64 and a limited number of training trajectories. This raises concerns about whether the experimental findings generalize to larger physical simulation benchmarks. Specifically, while latent-space flow matching may be necessary to reduce inference costs in high-dimensional settings, as noted in L23, training CRPS-based ensembles directly in ambient space may also become computationally prohibitive at scale, particularly given the longer per-epoch training times observed for CRPS in the current experiments. Therefore, it seems premature to conclude that ambient-space CRPS-trained ensembles generally offer a better trade-off between uncertainty reliability and computational efficiency than latent-space generative models without further evaluation on larger, higher-dimensional datasets.
  • According to Appendix A, the datasets contain 200 training, 20 validation, and 20 test trajectories, except for Gray–Scott (GS), which contains 240, 24, and 24 trajectories, respectively. It is unclear whether these dataset sizes are sufficient for adequately training flow-based generative models, particularly given their potentially higher data requirements. For comparison, [1] uses a 2D Kolmogorov flow dataset with 819 training, 102 validation, and 50 test trajectories. Similarly, The Well dataset [2] contains 1,200 GS trajectories across six parameter configurations, with 200 initial conditions per configuration. In contrast, the GS dataset considered in this paper includes only 40 training trajectories per configuration. Given that generative models may benefit from larger training datasets, it seems important to investigate whether the observed performance gap between FM and CRPS-trained ensembles persists as the dataset size increases.
  • The main experiments use 50 ODE sampling steps for flow model, resulting in higher inference latency than the CRPS-trained ensemble. However, Appendix F.8 (Fig. 16) suggests that fewer ODE steps may achieve comparable predictive performance at a lower computational cost. In particular, the VRMSE curves appear nearly identical across different numbers of ODE steps, while the empirical coverage curves also exhibit substantial overlap for some datasets, particularly CNS and GS. Although the authors report that increasing the number of ODE steps generally improves coverage, the results are presented only as plots, making it difficult to assess the trade-off between sampling efficiency and uncertainty calibration.
  • Although the paper investigates alternative network architectures in Appendix F.10, the comparisons are limited to flow versus diffusion model using ViT backbones and CRPS-trained ensembles using ViT versus U-Net backbones. In particular, the paper does not evaluate flow matching or diffusion models with a U-Net backbone, leaving it unclear whether the observed advantages of CRPS-trained ensembles over generative models generalize across different architectures.

Reference

  1. On conditional diffusion models for PDE simulations, Shysheya et al., NeurIPS 2024
  2. The Well: a Large-Scale Collection of Diverse Physics Simulations for Machine Learning, Ohana et al., NeurIPS 2024 Datasets and Benchmarks Track
Questions for authors
  1. The experiments are conducted on relatively small datasets with a spatial resolution of 64×6464 \times 64. Could the authors discuss whether the observed advantages of CRPS-trained ensembles are expected to persist on physics simulations conducted on larger spatial and temporal resolutions? Additional experiments at higher spatial resolutions or on larger datasets would help establish the generalizability of the findings.
  2. Given that the training datasets contain only approximately 200–240 trajectories, could the authors investigate how the relative performance of flow model and CRPS-trained ensembles changes as the training dataset size increases, beyond the currently tested settings? In particular, it would be informative to examine whether the flow model benefits more from additional training data. Since CRPS training also incurs a higher per-epoch cost, analyzing the interaction between dataset size, training time, and predictive performance would be valuable.
  3. Appendix F.8 (Figure 16) suggests that reducing the number of ODE sampling steps may preserve predictive accuracy and, in some cases, comparable empirical coverage. Could the authors provide quantitative results for VRMSE, coverage MAE, and inference latency across different ODE step counts (not as plots)? In particular, how does the comparison between flow model and CRPS-trained ensembles change when flow model uses fewer than 50 ODE steps?
  4. Appendix F.10 compares flow model against diffusion model using ViT backbones (i.e., different formulations) and CRPS-trained ensembles using ViT and U-Net backbones (i.e., different architectures). However, it does not evaluate flow or diffusion model with a U-Net backbone. Could the authors provide a direct comparison between U-Net-based generative models and CRPS-trained U-Net ensembles under matched model sizes and training budgets?
3 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

We thank the reviewer for their thoughtful review and suggestions. We have run additional experiments on training-data size, training time and U-Net backbones, and address the questions raised below.

W1/Q1: Larger physical systems and resolution

We agree that the present results do not establish whether the observed advantages persist at higher resolution or on larger physical systems. Our study aims to provide a controlled benchmark across the four physical systems while matching on a common dataset resolution and format, processor sizes and maximum processor-training budgets. These controls allow us to compare forecast reliability alongside accuracy and to test sensitivity to modelling choices. At larger resolution, these choices would need further careful consideration.

We have removed the discussion sentence expressing an expectation that the conclusions will extend to higher resolution and made clear that the findings concern the evaluated systems and budgets. A controlled study of how reliability and accuracy change with scale is needed to establish broader applicability.

W2/Q2: More training trajectories and training time

To investigate the reviewer's suggestion, we have run two additional tests specifically for Gray–Scott since this was mentioned in W2:

  1. We increased the training set from 240 to 1200 trajectories for both CRPS and flow matching (FM), as well as training the autoencoder on this larger dataset.
  2. We doubled FM's nominal processor-training budget from 24 to 48 hours on the larger training set to ablate the effect of increased training time.

The table below uses the original 24 test trajectories, matching the main benchmark. For the larger training set, each epoch uses a shuffled 20% of the training batches.

Below is a summary of the results for the next window task and late rollout performance (steps [31:99)). Parentheses give standard errors across held-out test trajectories for the models trained on 1200 trajectories.

Model and training trajectoriesSingle-step VRMSESingle-step coverage MAELate-rollout VRMSELate-rollout coverage MAE
CRPS, 2401.1e-030.1989.5e-020.300
FM, 2401.3e-030.1831.6e-010.328
CRPS, 1200 (24h)6.3e-04 (2.0e-05)0.062 (0.003)6.6e-02 (4.4e-02)0.10 (0.02)
FM, 1200 (24h)7.8e-04 (2.5e-05)0.097 (0.006)4.2e-02 (1.5e-02)0.20 (0.01)
FM, 1200 (48h)5.9e-04 (2.4e-05)0.145 (0.005)6.7e-02 (4.4e-02)0.23 (0.01)

With respect to these two tests:

  1. Both approaches benefit from the additional data with lower VRMSE, CRPS and coverage MAE. CRPS-training retains better late-rollout coverage.
  2. FM (48h) improves single-step VRMSE and CRPS score relative to FM (24h), but does not improve coverage.

An additional evaluation of the same checkpoints on 120 independent test trajectories preserved the coverage conclusions, although late-rollout accuracy rankings differed. Appendix F.15 reports this check alongside the approach, results and rollout ablation figure, with a short summary in the main text.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

W3/Q3: Fewer ODE sampling steps

We agree that fewer than 50 ODE steps can improve inference latency with little change in VRMSE as noted in the main text and examined in Appendix F.8, Figure 16.

Below we provide the results comparing 10 with 50 steps for FM on all four systems. Ten steps give about 3.6–4.4 times faster inference, with similar rollout VRMSE. The coverage differences vary by system and horizon. Fewer steps reduce the gap in latency compared to CRPS, though do not remove it. This is because CRPS-trained ensembles require one forward pass per ensemble member.

DatasetModelSingle-step VRMSESingle-step coverage MAELate-rollout VRMSELate-rollout coverage MAELatency (ms/ten-member forecast)
ADCRPS0.00017780.04130.0011470.031815.0
ADFM, 10 steps0.00028790.10120.067070.2809185.8
ADFM, 50 steps0.00022350.01930.069080.1413672.4
CNSCRPS0.0055950.12370.05690.255115.4
CNSFM, 10 steps0.008610.31700.082370.3556170.2
CNSFM, 50 steps0.008680.28420.081850.3367715.0
GSCRPS0.0010690.19840.095210.299713.7
GSFM, 10 steps0.0013710.22040.15240.3574156.7
GSFM, 50 steps0.0013060.18290.15820.3282687.1
GPECRPS0.027670.03730.41380.162915.4
GPEFM, 10 steps0.032070.26120.45350.3026147.8
GPEFM, 50 steps0.031990.23770.45450.3053647.4

We have extended Appendix F.8 with a numerical table across all four datasets and the six evaluated step counts (1, 5, 10, 25, 50 and 100), alongside CRPS (Table 8). The table reports VRMSE, coverage MAE and inference latency, distinguishing single-step and rollout results. We have also qualified the coverage discussion: its dependence on step count varies by dataset and horizon and is not consistently monotonic.

W4/Q4: U-Net generative models

We have evaluated latent FM models with a U-Net backbone on the four systems, using both 0% and 20% dropout (we observed a sensitivity to overfitting early into the training for 0% dropout). The FM U-Net had overall worse VRMSE relative to the ViT backbone although coverage performance varied by dataset. The 20% dropout FM model has low late-rollout coverage MAE but this comes at the cost of a large forecast error on the GPE dataset.

We have added these results to Appendix F.11, including numerical tables, a rollout ablation figure and the training and checkpoint-selection details, alongside a brief summary in the main text.

We hope these additional experiments and clarifications address the reviewer's concerns.

Dear authors,

Thank you for your response to my initial review.

While I still have a concern about the evaluation being limited to small-scale settings, I appreciate the additional results provided in response to my request. As my concerns have mostly been addressed, I update my rating.

Reviewer_BKjxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the reliability of probabilistic surrogate models for physical dynamical systems, comparing flow-matching generative models with noise-injected ensemble approaches. So the authors evaluate not just the predictive accuracy of the models, but also whether the resulting uncertainty estimates are appropriately accurate, using probabilistic scoring and coverage metrics. They study these approaches across several physical systems simulations and further investigate their findings with ablation studies and supplementary experiments. The authors also provide coding frameworks to reproduce their work or apply their comparative methods to potential future use cases.

Critical strengths

This is a well-written and valuable empirical study of the reliability of UQ for AI/ML emulators of physical systems. Rather than proposing a fundamentally new method, it examines whether established probabilistic approaches provide trustworthy uncertainty estimates. This is an important topic which will become increasingly relevant as surrogate models progressively start replacing conventional simulations in digital twins for engineering design and R&D. The comparison of flow-matching models and noise-injected ensembles provides an interesting perspective, while the supplementary experiments in the appendix explore factors that could influence the conclusions and open up ideas for future avenues. Overall, the paper is well written and well structured, it is informative and engaging, with significance arising from the question it investigates and the supporting analysis. We believe it will be of interest to the wider ICLR community.

Critical weaknesses

The paper lacks novelty in terms of AI/ML methods, but still provides valuable experiments and approaches to the community and addresses the important subject of UQ for surrogate models.

Questions for authors

n/a

1 comment

We thank the reviewer for their positive and thoughtful assessment, and for recognising the importance of assessing whether established probabilistic approaches provide trustworthy uncertainty estimates as surrogate models increasingly replace conventional simulations.

Prompted by the other reviewers, we have extended the supplementary experiments, including a larger training set and longer flow matching training budget, numerical comparisons across ODE sampling step counts, and flow matching with a U-Net backbone. We have added these to the appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.