- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper systematically compares two classes of methods for predicting the evolution of 2D dynamical systems, focusing on predictive accuracy and the reliability of their uncertainty estimates.
Critical strengths
- Good presentation and clear structure
- Important and relatively underexplored problem
- Careful experimental protocol, ablations, and robustness checks
- Fair and balanced presentation of results
Critical weaknesses
- The uncertainty setup is conceptually unclear. The simulators are mostly deterministic, so the true conditional distribution is essentially a point mass, while standard diffusion or flow matching does not directly represent epistemic uncertainty over the learned model.
- Relatedly, this may simply be a problem class where generative models are not especially needed. Their advantage should be more relevant when the conditional target is genuinely stochastic or multimodal.
- The main comparison may reflect latent compression rather than a fundamental difference between model classes.
- There is no real-world evaluation. A weather or climate dataset would be a natural test case.
- The title is broader than the evidence supports.
- Flow matching involves many architectural and training choices, and the paper does not explore enough of this design space to support broad conclusions about the model class.
- Overall, the contribution feels more like a careful empirical comparison and benchmarking framework than a sufficiently novel contribution for ICLR.
Questions for authors
- Why are these model classes and losses appropriate for the uncertainty considered here, especially flow matching for epistemic uncertainty?
- Why should the conclusions from these mostly deterministic simulations transfer to real-world settings?
2 comments
Official Comment by Authors (2/2)
W5 (Title)
We would respectfully like to keep the current title. It names the question the paper studies, the reliability of probabilistic emulation of physical systems, rather than asserting a conclusion. The evidence addresses this question broadly:
- four physical systems with distinct dynamics: transport, buoyancy-driven flow, reaction–diffusion pattern formation and a driven quantum fluid;
- the two most prominent probabilistic approaches in the field;
- further variants, including denoising diffusion, U-Net and FNO backbones, MC dropout and post-hoc conformal calibration.
The scope of the study is stated in the opening sentences of the abstract. As described in response to W4, we have further qualified our claims so that the conclusions are explicitly tied to the evaluated systems and experimental setting.
We would nonetheless be happy to consider a specific alternative if the reviewer feels the current title remains misleading given these clarifications.
W6 (Flow matching design space)
We agree that FM involves many design choices and that we have not explored them exhaustively. Our goal was not to determine the best achievable FM model, but to compare the two approaches under matched backbones, parameter counts and training budgets. We used configurations that follow established latent physics-emulation practice [4].
Within that scope, the paper examines the FM choices we expected to matter most for uncertainty reliability:
- latent versus ambient-space training (Appendix F.1);
- autoencoder compression (Appendix F.2);
- the number of ODE sampling steps (Appendix F.8);
- EMA weights (Appendix F.9);
- FM versus denoising diffusion with a matched backbone (Appendix F.10);
- training seed and training data (Appendices F.13 and F.14).
Motivated by the reviewer's comment, and by a related request from Reviewer ixhz, we have expanded the comparison to include FM with a U-Net backbone. These complement the CRPS-trained U-Net ensemble already reported in Appendix F.11. We have added these results to Appendix F.11.
W7 (Contribution)
We acknowledge that the paper does not propose a new method. Its contribution is, to our knowledge, the first controlled comparison of the uncertainty reliability of the two dominant probabilistic forecasting paradigms for physical systems under matched model size and compute. Reviewer BKjx similarly notes that the value of the work lies in examining whether established probabilistic approaches provide reliable uncertainty estimates.
We believe such evidence is important because these models are increasingly used as replacements for simulators in settings where their uncertainty informs decisions. The study yields a concrete and somewhat unexpected finding: the calibration gap is driven largely by latent-space training rather than by the paradigm itself. This finding motivates a specific direction for future work, namely autoencoders trained with a distributional objective so that latent representations preserve calibration (Section 5). Together with the released AutoCast and AutoSim frameworks, we believe this makes the paper a useful contribution to the ICLR community working on probabilistic emulation.
References
- [4] Lost in Latent Space: An Empirical Study of Latent Diffusion Models for Physics Emulation, Rozet et al.
Official Comment by Authors (1/2)
We thank the reviewer for their thoughtful review. We greatly appreciate the opportunity to clarify the scope of the paper. Below we address each of the points raised. Where weaknesses and questions overlap, we answer them together.
W1, W2 and Q1 (Uncertainty in deterministic systems and choice of model classes)
Even though the simulators are deterministic (Section 3.2), there is still uncertainty for the emulator to learn arising from approximating the dynamics from finite training data and finite model capacity. Neither the CRPS-trained ensembles nor the generative models considered place a posterior over model parameters or decompose epistemic and aleatoric uncertainty; both learn the overall predictive distribution. This follows current practice in weather forecasting [1, 2, 3] and in deterministic PDE emulation [4, 5], where learned spread is interpreted as forecast uncertainty. Our paper evaluates whether this uncertainty is reliable. We find that the answer depends on how the models are deployed: ambient-space flow matching achieves coverage comparable to CRPS-trained ensembles, while latent-space flow matching (FM) generally does not (Appendix F.1).
The paper also includes a conventional approximate Bayesian baseline, MC dropout trained with a regularized MSE loss [6], which was less competitive than both main approaches (Appendix F.17). Stochastic PDEs are a natural extension, which we have added to the discussion of future work.
W3 (Latent compression versus model class)
We agree that latent-space training plays a key role. Ambient FM achieves coverage comparable to CRPS-trained ensembles (Appendix F.1), while stronger compression further degrades FM performance on CNS (Appendix F.2). With the same autoencoder, latent CRPS shows limited coverage degradation on AD, CNS and GS, although accuracy worsens on CNS and GS and the GPE run collapsed. These findings motivate latent representations that more directly target the forecast distribution (Section 5). Our main comparison uses latent FM and ambient CRPS, with latent training motivated by the computational cost of generative emulation [4]. We clarified in the introduction that the crossed ambient/latent ablation is integral to interpreting this comparison.
W4 and Q2 (Real-world evaluation and transfer)
We agree that our claims should reflect the scope of the evaluation, and that we do not establish that the conclusions transfer to real-world systems. We chose controlled, smaller-scale simulations to compare the two approaches across multiple systems through ablations and robustness checks.
The 64×64 resolution made this breadth of experiments feasible within our compute budget. Scaling to higher resolutions and real-world data is a natural next step, which this study helps target by identifying design choices that affect reliability. AutoCast is interoperable with The Well, facilitating follow-up evaluation on larger datasets.
The effect of latent compression on reliability is likely to be relevant to real-world applications where latent spaces are commonly used, particularly when autoencoders are trained for reconstruction rather than forecasting. Whether the effect persists at that scale nevertheless requires further experiments.
We have qualified the claims in the Abstract and Conclusions to refer explicitly to "the four evaluated systems" or "our experimental setting", and have revised the limitations in Section 5 to read:
References