- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper presents a suite of different evaluation methods for neural PDE surrogates, based on generating perturbed initial condition ensembles with a trained surrogate and having access to a DNS solver. There are several conclusions: firstly, the step-wise residual error does not usually correlate with rollout error, second, matching the ensemble spread is generally not enough, and lastly, rare events are generally under or over-predicted by surrogates even with low bias.
Critical strengths
-
The evaluation seems to be thorough (plenty of surrogate benchmarks). Access to a DNS solver allows for some interesting experiments
-
The error decomposition is interesting, and can highlight where some of the rollout error comes from.
Critical weaknesses
-
My main concern is that the paper is written somewhat poorly and the main figures are hard to follow. This makes it difficult to understand the main points of the paper. For example:
- H1/ H4 doesn’t seem to be explained in Figure 2 or before line 191, when it is introduced.
- I’m also assuming the reference state x_k comes from the surrogate? (if so, reference state usually means ground truth which would be a bit confusing)
- What are the triangles/circles in Figure 3? These don’t seem to be labeled.
- I would recommend using variables to denote plotted/measured quantities (instead of “conditional spread” or “dynamics-solver subspace overlap”). I was able to find these quantities in the text, but using variables would be more clear as to what metric is plotted.
-
Somewhat of a costly metric (needs access to DNS to generate a ground truth ensemble). This probably isn’t feasible for unstructured solvers, or for high Reynolds numbers / more complex fluids. Therefore, it is unlikely that similar analyses can be done for other systems (for example weather/climate?) and these takeaways may be restricted to Kolmogorov Flow.
Questions for authors
- I am curious about the interpretation of the first experiment (Figure 2). Unet H1 has better rollout error but more residual error, but then the next plot for FNO shows that higher residual errors correspond to failed members. If this interpretation is correct, does it mean that residual error isn’t indicative of rollout stability? These seem to contradict each other.
5 comments
Response to Reviewer Zfv6 (Part 2/3)
2. Clarify the "contradiction" about local prediction error (Q1)
We thank the reviewer for this insightful question. We realize that our original presentation lacked clarity, making these two observations appear contradictory. In fact, they are entirely consistent and together reveal a critical flaw in evaluating models solely by rollout prediction error.
For clarity, we define the rollout prediction error as and the local prediction error as (formerly termed the "residual"). All notation here and below follows revised Section 3.1. The cumulative local prediction error is
Our core motivation is simple: a good solver should first and foremost have good local accuracy, which is why we choose to focus on it explicitly.
- The U-Net H1 vs. H4 case demonstrates how a lower can obscure a larger cumulative local prediction error .
- The FNO case demonstrates the harm of this obscuring effect. Before blow-up happens, the blown-up FNO members have almost identical to stable members. However, their local prediction error is consistently and significantly higher. A low here masks a severely flawed local model, making it a precursor to hidden failure modes.
In summary, these two cases jointly motivate our explicit focus on the local prediction error (). A low rollout prediction error () can obscure a large (U-Net H1 vs. H4) and mask a fragile state that precedes hidden failure modes (FNO).
To explain this mechanism, we have added Figure R1 in Section 3.1 and a detailed discussion in the revised manuscript. As shown in Figure R1b, the propagation of existing errors by the ground-truth dynamics () is highly complex and fluctuates in sign. Its negative short-term contribution can mask the severe local prediction error () (Figure R1c), leading to a deceptively low (Figure R1a). This demonstrates that the error formation process is far more complex than simple autoregressive amplification.
Response to Reviewer Zfv6 (Part 3/3)
3. Discussion on computational cost and Clarification of our purpose (W2)
We thank the reviewer for raising this practical and important concern. We agree that the ensemble evaluation is computationally demanding. We would like to clarify two aspects: (1) the varying computational costs of our proposed metrics and their feasibility, and (2) our core purpose, which is to reveal certain problems with neural PDE solver based on physical observation rather than a universal benchmark.
3.1. Computational cost and feasibility
We acknowledge the computational cost and breakdown the cost different metrics:
- Low-cost component (Section 3.1): The error decomposition () only requires single trajectory. It can be computed by launching a short numerical simulation at a selected model state to obtain , making it extendable to other systems.
- High-cost components (Sections 3.2, 4, 5): The ensemble error formation, subspace geometry, and extreme event analysis indeed require a full numerical simulation ensemble.
Regarding feasibility:
- For solvers and simulation methods: The paradigm applies to any neural solver trained on numerical simulation data. Provided that the underlying simulation code is accessible, that specific simulation solver can directly serve the exact same role as DNS in our framework.
- For systems: We agree that for high-Reynolds-number turbulence, complex fluids, or weather/climate models, the cost of generating sufficient DNS ensembles will increase drastically, limiting the direct application of the ensemble metrics.
3.2 Core purpose: pointing out unphysical behaviors in neural solvers
We emphasize that the goal of this work is not to position these ensemble metrics as universal benchmarks. Instead, the ensemble acts as a dense observational tool to reveal physical inconsistencies that standard rollout errors mask.
Our motivation stemmed from several physical intuitions that puzzled us when considering neural solver works. We provide the DPOT example:
- Fluid dynamics is a deterministic chaotic system. The mapping from to is deterministic, and due to positive Lyapunov exponents, ensemble of should expand compared to .
- However, DPOT's denoising training maps these perturbed states to the same clean target, despite them being supposed to map to distinct states. This physically corresponds to a contraction of the ensemble, contrary to the true dynamical expansion.
- Intuitively, this means its mapping is relatively more contractive than the true dynamics.
- Despite this physically contradictory behavior, DPOT achieves very good rollout prediction error () across broad cases.
Ensemble evaluation is a appropriate and necessary tool to reveal this hidden contradiction. Indeed, we find that this simultaneous occurrence of lower error and greater contraction is widespread across other neural solvers(Section 3.2).
This prompted us to investigate two follow-up questions: the origin and the consequences of this contraction design.
Regarding its origin, the common intuition is that autoregressive models suffer from iterative error amplification. Motivated by this, we further decomposed the error to understand error formation mechanism.
Regarding its consequences, we realized that this artificial contraction might suppress the natural dynamical amplification of the chaotic system. Consequently, we deliberately evaluated false-negative cases in extreme event forecasts to uncover this impact. These inquiries collectively form the core motivation of our work.
In summary, we appreciate the reviewer's concern and have clarified the cost breakdown and limitations in the paper. We hope the reviewer agrees that while computationally expensive and restricted on some fronts, the ensemble perspective provides unique physical insights that single-trajectory metrics fundamentally cannot offer. While we have not yet conducted broader tests, we believe that, based on the logic of our physical reasoning, these findings are not specific to the Kolmogorov flow but reflect general behaviors of deterministic chaotic dynamics.
Response to Reviewer Zfv6 (Part 1/3)
We thank Reviewer Zfv6 for the thoughtful review and for finding our decomposition and analysis interesting. Below, we respond to the presentation concern, clarify the interpretation of error formation experiments, and discuss the computational cost of the metrics.
1. The presentation concerns (W1)
Following the reviewer's suggestion, we have carefully clarified definitions and notation in revised paper, and have updated the figure captions and legends for better clarity.
To directly answer the reviewer's specific concerns:
Definitions of H1/H4 (Figure 2): We have explicitly defined H1 and H4 in the caption of Figure 2. Hi indicates that the training input uses the past i frames. While originally defined in Section 2 ("Solvers and Forecast Evaluation"), repeating these definitions at the point of specific elaboration is indeed necessary for clarity.
"Ground Truth" vs. "Reference": The reference state xk is computed via direct numerical simulation (DNS). Unlike weather forecasting, where a true physical system exists alongside its numerical approximation, our task directly learns from numerical simulations. Therefore, following the convention in the neural solver literature, we directly refer to the DNS results as the "ground truth" or "reference."
Clarifying the markers in Section 3.2 and Figure 3c: The red triangles in the original Figure 3c represent the noise-trained DPOT. The original Figure 3c aimed to highlight one key finding: while the noise-trained DPOT with lower rollout prediction error compresses the ensemble cloud during propagation, this compression does not bring the ensemble cloud closer to the ground truth than DNS. The detailed definitions are in the Section 3.2. With clearer definitions, we now convey this finding more effectively through text and notation. Consequently, we have replaced Figure 3c to align with the broader discussion of neural solver performance in Section 3.2.
Notation Optimization: We have optimized our notation and use variables as clear indicators. These adjustments cover Sections 3.1, 3.2, and Section 4.