- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper investigates if different fidelity in diverse problem settings affect model performance using diverse architectures. Therefore, they introduce a new benchmark, PDE-Fidelity, which contains 1,191 physical scenarios of four different systems. Each of the scenarios is simulated using discontinuous Galerkin solvers, with degrees 1, 3, and 5 on a grid of . Evaluation is performed with a variety of architectures using one-step prediction and rollout.
In summary, degree 5 data improves label and one-step accuracy for all problems, while rollout performance depends on the system. For the GLM-MHD dataset, the rollout results are three times worse. Additionally, the authors present a simplified DG example, which indicates how such a reversal can arise, and an error propagation analysis that reconstructs rollout errors over short horizons.
Critical strengths
- The topic is overall important, looking at different fidelities and their effects when training different networks, and the paper provides an honest framing of its results.
Critical weaknesses
- Writing / Presentation: The paper in its current form cannot be evaluated since central arguments cannot be extracted from the text. Just to give an example, sentences such as "Numerical source is the treatment condition, whereas reference error is a measured property of that source:" or "Finite-horizon propagation yields conditional vector-error bounds that retain defect directions and temporal interactions" are unclear even after repeated reading.
Additionally, there are a lot of notations and labels without definition, or they are used before they are introduced, for example: H, K1/K4, TF4/SG4/BP4.
-
Limited insights: The paper is structured in a sequence of experiments, with rather limited expressiveness. The outlined finding of having reversed effects for high-order supervision only appears in one system (GLM-MHD), where all others either improve or show no reaction. Based on the text, one understands that the reversal can occur but not why.
-
Narrow experimental design: The paper limits the grid to a size of 128^2 and the DG polynomial degree to three values. Problem settings such as GLM-MHD or RMI require higher resolutions, since only when being able to represent the fine-scale structure one would know how the PDEs actually behave. Not resolving high-order solutions correctly, or, in the case of the paper, averaging them over a specific grid, removes information that distinguishes them from the low-resolution case. The only experiment where resolution is varied shows the biggest improvement, and yet it's the only one out of the 4 datasets. 7 architectures for only one resolution is not as informative as more resolution and less architectures.
Questions for authors
- How do the results change with an observation grid larger than ? Does a finer-grid source also improve rollouts in the other three systems?
- Why do SWE and RMI show no clear rollout improvement with degree-5 data, despite large improvements in label and one-step accuracy?
- Since only one of four systems shows a clear reversal, what evidence supports the general claim that numerical fidelity does not determine learning value, rather than a claim specific to GLM–MHD in this setup?
- Beyond the results of the datasets in the rollouts, what concrete recommendation does the paper give for choosing a numerical data source?
5 comments
Resolution, simulation cost and practical recommendations
W3. Resolution and representation
Source and observation refinement preserve the main finding: SWE gains are limited, RMI depends on the model, and GLM U-Net still shows reversed rankings under single-stage SG4 training. The added source256 P1/P5 and observation128/256 tests cover all four systems. Stored fields are DG cell averages; observation256→128 adds a 2×2 average.
Q1. Larger observations and finer sources
| System | P5 obs256/obs128 error, U-Net/FNO | P1 source256/source128 error, U-Net/FNO |
|---|---|---|
| Allen | 0.848/0.979 | 0.536/0.608 |
| GLM | 1.577/314.606 | 0.634/1.040 |
| RMI | 1.766/0.985 | 0.454/0.886 |
| SWE | 1.087/1.017 | 0.935/0.929 |
Both columns score at 128. The first fixes source256; the second fixes observations128. References are GLM 512/P1, RMI 1024/P1, and Allen/SWE 512/P3; the SWE source-grid control uses 384/P3.
For GLM, source refinement lowers U-Net rollout error by 36.55% but raises FNO error by 3.99%. Both one-step errors decrease; physical failures change from 9 to 2 for U-Net and from 10 to 13 for FNO.
SWE gains about 7% from source refinement. RMI depends on architecture: U-Net mean error improves but physical failures rise from 1 to 20, while FNO error falls about 11%. GLM's H/P1 ratio is 2.06 for single-stage U-Net SG4; a separate two-stage comparison gives 0.39 (Section 6.2).
At native256, P5/P1 rollout ratios remain above one: RMI 2.57/1.76, SWE 1.1/1.13, and GLM 3.29/1048.53 (U-Net/FNO; Section 6.3).
Enlarging network observations along with source-grid refinement increases training and inference work. We therefore compare degrees at a fixed network grid to vary source accuracy without enlarging network inputs.
Q3. Scope beyond GLM reversal
The central issue is forecasting return on additional simulation work:
| System | P5/P1 generation time | Label error | Rollout error |
|---|---|---|---|
| Allen | 5.830 | 1.4×10⁻³ | 0.327 |
| GLM | 34.290 | 0.144 | 2.971 |
| RMI | 20.090 | 0.535 | 1.071 |
| SWE | 19.650 | 0.036 | 0.934 |
Errors are aggregate P5/P1 ratios; times are recorded within-system median ratios. RMI and SWE require about 20 times the generation time without a clear aggregate rollout gain. GLM worsens; Allen retains a clear benefit.
Q4. Concrete recommendations
Select source, model and training objective jointly for the intended forecast horizon, recording generation and training costs.
For existing P5 data, the loss reweighting in Part 1 is an option without regenerating trajectories. Applying the same change to P1, P5/P1 rollout ratios change from 2.07 to 1.02 for RMI and from 1.02 to 0.8 for SWE. Compare these options on validation forecasts.
Thank you. We would be happy to discuss any further questions.
Response to rebuttal
I thank the authors for their rebuttal and their effort.
The revision of this paper is not incremental. The uploaded paper is effectively a new paper. About ~70% of the main-text lines are new or rewritten. The abstract has been replaced, Sections 4 to 7 have been restructured, several experiments are new, and the central claim has been narrowed. None of the changes are marked. Evaluating the new text and results would require a full new review, which is not feasible in the discussion period. I therefore keep my original evaluation.
Writing clarity and mechanisms beyond GLM
Thank you for your detailed suggestions on improving the presentation and extending the experimental evidence.
W1. Main-text clarity
We have revised the manuscript, clarified the explanations, and moved definitions before the results. Section 3.1 now states: “A numerical source is a particular DG solver configuration. Polynomial degree specifies how training trajectories are generated; label error measures how closely their stored fields match the separately generated reference.”
Figure 1 explains the inputs once: one-step prediction uses five reference frames at every step; rollout starts from five reference frames and then feeds each prediction into the next input. H is defined in the Introduction, before Figure 1. Section 3.2 introduces these training labels before the results:
The abstract now describes one-step error accumulation and short-window neural error reconstruction. Section 7 presents neural error reconstruction, then the simplified DG example.
W2. Mechanistic insight
The experiments separate the accuracy of recorded targets from behavior after predictions enter later inputs. The original GLM test changes from reference inputs to rollout at fixed weights. The RMI/SWE intervention below changes the loss on the same data. Together, they show how feedback and training priorities affect forecasting gains.
Q2. Why do RMI/SWE local gains not yield rollout gains?
We reduce the relative weight on small-scale prediction errors (Fourier index radius above 12) from 1 to 0.1 in matched FNO training:
The same data yield better long forecasts despite worse one-step scores. SWE's source error falls sharply, while the squared difference between autonomous and reference-input predictions barely changes; for RMI, this difference changes in opposite directions across FNO and U-Net (Appendix L).
A separate SWE control at 256 resolution trains on predicted histories. Rollout error decreases 5.47% for P1 and 6.88% for P5 relative to recorded-history training. This extends the feedback-training evidence beyond GLM.