acceptodds
Under review as a conference paper at ICLR 2027

Better Simulations Do Not Guarantee Better Surrogates: A Paired Study of Numerical Fidelity

Abstract

Numerical simulations approximate PDE solutions, but their usefulness as training data also depends on the dynamics learned from them. We investigate this distinction with PDE-FIDELITY, a paired benchmark that varies the polynomial degree of one, three, and five in the discontinuous Galerkin (DG) method while keeping physical instances and the observation interface fixed. This design separates numerical label accuracy, prediction from common reference histories, and autonomous rollout. Across four systems and seven architectures, degree-five supervision improves label accuracy and aggregate one-step prediction over degree one, yet the rollout benefit is system-dependent and can reverse. In GLM–MHD, the aggregate degree-five/degree-one rollout error ratio is roughly three; the reversal persists under the tested training controls and is confirmed for U-Net on fresh physical instances. An exact DG construction shows that such a reversal can arise even at the unique training-loss minimizers. Within this construction, enforcing mean preservation removes the reversal without adding learned parameters, demonstrating how model structure can change the value of a numerical source. Finite-horizon propagation yields conditional vector-error bounds that retain defect directions and temporal interactions. Corresponding reconstruction experiments show close agreement with neural rollout errors over short windows, with model-dependent limits at longer horizons. Complementary finer-grid experiments also produce substantial U-Net gains, showing that accurate supervision can help without implying a universal source ranking. These results motivate evaluating numerical supervision through the autonomous dynamics learned from it, rather than inferring its value from label or one-step accuracy alone.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Neural operator, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uhKLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates if different fidelity in diverse problem settings affect model performance using diverse architectures. Therefore, they introduce a new benchmark, PDE-Fidelity, which contains 1,191 physical scenarios of four different systems. Each of the scenarios is simulated using discontinuous Galerkin solvers, with degrees 1, 3, and 5 on a grid of 1282128^2. Evaluation is performed with a variety of architectures using one-step prediction and rollout.

In summary, degree 5 data improves label and one-step accuracy for all problems, while rollout performance depends on the system. For the GLM-MHD dataset, the rollout results are three times worse. Additionally, the authors present a simplified DG example, which indicates how such a reversal can arise, and an error propagation analysis that reconstructs rollout errors over short horizons.

Critical strengths
  1. The topic is overall important, looking at different fidelities and their effects when training different networks, and the paper provides an honest framing of its results.
Critical weaknesses
  1. Writing / Presentation: The paper in its current form cannot be evaluated since central arguments cannot be extracted from the text. Just to give an example, sentences such as "Numerical source is the treatment condition, whereas reference error is a measured property of that source:" or "Finite-horizon propagation yields conditional vector-error bounds that retain defect directions and temporal interactions" are unclear even after repeated reading.

Additionally, there are a lot of notations and labels without definition, or they are used before they are introduced, for example: H, K1/K4, TF4/SG4/BP4.

  1. Limited insights: The paper is structured in a sequence of experiments, with rather limited expressiveness. The outlined finding of having reversed effects for high-order supervision only appears in one system (GLM-MHD), where all others either improve or show no reaction. Based on the text, one understands that the reversal can occur but not why.

  2. Narrow experimental design: The paper limits the grid to a size of 128^2 and the DG polynomial degree to three values. Problem settings such as GLM-MHD or RMI require higher resolutions, since only when being able to represent the fine-scale structure one would know how the PDEs actually behave. Not resolving high-order solutions correctly, or, in the case of the paper, averaging them over a specific grid, removes information that distinguishes them from the low-resolution case. The only experiment where resolution is varied shows the biggest improvement, and yet it's the only one out of the 4 datasets. 7 architectures for only one resolution is not as informative as more resolution and less architectures.

Questions for authors
  1. How do the results change with an observation grid larger than 1282128^2? Does a finer-grid source also improve rollouts in the other three systems?
  2. Why do SWE and RMI show no clear rollout improvement with degree-5 data, despite large improvements in label and one-step accuracy?
  3. Since only one of four systems shows a clear reversal, what evidence supports the general claim that numerical fidelity does not determine learning value, rather than a claim specific to GLM–MHD in this setup?
  4. Beyond the results of the datasets in the rollouts, what concrete recommendation does the paper give for choosing a numerical data source?
5 comments
AuthorsOpenReview ↗

Writing clarity and mechanisms beyond GLM

Thank you for your detailed suggestions on improving the presentation and extending the experimental evidence.

W1. Main-text clarity

We have revised the manuscript, clarified the explanations, and moved definitions before the results. Section 3.1 now states: “A numerical source is a particular DG solver configuration. Polynomial degree specifies how training trajectories are generated; label error measures how closely their stored fields match the separately generated reference.”

Figure 1 explains the inputs once: one-step prediction uses five reference frames at every step; rollout starts from five reference frames and then feeds each prediction into the next input. H is defined in the Introduction, before Figure 1. Section 3.2 introduces these training labels before the results:

LabelMeaning
K1/K4One next-target loss / four recursive target losses
TF4Recorded sliding histories
SG4Predicted histories; stop gradients before each prediction
BP4Backpropagate through predicted histories

The abstract now describes one-step error accumulation and short-window neural error reconstruction. Section 7 presents neural error reconstruction, then the simplified DG example.

W2. Mechanistic insight

The experiments separate the accuracy of recorded targets from behavior after predictions enter later inputs. The original GLM test changes from reference inputs to rollout at fixed weights. The RMI/SWE intervention below changes the loss on the same data. Together, they show how feedback and training priorities affect forecasting gains.

Q2. Why do RMI/SWE local gains not yield rollout gains?

We reduce the relative weight on small-scale prediction errors (Fourier index radius above 12) from 1 to 0.1 in matched FNO training:

SystemP5 one-step error change (%)P5 rollout error change (%)
RMI+11.200−51.020
SWE+13.390−26.280

The same data yield better long forecasts despite worse one-step scores. SWE's source error falls sharply, while the squared difference between autonomous and reference-input predictions barely changes; for RMI, this difference changes in opposite directions across FNO and U-Net (Appendix L).

A separate SWE control at 256 resolution trains on predicted histories. Rollout error decreases 5.47% for P1 and 6.88% for P5 relative to recorded-history training. This extends the feedback-training evidence beyond GLM.

AuthorsOpenReview ↗

Resolution, simulation cost and practical recommendations

W3. Resolution and representation

Source and observation refinement preserve the main finding: SWE gains are limited, RMI depends on the model, and GLM U-Net still shows reversed rankings under single-stage SG4 training. The added source256 P1/P5 and observation128/256 tests cover all four systems. Stored fields are DG cell averages; observation256→128 adds a 2×2 average.

Q1. Larger observations and finer sources

SystemP5 obs256/obs128 error, U-Net/FNOP1 source256/source128 error, U-Net/FNO
Allen0.848/0.9790.536/0.608
GLM1.577/314.6060.634/1.040
RMI1.766/0.9850.454/0.886
SWE1.087/1.0170.935/0.929

Both columns score at 128. The first fixes source256; the second fixes observations128. References are GLM 512/P1, RMI 1024/P1, and Allen/SWE 512/P3; the SWE source-grid control uses 384/P3.

For GLM, source refinement lowers U-Net rollout error by 36.55% but raises FNO error by 3.99%. Both one-step errors decrease; physical failures change from 9 to 2 for U-Net and from 10 to 13 for FNO.

SWE gains about 7% from source refinement. RMI depends on architecture: U-Net mean error improves but physical failures rise from 1 to 20, while FNO error falls about 11%. GLM's H/P1 ratio is 2.06 for single-stage U-Net SG4; a separate two-stage comparison gives 0.39 (Section 6.2).

At native256, P5/P1 rollout ratios remain above one: RMI 2.57/1.76, SWE 1.1/1.13, and GLM 3.29/1048.53 (U-Net/FNO; Section 6.3).

Enlarging network observations along with source-grid refinement increases training and inference work. We therefore compare degrees at a fixed network grid to vary source accuracy without enlarging network inputs.

Q3. Scope beyond GLM reversal

The central issue is forecasting return on additional simulation work:

SystemP5/P1 generation timeLabel errorRollout error
Allen5.8301.4×10⁻³0.327
GLM34.2900.1442.971
RMI20.0900.5351.071
SWE19.6500.0360.934

Errors are aggregate P5/P1 ratios; times are recorded within-system median ratios. RMI and SWE require about 20 times the generation time without a clear aggregate rollout gain. GLM worsens; Allen retains a clear benefit.

Q4. Concrete recommendations

Select source, model and training objective jointly for the intended forecast horizon, recording generation and training costs.

For existing P5 data, the loss reweighting in Part 1 is an option without regenerating trajectories. Applying the same change to P1, P5/P1 rollout ratios change from 2.07 to 1.02 for RMI and from 1.02 to 0.8 for SWE. Compare these options on validation forecasts.

Thank you. We would be happy to discuss any further questions.

Reviewer_uhKLOpenReview ↗

Response to rebuttal

I thank the authors for their rebuttal and their effort.

The revision of this paper is not incremental. The uploaded paper is effectively a new paper. About ~70% of the main-text lines are new or rewritten. The abstract has been replaced, Sections 4 to 7 have been restructured, several experiments are new, and the central claim has been narrowed. None of the changes are marked. Evaluating the new text and results would require a full new review, which is not feasible in the discussion period. I therefore keep my original evaluation.

Reviewer_Lws8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper explores the effect of the fidelity of the training data on the resulting neural surrogate performance. The paper finds that in certain systems, training on more refined training data can reduce rollout performance. However, one-step performance always improves with more accurate training data. The authors also investigate the effect of unrolled training and different refinement strategies.

Critical strengths
  • This is an interesting question. Although there is some prior works (Neural Emulator Superiority, Compute-Optimal data scaling for neural surrogates), the question of data fidelity is under-explored

  • The provided studies are very thorough and explore some interesting questions about data fidelity.

Critical weaknesses
  • The questions are interesting, but at the same time, the practical implications of this work may be limited. Sometimes we can only use existing data (for example from prior benchmarks or from historical simulations/observations), and in general, it is impossible to know the reversal point a priori and therefore we cannot find the ideal resolution before generating the data (which is the costly part). This would mean that we always need to train multiple surrogates at different resolutions to quantify the reversal point, which may not be feasible in practice.
Questions for authors
  • What do the different colors represent in Figure 4?
  • Would it be correct to say that we should use h-refinement instead of p-refinement if we want to increase the numerical fidelity of the source data? If so, this would be a good takeaway.
2 comments
AuthorsOpenReview ↗

Simulation Cost, Forecasting Gains, and Grid Refinement

Thank you for your constructive comments on simulation costs, refinement choices, and the practical value of our findings.

W1. Practical value with existing data and costly generation

The practical question is how much long-term forecasting improvement the additional simulation work provides. Section 4.3 now compares generation cost and rollout benefit:

SystemP5/P1 generation time (median)P5/P1 rollout error
Allen–Cahn5.8300.327
GLM34.2902.971
RMI20.0901.071
SWE19.6500.934

Allen–Cahn gains substantially. GLM forecasts worsen; RMI and SWE show no clear aggregate gain despite roughly 20 times the generation cost.

For the existing RMI and SWE data, downweighting high-frequency errors lowers P5 FNO rollout errors by 51.02% and 26.28%. Their P5/P1 rollout-error ratios change from 2.07 to 1.02 and from 1.02 to 0.8, respectively (§5.1). This training change improves models whose long-term performance was insufficient, using the same trajectories.

With fixed data, compare training choices using long-term validation forecasts. Before bulk simulation, compare candidate sources on a small paired set and weigh forecast gains against generation and training costs.

Q1. What do the different colors represent in Figure 4?

Figure 4 (Appendix I) now identifies P1 as blue circles, P3 as ochre squares, P5 as orange diamonds, and H as teal triangles. H is the 512×512, degree-one source averaged to 128×128 observations, and is defined in the introduction.

Q2. Should h-refinement replace p-refinement when increasing source fidelity?

h-refinement leaves the main conclusion unchanged: higher fidelity does not reliably improve long forecasts. At fixed 128×128 observations, refining the degree-one source from 128×128 to 256×256 gives these finer/coarser rollout-error ratios (§6.2):

SystemU-NetFNO
Allen–Cahn0.5360.608
RMI0.4540.886
SWE0.9350.929

Allen–Cahn benefits clearly, whereas SWE gains about 7%. RMI depends on the architecture: U-Net's mean error falls substantially, but physical failures rise from 1/20 to 20/20; FNO's error falls by about 11%. SWE FNO failures rise from 0/20 to 1/20.

Separate GLM experiments give a U-Net H/P1 ratio of 2.06 under single-stage SG4 and 0.39 under two-stage training (Appendices E.5 and I). The single-stage result shows that a GLM reversal persists under h-refinement.

When h-refinement also enlarges network observations, training and inference require more computation. This motivates our p-refinement comparison at a fixed network grid. Moving observations from 128×128 to 256×256 quadruples the grid points per frame. The source-only comparison above keeps observations fixed. The practical choice should combine rollout gains, physical outcomes, and generation, training, and inference costs.

Thank you again for your time and feedback.

Thank you for the response. I have spent a few hours reviewing the updated manuscript and reading response to other comments, and unfortunately the message of the work has gotten even more unclear to me.

It seems the results are rather convoluted, and dependent on the PDE, model architecture, training loss, and model input. This is further conflated by the fact that although more accurate simulations do not guarantee better long-term forecasts, sometimes we actually do need more accurate simulations to resolve important phenomena (such as DNS/LES). It may just be that higher resolution simulations have more difficult physics to predict, due to having finer-scale features for example.

Unfortunately, the best I can do is maintain my score, although I am no longer convinced about this work .

Reviewer_1jUMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces a paired PDE dataset and benchmark with 4 PDE systems solved with the discontinuous Galerkin (DG) method using 3 polynomial degrees and 7 neural surrogate model architectures. The authors run a controlled study of one step forecasting and rollout/autoregressive forecasting performance across all systems at varying DG degrees with all neural surrogate models. Their primary finding is that the GLM-MHD system with higher polynomial degree (p=5) has better one-step forecasting performance than the GLM using p=1, however when comparing the same data on autoregressive forecasting, the GLM p=5 data paired with a neural surrogate performs much worse than the p=1 data. The paper also shows some theory as to why the DG method can result in negatively correlated fluctuations that may cause this problem for GLM.

Critical strengths
  • The GLM-MHD system with varying polynomial degree results are convincing empirical evidence that higher accuracy simulations don't always lead to better neural network rollout performance.
  • The benchmark and experiments are very thorough and compare all methods in a fair way. The extra experiments on the finer grid H and fresh unseen physical identities also increase confidence in the empircal results.
Critical weaknesses
  • Even though the paper has extensive experiments, it does not explain why the reversal occurs for GLM. This reduces the novelty of the paper, with only one system showing the reversal effect.
  • The way the paper is written was difficult for me to follow and I had to put in significant effort while reading to understand the methods.
Questions for authors

Do you have any mechanistic explanation for why specifically the GLM system undergoes reversal and the others do not?

1 comment
AuthorsOpenReview ↗

Forecasting Gains, Error Accumulation, and Training Choices

Thank you for your careful reading and constructive comments.

W1. The finding and its explanation

We ask whether the extra work used to generate more accurate simulations brings a clear long-term forecasting benefit. The benchmark shows lower label and aggregate one-step errors in all four systems, but different rollout gains:

SystemP5/P1 generation timeP5/P1 rollout error
Allen–Cahn5.8300.327
GLM34.2902.971
RMI20.0901.071
SWE19.6500.934

P5/P1 compares degrees five and one. Section 4 reports these median generation-time ratios and aggregate rollout ratios. Allen–Cahn benefits substantially. RMI depends on the architecture, and SWE's overall gain is small. Their much greater generation cost brings no clear aggregate forecasting gain. GLM forecasts worsen.

Changing the loss improves P5 FNO models in GLM, RMI, and SWE, which had gained little or nothing from the more accurate data. Reducing the high-frequency (HF) error weight in the training loss to 0.1 gives (Section 5):

SystemP5 one-step error change (%)P5 rollout error change (%)
GLM+4.070−19.680
RMI+11.200−51.020
SWE+13.390−26.280

These P5 models improve their long forecasts even though their one-step errors increase. Changing how accurate data are used during training can therefore improve their forecasting value.

W2. Main-text clarity

We have revised the manuscript to make the explanations clearer. Section 3.1 states: “A numerical source is a particular DG solver configuration.” It then distinguishes polynomial degree from measured label error. Figure 1's caption explains the inputs:

“One-step prediction uses five reference frames at every step. Rollout (closed loop in the figures) starts from five reference frames and then feeds each prediction into the next input.”

The Introduction defines H as a degree-one source generated on a 512×512 grid and averaged to 128×128 observations. Section 3.2 introduces K1/K4 and TF4/SG4/BP4 before the results, distinguishing recorded and predicted inputs and gradients through predictions.

The abstract now says: “We also analyze how one-step errors accumulate and reconstruct neural forecast errors over short prediction windows.” Section 7 first presents neural error reconstruction, then a DG example showing how a model restriction changes the value of its training data.

Q1. Why do GLM and the other systems differ?

Appendix L tests error timing and error exchange in fixed models, using future reference frames to construct the replacement errors.

For GLM, longer intervals between reference-history resets gradually remove the P5 advantage (Section 5.2). Alternating the error signs over time, while preserving each frame's frequency magnitudes and energy, reduces P5 rollout error to 0.27/0.28 of the constant-sign values for FNO/U-Net. This shows that error timing changes how local errors accumulate.

RMI depends more on the architecture: the same timing intervention gives P5 ratios of 0.3 for FNO and 0.99 for U-Net. HF training changes the FNO P5/P1 rollout ratio from 2.07 to 1.02. At native 256×256 resolution, P5 rollout errors exceed P1 errors for both U-Net and FNO, with ratios of 2.57/1.76 (Section 6.3).

SWE has much lower source error but only a small aggregate rollout gain. Training on predicted histories lowers P5 rollout error by 6.88%, while its error remains 1.11 times P1's (Section 5.3). Improving training helps, but does not automatically give P5 an advantage.

Allen–Cahn benefits from the more accurate source. Replacing the P1 model's reference-input errors with P5 errors lowers rollout error by about 79% in both architectures; the reverse exchange worsens P5. Together, these comparisons show that the forecasting value of source accuracy depends on how the model learns and feeds predictions back into later inputs.

Thank you again for your feedback. We look forward to your response.

Comments on the paper

1 comment

Thank you for the detailed reviews.

We have revised the manuscript to improve readability and make the central claim clearer. We have reorganized the experiments, introduced definitions before use, removed repeated explanations, and clarified the purpose of the error-accumulation analysis. These changes clarify the wording and presentation of our original argument; they do not change our central claim or primary findings.

Our central claim remains that more accurate simulations do not guarantee better long-term forecasts, so their extra generation cost does not necessarily yield a clear forecasting benefit. RMI and SWE incur much greater generation costs without a clear aggregate forecasting gain: RMI's benefits depend on architecture, while SWE's overall improvement is limited. GLM forecasts become worse in the original degree comparison. Similar findings hold in the tested finer-source and larger-observation comparisons.

We have also extended the training experiments beyond GLM. Adjusting the loss improves models that previously gained little from more accurate data, and training on predicted histories improves SWE forecasts. These results show how training choices can increase the value of existing data. The additional experiments supplement the original evidence rather than replace the original results or redefine the central claim.

For existing data, the experiments provide training options that require no new trajectories. Before generating new data at scale, we recommend a small paired comparison of forecasting gains and costs.

We have highlighted the manuscript's core points and key clarifications in yellow to help reviewers quickly locate the central argument and its supporting evidence. These highlights provide a focused reading guide rather than mark every editorial change.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.