- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the adaptation of neural PDE forecasters from simulation training to real experimental data. The authors propose FROST, an adaptation scheme that freezes the forecaster and trains a corrector to learn the conditional distribution of its residual, given the measured history and the forecast, using stochastic interpolants. On five RealPDEBench systems, the resulting ensemble improves both the relative L2 error of its mean and the fair CRPS over the fine-tuned forecaster.
Critical strengths
Extensive real-data evaluation. The method is evaluated on five real experimental systems, including parameter settings held out from the real training data. FROST improves both accuracy and fair CRPS on seen and held-out parameters, supporting the claim that it is an adaptation method that generalizes across different real-data shifts.
Backbone-agnostic correction. On the cylinder and public foil datasets, FROST is applied to six frozen forecasters, including task-specific architectures and one pretrained foundation model, and improves all of them. This supports FROST as a general post-hoc module that does not require a specific architecture.
Critical weaknesses
Weak calibration evidence. Headline results use only K=4 ensemble members, and the paper's own calibration metrics (spread/skill ratios and Gaussian coverage) indicate strong under-dispersion. Moreover, the main probabilistic metric, the fair CRPS, is a marginal score and therefore ignores the spatio-temporal dependence that the paper itself motivates. This makes it unclear whether FROST learns a calibrated residual distribution.
Weakly supported sim-to-real framing. A forecaster trained only on real data and a weak simulation-only forecaster both reach, once corrected, almost the same accuracy as the simulation-pretrained and fine-tuned one. The corrected error barely depends on the base forecaster, and no generator of equal capacity trained on the history alone is reported on the main systems. We therefore cannot assess whether the gains come from a large corrector that compensates for weak forecasts, rather than from correcting a sim-to-real mismatch.
Questions for authors
-
Can the authors report results on the main systems for a generator of the same capacity, conditioned only on the history (without the forecast) and trained with the same budget as FROST?
-
Can the authors compare FROST with other probabilistic correctors, such as score-based diffusion or a CRPS-trained corrector?
-
Can the authors report evaluations with more ensemble members and with dependence-sensitive scores such as the Variogram Score?
-
Can the authors clarify whether FROST is a general correction method, or what specifically distinguishes it for sim-to-real adaptation?
7 comments
W2. Weakly supported sim-to-real framing. A forecaster trained only on real data and a weak simulation-only forecaster both reach, once corrected, almost the same accuracy as the simulation-pretrained and fine-tuned one. The corrected error barely depends on the base forecaster, and no generator of equal capacity trained on the history alone is reported on the main systems. We therefore cannot assess whether the gains come from a large corrector that compensates for weak forecasts, rather than from correcting a sim-to-real mismatch.
FROST is motivated by the analysis in Section 2: a forecaster trained by regression predicts the conditional mean, so it cannot represent the stochastic part of real measurements, and its errors on real data are systematic. Table 2 shows that this gap appears for every forecaster architecture we tested and comes from the regression objective rather than from the training data mixture. The corrected errors are close across forecasters because the corrector is trained on an ensemble of forecasters and models what all of them miss.
To isolate the role of the forecast, we trained a history-only generator with the same network, data and training budget as FROST. It generates the future field directly from the history, while FROST uses the time-resolved forecast as a condition and generates a residual around it. Both use four members and six corrector calls per member per block. On all five systems, FROST has 1.5–4.2× lower rollout error and 1.3–6.5× lower fair CRPS than the history-only generator.
Table R3. History-only generator against FROST: the same network, data and training budget, four members and six corrector calls per member per block; FROST uses the time-resolved forecast as a condition, the history-only generator does not. Cylinder on all 320 windows. (supplementary Table S6)
| System | History only: rollout L2 | FROST: rollout L2 | Ratio | History only: fair CRPS | FROST: fair CRPS | Ratio |
|---|---|---|---|---|---|---|
| Cylinder | 0.3875 | 0.1571 | 2.47× | 0.04708 | 0.01772 | 2.66× |
| Controlled cylinder | 0.6203 | 0.1490 | 4.16× | 0.04893 | 0.00826 | 5.92× |
| FSI | 0.5120 | 0.2134 | 2.40× | 0.03825 | 0.01264 | 3.03× |
| Public foil | 0.0774 | 0.0298 | 2.60× | 0.02213 | 0.00339 | 6.53× |
| Combustion | 1.1395 | 0.7750 | 1.47× | 0.00929 | 0.00694 | 1.34× |
To test whether a large corrector compensates for weak forecasts, we reduced the corrector from 41 M to 10 M parameters. Rollout error changes by at most 1.6% and fair CRPS by at most 2.4% on all five systems, at six and at thirty steps (Table R4). On cylinder we went further, and every corrector from 0.6 M to 163 M keeps the six-step rollout error below the FNO's.
Table R4. The 41 M corrector of the paper against a 10 M corrector of the same architecture; rollout relative L2 / fair CRPS, four members (supplementary Table S7)
| System | Steps | 41 M | 10 M | Change in L2 | Change in CRPS |
|---|---|---|---|---|---|
| Cylinder | 6 | 0.1571 / 0.01772 | 0.1579 / 0.01810 | +0.5% | +2.1% |
| 30 | 0.1535 / 0.01658 | 0.1549 / 0.01698 | +0.9% | +2.4% | |
| Controlled cylinder | 6 | 0.1490 / 0.00826 | 0.1500 / 0.00827 | +0.7% | +0.1% |
| 30 | 0.1478 / 0.00729 | 0.1502 / 0.00740 | +1.6% | +1.4% | |
| FSI | 6 | 0.2134 / 0.01264 | 0.2149 / 0.01278 | +0.7% | +1.1% |
| 30 | 0.2147 / 0.01208 | 0.2162 / 0.01221 | +0.7% | +1.1% | |
| Public foil | 6 | 0.0298 / 0.00339 | 0.0296 / 0.00331 | −0.6% | −2.4% |
| 30 | 0.0298 / 0.00316 | 0.0300 / 0.00315 | +0.6% | −0.4% | |
| Combustion | 6 | 0.7750 / 0.00694 | 0.7740 / 0.00688 | −0.1% | −0.8% |
| 30 | 0.7722 / 0.00649 | 0.7733 / 0.00645 | +0.1% | −0.5% |
Q1. Can the authors report results on the main systems for a generator of the same capacity, conditioned only on the history (without the forecast) and trained with the same budget as FROST?
Yes. Please see our response to W2: with the same capacity and training budget, FROST has 1.5–4.2× lower rollout error and 1.3–6.5× lower fair CRPS than the history-only generator on all five systems.
Q2. Can the authors compare FROST with other probabilistic correctors, such as score-based diffusion or a CRPS-trained corrector?
We trained two additional probabilistic correctors with the same data. First, an EDM residual-diffusion corrector with FROST's backbone, residual target, and history-and-forecast conditioning. Six-call FROST has lower rollout error and CRPS than seven-call EDM on all five systems, and at eleven calls each FROST has the lower rollout error on all five.
Table R5. EDM residual corrector against FROST, with the same network, residual target, measured-history and time-resolved-forecast conditioning, training data and random seeds. Mean rollout relative L2 and fair CRPS (lower is better), four members, three seeds; each call is a corrector evaluation per member per prediction block; cylinder on all 320 windows. (supplementary Table S8)
| System | FROST, 6 calls | FROST-EDM, 7 calls | FROST, 11 calls | FROST-EDM, 11 calls | FROST-EDM, 19 calls |
|---|---|---|---|---|---|
| (a) Rollout relative L2 ↓ | |||||
| Cylinder | 0.1571 | 0.2131 | 0.1548 | 0.1658 | 0.1591 |
| Controlled cylinder | 0.1490 | 0.3354 | 0.1472 | 0.2044 | 0.1641 |
| FSI | 0.2134 | 0.2642 | 0.2136 | 0.2251 | 0.2196 |
| Public foil | 0.0298 | 0.0596 | 0.0299 | 0.0328 | 0.0310 |
| Combustion | 0.7750 | 1.1863 | 0.7711 | 0.8201 | 0.7828 |
| (b) Fair CRPS ↓ | |||||
| Cylinder | 0.01772 | 0.02060 | 0.01717 | 0.01708 | 0.01700 |
| Controlled cylinder | 0.00826 | 0.01476 | 0.00774 | 0.00886 | 0.00764 |
| FSI | 0.01264 | 0.01393 | 0.01230 | 0.01194 | 0.01199 |
| Public foil | 0.00339 | 0.00643 | 0.00327 | 0.00359 | 0.00358 |
| Combustion | 0.00694 | 0.01258 | 0.00674 | 0.00619 | 0.00614 |
Second, we trained a one-call corrector directly under fair CRPS on cylinder, with the same network, history-and-forecast conditioning, training split and update count as FROST (three seeds). It generates each member in a single forward pass from noise and is trained on the fair CRPS of four samples per training example.
Table R6. Cylinder, four members, mean over three seeds, as trained (no calibration). All 320 test windows. (supplementary Table S9)
| Method | Rollout L2 ↓ | Fair CRPS ↓ |
|---|---|---|
| Fine-tuned FNO | 0.1632 | 0.02201 |
| CRPS-trained corrector | 0.1616 | 0.01791 |
| FROST, 6 steps | 0.1571 | 0.01772 |
| FROST, 30 steps | 0.1535 | 0.01658 |
Training directly on fair CRPS does not close the gap. FROST is ahead of it on both.
Q3. Can the authors report evaluations with more ensemble members and with dependence-sensitive scores such as the Variogram Score?
Yes. Please see our response to W1. From 4 to 16 members, fair CRPS changes by at most 0.2%, spread/skill by at most 0.04, and ensemble-mean rollout error improves by 1.1–4.5% on the five systems at six and thirty steps; Table R1 adds the calibrated 16-member results. On the order-0.5 variogram score, thirty-step FROST is lowest on cylinder, controlled cylinder, FSI and public foil and level with EDM diffusion on combustion (Table R2), and shuffling members between blocks worsens it on four systems.
Q4. Can the authors clarify whether FROST is a general correction method, or what specifically distinguishes it for sim-to-real adaptation?
FROST is a general forecast-conditioned correction interface that we evaluate primarily for sim-to-real PDE forecasting, where a simulation-trained operator must be adapted using scarce, partial, and noisy measurements. It learns a distribution over discrepancies between observed future states and the frozen forecaster's time-resolved predictions.
Thank you again for taking the time to review our paper and for your valuable feedback! We're happy to provide any further clarification.
References
- Johnson, C., and Bowler, N. (2009). On the reliability and calibration of ensemble forecasts. Monthly Weather Review, 137(5), 1717–1720.
- Van Schaeybroeck, B., and Vannitsem, S. (2015). Ensemble post-processing using member-by-member approaches: theoretical aspects. Quarterly Journal of the Royal Meteorological Society, 141(688), 807–818.
Response to Reviewer ro2v
Thank you for highlighting our thorough experimentation, generalization to held-out physical parameters, and the ability to correct multiple forecasters!
We believe the under-dispersion comes from the sim-to-real setting itself. The corrector is trained on a few hundred measured trajectories, and with this little data and few sampling steps the flow sampler places its members close to the conditional mean. Consistent with this, increasing the steps from six to thirty raises FROST's spread/skill on all five systems without retraining.
Because the mismatch is mostly one of scale, FROST is compatible with a common post-hoc calibration technique (Johnson and Bowler, 2009; Van Schaeybroeck and Vannitsem, 2015), which rescales each ensemble member's deviation from the ensemble mean using a validation-set-derived factor, fitted for each system and seed on validation windows of the training recordings. Once calibrated, six-step FROST's spread/skill is 0.75–0.98 on controlled cylinder, FSI, public foil and combustion and 0.96 on cylinder without the outlier recording (Table R1). The result is robust to the ensemble size: from 4 to 16 members, rollout error improves by 1.1–4.5% at six and thirty steps, and the calibrated spread/skill is the same (Table R1).
Table R1. Rollout relative L2 / spread/skill (ideal 1), six steps. Cylinder on all 320 windows, with spread/skill on the 314 windows without the outlier recording in brackets. (supplementary Table S4)
We also evaluated the order-0.5 variogram score over spatial, temporal and space–time pairs of grid points. It is a proper multivariate score: it compares how the differences between pairs of values vary across members with how they vary in the measurement, so an ensemble with the right marginals but the wrong correlations is penalised, which fair CRPS cannot detect.
Table R2. Order-0.5 variogram score in units of 10⁻³ (lower is better), four members unless stated, mean over three seeds; cylinder on all 320 windows. Bold marks the lowest four-member value; the score improves with the member count, so the 16-member row is shown for reference. (supplementary Table S5)
Thirty-step FROST has the lowest four-member variogram score on cylinder, controlled cylinder, FSI and public foil, and is level with EDM diffusion on combustion (6.89 against 6.88). It lowers the score of the forecaster it corrects on all five systems, from 2.78 to 1.95 on controlled cylinder and from 2.10 to 0.97 on public foil, and with 16 members it is lower again on every system. As a direct test of temporal dependence, shuffling member identities between successive rollout blocks, which keeps every block's marginals, worsens FROST's cross-block variogram score on four of five systems. FROST's members therefore carry dependence across space and time beyond what fair CRPS measures.