- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes PDE-KD, a population-level metric for neural simulators of chaotic PDEs. It is the KID-style cubic-kernel MMD computed in the 512-d mean-pool features of a ResNet-18. The network is trained with supervision on nine Well/PDEgym families to classify family and regime and to regress coefficients from single frames. The estimator treats the initial condition (IC) as the unit of independence by excluding same-IC pairs, and it serves as a permutation-test statistic. In my view it is a modest variant of KID.
Critical strengths
Strengths
- Important problem: after chaotic decorrelation, pointwise errors such as vRMSE cannot distinguish failure from plausible evolution and even favor blurred predictions (Fig. 1). The two-sample framing, which compares metrics by test power and n80, is a sound way to compare them across incomparable scales.
- Strong statistical rigor:
- IC-level U-statistic with same-IC exclusion, plus a clear demonstration of frame-level bias.
- Exact permutation tests with a code-level exactness check.
- Disclosed false-positive rates per cell.
- Untrained-network and PCA controls.
- Transparent and broad: negative and post-hoc results are openly reported, and the evaluation spans nine feature spaces, multiple PDE systems and real data.
Critical weaknesses
Weaknesses
- Limited novelty with mixed evidence for the learned features. The metric is essentially KID (cubic-kernel MMD) with a domain-trained encoder, an idea FPD already explored with Poseidon features, so the contribution depends on showing the new features are clearly better. They often aren't: an untrained copy of the network matches or beats PDE-KD on physical alignment; ImageNet features match or exceed it on perceptual disturbances, FSI and the unseen turbulent radiative layer; PCA-white reaches 80% power on more canonical spectral rungs.
- Post-hoc headline result. The main spectral claim in Table 2 relies on a "severe" ladder chosen for display after results were known (App. E.2). On the canonical ladder, PDE-KD's L2 power is only 0.07 for high-k injection and 0.10 for high-k removal.
- Generality is not directly tested. Most evaluated systems are training families, and Fig. 1's coefficient row is a training trajectory of a supervised target. About 80% of §4.2's trajectories come from the training split, and the held-out-family feature map was trained (App. I.6) but never scored with KD.
- Real-model evidence is thin and partly negative. Counting only orderings where both models are resolved from the reference floor, PDE-KD resolves 1 RB and 2 Euler pairs, against 4 and 4 for ImageNet (App. G.5). On WeatherBench-2, the only out-of-distribution model ladder, its ranking anti-correlates with published errors. This result and the random-init parity appear only in the ~75-page appendix, so the "general metric" framing of the main text overclaims.
Questions for authors
Questions
- Can you score the shear-flow-held-out feature map (App. I.6) with KD on shear flow, using the disturbance and Fig. 1 protocols? This is the most direct test of the generality claim.
- Can you report Table 2 on the canonical spectral ladder (or both ladders), and state in §4.1 how the severe ladder and the temporal columns were selected?
- Given that untrained and ImageNet features match PDE-KD in several analyses, what does physics supervision add? Is there a result where the trained features beat both by more than the variability across a few feature-map training seeds?
- How do you reconcile the WeatherBench-2 ranking reversal with the claim that PDE-KD is a general metric, and could this result be summarized in the main text?
2 comments
Thanks for clarification, I would like to increase my score accordingly.
Thank you for recognizing the importance of the problem and the statistical rigor of the evaluation. You found no substantive soundness concerns, and you call the two-sample framing "a sound way to compare" metrics. We ask you to reconsider the score, because that framing is half of our contribution.
We disagree that the paper is a modest KID variant. It contributes a practical metric for generative physical models and a framework for evaluating the metrics themselves: which physical discrepancies each metric detects, how many samples it needs, and how it ranks trained simulators. KID provides the statistic, and FPD applied Poseidon features to PDE fields. Neither asks these questions, and no earlier study of PDE metrics has.
PDE-KD does not need to outperform every metric on every task. We want a distributional metric for generative models after chaotic decorrelation. Such models can fail in many ways, so we need one score that detects a broad range of physical discrepancies without a new observable for each suspected error. Although each rival wins somewhere, each also has a blind spot in Table 2 that PDE-KD does not: ImageNet never reaches 80% power on complete low-wavenumber removal, where PDE-KD needs about 9 samples; PCA-white misses Gaussian blur; Random init misses four of the eight disturbances.
Direct paired comparisons resolve all four expected model orderings in each system. The counts you cite ask whether each model differs from the reference, which can exclude a useful comparison when one model is already close to it. Model selection asks a different question: do the two models' scores differ? We tested that directly for the Lola compression variants and the fine-tuned and pretrained Walrus models, on shared initial conditions. PDE-KD resolves all four expected orderings on RB and on periodic Euler after Holm correction, matching ImageNet; Random init resolves none on RB (Appendix C.4.4).
For training-data overlap in the initial-condition experiment, see our response to Reviewer RQGY, Q2.
Q1. Feature map trained without shear flow
We scored it on both requested protocols: coarse physics transfers, and fine sensitivity comes from training on the family (Appendix C.7.1).
Generality does not rest on one family: on HRRR weather, which resembles nothing it was trained on, PDE-KD detects all 11 kinds of spectral disturbance, more than any other feature space (Table 3, Appendix C.7).
Q2. Canonical ladder and the selection of Table 2's columns
The canonical injections were too weak for an informative comparison of the metrics, so Table 2 displays stronger ones. On the canonical ladder no feature space detects high-wavenumber injection, even at the strongest level. At level 2, where you quote 0.07 and 0.10, no per-frame feature space scores above 0.15 (Table 14). A test that every metric fails cannot rank them: the injected energy is within the natural variation between initial conditions. The revision reports both ladders (Appendix B.1), and Section 4.1 now states how the severe ladder and the temporal columns were selected. Although PCA-white detects more of the mild changes, PDE-KD still detects the most families at full strength on the canonical ladder (10 of 11).
Q3. What physics supervision adds
Physics supervision makes physical discrepancies detectable with far fewer samples. Yes, there is such a result. On complete low-wavenumber removal, PDE-KD reaches 80% power with about 9 samples (independent initial conditions). Random init needs about 33 (31 to 35 across three initializations), and ImageNet does not get there within 175. Although we trained the released map once, the gap is far outside the spread of the random initializations. Training also adds breadth: with the same input and architecture, Random init catches fewer spectral disturbance families than PDE-KD on every system of Table 3. For the intuition behind this, see our response to Reviewer aXjT, Q3.
Q4. WeatherBench-2
RMSE measures forecast skill; PDE-KD measures distributional realism. The reversal shows they are different things, not that PDE-KD fails to generalize. Although RMSE is the more appropriate measure for this forecast task, and our physical checks favor it, RMSE rewards smoothing: ERA5 blurred to a forecast's variability has the lowest RMSE at every lead. PDE-KD orders the four deterministic forecasters by the variability they keep, so it reverses the RMSE order at 36 of 40 leads (Appendix C.7.2). "General" means across systems and failure types, not a forecast-skill score. Section 4.3 now summarizes the result and says PDE-KD should not rank these forecasts by accuracy.