acceptodds
Under review as a conference paper at ICLR 2027

A Distributional Evaluation Metric for Chaotic Physical Systems

Abstract

Long-horizon predictions of chaotic physical systems eventually decorrelate from reference trajectories, making pointwise errors unreliable for distinguishing simulation failures from plausible alternative evolutions. We address this evaluation gap by comparing distributions of predicted and reference states using a limited number of independent trajectories. We introduce PDE-KD, a distributional evaluation metric for neural physics simulators based on a kernel discrepancy between predicted and reference distributions of states embedded in a learned feature space. To evaluate feature spaces for this application, we assess their statistical discriminative power under controlled perturbations of the spatial field and the initial conditions. PDE-KD is sensitive to disturbances across a broad range of spatial scales on several physical systems; it is sample efficient, requiring few trajectories to reliably detect discrepancies, and it reliably distinguishes conditional distributions. On trained neural simulators, it orders the models of each family consistently with measures of rollout quality, whereas pixel-space errors favor over-smoothed predictions. We thus present PDE-KD as a general, sample-efficient distributional evaluation metric that complements pointwise errors and targeted physical diagnostics for model evaluations.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Partial differential equation, chaotic dynamical systems, long-horizon prediction, scientific machine learning.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7MyoOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes PDE-KD, a population-level metric for neural simulators of chaotic PDEs. It is the KID-style cubic-kernel MMD computed in the 512-d mean-pool features of a ResNet-18. The network is trained with supervision on nine Well/PDEgym families to classify family and regime and to regress coefficients from single frames. The estimator treats the initial condition (IC) as the unit of independence by excluding same-IC pairs, and it serves as a permutation-test statistic. In my view it is a modest variant of KID.

Critical strengths

Strengths

  • Important problem: after chaotic decorrelation, pointwise errors such as vRMSE cannot distinguish failure from plausible evolution and even favor blurred predictions (Fig. 1). The two-sample framing, which compares metrics by test power and n80, is a sound way to compare them across incomparable scales.
  • Strong statistical rigor:
    • IC-level U-statistic with same-IC exclusion, plus a clear demonstration of frame-level bias.
    • Exact permutation tests with a code-level exactness check.
    • Disclosed false-positive rates per cell.
    • Untrained-network and PCA controls.
  • Transparent and broad: negative and post-hoc results are openly reported, and the evaluation spans nine feature spaces, multiple PDE systems and real data.
Critical weaknesses

Weaknesses

  • Limited novelty with mixed evidence for the learned features. The metric is essentially KID (cubic-kernel MMD) with a domain-trained encoder, an idea FPD already explored with Poseidon features, so the contribution depends on showing the new features are clearly better. They often aren't: an untrained copy of the network matches or beats PDE-KD on physical alignment; ImageNet features match or exceed it on perceptual disturbances, FSI and the unseen turbulent radiative layer; PCA-white reaches 80% power on more canonical spectral rungs.
  • Post-hoc headline result. The main spectral claim in Table 2 relies on a "severe" ladder chosen for display after results were known (App. E.2). On the canonical ladder, PDE-KD's L2 power is only 0.07 for high-k injection and 0.10 for high-k removal.
  • Generality is not directly tested. Most evaluated systems are training families, and Fig. 1's coefficient row is a training trajectory of a supervised target. About 80% of §4.2's trajectories come from the training split, and the held-out-family feature map was trained (App. I.6) but never scored with KD.
  • Real-model evidence is thin and partly negative. Counting only orderings where both models are resolved from the reference floor, PDE-KD resolves 1 RB and 2 Euler pairs, against 4 and 4 for ImageNet (App. G.5). On WeatherBench-2, the only out-of-distribution model ladder, its ranking anti-correlates with published errors. This result and the random-init parity appear only in the ~75-page appendix, so the "general metric" framing of the main text overclaims.
Questions for authors

Questions

  1. Can you score the shear-flow-held-out feature map (App. I.6) with KD on shear flow, using the disturbance and Fig. 1 protocols? This is the most direct test of the generality claim.
  2. Can you report Table 2 on the canonical spectral ladder (or both ladders), and state in §4.1 how the severe ladder and the temporal columns were selected?
  3. Given that untrained and ImageNet features match PDE-KD in several analyses, what does physics supervision add? Is there a result where the trained features beat both by more than the variability across a few feature-map training seeds?
  4. How do you reconcile the WeatherBench-2 ranking reversal with the claim that PDE-KD is a general metric, and could this result be summarized in the main text?
2 comments

Thank you for recognizing the importance of the problem and the statistical rigor of the evaluation. You found no substantive soundness concerns, and you call the two-sample framing "a sound way to compare" metrics. We ask you to reconsider the score, because that framing is half of our contribution.

We disagree that the paper is a modest KID variant. It contributes a practical metric for generative physical models and a framework for evaluating the metrics themselves: which physical discrepancies each metric detects, how many samples it needs, and how it ranks trained simulators. KID provides the statistic, and FPD applied Poseidon features to PDE fields. Neither asks these questions, and no earlier study of PDE metrics has.

PDE-KD does not need to outperform every metric on every task. We want a distributional metric for generative models after chaotic decorrelation. Such models can fail in many ways, so we need one score that detects a broad range of physical discrepancies without a new observable for each suspected error. Although each rival wins somewhere, each also has a blind spot in Table 2 that PDE-KD does not: ImageNet never reaches 80% power on complete low-wavenumber removal, where PDE-KD needs about 9 samples; PCA-white misses Gaussian blur; Random init misses four of the eight disturbances.

Direct paired comparisons resolve all four expected model orderings in each system. The counts you cite ask whether each model differs from the reference, which can exclude a useful comparison when one model is already close to it. Model selection asks a different question: do the two models' scores differ? We tested that directly for the Lola compression variants and the fine-tuned and pretrained Walrus models, on shared initial conditions. PDE-KD resolves all four expected orderings on RB and on periodic Euler after Holm correction, matching ImageNet; Random init resolves none on RB (Appendix C.4.4).

For training-data overlap in the initial-condition experiment, see our response to Reviewer RQGY, Q2.

Q1. Feature map trained without shear flow

We scored it on both requested protocols: coarse physics transfers, and fine sensitivity comes from training on the family (Appendix C.7.1).

Shear flowReleased mapTrained without shear flow
Reynolds number 10⁵ → 10⁴ (Fig. 1 protocol)separatedseparated
Reynolds number 10⁵ → 5×10⁴ (Fig. 1 protocol)separatednot consistently
Spectral disturbance families caught (of 11)94

Generality does not rest on one family: on HRRR weather, which resembles nothing it was trained on, PDE-KD detects all 11 kinds of spectral disturbance, more than any other feature space (Table 3, Appendix C.7).

Q2. Canonical ladder and the selection of Table 2's columns

The canonical injections were too weak for an informative comparison of the metrics, so Table 2 displays stronger ones. On the canonical ladder no feature space detects high-wavenumber injection, even at the strongest level. At level 2, where you quote 0.07 and 0.10, no per-frame feature space scores above 0.15 (Table 14). A test that every metric fails cannot rank them: the injected energy is within the natural variation between initial conditions. The revision reports both ladders (Appendix B.1), and Section 4.1 now states how the severe ladder and the temporal columns were selected. Although PCA-white detects more of the mild changes, PDE-KD still detects the most families at full strength on the canonical ladder (10 of 11).

Q3. What physics supervision adds

Physics supervision makes physical discrepancies detectable with far fewer samples. Yes, there is such a result. On complete low-wavenumber removal, PDE-KD reaches 80% power with about 9 samples (independent initial conditions). Random init needs about 33 (31 to 35 across three initializations), and ImageNet does not get there within 175. Although we trained the released map once, the gap is far outside the spread of the random initializations. Training also adds breadth: with the same input and architecture, Random init catches fewer spectral disturbance families than PDE-KD on every system of Table 3. For the intuition behind this, see our response to Reviewer aXjT, Q3.

Q4. WeatherBench-2

RMSE measures forecast skill; PDE-KD measures distributional realism. The reversal shows they are different things, not that PDE-KD fails to generalize. Although RMSE is the more appropriate measure for this forecast task, and our physical checks favor it, RMSE rewards smoothing: ERA5 blurred to a forecast's variability has the lowest RMSE at every lead. PDE-KD orders the four deterministic forecasters by the variability they keep, so it reverses the RMSE order at 36 of 40 leads (Appendix C.7.2). "General" means across systems and failure types, not a forecast-skill score. Section 4.3 now summarizes the result and says PDE-KD should not rank these forecasts by accuracy.

Thanks for clarification, I would like to increase my score accordingly.

Reviewer_aXjTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes using an MMD metric applied to a learned feature space to evaluate the quality of PDE surrogate models.

Critical strengths

The paper provides a nice exposition of the methodology.

Critical weaknesses

Ultimately, the proposed metric just doesn't appear to be very useful compared with standard evaluation methods. There is also no justification or comparison with other standard distributional metrics. Unlike for images, the metrics used for comparing prediction quality for PDEs are often physically and statistically motivated, providing an interpretable measure of quality that is critical for scientific applications. Replacing them with a black-box feature space metric by borrowing ideas from computer vision just doesn't seem scientifically useful if the primary goal is trustworthy evaluation.

Questions for authors
  1. Pixel-space errors are not the primary evaluation tool for most chaotic systems, so they are a weak baseline. Does your metric have any clear benefits over standard physical or statistical distribution-based metrics?
  2. Can you demonstrate how to use your metric to provide insight into model quality that can't be done with standard distributional metrics?
  3. How are you interpreting your metric physically, and does it have the same interpretation across different PDEs?
  4. In Fig 1, PDE-KD seems to think the symmetric augment variant is somehow less similar than one with a different noise seed. This doesn't make sense to me. The spectral error seems much more reasonable?
  5. How broadly generalizable is it, e.g. what if we give it a PDE not in the training set for the feature extractor?
5 comments

Thank you for pressing us on what this metric is for. The paper did not say it plainly enough: our goal is to help build generative simulators that reproduce the reference distribution after chaotic decorrelation.

We need a holistic score. When we evaluate a generative simulator, we often do not know in advance which physical diagnostic will be most informative. So we start with PDE-KD, a single score that compares the generated and reference distributions. If the discrepancy exceeds what we typically see between reference samples, we turn to targeted physical diagnostics to find which properties of the fields differ. PDE-KD does not replace those diagnostics; it tells us when to reach for them.

A physically motivated metric is trustworthy only for the failure it measures. A generative model does not fail on request. We therefore test two distinct ways a generated distribution can be wrong: off-manifold damage to the fields (lost structure, wrong energy across scales) and on-manifold shifts, where every solution is valid but the population is wrong. For each, statistical power tells us which discrepancies a metric reliably detects and how many independent samples it needs.

The review also says the paper has no comparison with standard distributional metrics. It has several: power spectra, energy distance, CRPS, a Fréchet distance, a buoyancy profile and physical diagnostics (Figure 1, Section 4, Appendices A.3.4, B.2, C.2, C.3). We ask you to reconsider your score.

Q1

Yes. Existing metrics each test only a subspace of plausible failures; PDE-KD is the only one that detects every disturbance in Table 2. Take the power spectrum. Our phase-scrambling experiment changes the Fourier phases and leaves the spectrum untouched, yet it destroys the field's spatial organization (Table 2 shows the fields). Physical observables have the same blind spots. We put each one through the same distributional test as PDE-KD:

RB disturbance (Table 2, level 2)Physical observablePDE-KD powerObservable power
Gaussian blurNusselt number0.890.16
High-wavenumber removalKinetic energy1.000.06

With 175 samples, a diagnostic of heat transport can miss a loss of spatial detail, and a diagnostic of kinetic energy can miss a loss of small-scale structure. At full strength PDE-KD detects all eight disturbances; the Nusselt number detects six, kinetic energy five and enstrophy three (Appendix B.2).

Q2

Yes. PDE-KD is a many-in-one test for physical failures, and it changes which model a developer would pick. Two examples from released simulators (Section 4.3):

  1. Compression. On RB, compressing the Lola models harder improves conventional forecast scores, including ensemble CRPS, while the rollouts lose thin plumes and small-scale structure (Appendix C.8). PDE-KD says the opposite: from lead 21 onward, in all 16 comparisons that both scores resolve, it prefers less compression (Appendix C.4.4). A developer choosing by forecast scores alone would pick the degraded model.
  2. Fine-tuning. On periodic Euler, PDE-KD ranks fine-tuned Walrus ahead of the pretrained model, while power spectra rank them the other way. Two physical diagnostics side with PDE-KD: after fine-tuning, rollouts in which total entropy decreases fall from 82.4% to 33.6%, and the equation-of-state residual also decreases (Table 20, Appendix C.4.4).

Q3

Intuition: predicting PDE coefficients teaches the feature map flow morphology and characteristic scales. The feature map is trained to predict PDE family, coefficients and regime from single frames. Take the Reynolds number, which measures the relative importance of inertia and viscosity: predicting it may encourage the network to recognize changes in vortex size and flow organization.

The interpretation is the same across PDEs. Each score is read against the spread between reference samples of its own system, which says whether the generated population can be told apart from the reference.

Q4

Both scores you compare sit inside the seed-to-seed range, so their order is not meaningful. Spectral error is not more reasonable: it gives a physically invalid mirror the same zero. We revised Figure 1 to show this. It now shows the range of scores between noise seeds and adds a mirror image that is not a valid solution. PDE-KD keeps the valid symmetric copy within seed-to-seed variation and scores the invalid mirror above it; spectral error reads zero for both (Appendix C.2).

Q5

It transfers at the coarse level. Although it misses a twofold change, a feature map trained without any shear-flow examples still separates a tenfold Reynolds-number change on shear flow (Appendix C.7.1; see our response to Reviewer 7Myo, Q1).

Please let us know if there are any further questions.

Thank you for the careful response. Positioning the method as a "screening" test prior to additional analysis is probably a better framing for what you are aiming to do. However, I do not think there is sufficient evidence that this kind of metric is much more informative than traditional physical metrics, which are more interpretable, have known failure modes, and perform just as well when taken together (as they should when evaluating surrogate models).

Q1: Yes, all metrics have blind spots. The important thing is that they are understood and can be covered by other metrics. Your approach has them too, but they are difficult to characterize clearly.

Q2: The two examples given here do not really answer the question. In Table 4, for Lola, the degradation in c4 is visible across many metrics, including the spectrum. The difference between c64 and c16 is small, and this is reflected across all the metrics. For Walrus, both FT and PT are both rather poor, which is also reflected in the physical metrics. There is no evidence here that PDE-KD provides any significant added value.

Q3: Again, this vague description of the metric is not really interpretable or actionable. You would still need to run a suite of physical metrics to understand what is happening.

Q4: If a value of 0.16 is not meaningful in Figure 1, it's hard to understand what is meaningful here. For example, PDE-KD fails to flag how poorly Walrus FT performs in Table 4, scoring it close to Lola-c16 and better than Lola-c4, even though it is clearly performing poorly according to the physical metrics. The point remains that a suite of physical metrics is clearly informative and has better-understood failure modes (which allow you to engineer the examples here) compared to PDE-KD.

Q5: The claimed generalization here is rather weak, so clearly physical metrics are preferred.

AuthorsOpenReview ↗

Point-by-point response to "Comment by Reviewer aXjT"

Thank you for the follow-up. We agree that screening before targeted diagnostics is the right description of the role of PDE-KD. Table numbers refer to the revised PDF.

Your follow-up compares PDE-KD with physical metrics "taken together". That suite has to be chosen before it can be compared, and what it shows depends on the choice. Our separate comment in this thread, Which physical suite?, takes this up.

PDE-KD adds one reusable screening test with broad, measured sensitivity. It complements a suite of diagnostics and does not replace one; explaining a failure remains the role of physical diagnostics. Three results support this:

  1. The same test on every system. The required physical diagnostics change with the system: the Nusselt number on Rayleigh–Bénard (RB), entropy and the equation of state on Euler (Table 19). PDE-KD runs unchanged on the five systems of Table 3 and rejects at least 9 of 11 spectral disturbance families on each.
  2. No choice of diagnostics needed. You noted that other metrics cover each blind spot. They do, given the right suite. On RB, the Nusselt number, kinetic energy and enstrophy all miss Gaussian blur and high-wavenumber removal (Table 15). The screen needs no such choice and reaches 80% power on all eight disturbances of Table 2 at full strength, with at most 42 initial conditions.
  3. One screening decision. PDE-KD answers a single question with a single test at the 5% level: does the generated population differ from the reference? A suite needs one test per diagnostic, or a rule that combines them.

On your specific points:

Q1: A blind spot is known in kind; its size has to be measured, for a physical diagnostic as for PDE-KD. Table 15 measures both with one test and one sample budget. At level 2, the Nusselt number has power 0.16 on blur and PDE-KD 0.89; on phase scrambling it is the reverse, 1.00 against 0.21.

Q2: PDE-KD adds a single metric that orders every expected model pair on both systems. To clarify our first reply: its compression example was on Rayleigh–Bénard (Table 21); Table 4 is Euler. On each system, we expect less compression to score better (three Lola pairs) and fine-tuning to score better (one Walrus pair). PDE-KD orders all eight pairs as expected, each significantly (paired test, Appendix C.4.4); power-spectrum KD orders three of four on each. On the Euler Walrus pair, the physical diagnostics split two against two (Table 20); PDE-KD favors fine-tuning.

Q2: On Euler, PDE-KD resolves a physically substantial difference between c16 and c64 that forecast scores barely reflect. Total entropy decreases, a violation of the second law, in 1.2% of Lola-c64 and 14.4% of Lola-c16 rollouts (Table 20). Table 4 makes the difference look small because it rescales each row to [0, 1]. PDE-KD scores c64 at 1.43, not resolved from its split-half floor, and c16 at 5.09; a paired test resolves the pair. vRMSE and CRPS change by less than 1% and favor c16.

Q3: The score is interpreted as a test of distinguishability from the reference population; this meaning is unchanged across PDE systems. The null hypothesis is that generated and reference fields come from the same distribution. A permutation test over initial conditions gives the p-value, and we check its 5% false-positive rate on clean-versus-clean data (Appendix A.3.5). So the score is actionable: a rejection is a signal to run the diagnostics.

Q4: A score is meaningful when it lies outside the spread between reference samples. Figure 1 shows that spread for each metric over 45 seed pairs. The valid symmetric copy, whose score you ask about, is inside that spread (permutation test, p = 0.30; Appendix C.2). The blur scores about five times the top of the spread and the invalid mirror about ten times. In Table 4, the scores that don't clear their split-half floor are in parentheses.

Q4: PDE-KD places Walrus-FT away from the reference. It is not a positivity check and does not replace one. Walrus-FT's score clears its split-half floor (Table 4, no parentheses). Positivity is a hard constraint: one cell with nonpositive density or pressure makes a state unphysical, which happens on 39.4% of initial conditions for Walrus-FT and on none for any Lola model (Table 20). Where positivity is required that settles the choice, and a comparison of whole-field statistics is not what catches it. Although three of the four Euler diagnostics put Lola-c4 ahead of Walrus-FT, entropy decrease puts Walrus-FT ahead (33.6% against 57.6%), as PDE-KD does. On RB, PDE-KD scores Walrus-FT behind every Lola model, as the consensus does (a paired test resolves two of three pairs).

Q5: PDE-KD transfers to data outside its training set; physical diagnostics are specific to each system. On HRRR weather data, which is outside the training set of PDE-KD and, unlike its training data, assimilates real observations, it detects every disturbance family tested (Table 26).

Reviewer_RQGYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PDE-KD compares distributions of physical states using a cubic kernel on features trained to predict PDE properties. The paper tests disturbance detection, initial-condition differences, and simulator rankings. It addresses a relevant problem: after chaotic decorrelation, pointwise error may favor smooth predictions over plausible sharp ones.

Critical strengths
  1. Same-IC exclusion, sample-count checks, and power comparisons at a fixed false-positive rate make the metric easier to interpret than a raw feature distance. The paper distinguishes marginal distribution agreement from individual forecast accuracy.

  2. The experiments cover several physics settings and include random networks, ImageNet, spectra, pixels, and PCA. They extend evidence on feature-space evaluation of PDE solutions. The reported dependence on disturbance severity, small-sample failures, and weather-ranking limitations help establish the scope of the method.

Critical weaknesses
  1. The Poseidon comparison does not isolate representation quality. Appendix C.1 and Table 15 state that Euler energy is placed in its density input and momentum in its velocity inputs; actual density is dropped, and inputs are resized to 128 × 128. The adapter may therefore disadvantage the comparator. The 3D/JEPA comparisons also differ in crops, clip lengths, splits, and training. These disclosed differences still limit the comparative claim.

  2. The main IC test does not establish performance on data excluded from feature training. Appendix F.1 says 80% of its trajectories were used for feature training, with no held-out control for this permutation design. F.2 uses a different estimator and retains thresholds from the full pool, so it does not resolve this concern. The horizontal-mean buoyancy profile also reaches power 0.975 versus about 0.895 for PDE-KD. The added value of the learned features needs to be established for this test.

Questions for authors
  1. On the existing Euler evaluation, does the comparison change when the Poseidon adapter preserves density and follows the pretrained variable meanings? This limited check would help separate the adapter effect from the feature comparison.

  2. Can the main IC test use comparison groups and calibration data excluded from feature training, with the same estimator? If the available pool is too small, please state the limitation. Please include the buoyancy-profile reference with that result.

3 comments

Thank you for recognizing the statistical rigor and breadth of our experiments. Your review rates the soundness concerns as minor or fixable and names two: the Poseidon input adapter and training exposure in the initial-condition (IC) test. We ran both checks you asked for, and neither changes a conclusion. With both concerns resolved, we ask you to reconsider your score. All section, table and appendix numbers below refer to the revised PDF.

Your summary also understates two contributions.

Statistical power is a metric of metrics. Scores from different metrics live on different scales and cannot be compared directly. Power can: it measures how reliably each metric detects the same planted discrepancy, and how many samples it needs. Section 4.1 uses it on off-manifold disturbances that damage the fields, and Section 4.2 on on-manifold differences between populations evolved from different initial conditions. The same test applies to any future metric.

After chaotic decorrelation, populations are the right thing to compare (Section 2.1). A prediction can drift away from its reference trajectory and still be physically plausible. So we ask whether the generated population reproduces the reference distribution.

W1, Q1. Poseidon and 3D/JEPA comparisons

Correcting Poseidon's inputs does not change the result: PDE-KD still catches far more disturbance families. You were right that our adapter could disadvantage Poseidon, so we reran the Euler evaluation with density, velocity and pressure in Poseidon's own normalization and changed nothing else. The table counts the disturbance families each feature space catches (its test rejects at the 5% level):

Euler-periodicPerceptual L2Spectral L5
PDE-KD4/410/11
ImageNet4/49/11
Poseidon, original inputs2/44/11
Poseidon, matched inputs1/46/11

The correction changes which disturbances Poseidon catches, not its rank. The 128×128 resize is Poseidon's own pretraining resolution. Input handling does not explain the gap either: Random init receives exactly PDE-KD's input and catches far fewer families (Table 3). Appendix A.2.1 has the details and the false-positive checks.

Matching the IC split does not change the JEPA result. Re-scored on the split of PDE-KD (3D), JEPA still catches 17 of 35 temporal disturbance levels, against 23 for PDE-KD (3D), and still does not reach 80% power on either frame-swap disturbance (Appendix A.2.3). Crop, clip length and training still differ, but crop and clip length cannot explain the frame-swap result: a swap moves whole frames, so no crop hides it, and JEPA's shorter clip makes a swap a larger change, not a smaller one.

W2, Q2. Training exposure and the buoyancy profile

Training exposure does not explain the IC result: PDE-KD separates the populations as often on held-out trajectories as on training ones. The test never used a training label. The feature map is trained to predict Ra and Pr; the test holds both fixed and varies the initial temperature profile, which is not a training target. For the held-out check we kept the same estimator and drew both the comparison groups and the calibration data from trajectories excluded from feature training. For each of the 34 testable coefficient settings we asked whether the profile levels differ, and repeated the test on matched draws of training trajectories:

Final window (frames 180–199)Held-out settings separated (of 34)Training draws separated
Buoyancy profile (reference)3397%
PDE-KD3188%
Random init (mean of 3 seeds)29.380%
Poseidon2878%
ImageNet1534%

Averaged over frames 60–199, PDE-KD separates the profiles in 95.4% of held-out settings and 93.5% of training draws. Although the held-out pool is too small for the original five-per-arm design, as you anticipated, the four profile pairs that can be tested agree: PDE-KD separates three of them in every time window (Appendix C.3).

Although the buoyancy profile measures exactly what we varied, PDE-KD comes closest to it among the feature spaces without being told what varies. The profile works only when the mechanism is known in advance, is specific to RB, and cannot respond to any change with zero horizontal mean. The same features, unchanged, also detect the spatial disturbances of Section 4.1. That is the added value: a simulator can produce valid solutions in the wrong proportions, and one common metric catches it without a new observable for each failure.

We thank the reviewer for helping us improve the paper. Please reconsider the score if we address the concerns.

All section, table and figure numbers in our response refer to the revised PDF (uploaded 15 Nov). The appendix was reorganized, so the locations cited in your review have moved:

  • Poseidon inputs (was Appendix C.1, Table 15): Appendix A.2.1, Table 10
  • JEPA on the matched split: Appendix A.2.3
  • Held-out initial-condition test (was Appendix F): Appendix C.3, Table 17, with the buoyancy profile as a row

New analysis on the initial-condition test: PDE-KD detects 31 of 34 held-out population differences; the RB physical diagnostics taken together detect 19 to 28.

Your review notes that the buoyancy profile outperforms PDE-KD on this test, and asks what PDE-KD adds there. Our first reply argued that the profile works only when the mechanism is known in advance. We have now measured this.

We ran the other RB physical diagnostics through the same held-out test (the design of Table 17). Each row is a single test at the 5% level; the combined rows merge their diagnostics into one test. The table reports the last window of the record (frames 180–199), as Table 17 does.

TestCoefficient cells rejecting, of 34
PDE-KD31
Nusselt number alone2
Five RB scalars (Table 19) and power spectrum, combined19
The same with buoyancy and heat-flux profiles28
Buoyancy profile alone (as in Table 17)33

PDE-KD rejects in 31 of the 34 cells without being told what varies. Run as a whole suite, the diagnostics reject in 28 cells with the profiles and in 19 without, because combining tests costs power. The order is the same over all windows from frame 60 onward: the mean rejection rate is 0.95 for PDE-KD, against 0.85 and 0.65 for the two combined suites.

The buoyancy profile alone still does better (33 cells, mean rate 0.98), as Table 17 reports. Its advantage belongs to that one diagnostic, chosen with knowledge of what was varied. The Nusselt number, the standard RB diagnostic, rejects in 2. So on this test PDE-KD adds a detection that needs no diagnostic chosen for it.

This analysis is new and covers one system. It will be added to Appendix C.3 in the final version of the paper. The table is the same one as in our comment of 18 Nov to Reviewer aXjT.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.