- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper considers a benchmark for the problem of mismatched mask distributions between training and testing in completion tasks on PDE datasets. The paper attempts to construct multiple PDE systems and various mask distributions, and considers two tasks for evaluation. The overall results are also consistent with intuition, showing that in-distribution performance is much better than OOD performance.
Critical strengths
Comprehensive and well-structured benchmark. The benchmark has a clear problem formulation and covers diverse evaluation settings, including multiple PDE systems, prediction tasks, observation patterns, and backbone architectures. This provides a systematic framework for evaluating model robustness under different observation conditions.
Critical weaknesses
-
Limited evaluation on real-world data. The benchmark relies entirely on simulated PDE systems, without evaluations on real-world observations. This limits its practical relevance, especially since real-world measurements often involve noise, irregular sensor placements, and other complexities not captured by the current benchmark.
-
Limited insights from the experimental results. The main findings are largely expected: models perform substantially better when the training and testing mask distributions match than when they differ. While the experiments confirm this phenomenon across different settings, they provide limited new insights into why models fail under distribution shifts or how such failures could be mitigated. As a result, the scientific takeaways beyond the construction of the benchmark are relatively limited.
-
Insufficient motivation for studying mask distribution shifts. The practical motivation for this benchmark is not sufficiently convincing. The paper assumes access to complete PDE solutions during training, allowing arbitrary observation masks to be generated. If the mask distribution at inference time is known, why not simply train models using the same distribution? If the inference-time distribution is unknown or variable, training with a mixture of observation patterns seems like a natural and straightforward solution. Although the paper includes preliminary mixed-pattern training experiments, it does not sufficiently explain why these strategies are inadequate or why train-test mask distribution mismatch represents an important practical challenge under its assumptions. The authors should better justify the scenarios in which such distribution shifts are unavoidable and explain why the proposed benchmark is necessary.
Questions for authors
-
Could the authors clarify the practical scenarios in which evaluating robustness to out-of-distribution (OOD) observation masks is particularly useful? The paper considers changes in observation patterns, such as sensor failures, as a motivation for studying train-test mask distribution shifts. However, since the benchmark assumes access to complete PDE solutions during training, it seems straightforward to simulate realistic sensor failures and generate training masks that better approximate the expected test-time observation distribution, rather than relying on a mismatched pixel-level random masking strategy. If the test-time observation distribution is known, models could be trained directly using the corresponding masks. If it is unknown or variable, mixed-pattern training also appears to be a natural solution, as partially explored in the paper. I would appreciate further clarification on what practical constraints make such distribution mismatches difficult to avoid, and what additional insights the proposed OOD evaluation provides beyond demonstrating the expected performance degradation under distribution shift.
-
Did the authors consider matching the training mask distribution through test-time subsampling? The paper reports that increasing the observation density from 50% to 80% at test time does not consistently improve the performance of models trained with 50% random observation masks. However, a straightforward alternative would be to randomly discard some of the available observations at inference time, reducing the observation density from 80% to 50% to match the training mask distribution. For uniformly random masks, such subsampling can recover the same mask distribution used during training without requiring any model retraining. Did the authors evaluate this simple test-time strategy? I believe this would be an important baseline for interpreting the reported results, as it could help distinguish whether the performance degradation reflects a fundamental difficulty in utilizing additional observations or simply a distribution mismatch that can be mitigated through straightforward preprocessing.
7 comments
Weakness 2
Limited insights from the experimental results. The main findings are largely expected: models perform substantially better when the training and testing mask distributions match than when they differ. While the experiments confirm this phenomenon across different settings, they provide limited new insights into why models fail under distribution shifts or how such failures could be mitigated. As a result, the scientific takeaways beyond the construction of the benchmark are relatively limited.
Response:
Thank you for this constructive criticism. The revised experiments go beyond documenting a transfer gap: they identify effects of training recipes, input representation, acquisition exposure, and the fitted physical prior. For density changes they also give an explanation: the failure of the zero-fill FNO comes mainly from its input encoding, not from missing information. The corresponding controls establish what can be improved [D1–D2] and where those remedies fail [D3].
[D1] Recovery and optimization are distinct explanations. Classical references reveal that some original fields are easy to recover: gappy POD gives 2.63e-8 on Poisson/R50, while the original matched FNO is approximately 1e-2. On the four static V2 slices, the covariance reference has lower error than the network in all 1,425 paired per-seed comparisons (Table 110). We state this as a finding: on these in-distribution simulated fields, the evaluated networks do not reach the accuracy of a training-fitted linear reference. Strong classical references are therefore retained as essential calibration rather than omitted from the comparison.
On the later Poisson-medium CNO study, reducing the learning rate without normalization improves error from 0.8356 to 0.167; adding normalization at that learning rate further improves it to 0.0593. The latter isolates normalization, whereas the full change is a joint recipe effect. These V2 controls do not overwrite the original 441-model grid. They show that scores can reflect the training recipe, which is why we read the grid as a comparison across observation patterns within a PDE–method pair and not as an architecture ranking (Section 5).
[D2] Representation and exposure have separable effects. In the five-setting FNO factorial, architecture, recipe, and initial weights are shared within each setting and seed. The table gives U256 relative L2 error averaged over three seeds.
| Setting | Uniform training: zero fill | Uniform training: nearest fill | Mixed training: zero fill | Mixed training: nearest fill |
|---|---|---|---|---|
| Poisson medium | 0.02644 | 0.01638 | 0.04304 | 0.02815 |
| Poisson rough | 0.10200 | 0.08657 | 0.11719 | 0.09663 |
| Darcy medium | 0.02487 | 0.01646 | 0.03404 | 0.02388 |
| Darcy rough | 0.03165 | 0.02948 | 0.03515 | 0.03308 |
| RPB Poisson, ID | 0.52375 | 0.49037 | 0.37638 | 0.36251 |
Nearest filling improves both exposure policies in all five ID settings. Mixed exposure sacrifices U256 accuracy on the four V2 settings but improves it on RPB. These are trained input policies, not test-time preprocessing of an unchanged zero-fill model.
The interventions also affect transfer: on the V2 NS-medium U1024 comparison in Table 118, errors are 0.577 for U256-trained zero-fill FNO, 0.103 with nearest-fill training, and 0.0275 with two-density training.
Why the fixed FNO fails under denser input. Three results point to the zero-filled input encoding and not to missing information. Thinning the input back to the training count removes the resolved penalty in 118 of 122 cells (S-Q2). Training the same FNO with nearest-observation fill lowers the U1024 error above from 0.577 to 0.103. Voronoi-CNN, whose input is a nearest-sensor fill, shows no such penalty (U256-trained, Poisson medium: 0.0200 at U256 and 0.0167 at U1024; Table 100). Layout changes differ: clustered and line layouts also carry far less information (G-Q1), so both the measurements and the predictor contribute.
[D3] The boundaries are informative. Nearest filling is not universally beneficial: on RPB OOD it worsens the mixed arm by 0.00337, and on Darcy-rough C256 it worsens the uniform-trained arm by 0.362. Covariance similarly changes from 0.0715 ID to 39.3 OOD at RPB U256. In forecasting, holding initialization fixed reveals a separate benefit from learned rather than discovered dynamics.
These controls explain which experimental factor changes performance, while retaining counterexamples. The encoding explanation above is specific to density changes of zero-fill models; we do not extend it to every architecture or shift.
Revision locations. Section 4; Appendix H.8 (pp.82–87), especially Tables 108, 112, 118, and 120–121; the Darcy-rough nearest-fill effect in Table 134E (p.93); external-shift evidence in H.4 and dynamics controls in H.3.
Weakness 3
Insufficient motivation for studying mask distribution shifts. The practical motivation for this benchmark is not sufficiently convincing. The paper assumes access to complete PDE solutions during training, allowing arbitrary observation masks to be generated. If the mask distribution at inference time is known, why not simply train models using the same distribution? If the inference-time distribution is unknown or variable, training with a mixture of observation patterns seems like a natural and straightforward solution. Although the paper includes preliminary mixed-pattern training experiments, it does not sufficiently explain why these strategies are inadequate or why train-test mask distribution mismatch represents an important practical challenge under its assumptions. The authors should better justify the scenarios in which such distribution shifts are unavoidable and explain why the proposed benchmark is necessary.
Response:
Thank you for clarifying the motivation. We agree that matched and mixed-pattern training are appropriate strategies. Mixed training nevertheless has measurable costs and limits: it raises FNO error at the uniform destination by 11–63% on the four V2 settings. The benchmark's role is to determine when these strategies suffice and how their tradeoffs depend on the deployment objective. We distinguish those decisions [D1], the relevant evaluation criteria [D2], and the resulting comparisons [D3].
[D1] A decision problem, not a requirement that mixtures fail. Complete training fields allow acquisition-aware training. For a known stable pattern, a matched specialist is a natural reference; for anticipated variability, a mixture is appropriate. If acquisition changes after training, reuse, preprocessing, and retraining remain alternatives to compare. Their success is useful benchmark evidence rather than an objection to evaluating them.
[D2] Single-destination accuracy and deployment-average risk differ. We distinguish one fixed model serving all destinations from a collection that selects a matched specialist for each destination. The V2 comparison averages error equally over uniform, clustered, and line layouts with 256 observations each.
For example, on Poisson rough the three matched specialist errors are 0.102, 0.593, and 0.588, whereas one mixed model gives 0.117, 0.470, and 0.467. Its slightly worse uniform-view accuracy is offset by better clustered/line accuracy. This explains how mixed training can lose at one destination yet improve the deployment average, without assuming that matching is always optimal.
[D3] Complete comparison and counterexamples.
| Method-tuned FNO setting | One mixed model: equal-weight risk | Destination-specialist collection |
|---|---|---|
| Poisson medium | 0.2233 | 0.2698 |
| Poisson rough | 0.3513 | 0.4276 |
| Darcy medium | 0.09724 | 0.1085 |
| Darcy rough | 0.05745 | 0.05606 |
The mixed FNO is better in three of these four settings, while the specialist collection is better on Darcy rough. Voronoi-CNN's specialist collections are better on all four corresponding slices. Across the ten complete V2 single-model policy matrices, mixtures have the lowest point estimate in nine and resolved lower risk in eight; these counts are distinct from comparisons with specialist collections.
Mixing has a cost and a range. At the uniform destination, mixed training raises FNO error relative to U256-only training by 63% on Poisson medium (0.0264 to 0.0430), 15% on Poisson rough, 37% on Darcy medium and 11% on Darcy rough (Table 120). The three-layout mixture trained at 256 sensors gives 0.503 at U1024 on Poisson medium, against 0.043 at U256 (Table 80), and the two-density Navier–Stokes mixture gives 0.027 at 1,024 sensors but 0.246 at 4,096 (Table 115).
The original five MIX models also give relative errors of 0.0839–0.2321 on an unseen random-20% protocol (Table 68). No specialist was trained for that protocol, so these errors have no matched reference to compare with.
Revision locations. The revised motivation and interpretation in Sections 1, 4, and 5; Appendix G, Table 68 (p.61); Appendix H.9, Tables 122–131 (pp.88–91); comparison counts in Table 135 (p.94).
Weakness 1
Response:
Thank you for emphasizing real-world evaluation. We now report reconstruction on experimental cylinder-wake PIV, including a fixed-count regional-exclusion test. The experiment measures how loss of spatial coverage affects reconstruction and whether mixed training mitigates it. Data and scoring are specified in [D1], results in [D2], and reusable measurement support in [D3].
[D1] Measured-data protocol. We use the measured RealPDEBench Cylinder data with whole-run splits: 62 training, 10 validation, 10 official ID-test, and 10 official OOD-test runs. Each run contributes 100 frames. Errors jointly score both velocity channels on hidden cells within a training-defined support and are averaged within each run before averaging across runs.
XW256 preserves 256 observations but excludes a specified near-wake region. It changes acquisition on measured fields without retraining the neural models, and it is in neither training policy: the mixture contains U256 and C256 only. Each view has its own hidden scoring cells. The source's ID/OOD labels are retained; they do not represent a pure Reynolds-number extrapolation split.
[D2] Regional exclusion and mitigation. Representative official-ID results are:
The complete ten-estimator/policy comparison shows increased error under regional exclusion on both official lists. Across the two networks, five views, and two lists, mixed training improves all 20 point estimates relative to uniform-only training, with 19 paired differences resolved. The training-fitted covariance reference is comparable to the mixture-trained networks on both views.
[D3] Reusable interface and scope. The measured-data extension supports target-free prediction, separate reference-based scoring, and grid/channel/run-split checks. The anonymous code repository includes the interface and a measured-flow example; Appendix B.10 and Table 136 document its verification.
This is one measured physical system with a processed reference containing interpolation and no per-location validity flags. We retain those limitations and do not interpret the synthetic exclusion as an observed device failure.
Revision locations. Sections 3.2/3.5; Appendix H.7, Tables 104–106 (pp.80–82); comparison counts in Table 135 (p.94); interface verification in B.10 (p.33) and Table 136 (p.94).