acceptodds
Under review as a conference paper at ICLR 2027

PDE-OBS: Controlled Evaluation Across Observation Patterns

Abstract

Physical-field reconstruction and forecasting depend on both measurement density and spatial layout, yet evaluation under a single observation pattern does not characterize performance when that pattern changes. We introduce , an integrated benchmarking platform spanning numerical data generation, model training, and inference and evaluation under varying observation conditions. It combines 560,000 fields and trajectories from seven partial differential equation families with configurable observation operators and seven adapted baseline methods for stationary reconstruction and short-horizon forecasting. Separating observation construction from physical records allows users to specify parameterized patterns and deterministic mixtures for training and testing while preserving prediction targets and data splits. The evaluation protocol uses references trained for each test pattern to compare models on identical test observations and targets, alongside equal-count groups for spatial-layout comparisons. On a 14,000-record subset, we evaluate 441 trained models under nine test patterns, yielding 3,969 evaluations. Mean cross-pattern error exceeds mean matched-pattern error in all 49 PDE–method pairs, and this finding persists in a configuration-matched subset of 117 models. Denser test observations do not consistently reduce error for a fixed model. Mixed-pattern training on five completed pairs reduces large single-pattern transfer errors, although destination-trained references usually remain more accurate. Together, the benchmark and findings support systematic evaluation of observation-pattern sensitivity and provide a reusable workflow for developing methods under changing measurement conditions.

Primary area:
datasets and benchmarks
Keywords:
partial differential equations, scientific machine learning, PDE benchmarks, partial observations, neural operators, inverse problems, out-of-distribution generalization, physics-informed learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_StGhOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2(+1)
Presentation
3
Soundness
3
Read review
Paper summary

This paper considers a benchmark for the problem of mismatched mask distributions between training and testing in completion tasks on PDE datasets. The paper attempts to construct multiple PDE systems and various mask distributions, and considers two tasks for evaluation. The overall results are also consistent with intuition, showing that in-distribution performance is much better than OOD performance.

Critical strengths

Comprehensive and well-structured benchmark. The benchmark has a clear problem formulation and covers diverse evaluation settings, including multiple PDE systems, prediction tasks, observation patterns, and backbone architectures. This provides a systematic framework for evaluating model robustness under different observation conditions.

Critical weaknesses
  1. Limited evaluation on real-world data. The benchmark relies entirely on simulated PDE systems, without evaluations on real-world observations. This limits its practical relevance, especially since real-world measurements often involve noise, irregular sensor placements, and other complexities not captured by the current benchmark.

  2. Limited insights from the experimental results. The main findings are largely expected: models perform substantially better when the training and testing mask distributions match than when they differ. While the experiments confirm this phenomenon across different settings, they provide limited new insights into why models fail under distribution shifts or how such failures could be mitigated. As a result, the scientific takeaways beyond the construction of the benchmark are relatively limited.

  3. Insufficient motivation for studying mask distribution shifts. The practical motivation for this benchmark is not sufficiently convincing. The paper assumes access to complete PDE solutions during training, allowing arbitrary observation masks to be generated. If the mask distribution at inference time is known, why not simply train models using the same distribution? If the inference-time distribution is unknown or variable, training with a mixture of observation patterns seems like a natural and straightforward solution. Although the paper includes preliminary mixed-pattern training experiments, it does not sufficiently explain why these strategies are inadequate or why train-test mask distribution mismatch represents an important practical challenge under its assumptions. The authors should better justify the scenarios in which such distribution shifts are unavoidable and explain why the proposed benchmark is necessary.

Questions for authors
  1. Could the authors clarify the practical scenarios in which evaluating robustness to out-of-distribution (OOD) observation masks is particularly useful? The paper considers changes in observation patterns, such as sensor failures, as a motivation for studying train-test mask distribution shifts. However, since the benchmark assumes access to complete PDE solutions during training, it seems straightforward to simulate realistic sensor failures and generate training masks that better approximate the expected test-time observation distribution, rather than relying on a mismatched pixel-level random masking strategy. If the test-time observation distribution is known, models could be trained directly using the corresponding masks. If it is unknown or variable, mixed-pattern training also appears to be a natural solution, as partially explored in the paper. I would appreciate further clarification on what practical constraints make such distribution mismatches difficult to avoid, and what additional insights the proposed OOD evaluation provides beyond demonstrating the expected performance degradation under distribution shift.

  2. Did the authors consider matching the training mask distribution through test-time subsampling? The paper reports that increasing the observation density from 50% to 80% at test time does not consistently improve the performance of models trained with 50% random observation masks. However, a straightforward alternative would be to randomly discard some of the available observations at inference time, reducing the observation density from 80% to 50% to match the training mask distribution. For uniformly random masks, such subsampling can recover the same mask distribution used during training without requiring any model retraining. Did the authors evaluate this simple test-time strategy? I believe this would be an important baseline for interpreting the reported results, as it could help distinguish whether the performance degradation reflects a fundamental difficulty in utilizing additional observations or simply a distribution mismatch that can be mitigated through straightforward preprocessing.

7 comments
AuthorsOpenReview ↗

Weakness 1

Limited evaluation on real-world data. The benchmark relies entirely on simulated PDE systems, without evaluations on real-world observations. This limits its practical relevance, especially since real-world measurements often involve noise, irregular sensor placements, and other complexities not captured by the current benchmark.

Response:

Thank you for emphasizing real-world evaluation. We now report reconstruction on experimental cylinder-wake PIV, including a fixed-count regional-exclusion test. The experiment measures how loss of spatial coverage affects reconstruction and whether mixed training mitigates it. Data and scoring are specified in [D1], results in [D2], and reusable measurement support in [D3].

[D1] Measured-data protocol. We use the measured RealPDEBench Cylinder data with whole-run splits: 62 training, 10 validation, 10 official ID-test, and 10 official OOD-test runs. Each run contributes 100 frames. Errors jointly score both velocity channels on hidden cells within a training-defined support and are averaged within each run before averaging across runs.

XW256 preserves 256 observations but excludes a specified near-wake region. It changes acquisition on measured fields without retraining the neural models, and it is in neither training policy: the mixture contains U256 and C256 only. Each view has its own hidden scoring cells. The source's ID/OOD labels are retained; they do not represent a pure Reynolds-number extrapolation split.

[D2] Regional exclusion and mitigation. Representative official-ID results are:

Method / trainingUniform 256Regional exclusion, 256Increase [95% run-bootstrap interval]
FNO / uniform0.10560.18970.0841 [0.0763, 0.0917]
FNO / mixture0.08990.12790.0381 [0.0336, 0.0421]
Voronoi-CNN / uniform0.08870.21350.1248 [0.1120, 0.1373]
Voronoi-CNN / mixture0.08260.12730.0447 [0.0406, 0.0485]
Covariance0.08600.12700.0410 [0.0371, 0.0446]

The complete ten-estimator/policy comparison shows increased error under regional exclusion on both official lists. Across the two networks, five views, and two lists, mixed training improves all 20 point estimates relative to uniform-only training, with 19 paired differences resolved. The training-fitted covariance reference is comparable to the mixture-trained networks on both views.

[D3] Reusable interface and scope. The measured-data extension supports target-free prediction, separate reference-based scoring, and grid/channel/run-split checks. The anonymous code repository includes the interface and a measured-flow example; Appendix B.10 and Table 136 document its verification.

This is one measured physical system with a processed reference containing interpolation and no per-location validity flags. We retain those limitations and do not interpret the synthetic exclusion as an observed device failure.

Revision locations. Sections 3.2/3.5; Appendix H.7, Tables 104–106 (pp.80–82); comparison counts in Table 135 (p.94); interface verification in B.10 (p.33) and Table 136 (p.94).

AuthorsOpenReview ↗

Weakness 2

Limited insights from the experimental results. The main findings are largely expected: models perform substantially better when the training and testing mask distributions match than when they differ. While the experiments confirm this phenomenon across different settings, they provide limited new insights into why models fail under distribution shifts or how such failures could be mitigated. As a result, the scientific takeaways beyond the construction of the benchmark are relatively limited.

Response:

Thank you for this constructive criticism. The revised experiments go beyond documenting a transfer gap: they identify effects of training recipes, input representation, acquisition exposure, and the fitted physical prior. For density changes they also give an explanation: the failure of the zero-fill FNO comes mainly from its input encoding, not from missing information. The corresponding controls establish what can be improved [D1–D2] and where those remedies fail [D3].

[D1] Recovery and optimization are distinct explanations. Classical references reveal that some original fields are easy to recover: gappy POD gives 2.63e-8 on Poisson/R50, while the original matched FNO is approximately 1e-2. On the four static V2 slices, the covariance reference has lower error than the network in all 1,425 paired per-seed comparisons (Table 110). We state this as a finding: on these in-distribution simulated fields, the evaluated networks do not reach the accuracy of a training-fitted linear reference. Strong classical references are therefore retained as essential calibration rather than omitted from the comparison.

On the later Poisson-medium CNO study, reducing the learning rate without normalization improves error from 0.8356 to 0.167; adding normalization at that learning rate further improves it to 0.0593. The latter isolates normalization, whereas the full change is a joint recipe effect. These V2 controls do not overwrite the original 441-model grid. They show that scores can reflect the training recipe, which is why we read the grid as a comparison across observation patterns within a PDE–method pair and not as an architecture ranking (Section 5).

[D2] Representation and exposure have separable effects. In the five-setting FNO factorial, architecture, recipe, and initial weights are shared within each setting and seed. The table gives U256 relative L2 error averaged over three seeds.

SettingUniform training: zero fillUniform training: nearest fillMixed training: zero fillMixed training: nearest fill
Poisson medium0.026440.016380.043040.02815
Poisson rough0.102000.086570.117190.09663
Darcy medium0.024870.016460.034040.02388
Darcy rough0.031650.029480.035150.03308
RPB Poisson, ID0.523750.490370.376380.36251

Nearest filling improves both exposure policies in all five ID settings. Mixed exposure sacrifices U256 accuracy on the four V2 settings but improves it on RPB. These are trained input policies, not test-time preprocessing of an unchanged zero-fill model.

The interventions also affect transfer: on the V2 NS-medium U1024 comparison in Table 118, errors are 0.577 for U256-trained zero-fill FNO, 0.103 with nearest-fill training, and 0.0275 with two-density training.

Why the fixed FNO fails under denser input. Three results point to the zero-filled input encoding and not to missing information. Thinning the input back to the training count removes the resolved penalty in 118 of 122 cells (S-Q2). Training the same FNO with nearest-observation fill lowers the U1024 error above from 0.577 to 0.103. Voronoi-CNN, whose input is a nearest-sensor fill, shows no such penalty (U256-trained, Poisson medium: 0.0200 at U256 and 0.0167 at U1024; Table 100). Layout changes differ: clustered and line layouts also carry far less information (G-Q1), so both the measurements and the predictor contribute.

[D3] The boundaries are informative. Nearest filling is not universally beneficial: on RPB OOD it worsens the mixed arm by 0.00337, and on Darcy-rough C256 it worsens the uniform-trained arm by 0.362. Covariance similarly changes from 0.0715 ID to 39.3 OOD at RPB U256. In forecasting, holding initialization fixed reveals a separate benefit from learned rather than discovered dynamics.

These controls explain which experimental factor changes performance, while retaining counterexamples. The encoding explanation above is specific to density changes of zero-fill models; we do not extend it to every architecture or shift.

Revision locations. Section 4; Appendix H.8 (pp.82–87), especially Tables 108, 112, 118, and 120–121; the Darcy-rough nearest-fill effect in Table 134E (p.93); external-shift evidence in H.4 and dynamics controls in H.3.

AuthorsOpenReview ↗

Weakness 3

Insufficient motivation for studying mask distribution shifts. The practical motivation for this benchmark is not sufficiently convincing. The paper assumes access to complete PDE solutions during training, allowing arbitrary observation masks to be generated. If the mask distribution at inference time is known, why not simply train models using the same distribution? If the inference-time distribution is unknown or variable, training with a mixture of observation patterns seems like a natural and straightforward solution. Although the paper includes preliminary mixed-pattern training experiments, it does not sufficiently explain why these strategies are inadequate or why train-test mask distribution mismatch represents an important practical challenge under its assumptions. The authors should better justify the scenarios in which such distribution shifts are unavoidable and explain why the proposed benchmark is necessary.

Response:

Thank you for clarifying the motivation. We agree that matched and mixed-pattern training are appropriate strategies. Mixed training nevertheless has measurable costs and limits: it raises FNO error at the uniform destination by 11–63% on the four V2 settings. The benchmark's role is to determine when these strategies suffice and how their tradeoffs depend on the deployment objective. We distinguish those decisions [D1], the relevant evaluation criteria [D2], and the resulting comparisons [D3].

[D1] A decision problem, not a requirement that mixtures fail. Complete training fields allow acquisition-aware training. For a known stable pattern, a matched specialist is a natural reference; for anticipated variability, a mixture is appropriate. If acquisition changes after training, reuse, preprocessing, and retraining remain alternatives to compare. Their success is useful benchmark evidence rather than an objection to evaluating them.

[D2] Single-destination accuracy and deployment-average risk differ. We distinguish one fixed model serving all destinations from a collection that selects a matched specialist for each destination. The V2 comparison averages error equally over uniform, clustered, and line layouts with 256 observations each.

For example, on Poisson rough the three matched specialist errors are 0.102, 0.593, and 0.588, whereas one mixed model gives 0.117, 0.470, and 0.467. Its slightly worse uniform-view accuracy is offset by better clustered/line accuracy. This explains how mixed training can lose at one destination yet improve the deployment average, without assuming that matching is always optimal.

[D3] Complete comparison and counterexamples.

Method-tuned FNO settingOne mixed model: equal-weight riskDestination-specialist collection
Poisson medium0.22330.2698
Poisson rough0.35130.4276
Darcy medium0.097240.1085
Darcy rough0.057450.05606

The mixed FNO is better in three of these four settings, while the specialist collection is better on Darcy rough. Voronoi-CNN's specialist collections are better on all four corresponding slices. Across the ten complete V2 single-model policy matrices, mixtures have the lowest point estimate in nine and resolved lower risk in eight; these counts are distinct from comparisons with specialist collections.

Mixing has a cost and a range. At the uniform destination, mixed training raises FNO error relative to U256-only training by 63% on Poisson medium (0.0264 to 0.0430), 15% on Poisson rough, 37% on Darcy medium and 11% on Darcy rough (Table 120). The three-layout mixture trained at 256 sensors gives 0.503 at U1024 on Poisson medium, against 0.043 at U256 (Table 80), and the two-density Navier–Stokes mixture gives 0.027 at 1,024 sensors but 0.246 at 4,096 (Table 115).

The original five MIX models also give relative errors of 0.0839–0.2321 on an unseen random-20% protocol (Table 68). No specialist was trained for that protocol, so these errors have no matched reference to compare with.

Revision locations. The revised motivation and interpretation in Sections 1, 4, and 5; Appendix G, Table 68 (p.61); Appendix H.9, Tables 122–131 (pp.88–91); comparison counts in Table 135 (p.94).

Reviewer_GNGXOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PDE-OBS, a benchmark that evaluate the generalization capability of physical field reconstruction or prediction models under variation in observation density and different spatial layout. The proposed benchmark decoupling the observation operator from the physical data, providing an cross-evaluation scheme of training and testing.

Based on different 7 types of PDE families, totally 14000 data, the paper benchmarking different widely used baseline methods and demonstrates that model's performance is highly sensitive to the density and spatial layout of observations. The results show that the prediction errors will generally increase when the observational patterns used for training and testing are inconsistent.

Critical strengths
  1. This study focuses on important issues in neural operator / neural PDE solver field. When learning from data, the neural operators are designed to learn mapping between function spaces, but their abilities are actually learned from learning discrete data space. This training scheme cannot guarantee robustness when observation method changes. This paper focused on benchmarking this issue and provides valuable insights into the reliability of models under varying observation.

  2. This study provides a reusable and extensible benchmark and framework for neural operator studies. The framework integrates data generation, observational construction, training, and evaluation. providing a foundation for future research on physical field predicting task under varying observational conditions.

Critical weaknesses
  1. The evaluation scenarios in this study are limited. Although seven classes of PDE families are covered, the main experiments are restricted to two-dimensional regular grids with fixed resolution, smooth fields, and noise-free observations.

  2. This study also lack of strong baselines. Some papers, RINO[1] and SDIFT[2], specifically designed to deal with irregular and sparse observation problem. Including these methods would help determine if the performance can be improved with these flexible observation settings.

[1] Bahmani, Bahador, et al. "A resolution independent neural operator." Computer Methods in Applied Mechanics and Engineering 444 (2025): 118113.

[2] Chen, Panqi, et al. "Generating full-field evolution of physical dynamics from irregular sparse observations." Advances in Neural Information Processing Systems 38 (2026): 133720-133756.

Questions for authors

Even with the same number of observation points, observations from different locations may provide different information based on PDEs. Did the authors quantify the information content of different observation methods to determine the difficulty is caused by insufficient input information or the mismatch of training and test observation distributions?

5 comments
AuthorsOpenReview ↗

Weakness 1

The evaluation scenarios in this study are limited. Although seven classes of PDE families are covered, the main experiments are restricted to two-dimensional regular grids with fixed resolution, smooth fields, and noise-free observations.

Response:

Thank you for identifying the limits of the original evaluation. We have expanded the study to sparse and noisy observations, different physical-field constructions and native resolutions, external numerical data, experimental measurements, and a one-dimensional dynamics extension. We summarize coverage [D1], what these extensions reveal [D2], and the remaining scope [D3].

[D1] Expanded coverage.

ExtensionCompleted evaluationRevised location
Physical-field complexityPoisson, Darcy, and Navier–Stokes at three construction scales; 9,000/900/1,800 training/validation/test records per sliceH.4, Tables 78–87
Sparse and noisy acquisitionLayouts with 256 or 1,024 sensors; clustered/line/grid observations; Gaussian noise and noise-aware controlsH.4, Tables 89–92
Native resolutionsSeparate training/evaluation at 64×64, 128×128, and 256×256 on selected rough Poisson/NS settingsH.4, Table 88
External numerical dataRPB Poisson at 64×64, with 256 ID and 256 OOD test fieldsH.4, Tables 93–95
Additional dynamicsNative-1D KS, with 300 test trajectories and five observation viewsH.3, Table 72
Measured fieldsCylinder-wake PIV, with 20 held-out experimental runs and five acquisition conditionsH.7, Tables 104–106

Sensor displacement and the five-setting representation/exposure factorial provide additional controlled acquisition studies in H.8.

[D2] The main finding persists, and the extensions change its interpretation. The dependence on the training view remains on rougher fields with 256 sensors (Tables 79–87): on Poisson medium, FNO trained on U256 gives 0.026 at U256 and 0.661 at C256, against 0.408 for the C256-trained model. It remains at each native resolution (Poisson rough, U256-trained FNO at C256: 0.865, 0.720 and 0.766 on the 64×64, 128×128 and 256×256 grids, against 0.549, 0.593 and 0.590 for the C256-trained models; Table 88) and on measured PIV (Table 104). Test noise up to 5% changes the mixture-trained networks little (Poisson-medium FNO at C256: 0.326 to 0.329; Table 89).

The classical references are more sensitive to shift and noise. On external Poisson at U256, covariance error changes from 0.0715 ID to 39.3 OOD, while mixed-training FNO gives 0.376 and 0.539. An error above 1 is worse than predicting the training mean: the prior fitted to the ID training fields does not represent the OOD fields, and conditioning 256 scattered values on it extrapolates (on a regular grid or with 1,024 sensors the covariance error is 0.411 and 0.335; Table 95). Under 5% observation noise on Poisson-medium clustered measurements, the covariance error is approximately 20.3 with the regularization selected on noise-free data and 0.222 with regularization selected for the noise level, which is below the networks' 0.290–0.375 on the same view (Tables 89–90).

Training-set-size effects also depend on the optimization budget. For Poisson-rough FNO, 1,800 versus 9,000 training records give 0.1166 versus 0.1172 at the same number of updates, but 0.1436 versus 0.1172 at 250 passes over the respective datasets. Finer native grids likewise do not produce a uniform improvement across methods and observation conditions.

[D3] Scope. These are explicitly separated datasets and protocols. Each native resolution has its own trained models, so transfer of one checkpoint across resolutions is untested. All data are on regular grids; unstructured meshes and 3D systems remain outside the evaluated scope. The measured evidence covers one physical system, and forecasting remains three-step. V2 has no Burgers slice: the two generation routes we tried did not meet our numerical-accuracy limit, so Burgers remains in the original grid only.

Revision locations. Section 3.2 and Appendix H's data overview (p.63); H.4 (pp.68–76); the training-size comparison in Table 134B (p.93); related dynamics, measurements, and intervention tables in H.3, H.7, and H.8. Section 5 retains the remaining limitations.

AuthorsOpenReview ↗

Weakness 2

This study also lack of strong baselines. Some papers, RINO[1] and SDIFT[2], specifically designed to deal with irregular and sparse observation problem. Including these methods would help determine if the performance can be improved with these flexible observation settings.

Response:

Thank you for recommending these observation-flexible methods. The RINO adaptation is stable when the sensor count increases, whereas the zero-fill FNO deteriorates. It is less accurate than FNO at the training density, and less accurate than Voronoi-CNN at both densities. We report that comparison [D1], the SDIFT evaluations [D2], and the method-specific qualifications needed to interpret both [D3].

[D1] RINO density transfer. All checkpoints below are trained on 256 uniform-random observations and evaluated without retraining. Errors average three training seeds.

SettingRINO v2: 256 sensorsRINO v2: 1,024 sensorsFNO: 256 sensorsFNO: 1,024 sensorsVoronoi-CNN: 256 sensorsVoronoi-CNN: 1,024 sensors
Poisson medium0.042360.041680.026440.62650.02000.0167
Poisson rough0.16950.15240.1020.57030.08440.0545
Darcy medium0.030630.028670.024870.46760.02230.0188
Darcy rough0.044740.041640.031650.29030.03240.0249

RINO has lower error than the zero-fill FNO at the denser destination in all four comparisons, despite higher matched-density error. On Darcy-medium clustered observations, the U256-trained RINO adaptation also gives 0.135 versus 0.420 for FNO. Two qualifications apply. First, the FNO deterioration is specific to feeding it the denser zero-filled input directly: in the V2 panel, thinning to the training count leaves no resolved residual gap in all 21 cells that had a density penalty (S-Q2). Second, the U256-trained Voronoi-CNN (Table 100) is also stable under denser input and has lower error than the RINO adaptation in all eight cells of the table. In our adaptation, RINO therefore adds robustness to density but does not improve on the best existing baseline. Results for the earlier v1 adaptation are labelled separately (Tables 97–98).

[D2] SDIFT results and sampling definition. We evaluated a local SDIFT adaptation on V2 Navier–Stokes and on the four original forecasting families. On the 300 test parents evaluated for SDIFT (V2 NS-medium, line observations), the eight-forecast averaged prediction gives 0.695, versus 0.905 for mean per-draw error; on the same parents the frozen line-trained FNO gives 0.750 and the mixed-trained FNO 0.558 (Table 134A; on the full test set, Table 86 gives 0.746 and 0.552). Thus, the comparison includes both favorable and unfavorable cases.

In the original-family forecasting extension, the SDIFT adaptation is less accurate than PDE-FIND + IC on all 36 family–view cells. Tables 73–74 report error after averaging eight predictions separately from the average of eight individual errors.

[D3] Qualification. These are task-specific adaptations, not fully reproduced upper-performance estimates of the original methods. On Darcy rough, the RINO dictionary reconstructs fields less accurately than a POD basis of equal rank (error ratio 1.266, above our 1.25 acceptance limit), so its scores there may understate the method. For SDIFT, none of the 13 guidance settings we tested passes all four of our stability checks on both V2 NS slices. We retain these limitations alongside the positive transfer findings and do not use the weak SDIFT scores to claim that the original method is ineffective. There is no native-1D SDIFT result or RINO-RPB/PIV result.

Revision locations. Section 3.4; Appendix H.5, Tables 96–99 (pp.76–78); the matched SDIFT/FNO and clustered RINO comparisons in Table 134A and 134C (p.93), with qualification counts in Table 135 (p.94); the separately qualified SDIFT forecasting comparisons in H.3, Tables 73–74 (pp.66–67).

AuthorsOpenReview ↗

Question 1

Even with the same number of observation points, observations from different locations may provide different information based on PDEs. Did the authors quantify the information content of different observation methods to determine the difficulty is caused by insufficient input information or the mismatch of training and test observation distributions?

Response:

Thank you for this important question. Yes, and both causes occur. Layout changes the information: with the same 256 sensors on Poisson medium, the posterior/prior covariance-trace ratio is 3.93e-5 for uniform-random observations and 5.20e-2 for line observations. Density transfer is mismatch: going from 256 to 1,024 sensors on Poisson rough lowers the ratio from 3.04e-3 to 1.12e-4, yet the U256-trained FNO error rises from 0.103 to 0.566, while a U1024 specialist reaches 0.0380. The locations therefore leave very different modeled uncertainty at an equal measurement budget. We define the diagnostic [D1], test its relationship to achieved error [D2], and combine it with controlled predictor comparisons [D3].

[D1] Information diagnostic. We condition a training-fitted, rank-2,048 Gaussian field prior on the observations using the stated noise model. The trace ratio measures remaining uncertainty relative to the prior; a smaller ratio means more uncertainty has been removed under that model. It is not the relative L2 error or a universal measure of information for every PDE distribution.

[D2] Empirical calibration and predictor response.

TestResultInterpretation
Equal-count Poisson-medium layoutsU256: 3.93e-5; L256: 0.0520Count alone does not characterize the constraints.
Nested Poisson-rough observations, 256 → 1,024Trace ratio: 3.04e-3 → 1.12e-4; fixed U256-trained FNO error: 0.103 → 0.566; U1024 specialist: 0.0380. This study adds sensors to the same 256, so its values differ slightly from the standard U1024 view in G-W2 (0.102 → 0.570).More recoverable information and better fixed-model prediction are different properties.
Poisson-medium error-model calibrationPosterior-risk R² is 0.93 for layout changes within a record, but 0.0029 across records at fixed U256Explaining layout sensitivity does not imply predicting which physical records are difficult.
External-prior shiftAchieved squared error / (posterior risk + truncation tail): 0.997–1.577 ID and 12.6–1,725 OODCalibration depends on whether the fitted prior represents the evaluation population.

The calibration models are fitted on validation records and scored on test records. The seven-layout correlations and the validation-calibration studies have distinct populations; we report their actual method coverage rather than extending the calibrated relationship to every network.

[D3] Separating mismatch empirically. At a fixed test observation set, we compare a transferred predictor with a same-method destination-trained reference on identical records and targets. A strong recovery reference establishes achievable accuracy, while changing input representation or training exposure tests whether the predictor can use those same observations more effectively.

Together, these controls identify cases in which poor prediction is not explained solely by missing measurements. They do not provide an exact additive decomposition: the fitted prior, optimization, representation, and model capacity still matter.

Revision locations. Sections 3.2–3.3; Appendix H.6, Tables 100–103 (pp.78–80); the nested-observation values in Table 134D (p.93); complementary controlled interventions in H.8, Tables 113–121.

Reviewer_AzwjOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PDE-OBS a benchmark dataset that evaluates how different PDE forecasting techniques can forecast PDE simulations from partial observations. The benchmark has atleast 7 different classes of PDEs and several methods of occluding observations. THe dataset is split into train and test where the train data spans the same distribution of initial conditions as test data. The overall paper is well motivated and well written however, there are some doubts about the novelty and triviality of creating this dataset from already existing dataset such as PDEFind or RealPDEBench.

Critical strengths

Good problem motivation. PDE reconstruction in the real world is almost always from partial observation in sub sampled regimes. So a benchmark specifically targetted for this problem is necessary.

The dataset is fairly large with 49 PDE and method pairs.

A slew of machine learning neural predictors are tested as a part of evaluation.

Establishes a standardized evaluation protocol and metrics.

Critical weaknesses

I did not understand why cant I build the PDE-OBS benchmark from RealPDEBench or PDEFInd by just adding a mask with simple python script. What is novel beyond the masking strategy. Is standardization of evaluation the only novel component?

There is a concern about overlapping of initial conditions between training and testing. The issue is even if test has initial conditions that are not in training data, the initial conditions of test data may still be present in some trajectory of train data. If that is the case, then an approach like the following should have very good results:

Approach: Use training data and PDEFind like equation discovery approach. Search for initial condition by assuming one, simulating the recovered PDE and computing RMSE by a small network. This technique should be able to outperform any other neural technique. This should be used as a baseline I believe. If this technique outperforms every other neural baselines then may be the benchmark is not varied enough.

Questions for authors

Can you add a baseline using PDEFind and initial condition search with a small neural network and see how it performs on the benchmark?

Rebuttal addressed my concerns significantly hence I update my score.

5 comments
AuthorsOpenReview ↗

Weakness 1

I did not understand why cant I build the PDE-OBS benchmark from RealPDEBench or PDEFInd by just adding a mask with simple python script. What is novel beyond the masking strategy. Is standardization of evaluation the only novel component?

Response:

Thank you for this thoughtful question. Your point about reusing existing physical data is well taken, and we apologize that our presentation did not make the scope sufficiently clear. Our contribution is an integrated benchmarking pipeline with reusable APIs, controlled references, and scientific findings—not a new masking algorithm. Existing datasets and PDE-OBS are complementary [D1]; the reusable experimental structure [D2] enables the calibration and intervention findings [D3].

[D1] Complementary objectives.

ResourcePrimary roleRelationship to PDE-OBS
RealPDEBenchPrediction on simulated and measured systemsSupplies the experimental cylinder-flow evaluation.
PDE-FINDIdentification of governing equationsSupplies the discovered-dynamics forecasting component.
PDE-OBSControlled reconstruction and forecasting across observation conditionsConnects data, observations, training, prediction, and scoring through reusable interfaces.

Reusing existing data is part of the platform's purpose. Complete simulated targets support evaluation at hidden locations; measured references retain their own quality limitations.

[D2] What the pipeline makes possible. A masking script reproduces the observation step only. It does not supply the trained reference models, the recovery references, or the per-record results that the comparisons below need. For each destination in the original complete matrices, we train a same-method reference and compare it with transferred models using identical test measurements, physical records, and targets. Equal-count groups additionally control the sensor budget. Training-fitted recovery references and representation/exposure controls then help interpret the differences; destination training is an empirical reference, not an information-theoretic optimum.

The APIs connect numerical generation or external loading, observation construction, model training/loading, prediction, and independent scoring. Researchers can add compatible methods, observations, and metrics without rebuilding the evaluation workflow: Appendix B.7 registers a new observation pattern in ten lines (Listing 3), and Appendix B.5 runs training, prediction and scoring on a small example (Listing 2). Code, configurations and tests are in the anonymous repository linked in the abstract. The same repository includes the measured-data interface (pdeobs.measurements), which supports target-free prediction; Appendix B.10 documents it and its verification.

[D3] Scientific findings enabled by this structure.

FindingConcrete evidence
Recovery calibration separates attainable accuracy from neural performance.On original Poisson/R50, gappy POD gives 2.63e-8 versus approximately 1e-2 for the original matched FNO.
Geometry-sensitive errors can be mitigated while preserving the task.On Poisson-rough jittered sensors, grid-trained FNO gives 0.391, covariance 0.033, and jitter-augmented FNO approximately 0.051.
Representation and exposure have distinct, condition-dependent effects.The five-setting factorial improves U256 accuracy with nearest filling across the ID settings, but finds deterioration for some shifted destinations.

These results provide practical guidance, including limitations of the original smooth data and of particular remedies, rather than only standardizing a score.

Revision locations. Sections 1–3.5; Appendix H.1 (p.63); recovery and interventions in H.8, Tables 107–121 (pp.82–87); reusable interfaces in Appendix B, including B.10 (p.33).

AuthorsOpenReview ↗

Weakness 2

There is a concern about overlapping of initial conditions between training and testing. The issue is even if test has initial conditions that are not in training data, the initial conditions of test data may still be present in some trajectory of train data. If that is the case, then an approach like the following should have very good results:

Approach: Use training data and PDEFind like equation discovery approach. Search for initial condition by assuming one, simulating the recovered PDE and computing RMSE by a small network. This technique should be able to outperform any other neural technique. This should be used as a baseline I believe. If this technique outperforms every other neural baselines then may be the benchmark is not varied enough.

Response:

Thank you for raising this important concern. Test initial states are not measurably closer to the stored training states than a development reference group (median relative distance to the nearest stored state: 0.35–0.39 on the four original families and 0.12 on KS), and observation-only retrieval is substantially less accurate than the learned forecasting reference. We address saved-state similarity [D1], whether retrieval explains performance [D2], and the separate question of dataset recoverability [D3].

[D1] Cross-time state checks. For each family we search all 27,000 saved training states, which include the later frames of every training trajectory, for the state nearest to each test initial state. The median relative distance to that nearest state is 0.3453–0.3904 for the four original families and 0.1209 for KS. There are also no exact duplicates of the 200 test initial states of each original family.

The audit also compares these distances with a 300-state development reference group per family, excluding a reference state's own trajectory when it appears in the search bank. The test-minus-development median intervals include zero in all five families, so test states are not measurably closer to the training set than this reference group. These checks cover all saved states; [D2] tests directly whether similarity to stored training states helps prediction.

[D2] Observation-only retrieval. We rank training states using only acquired measurements and predict from their continuations. The table uses common test identities and masks. The FNO comparator is the separately trained, three-seed mixture model in the expanded forecasting study, not an original-grid specialist.

Family / test viewRetrievalMixture-trained FNO
Heat / R500.2430.0502
Burgers / R500.2740.0453
Reaction–diffusion / R500.2480.0545
Navier–Stokes / R500.2520.00357
KS / KU640.2270.0178

Errors are relative L2 over forecast frames 1–3. R50 observes half of the two-dimensional grid; KU64 observes 64 of 128 KS sites. The implemented retrieval strategy does not reproduce the learned predictor's accuracy. The table is an excerpt with one view per family: retrieval on all nine original views and all five KS views is in Tables 72–74 (pp.66–67), where it has higher error than this FNO in all 41 family–view cells. Table 71 (p.65) compares it with the original-grid specialists and shows that extending the retrieval bank from the training initial states to every stored training frame does not lower its error in any of the 36 two-dimensional cells.

[D3] Diversity and recoverability. Beyond nonduplication, we applied the test you propose: the requested baseline has lower error than every end-to-end network in 22 of the 41 family–view cells, and an end-to-end network is lower in the other 19 (A-Q1). On Heat and reaction–diffusion its error is the same on all nine views to three digits (0.0548 and 0.0594), and a training-fitted covariance initializer in place of the network gives the same values (Tables 73–74), because the smooth initial state is recovered almost exactly from any of them. Its forecast error on these families therefore comes from the dynamics stage rather than from state recovery. Training-fitted gappy POD likewise nearly exactly reconstructs original Poisson and Helmholtz fields on several views. The revised study therefore adds medium and rough construction scales, on which gappy POD errors are 0.02–0.73 (Table 109), together with physical shifts and KS, with classical references kept in the comparisons. The requested PDE-FIND pipeline and same-initializer dynamics controls further distinguish state recovery from forecasting accuracy: with the same inferred state, learned dynamics have lower error than the discovered equation in all 41 cells (A-Q1).

Revision locations. Section 3.2; Appendix H.2, Tables 69–71 (pp.64–65); expanded retrieval and forecasting in H.3, Tables 72–74 (pp.66–67); recovery calibration in Tables 108–109 (p.83).

AuthorsOpenReview ↗

Question 1

Can you add a baseline using PDEFind and initial condition search with a small neural network and see how it performs on the benchmark?

Response:

Thank you for this useful suggestion. The equation-discovery baseline adds a perspective that complements the learned predictors, and it is now part of the revised manuscript (Appendix H.3). We implemented PDE-FIND with learned initial-condition inference on five dynamic PDE families. It has lower joint error than every end-to-end network in 22 of the 41 family–view cells, and an end-to-end network is lower in the other 19. It is therefore competitive, especially under clustered observations, but does not uniformly outperform learned predictors. We explain its information access [D1], show five families in a component-controlled comparison [D2], and summarize the outcome [D3].

[D1] Implemented baseline. PDE-FIND discovers sparse dynamics from complete training frames 0–3. A small supervised network estimates the initial state from partial observations, measurement-based refinement enforces those observations, and numerical rollout predicts frames 1–3. Test-time inference does not use hidden future targets or true per-record test parameters. Navier–Stokes uses a Biot–Savart closure and a fixed candidate forcing basis.

The pipeline has complete-initial-state training supervision in addition to the future targets used by end-to-end masked-input predictors. To control this distinction, we freeze its inferred initial state and refinement and replace only the evolution stage with learned full-state FNO dynamics trained on the same training trajectories.

[D2] Five-family evidence. The table reports joint relative L2 errors on one random and one clustered view per family, averaged over three fits. All methods share the test records and masks. R50 and CL are the original random-50% and clustered views; KU64 and KC64 are the 64-observation uniform and clustered KS views. Bold marks the lower of PDE-FIND + IC and the best end-to-end network in each row.

Family / viewPDE-FIND + ICEnd-to-end FNOCNO forecasterDeepONetSame inferred IC + learned dynamics
Heat / R500.05480.05020.05250.09360.0498
Heat / CL0.05480.05770.1510.1360.0498
Burgers / R500.05300.04530.05100.1260.0445
Burgers / CL0.05250.05440.1710.1740.0445
Reaction–diffusion / R500.05940.05450.05660.1100.0551
Reaction–diffusion / CL0.05940.06200.1470.1570.0551
Navier–Stokes / R500.004340.003570.02520.07490.00335
Navier–Stokes / CL0.005100.02670.1780.1240.00335
KS / KU640.003470.0178–0.09250.000671
KS / KC640.005460.0318–0.2200.00318

This table is an excerpt: two of the nine views of each two-dimensional family and two of the five KS views. The complete results are in Appendix H.3, Tables 72–74 (pp.66–67). Learned dynamics have lower joint error in all 41 same-initializer comparisons and lower error at each forecast step in all 123 corresponding comparisons; paired intervals exclude zero conditional on the fitted models. Between PDE-FIND + IC and the end-to-end networks the comparison goes both ways, in the excerpt (six rows against four) and over all 41 cells (point estimates): PDE-FIND + IC is lower than every end-to-end network (FNO, CNO and DeepONet) in 22 (all five KS views, eight of nine Navier–Stokes views, and CL, BL, BD of Heat, Burgers and reaction–diffusion), and FNO or CNO is lower in the other 19 (R50, R65, R80, H, V, LI of those three families, and Navier–Stokes R50). The SDIFT adaptation is reported separately with its qualifications.

[D3] Takeaway. Thank you for proposing this method. It is applicable across the benchmark: it runs on all five dynamic families and every observation view, with the same test records and masks as the networks. It is also competitive: it has lower error than every end-to-end network in 22 of the 41 cells, including all five clustered views. We keep it as a baseline alongside the learned predictors.

Revision locations. Appendix H.3 (pp.65–68), with full comparisons in Tables 72–74, seed variation and paired component effects in Tables 75–76, horizon intervals in Table 77, and the comparison counts in Table 135 (p.94).

Comments on the paper

2 comments

Revision map. The benchmark itself remains the original protocol and grid (Sections 3–4); the V2, RPB, KS and PIV studies are separately labelled extensions with their own data splits (Appendix H). Tables 134–136 (pp.93–94) collect the selected comparisons, counts and timings quoted in this response. Sections 1–3 clarify the platform, task and information-access definitions. Section 4 and Appendices H.1–H.11 organize the evidence by the eleven reviewer issues. Appendix B documents reusable interfaces, including the measured-data extension in B.10; Appendix C.5 reports representative fitting and inference costs. Full comparisons retain unfavorable results and the qualifications of task-specific adaptations.

Terminology. Errors are relative L2 values, not percentages, unless stated otherwise. V2 denotes the expanded Poisson/Darcy/Navier–Stokes simulation suite. RPB denotes external Representative PDE Benchmarks Poisson data; PIV denotes particle image velocimetry; KS denotes the new one-dimensional Kuramoto–Sivashinsky extension. U, C, L, and G denote uniform-random, clustered, line, and regular-grid observations, followed by the sensor count. XW256 excludes the near-wake region at the same count. R50, H, and CL denote the original random-50%, horizontal-row, and clustered views. Training-seed variability and test-record/run confidence intervals are reported separately.

Dear Area Chair and Reviewers,

Thank you for your thoughtful comments. PDE-OBS contributes an integrated, reusable pipeline for testing what different observations allow us to recover, how trained predictors respond to acquisition changes, and which interventions improve their performance. The revised manuscript strengthens the scientific interpretation through classical recovery references, controlled interventions, broader physical settings, and measured-data evaluation.

The original 441-model grid is unchanged and is reported in full. It contains 49 complete training-pattern-by-test-pattern matrices; mean cross-pattern error exceeds the matched-pattern mean in all 49 PDE–method pairs (median ratio 10.46). Each model is the final checkpoint of its recorded run, and all 3,969 evaluations are retained, including large errors. Training histories differ across the grid (Table 12), so we also report the 13 pairs (117 models) whose nine models all completed the same 500-epoch recipe and differ only in the observation pattern. This subset was fixed from configuration metadata before scoring, and the direction holds in all 13 pairs (median ratio 10.74; Appendix D). The grid compares observation patterns within a PDE–method pair for the evaluated trained models. It does not rank architectures (Section 5), and it does not measure attainable accuracy, which the classical references below address. Later recipe controls, retraining studies, and external-data comparisons are reported separately rather than substituted into the original grid.

What the expanded experiments establish.

Scientific findingDirect evidenceLocation in the revised manuscript
For in-distribution static reconstruction, training-fitted classical references are more accurate than the networks on the tested simulated fields and comparable on measured PIV.On original Poisson fields with 50% random observations, training-fitted gappy POD gives 2.63e-8 error; the original matched FNO is approximately 1e-2. On the four static V2 slices, the covariance reference has lower error than the network in all 1,425 paired per-seed comparisons. On measured PIV it is comparable to the best network (0.0860 versus 0.0826 at U256). Neural error alone is therefore not a measure of missing information; the ranking reverses under physical shift (see the physical-shift row below).Tables 25, 104–105 and 107–110; Appendix H.8
Equal measurement counts leave different uncertainty.With 256 sensors on Poisson medium, the posterior/prior covariance-trace ratio is 3.93e-5 for uniform-random observations and 5.20e-2 for line observations.Appendix H.6, Table 100
Representation and training exposure are actionable, distinct factors.The FNO factorial improves U256 accuracy with nearest-observation filling under both exposure policies across five in-distribution settings; it also identifies adverse shifted-view cases.Appendix H.8, Tables 120–121
The suggested equation-based predictor is competitive, and its components matter.Replacing discovered dynamics with learned dynamics while keeping the inferred state fixed lowers joint forecast error in all 41 family–view comparisons.Appendix H.3, Tables 72–77
Matching, mixtures, and thinning are useful policies with different objectives.Mixed training has the lowest single-model deployment-risk estimate in 9/10 complete V2 matrices, with resolved advantages in 8/10. For the original grid's 65 density-penalty cells, thinning leaves no statistically resolved residual gap in 62 and partially repairs three.Appendices H.9 and H.11, Tables 122, 132 and 135
Physical shift can reverse an in-distribution ranking.At U256 on external Poisson data, covariance error changes from 0.0715 ID to 39.3 OOD; mixed-training FNO gives 0.376 and 0.539.Appendix H.4, Tables 93–95
Regional coverage matters on measured fields even at fixed count.Excluding a near-wake region while retaining 256 observations raises ID FNO error from 0.1056 to 0.1897; mixed training reduces the exclusion penalty from 0.0841 to 0.0381.Appendix H.7, Tables 104–106

How the interpretation changes. We now distinguish recoverability from predictor accuracy, rather than treating every large neural error as evidence of insufficient observations. We interpret the density result as sensitivity of a fixed predictor/input policy, with subsampling as an effective remedy in many cases. Matching and mixtures are evaluated as valid solutions, not strategies that the benchmark requires to fail. The recipe controls in Table 112 concern later V2 models and do not replace the original 441-model statistics.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.