acceptodds
Under review as a conference paper at ICLR 2027

When Does AI for PDEs Yield Scientific Evidence?

Abstract

Existing AI-for-PDE benchmarks primarily assess models in terms of predictive or approximation accuracy. In physics research, however, AI outputs often serve as evidence for scientific claims. These two objectives are not equivalent: the former measures an output's agreement with a reference target or satisfaction of governing constraints; the latter asks whether, given a specified object of study, scientific claim, assumptions, and evidence standard, the output provides sufficient evidence for that claim. To bridge this gap, we extend a widely used PDE-simulation benchmark and a comprehensive benchmark for PDE inverse problems to enable, for the first time in AI for PDEs, evaluation of whether and to what extent model outputs support specified scientific claims. Our results show that numerical accuracy and evidential support can rank models differently, explain when and why they do so, and reveal that existing benchmarks can favor methods whose outputs provide weaker support for the scientific claims of interest. Together, we formalize, empirically demonstrate, and explain this evaluation–use mismatch in AI for PDEs.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
AI for PDEs, benchmarking, scientific evidence, numerical accuracy, evidential support

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_5awvOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper considers the problem of the use of AI for scientific "evidence" or claims validation and uses two existing benchmark datasets to validated the proposed model.

The paper proposes to model the evaluation of the scientific claim with a generic function of the model output, the target, and a given validation protcol.

The paper provides code with the experimental validation of the proposed metric and comparing to the error of the single data point.

The main conclusion is that a model that has the best accuracy is now always the model that helps with support for scientific claim.

Critical strengths

The paper is addressing an important concern on the scientific community, the validation of a scientific claim. The paper proposes for example the validation of the presence of a symmetry in a specific "object of study". This seems a genuinely interesting prospective.

The paper also provides experimental validation on two benchmark problems and datasets: PDEBench and PDEInvBench. This makes the contribution more concrete.

The paper compares the metric for accuracy and for claim validation, showing that these metric leads to different ranking of the underly "AI-for-PDE methods"

The paper introduces a probabilistic model for the scientific claim validation.

Critical weaknesses

The presentation is substandard.

There is repetition in the description of the metric.

The "Evaluation-Use Mismatch" and its probabilistic version is overly complicated. All the definition are non used in the rest of the paper.

In general the description is not clear:

  1. What is "AI-for-PDE benchmarks"?
  2. what is an “AI outputs" ?
  3. what is "AI for PDEs"?

In the introduction, the paper mention "PINN-derived self-similar profiles provide numerical evidence for candidate singularity scenarios in the Boussinesq and Euler equations" with the reference, but it is not really clear what this means. Also later in "USE OF MODEL OUTPUTS AS SCIENTIFIC EVIDENCE" the description of the two motivating examples are not really clear. "with Amery as an explicit exception". it seems that the paper want for example in the inverse problem compare inverse problem paramters derived by the "AI for PDE methods" matches the experimental values.

One important aspect seem to be the "evidence standard" . This would require a dericated description.

In general, the paper is hard to understand, it is not coherent. For example naming in the table and in the text have not clear correspondance: is "Reference target" the same as "object of Study"?

Having a question of the title is suggest that the paper is not able to address the question?

"In this section, we answer three questions []" Also in Chapter 5 the paper spels the questions, while it would have been better to describe what the problem and how the results of the paper address this problem.

The paper seems to claim to provide a benchmark, but as I understand they paper provides a "Mismatch" measure and its evaluation.

Questions for authors

Question:

I do not have question for the authors. I think the current state of the paper is not enough to be able to ask questions.

Suggestions:

While the authors aknowledge the use of AI, this should be use to clarify and not to generate not coherent text.

I personally find interesting the underly problem the authors want to address in this work but the approach proposed is experimental and does not help the understanding.

I would suggest the authors to have a critical review of thier work and separate the method to the evaluation.

Futher a deeper analysis of the experiments and the proposed "EVALUATION–USE MISMATCH" would increase the contribution.

Reviewer_r8MHOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper addressed a very important gap in AI4PDE between the PDE models and their application. I appreciate the author explicitly pointing it out and designing systematic experiments to justify their claim. However, I find that this paper provides no insight into this topic, nor does it offer any takeaways to guide other model training, evaluation, or benchmarking, which could distinguish it from other publications (see weakness). For the paper, I appreciate the authors’ effort to formalize the gap, but it is not useful to my understanding of the claims because too many variables are flooding the paper. The paper's structure is not well organized either. (e.g., duplication of the central gap claim on pages 1 and 4; Table 1 doesn’t provide any useful information but occupies one page; …)

Critical strengths

Focusing on an interesting and important topic in AI4PDE.

Critical weaknesses
  1. The paper attempts to address the aforementioned gap from a general perspective, but I couldn’t see any new insights or takeaways by doing so. As far as I know, a similar evaluation-use mismatch is known to the community and has been studied in several publications that the paper failed to cite. (e.g., in WeatherBench 2, JAMES 2024; REFORMS, SCIENCE ADVANCE 2025, etc.) These works have already shown that model rankings can reverse when evaluated on different metrics and suggest using task-specific metrics for model evaluation and selection, which weakens the novelty and contribution of the current paper.
  2. Although the paper tried to give an “explanation” of why this mismatch happens, I don’t think the current one is fundamental and convincing enough. The paper attributes the mismatched one-sided claim case to the reference-to-boundary clearance, which is self-evident and meaningless. To my understanding, the root cause of the mismatch lies in the error distributions across different models (e.g., some models may have large errors on high- or low-frequency components), which would be more meaningful and would help the community understand and update our methods.
Questions for authors
  1. I found it hard to identify the key parameters of the models being compared, e.g., model size and training setting (from scratch or fine-tuned), which are critical to the results.
  2. If I understand it correctly, the reported rank could be changed or even reversed simply by changing \tau. If so, how did you choose the \tau (and other claim parameters) in the experiments? How much will the choice of these claim parameters affect the ranking?
  3. How are the common evaluation metrics (e.g., RMSE) related to the scientific claims the authors proposed in the paper? E.g., it seems to be a direct mathematical relation between the region mean and the global RMSE.
Reviewer_v2w8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper uses conformal prediction to evaluate whether AI outputs support specific scientific claims and shows that numerical accuracy and claim support can select different models for forward and inverse PDE tasks.

Critical strengths
  1. A model with a smaller prediction error may not provide stronger support for a specific scientific claim. The experiments show this difference in both forward and inverse PDE tasks. The main ranking differences also remain when the comparison is restricted to claims that are true according to the reference.

  2. The authors separate model fitting, validation, calibration and final evaluation. They compare methods on the same eligible examples. They also keep dependent observations together and report cases where the results are inconclusive. These choices make the main comparisons easier to assess.

Critical weaknesses
  1. My first concern is about what each part of the proposed evaluation contributes. The method combines a claim-related quantity with uncertainty calibration and an interval-containment rule. However, the bootstrap uncertainty analysis conditions on the realized calibration sample: the interval radii and claim-support verdicts are not recomputed within bootstrap draws. Although the paper includes sensitivity analyses, it remains unclear whether different calibration samples would lead to similar model selections.

  2. My second concern is about the uncertainty and stability of the experimental results. Appendix A.3 uses only 19 calibration groups in each primary stratum. The 95% interval radius is therefore the largest calibration error. However, the ranking uncertainty and robustness analyses keep these radii fixed. It is not clear whether the same models would be selected with another calibration sample. There is also a specific issue in the secondary prediction experiment. Appendix B.6 states that some panels share FNO and U-Net runs but resample their clusters separately. This does not preserve the dependence caused by those shared runs. The authors acknowledge these limitations, but the reported predictive improvement still needs a check using the shared runs as a common resampling unit.

Questions for authors
  1. Could the authors compare model rankings using error in the same claim-related quantity and the final calibrated support score? Extending the Darcy analysis in Appendix B.2 to the main panels would help explain what the evidence test adds beyond task-specific accuracy.

  2. Are the selected models stable when different calibration groups are used and the interval radii are recomputed? For the secondary prediction experiment, does the improvement remain statistically supported when panels sharing the same model runs are resampled together?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.