- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper considers the problem of the use of AI for scientific "evidence" or claims validation and uses two existing benchmark datasets to validated the proposed model.
The paper proposes to model the evaluation of the scientific claim with a generic function of the model output, the target, and a given validation protcol.
The paper provides code with the experimental validation of the proposed metric and comparing to the error of the single data point.
The main conclusion is that a model that has the best accuracy is now always the model that helps with support for scientific claim.
Critical strengths
The paper is addressing an important concern on the scientific community, the validation of a scientific claim. The paper proposes for example the validation of the presence of a symmetry in a specific "object of study". This seems a genuinely interesting prospective.
The paper also provides experimental validation on two benchmark problems and datasets: PDEBench and PDEInvBench. This makes the contribution more concrete.
The paper compares the metric for accuracy and for claim validation, showing that these metric leads to different ranking of the underly "AI-for-PDE methods"
The paper introduces a probabilistic model for the scientific claim validation.
Critical weaknesses
The presentation is substandard.
There is repetition in the description of the metric.
The "Evaluation-Use Mismatch" and its probabilistic version is overly complicated. All the definition are non used in the rest of the paper.
In general the description is not clear:
- What is "AI-for-PDE benchmarks"?
- what is an “AI outputs" ?
- what is "AI for PDEs"?
In the introduction, the paper mention "PINN-derived self-similar profiles provide numerical evidence for candidate singularity scenarios in the Boussinesq and Euler equations" with the reference, but it is not really clear what this means. Also later in "USE OF MODEL OUTPUTS AS SCIENTIFIC EVIDENCE" the description of the two motivating examples are not really clear. "with Amery as an explicit exception". it seems that the paper want for example in the inverse problem compare inverse problem paramters derived by the "AI for PDE methods" matches the experimental values.
One important aspect seem to be the "evidence standard" . This would require a dericated description.
In general, the paper is hard to understand, it is not coherent. For example naming in the table and in the text have not clear correspondance: is "Reference target" the same as "object of Study"?
Having a question of the title is suggest that the paper is not able to address the question?
"In this section, we answer three questions []" Also in Chapter 5 the paper spels the questions, while it would have been better to describe what the problem and how the results of the paper address this problem.
The paper seems to claim to provide a benchmark, but as I understand they paper provides a "Mismatch" measure and its evaluation.
Questions for authors
Question:
I do not have question for the authors. I think the current state of the paper is not enough to be able to ask questions.
Suggestions:
While the authors aknowledge the use of AI, this should be use to clarify and not to generate not coherent text.
I personally find interesting the underly problem the authors want to address in this work but the approach proposed is experimental and does not help the understanding.
I would suggest the authors to have a critical review of thier work and separate the method to the evaluation.
Futher a deeper analysis of the experiments and the proposed "EVALUATION–USE MISMATCH" would increase the contribution.