- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that existing scientific AI benchmarks either assess:
- scientific knowledge, -closed-form problem solving, or -autonomous scientific research,
but rarely connect these levels into a unified framework.
To address this gap, the authors propose FluidBench, illustrated in Figure 1.
FluidBench consists of three components: -FluidBench-Fundamental 625 textbook-derived questions from six fluid mechanics textbooks. Evaluates foundational knowledge.
-FluidBench-Application 329 expert-reviewed applied fluid-mechanics problems. Includes matched prerequisite probes and six-dimensional failure diagnosis.
-FluidBench-Research 10 open-ended scientific research tasks spanning five research themes. Evaluates autonomous scientific workflows.
The benchmark construction process is described using a multi-stage generation and auditing pipeline.
Experiments on eight frontier LLMs show: -Strong foundational performance (>80% overall accuracy). -Substantially weaker application performance. -Frequent application failures despite successful prerequisite recall. -Very weak performance on research tasks, with average scores far below reference-study quality.
The authors conclude that scientific knowledge alone is insufficient for robust scientific action.
Critical strengths
1 Strong and Original Benchmarking Question The paper investigates an important and underexplored question: Where does scientific AI fail when moving from knowing something to actually using it? Most benchmarks measure final performance but not the transition between knowledge and action. The proposed “knowing-doing gap” provides a compelling conceptual lens.
2 Well-Designed Three-Level Evaluation Structure The benchmark spans: foundational knowledge, application-level reasoning, autonomous research. This layered design provides a richer analysis than most existing scientific benchmarks.
3 Diagnostic Innovation Through Matched Prerequisite Probes A particularly strong contribution is the use of prerequisite probes. Rather than simply observing failure, the benchmark asks whether the model already knows the required concepts, equations, and principles independently. This directly tests the hypothesised knowledge-to-application gap.
4 Fine-Grained Error Taxonomy The six-dimensional diagnostic framework: Information Extraction Physical Assumption Identification Governing Equation Selection Physical Modeling Mathematical Execution Physical Verification is well motivated and corresponds naturally to engineering problem-solving workflows.
5 Empirical Results Support Main Claims The strongest empirical result is the mismatch between: high prerequisite-probe success rates (79–94%) much lower application accuracy (55–72%) across all tested models. This is reasonably persuasive evidence that failure is not primarily due to missing factual knowledge.
6 Insightful Failure Analysis The paper goes beyond aggregate scores. Physical modeling (D4) is consistently the dominant failure mode across models, suggesting a common bottleneck in operationalising knowledge. The concrete example in Figure 4 clearly demonstrates the distinction between recalling an equation and using the correct equation within a problem context.
7 Research-Level Evaluation Is Ambitious The work attempts something still uncommon in benchmark literature: evaluating entire scientific workflows rather than isolated tasks. The identification of workflow failures such as: Evidence & Completeness Controls & Robustness Evaluation & Data Integrity is potentially valuable for future agent research.
Critical weaknesses
1 Heavy Dependence on LLM Judges The benchmark relies extensively on LLMs for: question review, answer grading, probe validation, failure diagnosis, research evaluation. Although expert involvement is described, quantitative validation of judge reliability is limited in the main paper. Key missing evidence: Inter-annotator agreement. Human-vs-LLM grading agreement. Calibration studies for failure labels. Without such analysis, it is difficult to assess whether diagnostic conclusions are robust.
2 Limited Validation of the Six-Dimensional Taxonomy The six-dimensional framework is intuitively appealing but appears largely designed by the authors. Missing evidence includes: annotation agreement, ambiguity assessment, alternative taxonomy comparison, expert consistency studies. Many real failures may plausibly fit multiple categories.
3 Small Research Benchmark FluidBench-Research includes only: 10 tasks, 5 topics, 2 tasks per topic. This limits statistical confidence. The core claim regarding the knowledge-to-research gap may therefore be sensitive to task selection.
4 Reproducibility Concerns for Research Evaluation The research evaluation framework introduces: custom rubrics, weighted scoring, hard caps, workspace-aware agent judging. While sensible, significant evaluator discretion remains. The paper does not sufficiently demonstrate that different judges would produce similar scores.
5 Benchmark Domain Restriction All findings are derived from fluid mechanics. The authors imply broader relevance to scientific AI, but evidence presented only directly supports conclusions within this domain. Further validation beyond fluid mechanics would strengthen claims of generality.
Questions for authors
-
How reliable is the evaluation and diagnosis pipeline? Question: The benchmark relies heavily on LLM judges for answer grading, prerequisite-probe validation, six-dimensional failure diagnosis, and research-task assessment. What are the human-vs-LLM agreement rates for each stage, and how sensitive are the reported conclusions to judge choice?
-
Does the "knowledge-to-application gap" remain after controlling for probe construction? Question: The paper reports that 79–94% of failed application problems still pass all matched prerequisite probes. How do the authors validate that these probes are truly complete and sufficient representations of the knowledge needed for the original problem, rather than simplified recall tests that omit essential reasoning components?
-
How robust and generalisable are the research-level conclusions? Question: FluidBench-Research contains 10 tasks across 5 topics. What evidence supports the claim that the observed "knowledge-to-research gap" generalises beyond these tasks, and how stable are the results under different task selections, judges, and scientific domains?
6 comments
Response to Reviewer k5W7 (2/4)
W2 & Q1: Validity and Reliability of the Six-Dimensional Failure Taxonomy
Response:
We thank the reviewer for highlighting the importance of taxonomy validity. We provide the following clarification on the specific details of the six-dimensional diagnosis.
1. Workflow-oriented taxonomy.
Our six-dimensional taxonomy follows the sequential reasoning process of solving applied fluid-mechanics problems:
- D1: Information Extraction
- D2: Physical Assumption Identification
- D3: Governing Equation Selection
- D4: Physical Modeling
- D5: Mathematical Execution
- D6: Physical Verification
A single incorrect solution may involve multiple failures. To distinguish the initiating error from its consequences, our framework explicitly separates primary, contributing, and downstream failures, as defined in Section 3.2 of the paper. For our analysis, we record the earliest identifiable primary failure responsible for the incorrect solution. For example, selecting an inappropriate governing equation (D3) may subsequently lead to incorrect physical modeling (D4) and mathematical execution (D5). In this case, D3 is identified as the primary failure, while the subsequent errors are treated as downstream consequences.
This stage-based attribution strategy helps distinguish the primary failure from its downstream consequences, thereby reducing ambiguity when multiple errors occur within the same solution.
2. Independent expert validation and annotation consistency.
To quantitatively assess the reliability of the taxonomy, we sampled 80 incorrect Application responses, with 10 instances from each of the eight evaluated models. Two domain experts independently annotated the primary failure dimension.
The experts initially disagreed on only 2 of the 80 instances (97.5% initial agreement). After further clarification of the diagnostic definitions and expert discussion, they reached consensus on all 80 cases. The resulting consensus labels were then compared against the automated diagnoses. The automated diagnosis agreed with the expert consensus in 69/80 cases (86.25%). The detailed category-level results are shown below.
| Primary failure | Human experts | LLM Judge | Agreement |
|---|---|---|---|
| D1: Information Extraction | 9 | 7 | 5 |
| D2: Physical Assumption Identification | 5 | 6 | 5 |
| D3: Governing Equation Selection | 26 | 29 | 25 |
| D4: Physical Modeling | 38 | 36 | 32 |
| D5: Mathematical Execution | 2 | 2 | 2 |
| D6: Physical Verification | 0 | 0 | 0 |
| Total | 80 | 80 | 69 |
Beyond instance-level agreement, the expert and automated diagnoses reveal highly consistent failure distributions. In particular, governing-equation selection (D3) and physical modeling (D4) account for 80.0% (64/80) of expert-identified primary failures, compared with 81.25% (65/80) under automated diagnosis. These results provide evidence that our central diagnostic finding—the concentration of application failures in governing-equation selection and physical modeling—is preserved under independent expert assessment.
We will incorporate these quantitative validation results and further clarify the failure-attribution protocol in our paper.
W3: Scale and Representativeness of FluidBench-Research
Response:
We thank the reviewer for this observation. We emphasize that FluidBench-Research prioritizes domain value, research representativeness, and execution feasibility rather than task count alone. Our 10 tasks are carefully designed across five distinct research directions—Aerodynamic Modeling, Physics-Informed Neural Networks, Super-Resolution, Scientific Discovery, and Neural Operators—requiring different scientific knowledge, computational methods, and research capabilities.
Importantly, these are not isolated coding or problem-solving tasks, but open-ended, end-to-end scientific research workflows. Agents must autonomously formulate research approaches, design and execute experiments, analyze results, validate findings, and draw evidence-supported conclusions.
This task scale is also characteristic of existing benchmarks evaluating realistic scientific research workflows:
| Benchmark | # Tasks | Research Scope |
|---|---|---|
| PaperBench [1] | 20 | 12 ML research topics |
| AIRS-Bench [2] | 20 | Language modeling, mathematics, bioinformatics, time series, etc. |
| PRBench [3] | 30 | 11 physics subfields |
| ResearchClawBench [4] | 40 | 10 scientific domains |
| FluidBench-Research (Ours) | 10 | 5 fluid-mechanics research directions |
Compared with conventional problem-solving benchmarks, constructing and evaluating these research-intensive tasks requires substantial domain expertise, computational resources, and detailed scientific evaluation criteria. In future work, we plan to expand the benchmark with additional research tasks covering more diverse scientific problems
Continued in the next comment.
Response to Reviewer k5W7 (3/4)
Continued from our previous comment.
References
- [1] PaperBench: Evaluating AI's Ability to Replicate AI Research
- [2] AIRS-Bench: A Suite of Tasks for Frontier AI Research Science Agents
- [3] PRBench: End-to-end Paper Reproduction in Physics Research
- [4] ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research
W4: Reproducibility of Research-Level Evaluation
Response:
We thank the reviewer for highlighting the importance of reproducibility in open-ended research evaluation. We agree that different Coding-Agent-as-a-Judge may exhibit scoring differences, making empirical judge selection and evaluation consistency essential.
1. Human calibration and judge selection.
We compared two Coding-Agent-as-a-Judge configurations against independent human assessments of 20 research workspaces. The selected Codex + GPT-5.6 Luna evaluator achieved a Pearson correlation of 0.904 with mean human scores, compared with 0.769 for Claude Code + Claude Sonnet 4.6. So rather than assuming different judges are alternative, we selected the evaluator demonstrating stronger agreement with independent human assessments.
2. Test–retest reproducibility.
To assess scoring stability, we sampled 10 research workspaces and independently repeated the evaluation three times, using the selected judge and identical evaluation criteria. The results are as follows.
| Evaluation | AM1 | AM2 | SD1 | SD2 | PINN1 | PINN2 | NO1 | NO2 | SR1 | SR2 |
|---|---|---|---|---|---|---|---|---|---|---|
| Run 1 | 36.31 | 6.04 | 24.12 | 25.97 | 5.70 | 6.84 | 30.00 | 17.49 | 24.95 | 18.26 |
| Run 2 | 33.26 | 7.98 | 27.33 | 26.11 | 6.10 | 8.00 | 25.36 | 19.86 | 25.22 | 14.38 |
| Run 3 | 35.98 | 6.55 | 24.12 | 27.31 | 5.83 | 6.50 | 31.80 | 18.96 | 27.88 | 16.54 |
| Metric | Result |
|---|---|
| Mean scores across three runs | 19.57 / 19.36 / 20.15 |
| Pairwise Pearson correlations | 0.969–0.993 |
| Mean within-workspace standard deviation | 1.43 points |
The repeated evaluations produced highly consistent scores and relative rankings, demonstrating strong test–retest reproducibility under the same judging configuration.
3. Structured rubrics and reproducibility materials.
Our task-specific rubrics were developed through multiple rounds of human expert iteration, together with explicit hard-cap criteria. To facilitate review, we have included our task-specific rubrics, criterion weights, and hard-cap definitions in the supplementary materials. We will publicly release the complete dataset, rubrics, and evaluation code in the future.
Q2: Does the Knowledge-to-Application Gap Remain After Controlling for Probe Construction?
Response:
We thank the reviewer for this important question. To address this concern, we clarify the intended scope of the probes and provide additional expert validation to assess whether the gap persists after controlling for probe construction.
1. What do prerequisite probes measure?
Our prerequisite probes are designed to assess the independently retrievable foundational knowledge required by an application problem, including foundational concepts, formulas, governing equations, and physical principles.
Crucially, we distinguish knowing the required knowledge from applying it in a specific physical context. The probes test whether models can recall the required knowledge, while application problems assess whether they can apply it through physical reasoning.
Therefore, the probes do not reproduce the complete problem-specific reasoning process. This is not an omission of the evaluation, but the intended distinction between knowledge recall and knowledge application.
2. Expert validation of prerequisite coverage.
We invited two domain experts to conduct an additional validation. Across all evaluated models, there were 193 unique incorrectly probe-eligible problems after deduplication. Both experts independently assessed the completeness of prerequisite coverage. Overall, 167 of 193 eligible problems (86.53%) passed the joint expert audit, retaining 457 prerequisite probes.
3. Does the knowledge-to-application gap persist?
Yes. We repeated the All-Probe Pass analysis using only the problems whose prerequisite probes passed the expert review. The results are shown below.
| Model | Original All-Probe Pass | Validated All-Probe Pass |
|---|---|---|
| Qwen3.8-27B | 82.03% (105/128) | 80.00% (92/115) |
| Qwen3.5-397B-A17B | 94.29% (99/105) | 90.53% (86/95) |
| MiniMax M3 | 87.12% (115/132) | 84.50% (109/129) |
| GLM-5.3-Flash | 91.07% (102/112) | 89.11% (90/101) |
| GPT-5.6 Sol | 89.72% (96/107) | 88.00% (88/100) |
| Claude Sonnet 4.6 | 79.43% (112/141) | 79.69% (102/128) |
| Claude Opus 4.8 | 83.56% (122/146) | 82.31% (107/130) |
| Gemini 3.6 Flash | 91.11% (82/90) | 91.76% (78/85) |
Continued in the next comment.
Response to Reviewer k5W7 (1/4)
Dear Reviewer,
Thank you for your careful review and positive feedback. We address your concerns as follows.
Response:
We thank the reviewer for highlighting the importance of evaluation reliability. We agree that systematic human validation are essential for ensuring that benchmark conclusions are reliable.
To address this concern, we conducted additional human evaluations across different evaluation stages.
1. Reliability and judge sensitivity of FluidBench-Fundamental and FluidBench-Application grading.
The first two levels of FluidBench primarily consist of problems with well-defined reference answers. Fundamental questions include multiple-choice, fill-in-the-blank, numerical, and symbolic problems, while prerequisite probes assess explicit concepts, formulas, and symbolic expressions. These structured answer formats substantially constrain grading ambiguity.
To validate grading reliability, we sampled 200 FluidBench-Fundamental responses and 100 FluidBench-Application responses for human verification. We evaluated three candidate judges: Gemini 3.5 Flash, DeepSeek V4 Flash, and MiniMax M3.
All three judges achieved high agreement with human evaluations. DeepSeek V4 Flash achieved perfect agreement on both validation subsets and was therefore selected as the final grading evaluator.
We further manually verified 200 prerequisite probes, DeepSeek V4 Flash agreed with human grading on 198 of 200 prerequisite-probe responses (99%). This provides critical evidence for the reliability of structured prerequisite assessment.
2. Human calibration and judge selection for Research-level evaluation.
Research-level evaluation is inherently more open-ended and requires assessment of complete scientific workflows rather than exact-match answers. We therefore employ a workspace-aware Coding-Agent-as-a-Judge approach, supported by task-specific rubrics.
To validate this approach, we sampled 20 research workspaces—specifically, two workspaces per task—ensuring coverage across all harness configurations. These 20 sampled workspaces were subsequently evaluated by two domain experts.
Both experts independently evaluated the workspaces using the same task-specific rubrics as the original Coding-Agent-as-a-Judge. The results are summarized below.
First half of the human evaluation results.
Second half of the human evaluation results.
CC = Claude Code; GPT = GPT-5.6 Sol; Opus = Opus 4.8; M3 = MiniMax M3.
The two human experts demonstrate high inter-rater consistency (Pearson = 0.978; Spearman = 0.979), with a mean absolute difference (MAE) of 1.48 points. The original LLM Judge also shows strong correlations with the mean human scores (Pearson = 0.904; Spearman = 0.895). The MAEs between the LLM Judge and Expert 1, Expert 2, and their mean scores are 2.72, 3.47, and 3.07 points, respectively.
We additionally compared two Coding-Agent-as-a-Judge configurations: Codex + GPT-5.6 Luna and Claude Code + Claude Sonnet 4.6. Their Pearson correlations with mean human scores were 0.904 and 0.769, respectively. Based on this comparison, we selected Codex + GPT-5.6 Luna as the final research evaluator.
3. Evaluation Reliability and Manuscript Revisions.
Together, these results demonstrate that our evaluator selection is empirically grounded in human validation across all three levels of FluidBench. The high human–LLM agreement support the reliability of our evaluation pipeline.