acceptodds
Under review as a conference paper at ICLR 2027

FluidBench: Diagnosing the Knowing–Doing Gap in Scientific AI through Fluid Mechanics

Abstract

As large language models and AI agents move from answering scientific questions toward solving real scientific problems, a critical question is whether scientific knowledge can be reliably translated into scientific action. Fluid mechanics offers a revealing testbed: success requires both applying physical knowledge to concrete problems and sustaining reliable computational research workflows. Existing benchmarks, however, typically evaluate scientific knowledge, problem solving, or research execution in isolation. We introduce **FluidBench**, a multi-level benchmark for diagnosing the *knowing–doing gap* in fluid mechanics. **FluidBench-Fundamental** contains 625 textbook-grounded problems assessing foundational knowledge; **FluidBench-Application** comprises 329 multi-stage problems with matched prerequisite probes and six-dimensional failure diagnosis; and **FluidBench-Research** contains 10 open-ended tasks across five research topics for evaluating end-to-end scientific agents. Experiments reveal two distinct gaps. First, a *knowledge-to-application gap*: models frequently answer all prerequisite probes correctly yet fail the original problem, with physical modeling emerging as the major bottleneck. Second, a *knowledge-to-research gap*: despite strong foundational fluid-mechanics performance, current agent systems remain far below reference-study quality, with failures concentrated in sustaining controlled experiments and complete, verifiable evidence. Together, these results show that strong scientific knowledge alone is insufficient for either reliable problem-specific reasoning or autonomous scientific research.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, AI Agents, Fluid Mechanics, Scientific Reasoning, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_k5W7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that existing scientific AI benchmarks either assess:

  • scientific knowledge, -closed-form problem solving, or -autonomous scientific research,

but rarely connect these levels into a unified framework.

To address this gap, the authors propose FluidBench, illustrated in Figure 1.

FluidBench consists of three components: -FluidBench-Fundamental 625 textbook-derived questions from six fluid mechanics textbooks. Evaluates foundational knowledge.

-FluidBench-Application 329 expert-reviewed applied fluid-mechanics problems. Includes matched prerequisite probes and six-dimensional failure diagnosis.

-FluidBench-Research 10 open-ended scientific research tasks spanning five research themes. Evaluates autonomous scientific workflows.

The benchmark construction process is described using a multi-stage generation and auditing pipeline.

Experiments on eight frontier LLMs show: -Strong foundational performance (>80% overall accuracy). -Substantially weaker application performance. -Frequent application failures despite successful prerequisite recall. -Very weak performance on research tasks, with average scores far below reference-study quality.

The authors conclude that scientific knowledge alone is insufficient for robust scientific action.

Critical strengths

1 Strong and Original Benchmarking Question The paper investigates an important and underexplored question: Where does scientific AI fail when moving from knowing something to actually using it? Most benchmarks measure final performance but not the transition between knowledge and action. The proposed “knowing-doing gap” provides a compelling conceptual lens.

2 Well-Designed Three-Level Evaluation Structure The benchmark spans: foundational knowledge, application-level reasoning, autonomous research. This layered design provides a richer analysis than most existing scientific benchmarks.

3 Diagnostic Innovation Through Matched Prerequisite Probes A particularly strong contribution is the use of prerequisite probes. Rather than simply observing failure, the benchmark asks whether the model already knows the required concepts, equations, and principles independently. This directly tests the hypothesised knowledge-to-application gap.

4 Fine-Grained Error Taxonomy The six-dimensional diagnostic framework: Information Extraction Physical Assumption Identification Governing Equation Selection Physical Modeling Mathematical Execution Physical Verification is well motivated and corresponds naturally to engineering problem-solving workflows.

5 Empirical Results Support Main Claims The strongest empirical result is the mismatch between: high prerequisite-probe success rates (79–94%) much lower application accuracy (55–72%) across all tested models. This is reasonably persuasive evidence that failure is not primarily due to missing factual knowledge.

6 Insightful Failure Analysis The paper goes beyond aggregate scores. Physical modeling (D4) is consistently the dominant failure mode across models, suggesting a common bottleneck in operationalising knowledge. The concrete example in Figure 4 clearly demonstrates the distinction between recalling an equation and using the correct equation within a problem context.

7 Research-Level Evaluation Is Ambitious The work attempts something still uncommon in benchmark literature: evaluating entire scientific workflows rather than isolated tasks. The identification of workflow failures such as: Evidence & Completeness Controls & Robustness Evaluation & Data Integrity is potentially valuable for future agent research.

Critical weaknesses

1 Heavy Dependence on LLM Judges The benchmark relies extensively on LLMs for: question review, answer grading, probe validation, failure diagnosis, research evaluation. Although expert involvement is described, quantitative validation of judge reliability is limited in the main paper. Key missing evidence: Inter-annotator agreement. Human-vs-LLM grading agreement. Calibration studies for failure labels. Without such analysis, it is difficult to assess whether diagnostic conclusions are robust.

2 Limited Validation of the Six-Dimensional Taxonomy The six-dimensional framework is intuitively appealing but appears largely designed by the authors. Missing evidence includes: annotation agreement, ambiguity assessment, alternative taxonomy comparison, expert consistency studies. Many real failures may plausibly fit multiple categories.

3 Small Research Benchmark FluidBench-Research includes only: 10 tasks, 5 topics, 2 tasks per topic. This limits statistical confidence. The core claim regarding the knowledge-to-research gap may therefore be sensitive to task selection.

4 Reproducibility Concerns for Research Evaluation The research evaluation framework introduces: custom rubrics, weighted scoring, hard caps, workspace-aware agent judging. While sensible, significant evaluator discretion remains. The paper does not sufficiently demonstrate that different judges would produce similar scores.

5 Benchmark Domain Restriction All findings are derived from fluid mechanics. The authors imply broader relevance to scientific AI, but evidence presented only directly supports conclusions within this domain. Further validation beyond fluid mechanics would strengthen claims of generality.

Questions for authors
  1. How reliable is the evaluation and diagnosis pipeline? Question: The benchmark relies heavily on LLM judges for answer grading, prerequisite-probe validation, six-dimensional failure diagnosis, and research-task assessment. What are the human-vs-LLM agreement rates for each stage, and how sensitive are the reported conclusions to judge choice?

  2. Does the "knowledge-to-application gap" remain after controlling for probe construction? Question: The paper reports that 79–94% of failed application problems still pass all matched prerequisite probes. How do the authors validate that these probes are truly complete and sufficient representations of the knowledge needed for the original problem, rather than simplified recall tests that omit essential reasoning components?

  3. How robust and generalisable are the research-level conclusions? Question: FluidBench-Research contains 10 tasks across 5 topics. What evidence supports the claim that the observed "knowledge-to-research gap" generalises beyond these tasks, and how stable are the results under different task selections, judges, and scientific domains?

6 comments
AuthorsOpenReview ↗

Response to Reviewer k5W7 (1/4)

Dear Reviewer,

Thank you for your careful review and positive feedback. We address your concerns as follows.

W1 & Q1: Reliability of the Evaluation and Diagnosis Pipeline

Response:

We thank the reviewer for highlighting the importance of evaluation reliability. We agree that systematic human validation are essential for ensuring that benchmark conclusions are reliable.

To address this concern, we conducted additional human evaluations across different evaluation stages.

1. Reliability and judge sensitivity of FluidBench-Fundamental and FluidBench-Application grading.

The first two levels of FluidBench primarily consist of problems with well-defined reference answers. Fundamental questions include multiple-choice, fill-in-the-blank, numerical, and symbolic problems, while prerequisite probes assess explicit concepts, formulas, and symbolic expressions. These structured answer formats substantially constrain grading ambiguity.

To validate grading reliability, we sampled 200 FluidBench-Fundamental responses and 100 FluidBench-Application responses for human verification. We evaluated three candidate judges: Gemini 3.5 Flash, DeepSeek V4 Flash, and MiniMax M3.

LLM JudgeFundamental (n=200)Application (n=100)
Gemini 3.5 Flash100% (200/200)98% (98/100)
DeepSeek V4 Flash100% (200/200)100% (100/100)
MiniMax M398.5% (197/200)97% (97/100)

All three judges achieved high agreement with human evaluations. DeepSeek V4 Flash achieved perfect agreement on both validation subsets and was therefore selected as the final grading evaluator.

We further manually verified 200 prerequisite probes, DeepSeek V4 Flash agreed with human grading on 198 of 200 prerequisite-probe responses (99%). This provides critical evidence for the reliability of structured prerequisite assessment.

2. Human calibration and judge selection for Research-level evaluation.

Research-level evaluation is inherently more open-ended and requires assessment of complete scientific workflows rather than exact-match answers. We therefore employ a workspace-aware Coding-Agent-as-a-Judge approach, supported by task-specific rubrics.

To validate this approach, we sampled 20 research workspaces—specifically, two workspaces per task—ensuring coverage across all harness configurations. These 20 sampled workspaces were subsequently evaluated by two domain experts.

Both experts independently evaluated the workspaces using the same task-specific rubrics as the original Coding-Agent-as-a-Judge. The results are summarized below.

First half of the human evaluation results.

MetricAM1 Codex+GPTAM1 CC+OpusAM2 Codex+M3AM2 CC+OpusSD1 Codex+GPTSD1 CC+M3SD2 Codex+GPTSD2 CC+M3PINN1 Codex+GPTPINN1 CC+Opus
Original Score36.3115.5120.506.0429.0824.1225.9716.4816.105.70
Expert 130.8712.7521.965.5827.6123.2728.9216.6722.404.50
Expert 231.288.3722.565.1328.2024.2827.2217.3922.256.00

Second half of the human evaluation results.

MetricPINN2 Codex+GPTPINN2 Codex+M3NO1 CC+OpusNO1 CC+M3NO2 Codex+M3NO2 CC+M3SR1 Codex+M3SR1 CC+OpusSR2 Codex+M3SR2 CC+M3Mean
Original Score27.646.8420.7130.0017.4913.8324.9510.2211.4618.2618.86
Expert 122.119.9922.3228.5614.9812.7616.4910.209.4612.7517.71
Expert 220.687.8224.9025.3412.1512.9915.999.106.6411.4916.99

CC = Claude Code; GPT = GPT-5.6 Sol; Opus = Opus 4.8; M3 = MiniMax M3.

The two human experts demonstrate high inter-rater consistency (Pearson = 0.978; Spearman = 0.979), with a mean absolute difference (MAE) of 1.48 points. The original LLM Judge also shows strong correlations with the mean human scores (Pearson = 0.904; Spearman = 0.895). The MAEs between the LLM Judge and Expert 1, Expert 2, and their mean scores are 2.72, 3.47, and 3.07 points, respectively.

We additionally compared two Coding-Agent-as-a-Judge configurations: Codex + GPT-5.6 Luna and Claude Code + Claude Sonnet 4.6. Their Pearson correlations with mean human scores were 0.904 and 0.769, respectively. Based on this comparison, we selected Codex + GPT-5.6 Luna as the final research evaluator.

3. Evaluation Reliability and Manuscript Revisions.

Together, these results demonstrate that our evaluator selection is empirically grounded in human validation across all three levels of FluidBench. The high human–LLM agreement support the reliability of our evaluation pipeline.

AuthorsOpenReview ↗

Response to Reviewer k5W7 (2/4)

W2 & Q1: Validity and Reliability of the Six-Dimensional Failure Taxonomy

Response:

We thank the reviewer for highlighting the importance of taxonomy validity. We provide the following clarification on the specific details of the six-dimensional diagnosis.

1. Workflow-oriented taxonomy.

Our six-dimensional taxonomy follows the sequential reasoning process of solving applied fluid-mechanics problems:

  • D1: Information Extraction
  • D2: Physical Assumption Identification
  • D3: Governing Equation Selection
  • D4: Physical Modeling
  • D5: Mathematical Execution
  • D6: Physical Verification

A single incorrect solution may involve multiple failures. To distinguish the initiating error from its consequences, our framework explicitly separates primary, contributing, and downstream failures, as defined in Section 3.2 of the paper. For our analysis, we record the earliest identifiable primary failure responsible for the incorrect solution. For example, selecting an inappropriate governing equation (D3) may subsequently lead to incorrect physical modeling (D4) and mathematical execution (D5). In this case, D3 is identified as the primary failure, while the subsequent errors are treated as downstream consequences.

This stage-based attribution strategy helps distinguish the primary failure from its downstream consequences, thereby reducing ambiguity when multiple errors occur within the same solution.

2. Independent expert validation and annotation consistency.

To quantitatively assess the reliability of the taxonomy, we sampled 80 incorrect Application responses, with 10 instances from each of the eight evaluated models. Two domain experts independently annotated the primary failure dimension.

The experts initially disagreed on only 2 of the 80 instances (97.5% initial agreement). After further clarification of the diagnostic definitions and expert discussion, they reached consensus on all 80 cases. The resulting consensus labels were then compared against the automated diagnoses. The automated diagnosis agreed with the expert consensus in 69/80 cases (86.25%). The detailed category-level results are shown below.

Primary failureHuman expertsLLM JudgeAgreement
D1: Information Extraction975
D2: Physical Assumption Identification565
D3: Governing Equation Selection262925
D4: Physical Modeling383632
D5: Mathematical Execution222
D6: Physical Verification000
Total808069

Beyond instance-level agreement, the expert and automated diagnoses reveal highly consistent failure distributions. In particular, governing-equation selection (D3) and physical modeling (D4) account for 80.0% (64/80) of expert-identified primary failures, compared with 81.25% (65/80) under automated diagnosis. These results provide evidence that our central diagnostic finding—the concentration of application failures in governing-equation selection and physical modeling—is preserved under independent expert assessment.

We will incorporate these quantitative validation results and further clarify the failure-attribution protocol in our paper.

W3: Scale and Representativeness of FluidBench-Research

Response:

We thank the reviewer for this observation. We emphasize that FluidBench-Research prioritizes domain value, research representativeness, and execution feasibility rather than task count alone. Our 10 tasks are carefully designed across five distinct research directions—Aerodynamic Modeling, Physics-Informed Neural Networks, Super-Resolution, Scientific Discovery, and Neural Operators—requiring different scientific knowledge, computational methods, and research capabilities.

Importantly, these are not isolated coding or problem-solving tasks, but open-ended, end-to-end scientific research workflows. Agents must autonomously formulate research approaches, design and execute experiments, analyze results, validate findings, and draw evidence-supported conclusions.

This task scale is also characteristic of existing benchmarks evaluating realistic scientific research workflows:

Benchmark# TasksResearch Scope
PaperBench [1]2012 ML research topics
AIRS-Bench [2]20Language modeling, mathematics, bioinformatics, time series, etc.
PRBench [3]3011 physics subfields
ResearchClawBench [4]4010 scientific domains
FluidBench-Research (Ours)105 fluid-mechanics research directions

Compared with conventional problem-solving benchmarks, constructing and evaluating these research-intensive tasks requires substantial domain expertise, computational resources, and detailed scientific evaluation criteria. In future work, we plan to expand the benchmark with additional research tasks covering more diverse scientific problems

Continued in the next comment.

AuthorsOpenReview ↗

Response to Reviewer k5W7 (3/4)

Continued from our previous comment.

References

  • [1] PaperBench: Evaluating AI's Ability to Replicate AI Research
  • [2] AIRS-Bench: A Suite of Tasks for Frontier AI Research Science Agents
  • [3] PRBench: End-to-end Paper Reproduction in Physics Research
  • [4] ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research

W4: Reproducibility of Research-Level Evaluation

Response:

We thank the reviewer for highlighting the importance of reproducibility in open-ended research evaluation. We agree that different Coding-Agent-as-a-Judge may exhibit scoring differences, making empirical judge selection and evaluation consistency essential.

1. Human calibration and judge selection.

We compared two Coding-Agent-as-a-Judge configurations against independent human assessments of 20 research workspaces. The selected Codex + GPT-5.6 Luna evaluator achieved a Pearson correlation of 0.904 with mean human scores, compared with 0.769 for Claude Code + Claude Sonnet 4.6. So rather than assuming different judges are alternative, we selected the evaluator demonstrating stronger agreement with independent human assessments.

2. Test–retest reproducibility.

To assess scoring stability, we sampled 10 research workspaces and independently repeated the evaluation three times, using the selected judge and identical evaluation criteria. The results are as follows.

EvaluationAM1AM2SD1SD2PINN1PINN2NO1NO2SR1SR2
Run 136.316.0424.1225.975.706.8430.0017.4924.9518.26
Run 233.267.9827.3326.116.108.0025.3619.8625.2214.38
Run 335.986.5524.1227.315.836.5031.8018.9627.8816.54
MetricResult
Mean scores across three runs19.57 / 19.36 / 20.15
Pairwise Pearson correlations0.969–0.993
Mean within-workspace standard deviation1.43 points

The repeated evaluations produced highly consistent scores and relative rankings, demonstrating strong test–retest reproducibility under the same judging configuration.

3. Structured rubrics and reproducibility materials.

Our task-specific rubrics were developed through multiple rounds of human expert iteration, together with explicit hard-cap criteria. To facilitate review, we have included our task-specific rubrics, criterion weights, and hard-cap definitions in the supplementary materials. We will publicly release the complete dataset, rubrics, and evaluation code in the future.

Q2: Does the Knowledge-to-Application Gap Remain After Controlling for Probe Construction?

Response:

We thank the reviewer for this important question. To address this concern, we clarify the intended scope of the probes and provide additional expert validation to assess whether the gap persists after controlling for probe construction.

1. What do prerequisite probes measure?

Our prerequisite probes are designed to assess the independently retrievable foundational knowledge required by an application problem, including foundational concepts, formulas, governing equations, and physical principles.

Crucially, we distinguish knowing the required knowledge from applying it in a specific physical context. The probes test whether models can recall the required knowledge, while application problems assess whether they can apply it through physical reasoning.

Therefore, the probes do not reproduce the complete problem-specific reasoning process. This is not an omission of the evaluation, but the intended distinction between knowledge recall and knowledge application.

2. Expert validation of prerequisite coverage.

We invited two domain experts to conduct an additional validation. Across all evaluated models, there were 193 unique incorrectly probe-eligible problems after deduplication. Both experts independently assessed the completeness of prerequisite coverage. Overall, 167 of 193 eligible problems (86.53%) passed the joint expert audit, retaining 457 prerequisite probes.

3. Does the knowledge-to-application gap persist?

Yes. We repeated the All-Probe Pass analysis using only the problems whose prerequisite probes passed the expert review. The results are shown below.

ModelOriginal All-Probe PassValidated All-Probe Pass
Qwen3.8-27B82.03% (105/128)80.00% (92/115)
Qwen3.5-397B-A17B94.29% (99/105)90.53% (86/95)
MiniMax M387.12% (115/132)84.50% (109/129)
GLM-5.3-Flash91.07% (102/112)89.11% (90/101)
GPT-5.6 Sol89.72% (96/107)88.00% (88/100)
Claude Sonnet 4.679.43% (112/141)79.69% (102/128)
Claude Opus 4.883.56% (122/146)82.31% (107/130)
Gemini 3.6 Flash91.11% (82/90)91.76% (78/85)

Continued in the next comment.

Reviewer_GbpROpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces a benchmark that evaluates the capabilities of LLM-based agents to translate scientific knowledge into actions in fluid mechanics. The benchmark evaluations have three levels: foundational knowledge, physical modeling and reasoning, and agent-level autonomous scientific research. It provides diagnostics for the transition gap between knowledge and action with matched prerequisite probes, analysis of failures at the application level, and end-to-end task-specific evaluation of research workflows. Experiments across eight models at fundamental and application levels. Results show that the models generally have strong knowledge of fluid mechanics and aerodynamics, whereas their knowledge of CFD principles and applications is limited in some models. However, diagnostic results show that possessing the relevant physical knowledge does not guarantee reliable operationalization in context. They still fail on end-to-end research tasks, where the dominant bottlenecks are sustaining scientifically valid experimental workflows, including experimental control, execution, and evidence construction.

Critical strengths
  • The benchmark evaluates meaningful problems in scientific agents, and the conclusions are clear and easy to interpret.
  • The benchmark is organized well into three modules and a six-dimensional diagnostic framework to identify failure modes from different perspectives.
  • The benchmark dataset contains 625 problems to test fundamental knowledge, 329 multi-stage problems with matched prerequisite probes to evaluate problem-solving skills, and 10 open-ended tasks across five research topics to assess the end-to-end capabilities. The construction of the dataset involves human evaluation, making it more reliable.
Critical weaknesses
  • Despite the results identifying gaps, the underlying root causes have not been explained well in the diagnosis.
  • The distinction between the knowledge-to-application gap and the real task difficulty needs further clarification.
  • There is no human evaluation for open-ended scientific research workflows.
  • The workflow comparison is limited to two harnesses, with two backbone models for each harness. While the MiniMax M3 model is used for both harnesses, the comparison does not sufficiently isolate the effect of the harness. There
  • The formatting of the manuscript can be improved (e.g., the font size of some figures is too small to read; there is nothing in Fig. 10b).
Questions for authors
  • Can the authors provide detailed analyses of the major bottleneck in physical modeling? For example, what are the possible root causes of the bottleneck?
  • For open-ended workflow evaluation, does the score consider efficiency (e.g., resource consumption, time usage)?
  • Can the authors report the statistical significance of the results?
  • How can the benchmark be generalized beyond fluid mechanics?
8 comments
AuthorsOpenReview ↗

Response to Reviewer GbpR (1/4)

Dear Reviewer,

Thank you for your careful review. We address your concerns as follows.

W2: Distinguishing the Knowledge-to-Application Gap from Task Difficulty

Response:

We thank the reviewer for raising this important distinction. Before discussing the potential root causes of the physical modeling bottleneck (W1 & Q1), we would like to clarify what we mean by the knowledge-to-application gap and how FluidBench identifies it, as this distinction is important for understanding our primary contribution.

1. The knowledge-to-application gap is not merely a performance difference between two levels of difficulty.

We would like to emphasize that our central finding is not simply that models perform better on foundational questions (Level 1) than on more challenging application problems (Level 2). Such a performance difference alone would not establish a knowledge-to-application gap, since application problems naturally require more complex reasoning. Instead, our core contribution is a matched prerequisite–application evaluation design, which directly examines whether models can successfully apply the specific knowledge they can independently recall.

2. Matched prerequisite probes reveal the knowledge-to-application gap.

For each incorrectly answered application problem, we construct matched prerequisite probes based on its reference solution. These probes independently assess whether the model can recall the foundational concepts, formulas, governing equations, and physical principles required to solve that particular problem.

We then evaluate the model on these prerequisite probes. For most evaluated models, the All-Probe Pass rate exceeds 80%. This means that, for more than 80% of their incorrectly answered problems, the models can correctly answer all the corresponding prerequisite probes. In other words, models often fail to solve a problem even though they can correctly recall the foundational knowledge needed for its solution.

A concrete example is illustrated in Figure 4 of our manuscript. In this drop-structure problem, GPT-5.6 Sol correctly answers all three matched prerequisite probes, including the critical-depth relation, the hydrostatic-force relation, and the steady one-dimensional momentum equation. In particular, the model correctly recalls the momentum equation required to solve the problem. However, when solving the original problem, the same model incorrectly applies a lossless energy balance instead of the required control-volume momentum balance, resulting in an incorrect solution. Our six-dimensional diagnosis identifies this as a D3 governing-equation selection failure.

This example directly illustrates the knowledge-to-application gap: the model knows the relevant governing equation and can reproduce it correctly when tested independently, yet fails to select and apply it in the actual physical problem.


W1 & Q1: Underlying Causes of the Physical Modeling Bottleneck

Response:

We thank the reviewer for this insightful question. We agree that understanding the underlying causes of physical modeling failures is an important direction for future research.

As clarified in W2, the primary objective of FluidBench is to identify the knowing–doing gap and diagnose where the transition from scientific knowledge to its practical application breaks down. Our matched prerequisite probes and six-dimensional failure diagnosis reveal that physical modeling (D4) is the dominant primary-failure category across all eight evaluated models.

1. Empirical analysis of representative physical modeling failures.

To better understand this bottleneck, we collected D4 failure cases and conducted a preliminary analysis. Several representative examples are summarized below:

CaseObserved physical-modeling failure
#310: Oblique-shock reflectionIncorrectly transforms the reflected shock angle into the wall-relative reference frame, resulting in an erroneous geometric relationship.
#105: Jet–plate interactionFails to update the jet impact location and corresponding moment arm after the plate rotates.
#75: Compound-channel flowApplies a standard trapezoidal cross-section model without accounting for the geometric transition at the floodplain elevation.
#135: Flip-bucket flowOmits the inlet hydrostatic pressure force when constructing the control-volume momentum balance.

Continued in the next comment.

AuthorsOpenReview ↗

Response to Reviewer GbpR (2/4)

Continued from our previous comment.

2. Potential explanations for the physical modeling bottleneck.

Based on these observations, we hypothesize two possible contributing factors:

  1. Insufficient context-dependent physical grounding. Although models can recall general physical principles, they may struggle to translate problem-specific geometry, boundary conditions, and physical configurations into correct mathematical representations. For example, Cases #310, #105, and #75 show that incorrect geometric interpretations can lead to invalid physical models.

  2. Difficulty in integrating knowledge and constraints. Models may correctly identify individual governing equations but fail to combine all relevant physical relations and constraints into a coherent and complete model. For example, in Case #135, the model applies the correct momentum principle but fails to account for all forces acting on the control volume.

At a deeper level, these difficulties may reflect a broader generalization challenge in LLMs. Models may learn physical laws and familiar solution patterns from training data but struggle to adapt them to unfamiliar geometric configurations, boundary conditions, or flow regimes. In such cases, even seemingly minor changes in the physical setup may lead to incorrect modeling assumptions or mathematical representations.

As we have clarified before, the primary contribution of FluidBench is to reveal the knowing–doing gap and systematically localize where the transition from scientific knowledge to its practical application breaks down. Our findings identify physical modeling as a major bottleneck, providing concrete evidence of the difficulties models encounter when applying available knowledge to specific physical problems. We believe these diagnostic findings can inform future improvements in scientific AI. In particular, future training data and algorithmic approaches could place greater emphasis on context-dependent physical grounding, knowledge and constraint integration, with the goal of improving the generalization and reliability of scientific reasoning.

W3: Lack of Human Evaluation for Open-Ended Research Workflows

Response:

We agree that human validation is important for assessing the reliability of open-ended scientific research evaluation. To address this concern, we conducted an additional human evaluation. We sampled 20 research workspaces—specifically, two workspaces per task—ensuring coverage across all harness configurations. These 20 sampled workspaces were subsequently evaluated by two domain experts.

Both experts independently evaluated the workspaces using the same task-specific rubrics as the original Coding-Agent-as-a-Judge. The results are summarized below.

Table 1a. First half of the human evaluation results.

MetricAM1 Codex+GPTAM1 CC+OpusAM2 Codex+M3AM2 CC+OpusSD1 Codex+GPTSD1 CC+M3SD2 Codex+GPTSD2 CC+M3PINN1 Codex+GPTPINN1 CC+Opus
Original Score36.3115.5120.506.0429.0824.1225.9716.4816.105.70
Expert 130.8712.7521.965.5827.6123.2728.9216.6722.404.50
Expert 231.288.3722.565.1328.2024.2827.2217.3922.256.00

Table 1b. Second half of the human evaluation results.

MetricPINN2 Codex+GPTPINN2 Codex+M3NO1 CC+OpusNO1 CC+M3NO2 Codex+M3NO2 CC+M3SR1 Codex+M3SR1 CC+OpusSR2 Codex+M3SR2 CC+M3Mean
Original Score27.646.8420.7130.0017.4913.8324.9510.2211.4618.2618.86
Expert 122.119.9922.3228.5614.9812.7616.4910.209.4612.7517.71
Expert 220.687.8224.9025.3412.1512.9915.999.106.6411.4916.99

CC = Claude Code; GPT = GPT-5.6 Sol; Opus = Opus 4.8; M3 = MiniMax M3.

The two human experts demonstrate high inter-rater consistency (Pearson = 0.978; Spearman = 0.979), with a mean absolute difference (MAE) of 1.48 points. The original LLM Judge also shows strong correlations with the mean human scores (Pearson = 0.904; Spearman = 0.895). The MAEs between the LLM Judge and Expert 1, Expert 2, and their mean scores are 2.72, 3.47, and 3.07 points, respectively.

Importantly, both experts rated all 20 workspaces below the reference-study quality anchor of approximately 50, consistent with the original evaluation. These findings provide independent human validation of our workspace-aware evaluation and support our conclusion that current scientific agents remain substantially below reference-study level research quality.

AuthorsOpenReview ↗

Response to Reviewer GbpR (3/4)

W4 & Q3: Harness Comparison and Statistical Significance

Response:

1. Statistical Comparison Under a Shared Backbone Model

Our original experiments already include a controlled comparison where MiniMax M3 is evaluated using both Codex and Claude Code on the same ten research tasks. The two-sided paired t-test results on task-level research-quality scores are summarized below:

ComparisonMean Score Difference95% CIp-value
Claude Code − Codex (MiniMax M3)+0.67[-2.65, 3.98]0.660
GPT-5.6 Sol − MiniMax M3 (Codex)+6.37[1.62, 11.12]0.014

Under the same MiniMax M3 backbone, Codex and Claude Code achieve similar average research-quality scores (18.63 and 19.29, respectively), with no statistically significant difference detected (p = 0.660). In contrast, under the same Codex harness, GPT-5.6 Sol achieves a significantly higher average score than MiniMax M3 (p = 0.014). This suggests that backbone-model choice influences research quality.

2. Differences in Execution Time and API Cost

Although the two harnesses achieve similar average research-quality scores with MiniMax M3, their execution profiles differ considerably.

Harness + BackboneAvg. ScoreTotal Execution TimeTotal API Cost
Codex + MiniMax M318.6333.69 h$7.91
Claude Code + MiniMax M319.2918.61 h$16.83

With the backbone model held constant, Claude Code reduces total execution time by approximately 44.8% compared with Codex, while incurring approximately 2.13 times the estimated API cost. This reveals an interesting trade-off: different harnesses can exhibit substantially different execution and resource-consumption profiles even when their final research-quality scores are similar.

We will include the execution-time and API-cost comparison in the main text of the revised manuscript to present a more comprehensive account of the evaluation details.

W5: Figure Readability and Figure 10b

Response:

1. Figure readability. We have revised the font used in Figure 2 and increased the font size in both Figures 2 and 3, with appropriate layout adjustments to improve readability.

2. Figure 10b. We would like to clarify that the blank plot in Figure 10b is not due to any omissive content, but an actual artifact from an agent-generated research workspace. During execution, model training failed before completing the first epoch. Nevertheless, the agent reported seemingly complete quantitative results using manually specified synthetic values, despite the absence of successful experimental runs. This is precisely an example of the agent's failure in Evidence & Completeness: it presented quantitative claims without sufficient experimental evidence to support them.

Reviewer_pVAzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces FluidBench to evaluate whether scientific AI systems can apply domain knowledge to problem solving and research. The benchmark comprises 625 foundational questions, 329 application problems, and 10 open-ended research tasks, featuring matched prerequisite probes, six-dimensional failure diagnosis, and workspace-aware agent evaluation. Results reveal a substantial gap between knowledge recall and application performance, while research agents often fail to produce complete and scientifically reliable evidence. Expert audits partially validate question quality and error diagnosis.

Critical strengths
  1. The matched prerequisite, application design provides direct evidence of the knowing, doing gap by showing that models can recall relevant knowledge yet fail to apply it to corresponding problems.

  2. Workspace-level inspection of code and experimental artifacts identifies consequential research failures, including data leakage and reported findings unsupported by completed experiments.

Critical weaknesses
  1. The approximately 50-point reference-study scoring anchor lacks demonstrated expert calibration. Missing task-specific scoring rules and per-run resource budgets further limit reproducibility and make it difficult to distinguish model limitations from evaluation or resource constraints.

  2. Short prerequisite probes may not capture all knowledge required for application problems, particularly those that are only partially probeable. Without independent validation of prerequisite coverage and clearer eligibility statistics, the observed performance gap cannot be fully attributed to failures in knowledge application.

Questions for authors

1.What execution budgets and stopping criteria were used for research agents? How was the reference-study scoring anchor calibrated?

4 comments
AuthorsOpenReview ↗

Response to Reviewer pVAz (1/2)

Dear Reviewer,

Thank you for your careful review. We appreciate your constructive comments and have conducted additional expert validation and analyses to address your concerns.

Q1 & W1: How were the execution budgets, stopping criteria, and reference-study scoring anchor determined?

Response:

1. Execution budgets and stopping criteria. FluidBench-Research involves heterogeneous computational workloads. Some fluid-mechanics tasks require modeling large-scale flow-field simulation data, including neural network training that may take several hours.

Therefore, we did not impose a uniform time or token limit. Each agent operated autonomously until it reported task completion. All experiments were conducted on a server equipped with eight NVIDIA RTX 4090 GPUs, providing substantial computational resources for these tasks. This design allows the agents' true capabilities to be observed as fully as possible.

2. Execution time and cost. We additionally collected the execution time and API costs. Here are the results:

Harness + ModelAvg. ScoreAvg. TimeTotal TimeTotal API Cost
Codex + GPT-5.6 Sol25.0000:31:3505:15:45$12.03
Codex + MiniMax M318.6303:22:0933:41:32$7.91
Claude Code + MiniMax M319.2901:51:3818:36:22$16.83
Claude Code + Opus 4.813.0600:15:5902:39:51$71.43

The results reveal substantial differences in execution behavior. Notably, longer execution time does not necessarily lead to higher research quality.

3. Reference-study scoring calibration. Our evaluation adopts a paper-relative scoring framework, in which approximately 50 points represent research quality comparable to the reference study. The task-specific rubrics were iteratively developed by human domain experts through multiple rounds of expert review; they assess textual evidence, source code, and figures. To calibrate this anchor, we packaged each original reference paper together with its associated source code and provided the package to three independent domain experts who were not involved in rubric construction.Each expert independently scored each reference study using its corresponding task-specific weighted rubric. The results are as follows:

ExpertAM1AM2SD1SD2PINN1PINN2NO1NO2SR1SR2Average
Expert 149.1447.8445.8449.2651.0843.1849.5649.2046.4149.3848.09
Expert 252.0752.2350.1454.2549.1852.1945.8747.8450.8648.2450.29
Expert 345.8243.8150.9249.9353.8550.0154.6745.7950.4751.8149.71
Mean49.0147.9648.9751.1551.3748.4650.0347.6149.2549.8149.36

Across all 10 tasks, the reference studies achieved an average score of 49.36, with task-level means ranging from 47.61 to 51.37. These results provide empirical support for our approximately 50-point reference anchor.

4. Reproducibility. To facilitate review, we have included our task-specific rubrics, criterion weights, and hard-cap definitions in the supplementary materials. We will publicly release the complete dataset, rubrics, and evaluation code in the future.

We will add these details about execution time, budget, and the 50-point calibration to the paper PDF.

AuthorsOpenReview ↗

Response to Reviewer pVAz (2/2)

W2: Do the prerequisite probes sufficiently cover the knowledge required for application problems?

Response:

1. Clarification of probeability and the scope of knowledge. We appreciate this concern and would like to clarify that the "knowing" component of our knowledge-to-application gap specifically refers to transferable foundational knowledge, including foundational concepts, formulas, governing equations, and physical principles.

Accordingly, we distinguish three categories:

  • Probeable: All required foundational prerequisites can be independently tested.
  • Partially probeable: The problem additionally involves task-specific information, such as empirical coefficients or parameters provided in the problem statement or accompanying tables.
  • Not probeable: No meaningful independent foundational recall probes can be constructed.

Importantly, partially probeable does not mean that only part of the targeted foundational knowledge can be probed. For example, one problem asks to “determine the jump type downstream of an overflow dam and design the dimensions of a baffle energy dissipator.” The targeted prerequisite knowledge includes the free-overflow formula for a practical weir, the critical-depth formula for a rectangular cross-section, the energy equation at the contracted section, and the conjugate-depth relation for a hydraulic jump. In addition, solving the problem requires empirical coefficients such as σ\sigma, m′m', and σ′\sigma', which are provided in a table accompanying the problem statement. These coefficients are therefore excluded from the probe set, and the problem is classified as partially probeable. Although task-specific information is also necessary for solving these problems, it is outside the scope of foundational knowledge. For partially probeable problems, we exclude such information and require the probes to cover all remaining independently testable foundational prerequisites.

Therefore, these problems remain valid for examining whether models can apply foundational knowledge that they can correctly recall in isolation.

2. Independent human validation. We invited two domain experts to independently audit whether the probes generated for all probe-eligible incorrect problems fully cover the foundational knowledge required by their reference solutions.

Across all evaluated models, there were 196 unique incorrectly answered problems after deduplication, comprising 154 probeable, 39 partially probeable, and 3 not-probeable problems. The latter were excluded from probe-based evaluation.

Both experts independently assessed the completeness of prerequisite coverage and the results are as follows.

CategoryOriginalExpert 1 ApprovedExpert 2 ApprovedBoth Approved
Probeable154141147139
Partially probeable39363028
Total193177177167

Approved indicates that the complete probe set covers all targeted foundational prerequisites.

Overall, 167 of 193 eligible problems (86.53%) passed the joint expert audit, retaining 457 prerequisite probes.

3. Robustness of the knowledge-to-application gap. We recomputed All-Probe Pass using only the expert-validated problems. The results before and after validation are summarized below:

ModelOriginal All-Probe PassValidated All-Probe Pass
Qwen3.8-27B82.03% (105/128)80.00% (92/115)
Qwen3.5-397B-A17B94.29% (99/105)90.53% (86/95)
MiniMax M387.12% (115/132)84.50% (109/129)
GLM-5.3-Flash91.07% (102/112)89.11% (90/101)
GPT-5.6 Sol89.72% (96/107)88.00% (88/100)
Claude Sonnet 4.679.43% (112/141)79.69% (102/128)
Claude Opus 4.883.56% (122/146)82.31% (107/130)
Gemini 3.6 Flash91.11% (82/90)91.76% (78/85)

After human validation, the All-Probe Pass rates remain high across all eight models (79.69%–91.76%), confirming that the observed knowledge-to-application gap persists.

We will update the All-Probe Pass results in the main paper using the expert-validated statistics and provide the detailed human audit protocol and results in the revised appendix.


Thank you again for your constructive feedback. We welcome any further questions or suggestions and would be happy to discuss them.

AuthorsOpenReview ↗

Revised Manuscript Update

Dear Reviewer,

Thank you again for your constructive feedback. We have uploaded the revised manuscript, incorporating the additional experiments and clarifications presented in our rebuttal. For ease of review, key revisions are highlighted in blue in the revised PDF. The relevant revisions are summarized below.

W1 & Q1: Execution Budgets, Stopping Criteria, and Reference-Study Calibration

  • Section 4.1 (p. 7): Clarified the execution protocol.
  • Section 4.4, Table 4 (p. 9): Added execution-time and API-cost statistics for all four agent configurations.
  • Appendix C.2, Table 8 (pp. 30–31): Detailed the rubric construction process and the calibration of the 50-point reference-study anchor by three experts.

W2: Completeness of Prerequisite Probes

  • Section 3.2 (pp. 4–5): Expanded the description of the matched prerequisite-probe methodology and clarified what foundational knowledge the probes measure. Clarified the distinction between probeable, partially probeable, and not probeable problems.
  • Section 4.3, Table 2 (pp. 7–8): Updated All-Probe Pass results using only expert-validated prerequisite sets.
  • Appendix D.3, Tables 10–11 (pp. 34–35): Added the independent two-expert audit protocol, complete audit statistics, and the results before and after human validation.

Thank you for your time and consideration. We welcome any further questions or suggestions.

Best regards,
The Authors

Dear reviewer pVAz,

With the discussion period ending soon, we wanted to follow up in case there are any questions or concerns we haven't fully addressed. We'd be happy to provide additional information. Thank you for your time to review our response!

Comments on the paper

2 comments

Dear reviewers, as we approach the end of the discussion period, I would appreciate a brief follow-up on whether the authors’ rebuttal resolves your concerns. Please consider updating your scores where appropriate, and thank you for your time!

AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chair,

As the discussion period approaches its end, we would like to summarize our main contributions and rebuttal outcomes. We sincerely appreciate the reviewers' constructive feedback and the Area Chair's efforts in facilitating the discussion.

1. Overview

We have provided substantive, evidence-supported responses to the major concerns raised by all three reviewers:

  • Reviewer pVAz: We addressed concerns about the 50-point reference-study calibration and prerequisite-probe coverage through expert validation, and clarified execution protocols and costs.
  • Reviewer GbpR: The reviewer acknowledged our responses concerning the knowledge-to-application gap, human evaluation, and execution costs, and increased the recommendation to Weak Accept.
  • Reviewer k5W7: The reviewer initially recommended Weak Accept, recognizing the original research question, three-level benchmark design, and supporting empirical findings. We further addressed evaluation reliability and reproducibility.

2. Main Contributions

FluidBench introduces a multi-level benchmark for diagnosing the knowing–doing gap in scientific AI through fluid mechanics, comprising 625 fundamental questions, 329 application problems, and 10 open-ended research tasks.

Our main contributions are:

  1. Multi-level Benchmark: Evaluating LLMs and scientific agents across foundational knowledge, applied physical reasoning, and open-ended scientific research in fluid mechanics.
  2. Diagnostic Framework: Combining matched prerequisite probes, six-dimensional failure diagnosis, and workspace-aware Coding-Agent-as-a-Judge evaluation of research workflows.
  3. Empirical Findings: Revealing the knowledge-to-application and knowledge-to-research gaps, showing that strong scientific knowledge does not guarantee reliable application or autonomous research.

3. Rebuttal Improvements and Additional Evidence

We conducted additional experiments and independent expert validation to address the major concerns raised by all three reviewers.

Reviewer pVAz:

  1. Execution Protocol and Scoring Calibration: Clarified the execution protocol and stopping criteria for research agents, reported agent execution times and API costs, and independently calibrated the 50-point reference-study anchor with three experts.
  2. Prerequisite-Probe Coverage: Domain experts independently audited prerequisite coverage, retaining 457 probes across 167 jointly approved problems. The validated All-Probe Pass rates remained high (79.69%–91.76% across eight models), confirming that the knowledge-to-application gap persists after expert validation.

Reviewer GbpR:

  1. Knowing–Doing Gap: Distinguished failures to apply independently recalled knowledge from differences in task difficulty.
  2. Physical Modeling: Identified two potential factors: insufficient context-dependent physical grounding and difficulty in integrating knowledge and constraints.
  3. Human Evaluation: Two experts evaluated 20 research workspaces, showing strong agreement between experts and with the LLM judge.
  4. Agent Harness: Analyzed statistical significance, runtime, and API costs, revealing efficiency differences despite similar research scores.

Reviewer k5W7:

  1. Evaluation and Diagnostic Reliability: Conducted independent human validation of answer grading, prerequisite-probe grading, and six-dimensional failure diagnosis, demonstrating strong agreement between automated evaluations and expert judgments.
  2. Research Evaluation Reproducibility: Repeated the evaluation three times, demonstrating highly consistent scores and relative rankings across runs.
  3. Prerequisite-Probe Coverage: Clarified that prerequisite probes assess independently retrievable foundational knowledge, and verified that the knowledge-to-application gap persists after expert validation of probe coverage.
  4. Research Scope and Generalizability: Clarified the diversity of the 10 open-ended research tasks across five research directions, and outlined a transferable prerequisite-probe methodology for investigating knowing–doing gaps in other scientific disciplines.

We have addressed all major reviewer concerns with additional evidence and incorporated the improvements into the revised manuscript, with changes highlighted in blue.

We sincerely appreciate Reviewer GbpR's decision to increase the recommendation to Weak Accept, as well as Reviewer k5W7's initially positive assessment. We respectfully hope that our additional experimental evidence and manuscript revisions will be carefully considered in the final assessment.

Thank you for your time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.