- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper addresses two challenges in using large language models for computational fluid dynamics: domain knowledge is difficult to reuse, and successful workflow execution does not necessarily mean that the results satisfy numerical and physical requirements. The authors propose EasyFoam, a skill-oriented framework that uses simulation evidence to diagnose and repair failures within a task, then distills experience across tasks into reusable skills. The study also introduces ExtFoamBench, which contains 100 tasks spanning ten families of physical problems. Experiments compare EasyFoam with three agent baselines on both the existing FoamBench and ExtFoamBench, and examine the effect of skill evolution on held-out tasks.
Critical strengths
None
Critical weaknesses
None
Questions for authors
1.Were all 100 benchmark tasks reviewed by independent computational fluid dynamics experts? If so, please describe the review criteria and how the consistency of task descriptions, reference cases, and acceptance criteria was assessed.
2.How does the reference-based evaluation avoid rejecting physically valid solutions produced using different but reasonable meshes or numerical schemes? Please clarify how the acceptance tolerances were determined and whether alternative valid solutions were considered.
4 comments
Response to Reviewer W6Fd (Part 2/2)
Q1 (continued)
(3) Consistency among task descriptions, reference cases, and acceptance criteria.
To ensure consistency between the task descriptions and the reference cases, we manually checked the correspondence between the natural-language task requirements and the reference configuration files (0/, constant/, and system/). This review covered solver selection, physical models, initial and boundary conditions, material properties, numerical settings, termination criteria, and output requirements, ensuring that the explicitly specified computational conditions in each task are consistent with the corresponding reference case.
The task-coverage matrix in the appendix further records the major configuration requirements for each physical family and its variants using a unified format, providing a clear and traceable correspondence among task descriptions, reference configurations, and evaluation metrics.
For the acceptance criteria, we consistently use three evaluation dimensions: Execution Completion (E), Numerical Soundness (N), and QoI Satisfaction (Q). Execution Completion checks whether the required computational workflow is successfully completed. Numerical Soundness evaluates the stability and convergence behavior of the simulation together with relevant numerical constraints. QoI Satisfaction evaluates whether the key physical quantities, spatial distributions, or response trends satisfy the predefined task-specific requirements.
The specific evaluation metrics, comparison procedures, and tolerances are determined according to the physical characteristics of each task and its reference case. For each task, the same acceptance criteria are applied to all methods, ensuring fair cross-method comparison while accounting for the physical characteristics and evaluation requirements of different CFD problems.
Q2. Reference-Based Evaluation and Acceptance Tolerances
A2. In designing ExtFoamBench, we explicitly considered that different meshes and numerical schemes may lead to reasonable numerical variations. The primary goal of the current benchmark is to evaluate whether an agent can correctly understand and complete a specified CFD task. Therefore, the task descriptions generally specify the mesh resolution, numerical method, and relevant computational requirements to keep the evaluation target well defined and comparable across methods.
Even under these explicitly specified conditions, our experiments show that simultaneously satisfying Execution Completion, Numerical Soundness, and QoI Satisfaction remains substantially challenging for existing agents. We therefore believe that the current benchmark can effectively evaluate the fundamental computational capabilities of agents on complex CFD tasks.
In the evaluation, the acceptance tolerance is essentially a benchmark-design hyperparameter that allows reasonable numerical deviations between the candidate result and the reference result. We set the tolerance according to the physical characteristics and target QoIs of each task and apply the same criteria to all evaluated methods.
To examine whether the experimental conclusions are sensitive to the choice of this hyperparameter, we further conducted a tolerance-sensitivity analysis by tightening the original tolerances to and relaxing them to . EasyFoam maintains the best performance under all three tolerance settings, indicating that the main comparative conclusions are not sensitive to this particular hyperparameter choice.
The reviewer’s suggestion to further consider multiple reasonable meshes, numerical methods, and their corresponding valid solutions is very valuable. Such an extension would move the benchmark from the current setting of relatively well-specified computational tasks toward more open-ended CFD problems with a broader space of valid numerical choices. We consider this an important direction for future extensions of ExtFoamBench and EasyFoam.
Thank author for addressing all my questions. I have no further questions.
Thank you very much for your prompt feedback and for taking the time to consider our clarifications. We sincerely appreciate the opportunity to address your questions and are truly grateful for your thoughtful comments, consideration, and help in improving our manuscript. Thank you again for your time and support!
Response to Reviewer W6Fd (Part 1/2)
We sincerely thank the reviewer for the careful reading of our manuscript and for the constructive questions regarding the construction and evaluation of ExtFoamBench. We appreciate the reviewer’s attention to the consistency of the benchmark design, the validation of reference cases, and the potential influence of reference-based evaluation and acceptance tolerances. Below, we clarify these points in detail.
Q1. Consistency of ExtFoamBench Evaluation Criteria
A1. All tasks in ExtFoamBench were jointly constructed, reviewed, and finalized by two authors with backgrounds in aerodynamics and CFD. To ensure the scientific validity, physical reasonableness, and reproducibility of the benchmark, we followed a unified review and validation procedure covering three main aspects.
(1) Task sources and coverage design.
ExtFoamBench is constructed from authoritative and publicly available CFD cases, including OpenFOAM official tutorials, the NASA CDV validation cases, the CFD-China/dyfluid open-source repositories, and classical fluid mechanics literature. We first examined the limitations of existing OpenFOAM agent benchmarks in terms of task diversity and complexity, and then extended the evaluation to more challenging computational scenarios, including compressible flows, VOF multiphase flows, rotating machinery, and external aerodynamics. The resulting benchmark contains 10 physical families and 100 tasks, with one baseline case and nine variants in each family.
For the task difficulty design, we adopted a principle that separates geometric complexity from physical-condition complexity:
This structured difficulty design helps diagnose and distinguish the capability boundaries of agents across geometry handling, boundary-condition specification, and numerical solution stages.
(2) Systematic task construction and reference-case validation.
All tasks follow a unified construction protocol requiring a traceable public source, a reproducible reference case, and clearly defined, extractable, and verifiable quantities of interest (QoIs). Each physical family contains one baseline task and nine variants, generated by varying factors such as mesh resolution, boundary conditions, physical parameters, turbulence models, solvers and numerical schemes, execution procedures, and target QoIs. These variations are designed to evaluate the agent’s ability to adapt to diverse computational conditions.
The verification of the reference cases is all based on rigorous real numerical calculation verification:
checkMeshwas used to examine mesh topology and geometric quality, including key indicators such as aspect ratio, skewness, and non-orthogonality, ensuring that the meshes satisfy the basic requirements for the corresponding numerical simulations.