acceptodds
Under review as a conference paper at ICLR 2027

EasyFoam: Skill-Oriented OpenFOAM Agents with Evidence-Driven Evolution

Abstract

Large language model (LLM) agents are increasingly capable of automating end-to-end OpenFOAM workflows for computational fluid dynamics (CFD). However, current progress is bottlenecked by two critical limitations. First, existing evaluations are dominated by tutorial-level cases and execution-oriented metrics, masking a substantial executability–acceptability gap: successfully executed simulations often fail to satisfy prescribed numerical or quantity-of-interest (QoI) criteria. Second, domain expertise in existing systems is tightly coupled with bespoke agent architectures, preventing such knowledge from being externalized, reused, and continuously improved. To address these challenges, we introduce ExtFoamBench, a challenging benchmark for OpenFOAM agents that spans five axes of workflow variation, and separately assesses execution completion, numerical soundness, and QoI satisfaction. Complementing this, we propose EasyFoam, an agent framework that decouples CFD expertise into modular, externalized skills driven by a dual-loop mechanism. An execution loop is designed to fulfill user instructions, utilizing structured simulation evidence to diagnose errors and perform targeted repairs. Meanwhile, an evolution loop distills task experience into reusable skill updates selected through paired evaluation. Empirical evaluations demonstrate that EasyFoam outperforms baseline agents in bridging the executability-acceptability gap, establishing this work as a starting point for future research toward self-improving CFD agents.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
LLM Agents, CFD, OpenFOAM, Skills

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_W6FdOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses two challenges in using large language models for computational fluid dynamics: domain knowledge is difficult to reuse, and successful workflow execution does not necessarily mean that the results satisfy numerical and physical requirements. The authors propose EasyFoam, a skill-oriented framework that uses simulation evidence to diagnose and repair failures within a task, then distills experience across tasks into reusable skills. The study also introduces ExtFoamBench, which contains 100 tasks spanning ten families of physical problems. Experiments compare EasyFoam with three agent baselines on both the existing FoamBench and ExtFoamBench, and examine the effect of skill evolution on held-out tasks.

Critical strengths

None

Critical weaknesses

None

Questions for authors

1.Were all 100 benchmark tasks reviewed by independent computational fluid dynamics experts? If so, please describe the review criteria and how the consistency of task descriptions, reference cases, and acceptance criteria was assessed.

2.How does the reference-based evaluation avoid rejecting physically valid solutions produced using different but reasonable meshes or numerical schemes? Please clarify how the acceptance tolerances were determined and whether alternative valid solutions were considered.

4 comments
AuthorsOpenReview ↗

Response to Reviewer W6Fd (Part 1/2)

We sincerely thank the reviewer for the careful reading of our manuscript and for the constructive questions regarding the construction and evaluation of ExtFoamBench. We appreciate the reviewer’s attention to the consistency of the benchmark design, the validation of reference cases, and the potential influence of reference-based evaluation and acceptance tolerances. Below, we clarify these points in detail.

Q1. Consistency of ExtFoamBench Evaluation Criteria

A1. All tasks in ExtFoamBench were jointly constructed, reviewed, and finalized by two authors with backgrounds in aerodynamics and CFD. To ensure the scientific validity, physical reasonableness, and reproducibility of the benchmark, we followed a unified review and validation procedure covering three main aspects.

(1) Task sources and coverage design.

ExtFoamBench is constructed from authoritative and publicly available CFD cases, including OpenFOAM official tutorials, the NASA CDV validation cases, the CFD-China/dyfluid open-source repositories, and classical fluid mechanics literature. We first examined the limitations of existing OpenFOAM agent benchmarks in terms of task diversity and complexity, and then extended the evaluation to more challenging computational scenarios, including compressible flows, VOF multiphase flows, rotating machinery, and external aerodynamics. The resulting benchmark contains 10 physical families and 100 tasks, with one baseline case and nine variants in each family.

For the task difficulty design, we adopted a principle that separates geometric complexity from physical-condition complexity:

  • Simple geometry with complex physical boundary conditions: these tasks mainly evaluate the agent’s ability to configure boundary conditions and physical models, such as multiple inlet boundaries and shock interactions in supersonic jet-in-crossflow cases, or strong pressure gradients and interface evolution in compressible bubble collapse.
  • Complex geometry with clearly specified physical conditions: these tasks mainly evaluate mesh handling, far-field boundary treatment, and aerodynamic quantity extraction, such as transonic NACA0012 airfoil flows and DrivAer external aerodynamics.
  • Coordinated configuration across multiple physical regions: for example, the ERCOFTAC centrifugal-pump case evaluates whether the agent can correctly coordinate MRF rotating regions, rotating walls, and the associated boundary conditions.

This structured difficulty design helps diagnose and distinguish the capability boundaries of agents across geometry handling, boundary-condition specification, and numerical solution stages.

(2) Systematic task construction and reference-case validation.

All tasks follow a unified construction protocol requiring a traceable public source, a reproducible reference case, and clearly defined, extractable, and verifiable quantities of interest (QoIs). Each physical family contains one baseline task and nine variants, generated by varying factors such as mesh resolution, boundary conditions, physical parameters, turbulence models, solvers and numerical schemes, execution procedures, and target QoIs. These variations are designed to evaluate the agent’s ability to adapt to diverse computational conditions.

The verification of the reference cases is all based on rigorous real numerical calculation verification:

  • Mesh-quality checks: OpenFOAM checkMesh was used to examine mesh topology and geometric quality, including key indicators such as aspect ratio, skewness, and non-orthogonality, ensuring that the meshes satisfy the basic requirements for the corresponding numerical simulations.
  • Numerical convergence and stability: solver logs, residual histories, and the numerical behavior of key physical fields were examined to assess convergence and stability and to identify numerical divergence, non-physical oscillations, or abnormal numerical values.
  • Physical plausibility: the computed results were checked against the expected physical behavior of each task and its target QoIs. Examples include near-wall y+y^+ distributions under turbulence models, interface evolution and phase-volume conservation in VOF rising-bubble simulations, and flow structures and pressure distributions in NASA CDV nozzle cases under different pressure conditions.
AuthorsOpenReview ↗

Response to Reviewer W6Fd (Part 2/2)

Q1 (continued)

(3) Consistency among task descriptions, reference cases, and acceptance criteria.

To ensure consistency between the task descriptions and the reference cases, we manually checked the correspondence between the natural-language task requirements and the reference configuration files (0/, constant/, and system/). This review covered solver selection, physical models, initial and boundary conditions, material properties, numerical settings, termination criteria, and output requirements, ensuring that the explicitly specified computational conditions in each task are consistent with the corresponding reference case.

The task-coverage matrix in the appendix further records the major configuration requirements for each physical family and its variants using a unified format, providing a clear and traceable correspondence among task descriptions, reference configurations, and evaluation metrics.

For the acceptance criteria, we consistently use three evaluation dimensions: Execution Completion (E), Numerical Soundness (N), and QoI Satisfaction (Q). Execution Completion checks whether the required computational workflow is successfully completed. Numerical Soundness evaluates the stability and convergence behavior of the simulation together with relevant numerical constraints. QoI Satisfaction evaluates whether the key physical quantities, spatial distributions, or response trends satisfy the predefined task-specific requirements.

The specific evaluation metrics, comparison procedures, and tolerances are determined according to the physical characteristics of each task and its reference case. For each task, the same acceptance criteria are applied to all methods, ensuring fair cross-method comparison while accounting for the physical characteristics and evaluation requirements of different CFD problems.

Q2. Reference-Based Evaluation and Acceptance Tolerances

A2. In designing ExtFoamBench, we explicitly considered that different meshes and numerical schemes may lead to reasonable numerical variations. The primary goal of the current benchmark is to evaluate whether an agent can correctly understand and complete a specified CFD task. Therefore, the task descriptions generally specify the mesh resolution, numerical method, and relevant computational requirements to keep the evaluation target well defined and comparable across methods.

Even under these explicitly specified conditions, our experiments show that simultaneously satisfying Execution Completion, Numerical Soundness, and QoI Satisfaction remains substantially challenging for existing agents. We therefore believe that the current benchmark can effectively evaluate the fundamental computational capabilities of agents on complex CFD tasks.

In the evaluation, the acceptance tolerance is essentially a benchmark-design hyperparameter that allows reasonable numerical deviations between the candidate result and the reference result. We set the tolerance according to the physical characteristics and target QoIs of each task and apply the same criteria to all evaluated methods.

To examine whether the experimental conclusions are sensitive to the choice of this hyperparameter, we further conducted a tolerance-sensitivity analysis by tightening the original tolerances to 0.5×0.5\times and relaxing them to 2×2\times. EasyFoam maintains the best performance under all three tolerance settings, indicating that the main comparative conclusions are not sensitive to this particular hyperparameter choice.

The reviewer’s suggestion to further consider multiple reasonable meshes, numerical methods, and their corresponding valid solutions is very valuable. Such an extension would move the benchmark from the current setting of relatively well-specified computational tasks toward more open-ended CFD problems with a broader space of valid numerical choices. We consider this an important direction for future extensions of ExtFoamBench and EasyFoam.

Thank author for addressing all my questions. I have no further questions.

Thank you very much for your prompt feedback and for taking the time to consider our clarifications. We sincerely appreciate the opportunity to address your questions and are truly grateful for your thoughtful comments, consideration, and help in improving our manuscript. Thank you again for your time and support!

Reviewer_pC89OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

I understand this paper as an attempt to make CFD agents solve the requested problem rather than just finish a solver run. The authors build ExtFoamBench with 100 tasks from ten physical families and evaluate execution, numerical soundness, and QoI satisfaction separately. They also introduce EasyFoam, which puts CFD knowledge into external skills, uses evidence to repair the current workflow, and updates the skills through paired validation. They report improvements over three CFD agents and further improvements on 57 held-out tasks after evolution.

Critical strengths
  1. I think this is very much of an engineering contribution. Preparing cases across different physics, making the workflow variations runnable, and collecting comparable quantities require more CFD work than the framework diagram shows. I would count this case preparation and measurement work as a contribution, since a useful benchmark does not have to come with a completely new agent algorithm.

  2. I find the reported gains worth looking into. EasyFoam gets 41.3% TAR on FoamBench while the best external baseline gets 16.7%, and the evolved library passes 27/57 held-out tasks instead of 18/57. I also appreciate the reference repeats, physical negative controls, and tolerance changes, since the authors do try to check their measurement. My problem is that I find other measurement errors which these checks do not cover.

Critical weaknesses

I feel like the novelty description is larger than the actual method difference. CFDLLMBench, Section 3.3, already combines execution and solution accuracy; ChatCFD, Section 3.2, checks physical fidelity; and EvoSkill and SkillOpt already update skills from feedback and validation. I do not think every component has to be new, since a useful benchmark and convincing system improvements could already support acceptance. However, I do not yet find this evaluation worth-trusting.

I checked the scoring code with constructed inputs and reference replay. For workflow checks I supplied fixed command results. I did not run CFD or call the models.

  1. I have problems with what the acceptance scores count.

    In benchmarks/evaluation/comparison.py:94, the code removes reference coordinates outside the candidate range, while it counts profile-field combinations for coverage. I keep the other reference quantities unchanged and leave one sample in every principal profile, and still get QoI pass for all 110 Basic and 16 Advanced tasks. Cavity/1 gives full coverage and zero profile error with one sample per centerline. I find this coverage kind of suspicious, since almost no spatial response remains. These are constructed inputs, so I have not shown that historical outputs contain such profiles. Their real sampling intervals and rescoring would tell the influence on Table 1.

    I also replayed the complete Riemann reference profiles at their declared paths. All eight multibranch tasks fail, while both single-branch controls pass. profile_key removes the parent path and final_riemann_profiles keeps only the latest horizontal and vertical profiles, therefore the six profiles in riemann2D/2 become two and event coverage becomes one third. I should not blame EasyFoam's relative gain on this shared failure, as the extractor loses branches for every method. The low TAR is not clean evidence of task difficulty either.

    I have more difficulty with squareBend/4 in Section 4.3 and Appendix E.1. The reference itself contains residualControl and records a latest time of 145, while both the prompt and controlDict request 500. In benchmarks/evaluation/profiles.py:70, the evaluator takes 145 from the reference as the expected endpoint, and I get E=pass at 145 without even adding a convergence marker. Therefore reaching 500 does not explain why one rollout passes and the other fails under this evaluator. I want to know which check the w/o Repair rollout actually failed. Separately, I get a numerical pass for a Riemann input with an empty final-time directory and very large residual, Courant, and continuity values, since missing reference diagnostics remove those magnitude checks.

    The README already states the coverage definition and the early-convergence behavior. What I have difficulty matching is the measurement described in the paper. I would separate the shared Riemann false rejection from profile, numerical, and log-selection errors, since the latter may accept different failures across methods. Their actual influence needs the frozen outputs.

  2. I get different execution and evolution rules when I follow the code.

    After a successful run, I reopen Write and change the inputs and mesh, but the summary still gives execution=passed before another run. This breaks the claimed invalidation. A wrong agent summary alone does not imply a wrong TAR, since the offline evaluator decides TAR. However, its adapter can also pick an old successful formal log and leave out a newer failed formal run. I want to know how many accepted tasks actually have this second problem.

    Appendix C.5.3 says the offline evaluator computes the binary E/N/Q used in Eq. (1). In the default reward code, I change only the agent's claim from none to validated_for_requested_qois and get reward 0.2 instead of 0, although execution and numerical assessments still fail and QoI quality is still zero. If the authors show that the historical runs used the available independent_gates option, I would drop this concern for those runs.

    For Eq. (2), I find zero regression penalty in the non-curriculum path at skills/evolution/engine/campaign.py:255. The curriculum path penalizes after aggregation by dataset, so one family's improvement can first cancel another family's regression. With two equal-weight families gaining +0.4 and -0.3 and lambda=0.5, I get -0.025 from the paper's formula, which should keep the parent, but both tested paths accept the candidate with +0.05. The README also states that these modes should not count as the manuscript's equal-family objective. I get opposite choices from the formula and the implementation.

    I should be careful about this last example. With a separate test set and correct scoring, the authors can still compare the initial and actual final libraries even if they used another selection rule. If the result remains 18 to 27, that improvement belongs to those two libraries. Whether it supports Eq. (2) needs the actual candidate comparisons.

The supplement omits historical checkpoints, fields, and trajectories, and the README does not certify the configuration used for the tables. I have not demonstrated that the rankings reverse. I guess rescoring saved outputs and checking the six candidate comparisons should be the first work here, before new CFD runs. The single-run Table 3 also lacks paired outcomes or uncertainty from repeated runs, while the ablation only covers 63 FoamBench tasks. I would give these results a smaller conclusion range, but the scoring questions mainly hold my score down.

I think there is enough work here to consider acceptance if the gains survive a correct measurement. For now I lean toward rejection, since I have reproduced scoring and selection errors but do not have the saved results to measure their influence. Rescoring and the actual evolution decisions could change my score.

Questions for authors
  1. I want to know which code revision, evaluator settings, reward mode, split IDs, ablation prompts, and initial/evolved libraries produced Tables 1, 2, and 3. For squareBend/4, which recorded criterion did w/o Repair fail, since the reference itself already stops at 145? How does that criterion explain the repair story in Section 4.3 and Appendix E.1?

  2. Can the authors rescore the frozen Table 1 outputs and show the E/N/Q/TAR changes for each method, together with the task IDs that change? I want to see profile sample counts and spatial intervals, and how many accepted tasks have a failed latest formal run or different inputs. I also ask for the corrected Table 2 and 3 results with paired initial/evolved outcomes. For the six candidate-parent comparisons, can the authors recompute Eqs. (1) and (2) by family and show whether any of the three promotion decisions changes? I would find these measurements more useful than an explanation of why the code behaves this way.

8 comments
AuthorsOpenReview ↗

Response to Reviewer pC89 (Part 1/5)

We thank the reviewer for carefully checking the evaluation code and experimental records. We corrected the profile coverage check, multibranch Riemann profile extraction, numerical diagnostics, and formal-run log selection, and re-evaluated the saved experimental outputs. We first summarize the corrected results and then respond to each question in turn.

(1) Corrected experimental results

As you suggested, we rescored the saved solver outputs with the corrected evaluator. Tables 1–3 list Execution Completion (EE), Numerical Soundness (NN), QoI Satisfaction (QQ), and Task Acceptance Rate (TAR) for each method.

Table 1. Main experimental results

TrackMethodExecution CompletionNumerical SoundnessQoI SatisfactionTAR
FoamBenchMetaOpenFOAM 2.036/126 (28.6%)29/126 (23.0%)10/126 (7.9%)9/126 (7.1%)
FoamBenchFoam-Agent74/126 (58.7%)46/126 (36.5%)5/126 (4.0%)5/126 (4.0%)
FoamBenchChatCFD39/126 (31.0%)37/126 (29.4%)9/126 (7.1%)9/126 (7.1%)
FoamBenchEasyFoam123/126 (97.6%)98/126 (77.8%)54/126(42.9%)52/126 (41.3%)
ExtFoamBenchMetaOpenFOAM 2.08/100 (8.0%)6/100 (6.0%)0/100 (0.0%)0/100 (0.0%)
ExtFoamBenchFoam-Agent25/100 (25.0%)14/100 (14.0%)0/100 (0.0%)0/100 (0.0%)
ExtFoamBenchChatCFD10/100 (10.0%)9/100 (9.0%)0/100 (0.0%)0/100 (0.0%)
ExtFoamBenchEasyFoam71/100 (71.0%)47/100 (47.0%)9/100 (9.0%)9/100 (9.0%)

After re-evaluation, EasyFoam still has the highest TAR on both benchmarks. On FoamBench, EasyFoam still passes 52/126 tasks, and the best-performing baseline passes 9/126. On ExtFoamBench, EasyFoam now also passes riemann2D/1, riemann2D/8, and riemann2D/10, increasing the number of accepted tasks from 6/100 to 9/100. All three baselines remain at 0/100. EasyFoam's lead on ExtFoamBench increases from 6 to 9 percentage points.

Table 2. Ablation results

Method VariantExecution CompletionNumerical SoundnessQoI SatisfactionTAR
EasyFoam63/63 (100.0%)55/63 (87.3%)37/63 (58.7%)35/63 (55.6%)
w/o Stage-Scoped Repair57/63 (90.5%)47/63 (74.6%)33/63 (52.4%)32/63 (50.8%)
w/o Repair51/63 (81.0%)45/63 (71.4%)32/63 (50.8%)31/63 (49.2%)
w/o Repair & Skills49/63 (77.8%)45/63 (71.4%)23/63 (36.5%)23/63 (36.5%)

After re-evaluation, the TAR values in Table 2 remain unchanged, and full EasyFoam achieves the highest results on all four metrics.

Table 3. Skill evolution results

Test TrackSkill LibraryExecution CompletionNumerical SoundnessQoI SatisfactionTAR
FoamBench (32)Initial30/32 (93.8%)25/32 (78.1%)20/32 (62.5%)16/32 (50.0%)
FoamBench (32)Evolved32/32 (100.0%)32/32 (100.0%)22/32 (68.8%)22/32 (68.8%)
ExtFoamBench (25)Initial15/25 (60.0%)9/25 (36.0%)2/25 (8.0%)2/25 (8.0%)
ExtFoamBench (25)Evolved22/25 (88.0%)17/25 (68.0%)5/25 (20.0%)5/25 (20.0%)
Overall (57)Initial45/57 (78.9%)34/57 (59.6%)22/57 (38.6%)18/57 (31.6%)
Overall (57)Evolved54/57 (94.7%)49/57 (86.0%)27/57 (47.4%)27/57 (47.4%)

After re-evaluation, the TAR values for both the initial and the evolved skill libraries remain unchanged. On the same 57 held-out tasks, the number of accepted tasks increases from 18 to 27, and overall TAR rises from 31.6% to 47.4%.

AuthorsOpenReview ↗

Response to Reviewer pC89 (Part 2/5)

(2) Point-by-point responses

Q1. Which experimental configurations were used for Tables 1–3? Can you provide the corresponding records?

A1. The original supplementary material did not make clear which experimental settings corresponded to each table. We have added EasyFoam_Task_IDs.xlsx to the supplementary material. Its three worksheets list all task IDs for Tables 1–3, respectively. The configurations for each table can be found at the following locations.

Table 1 (main results). All four methods were evaluated on the same 226 tasks. The evaluator settings for all three datasets are in benchmarks/evaluation/profiles/, which specify the reference quantities and the rules for the numerical checks. A task counts toward TAR only if it passes all three evaluation dimensions: Execution Completion, Numerical Soundness, and QoI Satisfaction.

Table 2 (ablation study). All four experiments used the same 63 tasks, the same model, and a 7,200-second per-task rollout limit. The four prompts are in benchmarks/campaign/easyfoam.py and correspond to stage-scoped repair, whole-case retry, retaining skills without repair, and removing both repair and skills, respectively.

Table 3 (skill evolution). We evaluated the initial and evolved skill libraries on the same 57 held-out tasks. The two frozen versions are in reproducibility/table3/initial/ and reproducibility/table3/evolved/, respectively. The configurations for the three evolution rounds are in configs/evolution.json, and the task IDs for the construction and validation subsets in each round are in benchmarks/evolution_round_tasks.json.

Q2. A single-point profile can pass the QoI check. What were the sample counts and spatial intervals of the saved profiles?

A2. We corrected the profile coverage check and rescored the 904 historical results in Table 1. This correction changed no task's QoI Satisfaction verdict and left TAR unchanged.

The previous evaluator computed the error only over the overlap between the candidate and reference profiles, so even a single sample could pass the check with zero error. After the correction, each candidate profile must have at least two distinct sample locations. At every sample location in the reference profile, the required physical quantity must be available directly from the candidate profile or by interpolation between adjacent samples, allowing for small rounding differences at the profile endpoints. Profiles with insufficient coverage fail the check.

We examined the 88 FoamBench results that passed the previous evaluator's QoI check and found no case that passed on the basis of a single sample or a short segment of a curve. These results contained 189 profiles. Each had the same number of sample locations as its reference profile (10–41) and covered nearly the entire spatial extent of that reference profile. The minimum spatial coverage was 99.9967% of the reference sampling interval. The corrected implementation is in compare_profiles() in benchmarks/evaluation/comparison.py.

Q3. Why did even the reference results fail on the eight multibranch Riemann tasks? How does the fix affect the scores and the assessment of task difficulty?

A3. We reproduced what you observed on riemann2D/2. The old evaluator discarded the branch directories and incorrectly merged the horizontal and vertical profiles from three meshes, reducing six profiles to two. As a result, even the complete reference output was rejected. After the fix, the evaluator selects profiles separately within each branch. When rechecked at the original tolerances, all 10 complete reference cases passed the QoI check. The corresponding test is in benchmarks/scripts/test_riemann_branch_evaluation.py.

We rescored the Riemann family with the corrected evaluator. The updated Table 1 shows that EasyFoam's TAR on ExtFoamBench increases from 6/100 to 9/100, while all three baselines remain at 0/100. After the correction, EasyFoam's lead over the three baselines therefore increases from 6 to 9 percentage points. At the same time, even EasyFoam, the best-performing method, passes all three evaluation dimensions (Execution Completion, Numerical Soundness, and QoI Satisfaction) on only nine tasks. This indicates that ExtFoamBench remains challenging for the four methods evaluated here.

AuthorsOpenReview ↗

Response to Reviewer pC89 (Part 3/5)

Q4. If squareBend/4 passes the endpoint check at step 145, which recorded criterion did w/o Repair fail? How does this explain the result after repair and running to step 500?

A4. On squareBend/4, the w/o Repair variant did pass the endpoint check at step 145. Both the task prompt and controlDict requested a run through step 500 with output every 100 steps, but the reference case retained residualControl, so the solver exited normally when it met the convergence criterion at step 145. The old evaluator used the actual endpoint of the reference run, so the endpoint check passed. The runner used in the final evaluation assessed completion against the 500 steps requested by the task and recorded this run as incomplete, so the corresponding formal_run_succeeded check failed. Although the solver generated a log, the old evaluator read logs only from successful runs. It therefore did not read this file, and solver_log_exists also failed.

During repair, full EasyFoam used the check results to remove the early stopping condition, reran the case through step 500, and produced the missing outputs.

We also selected pitzDaily/3 from the same ablation study to illustrate repair after a solver failure. Both independent runs reported “Continuity error cannot be removed by adjusting the outflow.” at the first time step (t=0.0001 st=0.0001\,\mathrm{s}). The logs showed that there was already incoming flux at the inlet, while the flux available for adjustment at the outlet was close to zero, so the solver could not complete the continuity correction. The w/o Repair variant reported the error and stopped at this step. Full EasyFoam adjusted the initialization settings based on the log, first confirmed on a copy of the case that the solver could continue, then returned to the Write stage to review the configuration and start the formal run.

Q5. Why could a case pass the N check with an empty final-time directory and large residuals or other diagnostic values? Did missing reference diagnostics affect the scores of actual tasks?

A5. We reproduced the counterexample you constructed. The previous evaluator failed to detect an empty final-time directory and skipped the corresponding magnitude checks when reference diagnostic data were missing. As a result, a candidate could be judged numerically sound (an NN pass) even with a large residual or Courant number. We corrected both checks and re-examined the affected historical records using the saved solver logs and final result fields. Correcting the numerical checks alone changed neither the number of NN passes nor the final task acceptance rate (TAR) for any of the four methods in Table 1.

The corrected evaluator requires the final-time directory to contain parseable result fields. For the residual, Courant number, and continuity error checks required by a task, a candidate value above the reference threshold is recorded as a failure, and missing candidate or reference data are recorded as “unknown.” In either case, the NN check cannot pass. Table Q5-1 gives the results of the historical audit for each check.

Table Q5-1. Numerical diagnostic audit and re-evaluation of historical experiments

Item examinedCorrection and auditResult
Riemann familyIncluded the separately saved reference logs and rechecked residuals, Courant numbers, and final result fields for eight records with successful execution.All eight retained their N pass. N and TAR were unchanged.
nozzleCDV/4Checked whether omission of the reference log led to incorrect acceptance.Execution failed for all four methods. All original N verdicts were false. There were no false passes.
shallowWater/SquareBump familyCorrected the recognition of Courant numbers in the logs and rechecked 14 records with successful execution.All 14 passed the added Courant check. The original seven N passes and seven N failures were unchanged.
Final-time directories and result fieldsChecked each historical record that reached the numerical checks and its result files.All contained parseable result fields. No task passed N on the basis of an empty directory.

The fix is implemented in benchmarks/evaluation/numerical.py. The corresponding regression test, test_numerical_missing_evidence.py, covers empty directories, missing required diagnostics, and abnormal residuals or Courant numbers.

Reviewer_m4JtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ExtFoamBench to evaluate complex OpenFOAM tasks along three dimensions: Execution Completion, Numerical Soundness, and QoI Satisfaction. It also proposes EasyFoam, which organizes domain knowledge into externalized skills. Stage-scoped repair and cross-task skill evolution improve simulation workflows.

Critical strengths
  1. A more complex and auditable CFD benchmark. ExtFoamBench covers ten physical families and five axes of workflow variation. It provides reference cases and task-specific QoI acceptance criteria. Reference-case reproducibility tests, physical negative controls, and tolerance sensitivity analyses provide empirical support for the evaluation's reliability. (Section 2; Appendices B.3–B.5)

  2. Clear gains from the system and skill updates. With the same backbone model, EasyFoam achieves a Task Acceptance Rate (TAR) of 41.3% on FoamBench. The strongest baseline achieves 16.7%. With model weights fixed and execution budgets matched, skill evolution further raises TAR on 57 held-out tasks from 31.6% to 47.4%. (Tables 1 and 3)

Critical weaknesses
  1. The methodological contribution of skill evolution is not sufficiently established. Trajectory reflection, experience-driven skill updates, and candidate selection are closely related to methods such as GEPA and ACE. The experiments mainly compare the initial and evolved skill libraries. They do not test the added value of numerical and QoI feedback or selection with family-level regression penalties over generic reflection-based updates. (Section 3.3; Table 3)

  2. The role of stage-scoped repair on complex tasks lacks direct validation. The ablations cover 63 FoamBench tasks. Stage-scoped repair increases the number of accepted tasks from 32 to 35 compared with whole-case retry. No corresponding comparison is provided on ExtFoamBench. It remains unclear whether this mechanism effectively addresses the main bottleneck: failure to satisfy physical QoI requirements. (Table 2; Sections 4.2–4.3)

Questions for authors
  1. Could you compare with a generic method that updates skills through trajectory reflection? Please start from the same initial skill library and use the same feedback data and evolution budget. Report TAR on the held-out tasks and performance regressions in each task family.

  2. Could you compare stage-scoped repair with whole-case retry on a representative subset of ExtFoamBench? Please report TAR and the QoI Satisfaction rate to assess its contribution to complex tasks.

2 comments
AuthorsOpenReview ↗

Response to Reviewer m4Jt

We sincerely thank the reviewer for the careful reading of our manuscript and for the constructive comments and suggestions. We appreciate the reviewer’s concerns regarding the methodological contribution of skill evolution and the effectiveness of stage-scoped repair on more complex CFD tasks. In response, we have conducted additional experiments to directly address these questions and further clarify the empirical contribution of the proposed design.

Q1. Comparison with Generic Trajectory-Reflection Methods

A1. To evaluate the effectiveness of EasyFoam’s skill evolution mechanism relative to generic trajectory-reflection approaches, we conducted additional comparisons with GEPA and ACE. All methods start from the same initial skill library, use the same evolution budget, and are evaluated on the same 57 held-out tasks.

Test TrackInitial Skill Library TARGEPA (Adapted to EasyFoam) TARACE (Adapted to EasyFoam) TARFull EasyFoam TAR
FoamBench (32)16/32 (50.0%)19/32 (59.4%)18/32 (56.3%)22/32 (68.8%)
ExtFoamBench (25)2/25 (8.0%)2/25 (8.0%)1/25 (4.0%)5/25 (20.0%)
Overall (57)18/57 (31.6%)21/57 (36.8%)19/57 (33.3%)27/57 (47.4%)

After evolution, EasyFoam improves the overall TAR from 31.6% with the initial skill library to 47.4%, outperforming GEPA at 36.8% and ACE at 33.3%, and achieving the largest overall improvement. On the FoamBench and ExtFoamBench held-out tasks, EasyFoam reaches TARs of 68.8% and 20.0%, respectively, both higher than those achieved by the trajectory-reflection-based evolution schemes of GEPA and ACE.

These results support the effectiveness of our skill evolution design for scientific computing. By incorporating numerical soundness and QoI satisfaction into the evolution evaluation and combining them with a performance-regression penalty, EasyFoam improves the overall acceptance performance of the evolved skill library across diverse CFD tasks.

Q2. Stage-Scoped Repair on ExtFoamBench

A2. We further conducted an additional ablation study on 50 complex ExtFoamBench tasks to compare Stage-Scoped Repair with Whole-Case Retry, while keeping the model, task settings, and execution budget identical.

MethodExecution Completion (E)Numerical Checks (N)QoI Satisfaction (Q)Task Acceptance Rate (TAR)
Stage-Scoped Repair35/50 (70%)23/50 (46%)6/50 (12%)6/50 (12%)
Whole-Case Retry28/50 (56%)17/50 (34%)3/50 (6%)3/50 (6%)

The results show that Stage-Scoped Repair outperforms Whole-Case Retry in Execution Completion, Numerical Checks, and QoI Satisfaction. In particular, both the QoI Satisfaction rate and the final TAR increase from 6% to 12%, achieving a doubling.

These results further validate the effectiveness of Stage-Scoped Repair in complex CFD workflows. It not only improves workflow execution capability, but also leads to better numerical reliability and stronger satisfaction of the final physical objectives.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Additional Experiments

Thank you again for your detailed and constructive comments. As the discussion period is drawing to a close, we would greatly appreciate it if you could review our responses and share your feedback before the deadline.

We have responded individually to all your weaknesses and questions. For W1 and Q1, we added controlled comparisons against generic trajectory-reflection methods (GEPA and ACE) under the same initial skill library and evolution budget, showing that EasyFoam achieves the highest overall TAR (47.4%) across the 57 held-out tasks. For W2 and Q2, we conducted an additional ablation on 50 ExtFoamBench tasks, demonstrating that Stage-Scoped Repair doubles both QoI Satisfaction and final TAR compared with Whole-Case Retry (from 6% to 12%).

We would appreciate knowing whether these additions address your concerns and would be happy to promptly clarify any remaining points. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.