- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether successful execution of scientific simulation code is a reliable indicator of scientific fidelity. It constructs a controlled OpenFOAM benchmark with hidden scientific contracts and evaluates generated simulation artifacts using separate execution and requirement-satisfaction checks. The paper further examines how these criteria affect model rankings and how runtime feedback influences the repair of failed artifacts. Experiments across six language models reveal discrepancies between execution success and scientific fidelity, including changes in model rankings and different responses to execution feedback.
Critical strengths
S1. The paper addresses an important and practical problem in scientific code evaluation. Distinguishing solver execution from adherence to scientific requirements is meaningful for assessing the reliability of scientific coding agents.
S2. The study combines real solver execution, explicit scientific requirement checks, repeated model comparisons, and a blinded human audit. These analyses provide concrete evidence of different failure modes and help make the evaluation process more transparent.
Critical weaknesses
W1. The validity of the scientific-fidelity assessment under deliberately incomplete specifications is not sufficiently established. The evaluation may conflate scientific errors with disagreement over information withheld from the models.
W2. The experimental evidence is insufficient to fully support the claimed separation of model capabilities. The evaluation is limited to a small set of canonical simulation problems, and the repair study lacks an appropriate control for additional generation attempts.
Questions for authors
Q1. In Appendix A (Tables 5 and 6), the model-facing specifications omit some scientific quantities while retaining the full hidden contracts for evaluation. Are the omitted quantities uniquely recoverable from the remaining constraints? For example, some physical parameters may be inferred if the Reynolds number and other necessary quantities remain specified. Otherwise, multiple scientifically valid configurations may satisfy the visible request. Could the authors clarify how such cases are distinguished from genuine scientific errors?
Q2. According to Tables 2 and 10, all 18 executable-but-scientifically-wrong (E1S0) artifacts in the six-model matched panel occur under specification-omission conditions, while none occur under FULL specifications. How do the main conclusions change when fully specified and incomplete tasks are analyzed separately? This distinction seems important for determining whether the observed errors arise from incorrect scientific implementations or missing information in the task specifications.
Q3. In Section 5, models receive up to two additional repair rounds with runtime error feedback. However, there is no corresponding control that allows additional repair attempts without such feedback. Could the authors clarify how much of the reported execution improvement can be attributed specifically to runtime information rather than additional generation attempts? A controlled comparison would strengthen the interpretation of the intervention results.
Q4. The main comparison uses 25 matched instances derived from only five canonical OpenFOAM problems, with just two cylinder-flow instances retained (Table 10). Furthermore, the reported confidence interval for the Spearman correlation is relatively wide. Could the authors explain the excluded instances and provide additional evidence that the observed capability differences are robust to problem selection and generalize beyond these canonical simulations?
5 comments
Response to Reviewer GQ5D (2/3)
Q1: Are withheld quantities recoverable?
In Appendix A (Tables 5 and 6), the model-facing specifications omit some scientific quantities while retaining the full hidden contracts for evaluation. Are the omitted quantities uniquely recoverable from the remaining constraints? [...]
The reviewer's intuition is right. Omissions should be distinguished by whether the withheld value can be recovered. No specification states the Reynolds number, so the revision classifies each omission by what the withheld quantity controls, using a single rule (Section 3; Appendix M, Table 14):
| Class | What is withheld | Cells | S = 1 |
|---|---|---|---|
| Complete | nothing (FULL) | 4 | 22/24 (92%) |
| Admissible: numerics | mesh and timestep (checked as bounds) | 5 | 24/30 (80%) |
| Admissible: stopping | end time (checked as a lower bound) | 3 | 10/18 (56%) |
| Regime-setting | a velocity, viscosity or length that sets the flow regime | 10 | 25/60 (42%) |
| Scale-only | conduction wall temperature, diffusivity or slab size (rescales the solution) | 3 | 1/18 (6%) |
Fidelity falls across the classes (Figure 2a; regime-setting vs scale-only, Fisher p = 0.004). Problem difficulty does not explain this drop. The conduction problem is translated correctly when it is fully described (6/6 under FULL).
How do we tell errors from admissible completions? In the admissible cells, the contract checks a bound, so any stable discretization or long enough run passes. In the scale-only cells, the withheld quantity only rescales the conduction solution. We report these cells separately. Of the 18 executable-but-wrong artifacts, 11 lie there. The ranking reversal holds without them (GLM-5.3: E = 10/22, S = 13/22; DeepSeek-V4-Pro: E = 3/22, S = 17/22; Section 7). For the rest, the source of each departure is visible in the filled-in values (Section 5, Figure 2b,c):
(a) Cavity, viscosity withheld: three of six models write = 0.01, the value used in OpenFOAM's own cavity tutorial. The case runs at Re = 10 instead of 100.
(b) Cavity, size withheld: three models write L = 1 m, which runs the cavity at Re = 1000.
(c) Conduction, left wall temperature withheld: all five models that write a value copy the 300 K initial temperature from the specification, instead of the intended 373 K.
The models complete the physical model by convention, and these completions run without any error.
Q2: Complete and incomplete specifications analyzed separately
According to Tables 2 and 10, all 18 executable-but-scientifically-wrong (E1S0) artifacts in the six-model matched panel occur under specification-omission conditions, while none occur under FULL specifications. How do the main conclusions change when fully specified and incomplete tasks are analyzed separately? [...]
When the two subsets are analyzed separately, each subset supports one half of the paper's claim (Section 4, Table 1):
| Subset | n | E | S | E1S0 | E0S1 | E0S0 |
|---|---|---|---|---|---|---|
| Complete (FULL) | 24 | 7/24 (29%) | 22/24 (92%) | 0 | 15 | 2 |
| Omission | 126 | 34/126 (27%) | 60/126 (48%) | 18 | 44 | 48 |
| All | 150 | 41/150 (27%) | 82/150 (55%) | 18 | 59 | 50 |
Complete specifications: the whole gap lies in the encoding. The models encode the intended model in 22 of 24 cases, but 15 of these do not run, and none runs with a different model (15 : 0, p = 6e-5). This is where execution-based evaluation under-rates fidelity most clearly.
Omission: the model-layer distance appears. Fidelity drops from 92% to 48% while execution barely changes (29% vs 27%). All 18 executable-but-wrong artifacts are here. The two layers are close to independent: 16 of the 34 executed artifacts and 44 of the 92 non-executed artifacts encode the intended model.
The two subsets together give the paper's conclusion. Which layer holds the distance depends on whether the specification is complete. The revised Section 4 is organized around this split.
Response to Reviewer GQ5D (3/3)
Q3: Control for additional repair attempts
In Section 5, models receive up to two additional repair rounds with runtime error feedback. However, there is no corresponding control that allows additional repair attempts without such feedback. [...]
Following this request, the revision includes the control (Appendix J, Table 12). The same 20 execution failures are repaired with the same instruction and the same two rounds. Only the feedback changes:
| Repair model | Feedback | Round-1 E | Final E | Final S |
|---|---|---|---|---|
| DeepSeek-V4.1-Flash | one-line notice that the case failed | 6/20 | 8/20 | 16/20 |
| DeepSeek-V4.1-Flash | solver error block from the failing stage | 18/20 | 18/20 | 16/20 |
| DeepSeek-V4-Pro | the case's own runtime error | 15/20 | 18/20 | 14/20 |
Additional attempts alone repair 8 of 20 cases. The error information repairs 10 more (10 vs 0 discordant pairs, exact McNemar p = 0.002), and fidelity is the same in both arms.
In the error-block arm, the block comes from a different case that failed at the same stage. It is still enough to repair 18 of 20 cases. The repair is therefore driven by encoding-layer information, namely which stage failed and how the solver reports it.
Q4: Excluded instances and robustness to problem selection
The main comparison uses 25 matched instances derived from only five canonical OpenFOAM problems, with just two cylinder-flow instances retained (Table 10). Furthermore, the reported confidence interval for the Spearman correlation is relatively wide. [...]
The matched panel requires a usable generation from all six models. Five cells do not meet this condition and are excluded: cylinder/FULL, cylinder/NO_GEOMETRY, cylinder/NO_PHYSICS, cylinder/NO_STOPPING and couette/NO_STOPPING (Appendix K). All models are compared on the same 25 instances.
On robustness to problem selection, in all five leave-one-problem-out panels, GLM-5.3 executes more often than DeepSeek-V4-Pro, and DeepSeek-V4-Pro is more faithful than GLM-5.3. Per instance, fidelity exceeds execution in 20 of 25 instances (sign test p = 0.0005), and the direction is the same for all six models (Appendix D). The wide interval on does not affect these conclusions, because is reported only as a description. The claim rests on the pairwise reversal.
Canonical problems give an unambiguous intended model, which is a precondition for the measurement. To test whether the results depend on their conventional values, we ran a preregistered off-convention replication (Section 5, Appendix N; the table is in the general response). It moves every value that a condition can withhold off convention and keeps each flow's Reynolds number. Under complete specifications, no executable artifact encodes a different model (7 : 0, p = 0.016). Under omission, fidelity in the regime-setting cells falls from 41.7% to 8.5%, and 21 of 57 wrong completions copy the original benchmark's values. All five models that write the withheld wall temperature copy the 309 K initial temperature from the specification, and only 1 of 123 omission artifacts asks for clarification. So these two results do not depend on the specific values of the canonical problems.
If the revision addresses your concerns, we would appreciate it if you would consider updating your score.
Response to Reviewer GQ5D (1/3)
We thank the reviewer for a careful and constructive review and for recognizing the importance of the problem and the rigor of our evaluation.
Below we respond to each point. The cleanest evidence comes from the complete specifications, where no information is withheld. There the two mismatch states split 15 to 0. Under omission, the classification of omissions and the analysis of filled-in values keep scientific errors apart from admissible completions.
As stated in the general response, the revision no longer uses the wording about model capabilities. A generated simulation case is an encoding of a physical model. We measure two things separately: how far that model is from the intended one (the model layer, S), and whether the solver accepts its encoding (the encoding layer, E). Section, figure and table numbers refer to the revised PDF.
W1: Validity of fidelity under incomplete specifications
In our design, the completeness of the specification is a variable that we control.
First, the core evidence for the separation comes from the complete specifications (FULL). Here nothing is withheld, so there is no assumption about missing information to disagree with. In this subset, 22 of 24 artifacts encode the intended physical model, and the solver rejects 15 of them. No artifact runs while encoding a different model. The two mismatch states split 15 to 0 (exact McNemar p = 6e-5; Section 4, Table 1). The separation between execution and fidelity therefore holds before any omission is considered.
Second, under omission we classify every cell by one rule, based on what the withheld quantity controls (Section 3, Appendix M). Third, we decode the value each model wrote. For each artifact with S = 0, we can therefore say which value departs from the intended model and where it comes from (Section 5, Figure 2b,c). Please see our response to Q1 for the classification and the examples.
W2: Evidence for the separation
The revision describes this as a separation between two measured layers. Four independent lines of evidence support it:
For the control on additional repair attempts, please see Q3. For the problem set and the off-convention replication, please see Q4.