acceptodds
Under review as a conference paper at ICLR 2027

Beyond Execution: Operationality and Scientific Fidelity Are Distinct Capabilities of Scientific Coding Agents

Abstract

Scientific coding agents are increasingly judged by whether the artifacts they produce execute. For simulation code, execution is only operational evidence: a case can run while instantiating the wrong physical parameter, geometry, boundary condition, or numerical intent. We ask whether *operationality* and *scientific fidelity* are the same capability. We encode five canonical OpenFOAM problems as hidden machine-readable scientific contracts and evaluate six contemporary language models under real solver execution and deterministic requirement checks. Both mismatch directions occur: executable artifacts violate the intended science, and scientifically faithful artifacts fail to run. Across the six evaluated models, the two rankings have a Spearman correlation of , and model ordering can reverse between the criteria; the reversal between the best executor and the most faithful model persists across three independent generations. We then intervene on the information the agent receives: given only ordinary runtime feedback, operationality rises by while scientific fidelity changes by , and 79% of doubly-failing artifacts whose execution is repaired remain scientifically wrong. Execution feedback repairs operationality but provides little signal for hidden scientific errors. Evaluation that collapses the two axes into a single execution signal can change which model is judged best.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
scientific coding agents, LLM agents, agent evaluation, scientific machine learning, code generation, scientific computing, execution-based evaluation, AI for science

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_GQ5DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether successful execution of scientific simulation code is a reliable indicator of scientific fidelity. It constructs a controlled OpenFOAM benchmark with hidden scientific contracts and evaluates generated simulation artifacts using separate execution and requirement-satisfaction checks. The paper further examines how these criteria affect model rankings and how runtime feedback influences the repair of failed artifacts. Experiments across six language models reveal discrepancies between execution success and scientific fidelity, including changes in model rankings and different responses to execution feedback.

Critical strengths

S1. The paper addresses an important and practical problem in scientific code evaluation. Distinguishing solver execution from adherence to scientific requirements is meaningful for assessing the reliability of scientific coding agents.

S2. The study combines real solver execution, explicit scientific requirement checks, repeated model comparisons, and a blinded human audit. These analyses provide concrete evidence of different failure modes and help make the evaluation process more transparent.

Critical weaknesses

W1. The validity of the scientific-fidelity assessment under deliberately incomplete specifications is not sufficiently established. The evaluation may conflate scientific errors with disagreement over information withheld from the models.

W2. The experimental evidence is insufficient to fully support the claimed separation of model capabilities. The evaluation is limited to a small set of canonical simulation problems, and the repair study lacks an appropriate control for additional generation attempts.

Questions for authors

Q1. In Appendix A (Tables 5 and 6), the model-facing specifications omit some scientific quantities while retaining the full hidden contracts for evaluation. Are the omitted quantities uniquely recoverable from the remaining constraints? For example, some physical parameters may be inferred if the Reynolds number and other necessary quantities remain specified. Otherwise, multiple scientifically valid configurations may satisfy the visible request. Could the authors clarify how such cases are distinguished from genuine scientific errors?

Q2. According to Tables 2 and 10, all 18 executable-but-scientifically-wrong (E1S0) artifacts in the six-model matched panel occur under specification-omission conditions, while none occur under FULL specifications. How do the main conclusions change when fully specified and incomplete tasks are analyzed separately? This distinction seems important for determining whether the observed errors arise from incorrect scientific implementations or missing information in the task specifications.

Q3. In Section 5, models receive up to two additional repair rounds with runtime error feedback. However, there is no corresponding control that allows additional repair attempts without such feedback. Could the authors clarify how much of the reported execution improvement can be attributed specifically to runtime information rather than additional generation attempts? A controlled comparison would strengthen the interpretation of the intervention results.

Q4. The main comparison uses 25 matched instances derived from only five canonical OpenFOAM problems, with just two cylinder-flow instances retained (Table 10). Furthermore, the reported confidence interval for the Spearman correlation is relatively wide. Could the authors explain the excluded instances and provide additional evidence that the observed capability differences are robust to problem selection and generalize beyond these canonical simulations?

5 comments
AuthorsOpenReview ↗

Response to Reviewer GQ5D (1/3)

We thank the reviewer for a careful and constructive review and for recognizing the importance of the problem and the rigor of our evaluation.

Below we respond to each point. The cleanest evidence comes from the complete specifications, where no information is withheld. There the two mismatch states split 15 to 0. Under omission, the classification of omissions and the analysis of filled-in values keep scientific errors apart from admissible completions.

As stated in the general response, the revision no longer uses the wording about model capabilities. A generated simulation case is an encoding of a physical model. We measure two things separately: how far that model is from the intended one (the model layer, S), and whether the solver accepts its encoding (the encoding layer, E). Section, figure and table numbers refer to the revised PDF.

W1: Validity of fidelity under incomplete specifications

The validity of the scientific-fidelity assessment under deliberately incomplete specifications is not sufficiently established. The evaluation may conflate scientific errors with disagreement over information withheld from the models.

In our design, the completeness of the specification is a variable that we control.

First, the core evidence for the separation comes from the complete specifications (FULL). Here nothing is withheld, so there is no assumption about missing information to disagree with. In this subset, 22 of 24 artifacts encode the intended physical model, and the solver rejects 15 of them. No artifact runs while encoding a different model. The two mismatch states split 15 to 0 (exact McNemar p = 6e-5; Section 4, Table 1). The separation between execution and fidelity therefore holds before any omission is considered.

Second, under omission we classify every cell by one rule, based on what the withheld quantity controls (Section 3, Appendix M). Third, we decode the value each model wrote. For each artifact with S = 0, we can therefore say which value departs from the intended model and where it comes from (Section 5, Figure 2b,c). Please see our response to Q1 for the classification and the examples.

W2: Evidence for the separation

The experimental evidence is insufficient to fully support the claimed separation of model capabilities. The evaluation is limited to a small set of canonical simulation problems, and the repair study lacks an appropriate control for additional generation attempts.

The revision describes this as a separation between two measured layers. Four independent lines of evidence support it:

EvidenceResultWhere
Complete specifications onlyE0S1 vs E1S0 = 15 : 0, exact McNemar p = 6e-5Section 4, Table 1
Repair, same 46 targetsE restored in 32 of 46; S changes 5 : 3 (p = 0.73), none a correctionSection 6, Figure 3
Same-checkpoint controlerror block 18/20 vs one-line notice 8/20 (p = 0.002); same S in both armsAppendix J, Table 12
Cluster-level robustnessS > E in 20 of 25 instances, S < E in 3 (p = 0.0005); same direction for all six modelsAppendix D, Table 8

For the control on additional repair attempts, please see Q3. For the problem set and the off-convention replication, please see Q4.

AuthorsOpenReview ↗

Response to Reviewer GQ5D (2/3)

Q1: Are withheld quantities recoverable?

In Appendix A (Tables 5 and 6), the model-facing specifications omit some scientific quantities while retaining the full hidden contracts for evaluation. Are the omitted quantities uniquely recoverable from the remaining constraints? [...]

The reviewer's intuition is right. Omissions should be distinguished by whether the withheld value can be recovered. No specification states the Reynolds number, so the revision classifies each omission by what the withheld quantity controls, using a single rule (Section 3; Appendix M, Table 14):

ClassWhat is withheldCellsS = 1
Completenothing (FULL)422/24 (92%)
Admissible: numericsmesh and timestep (checked as bounds)524/30 (80%)
Admissible: stoppingend time (checked as a lower bound)310/18 (56%)
Regime-settinga velocity, viscosity or length that sets the flow regime1025/60 (42%)
Scale-onlyconduction wall temperature, diffusivity or slab size (rescales the solution)31/18 (6%)

Fidelity falls across the classes (Figure 2a; regime-setting vs scale-only, Fisher p = 0.004). Problem difficulty does not explain this drop. The conduction problem is translated correctly when it is fully described (6/6 under FULL).

How do we tell errors from admissible completions? In the admissible cells, the contract checks a bound, so any stable discretization or long enough run passes. In the scale-only cells, the withheld quantity only rescales the conduction solution. We report these cells separately. Of the 18 executable-but-wrong artifacts, 11 lie there. The ranking reversal holds without them (GLM-5.3: E = 10/22, S = 13/22; DeepSeek-V4-Pro: E = 3/22, S = 17/22; Section 7). For the rest, the source of each departure is visible in the filled-in values (Section 5, Figure 2b,c):

(a) Cavity, viscosity withheld: three of six models write ν\nu = 0.01, the value used in OpenFOAM's own cavity tutorial. The case runs at Re = 10 instead of 100.

(b) Cavity, size withheld: three models write L = 1 m, which runs the cavity at Re = 1000.

(c) Conduction, left wall temperature withheld: all five models that write a value copy the 300 K initial temperature from the specification, instead of the intended 373 K.

The models complete the physical model by convention, and these completions run without any error.

Q2: Complete and incomplete specifications analyzed separately

According to Tables 2 and 10, all 18 executable-but-scientifically-wrong (E1S0) artifacts in the six-model matched panel occur under specification-omission conditions, while none occur under FULL specifications. How do the main conclusions change when fully specified and incomplete tasks are analyzed separately? [...]

When the two subsets are analyzed separately, each subset supports one half of the paper's claim (Section 4, Table 1):

SubsetnESE1S0E0S1E0S0
Complete (FULL)247/24 (29%)22/24 (92%)0152
Omission12634/126 (27%)60/126 (48%)184448
All15041/150 (27%)82/150 (55%)185950

Complete specifications: the whole gap lies in the encoding. The models encode the intended model in 22 of 24 cases, but 15 of these do not run, and none runs with a different model (15 : 0, p = 6e-5). This is where execution-based evaluation under-rates fidelity most clearly.

Omission: the model-layer distance appears. Fidelity drops from 92% to 48% while execution barely changes (29% vs 27%). All 18 executable-but-wrong artifacts are here. The two layers are close to independent: 16 of the 34 executed artifacts and 44 of the 92 non-executed artifacts encode the intended model.

The two subsets together give the paper's conclusion. Which layer holds the distance depends on whether the specification is complete. The revised Section 4 is organized around this split.

AuthorsOpenReview ↗

Response to Reviewer GQ5D (3/3)

Q3: Control for additional repair attempts

In Section 5, models receive up to two additional repair rounds with runtime error feedback. However, there is no corresponding control that allows additional repair attempts without such feedback. [...]

Following this request, the revision includes the control (Appendix J, Table 12). The same 20 execution failures are repaired with the same instruction and the same two rounds. Only the feedback changes:

Repair modelFeedbackRound-1 EFinal EFinal S
DeepSeek-V4.1-Flashone-line notice that the case failed6/208/2016/20
DeepSeek-V4.1-Flashsolver error block from the failing stage18/2018/2016/20
DeepSeek-V4-Prothe case's own runtime error15/2018/2014/20

Additional attempts alone repair 8 of 20 cases. The error information repairs 10 more (10 vs 0 discordant pairs, exact McNemar p = 0.002), and fidelity is the same in both arms.

In the error-block arm, the block comes from a different case that failed at the same stage. It is still enough to repair 18 of 20 cases. The repair is therefore driven by encoding-layer information, namely which stage failed and how the solver reports it.

Q4: Excluded instances and robustness to problem selection

The main comparison uses 25 matched instances derived from only five canonical OpenFOAM problems, with just two cylinder-flow instances retained (Table 10). Furthermore, the reported confidence interval for the Spearman correlation is relatively wide. [...]

The matched panel requires a usable generation from all six models. Five cells do not meet this condition and are excluded: cylinder/FULL, cylinder/NO_GEOMETRY, cylinder/NO_PHYSICS, cylinder/NO_STOPPING and couette/NO_STOPPING (Appendix K). All models are compared on the same 25 instances.

On robustness to problem selection, in all five leave-one-problem-out panels, GLM-5.3 executes more often than DeepSeek-V4-Pro, and DeepSeek-V4-Pro is more faithful than GLM-5.3. Per instance, fidelity exceeds execution in 20 of 25 instances (sign test p = 0.0005), and the direction is the same for all six models (Appendix D). The wide interval on ρ\rho does not affect these conclusions, because ρ\rho is reported only as a description. The claim rests on the pairwise reversal.

Canonical problems give an unambiguous intended model, which is a precondition for the measurement. To test whether the results depend on their conventional values, we ran a preregistered off-convention replication (Section 5, Appendix N; the table is in the general response). It moves every value that a condition can withhold off convention and keeps each flow's Reynolds number. Under complete specifications, no executable artifact encodes a different model (7 : 0, p = 0.016). Under omission, fidelity in the regime-setting cells falls from 41.7% to 8.5%, and 21 of 57 wrong completions copy the original benchmark's values. All five models that write the withheld wall temperature copy the 309 K initial temperature from the specification, and only 1 of 123 omission artifacts asks for clarification. So these two results do not depend on the specific values of the canonical problems.

If the revision addresses your concerns, we would appreciate it if you would consider updating your score.

Reviewer_tYU6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper asks whether judging scientific coding agents by successful execution leads to the same conclusions as checking whether they implement the intended scientific setup. It evaluates generated OpenFOAM cases using two criteria. Operationality (E) records whether the case completes the required solver run. Scientific fidelity (S) checks its parameters, geometry, boundary conditions, and other settings against a fixed scientific contract.

The benchmark contains five canonical problems, each with a complete specification and five versions that omit selected information. The authors compare six models on a shared panel of 25 task instances. The two criteria produce different rankings: GLM-5.3 has the highest execution rate, while DeepSeek-V4-Pro has the highest contract-satisfaction rate. This pairwise reversal persists across three independent generations.

The authors then give runtime error messages to models whose cases failed to execute and allow up to two repair rounds. Of 46 cases, 32 become executable, while the number satisfying the scientific contract rises from 25 to 27. A blinded audit of 50 cases by two human reviewers identifies an error in how the automatic checker handles equivalent boundary-condition representations. The authors correct the checker and report that the main results remain. Together, these experiments motivate reporting execution and contract satisfaction separately.

Critical strengths

None

Critical weaknesses
  1. The experiments do not clearly support the broad capability claim. All 18 executable contract violations in the matched panel occur when information is omitted, so S mixes implementing requirements with recovering hidden targets. The repair results are also shaped by the feedback: runtime logs identify software errors but do not reveal hidden scientific values. These results establish differences between evaluation outcomes more directly than distinct model capabilities.

  2. The additional insight is limited. ScienceAgentBench and SciCode already evaluate correctness beyond execution, and PhysCodeBench reports changes in system ordering across metrics. This paper adds contract checking even when execution fails, but studies only five canonical OpenFOAM problems and their omission variants. The current analysis does not establish enough new understanding or benchmark value to support acceptance.

  3. The figures need more careful preparation. For example, Figure 1(a) shows four gray points without descriptive labels for the states they represent. The reader must rely on the caption to interpret a very simple diagram.

Questions for authors
  1. What evidence supports the stronger capability interpretation beyond showing that this runtime-feedback workflow changes E much more than S? In particular, how does the experiment justify the Section 8 interpretation of the result after repair, given that runtime logs do not reveal the hidden target?

  2. Which omitted targets can be recovered from the remaining specification, and which require an assumption? Please explain how that distinction affects the interpretation of contract failures and model rankings. A few concrete task examples would make the meaning of S much clearer.

  3. What is the main advance over earlier evaluations that separately check execution and scientific results, including the ranking changes in PhysCodeBench Section 6.2?

5 comments
AuthorsOpenReview ↗

Response to Reviewer tYU6 (1/4)

We thank the reviewer for the careful and accurate summary of our paper. Following the reviewers' suggestion, we have restated the contribution as a claim about what is measured, and removed the wording about model capabilities. The new title names the two layers of distance that the paper measures.

Section 2 of the revision states the object of study explicitly. A generated simulation case is an encoding of a physical model. We measure two things separately: how far that model is from the one the user intended (the model layer, S), and whether the solver accepts its encoding (the encoding layer, E). Section, figure and table numbers refer to the revised PDF.

W1: What the experiments support

The experiments do not clearly support the broad capability claim. All 18 executable contract violations in the matched panel occur when information is omitted, so S mixes implementing requirements with recovering hidden targets. [...]

These three observations led us to run a new experiment. We report it first, because it tests the main concern, which is whether S under omission reflects the models or the withheld targets.

The experiment is a preregistered off-convention replication (Section 5, Appendix N). It moves every value that a condition can withhold off convention, and keeps each flow's Reynolds number. The same six models give 147 matched artifacts:

MeasureOriginal benchmarkOff-convention
Complete specifications: E0S1 : E1S015 : 0 (p = 6e-5)7 : 0 (p = 0.016)
Regime-setting omissions: S = 125/60 (41.7%)5/59 (8.5%)
Withheld values restored44/74 (59%)13/70 (19%), Fisher p = 5e-7
Omission artifacts that run with a different model (E1S0)1828
Wrong completions that copy the original benchmark's valuen/a21/57

With a complete specification, nothing is withheld, and the result is the same as before. No executable artifact encodes a different model (7 : 0). Under omission, the conventions no longer match the intent, and fidelity in the regime-setting cells falls from 41.7% to 8.5%. Of the 57 wrong completions, 21 copy the original benchmark's values. So the models implement the requirements when they are stated, and they fill a gap with conventional values when they are not.

We now take the three observations in turn. S asks whether the physical model that an artifact encodes is the one the user intended (Section 2), and the revision no longer presents it as a capability score.

(i) All 18 executable-but-wrong artifacts occur under omission. This shows that with a complete specification, the whole gap lies in the encoding. Of 24 artifacts, 22 encode the intended model, and the solver rejects 15 of them. None runs with a different model (15 : 0, exact McNemar p = 6e-5; Section 4, Table 1). Model-layer distance appears mainly when the model has to complete the specification.

(ii) The revision now separates implementing requirements from recovering withheld targets. We assign each omission cell to one class by a single rule. Fidelity falls across the classes (Section 3, Figure 2a). It is 92% for complete specifications, and 80% and 56% for admissible omissions. It is 42% when the withheld quantity sets the flow regime, and 6% when it only rescales the conduction solution.

(iii) The feedback carries information about the encoding only. The two-layer formulation predicts that it moves execution and not fidelity, and the revision tests this. On the same 46 targets, execution is restored in 32. Fidelity rises on 5 and falls on 3, with no consistent direction (p = 0.73), and none of these changes is a correction (Section 6, Figure 3). In a same-checkpoint control, an error block from the failing stage repairs 18 of 20 cases, against 8 of 20 for a one-line notice (p = 0.002). A stage-typical block taken from a different case is enough.

We have followed the reviewer's suggestion, and the revised claim is about evaluation outcomes. The revision also shows where these outcomes come from.

AuthorsOpenReview ↗

Response to Reviewer tYU6 (2/4)

W2: Advance over existing benchmarks

The additional insight is limited. ScienceAgentBench and SciCode already evaluate correctness beyond execution, and PhysCodeBench reports changes in system ordering across metrics. This paper adds contract checking even when execution fails, but studies only five canonical OpenFOAM problems and their omission variants. [...]

The revised Section 8 cites all three benchmarks. ScienceAgentBench and SciCode evaluate correctness beyond execution, as the reviewer says. They do so after execution, so a program must run before its output can be scored.

One state stays hidden in this design. It is the intended physical model inside an encoding that the solver rejects. In our data, this is the most common case under complete specifications (15 of 24 under FULL; 59 of 150 overall). A benchmark that executes first and scores second counts all of these as failures. It cannot tell them apart from getting the physics wrong.

Measuring the encoded physical model directly, without running it, complements these benchmarks. It makes this state visible, and it allows the paper to report the following findings:

FindingEvidence
Which layer holds the distance depends on whether the specification is completeComplete specifications: 15 : 0, p = 6e-5. Omission: fidelity drops from 92% to 48% while execution barely changes (Section 4)
Missing parts of the model are filled in by convention23 of the 30 missed values are round or copied values; 3 of 6 models use OpenFOAM's cavity-tutorial viscosity ν\nu = 0.01 (Section 5)
Runtime feedback acts on the encoding layer onlyE restored in 32 of 46 targets; S changes 5 : 3 (Section 6)

On the five problems, we chose canonical problems on purpose. They give an unambiguous intended model and a contract that can be checked requirement by requirement. This is a precondition for measuring model-layer distance. The revision also adds cluster-level tests. Fidelity is higher than execution in 20 of 25 instances and lower in 3 (p = 0.0005), and the direction is the same for all six models (Appendix D). The off-convention replication in W1 shows that both findings also hold on a new set of values.

An execution-first benchmark cannot see either result. The artifacts in question either fail to run with the intended model, or run without error with a different one.

W3: Figures

The figures need more careful preparation. For example, Figure 1(a) shows four gray points without descriptive labels for the states they represent. The reader must rely on the caption to interpret a very simple diagram.

Following this comment, we have redrawn Figure 1. Panel (a) follows one description through the language model to two generated cases and their solver runs. Each case is labeled with its verdict on both layers and its state (E1S0, E0S1), and the intended model is marked on the model layer. Panel (b) gives the counts of the four artifact states on labeled axes, split by complete and incomplete specifications. The figure can now be read without the caption. All data figures are regenerated with labeled axes (Figures 2 to 4).

AuthorsOpenReview ↗

Response to Reviewer tYU6 (3/4)

Q1: Interpretation of the result after repair

What evidence supports the stronger capability interpretation beyond showing that this runtime-feedback workflow changes E much more than S? In particular, how does the experiment justify the Section 8 interpretation of the result after repair, given that runtime logs do not reveal the hidden target?

The revision no longer gives the post-repair result a capability interpretation.

Runtime logs do not reveal the hidden target, as the reviewer notes. The experiment is built on this property. Execution feedback carries information about the encoding only, so it cannot direct a correction of the physical model.

The experiment measures whether repair changes the physical model, because the model rewrites the whole case. If repairing an encoding tended to restore the intended model, fidelity would rise together with execution. It does not. Execution is restored in 32 of 46 targets, while fidelity changes 5 : 3 (p = 0.73), and none of the changes is a correction (Figure 3).

The revised Section 9 states the practical consequence. Runtime feedback informs only the encoding, so reducing the model-layer distance needs a model-layer signal. Execution feedback is the default repair signal of coding agents, and it cannot fix the physical model.

Q2: Recoverable and assumed targets

Which omitted targets can be recovered from the remaining specification, and which require an assumption? Please explain how that distinction affects the interpretation of contract failures and model rankings. A few concrete task examples would make the meaning of S much clearer.

Omissions differ in whether the withheld value can be recovered or must be assumed, and this matters for reading S. No specification states the Reynolds number, so the revision classifies each omission by what the withheld quantity controls (Section 3; Appendix M, Table 14). A withheld velocity, viscosity or length of a flow problem sets the flow regime. A withheld wall temperature, diffusivity or slab size of the conduction problem only rescales the solution. A withheld mesh, timestep or end time is checked against a bound. The full table is in our response to Reviewer GQ5D (Q1).

Fidelity is 42% when the withheld quantity sets the flow regime and 6% when it only rescales the conduction solution (Fisher p = 0.004). Problem difficulty does not explain this gap. The conduction problem is solved correctly when it is fully described (6/6 under FULL).

On contract failures, in the scale-only cells, the withheld quantity only rescales the conduction solution. We report these cells separately, and 11 of the 18 executable-but-wrong artifacts lie there.

On rankings, each criterion is reported on the full matched panel, and the key pair is replicated over three generations (Section 7). The reversal also holds when the scale-only cells are excluded (GLM-5.3: E = 10/22, S = 13/22; DeepSeek-V4-Pro: E = 3/22, S = 17/22).

As the reviewer suggested, here are three concrete examples (Section 5, Figure 2b,c):

(a) Cavity, viscosity withheld: three of six models write ν\nu = 0.01, the value from OpenFOAM's own cavity tutorial. The case runs without error, but at Re = 10 instead of 100.

(b) Cavity, size withheld: three models write L = 1 m, and the case runs at Re = 1000.

(c) Conduction, left wall temperature withheld: all five models that write a value copy the 300 K initial temperature from the specification, instead of the intended 373 K.

In all three, the model completes the physical model by convention, and the completed case runs without any error.

Comments on the paper

7 comments
AuthorsOpenReview ↗

General Response to All Reviewers (1/2)

We thank all three reviewers for their careful reading. Their comments on the scope of our claims and on the omission design led to most of the changes in this revision. We are glad that the reviewers recognize the core problem of the paper. A simulation can run successfully while it simulates the wrong physics (Reviewers Psr9 and GQ5D). The intervention isolates the two dimensions cleanly (Reviewer Psr9). The evaluation combines real solver execution, explicit requirement checks, repeated comparisons and a blinded human audit (Reviewer GQ5D).

As the reviewers pointed out, the submission did not state its object of measurement explicitly. Section 2 of the revision now does, and the revision adds the analyses that the reviewers asked for. This response summarizes the revision. Each reviewer also receives a point-by-point reply. Section, figure, table and appendix numbers refer to the revised PDF.

1. What the paper measures

The revised title is "Beyond Execution: Two Layers of Distance in LLM-Generated Physical Simulations". It names what the paper measures.

A generated OpenFOAM case is an encoding of a physical model: governing equations, parameters, geometry, boundary conditions, numerics and time horizon (Section 2). Generating it takes two steps. The model chooses a physical model, and then encodes it for the solver. A solver run tests only the second step. We therefore measure the two layers separately. At the model layer, a deterministic decoder reads the physical model from the case and checks it against a hidden contract of the intended model (S). At the encoding layer, the solver tests whether the case runs (E).

Following the reviewers' suggestion, we have restated the contribution as a claim about what is measured, and removed the wording about model capabilities. The original experiments and their results are kept. All claims are about how far generated simulations are from the intended model, and where that distance lies.

2. Main results

The revised paper asks three questions and draws one consequence for evaluation. The table gives the answer to each, and where to find it in the revised PDF.

QuestionResultWhere
Where does the distance lie?Complete specifications: 22/24 encode the intended model, but only 7 run. The mismatch states split 15 : 0 (exact McNemar p = 6e-5). Under omission, fidelity falls from 92% to 48% while execution stays near 28%.Section 4, Table 1
How do models complete a specification?By convention. Three of six models use the cavity viscosity of OpenFOAM's tutorial, which runs the cavity at Re = 10 instead of 100. All five models that write the withheld 373 K wall temperature copy the 300 K initial temperature.Section 5, Figure 2
Which layer does runtime feedback act on?The encoding only. Execution is restored in 32 of 46 targets. Fidelity changes in 8, 5 up and 3 down (p = 0.73). 11 of the 14 doubly failing targets whose execution is repaired now run a different model.Section 6, Figure 3
Consequence for rankingGLM-5.3 ranks first by E and fourth by S. DeepSeek-V4-Pro ranks fifth by E and first by S. On the same tasks, GLM-5.3 executes 7 more (p = 0.016), while DeepSeek-V4-Pro is more faithful on 5 tasks against 1.Section 7, Figure 4
AuthorsOpenReview ↗

General Response to All Reviewers (2/2)

3. Shared concerns

Generality beyond five canonical problems (all reviewers). We chose canonical problems on purpose. Distance to the intended model can only be measured when the intended model is unambiguous and every requirement can be checked. Fidelity exceeds execution in 20 of 25 instances and falls below it in 3 (sign test p = 0.0005), and the direction is the same for all six models (Appendix D). We also ran a preregistered off-convention replication (Section 5, Appendix N). Every value that a condition can withhold is moved off convention, while each flow keeps its Reynolds number. On 147 matched artifacts, the two-layer pattern holds:

MeasureOriginalOff-convention
Complete: E0S1 : E1S015 : 0 (p = 6e-5)7 : 0 (p = 0.016)
Regime-setting omissions: S = 125/60 (41.7%)5/59 (8.5%)
Withheld values restored44/74 (59%)13/70 (19%), p = 5e-7
Wrong completions that copy the original benchmark's valuen/a21/57

Under complete specifications, no executable artifact encodes a different model, as before. Under omission, the models still complete by convention, and once the conventions no longer match the intent, fidelity collapses.

Incomplete specifications (Reviewers GQ5D and tYU6). Specification completeness is a variable we control. The core separation comes from the complete specifications, where nothing is withheld (15 : 0). Under omission, every cell is assigned to one class by a single rule on what the withheld quantity controls (Section 3, Appendix M). The completion analysis then shows which value each model chose, and where it came from (Section 5).

Is the repair result built into the design? (Reviewers Psr9 and tYU6) The feedback carries no information about the intended model, so it cannot direct a correction. During repair, however, the model rewrites the whole case, so whether S moves with E is an empirical question. The data show that execution is restored in 32 of 46 targets, fidelity does not move systematically, and none of the eight fidelity changes is a correction. A same-checkpoint control, requested by Reviewer GQ5D, shows what drives the repair. A solver error block restores 18 of 20 cases, against 8 of 20 for a one-line notice (p = 0.002), with the same fidelity in both arms. A stage-typical error block is enough, even when it comes from a different case (Appendix J).

Statistics on six models (Reviewer Psr9). The rank correlation is reported only as a description. The ranking claim rests on the pairwise comparison above and on three independent generations (P(full reversal) = 0.87). The reversal also holds in every leave-one-problem-out panel (Figure 4c, Appendix D).

Relation to prior work (Reviewer tYU6). ScienceAgentBench, SciCode and PhysCodeBench evaluate correctness beyond execution, and they do so after execution. Our measurement complements them, because it can see a correct model behind an encoding that the solver rejects. This is the most common state under complete specifications (15 of 24). Decoding the physical model without running it is what makes these three results measurable (Section 8).

4. Summary of changes

The table lists the main changes in the revised PDF and where to find them. Most of them follow directly from the reviewers' comments.

ChangeWhere
New title and two-layer formulation of the measured objectTitle, Sections 1-2
Split by specification completenessSection 4, Table 1
Classification of omissions by what the withheld quantity controlsSection 3, Appendix M
Analysis of the values models write for withheld quantitiesSection 5, Figure 2
Both layers measured on the same repair targetsSection 6, Figure 3
Same-checkpoint feedback controlAppendix J, Table 12
Cluster-level robustness testsAppendix D, Table 8
Preregistered off-convention replicationSection 5, Appendix N
Figure 1 redrawn; all data figures regenerated with labeled axesFigures 1-4

Please let us know if you have any further questions.

AuthorsOpenReview ↗

Response to Reviewer Psr9 (1/3)

We thank the reviewer for a careful and constructive review and for recognizing the two core ideas of the paper. A program can run and still implement the wrong science, and the intervention separates the two dimensions.

Following the reviewers' suggestion, we have restated the contribution as a claim about what is measured, and removed the wording about model capabilities (see the general response). A generated simulation case is an encoding of a physical model. We measure two things separately: how far that model is from the intended one (the model layer, S), and whether the solver accepts its encoding (the encoding layer, E). Section, figure and table numbers refer to the revised PDF.

W1: Scale and statistical robustness

Severe Lack of Scale and Statistical Robustness: The paper makes sweeping claims about the capabilities of "Scientific Coding Agents" based on an extraordinarily narrow dataset: exactly 5 canonical OpenFOAM problems and a matched panel of only 25 instances. [...]

The rank correlation ρ\rho is reported only as a description (Section 7, Appendix D). The claim is a pairwise reversal, and the revision states it in the main text.

For the same pair of models, execution and fidelity give opposite verdicts. GLM-5.3 ranks first by execution (E = 0.48) and fourth by fidelity (S = 0.52). DeepSeek-V4-Pro ranks fifth by execution (E = 0.20) and first by fidelity (S = 0.68). On the same tasks, the execution comparison is one-sided: GLM-5.3 executes every task that DeepSeek-V4-Pro executes, and 7 more (exact McNemar p = 0.016). In fidelity the direction reverses, 5 tasks against 1. The reversal holds across three independent generations. A task-level bootstrap gives P(E_GLM > E_DS) = 0.89, P(S_DS > S_GLM) = 0.98, and P(full reversal) = 0.87 (Section 7, Appendix D). This claim rests on two models compared on the same tasks. It still holds when any one problem is removed. In all five leave-one-problem-out panels, GLM-5.3 executes more often than DeepSeek-V4-Pro, and DeepSeek-V4-Pro is more faithful than GLM-5.3 (Appendix D).

The gap between the two layers is stable at every level (Appendix D, Table 8). Per instance, fidelity is higher than execution in 20 of the 25 matched instances and lower in only 3 (exact sign test p = 0.0005). Per model, artifacts that encode the intended model but fail to run (E0S1) outnumber artifacts that run a different model (E1S0) for all six models.

At the artifact level, the two layers are nearly independent (phi = 0.018). Among executed artifacts, 23 of 41 encode the intended model. Among non-executed artifacts, 59 of 109 do. So the gap does not come from a few instances or from one model.

We chose five canonical problems on purpose. The distance to the intended physical model can only be measured well when the intended model is unambiguous and every requirement can be checked. We also ran a preregistered off-convention replication (Section 5, Appendix N; the table is in the general response). It moves every value that a condition can withhold off convention and keeps each flow's Reynolds number. Under complete specifications, no executable artifact encodes a different model (7 : 0, p = 0.016). Under omission, fidelity in the regime-setting cells falls from 41.7% to 8.5%, and 21 of 57 wrong completions copy the original benchmark's values. This is consistent with Section 5 and extends it to a new set of values.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.