acceptodds
Under review as a conference paper at ICLR 2027

MDGym: Benchmarking AI Agents on Molecular Simulations

Abstract

The promise of AI-driven scientific discovery hinges on whether AI agents can autonomously design and execute the computational workflows that underpin modern science. Molecular dynamics (MD) simulation provides a demanding test of this capability: it requires translating a physical problem into syntactically and semantically correct input scripts, choosing force fields, ensembles, and initial and boundary conditions, diagnosing engine errors and unstable trajectories, and interpreting outputs against known physical behavior. We introduce , a benchmark of 200 expert-curated MD simulation problems in and , two widely used MD engines, organized into three difficulty levels and comprising 366 target quantities scored against reference values from expert simulations. We evaluate three general-purpose command-line agent frameworks, Claude Code, Codex, and OpenHands, each with four LLMs. The best configuration solves 68% of easy problems, 30% of medium problems, and 21% of hard problems, and the choice of model changes performance far more than the choice of framework. Trajectory and engine-error analyses show that most engine errors arise while agents assemble the simulation input, that weaker models fail early through syntax errors and premature termination, and that stronger models complete more simulations but report incorrect values from error-free runs. Current agents therefore lack three capabilities required for autonomous simulation: coherent specification of simulation inputs, domain-aware recovery from engine errors, and physical and methodological validation of results.

Primary area:
datasets and benchmarks
Keywords:
AI Agents, Benchmark, Molecular Dynamics Simulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GF1aOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a benchmark for end-to-end molecular dynamics simulations, comprising 200 simulation tasks, 366 target quantities, and 160 knowledge questions. It covers LAMMPS and GROMACS across three difficulty levels. The tasks were developed and reviewed by domain experts, with a subset independently reproduced by additional researchers. The authors compare different LLMs and agent frameworks and analyze failure stages and error types. The benchmark has practical value, but several concerns remain:

  1. Insufficient detail on reference values and scoring tolerances.
    Although the authors mention independent reference runs and expert reproduction of a subset of tasks, they do not clearly report how many times each reference simulation was repeated, how the final reference values were calculated, or how the scoring tolerances were determined. Without this information, it is difficult to distinguish normal simulation variability from genuine task errors.

  2. Unclear evaluation procedure for the knowledge questions.
    The paper reports results on 160 knowledge questions but does not sufficiently describe the question formats, reference answers, or scoring procedure. This makes the reported accuracy difficult to independently verify or reproduce.

  3. Inconsistencies in the reported results and experimental settings.
    For example, the abstract reports the best success rates as 68%, 30%, and 21% for Easy, Medium, and Hard tasks, whereas the conclusion reports 68%, 35%, and 33%. The evaluation denominators in Table I.5 sum to 197 tasks rather than the stated 200. In addition, the main text and Appendix E.2 name different models for error classification. These inconsistencies should be clarified and corrected.

  4. Limited evidence of evaluation stability.
    Each model–framework configuration was evaluated only once on the full benchmark. Given the variability in agent outputs and execution behavior, small differences in success rates may not be stable across repeated runs. Repeating the evaluation on at least a representative subset would help establish the reliability of the main comparisons.

Critical strengths
  1. The paper introduces a comprehensive, expert-validated benchmark comprising 200 molecular dynamics simulation tasks with 366 target quantities spanning 76 physical properties, alongside 160 knowledge questions. The simulation tasks cover LAMMPS and GROMACS across three difficulty levels, with a subset independently reproduced by additional domain experts. The authors evaluate multiple LLMs and agent frameworks on the benchmark and analyze the causes of task failures.
Critical weaknesses
  1. Insufficient detail on reference values and scoring tolerances. Although the authors mention independent reference runs and expert reproduction of a subset of tasks, they do not clearly report how many times each reference simulation was repeated, how the final reference values were calculated, or how the scoring tolerances were determined. Without this information, it is difficult to distinguish normal simulation variability from genuine task errors.
Questions for authors

The reliability of the current scoring tolerances also needs to be established, as molecular dynamics simulations involve inherent uncertainty. The benchmark would benefit from a more systematic validation of these tolerances, as well as repeated evaluation runs to assess the stability of the reported results.

2 comments
AuthorsOpenReview ↗

Response to Reviewer GF1a (Part 1/2)

W1 / Q1. Reference values, scoring tolerances, and their reliability

TL;DR: Tolerances now come from each reference value’s own statistical uncertainty (±2.77σ\pm 2.77\sigma, with a 5% floor) and no longer depend on expert or agent output. Rescoring changes 30 of 2,400 runs.

Thank you for this comment! Each configuration was run only once on the full benchmark, although the MD runs has a production run on which statistical sampling could be performed. Moreover, ergodicity of MD simulations ensure that the statistics obtained from spatial and temporal sampling are consistent. Since every problem runs real MD, so one configuration takes about 82 hours of wall-clock time, and all 12 take over 40 days.

We agree that the tolerance needed a clearer and more principled treatment. In the submitted version, the tolerance was a flat 5%, relaxed for eight properties after the human reproduction study. This made it dependent on expert output. We have now replaced it with a single rule that is independent of any agent or expert output:

  • Reference uncertainty. Since each reference value comes from the expert simulation, we estimate its standard error σ\sigma by Flyvbjerg–Petersen blocking over the production run for time averages or from the fit uncertainty for derived quantities (e.g., Young’s modulus). Out of total 366 metics, 29.5% has a wider tolerance than 5%. The dataset JSON stores σ\sigma for every target.
  • Acceptance band. The agent’s simulation and the reference are independent runs of equal length, so their difference has standard error 2σ\sqrt{2}\sigma. A two-sided 95% band is therefore ±1.962σ≈±2.77σ\pm1.96\sqrt{2}\sigma\approx\pm2.77\sigma.
  • Acceptance rule. A value is accepted if ∣x−xref∣≤max(0.05∣xref∣,2.77σ)|x-x_{ref}| \leq \mathrm{max}(0.05 |x_{ref}|, 2.77\sigma). The 5% floor covers differences that σ\sigma does not capture, such as legitimate setup choices (random seed, thermostat implementation, equilibration protocol) and rounding of the stored reference value.

Rescoring all 12 configurations under this rule changes 30 of the 2,400 runs from failure to success and gives partial credit to 11 more. Section 3 and the human-validation paragraph now describe this procedure.

W2. Evaluation procedure for the knowledge questions

TL;DR: The knowledge questions are single-answer multiple choice, asked zero-shot and scored by exact match. Details and examples are now in the paper.

We evaluate LLMs on two QA sets: MD-QA and the MDGym knowledge questions. Each question is multiple-choice with exactly one correct option. Models are queried zero-shot, and answers are scored by exact match on the selected option:

Pass = 1\mathbf{1}(selected option = ground truth)

This is now described in Section 4 (Evaluation metrics), and example questions are in Appendix C.6. Complete set of QA are provided already in the Github.

W3. Inconsistencies in reported results and settings

TL;DR: All numbers are regenerated from the final results. The three inconsistencies you identified are fixed, and no conclusion changes.

We apologise for these errors as some of the simulation runs were completed close to the submission and the numbers got mixed with the intermediate runs. We have now regenerated all the results from the final runs of 200 simulations. The full list of corrections is in our response to Reviewer HMnw, W2. For the three points raised here:

  • The abstract and conclusion now both report 69% / 30% / 30% (Easy / Medium / Hard) for Claude Opus 4.8 + OpenHands.
  • Table I.5 was computed on an earlier 197-problem version. It is now recomputed on all 200 problems (64/70/66).
  • The error-classification model is Claude Sonnet 5, now stated consistently in the main text and Appendix E.2.
AuthorsOpenReview ↗

Response to Reviewer GF1a (Part 2/2)

W4. Stability of the evaluation

TL;DR: Paired tests confirm the main comparisons, and repeated runs show small run-to-run variation (≈\approx 1–2 points) relative to the differences between model tiers.

We address this in two ways.

Paired statistical analysis. We have strengthened the significance tests in Appendix I with a paired analysis that compares configurations on the same problems and models problem identity explicitly. Details and results are in our response to Reviewer gzuV, Q3.

Repeated runs. A full evaluation of the 12 configurations costs about $4,000 in API usage and takes 82 hours of wall-clock time per configuration, because each run executes MD simulations. Repeating the full benchmark is therefore impractical. To estimate run-to-run variability, we ran two configurations, Claude Code + GPT-5.5 and Claude Code + DeepSeek V4 Pro, two additional times on a 50-problem subset (n = 3 runs each):

ConfigurationSuccess rate (mean ±\pm SD)Problems with the same verdict in all 3 runs
Claude Code + GPT-5.514.7 ±\pm 1.2%80%
Claude Code + DeepSeek V4 Pro2.0 ±\pm 2.0%94%

Run-to-run variation is small compared with the gap between model tiers, so the aggregate success rates and tier separation are stable. We have now added this discussion in Appendix I.1.

We believe the revision addresses each of your concerns. If any remain, please let us know and we will be happy to address those during the discussion period. Otherwise, we would be grateful if you would reconsider your score in light of these changes. Looking forward to your response.

Reviewer_gzuVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MDGym evaluates AI agents on 200 molecular-dynamics tasks in LAMMPS and GROMACS, with 366 numerical targets and 160 knowledge questions. It compares 12 model–framework configurations using expert references and workflow diagnostics. The main results report best success rates of 68%, 30%, and 21% across three difficulty levels.

Critical strengths
  1. Real engine execution and numerical targets test the gap between completing a workflow and obtaining correct physical results. Broad coverage could make MDGym a useful community resource.
  2. The crossed model–framework comparison, expert task review, human reproduction study, and post-processing intervention are informative design choices.
Critical weaknesses
  1. MDGym runs real simulation engines and checks numerical results. This shows whether completed workflows produce correct physical results. Its range of tasks could make it useful to the research community.

  2. The paper compares models across simulation frameworks, asks experts to review the tasks, studies how people reproduce the results, and tests the effect of post-processing. These analyses help explain the results.

Questions for authors
  1. Which task set and result version should readers use? Please explain the conflicting task counts and success rates.
  2. What tolerance is used for each property, and how uncertain are the reference runs? Were these tolerances set before the human validation study?
  3. Do the main conclusions still hold when experts verify the failure labels and the analysis accounts for paired tasks?
2 comments
AuthorsOpenReview ↗

Response to Reviewer gzuV (Part 1/2)

Q1. Which task set and result version should readers use?

TL;DR: Use the final 200-problem set and the 69/30/30% best results. Every number is now regenerated from the final result files, and all conclusions are unchanged.

All results in the revision use the final 200-problem set (64/70/66 Easy/Medium/Hard). The conflicting numbers came from intermediate results since the final results were obtained close to the manuscript submission. We apologize for our oversight. The best configuration, Claude Opus 4.8 + OpenHands, solves 69% / 30% / 30% of Easy / Medium / Hard problems, and the abstract and conclusion now are consistent with each other. The revision also replaces the flat 5% tolerance with the statistical criterion described in Section 3 and Appendix C.3 (see Q2). The full list of corrections is in our response to Reviewer HMnw, W2.

Q2. What tolerance is used for each property, how uncertain are the reference runs, and were tolerances set before the human validation study?

TL;DR: Tolerances now follow one rule based on each reference value’s statistical uncertainty from the ground truth simulations, with a 5% floor. The rule depends neither on the expert nor on the agent outputs.

Thank you. The full answer is in our response to Reviewer GF1a, W1. In brief:

  • Reference uncertainty. Since each reference value comes from the expert simulation, we estimate its standard error σ\sigma by Flyvbjerg–Petersen blocking over the production run for time averages or from the fit uncertainty for derived quantities (e.g., Young’s modulus). Out of total 366 metics, 29.5% has a wider tolerance than 5%. The dataset JSON stores σ\sigma for every target.
  • Tolerance. A value is accepted if ∣x−xref∣≤max(0.05∣xref∣,2.77σ)|x-x_{ref}| \leq \mathrm{max}(0.05 |x_{ref}|, 2.77\sigma). This is a two-sided 95% band on the difference between two independent runs of equal length, with a 5% floor.
  • Timing relative to validation. You are right that the submitted tolerance depended on the human study, since we relaxed it for eight properties afterwards. The revised rule uses only the reference run’s statistics and a fixed floor. Further, We rescored all the simulations under it and the human evaluators still reproduce 95 of 95 quantities.

To address this comment, the uncertainty associated with each of the reference values is added in the data JSON in the github [https://anonymous.4open.science/r/MDGym-Dataset-CFE8/].

AuthorsOpenReview ↗

Response to Reviewer gzuV (Part 2/2)

Q3. Do the conclusions hold when experts verify the failure labels and the analysis accounts for paired tasks?

TL;DR: Yes. Detailed response is given below with additional analysis during revision.

Verification of Failure Labels. The failure labels come from two automated components. Trajectory-level categories (e.g., Fabricated Answer, Premature Termination, Syntax Error) are assigned by deterministic rules over the agent’s files and logs (Appendix E.1). Engine errors are classified by an LLM judge (Claude Sonnet 5, majority vote over three classifications) into predefined LAMMPS and GROMACS categories.

We checked these in three ways as follows.

  1. Expert check of rules. The trajectory level error rules were developed on the basis of 20 failed trajectories and then frozen for the rest of the experiments. A domain expert (one of the authors) independently labeled a separate sample of 250 failed trajectories, which agreed with all 250.
  2. Judge consistency. The three LLM-as-a-judge votes were unanimous for over 95% of engine errors, and the expert agrees with every unanimous case checked. All the non-unanimous cases involved the None and Unclassified categories. These categories are not interpreted in the analysis.
  3. Second-model check. To address this concern further, we now used Claude Opus 5.5 to independently re-classify 250 engine errors using the same definitions. The two models disagreed on about 4% of errors, all in the MPI / Parallel category, which is 1.2% of engine errors.

None of these checks changed a label in any category the analysis relies on. Hence, the conclusions of RQ2 and RQ3 remain unchanged.

We have updated Appendix E.1 and E.2.

Paired Analysis. All 12 configurations attempt the same 200 problems, so their scores are not independent, and our original pooled tests did not account for this. We have added a paired analysis for every claim (Appendix I):

  • Configurations are compared on the same problems using Friedman, Wilcoxon signed-rank (Holm-corrected) and Cochran’s Q tests.
  • Problem identity is modelled as an ANOVA factor and as a random intercept in a mixed model.
  • Confidence intervals on success-rate differences come from a bootstrap that resamples whole problems.
  • For difficulty and engine, which are properties of the problem, we test on per-problem means so each problem counts once.

All conclusions hold with the paired tests (see App. I):

  1. Difficulty: Performance declines with difficulty (from 0.425 (Easy) to 0.286 (Medium) and 0.243 (Hard); H = 19.5, p < 0.001). Most of the drop is from Easy to Medium.
  2. Model vs Harness: Model choice remains the dominant factor. With problem identity included, the model explains 16.4% of variance and the harness explains 0.14%.The harness main effect becomes marginally significant (p = 0.048), but in the mixed model neither harness coefficient differs from zero. Within a model the harness does matter (Table I.6), and its direction depends on the model, which explains the interaction term.
  3. Model tiers: The three model tiers are clearly separated: all pairwise Wilcoxon tests are significant after Holm correction. The Opus 4.8 vs GPT 5.5 gap comes entirely from OpenHands, so we report three tiers and claim no general order between the two frontier models.
  4. Ceiling: No configuration’s success rate at Hard has an upper confidence bound above 42.2%.

Results of the paired analysis are added in Appendix I.

We believe the revision addresses each of your concerns. If any remain, please let us know and we will be happy to address those during the discussion period. Otherwise, we would be grateful if you would reconsider your score in light of these changes. Looking forward to your response.

Reviewer_HMnwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

MDGym introduces 200 molecular dynamics tasks, with 366 target quantities across 76 physical properties and 160 knowledge questions. Across 12 agent configurations, the best configuration achieves success rates of about 68%, 30%, and 21% on Easy, Medium, and Hard tasks. These results show that scientific workflows reveal challenges that general coding benchmarks do not fully cover.

Critical strengths
  1. This is an interesting paper. It puts AI in the role of a research assistant that operates simulation software to help run experiments. This is timely work for AI4Science.
  2. The benchmark involves experts and covers diverse tasks. It includes two engines, multiple material categories, and 76 physical properties. Checking whether the final results are physically reasonable is also a strong way to evaluate agents.
Critical weaknesses
  1. The current task success rate seems to measure the ability to reproduce results. It is still some distance from measuring independent scientific judgment. The example in Appendix C.4 already specifies the potential, ensemble, timestep, and run length. This leaves limited room for the agent to design the experiment.
  2. I am concerned about several inconsistencies in the reported results. The best success rates differ between the abstract and conclusion. The Mechanical and Structural counts differ between Figure 1 and Tables C.2/C.3. The denominators also differ between Tables C.1 and I.5. These errors make the results unconvincing to me.
Questions for authors
  1. Appendix E.1 labels runs with no engine errors but an incorrect submitted answer as Incorrect Post-Processing. Could these failures also come from an incorrect physical setup? Also, errors in the log alone do not establish that an answer was fabricated. The agent may have fixed the errors and rerun the simulation.
4 comments
AuthorsOpenReview ↗

Response to Reviewer HMnw (Part 1/2)

W1. Do the tasks measure independent scientific judgment if the prompt fixes the potential, ensemble, timestep and run length?

TL;DR: Fixing the physical parameters is what lets each task have a unique, scorable reference value. The agent still has to design the simulation and the measurement, and this design work increases from Easy to Hard. Examples are now in Appendix C.5.

We agree that MDGym measures whether an agent can perform a specified simulation and post-process correctly to obtain the result, not whether it can design a study from an open question. This is a deliberate choice as making the question unambiguous in terms of potential, ensemble, timestep and run length gives each task one reference value that can be scored automatically. Leaving them open would make the answer depend on legitimate modelling choices and would require expert grading of every run.

MDGym therefore evaluates the judgment needed to turn a stated scientific question into a correct measurement, given a fixed physical model. We treat this as the first requirement for autonomous simulation, since an agent that cannot reproduce a specified simulation cannot be trusted to design one. While the prompt states what to simulate, it does not take the agency on how to set it up in the engine or how to measure the target through appropriate post-processing. These decisions left to the agent grow with difficulty as follows:

LevelIndependent scientific judgement of the agent evaluated (not specified in the prompt)
1 (Easy)Translate the description into a valid engine input (e.g., correct parameter file and element assignment); build or replicate the appropriate input structure; configure output so the relevant quantity is recorded; extract and normalize it correctly (per particle, over the production run, in the relevant units).
2 (Medium)Choose the analysis definition and its parameters: the coordination cutoff (e.g., from the first minimum of the radial distribution function); the atom groups to analyze; the hydrogen-bond geometric criteria; the frames and averaging window; whether to compute the quantity at run time or by post-processing.
3 (Hard)Design the measurement protocol and analysis: the deformation direction, strain rate, and strain range used for the linear fit; the stage sequence of multi-stage runs; the fitting window of the mean-squared displacement, excluding the ballistic regime; the estimator (e.g., Green–Kubo or non-equilibrium) and its correlation or integration length.

The revised Introduction and Limitations now clearly state this scope.

W2. Inconsistencies in the reported numbers

TL;DR: The inconsistencies came from mixing intermediate and final results. All numbers are now regenerated from the final result files. The benchmark has 200 problems split 64/70/66, the best configuration scores 69/30/30%, and no conclusion changes.

Thank you for the careful reading and we apologise for these errors; they came as some of the results were obtained close to the submission resulting in the mixing of intermediate and final result files. All numbers are now regenerated from the final files. The corrections in the manuscript are as follows.

  • Best success rates. The abstract and conclusion now report the final results for the best configuration, Claude Opus 4.8 + OpenHands: 69% Easy, 30% Medium, 30% Hard. These numbers also reflect the revised scoring tolerance (see our response to Reviewer GF1a, W1).

  • Mechanical and Structural counts (Figure 1, Tables C.2 and C.3). The two totals were swapped in Table C.2. The correct totals are Structural 94 (25.7%) and Mechanical 41 (11.2%), which match Figure 1. Table C.3 lists only the 5 most frequent and 5 least frequent metrics, so its counts are a subset of the category totals rather than the totals themselves.

  • Denominators (Tables C.1 and I.5). Table C.1 had the engine labels swapped and two counts misreported. The correct split is GROMACS 30/40/30 and LAMMPS 34/30/36, for a total of 64/70/66 (Easy/Medium/Hard). Table I.5 had been computed on an earlier 197-problem version (63/70/64). We have recomputed it on all 200 problems, and it now uses the same 64/70/66 as Table C.1.

  • Error Classification Model. The model used is Claude Sonnet 5, which is stated in both Section 4 and App. E.2.

We note that all original conclusions remain unchanged after fixing these minor inconsistencies. The updated statistical tests are reported in Appendix I.

AuthorsOpenReview ↗

Response to Reviewer HMnw (Part 2/2)

Q1. Could “Incorrect Post-Processing” failures come from an incorrect setup? Do log errors prove fabrication?

TL;DR: We split the category into setup errors and post-processing errors by running our reference analysis scripts on the agent’s own trajectories. Most error-free failures turn out to be incorrect simulations. “Fabricated Answer” only looks at the last run before the answer was submitted, so fixed-and-rerun simulations are not counted as fabrication.

Incorrect Post-Processing. We agree with you that a run that completes without engine errors may still rest on a physically incorrect setup. To address this, we rename this category as "Failure Despite Error Free Run" and split it into two subcategories (Incorect simulation and incorrect post-processing) using a direct test -- applying the reference post-processing scripts to the agent’s own simulation output. For trajectories where the agents' output format was different, we manually update the input format of the post processing script.

  • If the scripts recover the correct value, we conclude that the simulation was sound and the agent’s post-processing caused the failure (Incorrect Post-processing).
  • If they do not, we conclude that the simulation itself was physically incorrect (Incorrect Simulation).

This test is valid because each problem description is unambigous as it fixes the method and physical setup (force field, ensemble, timestep, run length). A physically correct simulation should therefore reproduce the reference value within the scoring tolerance.

ModelEnv MisunderstandingFabricated AnswerIncorrect Post-processingIncorrect SimulationPremature TerminationSyntax Error
Claude Opus 4.80%1%3%18%15%30%
GPT-5.50%6%7%31%4%43%
DeepSeek V4 Pro5%3%2%4%25%30%
Qwen 3.8 27b3%2%1%1%18%59%

Note that for the two strongest models, incorrect simulations outnumber incorrect analyses six to one (Opus 18% vs 3%) and four to one (GPT 31% vs 7%). This motivated the post-processing ablation already present in the manuscript. Agents given correct outputs still fail 28% of problems at analysis, but in end-to-end runs the simulation is usually already wrong, so the analysis error rarely decides the outcome.

Fabricated Answer. The detection rule checks only the log of the last simulation run executed before final_answer.json was written. It does not check every log in the trajectory. An agent that hit an error, fixed it, and reran the simulation is therefore not labelled as fabricating. Appendix E.1 did not state this clearly, and we have revised it to give the exact rule.

We have updated the Figure 5, the attributed discussion in Section 5 RQ2 and Appendix E.1.

We believe the revision addresses each of your concerns. If any remain, please let us know and we will be happy to address those during the discussion period. Otherwise, we would be grateful if you would reconsider your score in light of these changes. Looking forward to your response.

Thank you for the detailed responses and additional analyses, which address some of my concerns. However, numerical inconsistencies remain between Figure 5 and Table F.5 in the revised manuscript. My initial assessment was borderline between weak rejection and weak acceptance, and these unresolved consistency issues leave me insufficiently confident to raise my score. I believe the work would benefit from more thorough verification and revision before resubmission. I therefore maintain my current rating.

AuthorsOpenReview ↗

Clarification on Figure 5 and Table F.5

Thank you for checking the revised manuscript closely. Figure 5 and Table F.5 report the same data and are consistent. They differ in two ways.

  1. Figure 5 merges two categories. In the revision, we split "Failure Despite Error-Free Run" into Incorrect Post-Processing and Incorrect Simulation (our response to Reviewer HMnw, Q1). Table F.5 reports the two subcategories separately. Figure 5 shows their sum as one bar.

  2. Figure 5 shows values rounded to whole percentages. Table F.5 gives two decimals. Rounding each subcategory before adding them can differ from rounding their sum by one point. For Claude Opus 4.8, 3% + 18% = 21%, while the exact sum 21.54% rounds to 22%.

The exact values for both are below. The sum of the two subcategories in Table F.5 equals the Figure 5 value in every row.

ModelIncorrect Post-Processing (Table F.5)Incorrect Simulation (Table F.5)SumFailure Despite Error-Free Run (Fig. 5)
Claude Opus 4.83.46%18.08%21.54%21.54%
GPT-5.57.14%30.55%37.69%37.69%
DeepSeek V4 Pro1.79%4.46%6.25%6.25%
Qwen 3.8 27b0.78%0.52%1.30%1.30%

The remaining four categories (Env Misunderstanding, Fabricated Answer, Premature Termination, Syntax Error) are identical in Figure 5 and Table F.5.

We have proofread the revised manuscript and checked every reported number against the final result files. In the final version, we will state in the caption of Figure 5 that its bar is the sum of the two subcategories in Table F.5.

If you find any other number that does not match, please tell us which one and we will respond before the discussion closes. Since the inconsistency you raised resolves as above, we would be grateful if you would reconsider your rating.

Comments on the paper

2 comments

Hi reviewers,

Thank you for your reviewing service. The authors have provided rebuttals with some new results. Please confirm that you have read them, state which of your concerns are or are not addressed, and whether that changes your recommendations.

AuthorsOpenReview ↗

Awaiting reviewer feedback

Dear Reviewers,

Thank you again for your reviews and thanks to the AC for initiating the discussion. With about two days left in the discussion period, we would like to know whether our responses have addressed your concerns. The general response below summarises the key revisions and the detailed responses address the comments in a point-by-point fashion. Revised text is in blue in the manuscript.

If these changes resolve your concerns, we would be grateful if you would update your assessment. If anything is still open, please let us know and we will be happy to address those within the remaining window.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.