- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a benchmark for end-to-end molecular dynamics simulations, comprising 200 simulation tasks, 366 target quantities, and 160 knowledge questions. It covers LAMMPS and GROMACS across three difficulty levels. The tasks were developed and reviewed by domain experts, with a subset independently reproduced by additional researchers. The authors compare different LLMs and agent frameworks and analyze failure stages and error types. The benchmark has practical value, but several concerns remain:
-
Insufficient detail on reference values and scoring tolerances.
Although the authors mention independent reference runs and expert reproduction of a subset of tasks, they do not clearly report how many times each reference simulation was repeated, how the final reference values were calculated, or how the scoring tolerances were determined. Without this information, it is difficult to distinguish normal simulation variability from genuine task errors. -
Unclear evaluation procedure for the knowledge questions.
The paper reports results on 160 knowledge questions but does not sufficiently describe the question formats, reference answers, or scoring procedure. This makes the reported accuracy difficult to independently verify or reproduce. -
Inconsistencies in the reported results and experimental settings.
For example, the abstract reports the best success rates as 68%, 30%, and 21% for Easy, Medium, and Hard tasks, whereas the conclusion reports 68%, 35%, and 33%. The evaluation denominators in Table I.5 sum to 197 tasks rather than the stated 200. In addition, the main text and Appendix E.2 name different models for error classification. These inconsistencies should be clarified and corrected. -
Limited evidence of evaluation stability.
Each model–framework configuration was evaluated only once on the full benchmark. Given the variability in agent outputs and execution behavior, small differences in success rates may not be stable across repeated runs. Repeating the evaluation on at least a representative subset would help establish the reliability of the main comparisons.
Critical strengths
- The paper introduces a comprehensive, expert-validated benchmark comprising 200 molecular dynamics simulation tasks with 366 target quantities spanning 76 physical properties, alongside 160 knowledge questions. The simulation tasks cover LAMMPS and GROMACS across three difficulty levels, with a subset independently reproduced by additional domain experts. The authors evaluate multiple LLMs and agent frameworks on the benchmark and analyze the causes of task failures.
Critical weaknesses
- Insufficient detail on reference values and scoring tolerances. Although the authors mention independent reference runs and expert reproduction of a subset of tasks, they do not clearly report how many times each reference simulation was repeated, how the final reference values were calculated, or how the scoring tolerances were determined. Without this information, it is difficult to distinguish normal simulation variability from genuine task errors.
Questions for authors
The reliability of the current scoring tolerances also needs to be established, as molecular dynamics simulations involve inherent uncertainty. The benchmark would benefit from a more systematic validation of these tolerances, as well as repeated evaluation runs to assess the stability of the reported results.
2 comments
Response to Reviewer GF1a (Part 2/2)
W4. Stability of the evaluation
TL;DR: Paired tests confirm the main comparisons, and repeated runs show small run-to-run variation ( 1–2 points) relative to the differences between model tiers.
We address this in two ways.
Paired statistical analysis. We have strengthened the significance tests in Appendix I with a paired analysis that compares configurations on the same problems and models problem identity explicitly. Details and results are in our response to Reviewer gzuV, Q3.
Repeated runs. A full evaluation of the 12 configurations costs about $4,000 in API usage and takes 82 hours of wall-clock time per configuration, because each run executes MD simulations. Repeating the full benchmark is therefore impractical. To estimate run-to-run variability, we ran two configurations, Claude Code + GPT-5.5 and Claude Code + DeepSeek V4 Pro, two additional times on a 50-problem subset (n = 3 runs each):
| Configuration | Success rate (mean SD) | Problems with the same verdict in all 3 runs |
|---|---|---|
| Claude Code + GPT-5.5 | 14.7 1.2% | 80% |
| Claude Code + DeepSeek V4 Pro | 2.0 2.0% | 94% |
Run-to-run variation is small compared with the gap between model tiers, so the aggregate success rates and tier separation are stable. We have now added this discussion in Appendix I.1.
We believe the revision addresses each of your concerns. If any remain, please let us know and we will be happy to address those during the discussion period. Otherwise, we would be grateful if you would reconsider your score in light of these changes. Looking forward to your response.
Response to Reviewer GF1a (Part 1/2)
W1 / Q1. Reference values, scoring tolerances, and their reliability
Thank you for this comment! Each configuration was run only once on the full benchmark, although the MD runs has a production run on which statistical sampling could be performed. Moreover, ergodicity of MD simulations ensure that the statistics obtained from spatial and temporal sampling are consistent. Since every problem runs real MD, so one configuration takes about 82 hours of wall-clock time, and all 12 take over 40 days.
We agree that the tolerance needed a clearer and more principled treatment. In the submitted version, the tolerance was a flat 5%, relaxed for eight properties after the human reproduction study. This made it dependent on expert output. We have now replaced it with a single rule that is independent of any agent or expert output:
Rescoring all 12 configurations under this rule changes 30 of the 2,400 runs from failure to success and gives partial credit to 11 more. Section 3 and the human-validation paragraph now describe this procedure.
W2. Evaluation procedure for the knowledge questions
We evaluate LLMs on two QA sets: MD-QA and the MDGym knowledge questions. Each question is multiple-choice with exactly one correct option. Models are queried zero-shot, and answers are scored by exact match on the selected option:
Pass = 1(selected option = ground truth)
This is now described in Section 4 (Evaluation metrics), and example questions are in Appendix C.6. Complete set of QA are provided already in the Github.
W3. Inconsistencies in reported results and settings
We apologise for these errors as some of the simulation runs were completed close to the submission and the numbers got mixed with the intermediate runs. We have now regenerated all the results from the final runs of 200 simulations. The full list of corrections is in our response to Reviewer HMnw, W2. For the three points raised here: