- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces ElectrolyteMD-Bench, a benchmark for evaluating whether AI agents can autonomously conduct scientifically reliable molecular dynamics studies of lithium-battery electrolytes. It evaluates eight model–harness configurations across dilute, high-concentration, and localized high-concentration electrolyte systems.
The proposed SAEO framework assesses simulation validity, analysis validity, evidence integrity, and outcome calibration. Although all 24 system runs generated production trajectories judged valid, none of the eight studies satisfied all scientific requirements. The authors identify failures in physical definitions, numerical implementations, statistical adequacy, research evidence, and final completion judgments.
Critical strengths
- Scientifically meaningful end-to-end evaluation.
The benchmark goes beyond checking whether agents can execute simulation software. It incorporates interconnected structural, dynamical, and collective-transport analyses, requiring agents to maintain scientific consistency across a complete research process. This provides a useful and challenging test of autonomous scientific reasoning.
- Informative analysis of scientific failure modes.
The paper distinguishes missing analyses, deterministic methodological errors, insufficient statistical sampling, and unsupported completion claims. The examples involving inconsistent Onsager coefficient conventions, numerical underflow, and incorrect final completion declarations are particularly informative. These findings demonstrate why technically successful simulations do not necessarily establish scientifically reliable conclusions.
Critical weaknesses
- Evaluation validity and potential retrospective grading bias.
The evaluation framework includes criteria that were specified retrospectively after examining the agent artifacts. Although the authors acknowledge these changes, they do not fully demonstrate that the final judgments remain invariant under the original task specification.
A particular concern is the requirement for a correlated transference-number estimator, whereas the original task prompt specifies a clearly defined transference number without unambiguously requiring that convention. The paper should establish whether alternative scientifically valid definitions were fairly assessed.
Furthermore, the LLM-assisted expert evaluation lacks reported inter-rater agreement, independent calibration, and quantitative analysis of adjudication uncertainty. These issues affect confidence in the reported scores and model comparisons.
- Limited reproducibility and experimental calibration.
The supplied manuscript does not provide access to the complete benchmark input package, archived agent execution artifacts, evaluation scripts, or detailed expert-adjudication records. These materials are important for independently verifying the scientific failure classifications.
The main comparison includes only one evaluation episode per model–harness configuration, with repeated experiments restricted to the strongest configuration. The observed run-to-run differences raise concerns about the stability of the reported rankings.
Additionally, the absence of an expert reference study under the same 100 ns simulation budget leaves uncertainty about whether all required transport analyses can realistically achieve the benchmark's completion criteria within the available resources.
Questions for authors
Q1 — Retrospective evaluation: Can the authors provide a criterion-level sensitivity analysis comparing judgments under the original task specification and the retrospective refinements? In particular, would any reported failure classifications change when alternative, scientifically valid transference-number conventions are permitted?
Q2 — Evaluation reliability: How many domain experts independently evaluated each study, and what was their agreement before adjudication? Can the authors provide evaluator prompts, expert grading protocols, and representative evidence supporting the final decisions?
Q3 — Benchmark feasibility and reproducibility: Can the authors demonstrate an expert-designed reference workflow satisfying the benchmark under the same simulation budget, and provide the full reproducibility package? Can additional repeated runs establish whether model rankings are stable?
9 comments
Q1&W1 (Part 2):
Impact on Results
4 cells modified in the main matrix: A1 and A2 for the two NE configurations (see Fig. R4 in Response_Letter.pdf (Supplementary Materials)).
2 cells modified in the repeat-run matrix: A2 and A3 for Sol repeat 03, with no net change in score.
Codex + DeepSeek-V4-Flash: 60.7 → 66.1, remaining in 2nd place (see Fig. R5 in Response_Letter.pdf (Supplementary Materials)).
OpenCode + GLM-5.2: 50.7 → 54.3, moving from 5th to 4th place.
OpenCode + GPT-5.6-Luna: Score remains at 52.5, with its rank shifting correspondingly from 4th to 5th.
All study-validity statuses remain unchanged.
This sensitivity audit acknowledges scientifically sound alternative definitions and revises both the rationales and corresponding scores that relied on post-hoc rules. At the same time, it confirms that even after accepting the NE transference number definition, the two runs still lack the required 4-block analytical results. And even after accepting the self-selected contact threshold in Sol repeat 03, the issue that its code failed to execute that threshold still holds. The revisions to individual sub-item judgments do not alter the conclusion that these runs have not met the requirements of a complete valid study, nor do they change the paper's core observation that "generating valid production trajectories is still insufficient to guarantee the reliable completion of scientific research."
Q2&W1: Evaluation Reliability, Joint Expert Review, and Supporting Evidence
We thank the reviewer for raising this. We described the evaluation workflow in Section 3.4. The LLM judge first reads the README and evaluation rule files in the ElectrolyteMD-Bench-admin folder. These files specify how to inspect the scientific validity of simulations and analyses, judge whether sampling is sufficient, assign scores, distinguish errors from research termination states, and report the evidence supporting each judgment. Subsequently, the LLM judge inspects—in read-only mode—the saved simulation inputs, run logs, trajectories, analysis scripts, numerical outputs, and final reports for each study, providing preliminary item-by-item judgments while citing the specific files, code, or results relied upon. Domain researchers then verify these judgments and their supporting evidence to confirm the final adjudication. The final SAEO evaluation criteria, scoring and state definitions, and evidence requirements adopted in the paper are detailed in Appendix A.2. Specifically, Tables A1–A2 list the exact items across the four evaluation dimensions, and Table A3 details the methodological checks and sampling evidence requirements for each physical property.
The manual review was jointly conducted by three researchers with research backgrounds in computational materials science. We adopted a collaborative scientific review process involving verification, discussion, and revision among the three reviewers, with all judgments and modifications uniformly documented in a consolidated review record. The review covered physical definitions, numerical implementations, sampling adequacy, evidence completeness, and disputed judgments; the discussions and corresponding revisions were compiled into a consolidated record. The table, organized from these records, illustrates what was specifically inspected and how the evaluation criteria were determined accordingly (see Table R1 in Response_Letter.pdf, provided in the Supplementary Materials).
Representative evidence is also presented in the original manuscript. Fig. 4 illustrates specific cases such as inconsistent Onsager conventions, non-finite numerical outputs, and missing required analyses. Figs. S1, S2a–b, S3, and S4 report the item-by-item judgments across the Simulation, Analysis, Evidence, and Outcome dimensions, respectively. Consequently, the basis for our final judgments can be traced down to concrete definitions, implementations, numerical outputs, and deliverables, rather than relying solely on an aggregate score.
We thank the reviewer for the positive assessment of our benchmark design and failure analysis, as well as for the valuable suggestions regarding evaluation criteria, expert review, and scientific reproducibility. We provide point-by-point responses below.
Q1&W1: Clarification of Evaluation Criteria and Localized Revisions
We thank the reviewer for raising the concern about retrospective evaluation criteria. We now accept clearly defined, scientifically sound alternative methods and no longer deduct points solely because they differ from the definitions, thresholds, or durations used in our post-hoc unified recalculation. Meanwhile, we separately inspect whether the required physical properties were delivered, whether the code correctly executed the stated method, and whether the block-averaged results and convergence evidence are sufficient.
The specific modifications and their correspondence to the audit matrices (see Fig. R3 in Response_Letter.pdf (Supplementary Materials)) are as follows:
Nernst–Einstein (NE) transference numbers for two configurations: Recognizing the definition and adjusting Coverage and Definition scores.
This corresponds to the two purple cells in the "Conductivity & transference formulas" column of Fig. R3(b) in Response_Letter.pdf (Supplementary Materials) for Codex + DeepSeek-V4-Flash and OpenCode + GLM-5.2. Because the original task did not restrict the evaluation to a single transference number convention, we no longer deduct points for adopting an explicitly labeled NE definition.
Codex + DeepSeek-V4-Flash: A1 Coverage 0 → 1; A2 Definition 0.5 → 1. Once the NE definition is accepted, all required physical properties have analytical outputs. 4-block completeness and implementation correctness are evaluated under their respective items.
OpenCode + GLM-5.2: A1 Coverage 0 → 0.5; A2 Definition 0.5 → 1. Its NE output is accepted, but the required potential energy output is still missing. Therefore, it receives only partial credit for coverage.
Although the transference number definitions for both configurations are accepted, their required 4-block analyses remain incomplete, and neither passes the study-validity check.
Contact definition in Sol repeat 03: Credit added for Definition, deducted for Implementation.
This corresponds to the purple cell under "Contact & cluster definitions" in Fig. R3(b) in Response_Letter.pdf (Supplementary Materials), and the error cell under "Following stated methods" in Fig. R3(a) in Response_Letter.pdf (Supplementary Materials).
A2 Definition 0 → 1. Its registered 0.32 nm contact threshold is a permissible methodological choice under the original task and is not penalized for differing from the unified threshold.
A3 Implementation 1 → 0. In practice, the code directly used neighbor pairs within 1.2 nm to construct edges without executing its own stated 0.32 nm filter.
Consequently, these two cells are updated in the repeat-run matrix. Because both items carry equal weight, the total score remains unchanged.
Analysis durations for GLM and Sol repeat 03: Removing the unified duration veto while retaining evidence-based scores.
This corresponds to the two purple cells under "Fitting windows & analysis durations" in Fig. R3(b) in Response_Letter.pdf (Supplementary Materials).
GLM: Both A5 and A6 remain at 0.5. The report has not demonstrated a stable plateau for the viscosity integral, and 4-block results are missing for CN, transference number, and AGG.
Sol repeat 03: Both A5 and A6 remain at 0.5. Partial fitting and block diagnostics are present, but block fitting failed for multiple collective transport properties, which is insufficient to support the reliability of all transport results.
These two updates revise the rationale for the judgments without awarding additional points.
Viscosity in Codex + DeepSeek-V4-Pro: Correcting the attribution while keeping the score unchanged.
This corresponds to the yellow cell under "Viscosity calculation & data requirements" in Fig. R3(b) in Response_Letter.pdf (Supplementary Materials). Its pressure output interval is 0.2 ps, and the 4-block coefficients of variation for viscosity across the three systems are approximately 2.9%, 1.9%, and 3.6%, all passing the 25% threshold in the code. Therefore, "overly sparse pressure output" or "4-block instability" is no longer used as a reason for rejection.
A5 remains at 0.5. An integral analysis is present, but taking the average over the tail of a fixed window is insufficient to demonstrate a reliable plateau.
A6 remains at 0. This item checks the block-averaged results across all required physical properties. But the collective conductivity blocks across the three systems still exhibit inf/nan, precluding a valid consistency check. Passing the viscosity block check cannot substitute for these failed results.