acceptodds
Under review as a conference paper at ICLR 2027

ELECTROLYTEMD-BENCH: HOW WELL DO AI AGENTS CONDUCT MOLECULAR DYNAMICS STUDIES ACROSS ELECTROLYTE SOLVATION REGIMES?

Abstract

AI agents are advancing the automation of molecular dynamics (MD) research, with the potential to accelerate discovery and design of lithium-battery electrolytes. However, whether they can autonomously complete scientifically reliable electrolyte MD studies remains insufficiently evaluated. We introduce ElectrolyteMD-Bench, which unifies dilute, high-concentration, and localized high-concentration electrolytes in a comparative study across distinct solvation regimes. The benchmark connects local coordination, ionic association, single-particle dynamics, and collective transport within one continuous research process. Agents autonomously perform system construction, equilibration, production sampling, property analysis, and cross-system interpretation, while deciding whether to correct errors, continue sampling, or stop. We audit each study along four dimensions: simulation validity, analysis validity, evidence integrity, and outcome calibration. Across eight model–harness configurations, all 24 system runs produced valid production trajectories, yet none of the eight studies satisfied all scientific requirements. Failures included missing analyses, incorrect physical definitions or numerical implementations, insufficient statistical support, and completion claims inconsistent with retained evidence. These results expose a gap between successful MD workflow execution and reliable scientific study completion. ElectrolyteMD-Bench therefore provides a foundational test and defines a key capability threshold that autonomous electrolyte simulation must cross to advance toward computation-first materials exploration.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
AI for Science, AI Agents, Electrolyte Materials, Molecular Dynamics Simulation, Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_FEMJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces ElectrolyteMD-Bench, a benchmark for evaluating whether AI agents can autonomously conduct scientifically reliable molecular dynamics studies of lithium-battery electrolytes. It evaluates eight model–harness configurations across dilute, high-concentration, and localized high-concentration electrolyte systems.

The proposed SAEO framework assesses simulation validity, analysis validity, evidence integrity, and outcome calibration. Although all 24 system runs generated production trajectories judged valid, none of the eight studies satisfied all scientific requirements. The authors identify failures in physical definitions, numerical implementations, statistical adequacy, research evidence, and final completion judgments.

Critical strengths
  1. Scientifically meaningful end-to-end evaluation.

The benchmark goes beyond checking whether agents can execute simulation software. It incorporates interconnected structural, dynamical, and collective-transport analyses, requiring agents to maintain scientific consistency across a complete research process. This provides a useful and challenging test of autonomous scientific reasoning.

  1. Informative analysis of scientific failure modes.

The paper distinguishes missing analyses, deterministic methodological errors, insufficient statistical sampling, and unsupported completion claims. The examples involving inconsistent Onsager coefficient conventions, numerical underflow, and incorrect final completion declarations are particularly informative. These findings demonstrate why technically successful simulations do not necessarily establish scientifically reliable conclusions.

Critical weaknesses
  1. Evaluation validity and potential retrospective grading bias.

The evaluation framework includes criteria that were specified retrospectively after examining the agent artifacts. Although the authors acknowledge these changes, they do not fully demonstrate that the final judgments remain invariant under the original task specification.

A particular concern is the requirement for a correlated transference-number estimator, whereas the original task prompt specifies a clearly defined transference number without unambiguously requiring that convention. The paper should establish whether alternative scientifically valid definitions were fairly assessed.

Furthermore, the LLM-assisted expert evaluation lacks reported inter-rater agreement, independent calibration, and quantitative analysis of adjudication uncertainty. These issues affect confidence in the reported scores and model comparisons.

  1. Limited reproducibility and experimental calibration.

The supplied manuscript does not provide access to the complete benchmark input package, archived agent execution artifacts, evaluation scripts, or detailed expert-adjudication records. These materials are important for independently verifying the scientific failure classifications.

The main comparison includes only one evaluation episode per model–harness configuration, with repeated experiments restricted to the strongest configuration. The observed run-to-run differences raise concerns about the stability of the reported rankings.

Additionally, the absence of an expert reference study under the same 100 ns simulation budget leaves uncertainty about whether all required transport analyses can realistically achieve the benchmark's completion criteria within the available resources.

Questions for authors

Q1 — Retrospective evaluation: Can the authors provide a criterion-level sensitivity analysis comparing judgments under the original task specification and the retrospective refinements? In particular, would any reported failure classifications change when alternative, scientifically valid transference-number conventions are permitted?

Q2 — Evaluation reliability: How many domain experts independently evaluated each study, and what was their agreement before adjudication? Can the authors provide evaluator prompts, expert grading protocols, and representative evidence supporting the final decisions?

Q3 — Benchmark feasibility and reproducibility: Can the authors demonstrate an expert-designed reference workflow satisfying the benchmark under the same simulation budget, and provide the full reproducibility package? Can additional repeated runs establish whether model rankings are stable?

9 comments

We thank the reviewer for the positive assessment of our benchmark design and failure analysis, as well as for the valuable suggestions regarding evaluation criteria, expert review, and scientific reproducibility. We provide point-by-point responses below.

Q1&W1: Clarification of Evaluation Criteria and Localized Revisions

We thank the reviewer for raising the concern about retrospective evaluation criteria. We now accept clearly defined, scientifically sound alternative methods and no longer deduct points solely because they differ from the definitions, thresholds, or durations used in our post-hoc unified recalculation. Meanwhile, we separately inspect whether the required physical properties were delivered, whether the code correctly executed the stated method, and whether the block-averaged results and convergence evidence are sufficient.

The specific modifications and their correspondence to the audit matrices (see Fig. R3 in Response_Letter.pdf (Supplementary Materials)) are as follows:

Nernst–Einstein (NE) transference numbers for two configurations: Recognizing the definition and adjusting Coverage and Definition scores.

This corresponds to the two purple cells in the "Conductivity & transference formulas" column of Fig. R3(b) in Response_Letter.pdf (Supplementary Materials) for Codex + DeepSeek-V4-Flash and OpenCode + GLM-5.2. Because the original task did not restrict the evaluation to a single transference number convention, we no longer deduct points for adopting an explicitly labeled NE definition.

Codex + DeepSeek-V4-Flash: A1 Coverage 0 → 1; A2 Definition 0.5 → 1. Once the NE definition is accepted, all required physical properties have analytical outputs. 4-block completeness and implementation correctness are evaluated under their respective items.

OpenCode + GLM-5.2: A1 Coverage 0 → 0.5; A2 Definition 0.5 → 1. Its NE output is accepted, but the required potential energy output is still missing. Therefore, it receives only partial credit for coverage.

Although the transference number definitions for both configurations are accepted, their required 4-block analyses remain incomplete, and neither passes the study-validity check.

Contact definition in Sol repeat 03: Credit added for Definition, deducted for Implementation.

This corresponds to the purple cell under "Contact & cluster definitions" in Fig. R3(b) in Response_Letter.pdf (Supplementary Materials), and the error cell under "Following stated methods" in Fig. R3(a) in Response_Letter.pdf (Supplementary Materials).

A2 Definition 0 → 1. Its registered 0.32 nm contact threshold is a permissible methodological choice under the original task and is not penalized for differing from the unified threshold.

A3 Implementation 1 → 0. In practice, the code directly used neighbor pairs within 1.2 nm to construct edges without executing its own stated 0.32 nm filter.

Consequently, these two cells are updated in the repeat-run matrix. Because both items carry equal weight, the total score remains unchanged.

Analysis durations for GLM and Sol repeat 03: Removing the unified duration veto while retaining evidence-based scores.

This corresponds to the two purple cells under "Fitting windows & analysis durations" in Fig. R3(b) in Response_Letter.pdf (Supplementary Materials).

GLM: Both A5 and A6 remain at 0.5. The report has not demonstrated a stable plateau for the viscosity integral, and 4-block results are missing for CN, transference number, and AGG.

Sol repeat 03: Both A5 and A6 remain at 0.5. Partial fitting and block diagnostics are present, but block fitting failed for multiple collective transport properties, which is insufficient to support the reliability of all transport results.

These two updates revise the rationale for the judgments without awarding additional points.

Viscosity in Codex + DeepSeek-V4-Pro: Correcting the attribution while keeping the score unchanged.

This corresponds to the yellow cell under "Viscosity calculation & data requirements" in Fig. R3(b) in Response_Letter.pdf (Supplementary Materials). Its pressure output interval is 0.2 ps, and the 4-block coefficients of variation for viscosity across the three systems are approximately 2.9%, 1.9%, and 3.6%, all passing the 25% threshold in the code. Therefore, "overly sparse pressure output" or "4-block instability" is no longer used as a reason for rejection.

A5 remains at 0.5. An integral analysis is present, but taking the average over the tail of a fixed window is insufficient to demonstrate a reliable plateau.

A6 remains at 0. This item checks the block-averaged results across all required physical properties. But the collective conductivity blocks across the three systems still exhibit inf/nan, precluding a valid consistency check. Passing the viscosity block check cannot substitute for these failed results.

Q1&W1 (Part 2):

Impact on Results

4 cells modified in the main matrix: A1 and A2 for the two NE configurations (see Fig. R4 in Response_Letter.pdf (Supplementary Materials)).

2 cells modified in the repeat-run matrix: A2 and A3 for Sol repeat 03, with no net change in score.

Codex + DeepSeek-V4-Flash: 60.7 → 66.1, remaining in 2nd place (see Fig. R5 in Response_Letter.pdf (Supplementary Materials)).

OpenCode + GLM-5.2: 50.7 → 54.3, moving from 5th to 4th place.

OpenCode + GPT-5.6-Luna: Score remains at 52.5, with its rank shifting correspondingly from 4th to 5th.

All study-validity statuses remain unchanged.

This sensitivity audit acknowledges scientifically sound alternative definitions and revises both the rationales and corresponding scores that relied on post-hoc rules. At the same time, it confirms that even after accepting the NE transference number definition, the two runs still lack the required 4-block analytical results. And even after accepting the self-selected contact threshold in Sol repeat 03, the issue that its code failed to execute that threshold still holds. The revisions to individual sub-item judgments do not alter the conclusion that these runs have not met the requirements of a complete valid study, nor do they change the paper's core observation that "generating valid production trajectories is still insufficient to guarantee the reliable completion of scientific research."

Q2&W1: Evaluation Reliability, Joint Expert Review, and Supporting Evidence

We thank the reviewer for raising this. We described the evaluation workflow in Section 3.4. The LLM judge first reads the README and evaluation rule files in the ElectrolyteMD-Bench-admin folder. These files specify how to inspect the scientific validity of simulations and analyses, judge whether sampling is sufficient, assign scores, distinguish errors from research termination states, and report the evidence supporting each judgment. Subsequently, the LLM judge inspects—in read-only mode—the saved simulation inputs, run logs, trajectories, analysis scripts, numerical outputs, and final reports for each study, providing preliminary item-by-item judgments while citing the specific files, code, or results relied upon. Domain researchers then verify these judgments and their supporting evidence to confirm the final adjudication. The final SAEO evaluation criteria, scoring and state definitions, and evidence requirements adopted in the paper are detailed in Appendix A.2. Specifically, Tables A1–A2 list the exact items across the four evaluation dimensions, and Table A3 details the methodological checks and sampling evidence requirements for each physical property.

The manual review was jointly conducted by three researchers with research backgrounds in computational materials science. We adopted a collaborative scientific review process involving verification, discussion, and revision among the three reviewers, with all judgments and modifications uniformly documented in a consolidated review record. The review covered physical definitions, numerical implementations, sampling adequacy, evidence completeness, and disputed judgments; the discussions and corresponding revisions were compiled into a consolidated record. The table, organized from these records, illustrates what was specifically inspected and how the evaluation criteria were determined accordingly (see Table R1 in Response_Letter.pdf, provided in the Supplementary Materials).

Representative evidence is also presented in the original manuscript. Fig. 4 illustrates specific cases such as inconsistent Onsager conventions, non-finite numerical outputs, and missing required analyses. Figs. S1, S2a–b, S3, and S4 report the item-by-item judgments across the Simulation, Analysis, Evidence, and Outcome dimensions, respectively. Consequently, the basis for our final judgments can be traced down to concrete definitions, implementations, numerical outputs, and deliverables, rather than relying solely on an aggregate score.

Reviewer_tnjFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors benchmark agentic systems designing electrolytes by molecular dynamics, auditing decisions across key steps of the process.

Critical strengths

S1: Well-designed and well-written

Overall, I think the paper is well-written, informative and easy to follow, even despite my own limited experience with MD simulations. As far as I can tell, the MD experiment pipeline is carefully designed around current practices, and the analysis of the agents' performance across steps is well-designed. I think the large spread of scores in Figure 2 lends credibility to the benchmark as nontrivial and probing meaningful differences across systems. Also, a meaningful range of models and harnesses are evaluated.

Critical weaknesses

W1: Intended use

I don't fully understand how you intend for this to be used. As a one-off benchmarking study it has a much lower value than if you made it a continually-available and maintained public benchmark. I understand that the approach with domain experts reviewing the LLM judge will not scale over time, but I think that once you've established that the LLM judge is sufficiently reliable, you could skip the manual reviewing.

While you provide ample detail in the appendix, I think that you should strive to fully open-source the code. Even if you are unable to maintain a benchmarking challenge yourself, open-sourcing the code would greatly simplify reproducibility such that others could potentially do it.

W2: Scoring

Regarding the score in eq. (1), is a plain arithmetic mean really the best for accurately capturing the performance? Its simplicity is good, of course, but I can also imagine that the severity of the different items in a dimension ranges from mild to critical - where the latter type would imply cascading failures in subsequent steps. Designing the score judiciously would likely be especially important if is later used a reward when finetuning an agentic system with, e.g., reinforcement learning.

Tip: it generally looks better when floats are top-aligned. Try it yourself with Figures 2 and 3.

Questions for authors

Q1: LLM judge reliability What was the verdict on the LLM judges' performance? How did you assess that?

5 comments

We thank the reviewer for the positive assessment of the benchmark design and presentation, and for the constructive suggestions.

W1: Benchmark Positioning and Open-Source Release

We thank the reviewer for the valuable feedback on benchmark reuse and open-source availability. We agree that open materials will help ElectrolyteMD-Bench support the continued evaluation and improvement of agent systems. The benchmark’s primary purpose is to localize specific failures across the scientific workflow. Relevant materials are provided in the Supplementary Materials, enabling researchers to maintain and extend the benchmark and evaluate new systems.

Our supplementary experiments also illustrate a further use of our work (see our response to Reviewer 1, Q2, and Fig. R1 in Response_Letter.pdf (Supplementary Materials)), where we translated identified failure modes into a reusable methodological skill and conducted corresponding evaluation. This demonstrates a practical path from failure localization to skill development and subsequent re-evaluation. Opening these materials will allow the community to build on documented failures, develop methodological support, and test whether proposed improvements address the problems identified by the benchmark.

W2: Composite Scoring and Error Severity

We thank the reviewer for pointing out that arithmetic averaging may obscure differences in the severity of errors. The composite score summarizes the overall performance of each configuration across the four SAEO dimensions, and Figure 2 shows clear differences among configurations. Meanwhile, our evaluation preserves the upstream–downstream dependencies in the research workflow and their scientific consequences. As specified in Appendix A.2.1, invalid upstream data cannot establish the scientific validity of downstream results. Appendix A.2.2 further clarifies that high scores in other dimensions cannot compensate for scientific errors or unsupported study-completion claims. The independent "Valid study" judgments in Figures 2 and 3 reflect this non-compensatory assessment of scientific validity.

We also retain item-level judgments and analyze specific failure modes and their scientific consequences in Sections 4.2–4.3. The three-level item scoring scheme is itself designed to improve discrimination: scores of 1, 0.5, and 0 indicate satisfied, partially satisfied, and unsatisfied criteria, respectively. This allows us to distinguish insufficient sampling (SI) under a valid estimator from deterministic methodological or numerical errors (F) and missing required analyses (M). The scoring and scientific-state definitions are provided in Appendix A.2.2, while Figures S1–S4 document the individual judgments and their supporting evidence.

For reinforcement learning, the reward signal could be derived from item-level scientific judgments rather than directly from the composite score. Positive or negative rewards could be assigned to specific actions based on their scientific validity, such as implementing estimators correctly, identifying sampling insufficiency, repairing methodological errors, and making evidence-consistent stopping and completion decisions. This would allow the reward design to distinguish scientifically appropriate decisions from errors with different downstream consequences, while incorporating the final study-validity judgment as a terminal reward.

Q1. Evaluating the Reliability of the LLM Judge

We thank the reviewer for requesting clarification on the performance of the LLM judge. As described in Section 3.4, the LLM examines archived study artifacts item by item and produces preliminary judgments with supporting evidence citations. Domain experts subsequently review these judgments and the underlying evidence to establish the final decisions.

We additionally conducted three independent evaluation runs using GPT-5.6-Sol (high) across the eight main configurations, with 168 item-level judgments per run. Across the three pairwise comparisons, the mean exact agreement was 73.4%, the mean Pearson correlation for item-level scores was 0.721, and the mean Spearman rank correlation for overall configuration scores was 0.802, indicating a degree of reproducibility in the LLM judge. All three runs identified the three representative issues discussed in the manuscript: inconsistent charge-sign conventions between the transference-number formula and its underlying calculations, non-finite blockwise transport outputs, and a conflict between missing required analyses and a clean-valid completion claim. Item-level disagreements across runs involved differences in partial-credit assignment and interpretation of criterion applicability, with 98.5% occurring between adjacent satisfaction categories.

Accordingly, we position the LLM judge as an evidence-supported assistant in the evaluation process, while retaining domain-expert verification of the original artifacts and final judgments. This is consistent with the workflow described in Section 3.4.

Reviewer_W84MOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduce ElectrolyteMDBench, which evaluates agent's ability to autonomously run and analyze MD simulations for electrolytes. The results conclude that the systems all produce valid productions, but often fail in making proper scientific decisions to perform meaningful analysis and hypothesis updates. In other words, there is a gap between successful MD workflow execution and reliable scientific study, as the authors describe in the abstract.

Critical strengths
  • It is nice the the authors present a way to evaluate agents on making reasonable and theory-bound decisions during scientific research. This benchmark identifies scientific criteria needed for electrolyte studies such as relevant physical formulas, periodic boundary handling, ion correlation terms, and general scientific criteria such as statistical convergence and uncertainties.
Critical weaknesses
  • Information leakage/memorization concern: The paper does not mention if they investigated whether the previous studies of chosen electrolyte systems (for example, cited Mohanakrishnan et al.) were seen by the agents, and whether that knowledge could have contributed to scientific decisions in any way.
  • I feel like many of the problems and failure modes identified here can be solved with better domain-specific harness or skills. And that is a more realistic scenario practically (agents being used with predefined domain-specific harness or skills). All harnesses tested here (such as Codex) are general purpose coding harness and not the scientific or domain specific ones, even though the paper mentions quite a few domain specific ones (or MD-specialized ones) in the Related Works section. Benchmarking general-purpose coding agents on a challenging scientific task can be still valuable, but then I think it is less significant than testing more realistic cases.
  • Plus, I think it is not very clear if the failure is coming from lack of knowledge vs. poor reasoning. There are some discussions about this matter in the paper, but the distinctive test would be giving all necessary knowledge to the agents and then see if they can perform properly. If they do, then the failure comes from lack of knowledge, if they don't, then poor reasoning is contributing.
  • I agree with the authors that it is important to check if agents can make scientifically coherent, domain specific decisions, but I think what would be more valuable is whether they can draw scientifically meaningful and accurate conclusions from their simulations and analysis. I don't think the current version tests that. It seems like there are some known facts about the chosen electrolyte systems - we could test if the agents can draw mechanistic conclusions that agree with the experts. Only then, we can say we evaluated "reliable scientific study completion" beyond evaluating "MD workflow execution". Of course, to do this, it would be important to control the information leakage. If we just benchmark MD workflow execution (including both simulation and analysis), then I think it makes sense to make the domain broader to increase the significance of this benchmark.
Questions for authors
  • Can you investigate for the information leakage/memorization concern?
  • Can you provide results with more MD- or domain specialized agentic workflows, like the ones cited in the Related Works section? Even if such works are not specialized for electrolytes, they could still do much better than general purpose agents like Codex, given some light modification.
  • Can you additionally assess whether agent generated conclusions recover the known solvation trends, to see if any of them can correctly draw mechanistic conclusions from their simulation and analysis?
9 comments

We thank the reviewer for the positive assessment of our benchmark and provide point-by-point responses to their valuable comments below.

Q1&W1: Auditing Potential Information Leakage

We thank the reviewer for this helpful suggestion. We agree that examining the agents’ exposure to prior literature helps clarify how their conclusions were reached. We audited the fixed input package and preserved execution logs from all 10 runs. Materials from Kim et al. were provided for force-field parameters, and information on anion-rich solvation structures appeared in tool outputs during some runs. The paper by Mohanakrishnan et al. was not included in the input package, and we found no record in the preserved logs of agents attempting to retrieve it.

We acknowledge that the supplied literature may have influenced the agents’ methodological choices or interpretations. However, our benchmark permits the use of prior scientific knowledge, while requiring agents to generate valid trajectories and support their conclusions with their own computations and sampling diagnostics. Agents must base their conclusions on the files they generate and show whether those files support the conclusions, while familiarity with an expected trend does not invalidate a conclusion supported by these computations. As detailed in our response to Q3, we checked whether each agent’s analysis code and numerical results supported the solvation trends it reported. Thus, agreement with published trends alone is insufficient.

Q2&W2, W3: Domain-Specific Methodological Support vs. Reasoning Capability

We thank the reviewer for this helpful suggestion. We agree that domain-specific methodological support is an important application mode for scientific agents. In our main experiments, all 24 system runs across the eight model–harness configurations passed the Simulation (S) dimension. Therefore, new experiments are added focusing on A/E/O—namely, the correctness of analytical methods, the completeness of evidence, and whether diagnostic outcomes are properly handled and reflected in the final conclusions.

To address the reviewer's question regarding domain knowledge versus reasoning capability, we introduced two intervention factors. First, alongside GPT-5.6-Sol (high), we introduced GPT-6-Astra (ultra) to examine performance under an upgraded model and reasoning configuration. Second, we distilled the failure modes identified in the original manuscript into a domain-methodology skill that explicitly supplies knowledge and verification steps regarding physical definitions, charge conventions, numerical precision, contact criteria, and outcome adjudication (see Fig. R1(a) in Response_Letter.pdf (Supplementary Materials)). This skill was frozen prior to the experiments and contains no reference numerical values, expected trends, or candidate-specific analysis code. It is also provided in the Supplementary Materials.

Of the four configurations shown in Fig. R1(b) in Response_Letter.pdf (Supplementary Materials), only GPT-6-Astra (ultra) with the domain-methodology skill met the study-validity criteria for the fixed-trajectory analysis. The primary reason Sol + skill did not pass is that it accepted the dilute-solution viscosity before sampling adequacy had been established. Its viscosity integral curve still exhibited pronounced variations in the later time regime without forming a stable plateau backed by sufficient evidence, yet the final status retained an acceptance judgment. The primary reason Astra without skill did not pass is that identified sampling limitations failed to constrain its final acceptance judgment. This run found that conductivity estimates for HCE and LHCE varied significantly with the fitting time window and acknowledged that a reliable long-time limit had not yet been established, but failed to adjust the final acceptance status for those two results accordingly.

In contrast, Astra + skill completed the required analyses and revised its acceptance judgments based on further diagnostics. After completing block-by-block window checks, it withdrew the accepted status for HCE conductivity and correlated transference number, while also fixing a floating-point boundary issue in the viscosity integration window. For results still constrained by sampling limitations, it retained the computed values and diagnostic evidence but refrained from making transport ranking claims based on them. Under the original rules, insufficient sampling under a valid estimator with thorough diagnostic support qualifies as a legitimate terminal state, provided that stopping is justified and final conclusions align with the evidence. Consequently, this configuration met the study-validity requirements within the scope of fixed-trajectory analysis.

In this comparison, neither adopting a stronger model and reasoning configuration alone nor providing the domain-methodology skill alone enabled the study to pass; only the combination of both met the valid-study standard. This result indicates that failures do not stem from a single knowledge gap or reasoning deficit alone, suggesting instead that model capability and domain-methodological support may play complementary roles. The combined condition passed verification, which likely reflects that the two enhancements improved distinct stages of the workflow, jointly bringing the study across the validity threshold. This experiment also demonstrates the benchmark's value toward achieving scientifically reliable electrolyte molecular dynamics research. Beyond identifying specific failures, ElectrolyteMD-Bench enables their abstraction into reusable domain-methodology skills and evaluates the resulting improvements in electrolyte MD research under the same criteria.

Q3&W4: Recovery of Known Solvation Trends and Scientific Interpretation.

We thank the reviewer for raising this. Several agents successfully recovered scientifically meaningful local solvation trends. Section 4.4 and Figure 5a of the original manuscript already showed that GPT-5.6-Sol recovered these trends across all three independent runs, with the original reports containing the corresponding numerical values and interpretations. These represent local scientific successes.

We further audited the 8 main comparison configurations, examining two separate questions: whether the corresponding trends can be recovered from the trajectories, and whether the conclusions formulated by agents are supported by its own analysis. A unified reanalysis of the archived trajectories shows that all of them exhibit local structural trends consistent with prior studies:

  • From DE to HCE, the DMC oxygen coordination number decreases while the FSI⁻ oxygen coordination number increases.

  • Local DMC/FSI⁻ coordination numbers in HCE and LHCE are similar.

  • Concentrated systems exhibit a higher proportion of ion aggregation.

  • In LHCE, close contacts with TTE as defined by the specified cutoff are rare.

It is important to note that recovering a trend in the trajectory is not equivalent to the original agent having completed the same analysis. After cross-checking the original reports, code, and numerical outputs one by one across the 8 main configurations, 4 correctly identified the decrease in DMC coordination and increase in FSI⁻ coordination from DE to HCE, 3 noted the similarity in local coordination between HCE and LHCE, 6 identified the low close-coordination contribution of TTE, and 5 provided evidence-backed conclusions regarding changes in ion association or aggregation (see Fig. R2 in Response_Letter.pdf (Supplementary Materials)).

Directional correctness and computational correctness can decouple. A concrete example is OpenCode + DeepSeek-V4-Flash: the direction of its FSI⁻ coordination change matched expectations, but its oxygen-site RDF integration omitted the normalization factor for the four selected oxygen atoms per FSI⁻ ion. Upon re-integrating the exact same RDF, the values for DE / HCE / LHCE changed from approximately 0.249 / 0.686 / 0.681 to 0.995 / 2.743 / 2.723. While the relative ordering remained unchanged, the quantitative calculation contained a clear error. This case illustrates that although an agent may arrive at the correct physical trend, that trend is not necessarily obtained through reliable computation.

At the stronger level of structure–transport mechanisms, among the 8 main configurations, 2 received only qualitative or partial support, 3 offered interpretations not backed by sufficient evidence, 2 proposed no corresponding mechanism, and 1 lacked required deliverables; none established a fully supported, complete mechanistic chain. Therefore, we explicitly credit agents for recovering meaningful known structural trends while separately evaluating their quantitative implementation, sampling support, and mechanistic inference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.