- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces MOLSTRUCT, a benchmark for molecular structural reasoning organized around an intervention–relaxation pipeline. Starting from stratified QM9 samples, the authors apply five intervention types (molecular coupling, atom substitution, bond rotation, functional-group editing, thermal perturbation), generate relaxed trajectories via MACE-OFF23 + BFGS optimization or ReaxFF/LAMMPS reactive MD, and programmatically derive ground truth from geometry/energy/bond-order records (Eq. 3). The benchmark spans three structural granularities (atoms & bonds, functional groups, whole molecules) and four capability axes (structural perception, inverse change reasoning, stability reasoning, structural prediction), totaling 6,100 questions with paired CIF-text and rendered-image modalities. Main findings: models perform reasonably on perception but struggle with inverse/stability reasoning and post-relaxation prediction; most models exhibit a systematic bias toward selecting the unrelaxed (least stable) structure in stability tasks.
Critical strengths
- Well-motivated and clearly differentiated positioning. Unlike editing benchmarks such as MolLangBench and AtomWorld where the edit itself is the endpoint, this work treats the post-intervention relaxed outcome as the evaluation target — a genuinely under-explored capability axis. The coverage comparison in Table 1 is persuasive.
- Physics-simulation-derived labels rather than LLM-generated ones. Ground truth is programmatically exported from simulation records, reducing label noise; the AI Use Statement is also properly disclosed.
- Clean task matrix. 3 scales × 4 capabilities with a natural difficulty progression; CIF/image dual versions of the same questions share ground truth, enabling controlled modality comparisons.
- The similarity-bias finding is the most informative result. Models tend to select candidates most similar to the pre-relaxation structure (i.e., the least stable one), pushing accuracy below random guessing; the observation that image modality can score higher precisely because weaker visual similarity judgment attenuates this bias is a nice mechanistic insight.
- Useful failure taxonomy for SMILES reconstruction (syntax errors → formula errors → isomer confusion).
Critical weaknesses
- Reliability of stability labels is not validated (most critical). Stability labels are the arg min of single-point potential energies from MACE-OFF23-small: (a) no DFT-level spot-check of the small MLFF's accuracy on these tasks; (b) lowest potential energy ≠ thermodynamic stability (entropy/free energy ignored, and BFGS endpoints depend on initial points); (c) "retain only geometrically and energetically distinguishable states" is stated without any thresholds. If labels are noisy, the central claim that "models fail at stability reasoning" does not hold.
- The "Expert" row in Table 4 (87.3) is entirely undocumented — who the expert is, which questions, what protocol, how many samples. This is a significant reporting gap, and the large expert–model gap underpins the paper's central narrative.
- No random baselines or statistical significance. Most single-choice accuracies fall in the 10–30% range, but without per-task chance levels (candidate counts vary) these numbers are uninterpretable; the text claims "below random guessing" without ever reporting the random baseline. Per-task sizes are small (200–1000) with no confidence intervals.
Questions for authors
see Weaknesses
3 comments
Official Comment by Authors
We sincerely thank the reviewers for their valuable review. We have substantially revised the manuscript and hope that these revisions adequately address their concerns. We have uploaded an updated version of the paper, with the revised text marked in blue for easier review.
As the Author–Reviewer Discussion period is approaching its end, we welcome any further questions or concerns. We greatly appreciate the reviewers' time and effort in evaluating our work.
A Brief Update on the Discussion
Dear Reviewer FT5C,
Apologies for the additional follow-up. We wanted to share a brief update before the discussion period closes.
Reviewer Lj2J, who raised several concerns overlapping with yours, particularly regarding the validity and interpretation of stability-reasoning tasks, the reliability of simulation-derived labels, candidate-selection criteria, expert evaluation protocols, and the interpretation of below-chance performance, has confirmed that our responses and revisions addressed their concerns and has raised their score.
We hope our additional experiments and clarifications will also helped address your concerns. If you have any remaining questions or suggestions, we would sincerely welcome further discussion before the deadline.
Thank you again for your time and constructive feedback.
Response to Reviewer FT5C
We sincerely thank the reviewer for their careful reading and constructive feedback. We appreciate their recognition of our benchmark's positioning, simulation-derived labels, and the findings on stability-reasoning biases. We address each concern below.
Weakness 1:
(a) DFT validation. To assess label reliability, we conducted additional DFT single-point energy calculations on 40% of the CIF-text questions from each of the four stability-reasoning tasks (M4, Q2, Q5, and Q10). For each question, we computed the DFT energies of all four candidate structures and compared the lowest-energy candidate with the MACE-OFF23-small ground truth. The agreement rates are 95.0% (M4), 82.5% (Q2), 100% (Q5), and 92.0% (Q10), providing strong evidence that the majority of our stability labels are consistent with DFT-level energy rankings. We added this DFT validation to the supplement.
(b) Thermodynamic stability. We agree that potential energy does not fully capture thermodynamic stability, as entropy and free-energy contributions are not considered. More broadly, computational simulations do not necessarily reproduce real chemical processes in full. Therefore, our benchmark deliberately focuses on the simulated structural relaxation following molecular interventions, rather than claiming to reproduce real chemical reactions or thermodynamic behavior. We have revised the manuscript to clarify this distinction, explicitly defining stability reasoning in terms of relative potential energy and distinguishing geometry optimization from physical reaction dynamics in Sections 3.1 and 3.2.
(c) Candidate selection thresholds. For stability reasoning, we exclude the first 25% of BFGS optimization steps and require four candidates with pairwise potential-energy differences of at least 0.02 eV and aligned RMSD differences of at least 0.05 Å. BFGS uses a force convergence threshold of 0.03 eV/Å, with up to 200 optimization steps for stability-reasoning trajectories. These criteria ensure that the selected candidates are geometrically and energetically distinguishable. We have added these details to Sections 3.1 and 3.2 in the revised manuscript.
Weakness 2:
We thank the reviewer for pointing out this missing detail. The expert evaluation was conducted by three PhD researchers with backgrounds in physical chemistry. Each independently evaluated 610 questions (10% of the benchmark), covering 1,830 non-overlapping questions (30%) in total. Questions were sampled across task categories, and the experts were allowed to inspect molecular structures using the Jmol 3D visualization tool. The reported 87.3% accuracy is the aggregate expert performance across these evaluations. We have clarified the expert qualifications, sampling procedure, and evaluation protocol in Section 4.1 of the revised manuscript.
Weakness 3:
We clarify that all stability-reasoning tasks are four-option single-choice questions, yielding a uniform random-guessing baseline of 25%. Other tasks include different answer formats, so a unified random baseline is not applicable to the entire benchmark. Several models achieve substantially lower accuracy on stability reasoning (e.g., 1.8% for Gemma 4-31B and 4.4% for DeepSeek-V4.1-Flash in the CIF-text setting) compared to 25%.
To investigate why several models perform substantially below chance on stability reasoning, we further examined representative failure cases. For example, on an M4 molecular-assembly question, InternVL3.5 selected the highest-energy candidate after explicitly reasoning that the most stable structure should be the one closest to the immediately edited state. This illustrates how reliance on pre-relaxation geometric similarity can lead to incorrect stability judgments, consistent with the broader preference for earlier relaxation states observed in Table 5(b). The detailed information is added to the appendix in the revised version.