acceptodds
Under review as a conference paper at ICLR 2027

MolStruct: A Benchmark for Multi-Scale Molecular Structure Understanding, Reasoning, and Prediction

Abstract

Foundation models have shown growing capabilities in molecular understanding and structure editing, yet their ability to reason about how molecular interventions shape dynamic behavior remains largely unexplored. In this work, we introduce MolStruct, a benchmark for molecular structural reasoning built around an intervention-relaxation pipeline. MolStruct covers molecular structure at three levels of granularity: atoms and bonds, functional groups, and whole molecules and evaluates four capabilities: structural perception, inverse change reasoning, stability reasoning, and structural prediction, in both text-based and multimodal settings. Experiments across multiple open-source and proprietary models show that current models perform reasonably well on perception, but struggle substantially with inverse reasoning, stability reasoning and predicting post-intervention relaxed outcomes. The results indicate that models are relatively stronger at recognizing molecular structures than at judging their stability or predicting how they relax. We further find that increasing model scale or reasoning depth provides only limited improvement, highlighting molecular stability reasoning as an important unresolved challenge for current foundation models.

Primary area:
datasets and benchmarks
Keywords:
molecular reasoning, large language models, multimodal large language models, scientific benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_FT5COpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MOLSTRUCT, a benchmark for molecular structural reasoning organized around an intervention–relaxation pipeline. Starting from stratified QM9 samples, the authors apply five intervention types (molecular coupling, atom substitution, bond rotation, functional-group editing, thermal perturbation), generate relaxed trajectories via MACE-OFF23 + BFGS optimization or ReaxFF/LAMMPS reactive MD, and programmatically derive ground truth from geometry/energy/bond-order records (Eq. 3). The benchmark spans three structural granularities (atoms & bonds, functional groups, whole molecules) and four capability axes (structural perception, inverse change reasoning, stability reasoning, structural prediction), totaling 6,100 questions with paired CIF-text and rendered-image modalities. Main findings: models perform reasonably on perception but struggle with inverse/stability reasoning and post-relaxation prediction; most models exhibit a systematic bias toward selecting the unrelaxed (least stable) structure in stability tasks.

Critical strengths
  1. Well-motivated and clearly differentiated positioning. Unlike editing benchmarks such as MolLangBench and AtomWorld where the edit itself is the endpoint, this work treats the post-intervention relaxed outcome as the evaluation target — a genuinely under-explored capability axis. The coverage comparison in Table 1 is persuasive.
  2. Physics-simulation-derived labels rather than LLM-generated ones. Ground truth is programmatically exported from simulation records, reducing label noise; the AI Use Statement is also properly disclosed.
  3. Clean task matrix. 3 scales × 4 capabilities with a natural difficulty progression; CIF/image dual versions of the same questions share ground truth, enabling controlled modality comparisons.
  4. The similarity-bias finding is the most informative result. Models tend to select candidates most similar to the pre-relaxation structure (i.e., the least stable one), pushing accuracy below random guessing; the observation that image modality can score higher precisely because weaker visual similarity judgment attenuates this bias is a nice mechanistic insight.
  5. Useful failure taxonomy for SMILES reconstruction (syntax errors → formula errors → isomer confusion).
Critical weaknesses
  1. Reliability of stability labels is not validated (most critical). Stability labels are the arg min of single-point potential energies from MACE-OFF23-small: (a) no DFT-level spot-check of the small MLFF's accuracy on these tasks; (b) lowest potential energy ≠ thermodynamic stability (entropy/free energy ignored, and BFGS endpoints depend on initial points); (c) "retain only geometrically and energetically distinguishable states" is stated without any thresholds. If labels are noisy, the central claim that "models fail at stability reasoning" does not hold.
  2. The "Expert" row in Table 4 (87.3) is entirely undocumented — who the expert is, which questions, what protocol, how many samples. This is a significant reporting gap, and the large expert–model gap underpins the paper's central narrative.
  3. No random baselines or statistical significance. Most single-choice accuracies fall in the 10–30% range, but without per-task chance levels (candidate counts vary) these numbers are uninterpretable; the text claims "below random guessing" without ever reporting the random baseline. Per-task sizes are small (200–1000) with no confidence intervals.
Questions for authors

see Weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer FT5C

We sincerely thank the reviewer for their careful reading and constructive feedback. We appreciate their recognition of our benchmark's positioning, simulation-derived labels, and the findings on stability-reasoning biases. We address each concern below.

Weakness 1:

(a) DFT validation. To assess label reliability, we conducted additional DFT single-point energy calculations on 40% of the CIF-text questions from each of the four stability-reasoning tasks (M4, Q2, Q5, and Q10). For each question, we computed the DFT energies of all four candidate structures and compared the lowest-energy candidate with the MACE-OFF23-small ground truth. The agreement rates are 95.0% (M4), 82.5% (Q2), 100% (Q5), and 92.0% (Q10), providing strong evidence that the majority of our stability labels are consistent with DFT-level energy rankings. We added this DFT validation to the supplement.

(b) Thermodynamic stability. We agree that potential energy does not fully capture thermodynamic stability, as entropy and free-energy contributions are not considered. More broadly, computational simulations do not necessarily reproduce real chemical processes in full. Therefore, our benchmark deliberately focuses on the simulated structural relaxation following molecular interventions, rather than claiming to reproduce real chemical reactions or thermodynamic behavior. We have revised the manuscript to clarify this distinction, explicitly defining stability reasoning in terms of relative potential energy and distinguishing geometry optimization from physical reaction dynamics in Sections 3.1 and 3.2.

(c) Candidate selection thresholds. For stability reasoning, we exclude the first 25% of BFGS optimization steps and require four candidates with pairwise potential-energy differences of at least 0.02 eV and aligned RMSD differences of at least 0.05 Å. BFGS uses a force convergence threshold of 0.03 eV/Å, with up to 200 optimization steps for stability-reasoning trajectories. These criteria ensure that the selected candidates are geometrically and energetically distinguishable. We have added these details to Sections 3.1 and 3.2 in the revised manuscript.


Weakness 2:

We thank the reviewer for pointing out this missing detail. The expert evaluation was conducted by three PhD researchers with backgrounds in physical chemistry. Each independently evaluated 610 questions (10% of the benchmark), covering 1,830 non-overlapping questions (30%) in total. Questions were sampled across task categories, and the experts were allowed to inspect molecular structures using the Jmol 3D visualization tool. The reported 87.3% accuracy is the aggregate expert performance across these evaluations. We have clarified the expert qualifications, sampling procedure, and evaluation protocol in Section 4.1 of the revised manuscript.


Weakness 3:

We clarify that all stability-reasoning tasks are four-option single-choice questions, yielding a uniform random-guessing baseline of 25%. Other tasks include different answer formats, so a unified random baseline is not applicable to the entire benchmark. Several models achieve substantially lower accuracy on stability reasoning (e.g., 1.8% for Gemma 4-31B and 4.4% for DeepSeek-V4.1-Flash in the CIF-text setting) compared to 25%.

To investigate why several models perform substantially below chance on stability reasoning, we further examined representative failure cases. For example, on an M4 molecular-assembly question, InternVL3.5 selected the highest-energy candidate after explicitly reasoning that the most stable structure should be the one closest to the immediately edited state. This illustrates how reliance on pre-relaxation geometric similarity can lead to incorrect stability judgments, consistent with the broader preference for earlier relaxation states observed in Table 5(b). The detailed information is added to the appendix in the revised version.

AuthorsOpenReview ↗

Official Comment by Authors

We sincerely thank the reviewers for their valuable review. We have substantially revised the manuscript and hope that these revisions adequately address their concerns. We have uploaded an updated version of the paper, with the revised text marked in blue for easier review.

As the Author–Reviewer Discussion period is approaching its end, we welcome any further questions or concerns. We greatly appreciate the reviewers' time and effort in evaluating our work.

AuthorsOpenReview ↗

A Brief Update on the Discussion

Dear Reviewer FT5C,

Apologies for the additional follow-up. We wanted to share a brief update before the discussion period closes.

Reviewer Lj2J, who raised several concerns overlapping with yours, particularly regarding the validity and interpretation of stability-reasoning tasks, the reliability of simulation-derived labels, candidate-selection criteria, expert evaluation protocols, and the interpretation of below-chance performance, has confirmed that our responses and revisions addressed their concerns and has raised their score.

We hope our additional experiments and clarifications will also helped address your concerns. If you have any remaining questions or suggestions, we would sincerely welcome further discussion before the deadline.

Thank you again for your time and constructive feedback.

Reviewer_Lj2JOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper introduces MolStruct, a benchmark for molecular structural reasoning built around an intervention–relaxation pipeline. Starting from QM9 molecules, it applies five classes of controlled interventions followed by geometry optimization or reactive molecular dynamics. The resulting benchmark contains 6,100 examples across 16 tasks spanning molecular, atom-and-bond, and functional-group scales. It evaluates structural perception, inverse change reasoning, stability reasoning, and structural prediction using CIF text or rendered molecular images. The authors evaluate eight foundation models, report substantial differences across models and representations, and identify a tendency for some models to favor candidates resembling the immediately edited structure in stability judgments. The primary contributions are the benchmark and an empirical analysis of current models’ limitations.

Critical strengths
  1. The intervention–relaxation framing addresses a meaningful evaluation gap. The benchmark focuses on the structural consequences of an edit rather than only molecular recognition or execution of a prescribed modification. Its separation into perception, inverse reasoning, stability assessment, and outcome prediction provides a systematic framework for examining whether models connect molecular structure with subsequent responses. This is the paper’s main contribution.
  2. Simulation-derived labels and paired modalities support controlled evaluation. Labels are derived from geometric differences, computed potential energies, and simulation records rather than subjective judgments from another LLM. For tasks available in both CIF and image formats, the underlying examples and answers are shared, enabling relatively direct comparisons of representation effects. This provides a useful foundation for future evaluation, conditional on adequate validation and disclosure of the simulation and sampling protocols.
Critical weaknesses
  1. The stability tasks and their central interpretation are insufficiently validated. Stability candidates are sampled from different steps of a single BFGS optimization trajectory. The task therefore primarily measures potential-energy ranking among optimization intermediates, which is not directly equivalent to comparing independently converged conformers. The paper does not sufficiently establish how this construction separates the intended molecular stability reasoning from generic geometric-quality or trajectory-related cues. Candidate separation thresholds, convergence criteria, and label robustness are also insufficiently reported. Furthermore, candidate selection frequencies alone do not establish the authors’ explanation that low accuracy results from a preference for similarity to the initial structure. The actual input is particularly unclear: Section 3.2 describes candidate relaxed states, whereas Section 4.3 refers to an initial structure supplied in the prompt. These issues directly affect the scientific interpretation of the benchmark results.
  2. Reporting and experimental transparency are insufficient to substantiate the main conclusions. The aggregation underlying Table 4 is unclear. For example, the simple average of GPT-5.5’s seven displayed scores is approximately 30.46%, whereas the reported Avg. is 26.7%; some other rows do agree with the simple average. This does not by itself establish a calculation error, but a consistent aggregation formula and task-level results are needed. The Expert score of 87.3% is also reported without an accompanying protocol specifying participants, evaluated samples, modalities, or tool access. Moreover, complete prompts, actual image-input examples, and task-specific chance baselines are missing, making it difficult to distinguish scientific reasoning failures from visual-input limitations and output-format failures. The abstract’s claim that increasing model scale or reasoning depth yields limited improvement is not supported by clearly presented controlled comparisons. Because the benchmark and its empirical conclusions constitute the central contribution, these shortcomings are consequential.
Questions for authors
  1. What exactly is provided in stability-reasoning prompts? Is the initial or immediately edited structure included in addition to the candidates? Please provide a complete prompt and corresponding image example, specify how similarity in Table 5(b) is computed and which modalities it covers, and clarify whether the effect of initial-structure information or similarity cues on candidate preferences has been tested.
  2. Candidate selection and label reliability in stability reasoning. The paper states that only geometrically and energetically distinguishable candidate states are retained. Please specify the geometric and energy separation thresholds used for candidate selection, as well as the BFGS convergence criteria. Were any label-robustness checks performed to ensure that the energy differences between candidates support a reliable ranking?
  3. How are the Avg. and Expert results in Table 4 calculated? Please provide a consistent aggregation formula and task-by-modality scores, and explain the GPT-5.5 aggregate. Please also specify the number and expertise of human participants, evaluated samples, input modalities, and permitted tools.
  4. Which experiments support the claim that increasing model scale or reasoning depth provides only limited gains? Were comparisons conducted within the same model family across scales, or for the same model under different reasoning budgets? If not, would the authors narrow this claim to the observations directly supported by the reported experiments?
5 comments
AuthorsOpenReview ↗

Response to Reviewer Lj2J

We sincerely thank the reviewer for their careful evaluation and constructive feedback. We appreciate their recognition of the value of our intervention–relaxation framework and the controlled evaluation enabled by simulation-derived labels and paired modalities. We also acknowledge that the original submission lacked sufficient clarity and rigor in several aspects of its methodology and experimental reporting. We have taken these concerns seriously and made substantial revisions. We address the reviewer's concerns in detail below.

W1-1 Stability:

We agree that ranking intermediate states along a BFGS trajectory is not equivalent to comparing independently converged conformers. Our benchmark specifically focuses on whether models can reason about structural changes following molecular interventions, rather than accurately predicting real chemical reactions or physical dynamics. Accordingly, we carefully frame this process as structural relaxation and optimization rather than a chemical reaction. We have clarified these criteria in Sections 3.1 and 3.2 of the revised manuscript.

W1-2&Q2-1 Candidate separation thresholds and convergence criteria:

Regarding candidate construction, we use MACE-OFF23-small with BFGS optimization, a maximum atomic force convergence threshold of 0.03 eV/Å, and a maximum step displacement of 0.04 Å. For stability reasoning, we exclude the first 25% of optimization steps and select four intermediate structures satisfying pairwise energy differences of at least 0.02 eV and aligned RMSD differences of at least 0.05 Å. These criteria ensure that the candidates are geometrically and energetically distinguishable. We have added these details to Sections 3.1 and 3.2 in the revised manuscript.

W1-3&Q2-2 Label robustness:

To further validate label reliability, we performed DFT single-point energy calculations on 40% of the CIF-text questions from each stability-reasoning task, evaluating all four candidates per question. The agreement between the lowest-energy candidates identified by DFT and MACE-OFF23-small reaches 95.0% (M4), 82.5% (Q2), 100% (Q5), and 92.0% (Q10), respectively. These results support the reliability of most energy-ranking labels. We added DFT validation in the supplement of the revised manuscript.

W1-4&Q1-3 Preference for similarity to the initial structure:

All stability-reasoning tasks contain four options, randomly shuffled with a fixed seed, resulting in approximately uniform ground-truth positions (25% each). Table 5(b) reports the frequency of selecting the earliest candidate along the BFGS relaxation trajectory, rather than explicitly computing structural similarity. The original results cover the CIF-text modality only.

We further evaluated this bias in the image modality and observed substantial reductions: from 64.8% to 27.6% for Gemma 4-12B, 74.9% to 42.4% for Gemma 4-31B, 77.6% to 42.0% for DeepSeek-V4.1-Flash, 67.1% to 23.5% for Claude Opus 4.7, and 34.7% to 26.2% for Gemini 3.7 Flash. We refined the formulation of the "earliest frame" and added a discussion regarding the image results in Sec. 4.3.

We also appreciate the reviewer's suggestion that providing the immediately edited structure might reinforce this bias. To investigate this, we removed the immediately edited structure from the prompt and reevaluated DeepSeek-V4.1-Flash, which exhibited the strongest original bias. Its preference remained high at 72.7%, compared with 77.6% originally, suggesting that the bias cannot be primarily attributed to the presence of this reference structure alone.

To further investigate this bias, we examined representative model reasoning traces. In an M4 example presented in the Appendix, InternVL3.5 selected the highest-energy candidate, reasoning that "the most stable would be the one that is closest to the ideal, as any deviation would increase energy." This illustrates the model's difficulty in assessing relative potential energy from molecular structures, despite recognizing the importance of energy minimization.

ALL this evidence together suggest that models struggle to assess relative stability from molecular structures and instead rely on unreliable geometric heuristics, contributing to their below-chance performance.

W1-5&Q1-1 Actual input:

Sections 3.2 and 4.3 are not contradictory. Both the original and immediately edited structures are provided in the prompt, while the four candidates are sampled exclusively from subsequent relaxation steps. This design intentionally focuses on reasoning about post-intervention relaxation rather than predicting the immediate edited structure, as explored in prior benchmarks. We have clarified this in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer Lj2J (2)

W2-1&Q3-1 GPT-5.5:

We apologize for this error. During the initial development of the benchmark, we constructed a smaller version of the dataset and evaluated GPT-5.5 as a baseline to assess task difficulty. After expanding the benchmark to its current scale, we updated the individual performance metrics for GPT-5.5 but inadvertently retained its overall accuracy from the earlier version. We have corrected the overall accuracy and averaging error in the revised version. We have rechecked GPT-5.5's results on the complete benchmark and confirmed that all other reported performance metrics are correct. We have also further confirmed that these corrections do not alter the main findings or conclusions of our study.

W2-2&Q3-2 Experts' results:

We have also added the description of experts' results in Sec. 4.1 in the revised version. Three PhD researchers with backgrounds in physical chemistry each independently evaluated 610 questions (10% of the benchmark), covering 1,830 questions (30%) in total. The questions were sampled across task categories, and evaluators were allowed to inspect molecular structures using the Jmol 3D visualization tool. The resulting expert accuracy was 87.3%.

W2-3&Q1-2 Prompts and actual image-input examples:

To improve the transparency of our evaluation protocol, we have added complete evaluation examples for four representative tasks (Q1, M3, Q10, and Q6) in the Appendix, covering all four reasoning capabilities. Each example includes the full text prompt and the corresponding molecular images provided to the evaluated models. In particular, the Q10 example explicitly demonstrates that both the initial and immediately edited structures are supplied as references, while the four candidate structures are sampled from subsequent relaxation steps. These additions clarify the exact inputs and output requirements of our evaluation.

W2-4&Q4-1 Model scale:

We acknowledge that the original cross-model comparisons were not sufficiently controlled to support our claim regarding model scaling. To address this limitation, we have conducted additional experiments with different model sizes within the same families, namely Gemma 4 (12B and 31B) and Qwen3 (8B and 32B). The results show that increasing model size does not consistently improve performance. We have added a dedicated discussion of these findings in Section 4.3 of the revised manuscript.

W2-5&Q4-2 Reasoning depth:

To investigate the effect of reasoning effort, we selected Gemini 3.7 Flash, the best-performing model in our original evaluation (37.8% overall accuracy), and reevaluated it under the maximum reasoning-effort setting. Despite increased reasoning effort, its overall accuracy decreased from 37.8% to 33.6%. The effects also varied across tasks and modalities: stability-reasoning accuracy decreased from 44.5% to 30.3% with CIF-text inputs, while increasing from 25.1% to 30.3% with image inputs. These results suggest that increasing reasoning effort does not consistently improve performance, even for our strongest evaluated model. We have added this controlled comparison and the corresponding discussion to Section 4.3 of the revised manuscript.

AuthorsOpenReview ↗

Official Comment by Authors

We sincerely thank the reviewers for their valuable review. We have substantially revised the manuscript and hope that these revisions adequately address their concerns. We have uploaded an updated version of the paper, with the revised text marked in blue for easier review.

As the Author–Reviewer Discussion period is approaching its end, we welcome any further questions or concerns. We greatly appreciate the reviewers' time and effort in evaluating our work.

Reviewer_qtd8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MolStruct evaluates whether foundation models can reason about the structural consequences of molecular edits. It contains 6,100 examples across 16 tasks, covering structural perception, recovery of hidden edits, stability ranking and prediction of post-intervention outcomes. The data come from edited QM9 molecules followed by MACE-OFF23 geometry optimization or ReaxFF simulations. Reasoning and prediction tasks have paired CIF-text and image versions. Eight models are evaluated, with large differences between model families and frequent failures on stability tasks. The paper also reports a tendency to select candidates resembling the initial edited structure.

Critical strengths
  1. Evaluating what happens after a molecular edit is a useful direction. The distinction between identifying an edit and predicting its relaxed outcome gives this benchmark a concrete purpose beyond structure recognition. The intervention records also make it possible to generate targets without asking an LLM to supply the answers.
  2. Using the same underlying examples and labels for CIF-text and image evaluation is a good design choice. It supports comparisons of how representations affect performance.
Critical weaknesses
  1. I have questions about that the stability labels have been validated for the claims made here. In Section 3.2, candidates are intermediate states from one BFGS trajectory. It does not establish performance on separately converged conformers or physical dynamics.

  2. As for the edited structures. It describes substitutions such as C to N while preserving connectivity, but do not explain how hydrogens, valence and charge are handled.

  3. The evaluation does not separate molecular reasoning failures from input and response-format difficulties. CIF inputs omit connectivity, and Section 4.3 reports 82.0% unparsable responses for InternVL3.5 on Q9. The paper reports no candidate-order controlto support the proposed explanation. The abstract also claims limited gains from model scale or reasoning depth without a controlled scaling or reasoning-budget comparison.

  4. I also cannot reconcile the GPT-5.5 averages. The seven entries in Table 4 average to 30.46%, while the reported average is 26.7%. The four entries in Table 5(b) average to 50.78%, rather than 40.8%. Most other rows appear to use equal weighting. If a different rule was used, it needs to be stated. Task-level results and a description of the unexplained 87.3% Expert result would make the model comparisons much easier to assess.

Questions for authors
  1. What energy-gap and convergence thresholds were used?
  2. Does the early-stage preference persist after randomizing candidate order?
  3. What averaging rule produces the GPT-5.5 values in Tables 4 and 5(b)?
  4. Which controlled experiment supports the claim about limited benefits from scaling or deeper reasoning?
6 comments
AuthorsOpenReview ↗

Response to Reviewer qtd8

We thank the reviewer for their careful reading and constructive feedback. We appreciate their recognition of the value of studying post-intervention molecular structures and comparing CIF-text and image representations. We have addressed the concerns raised in the review through additional experiments, further clarification of our methodology, and corrections to the reported results. We respond to each point below.

Weakness 1:

We agree that ranking intermediate states along a single BFGS trajectory is not equivalent to comparing independently converged conformers or modeling physical dynamics. Our benchmark specifically focuses on whether models can reason about structural changes following molecular interventions, rather than accurately predicting real chemical reactions or thermodynamic stability. Therefore, we addressed this process as structural relaxation and optimization rather than a chemical reaction. We have revised the manuscript to clarify this distinction, explicitly defining stability reasoning in terms of relative potential energy and distinguishing geometry optimization from physical reaction dynamics in Sections 3.1 and 3.2.

To ensure that stability-reasoning questions involve meaningful distinctions between candidate structures rather than negligible geometric or energy differences, we apply explicit criteria during candidate selection. Specifically, we use MACE-OFF23-small with BFGS optimization, a force convergence threshold of 0.03 eV/Å, and a maximum step displacement of 0.04 Å. For stability reasoning, we exclude the first 25% of optimization steps and sample four intermediate structures with pairwise potential-energy differences of at least 0.02 eV and aligned RMSD differences of at least 0.05 Å. Each trajectory contains up to 200 optimization steps. These criteria ensure that the candidates are sufficiently distinguishable for evaluating relative potential-energy reasoning. We have clarified these criteria in Sections 3.1 and 3.2 of the revised manuscript.

To further assess label reliability, we conducted DFT single-point energy calculations on 40% of the CIF-text questions from each stability-reasoning task, evaluating all four candidates per question. The agreement between DFT and MACE-OFF23-small energy rankings in identifying the lowest-energy candidate reaches 95.0% (M4), 82.5% (Q2), 100% (Q5), and 92.0% (Q10). These results support the reliability of the majority of our energy-ranking labels. We have added the construction criteria and validation results to the revised manuscript.


Weakness 2:

To ensure that atom substitutions produce chemically valid structures rather than arbitrary element replacements, we apply explicit constraints on valence, hydrogen counts, and formal charges. Specifically, atom substitution is restricted to non-aromatic molecules with zero total formal charge, and the substituted atom must also have zero formal charge. We preserve heavy-atom connectivity and bond orders while adjusting the number of attached hydrogens according to the target element's standard neutral valence. Substitutions requiring invalid hydrogen counts are rejected, and all resulting structures undergo RDKit sanitization. These constraints prevent invalid valence assignments and ensure that the edited molecular graphs satisfy the specified chemical validity checks. We have clarified this procedure in Section 3.1.

AuthorsOpenReview ↗

Response to Reviewer qtd8 (2)

Weakness 3:

Cif input: We acknowledge that inferring connectivity from atomic coordinates introduces an additional challenge. This is an intentional aspect of our coordinate-based structural reasoning setting, rather than an attempt to isolate chemical reasoning from all representation-related difficulties. Investigating this is not the primary focus of our work; furthermore, similar studies—such as AtomWorld (ICML 2026) and LLM4Mat-Bench (2025)—also utilize CIF files as input.

Q9 results: We manually examined the 82.0% unparsable Q9 responses from InternVL3.5 and confirmed that these errors could not be reliably recovered through more permissive parsing. The prompt explicitly specifies the required JSON fields (attachment_atom_index, removed_group_smiles, and added_group_smiles) and provides an example. Since strict formatting is not the focus of our benchmark, our parser accepts alternative field names (e.g., attachment_site, attachment, attachment_index, or site for attachment_atom_index). Despite this flexibility, 31.5% of responses omit required fields, making the intended edits incomplete. As in other SMILES-based tasks, we compare parsed molecular structures rather than requiring exact string matches. However, 38.5% contain invalid SMILES that cannot be parsed into molecular structures, while 9.0% omit the required * marker indicating the attachment site. The remaining 3.0% output an integer array instead of the required nested JSON structure, making their intended meaning ambiguous. We add these detailed descriptions to the revised version.

Candidate-order bias: All stability-reasoning tasks contain four options, which were randomly shuffled using a fixed seed during benchmark construction. To examine whether the observed preference for earlier relaxation states could be attributed to candidate ordering, we analyzed both the correct-answer positions and the earliest-candidate positions across the 1,100 stability-reasoning questions.The correct-answer distribution across A–D is 23.4%, 23.2%, 28.4%, and 25.1%, respectively, while the earliest-candidate distribution is 26.8%, 24.0%, 24.5%, and 24.7%. These results suggest that the observed preference for earlier relaxation states is unlikely to arise simply from their systematic placement at particular option positions. We have added the detailed task-level distributions to the Appendix and clarified the candidate-order randomization procedure in Section 4.1.

Impact of model scale: We acknowledge that the original cross-model comparisons were not sufficiently controlled to support our claim regarding model scaling. To address this limitation, we have conducted additional experiments with different model sizes within the same families, namely Gemma 4 (12B and 31B) and Qwen3 (8B and 32B). The results show that increasing model size does not consistently improve performance. We have added a dedicated discussion of these findings in Section 4.3 of the revised manuscript.

Impact of model depth: To investigate the effect of reasoning effort, we selected Gemini 3.7 Flash, the best-performing model in our original evaluation, and reevaluated it under the maximum reasoning-effort setting. Despite increased reasoning effort, its overall accuracy decreased from 37.8% to 33.6%. The effects also varied across tasks and modalities: stability-reasoning accuracy decreased from 44.5% to 30.3% with CIF-text inputs, while increasing from 25.1% to 30.3% with image inputs. These results suggest that increasing reasoning effort does not consistently improve performance, even for our strongest evaluated model. We have added this controlled comparison and the corresponding discussion to Section 4.3 of the revised manuscript.


Weakness 4:

We apologize for this error and any confusion it may have caused. During the initial development of the benchmark, we constructed a smaller version of the dataset and evaluated GPT-5.5 as a baseline to assess task difficulty. After expanding the benchmark to its current scale, we updated the individual performance metrics for GPT-5.5 but inadvertently retained its overall accuracy from the earlier version. We have corrected the overall accuracy and averaging error in the revised paper in Table 4 and Table 5 (B). We have rechecked and confirmed that all other reported performance metrics are correct. We have also added the description of experts' results in Sec. 4.1 in the revised version. Three PhD researchers with backgrounds in physical chemistry each independently evaluated 610 questions (10% of the benchmark), covering 1,830 questions (30%) in total. The questions were sampled across task categories, and evaluators were allowed to inspect molecular structures using the Jmol 3D visualization tool. The resulting expert accuracy was 87.3%.

AuthorsOpenReview ↗

Official Comment by Authors

We sincerely thank the reviewers for their valuable review. We have substantially revised the manuscript and hope that these revisions adequately address their concerns. We have uploaded an updated version of the paper, with the revised text marked in blue for easier review.

As the Author–Reviewer Discussion period is approaching its end, we welcome any further questions or concerns. We greatly appreciate the reviewers' time and effort in evaluating our work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Conclusion on the Rebuttal

We thank the reviewers for their careful evaluations and constructive feedback. We appreciate their recognition of the motivation and contributions of MolStruct, particularly the intervention–relaxation framework, simulation-derived ground truth, and systematic evaluation across molecular scales and modalities. We believe MolStruct addresses an important gap in existing molecular benchmarks by evaluating whether foundation models can reason about structural consequences beyond immediate molecular edits, and we hope it will serve as a valuable resource for the research community.

The reviewers generally acknowledged the value of our contributions, and we carefully addressed the reviewers' concerns about additional experiments through detailed point-by-point responses and substantial revisions to the manuscript. Our main updates include:

  1. Stability-label validation: Conducted DFT calculations to further assess the reliability of our stability-reasoning labels.
  2. Dataset construction and evaluation: Clarified candidate-selection criteria, structural relaxation procedures, evaluation protocols, and expert evaluation details.
  3. Stability-reasoning analysis: Added candidate-order distribution analysis, image-modality comparisons, and qualitative case studies to better understand model biases.
  4. Additional experiments: Conducted controlled comparisons across model sizes and reasoning settings, and clarified random baselines and reported results.

The three reviewers raised several closely related concerns. After our detailed rebuttal and revisions, two reviewers (Lj2J and qtd8) acknowledged our responses and raised their ratings to weak acceptance. We have not received any follow-up feedback from the remaining reviewer FT5C, and their concerns substantially overlap with those raised by the other two reviewers. We therefore hope that our additional experiments and revisions have addressed their concerns as well.

Finally, we would like to express our sincere gratitude to the Program Chairs, Senior Area Chairs, Area Chairs, and all reviewers for their time and effort throughout the review process. Their constructive feedback has helped us improve the reliability, clarity, and overall quality of our work. We greatly appreciate their consideration of our revised submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.