acceptodds
Under review as a conference paper at ICLR 2027

MolP2SBench: Do LLMs Really Reason from Molecular Properties to Structure?

Abstract

Large language models (LLMs) are increasingly used in molecular discovery and structure elucidation, where decisions depend on physical observations. Existing evaluations primarily focus on end-to-end molecular reconstruction, where correct predictions do not necessarily indicate that these results are grounded in the supplied properties. To address this, we introduce MolP2SBench, a benchmark for evaluating molecular property-to-structure (P2S) reasoning in LLMs. Each task presents a masked molecular scaffold with given properties, requiring the model to select the missing structure from plausible candidates. To assess whether predictions genuinely rely on the provided properties, we test how models respond when properties are shuffled or replaced, thereby evaluating their reliance on correct property-structure pairing and their responsiveness to counterfactual property changes. We also establish a channel-wise leaderboard using Property-Grounded Recovery (PGR), which gives equal weight to source and counterfactual recovery. In our evaluation of seven models, we find that LLMs can leverage physical properties for molecular structure prediction, but their performance varies substantially across property types. More importantly, higher accuracy with a given property does not necessarily imply greater responsiveness to that property. For example, a model may benefit more from H NMR than from C NMR, yet be less likely to change its prediction when the H data is replaced. Interestingly, LLM performance does not necessarily reflect the relative difficulty of these properties for human chemists. For instance, although UV–Vis spectra provide less direct structural information than NMR spectra, Astra achieves higher accuracy with UV–Vis data than with NMR peaks. In summary, MolP2SBench provides a framework for studying how LLMs extract and use information from different physical properties, revealing performance patterns that may differ from expectations based on conventional chemical intuition and offering a foundation for developing more reliably property-grounded models for molecular and quantum-chemical inference.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, AI for Science, Molecular Property-to-Structure Prediction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_cnMsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MolP2SBench evaluates property-conditioned local chemical-graph completion. Given a scaffold and a small set of candidate molecules represented as SMILES, models select the candidate matching supplied physical properties. Experiments across seven models use property removal, shuffling, and within-family substitutions to assess behavioral reliance and responsiveness.

Critical strengths
  1. Holding the scaffold and candidates fixed while substituting properties provides an interpretable test of whether candidate selections respond to changing evidence. This offers useful diagnostic information beyond answer accuracy alone.
  2. The evaluation covers multiple models and property representations, supported by documented molecular alignment, unconditional counterfactual scoring, and sensitivity analyses. These components provide a useful foundation for controlled comparisons.
Critical weaknesses
  1. The task supplies most of the chemical connectivity and leaves limited inference to the model. Only 1-3 heavy atoms are hidden, while the scaffold, attachment constraints, boundary bond orders, and 2-4 complete candidates are provided. Performance therefore measures highly constrained local discrimination, with insufficient evidence of capabilities beyond familiar pattern matching. Although “chemical structure” is valid terminology, the manuscript should consistently specify SMILES-based chemical-graph completion and distinguish it from three-dimensional geometry or conformational reconstruction. The limited task complexity and uncertain broader relevance weaken the significance of the contribution.
  2. The diagnostic interpretations require stronger calibration. PGR can reward property-invariant predictions, but actual no-property predictions are not evaluated under the same scoring rule. CFR conditions on different initially correct cohorts, and the common-cohort contrast in Appendix G.3 remains statistically uncertain. Cross-channel comparisons additionally confound modality with observation scope: NMR excludes hidden-fragment nuclei, whereas other channels describe complete molecules. These limitations weaken conclusions about distinct reasoning capabilities and relative difficulty across properties.
Questions for authors
  1. Could you explicitly define the evaluated notion of “structure” and consistently delimit the claims to local chemical-graph completion throughout the title, abstract, and conclusions?
  2. How do the findings generalize as masked fragments and candidate sets become larger, particularly on previously unseen scaffolds?
  3. How do simple chemical-rule, local-pattern, or dedicated property-prediction baselines perform under the same conditions?
  4. Could you report PGR for unchanged no-property predictions and establish whether the reliance-responsiveness contrasts persist with common cohorts, paired uncertainty estimates, and repeated inference?
  5. Do cross-channel comparisons persist under more comparable observation scopes and information content, including whole-molecule NMR?
3 comments
AuthorsOpenReview ↗

Response to cnMs

Thank you for highlighting task scope and diagnostic calibration. We address Q4 in two parts.

W1 & Q1. Task definition and scope of claims

Response. We agree that the evaluated notion of structure should be explicit. Here, chemical structure refers to molecular connectivity represented as SMILES. This terminology is consistent with related benchmarks: MolLangBench [1] evaluates molecular structure recognition and editing using SMILES representations, and MolPuzzle [2] formulates molecular structure elucidation with SMILES outputs. These examples illustrate that SMILES is a representation of molecular structure, and that the term does not require prediction of three-dimensional coordinates. MassSpecGym [3] also distinguishes de novo generation from spectrum-based retrieval among supplied candidate molecular graphs, providing a precedent for evaluating candidate-based structure identification as a distinct task.

In MolP2SBench, the specific task is property-guided masked-scaffold completion with supplied candidate structures, as specified in the abstract and task definition. Holding the scaffold and candidates fixed allows us to test whether correct property pairing improves selection and whether substituting an alternative candidate's properties redirects the answer toward that candidate. The contribution is a controlled diagnostic of property use in structural decisions, rather than evidence of unrestricted structure elucidation. Our claims concern this completion setting; they do not imply open-ended reconstruction of an entire molecule or prediction of three-dimensional geometry or conformations. The Limitation section also explicitly distinguishes this setting from open-ended structure elucidation.

W1 & Q2. Generalization to more difficult tasks

Response. MolP2SBench continues to expand its coverage within a consistent task definition. C1 reports the current 499-family evaluation, and C2 compares candidate sets within the same family. The present version uses QM9 molecules with 1–3 masked heavy atoms; larger masks and an independent unseen-scaffold split are directions for continued development.

Q3. Simple and specialized baselines

Response. The evaluation includes the measured no-property reference below to calibrate recovery from unchanged structural predictions. Chemical-rule and dedicated property-prediction baselines remain outside the present comparison. A forward predictor could rank candidates by agreement with the observation, with its training data and candidate-property access specified. This would provide a complementary reference for task difficulty.

W2 & Q4(a). PGR calibration against unchanged predictions

Response. We evaluated the requested reference for all six models on the same 499 families. Each model's no-property choice is held unchanged when scored against the source and CF-donor targets. Structural inputs, candidate order, donor assignments, and scoring match the property-conditioned evaluation; invalid outputs count as errors.

Table R4. Unchanged no-property PGR on the common 499-family cohort. Each model has one shared reference across the eleven representations because their structural inputs and paired targets coincide.

ModelUnchanged no-property PGR (%)
Qwen23.45
Gemma22.14
Luna24.45
Sol23.75
Opus 5.523.95
Astra24.65

Manuscript Table 2 places this reference alongside the PGR for each property representation, allowing direct comparison without averaging across representations. Appendix F.2 details the unchanged-prediction scoring protocol and further comparisons. These recovery comparisons complement RM and CFR's diagnostics of correct pairing and donor-directed responsiveness.

Under the single-donor PGR definition, a fixed source or fixed donor choice scores 50%, and a fixed third candidate scores zero; uniform guessing yields 100/K%. The unchanged reference and property-conditioned results use this same scoring rule.

W2 & Q4(b). CFR comparisons and statistical uncertainty

Response. We agree that a shared input cohort alone does not align CFR eligibility. We therefore examine the two RM–CFR ordering examples discussed in the manuscript using the saved results on all 499 families. For each comparison, CFR is recomputed on the intersection of families for which both representations produce a correct source answer and a valid CF answer. RM retains its standard definition on all 499 families. These are representation-specific comparisons, with no averaging across representations.

Table R5. Paired differences for the two manuscript examples. Each difference is A minus B, in percentage points, with a pointwise 95% confidence interval. The common-eligibility count applies to CFR; RM uses all 499 families.

ModelComparison (A − B)ΔRM [95% CI]Common CFR familiesΔCFR [95% CI]
Luna¹³C peaks + degeneracy − ¹H peaks only+7.01 [+0.60, +13.43]55−16.36 [−28.85, −4.26]
SolHOMO–LUMO gap − dipole magnitude+7.82 [+0.80, +14.83]98−14.29 [−26.67, −1.98]

For Luna, common-eligibility CFR is 8/55 (14.55%) for ¹³C peaks with degeneracy and 17/55 (30.91%) for ¹H peaks only. For Sol, it is 41/98 (41.84%) for the gap and 55/98 (56.12%) for dipole magnitude. Thus, the opposite RM and CFR orderings persist after matching CFR eligibility in both examples. Eligibility alignment matters: Luna's original, separately eligible CFR difference is only −1.42 points (95% CI [−10.30, +7.51]); its two eligibility sets each contain 141 families but share only 55.

For these contrasts, we use 100,000 paired family-bootstrap resamples (seed 20261115), drawing 499 families with replacement and retaining matched responses across representations and conditions. CFR eligibility denominators are recomputed in each resample. All four pointwise 95% intervals in Table R5 exclude zero. However, with approximate Bonferroni adjustment across these four contrasts (98.75% intervals), both RM intervals and Sol's common-eligibility CFR interval include zero. As an additional sensitivity check, restricting RM itself to the same 55/98 families preserves the positive differences, but their 95% intervals include zero: Luna +10.91 [−5.77, +27.45] and Sol +5.10 [−6.74, +17.00]. This latter analysis is conditioned on initial correctness and does not replace standard RM.

These examples provide supplementary evidence that higher RM need not coincide with higher CFR; they do not establish a universal ordering or distinct internal reasoning mechanisms. These are the updated contrasts reported in Appendix G.3 (Table 19), subject to the multiplicity and common-cohort sensitivity qualifications stated above. As in C1, uncertainty is conditional on saved responses and fixed donor assignments, without additional clustering for reused shuffle donors. Repeated-inference variability remains untested: resampling families does not measure variation from rerunning the same input or changing prompts.

W2 & Q5. Comparable information across property channels

Response. C3 explains the scaffold-only design and reports the matched full-molecule NMR comparison (manuscript Section 4.2, Figure 2). The results show that observation scope contributes to measured performance, so cross-channel differences should not be attributed solely to reasoning ability. The comparison does not equalize the information content of different channels; conclusions remain tied to the specified observation conditions.

Thank you again for your constructive feedback, which has helped us clarify the task scope and strengthen the calibration and interpretation of our evaluation metrics.

References

[1] MolLangBench: A Comprehensive Benchmark for Language-Prompted Molecular Structure Recognition, Editing, and Generation. ICLR 2026.

[2] Can LLMs Solve Molecule Puzzles? A Multimodal Benchmark for Molecular Structure Elucidation. NeurIPS 2024.

[3] MassSpecGym: A Benchmark for the Discovery and Identification of Molecules. NeurIPS 2024.

Reviewer_Mgp8OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work investigates LLMs’ ability to utilize molecular physical properties for molecular structure prediction. The authors argue that existing evaluations often conflate property interpretation with structure prediction, making it difficult to assess whether LLMs genuinely rely on physical evidence. To address this issue, the authors introduce MolP2SBench, which employs three controlled interventions: (1) removing the supplied physical properties, (2) replacing them with properties of the same type from another molecular family, and (3) substituting counterfactual properties from alternative candidates. These interventions aim to distinguish whether LLMs rely on physical evidence and whether their predictions respond appropriately to changes in that evidence. Experimental results across multiple LLMs demonstrate substantial variations in their ability to utilize different physical properties. Furthermore, the observed difficulty across property types does not necessarily align with the expectations of human chemistry experts.

Critical strengths
  1. Important research problem. Understanding LLMs’ ability to interpret and utilize physical evidence for molecular structure prediction is important for developing reliable molecular structure generation systems. The proposed evaluation provides an opportunity to better understand how LLMs incorporate physical properties into their predictions.
Critical weaknesses
  1. Limited insights and implications for future research. Although the reported findings are reasonable, their contributions to the broader research community and implications for future work remain limited. For example, one of the main conclusions is that “LLMs can leverage physical properties for molecular structure prediction, but their performance varies substantially across property types.” While this observation is reasonable, it is somewhat intuitive and does not sufficiently explain why LLMs perform better with certain property types than others. Without investigating the underlying reasons for these differences, the findings provide limited actionable insights for improving future models. This weakens the overall contribution of the work.
  2. Insufficient depth of experimental analysis. Most of the experimental analyses focus on reporting observed performance differences rather than investigating the underlying factors responsible for these differences. Although the paper provides several observations and case studies, it remains unclear what generalizable insights can be derived from them. As an empirical analysis paper, a deeper investigation is necessary to move beyond descriptive findings and provide meaningful explanations. The lack of such analysis limits the significance of the conclusions and the overall contribution of the work.
Questions for authors
  1. Are there any underlying factors that could explain why LLMs perform differently across molecular property types, particularly why their performance does not align with the relative difficulty perceived by human chemistry experts? Can the authors provide deeper insights into these observations beyond the reported performance differences?
  2. Could the authors clarify the inference settings used for the evaluated LLMs? Specifically, was reasoning enabled during the experiments, and if so, what reasoning effort was used? Additionally, what were the values of the inference parameters, such as temperature, top-p, and top-k?
2 comments
AuthorsOpenReview ↗

Response to Mgp8

Thank you for requesting deeper explanations of the observed performance differences.

W1 & Q1. Reasons for performance differences across properties

Response. A chemist’s heuristic expectation is that spectra provide more directly interpretable structural cues than dipole magnitude or the HOMO–LUMO gap. Our results suggest that realizing this advantage depends on both observation scope and the model’s ability to turn spectral information into consistent, candidate-specific judgments.

1. Full-molecule NMR changes, but does not eliminate, reverse orderings. We compare existing full-molecule NMR and electronic-property results on identical families, candidate orders, and source/CF targets: 85 ¹H and 52 ¹³C families with complete coverage, reporting each representation separately. This cross-property comparison uses the common subset with complete source and CF coverage for dipole, gap, and UV–Vis. Eight ¹H and five ¹³C families lack dipole and UV–Vis CF records, reducing the 93/57-family observation-scope cohorts to 85/52 families. For all four closed-weight models, each full-molecule ¹H representation has higher PGR than dipole and gap. However, Opus 5.5’s gap PGR (74.04%) exceeds full-molecule ¹³C peak-position PGR (68.27%), while adding degeneracy raises NMR PGR to 76.92%. These point estimates show that the ordering also depends on representation. C3 and manuscript Section 4.2, Figure 2 report the broader observation-scope comparison.

2. Recognizing a chemical feature need not distinguish the candidates. Among 771 NMR correct-to-incorrect pairs from the four closed-weight models, 603 reduced-set explanations (78.2%) lack sufficient candidate-specific discrimination. A plausible description of why an observation fits one candidate does not establish why it favors that candidate over another. This limits the benefit of richer spectral information when the decisive structural comparison remains unresolved.

3. Fixed observations can receive changing interpretations. In the same pairs, 275 (35.7%) exhibit explicit changes in reference estimates, assignments, or qualitative interpretations; 91 (11.8%) specifically change peak or atom assignments. For example, Opus assigns the unchanged ¹³C shielding value of 123.3 ppm to different scaffold carbons. This example shows that the same observation can receive different assignments across candidate contexts. We record this as an interpretation change, without independently establishing which assignment is correct (C4; Appendix F.1).

Together, these findings suggest why models may depart from the chemical heuristic: a scalar trend may suffice for some candidate comparisons, whereas exploiting spectral information requires consistent assignments and discrimination across multiple observations. More information therefore need not yield better model performance, and higher electronic-property scores need not indicate deeper physical understanding. The explanation statistics concern scaffold-NMR regression pairs, not a separate audit of the full-molecule runs; they support this interpretation without establishing causality. The human reference is a heuristic expectation, not a measured human–model comparison.

W1–W2 & Q1. Deeper insights and implications for future models

Response. Our candidate-set ablation and analysis of all 1,362 valid correct-to-incorrect pairs reveal three findings (C2 and C4; manuscript Section 4.6 and Appendix F.1). In every pair, the physical observations remain unchanged and the previously correct candidate remains available.

1. A correct selection can coexist with an insufficiently supported explanation. 948 initially correct explanations lack sufficient candidate-specific discrimination, invoking plausible chemical trends without adequately explaining why the observation favors the selected candidate. This issue persists in both explanations in 882 pairs, despite the answer changing from correct to incorrect. Initially correct explanations also contain 104 structural and 95 observation-related errors. These findings expose limitations that selection accuracy alone does not capture: a correct answer need not have a sufficiently supported explanation.

2. The interpretation of fixed evidence can change with the candidate context. In 459 pairs, explanations change a reference estimate, peak or atom assignment, or qualitative interpretation. For example, Opus describes the unchanged gap of 0.2687 Ha as relatively large in the original set and relatively small in the reduced set, supporting different candidates. Thus, changes extend beyond the final choice to the interpretation used to justify it, motivating consistency checks across candidate contexts.

3. Recognizing a relevant feature does not necessarily distinguish the candidates. 126 reduced-set explanations use a shared candidate feature to support a particular choice without an adequate distinguishing argument; this issue appears only after reduction in 71 pairs. For example, Astra uses a carbonyl-related UV–Vis transition to support a ketone in the original set and an aldehyde in the reduced set. Recognizing the shared carbonyl feature leaves the decisive comparison unresolved, even with fewer alternatives.

Implications. Future models should connect observations to actual candidate differences, respect observation scope, and maintain supported assignments and reference estimates across candidate-set changes. Paired evaluations complement PGR, RM, and CFR by testing these requirements. Counts describe saved explanations in the complete analyzed regression set, with overlapping categories, not overall error rates or hidden reasoning. The intervention changes candidate number, composition, and potentially positions together; one response per input does not isolate these effects or repeated-inference variability.

Q2. Reasoning and decoding settings

Response. The protocol is zero-shot with no external tools. The saved request settings are:

  • Luna, Sol, and Astra: reasoning_effort=low, with an 8,000-token output limit; temperature, top-p, and top-k are not explicitly supplied.
  • Opus 5.5: native Anthropic output_config.effort=low, with max_tokens=8000; temperature, top-p, top-k, and a separate thinking field are not explicitly supplied. The low-effort setting is distinct from explicit activation of extended thinking.
  • Qwen and Gemma: temperature 0.2, with output limits of 4,096 and 900 tokens, respectively. Requests do not set a reasoning-effort or thinking override, top-p, or top-k; omitted settings are governed by the serving configuration.

These are the evaluated request settings, not equal inference budgets or identical internal reasoning modes across models. Opus 5.5 is identified separately from the earlier Opus version in legacy analyses.

Thank you again for your constructive feedback, which has helped us strengthen the empirical analysis and clarify the implications of our findings.

Reviewer_XQisOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This is a benchmark that tests whether LLMs actually use physical-property evidence to pick a molecular structure. The question matters, and the intervention design is the best thing about the paper. But the benchmark is too small and too easy to support its conclusions.

Critical strengths

The counterfactual swap is the right idea. A model that really uses the evidence should change its answer when the evidence changes.

Critical weaknesses

The intervention design is worth building on, but 136 easy multiple-choice tasks with simulated spectra can't answer "do LLMs really infer structure from properties." Scaling up, adding harder tasks with experimental spectra, and comparing channels on a shared task set would make this a solid paper.

It's tiny. There are only 136 source tasks and 394 candidates, and each channel is evaluated on an even smaller subset. With this n, differences between channels and models of a few points are mostly noise. Yet the leaderboard ranks models by point estimates, with no CIs in Table 2.

The properties look computed, not measured. The NMR inputs are QM shieldings with degeneracy and integration added. Real spectra have noise, solvent effects and missing peaks. Models may pick up regularities of the simulation rather than chemistry, which matters for the paper's claims about "physical evidence."

The UV–Vis > NMR finding is probably a filtering artifact. Each channel keeps a different set of families, chosen by a channel-specific threshold. The "surprising" ranking may just reflect which tasks survive each filter. Comparisons across channels need a shared task set, or a check that difficulty is matched.

CFR is conditioned on the source answer being correct. That makes the cohort different for every model and channel, so CFR values can't be compared across models. Weak models get evaluated on a handful of easy tasks. Shortcuts aren't fully ruled out. Shuffled properties come from other tasks, so they can be wrong in scale or format, which makes them easy to ignore. That makes RM an easy test.

A stronger control is a plausible decoy property, for example one perturbed within the noise of the real channel. There's no analysis of position bias or of how sensitive results are to the prompt.

Reproducibility is weak. Most of the evaluated models are closed and versioned APIs. There's one prompt, no tools, and no few-shot examples. The leaderboard may not survive the next model update.

The justification analysis is thin. The paper says it separates correct selection from valid reasoning, but there's little systematic evaluation of whether the stated reasons are chemically correct.

Questions for authors

please see in above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer XQis

Thank you for recognizing the intervention design and identifying these evaluation concerns. W1–W8 below follow the eight paragraphs of your Critical Weaknesses section.

W1. Benchmark scale and task difficulty

Response. The benchmark is under continued development, with the current version comprising 499 task families evaluated across six models (C1). Each task is manually reviewed by domain experts. The present scope is local QM9 completion with 1–3 masked heavy atoms and 2–8 candidates. C2–C4 characterize candidate-set sensitivity, observation scope, and explanation quality within this benchmark.

W2. Confidence intervals and statistical support

Response. Tables R1a–R1b provide pointwise family-bootstrap 95% intervals for every model and representation, with PGR and RM point estimates matching manuscript Tables 2 and 3, respectively. These intervals are computed from the existing responses on the same 499 families. Comparisons should use the corresponding representation-specific effects; interval overlap alone is not a paired test of a difference between models.

W3. Experimental spectra and measurement conditions

Response. The benchmark uses computed properties as defined references for pairing and within-family substitution. C3 varies the scope of computed NMR observations. The findings apply to this computational setting; measured spectra with solvent effects, noise, and missing peaks remain a further validation setting.

W4. Channel selection bias and matched comparisons

Response. C1 uses the same families across channels. On this cohort, Astra's source accuracy is 50.7% for UV–Vis, 51.5% for ¹³C peaks with degeneracy, and 50.7% for atomwise ¹³C. These close point estimates replace a general UV–Vis > NMR ordering with a representation-specific comparison. C3 further shows the importance of observation scope.

W5. Comparability of conditional CFR

Response. As specified in C1, manuscript Table 3 reports conditional CFR descriptively on each model's and representation's eligible cases. cnMs W2 & Q4(b), Table R5 additionally compares two pairs of representations within models using explicitly shared CFR eligibility on the current cohort, with paired difference intervals and sensitivity analyses. These contrasts are also reported in the revised Appendix G.3 (Table 19); they do not make all model–representation CFR values directly comparable.

W5–W6. Plausible controls and sensitivity to presentation

Shuffle compatibility. Our shuffle control preserves the scaffold, mask specification, candidate structures and order, property format, units, and observation scope. Donors are matched on observable representation structure: IR/Raman and NMR peak counts, atomwise NMR list lengths, and UV–Vis transition counts and state indices remain unchanged. For integrated ¹H and degeneracy-bearing ¹³C, the entire count sequence is preserved in input order, not just its total. The three representations of each NMR nucleus use the same donor and retain consistent peak positions and counts. Expanding the donor pool preserves these constraints rather than relaxing them. We verified these input-matching constraints across all 5,489 correct–shuffle pairs (499 families × 11 representations). Thus, RM tests the benefit of correct structure–property pairing while controlling format and count differences that could otherwise reveal the shuffled condition.

Plausible decoys and noise. Within-family CF supplies the computed property of an actual alternative candidate, providing a complementary test of donor-directed responsiveness. Shuffle compatibility does not enforce numerical proximity to the original observation or equal chemical plausibility for the recipient scaffold. Perturbations calibrated to experimental noise would test a different form of robustness and have not been evaluated.

Position and prompt sensitivity. C2 examines candidate-context sensitivity with fixed physical observations, but changes candidate composition, number, and potentially positions together. It therefore does not isolate position effects. Independent candidate-order randomization and semantically equivalent prompt paraphrases remain untested.

W7. API dependence and the scope of the inference protocol

Response. The aligned comparison includes hosted and open models under a zero-shot protocol without tools. Saved inputs, configurations, raw trajectories, parsed predictions, and invalid-output records support recomputation. Appendix E records the evaluated model identifiers and request settings, including model-dependent token limits. These results describe the evaluated configurations and do not guarantee unchanged behavior from later API versions.

W8. Chemical correctness of model justifications

Response. C4 (Table R3; manuscript Section 4.6, Table 5, and Appendix F.1) reports the complete analysis of valid correct-to-wrong pairs in the smallest-subset comparison. It distinguishes factual errors, insufficient discriminating support, paired explanation changes, and choice–explanation disagreement. Errors already present in initially correct answers show why justification quality adds information beyond selection accuracy. The stated denominator confines these findings to the analyzed failure pairs.

Thank you again for your constructive comments and suggestions, which have helped us further strengthen the manuscript. We would appreciate any additional feedback and welcome continued discussion.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Common Response to All Reviewers

We thank the reviewers for their constructive feedback. MolP2SBench is an evolving benchmark whose coverage and evaluation protocols continue to expand. It currently comprises 499 task families, with each task manually reviewed by domain experts. The current evaluation covers six models and combines matched property interventions with no-property PGR calibration. Complementary analyses examine 459 nested candidate subsets from 210 families, full-molecule NMR, and 2,724 explanations from 1,362 pairs that lose a correct answer after candidate reduction.

The central findings are that higher aggregate recovery can coexist with lost correct decisions, NMR performance depends on observation scope, and explanations can lack candidate-specific support even when they invoke plausible chemical trends. We summarize the shared evidence below. The no-property comparison appears under cnMs Q4(a), Table R4. Manuscript Section 4.6 and Table 5 present the candidate-set ablation and explanation analysis; Section 4.2 and Figure 2 present full-molecule NMR. Appendix F.1 contains the detailed ablation and audit, and Appendix F.3 gives the NMR values.

Common Response

C1. Current benchmark coverage and uncertainty (Reviewers cnMs and XQis)

Response. MolP2SBench is under continuous development, with the goal of broadening its molecular coverage and enriching the evaluation of property use. As part of this ongoing effort, the benchmark has expanded to 499 task families. All six models are evaluated on these same families with 2–8 candidates, identical candidate sets, and a fixed single CF donor per family. Each model has 5,489 observations per condition—correct, shuffle, and CF—across eleven representations of seven physical channels. Terminal-invalid outputs count as errors in unconditional accuracies.

The revised main evaluation uses six models, including Opus 5.5, whereas the separately reported 136-family analyses retain their original cohorts, CF records, and model versions, including Opus 5 and ChemDFM where reported. Channel-specific separation thresholds serve as diagnostics rather than universal inclusion requirements for the 499-family cohort (Appendix B.1–B.2).

The representation-specific results are reported in manuscript Table 2 (PGR) and Table 3 (RM and CFR). Tables R1a–R1b below supplement those results with a 95% confidence interval for each model and representation; their point estimates match the corresponding manuscript entries, without averaging across representations.

Intervals use 10,000 paired family-bootstrap resamples (seed 20261114). Each resample draws 499 families with replacement, retaining matched correct, shuffle, and CF responses and using the same sampled families across models and representations. PGR = ½(A_src + A_CF), and RM = A_src − A_shuffle. The 2.5th and 97.5th percentiles give pointwise 95% intervals, conditional on saved responses and fixed donor assignments. These intervals measure variation across recipient families, not repeated-inference variability; dependencies from reused shuffle donors are not additionally clustered, and no multiple-comparison adjustment is applied.

Table R1a. PGR (%) by representation on the same 499 families. Each cell gives the point estimate [95% CI], corresponding to manuscript Table 2.

RepresentationQwenGemmaLunaSolOpus 5.5Astra
Dipole magnitude26.7 [24.0, 29.4]30.7 [27.9, 33.5]36.4 [33.2, 39.7]38.5 [35.3, 41.8]41.4 [38.2, 44.6]42.6 [39.3, 45.9]
HOMO–LUMO gap22.4 [20.0, 24.8]24.3 [22.0, 26.7]29.0 [26.3, 31.7]40.5 [37.5, 43.6]38.5 [35.4, 41.6]40.5 [37.3, 43.7]
UV–Vis22.8 [20.4, 25.3]27.5 [24.8, 30.1]35.4 [32.5, 38.3]45.5 [42.4, 48.7]47.7 [44.6, 50.8]52.5 [49.4, 55.7]
IR38.5 [35.6, 41.4]49.8 [46.8, 52.8]67.5 [64.5, 70.5]78.0 [75.4, 80.7]82.7 [80.4, 85.0]81.7 [79.4, 84.0]
Raman39.2 [36.5, 41.9]45.3 [42.4, 48.2]65.4 [62.5, 68.3]76.0 [73.3, 78.6]77.9 [75.5, 80.3]78.0 [75.4, 80.6]
¹H NMR: peaks only21.9 [19.6, 24.3]22.1 [19.8, 24.4]24.6 [22.0, 27.4]35.5 [32.6, 38.4]31.3 [28.5, 34.2]43.6 [40.3, 46.9]
¹H NMR: peaks + integration22.0 [19.8, 24.3]23.0 [20.8, 25.3]24.3 [21.7, 27.0]34.6 [31.7, 37.5]29.9 [27.0, 32.8]43.8 [40.5, 47.0]
¹H NMR: atomwise22.1 [19.8, 24.5]22.0 [19.8, 24.3]24.9 [22.3, 27.6]34.9 [31.9, 37.9]29.6 [27.0, 32.3]44.2 [41.0, 47.4]
¹³C NMR: peaks only23.1 [20.8, 25.5]21.5 [19.3, 23.7]26.8 [24.0, 29.6]38.1 [34.8, 41.3]36.2 [33.2, 39.3]49.3 [46.2, 52.4]
¹³C NMR: peaks + degeneracy22.0 [19.8, 24.3]21.2 [18.9, 23.5]24.8 [22.2, 27.6]38.6 [35.6, 41.7]36.9 [33.9, 39.8]51.6 [48.5, 54.8]
¹³C NMR: atomwise21.8 [19.4, 24.2]22.2 [19.9, 24.5]28.5 [25.7, 31.3]37.3 [34.3, 40.4]39.5 [36.7, 42.5]50.8 [47.7, 53.9]

Table R1b. RM (pp) by representation on the same 499 families. Each cell gives the point estimate [95% CI], corresponding to manuscript Table 3.

RepresentationQwenGemmaLunaSolOpus 5.5Astra
Dipole magnitude1.4 [-2.6, 5.4]5.4 [1.0, 9.8]15.4 [10.0, 20.6]14.8 [9.6, 20.0]19.6 [14.6, 24.8]21.2 [16.0, 26.7]
HOMO–LUMO gap-1.2 [-4.2, 1.8]1.6 [0.2, 3.0]3.6 [-1.0, 8.2]22.6 [17.4, 27.7]15.4 [11.2, 19.8]19.6 [14.6, 24.6]
UV–Vis0.8 [-1.6, 3.2]1.4 [-2.0, 4.8]14.2 [9.4, 19.0]23.8 [19.0, 28.9]27.1 [22.2, 31.7]29.9 [24.6, 35.1]
IR19.6 [15.6, 23.8]25.5 [20.8, 30.1]41.1 [36.1, 46.1]50.9 [46.1, 55.7]56.9 [52.3, 61.7]54.3 [49.7, 58.9]
Raman18.2 [14.0, 22.4]23.2 [19.0, 27.5]41.5 [36.3, 46.7]50.5 [45.7, 55.5]53.9 [49.3, 58.5]50.9 [46.1, 55.5]
¹H NMR: peaks only-1.0 [-3.4, 1.4]-1.2 [-2.6, 0.2]0.4 [-4.2, 5.0]11.0 [6.4, 15.8]6.6 [2.0, 11.4]21.8 [16.6, 27.1]
¹H NMR: peaks + integration3.2 [0.8, 5.6]-0.2 [-1.8, 1.2]2.8 [-1.6, 7.2]8.8 [4.0, 13.4]5.8 [1.6, 10.0]19.2 [14.2, 24.2]
¹H NMR: atomwise-1.0 [-3.4, 1.6]-0.8 [-3.0, 1.4]2.6 [-1.8, 7.2]12.2 [7.4, 16.8]2.4 [-2.2, 7.2]19.4 [14.2, 24.6]
¹³C NMR: peaks only0.0 [-2.8, 2.6]-1.6 [-4.2, 0.8]3.2 [-1.6, 8.0]13.4 [8.4, 18.4]13.2 [8.4, 18.0]24.2 [18.8, 29.7]
¹³C NMR: peaks + degeneracy1.0 [-1.8, 3.8]-1.6 [-3.8, 0.6]7.4 [2.8, 12.0]10.2 [5.2, 15.2]13.6 [8.6, 18.6]26.9 [21.6, 32.3]
¹³C NMR: atomwise-0.2 [-3.2, 2.8]-0.8 [-3.4, 1.8]8.4 [3.6, 13.2]11.4 [6.4, 16.4]18.0 [13.4, 22.8]27.1 [21.8, 32.1]

CFR in manuscript Table 3 remains conditional on each model’s and representation’s source-correct, CF-valid cases. The shared input cohort aligns PGR and RM comparisons but does not make CFR eligibility identical. Below, cnMs W2 & Q4(b), Table R5 reproduces the paired contrasts reported in the revised Appendix G.3 (Table 19), using the 499-family cohort and explicitly matched CFR eligibility.

C2. Candidate-set dependence (Reviewers cnMs, Mgp8, and XQis)

Response. Manuscript Section 4.6 and Table 5, with additional details in Appendix F.1, report the candidate-set ablation. We construct 459 nested subsets from 210 families originally containing 6–8 candidates, retaining the source, fixed CF donor, scaffold, and condition-specific property input. The main comparison uses the smallest subset for each family: two or three candidates.

The reported aggregate recovery improves after candidate reduction, but 9.0–24.2% of initially correct source answers become incorrect. Higher overall recovery therefore does not ensure preservation of previously successful decisions under the same physical evidence. Uniform-guess PGR also rises from 14.9% to 43.3%, and pruning changes distractor composition and potentially option positions alongside candidate number. C4 discusses the full analysis of explanations associated with these lost correct decisions.

C3. Full-molecule NMR and observation scope (Reviewers cnMs, Mgp8, and XQis)

Response. The scaffold-only NMR setting reflects a constraint in constructing matched molecular families. Families must share the retained scaffold while admitting valid alternative completions. Using full-molecule NMR while also controlling nuclear-count cues imposes additional restrictions: integrated or atomwise inputs can reveal H or C counts that distinguish candidates without requiring detailed interpretation of the property values. Finding families that satisfy both the structural and count-matching conditions substantially restricts coverage. We therefore use NMR observations from the retained scaffold in the main evaluation, while evaluating full-molecule NMR on smaller matched subsets.

This is a limitation of observation scope: scaffold-only NMR omits potentially informative nuclei in the masked group, whereas other channels describe the whole molecule. Consequently, the main comparison does not establish the relative difficulty of these channels under equally complete observations.

To assess the effect of this restriction, we compare retained-scaffold computed NMR with full-molecule NMR that also exposes nuclei in the masked group, fixing the structural question, candidate order, and source/CF donor. The paired comparison uses 93 ¹H families and 57 ¹³C families, each with three representations. Source and donor have equal H counts in the ¹H set; all candidates have equal C counts in the ¹³C set. Other ¹H distractors may still carry nuclear-count cues.

Table R2. Retained-scaffold → full-molecule NMR (%), averaged over the three representation-specific PGR and CFR scores. PGR counts invalid outputs as errors; CFR uses source-correct, CF-valid pairs separately in each setting.

Model¹H PGR (%)¹H CFR (%)¹³C PGR (%)¹³C CFR (%)
Qwen33.9 → 31.910.8 → 9.540.1 → 42.110.5 → 1.6
Gemma33.3 → 34.91.0 → 4.738.0 → 41.80.0 → 7.1
Luna32.6 → 47.335.1 → 42.948.8 → 64.042.3 → 67.8
Sol47.8 → 63.440.0 → 63.261.4 → 75.465.0 → 75.5
Opus 5.538.4 → 59.141.3 → 56.161.4 → 73.457.9 → 71.1
Astra56.5 → 86.067.4 → 86.473.4 → 95.073.9 → 95.1

All four closed-weight models show higher mean PGR and conditional CFR point estimates for both nuclei, while Qwen has lower ¹H PGR/CFR and ¹³C CFR. PGR compares the matched family cohort, whereas CFR eligibility is determined separately in each setting; CFR differences therefore do not isolate responsiveness changes on a fixed eligible cohort. For Astra, paired source gains are 28.0 points for ¹H (95% CI [20.4, 35.5]) and 27.5 for ¹³C ([16.4, 39.2]); gains vary substantially across models. Apparent channel difficulty therefore depends on which molecular observations are supplied. These results concern computed shieldings and the specified count-matching conditions; they motivate interpreting NMR comparisons jointly with observation scope.

C4. Explanations behind lost correct decisions (Reviewers Mgp8 and XQis)

Response. We examined every valid correct-to-incorrect pair in the smallest-subset comparison, not a sample. A pair consists of two saved responses to the same structural question: the model answers correctly with the original 6–8 candidates, but incorrectly after reduction to two or three candidates. The relevant target remains available, and the scaffold and condition-specific property input are unchanged. This gives 1,362 pairs and 2,724 explanations, covering all six models and eleven representations: 688 pairs under correct properties and 674 under CF properties. In the CF condition, correctness is judged against the designated donor. These are response pairs, not 1,362 distinct molecular families; a family can recur across models, representations, and conditions. Eighteen additional Qwen cases with invalid reduced outputs remain errors in performance scoring but are excluded from the valid-explanation comparison.

Appendix F.1 provides the annotation procedure, detailed category definitions, and actual examples for each category. Domain experts reviewed the annotations for all 1,362 pairs. We distinguish factual errors about structures or supplied observations; unsupported numerical assertions; insufficient candidate discrimination, including reliance on shared features; and disagreement between the selected index and the textual recommendation. These categories concern the reduced, incorrect explanations. Separately, changed evidence interpretation records a change in numerical reference, peak/atom assignment, or qualitative interpretation between the two explanations under unchanged property input. A changed choice alone does not qualify, and an interpretation change is not automatically a factual error.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.