- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
MolP2SBench evaluates property-conditioned local chemical-graph completion. Given a scaffold and a small set of candidate molecules represented as SMILES, models select the candidate matching supplied physical properties. Experiments across seven models use property removal, shuffling, and within-family substitutions to assess behavioral reliance and responsiveness.
Critical strengths
- Holding the scaffold and candidates fixed while substituting properties provides an interpretable test of whether candidate selections respond to changing evidence. This offers useful diagnostic information beyond answer accuracy alone.
- The evaluation covers multiple models and property representations, supported by documented molecular alignment, unconditional counterfactual scoring, and sensitivity analyses. These components provide a useful foundation for controlled comparisons.
Critical weaknesses
- The task supplies most of the chemical connectivity and leaves limited inference to the model. Only 1-3 heavy atoms are hidden, while the scaffold, attachment constraints, boundary bond orders, and 2-4 complete candidates are provided. Performance therefore measures highly constrained local discrimination, with insufficient evidence of capabilities beyond familiar pattern matching. Although “chemical structure” is valid terminology, the manuscript should consistently specify SMILES-based chemical-graph completion and distinguish it from three-dimensional geometry or conformational reconstruction. The limited task complexity and uncertain broader relevance weaken the significance of the contribution.
- The diagnostic interpretations require stronger calibration. PGR can reward property-invariant predictions, but actual no-property predictions are not evaluated under the same scoring rule. CFR conditions on different initially correct cohorts, and the common-cohort contrast in Appendix G.3 remains statistically uncertain. Cross-channel comparisons additionally confound modality with observation scope: NMR excludes hidden-fragment nuclei, whereas other channels describe complete molecules. These limitations weaken conclusions about distinct reasoning capabilities and relative difficulty across properties.
Questions for authors
- Could you explicitly define the evaluated notion of “structure” and consistently delimit the claims to local chemical-graph completion throughout the title, abstract, and conclusions?
- How do the findings generalize as masked fragments and candidate sets become larger, particularly on previously unseen scaffolds?
- How do simple chemical-rule, local-pattern, or dedicated property-prediction baselines perform under the same conditions?
- Could you report PGR for unchanged no-property predictions and establish whether the reliance-responsiveness contrasts persist with common cohorts, paired uncertainty estimates, and repeated inference?
- Do cross-channel comparisons persist under more comparable observation scopes and information content, including whole-molecule NMR?
3 comments
Continued
W2 & Q4(a). PGR calibration against unchanged predictions
Response. We evaluated the requested reference for all six models on the same 499 families. Each model's no-property choice is held unchanged when scored against the source and CF-donor targets. Structural inputs, candidate order, donor assignments, and scoring match the property-conditioned evaluation; invalid outputs count as errors.
Table R4. Unchanged no-property PGR on the common 499-family cohort. Each model has one shared reference across the eleven representations because their structural inputs and paired targets coincide.
| Model | Unchanged no-property PGR (%) |
|---|---|
| Qwen | 23.45 |
| Gemma | 22.14 |
| Luna | 24.45 |
| Sol | 23.75 |
| Opus 5.5 | 23.95 |
| Astra | 24.65 |
Manuscript Table 2 places this reference alongside the PGR for each property representation, allowing direct comparison without averaging across representations. Appendix F.2 details the unchanged-prediction scoring protocol and further comparisons. These recovery comparisons complement RM and CFR's diagnostics of correct pairing and donor-directed responsiveness.
Under the single-donor PGR definition, a fixed source or fixed donor choice scores 50%, and a fixed third candidate scores zero; uniform guessing yields 100/K%. The unchanged reference and property-conditioned results use this same scoring rule.
W2 & Q4(b). CFR comparisons and statistical uncertainty
Response. We agree that a shared input cohort alone does not align CFR eligibility. We therefore examine the two RM–CFR ordering examples discussed in the manuscript using the saved results on all 499 families. For each comparison, CFR is recomputed on the intersection of families for which both representations produce a correct source answer and a valid CF answer. RM retains its standard definition on all 499 families. These are representation-specific comparisons, with no averaging across representations.
Table R5. Paired differences for the two manuscript examples. Each difference is A minus B, in percentage points, with a pointwise 95% confidence interval. The common-eligibility count applies to CFR; RM uses all 499 families.
| Model | Comparison (A − B) | ΔRM [95% CI] | Common CFR families | ΔCFR [95% CI] |
|---|---|---|---|---|
| Luna | ¹³C peaks + degeneracy − ¹H peaks only | +7.01 [+0.60, +13.43] | 55 | −16.36 [−28.85, −4.26] |
| Sol | HOMO–LUMO gap − dipole magnitude | +7.82 [+0.80, +14.83] | 98 | −14.29 [−26.67, −1.98] |
For Luna, common-eligibility CFR is 8/55 (14.55%) for ¹³C peaks with degeneracy and 17/55 (30.91%) for ¹H peaks only. For Sol, it is 41/98 (41.84%) for the gap and 55/98 (56.12%) for dipole magnitude. Thus, the opposite RM and CFR orderings persist after matching CFR eligibility in both examples. Eligibility alignment matters: Luna's original, separately eligible CFR difference is only −1.42 points (95% CI [−10.30, +7.51]); its two eligibility sets each contain 141 families but share only 55.
For these contrasts, we use 100,000 paired family-bootstrap resamples (seed 20261115), drawing 499 families with replacement and retaining matched responses across representations and conditions. CFR eligibility denominators are recomputed in each resample. All four pointwise 95% intervals in Table R5 exclude zero. However, with approximate Bonferroni adjustment across these four contrasts (98.75% intervals), both RM intervals and Sol's common-eligibility CFR interval include zero. As an additional sensitivity check, restricting RM itself to the same 55/98 families preserves the positive differences, but their 95% intervals include zero: Luna +10.91 [−5.77, +27.45] and Sol +5.10 [−6.74, +17.00]. This latter analysis is conditioned on initial correctness and does not replace standard RM.
These examples provide supplementary evidence that higher RM need not coincide with higher CFR; they do not establish a universal ordering or distinct internal reasoning mechanisms. These are the updated contrasts reported in Appendix G.3 (Table 19), subject to the multiplicity and common-cohort sensitivity qualifications stated above. As in C1, uncertainty is conditional on saved responses and fixed donor assignments, without additional clustering for reused shuffle donors. Repeated-inference variability remains untested: resampling families does not measure variation from rerunning the same input or changing prompts.
Continued
W2 & Q5. Comparable information across property channels
Response. C3 explains the scaffold-only design and reports the matched full-molecule NMR comparison (manuscript Section 4.2, Figure 2). The results show that observation scope contributes to measured performance, so cross-channel differences should not be attributed solely to reasoning ability. The comparison does not equalize the information content of different channels; conclusions remain tied to the specified observation conditions.
Thank you again for your constructive feedback, which has helped us clarify the task scope and strengthen the calibration and interpretation of our evaluation metrics.
References
[1] MolLangBench: A Comprehensive Benchmark for Language-Prompted Molecular Structure Recognition, Editing, and Generation. ICLR 2026.
[2] Can LLMs Solve Molecule Puzzles? A Multimodal Benchmark for Molecular Structure Elucidation. NeurIPS 2024.
[3] MassSpecGym: A Benchmark for the Discovery and Identification of Molecules. NeurIPS 2024.
Response to cnMs
Thank you for highlighting task scope and diagnostic calibration. We address Q4 in two parts.
W1 & Q1. Task definition and scope of claims
Response. We agree that the evaluated notion of structure should be explicit. Here, chemical structure refers to molecular connectivity represented as SMILES. This terminology is consistent with related benchmarks: MolLangBench [1] evaluates molecular structure recognition and editing using SMILES representations, and MolPuzzle [2] formulates molecular structure elucidation with SMILES outputs. These examples illustrate that SMILES is a representation of molecular structure, and that the term does not require prediction of three-dimensional coordinates. MassSpecGym [3] also distinguishes de novo generation from spectrum-based retrieval among supplied candidate molecular graphs, providing a precedent for evaluating candidate-based structure identification as a distinct task.
In MolP2SBench, the specific task is property-guided masked-scaffold completion with supplied candidate structures, as specified in the abstract and task definition. Holding the scaffold and candidates fixed allows us to test whether correct property pairing improves selection and whether substituting an alternative candidate's properties redirects the answer toward that candidate. The contribution is a controlled diagnostic of property use in structural decisions, rather than evidence of unrestricted structure elucidation. Our claims concern this completion setting; they do not imply open-ended reconstruction of an entire molecule or prediction of three-dimensional geometry or conformations. The Limitation section also explicitly distinguishes this setting from open-ended structure elucidation.
W1 & Q2. Generalization to more difficult tasks
Response. MolP2SBench continues to expand its coverage within a consistent task definition. C1 reports the current 499-family evaluation, and C2 compares candidate sets within the same family. The present version uses QM9 molecules with 1–3 masked heavy atoms; larger masks and an independent unseen-scaffold split are directions for continued development.
Q3. Simple and specialized baselines
Response. The evaluation includes the measured no-property reference below to calibrate recovery from unchanged structural predictions. Chemical-rule and dedicated property-prediction baselines remain outside the present comparison. A forward predictor could rank candidates by agreement with the observation, with its training data and candidate-property access specified. This would provide a complementary reference for task difficulty.