- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces IRSpectra-Bench, a benchmark of 500 molecular structure elucidation problems containing molecular formulae and literature-reported IR, ¹H NMR, and ¹³C NMR peak lists. It evaluates a closed-book frontier-model workflow and decomposes structure recovery into candidate-pool recall and conditional ranking accuracy. The principal conclusion is that candidate generation constitutes the main bottleneck in the evaluated workflow.
Critical strengths
- Realistic and understudied input format. Printed peak lists as they appear in experimental sections are what chemists actually have, and evaluating models on them is a worthwhile goal.
- Simple and reusable evaluation. The InChIKey-14 scorer and the deposit format are easy to reuse, and reporting recall and precision given recall separately is good practice.
Critical weaknesses
- Insufficiently specified benchmark construction and sampling procedure The paper uses existing literature data but fails to adequately explain how its core 500 benchmark problems were selected. The authors identify the structure-complete split and describe balancing by difficulty, but do not clearly specify the sampling procedure within each stratum. This critical omission seriously undermines confidence in the benchmark’s representativeness and weakens the conclusions drawn from it. Furthermore, the missing random seeds for the early draws further limit the reproducibility of the sample-selection process.
- Insufficient evidence for the main conclusion The central “generation wall” conclusion is insufficiently supported. The restrictive candidate budget constrains recall, yet the paper does not systematically evaluate sensitivity to pool size. The results therefore fail to distinguish budget-induced limitations from a bottleneck in generation capability, undermining the recommendation to prioritize generation over ranking.
- Unclear contribution and limited comparative evaluation Listing unevaluated systems does not constitute a substantive contribution. Without sufficiently specific reasons for excluding relevant approaches, the paper leaves its empirical contribution inadequately contextualized.
Questions for authors
- Please clarify the contribution of Point 4 (line 97). What value does listing systems not evaluated in this work provide? Also, please identify each system mentioned in Section 2 that was not evaluated on IRSpectra-Bench and state the specific reason it was excluded.
- Please specify the complete procedure used to construct the 500-item roster from eligible IRexp records, including sampling within each difficulty stratum, treatment of source-paper concentration and structural redundancy, and the rule selecting 200 of the 224 validate-clean expansion items. Prespecifying question IDs fixes the evaluation set, but does not by itself establish its representativeness.
- Given the scale of the IRexp resource, what considerations motivated evaluating only 500 problems? Could this relatively small sample limit coverage of the chemical space or the robustness of subgroup analyses, and might evaluation on a larger sample provide a more representative assessment of model performance?
- Can the authors provide a fully specified evaluation on a common subset under one consistent inference protocol, including the complete prompts, model identifier, reasoning settings, and candidate budget?
- How do recall and precision given recall change as the candidate budget k increases (e.g., 5, 10, 20, 50)? What happens with an exhaustive formula-constrained isomer pool plus a ranker?