acceptodds
Under review as a conference paper at ICLR 2027

Molecular Structure Elucidation with Frontier Models: A Benchmark on Literature-Reported IR, H, and C NMR Peak Lists

Abstract

Frontier models are often presented as near-solved structure elucidators on curated spectra. We ask an operational question: given the molecular formula and literature-reported IR, H, and C NMR peak lists, printed wavenumbers and shift tables, not digitised spectra images, can an out-of-the-box frontier model in an agentic workflow recover the correct constitution, and which stage fails when it does not? We release IRSpectra-Bench as a frozen roster of problems, an RDKit InChIKey-14 contract, and the prediction deposits a mechanical scorer can replay. The object is a factorised diagnosis of one closed-book harness. One trained NMR-only baseline is scored on the same roster: NMRTrans, formula and printed H/C lists, IR unused (79/500 top-1; Appendix A.13). It is not a full multi-modal peer of the IR+H+C task. On one closed-book frontier harness, generation top-1 is 45.4% (227/500) and recall (the fraction of problems whose true structure enters the proposed pool) is 49.8% (249/500). Self-ranking places the true structure first in 91.2% of the cases (). The generation wall is 227 exact top-1 / 22 in-pool not top-1 / 251 never proposed. The roster is near-50/50 simple/complex; reweighted to the eligible corpus mix, top-1 on a validate-clean subset is 26.5%. Forward verification of the same pools is a secondary drop, not a second wall: top-1 falls from 227/500 to 204/500 (; diagnostic ; Appendix A.8). On a 60-compound arm only, formula-only recovery is 3/60, accuracy is flat in source-paper year, and recall stays below precision across four vendor families.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
molecular spectroscopy, frontier models, Molecular Structure Elucidation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_ebxWOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces IRSpectra-Bench, a benchmark of 500 molecular structure elucidation problems containing molecular formulae and literature-reported IR, ¹H NMR, and ¹³C NMR peak lists. It evaluates a closed-book frontier-model workflow and decomposes structure recovery into candidate-pool recall and conditional ranking accuracy. The principal conclusion is that candidate generation constitutes the main bottleneck in the evaluated workflow.

Critical strengths
  • Realistic and understudied input format. Printed peak lists as they appear in experimental sections are what chemists actually have, and evaluating models on them is a worthwhile goal.
  • Simple and reusable evaluation. The InChIKey-14 scorer and the deposit format are easy to reuse, and reporting recall and precision given recall separately is good practice.
Critical weaknesses
  • Insufficiently specified benchmark construction and sampling procedure The paper uses existing literature data but fails to adequately explain how its core 500 benchmark problems were selected. The authors identify the structure-complete split and describe balancing by difficulty, but do not clearly specify the sampling procedure within each stratum. This critical omission seriously undermines confidence in the benchmark’s representativeness and weakens the conclusions drawn from it. Furthermore, the missing random seeds for the early draws further limit the reproducibility of the sample-selection process.
  • Insufficient evidence for the main conclusion The central “generation wall” conclusion is insufficiently supported. The restrictive candidate budget constrains recall, yet the paper does not systematically evaluate sensitivity to pool size. The results therefore fail to distinguish budget-induced limitations from a bottleneck in generation capability, undermining the recommendation to prioritize generation over ranking.
  • Unclear contribution and limited comparative evaluation Listing unevaluated systems does not constitute a substantive contribution. Without sufficiently specific reasons for excluding relevant approaches, the paper leaves its empirical contribution inadequately contextualized.
Questions for authors
  • Please clarify the contribution of Point 4 (line 97). What value does listing systems not evaluated in this work provide? Also, please identify each system mentioned in Section 2 that was not evaluated on IRSpectra-Bench and state the specific reason it was excluded.
  • Please specify the complete procedure used to construct the 500-item roster from eligible IRexp records, including sampling within each difficulty stratum, treatment of source-paper concentration and structural redundancy, and the rule selecting 200 of the 224 validate-clean expansion items. Prespecifying question IDs fixes the evaluation set, but does not by itself establish its representativeness.
  • Given the scale of the IRexp resource, what considerations motivated evaluating only 500 problems? Could this relatively small sample limit coverage of the chemical space or the robustness of subgroup analyses, and might evaluation on a larger sample provide a more representative assessment of model performance?
  • Can the authors provide a fully specified evaluation on a common subset under one consistent inference protocol, including the complete prompts, model identifier, reasoning settings, and candidate budget?
  • How do recall and precision given recall change as the candidate budget k increases (e.g., 5, 10, 20, 50)? What happens with an exhaustive formula-constrained isomer pool plus a ranker?
Reviewer_ndRhOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors introduce IRSpectra-Bench, a benchmark comprising 500 molecular structure elucidation tasks based on molecular formulas and literature-reported IR and NMR peak lists. The proposed evaluation protocol enables the assessment of frontier LLMs on this task and distinguishes between failures in candidate generation and subsequent ranking.

Critical strengths
  1. The authors introduce a standardized benchmark for molecular structure elucidation based on literature-reported spectral data in textual form, which could serve as a useful resource for evaluating and comparing specialized approaches in this field.
  2. The work investigates the applicability of frontier LLMs to molecular structure elucidation, distinguishing between errors in candidate generation and subsequent ranking, while also analyzing the impact of molecular complexity.
Critical weaknesses
  1. Significant issues with the organization and clarity of presentation make the paper difficult to follow and evaluate. In particular, the abstract is overloaded with numerical results, the stated contributions are mixed with limitations of the study, and the relationship to prior work is insufficiently discussed. The procedures for constructing and validating the benchmark are also not described transparently enough. Furthermore, experimental results are reported across different subsets without sufficiently clear distinctions, and there are errors in cross-references and citations.
  2. Given that one of the main contributions is a standardized benchmark for evaluating LLMs, the heterogeneity and limited reproducibility of the experimental protocol are particularly concerning. The main results combine runs with and without thinking under undocumented generation settings, while cross-model comparisons are conducted on a small subset with different numbers of generated candidates. Moreover, the central conclusion that generation errors dominate ranking errors is based on ranking among at most three candidates, while the additional Generate-wide experiment does not isolate the effect of candidate pool size.
Questions for authors
  1. What validation procedures were applied to IRSpectra-Bench, and what motivated the selection of the validate-clean subset from the full set of 500 compounds?
  2. Could the authors report the main experimental results using a consistent model configuration or conduct a controlled comparison of thinking and non-thinking modes on the same subset?
  3. Given that the full prompt, exact model version, and generation parameters were not recorded, could the authors provide a complete evaluation under fully documented and reproducible conditions?
  4. Does the conclusion that generation errors dominate ranking errors still hold when the number of generated candidates is varied (e.g., k = 3, 5, 10) under otherwise fixed generation conditions? How would this affect cross-model comparisons?
  5. Could the authors extend the study with a direct comparison against closely related state-of-the-art specialized methods, such as MMST (10.1002/anie.202517611), MolSpectLLM (10.48550/arXiv.2509.21861), SECS (10.1038/s41467-026-73846-y) and others, on IRSpectra-Bench, or explain why such comparisons are not feasible?
Reviewer_MVmYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes IRSpectra-Bench, a benchmark that measures whether frontier LLMs can determine a molecule’s constitution when given only its molecular formula and the IR band lists and 1H/13C NMR peak lists written as text in the experimental sections of papers.

Critical strengths
  • The input format is realistic. Rather than simulated spectra or educational problems, it uses the incomplete peak lists actually reported in papers. It is clearly distinguished from existing benchmarks, and its focus on literature-based structure confirmation is a strength.
  • A variety of experiments are conducted, including error analysis.
Critical weaknesses
  • The core conclusion, that models rank well and generation is the bottleneck, is largely determined by the candidate budget. k≤3, and Claude produced only 2.2 candidates on average (based on n=60). The fewer candidates a model produces, the closer recall gets to top-1 and the closer precision|recall gets to 1, so the conclusion is one that naturally follows when k is small. Comparisons that control for the number of candidates are needed.
  • It is difficult to identify any technical contribution. The paper provides a benchmark and proposes an evaluation scheme, but it is a combination of existing evaluation methods, and it is hard to find technical contribution or depth.
  • Claude is the only frontier model that solved all 500 problems.
Questions for authors

None.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.