acceptodds
Under review as a conference paper at ICLR 2027

SupraBench: A Benchmark for Supramolecular Chemistry

Abstract

Supramolecular chemistry, which includes the study of non-covalent host–guest assemblies, has advanced various applications. However, designing host–guest systems remains time-consuming, requiring days of dry-lab verification per candidate pair. Although LLMs have emerged as a fast alternative with strong performance on molecular binding tasks, no benchmark currently systematically evaluates LLMs for host–guest reasoning across fundamental supramolecular chemistry tasks, e.g., binding affinity prediction. To this end, we collaborate with domain experts to release the first Supramolecular benchmark, called SupraBench, to evaluate LLMs in chemistry reasoning. Specifically, we design four fundamental tasks, i.e., binding affinity prediction, top-binder selection, solvent identification, and host–guest description, plus an auxiliary vision-based task for molecular identification. We also release SupraPMC, a curated 16M-token corpus of Supramolecular chemistry articles distilled from Europe PMC, to support the adaptation to the supramolecular domain. We benchmark a broad range of open and proprietary LLMs and find that general-purpose LLMs leave substantial headroom on every task, DAPT over SupraPMC transfers cleanly to in-distribution regression but trades off against strict letter-format output. Moreover, the difficulty profile differs sharply across task families, revealing distinct failure modes that indicate specific gaps in current supramolecular chemical reasoning.

Primary area:
datasets and benchmarks
Keywords:
Chemistry Benchmark, Supramolecular Chemistry, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_uA5MOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces a benchmark paper to evaluate the capabilities of LLMs on supramolecular chemistry, particularly on the study of host-guest interactions. The paper mainly includes four tasks: binding affinity prediction, top-binder selection, solvent identification, and host-guest description, and an additional molecular identification task. In addition, the authors collect a domain-specific text corpus (SupraPMC), and evaluate LLMs under different prompting (few shot, chain of thought, etc) and pretraining strategies. The results of the paper show that current LLMs still suffer with these tasks, even with few-shot, CoT, and domain adaptation.

Critical strengths

The paper tests LLM’s capability from a new and niche angle, the host-guest interactions in supramolecular chemistry, because most existing benchmarks focus on tasks such as molecular prediction, reaction prediction, or general chemical knowledge questions. Therefore, such evaluation on binding affinity and selectivity is an interesting direction, and the data collected might be useful for later work.

Critical weaknesses
  1. My main concern is the limited novelty of the work, which designs a specific domain benchmark on supramolecular chemistry and tests LLM capabilities on it with prompting and pretraining strategies. While host-guest chemistry is important, I’m not convinced that evaluating these models is by itself a strong contribution. In addition, some LLMs tested in this paper are no longer considered frontier models. Although we understand that newer models might not have been available when the paper was written, if the main contribution of the paper is benchmarking LLMs, this makes me wonder how more recent frontier models would actually perform on these tasks.

  2. Main findings are not particularly surprising. For example, the observation that CoT can sometimes hurt model performance has been pointed out by prior work, such as Li et al., When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMs (NeurIPS 2025). I understand supramolecular chemistry is a different domain and the tasks are different, but it is not clear to me what new insights this adds besides showing that similar situations also exist in chemistry.

  3. Limited evidence that the benchmark measures generalizable chemical reasoning. I also have concerns about how strongly the experimental results support the paper’s conclusions. For example, the benchmark is mainly built from public binding records, so there is a potential risk of overlap with LLM pretraining data. Even though the authors introduce 63 new host-guest pairs curated by experts, the paper still does not establish how reliable the reference binding affinities are for these pairs, particularly whether they are experimentally measured in a wet lab or not.

Questions for authors
  1. what's the main new insight from this paper beyond LLM evaluations in a different chemistry domain? Particularly, how does the finding that CoT prompting could hurt model performance extend what has already been established in the LLM reasoning community?

  2. how did the authors get the reference binding affinities for the 63 host-guest pairs? Were these values measured experimentally, or estimated by computation? Could they also be based on expert judgment? If so, how reliable are they?

  3. Could the authors clarify how the solvent identification dataset was constructed? Appendix C describes filtering out organic solvent measurement, but then D.3 constructs a six-class task that includes several organic solvents. I find it hard to follow here. So, were different data processing pipelines used for these tasks?

  4. The paper uses domain-adaptive pretraining on a mixture of data (chemistry, general-domain, and instruction data). I wonder whether the authors have considered that reinforcement learning with verifiable rewards (RLVR) could be more effective, especially for tasks with clear ground truth (binding affinity prediction and top-binder selection). For questions involving open-ended chemical reasoning, LLMs can be used as judges to construct rewards. This comparison is important, as it could clarify whether domain knowledge or task-specific optimization is the bottleneck for current models.

  5. Although supramolecular chemistry is an important area, a general discussion of previous work on AI foundation models for chemistry is important. Such a discussion could strengthen the work and make it more impactful. For example: (1) Omni-Mol: Multitask Molecular Model for Any-to-any Modalities (NeurIPS 2025 Main), and (2) ChemFM as a Scaling Law Guided Foundation Model Pre-trained on Informative Chemicals (Communications Chemistry, 2025/2026).

Reviewer_3P6aOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SupraBench, a supramolecular host–guest chemistry benchmark composed of tasks with literature-derived ground-truth labels. Specifically, the tasks consist of four types plus one auxiliary vision-based type: (1) binding affinity prediction, (2) top-binder selection, (3) solvent identification, (4) host–guest description, and (5) image-to-SMILES molecular identification. The dataset is cleaned from 5,362 crawled records into 4,635 records over 2,008 components, with 63 additionally expert-annotated pairs. In addition, the paper releases a 16M-token domain corpus, SupraPmc, and applies it to domain-adaptive post-training. The experimental results show that current LLMs leave substantial headroom on every task, with failure modes differing sharply across task families.

Critical strengths
  • The benchmark covers an evaluation axis that prior chemistry benchmarks leave untested, scoring host–guest pairs rather than single molecules.
  • The released 16M-token corpus gives the community a reusable resource for low-budget domain adaptation.
Critical weaknesses
  • W1. The DAPT gains may be affected by overlap between SUPRAPMC (from Europe PMC) and the source papers underlying SUPRABENCH (from SupraBank records). The paper does not quantify this overlap, leaving it unclear whether Table 3's gains reflect transferable domain knowledge or potential exposure to test records' source papers during adaptation. Verification is further precluded since no repository link to code and data, and no details on the domain experts' qualifications appear in the submission.
  • W2. The 63 expert-constructed novel host–guest pairs in §4.2 offer a useful test of generalization beyond potential literature exposure. However, they are evaluated only on the three proprietary models, not the three DAPT-adapted open-weight models in Table 3. Evaluating the latter on these pairs would help clarify whether the gains reported in §4.4 reflect transferable domain knowledge or are limited to the original benchmark distribution.
Questions for authors

Q1. Step 2 (p.21) discards any record whose solvent field contains tokens like methanol/DMSO/chloroform, yet SID (§D.3) is "constructed from the same cleaned records" and covers only six solvent classes with 2,172 questions; is Step 2 a literal substring filter that deuterated variants (DMSO-d6, CD2Cl2) escape, or does SID use a pre-Step-2 split?

Q2. §4.4(iii) states that DAPT-adapted Llama3.1-8B fails to follow the strict letter format for solvent identification, yet Table 3 shows improvements in both SID F1 (0.151 → 0.173) and balanced accuracy (0.225 → 0.268). Could the authors report the format- or parse-failure rate before and after DAPT, and clarify how such failures are handled when computing these metrics?

Reviewer_VuHVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces SupraBench, an LLM evaluation suite for supramolecular host-guest chemistry, with binding-affinity prediction, top-binder selection, solvent identification, host-guest description, and an auxiliary image-to-SMILES task. It derives the benchmark from SupraBank records and introduces SupraPMC, a reported 16M-token domain corpus. The evaluation compares eight LLMs under direct, few-shot and chain-of-thought prompting, with additional small-model, domain-adaptation, host-disjoint specialized-ML and description-judge analyses. A separate set of 63 expert-annotated novel pairs is intended to probe generalization. The reported results suggest substantial remaining difficulty and strongly task-dependent prompting and adaptation effects.

Critical strengths

A useful, focused benchmark scope. The combination of affinity regression, relative binder selection, solvent context, descriptions and molecular-image recognition targets several distinct aspects of host-guest chemistry. This breadth, together with a domain-adaptation corpus, could make the resource useful beyond a single leaderboard.

Evaluation extends beyond a basic model comparison. The paper includes host-disjoint ChemProp/CGIB comparisons, DAPT recipe ablations, a chemistry-specific description judge and per-host calibration analyses. These are meaningful attempts to contextualize LLM scores and investigate failure modes; the novelty test is also a useful direction, subject to validating its reference labels.

Critical weaknesses

W1. The reported benchmark is not internally consistent enough to audit. Appendix B.2, Eq. (14), assigns zero Tanimoto similarity to invalid predictions and averages over the full split. Tanimoto must therefore be no greater than SMILES validity. However, Table 5 reports Qwen3.5-27B Base validity 0.738 versus Tanimoto 0.814, and CoT validity 0.626 versus Tanimoto 0.838. Appendix B.2 also averages per-example ROUGE-1 precision, recall and F1 over the same examples. Consequently mean F1 cannot exceed (mean precision + mean recall)/2, but Table 1 reports Qwen3.5-9B Base recall 0.040, precision 0.023 and F1 0.043, exceeding the bound 0.0315. These differences cannot be explained by the displayed rounding. Separately, Appendix C Step 2 removes organic-solvent records, while Appendix D.3 constructs a six-solvent task from the same cleaned records. A separate retained branch may exist, but it is not specified. The paper needs consistent data lineage, metric denominators and recomputed results before the affected comparisons can be trusted.

W2. Reference validity and the claim of contamination-independent reasoning are insufficiently established. Appendix C imputes temperature for 2,238 records and pH for 3,358 records, uses host-level average enthalpies for temperature correction, and retains distinct pH bins. Appendix D.1 nevertheless asks for affinity under standard aqueous conditions without explaining how differing pH bins are filtered or reconciled for that prompt. Thus, the reference may contain condition-dependent variation absent from the model input. More critically, Section 4.2 describes expert annotation of affinities for 63 unpublished pairs, without specifying whether the values come from new measurements, validated calculations or expert estimates, or reporting their uncertainty. This set is central to the assertion that the benchmark measures inference rather than memorization. Comparable error magnitude and prompt-dependent gaps do not by themselves exclude contamination. No document/pair-level overlap audit between SupraPMC and benchmark sources is reported. These issues limit both the interpretation of absolute affinity error and the generalization claims.

Questions for authors
  1. Clarification/correction: What exact sample sets and denominators produced Tables 1 and 5? Please reconcile the two metric bounds above and provide corrected tables or the corresponding implementation details. Does this change model rankings or prompting conclusions?

  2. Clarification: Is SID built from records before organic-solvent filtering? Please provide per-task record counts after each relevant filter and per-solvent counts. For BAP, how are retained pH-specific records mapped to the standard-condition prompt?

  3. Validation needed for the current claims: How were the 63 novel affinity labels obtained, with what uncertainty and independent verification? Please also report benchmark-source overlap with SupraPMC and whether DAPT improvements persist after removing overlapping sources. If these checks are unavailable, the contamination and generalization conclusions should be narrowed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.