- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces a benchmark paper to evaluate the capabilities of LLMs on supramolecular chemistry, particularly on the study of host-guest interactions. The paper mainly includes four tasks: binding affinity prediction, top-binder selection, solvent identification, and host-guest description, and an additional molecular identification task. In addition, the authors collect a domain-specific text corpus (SupraPMC), and evaluate LLMs under different prompting (few shot, chain of thought, etc) and pretraining strategies. The results of the paper show that current LLMs still suffer with these tasks, even with few-shot, CoT, and domain adaptation.
Critical strengths
The paper tests LLM’s capability from a new and niche angle, the host-guest interactions in supramolecular chemistry, because most existing benchmarks focus on tasks such as molecular prediction, reaction prediction, or general chemical knowledge questions. Therefore, such evaluation on binding affinity and selectivity is an interesting direction, and the data collected might be useful for later work.
Critical weaknesses
-
My main concern is the limited novelty of the work, which designs a specific domain benchmark on supramolecular chemistry and tests LLM capabilities on it with prompting and pretraining strategies. While host-guest chemistry is important, I’m not convinced that evaluating these models is by itself a strong contribution. In addition, some LLMs tested in this paper are no longer considered frontier models. Although we understand that newer models might not have been available when the paper was written, if the main contribution of the paper is benchmarking LLMs, this makes me wonder how more recent frontier models would actually perform on these tasks.
-
Main findings are not particularly surprising. For example, the observation that CoT can sometimes hurt model performance has been pointed out by prior work, such as Li et al., When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMs (NeurIPS 2025). I understand supramolecular chemistry is a different domain and the tasks are different, but it is not clear to me what new insights this adds besides showing that similar situations also exist in chemistry.
-
Limited evidence that the benchmark measures generalizable chemical reasoning. I also have concerns about how strongly the experimental results support the paper’s conclusions. For example, the benchmark is mainly built from public binding records, so there is a potential risk of overlap with LLM pretraining data. Even though the authors introduce 63 new host-guest pairs curated by experts, the paper still does not establish how reliable the reference binding affinities are for these pairs, particularly whether they are experimentally measured in a wet lab or not.
Questions for authors
-
what's the main new insight from this paper beyond LLM evaluations in a different chemistry domain? Particularly, how does the finding that CoT prompting could hurt model performance extend what has already been established in the LLM reasoning community?
-
how did the authors get the reference binding affinities for the 63 host-guest pairs? Were these values measured experimentally, or estimated by computation? Could they also be based on expert judgment? If so, how reliable are they?
-
Could the authors clarify how the solvent identification dataset was constructed? Appendix C describes filtering out organic solvent measurement, but then D.3 constructs a six-class task that includes several organic solvents. I find it hard to follow here. So, were different data processing pipelines used for these tasks?
-
The paper uses domain-adaptive pretraining on a mixture of data (chemistry, general-domain, and instruction data). I wonder whether the authors have considered that reinforcement learning with verifiable rewards (RLVR) could be more effective, especially for tasks with clear ground truth (binding affinity prediction and top-binder selection). For questions involving open-ended chemical reasoning, LLMs can be used as judges to construct rewards. This comparison is important, as it could clarify whether domain knowledge or task-specific optimization is the bottleneck for current models.
-
Although supramolecular chemistry is an important area, a general discussion of previous work on AI foundation models for chemistry is important. Such a discussion could strengthen the work and make it more impactful. For example: (1) Omni-Mol: Multitask Molecular Model for Any-to-any Modalities (NeurIPS 2025 Main), and (2) ChemFM as a Scaling Law Guided Foundation Model Pre-trained on Informative Chemicals (Communications Chemistry, 2025/2026).