acceptodds
Under review as a conference paper at ICLR 2027

LAO-Eval: A Diagnostic Benchmark for LLM Capabilities in Structure-Based Ligand Affinity Optimization

Abstract

Structure-based small-molecule ligand affinity optimization—the iterative editing of hit compounds to improve protein-binding affinity under binding-pocket constraints—is a fundamental task in computer-aided drug discovery. Despite rapid progress in scientific reasoning, the capabilities of large language models (LLMs) across the component steps of this task remain poorly understood. We first show through end-to-end experiments that frontier LLMs achieve only marginal affinity improvements and frequently degrade higher-affinity starting ligands. To identify where these failures arise, we introduce LAO-Eval, a diagnostic benchmark for Ligand Affinity Optimization that decomposes the reasoning process into three independently evaluable stages: Structural Perception, Strategy Discovery, and Strategy Execution, supplemented by chain-of-thought quality analysis. We evaluate 14 frontier LLMs on 10 diagnostic tasks spanning 30 DUD-E protein targets. Our results show that: (i) LLMs perform well on 2D molecular structure recognition (micro-F1 = 0.92) and protein secondary-structure classification (accuracy up to 90%), but their performance drops sharply on 3D ligand–protein interaction perception (F1 0.40); (ii) proposed strategies are usually chemically plausible (93%), and most models generate valid molecules in more than 95% of completed responses, yet at most 25% of intended interactions are realized after Boltz-2 co-folding and structural analysis; (iii) performance differences across model families narrow substantially on interaction realization, indicating a shared limitation of current LLM paradigms; and (iv) controlled distance perturbations show that LLMs respond to spatial information but cannot reliably translate it into interaction predictions or structurally effective edits. These findings identify 3D ligand–pocket reasoning as the principal bottleneck in LLM-based ligand affinity optimization and motivate the development of molecular foundation models with spatially grounded representations and physics-aware structural feedback. Code, data, and reproduction instructions are included in the supplementary reproducibility archive.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Structure-Based Drug Design, Ligand Affinity Optimization, Diagnostic Benchmark, Protein-Ligand Interaction, Spatial Reasoning, Chain-of-Thought Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SNf8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LAO-Eval, a diagnostic benchmark for evaluating LLM capabilities in structure-based ligand affinity optimization. It decomposes the task into structural perception, strategy discovery, strategy execution, and inference quality, evaluating 14 frontier LLMs across 30 protein targets. The authors find that LLMs perform well on basic molecular recognition and editing tasks but struggle with 3D ligand–protein interaction reasoning and realization, which they identify as the main bottleneck in affinity optimization.

Critical strengths
  1. Clear benchmark structure: The paper decomposes ligand affinity optimization into perception, strategy discovery, and execution, providing a systematic framework for evaluating different LLM capabilities.

  2. Broad empirical comparison: The evaluation covers 14 frontier LLMs across multiple model families and uses chemical validation and structure-based analysis, offering a useful comparison of their strengths and limitations.

Critical weaknesses
  1. Weak motivation (Figure 1): The affinity changes are highly noisy, and the regression trends are unconvincing. The experiment does not establish a clear failure pattern that motivates the proposed diagnostic benchmark.

  2. Limited novelty and weak spatial reasoning evaluation (Stages 1–2): The findings largely confirm that LLMs perform well on sequence-based molecular tasks but struggle with geometry-dependent interactions. Chemical plausibility mainly tests basic knowledge, while distance perturbation can be explained by simple proximity heuristics rather than actual spatial reasoning. This point has been well discussed in the community.

  3. Unreliable execution evaluation (Stage 3): SMILES validity is a weak measure of molecular-editing success. Interaction realization relies on an imperfect Boltz-2 predictor and does not verify whether the specified ligand atom forms the intended interaction.

  4. Limited benchmark value and overstated conclusions: The dataset covers only 30 protein targets without connected optimization trajectories. Several metrics are nearly saturated or uniformly poor, limiting their ability to distinguish model capabilities. Overall, the benchmark mainly confirms expected LLM limitations rather than identifying or validating a new failure mechanism for better LLM evaluation.

Questions for authors

The weaknesses discussed above are the main issues I would like the authors to address. Beyond these, I have two additional concerns:

  1. Paper positioning: Considering the limited benchmark size, lack of complete optimization trajectories, and emphasis on the limitations of current LLMs rather than new diagnostic findings, I believe this work reads more like a position paper supported by empirical observations. How do the authors justify positioning it as a benchmark paper, and what makes this benchmark valuable for evaluating future models?

  2. Writing and presentation quality: The overall logic is difficult to follow, with many small, disconnected explanations and qualifications that make the manuscript appear insufficiently integrated. Some visual illustrations also seem to lack careful human verification, including inconsistencies in figure content, very small text in the main overview figure, and layout issues. Given the disclosed AI-assisted writing, I encourage the authors to carefully review the consistency and presentation of the manuscript. I believe the paper could be improved substantially through a comprehensive revision rather than isolated fixes.

Reviewer_nb11OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

LAO-Eval is a diagnostic benchmark for LLM performance on structure-based ligand affinity optimization. It breaks the task into structural perception, strategy discovery and strategy execution, plus CoT quality analysis, and evaluates 14 frontier models on 10 tasks across 30 DUD-E targets. The main result is that models are weak at 3D ligand–pocket interaction perception and at realizing the intended interactions after editing.

Critical strengths
  1. Breaking the task into stages makes it clearer where models fail, instead of only looking at end-to-end affinity change.
  2. It covers a wide set of models under the same evaluation protocol.
Critical weaknesses
  1. Boltz-2 is a limited affinity oracle. Wan et al. (JCTC 2026) report only weak correlations with physics-based free energies across large compound sets, along with a strong “regression-to-the-centre” effect and no meaningful correlation for top-ranked compounds [1]. Figure 1 and Table 5 in LAO-Eval therefore do not provide reliable evidence of how LLM edits affect binding affinity.
  2. The execution task relies on a fixed set of strategies generated by Claude Opus 4.6 and filtered by experts. Using the same strategies makes comparisons fairer, but their source and difficulty may still affect realization rates, making it harder to attribute the results to general LLM ability.
  3. Post-edit interaction retention (~45%) is close to Boltz-2’s crystal fidelity (45.7%). Since interaction realization also depends on Boltz-2, the absolute success rates are difficult to interpret.
  4. The ≤25.1% interaction realization rate is used as key evidence of a bottleneck in 3D ligand–pocket reasoning. However, there is no clear upper bound for this rate, and it is affected by the curated strategies, SMILES editing, and noise in Boltz-2 predictions. Moreover, realizing a target interaction does not necessarily improve binding affinity.
  5. The interaction-perception prompt appears to use simplified distance criteria (interaction_perception.py), while the ground truth from PLIP also considers angles, atom types and other distance constraints. This mismatch may underestimate LLM performance.

[1] Wan, S., Zhang, X., Xue, X., & Coveney, P. V. (2026). Reliability of AI methods in drug discovery: Evaluation of Boltz-2 for structure and binding affinity prediction. Journal of Chemical Theory and Computation, 22(15), 7811-7824.

Questions for authors

Could the authors include PLIP’s full criteria in the interaction-perception prompt and rerun the evaluation?

Reviewer_8xLXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces LAO-Eval, a diagnostic benchmark that splits LLM-driven structure-based ligand affinity optimization into three stages: Structural Perception, Strategy Discovery, and Strategy Execution. A separate analysis of chain-of-thought quality runs alongside them. It covers 10 tasks over 30 DUD-E targets (150 protein–ligand complexes) and 14 frontier LLMs. Execution is validated with Boltz-2 co-folding and PLIP. The authors report that models do well on 2D functional-group recognition and protein secondary structure, but poorly on ligand–pocket interaction perception.

Critical strengths
  • A useful diagnostic decomposition. Splitting the pipeline into perception, strategy, execution, and CoT quality is a sensible way to localize failures. It is more informative than the aggregate end-to-end scores in prior work. Models handle 3D input for a single entity (protein fold, up to 90%) but fail on cross-entity interaction perception (F1 < 0.40).
  • The study covers 14 models from five families under identical prompts, and 30 targets spanning 15 protein families. Tasks are paired with expert-curated strategies shared across models, and the full compute scale is documented. The distance-perturbation and shrink experiments are a good idea because they test whether models use spatial information rather than templates.
Critical weaknesses
  • The central claim (“3D ligand–pocket reasoning is the principal bottleneck”) is not supported by the evidence. Interaction Realization is confounded by Boltz-2 noise.
  • Strategy Execution uses expert-curated strategies rather than each model’s own proposals. So the autonomous discover-and-execute pipeline is never evaluated, and the paper concedes it cannot rule out strategy quality as an independent failure source.
  • The Interaction Recognition task (1.3) gives models a pre-digested text description of neighboring atoms and residues. This is arguably a text-comprehension and chemistry-rule task, not 3D perception. The Task 1.2 vs. 1.3 contrast also differs in input format, label definition, and difficulty, so “cross-entity” is only one of several possible explanations.
  • The CoT error analysis relies on two LLM judges with 83.6% agreement and no human-labeled subset.
Questions for authors
  • Is the Task 1.3 F1 comparable to a simple baseline, such as a distance-threshold rule applied to the same text description, or human expert performance on a subsample? This would show whether F1 < 0.40 reflects a reasoning failure or a mismatch between task format and PLIP’s criteria.
  • How did you handle the 27 target–hit pairs where Boltz-2 failed, and the pose generation via Glide for additional actives? Were failures correlated with particular targets or families?
  • Could the Claude Opus 4.6 involvement in generating the candidate strategies (and the Claude judge) have given Claude models an advantage in the results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.