- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces LAO-Eval, a diagnostic benchmark for evaluating LLM capabilities in structure-based ligand affinity optimization. It decomposes the task into structural perception, strategy discovery, strategy execution, and inference quality, evaluating 14 frontier LLMs across 30 protein targets. The authors find that LLMs perform well on basic molecular recognition and editing tasks but struggle with 3D ligand–protein interaction reasoning and realization, which they identify as the main bottleneck in affinity optimization.
Critical strengths
-
Clear benchmark structure: The paper decomposes ligand affinity optimization into perception, strategy discovery, and execution, providing a systematic framework for evaluating different LLM capabilities.
-
Broad empirical comparison: The evaluation covers 14 frontier LLMs across multiple model families and uses chemical validation and structure-based analysis, offering a useful comparison of their strengths and limitations.
Critical weaknesses
-
Weak motivation (Figure 1): The affinity changes are highly noisy, and the regression trends are unconvincing. The experiment does not establish a clear failure pattern that motivates the proposed diagnostic benchmark.
-
Limited novelty and weak spatial reasoning evaluation (Stages 1–2): The findings largely confirm that LLMs perform well on sequence-based molecular tasks but struggle with geometry-dependent interactions. Chemical plausibility mainly tests basic knowledge, while distance perturbation can be explained by simple proximity heuristics rather than actual spatial reasoning. This point has been well discussed in the community.
-
Unreliable execution evaluation (Stage 3): SMILES validity is a weak measure of molecular-editing success. Interaction realization relies on an imperfect Boltz-2 predictor and does not verify whether the specified ligand atom forms the intended interaction.
-
Limited benchmark value and overstated conclusions: The dataset covers only 30 protein targets without connected optimization trajectories. Several metrics are nearly saturated or uniformly poor, limiting their ability to distinguish model capabilities. Overall, the benchmark mainly confirms expected LLM limitations rather than identifying or validating a new failure mechanism for better LLM evaluation.
Questions for authors
The weaknesses discussed above are the main issues I would like the authors to address. Beyond these, I have two additional concerns:
-
Paper positioning: Considering the limited benchmark size, lack of complete optimization trajectories, and emphasis on the limitations of current LLMs rather than new diagnostic findings, I believe this work reads more like a position paper supported by empirical observations. How do the authors justify positioning it as a benchmark paper, and what makes this benchmark valuable for evaluating future models?
-
Writing and presentation quality: The overall logic is difficult to follow, with many small, disconnected explanations and qualifications that make the manuscript appear insufficiently integrated. Some visual illustrations also seem to lack careful human verification, including inconsistencies in figure content, very small text in the main overview figure, and layout issues. Given the disclosed AI-assisted writing, I encourage the authors to carefully review the consistency and presentation of the manuscript. I believe the paper could be improved substantially through a comprehensive revision rather than isolated fixes.