- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces H2L-Bench, which is a test environment for sequential experimental decisions under uncertainty, focusing on small-molecule hit-to-lead discovery. In this task, an agent has a fixed budget to run 8 different assays on 24 compounds, and finally picks one compound that meets several hidden thresholds. The authors use an approximate Bayesian value-of-information (VoI) policy as a reference, and compare it with four LLMs, some heuristic/learned baselines, and a few inference-time scaffolds. The VoI method gets the highest win rate in each difficulty tier, but its advantage over the best LLM is not statistically significant after correction. Also, the paper argues that LLMs sometimes succeed by focusing confirmatory tests on good compounds, while the myopic, cost-normalized VoI policy spreads its budget too wide.
Critical strengths
- I think this is very relevant to practical drug discovery problems. By combining noisy observations, assay artifacts, different costs, and strict budgets, it captures real problems in scientific experimentation that normal benchmarks often miss.
- The Bayesian VoI policy provides a very useful baseline for comparison beyond just LLM rankings. The authors are also very honest about its privileged knowledge of the generative model, treating it as a reference instead of claiming it is an optimal policy.
Critical weaknesses
-
The explanation for the mechanism is a bit too strong based on the current evidence. The task uses a fixed list of candidates and just asks the model to choose which experiments to run. It doesn't really test if the model can design new molecules or understand how molecular structures affect activity. Therefore, the main question is whether the model's success just comes from making early commitments when new evidence shows up. Although VoI performs worse and LLMs perform better in some cases, it doesn't prove this is the exact cause. Maybe LLMs just make stronger commitments because early evidence is clearer, not because their strategy for allocating experiments is actually better.
-
The depth-2 comparison is not very clear for testing the planning horizon. Appendix I changes many things at the same time—such as which actions are removed, whether batch screening is available, how accurate the sampling is, and when the process stops. It also measures two-step planning using only the cost of the first action. Because of this, its worse performance doesn't clearly prove that cost normalization is why VoI fails. The problem could also come from limited lookahead or approximation choices.
-
The PK case study does not support the explanation very well. Section 5.2 mentions that in_vivo_pk addresses a difficult axis in antibiotic-medium. However, Table 4 looks at potency, selectivity, and metabolic stability, while PK measures oral bioavailability and half-life. PK might help confirm basic performance, but the paper needs to fix this explanation and show evidence of how these results actually changed the final candidate selection.
Questions for authors
- For the depth-1 versus depth-2 comparison, could you use the same action sets, stopping rules, and numerical-accuracy standards in both conditions, and explain how the costs of follow-up evaluations are accounted for?
- In the antibiotic-medium experiment, which specific constraint did PK resolve, and did the resulting observations actually change the final nominated compound?
2 comments
Response to Reviewer wjMc (part 2 of 2)
W1: mechanistic explanation. We agree that the trajectory analysis showed an association, not a cause. We ran two tests (Appendix O):
- Is it the early evidence? For each hard-tier game of base Sonnet 5 and Opus 5 (250 each), we replayed the agent's own observations through VoI's posterior up to its first confirmatory assay, and measured how clear the evidence was at that point: the P(win) gap between the two leading compounds, and the leader's share of total P(win). In logistic regressions of winning with archetype fixed effects, commitment's coefficient does not change when either measure is added (Sonnet 5: +0.58 per SD in all three models; Opus 5: +0.70, +0.67 and +0.70; all p < 0.001), and neither clarity measure predicts winning (all p > 0.19; Table 27). Within every tercile of the P(win) gap, games with above-median commitment win more often.
- Does imposing commitment help? We built a VoI variant that, from its first confirmatory choice on, runs confirmatory assays only on its top three compounds by P(win), and ran it against standard VoI on 250 fresh hard-tier games (seeds 200–249). It concentrated its budget as intended (0.58 of confirmatory spend on the top compound, the same as base Sonnet 5, spread over 2.8 compounds instead of 4.8) but did not win more often: 39.6% against 42.8% (p = 0.31; paired difference −3.2 points, approximate 95% CI −8.6 to +2.2; Table 28).
Commitment thus predicts the LLMs' wins after controlling for how clear the early evidence was, but imposing it on VoI does not help. We now describe it as a behavioural marker of the LLMs' successful games, not a cause, and revised the abstract, Section 5.3 and the conclusion accordingly. One explanation consistent with both results, which we have not tested, is that the LLMs concentrate their budget on compounds whose confirmatory results come back favourable, so that concentration follows good evidence rather than producing it.
We also agree that the task does not test molecule design or structure–activity reasoning. This was already stated in the limitations, but we have now also stated it in the introduction.
Thank you again for the specific questions; the controlled depth comparison in particular changed what the paper claims.
Response to Reviewer wjMc (part 1 of 2)
Thank you for the thoughtful and careful review. All three weaknesses pointed to real problems, and we ran the experiments needed to address them. The revised PDF is uploaded; numbers below refer to it. We believe this has greatly strengthened the manuscript.
PK in antibiotic-medium. You are right, and our explanation was wrong.
in_vivo_pkreads only oral bioavailability and half-life. In antibiotic-bacterial these are easy floors, not hard axes (Table 5, formerly Table 4), and the generator draws them independently of every other trait. So PK resolves no hard constraint: it checks two floors that no other assay reads.To check this, we went through the 8 seeds behind Figure 3 (right), where GPT-5.4 won and VoI lost:
PK is therefore the last step of fully profiling the lead, not a decisive assay. The win rates are unaffected; only the explanation changes. We rewrote Strategy 2 in Section 5.2 as "complete the lead's profile", corrected the Figure 3 caption and label, and removed the claim from the abstract.
W2 / Q1: depth-2 comparison. We agree that the original comparison did not isolate the horizon. We ran five conditions that change one factor at a time, on 30 fresh seeds (200–229) in each of the two cells (ppi-hard and antibiotic-medium), 300 games in all (Appendix I, Tables 20–21):
C2 and C3 share action sets, stopping rule and Monte Carlo accuracy, so they differ only in horizon, as Q1 asks. Paired exact McNemar tests, Holm-corrected over the four contrasts:
On the second part of Q1: in the original depth-2 agent (C3), the follow-up must be affordable after the first action, but its cost does not enter the denominator. C5 charges it, and that does not help.
Depth-2's shortfall therefore comes from the approximations needed to make it tractable, not from the extra step, and nothing here shows that cost normalisation is why VoI loses. We removed that claim from the abstract, Section 4 and Appendix H. Appendix I now reports the controlled comparison, and states that depth-2 uses 16 imagined outcomes of each first action and 64 of each follow-up.