acceptodds
Under review as a conference paper at ICLR 2027

UMO-Agent: Uncertainty-Guided Molecular Optimization Agent for Drug Discovery

Abstract

Molecular optimization against a protein target is an important step in drug discovery, seeking to improve an initial ligand's binding affinity while maintaining drug-likeness, synthetic accessibility, and structural feasibility. We focus on fragment-based optimization, which enables controlled molecular refinement through modular chemical edits. However, the problem poses several technical challenges, including competing objectives, reward hacking, uncertain evaluation feedback, and combinatorial search. Earlier techniques primarily employ heuristic or evolutionary search, which requires careful operator and objective design, or reinforcement learning, which introduces additional policy-training costs and difficulties. Recent LLM-based agents support molecular optimization through tool use and iterative feedback, motivating mechanisms that limit the propagation of unreliable evaluations through molecular edits and accumulated experience. To fill this gap, we introduce UMO-Agent, a training-free LLM-based agent that integrates predictive uncertainty into the control of iterative fragment-based optimization. The agent combines Monte Carlo dropout estimates from a neural affinity predictor with docking consistency, interaction evidence, and molecular-property assessments. Its central design separates candidate screening from acceptance and links the resulting reliability assessment to memory updates and subsequent fragment search. We evaluate UMO-Agent on the CrossDocked2020 benchmark, and the results show that it achieves an average joint Success Rate of 60% across three independent runs, compared with approximately 35% reported for the agent-based baseline CIDD under its original evaluation protocol.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
AI for Drug Discovery, Structure-Based Drug Design, Agentic AI, Molecular Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.60 over 5 reviews, above 10% of the others.
Reviewer_qqkzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces UMO-Agent, an LLM-based agentic framework for fragment-based molecule optimzation. UMO works by editing the initial molecule iteratively. In each iteration, a challenger molecule is selected and compared with the currently retained one, from different aspects. Compared to existing works of agentic molecule optimization, the key claimed technical novelty of UMO is that it takes uncertainty and risks within the editing and decision steps into consideration. On the CrossDocked benchmark, UMO achieves a higher success rate than the reported baselines. Ablation studies support the effectiveness of different components in UMO.

Critical strengths
  1. Considering uncertainty and risks in the decision process of molecule optimization is meaningful and well-motivated.

  2. Perfoming local optimization and then full docking can effectively reduce cost.

  3. Reported experimental results are good. Ablation studies clearly show the performance gains from different components.

Critical weaknesses
  1. As highlighted in the title, abstract and introduction section, considering uncertainty is the core technical contribution of this paper. However, as can be seen from the experimental results, even UMO w\o MCDropout outperforms baselines like CIDD significantly. This indicates that the good performance of UMO comes from the workflow as a whole, in which the contribution of other components also plays an important role.

  2. A higher SR alone cannot explain how and when the uncertainty-based component works, particularly given that uncertainty signal does not trigger the critical "RED" flag on its own. The authors should design and conduct experiments to investigate how this components works, for example, how uncertainty signals alter the decision of the critic and the controller.

  3. Some important details are missing: the precise definitions and computation of several evidence-vector components, the calculation of "edit radius", the rules for building the library, etc.

  4. MCDropout requires access to a predictor with dropout enabled at inference time. How can UMO accommodate a strong black-box predictor for which MC dropout is unavailable? Additionally, five forward passes for MCDrop may be not enough; please provide an sensitivity analysis on the number of forward passes. Furthermore, it is not clear whether the uncertainty quantified by MCDrop correlates well with prediction errors; please provide evidence for this.

Questions for authors

Please see "weaknesses". One more: why is the Boltzmann distribution chosen? How sensitive is the perforance to the choice of the distribution family here?

1 comment

Response to Reviewer qqkz

Thanks for your constructive comments and questions. Below we address Weaknesses 1–4 and Question 1.

W1: Isolating the Contribution of Uncertainty Quantification Within the Full Workflow.

Removing only MC-Dropout reduces Joint SR from 60% to 50% and increases mean ΔHA\Delta\text{HA} from +2.2 to +6.6 (Table 2). Table G4 further isolates UQ with and without LLM guidance: enabling UQ improves SR from 55% to 56% without the LLM, and from 50% to 60% with the LLM, while reducing mean HA from 29.2 to 24.8 in the latter setting. These results suggest that UQ is particularly useful in the LLM-guided workflow.

W2: How Uncertainty Signals Alter Critic Assessments, Controller Risk States, and Closed-Loop Trajectories.

In Figure 1 and Eq. (4), utu_t enters ete_t as an auxiliary signal for the Evidence Critic (CtevidC_t^{\text{evid}}) and Ctrl. It can affect risk assessment for borderline proposals and corroborate other anomalies, but does not trigger RED on its own. In our 100-target audit (General Response Table G3c), enabling utu_t increases GREEN-conditional acceptance (85.1% -> 94.9%), reduces verified steps (428 -> 341) and committed edits (236 -> 191), and limits molecular growth (ΔHA:+6.6→+2.2\Delta\text{HA}: +6.6 \rightarrow +2.2). Because these are different stochastic trajectories, we do not interpret the aggregate differences as matched-state decision changes.

W3: Reproducibility Details for Evidence Vector ete_t, Edit Radius, and Fragment Library Construction.

Thanks for pointing this out. We will detail all three in Appendix A.3–A.7:

  • Evidence vector ete_t: compares candidate vs. current ligand via redocking gain (gtcertg_t^{\text{cert}}), screening–docking reversal, pose RMSD, interaction retention, Vina–NN conflict, utu_t, QED/SA shifts, and failure streak to assign {GREEN, YELLOW, RED}.
  • Edit radius: defined as 1−Tanimoto(candidate,current)1 - \text{Tanimoto}(\text{candidate}, \text{current}) (LOCAL ≤0.52\le 0.52, BROAD ≤0.75\le 0.75).
  • Library rules: starting from the public DeepFrag library [1] (5,563 fragments), we retain 1,164 fragments that pass RDKit validity checks and contain at most six heavy atoms (≤6\le 6 HA).

W4: Black-Box Predictor Compatibility, Sensitivity to Forward Passes (TT), and Correlation with Errors.

Black-box compatibility. MC-Dropout can be replaced by other uncertainty estimates when available. With point predictions only, UMO-Agent retains two-tier verification, evaluator disagreement, and memory. However, removing UQ reduces Joint SR from 60% to 50% and increases molecular growth (+2.2 to +6.6 HA), highlighting its contribution.

Forward-pass sensitivity. Using the same populations as Table G3a, varying TT from 3 to 30 yields AUROC of 0.876–0.908 (Table G3b). Five passes (0.883 AUROC) provide a practical balance between discrimination and cost.

Relationship with prediction errors. Lead optimization requires correctly ranking similar ligands by affinity to determine whether an edit improves upon the current ligand. In Section 4.4, even FEP+ achieves a median Spearman correlation of only 0.62 on congeneric series (docking: 0.32; NN: 0.24). Thus, we combine UQ with LLM-guided decisions to mitigate risk, even at the cost of rejecting potentially beneficial edits when evidence is insufficient. Uncertainty remains relatively stable on real analogues but rises on potentially reward-hacked states even for small heavy-atom changes (Table G3a). Adding UQ to the LLM-guided workflow improves SR from 50% to 60% and reduces molecular growth (Tables G3c–G4). This supports UQ as a complementary risk signal, not a direct detector of ranking errors.

Q1: Rationale and Sensitivity of the ϵ\epsilon-Mixed Boltzmann Sampling Distribution.

We use an ϵ\epsilon-mixed Boltzmann distribution to convert combined proposal scores into stochastic action preferences, with temperature (T=0.70T=0.70), uniform exploration (ϵ=0.20\epsilon=0.20), and probability capping (0.700.70) to balance exploration and exploitation (60%±3%60\% \pm 3\% SR across three seeds). Replacing this score-guided Boltzmann distribution with uniform sampling over edit factors while keeping verification, UQ, and risk control intact (w/o Decision, Table 2) reduces Joint SR to 40%±2%40\% \pm 2\%.

We appreciate your constructive feedback and hope these responses help address your questions.

References

  • [1] Green H, Koes DR, Durrant JD. DeepFrag: a deep convolutional neural network for fragment-based lead optimization. Chem. Sci., 2021.
Reviewer_AC83OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces UMO-Agent, an agentic pipeline for molecular lead optimization. The method consists of three steps, in which an agent samples different actions to modify existing candidate ligands by pulling from a fragment library. It uses binding-affinity results, statistical feedback, uncertainty estimation, and LLM decision-making to decide whether to accept or reject the resulting molecules; either way, it adds knowledge to memory to guide future modifications.

Critical strengths
  • The core methodology and ideas in this paper are very good.
  • The way the agent interacts with the molecule is very robust; not only is it given structural and physical information about the current ligand-protein fit, but it is also constrained to only make edits via particular operations, which I believe is a smart choice that avoids the syntactic and semantic headaches that often come with having LLMs output raw SMILES.
  • Having all molecule acceptances and rejections update a continuously maintained knowledge base is also a nice way to make every docking evaluation and generation valuable in some way, and for past mistakes and successes to guide future decisions.
  • The uncertainty quantification is also well done and folds logically into the pipeline.
Critical weaknesses
  • The evaluation and results keep this work from being fully convincing, despite its meticulous methodology.
  • My main concern is with this joint Success Rate (SR) metric. Firstly, on CrossDocked2020, UMO-Agent only beats the other baselines on this SR metric, and is entirely outclassed in the others. The SR metric itself is not very convincing to me either; it utilizes cutoffs of QED > 0.25 and normalized SA > 0.59 (which should equate to < 4.69 for regular unnormalized SA). These cutoffs are rather on the low end for both metrics, and do not reliably say that the molecules would be good.
  • The authors also mention that SR remains between 60% and 62% when using Claude-4.8 Opus, GPT-5.4, and GPT-OSS. This to me is actually also slightly concerning, given that Claude 4.8 and GPT 5.4 are much larger, more intelligent closed source models, while GPT-OSS is max 120B parameters, and therefore I would expect the closed source models to make better decisions and have better performance. The fact that they all remain within the same very tight range makes me question whether or not the agent actually has a significant role in the generation pipeline, or if the other method architecture is primarily driving the gains in metrics and results.
  • In Appendix A.4, the deficits for each metric (that determine how prioritized each objective is at intermediate optimization steps) are exactly the SR cutoffs used in evaluation. This means that the pipeline is optimizing exactly for these cutoffs, and likely produces a large pool of molecules that all just barely meet this arbitrarily defined criteria, which may be a very large reason as to why UMO-Agent beats the other methods so much in SR but gets beaten in all of the singular objectives; it is optimizing for that exact threshold, which seems to me as unfair to the baselines, which presumably have no knowledge of this threshold at generation time.
Questions for authors
  • I would be interested in seeing how the method compares to the baselines when these cutoffs are raised to a stricter degree, e.g., QED > 0.50 and SA < 3.0.

  • Improving evaluation in general would be useful, and if the results remain the same or similar, an explanation of why this is the case would be useful.

  • The authors also mention that SR remains between 60% and 62% when using Claude-4.8 Opus, GPT-5.4, and GPT-OSS. This to me is actually also slightly concerning, given that Claude 4.8 and GPT 5.4 are much larger, more intelligent closed-source models, while GPT-OSS is max 120B parameters, and therefore I would expect the closed-source models to make better decisions and have better performance. Could you elaborate why that isn't the case?

  • The fact that they all remain within the same very tight range makes me question whether or not the agent actually has a significant role in the generation pipeline, or if the other method architecture is primarily driving the gains in metrics and results.

-It would be very informative and a much stronger result if the authors could include an ablation that has a random / heuristic-based operator that makes the modification decisions instead of an LLM, so that we could see the true impact of the agent within this fundamentally agentic framework.

With stronger results, more convincing metrics (e.g., stricter thresholds), isolation between the optimization criteria and evaluation metrics, and more rigorous ablations, this work could be greatly improved.

3 comments

Response to Reviewer AC83 (1/2)

Thanks for your detailed comments and questions. Below we address Weaknesses 1–4 in Part 1 and Questions 1–5 in Part 2.

W1 & W2: Joint SR Cutoffs and Single-Objective Metric Performance Across CrossDocked2020 and FEP+.

Because de novo methods in Table 1 (DecompOpt, MolJO) generate new molecules from scratch and cannot optimize a starting ligand x0x_0, we conducted controlled same-x0x_0 lead-optimization comparisons under a 5 Å Vina box in General Response Tables G1a & G1b. On CrossDocked2020 (Table G1a), UMO-Agent outperforms CIDD [1] (58.0% vs. 37.04% SR) and PROBE [2] on both Joint SR (58.0% vs. 52.0%) and mean QED (0.542 vs. 0.510) with comparable SA and Vina; on 100 targets from the external FEP+ benchmark [3] (Table G1b), UMO-Agent reaches 81.0% vs. 63.0% SR (Reference 56.0%), improving both ΔVina\Delta\text{Vina} (+1.022+1.022 vs. +0.457+0.457) and SA (0.7360.736 vs. 0.7110.711). Furthermore, Table G2 (Q1) shows that our delivered ligands remain competitive when evaluated post-hoc under stricter QED > 0.50 and Raw SA < 3.0 cutoffs.

W3: Why Different LLM Backends Yield Consistent 60%–62% SR.

Because it is difficult to be certain whether free-form LLM molecular outputs reflect reliable chemistry or potential hallucinations, and because our paper's primary goal is risk reduction, we place strict boundaries on how the LLM participates (Figure 1, Appendix A.8, B). Specifically, we ground all structural edits in a curated library of real, existing chemical fragments and keep the LLM out of direct molecule generation; instead, it guides search indirectly via prior chemical preferences (Pocket Planner), non-numerical semantic evidence (Evidence Critic), and trajectory experience summaries (Memory Summarizer). Within this bounded interface, GPT-OSS-120B, GPT-5.4, and Claude-4.8 Opus perform comparably (60%–62% SR, Table 4), while our non-LLM ablation (General Response Table G4, Q5) indicates that this indirect guidance still improves QED, compactness, Vina, and Joint SR (56% →\rightarrow 60%).

W4: Isolating Optimization Criteria from Evaluation Cutoffs (Appendix A.4).

To examine whether UMO-Agent's gains come from cutoff awareness in Appendix A.4, we isolated internal optimization criteria (-8.18 / 0.25 / 0.59) from evaluation metrics by evaluating on unseen stricter cutoffs, where UMO-Agent more than doubles Reference on Both Strict (16.0% vs. 7.0%) and stays comparable to or slightly ahead of PROBE across 5 of the 6 settings (Q1), which is shown in General Response Table G2.

References

  • [1] CIDD: Collaborative intelligence for structure-based drug design empowered by LLMs
  • [2] Probe before you edit: Probing-guided molecular optimization for LLM agents in structure-based drug design
  • [3] The maximal and current accuracy of rigorous protein-ligand binding free energy calculations


Response to Reviewer AC83 (2/2)

Q1: Comparison Against Baselines Under Unseen Stricter Property Cutoffs (QED > 0.50, Raw SA < 3.0).

Following your suggestion, we evaluated UMO-Agent and PROBE post-hoc under Strict QED > 0.50, Strict Raw SA < 3.0 (normalized SA >0.778>0.778), Both Strict, and stricter Vina cutoffs with no access to these thresholds during optimization (General Response Table G2). Despite optimizing internally only for -8.18 / 0.25 / 0.59, UMO-Agent more than doubles Reference's pass rate on Both Strict (16.0% vs. 7.0%; PROBE 14.0%) and Strict QED > 0.50 (37.0% vs. 11.0%; PROBE 36.0%), remains ahead on Vina < -8.50 (46.0% vs. 43.0%) and Vina < -9.00 (35.0% vs. 32.0%), and stays close on Strict Raw SA < 3.0 alone (19.0% vs. 21.0%; Ref. 12.0%).

Q2: Explanation of Evaluation Behavior and Metric Trade-offs Across Threshold Regimes.

Under the stricter SA constraint (Raw SA < 3.0), PROBE achieves a slightly higher joint SR (21.0% vs. 19.0%). When both stricter QED and SA constraints are applied, UMO-Agent remains competitive (16.0% vs. 14.0%). As these stricter thresholds were not used during optimization, the results suggest that the observed improvements are not limited to the original evaluation thresholds.

Q3: Elaboration on Why Model Scale (Claude-4.8 Opus, GPT-5.4, GPT-OSS-120B) Yields Similar SR.

Building on W3, we cannot always be certain whether an unconstrained LLM output reflects reliable chemical reasoning or an unverified hallucination. Because our paper focuses on reducing optimization risk, we place strict guardrails on the LLM: structural assembly is handled by a real fragment library, LLM outputs are restricted to bounded categorical evidence and experience summaries, and every candidate is gated by paired Vina.dock() verification and our RED controller. Under these constraints, GPT-OSS-120B appears sufficient to match GPT-5.4 and Claude-4.8 Opus.

Q4: Role of the LLM Agent Relative to the Overall Optimization Architecture.

Deterministic verification, memory, and UQ (LLM- UQ+, 56% SR in General Response Table G4) are intentionally designed to provide a reliable optimization backbone, while numerical rules alone only track threshold deficits (leaving mean QED at 0.494–0.496). The LLM complements this backbone in three places (Figure 1): Pocket Planner supplies prior chemical preferences, Evidence Critic provides non-numerical semantic chemistry evidence (Ctchem,CtevidC_t^{\text{chem}}, C_t^{\text{evid}}, Section 4.5), and Memory Summarizer distills trajectory experience. In Table G4, adding the LLM elevates mean QED (0.526–0.530) and SA (0.700 in LLM+ UQ-), and when paired with UQ (LLM+ UQ+) to curb excess molecular growth (HA 29.2→24.829.2 \rightarrow 24.8), yields the most compact molecules, highest QED pass rate (92%), and highest Joint SR (60%).

Q5: Ablation Replacing the LLM with a Random or Heuristic / Rule-Based Operator.

We agree that replacing the LLM with a non-LLM baseline is the clearest test of its contribution. Because our proposal distribution (Eq. 2) is already a stochastic sampling process, adding a random operator in place of the LLM would compound randomness and make variance harder to interpret. We therefore fixed all three LLM outputs to deterministic neutral values (bLLM=1.0b_{\text{LLM}} = 1.0, neutral critic concern; Appendix A.8, B.4) in General Response Table G4 and Q4.

We appreciate your constructive feedback and hope these additional benchmarks, stricter-threshold evaluations, and ablations help address the concerns raised in the review.


I thank the authors for responding to my concerns. I will discuss this with other reviewers and make a final decision. I dont have further questions

Reviewer_jDnxOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces UMO-Agent, a training-free LLM-based agent for fragment-based molecular optimization in drug discovery. Given a protein pocket and an initial ligand, the agent iteratively proposes fragment edits from a fixed library, screens candidates with cheap local docking, and applies full re-docking verification before accepting a molecule. A MC Dropout-based uncertainty estimate from a frozen neural affinity predictor is combined with docking consistency, interaction evidence, and molecular property assessments into an evidence vector, which an LLM critic and deterministic controller use to assign a reliability state that gates acceptance and memory updates. A memory module tracks verified edit outcomes per target and biases subsequent sampling away from failed or risky directions. Evaluated on the CrossDocked2020 100-pocket benchmark, UMO-Agent achieves a joint SR of 60% across three runs.

Critical strengths
  1. The two-tier screening and verification design is a practical and well-motivated contribution. By separating cheap local pose optimization from full docking applied only to the selected challenger, the framework efficiently filters candidates whose screening gains do not survive broader pose exploration, addressing a known failure mode in fragment-based optimization.

  2. Using historical verified edit outcomes to bias future sampling aligns well with how medicinal chemists iteratively reason about structure-activity relationships. Tracking which edit directions produce verification-supported improvements on the current target, and down-weighting failed or risky directions, is a reasonable approach to making search more efficient as experience accumulates.

Critical weaknesses
  1. The underlying justification for why the method works is insufficient. The authors clearly identify problems in existing methods, but do not provide adequate explanation for how their design resolves them. The optimization is inherently context-dependent — each editing step changes the molecular context for the next — yet the paper does not explain how the proposed gating and scoring mechanisms adapt to this dynamic context in a principled way. The method appears as a complex combination of gates, scoring terms, and heuristic rules. From a chemical perspective, the design is not grounded in clear chemical or physical reasoning: using Vina to verify Vina means screening and verification share the same systematic errors, and MC Dropout on a frozen predictor does not provide meaningful uncertainty over true binding affinity. The method lacks a coherent scientific narrative connecting its components to the problem it claims to solve.

  2. The experimental evaluation does not adequately support the claims. PROBE (Yang et al., 2026) is explicitly identified as the closest related method but is not compared against. The comparison with de novo generative models is not fair because those methods generate molecules from scratch while UMO-Agent starts from a reference ligand, making the SR comparison misleading. Baseline numbers are taken directly from original papers without re-evaluation under a unified protocol, introducing uncontrolled experimental variation. Furthermore, UMO-Agent does not perform well on all metrics, but the paper lacks detailed analysis of when and why the method underperforms, which is necessary to understand the actual scope and limitations of the contribution.

Questions for authors
  1. The joint SR threshold (Vina < -8.18, QED > 0.25, SA > 0.59) is adopted from prior benchmarks, but these thresholds are relatively loose — a QED of 0.25 is below the typical drug-likeness standard. How many of UMO-Agent’s delivered molecules would satisfy stricter thresholds commonly used in lead optimization, and does the method’s advantage over baselines hold under those criteria?
1 comment

Response to Reviewer jDnx

Thanks for your constructive comments and suggestions. Below we address Weaknesses 1–2 and Question 1.

W1: Unifying Motivation, Physical Role of Verification and UQ, and Dynamic Context Adaptation.

We clarify that our design addresses a specific failure mode in iterative molecular optimization: unreliable evaluator feedback can be repeatedly exploited and propagated through subsequent decisions and memory updates.

Our framework is organized around controlling this feedback loop, rather than independently combining several heuristics.

First, MC-Dropout provides an auxiliary signal of neural predictive instability (General Response Table G3a), cross-evaluator checks flag conflicts between Vina and the neural predictor, and full Vina docking checks whether improvements observed under local pose optimization persist under a more extensive pose search. None of these mechanisms is assumed to recover true binding affinity or eliminate intrinsic evaluator bias.

Second, the risk-aware controller determines whether the available evidence is sufficient to commit a proposed edit. Importantly, subsequent memory updates depend on verification-supported outcomes, so that unverified improvements are not directly reinforced in future proposal preferences.

Third, since each accepted edit changes the molecular context, subsequent attachment sites, molecular properties, interaction descriptors, and proposal constraints are recomputed from the current ligand. Target-specific memory supplies historical factor-level preferences rather than assuming that previous edits remain universally beneficial.

Our ablations support the importance of these mechanisms: removing the evidence-based verification gate reduces SR from 60% to 43%, removing memory reduces it to 51%, and disabling MC-Dropout reduces it to 50% (Table 2). The additional UQ OFF vs. UQ ON trajectory analysis (General Response Table G3c) further shows reductions in verified steps (428 vs. 341), GREEN redocking failures (14.9% vs. 5.1%), and molecular growth (+6.6 vs. +2.2 heavy atoms).

We acknowledge that these components do not provide theoretical guarantees of affinity correctness. Our methodological contribution is their coordinated use to regulate how imperfect evidence influences commitment and future search, with the goal of reducing feedback amplification during sequential optimization.

W2: Controlled Head-to-Head Comparisons with PROBE Across CrossDocked2020 and FEP+, and Metric Trade-offs.

Evaluation. We agree that de novo baselines in Table 1 are not directly comparable. Since CIDD's [1] code omits required files, we matched its published CrossDocked2020 setup (same 100 x0x_0, 5 Å box) and evaluated PROBE [2] under the same initialization and box. UMO-Agent achieves 58% SR vs. 37.04% for CIDD and 52% for PROBE (Table G1a). On 100 external FEP+ [3] complexes, it achieves 81% vs. 63% SR, with greater Vina improvement (+1.022 vs. +0.457) and higher SA (0.736 vs. 0.711) than PROBE (Table G1b).

Limitations and underperforming cases. We acknowledge that UMO-Agent does not consistently outperform competing methods on every individual metric. For example, on FEP+, UMO-Agent achieves lower mean QED than PROBE (0.569 vs. 0.612), and under the stricter SA constraint on CrossDocked2020, its joint SR is slightly lower (19% vs. 21%, Table G2). The restricted fragment-editing space and conservative verification mechanism may limit improvements on individual objectives. UMO-Agent is primarily designed for risk-aware optimization of existing ligands under imperfect evaluators, rather than unrestricted molecular generation or single-property maximization. We will clarify this scope and discuss these limitations in the revised manuscript.

Q1: Performance Under Stricter Medicinal-Chemistry Cutoffs (QED > 0.50, Raw SA < 3.0).

Re-evaluating our CrossDocked2020 molecules post-hoc under unseen stricter cutoffs without retuning internal parameters (General Response Table G2), UMO-Agent more than doubles Reference's pass rate on Both Strict (QED > 0.50 & Raw SA < 3.0: 16.0% vs. 7.0%) and remains comparable to or slightly ahead of PROBE across 5 of the 6 settings (including Vina < -8.50: 46.0% vs. 43.0%, and Vina < -9.00: 35.0% vs. 32.0%).

We appreciate your constructive feedback and hope these additional experiments and clarifications help address the concerns raised in the review.

References

  • [1] CIDD: Collaborative intelligence for structure-based drug design empowered by LLMs
  • [2] Probe before you edit: Probing-guided molecular optimization for LLM agents in structure-based drug design
  • [3] The maximal and current accuracy of rigorous protein-ligand binding free energy calculations


Reviewer_2WJmOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces a framework called UMO-Agent, which performs fragment-based molecular optimization for protein targets without the need for training. The method addresses the challenge of unreliable computational feedback during iterative molecular editing—specifically, how inaccurate affinity estimates can lead to the acceptance of unsuitable candidate fragments, thereby misdirecting the subsequent search process.

Critical strengths
  1. Decoupling low-cost candidate screening from more cautious acceptance decisions is an effective design choice. Distinguishing between verified improvements and unsuccessful attempts also provides a sound basis for updating the local memory regarding the target.
  2. Beyond docking scores, the evaluation incorporates metrics such as drug-likeness, synthetic accessibility, heavy-atom growth, and ligand efficiency. These indicators help characterize trade-offs that might otherwise be overlooked if optimization were focused solely on docking affinity.
  3. Section 4.4 examines the consistency between computational evaluation results and experimental binding affinities for 34 congeneric series, highlighting the limitations of relying exclusively on low-cost affinity estimates.
Critical weaknesses
  1. Table 1 references published baseline results from methods such as RGA, DecompOpt, MolJO, and CIDD. However, these methods differ in their initialization, candidate generation processes, computational budgets, and evaluation protocols. Even in the case of CIDD—which shares a more similar setup for lead discovery—the optimization schemes are not identical. Although the authors acknowledge these discrepancies, they make it difficult to definitively attribute the reported advantages in success rates solely to the proposed framework. Re-evaluating the strongest baseline methods under consistent conditions—using identical initial ligands, evaluation budgets, and scoring procedures—would provide more compelling evidence.
  2. The core motivation is to leverage predictive uncertainty to prevent unreliable molecular modifications from adversely affecting subsequent optimization. Table 2 shows that removing MC-Dropout causes the overall success rate to drop from 60% to 50%, suggesting that this signal contributes to the framework's performance. However, the results do not clarify whether higher uncertainty actually corresponds to larger errors in affinity prediction, or if it correlates with a higher probability of making incorrect optimization decisions. The evaluator analysis in Section 4.4 demonstrates that prediction errors do indeed exist; however, it does not directly assess the calibration or discrimination capability of the uncertainty estimates provided by MC-Dropout. Additional analyses—specifically regarding the correlation between uncertainty and error, as well as calibration—would strengthen the core methodology.
  3. The framework combines deterministic scoring rules, memory-guided sampling, LLM-generated preferences, and LLM critics. However, the w/o Decision variant removes planner and memory-guided proposal bias, making it difficult to independently determine the contribution of LLM-based decisions. Likewise, removing the verification gate does not isolate the impact of LLM critics from other reliability signals. Although the authors studied different LLM backends, additional controlled comparisons with deterministic or rule-based alternatives would help clarify whether the LLM component provides benefits beyond structured optimization rules.
  4. The computational efficiency of this framework remains undercharacterized. UMO-Agent performs multiple candidate evaluations, iterative docking, neural affinity prediction, and LLM interactions during optimization. Table 2 reports average trajectory lengths, but the authors correctly point out that these values ​​do not determine computational efficiency because the methods differ in candidate pool expansion and evaluation procedures. Comparisons of docking calls, neural inference calls, LLM usage, and wall-clock times under matching computational budgets are valuable for assessing the practical applicability of the proposed framework.
  5. Experimental evaluation is limited to one molecular optimization benchmark and threshold-based success metrics. The evaluation uses 100 CrossDocked2020 protein bags, with primary success rates defined by fixed thresholds on Vina, QED, and normalized SA. Although this method improved the joint metric, its average Vina score was inferior to that reported by DecompOpt and MolJO, while its normalized SA decreased relative to the initial reference ligand. These results indicate that the approach emphasizes a particular balance between the chosen objectives. Additional evaluation using alternative thresholds, Pareto-based metrics, or other protein target benchmarks will help determine the robustness and generalizability of the reported improvements.
Questions for authors

Weaknesses

2 comments

Response to Reviewer 2WJm (1/2)

Thanks for your detailed comments and constructive suggestions. Below we address Weaknesses 1–2 in Part 1 and Weaknesses 3–5 in Part 2.

W1: Controlled Baseline Comparison Under Identical Initialization and Evaluation Protocols.

Because de novo and prior-guided methods in Table 1 (RGA, DecompOpt, MolJO) generate ligands from scratch and cannot optimize a given starting ligand, we conducted controlled comparisons against the two ligand-editing agent baselines CIDD [1] and PROBE [2] (General Response Section 1, Tables G1a & G1b). Since CIDD's repository omits required files (UniMAP and its fragment bank), we aligned our CrossDocked2020 setup to CIDD, using the same 100 starting ligands (x0x_0) and 5 Å Vina evaluation bounding box, and evaluated PROBE under the identical x0x_0, 5 Å box, and aligned fragment space (≤6\le 6 HA). In Table G1a, UMO-Agent achieves 58.0% Joint SR and 0.542 QED, outperforming CIDD (37.04% SR) and PROBE (52.0% SR, 0.510 QED) with comparable SA and Vina. On 100 targets from the external FEP+ benchmark [3] (Table G1b), UMO-Agent also outperforms PROBE (81.0% vs. 63.0% SR, ΔVina\Delta\text{Vina} +1.022+1.022 vs. +0.457+0.457; detailed in W5).

W2: Uncertainty Discrimination Capability and Closed-Loop Decision Reliability.

To clarify the role of utu_t, we conducted two targeted analyses in General Response Section 3 (Tables G3a & G3c):

  1. Distinguishing normal analogues from greedy-generated molecules (Tables G3a & G3b): On the same FEP+ dataset [3], we compared 5-pass MC-Dropout (ut=σu_t = \sigma) across 1,254 real congeneric analogues and 217 states generated by NN-guided Greedy Search. Neural uncertainty remains relatively stable across real congeneric analogues, whereas potentially reward-hacked states exhibit elevated uncertainty even with relatively small changes in heavy-atom count (1.253 vs. 0.816 at ΔHA=1\Delta\text{HA}=1–33). Five MC-Dropout passes achieve an AUROC of 0.883 in distinguishing these two populations. These results suggest that utu_t can provide an early warning signal for potentially unreliable optimization states without exhibiting a comparable increase on real congeneric analogues. Consistent with Sections 4.4 and 5, our objective is to mitigate optimization risk under imperfect evaluators rather than guarantee correct affinity rankings.
  2. Closed-loop acceptance and decision reliability (Table G3c): Auditing UQ OFF vs. UQ ON across 100 CrossDocked2020 trajectories indicates that disabling utu_t (UQ OFF) increases global-redocking failure rates among GREEN candidates from 5.1% to 14.9% (GREEN acceptance 94.9% -> 85.1%), increases the number of committed edits (191→236191 \rightarrow 236; ΔHA:+2.2→+6.6\Delta\text{HA}: +2.2 \rightarrow +6.6), and lowers Joint SR from 60.0% to 50.0%.

References

  • [1] CIDD: Collaborative intelligence for structure-based drug design empowered by LLMs
  • [2] Probe before you edit: Probing-guided molecular optimization for LLM agents in structure-based drug design
  • [3] The maximal and current accuracy of rigorous protein-ligand binding free energy calculations


Response to Reviewer 2WJm (2/2)

W3: Isolating LLM Components from Deterministic Scoring Rules, Memory, and UQ.

We agree that w/o Decision and w/o Verification did not separately isolate the LLM modules from our deterministic rules and statistical memory. We therefore fixed all three LLM roles (Pocket Planner, Memory Summarizer, Evidence Critic) to deterministic neutral values (bLLM=1.0b_{\text{LLM}}=1.0, neutral critic concern; Appendix A.8, B.4) in our 2×22 \times 2 ablation (General Response Table G4). Pure numerical rules (LLM-) only track threshold deficits, leaving mean QED lower (0.494–0.496) and shifting little when UQ is added (55% -> 56% SR). Adding the LLM (LLM+) introduces chemical priors and semantic evidence (Ctchem,CtevidC_t^{\text{chem}}, C_t^{\text{evid}}) that elevate mean QED (0.526–0.530) and SA (0.700 in LLM+ UQ-), but without UQ (LLM+ UQ-), active LLM-guided editing lacks an OOD boundary signal and accumulates excess growth (HA 29.2, 50% SR). Pairing UQ with the LLM (LLM+ UQ+) curbs over-growth (HA 24.8) while retaining higher QED (0.526 vs. 0.496), raising Joint SR by +10 pp (50% -> 60%) over LLM+ UQ- and +4 pp over LLM- UQ+.

W4: Computational Efficiency and Head-to-Head Resource Comparison.

We profiled docking evaluations, LLM usage, neural UQ overhead, and wall-clock time across the 100 CrossDocked2020 targets under the aligned fragment space, comparing UMO-Agent directly against PROBE [1] (UMO-Agent median per-target usage is 10.0 full dockings, 7.0 LLM calls, and 7.9 min wall-clock time [mean 19.2 min]):

Per-Target Resource MetricPROBE [1]UMO-AgentEfficiency Gain
Full Docking Evaluations28.213.92.0×2.0\times fewer
LLM API Calls63.98.97.2×7.2\times fewer
Total LLM Tokens632,62318,76733.7×33.7\times fewer
5-Pass MC-Dropout Inference—<0.5 sNegligible
End-to-End Wall-Clock (Median)28.6 min7.9 min3.6×3.6\times faster

W5: Additional Target Benchmark (FEP+), Objective Balance, and Stricter Thresholds.

To evaluate generalizability beyond CrossDocked2020 and fixed cutoffs, we added the 100-target FEP+ benchmark (General Response Table G1b) and five stricter post-hoc threshold regimes alongside the standard setting (Table G2). On FEP+ (Table G1b), UMO-Agent reaches 81.0% Joint SR (vs. 63.0% for PROBE and 56.0% for Reference) with stronger Vina (−9.860-9.860 vs. −9.295-9.295) and higher SA (0.7360.736 vs. 0.7110.711). Mean QED is lower than PROBE (0.569 vs. 0.612), and mean SA is lower than Reference (0.736 vs. 0.775). Under unseen stricter cutoffs (Table G2), UMO-Agent more than doubles Reference's pass rate on Both Strict (16.0% vs. 7.0%) and stays comparable to or slightly ahead of PROBE across 5 of the 6 regimes.

We appreciate your constructive feedback and hope these additional experiments and clarifications help address the concerns raised in the review.

References

  • [1] Probe before you edit: Probing-guided molecular optimization for LLM agents in structure-based drug design


Reviewer_uLHtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes UMO-Agent, a training-free framework for uncertainty-guided molecular optimization. It combines LLM-guided fragment editing, Vina docking verification, MC Dropout uncertainty estimation, and experience memory. A risk controller determines whether to accept candidate modifications, while verified outcomes guide subsequent search. On 100 test pockets from CrossDocked2020, the method achieves a joint success rate of 60% ± 3%.

Critical strengths

The paper addresses an important challenge in sequential molecular optimization: unreliable evaluation feedback may propagate through subsequent search decisions. The authors integrate candidate screening, docking-based verification, risk control, and experience memory into a unified framework. The deterministic acceptance mechanism also provides explicit control over molecular updates. The experiments report molecular growth and ligand efficiency alongside conventional molecular-property metrics, and the component ablations provide useful evidence supporting the overall framework.

Critical weaknesses

1、A central claim of the paper is that reliability information guides candidate acceptance, experience memory, and subsequent search. Although removing MC Dropout reduces the joint success rate from 60% to 50%, the experiments do not directly establish how predictive uncertainty affects risk classifications, acceptance decisions, or subsequent sampling behavior. Furthermore, Section 4.4 demonstrates that Vina and the neural affinity predictor can produce incorrect affinity rankings, but does not examine whether MC Dropout uncertainty can identify these errors. Consequently, the existing ablation suggests that uncertainty information provides additional benefits, but does not sufficiently explain whether and how it improves the reliability of optimization decisions.

2、Table 1 shows that UMO-Agent achieves the highest joint success rate (60%), but does not achieve the best mean Vina, QED, or SA scores. A similar pattern appears in Table 2: removing MC Dropout slightly improves all three mean metrics, while joint success decreases from 60% to 50%. Although the full method performs better in terms of molecular growth and ligand efficiency, these results suggest that its advantage is primarily reflected in satisfying the joint constraints rather than consistently improving individual molecular properties. Moreover, Appendix A.4 shows that the optimization procedure explicitly uses the same Vina, QED, and SA thresholds that define the final success metric. While this is a reasonable goal-directed optimization strategy, it raises the question of whether the reported advantage primarily reflects optimization toward these particular thresholds. The existing w/o MC-Dropout ablation does provide evidence of additional gains under the same objective, but it remains unclear whether these gains generalize to other constraint thresholds or broader multi-objective optimization settings.

Questions for authors

1、Could the authors examine whether MC Dropout uncertainty is associated with candidate verification failures or actual affinity prediction errors? In particular, could the experimental affinity data used in Section 4.4 be leveraged to evaluate whether the uncertainty estimates help identify unreliable predictions?

2、Could the authors report how frequently predictive uncertainty changes risk classifications, candidate acceptance decisions, or subsequent edit-sampling probabilities? Additionally, could a targeted ablation distinguish reliability-aware acceptance from reliability-guided memory and sampling, thereby clarifying the benefits of coupling these feedback pathways?

3、Could the authors explain why removing MC Dropout slightly improves the mean Vina, QED, and SA metrics while reducing joint success rate? Reporting individual constraint satisfaction rates, per-target property changes, and the distributions of molecular properties could help clarify whether the full method improves the balance among objectives.

4、Could the authors evaluate whether the reported advantages remain consistent under alternative Vina, QED, and SA thresholds? A sensitivity analysis or Pareto-based evaluation would help establish whether the benefits extend beyond the particular joint success criterion used in the current experiments

2 comments

Response to Reviewer uLHt (1/2)

Thanks for your detailed comments and questions. Below we address Weaknesses 1–2 in Part 1 and Questions 1–4 in Part 2.

W1: Role of Predictive Uncertainty in Verification, Memory, and Ranking Errors.

We agree that our original ablations did not directly display how predictive uncertainty (utu_t) alters internal risk states or behaves on the Section 4.4 congeneric series. As noted in Sections 4.4 and 5, we do not claim to guarantee true physical affinity rankings, nor does a fast, practical alternative exist for accurate affinity estimation during iterative search (even costly FEP+ reaches only median Spearman ρ=0.62\rho = 0.62 in Figure 2b); instead, UMO-Agent aims to reduce sequential optimization risk under unstable evaluation tools by combining utu_t with paired global redocking (gtcertg_t^{\text{cert}}), LLM evidence, and cross-evaluator disagreement. To examine how utu_t contributes in practice, we conducted the analyses in General Response Section 3 (Tables G3a & G3c): utu_t stays low and flat on real shared-scaffold analogues across ΔHA\Delta\text{HA} while rising on greedy-generated states (Table G3a, detailed in Q1), and in closed-loop search (Table G3c, detailed in Q2), enabling utu_t improves GREEN-conditional redocking acceptance (85.1% -> 94.9%), curbing excess molecular growth (+6.6 -> +2.2 HA), and raising Joint SR from 50.0% to 60.0%.

W2: Divergence Between Marginal Property Means and Joint SR, and Generalization Across Thresholds.

First, single-property averages and pass rates / Joint SR follow different trends in Tables 1–2 because arithmetic means are sensitive to high-growth outliers whereas Joint SR requires the same molecule to satisfy Vina, QED, and SA together: as shown in the pass-rate breakdown in Q3, removing MC-Dropout increases heavy-atom growth, slightly raising sample means on easy targets while lowering all three individual pass rates as well as Joint SR. Second, to test generalization beyond these cutoffs without retuning internal parameters, we evaluated UMO-Agent under the standard and five stricter evaluation settings (General Response Table G2).


Response to Reviewer uLHt (2/2)

Q1: Behavior of MC-Dropout Uncertainty on Verification Failures and Section 4.4 Experimental Series.

We clarify that MC-Dropout uncertainty is not intended to directly identify incorrect affinity rankings. As shown in Section 4.4, even computationally expensive FEP+ achieves a median Spearman correlation of only 0.62 when ranking congeneric ligands. Given these limitations, our objective is to assess whether a proposed improvement is sufficiently reliable to accept, rather than determine whether its predicted affinity ranking is correct.

As shown in Tables G3a and G3b, neural uncertainty remains relatively stable on real congeneric analogues but increases substantially on NN-guided greedy-generated states, even under relatively small changes in heavy-atom count. Five MC-Dropout passes achieve an AUROC of 0.883 in distinguishing these two populations. In closed-loop optimization (Table G3c), enabling utu_t reduces GREEN-conditional verification failure rates from 14.9% to 5.1%. These results support the use of neural uncertainty as a complementary risk signal, rather than a direct detector of affinity ranking errors.

Q2: Frequency of Risk, Acceptance, and Sampling Changes, and Decoupling Feedback Pathways.

In General Response Table G3c, enabling utu_t raises GREEN-conditional acceptance (85.1% -> 94.9%) while reducing verified steps (428 -> 341), committed edits (236 -> 191), and molecular growth (ΔHA:+6.6→+2.2\Delta\text{HA}: +6.6 \rightarrow +2.2), with Joint SR improving from 50% to 60%. As the resulting trajectories differ, these aggregate statistics do not isolate individual state-level decisions. In Table 2, removing the evidence-based RED gate (w/o Verification) leads to excess growth (ΔHA=+6.5\Delta\text{HA} = +6.5, 43% SR), whereas removing memory (w/o Memory) retains relatively compact molecules (ΔHA=+2.6\Delta\text{HA} = +2.6) but requires more steps (3.2 -> 4.8) and reaches 51% SR.

Q3: Individual Constraint Pass Rates and Why Removing MC-Dropout Alters Marginal Means vs. Joint SR.

Comparing individual constraint pass rates (Vina < -8.18, QED > 0.25, SA > 0.59) alongside marginal means on CrossDocked2020 (Table 2) clarifies this trend:

Variant (Table 2)Vina ↓\downarrowQED ↑\uparrowSA ↑\uparrowΔHA\Delta\text{HA}Vina PassQED PassSA PassJoint SR ↑\uparrow
w/o MC-Dropout-8.6500.5300.700+6.657%88%86%50%
Full (UMO-Agent)-8.6360.5260.683+2.264%92%89%60%

Full achieves higher pass rates on all three individual constraints as well as Joint SR, with 60 of the 64 affinity-passing ligands also satisfying QED and SA (mean ΔVina=−1.19\Delta\text{Vina} = -1.19 kcal/mol, ΔQED=+0.046\Delta\text{QED} = +0.046; full per-target distributions will be added to Appendix A.10). In w/o MC-Dropout, larger heavy-atom growth degrades ligand efficiency (−0.309→−0.289-0.309 \rightarrow -0.289): extreme scores on a subset of easier targets pull the arithmetic means slightly higher, while more borderline targets miss individual and joint cutoffs.

Q4: Sensitivity Analysis Under Alternative and Stricter Vina, QED, and SA Thresholds.

Evaluating our CrossDocked2020 molecules post-hoc under the standard and five stricter evaluation settings without retuning internal parameters (General Response Table G2), UMO-Agent more than doubles Reference's pass rate on Both Strict (16.0% vs. 7.0%) and stays comparable to or slightly above PROBE on 5 of the 6 settings (including Vina < -8.50: 46.0% vs. 43.0%, and Vina < -9.00: 35.0% vs. 32.0%).

We appreciate your constructive feedback and hope these additional experiments and clarifications help address the concerns raised in the review.


Comments on the paper

3 comments

General Response: Additional Experiments and Clarifications (1/3)

Thanks for your detailed comments and constructive suggestions. Below we present four sets of additional experiments across three parts addressing shared questions across reviews, with point-by-point responses in our individual replies.

1. Controlled Lead-Optimization Comparisons on CrossDocked2020 and FEP+

Reviewers 2WJm W1, W5; jDnx W2; AC83 W1, W2

Most de novo and prior-guided methods in Table 1 (RGA, DecompOpt, MolJO) generate molecules from scratch and cannot take an existing bound ligand as input for iterative optimization. For direct lead-optimization comparisons, we therefore compare against the two closely related ligand-editing baselines: CIDD [1] and PROBE [2]. Because CIDD's repository omits key required files (UniMAP and its fragment bank), it cannot be re-run under new settings. For a fair comparison on CrossDocked2020, we aligned our setup to CIDD, using the same 100 starting ligands (x0x_0) and 5 Å Vina evaluation bounding box, and evaluated PROBE under the identical x0x_0, 5 Å box, and aligned fragment space (≤6\le 6 HA) (Table G1a, kept separate from our 12 Å Table 1 benchmark). On the external FEP+ dataset [3], where CIDD cannot be run, we randomly sampled 100 complexes and compared UMO-Agent head-to-head against PROBE from identical starting ligands (Table G1b).

  • CrossDocked2020 (Table G1a): UMO-Agent achieves 58.0% Joint SR and 0.542 QED, outperforming CIDD (37.04% SR) and PROBE (52.0% SR, 0.510 QED) with comparable SA and Vina.
  • External FEP+ benchmark (Table G1b): UMO-Agent achieves 81.0% Joint SR (vs. 63.0% for PROBE and 56.0% for Reference), with greater Vina-score improvement (+1.022+1.022 vs. +0.457+0.457) and higher SA (0.7360.736 vs. 0.7110.711) than PROBE, although its mean QED is lower (0.5690.569 vs. 0.6120.612).

Table G1a. Controlled lead-optimization comparison on CrossDocked2020 (100 targets, identical x0x_0, 5 Å box).

MethodVina ↓\downarrowQED ↑\uparrowNorm. SA ↑\uparrowJoint SR (%) ↑\uparrow
Reference-7.3760.4740.72725.0%
CIDD (hit-to-lead*)-8.2200.5280.75237.04%
PROBE-8.3800.5100.68852.0%
UMO-Agent-8.3540.5420.68858.0%

*Published hit-to-lead result from [1] using the same starting ligands and 5 Å Vina box.

Table G1b. Controlled comparison on 100 randomly sampled targets from the FEP+ benchmark (identical x0x_0).

MethodVina ↓\downarrowQED ↑\uparrowNorm. SA ↑\uparrowHAΔVina\Delta\text{Vina}Joint SR (%) ↑\uparrow
Reference-8.8380.6140.77524.7+0.00056.0%
PROBE-9.2950.6120.71125.0+0.45763.0%
UMO-Agent-9.8600.5690.73627.3+1.02281.0%

2. Post-Hoc Evaluation Under Unseen Stricter Thresholds

Reviewers uLHt W2, Q3, Q4; 2WJm W5; jDnx Q1; AC83 W1, W2, W4, Q1, Q2

To separate internal optimization criteria (Vina<−8.18,QED>0.25,SA>0.59\text{Vina} < -8.18, \text{QED} > 0.25, \text{SA} > 0.59) from evaluation cutoffs, we re-evaluated the CrossDocked2020 molecules from Table G1a post-hoc under unseen stricter property thresholds (Strict QED > 0.50, Strict Raw SA < 3.0 [normalized SA >0.778>0.778], and Both Strict) and stricter affinity thresholds (Vina < -8.50 and Vina < -9.00) without retuning internal parameters (Table G2). Despite never targeting these stricter cutoffs during search, UMO-Agent more than doubles Reference's pass rate on Both Strict (7.0% -> 16.0%) and remains comparable to or slightly ahead of PROBE across 5 of the 6 evaluation settings.

Table G2. Joint SR under standard and five stricter post-hoc evaluation settings on CrossDocked2020 (100 targets, 5 Å box). Unspecified thresholds retain their standard values.

MethodStandard SRStrict QED (>.50)Strict SA (Raw<3.0)Both Strict (QED>.50 & SA<3.0)Vina<-8.50Vina<-9.00
Reference25.0%11.0%12.0%7.0%24.0%19.0%
PROBE52.0%36.0%21.0%14.0%43.0%32.0%
UMO-Agent58.0%37.0%19.0%16.0%46.0%35.0%

References

  • [1] CIDD: Collaborative intelligence for structure-based drug design empowered by LLMs
  • [2] Probe before you edit: Probing-guided molecular optimization for LLM agents in structure-based drug design
  • [3] The maximal and current accuracy of rigorous protein-ligand binding free energy calculations


General Response: Additional Experiments and Clarifications (2/3)

3. Uncertainty Quantification: Reward-Hacking Discrimination, Pass Sensitivity, and Closed-Loop Audit

Reviewers uLHt W1, Q1, Q2; 2WJm W2; jDnx W1; qqkz W1, W2, W4

3.1 Distinguishing Normal Analogues from Greedy-Generated Molecules (Table G3a). To examine whether MC-Dropout uncertainty (ut=σu_t = \sigma) can identify potentially reward-hacked states, we compare two molecular populations from the same FEP+ dataset [1]: (1) real congeneric analogues (1,254 synthesized ligands across 34 series in Section 4.4); and (2) 217 states generated through NN-guided Greedy Search without verification or UQ. We perform three editing steps to approximate the typical optimization horizon of UMO-Agent (3.2 steps on average in Table 2).

We bin both populations by heavy-atom difference (ΔHA\Delta\text{HA}) relative to their respective reference ligands. Uncertainty remains relatively stable across real congeneric analogues. In contrast, NN-guided greedy-generated states exhibit elevated uncertainty even with relatively small molecular modifications (1.253 vs. 0.816 at ΔHA=1\Delta\text{HA}=1–33). These observations suggest that neural uncertainty can provide an early warning signal for potentially unreliable optimization states without exhibiting a comparable increase on real congeneric analogues.

Table G3a. 5-pass MC-Dropout std (σ\sigma) on real congeneric analogues [1] vs. NN-guided greedy-generated states across ΔHA\Delta\text{HA}.

ΔHA\Delta\text{HA}Sec. 4.4 Avg. σ\sigmaGreedy Avg. σ\sigma
00.7920.810
1–30.8161.253
4–60.8321.757
7–100.8642.828
11–200.751—

3.2 Forward-Pass Sensitivity (Table G3b). Using the same molecular populations as Table G3a, we evaluate uncertainty-based discrimination across different MC-Dropout pass counts (T∈{3,5,10,20,30}T \in \{3, 5, 10, 20, 30\}). AUROC measures discrimination between real analogues and greedy-generated states. The relatively stable AUROC and variance ratios across pass counts suggest that T=5T=5 already provides a useful uncertainty signal.

Table G3b. Sensitivity of uncertainty-based discrimination to MC-Dropout forward passes.

Passes (TT)T=3T=3T=5T=5 (Default)T=10T=10T=20T=20T=30T=30
AUROC0.8760.8830.8920.9050.908
Avg. Variance σ2\sigma^2 (Nat. / Gen.)0.86 / 7.731.02 / 8.231.23 / 8.681.22 / 9.381.26 / 10.34
Variance Ratio (Gen. / Nat.)9.0×8.0×7.1×7.7×8.2×

3.3 Closed-Loop Trajectory Audit (Table G3c). With all other optimization components fixed, enabling UQ changes the resulting closed-loop trajectories: verified steps decrease from 428 to 341, committed edits from 236 to 191, and mean heavy-atom growth from +6.6 to +2.2, while Joint SR increases from 50% to 60%. GREEN-conditional acceptance also increases from 85.1% to 94.9%. These results indicate that UQ affects overall search dynamics and final outcomes. As the stochastic trajectories visit different states, these statistics do not isolate matched-state decision flips.

Table G3c. Closed-loop trajectory audit (UQ OFF vs. UQ ON, 100 CrossDocked2020 targets).

Closed-Loop Audit MetricUQ OFF (w/o MC-Dropout)UQ ON (Full UMO-Agent)
Verified Steps / Committed Edits428 / 236341 / 191
GREEN Share / Cond. Acceptance11.0% / 85.1%11.4% / 94.9%
YELLOW Share / Cond. Acceptance62.6% / 73.1%60.7% / 74.4%
RED Share / Cond. Acceptance26.4% / 0.0%27.9% / 0.0%
Final Mean ΔHA\Delta\text{HA} / Joint SR+6.6 / 50.0%+2.2 / 60.0%

References

  • [1] The maximal and current accuracy of rigorous protein-ligand binding free energy calculations


General Response: Additional Experiments and Clarifications (3/3)

4. Orthogonal 2×22 \times 2 Ablation of LLM Modules and Neural UQ

Reviewers 2WJm W3; AC83 W3, Q3, Q4, Q5; qqkz W1

To isolate the LLM modules from our deterministic verification rules and statistical memory, we replaced all three LLM roles (Planner, Summarizer, Critic) with hard-coded neutral fallbacks (bLLM=1.0b_{\text{LLM}}=1.0, neutral critic concern; Appendix A.8, B.4) and crossed this with neural UQ (LLM ± ×\times UQ ±) on CrossDocked2020 (Table G4).

  • Findings from Table G4: Pure numerical rules (LLM-) only track threshold deficits, leaving mean QED lower (0.494–0.496) and changing little when UQ is added (55% -> 56% SR). Adding the LLM (LLM+) introduces chemical priors and semantic evidence that elevate mean QED (0.526–0.530) and SA (0.700 in LLM+ UQ-), but without UQ (LLM+ UQ-), active LLM-guided editing lacks the additional uncertainty-based risk signal and accumulates excess growth (HA 29.2), lowering pass rates (50% SR). Pairing the LLM with UQ (LLM+ UQ+) curbs over-growth (HA 24.8) while retaining the LLM's drug-likeness gains (QED 0.526, 92% pass), raising Joint SR by +10 pp (50% -> 60%) over LLM+ UQ- and +4 pp over LLM- UQ+.

Table G4. Orthogonal 2×22 \times 2 ablation of LLM modules and neural UQ on CrossDocked2020.

Variant (LLM ×\times UQ)Vina ↓\downarrowQED ↑\uparrowSA ↑\uparrowHAVina PassQED PassSA PassJoint SR ↑\uparrow
LLM- UQ- (Rules + Memory)-8.5320.4940.68525.562%89%89%55%
LLM- UQ+ (Rules + Memory + UQ)-8.5830.4960.68625.864%89%90%56%
LLM+ UQ- (w/o MC-Dropout)-8.6500.5300.70029.257%88%86%50%
LLM+ UQ+ (Full UMO-Agent)-8.6360.5260.68324.864%92%89%60%


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.