- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3(+1)
- Soundness
- 2(+1)
Read review
Paper summary
DrugHarness integrates scientific tools and code execution for computational drug discovery. It learns conditional skills from execution experience, reviews them through an independent agent, and revises them using subsequent feedback. Evaluations cover molecular screening, optimization, and constrained design, including retrospective skill assessment and prospective HPK1 inhibitor design with experimental validation.
Critical strengths
-
DrugHarness provides a functional computational drug discovery harness that integrates scientific tools and code execution with conditional skill learning, enabling agents to derive reusable guidance from execution experience, review its evidence and applicability, and refine it through subsequent feedback.
-
The evaluation covers diverse molecular tasks, including screening, affinity comparison, molecular editing, optimization, and constrained design, demonstrating the harness’s capabilities across computational drug discovery workflows. A prospective HPK1 study further supports its practical utility through experimental validation of a designed compound.
Critical weaknesses
-
The novelty and domain-specific benefits of skill learning are insufficiently established. Experience-based guidance and iterative refinement have been explored in ExpeL [1] and ACE [2]. The evaluation does not clearly establish which proposed components improve particular drug-discovery tasks. It would be helpful to provide detailed analyses of skill-learning trajectories, showing how guidance evolves and affects scientific decisions, including cases where it helps or hurts performance.
-
The evaluation does not adequately support the claimed performance advantages. The baseline coverage is limited, notably lacking controlled comparisons with general coding agents, such as Claude Code and Codex, without DrugHarness. Table 1 also does not evaluate the original SMDD harness with GPT-5.6 Terra, making backbone and harness improvements difficult to distinguish. Furthermore, DrugHarness’s lead-optimization results use 40 selected tasks, whereas the reported baselines use the full 340-task set. The authors need to compare DrugHarness with these baselines using the same models, tasks, and resource limits to show whether the harness itself improves performance.
[1] Zhao, A., et al. ExpeL: LLM Agents Are Experiential Learners. AAAI, 2024.
[2] Zhang, Q., et al. Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models. ICLR, 2026.
Questions for authors
- Could the authors clarify the technical novelty of the proposed skill-learning framework beyond established experience-based learning and refinement schemes?
- Could the authors provide a detailed skill-learning trajectory for a specific drug-discovery task, showing how guidance changes, when its applicability conditions matter, and how it improves scientific decisions beyond execution reliability?
- Could the authors explain why Tables 1–2 report only results without evolution, and whether learned skills improve performance on SMDD-Bench and with the strongest backbone?
- Could the authors investigate why evolution reduces virtual-screening Hit@3 from 60% to 48% and mean Hits@3 from 0.72 to 0.64 (Table 3)? This raises concerns that some learned guidance may be counterproductive.
- Could the authors evaluate the original SMDD harness with GPT-5.6 Terra to distinguish the contribution of the harness from that of the backbone?
- Could the authors compare DrugHarness with Claude Code or Codex without DrugHarness, using the same backbone and resource limits?
- Did the authors rerun the baselines on the same 40 lead-optimization tasks used to evaluate DrugHarness? 7. Table 1 reports baseline results from Han et al. (2026), while DrugHarness is evaluated on a 40-task lead-optimization subset of the original 340 tasks, as specified in section 4.1.
- Could the authors specify Biomni’s backbone model? Biomni may be outdated; benchmarking against Phylo with a matched backbone would help make the comparison fairer and more up to date.
13 comments
Response to Reviewer pDXC [2/6]: W1 & Q2
W1 & Q2: How learned guidance shapes decisions on a new target (Part 2/2)
We appreciate your request for a detailed trajectory showing when applicability conditions affect scientific decisions. Continuing Part 1/2 on method and comparative evidence, this reply follows the affinity skill from development feedback to its use on a target excluded from development.
1. Development feedback produces a reusable evidence procedure
The central problem was converting retrieved records into usable measurements for the current candidates. Task 029 provided one source experience; the learner also used other selected development experiences and diagnostic feedback.
| Stage | Observation and change | Contribution to the skill |
|---|---|---|
| Development experience | In 029, the agent did not match candidates to measurements and answered incorrectly using properties and structural reasoning. | Identify the need to connect retrieved records to the actual candidates before comparing them. |
| Skill induction | The learner developed complete-record retrieval, target verification, structure/identifier matching, and measurement comparison, with supporting scripts. | Specify how to obtain and associate the evidence needed for a comparison. |
| Review and revision | A real-data trial found that a molecule's presence was reported as a measurement match despite having no activity record. The learner separated these states; a repeated trial with the same inputs confirmed the correction. | Report a missing measurement explicitly even when the molecule's identity has been found. |
The resulting full skill combined written guidance with matching scripts. Its evidence procedure can be summarized as three checks:
-
Candidate identity: Determine whether a record corresponds to the specific candidate, including its stereochemistry. A match in atom connectivity alone can remain a search lead.
-
Measurement availability: Check that the record contains an activity measurement for that candidate and target. Finding the molecule in a source is not itself a measurement.
-
Comparison: Check the measurement endpoints and units, then compare usable values in the direction requested by the task. If a preview does not resolve the candidate matches, retrieve and examine the complete records.
These checks connect the learning process to later execution: the skill specifies both what qualifies as evidence and what to do when the evidence is incomplete. The skill was then frozen for testing on targets outside development.
2. Candidate identity determines which measurements enter the comparison
Task 022, a MAP3K14 affinity comparison outside the development set, illustrates the first check in action. It asks which of two molecules binds more weakly. A and B have the same atom connectivity but different stereochemical configurations. They are distinct candidates, so a measurement for one cannot be assigned to the other merely because the connectivity matches.
The agent used the full skill's matching procedure to associate records with the specific candidate structures. It obtained a Ki of 0.16 nM for A and 634 nM for B, from the same publication. For each candidate, records matching its connectivity but not its stereochemistry were retained as search leads and excluded from that candidate's direct measurement comparison. The identity condition therefore determined which records could supply the values being compared.
With those assignments established, the agent correctly selected B. The historical no-skill run had analyzed stereochemistry and drawn analogies to a crystal structure but incorrectly selected A. In the full-skill run, the learned evidence-selection procedure was actually used to link each candidate to its own measurement, making the final choice depend on their measured affinity order.
3. Candidate-matched measurements guide an external OX2R decision
The same frozen skill was applied in the MoleculeACE evaluation introduced in Part 1/2. An orexin receptor 2 (OX2R) task asked for the weaker binder. Both arms obtained the same 12,926 activity records, including measurements for the current candidates.
The no-skill run inferred a trend from other analogues with Ki values of 1.3 and 4.5 nM and incorrectly selected A. The full-skill agent executed the matching procedure, linking A and B to their own Ki records of 620 and 6,200 nM, and correctly selected B.
Here, the learned matching requirement determined which measurements supported the comparison. The procedure developed from MolBench experience was used to select candidate-specific evidence for an external scientific decision.
Response to Reviewer pDXC [3/6]: W2 & Q5–Q7
W2 & Q5–Q7: Matched-backbone, common-task, and native Codex comparisons
Thank you for asking for comparisons using matched backbones and task sets. We report the completed original-SMDD/Terra comparison, summarize the official baselines on the same 40 lead-optimization tasks, and add a native Codex baseline on MolBench.
1. W2 & Q5: Comparing the original SMDD harness and DrugHarness with Terra
The completed comparison below uses GPT-5.6 Terra for both the original SMDD harness and DrugHarness (DH) with Codex. DH uses the no-evolution configuration. The comparison covers Types 1–5, with lead optimization evaluated on the same 40-task subset. Entries are task success rates.
| Configuration | Type1 | Type2 | Type3 | Type4 | Type5 |
|---|---|---|---|---|---|
| Original SMDD + Terra | 16.0% | 0.0% | 9.6% | 50.0% | 0.0% |
| DH + Codex + Terra | 12.0% | 28.0% | 7.7% | 60.0% | 0.0% |
With the same Terra model, DH + Codex achieves 28.0% versus 0.0% for the original SMDD harness in interaction point discovery (Type2), and 60.0% versus 50.0% in lead optimization (Type4), gains of 28 and 10 percentage points.
2. W2 & Q7: Reaggregating official baselines on the same 40 tasks
For lead optimization, we additionally recalculated the seven published SMDD baselines on these same 40 tasks using the authors' released per-task evaluation records. Matching by task ID yielded all 40 records for each configuration.
| Original SMDD model | Successful tasks | Success rate |
|---|---|---|
| Claude Sonnet 4.6 | 19/40 | 47.5% |
| GPT-5.4 | 22/40 | 55.0% |
| Gemini 3.1 Pro | 19/40 | 47.5% |
| Kimi K2.5 Thinking | 13/40 | 32.5% |
| DeepSeek V3.2 | 9/40 | 22.5% |
| MiniMax M2.7 | 11/40 | 27.5% |
| Qwen3.5 397B A17B | 18/40 | 45.0% |
On this subset, DH + Codex + Terra achieves 24/40 (60.0%), while the best of the seven published configurations reaches 22/40 (55.0%).
3. W2 & Q6: Native Codex with the same Qwen model and task budget
We ran native Codex + Qwen3.7-Max on all 190 MolBench tasks. It could browse the web, install dependencies, and execute code. All three Qwen configurations use the same model and Codex CLI version, with a common default task budget of one hour. The native Codex column is from the new runs; the two DH columns retain the results reported in the original submission.
| Metric | Native Codex | DH w/o evolution | DH + frozen skills |
|---|---|---|---|
| Screening accuracy ↑ | 98.0% | 96.0% | 100.0% |
| Affinity accuracy ↑ | 56.8% | 62.2% | 81.1% |
| Virtual screening Hit@3 ↑ | 56.0% | 60.0% | 48.0% |
| Virtual screening mean Hits@3 ↑ | 0.64 | 0.72 | 0.64 |
| Virtual screening mean reference rank ↓ | 25.73 | 29.05 | 26.01 |
| Editing accuracy ↑ | 97.4% | 97.4% | 97.4% |
| Optimization positive-improvement rate ↑ | 76.9% | 79.5% | 87.2% |
| Optimization mean Δ ↑ | 0.7345 | 0.6054 | 0.7213 |
| Strict scaffold retention ↑ | 17.9% | 38.5% | 48.7% |
Affinity correctness is 21/37 for native Codex, 23/37 for DH without evolution, and 30/37 with frozen skills. The frozen-skill configuration answers nine more tasks correctly than native Codex, a gain of 24.3 percentage points. It also achieves higher optimization positive-improvement and strict scaffold-retention rates.
Response to Reviewer pDXC [1/6]: W1 & Q1
W1 & Q1: Scientific skill learning and comparative evidence (Part 1/2)
Thank you for asking us to clarify the technical contribution beyond existing experience-based learning. DrugHarness couples skill induction, construction, and review to address evidence applicability, strategy executability, and scientific validation before updates in drug discovery.
1. Q1: Scientific strategy learning for drug discovery
ExpeL and ACE demonstrate reusable learning from experience. DrugHarness addresses the dependence of scientific evidence on specific molecules, targets, and measurement conditions through three linked designs.
Conditional action hypotheses as learning units. The learner separates observations, executor interpretations, and feedback. It compares scientific objects, endpoints, and conditions to induce hypotheses specifying a goal, observable conditions, alternative and recommended actions, sources, and unknowns, retaining the scientific basis and scope of reusable guidance.
Executable skills from action hypotheses. Instructions, references, and accompanying procedures preserve the hypothesis's conditions and evidence. For affinity tasks, this turns evidence retrieval and candidate matching into operations executable in the shared scientific workspace, translating learned guidance into actual evidence processing.
Scientific review governs updates and retains revision evidence. An independent reviewer checks scientific premises, condition observability, and executability, using targeted trials when needed. Admit, reject, or defer decisions govern deployment; review findings and later execution feedback inform revisions to guidance and its applicability.
These mechanisms help the executor obtain and use candidate-specific scientific evidence, reducing decision errors from mismatched records or inapplicable analogies.
2. W1: Cross-target comparisons show gains in decision accuracy
We split the 37 MolBench affinity tasks into 11 development tasks covering 10 targets and 26 test tasks covering 23 other targets. Learning, revision, and version selection remained within the development split; guidance was frozen before testing, and test trajectories and feedback were excluded from learning.
ACE starts from the same original no-skill experiences within the development split, retains its Reflector, Curator, and playbook-update procedure, and completes feedback updates and validation within that split. The ablation removes conditional decision guidance from skill text, references, and scripts, including evidence-admission rules, measurement recommendations, and automatic comparison conclusions. It retains data parsing, structure/identifier matching, unit conversion, and autonomous computation. Full-skill, ACE, and ablation test runs use the same Codex + Qwen3.7-Max executor, scientific-tool versions, inputs, and per-task budgets.
The full skill answers five more tasks correctly than ACE and four more than the ablation, retaining every task either comparator answers correctly. Both the full skill and ablation submit 26 valid answers, so their difference concerns decision correctness. ACE and the historical baseline have equal totals but different outcomes: ACE corrects five historical errors and loses five previously correct answers.
3. W1: The frozen skill improves decisions on external affinity tasks
We further evaluated the same frozen skill on 12 pairwise affinity tasks constructed from 24 distinct molecules in the official test split of MoleculeACE, a public molecular bioactivity prediction benchmark. The tasks covered PIM1, OX2R, GR, and DRD4 (three tasks per target). These targets were excluded from skill development. Structurally similar pairs with clear Ki differences were selected and fixed before execution. Both arms were newly run using Codex + Qwen3.7-Max with matched tool versions, inputs, and per-task budgets. The skill remained frozen; external trajectories and feedback were not used for learning or version selection.
The full skill corrected all four no-skill errors and retained its eight correct answers, improving accuracy by 33.3 percentage points. Improvements covered all four targets. Both arms produced valid answers on all 12 tasks, so the gains again concern scientific decision correctness.