acceptodds
Under review as a conference paper at ICLR 2027

DrugHarness: An Evolvable Agent Harness for Computational Drug Discovery

Abstract

Molecular screening and optimization require agents to coordinate specialized computation and adapt their actions as evidence develops. We introduce DrugHarness, a unified system for composable scientific execution and experience-driven strategy learning. DrugHarness integrates over 80 scientific operations with biomedical resources and executable scientific code, and translates execution experience into skills with explicit applicability conditions. An independent review agent examines candidate skills against source evidence and targeted scientific trials when needed; admitted skills guide subsequent tasks and are revised using their feedback. Without evolution, DrugHarness achieves 91.9% affinity-comparison accuracy and 100% molecular-editing accuracy on MolBench. With frozen learned guidance, the Qwen configuration reaches 81.1% affinity-comparison accuracy on development tasks, compared with a historical no-skill result of 62.2%. In an HPK1 design task, a DrugHarness-designed compound was synthesized and experimentally evaluated by medicinal chemistry collaborators, achieving a cellular IC50 of 14.19\ nM versus 37.47\ nM for the best input reference. These results connect broad scientific execution, experience-derived guidance, and candidate-level computational and experimental evaluation in a unified approach to drug discovery.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
AI agents, drug discovery, scientific reasoning, self-evolving agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pDXCOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3(+1)
Soundness
2(+1)
Read review
Paper summary

DrugHarness integrates scientific tools and code execution for computational drug discovery. It learns conditional skills from execution experience, reviews them through an independent agent, and revises them using subsequent feedback. Evaluations cover molecular screening, optimization, and constrained design, including retrospective skill assessment and prospective HPK1 inhibitor design with experimental validation.

Critical strengths
  1. DrugHarness provides a functional computational drug discovery harness that integrates scientific tools and code execution with conditional skill learning, enabling agents to derive reusable guidance from execution experience, review its evidence and applicability, and refine it through subsequent feedback.

  2. The evaluation covers diverse molecular tasks, including screening, affinity comparison, molecular editing, optimization, and constrained design, demonstrating the harness’s capabilities across computational drug discovery workflows. A prospective HPK1 study further supports its practical utility through experimental validation of a designed compound.

Critical weaknesses
  1. The novelty and domain-specific benefits of skill learning are insufficiently established. Experience-based guidance and iterative refinement have been explored in ExpeL [1] and ACE [2]. The evaluation does not clearly establish which proposed components improve particular drug-discovery tasks. It would be helpful to provide detailed analyses of skill-learning trajectories, showing how guidance evolves and affects scientific decisions, including cases where it helps or hurts performance.

  2. The evaluation does not adequately support the claimed performance advantages. The baseline coverage is limited, notably lacking controlled comparisons with general coding agents, such as Claude Code and Codex, without DrugHarness. Table 1 also does not evaluate the original SMDD harness with GPT-5.6 Terra, making backbone and harness improvements difficult to distinguish. Furthermore, DrugHarness’s lead-optimization results use 40 selected tasks, whereas the reported baselines use the full 340-task set. The authors need to compare DrugHarness with these baselines using the same models, tasks, and resource limits to show whether the harness itself improves performance.

[1] Zhao, A., et al. ExpeL: LLM Agents Are Experiential Learners. AAAI, 2024.
[2] Zhang, Q., et al. Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models. ICLR, 2026.

Questions for authors
  1. Could the authors clarify the technical novelty of the proposed skill-learning framework beyond established experience-based learning and refinement schemes?
  2. Could the authors provide a detailed skill-learning trajectory for a specific drug-discovery task, showing how guidance changes, when its applicability conditions matter, and how it improves scientific decisions beyond execution reliability?
  3. Could the authors explain why Tables 1–2 report only results without evolution, and whether learned skills improve performance on SMDD-Bench and with the strongest backbone?
  4. Could the authors investigate why evolution reduces virtual-screening Hit@3 from 60% to 48% and mean Hits@3 from 0.72 to 0.64 (Table 3)? This raises concerns that some learned guidance may be counterproductive.
  5. Could the authors evaluate the original SMDD harness with GPT-5.6 Terra to distinguish the contribution of the harness from that of the backbone?
  6. Could the authors compare DrugHarness with Claude Code or Codex without DrugHarness, using the same backbone and resource limits?
  7. Did the authors rerun the baselines on the same 40 lead-optimization tasks used to evaluate DrugHarness? 7. Table 1 reports baseline results from Han et al. (2026), while DrugHarness is evaluated on a 40-task lead-optimization subset of the original 340 tasks, as specified in section 4.1.
  8. Could the authors specify Biomni’s backbone model? Biomni may be outdated; benchmarking against Phylo with a matched backbone would help make the comparison fairer and more up to date.
13 comments
AuthorsOpenReview ↗

Response to Reviewer pDXC [1/6]: W1 & Q1

W1 & Q1: Scientific skill learning and comparative evidence (Part 1/2)

Thank you for asking us to clarify the technical contribution beyond existing experience-based learning. DrugHarness couples skill induction, construction, and review to address evidence applicability, strategy executability, and scientific validation before updates in drug discovery.

1. Q1: Scientific strategy learning for drug discovery

ExpeL and ACE demonstrate reusable learning from experience. DrugHarness addresses the dependence of scientific evidence on specific molecules, targets, and measurement conditions through three linked designs.

Conditional action hypotheses as learning units. The learner separates observations, executor interpretations, and feedback. It compares scientific objects, endpoints, and conditions to induce hypotheses specifying a goal, observable conditions, alternative and recommended actions, sources, and unknowns, retaining the scientific basis and scope of reusable guidance.

Executable skills from action hypotheses. Instructions, references, and accompanying procedures preserve the hypothesis's conditions and evidence. For affinity tasks, this turns evidence retrieval and candidate matching into operations executable in the shared scientific workspace, translating learned guidance into actual evidence processing.

Scientific review governs updates and retains revision evidence. An independent reviewer checks scientific premises, condition observability, and executability, using targeted trials when needed. Admit, reject, or defer decisions govern deployment; review findings and later execution feedback inform revisions to guidance and its applicability.

These mechanisms help the executor obtain and use candidate-specific scientific evidence, reducing decision errors from mismatched records or inapplicable analogies.

2. W1: Cross-target comparisons show gains in decision accuracy

We split the 37 MolBench affinity tasks into 11 development tasks covering 10 targets and 26 test tasks covering 23 other targets. Learning, revision, and version selection remained within the development split; guidance was frozen before testing, and test trajectories and feedback were excluded from learning.

ACE starts from the same original no-skill experiences within the development split, retains its Reflector, Curator, and playbook-update procedure, and completes feedback updates and validation within that split. The ablation removes conditional decision guidance from skill text, references, and scripts, including evidence-admission rules, measurement recommendations, and automatic comparison conclusions. It retains data parsing, structure/identifier matching, unit conversion, and autonomous computation. Full-skill, ACE, and ablation test runs use the same Codex + Qwen3.7-Max executor, scientific-tool versions, inputs, and per-task budgets.

ConfigurationCorrectAccuracyValid answers
No skill (historical)20/2676.9%26/26
ACE20/2676.9%25/26
Without conditional decision guidance21/2680.8%26/26
Full skill25/2696.2%26/26

The full skill answers five more tasks correctly than ACE and four more than the ablation, retaining every task either comparator answers correctly. Both the full skill and ablation submit 26 valid answers, so their difference concerns decision correctness. ACE and the historical baseline have equal totals but different outcomes: ACE corrects five historical errors and loses five previously correct answers.

3. W1: The frozen skill improves decisions on external affinity tasks

We further evaluated the same frozen skill on 12 pairwise affinity tasks constructed from 24 distinct molecules in the official test split of MoleculeACE, a public molecular bioactivity prediction benchmark. The tasks covered PIM1, OX2R, GR, and DRD4 (three tasks per target). These targets were excluded from skill development. Structurally similar pairs with clear Ki differences were selected and fixed before execution. Both arms were newly run using Codex + Qwen3.7-Max with matched tool versions, inputs, and per-task budgets. The skill remained frozen; external trajectories and feedback were not used for learning or version selection.

ConfigurationCorrectAccuracyValid answers
No skill (new runs)8/1266.7%12/12
Full skill (new runs)12/12100.0%12/12

The full skill corrected all four no-skill errors and retained its eight correct answers, improving accuracy by 33.3 percentage points. Improvements covered all four targets. Both arms produced valid answers on all 12 tasks, so the gains again concern scientific decision correctness.

AuthorsOpenReview ↗

Response to Reviewer pDXC [2/6]: W1 & Q2

W1 & Q2: How learned guidance shapes decisions on a new target (Part 2/2)

We appreciate your request for a detailed trajectory showing when applicability conditions affect scientific decisions. Continuing Part 1/2 on method and comparative evidence, this reply follows the affinity skill from development feedback to its use on a target excluded from development.

1. Development feedback produces a reusable evidence procedure

The central problem was converting retrieved records into usable measurements for the current candidates. Task 029 provided one source experience; the learner also used other selected development experiences and diagnostic feedback.

StageObservation and changeContribution to the skill
Development experienceIn 029, the agent did not match candidates to measurements and answered incorrectly using properties and structural reasoning.Identify the need to connect retrieved records to the actual candidates before comparing them.
Skill inductionThe learner developed complete-record retrieval, target verification, structure/identifier matching, and measurement comparison, with supporting scripts.Specify how to obtain and associate the evidence needed for a comparison.
Review and revisionA real-data trial found that a molecule's presence was reported as a measurement match despite having no activity record. The learner separated these states; a repeated trial with the same inputs confirmed the correction.Report a missing measurement explicitly even when the molecule's identity has been found.

The resulting full skill combined written guidance with matching scripts. Its evidence procedure can be summarized as three checks:

  • Candidate identity: Determine whether a record corresponds to the specific candidate, including its stereochemistry. A match in atom connectivity alone can remain a search lead.

  • Measurement availability: Check that the record contains an activity measurement for that candidate and target. Finding the molecule in a source is not itself a measurement.

  • Comparison: Check the measurement endpoints and units, then compare usable values in the direction requested by the task. If a preview does not resolve the candidate matches, retrieve and examine the complete records.

These checks connect the learning process to later execution: the skill specifies both what qualifies as evidence and what to do when the evidence is incomplete. The skill was then frozen for testing on targets outside development.

2. Candidate identity determines which measurements enter the comparison

Task 022, a MAP3K14 affinity comparison outside the development set, illustrates the first check in action. It asks which of two molecules binds more weakly. A and B have the same atom connectivity but different stereochemical configurations. They are distinct candidates, so a measurement for one cannot be assigned to the other merely because the connectivity matches.

The agent used the full skill's matching procedure to associate records with the specific candidate structures. It obtained a Ki of 0.16 nM for A and 634 nM for B, from the same publication. For each candidate, records matching its connectivity but not its stereochemistry were retained as search leads and excluded from that candidate's direct measurement comparison. The identity condition therefore determined which records could supply the values being compared.

With those assignments established, the agent correctly selected B. The historical no-skill run had analyzed stereochemistry and drawn analogies to a crystal structure but incorrectly selected A. In the full-skill run, the learned evidence-selection procedure was actually used to link each candidate to its own measurement, making the final choice depend on their measured affinity order.

3. Candidate-matched measurements guide an external OX2R decision

The same frozen skill was applied in the MoleculeACE evaluation introduced in Part 1/2. An orexin receptor 2 (OX2R) task asked for the weaker binder. Both arms obtained the same 12,926 activity records, including measurements for the current candidates.

The no-skill run inferred a trend from other analogues with Ki values of 1.3 and 4.5 nM and incorrectly selected A. The full-skill agent executed the matching procedure, linking A and B to their own Ki records of 620 and 6,200 nM, and correctly selected B.

Here, the learned matching requirement determined which measurements supported the comparison. The procedure developed from MolBench experience was used to select candidate-specific evidence for an external scientific decision.

AuthorsOpenReview ↗

Response to Reviewer pDXC [3/6]: W2 & Q5–Q7

W2 & Q5–Q7: Matched-backbone, common-task, and native Codex comparisons

Thank you for asking for comparisons using matched backbones and task sets. We report the completed original-SMDD/Terra comparison, summarize the official baselines on the same 40 lead-optimization tasks, and add a native Codex baseline on MolBench.

1. W2 & Q5: Comparing the original SMDD harness and DrugHarness with Terra

The completed comparison below uses GPT-5.6 Terra for both the original SMDD harness and DrugHarness (DH) with Codex. DH uses the no-evolution configuration. The comparison covers Types 1–5, with lead optimization evaluated on the same 40-task subset. Entries are task success rates.

ConfigurationType1Type2Type3Type4Type5
Original SMDD + Terra16.0%0.0%9.6%50.0%0.0%
DH + Codex + Terra12.0%28.0%7.7%60.0%0.0%

With the same Terra model, DH + Codex achieves 28.0% versus 0.0% for the original SMDD harness in interaction point discovery (Type2), and 60.0% versus 50.0% in lead optimization (Type4), gains of 28 and 10 percentage points.

2. W2 & Q7: Reaggregating official baselines on the same 40 tasks

For lead optimization, we additionally recalculated the seven published SMDD baselines on these same 40 tasks using the authors' released per-task evaluation records. Matching by task ID yielded all 40 records for each configuration.

Original SMDD modelSuccessful tasksSuccess rate
Claude Sonnet 4.619/4047.5%
GPT-5.422/4055.0%
Gemini 3.1 Pro19/4047.5%
Kimi K2.5 Thinking13/4032.5%
DeepSeek V3.29/4022.5%
MiniMax M2.711/4027.5%
Qwen3.5 397B A17B18/4045.0%

On this subset, DH + Codex + Terra achieves 24/40 (60.0%), while the best of the seven published configurations reaches 22/40 (55.0%).

3. W2 & Q6: Native Codex with the same Qwen model and task budget

We ran native Codex + Qwen3.7-Max on all 190 MolBench tasks. It could browse the web, install dependencies, and execute code. All three Qwen configurations use the same model and Codex CLI version, with a common default task budget of one hour. The native Codex column is from the new runs; the two DH columns retain the results reported in the original submission.

MetricNative CodexDH w/o evolutionDH + frozen skills
Screening accuracy ↑98.0%96.0%100.0%
Affinity accuracy ↑56.8%62.2%81.1%
Virtual screening Hit@3 ↑56.0%60.0%48.0%
Virtual screening mean Hits@3 ↑0.640.720.64
Virtual screening mean reference rank ↓25.7329.0526.01
Editing accuracy ↑97.4%97.4%97.4%
Optimization positive-improvement rate ↑76.9%79.5%87.2%
Optimization mean Δ ↑0.73450.60540.7213
Strict scaffold retention ↑17.9%38.5%48.7%

Affinity correctness is 21/37 for native Codex, 23/37 for DH without evolution, and 30/37 with frozen skills. The frozen-skill configuration answers nine more tasks correctly than native Codex, a gain of 24.3 percentage points. It also achieves higher optimization positive-improvement and strict scaffold-retention rates.

Reviewer_cogpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduce DrugHarness, an evolvable agent harness that couples composable scientific execution with conditional skill induction. DrugHarness induces action hypotheses specifying observable applicability conditions, recommended actions, and supporting evidence. Evaluation on downstream benchmarks demonstrates capabilities of proposed DrugHarness.

Critical strengths
  1. This paper introduces conditional skills, explicitly writes when to use into each skill. This is novel.
Critical weaknesses
  1. Model coverage is incomplete, so the harness effect cannot be attributed. Tables 1 and 2 never run DrugHarness and a competing method on the same backbone. In Table 1, all seven baselines use the SMDD harness with Sonnet 4.6, GPT-5.4, Gemini 3.1 Pro, Kimi, DeepSeek, MiniMax, or Qwen3.5, while both DrugHarness rows use GPT5.6-Terra; harness and model are fully confounded. In Table 2, no baseline is run with GPT-5.6 Terra, yet the abstract's headline claims (+10.8 pp affinity, +55% mean Hits@3) come from that column, compared against MolClaw on Sonnet 4.6. The one backbone-matched comparison (Qwen3.7-Max) is far weaker and mixed: affinity 62.2% vs 54.1%, but mean Hits@3 0.72 vs MolClaw's 0.76 and scaffold-hard retention 38.5% vs 71.8%. Moreover, DrugHarness's own runtime swing (LO 37.5% vs 60.0% at identical model) exceeds its 2.4-point margin over the best baseline, so unreported run-to-run variance dominates every cross-system difference.

  2. No ablations, and conditionality is never tested against unconditional skills. The mechanism has at least four separable parts: applicability conditions with task-time Assess (Eq. 6), the independent review agent (Eq. 5), targeted execution trials, and feedback-driven revision (Eq. 8). None is ablated, and no experience-learning baseline (ACE, SkillOpt, ExpeL, DrugSAGE, TRACE) is run, despite all being discussed in related work. Critically, the paper's conceptual claim is that conditions preserve applicability, but it never compares against the same skill text with conditions stripped or always applied, and never reports the distribution of supported/contradicted/unknown.

Questions for authors
  1. Can you test DrugHarness on other base models?

  2. Can you report the ablation experiments?

4 comments
AuthorsOpenReview ↗

Response to Reviewer cogp [1/3]: W1 & Q1

W1 & Q1: Model coverage and comparisons with matched backbones

We appreciate your emphasis on model coverage and comparisons using the same backbone. The original submission evaluates DrugHarness with Qwen3.7-Max and GPT-5.6 Terra. We now report the completed Terra comparison on SMDD and add a native Codex baseline with Qwen, providing corresponding system comparisons under both backbones.

1. A shared Terra backbone for SMDD comparisons

The original SMDD harness and both DrugHarness (DH) configurations below use GPT-5.6 Terra. The DH rows use the no-evolution setting. The comparison covers Types 1–5, with lead optimization evaluated on the same 40-task subset. Entries are task success rates.

ConfigurationType1Type2Type3Type4Type5
Original SMDD + Terra16.0%0.0%9.6%50.0%0.0%
DH + Hermes + Terra12.0%20.0%3.8%37.5%0.0%
DH + Codex + Terra12.0%28.0%7.7%60.0%0.0%

With the same Terra backbone, DH + Codex achieves 28.0% versus 0.0% for the original SMDD harness in interaction point discovery (Type2), and 60.0% versus 50.0% in lead optimization (Type4), gains of 28 and 10 percentage points.

The Codex–Hermes comparison was designed to examine runtime selection and the interchangeability of execution backends. We use Codex as the primary runtime for code-intensive scientific tasks and evaluate performance with Hermes as an alternative execution backend. The 37.5% and 60.0% results come from the Hermes and Codex configurations, respectively, so their difference is not an estimate of run-to-run variance under a fixed configuration.

2. A native Codex baseline with Qwen on MolBench

We ran native Codex + Qwen3.7-Max on all 190 MolBench tasks. It could browse the web, install dependencies, and execute code. All three Qwen configurations use the same model and Codex CLI version, with a common default task budget of one hour. The native Codex column is from the new runs; the two DH columns retain the results reported in the original submission.

MetricNative CodexDH w/o evolutionDH + frozen skills
Screening accuracy ↑98.0%96.0%100.0%
Affinity accuracy ↑56.8%62.2%81.1%
Virtual screening Hit@3 ↑56.0%60.0%48.0%
Virtual screening mean Hits@3 ↑0.640.720.64
Virtual screening mean reference rank ↓25.7329.0526.01
Editing accuracy ↑97.4%97.4%97.4%
Optimization positive-improvement rate ↑76.9%79.5%87.2%
Optimization mean Δ ↑0.73450.60540.7213
Strict scaffold retention ↑17.9%38.5%48.7%

Affinity correctness is 21/37 for native Codex, 23/37 for DH without evolution, and 30/37 with frozen skills. The frozen-skill configuration answers nine more tasks correctly than native Codex, a gain of 24.3 percentage points. It also achieves higher optimization positive-improvement and strict scaffold-retention rates.

AuthorsOpenReview ↗

Response to Reviewer cogp [2/3]: W2 & Q2

W2 & Q2: Conditional guidance, ACE, and evidence use (Part 1/2)

Thank you for requesting ablations and experience-learning baselines. We compare the full skill with ACE and an ablation of conditional decision guidance, then examine how conditions affect evidence use.

1. Ablation and ACE comparisons on held-out targets

MolBench affinity tasks were split by target: 11 development tasks (10 targets) and 26 test tasks (23 other targets). Tasks sharing a target stay in one split. Guidance was frozen before testing; test trajectories and feedback did not enter learning or version selection.

The ablation retains full-data parsing, structure/identifier matching, unit conversion, and autonomous computation. It removes condition-to-action recommendations, evidence-admission decisions, measurement recommendations, and automatic comparison conclusions from the skill text, references, and scripts.

ACE starts from the same original no-skill experiences within the development split. We retain its Reflector, Curator, and playbook-update procedure, complete feedback updates and validation within that split, and freeze the playbook for testing. Full-skill, ACE, and ablation test runs use Codex + Qwen3.7-Max with matched scientific-tool versions, inputs, and per-task budgets. The no-skill row is historical.

ConfigurationCorrectAccuracyValid answers
No skill (historical)20/2676.9%26/26
ACE20/2676.9%25/26
Without conditional decision guidance21/2680.8%26/26
Full skill25/2696.2%26/26

The full skill retained every task answered correctly by either comparator, adding five correct answers over ACE and four over the ablation (19.2 and 15.4 percentage points). ACE matches the historical baseline's 20/26 total but corrects five prior errors and loses five previously correct answers. The full skill corrects five historical errors and retains all 20 previously correct answers.

In three of the five tasks where the full skill and ACE differed, the complete query results already contained the required measurements, but the ACE runs did not extract and use them for the decision.

Task 030 illustrates how the full skill addressed this gap by executing the learned matching procedure to associate candidates with their measurements.

The task asks for the weaker binder. The database returned 4,155 activity records, with 12 shown in the preview. The full-skill agent obtained the complete file and ran the skill's matching script, matching A and B to Ki records of 2,530 nM and 9.5 nM, respectively. It correctly selected A. ACE treated B's measurement as unavailable and incorrectly chose B using structural/property reasoning. Post-run inspection found B's measurement in row 16 of the complete result. The learned procedure converted available data into candidate-specific measurements used in the final comparison.

2. Conditions affect evidence selection and further investigation

From saved tool outputs and execution records for the 26 full-skill test runs, we annotated two observable applicability conditions: supported when established by evidence, contradicted when evidence shows a mismatch, and unknown when necessary information is missing.

CheckUnitSupportedContradictedUnknown
Do candidate-matched measurements support direct numerical ordering?26 tasks2204
Does a source record support the candidate's exact identity?50 candidate-record relations4451

Note: The first row counts tasks; the second counts observed candidate-record relations. One task can produce multiple such relations.

Conditions affected both further evidence gathering and record selection. Seven tasks initially lacked sufficient candidate measurements; processing the complete data supplied a comparison basis. All 22 tasks with a final direct measurement ordering, including these seven, used it in their answers.

Five candidate-record relations had conflicting stereochemical markings in the saved structures, and one lacked sufficient identity information. None of these six relations was used as direct measurement evidence for the corresponding candidate.

AuthorsOpenReview ↗

Response to Reviewer cogp [3/3]: W2 & Q2

W2 & Q2: Independent scientific review and feedback-driven revision (Part 2/2)

We appreciate your request to examine independent scientific review and feedback-driven skill revision. The deployment comparison below uses the same 26-task MolBench affinity test set as Part 1/2.

1. Independent review identifies defects and guides revision

Skill generation and review run as separate agent tasks with separate contexts. The shared scientific environment lets the review agent reproduce candidate operations and check their outputs against source evidence.

During affinity-skill development, the review agent ran a candidate's evidence-processing script on real database results, identified the following errors, and rejected the candidate. After the learner revised it, a later review repeated the trial with the same data and candidate molecules.

Review findingLearner revisionRepeated-trial result
A molecule was found in the source, but had no activity measurement; the script conflated these states.Separate molecule presence from measurement availability.Identity matched, with the missing activity measurement explicitly reported.
Measurements were read twice and lacked traceable record identifiers.Correct duplicate parsing and preserve measurement identifiers.Twelve duplicate outputs became six distinct, individually traceable records.

We evaluated the saved candidate from before this review round directly on the same 26 affinity test tasks and compared it with the version revised using the review feedback.

ConfigurationCorrectValid answers
Direct use of the pre-review candidate24/2625/26
Revised using review feedback (full skill)25/2626/26

Note: The molecular choices were identical on the 25 tasks with valid answers in both runs; the one-task score difference arose from a missing answer in the candidate run.

Together, these results show how independent review improves evidence processing in reusable skills: it identified defects in determining measurement availability, deduplicating records, and preserving source identifiers, prompted the learner to revise the skill, and verified the repairs through repeated trials with the same inputs. The revised full skill answered 25 of the 26 test tasks correctly, submitted valid answers to all 26, and retained all 24 correct answers obtained with the pre-review candidate.

2. Feedback-driven skill revision improves a subsequent affinity decision

A separate affinity-skill revision case illustrates the role of execution feedback in subsequent scientific decisions. Task 012 asked the agent to choose the weaker binder from two candidate molecules. The agent answered incorrectly with the initial skill (S0) and correctly with the feedback-revised skill (S1). This task was excluded from the learning inputs for this revision.

S0, learned from earlier task experiences, already instructed the agent to base decisions on measurements for the candidate molecules. Subsequent execution feedback exposed a gap: measurements present in the retrieved data were not successfully matched to the candidates and used in the decision. Using selected execution records and review feedback from other tasks, the learner revised S0 into S1 with concrete conditional steps: when the preview lacks measurements matched to the candidates, obtain the complete records, match the candidate structures, and extract the measurements. Independent review trials confirmed that these steps could recover the candidate records.

The two executions of task 012 both retrieved data containing the required measurements. With S0, the agent read only part of the source, failed to link the candidates to their measurements, and incorrectly selected B based on molecular properties and conformations. With S1, it followed the revised guidance to read the complete data and match the candidates, obtaining Ki values of 9,000 nM for A and 64 nM for B. It correctly selected A. The revised retrieval steps were thus used to turn measurements already present in the returned data into evidence for the final comparison.

Dear Reviewer cogp, we appreciate the time and care you devoted to reviewing our work. Following your suggestions, we have added the native Codex comparison, an ACE baseline, and evidence on conditional guidance, independent review, and feedback-driven revision. These are detailed in our posted responses and revised manuscript. We would be grateful for any further feedback when convenient.

Reviewer_rJdJOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

DrugHarness combines a broad scientific tool environment with a loop that induces conditional skills from agent trajectories, checks candidate skills with a separate review agent, and revises admitted guidance using later task feedback. The paper evaluates execution without evolution on MolBench and SMDD-Bench, then evaluates frozen skills retrospectively on MolBench. The integrated workflow is useful and the experimental example is encouraging.

Critical strengths
  1. The method represents source observations, agent interpretations, feedback, and deliverables separately. This is a sensible design for scientific agents, where context and provenance affect whether advice should be reused.

  2. The paper tests several task families rather than presenting a single configuration. The MolBench tables also report metrics that worsen with evolution, which is more informative than an aggregate success number alone.

  3. The HPK1 case connects a computational agent workflow to a measured cellular endpoint. A single compound with Jurkat p-SLP76 IC50 of 14.19 nM versus 37.47 nM for the best input reference is a meaningful proof of feasibility, subject to the study’s small scale.

Critical weaknesses
  1. The strongest cross-system numbers do not isolate the proposed harness. The 91.9% affinity-comparison result uses Codex/GPT-5.6 Terra, whereas the reported MolClaw-CC comparison uses Claude Sonnet 4.6; the external baseline values come from their original protocols (Table 2; Appendix A.1). The Qwen comparison reduces some model mismatch but still mixes different runtimes and tool environments. A same-model, same-runtime comparison that selectively adds the scientific tools and skill mechanism is needed for causal attribution.

  2. The evolution evaluation is explicitly retrospective and in-domain. Table 3 compares historical no-skill runs on MolBench with runs using frozen skills learned from related task experience. It does not establish transfer to new targets or task families, and there is no blinded held-out skill-development/test split clearly demonstrating generalization.

  3. The prospective comparison is too small and partly selected after inspection. Only four system-specific candidates per method remain after removing overlaps; four favorable inhibition endpoints in Figure 2 were chosen post hoc, while DrugHarness has worse predicted hERG inhibition and lower QED. One synthesized candidate supports feasibility but cannot establish a reliable advantage over Biomni or experienced chemists. The candidate structures are withheld, limiting independent reproduction of this example.

  4. The “independent review” step is an agent with access to the same tools and evidence environment. The paper specifies what it checks, but does not quantify review-agent false admissions, false rejections, or whether source verification and targeted trials improve outcomes relative to admitting all proposed skills. This is important to substantiate the proposed governance mechanism.

Questions for authors
  1. Can the authors evaluate skill evolution on held-out targets or task families whose feedback was not used during induction or revision?

  2. What is the effect of disabling the independent review agent, condition checks, and skill revision separately under the same model, runtime, tools, and task budget?

  3. Why does evolution reduce virtual-screening Hit@3 despite improving coverage and mean reference rank? Which candidate decisions changed?

  4. How many HPK1 candidates were proposed, selected for synthesis, and successfully synthesized in total, and what were the prespecified selection criteria and replicate uncertainties for the reported assays?

8 comments
AuthorsOpenReview ↗

Response to Reviewer rJdJ [1/5]: W1

W1: Native Codex and SMDD comparisons with matched backbones

Thank you for highlighting the effects of backbone and runtime differences on the comparisons. Given the API and runtime costs of full scientific-agent evaluations, the original submission used baseline results reported in the MolClaw paper and supplemented them with a comparison using the lower-cost Qwen model to align the backbone. We agree that matching both the model and runtime helps clarify DrugHarness's contribution. We therefore added a native Codex comparison with matched model and runtime settings, and report a second comparison using the same Terra model on SMDD.

1. Native Codex and DrugHarness with the same Qwen model

We ran native Codex + Qwen3.7-Max on all 190 MolBench tasks and compared it with the reported DrugHarness (DH) configurations without evolution and with frozen skills. All three use Qwen3.7-Max and the same Codex CLI version. Native Codex could browse the web, install dependencies, and execute code. All three configurations used the same default one-hour task budget.

The native Codex results are from the new runs; the two DH columns retain the results reported in the original submission.

MetricNative CodexDH w/o evolutionDH + frozen skills
Screening accuracy ↑98.0%96.0%100.0%
Affinity accuracy ↑56.8%62.2%81.1%
Virtual screening Hit@3 ↑56.0%60.0%48.0%
Virtual screening mean Hits@3 ↑0.640.720.64
Virtual screening mean reference rank ↓25.7329.0526.01
Editing accuracy ↑97.4%97.4%97.4%
Optimization positive-improvement rate ↑76.9%79.5%87.2%
Optimization mean Δ ↑0.73450.60540.7213
Strict scaffold retention ↑17.9%38.5%48.7%

With the same Qwen model and Codex runtime, DH without evolution and with frozen skills correctly answer 23/37 and 30/37 affinity tasks, respectively, compared with 21/37 for native Codex. The frozen-skill configuration answers nine more tasks correctly than native Codex, increasing accuracy by 24.3 percentage points. Its optimization positive-improvement rate and strict scaffold-retention rate reach 87.2% and 48.7%, respectively, compared with 76.9% and 17.9% for native Codex.

2. A Terra comparison on the same SMDD tasks

We also report the completed comparison between the original SMDD harness and DrugHarness with Codex, both using GPT-5.6 Terra. The DrugHarness row uses the no-evolution configuration. The comparison covers Types 1–5, with lead optimization evaluated on the same 40-task subset. Entries are task success rates.

ConfigurationType1Type2Type3Type4Type5
Original SMDD + Terra16.0%0.0%9.6%50.0%0.0%
DH + Codex + Terra12.0%28.0%7.7%60.0%0.0%

With the same Terra model, DrugHarness + Codex achieves higher success rates in interaction point discovery (Type2) and lead optimization (Type4): 28.0% versus 0.0%, and 60.0% versus 50.0%, respectively. These are gains of 28 and 10 percentage points.

AuthorsOpenReview ↗

Response to Reviewer rJdJ [2/5]: W2 & Q1

W2 & Q1: Skill transfer to held-out targets and an external data source

We appreciate your suggestion to evaluate skill transfer beyond development tasks. We added two complementary evaluations: target-disjoint testing within MolBench and external affinity tasks derived from the public molecular bioactivity prediction benchmark MoleculeACE, both using the same full skill frozen after MolBench development.

1. Separate development from testing and reuse the frozen skill

We divided the 37 tasks into 11 development tasks covering 10 targets and 26 test tasks covering 23 other targets. All tasks for a given target remain in the same split. Skill learning, revision, and version selection were completed within the development split. We then froze the full skill and evaluated it with Codex + Qwen3.7-Max. Test trajectories and feedback were not used for learning, revision, or version selection.

For external evaluation, we constructed 12 pairwise affinity tasks from 24 distinct molecules in MoleculeACE's official test split, with three tasks each for PIM1, orexin receptor 2 (OX2R), glucocorticoid receptor (GR), and dopamine D4 receptor (DRD4). These targets were excluded from skill development. We selected structurally similar pairs with clear Ki differences and fixed the tasks before execution. Task inputs specified the target, two molecular structures, and comparison direction; the agents obtained evidence through the scientific tools.

We reused the same frozen skill without further learning, revision, or version selection. Both external arms were newly run with Codex + Qwen3.7-Max, matched scientific-tool versions, task inputs, and per-task budgets. External test trajectories and feedback were not used to update the skill.

2. Accuracy improves in both evaluations

MolBench held-out-target results:

ConfigurationCorrectAccuracyValid answers
DrugHarness, no skill (historical)20/2676.9%26/26
DrugHarness, full skill25/2696.2%26/26

The full skill corrects five of the six historical errors and preserves all 20 previously correct answers, increasing accuracy by 19.2 percentage points.

External MoleculeACE-derived task results:

ConfigurationCorrectAccuracyValid answers
DrugHarness, no skill (new runs)8/1266.7%12/12
DrugHarness, full skill (new runs)12/12100.0%12/12

The full skill answered four more external tasks correctly, a gain of 33.3 percentage points, preserving all eight correct no-skill answers. Improvements spanned all four targets. Both arms submitted valid answers for all 12 tasks, so the difference concerns decision correctness.

3. Transferred guidance changes the use of scientific evidence

This reuse across targets involves checking candidate identity and determining which measurements apply. In the MolBench MAP3K14 task (022), whose target was excluded from skill development, the two candidates have identical atom connectivity but different stereochemical configurations, and the task asks which binds more weakly. Using the full skill, the agent matched A and B to Ki records of 0.16 nM and 634 nM, respectively, from the same publication. For each candidate, records corresponding to the other stereoisomer were retained only as search leads and excluded from its direct measurement comparison. The agent correctly selected B; the historical no-skill run had selected A.

Trace analysis of the four corrected external tasks showed that the full skill executed its matching procedures and used measurements for the current candidates. In the OX2R case, both arms retrieved the same scientific records, but the no-skill run used other analogues' measurements to infer the answer. The full skill matched the current candidates to their own Ki records and answered correctly.

Together, the evaluations support reuse of the learned evidence procedures across targets and in tasks from an external data source without further skill learning. We will provide further experimental details and supporting analyses in the revised supplementary material.

AuthorsOpenReview ↗

Response to Reviewer rJdJ [3/5]: W3 & Q4

W3 & Q4: HPK1 candidate selection, experimental validation, and computational comparison

Thank you for recognizing the HPK1 case as evidence of experimental feasibility and for your specific suggestions on candidate selection and assay information. This study examines whether DrugHarness can propose synthesizable candidates that improve cellular potency over the input references under medicinal chemistry constraints. The selected candidate was successfully synthesized and achieved a Jurkat p-SLP76 IC50 of 14.19 nM, lower than the best input reference at 37.47 nM. We provide further details on candidate selection, experimental settings, and computational comparison below.

1. Validation scale and candidate selection

Starting from four experimentally characterized reference compounds, DrugHarness generated ten candidates while preserving the specified core and double-bond geometry and modifying only terminal amide nitrogen substituents. Given the cost and time required for synthesis and subsequent biochemical and cellular assays, this round used a small-scale prospective validation. The medicinal chemistry team considered structural novelty, synthesis steps, and DrugHarness's activity predictions to select one candidate for synthesis. This candidate was successfully synthesized and evaluated in both the HPK1 biochemical inhibition assay and the Jurkat cellular p-SLP76 inhibition assay.

2. Measured activity, analysis, and replication

HPK1 biochemical inhibition was measured using ADP-Glo. Following CD3 antibody stimulation of Jurkat cells, p-SLP76 was detected by ELISA. OD450 readings were converted using a standard curve to calculate inhibition at each concentration, and IC50 was estimated by fitting a four-parameter concentration-response curve. The cellular assay used eight concentrations, with a maximum of 1 μM and one well per concentration. The results and replication settings are as follows:

MeasurementExperimental settingResult
HPK1 biochemical inhibitionOne independent experiment; two technical replicate wells at 1 nMInhibition of 96.96% and 96.87%; mean 96.92%
Jurkat cellular p-SLP76 inhibitionOne independent experiment; eight concentrations, one well per concentrationIC50 from the four-parameter fit: 14.19 nM

The candidate's cellular IC50 was lower than those of all four input references (37.47–227.77 nM). The concentration-response records for the candidate and the best input reference further illustrate this activity difference: for example, at 37.037 nM, their p-SLP76 inhibition rates were 76.53% and 38.61%, respectively. The corresponding IC50 values from the four-parameter fits were 14.19 and 37.47 nM, respectively. The manuscript also presents biochemical and cellular activity comparisons with two selected chemist-designed compounds.

We will include the concentration-response data, fitted curves, and assay methods for the candidate and best input reference in the revised appendix, clearly identifying the positive control and replication settings. Candidate structures remain confidential under the research collaboration and are reported using coded identifiers.

3. Scope and property trade-offs in the computational comparison

Computational evaluation characterized docking and related properties of the generated designs. We evaluated all ten DrugHarness outputs and ten of Biomni's twelve outputs, numbered D01–D10. Six structures were shared; Figure 2 shows the four system-specific candidates per group.

For these groups of four system-specific candidates, DrugHarness had a more favorable median MOE docking score (−8.180 versus −7.825) and lower median inhibition predictions for CYP2C19, CYP2B6, OATP1B1, and OATP1B3. These four endpoints were selected post hoc for display, as stated in the original caption. That caption also reports the higher median hERG inhibition prediction (0.949 versus 0.663) and lower median QED (0.3645 versus 0.4085) for the DrugHarness group.

The task's optimization objective was improved cellular p-SLP76 inhibition; hERG and QED were auxiliary computational characterizations rather than optimization constraints. The synthesized candidate's measured activity and concentration-response records support achievement of that objective. The higher median hERG prediction flags a risk for investigation during subsequent lead optimization; this computed score is not a measured channel-blocking percentage, and cardiac safety requires experimental assessment.

We will also consider expanding prospective validation to more candidates and targets in future versions of DrugHarness.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Response to the Area Chair and Reviewers

Dear Area Chair and Reviewers,

We sincerely appreciate the time and care you have devoted to reviewing our manuscript and the thoughtful feedback you have provided. Your constructive suggestions have helped us strengthen the rigor and clarity of the paper.

1. Fair comparisons with matched models and task sets

To distinguish the harness contribution from model choice, we added a native Codex baseline with the same Qwen backbone, Codex runtime, and default task budget, complemented by same-Terra SMDD comparisons and published baselines reaggregated on the same lead-optimization tasks. These comparisons show gains in affinity comparison and lead optimization with the backbone held fixed.

2. Technical design and evidence for scientific skill learning

We now explain the technical design through three connected elements: conditional action hypotheses, executable skills, and independent scientific review. Together, they connect reusable guidance to applicable scientific evidence and concrete actions. On 26 target-disjoint MolBench affinity tasks, the full skill achieves 25/26 correct decisions, versus 20/26 for ACE and 21/26 after removing conditional decision guidance. Case studies and analyses of execution traces further show how skills are refined and how learned guidance shapes evidence use and candidate selection.

3. Transfer to held-out targets and external data

The same guidance also transfers beyond the development data. Without further learning or revision, the frozen skill achieves 12/12 on MoleculeACE-derived affinity tasks across four targets excluded from skill development, compared with 8/12 for newly run no-skill controls. Both conditions deliver valid answers throughout, showing a gain in decision accuracy.

4. Experimental support for the HPK1 design case

We also strengthen the experimental basis of the HPK1 case by documenting candidate selection, assay methods, replication settings, and concentration-response data. These records support the candidate’s reported cellular IC50 of 14.19 nM relative to 37.47 nM for the best input reference, connecting the design task to measured activity.

Following your feedback, we have carefully revised the manuscript and added experiments and clarifications. We are grateful for your suggestions and have made every effort to address the concerns raised. We sincerely hope that the revised manuscript and our detailed responses provide clear answers to your questions. The corresponding manuscript changes are highlighted in blue.

Best regards,

Authors of Submission 40079

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.