acceptodds
Withdrawn from ICLR 2027

BRIDGE-bench: Benchmarking Real-world Issues in Drug Generation and Evaluation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) and tool-using agents are increasingly proposed as assistants for drug discovery, but their evaluation remains fragmented. Existing benchmarks often emphasize isolated capabilities, such as molecular property prediction and chemical question answering, while practical therapeutic research requires coordinated decisions across biology, chemistry, pharmacology, modeling, manufacturability, and translation. We introduce BRIDGE-Bench (Benchmarking Real-world Issues in Drug Generation and Evaluation), a pipeline-level benchmark built from real-world drug discovery and development questions. BRIDGE-Bench covers target biology, hit and lead discovery, binding pocket analysis, absorption, distribution, metabolism, and excretion (ADME), quantitative systems pharmacology (QSP), retrosynthesis, antibody-drug conjugate (ADC) design, drug delivery, chemistry, manufacturing, and controls (CMC), and wet-lab planning. Each instance is organized around a concrete decision scenario with reference answers and scoring criteria. We evaluated 13 harness-model configurations within the Claude Code and Codex agent frameworks. Final average scores ranged from 19.3 to 36.0, with the strongest aggregate result obtained by Claude Code with claude-opus-5. The benchmark supports a modular and trajectory-level evaluation of answer correctness, evidence grounding, tool use, uncertainty awareness, cross-stage trade-off reasoning, and error modes that matter for practical therapeutic decisions.

Primary area:
datasets and benchmarks
Keywords:
benchmark, drug discovery, agent ability

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_uRDMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces BRIDGE-Bench, a 38-question expert-curated benchmark evaluating LLM agents on realistic drug discovery and development tasks across the therapeutic pipeline (e.g., ADME, QSP, retrosynthesis, ADC design, CMC). Evaluating 13 harness–model configurations (including Claude Code and Codex) on answer quality and trajectory execution, the authors identify key agent bottlenecks in biological reasoning, workflow execution, evidence integration, and reproducibility.

Critical strengths
  • Practical, pipeline-wide coverage
  • Rich failure analysis
Critical weaknesses

W1. Independent tasks rather than pipeline integration: Tasks are standalone stage-specific problems rather than interconnected multi-stage workflows where decisions propagate downstream, undermining claims of true pipeline-level evaluation.

W2. Small scale and imbalanced distribution: With only 38 questions, distribution is heavily skewed (e.g., drug combination screening comprises ~26%, while several domains have only 1 question), making aggregate metrics unreliable across sparse categories.

W3. Unvalidated LLM judge: Relies on gpt-5.6-sol for rubric grading without proving alignment with human expert evaluations or controlling for self-evaluator bias (since gpt-5.6-sol is also a subject model).

W4. Single-run stochasticity: Evaluates each configuration only once per question, omitting confidence intervals or significance tests despite close score gaps and high agent execution variance.

W5. Lack of benchmark accessibility and reproducibility.

The paper does not provide a publicly accessible repository or a clear data release plan for BRIDGE-Bench. Although Appendix A summarizes the 38 benchmark tasks, the complete task instructions, input artifacts, reference answers, grading rubrics, and evaluation scripts are not made available in the manuscript. Without these resources, independent researchers cannot reproduce the reported evaluations or use the benchmark to assess new agent systems. The authors should clarify the benchmark release plan and ideally provide the complete evaluation resources.

Questions for authors

Provide above

Reviewer_qrm7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors introduce BRIDGE-Bench - a benchmark that consists of real-world drug discovery and development questions. 13 harness-model configurations are benchmarked and their errors are analyzed.

Critical strengths

I can't see any critical strengths, unfortunately. The idea to see how widely used agents perform in drug discovery tasks might be interesting, but the quality of the current experimental design leaves a lot to be desired.

Critical weaknesses
  1. The quality of the paper is poor. The paper suffers from serious shortcomings in its experimental reporting and reproducibility. Essential methodological details are missing (task prompts, agents execution settings, scoring procedure). Neither the benchmark files nor the evaluation code are made available. Appendix A provides only brief, one-sentence descriptions of the 38 benchmark tasks rather than the actual questions, input data (e.g. figures), or expected outputs. Without this the reported results cannot be verified, reproduced, or meaningfully compared against future methods.
  2. The motivation for benchmarking general-purpose agent harnesses, rather than agents equipped with domain-specific scientific tools, is unclear. E.g., Question 15 asks agents to analyze the binding pocket, interacting residues, docking score, and pharmacological implications of Gymnemaside VI binding to AKT1. Such a task would typically require molecular docking software and appropriate molecular structures. However, Section 4.1 only specifies that agents were allowed Internet access and the tools available within their respective harnesses. This raises a fundamental concern about what the benchmark actually measures: the scientific reasoning capabilities of the underlying models, the capabilities and limitations of their execution environments, or simply their ability to work around missing domain-specific tools. Furthermore, both Claude Code and Codex are proprietary frameworks whose internal implementations and execution behavior are not fully transparent, making it difficult to isolate and reproduce harness-specific effects. Although Section 4.4 attempts to compare identical backbone models across the two harnesses, the scientific motivation for this particular comparison remains unclear, especially given the paper's stated objective of evaluating real-world drug-discovery workflows.
  3. The authors use a single LLM to evaluate scientific correctness, evidence grounding, tool use, and reasoning trajectories. However, no experiments are provided to validate the agreement between LLM-generated scores and assessments by independent domain experts. Assessing the correctness of such tasks requires specialized scientific knowledge, and it is unclear whether a single LLM judge can reliably identify methodological errors or distinguish scientifically valid solutions from plausible but incorrect explanations.
  4. The benchmark contains only 38 questions. While the authors emphasize comprehensive pipeline coverage as a major contribution, the dataset is too small and unevenly distributed to support robust conclusions about performance.
Questions for authors

What is the main research question and intended contribution of this work? Is the primary goal to assess the capabilities of current LLMs in drug discovery, evaluate the impact of different agent frameworks, or establish a standardized benchmark for future research? The authors should clarify the central objective of the study and explain how the experimental design and reported results support it.

Reviewer_BkvxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces BRIDGE-Bench, a benchmark for evaluating LLM-based agents on realistic drug discovery and development tasks. Unlike existing benchmarks focusing on isolated prediction or reasoning problems, BRIDGE-Bench emphasizes scientific workflows, tool usage, and evidence-based decision-making. The benchmark consists of 38 expert-curated tasks spanning ten drug discovery stages, with reference answers and grading rubrics provided for each task. The authors evaluate 13 harness–model configurations using Claude Code and Codex, employing GPT-5.6-sol as an LLM judge to assess both final answers and execution trajectories. Experimental results show average scores ranging from 19.3 to 36.0, with Claude Code combined with Claude Opus 5 achieving the highest score. The paper further analyzes performance differences across task categories, harness effects, and common failure modes in scientific workflow execution.

Critical strengths

The paper addresses an important problem in drug discovery by evaluating LLM agents on realistic scientific tasks rather than isolated prediction or question-answering problems. The benchmark covers a broad range of drug discovery stages, including several relatively underexplored areas such as QSP modeling, retrosynthesis, and ADC design.

Critical weaknesses

1.The benchmark contains only 38 tasks across ten drug discovery stages, with a highly imbalanced distribution across task categories. As shown in Figure 3, drug-combination screening accounts for 26.3% of the benchmark, while several other categories contain only a single task. Such limited coverage raises concerns about the representativeness of the benchmark and the generalizability of the reported findings.
2.The claimed pipeline-level evaluation is not sufficiently demonstrated. Although the benchmark covers multiple stages of drug discovery, most tasks listed in Appendix A appear to be independent scientific problems rather than interconnected workflows requiring cross-stage reasoning or coordinated decision-making. This makes it difficult to distinguish the proposed benchmark from an aggregation of existing domain-specific evaluation tasks.
3.The reliability of the evaluation procedure remains insufficiently validated. Section 4.1 relies on GPT-5.6-sol to evaluate both final answers and agent trajectories, but no quantitative agreement with independent human expert assessments or alternative LLM judges is reported. Given the open-ended nature of these scientific tasks, the absence of such validation raises concerns about the reliability and consistency of the reported scores.
4.The experimental evaluation lacks sufficient statistical robustness. Each harness–model configuration is evaluated only once per task, without repeated trials or confidence intervals. The top configurations achieve very similar average scores, such as 36.0 and 35.9, making it difficult to determine whether the observed differences reflect meaningful performance variations.
5.The comparison between Claude Code and Codex may be affected by differences in execution environments and tool availability. Although the authors analyze the effects of different agent harnesses, the paper does not clearly establish whether tool access, execution budgets, and prompting strategies were sufficiently controlled. Consequently, the observed performance differences cannot be confidently attributed to the harness alone.

Questions for authors

1.How many of the 38 tasks explicitly require cross-stage reasoning? Could the authors provide an example in which decisions or outputs from one drug discovery stage directly influence subsequent stages?
2.Would the overall model rankings remain consistent if category-level macro-averaging were used to account for the imbalanced task distribution?
3.How are the 100-point grading rubrics constructed, and how does the evaluator handle alternative scientifically valid solutions, particularly for tasks such as retrosynthesis where multiple feasible routes may exist?
4.Has the GPT-5.6-sol judge been validated against independent human expert assessments? How sensitive are the evaluation results to the choice of LLM judge, particularly given that GPT-5.6-sol is also among the evaluated models?
5.Could the authors provide repeated evaluations for representative configurations to assess run-to-run variability and determine whether the small score differences among leading systems are statistically meaningful?
6.Were Claude Code and Codex evaluated under comparable tool access, computational budgets, and execution constraints? Could the authors additionally include OpenCode as an open-source agent harness to provide a more comprehensive comparison under consistent experimental settings?
7.Did the authors consider evaluating direct LLM baselines without agentic tool execution? Such comparisons could help distinguish improvements from agentic workflows from those attributable to the underlying models' scientific reasoning capabilities.
8.How many domain experts participated in task construction and quality control? Were the tasks and grading rubrics independently reviewed by multiple experts, and how were disagreements resolved?
9.Section 3.3 describes leakage prevention primarily through question rephrasing and modification of source materials. How effective are these measures in preventing semantic contamination, particularly for tasks derived from publicly available literature and databases?
10.Table 3 identifies six common failure modes, but their frequencies across different models and tasks are not reported. How were these failure categories determined, and could the authors provide a quantitative breakdown to support the failure-mode analysis?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.