- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors introduce DORA - an agent for drug repurposing supplied with a biomedical knowledge-graph, Meta-Pathfinder, an algorithm that performs graph search to find nontrivial drug-disease paths, and scientific tools such as docking, protein design tools etc. LLM-only approach is compared with DORA across 7 LLMs using continuous clinical trial success-based scores and categorical accuracy. DORA improves five of seven LLMs and surpasses biomedical AI agent Biomini. The authors demonstrate 3 positive examples of DORA results, discuss the observed failure modes and suggest that the knowledge-rich environment itself can be a key scaling axis for scientific discovery rather than scaling models.
Critical strengths
- The idea is practically important. The authors try to build a system that produces verifiable hypotheses through a transparent pipeline.
- The paper is of high quality, it is easy to follow, the experimentation is rigorous.
Critical weaknesses
- The paper claims that DORA discovers previously unrecognized drug–disease mechanisms. The three qualitative examples in Figure 5 illustrate biologically plausible connections, but do not establish that these mechanisms were previously unknown and that they are actually potent and useful. Probably, a small human expert study across several experts could strengthen this claim.
- The proposed metric uses asymmetric losses for ongoing and completed trials. For ongoing trials, predictions above the currently achieved phase incur no loss. This potentially rewards overly optimistic predictions, especially given the large proportion of ongoing trials in the dataset. For example, an ongoing Phase II trial can receive the maximum score even when the agent predicts eventual approval without sufficient evidence.
- The biomedical knowledge graph, presented as a central component of DORA, has a negative estimated contribution according to ablation studies. This requires more clarification because it undermines the conclusion that knowledge-rich environment itself can be a key scaling axis for scientific discovery rather than scaling models.
Questions for authors
- How do the authors distinguish genuinely novel discoveries from connections already documented in the literature? Could the authors conduct a small-scale evaluation involving several independent biomedical experts to assess the novelty, mechanistic plausibility, and potential therapeutic utility of the generated hypotheses?
- How do the authors explain the result that knowledge graph has negative contribution to DORA?
- How do the authors justify the proposed scoring scheme, and to what extent might it inflate the reported performance?
4 comments
Thank you for the clarifications! I am pleased with the answers. I have one additional question. Why is comparison with baselines limited to Biomini? Did you consider including other baselines that are mentioned in Related Work section?
Author response
We thank the reviewer for this helpful question. The baseline with the closest setting to our problem is K-Paths, which retrieves up to top-10 3-hop paths from a Hetionet graph augmented with training interactions, and primarily supports relation classification between entities represented in that graph. DORA instead accepts arbitrary drug–disease pairs and searches across independently maintained knowledge and literature providers with provenance-preserving integration. We implement K-Paths with its original graph, relation semantics, and retrieval settings. However, on our public-100 benchmark, K-Paths retrieved 0/100 paths covering both entities (drug candidates and diseases). If relaxed to one-endpoint coverage, it resolves for 32/100 pairs (28 drug and 4 disease endpoints). Running this model would be theoretically equivalent to the LLM-only baseline. This shows the challenge of our problem setting and a unique advantage of our method.
We hope this addresses your concersn and look forward to your final version of scores.
Thank you. I will increase my rating.
Thank you and our response
Thank you for recognizing the practical importance of DORA, the transparency of its pipeline, and the rigorousness of our evaluation. We appreciate your questions about mechanism discovery, asymmetric labels, and the Biomedical KG analysis.
Weakness 1 and Question 1: Mechanism discovery and expert evaluation. DORA is a system that provides evidence for any pair of drug candidates and diseases. Limited by available ground truth data, we are only able to evaluate based on existing trials and results. That said, the system can be deployed for any open pair of drug candidates and diseases. We have also been actively collaborating with users to discover new drug candidates that have never been discovered by humans, even though their evaluation will take much longer and rely on real-world experiments. In Section 4.3, we provide a few examples of such efforts, and Appendix D presents one complete pairwise DORA report with its force-directed Meta-Pathfinder visualization.
We appreciate the advice on letting human experts evaluate the insights and hypotheses provided by DORA. We asked two human evaluators to complete human evaluation of 19 DORA reports. For each drug–disease pair, evaluators first independently searched literature available on or before July 10, 2026 and froze a literature-based baseline. They then read the DORA report and assessed its strongest incremental insight for novelty beyond the baseline, biological plausibility, and therapeutic utility. On average, they spent 33.2 minutes manually searching the literature and identified 2.1 relevant literature or registry sources per pair. 14 of the 19 hypotheses were rated at least “plausible with gaps,” including nine rated “coherent and supported” or “supported by a strong convergent rationale.” Among them, 3 were also rated as containing a “specific new insight” or a “clearly distinct and consequential insight”: delgocitinib for vitiligo, through a candidate melanocyte-intrinsic mechanism involving inhibition of laminin-332-induced melanocyte dedifferentiation; linsitinib for thyroid eye disease, by connecting clinical response with IR-beta cross-inhibition and hyperglycemia; and ravulizumab for IgA nephropathy, through a candidate C5–EDNRA connection extending the complement-centered mechanism. All three also received high therapeutic-utility ratings and were considered worthy of further exploration or experimental verification. These results provide promising signals for DORA on this extremely challenging problem and illustrate how the system may inspire researchers to explore mechanistic gaps that could otherwise be overlooked within the vast volume of biomedical knowledge. We are continuing working on a larger human evaluation throughout the system deployment process, and might also update the results.
Weakness 2 and Question 3: Asymmetric scoring and optimistic predictions. We appreciate the question and the chance to clarify our metric design. In our evaluation set, whether a question has an exact label or a lower-bound label is hidden from every model, so a model cannot cheat by selectively over-reporting scores only on lower-bound labels, but not on exact label queries. Additionally, we provide a factor that upcurves the overconfident branch in exact label loss, so that in expectation, a blind model cannot gain more by overreporting scores in an uninformed way. We provided more details in Section 2.
Weakness 3 and Question 2: Biomedical KG coefficient. We appreciate the question and the chance to clarify the observation better. Knowledge graphs (KGs) are included in the system to provide more long-established, historically recognized human knowledge of the involved entities. They are usually built from older sources than rapidly updated literature, so they might provide information that is less up-to-date. This is reflected as a non-strictly-positive effect in the quantitative evaluation when the KG is used alone to predict the latest clinical trials. In the corrected additive Bayesian model, the KG effect is +1.43 normalized-score points (95% CrI [−6.36,9.21], P(β>0)=0.64). We keep this factor for multiple reasons: First, we envision that future versions of the system could potentially address this and reconcile the trade-off between knowledge credibility and recency; Second, knowledge graphs provide a network to which the latest literature can be anchored, providing more comprehensive and better-connected pathfinding results. Our human experts in the team still appreciate the value of knowledge graphs, so from the perspective of producing the most useful product for this challenging quest, we don’t want to remove that component at an early stage of the evolving system solely due to the negative effect in preliminary quantitative results.
Thank you again for the detailed questions. We would appreciate your follow-up comments and, if your concerns are addressed, an updated score.