- Rating
- 3(+1)
- Confidence
- 1(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces DeepQuest, a knowledge-gap-driven AI scientist framework for Alzheimer's disease (AD) research that automates the workflow from literature review to empirical discovery. It transforms unresolved limitations from domain reviews into traceable, testable hypotheses, and then select and prioritize tasks under data feasibility constraints. A two-level auditing mechanism separates experimental execution reliability from knowledge contribution relative to a frozen 200-paper corpus. On OASIS-3, DeepQuest generates 42 feasible hypotheses, achieves a 95.24% Scientific Finding Rate, and validates key clinical findings across independent ADNI and AIBL cohorts as well as external finding reproduction tasks.
Critical strengths
- Principled Coupling of Data Feasibility and Two-Level Auditing: By grounding hypothesis selection in structured multimodal data summaries and auditing both sandbox code execution and literature novelty, DeepQuest achieves a 95.24% Scientific Finding Rate on OASIS-3 while drastically reducing execution skips and uncertain outcomes
- Rigorous Cross-Cohort Biomedical Validation and Reproduction: Beyond LLM-based metrics, the framework produces clinically meaningful findings in OASIS-3 that replicate in independent ADNI and AIBL cohorts (covering NPS-related dementia risk, longitudinal amyloid topographic stability, and neocortical tau progression), and independently recovers core patterns from two recent 2026 clinical studies across different tracers and outcome definitions.
Critical weaknesses
- Reliance on Automated Evaluators and a Narrow Frozen Corpus Without Expert Calibration: In Section 4.2, the three LLM evaluators overlap with the baseline generators (potentially introducing self-preference bias), and in Section 3.6, assessing novelty solely against 200 papers published after February 16, 2026 may overstate the rate of "Candidate New Discoveries" by missing earlier foundational AD literature, with no blind evaluation by human clinical experts.
- Lack of presentation of the criteria of selecting the 200 clinical papers.
- The findings in Sec 4.4, unveiled by DeepQuest, are somehow established in the area. They may also stem from the model's parametric knowledge, not just the auto research process. An isolation between them can make more solid the claim that the model is able to give novel findings.
- Incomplete Component Ablations : As noted in Sections 4.2–4.3, the nested Top-10 evaluation and end-to-end execution comparison do not isolate the individual contributions of groupwise Elo ranking, within-round execution auditing, or cross-round planning updates.
Questions for authors
Clarifications:
- Why was the evidence bank restricted to 200 clinical papers published after February 16, 2026, and how many of the 20 "Candidate New Discoveries" would be reclassified as "Rediscovery" or "Extension" if audited against a broader historical PubMed corpus? What is the criteria of selecting the 200 clinical papers.
- Did the authors check for self-preference bias when GPT-5.6-sol, Claude Opus 4.8, and DeepSeek-v4-pro-0813 evaluated their own generated hypotheses, and how does the 95.24% SFR change if cross-round optimization or execution auditing is ablated on DeepQuest-H?
Suggestions:
- Adding a small-scale blind review by domain neurologists on the Top-10 hypotheses and audited claims would significantly strengthen the validation of clinical significance and novelty.