- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes HypoEvolve: a generational genetic algorithm coordinating specialized LLM agents (generation, pairwise comparison, crossover/mutation, etc.) to cast hypothesis refinement as fixed-size population search, with fitness from pairwise judgments driving selection and replacement. The core claim is that, with scientific roles and prompts fixed, collaboration/search rules can themselves be experimental variables. Validation focuses on a closed-set drug-repurposing setting with external biomedical association resources, plus comparisons to Tree-of-Thoughts and single-pass generation, and selection-related ablations.
Critical strengths
- Trying to decouple collaboration rules from scientific roles, and contrasting fitness-guided selection against weaker controls, is more methodologically intentional than stacking agent workflows; this is a useful framing for whether organization contributes independently.
- The external-evaluation protocol is comparatively careful: answers are submitted before scoring, external evidence does not enter fitness, and held-out settings reduce pure LLM self-scoring circularity.
Critical weaknesses
- The evaluation mainly measures biological associations implied by hypotheses, not the full scientific quality of those hypotheses. The paper claims to assess scientific hypothesis discovery with mechanistic rationales, yet the main results largely reduce to DepMap selectivity and Open Targets association on a closed drug set—closer to scoring whether chosen drug–cancer target links match external resources than to directly testing whether rationales are correct, novel, or mechanistically falsifiable.
- There is a risk that pairwise fitness judging is aligned with the external metrics, further weakening the discovery narrative. The scoring appears to emphasize cancer-specific dependency and target-evidence cues close to DepMap selectivity, raising the risk of using a nearby criterion as the search referee and then claiming discovery on similar external metrics; the appendix constant-drug control can also trade wins and losses with the main method on held-out settings, weakening the “evolution produced better hypotheses” interpretation.
Questions for authors
- After matching call counts and candidate-pool size, do advantages over main search baselines and simpler controls remain stable?
- If pairwise scoring is decoupled from external biomedical metrics (e.g., emphasizing mechanistic coherence and testability), do conclusions change?
- Does current evidence better support “improved drug–setting matching” or “improved scientific hypothesis discovery,” and how should the writing be scoped?
2 comments
Response to Reviewer QHLC (2/2)
Q2. A judge decoupled from the metrics. We ran the judge you describe. It ranks on mechanistic coherence, evidence and testability, and it does not mention cancer specificity. The conclusions change in part.
With this judge, HypoEvolve scores 0.103 on DepMap and 0.316 on Open Targets. With our judge (the pairwise scorer in the paper) it scores 0.120 and 0.400, and single-pass generation scores 0.026 and 0.147. The generic-judge arm still leads single-pass generation on Open Targets (+0.169, corrected p=0.011). The difference between the two judges is not resolved (+0.017 on DepMap, p=0.69; +0.084 on Open Targets, p=0.064).
However, three of the four operator prompts also ask for cancer specificity, so this arm keeps the cancer-specificity criterion in the search. We therefore ran a further arm with the generic judge and with operator prompts that do not ask for cancer specificity. It scores 0.037 and 0.200, and it is not shown to beat single-pass generation (+0.011, p=0.87; +0.053, p=0.44). The full system leads it on Open Targets by +0.201, with a corrected p-value of 0.005 over the ten tests of this arm.
This arm also changes which drugs the operators propose, so it does not isolate the wording. In short, the Open Targets lead survives a generic judge. We can no longer show it once the criterion is also removed from the operator prompts. Our reply to Reviewer CYkt gives the details.
W1. What the evaluation measures. We agree. Both metrics score the annotated targets of the proposed drug on a closed list of 61 drugs. We do not assess the rationale itself. We will state this as a limitation.
W2. Alignment of the judge with the metrics. The risk you describe is real, and it applies beyond the judge. The judge and three of the four operator prompts ask for cancer specificity, and the single-pass prompt does not. When we put the criterion of the judge into the single-pass prompt, its scores rise from 0.026 to 0.062 on DepMap and from 0.147 to 0.265 on Open Targets. That recovers about half of the Open Targets margin. HypoEvolve leads this prompt by +0.058 on DepMap (p=0.66) and +0.135 on Open Targets (p=0.021, or 0.17 after correction). On Luna, we therefore cannot show that the search adds to the criterion.
On the constant-drug control, we agree that it weakens a reading based on raw scores. The residual described under Q3 is our answer, because any constant drug scores zero on it. The final answers of HypoEvolve score +0.103 on DepMap and +0.100 on Open Targets on gpt-5.4-mini (p=0.018 and p=0.0086 against zero). On Luna they score +0.080 and +0.101 (p=0.16 and p=0.014).
Response to Reviewer QHLC (1/2)
Thank you for the review. Your three questions go to the core of the paper, so we ran the experiments they call for. The new runs use a second model, GPT-5.6 Luna, across all 34 cancer types. Each method has one run per cancer type, and single-pass generation has six draws. Scores are means over cancer types (31 for DepMap selectivity, 34 for Open Targets), and p-values are uncorrected unless stated. We answer the questions first.
Q3. Matching or discovery. The evidence supports drug-cancer matching in a narrow form. It does not support hypothesis discovery. We withdraw that claim and the collaboration framing. We will revise the title to match.
The result that holds on both models is higher Open Targets association than single-pass generation. On all 34 cancer types, the margin is +0.273 on gpt-5.4-mini (the model of the paper) and +0.253 on GPT-5.6 Luna, each with p<0.001.
To test matching directly, we use the cancer-specificity residual of Appendix C.4. It subtracts the mean score of the same drug over the other cancer types, so a drug that scores well everywhere gains nothing. On this residual, HypoEvolve leads single-pass generation by +0.089 on DepMap (p=0.014) and +0.092 on Open Targets (p=0.0055) on gpt-5.4-mini. On Luna the lead is +0.085 (p=0.084) and +0.086 (p=0.036).
Q1. Matched calls and candidate pool. The advantages do not remain stable.
A margin is the score of HypoEvolve minus the score of the method. We fixed 16 tests before the runs and applied Holm correction. Of these six margins, only the DepMap margin at 193 calls stays below 0.05 after correction (0.044). This result is fragile. Tree of Thoughts often ends with several leaves tied on the top score and returns the first. When we average the tied leaves, the corrected value is 0.18. Tokens are not matched in any comparison.