- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper introduces Hypothesis Portfolio Search (HyPS), a framework for selecting genetic perturbation experiments under a limited testing budget. The method is split in two submethods that interact hierarchically. The top method supervises K individual methods that perform individual searches. An LLM uses the scientific goal, the stated hypothesis and the history of each specific assay as input to decide whether each individual search will stop and propose budget and certain genes that will serve as anchors for each hypothesis. The anchors proposed will be used to derive the candidate pool, which are basically the closest genes to the anchors using their Achilles representation for similarity. A module updates the quotas for each hypothesis throughout the rounds. Inside each candidate level search, there isa score which is a model trained on the historical assays. They are small logistic-regressions with three features per gene, representing the experimental effects already observed in genes based on three different similarities. The whole method runs in a loop and is updated based on the update of the history.
Critical strengths
The work tackles a very important and interesting problem on building assays for genomic experiments to increase the discovery rate.
The inclusion of an LLM as supervisor and a common regression for each search is an interesting idea.
Critical weaknesses
To my understanding, the internal threshold is derived from the complete benchmark hit set. This means that both allocation and scorer training use information beyond completed experiments.
The description of the methodology is hard to follow Sections 3 and 4 could improve in clarity significantly. For example slots and anchors are not clearly defined. The whole paragraph on Bounded adaptive quotas is hard to follow.
The method is not compared to TYpiclust or ACS-FW or most importantly, random selection [1].
[1] Huang, Kexin, et al. "Sequential optimal experimental design of perturbation screens guided by multi-modal priors." International Conference on Research in Computational Molecular Biology. Cham: Springer Nature Switzerland, 2024.
The code is not available to reproduce the results, which, for a non-theretical paper, is vital.
The theoretical or technical novelty of the work is limited.
Questions for authors
-
How much do the reported gains depend on deriving from the complete benchmark hit set, and do they persist when the threshold is specified independently
-
Can you provide results for random selection, TypiClust and ACS-FW under the same experimental budgets?
-
Can you provide the implementation, configurations, and run records?
-
What is the principal technical advance beyond combining LLM based retrieval with scoring based on similarity?
10 comments
2. Slots, anchors, pools, and quotas are now defined with an example and Algorithm 1
The description of the methodology is hard to follow Sections 3 and 4 could improve in clarity significantly. For example slots and anchors are not clearly defined.
We thank the reviewer for this helpful feedback. §4.1 now defines each term with a concrete example:
-
Slot: One of persistent positions in the portfolio.
-
Unit: The content of a slot, i.e., a natural-language hypothesis plus three anchor genes. For example, "TCR proximal signaling loss reduces IL-2 output" with anchors LCP2, ZAP70, and LAT.
-
Pool: The untested genes retrieved by Achilles similarity to the anchors.
-
Quota: The number of tests the slot receives in the round.
-
Algorithm 1: Shows how these elements interact in one complete round.
-
Explicit protocol: §3 now defines what the agent observes in each round.
-
Clarification of the summary: In HyPS, the LLM keeps, revises, or replaces each unit, while testing budgets are assigned by the bounded allocation rule (Eq. 2–3) from observed outcomes. This keeps the budget decision quantitative and feedback-driven.
Revision in paper: §3; §4.1; §4.2; Algorithm 1.
3. The quota rule is now an explicit three-step procedure
The whole paragraph on Bounded adaptive quotas is hard to follow.
We thank the reviewer for this comment. The paragraph is rewritten as an explicit procedure:
- Priority: Each slot receives a priority (Eq. 2, moved from the appendix): a smoothed event rate plus a low-observation bonus, in the spirit of upper-confidence-bound allocation.
- Floor: Every slot first receives tests.
- Assignment: The remaining tests are assigned one at a time by priority, subject to the cap (Eq. 3).
- Design rationale: The floor keeps every direction explored, and the cap prevents over-concentration on early successes. With and , the bounds are 16–48 and the initial quotas are equal.
Revision in paper: §4.2, ¶"Bounded adaptive quotas", Eq. (2)–(3); Algorithm 1, line 8.
1. τ encodes the hit criterion, and HyPS's gains persist with a label-blind threshold
We thank the reviewer for raising this important point.
Labels only after testing: τ is a single scalar fixed before search that encodes the hit criterion; it contains no gene identities. HyPS labels a gene only after testing it and never observes any untested outcome.
τ conveys the hit definition: On four of six screens, the label of a tested gene is identical to its hit status (new Table 4). On Sanchez, τ carries no information at all, and HyPS still outperforms BDA for all nine backbones.
Label-blind experiment: We removed τ entirely. The threshold is computed each round from tested genes only (robust z ≥ 2, fixed before any run). On a representative subset fixed in advance (IFNG, IL2, Carnevale × GPT-5.6 Sol, GPT-5.6 Terra, Claude Sonnet 5; ten runs each):
Label-blind HyPS outperforms BDA in all 9 pairs on both metrics and retains 92% / 83% of the gain. Per-pair results are given in our response to Reviewer p8h2 (Comment 2).
Revision in paper: §3, ¶"Interactive protocol" and ¶"Hit criterion and internal feedback labels"; Table 4 (App. A.3); §5.2, ¶"Independent threshold"; Table 20 (App. C.10).