- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
PerturbOS is an agentic framework that selects and executes workflows using perturbation datasets and predictive models suited to a biological question, then uses the resulting predictions to prioritize intervention candidates. It is evaluated through one retrospective drug-prioritization case study and a separate routing benchmark.
Critical strengths
- S1. Assessing whether available datasets and models fit a biological question before execution is a practical problem formulation. The paper makes this suitability check explicit rather than assumed.
- S2. Comparing matched predictive feedback against permuted and absent feedback is a well-designed contrast that demonstrates the value of candidate-specific predictive information.
- S3. PerturbOS selects suitable dataset–model pairs more accurately than the LLM baselines, supporting the usefulness of structured resource records and suitability analysis.
Critical weaknesses
- W1. The contribution of the agentic discovery procedure is not separated from that of the predictive model.
- All BDA-based baseline shortlists underperform uniform random selection on the primary hit-rate metric. This raises questions about the effectiveness of LLM-driven candidate proposals as a search strategy.
- Candidate acceptance is largely determined by the predicted direction of effect, and STATE predictions are already computed for the full candidate set. Without a non-agentic baseline that directly ranks or filters these predictions, the added value of iterative agentic refinement remains unclear.
- W2. The routing benchmark's validity and downstream utility are insufficiently established.
- The construction and validation of the acceptable and preferred dataset–model annotations are not sufficiently described. This makes it difficult to assess whether agreement with these reference labels reflects genuine resource suitability.
- Under Opus 5, PerturbOS improves indirect routing but misclassifies most direct-route cases, as the authors acknowledge. The overall accuracy gain therefore does not reflect a consistent improvement across route types.
- Routing is evaluated against reference annotations, but whether the selected resources lead to better downstream predictions or intervention prioritization remains unverified.
- W3. The biological-discovery framing is broader than the problem and evaluation actually support.
- PerturbOS prioritizes interventions toward a prespecified molecular readout and direction of change. While this is a valid discovery task, the evaluation supports this narrower form of prediction-guided prioritization rather than the broader biological-discovery framing.
- End-to-end prioritization is evaluated on a single retrospective case, while the broader benchmark covers only routing. Although new biological findings are not required for a CS contribution, the current evaluation does not establish the generality implied by the broader discovery framing.
Questions for authors
- Q1. Does agentic refinement provide additional benefits over directly ranking or filtering the same STATE predictions?
- Q2. Who constructed the acceptable and preferred annotations, under what criteria, and were they validated independently of system development?
- Q3. Do the selected dataset–model pairs lead to better downstream prediction performance?
- Q4. Has end-to-end prioritization been evaluated on biological objectives beyond TOP2A, or is there other evidence supporting its broader discovery utility?
12 comments
Q2 Reply: Annotation authorship and independent validation
Thank you for your willingness to understand our work in detail. One author supplied the datasets and verified the AI-generated scenarios and acceptable/preferred annotations through a process independent of PerturbOS development. The author then checked the benchmark ground truth for correctness and fairness. Annotations were finalized without access to PerturbOS’s routing decisions.
Acceptable pairs satisfy the question's requirements for organism, intervention modality, cellular context, candidate/readout coverage, model capabilities, and inputs. Preferred pairs prioritize context and coverage, favoring an exact context match before justified biological surrogates.
We applied the following criteria when verifying acceptable and preferred dataset–model pairs:
| Criterion | Annotation rule |
|---|---|
| Biological compatibility | Match the organism, intervention modality, and requested cellular context, or identify a justified transfer setting. |
| Coverage | Check whether the dataset/model supports the requested interventions and expression readout. |
| Generalization capability | Require capabilities appropriate to the question: unseen interventions, context transfer, or batch/donor transfer. |
| Required inputs | Check requirements such as target-context controls or chemical structures. |
| Preference among acceptable pairs | Favor closer biological context and relevant coverage. Preference denotes suitability under this rubric, rather than measured predictive superiority. |
Takeaway & Claim Calibration: The annotations reflect an explicit suitability rubric, author verification, and finalization without access to the system’s routing decisions.
Q3 Reply: Downstream prediction performance
Thank you for this question. Currently, our focus is that the resource selection must first satisfy the question’s required generalization, biological context, intervention modality, candidate and readout coverage, and input requirements.
Model suitability: RACK1/K562: For example, the broader RACK1/K562 question includes genetic interventions absent from fitting. GEARS was used because it supports predictions for these unseen interventions. The evaluated one-hot STATE configuration does not qualify for this scope because it can score only intervention identities represented during fitting.
Dataset suitability: HSP90AA1/A549: Another example would be to reduce HSP90AA1 expression in A549 cells, including untested compounds with known structures. sci-Plex3 A549 + chemCPA ranked above ComboSciPlex + chemCPA. Holding chemCPA fixed, sci-Plex3 provides broader measured single-compound coverage—188 observed labels compared with six single drugs in the inspected ComboSciPlex release—and a closer recorded biological-context match. These properties make it better suited to broad single-compound discovery.
Q1 Reply: Value of agentic refinement
Full-pool ranking is stronger on TOP2A; the exploratory RACK1 results favor the complete feedback-guided workflow.
TOP2A baseline comparison: We sincerely thank the reviewer for pointing out this important baseline. We have decided to adopt full-pool ranking when the requested candidate scope contains no unseen perturbations and the supported search space is small. In this setting, exhaustive ranking makes full use of the available predictions. The comparison with the original PerturbOS workflow is:
PerturbOS(original final selection)Full-pool STATE reaches 70% Top-60 versus 46–53% for PerturbOS and also leads at Top-10 and Top-20. These numbers support adopting exhaustive ranking for this small panel.
End-to-end exploratory RACK1 case: On the other hand, we ran the separate exploratory RACK1 case. The task asks which single-gene CRISPRi interventions decrease RACK1 RNA expression in K562. The frozen GEARS predictor uses the Replogle K562 essential-gene dataset, with 465 fitting interventions and a supported pool of 1,861 seen and unseen targets.
The table below shows the workflow's value in a broader space that includes unseen interventions:
PerturbOS(matched feedback)RACK1 gives 75% Top-60 for PerturbOS versus 50% for full-pool GEARS and 60% for seen-only GEARS; the corresponding Top-10/20 rates are also higher. This supports the complete workflow on this exploratory objective, without identifying which component causes the advantage.
Random-50 reports mean ± sample SD. The seen/unseen breakdown is:
PerturbOS(matched feedback)The combined unseen hit rate is 67.3% for PerturbOS versus 40.0% for full-pool GEARS, alongside 81.5% versus 60.0% on seen targets. These rates support improved prioritization among both selected seen and selected unseen targets in this exploratory task.
Additionally, the separate shared-start comparison supports the value of correctly matched predictive information for the refinement-and-selection workflow:
PerturbOS(matched feedback)Matched feedback exceeds no feedback by 42 percentage points on TOP2A (44% versus 2%) and 30 points on RACK1 (75% versus 45%). .
Takeaway & Claim Calibration: The results support exhaustive ranking for small, observed panels, whereas iterative prior-guided proposal adds clear value over exhaustive predictor sorting in larger candidate spaces with unseen perturbations