acceptodds
Under review as a conference paper at ICLR 2027

PerturbOS: An End-to-End Agentic Framework for Prediction-Guided Single-Cell Perturbation Discovery

Abstract

Identifying interventions that reshape cellular behavior can reveal disease mechanisms and guide therapeutic research, but testing large genetic and chemical spaces with single-cell assays is costly. Existing perturbation datasets and predictive models offer a way to focus these experiments. However, their availability alone does not establish whether they cover the interventions and cellular context specified by a new discovery question. Using resources without assessing that fit can mislead intervention selection. To turn existing resources into discovery-specific guidance, we introduce PerturbOS, an end-to-end agentic framework that assesses resource suitability before prioritizing interventions. Given a natural-language question, PerturbOS selects and executes a suitable analysis workflow, then uses its quantitative output alongside biological knowledge to refine candidate interventions. In retrospective drug prioritization for increasing TOP2A expression in A549 cells, 49% of PerturbOS’s shortlisted drugs were among those with the largest measured increases, compared with 9% for an agentic baseline without predictive feedback. PerturbOS also outperformed LLM baselines in selecting appropriate workflows across 37 biological scenarios. By improving both workflow selection and intervention prioritization, PerturbOS offers a foundation for more systematic, question-driven biological discovery guided by existing perturbation resources.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
AI for Sci, Agentic AI, Computational biology, In silico perturbation screening, Single-cell perturbation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_apLHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PerturbOS is an agentic framework that selects and executes workflows using perturbation datasets and predictive models suited to a biological question, then uses the resulting predictions to prioritize intervention candidates. It is evaluated through one retrospective drug-prioritization case study and a separate routing benchmark.

Critical strengths
  • S1. Assessing whether available datasets and models fit a biological question before execution is a practical problem formulation. The paper makes this suitability check explicit rather than assumed.
  • S2. Comparing matched predictive feedback against permuted and absent feedback is a well-designed contrast that demonstrates the value of candidate-specific predictive information.
  • S3. PerturbOS selects suitable dataset–model pairs more accurately than the LLM baselines, supporting the usefulness of structured resource records and suitability analysis.
Critical weaknesses
  • W1. The contribution of the agentic discovery procedure is not separated from that of the predictive model.
    • All BDA-based baseline shortlists underperform uniform random selection on the primary hit-rate metric. This raises questions about the effectiveness of LLM-driven candidate proposals as a search strategy.
    • Candidate acceptance is largely determined by the predicted direction of effect, and STATE predictions are already computed for the full candidate set. Without a non-agentic baseline that directly ranks or filters these predictions, the added value of iterative agentic refinement remains unclear.
  • W2. The routing benchmark's validity and downstream utility are insufficiently established.
    • The construction and validation of the acceptable and preferred dataset–model annotations are not sufficiently described. This makes it difficult to assess whether agreement with these reference labels reflects genuine resource suitability.
    • Under Opus 5, PerturbOS improves indirect routing but misclassifies most direct-route cases, as the authors acknowledge. The overall accuracy gain therefore does not reflect a consistent improvement across route types.
    • Routing is evaluated against reference annotations, but whether the selected resources lead to better downstream predictions or intervention prioritization remains unverified.
  • W3. The biological-discovery framing is broader than the problem and evaluation actually support.
    • PerturbOS prioritizes interventions toward a prespecified molecular readout and direction of change. While this is a valid discovery task, the evaluation supports this narrower form of prediction-guided prioritization rather than the broader biological-discovery framing.
    • End-to-end prioritization is evaluated on a single retrospective case, while the broader benchmark covers only routing. Although new biological findings are not required for a CS contribution, the current evaluation does not establish the generality implied by the broader discovery framing.
Questions for authors
  • Q1. Does agentic refinement provide additional benefits over directly ranking or filtering the same STATE predictions?
  • Q2. Who constructed the acceptable and preferred annotations, under what criteria, and were they validated independently of system development?
  • Q3. Do the selected dataset–model pairs lead to better downstream prediction performance?
  • Q4. Has end-to-end prioritization been evaluated on biological objectives beyond TOP2A, or is there other evidence supporting its broader discovery utility?
12 comments
AuthorsOpenReview ↗

Q1 Reply: Value of agentic refinement

Full-pool ranking is stronger on TOP2A; the exploratory RACK1 results favor the complete feedback-guided workflow.

TOP2A baseline comparison: We sincerely thank the reviewer for pointing out this important baseline. We have decided to adopt full-pool ranking when the requested candidate scope contains no unseen perturbations and the supported search space is small. In this setting, exhaustive ranking makes full use of the available predictions. The comparison with the original PerturbOS workflow is:

MethodCandidate exploredTop-10Top-20Top-60
Full-pool STATE ranking18820%30%70%
PerturbOS (original final selection)504–14%8–19%46–53%

Full-pool STATE reaches 70% Top-60 versus 46–53% for PerturbOS and also leads at Top-10 and Top-20. These numbers support adopting exhaustive ranking for this small panel.

End-to-end exploratory RACK1 case: On the other hand, we ran the separate exploratory RACK1 case. The task asks which single-gene CRISPRi interventions decrease RACK1 RNA expression in K562. The frozen GEARS predictor uses the Replogle K562 essential-gene dataset, with 465 fitting interventions and a supported pool of 1,861 seen and unseen targets.

The table below shows the workflow's value in a broader space that includes unseen interventions:

MethodCandidate budgetCohort Top-10Cohort Top-20Cohort Top-60
GEARS ranking, all 1,861 targets1,86130%40%50%
GEARS ranking, 465 seen targets46525%40%60%
Random-50 + GEARS, 20 repeats505.5%8.5%20.75%
No feedback + LLM selection5019%32%45%
Mutated feedback + LLM selection5013%21%34%
PerturbOS (matched feedback)5046%61%75%

RACK1 gives 75% Top-60 for PerturbOS versus 50% for full-pool GEARS and 60% for seen-only GEARS; the corresponding Top-10/20 rates are also higher. This supports the complete workflow on this exploratory objective, without identifying which component causes the advantage.

Random-50 reports mean ± sample SD. The seen/unseen breakdown is:

RACK1/K562 method, Top-60RunsSeen hit rateUnseen hit rateOverall hit rate
GEARS ranking, all 1,861 targets160.0% (6 / 10)40.0% (4 / 10)50.0% (10 / 20)
GEARS ranking, 465 seen targets160.0% (12 / 20)N/A (0 / 0)60.0% (12 / 20)
Random-50 + GEARS, 20 repeats2023.4% (1.7 / 7.1)19.2% (2.45 / 12.4)20.8% (4.15 / 20)
No feedback + LLM selection549.4% (6 / 12.2)38.8% (3 / 7.8)45.0% (9 / 20)
Mutated feedback + LLM selection545.7% (4 / 9)31.0% (2.8 / 9.8)34.0% (6.8 / 20)
PerturbOS (matched feedback)581.5% (8.8 / 10.8)67.3% (6.2 / 9.2)75.0% (15 / 20)

The combined unseen hit rate is 67.3% for PerturbOS versus 40.0% for full-pool GEARS, alongside 81.5% versus 60.0% on seen targets. These rates support improved prioritization among both selected seen and selected unseen targets in this exploratory task.

Additionally, the separate shared-start comparison supports the value of correctly matched predictive information for the refinement-and-selection workflow:

MethodTOP2A Top-60RACK1 cohort Top-60
No feedback + LLM selection2% ± 2.7 pp45% ± 0.0 pp
Mutated feedback + LLM selection13% ± 9.7 pp34% ± 11.9 pp
PerturbOS (matched feedback)44% ± 4.2 pp75% ± 0.0 pp

Matched feedback exceeds no feedback by 42 percentage points on TOP2A (44% versus 2%) and 30 points on RACK1 (75% versus 45%). .

Takeaway & Claim Calibration: The results support exhaustive ranking for small, observed panels, whereas iterative prior-guided proposal adds clear value over exhaustive predictor sorting in larger candidate spaces with unseen perturbations

AuthorsOpenReview ↗

Q2 Reply: Annotation authorship and independent validation

Thank you for your willingness to understand our work in detail. One author supplied the datasets and verified the AI-generated scenarios and acceptable/preferred annotations through a process independent of PerturbOS development. The author then checked the benchmark ground truth for correctness and fairness. Annotations were finalized without access to PerturbOS’s routing decisions.

Acceptable pairs satisfy the question's requirements for organism, intervention modality, cellular context, candidate/readout coverage, model capabilities, and inputs. Preferred pairs prioritize context and coverage, favoring an exact context match before justified biological surrogates.

We applied the following criteria when verifying acceptable and preferred dataset–model pairs:

CriterionAnnotation rule
Biological compatibilityMatch the organism, intervention modality, and requested cellular context, or identify a justified transfer setting.
CoverageCheck whether the dataset/model supports the requested interventions and expression readout.
Generalization capabilityRequire capabilities appropriate to the question: unseen interventions, context transfer, or batch/donor transfer.
Required inputsCheck requirements such as target-context controls or chemical structures.
Preference among acceptable pairsFavor closer biological context and relevant coverage. Preference denotes suitability under this rubric, rather than measured predictive superiority.

Takeaway & Claim Calibration: The annotations reflect an explicit suitability rubric, author verification, and finalization without access to the system’s routing decisions.

AuthorsOpenReview ↗

Q3 Reply: Downstream prediction performance

Thank you for this question. Currently, our focus is that the resource selection must first satisfy the question’s required generalization, biological context, intervention modality, candidate and readout coverage, and input requirements.

Model suitability: RACK1/K562: For example, the broader RACK1/K562 question includes genetic interventions absent from fitting. GEARS was used because it supports predictions for these unseen interventions. The evaluated one-hot STATE configuration does not qualify for this scope because it can score only intervention identities represented during fitting.

Dataset suitability: HSP90AA1/A549: Another example would be to reduce HSP90AA1 expression in A549 cells, including untested compounds with known structures. sci-Plex3 A549 + chemCPA ranked above ComboSciPlex + chemCPA. Holding chemCPA fixed, sci-Plex3 provides broader measured single-compound coverage—188 observed labels compared with six single drugs in the inspected ComboSciPlex release—and a closer recorded biological-context match. These properties make it better suited to broad single-compound discovery.

Reviewer_H9XHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PerturbOS takes a natural-language intervention question, identifies the biological setting, selects a dataset-model pair from registries, prepares the model and uses predicted responses to guide repeated candidate selection. Its main experiment asks which drugs increase TOP2A RNA in A549 cells. STATE is fitted on sciPlex3 batch E09, then frozen; F10 control cells supply the target population and F10 treatments are reserved for evaluation. Five LLM backbones are tested with matched, permuted or absent predictive feedback. A separate bank of 37 questions evaluates direct analysis, indirect prediction, unsupported requests and resource selection.

Critical strengths
  1. The feedback controls test a real component of the proposed system

The study compares correctly matched predictions with the same kind of feedback permuted across drugs, as well as no feedback. In Table 2, the full workflow reaches Top-60 hit rates of 46-53% across the five backbones, while the permuted-feedback arms reach 17-28% and no-feedback arms 4-16%. This supports a specific conclusion: the association between a compound and its quantitative response helps correct candidate selection. Freezing the predictor before evaluating F10 treatments prevents the agent from learning from the retrospective evaluation labels during the search. The five trajectories characterize stochastic agent behavior conditional on that predictor.

  1. Resource compatibility is treated as part of the scientific workflow

The distinction between analyzing an observed response and invoking a model for an unmeasured setting is useful. The registries describe cell context, perturbation coverage, model inputs and implementation status, rather than listing model names alone. The registry-context baseline also makes the routing comparison more informative. Figure 3 shows where the gains and failures occur, including the loss on direct questions. These choices make the prototype easier to inspect and give a sensible starting point for a tool intended to prevent biologically inappropriate model use.

Critical weaknesses
  1. The principal discovery result could be explained by a simple predictor-based selector

Appendix B.5 says the frozen checkpoint is run on all 2,404 admitted F10 control cells for each of the 188 candidate compounds. The scalar TOP2A prediction therefore exists for every candidate. A deterministic selector can rank those 188 values and return the best twenty without any LLM search. That baseline is absent. For a single-gene objective, it is the first comparison needed to establish why the agent is useful after the predictor has been fitted.

The no-feedback and permuted-feedback arms answer whether accurate predictive information helps an LLM. They do not answer whether the LLM uses that information better than sorting it. There is also a filtering difference: prediction-feedback candidates are accepted only when the predicted effect is finite and positive, while no-feedback and uniform-random shortlists are not selected by this rule. A matched fifty-candidate random proposal pool, followed by the same filter and deterministic final selection, would isolate any benefit of sequential proposals. The result should be compared at the same number of scored candidates and final shortlist size.

Table 2's 49% versus 9% comparison for one backbone is striking, but the observed uniform-random Top-60 hit rate is 30%, close to the 60/188=31.9% expectation. The relevant practical question is how much the framework improves beyond an already useful frozen predictor. Section 5.2 reports TOP2A Spearman correlation of 0.766 and balanced directional accuracy of 0.822, yet only 0.20 overlap between predicted and measured top-ten compounds. Those results make a direct predictor-only ranking particularly informative and suggest that success on a broad top-sixty set should not be equated with reliably recovering the strongest candidates.

The task is also narrower than new-compound discovery. STATE uses one-hot drug identities and is trained on E09 measurements of the same candidate universe; the held-out challenge is transfer to F10. Ranking the observed E09 responses would show whether model-based transfer is needed. One question in one cell line, with dose-agnostic predictions assessed against pooled-dose effects, establishes a retrospective batch-transfer example. It does not yet demonstrate that the framework can discover unseen intervention identities or specify a useful dose for testing.

  1. The reported routing gain comes with a substantial failure on direct questions

Table 4 emphasizes route-accuracy improvements of 16.2 and 5.4 percentage points over the strongest LLM baselines. The class distribution is four direct, twenty-eight indirect and five unsupported questions. Always returning indirect would obtain 75.7% micro-accuracy, above both reported PerturbOS accuracies of 62.2% and 54.1%. Such a router would be scientifically inadequate, but the comparison exposes how strongly this metric depends on the largest class.

The supplied Opus 5 confusion matrix gives a better account. PerturbOS is correct on 1/4 direct, 14/28 indirect and 5/5 unsupported questions. The registry-context baseline gets 4/4, 9/28 and 5/5. The macro-average recall calculated from those counts is 58.3% for PerturbOS and 77.4% for the baseline. PerturbOS recovers five more indirect questions and loses three direct ones. The authors acknowledge the direct-case loss; it needs to shape the conclusion about improved routing, rather than remain a qualification after the aggregate result.

Absolute pair-selection reliability is also limited: acceptable dataset-model choices are obtained for 42.9-46.4% of indirect cases and preferred choices for 32.1-35.7%. Section 5.3 identifies target-gene coverage matching in nine of fifteen routing failures. This is precisely the compatibility issue the framework is intended to solve. Appendix C describes question construction but does not document an independent adjudication of the required capabilities and preferred pairs; expert-validated intermediate labels are left to future work. Finally, the executed discovery example uses the STATE adapter. Selecting GEARS or chemCPA in a registry question is not evidence that the corresponding preparation, training and evaluation actually succeed. A few complete execution traces across different route types would make the claimed end-to-end reliability assessable.

Questions for authors
  1. On the frozen E09/F10 experiment, what are the results for (i) directly ranking all 188 STATE TOP2A scores, (ii) sampling fifty candidates and applying the identical positive-effect filter and final selector, and (iii) ranking measured E09 effects? Compare these baselines at the Top-10/Top-20 thresholds already plotted in Figure 2 and at the primary Top-60 threshold, with matched candidate budgets and shortlist sizes.

  2. How do the routing conclusions change when the three routes receive equal weight or explicit error costs? The Opus 5 counts imply 58.3% versus 77.4% macro-recall for PerturbOS and registry-context. Please identify the three lost direct cases and explain what computation each wrong route would launch or prevent.

  3. Who established acceptable and preferred dataset-model pairs, and were labels adjudicated independently of framework development? Please show complete preparation/execution traces for representative non-STATE and direct-analysis cases, including whether target coverage was checked against local data rather than registry descriptions. This would test the advertised workflow with the resources already in the bank.

5 comments
AuthorsOpenReview ↗

Q1 Reply: Predictor-only and source-observation baselines

We sincerely thank the reviewer for highlighting the stronger performance of full-pool STATE ranking. Expanding the scored candidate pool from 50 to all 188 compounds improves shortlist quality in this setting. We will therefore adopt full-pool ranking when the supported candidate space is small and all requested perturbations are represented in the fitting data.

We report all three requested baselines alongside PerturbOS:

MethodCandidate scores or observationsTop-10Top-20Top-60
Full-pool STATE ranking188 scores20%30%70%
Random-50 + STATE, five runs50 scores5%14%46%
Observed E09 fitting-effect ranking188 observed effects30%40%80%
PerturbOS (original final selection)50 scores4–14%8–19%46–53%

End-to-end exploratory RACK1 case: Additionally, we ran the separate exploratory RACK1 case. The task asks which single-gene CRISPRi interventions decrease RACK1 RNA expression in K562. The frozen GEARS predictor uses the Replogle K562 essential-gene dataset, with 465 fitting interventions and a supported pool of 1,861 seen and unseen targets.

The table below shows the workflow's value in a broader space that includes unseen interventions:

MethodCandidate budgetCohort Top-10Cohort Top-20Cohort Top-60
GEARS ranking, all 1,861 targets1,86130%40%50%
GEARS ranking, 465 seen targets46525%40%60%
Random-50 + GEARS, 20 repeats505.5%8.5%20.75%
No feedback + LLM selection5019%32%45%
Mutated feedback + LLM selection5013%21%34%
PerturbOS (matched feedback)5046%61%75%

On RACK1, PerturbOS reaches 75% Top-60 versus 50% for full-pool GEARS and 20.75% for Random-50 + GEARS. Its Top-10/20 rates also exceed full-pool ranking (46/61% versus 30/40%), supporting shortlist quality across thresholds in this exploratory task.

Seen and unseen targets performance breakdone:

RACK1/K562 method, Top-60RunsSeen hit rateUnseen hit rateOverall hit rate
GEARS ranking, all 1,861 targets160.0% (6 / 10)40.0% (4 / 10)50.0% (10 / 20)
GEARS ranking, 465 seen targets160.0% (12 / 20)N/A (0 / 0)60.0% (12 / 20)
Random-50 + GEARS, 20 repeats2023.4% (1.7 / 7.1)19.2% (2.45 / 12.4)20.8% (4.15 / 20)
No feedback + LLM selection549.4% (6 / 12.2)38.8% (3 / 7.8)45.0% (9 / 20)
Mutated feedback + LLM selection545.7% (4 / 9)31.0% (2.8 / 9.8)34.0% (6.8 / 20)
PerturbOS (matched feedback)581.5% (8.8 / 10.8)67.3% (6.2 / 9.2)75.0% (15 / 20)

PerturbOS averages 15 hits per 20 slots versus 10 for full-pool GEARS, including 6.2 versus 4 unseen hits. Its unseen hit rate is 67.3% versus 40.0%, supporting useful prioritization beyond interventions represented during fitting.

Dependence on the predictive model: Observed E09 effects provide a strong baseline. We think the batch transfer in this case is still relatively strong, and perhaps existing models have yet to outperform it. However, PerturbOS is designed to accommodate this dependence on the predictive model. Its workflow orchestration is separated from model-specific adapters for data preparation, training, inference, and evaluation. Alternative predictors and their execution harnesses can therefore be integrated as model capabilities improve.

Takeaway & Claim Calibration: The results support exhaustive ranking for small, observed panels, whereas iterative prior-guided proposal adds clear value over exhaustive predictor sorting in larger candidate spaces with unseen perturbations.

AuthorsOpenReview ↗

Q2 Reply: Route weighting, error costs and lost direct cases

We thank the reviewer for their patience and attention to detail in reviewing our work. Equal-route weighting favors registry context in the original Opus 5 evaluation, whereas the stated illustrative costs favor PerturbOS among completed decisions; the three lost direct cases are S004, S006, and S008, with case retries recovering S004 and S008.

  • S004, A549/TOP2A: An unrequested novelty requirement inferred by Opus 5 in that run caused direct ranking to be replaced with ComboSciPlex + chemCPA.

  • S008, drug-tolerant PC9/AXL: Incomplete records and invalid capability labels returned by Opus 5 resulted in failure rather than a usable conclusion.

  • S006, basal K562/XBP1: Cell-state evidence remained unresolved because our strict harness did not match “K562” with “basal K562.” We believe this strictness may be a reasonable trade-off for biological integrity. As we add more diverse datasets to the registry, we believe it will help identify the best-performing dataset–model pairs.

Case repeats: We conducted two fresh Opus 5 repeats per case using the implementation, and both S004 and S008 were classified correctly. We consider the false negative in S006 an acceptable trade-off in overall recall.

Route weighting and error costs: When the three routes receive equal weight, Opus 5 + PerturbOS now achieves 75.0% macro recall, compared with 77.4% for the original registry-context baseline. With 4 direct cases, the equal-route weighting gives each direct-case error seven times the influence of an indirect-case error. We do think your other suggestion of explicit error costs is more informative for assessing practical consequences. We set the illustrative penalties based on the importance of consequences, unnecessary modeling u = 1, missed feasible analysis m = 2, unjustified execution v = 4,

The error cost is calculated as follows:

Observed case mix = (C_D + C_I + C_U) / (n_D + n_I + n_U).

Equal route weights = (C_D/n_D + C_I/n_I + C_U/n_U) / 3.

C_D, C_I, and C_U are the total penalties within each reference route, and n_D, n_I, and n_U are the corresponding numbers of completed decisions.

Error-cost matrix:

Reference route ↓ / Decision →DirectIndirectUnsupported
Direct0um
Indirectv0m
Unsupportedvv0
BackboneMethod / result setCompleted decisionsMean cost: observed case mix ↓Mean cost: equal route weights ↓
Opus 4.7Registry context37/371.5680.690
Opus 4.7PerturbOS37/371.0540.536
Opus 5Registry context36/371.4440.642
Opus 5PerturbOS, historical34/371.0000.632
Opus 5PerturbOS, partial update35/370.9430.494

Under the stated error penalties, PerturbOS achieves lower costs than registry context under both weighting schemes, supporting its practical value when routing errors have different consequences.

Takeaway & Claim Calibration: We thank the reviewer for pointing out the Opus 5 agent’s errors in cases S004 and S008. We have retained stricter biological verification for S006 and conducted error-cost calculations using the reported numbers.

AuthorsOpenReview ↗

Q3 Reply: Annotation provenance and complete execution traces

Thank you for this question. The annotation process and representative execution traces are documented. One author supplied the datasets and verified AI-generated scenarios and annotations through a process independent of PerturbOS development. Acceptable pairs satisfy task requirements; preferred pairs prioritize context and coverage. The benchmark ground truth is then checked by the author to ensure correctness and fairness. Annotations were finalized without access to PerturbOS’s routing decisions.

We applied the following criteria when verifying acceptable and preferred dataset–model pairs:

CriterionAnnotation rule
Biological compatibilityMatch the organism, intervention modality, and requested cellular context, or identify a justified transfer setting.
CoverageCheck whether the dataset/model supports the requested interventions and expression readout.
Generalization capabilityRequire capabilities appropriate to the question: unseen interventions, context transfer, or batch/donor transfer.
Required inputsCheck requirements such as target-context controls or chemical structures.
Preference among acceptable pairsFavor closer biological context and relevant coverage. Preference denotes suitability under this rubric, rather than measured predictive superiority.

Representative execution traces: The complete preparation/execution traces for representative non-STATE and direct-analysis cases:

WorkflowExecution and verification evidenceAssistance and scope
K562 + GEARSCompleted preparation, GEARS training, evaluation of frozen predictions and five HMBS discovery rounds, with readout, intervention and control coverage checked directly in local expression files. The fitted prediction cache was subsequently reused for the RACK1 prioritization reported in our response to Question 1, including interventions absent from fitting.Essential-gene development subset;
S002/G15, HSPA5 decrease in K562Completed direct ranking of 9,716 observed knockdowns; a separate reconstruction from source counts reproduced all candidate effects and cell counts across 31 numerical checks. In a separate day-6 K562 screen from the same study, all 16 evaluable top-20 candidates retained negative HSPA5 effects; rank correlation was 0.516 across 1,974 shared knockdowns.Original-question direct route; five preparation invocations, repairs, and RNA/day8 clarification

Target coverage check: We thank the reviewer for raising questions about the harness details. Coverage was verified against local data, not inferred solely from registry descriptions. Registry creation proceeds in two separate stages:

  1. First, the registry creation agent gathers dataset information from papers, Hugging Face, or websites and stores it in the registry to support routing decisions. At this stage, the dataset is marked as non-executable.

  2. Once a local dataset path is provided, the agent runs scripts on the local data to extract more detailed metadata, including target coverage. This verified metadata is then added to the dataset’s registry entry for further judgement.

Reviewer_WkTcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce an end-to-end agentic framework for perturbation response prediction. It retrieves appropriate resources to perform the requested analysis for prioritizing interventions. There are three components: scenario agent (process the query), dataset and model registry agents (retrieve resources), and routine agent (directly reply to query given resources without predictive modeling; indirectly reply by first building a predictive model; determine unsupported).

The agent is evaluated under three experimental setups: TOP2A drug prioritization, routed model preparation and held-out prediction, and registry-grounded routing.

Critical strengths

The authors evaluate on three different experimental setups.

PerturbOS and baselines are evaluated with different backbones (Claude Opus 4.7 and 5; GPT-5.6 and 6 Sol; DeepSeek Flash).

Critical weaknesses

The only baseline that PerturbOS is evaluated against is BDA and uniform random.

What proportion of the dataset only requires direct analysis? These samples are likely the easiest to predict. Could you stratify performance by whether the analysis was direct, indirect, or unsupported?

How extensive are the indirect analyses? In other words, how challenging are the settings in which indirect analyses are necessary to answer the query?

To test how prior biological knowledge could misguide performance, could you run sensitivity/ablation analyses in which you corrupt the correct biological priors? How does performance change?

In Table 3, the large standard deviations suggest that PerturbOS is comparable to both BDA strategies (i.e., not statistically significant). Additionally, PerturbOS is usually not the best-performing in Round 1. While PerturbOS seems to improve in Rounds 2-5, it's unclear how to decide the number of rounds to execute in practice. Could you share the performance of rounds 2-5 individually?

In Table 4, what is the standard deviation of the performance? Are the performance gains statistically significant?

Questions for authors

See above

9 comments
AuthorsOpenReview ↗

Reply1: The only baseline that PerturbOS is evaluated against is BDA and uniform random.

Dear Reviewer WkTc, we have now added full-pool predictor ranking and random proposals followed by predictor ranking, alongside the original PerturbOS workflow:

MethodCandidate exploredTop-10Top-20Top-60
Full-pool STATE ranking18820%30%70%
Random-50 + STATE, five runs505%14%46%
PerturbOS504–14%8–19%46–53%

Full-pool STATE reaches 70% Top-60 versus 46–53% for PerturbOS, a 17–24 percentage-point advantage. This supports exhaustive ranking on the small, already-scored panel; it explores the full panel of 188 candidates rather than the agent’s 50.

We have decided to adopt full-pool ranking when the requested candidate scope contains no unseen perturbations and the supported search space is small. Exhaustive ranking makes full use of the available predictions in this setting.

End-to-end exploratory RACK1 case: On the other hand, we ran the separate exploratory RACK1 case. The task asks which single-gene CRISPRi interventions decrease RACK1 RNA expression in K562. The frozen GEARS predictor uses the Replogle K562 essential-gene dataset, with 465 fitting interventions and a supported pool of 1,861 seen and unseen targets.

The table below shows the workflow's value in a broader space that includes unseen interventions:

MethodCandidate budgetCohort Top-10Cohort Top-20Cohort Top-60
GEARS ranking, all 1,861 targets1,86130%40%50%
GEARS ranking, 465 seen targets46525%40%60%
Random-50 + GEARS, 20 repeats505.5%8.5%20.75%
No feedback + LLM selection5019%32%45%
Mutated feedback + LLM selection5013%21%34%
PerturbOS5046%61%75%

On RACK1, PerturbOS reaches 75% Top-60 versus 50% for full-pool GEARS and 20.75% for Random-50 + GEARS. Its Top-10/20 rates also exceed full-pool ranking (46/61% versus 30/40%), supporting shortlist quality across thresholds in this exploratory task.

Seen and unseen targets performance breakdown:

RACK1/K562 method, Top-60RunsSeen hit rateUnseen hit rateOverall hit rate
GEARS ranking, all 1,861 targets160.0% (6 / 10)40.0% (4 / 10)50.0% (10 / 20)
GEARS ranking, 465 seen targets160.0% (12 / 20)N/A (0 / 0)60.0% (12 / 20)
Random-50 + GEARS, 20 repeats2023.4% (1.7 / 7.1)19.2% (2.45 / 12.4)20.8% (4.15 / 20)
No feedback + LLM selection549.4% (6 / 12.2)38.8% (3 / 7.8)45.0% (9 / 20)
Mutated feedback + LLM selection545.7% (4 / 9)31.0% (2.8 / 9.8)34.0% (6.8 / 20)
PerturbOS581.5% (8.8 / 10.8)67.3% (6.2 / 9.2)75.0% (15 / 20)

The combined unseen hit rate is 67.3% for PerturbOS versus 40.0% for full-pool GEARS, alongside 81.5% versus 60.0% on seen targets. These rates support improved prioritization among both selected seen and selected unseen targets in this exploratory task.

Takeaway & Claim Calibration: The results support exhaustive ranking for small, observed panels, whereas iterative prior-guided proposal adds clear value over exhaustive predictor sorting in larger candidate spaces with unseen perturbations.

AuthorsOpenReview ↗

Reply2: What proportion of the dataset only requires direct analysis? These samples are likely the easiest to predict. Could you stratify performance by whether the analysis was direct, indirect, or unsupported?

Thank you for this question. Only 4 of the 37 benchmark questions (10.8%) require direct analysis; 28 (75.7%) require indirect prediction, and 5 (13.5%) are unsupported.

Performance stratified by reference route is shown below. Each entry reports correctly routed questions out of the total in that route. The table distinguishes original results from the partial Opus 5 update described afterward.

BackboneMethod / result setDirectIndirectUnsupported
Opus 4.7Direct selection, original4/47/285/5
Opus 4.7Source lookup, original4/47/284/5
Opus 4.7Registry context, original4/48/285/5
Opus 4.7PerturbOS, original3/415/285/5
Opus 5Direct selection, original4/47/285/5
Opus 5Source lookup, original4/49/284/5
Opus 5Registry context, original4/49/285/5
Opus 5PerturbOS, partial update3/414/285/5

Updates from the manuscript: Following Reviewer H9XH’s suggestion, we double-checked the underperformance of Opus 5–backed PerturbOS on direct cases. We conducted two fresh Opus 5 repeats per case using the implementation. Both S004 and S008 were classified correctly in these repeats, suggesting that the original errors were isolated misjudgments.

S006, basal K562/XBP1**:** Cell-state evidence remained unresolved because our strict harness did not match “K562” with “basal K562.” We believe this strictness may be a reasonable trade-off for biological integrity. As we add more diverse datasets to the registry, we believe it will help identify the best-performing dataset–model pairs.

For detailed references, the original manuscript reports the benchmark composition and route-level evaluation in the following places:

  • Section 4.3, “Registry-Grounded Routing,” states the numbers of direct, indirect, and unsupported questions. The corresponding proportions are shown explicitly above.
  • Table 4 and its caption specify that model and dataset–model selection are evaluated only on the indirect questions; those gains therefore do not come from direct-analysis cases. Overall route and capability metrics use all questions.
  • Figure 3A, “Opus 5 routing and pair selection,” shows confusion counts by reference route for all four methods. Section 5.3, in the paragraph “PerturbOS helps resolve indirect cases,” explains that PerturbOS improves indirect routing while losing direct cases.

Takeaway & Claim Calibration: The main routing benefit is improved identification of indirect workflows and suitable dataset–model pairs; pair-selection scores are evaluated only on indirect questions, so this benefit cannot be explained by easy direct cases.

AuthorsOpenReview ↗

Reply3: How extensive are the indirect analyses? In other words, how challenging are the settings in which indirect analyses are necessary to answer the query?

Thank you for this question. The 28 indirect questions cover the following settings:

Required generalizationQuestions
Unseen genetic intervention6
Unseen chemical structure6
Known genetic intervention in a new cellular context5
Known compound in a new cellular context5
New experimental/sequencing batch4
New donor2

This distribution supports the stated breadth of predictive requirements rather than a benchmark composed mainly of direct lookups.

Generalization requirements and scope: These questions need different kinds of prediction. A model that transfers known drug responses between batches may still be unable to predict a previously untested compound. For context, batch, and donor transfer, the benchmark assumes that target-context controls are provided. Requests combining an unseen intervention with a new cellular context are labeled unsupported under the benchmark policy.

Comments on the paper

1 comment
AuthorsOpenReview ↗

With Appreciation: Our Response and Key Updates

We sincerely thank the Reviewers (WkTc, H9XH, apLH), Area Chair, Senior Area Chair, and Program Chairs for their rigorous and constructive evaluations. Thank you for the time and care you have devoted to reviewing our submission, PerturbOS: An End-to-End Agentic Framework for Prediction-Guided Single-Cell Perturbation Discovery. Your questions have helped us strengthen the evaluation and clarify where the framework is useful and what the current evidence supports.

We especially thank Reviewer WkTc for recognizing the breadth of our experimental settings and LLM backbones; Reviewer H9XH for highlighting the value of our feedback controls and resource-compatibility checks; and Reviewer apLH for recognizing the practical importance of resource suitability and dataset–model selection. We have used your concerns to guide six additions and clarifications:

  • Stronger baselines and a clearer choice of workflow. We added full-pool predictor ranking, random proposals followed by predictor ranking, and observed-data ranking. On TOP2A, full-pool STATE and observed E09 ranking achieve 70% and 80% Top-60 hit rates, respectively, versus 46–53% for the original PerturbOS workflow. These findings support our decision to adopt exhaustive ranking for small supported panels whose requested interventions are represented in the fitting data.
  • Evidence beyond TOP2A. The exploratory RACK1/K562 study extends predictive prioritization to genetic interventions, including targets absent from GEARS fitting. PerturbOS achieves a 75% overall cohort-reference hit rate, compared with 50% for full-pool GEARS and 20.75% for Random-50 + GEARS, with seen and unseen results reported separately. The assisted HSPA5 direct-analysis workflow ranks 9,716 observed knockdowns; all 16 evaluable members of its top-20 shortlist retain negative effects in a separate screen from the same study.
  • More direct tests of predictive feedback. Shared-start experiments compare matched, permuted, and absent feedback from identical initial candidate batches. These support the value of predictive information across the complete search-and-selection workflow, while leaving iteration’s isolated contribution unresolved.
  • More complete statistical analysis. We provide individual-round results, paired-change analyses, pooled comparisons, and five-run Opus 5 routing summaries. We distinguish improvement within a trajectory from superiority over another strategy, and significance from results that survive multiple-comparison correction.
  • A more precise account of routing performance. We report direct, indirect, and unsupported cases separately, characterize all 28 indirect questions, and examine macro recall and illustrative error costs. We identify the three original Opus 5 direct-route failures and distinguish case retries from the original benchmark. The conclusions emphasize indirect routing and resource selection while acknowledging sacrifices in small direct-route reliability.
  • Clearer annotation provenance and execution evidence. We document author verification of the annotations, the acceptable/preferred-pair criteria, and representative GEARS and HSPA5 execution traces with local-data coverage checks. We distinguish resource suitability from demonstrated predictive superiority.

Together, these additions support question-conditioned intervention prioritization and direct analysis using available perturbation resources. Our individual responses provide the detailed results, assistance required, and remaining limitations. We will incorporate these additions and clarifications into the revised manuscript. Thank you again for helping us make the contribution and its evidence clearer.

All Authors of Submission #39743

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.