- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CrossPerturb benchmarks differential-expression and response-direction prediction under drug and context shifts. It uses L1000 and Tahoe, disjoint source/target drug identities, and eight settings: two within-context references and six cell-line, dose, or duration shifts. Population- and gene-level metrics test reliance on gene-specific response priors. The accompanying ARCS workflow retrieves knowledge and experimental observations, critiques them, and combines an evidence score with an empirical prior. Comparisons and ablations show gains on several selected-response classification tasks, especially gene-level direction prediction. The benchmark is useful, but the role of reasoning versus additional evidence access needs clearer separation.
Critical strengths
-
The benchmark addresses a meaningful evaluation problem. Drug holdouts, context shifts, and gene-level metrics help distinguish contextual prediction from gains driven by majority labels. Using two data sources and several shift settings makes it a useful resource for testing this distinction.
-
ARCS provides inspectable evidence alongside measurable classification gains. Retrieval provenance and critique make its predictions easier to audit than unsupported biological explanations. In Table 2, gene-level direction scores reach 0.56 on both datasets, versus 0.47 and 0.41 for voting. These improvements are valuable even without a claim of mechanistic understanding.
Critical weaknesses
-
Reasoning is not cleanly separated from access to evidence. Voting uses a restricted observation set, while ARCS combines multichannel retrieval, graph information, descriptions, and an empirical prior (PDF pp. 18–20). Sharing a graph with GAT does not ensure identical observations or derived information. Figure 4 partially attributes the gains to Analyze/Critique; removing Critique even improves L1000-DE slightly. An information-matched non-LLM reader is still needed to isolate the reasoning contribution.
-
Several implementation details affect how the comparisons should be read. Binary AUROC is valid here and equals balanced accuracy (Appendix D). Equation 8 sets the ARCS threshold to 0.5. Appendix C states that expression models use benchmark DE criteria, but their application to generated profiles and validation of fusion settings remain unclear. Retrieval is also described inconsistently. Section 4.3.2 and Appendix E cap three criteria at five records each, whereas Appendix I uses four quotas (10/15/5/5) and reports 17 candidates.
-
The benchmark tests a restricted set of responses. It excludes experiments with fewer than 15 DEGs, selects response genes, and evaluates direction conditional on DEG status (Appendix A). Gene-level metrics pool four context settings within each dataset (Appendix D). The direction-flip example in Appendix I and hard negatives in shifted-context DE queries are useful controls. However, the results do not establish performance on weak-response experiments or how often ARCS succeeds when response directions change across contexts.
Questions for authors
-
How does ARCS compare with a weighted vote or learned non-LLM reader given exactly the same candidate observations and graph-derived information? This would separate evidence access and selection from the language-based reasoning stages.
-
Which source-only validation split selected fusion and retrieval settings? What replicate units and variance estimates were used when applying DE criteria to generated profiles? Appendix C already excludes target-query labels from selection. Please reconcile the three-channel cap with Appendix I's four quotas and 17 candidates, identifying the configuration used for the main tables.
-
Using existing predictions, can gene-level results be separated by context setting and by drug–gene pairs whose observed directions change across contexts? Please include class counts. This would extend the individual direction-flip example and show how broadly the gains depend on context-sensitive decisions.
8 comments
Response to Reviewer njjc — Part 2/3
- Context-Dependent Direction-flip (Q3, W3)
We clarify the scope of our benchmark evaluation by providing an additional analysis of context-dependent direction flips and explaining the rationale for the response-gene selection criterion.
-
Weak-response. We set a minimum of 15 DEGs per experimental condition to maintain a balanced number of perturbation pairs across drugs, as many drugs in our benchmark induce relatively few DEGs. Weak responses are less commonly studied as a separate prediction category, since statistical significance is typically the primary focus of DE analysis. In our benchmark, gene-level non-significant responses are already included in the negative class of the DE task. However, we do not separately evaluate weak responses as a distinct response-intensity category.
-
Context-dependent direction flips. To further evaluate context-dependent direction prediction, we identified (drug, gene) pairs that appear across multiple test contexts and exhibit direction-label reversals. Since the training and test sets contain completely disjoint drugs, this analysis was conducted exclusively on the test set, yielding 158 such pairs in L1000-DIR. Using L1 (the context represented in the training data) as the reference, we evaluated the label flip in the shifted contexts. Among these pairs, 80 were flipped to up-regulated and 78 to down-regulated. As shown below, ARCS achieves the best overall performance, with an accuracy of 0.57, AUROC of 0.57, and F1 score of 0.58, outperforming VCWorld, GAT, and Simple Voting. This additional evaluation provides evidence that ARCS better captures context-dependent direction reversals. Relevant results are added in Section 5.3 of the revised manuscript.
| Method | Acc | AUROC | F1 | True | up/down |
|---|---|---|---|---|---|
| ARCS | 0.57 | 0.57 | 0.58 | 90 | 47/43 |
| VCWorld | 0.43 | 0.43 | 0.37 | 68 | 26/42 |
| GAT | 0.51 | 0.51 | 0.52 | 81 | 41/40 |
| Simple Voting | 0.48 | 0.48 | 0.52 | 76 | 44/32 |
Response to Reviewer njjc — Part 3/3
- Knowledge-Source Provenance Audit (Q4, W4)
We clarify the provenance of KG records and retrieved descriptions, detail the controls against experimental overlap, and quantify both direct and multi-hop overlap with evaluated test pairs to assess potential label leakage. Complete details of this audit are provided in Section 5.6 of the revised manuscript.
Knowledge-source provenance. The expression and activity records are primarily derived from CTD-curated chemical-gene interactions. CTD specifies these interactions using an explicit outcome vocabulary, with each record mapped to one KG edge while preserving the original vocabulary term in its detail attribute. For example, "doxorubicin, decreases expression, A2M" is represented as a doxorubicin → A2M edge with the predicate inhibits and sign −1, whereas "doxorubicin, increases expression, AAMP" is mapped to activates with sign +1. When CTD reports an effect without specifying its direction, as in "doxorubicin, affects expression, ABAT", the edge is assigned the predicate modifies and remains unsigned. The reviewer's 86.6% corresponds precisely to these CTD-sourced compound-gene edges: 2,170,808 of 2,506,072 in total, comprising 901,277 activates, 804,668 inhibits, 261,481 interacts_with, and 203,382 modifies edges.
Other records with these predicates originate from DGIdb drug-label interaction verbs and smaller sets of repoHub, DrugBank, and GtoPdb records. Separately, STITCH contributes scored chemical-protein binding associations, represented by the binds relation rather than expression records. Every edge retains its source database list, which is also included in the retrieved sentence, allowing each piece of evidence to be traced to its supporting records.
Overlap. To assess overlap beyond direct edges, we examined the shortest KG path between each evaluated (drug, gene) pair in the test sets. Beyond the 1-hop paths mentioned, we present the 2-hop and 3-hop paths classified by their shortest path lengths, which together account for the majority of cases. We further assessed whether a path could directly determine the direction label, requiring every edge along the path to have an explicit direction and sign. As shown in the table, only a small fraction of test pairs have such label-determinable paths. The final column reports how many KG-inferred directions agree with the ground-truth labels, with percentages calculated over all test pairs.
| Task | Test pairs | 2-hop | 3-hop | Determinable | Correct |
|---|---|---|---|---|---|
| L1000-DE | 62,711 | 13,683 (21.82%) | 34,506 (55.02%) | 4,190 (6.68%) | 1,068 (1.70%) |
| L1000-DIR | 29,887 | 6,533 (21.86%) | 16,592 (55.52%) | 2,066 (6.91%) | 1,068 (3.57%) |
| Tahoe-DE | 74,753 | 18,878 (25.25%) | 38,487 (51.49%) | 4,473 (5.98%) | 1,224 (1.64%) |
| Tahoe-DIR | 37,770 | 9,852 (26.08%) | 18,413 (48.75%) | 2,344 (6.21%) | 1,234 (3.27%) |
Although multi-hop connections are common, only 5.98–6.91% of test pairs have paths that can determine a direction label, and the inferred directions match the ground truth for only 1.64–3.57% of all test pairs. These results indicate that multi-hop KG connectivity does not translate into direct access to ground-truth labels and thus does not provide a direct source of label leakage into the final predictions.
- Contribution of LLM-Based Evidence (Q5, W5)
We conducted an analysis on the L1000 dataset to quantify how ARCS modifies predictions from the empirical prior. With λ = 0.25, ARCS assigns 25% weight to retrieved evidence and 75% to the empirical prior. On L1000-DE, ARCS changes 58.4% of the prior's predictions, correcting 21,805 decisions while introducing 14,421 errors. On L1000-DIR, it changes 38.8% of the prior's predictions, correcting 8,653 decisions while introducing 2,694 errors, corresponding to a 3.2:1 correction-to-error ratio. These results show that retrieved evidence does not simply override the empirical prior, but provides complementary information that improves its predictions, particularly for direction prediction.
We agree that the selected mechanistic paths have not been independently validated as biological mechanisms. We therefore limit their interpretation to evidence retrieval, relevance assessment, and selection, and have revised the manuscript to avoid implying that these paths establish mechanistic validity.
Response to Reviewer njjc — Part 1/3
We thank the reviewer for the constructive feedback. We have conducted additional experiments, clarified methodological details, and revised the manuscript to address the five key concerns. Our point-by-point responses are provided below.
We conducted additional experiments on L1000-DIR to compare ARCS with two non-LLM baselines, AdaBoost and Weighted Voting, using the same candidate observations, biological descriptions, KG-derived information, and empirical prior. AdaBoost performs poorly, suggesting that a naive evidence integration approach struggles to reconcile heterogeneous information from multiple sources and filter out noisy evidence, capabilities that LLM-based reasoning in ARCS is designed to provide. Weighted Voting, which assigns weights to retrieved observations according to their similarity to the target query, performs better than Simple Voting in most contexts but remains inferior to ARCS. As shown below, ARCS achieves the highest AUROC across all four context settings (L1–L4), reaching 0.78, 0.63, 0.65, and 0.73, respectively. These results highlight the advantage of ARCS in integrating heterogeneous evidence and assessing its relevance to the target query. Relevant results are added in Section 5.5 of the revised manuscript.
We clarify the hyperparameter selection process, the conversion of generated expression profiles into DE/DIR predictions, and the retrieval configuration.
DE/DIR conversion for generative baselines. For scVI, ChemCPA, and State, we converted their predicted continuous expression profiles into binary DE/DIR predictions by comparing the predicted perturbed profiles with the corresponding control profiles (gene expression before perturbation). For DE, a gene is classified as differentially expressed if it satisfies the benchmark's standardized DE criteria, namely a Benjamini–Hochberg adjusted q-value ≤ 0.05 and |log2FC| ≥ 0.36. For DIR, only differentially expressed genes are considered, with the sign of the gene-wise log-fold change determining upregulation or downregulation. These criteria follow the benchmark label definitions described in Section 3 and Section 4.1, while Appendix C (Lines 967–970) explicitly states that continuous profiles generated by scVI, ChemCPA, and State are converted using the same standardized DE criteria. For completeness, Appendix A (Lines 673–682) details how the experimental gold labels are constructed from pseudobulk treatment-versus-control contrasts across biological replicates using limma moderated t-statistics.
Retrieval configuration. We acknowledge that the original description in Section 4.3.2 was inaccurate and clarify that the retrieval budget applies to the total candidate pool across all channels, rather than imposing an independent top-five limit on each channel. The Analyze module generates a query-specific retrieval plan that allocates the overall budget among channels according to query relevance and candidate availability. Accordingly, the four values (10/15/5/5) reported in Appendix I specify the channel-wise retrieval quotas under this adaptive allocation scheme. We have revised Section 4.3.2 to correct this description and ensure consistency between the main text and Appendix I.