acceptodds
Under review as a conference paper at ICLR 2027

CrossPerturb: A Cross-Context Benchmark and Evidence-Grounded Framework for LLM-Based Virtual Cell Perturbation Prediction

Abstract

Predicting cellular responses to drug perturbations is fundamental to virtual cell modeling and therapeutic discovery. While large language models (LLMs) show growing promise for biological reasoning, existing benchmarks evaluate them solely within the same cellular context. Consequently, simple voting baselines achieve comparable performance to complex LLM reasoning frameworks, obscuring whether models perform genuine biological reasoning or merely exploit majority labels. This limitation highlights the urgent need for more rigorous evaluation benchmarks that test whether LLMs can perform mechanistic reasoning in virtual cell modeling. To bridge this gap, we introduce CrossPerturb, a comprehensive benchmark for evaluating drug perturbation prediction across diverse cellular contexts. Derived from the LINCS L1000 and Tahoe datasets, CrossPerturb assesses transcriptional outcomes across diverse cell lines, drug dosage, and exposure durations. Alongside the benchmark, we introduce ARCS, an evidence-grounded reasoning framework that **A**nalyzes biological pathways underlying perturbations, **R**etrieves cross-context evidence, **C**ritiques candidate evidence, and **S**ynthesizes the final response score through mechanism-constrained aggregation. Extensive experiments demonstrate that our evidence integration framework surpasses current state-of-the-art methods, whereas existing LLM reasoning frameworks struggle to achieve robust performance, confirming that cross-context biological reasoning remains an open challenge for current LLMs.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Virtual Cell Modeling, Large Language Models, Cross-Context Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_njjcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CrossPerturb benchmarks differential-expression and response-direction prediction under drug and context shifts. It uses L1000 and Tahoe, disjoint source/target drug identities, and eight settings: two within-context references and six cell-line, dose, or duration shifts. Population- and gene-level metrics test reliance on gene-specific response priors. The accompanying ARCS workflow retrieves knowledge and experimental observations, critiques them, and combines an evidence score with an empirical prior. Comparisons and ablations show gains on several selected-response classification tasks, especially gene-level direction prediction. The benchmark is useful, but the role of reasoning versus additional evidence access needs clearer separation.

Critical strengths
  1. The benchmark addresses a meaningful evaluation problem. Drug holdouts, context shifts, and gene-level metrics help distinguish contextual prediction from gains driven by majority labels. Using two data sources and several shift settings makes it a useful resource for testing this distinction.

  2. ARCS provides inspectable evidence alongside measurable classification gains. Retrieval provenance and critique make its predictions easier to audit than unsupported biological explanations. In Table 2, gene-level direction scores reach 0.56 on both datasets, versus 0.47 and 0.41 for voting. These improvements are valuable even without a claim of mechanistic understanding.

Critical weaknesses
  1. Reasoning is not cleanly separated from access to evidence. Voting uses a restricted observation set, while ARCS combines multichannel retrieval, graph information, descriptions, and an empirical prior (PDF pp. 18–20). Sharing a graph with GAT does not ensure identical observations or derived information. Figure 4 partially attributes the gains to Analyze/Critique; removing Critique even improves L1000-DE slightly. An information-matched non-LLM reader is still needed to isolate the reasoning contribution.

  2. Several implementation details affect how the comparisons should be read. Binary AUROC is valid here and equals balanced accuracy (Appendix D). Equation 8 sets the ARCS threshold to 0.5. Appendix C states that expression models use benchmark DE criteria, but their application to generated profiles and validation of fusion settings remain unclear. Retrieval is also described inconsistently. Section 4.3.2 and Appendix E cap three criteria at five records each, whereas Appendix I uses four quotas (10/15/5/5) and reports 17 candidates.

  3. The benchmark tests a restricted set of responses. It excludes experiments with fewer than 15 DEGs, selects response genes, and evaluates direction conditional on DEG status (Appendix A). Gene-level metrics pool four context settings within each dataset (Appendix D). The direction-flip example in Appendix I and hard negatives in shifted-context DE queries are useful controls. However, the results do not establish performance on weak-response experiments or how often ARCS succeeds when response directions change across contexts.

Questions for authors
  1. How does ARCS compare with a weighted vote or learned non-LLM reader given exactly the same candidate observations and graph-derived information? This would separate evidence access and selection from the language-based reasoning stages.

  2. Which source-only validation split selected fusion and retrieval settings? What replicate units and variance estimates were used when applying DE criteria to generated profiles? Appendix C already excludes target-query labels from selection. Please reconcile the three-channel cap with Appendix I's four quotas and 17 candidates, identifying the configuration used for the main tables.

  3. Using existing predictions, can gene-level results be separated by context setting and by drug–gene pairs whose observed directions change across contexts? Please include class counts. This would extend the individual direction-flip example and show how broadly the gains depend on context-sensitive decisions.

8 comments
AuthorsOpenReview ↗

Response to Reviewer njjc — Part 1/3

We thank the reviewer for the constructive feedback. We have conducted additional experiments, clarified methodological details, and revised the manuscript to address the five key concerns. Our point-by-point responses are provided below.

  1. Non-LLM Baselines Comparison (Q1, W1)

We conducted additional experiments on L1000-DIR to compare ARCS with two non-LLM baselines, AdaBoost and Weighted Voting, using the same candidate observations, biological descriptions, KG-derived information, and empirical prior. AdaBoost performs poorly, suggesting that a naive evidence integration approach struggles to reconcile heterogeneous information from multiple sources and filter out noisy evidence, capabilities that LLM-based reasoning in ARCS is designed to provide. Weighted Voting, which assigns weights to retrieved observations according to their similarity to the target query, performs better than Simple Voting in most contexts but remains inferior to ARCS. As shown below, ARCS achieves the highest AUROC across all four context settings (L1–L4), reaching 0.78, 0.63, 0.65, and 0.73, respectively. These results highlight the advantage of ARCS in integrating heterogeneous evidence and assessing its relevance to the target query. Relevant results are added in Section 5.5 of the revised manuscript.

MethodL1L2L3L4
Adaboost0.550.520.570.51
Simple Voting0.680.610.630.65
Weighted Voting0.720.610.610.67
ARCS0.780.630.650.73
  1. Detailed Implementation (Q2, W2)

We clarify the hyperparameter selection process, the conversion of generated expression profiles into DE/DIR predictions, and the retrieval configuration.

  • Source-based hyperparameter selection. (1) We selected the fusion weight λ using 10,296 eligible queries from the L1000 source data, each with at least five related perturbation records available in the remaining source observations. For each query, its own observed response was excluded from the reference observations used for retrieval and prior estimation. The default (λ=0.25) achieved an AUROC of 0.784 in this source-data evaluation. (2) We set α=0.75 based on the biological consideration that perturbation-semantic similarity is more closely related to the underlying regulatory mechanisms shared by perturbations, while cellular-context similarity provides complementary contextual information. Accordingly, we assign a higher weight to perturbation-semantic similarity and choose 0.75 as the midpoint of the predefined range ([0.5,1.0]), rather than tuning it against target-test performance. The held-out test results further support the reasonableness of this choice.
λ0.00.150.250.500.751.0
AUROC0.5610.6930.7840.7830.7650.726
  • DE/DIR conversion for generative baselines. For scVI, ChemCPA, and State, we converted their predicted continuous expression profiles into binary DE/DIR predictions by comparing the predicted perturbed profiles with the corresponding control profiles (gene expression before perturbation). For DE, a gene is classified as differentially expressed if it satisfies the benchmark's standardized DE criteria, namely a Benjamini–Hochberg adjusted q-value ≤ 0.05 and |log2FC| ≥ 0.36. For DIR, only differentially expressed genes are considered, with the sign of the gene-wise log-fold change determining upregulation or downregulation. These criteria follow the benchmark label definitions described in Section 3 and Section 4.1, while Appendix C (Lines 967–970) explicitly states that continuous profiles generated by scVI, ChemCPA, and State are converted using the same standardized DE criteria. For completeness, Appendix A (Lines 673–682) details how the experimental gold labels are constructed from pseudobulk treatment-versus-control contrasts across biological replicates using limma moderated t-statistics.

  • Retrieval configuration. We acknowledge that the original description in Section 4.3.2 was inaccurate and clarify that the retrieval budget applies to the total candidate pool across all channels, rather than imposing an independent top-five limit on each channel. The Analyze module generates a query-specific retrieval plan that allocates the overall budget among channels according to query relevance and candidate availability. Accordingly, the four values (10/15/5/5) reported in Appendix I specify the channel-wise retrieval quotas under this adaptive allocation scheme. We have revised Section 4.3.2 to correct this description and ensure consistency between the main text and Appendix I.

AuthorsOpenReview ↗

Response to Reviewer njjc — Part 2/3

  1. Context-Dependent Direction-flip (Q3, W3)

We clarify the scope of our benchmark evaluation by providing an additional analysis of context-dependent direction flips and explaining the rationale for the response-gene selection criterion.

  • Weak-response. We set a minimum of 15 DEGs per experimental condition to maintain a balanced number of perturbation pairs across drugs, as many drugs in our benchmark induce relatively few DEGs. Weak responses are less commonly studied as a separate prediction category, since statistical significance is typically the primary focus of DE analysis. In our benchmark, gene-level non-significant responses are already included in the negative class of the DE task. However, we do not separately evaluate weak responses as a distinct response-intensity category.

  • Context-dependent direction flips. To further evaluate context-dependent direction prediction, we identified (drug, gene) pairs that appear across multiple test contexts and exhibit direction-label reversals. Since the training and test sets contain completely disjoint drugs, this analysis was conducted exclusively on the test set, yielding 158 such pairs in L1000-DIR. Using L1 (the context represented in the training data) as the reference, we evaluated the label flip in the shifted contexts. Among these pairs, 80 were flipped to up-regulated and 78 to down-regulated. As shown below, ARCS achieves the best overall performance, with an accuracy of 0.57, AUROC of 0.57, and F1 score of 0.58, outperforming VCWorld, GAT, and Simple Voting. This additional evaluation provides evidence that ARCS better captures context-dependent direction reversals. Relevant results are added in Section 5.3 of the revised manuscript.

MethodAccAUROCF1Trueup/down
ARCS0.570.570.589047/43
VCWorld0.430.430.376826/42
GAT0.510.510.528141/40
Simple Voting0.480.480.527644/32
AuthorsOpenReview ↗

Response to Reviewer njjc — Part 3/3

  1. Knowledge-Source Provenance Audit (Q4, W4)

We clarify the provenance of KG records and retrieved descriptions, detail the controls against experimental overlap, and quantify both direct and multi-hop overlap with evaluated test pairs to assess potential label leakage. Complete details of this audit are provided in Section 5.6 of the revised manuscript.

Knowledge-source provenance. The expression and activity records are primarily derived from CTD-curated chemical-gene interactions. CTD specifies these interactions using an explicit outcome vocabulary, with each record mapped to one KG edge while preserving the original vocabulary term in its detail attribute. For example, "doxorubicin, decreases expression, A2M" is represented as a doxorubicin → A2M edge with the predicate inhibits and sign −1, whereas "doxorubicin, increases expression, AAMP" is mapped to activates with sign +1. When CTD reports an effect without specifying its direction, as in "doxorubicin, affects expression, ABAT", the edge is assigned the predicate modifies and remains unsigned. The reviewer's 86.6% corresponds precisely to these CTD-sourced compound-gene edges: 2,170,808 of 2,506,072 in total, comprising 901,277 activates, 804,668 inhibits, 261,481 interacts_with, and 203,382 modifies edges.

Other records with these predicates originate from DGIdb drug-label interaction verbs and smaller sets of repoHub, DrugBank, and GtoPdb records. Separately, STITCH contributes scored chemical-protein binding associations, represented by the binds relation rather than expression records. Every edge retains its source database list, which is also included in the retrieved sentence, allowing each piece of evidence to be traced to its supporting records.

Overlap. To assess overlap beyond direct edges, we examined the shortest KG path between each evaluated (drug, gene) pair in the test sets. Beyond the 1-hop paths mentioned, we present the 2-hop and 3-hop paths classified by their shortest path lengths, which together account for the majority of cases. We further assessed whether a path could directly determine the direction label, requiring every edge along the path to have an explicit direction and sign. As shown in the table, only a small fraction of test pairs have such label-determinable paths. The final column reports how many KG-inferred directions agree with the ground-truth labels, with percentages calculated over all test pairs.

TaskTest pairs2-hop3-hopDeterminableCorrect
L1000-DE62,71113,683 (21.82%)34,506 (55.02%)4,190 (6.68%)1,068 (1.70%)
L1000-DIR29,8876,533 (21.86%)16,592 (55.52%)2,066 (6.91%)1,068 (3.57%)
Tahoe-DE74,75318,878 (25.25%)38,487 (51.49%)4,473 (5.98%)1,224 (1.64%)
Tahoe-DIR37,7709,852 (26.08%)18,413 (48.75%)2,344 (6.21%)1,234 (3.27%)

Although multi-hop connections are common, only 5.98–6.91% of test pairs have paths that can determine a direction label, and the inferred directions match the ground truth for only 1.64–3.57% of all test pairs. These results indicate that multi-hop KG connectivity does not translate into direct access to ground-truth labels and thus does not provide a direct source of label leakage into the final predictions.

  1. Contribution of LLM-Based Evidence (Q5, W5)

We conducted an analysis on the L1000 dataset to quantify how ARCS modifies predictions from the empirical prior. With λ = 0.25, ARCS assigns 25% weight to retrieved evidence and 75% to the empirical prior. On L1000-DE, ARCS changes 58.4% of the prior's predictions, correcting 21,805 decisions while introducing 14,421 errors. On L1000-DIR, it changes 38.8% of the prior's predictions, correcting 8,653 decisions while introducing 2,694 errors, corresponding to a 3.2:1 correction-to-error ratio. These results show that retrieved evidence does not simply override the empirical prior, but provides complementary information that improves its predictions, particularly for direction prediction.

We agree that the selected mechanistic paths have not been independently validated as biological mechanisms. We therefore limit their interpretation to evidence retrieval, relevance assessment, and selection, and have revised the manuscript to avoid implying that these paths establish mechanistic validity.

Reviewer_f1PaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This article introduces CrossPerturb, a benchmark built on the LINCS L1000 and Tahoe datasets for predicting drug-induced gene responses across cellular contexts. Compounds are split into disjoint source and target sets to test generalization to unseen cell lines, doses, and exposure durations, using population-level and per-gene metrics on differential-expression and direction-of-change tasks. The authors also propose ARCS, an LLM-guided framework that analyzes mechanisms over a biomedical knowledge graph, retrieves and critiques cross-context evidence, and aggregates retained evidence with empirical gene-response priors. ARCS outperforms existing LLM reasoning frameworks and baselines—more so on direction of change than on differential expression—with ablations and a case study supporting the design. A key finding is that within-context benchmarks let models exploit gene-specific response tendencies, so simple voting can beat advanced LLM frameworks, and cross-context reasoning remains an open challenge.

Critical strengths
  1. The benchmark addresses an important evaluation gap—generalization to unseen cellular contexts (cell line, dose, duration) rather than unseen compounds in familiar environments—and its gene-level evaluation exposes performance driven by aggregate response tendencies. Notably, it demonstrates the problem is real: simple voting beats advanced LLM frameworks, so cross-context reasoning remains an open challenge.
  2. ARCS offers a more inspectable pipeline than unconstrained LLM predictions: analysis, retrieval, critique, and aggregation are separated and individually checkable, with a case study recording evidence, critique decisions, and final scores, and typed knowledge-graph relations grounding the reasoning. Ablations that remove each module help attribute the gains.
Critical weaknesses
  1. The reported metrics appear inconsistent with the stated evaluation definitions. Appendix D states that AUROC is computed from binary predictions and is therefore equivalent to balanced accuracy, yet for at least one baseline setting the reported AUROC cannot be reconciled with the reported accuracy and F1 values under the stated confusion-matrix definitions, and conventional rounding does not account for the gap. Different aggregation schemes, prediction sets, or score-based AUROCs might explain the discrepancy, but the current description does not. Reproducible metric computation and a clarification of how predictions map to labels are needed before reliable performance claims can be made.
  2. The implementation description is not fully resolved. With the default fusion setting, the interpolation assigns most of the weight to the empirical prior, yet the appendix characterizes the configuration as anchored primarily by structured relational reasoning—a mismatch that leaves the source of the method's gains unclear, and the ablations do not compare LLM-based evidence selection against matched non-LLM selection under the same candidate pool and prior. The retrieval specification is also internally inconsistent: the main text defines a small number of matching criteria with a per-criterion cap on candidates, while the case study describes more retrieval sources with different quotas and a larger resulting pool. These descriptions need to be reconciled to establish which process produced the reported results.
Questions for authors
  1. Can you reconcile the reported results for a simple baseline across the main and appendix tables? Appendix D states that AUROC is computed from binary decisions; please confirm this applies to all reported values, and provide the confusion matrix and exact evaluation code or pseudocode, clarifying how the metrics can be inconsistent under the stated definitions.
  2. For the generative baselines, what exact model configurations, checkpoints or training procedures, and representations for unseen target compounds were used? The stated conversion criteria rely on significance testing over replicates; what served as replicates or variance estimates when converting a single predicted expression profile into DE significance and direction labels?
  3. Which retrieval specification was actually used: the criteria described in the method, or the sources with variable quotas described in the case study? Please also specify the critique quotas and deterministic fallback behavior. Since the hyperparameter sensitivity analysis reports gene-level results across all context settings, including held-out regimes, please state the exact selection protocol and confirm that no target-side performance informed the choice.
  4. Does the improvement persist against a matched non-LLM evidence-selection baseline using the same knowledge graph, descriptions, candidate pool, and empirical prior? Reporting macro per-gene results separately for the held-out-context regimes, rather than pooled across settings, would also help establish the claimed context-sensitive contribution.
6 comments
AuthorsOpenReview ↗

Response to Reviewer f1Pa — Part 1/3

We sincerely thank the reviewer for the constructive feedback and insightful questions. In response, we provide detailed metric calculations, clarify the evaluation protocols, and validate the effectiveness of LLM-based evidence selection.

  1. Metrics Computation (Q1, W1)

To clarify the metric computation, We use GAT's results to illustrate the confusion matrices on L1000 dataset. Reported AUROC values are computed from binary prediction, following the evaluation protocol of previous LLM-based reasoning frameworks, such as VCWorld. GAT and other baselines were evaluated using three random seeds (40/41/42), with mean and standard deviation reported. We show seed 42 as a representative example; its results fall within the ranges in Tables 1, 9, and 10.

GAT confusion matrices and metric computation.

For L1000-DE, the three prediction outcomes (up, down, and no-DE) are converted into two classes: DE (up or down) and no-DE. DE is treated as the positive class and no-DE as the negative class. We then compute Accuracy, AUROC, and F1 from the resulting binary confusion matrix:

Acc=TP+TNTP+FP+FN+TN,\mathrm{Acc}=\frac{TP+TN}{TP+FP+FN+TN}, AUROC=12(TPTP+FN+TNTN+FP),\mathrm{AUROC}=\frac{1}{2}\left(\frac{TP}{TP+FN}+\frac{TN}{TN+FP}\right), F1=2TP2TP+FP+FN.\mathrm{F1}=\frac{2TP}{2TP+FP+FN}.

The resulting confusion matrices and metrics for L1000-DE are:

TPFPFNTNAccAUROCF1
L1103979391184339720.5610.5730.649
L247844892157219170.5090.5170.597
L33522377785811970.5040.5220.603
L450635378184823000.5050.5160.584

For L1000-DIR, the gold labels are up or down, whereas GAT can predict up, down, or no-DE. We denote these outcomes by uu, dd, and cc, respectively. Specifically, uuuu represents gold-up genes predicted as up; udud represents gold-up genes predicted as down; ucuc represents gold-up genes predicted as no-DE; dudu represents gold-down genes predicted as up; dddd represents gold-down genes predicted as down; and dcdc represents gold-down genes predicted as no-DE.

Unlike L1000-DE, the three metrics for DIR are computed differently because the original predictions have three possible outcomes. Accuracy counts only exact up/down matches as correct:

AccDIR=uu+ddn,\mathrm{Acc}_{DIR}=\frac{uu+dd}{n},

where n=uu+ud+uc+du+dd+dcn=uu+ud+uc+du+dd+dc. F1 is the macro-average of the up-class and down-class F1 scores, with no-DE predictions counted as errors for their corresponding gold classes:

F1DIR=12(2uu2uu+du+ud+uc+2dd2dd+ud+du+dc).\mathrm{F1}_{DIR}=\frac{1}{2}\left(\frac{2uu}{2uu+du+ud+uc}+\frac{2dd}{2dd+ud+du+dc}\right).

For AUROC, we use a binary up-versus-down comparison, treating up as the positive class. No-DE predictions are folded into the non-up (negative) class. Thus, the confusion matrix is defined as: TP=uu,FP=du,FN=ud+uc,TN=dd+dc.TP=uu,\quad FP=du,\quad FN=ud+uc,\quad TN=dd+dc. AUROC is then calculated as the balanced accuracy of this binary classification:

AUROC=12(TPTP+FN+TNTN+FP).\mathrm{AUROC}=\frac{1}{2}\left(\frac{TP}{TP+FN}+\frac{TN}{TN+FP}\right).

The original prediction counts and the resulting metrics for L1000-DIR are summarized as follows:

uuuuududucucdududddddcdcTPFPFNTNAccAUROCF1
L1325611049291023501491432561023203359280.6760.7340.724
L2176675082874015287441766740157822720.5180.6410.591
L3103560049849813893601035498109817490.5530.6320.610
L4184155389667919909521841679144929420.5540.6860.640

This distinction is particularly relevant for genes classified as no-DE at the preceding DE stage and subsequently entering DIR as no-change predictions. A gold-up gene predicted as no-DE is an FN for both the up-class F1 and binary AUROC. In contrast, a gold-down gene predicted as no-DE is an FN for the down-class F1 but a TN for binary AUROC. Thus, the different treatment of no-DE predictions across metrics can contribute to apparent discrepancies among their values. We provide the complete metric definitions for DE and DIR task (especially for three class prediction) in the supplementary material to make these conventions and calculations explicit. We have added the relevant definitions to Appendix D.

Macro-per-gene evaluation. Unlike population-level evaluation, which pools predictions across genes, macro-per-gene evaluation computes metrics separately for each gene and then takes their unweighted average, giving each gene equal weight regardless of query count. This requires a separate confusion matrix for each gene; since over 900 genes are evaluated, we do not list all matrices here. AUROC includes only genes with both positive and negative queries; Accuracy and F1 include all genes with at least one query, with undefined F1 assigned zero. For overall gene-level results, queries from all four context settings are pooled before per-gene computation. For example, Simple Voting on L1000-DE yields 59 queries for LGMN (TP = 16, FP = 11, FN = 10, TN = 22), giving per-gene Acc = 0.644, AUROC = 0.641, and F1 = 0.604. The reported gene-level metrics are the unweighted averages across genes.

AuthorsOpenReview ↗

Response to Reviewer f1Pa — Part 2/3

  1. Baseline Configurations and DE/DIR Conversion (Q2)

We clarify the configurations and training procedures of the generative baselines, the representation of unseen target compounds, and the procedure for deriving DE/DIR predictions from continuous expression outputs.

  • Baseline configurations. ChemCPA was trained from scratch using its officially provided default model architecture. State was initialized with the model parameters released by the original ARC implementation and subsequently fine-tuned. For scVI, we used scVI as a cell-state encoder and incorporated the compound as a conditioning variable to generate perturbation-induced expression profiles. For compound representation, all methods consistently used RDKit embeddings, a commonly adopted, training-free molecular representation derived directly from compound structures, which allows unseen target compounds to be represented without requiring compound-specific learned embeddings.
  • DE/DIR label conversion. For methods that generate continuous expression profiles, we converted their predictions into binary DE/DIR labels using the same standardized differential expression criteria adopted during CrossPerturb dataset construction, as explicitly stated in Appendix C (Baseline Details), p. 18, Lines 967–970. Specifically, we compared the predicted perturbed expression profiles with the corresponding control profiles, quantified gene-wise changes using log-fold changes, and determined regulation directions from the signs of these changes. The statistical criteria follow the benchmark label construction procedure described in Appendix A (Label Construction), p. 13, Lines 673–677: differential expression statistics were computed from pseudobulk treatment-versus-control contrasts across biological replicates using limma moderated t-statistics, and a gene was designated as differentially expressed if its Benjamini–Hochberg adjusted p-value satisfied (q \leq 0.05) and its absolute log-fold change satisfied (|\log_2 FC| \geq 0.36). The same golden label construction procedure is also summarized in Section 3, p. 4, Lines 185–190, where the use of pseudobulk expression contrasts, limma-based differential expression analysis, and the sign of expression changes for determining regulation directions is described.
  1. Retrieval Strategy, Critique, and Hyperparameter Protocol (Q3, W2)

We clarify the retrieval and fallback procedures in ARCS, and provide further details on evidence fusion and hyperparameter settings.

  • Query-adaptive retrieval. We acknowledge that the fixed (k=5) description in the original manuscript was inaccurate and have revised it. Before retrieval, the Analyze stage generates a query-specific retrieval plan by ordering retrieval channels and assigning quotas based on an availability profile, which records candidate counts for the target gene, distinct compounds, and context compatibility. The planned quotas follow predefined constraints and are subsequently clipped by the actual candidate availability, with unavailable channels omitted and unmet quotas recorded. The total retrieval pool is bounded by the data-source capacity (15–40 for L1000 and 8–30 for Tahoe). We have revised the retrieval strategy in Section 4.3.2.

  • Critique and fallback. Critique selects a non-empty subset of retrieved candidates. If its output is invalid or empty, ARCS deterministically selects the top five candidates according to the original context-similarity ordering, consistent with the Simple Voting baseline. No additional retrieval is performed, and each fallback event is logged with its triggering reason. This occurs in 17.3% of DIR queries and 21.7% of DE queries.

  • Hyperparameter protocol. (1) We set α=0.75 based on the biological consideration that perturbation-semantic similarity is more closely related to the underlying regulatory mechanisms shared by perturbations, while cellular-context similarity provides complementary contextual information. Accordingly, we assign a higher weight to perturbation-semantic similarity and choose 0.75 as the midpoint of the predefined range ([0.5,1.0]), rather than tuning it against target-test performance. The held-out test results further support the reasonableness of this choice. (2) We selected λ using 10,296 eligible queries from the L1000 source data, each with at least five related perturbation records retrievable from the remaining source data. The default setting λ=0.25 achieved an AUROC of 0.784 on this source-data evaluation. We retained this prior-dominant setting to allow retrieved evidence to complement the empirical prior, rather than letting potentially noisy retrieved evidence dominate the prediction. Its performance on held-out target datasets further supports the suitability of this default setting.

λ0.00.150.250.500.751.0
AUROC0.5610.6930.7840.7830.7650.726
AuthorsOpenReview ↗

Response to Reviewer f1Pa — Part 3/3

  1. Matched Non-LLM Evidence Selection and Context-Specific Evaluation (Q4, W2)

To isolate the contribution of LLM-based evidence selection, we added matched non-LLM baselines using the same biological and experimental information, candidate pool, and empirical prior as ARCS. We also report gene-level macro AUROC across the four context regimes (L1000 L1–L4), providing a more detailed assessment of context-specific performance. We have added the relevant comparative experiments to Section 5.3 and 5.5.

  • Context-specific gene-level evaluation. We report gene-level macro AUROC separately for L1–L4, in addition to the pooled results. ARCS achieves the highest pooled performance on both DIR (0.5564) and DE (0.5248), outperforming the strongest competing baseline, KG-GAT, by 4.41 and 1.86 percentage points, respectively. For DIR, ARCS also achieves the best performance in all four context regimes.
MethodL1L2L3L4Pooled
DIRSimple Voting0.50860.43600.44140.40760.4660
DIRGAT0.52470.48670.50550.49730.5123
DIRVCWorld0.45770.44990.45110.45030.4743
DIRARCS0.60390.50110.51670.53530.5564
DESimple Voting0.53380.49050.48900.46020.5059
DEGAT0.51760.50020.50630.49560.5062
DEVCWorld0.42770.42950.43030.42770.4287
DEARCS0.54530.49600.50630.51210.5248
  • Matched non-LLM evidence-selection baselines. To ensure a controlled comparison, we further evaluated three non-LLM strategies on L1000-DIR using the same input evidence and empirical prior. AdaBoost directly predicts from encoded drug/gene descriptions, retrieved observations, and KG-derived drug–target relations. Same-gene aggregation uses retrieved observations sharing the target gene, while Voting aggregates all retrieved evidence without LLM-based assessment. ARCS consistently outperforms all three alternatives across L1–L4 and gene-level macro evaluation, achieving a macro score of 0.56 compared with 0.52, 0.53, and 0.54, respectively. This comparison indicates that the improvement is not simply attributable to access to biological information or retrieved evidence, but also to the LLM-guided assessment of candidate relevance, which helps filter contextually irrelevant evidence and mitigate retrieval noise.
MethodL1L2L3L4Macro
Adaboost0.550.520.570.510.49
same genes0.720.590.590.650.53
Voting0.740.610.600.660.54
ARCS0.780.630.650.730.56
Reviewer_wjyDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

CrossPerturb introduces compound-disjoint drug response classification benchmarks from LINCS L1000 and Tahoe, with shifts in cell line, dose, and exposure time. ARCS uses KG-informed descriptions and Qwen3-8B to propose hypotheses, retrieve source observations, and critique their applicability. It then averages the retained labels and blends them with a gene / context prior. The reported setup uses API-based LLM inference, with no task-specific LLM fine-tuning described.

Critical strengths
  1. A useful evaluation of held-out compounds and multiple context shifts. The benchmark explicitly varies cell line, dose, and time across complementary datasets. Hard negatives and per-gene reporting address weaknesses of the aggregate-only evaluation.

  2. ARCS leads the listed baselines in 7 out of 8 population DIR settings and improves macro per-gene DIR scores.

Critical weaknesses
  1. Appendices E and H describe Qwen3-8B prompting through the official API, with no task-specific LLM fine-tuning reported. Comparing ARCS and VCWorld under that shared setup is useful, but does not show the necessity of ARCS's particular solution. PertReason is a relevant training-based alternative using KG-grounded reasoning supervision. Its results cannot be transferred directly to this benchmark, but the design could be learned from. We can see DE gains are mixed, and macro per-gene scores remain 0.52/0.53 for DE and 0.56/0.56 for DIR. If we can see performance improvements while fine-tuning a general LLM with biologically solid KG, then we can draw a more concrete conclusion that the KG is good.

  2. The framework exposes hypotheses and KEEP/DROP decisions, but the experiments mainly evaluate final labels. There is no independent assessment of whether a hypothesis preserves the supplied biology, an observation justifies transfer to the target context, or a critique rejects an inappropriate analogue for the right reason. Removing Analyze or Critique establishes predictive usefulness, rather than correctness of the mechanism or transfer warrant. The direction flip case's correct labels do not validate the pharmacological analogy between the selected drugs. CrossPerturb needs an explicit assessment suited to its hypotheses and transfer decisions. Otherwise, its claimed reasoning improvement remains largely inferred from prediction gains.

Note: PertReason is a relatively new paper, so the authors do not need to directly compare with it or cite it. The reviewer refers to this paper because it reflects some current structural flaws in ARCS.

Questions for authors
  1. Is the intended contribution an effective inference-only evidence-selection procedure, or a broader conclusion about the limits of LLM perturbation reasoning? The reported setup does not include task-specific LLM fine-tuning. How should that limit the interpretation of the mixed DE results and modest macro per-gene scores?

  2. What observable characteristics defines evidence-grounded reasoning beyond improved prediction, and what existing evidence assesses it? In particular, can the method distinguish a biologically justified transfer from an inappropriate evidance that happens to produce the correct label?

  3. What do the existing results establish about the benefit of the generated hypotheses and critique beyond a simpler structured retrieval / voting method using the same biological and experimental information? The component removals are useful, but do not establish that the text-space mechanism is correct or needed.

3 comments
AuthorsOpenReview ↗

Response to Reviewer wjyD — Part 1/2

We sincerely thank the reviewer for the constructive feedback and insightful questions. In response, we provide additional comparisons and analyses to clarify the contributions of CrossPerturb, establish the benefits of KG-guided evidence integration, and further examine the biological grounding and evidence-selection process of ARCS.

  1. Main contributions and comparison with fine-tuned and alternative LLM baselines(Q1, W1)

(1) Main contributions.

Our study reveals a fundamental limitation of existing perturbation prediction benchmarks: simple statistical approaches can achieve state-of-the-art performance, raising questions about whether high-performing models genuinely capture the biological mechanisms underlying perturbation responses or merely exploit recurring statistical patterns. To address this issue, we introduce a cross-context perturbation benchmark, CrossPerturb, to assess generalization across diverse cellular and experimental contexts, together with ARCS, an LLM-based evidence integration framework that leverages biological knowledge to support more reliable cross-context predictions.Specifically, our contributions are two-fold:

  • Identifying limitations of existing perturbation benchmarks and establishing CrossPerturb. Existing perturbation benchmarks primarily evaluate models within familiar cellular contexts, where simple statistical approaches can achieve state-of-the-art performance. This raises questions about whether strong benchmark performance reflects a genuine ability to capture the biological mechanisms underlying perturbation responses or instead relies on recurring response patterns within familiar contexts. To address this limitation, we introduce CrossPerturb, a benchmark designed to assess generalization across diverse experimental contexts. It incorporates drug-disjoint splits and cross-context evaluation for both differential expression (DE) and direction of change (DIR) prediction, evaluating model's generalization to unseen cellular contexts.

  • Developing an evidence-grounded framework for cross-context prediction. We propose ARCS, an LLM-guided framework that analyzes perturbation-related biological knowledge, retrieves relevant evidence across contexts, evaluates its compatibility with the target context, and integrates selected evidence with empirical priors. By explicitly assessing evidence relevance rather than relying on straightforward evidence aggregation, ARCS combines complementary biological knowledge and empirical response patterns to support more reliable cross-context predictions.

(2) Comparison with fine-tuned and alternative LLM baselines.

We further compare ARCS with fine-tuning models (with and w/o KG) and VCWorld under its original backbone (Gemini-2.5-Flash). These comparisons demonstrate the contribution of KG evidence for perturbation prediction and the superior performance of ARCS across the evaluated settings. The results on the L1000 DIR task are summarized in the table above, with the corresponding experimental details and analysis provided in Section 5.6 and Appendix H of the revised manuscript.

  • Fine-tuning with and w/o KG. We fine-tune Qwen3-8B using LoRA with the same drug and gene descriptions and supervised question–answer pairs. Notably, fine-tuned models without KG underperform simple voting, suggesting that task-specific fine-tuning alone is insufficient to address cross-context perturbation prediction. Nevertheless, incorporating KG information consistently improves population-level AUROC across all four settings (L1–L4) and increases gene-level macro AUROC from 0.47 to 0.51, confirming the benefit of structured biological knowledge.
  • VCWorld with Gemini-2.5-Flash. Although using its original backbone improves VCWorld's gene-level macro AUROC to 0.49, ARCS achieves 0.56 and outperforms VCWorld in all population-level settings.
MethodPopulation-level(AUROC)Gene-level(AUROC)
L1L2L3L4macro
Fine-tune, w/o KG0.660.570.630.630.47
Fine-tune, with KG0.700.580.680.680.51
VCWorld (Qwen3-8B)0.640.530.560.480.47
VCWorld (Gemini-2.5-Flash)0.680.550.550.520.49
ARCS (Qwen3-8B)0.780.630.650.730.56
AuthorsOpenReview ↗

Response to Reviewer wjyD — Part 2/2

  1. Biological Grounding and Evidence-Selection Audit (W2, Q2)

Before addressing this concern, we clarify that ARCS is a KG-guided cross-context evidence integration framework, rather than a biological mechanism-reasoning framework like VCWorld or SUMMER. Its Analyze and Critique modules organize candidate mechanistic hypotheses and assess the applicability of retrieved cross-context observations to the target perturbation, primarily to facilitate evidence integration. We appreciate the importance of PertReason, which highlights the need for systematic construction and evaluation of biological mechanism ground truth, an important open challenge in virtual-cell perturbation reasoning. We have cited and discussed PertReason in the revised manuscript. However, its curated perturbation mechanisms do not cover the full scope of CrossPerturb. Constructing the required mechanism-level ground truth and establishing fair baseline comparisons following its evaluation protocol are beyond the time available for this rebuttal. We therefore commit to incorporating this evaluation, following PertReason's methodology, in the final version.

To address this concern, we audited all 29,887 L1000 DIR queries, examining the biological grounding of Analyze-generated hypotheses and the rationale and label consistency of Critique's evidence-selection decisions. We further evaluated ARCS on genuine cross-context queries with reversed response labels to assess its ability to transfer evidence when response directions differ across contexts. Complete details of this audit and evaluation are provided in Section 5.6 of the revised manuscript.

  • Analyze. We found that 89% of drug-side target relations in the generated hypotheses could be directly traced to BioRelKG, while 99.8% of gene-side regulatory relations had supporting biological connections in the KG. For drug-side associations without direct KG support, Analyze uses semantic similarity between drug and gene descriptions to generate candidate hypotheses. We further examined the KG connectivity between the drugs and target genes in the L1000-DIR queries. 79.5% of the drug–gene pairs are connected by paths of up to three hops, indicating that the hypotheses generated by Analyze are largely supported by closely connected biological knowledge in the KG.

  • Critique. Among the DROP decisions, 68.7% were attributed to insufficient biological or experimental support, including weak drug–gene or pathway associations and substantial incompatibilities in cellular context. Another 29.7% involved conflicting evidence associated with cross-context changes, such as inconsistent response directions across experimental conditions. Furthermore, 76.3% of KEEP evidence had labels consistent with the target query, compared with 43.0% of DROP evidence. This difference supports the rationale of Critique's evidence-selection mechanism. However, the mechanistic validity of individual KEEP/DROP decisions remains difficult to establish.

  • Cross-context transfer evaluation (dinstinct lables). On genuine cross-context queries with reversed response labels, ARCS consistently outperformed VCWorld, GAT, and Simple Voting across all three metrics. This demonstrates ARCS's advantage in cross-context evidence transfer over conventional KG-based methods and plausible LLM-based reasoning approaches.

MethodAccAUROCF1
ARCS0.570.570.58
VCWorld0.430.430.37
GAT0.510.510.52
Simple Voting0.480.480.52
  1. Comparison with Rule-Based Baselines (Q3)

We further compare ARCS with three baselines using the same biological and experimental information on L1000-DIR. AdaBoost directly predicts the output from the same input evidence, with drug and gene descriptions encoded as vectors, retrieved observations represented by (drug, gene, context, label), and KG-derived drug–target relations used as features. Same-gene aggregates retrieved observations sharing the target gene, while Voting directly aggregates all retrieved evidence. ARCS consistently outperforms all three baselines across the four population-level settings and gene-level macro evaluation. These results highlight the advantage of ARCS in using LLM-based reasoning to progressively assess retrieved evidence, identify contextually irrelevant information, and mitigate noise that simpler rule-based methods struggle to resolve.

MethodL1L2L3L4Macro
Adaboost0.550.520.570.510.49
same genes0.720.590.590.650.53
Voting0.740.610.600.660.54
ARCS0.780.630.650.730.56
AuthorsOpenReview ↗

Follow-Up on Our Rebuttal Response – Submission57623

Dear Reviewer,

Thank you for your valuable feedback on our paper. We have carefully addressed your concerns in our rebuttal and would greatly appreciate it if you could review our responses.

If you have any remaining questions or concerns, please feel free to let us know. We would be happy to provide further clarifications or make additional revisions.

Thank you again for your time and consideration.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.