acceptodds
Under review as a conference paper at ICLR 2027

CellChain: Benchmarking Cross-Scale Evidence Use for Perturbation Reasoning

Abstract

Recent advances in foundation models and large-scale cellular perturbation datasets have driven growing interest in using LLMs to predict and reason about cellular responses. Biological reasoning in this setting involves relating molecular mechanisms, cellular responses, and functional outcomes across scales. Evaluating foundation models therefore requires assessing not only whether their predictions are correct, but also how those predictions depend on the biological evidence provided. We introduce CellChain, an experimentally grounded benchmark comprising 590 biological cases and 2,360 matched evidence-condition instances across gene-dependency prediction, cytokine-axis identification, and drug-sensitivity ranking. CellChain links perturbation experiments and external biological resources through a provenance-aware evidence graph spanning molecular, pathway, transcriptional, and phenotypic information. Task-specific conditions vary evidence coverage, composition, or assignment while preserving questions, candidate answers, reference targets, and scoring rules. Across 18 foundation models, the highest accuracies under prespecified target-evidence conditions are 57.5%, 75.5%, and 70.8%, respectively, and broader evidence coverage does not consistently improve accuracy. On the drug-sensitivity benchmark cohort, matched reference histories outperform swapped histories by an average of 13.0 percentage points in cluster-weighted accuracy. Decision-transition analysis separates corrections, newly introduced errors, and wrong-answer switches, revealing substantial answer changes hidden by small net accuracy differences. CellChain provides controlled comparisons for attributing changes in model predictions to prespecified expansions of biological context, including changes in evidence coverage, composition, and biological assignment.

Primary area:
datasets and benchmarks
Keywords:
Cellular Perturbation, Biological Reasoning, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_4EZKOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

CellChain is a benchmark for evaluating how large language models use biological evidence in perturbation-related decisions. It comprises 590 cases across gene-dependency prediction, cytokine-axis identification, and drug-sensitivity ranking, with four matched evidence conditions per case. A provenance-aware evidence graph connects experimental measurements with external biological resources. Questions, candidate answers, and reference targets remain fixed while evidence coverage, composition, or assignment changes. Evaluation of 18 models combines endpoint accuracy with paired evidence contrasts and decision-transition analysis. The results show that additional evidence does not consistently improve accuracy and that small net performance changes can conceal substantial changes in individual predictions.

Critical strengths
  1. Controlled comparisons expose behavior hidden by accuracy. Holding decision tasks fixed while varying evidence enables informative paired analyses. Figure 3 shows that adding System context changes 24.1% of Cytokine predictions despite a net accuracy decrease of only 0.3 percentage points.
  2. Useful benchmark construction and transparent evaluation. The three tasks connect experimental targets with traceable evidence sources and cover 18 models. Biological-cluster bootstrap intervals and explicit limits on mechanistic interpretation help distinguish observed evidence dependence from stronger claims about reasoning.
Critical weaknesses
  1. Process annotations are selected using the recorded stimulus, while candidate answers expose ligand and receptor identities. Without genuine process-only, identity-masked, and permuted-process controls, the gain of 42.4 percentage points does not establish how much predictions depend on the measured cellular response.
  2. Joint reallocates the Process/System budget, and a simple drug reference-history rule already reaches 64.2% compound-weighted accuracy. The added value of combining biological evidence sources therefore remains uncertain, despite the paper's appropriate acknowledgment of these interpretive limits.
Questions for authors
  1. All current Cytokine conditions retain the measured transcriptional response, so the Process column in Table 6 is not a true process-only baseline. How much predictive value does the measured response add relative to a condition containing only the same process annotations?
  2. The Process condition produces a large gain over Response, but the comparison does not test whether that gain requires correctly matched process annotations. How does cytokine-axis accuracy change when these annotations are permuted across stimuli while the measured response and evidence length remain fixed?
  3. Table 8 provides a confidence interval for the aggregate Process-associated gain, but the Cytokine benchmark contains only 11 target axes with ten cases each. How consistent is this gain across individual axes, as assessed by per-axis accuracy differences and confidence intervals?
  4. Appendix D states that Process annotations are linked to the recorded cytokine stimulus, while the answer options contain explicit ligand and receptor names. What does a systematic audit show about the presence of ligand or receptor names, aliases, or other direct identity cues in the model-visible Process evidence?
Reviewer_3jQEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CellChain, a benchmark for evaluating how large language models utilize cross-scale biological evidence in cellular perturbation reasoning. It integrates three experimental perturbation datasets with 14 external biological resources, constructing 590 biological cases and 2,360 matched evidence-condition instances across three tasks: Gene Dependency Prediction, Cytokine-Axis Inverse Recovery, and Drug-Sensitivity Ranking. The main contribution is a controlled evidence evaluation framework that varies evidence coverage, composition, and biological assignment while keeping the underlying prediction questions and reference answers fixed. The authors also introduce decision-transition analysis to characterize corrections, newly introduced errors, and switches between incorrect answers. Experiments across 18 foundation models show that increasing biological evidence does not consistently improve prediction accuracy. Furthermore, substantial changes in individual predictions can occur even when aggregate accuracy changes very little. These findings highlight the importance of examining evidence-dependent prediction behavior beyond conventional endpoint accuracy.

Critical strengths

1. A systematic framework for evaluating evidence-dependent prediction behavior. The paper introduces a well-structured benchmark that systematically varies the biological evidence available to models while preserving the underlying prediction tasks. This enables controlled comparisons of how evidence coverage, composition, and biological assignment affect predictions. The integration of multiple biological resources and three complementary perturbation tasks also provides a useful foundation for studying evidence-dependent behavior across different biological contexts.

2. Informative analysis beyond conventional prediction accuracy. The decision-transition analysis provides useful insights that are not captured by aggregate accuracy alone. By distinguishing corrections, newly introduced errors, and wrong-answer switches, the authors demonstrate that relatively small changes in overall accuracy can conceal substantial changes in individual predictions. This offers a valuable additional perspective for evaluating model behavior under different evidence conditions.

Critical weaknesses

Limited ability to assess effective cross-scale evidence integration beyond evidence-dependent prediction behavior. The paper motivates the need to go beyond endpoint accuracy to better understand how LLMs utilize biological information across scales. However, while the proposed evaluation successfully characterizes how model predictions change under different evidence conditions, it does not sufficiently distinguish effective integration of complementary biological information from reliance on individual predictive cues.

This limitation is reflected in the main experimental findings:

Cytokine-Axis Inverse Recovery: The substantial improvement from adding biological-process annotations (+42.4 percentage points) may primarily reflect the predictive value of annotations associated with the target cytokine, rather than the integration of process knowledge with the observed transcriptional response. Without a Process-only condition, it remains unclear whether the measured transcriptional response contributes meaningful additional information when process annotations are available.

Drug-Sensitivity Ranking: The improvement of Matched over Swapped reference histories (+13.0 percentage points) demonstrates that model predictions are sensitive to the assignment of historical evidence. However, it does not establish whether models meaningfully integrate query-specific transcriptional responses with historical sensitivity information. This concern is particularly relevant given the uncertain improvement of Matched over Background (+6.2 percentage points, 95% CI [-0.3, +12.2]) and the 64.2% compound-weighted accuracy achieved by the simple reference-history heuristic reported in Appendix D.

Furthermore, although the decision-transition analysis reveals prediction changes that are obscured by aggregate accuracy, it provides limited insight into whether these changes reflect biologically meaningful evidence utilization or sensitivity to additional input information. The authors appropriately acknowledge that their evaluation does not directly identify underlying reasoning mechanisms. Nevertheless, this limitation remains important for a benchmark motivated by understanding cross-scale biological evidence use.

Questions for authors

Q1. Can the authors better distinguish the contributions of individual evidence sources from their integration in the Cytokine task?

Could the authors compare Process-only, Response-only, and Response+Process conditions? In particular, how much additional predictive value does the measured transcriptional response provide when target-linked process annotations are already available? A control using incorrectly assigned process annotations would also help clarify the extent to which predictions depend on target-associated cues rather than the integration of multiple evidence sources.

Q2. Can the authors demonstrate the incremental predictive value of combining query-specific responses with historical sensitivity evidence in the Drug task?

Could the authors compare History-only, Query Response-only, and Combined conditions under consistent evaluation protocols? Additionally, a direct comparison between LLM predictions and the simple reference-history heuristic, using the same cases and aggregation scheme, would help determine whether the models provide meaningful predictive improvements beyond historical sensitivity priors.

Q3. Can the authors further clarify whether the observed decision transitions reflect biologically informative evidence use?

The decision-transition analysis reveals substantial changes in predictions even when accuracy remains nearly unchanged. Could the authors provide additional controls, such as irrelevant or biologically mismatched evidence under comparable input conditions, to help distinguish responses to biologically relevant information from general prediction sensitivity? This would strengthen the interpretation of decision transitions as a diagnostic of the use of biological evidence.

Reviewer_TTLTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

CellChain benchmarks how biological evidence affects model predictions across three tasks: gene dependency prediction, cytokine-axis recovery, and drug sensitivity ranking. It varies evidence composition and assignment while keeping questions, answer choices, and reference answers fixed, measuring both accuracy and changes in individual predictions.

Critical strengths
  • S1. The paired design keeps questions, answer choices, reference answers, and scoring fixed, reducing confounding from task difficulty. Evidence provenance and experimental context are also preserved.
  • S2. The Drug Matched–Swapped comparison changes only how histories are assigned to candidates, holding query responses, history content, and instructions fixed. Reference histories are constructed from compounds disjoint from the query set.
  • S3. Prediction transition analysis reveals changes in individual decisions that can be masked by small net changes in accuracy.
Critical weaknesses
  • W1. The benchmark’s diagnostic contribution remains unclear.
    • In the Drug Matched condition, no model outperforms the mean-history baseline reported in Appendix D under the same compound-weighted evaluation. Case-level agreement with this rule is not analyzed, leaving unclear how model predictions differ from this simple baseline.
    • The contribution of measured responses remains unresolved. Cytokine gains mainly come from target-linked process annotations, without a control isolating the response contribution beyond those annotations. Gene and Drug also show no panel-average accuracy gain from adding measured responses.
  • W2. The input interventions are difficult to reproduce.
    • Some Drug contrasts change both history evidence and comparison instructions, but the abbreviated prompt examples do not specify these changes well enough to reconstruct the conditions independently.
    • The rules for candidate construction and evidence selection, ordering, and allocation are insufficiently specified. This is particularly consequential for Cytokine Joint, where adding system evidence reduces the available process evidence.
  • W3. The robustness of the Drug assignment effect remains unclear.
    • The confidence intervals account only for compound-level clustering, although cell models and reference histories are shared across compounds. Since the assignment-effect interval narrowly excludes zero, whether statistical support persists under this shared dependence needs verification.
    • The scope of inference needs clarification, particularly whether the Drug result is conditional on the fixed panel of cell models and histories. Sensitivity analyses should match that scope and establish whether the result is driven by a small subset of compounds.
Questions for authors
  • Q1. How often does each model agree with the mean-history rule and its reverse? Could negative assignment effects reflect confusion about lower AUC and higher percentiles both indicating greater sensitivity?
  • Q2. Do the Cytokine results persist when the measured response is removed or swapped while process annotations are held fixed?
  • Q3. Could you provide the full prompts for each condition, along with the rules for candidate construction and evidence selection, ordering, and allocation?
  • Q4. Does the Drug assignment effect remain statistically supported after accounting for shared cell models and histories? Is it stable under leave-one-compound-out analysis?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.