- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
CellChain is a benchmark for evaluating how large language models use biological evidence in perturbation-related decisions. It comprises 590 cases across gene-dependency prediction, cytokine-axis identification, and drug-sensitivity ranking, with four matched evidence conditions per case. A provenance-aware evidence graph connects experimental measurements with external biological resources. Questions, candidate answers, and reference targets remain fixed while evidence coverage, composition, or assignment changes. Evaluation of 18 models combines endpoint accuracy with paired evidence contrasts and decision-transition analysis. The results show that additional evidence does not consistently improve accuracy and that small net performance changes can conceal substantial changes in individual predictions.
Critical strengths
- Controlled comparisons expose behavior hidden by accuracy. Holding decision tasks fixed while varying evidence enables informative paired analyses. Figure 3 shows that adding System context changes 24.1% of Cytokine predictions despite a net accuracy decrease of only 0.3 percentage points.
- Useful benchmark construction and transparent evaluation. The three tasks connect experimental targets with traceable evidence sources and cover 18 models. Biological-cluster bootstrap intervals and explicit limits on mechanistic interpretation help distinguish observed evidence dependence from stronger claims about reasoning.
Critical weaknesses
- Process annotations are selected using the recorded stimulus, while candidate answers expose ligand and receptor identities. Without genuine process-only, identity-masked, and permuted-process controls, the gain of 42.4 percentage points does not establish how much predictions depend on the measured cellular response.
- Joint reallocates the Process/System budget, and a simple drug reference-history rule already reaches 64.2% compound-weighted accuracy. The added value of combining biological evidence sources therefore remains uncertain, despite the paper's appropriate acknowledgment of these interpretive limits.
Questions for authors
- All current Cytokine conditions retain the measured transcriptional response, so the Process column in Table 6 is not a true process-only baseline. How much predictive value does the measured response add relative to a condition containing only the same process annotations?
- The Process condition produces a large gain over Response, but the comparison does not test whether that gain requires correctly matched process annotations. How does cytokine-axis accuracy change when these annotations are permuted across stimuli while the measured response and evidence length remain fixed?
- Table 8 provides a confidence interval for the aggregate Process-associated gain, but the Cytokine benchmark contains only 11 target axes with ten cases each. How consistent is this gain across individual axes, as assessed by per-axis accuracy differences and confidence intervals?
- Appendix D states that Process annotations are linked to the recorded cytokine stimulus, while the answer options contain explicit ligand and receptor names. What does a systematic audit show about the presence of ligand or receptor names, aliases, or other direct identity cues in the model-visible Process evidence?