acceptodds
Under review as a conference paper at ICLR 2027

Locked Evaluation Surfaces Transfer Failure and Sampling-Depth Entanglement in CRISPRi Perturbation-Effect Prediction

Abstract

Predicting how held-out target genes respond to CRISPRi perturbation, and whether such predictions transfer across biological screens, is hard to evaluate: a representation can be informative within one screen yet fail across screens, while endpoint definitions and design factors such as sampling depth differ between datasets. We evaluate a frozen Geneformer representation under a locked, pre-registered protocol—heads and model selection frozen before test evaluation, external outcome labels withheld until final unblinding, and analysis-governing decisions fixed before the evaluations they govern. In-distribution on the Virtual Cell Challenge (VCC), the frozen representation carries measurable predictive information beyond a dimension-matched random-feature control (ΔR² = +0.1645, 95% CI [+0.1375, +0.1920]), satisfying the pre-registered informativeness gate required before interpreting transfer. It then fails zero-shot transfer on both external screens (Spearman ρ = −0.139 and −0.267), lying below that control on each. Adding a predefined magnitude block improves the representation externally (Δρ = +0.032 and +0.143) but, under the frozen primary head, does not rescue transfer: both remain negative. A pre-registered, count-adjusted max-response secondary is positively associated with the outcome on both screens; we report it as correlational and secondary, not as a recovered magnitude signal. Finally, the VCC endpoint is strongly sample-size associated—a count-only linear model reaches R² = +0.4325, versus +0.2589 for the four magnitude scalars; adding those scalars to cell count improves R² by only +0.0017—so much of the aggregate-magnitude signal overlaps with cell count. A locked evaluation thus surfaces a transfer failure and a sampling-depth entanglement that a less controlled evaluation could obscure.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
CRISPRi, perturbation prediction, single-cell foundation models, Geneformer, distribution shift, zero-shot transfer, Perturb-seq, sampling depth

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Mu7FOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper explores a frozen Geneformer encoding of observed CRISPRi expression profiles and the transferability of effect-strength predictions across biological screens. The paper considers in-distribution performance, zero-shot transfer and effects due to sample size and endpoint construction. The paper shows the frozen encoding provides predictive information on in-distribution tasks, but is not effective out of distribution. The paper shows in-domain magnitude signal is entangled with sampling depth and cell count.

Critical strengths

The paper evaluates Geneformer representations across different settings, including in-domain and zero-shot cross-screen, which provides a helpful characterization of its predictive performance. The experiments with random-feature, compressed-expression and shuffled-feature controls add insight.

The paper also characterizes relationships due to endpoint, expression magnitude and cell count, identifying important limitations to interpreting the results.

Critical weaknesses

Something unclear to this reviewer is the significance of the task. The model receives observed post-perturbation expression profiles and predicts an output that can be directly calculated from those measurements. The paper doesn’t establish the practical value of this mapping, or why frozen Geneformer representations should be expected to generalize it across biological screens. Therefore, the negative result presented in the paper appears to be tied to this specific pipeline. A different, potentially more meaningful task is to predict the unobserved effect of a proposed perturbation. Perhaps the rebuttal could comment on this.

Another weakness appears to be around training and external evaluation. The training endpoint is calculated using all available cells and is shown to be strongly correlated with cell count. However, the external validation uses fixed-size subsampling. The reported generalization failure may reflect this difference rather than a limitation of the representation’s biological information. An experiment that reconstructs the source endpoint using the same fixed-size procedure and retrains the predictors would help assess whether the generalization failure remains when the endpoint construction is matches.

Finally, the text is rather dense in places, uses awkward terms, and sounds overly AI-written, making it hard to read as a human reader. For example, words like “locked”, “governed”, “resolved” aren’t natural in this context and require parsing which gets in the way of understanding. Also it’s hard to follow some of the experiments as described, like Line 186, “The in-distribution gate (H-FM1)” – what is the gate, and how does it work? Line 300 talks about a “capacity floor the licenses every comparison below” which is hard to follow. The paper mentions several times that the scientific conclusion is changed, but it doesn’t describe clearly what conclusion is being changed or what new scientific understanding is generated.

Questions for authors
  1. What is the intended practical use of predicting an endpoint that can be calculated directly from the observed post-perturbation measurements? If the task is primarily a representation probe, what new understanding does it provide about Geneformer?

  2. How does the generalisation compare when constructing the source endpoint using the same fixed-size subsampling procedure as the external endpoints and retraining the predictors? What evidence distinguishes biological generalisation failure from the difference in endpoint construction?

  3. The paper repeatedly states that the evaluation changes the scientific conclusion. Please clarify which conclusion is being revised, what replaces it, and what this implies for future modelling or evaluation.

Reviewer_qZrpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates if a frozen Geneformer representation can predict CRISPRi perturbation-effect strength across biological screens. Using a locked evaluation protocol, the authors find that Geneformer contains predictive information within the Virtual Cell Challenge (VCC) dataset but fails on two external screens. They also identify a strong association between the VCC prediction and sampling depth.

Critical strengths
  1. Thoughtful experimental controls. The authors use random-feature, shuffled, and compressed-expression comparisons to investigate where predictive information comes from.
  2. Interesting findings. Distinguishing in-distribution predictive performance from cross-screen generalization is important, and the sampling-depth finding raises important questions about what the VCC measures.
Critical weaknesses
  1. The training and external measures are not exactly the same. The VCC labels are computed using all available cells and batch-specific controls, while external labels use fixed-size subsampling and pooled controls. This distinction matters because the authors demonstrate that the VCC measure is associated with sampling depth, while the external measures are largely independent of it by construction. So the central finding of negative external correlations might actually reflect differences in prediction target rather than failures of generalization.
  2. The attribution to the Geneformer representation is not clear. Geneformer outperforms random features in-distribution, but is outperformed by classical models. Also, the compressed-expression comparator has similarly negative external correlations. These results suggest that the observed failure is not specific to Geneformer.
  3. The methodological contribution is quite narrow. This study evaluates one frozen foundation model, one primary training datasets, and two external screens. The broader methodological novelty is less clear.
Questions for authors
  1. Can you repeat the transfer experiment with VCC labels reconstructed using the same procedure as the external measures?
Reviewer_65VYOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents a novel evaluation protocol to evaluate the geneformer embedding on different CRISPi perturbation situation.

Critical strengths

Nope

Critical weaknesses
  1. This manuscript contributes nothing to the perturbation prediction community. It only conducts a very simple evaluation of fixed Geneformer embeddings, without even considering more recent models such as scGPT or STACK. Mainstream approaches to perturbation prediction involve fine-tuning models or using specialized models such as STATE. This evaluation protocol is unusual, and I cannot derive any insights from it.

  2. The motivation is missing. Why evaluate Geneformer rather than other models, and what is the purpose of evaluating the contribution of these fixed embeddings? The evaluation does not provide any useful takeaways.

Questions for authors
  1. Please explain the motivation for using only Geneformer rather than fixed embeddings from scGPT or STACK.

  2. The paper is difficult to read and full of jargon. The motivation is unclear, and the presentation of related work is also confusing.

  3. Please clarify why this evaluation is necessary and which scientific problem the authors aim to address.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.