- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper explores a frozen Geneformer encoding of observed CRISPRi expression profiles and the transferability of effect-strength predictions across biological screens. The paper considers in-distribution performance, zero-shot transfer and effects due to sample size and endpoint construction. The paper shows the frozen encoding provides predictive information on in-distribution tasks, but is not effective out of distribution. The paper shows in-domain magnitude signal is entangled with sampling depth and cell count.
Critical strengths
The paper evaluates Geneformer representations across different settings, including in-domain and zero-shot cross-screen, which provides a helpful characterization of its predictive performance. The experiments with random-feature, compressed-expression and shuffled-feature controls add insight.
The paper also characterizes relationships due to endpoint, expression magnitude and cell count, identifying important limitations to interpreting the results.
Critical weaknesses
Something unclear to this reviewer is the significance of the task. The model receives observed post-perturbation expression profiles and predicts an output that can be directly calculated from those measurements. The paper doesn’t establish the practical value of this mapping, or why frozen Geneformer representations should be expected to generalize it across biological screens. Therefore, the negative result presented in the paper appears to be tied to this specific pipeline. A different, potentially more meaningful task is to predict the unobserved effect of a proposed perturbation. Perhaps the rebuttal could comment on this.
Another weakness appears to be around training and external evaluation. The training endpoint is calculated using all available cells and is shown to be strongly correlated with cell count. However, the external validation uses fixed-size subsampling. The reported generalization failure may reflect this difference rather than a limitation of the representation’s biological information. An experiment that reconstructs the source endpoint using the same fixed-size procedure and retrains the predictors would help assess whether the generalization failure remains when the endpoint construction is matches.
Finally, the text is rather dense in places, uses awkward terms, and sounds overly AI-written, making it hard to read as a human reader. For example, words like “locked”, “governed”, “resolved” aren’t natural in this context and require parsing which gets in the way of understanding. Also it’s hard to follow some of the experiments as described, like Line 186, “The in-distribution gate (H-FM1)” – what is the gate, and how does it work? Line 300 talks about a “capacity floor the licenses every comparison below” which is hard to follow. The paper mentions several times that the scientific conclusion is changed, but it doesn’t describe clearly what conclusion is being changed or what new scientific understanding is generated.
Questions for authors
-
What is the intended practical use of predicting an endpoint that can be calculated directly from the observed post-perturbation measurements? If the task is primarily a representation probe, what new understanding does it provide about Geneformer?
-
How does the generalisation compare when constructing the source endpoint using the same fixed-size subsampling procedure as the external endpoints and retraining the predictors? What evidence distinguishes biological generalisation failure from the difference in endpoint construction?
-
The paper repeatedly states that the evaluation changes the scientific conclusion. Please clarify which conclusion is being revised, what replaces it, and what this implies for future modelling or evaluation.