acceptodds
Under review as a conference paper at ICLR 2027

What Do Transcriptome Models Learn? From Expression Probing to Biological Prediction and RNA Question Answering

Abstract

Transcriptome foundation models are increasingly used for biological prediction and connected to language models to answer questions about transcriptomic data. However, differences in both pretraining methods and training data make it difficult to determine which approach works best for a given use case. To address this, we train six single-cell and four bulk models from scratch on one shared corpus per modality, following literature-inspired pretraining recipes. Our evaluation addresses three questions. *How does pretraining affect expression probing?* The recipe with the highest score after pretraining is not always the one that improves the most, and for some recipes linear probes already achieve high scores on representations from untrained models. *How do the recipes perform on biological tasks?* Pretraining improves classification over untrained models for every recipe, but its effect on perturbation prediction is inconsistent. Moreover, recipes that score higher on expression probes do not necessarily perform better on biological tasks. Early in pretraining, LogME transferability scores already correlate positively with recipe performance on all five biological tasks, without training any downstream predictor. *Do the differences observed in probing and biological prediction extend to RNA question answering?* We connect the frozen pretrained models to Qwen and evaluate the probability assigned to reference answers for released CellWhisperer questions. Among single-cell models, cell-type classification rankings agree most closely with reference-answer likelihood rankings on cell-identity questions, whereas expression-value probe rankings show no such correspondence. Cell-type LogME rankings also agree between RNA and Qwen representations with an untrained adapter, suggesting that, as in vision–language models, encoder strengths carry over into the language model.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
transcriptome foundation models, rna question answering, transcriptomic-aware question answering

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_aLkfOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines what transcriptome pretraining contributes to expression probing, biological prediction, and RNA question answering. Six single-cell and four bulk recipes are trained on a shared corpus within each modality. Pretrained models are compared with their untrained counterparts and simple expression baselines. Classification usually benefits from pretraining, whereas perturbation prediction does not improve consistently. Further analyses examine early LogME scores and reference-answer likelihood in RNA question answering. Multiple pretraining seeds, RNA replacement controls, and uncertainty analyses help distinguish information already present in a representation from improvements attributable to training.

Critical strengths
  1. The pretrained-versus-untrained comparison addresses a basic attribution problem in this field. Shared corpora and simple expression baselines make the comparison more informative than a ranking of pretrained models alone. The finding that good probing performance need not imply useful biological prediction has clear implications for model evaluation.

  2. The paper gives a useful account of where pretraining helps and where it does not. Classification gains, inconsistent perturbation gains, and competitive simple baselines are reported together. The RNA replacement control adds evidence about whether reference-answer likelihood actually depends on the RNA input.

Critical weaknesses
  1. The early LogME result is promising but still exploratory. Appendix D describes a retrospective analysis with only four or six recipes per comparison. Early selection does not improve over initialization on TCGA or PANACEA (Table 16), and simple expression baselines are excluded from the candidate set. The case for a broadly useful selection rule is therefore weaker than the main pretrained-versus-untrained result.

  2. The QA analysis has a narrower evidence base. It measures reference-answer likelihood, so it supports conclusions about that quantity rather than generated-answer correctness. It also uses one pretraining/adaptation seed. Appendix B.3 reports 94 uncertain category assignments and 21 potential reference/context issues (PDF p. 18). All 95 expression-interpretation questions are follow-up turns, where history can reveal cell identity. The history-free Tabula analysis is a useful control, but sensitivity of the main-source category results remains unclear. The selected subtype findings in Table 8 also fail multiple-testing correction; this does not invalidate all aggregate QA results.

Questions for authors
  1. How would the early LogME selection result change if simple expression baselines were included as candidates? Please distinguish the value of choosing among recipes from the value of computing LogME after pretraining. Table 16 already shows that early selection does not improve over initialization on TCGA or PANACEA.

  2. Do the main-source QA rankings and category correlations change after excluding uncertain assignments and flagged reference/context cases? Please distinguish evidence from history-free Tabula questions from expression-interpretation results based entirely on follow-up turns. This could be checked with the existing QA scores.

Reviewer_wvnZOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors train from scratch and contrast several transformed-based large transcriptomics models. The comparison is on several downstream tasks, including expression prediction and cell type classification. Importantly, downstream performance is compared using either pretrained or randomly initialized, frozen encoders.

Critical strengths

The paper has a well thought experimentation protocol. Models share the same pretraining corpus within each modality and have matched depth and width. The results shed some light on the actual advantages of pre-training.

Critical weaknesses

My main criticism, recognized by the authors as well, is that the LogME analysis described in Appendix D.2 is retrospective, and thus its generability is debatable.

Questions for authors

I would suggest the authors to add a further analysis where they apply their frozen models to one or more datasets not yet used in their analysis. It would be impotant to (a) fix selection and hold-out partitions in the new dataset, (b) fix the LogME procedure and model-selection rule (c) select an encoder on the basis of LogME and fit the downstream predictor using selection-partition, (c) evaluate the selected model on the hold-out partition.

Reviewer_yFtgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper compares the pre-training recipes of published single-cell and bulk transcriptomic foundation models under controlled conditions. Each recipe adapts the input encoding, pre-training objective and loss of a published model. The authors pre-train one shared backbone (8 layers, hidden size 512) from scratch with each of 6 single-cell and 4 bulk recipes, on one shared corpus per modality. To study what each recipe makes the model learn, they evaluate the pre-trained models on three types of tasks, and compare them with their untrained versions on the first two: (i) three gene-level linear probes (“expression probing”): gene presence, order of expressed genes and expression value; (ii) five biological tasks: cell-type and cancer-type classification, and genetic and chemical perturbation prediction; (iii) RNA question answering on four question types, with the frozen encoder connected to a general-purpose LLM (Qwen3-1.7B) through a Q-Former.

They report three main findings. (i) On expression probing, pre-training does not always improve over untrained models by a large margin, and the recipe with the highest score is not always the one with the largest gain. (ii) The recipes that lead on expression probing do not consistently lead on biological tasks. Pre-training consistently improves classification, but its effect on perturbation prediction is inconsistent. (iii) For single-cell models, recipe rankings on cell-type classification are close to rankings on cell-identity questions.

Finally, they report that LogME scores computed early in pre-training (25k steps) already correlate with the final recipe rankings on the five biological tasks. Cell-type LogME rankings also agree between the encoder embeddings and the LLM representations.

Critical strengths

1. An interesting question addressed with a controlled and broad design.  Whether pre-training helps transcriptomic models is an open question: prior work shows that simple baselines often match or beat these models on downstream tasks (Kedzierska et al., 2025; Ahlmann-Eltze et al., 2025). The authors build a comprehensive design to answer this question: Within each modality, all recipes share the pre-training corpus. All 10 recipes share the same backbone depth, width and seeds, and each pre-trained model is compared with its untrained copy on probing and biological tasks. This removes the data and scale differences that limit comparisons between released models. Also, the authors break the question down into three axes (gene-level probing, biological tasks and RNA QA), and extend it to bulk RNA-seq and to language-model interfaces, beyond the single-cell setting of prior controlled studies (Ovcharenko et al., 2025; De Waele et al., 2026).

2. Honesty. Throughout the paper, the authors transparently report negative or mixed results:

  • Simple baselines beat every bulk recipe (Table 5).
  • Pre-training improves perturbation in only 7/18 and 11/18 recipe-seed pairs (l.1008).
  • Recipe rankings do not translate between RQ1 and RQ2 (l.317-318).

Also, the authors clearly label the LogME analyses as retrospective (l.1072).

Critical weaknesses
  1. Presentation and novelty. Though this work is comprehensive, the conclusions are multiple, and the overall paper lacks, to me, a clear novel contribution for the field. The paper answers three RQs for each of the 10 recipes, but most of the results are weak correlations or hedged conclusions, making it hard to understand the overall message. The given recommendations (l.482-484) are already known as far as I know: (i) Kedzierska et al., 2025; Liu et al., 2026; De Waele et al., 2026; (ii) Liu et al., 2026; De Waele et al., 2026; Kedzierska et al., 2025; (iii) Bendidi et al., 2024; Kedzierska et al., 2025; Ahlmann-Eltze et al., 2025; and the title question, “What Do Transcriptome Models Learn?”, gets no direct answer.
  2. Experiment soundness.
    1. All probe-biology, LogME and QA correspondences are Spearman correlations computed over one mean per recipe: a three-seed mean for probing and biological tasks, and a single seed for RNA QA. Each correlation therefore has only n = 6 points for single-cell data and n = 4 for bulk data (App. D.1, l.1057). With so few points, a large rank correlation can easily arise by chance: with n = 4, random rankings give |ρ| ≥ 0.8 one time in three. Under an exact permutation test, no Spearman correlation with n = 4 can reach two-sided significance at p < 0.05, because the smallest possible p-value is 2/24 ≈ 0.083. With n = 6, |ρ| must be at least 0.886. The paper reports no p-values (Table 8 mentions a Benjamini-Hochberg correction but does not show them), and most of its correlations fall below this threshold.
    2. Each recipe score is a mean over three pre-training seeds, so its seed-to-seed uncertainty should carry over to any ranking or correlation computed from it. This is not the case: the correlations rank three-seed means as if they were exact (App. D.1, l.1057), although many recipes differ by less than their seed noise. On Norman, the whole spread (0.654 to 0.683) is smaller than the standard error of SC PRESENCE alone (about 0.032, Table 4b), so other seeds could reorder the recipes and change ρ. RNA QA uses one seed (l.369-370), so this noise is not estimated at all. A seed bootstrap would show whether the rankings are stable.
    3. The mean training perturbation is missing from Table 4b, and could give another point of comparison for single-cell perturbation tasks.
Questions for authors

Clarifications

  1. Could you clarify the main conclusions of this paper in the abstract? And explain the concrete impact and novelty of each of them, either in the introduction or in the related work section? This question (related to weakness n°1) is the one that would affect my assessment the most.
  2. For the main correlations (Tables 6, 15 and 17, Figure 2, App. D.4), can you report exact permutation p-values and seed-bootstrap intervals, together with the total number of correlations computed? Which ones remain significant after multiple-comparison correction?
  3. In Table 9, you report different learning rates per recipe. It seems no sweep or rationale is mentioned in the paper. Can you explain how you chose these values?
  4. How do simple baselines (PCA-50 or a randomly initialized MLP projecting into 512 dimensions) perform on RQ1 tasks? Adding these baselines would help quantify the gain of pre-training differently.

Suggestions for future work

  1. It is a known result that using representations from the last layer of encoders often leads to suboptimal results on downstream tasks (due to the impact of the pretraining objective on the last layer latent space geometry) (Valeriani et al., 2023). For transcriptomic models, the Geneformer documentation also recommends using the second-to-last layer, as “the last layer is most specifically weighted to optimize the given learning objective” (Theodoris et al., 2023; https://geneformer.readthedocs.io). Evaluating other layers (at least the second-to-last layer) on RQ1 and RQ2 tasks would be interesting to see if the rankings and performances hold.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.