- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper examines what transcriptome pretraining contributes to expression probing, biological prediction, and RNA question answering. Six single-cell and four bulk recipes are trained on a shared corpus within each modality. Pretrained models are compared with their untrained counterparts and simple expression baselines. Classification usually benefits from pretraining, whereas perturbation prediction does not improve consistently. Further analyses examine early LogME scores and reference-answer likelihood in RNA question answering. Multiple pretraining seeds, RNA replacement controls, and uncertainty analyses help distinguish information already present in a representation from improvements attributable to training.
Critical strengths
-
The pretrained-versus-untrained comparison addresses a basic attribution problem in this field. Shared corpora and simple expression baselines make the comparison more informative than a ranking of pretrained models alone. The finding that good probing performance need not imply useful biological prediction has clear implications for model evaluation.
-
The paper gives a useful account of where pretraining helps and where it does not. Classification gains, inconsistent perturbation gains, and competitive simple baselines are reported together. The RNA replacement control adds evidence about whether reference-answer likelihood actually depends on the RNA input.
Critical weaknesses
-
The early LogME result is promising but still exploratory. Appendix D describes a retrospective analysis with only four or six recipes per comparison. Early selection does not improve over initialization on TCGA or PANACEA (Table 16), and simple expression baselines are excluded from the candidate set. The case for a broadly useful selection rule is therefore weaker than the main pretrained-versus-untrained result.
-
The QA analysis has a narrower evidence base. It measures reference-answer likelihood, so it supports conclusions about that quantity rather than generated-answer correctness. It also uses one pretraining/adaptation seed. Appendix B.3 reports 94 uncertain category assignments and 21 potential reference/context issues (PDF p. 18). All 95 expression-interpretation questions are follow-up turns, where history can reveal cell identity. The history-free Tabula analysis is a useful control, but sensitivity of the main-source category results remains unclear. The selected subtype findings in Table 8 also fail multiple-testing correction; this does not invalidate all aggregate QA results.
Questions for authors
-
How would the early LogME selection result change if simple expression baselines were included as candidates? Please distinguish the value of choosing among recipes from the value of computing LogME after pretraining. Table 16 already shows that early selection does not improve over initialization on TCGA or PANACEA.
-
Do the main-source QA rankings and category correlations change after excluding uncertain assignments and flagged reference/context cases? Please distinguish evidence from history-free Tabula questions from expression-interpretation results based entirely on follow-up turns. This could be checked with the existing QA scores.