- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 3(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SPORE proposes post-training a single-cell foundation model once to produce a checkpoint that can be reused across studies and tasks without further backbone fine-tuning. It combines gene-semantic alignment using GenePT, ontology-aware cell-type supervision, and biological-context alignment using biomedical text embeddings. Its main contribution is integrating these complementary signals into a shared representation that supports both lightweight downstream predictors and direct cell annotation.
The authors evaluate five backbones across six downstream tasks, including cell-type annotation, cancer classification, drug sensitivity, gene-level prediction, and perturbation prediction. They also examine representation geometry, performance with limited downstream labels, and direct annotation on 11 study-disjoint datasets. Supervision ablations and data-scaling experiments help assess where the improvements come from.
The clearest gains are in cell-type transfer and cancer classification, with additional evidence of improved label efficiency and closer alignment with ontology structure. Gene-level and perturbation improvements are smaller or mixed, and the combined objective does not consistently outperform individual supervision branches. Overall, the results support the usefulness of global semantic post-training, particularly for cell-level applications.
Critical strengths
The paper addresses a practical limitation of single-cell foundation models: adapting a model separately for every downstream setting limits its reuse. SPORE combines biological supervision at the gene, cell, and context levels in a single post-training stage, then reuses the resulting checkpoint without target-specific backbone updates. This makes the contribution relevant beyond a single annotation task and offers a practical way to improve existing pretrained models.
The evaluation covers five backbones and several downstream tasks, with consistent improvements in cell-type hAcc and cancer-classification AUROC across the paired backbones. The study-disjoint annotation experiments and label-efficiency analyses provide complementary evidence that the benefits extend beyond fitting the post-training labels. This breadth strengthens confidence in the approach, while the mixed gene-level and perturbation results help clarify where its benefits are strongest.
Critical weaknesses
The evidence for the benefit of combining all three supervision branches is limited. In Table 3, ontology-only supervision nearly matches the full objective on cell-type transfer and cancer classification, while context-only supervision performs better on drug sensitivity, and several individual branches perform better on tissue specificity. But I believe that shouldn't be the reason to reject this paper simply and bluntly. On the contrary, this leads me to believe that these improvements and results are highly credible. It is worth noting that many current single-cell foundation models employ evaluation methods that favor them—sometimes even inflating their reported performance. This kind of honest reporting is precisely what the field needs.
Questions for authors
That said, there are still many critical issues that should be solved or explained further:
-
The post-training objectives may suppress biological variation not captured by the supplied annotations. In particular, cell-type supervision may encourage stronger separation between annotated populations, potentially obscuring the continuous transitions and developmental relationships within differentiating lineages. These relationships are not fully represented by the cell ontology hierarchy. Consequently, improved classification performance could come at the expense of preserving the connectivity of the underlying biological manifold. The current evaluations do not directly address this possibility. An additional analysis of trajectory inference or differentiation would help assess whether post-training preserves continuous developmental structure.
-
Using gene-semantic alignment as a post-training task should be approached with caution. If GenePT embeddings are weak on the relevant tasks, aligning pretrained gene representations to them may constrain rather than improve those representations. Their informativeness also depends on the underlying NCBI descriptions (used in GenePT), which can be sparse for poorly characterized genes. For example, the summary for the protein-coding gene C16orf78 states only “Located in nucleus.” Such descriptions provide limited gene-specific functional information. These concerns suggest that GenePT embeddings may be more suitable for initializing the gene dictionary than serving as fixed post-training targets, although a direct comparison is needed to establish this.
-
In the cell type annotation task across different experiments, it reports hAcc as the evaluation metric. However, as SPORE incorporates a hierarchy-related post-training task, ontology-aware cell identity, the improvement may be caused by the alignment between training task and evaluation metrics. So, it will be more persuasive if the authors can replace it as standard macro-F1 or accuracy.
-
The color coding in panels A and B of Figure 4 is very difficult to interpret on first inspection. The legend placement makes it unclear which legend applies to each panel and whether colors have the same meaning across panels. Placing the relevant legend beside each panel, or explicitly identifying the panels associated with each legend, would improve readability.
3 comments
# Part 2 — Gene-semantic target quality
Q2 — Gene-semantic target quality.
We appreciate the reviewer’s caution regarding the quality of GenePT as a fixed supervision target.
Target quality. To our knowledge, GenePT is among the most comprehensive text-derived gene-level knowledge bases available, providing broad coverage of NCBI descriptions. Our audit found that 88.65% of covered Geneformer genes have archived descriptions exceeding ten words, while 11.35% have short or unavailable descriptions, including C16ORF78. Looking ahead, the strong results on protein-informed backbones (Table 1) motivate exploring protein-derived knowledge as another promising source of semantic priors.
Functional evidence. We further compared Frozen Geneformer, raw GenePT, and Gene-only post-training under coverage-matched evaluation protocols (289 common genes for Gene–Tissue; coverage-matched edges for UGRN):
| Task | Frozen | Raw GenePT | Gene-only |
|---|---|---|---|
| Gene–Tissue accuracy | 0.8241 | 0.6000 | 0.8517 |
| UGRN AUPRC | 0.6791 | 0.7214 | 0.6858 |
Raw GenePT exhibits task-dependent utility, weaker on tissue but stronger on UGRN (including all STRING endpoints). Notably, Gene-only alignment improves tissue accuracy (+0.0276) and UGRN AUPRC (+0.0068) over Frozen. Thus, an external semantic prior need not be a superior standalone representation to provide useful auxiliary supervision.
Design rationale. SPORE aligns a learned projection of pretrained gene embeddings with GenePT, allowing the native representations to retain information beyond the external prior. Supervision also requires only partial gene coverage, permitting future quality-aware target selection.
Regarding the alternative of GenePT-based initialization, we agree that this is a plausible strategy, particularly when constructing a backbone from scratch, where the gene dictionary and backbone can be optimized jointly.
In the post-training of an existing backbone, however, GenePT-based initialization presents two related challenges. First, GenePT embeddings (1,536 dimensions) and the backbone's native gene embeddings (256 dimensions in Geneformer) differ in dimensionality, requiring an additional mapping step. More fundamentally, matching their dimensions does not ensure compatibility with the representation space expected by the pretrained backbone.
To illustrate this compatibility challenge, we tested a ridge-based GenePT embedding-replacement control on Geneformer. We mapped normalized GenePT embeddings to the corresponding original Geneformer gene embeddings for the 20,824 covered genes and replaced the original entries with the mapped vectors. The mapping achieved a mean cosine similarity of 0.6338 (median 0.6488) and a relative L2 reconstruction error of 0.7640, indicating substantial differences from the original gene representations.
We further compared the replacement variant with the original-vocabulary Cell/Context-only control (−Gene) under the same training budget (seed 37, 6,546 steps), using the native annotation head on the same 15,190 donor-held-out test cells:
| Metric | −Gene | GenePT replacement |
|---|---|---|
| Accuracy | 0.6350 | 0.5520 |
| Macro-F1 | 0.3159 | 0.2347 |
| hAcc | 0.8141 | 0.7481 |
| hMacro-F1 | 0.4615 | 0.3595 |
The replacement variant underperforms the original-vocabulary control across all four metrics, illustrating the practical challenges of directly incorporating external gene embeddings into an existing pretrained backbone. Developing more compatible ways to integrate complementary gene-semantic information while preserving pretrained representations remains an interesting direction for future work.
Part 3 — Evaluation metrics and figure readability
Q3 — Hierarchy-compatible versus conventional evaluation.
We thank the reviewer for raising this important concern. We use hAcc and hMacro-F1 to accommodate differences in annotation granularity, following scTab and HCE-scTab, where valid descendant predictions are not penalized for being more specific than reference annotations. To assess potential metric alignment, we additionally report standard exact-match accuracy and macro-F1 on the original frozen-feature transfer benchmark (revised Table A5):
| Backbone | hAcc (Frozen SPORE) | Accuracy (Frozen SPORE) | Macro-F1 (Frozen SPORE) |
|---|---|---|---|
| Geneformer | 0.649 0.822 | 0.546 0.647 | 0.167 0.323 |
| scCello | 0.747 0.833 | 0.683 0.729 | 0.269 0.321 |
| TEDDY-G | 0.665 0.824 | 0.572 0.674 | 0.262 0.404 |
| UCE | 0.839 0.874 | 0.781 0.845 | 0.462 0.564 |
| TF-Sapiens | 0.871 0.894 | 0.841 0.802 | 0.579 0.492 |
Both conventional metrics improve on four of five backbones. For TF-Sapiens, the proportion of valid but more-specific-than-reference predictions increases from 3.0% to 9.2%. This may reflect a backbone-specific interaction between autoregressive pretraining and ontology-aware supervision that favors finer-grained predictions.
We further evaluated frozen Geneformer representations on two independent datasets with more consistent, flat cell-type annotations:
| Dataset | Macro-F1 (Frozen SPORE) | Accuracy (Frozen SPORE) |
|---|---|---|
| BA10 (seven unseen PT labels) | 0.6032 0.7083 | 0.6289 0.7603 |
| Smart-Seq2 (cross-platform) | 0.4773 0.5141 | 0.7585 0.7821 |
Together, these results show that SPORE's representation gains are not solely attributable to hierarchy-compatible scoring. Moreover, our original direct-inference audit (Appendix E.3) distinguishes exact-label errors from descendant-compatible predictions and quantifies ontology displacement, revealing differences in prediction specificity and error severity that hAcc alone cannot capture.
The additional evaluations are included in revised Appendices E.1.1 and F.4.
Q4 — Figure 4 readability.
We thank the reviewer for identifying the ambiguity in the Figure 4 legend–panel associations. We have reorganized Figure 4 to clarify the legend–panel association: each row now has a shared legend placed directly below it, and each legend explicitly lists the panels to which it applies ("Biological compartment (A, C, E, F)" and "Ontology relation (B, D)"). Panel numbering and all corresponding references have been updated accordingly.
Official Response to Reviewer jPTP
Thank you for recognizing the practical value of a reusable checkpoint and the importance of transparently reporting task-dependent results. Our response is organized into three parts. Part 1 examines the task-dependent contributions of joint supervision and whether post-training preserves continuous developmental structure. Part 2 evaluates the quality and utility of GenePT embeddings as gene-semantic supervision targets. Part 3 addresses hierarchy-compatible versus exact-match evaluation and the presentation of Figure 4.
Part 1 — Joint supervision and developmental structure
W1 — Combining the supervision branches.
We appreciate the reviewer's thoughtful assessment of our mixed results and emphasis on transparent evaluation. We agree that ontology-only supervision accounts for much of the cell-type and cancer gains, while individual branches can outperform Full on specific tasks (Table 3). These findings motivate a closer examination of the relationships between supervision objectives and downstream tasks.
Task-dependent contributions. We added Geneformer leave-one-out experiments (Appendix E.4.1, Table A19; Figure A6A). Removing Cell reduces cell-type hAcc from 0.822 to 0.771 and Cancer AUROC from 0.743 to 0.721, while removing Context or Gene reduces tissue accuracy from 0.814 to 0.797 or 0.783, respectively. Interestingly, although Context-only achieves higher tissue accuracy than Full (0.881 vs. 0.814), removing Context from Full still reduces performance. This illustrates that an objective's standalone performance does not necessarily predict its contribution within joint training, revealing non-additive, task-dependent effects of combining supervision sources. Complementary gradient diagnostics (Figure A6B) further characterize their optimization interactions, showing weak positive Cell–Context alignment and nearly orthogonal Gene-related gradients on average.
Cross-study generalization. SPORE aims to consolidate heterogeneous biological supervision into one checkpoint reusable across studies and tasks. On 10,500 cells from 11 held-out studies, Full improves direct-annotation macro-F1 over ontology-only post-training from 0.3343 to 0.3658 (paired study-bootstrap Δ +0.0315, 95% CI [0.0008, 0.0492]; Appendix E.3, Table A18). Together, these results provide evidence of additional cross-study benefits while revealing meaningful task-dependent trade-offs in joint semantic post-training.
Q1 — Continuous developmental structure.
We thank the reviewer for raising the preservation of continuous developmental structure.
We investigated the preservation of continuous developmental structure through two trajectory-conservation evaluations comparing the original Frozen and SPORE-post-trained Geneformer checkpoints. SPORE uses an ontology-aware classification objective to make cell identity accessible from the learned representations, without explicitly constraining their pairwise geometry. This leaves room for the representations to retain variation beyond the supervised cell-type labels.
Trajectory conservation is measured by the scIB score (ρ+1)/2 (higher is better), where ρ is the Spearman correlation between reference and embedding-derived pseudotime:
Oetjen erythroid differentiation (Oetjen et al., 2018; Luecken et al., 2022). Using 2,629 cells and the scIB benchmark's fixed continuous DPT reference, we obtained:
Trajectory conservation improves in all three sample-batches.
Ranzoni Smart-Seq2 fetal hematopoiesis (Ranzoni et al., 2021). As an additional cross-platform test, we evaluated 2,070 cells against the original authors' saved continuous DPT reference:
Both checkpoints retain all cells in a single connected component with finite inferred pseudotime in each cohort. Across these two cohorts, SPORE maintains or improves agreement with the reference developmental trajectories, with no observed loss of global graph connectivity. Complete evaluation protocols, sample-level results, and trajectory visualizations are provided in the revised appendix.
The added trajectory protocols, results and visualizations are in Appendices F.3–F.4, Tables A25 and A27 and Figures A9–A10.