acceptodds
Under review as a conference paper at ICLR 2027

Prototype Guided Post-pretraining for Single-Cell Representation Learning

Abstract

Single-cell representation learning (SCRL) from gene expression data offers a way to uncover the complex regulatory logic underlying cellular function. Inspired by large language models in natural language modeling, several single-cell pretrained models have recently been proposed that treat genes as tokens and cells as sentences. However, these models are fundamentally limited by the long-tailed nature of cell-type distributions and struggle to generalize under covariate shifts in gene expression data. While fine-tuning is often used to mitigate these issues, we observe that performance remains bounded. To address this challenge, we introduce CellRefine, a post-pretraining method that operates between the pretraining and fine-tuning stages of a single-cell foundation model. CellRefine uses a multi-faceted objective that incorporates marker-gene sets as structural priors to guide post-pretraining and refine the latent embedding manifold of cells. Across multiple computational biology tasks, empirical results show that CellRefine consistently improves downstream performance, yielding gains up to 15%.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Single-cell representation learning, foundation models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NbFKOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CellRefine introduces a biologically guided adaptation stage between foundation-model pretraining and downstream fine-tuning. It aims to improve representations of rare cell types and robustness to distribution shifts. The method aligns cell embeddings with learned prototypes constructed from curated marker-gene sets, adds an ontology-based loss to distinguish closely related cell types, and uses Gaussian-mixture variational regularization to structure the latent space. These objectives supplement the backbone’s original pretraining loss.

The authors evaluate CellRefine with scBERT, scGPT, and Geneformer across cell-type prediction, spatial expression imputation, and perturbation-response prediction, reporting consistent improvements over the corresponding backbones. Validation includes cross-dataset classification, transfer from scRNA-seq to spatial data, and response prediction for held-out cell types. Additional experiments examine rare-cell performance, individual loss components, compute-matched training, random or shuffled marker controls, marker coverage, and hyperparameter sensitivity. The strongest evidence concerns improved cell-type discrimination, while gains in imputation and perturbation-response prediction are smaller.

Critical strengths

The paper makes a credible case that curated marker knowledge can improve pretrained cell representations beyond continued pretraining alone. Its compute-matched, random-prototype, random-marker, and shuffled-marker controls help distinguish the contribution of biological information from extra optimization or generic supervision. The marker-coverage experiments further support this interpretation. These controls strengthen the central methodological claim.

The improvements extend across three foundation-model backbones and several downstream settings, including cross-dataset cell annotation and transfer to spatial transcriptomics. Class-level results also show gains for rare and closely related populations, directly supporting the paper’s motivation. This breadth makes the approach more convincing than an improvement confined to one architecture or classification dataset.

Critical weaknesses

The method explicitly pushes apart cell types sharing a parent lineage and encourages a clustered latent structure, but does not establish that this preserves continuous biological variation. Closely related populations can represent adjacent developmental or activation states, so stronger separation is not inherently a better biological representation. The UMAPs and classification metrics largely reward the structure imposed by the training objectives. Without evaluating trajectories, transitional populations, or within-type state variation, the claim that CellRefine improves the underlying representation remains incomplete.

Evidence for broader functional transfer is limited. Spatial imputation is evaluated on two liver datasets, while perturbation-response prediction uses a single peripheral-blood dataset and compares primarily against the same three foundation-model backbones. The improvements are modest, and the absence of perturbation-specific methods or simple response-transfer baselines makes their practical significance difficult to assess. Consequently, the experiments support improved cell-type discrimination more convincingly than the broader claim that marker-guided refinement improves functional biological modeling.

Questions for authors

The paper is technically interesting. That said, several critical issues in the experimental design and implementation require further clarification or validation:

  1. Appendix B describes ordering marker genes by cell-type specificity through H(c)H(c), but how is specificity determined when a gene marks multiple populations? The authors should explain how ties, conflicting annotations, and missing markers are handled. Since this ordering defines the prototypes used for supervision, these choices affect both the biological interpretation and reproducibility of the method.

  2. Curated marker databases may contain signatures derived from the same studies used for evaluation, allowing benchmark-derived information to enter post-pretraining. The authors should document the overlap between these resources and the evaluation studies, clarify whether evaluation-derived marker lists were excluded, and assess how excluding them affects performance.

  3. CellRefine uses ground-truth cell-type labels to select marker prototypes, but it is unclear whether it outperforms an additional stage of ordinary supervised cell-type training under matched compute and data access. The existing contrastive and prototype controls are useful, but this comparison is still needed to establish what the biological structure adds beyond directly exploiting the same labels.

  4. Explicitly separating closely related cell types may obscure continuous developmental transitions and within-type state variation, which are not fully represented by the ontology hierarchy. Improved classification and clearer UMAP clusters therefore do not establish that the underlying biological structure is preserved. Trajectory or differentiation analyses before and after lineage regularization would help determine whether these gains come at the expense of developmental continuity.

Reviewer_mzAJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces and defines CellRefine, a novel post-pretraining framework for single-cell foundation models that defines four total loss terms in order to optimize for mitigating the long-tail problem and bounded-performance problem. The overall gains in the new methods last around 15%. This was validated on various tasks across 9 baselines and found that the Cell Refine post-pretraining is significantly better.

Critical strengths

Loss Ablations: the authors give comprehensive loss ablations removing each term of the final loss function effectively isolating how various terms affect each matrix. This greatly improves the argument being made about the ability of CellRefine.

Critical weaknesses

The paper does not compare a CellRefine method to a more computationally similar method for each downstream task. For example, many of the gains that CellRefine creates may just be due to more available information rather than true improvement of method.

  1. The Figure 4 evidence is partly circular, and the comparison is not like-for-like. Section 2.2.1 says the prototype loss is supervised: the ground-truth cell-type label selects each cell’s prototype. The silhouette and ARI in Figure 4 are computed against those same labels. Any objective that explicitly separates labeled classes should raise these scores, so the improvement (0.445→0.695 silhouette, 0.258→0.611 ARI) mostly shows that the supervised term was optimized. It does not show that the representation is better in a task-independent sense. The “before” panel is also an unsupervised MLM embedding, while the “after” panel has seen labels. A fairer comparison would use the label-supervised controls the authors already train in Appendix D.4: supervised contrastive (row 2) and learnable prototypes (row 3). • ARI requires cluster assignments, but the clustering method and its resolution are not stated (Leiden, k-means, or something else). • In Figure 4a, “NK” is annotated twice in addition to “CD56+ NK”, although Blood has only one NK class. • Neither panel annotates CD19+ B, CD14+ Monocyte, Dendritic or CD4+ T Helper2. Th2 is the rarest class (97 cells) and the one the long-tail argument most depends on.

  2. The “Shuffled marker alignment” control is undefined and only reported for one task. Table 10 row (5) is not described anywhere in Appendix D.4. It is also missing from Tables 11 and 12, where the row numbering shifts to accommodate this. This matters because the result is strong. Shuffled markers (avg 0.502) beat random markers (0.476) and 25% markers (0.490), and roughly match 50% markers (0.506). On Blood, the shuffled condition reaches 0.789, which is higher than 50% markers. The meaning of this result depends on the definition: • If marker sets are permuted across cell types, then the specific marker content matters less than the paper claims. • If only the order within each prototype is permuted, this is effectively an ablation of H(c) and should be presented that way. Either way, the authors should define the control and report it for all three tasks.

  3. Necessary baselines are missing. Several baselines are needed to tell whether the gains come from the proposed objectives or from the additional supervision and priors. • Marker-informed annotation methods. Since CellRefine’s distinguishing input is curated marker sets and ontology structure, it should be compared against methods that use the same information directly, such as CellAssign or Garnett. • Long-tail methods. Section 5 discusses class-balanced loss, LDAM, logit adjustment and adaptive resampling (Navidi et al.), but none of them is evaluated. Applying them during fine-tuning of the same backbones would test whether the long-tail gains are specific to CellRefine. • Text-anchored prototype methods. LangCell, GenePT and Cell2Sentence are presented as the closest related work, but none is compared. • Perturbation-specific models. Only the three foundation models are evaluated on perturbation. The Kang et al. dataset is the standard scGen benchmark, so scGen, CPA or GEARS should be included.

Minor issues / inconsistencies • LiverST cell counts are swapped. Appendix F gives LiverST1 = 90,408 and LiverST2 = 309,808 cells. In Table 19, the LiverST1 rows sum to 309,808 and the LiverST2 rows sum to 90,408. • Table 18 has several errors. Pancreas is listed with 17 cell types, but Table 19 lists 14 and Appendix F says 13 (reference) and 11 (query). The text cites “Custom3” (α ≈ 0.2), which does not appear in the table or anywhere else in the paper. Blood’s fit R² = 1.000 is not meaningful, since the fit uses the rarest 30% of 11 categories (about three points). The text says Liver has n = 5 categories, but Liver has 10 (Table 20); five is the LiverST datasets. • Appendix J has a heading and no text. Tables 19–20 on the following pages appear to be its content. However, the introduction (l. 51–52) promises further analyses of Figure 1 there, and none exist. • Figure 10 swaps the Liver and Pancreas panels. The “Liver” panel shows alpha, beta, ductal and acinar cells. The “Pancreas” panel shows hepatocytes and LSECs, which are LiverST cell types. • Figure 6 caption and Appendix D.7 cite the wrong subfigures. They refer to Figure 8a/8b instead of 6a/6b. Figure 8b also has no cluster annotations.

Questions for authors
  1. How are cell-type labels and marker-gene annotations used during post-pretraining relative to the train/test split? In particular, are any test-cell labels or test-derived annotations used to select prototypes to optimize CellRefine?

  2. How does CellRefine compare against a supervised baseline trained with the same labeled cells, biological annotations, and comparable compute? Given that the method explicitly incorporates cell-type labels and ontology structure, this comparison is important for establishing whether the proposed objectives provide benefits beyond additional supervision.

  3. Shuffled control. What exactly is the shuffled control (see Weakness 2), and why is it omitted from Tables 11 and 12?

  4. Clustering for ARI. Which clustering method and parameters produce the ARI values in Figure 4?

  5. Spatial training setup. Section 3.1 says the out-of-domain models are trained on scRNA-seq and evaluated zero-shot on spatial data. Appendix F says LiverST1 results come from models trained on LiverST2, and vice versa. Which is correct? Table 7 also reports LivST1 and LivST2 as on-domain.

Reviewer_whvVOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

CellRefine adds a supervised training step after pretraining single-cell foundation models. It combines marker-based cell-type prototypes, cell-type ontology information, and Gaussian-mixture regularization while keeping the original pretraining objective. The method improves cell annotation, spatial expression imputation, and perturbation-response prediction across three backbone models.

Critical strengths

Results on scBERT, scGPT, and Geneformer support the framework’s applicability beyond one architecture. For example, Geneformer’s average annotation macro-F1 increases from 0.522 to 0.593 in Table 1.

Critical weaknesses
  1. The method combines established prototype learning, margin-based separation, and mixture regularization. Its contribution should be positioned against scCello, which already incorporates cell-type coherence and ontology alignment alongside masked expression prediction, but is neither discussed nor evaluated.

  2. Limited comparisons with recent methods. The foundation-model comparisons primarily focus on scBERT, scGPT, and Geneformer, omitting recent approaches such as STATE, STACK, and Tabula. Comparisons on the tasks these methods support, under matched training and evaluation settings, would help establish whether CellRefine provides competitive performance beyond improving its selected backbones.

  3. Missing genetic perturbation benchmarks. Perturbation evaluation is restricted to the Kang dataset, without results on established genetic perturbation benchmarks such as Adamson and Norman. These additional evaluations would test generalization across perturbation targets and, for Norman, combinations of genetic interventions. The current experimental coverage is insufficient to support broad claims about genetic perturbation prediction.

Questions for authors

Can the authors narrow the perturbation claims to stimulus-response transfer or provide actual genetic perturbation evaluations, such as Adamson and Norman?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.