acceptodds
Withdrawn from ICLR 2027

Batch-Robust Cell Representations by Self-Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Zero-shot embeddings from masked-reconstruction single-cell foundation modelssuch as scGPT and Geneformer do not reliably beat HVG+PCA at batch integration.Self-distillation methods such as DINO, which optimise the embeddingspace directly, lead imaging benchmarks but have not yet shown clear gains onscRNA-seq. We train scRAPTOR (single-cell RAPid Teacher-student Optimisationof Representations), a 1.8M-parameter attention-free encoder, by prototypeself-distillation on a public corpus of 86 million cells from 1,065 human studies.Sinkhorn equipartition keeps the prototype head from collapsing, and a low-rankstudy term on the prototype logits, discarded after training, absorbs study identity,so the encoder needs no batch label at inference. Zero-shot, scRAPTOR beatstransductive HVG+PCA on scIB Total on the Pancreas, PBMC and DKD benchmarks,and leads every other zero-shot embedding we scored (scGPT, Geneformer,scJEPA, Concerto and two scVI models trained on our corpus) on Pancreas, PBMCand the three-dataset mean. Without Sinkhorn equipartition the head collapses to adozen prototypes and loses held-out-study accuracy; without the study term theembedding carries more study identity and scores lower on every benchmark. Werelease the weights of all three seeds, the cell-level split and the atlas build scripts.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
single-cell foundation models, self-supervised learning, self-distillation, batch integration, zero-shot representation learning, scRNA-seq, benchmarking, evaluation methodology, invariance

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_uUA4OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper uses the DINOv2 framework for self-supervision to learn representations for scRNA-seq data, which is tabular data. The proposed model improves performance with respect to classic baselines, which was not reported by other papers before. The experimental results were obtained with a corpus of data curated by the authors, and then three subsets are used for evaluation.

Critical strengths
  • The paper shows that self-supervised learning using self-distillation algorithms improves performance with respect to classical baselines.
Critical weaknesses
  • The paper was written with heavy use of AI, resulting a confusing manuscript that requires too much effort to follow and understand. It has no global coherence and reports either trivial facts summarized from sources or trivial data interpretations without sufficient explanation. The paper lacks the proper voice of scientists and has no well developed motivations, explanations, and interpretations of the results. It is a well conducted and well documented experiment, but it is not a fully developed scientific manuscript for this conference.
  • On the technical side, the paper adapts the DINOv2 framework for tabular, scRNA-seq data, but it lacks technical innovation. All the algorithmic elements introduced in the paper are mainly taken from the existing DINOv2 implementation, including the student-teacher loss, and the KoLeo regularizer. The Sinkhorn equipartition is known in the SSL literature as well, and was just added to the existing framework.
  • There is no reasoning or explanation for why the architecture was chosen. Presumably, the authors were motivated by the previous use of linear models for sequencing data. But other competing models that they cite use transformer networks, and there is no rational for using their models, nor experiments that evaluates their choice.
  • The contributions and novelty are limited. Beyond improving performance with respect to classical baselines, there is no major technical or conceptual contribution in this paper. The experiments are limited to comparing to baseline methods and the ablations or other evaluations to understand the methods or data are limited.
  • Missing critical baselines. No comparisons with Cell-JEPA and GeneJEPA are reported despite the fact that these are direct competitors with the proposed method. Also, they presented an evaluation with their own curated data. Why not adopt existing datasets and data partitions to report results comparable to previous literature?
Questions for authors
  • Can the authors make the effort of writing the paper in their own voice? Communication is one of the main building blocks of the scientific method, and if the authors are unable to explain their research, the manuscript cannot be published in this conference.
  • Why were Cell-JEPA and GeneJEPA left out of the evaluation?
  • Why was that architecture chosen?
Reviewer_s9HDOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes scRAPTOR, a small attention free encoder trained with gene dropout views, 1,024 prototypes, Sinkhorn equipartition, and a low rank study term that is discarded at inference. The goal is to make single cell embeddings less sensitive to batch while keeping cell biology useful without a batch label at test time. The method is carefully assembled and the PBMC result is persuasive. The main question is how much of the gain comes from a new idea rather than from combining familiar self distillation and batch correction ingredients.

Critical strengths
  1. The paper has a coherent method story. Gene dropout defines the invariance, the prototype loss shapes the representation, and the study term is removed before inference. The encoder is small and does not need a batch label at test time.

  2. The baseline set is stronger than a typical foundation model comparison. HVG, HVG plus PCA, corpus PCA, random projection, an untrained encoder, scVI, and self distillation baselines help separate the training objective from the encoder architecture.

Critical weaknesses
  1. The paper is methodologically incremental. Gene dropout views, prototype self distillation, Sinkhorn balancing, and a batch or study conditioning term are all familiar ingredients. The contribution is the particular combination and its scale, not a new learning principle. The paper should state this contribution plainly and show why this combination is preferable to simpler existing recipes.

  2. The central comparison is uneven across datasets. The reported Total gaps over HVG plus PCA are +0.043 on Pancreas, +0.223 on PBMC, and +0.020 on DKD, but Appendix Table 7 shows gaps of -0.003 and -0.003 on Pancreas and DKD after removing PCR. The broad claim of beating HVG plus PCA on all three datasets therefore depends heavily on one metric. PBMC is the convincing result; the other two should be presented as mixed or comparable outcomes.

  3. The component analysis does not fully establish what is doing the work. The paper does not compare the full recipe with equally simple alternatives that combine the same ingredients differently, and it does not test the dropout range, local views, or count downsampling. Table 3 also shows Sinkhorn only ahead of the full recipe on Pancreas and PBMC, which weakens the claim that every added component is useful.

Questions for authors
  1. What is the strict held out DKD result after removing the whole source or study, with PCR reported separately from the other metrics?

  2. Relative to existing prototype or self distillation methods, which part of scRAPTOR is intended to be the main new contribution, and what matched comparison isolates it?

  3. Why is the full recipe retained when Sinkhorn only is better on Pancreas and PBMC in Table 3? Please also separate the effect of study identity from the effect of gene dropout augmentation.

Reviewer_btMDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at zero-shot batch integration for single-cell RNA-seq, with the goal of reducing batch effects while keeping cell-type structure. It introduces scRAPTOR, a small attention-free model trained by self-distillation. After training on 86 million cells, it beats HVG+PCA on scIB Total across three benchmarks, with the clearest gains on PBMC.

Critical strengths
  • The motivation is clear, and Figure 1 makes it easy to see how the model is trained and what is kept at inference.
  • The authors are upfront about where the method helps and where it does not.
  • The ablations tell a clear story. Figure 3 and Table 3 make a convincing case that keeping prototypes in use matters more than simply keeping them apart.
  • The implementation details are well documented.
Critical weaknesses
  • I have to say that the novelty seems fairly limited. The method combines established ideas from DINO with a study-specific logit term. This is just an adaptation to single-cell data. More explanation is welcome.
  • PBMC shows a clear improvement, but the gains over PCA on Pancreas and DKD disappear when PCR is removed. On DKD, even the untrained encoder performs slightly better, which makes the benefit of pretraining less clear.
  • The biological significance is not clearly demonstrated. The evaluation focuses on integration metrics and cell-type separation, without showing whether the embeddings reveal better biological conservation. Better batch mixing does not necessarily mean better biological preservation.
Questions for authors

Please refer to the Weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.