acceptodds
Withdrawn from ICLR 2027

Compressing Single-Cell Foundation Models via Sparse Multi-Level Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Single-cell foundation models provide reusable representations for biological analysis, but their size and inference cost limit deployment. We introduce scPACK (Parameter-efficient Architecture Compression with Knowledge transfer), a frame-work for compressing these models while transferring their learned capabilities. The framework represents gene embeddings as sparse combinations of shared dictionary atoms and reduces backbone width and depth. Training combines ground-truth supervision and teacher-prediction matching for the same self-supervised tasks, alongside gene, cell, and internal feature matching; Nash multi-task learning adjusts the objective weights. Evaluated on scGPT, the 3.743M-parameter student retains 89.15% of the teacher’s fine-tuned Macro-F1 on the multi-study Broad collection and 99.49% on the Cardiac dataset, held blind during framework development, while using only 7.29% of the teacher’s deployed parameters. Under matched inference conditions, its forward throughput is 2.86× the teacher’s. Evaluation also covers few-shot annotation, clustering, donor mixing, and masked-expression reconstruction. Cell representations remain close to the teacher’s even though nearest-neighbor relationships among gene embeddings change substantially. At the same training budget, additional teacher supervision improves annotation over ground-truth-only training, and dynamic weighting improves matching to the teacher’s expression predictions during multi-objective training. With this framework, a compact model retains substantial teacher capability, supporting parameter-efficient single-cell deployment.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Single-cell foundation models, Model compression, Knowledge distillation, Computational biology, Sparse coding, Single-cell transcriptomics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WdrCOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces scPACK, a framework for compressing the scGPT foundation model using knowledge distillation. To this end, the authors propose sparse dictionary-based gene embeddings, backbone width and depth reduction, and Nash-MTL optimization. The evaluation shows that the compressed model performs competitively with scGPT on downstream tasks, including cell-type annotation, clustering, and donor mixing, while improving masked-expression reconstruction.

Critical strengths
  • The research question is clearly stated.
  • The manuscript is Easy-to-read (although the figures need more polish).
Critical weaknesses
  1. The main problem is not clearly defined in the introduction. Existing distillation methods may align teacher and student representations without preserving local neighborhood relationships. Is this the limitation the authors aim to address? Why is preserving gene neighborhood biologically important, and how does the proposed method address this issue? These points should be clarified while providing supporting evidence and a discussion of why previous works could not address it.

  2. No comparison with related approaches is reported.

  3. scKAN, a potentially strong and reproducible competitor in this field, is not included, although it also distills knowledge from scGPT and reports better performance than it on downstream annotation tasks.

  4. MicroSC, mentioned as a direct competitor, appears to have reproducibility issues due to the unclear definition of its distillation targets.

  5. The ablation studies are insufficient. The effects of PCA initialization, the advantage of Nash-MTL over alternative optimization methods, and the individual contributions of the five proposed objectives are not investigated.

  6. Table 2 shows that a simple PCA representation performs competitively with both teacher and student models on the cell-type annotation task. This raises questions about the added value of the learned representations compared with a simple baseline. I wonder why PCA is not also included in the evaluation of other tasks.

  7. Donor mixing is mentioned in the abstract but is not sufficiently discussed. Does it refer to batch effects? If so, how do the authors identify the batch effect? Does the proposed method have the capacity to address it to some extent? If so, how?

  8. Figure 6 seems relevant to this topic, but the representations are colored by cell type. Additional UMAP visualizations colored by gene expression and donor ID would be helpful.

  9. Despite the substantial parameter reduction, the students consume more GPU memory than the teacher at larger batch sizes. This is an important limitation for practical deployment and should be explicitly discussed in the paper in a "limitations" subsection.

  10. The figures need more polish. For example, in Figure 2, some axis labels are unclear, and the graphs overlap or are difficult to distinguish in some subfigures.

ref: "scKAN: interpretable single-cell analysis for cell-type-specific gene discovery and drug repurposing via Kolmogorov-Arnold networks." He, Haohuai, et al. Genome biology (2025)

Questions for authors
  1. In the loss design, representations generated at internal layers of the model are compared. How were these layers selected?

  2. The final GT-only stage substantially weakens prediction matching with the teacher, nearly eliminating the difference between fixed and "dynamic weighting". How does this affect the claimed benefit of Nash-MTL in the final deployed model?

  3. Most main results and ablation studies are reported without variability estimates or statistical significance tests. Were experiments repeated with multiple seeds to assess the robustness and significance of the reported improvements?

1 comment

Dear Reviewer,

Thank you for your detailed review and helpful suggestions on the experimental evaluation and presentation. Unfortunately, due to more time is needed to complete the remaining experiments and revise the paper, we have to decide to withdraw this submission. We appreciate the time and care you devoted to reviewing our work.

Best Regards

Reviewer_cTmMOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes scPACK, a framework for compressing single-cell foundation models, demonstrated using scGPT. The method replaces the dense gene embedding table with sparse combinations of shared dictionary atoms and reduces the Transformer backbone's depth and width. The compressed student is trained using both ground-truth expression reconstruction and multi-level knowledge distillation, including matching the teacher's expression predictions, gene and cell embeddings, intermediate features, and cell–cell relationships.

The authors evaluate two compressed models on cell-type annotation, few-shot learning, clustering, expression reconstruction, representation similarity, and inference efficiency. The smaller model reduces the parameter count from 51.33M to 3.74M (7.29% of the teacher) and achieves 2.86× higher inference throughput, while retaining 89.15% and 99.49% of the teacher's fine-tuned Macro-F1 on the Broad and Cardiac datasets, respectively. The paper also examines how compression affects learned representations, finding that cell-level representations remain relatively similar to the teacher's despite substantial changes in gene-embedding neighborhoods.

Critical strengths
  1. The paper established a comprehensive distillation/compression approach, covering from the gene embeddings, to loss matching and others. It showed a significant reduction in model size, retaining 89.15% and 99.49% of the teacher's Macro-F1 on Broad and Cardiac datasets, respectively. Also naturally makes the inference much faster (2.86 x throughput) These are meaningful practical improvements, especially for deploying single-cell foundation models under limited computational resources.

  2. The authors established comprehensive evaluations across varying tasks and metrics, including cell- gene- representation similarity, few-shot annotation, reconstruction, and inference efficiency. Particularly, the use of donor-source stratification between training and eval also set up more challenging and stringent analysis. These analysis shows that most of the performance can be retained despite the pruning and shrinking of model size.

Critical weaknesses
  1. The proposed approach includes several exciting distillation/compression techniques, including sparse dictionary representations, fewer layers, loss for feature matching. Currently, the experiments haven't thoroughly test the contributions of each one. It will be interesting to add such ablation studies showing if removing some of the techniques how much of the performances left. (apart from what the current section 8 already shows)
Questions for authors
  1. In the direction of the weakness point mentioned above, could the authors give a comparison with student models trained with a simpler distillation objective such as only one of the few loss and feature matching technique used.

  2. The solution is useful and interesting to the field, while the methodology novelty can be limited. Can the authors discuss their methodology contributions around this, and particularly point out the difference to the prior work of "MicroSC: A Tiny, Performant Single-Cell Foundation Model" (already mentioned in introduction)

  3. Is it possible to show the approach on other single-cell foundation models, such as geneformer and others. At least discussing the feasibility or potential limitations when doing so.

1 comment

Dear Reviewer,

Thank you for recognizing the practical value of our work and for your helpful suggestions. Following your suggestion, our Geneformer distillation experiments have yielded encouraging preliminary results. Unfortunately, due to more time is needed to complete the remaining experiments and revise the paper, we have to decide to withdraw this submission. We appreciate the time and care you devoted to reviewing our work, and for your kind encouragement.

Best Regards

Reviewer_z37ROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes scPACK, a framework for compressing single-cell foundation models through sparse gene embeddings, backbone reduction, and multi-level knowledge distillation with Nash-MTL. Experiments on scGPT demonstrate substantial parameter reduction while retaining much of the teacher's downstream performance.

Critical strengths
  1. The paper addresses a practically relevant problem of reducing the computational cost of single-cell foundation models.
  2. The experiments cover multiple downstream tasks, representation similarity, and inference efficiency, demonstrating promising compression results.
Critical weaknesses
  1. The proposed method mainly combines existing techniques, including sparse dictionary learning, multi-level knowledge distillation, and Nash-MTL. It remains unclear what new methodological insight this combination provides, particularly for single-cell foundation models.

  2. The paper lacks comparisons with parameter-matched low-rank embeddings, existing approaches such as MicroSC, and standard compression baselines. Furthermore, the contributions of sparse embeddings, backbone reduction, and individual distillation objectives are not independently evaluated, making it difficult to establish the necessity of the proposed design.

  3. Although Nash-MTL improves teacher-prediction matching during training, it consistently underperforms fixed-weight distillation on downstream annotation tasks (Table 10). Moreover, the final GT-only training stage substantially weakens the earlier matching improvements. These results do not justify introducing Nash-MTL as a central component of the framework.

  4. The compressed models preserve only 6.3–7.6% of the teacher's top-10 gene neighbors (Figure 3), despite explicitly incorporating gene-level representation distillation. The paper provides no functional evaluation of whether these substantially altered gene representations retain meaningful biological relationships, leaving the effectiveness of gene-level knowledge transfer unclear.

Questions for authors
  1. Beyond combining existing compression and distillation techniques, what is the core insight behind this approach? Why is this specific combination particularly well-suited for single-cell foundation models?
  2. How does scPACK perform compared to simpler compression baselines of similar parameter scale? Among the observed trade-offs between compression and performance, which components play the most critical role?
  3. Given that Table 10 indicates fixed-weight distillation yields better results on downstream annotation tasks, what practical benefits does Nash-MTL offer for the final deployed model—especially after it has been refined using ground-truth labels?
  4. Although the compressed gene embeddings show low overlap with the teacher model in terms of "nearest neighbors," do they still preserve meaningful biological relationships? Could the authors provide functional-level evidence to evaluate this?
  5. Can the authors provide evidence that a similar "compression vs. performance" trade-off exists in single-cell foundation models or backbone architectures other than scGPT?
1 comment

Dear Reviewer,

Thank you for your careful review and constructive suggestions. Following your suggestion to evaluate other foundation models, we conducted Geneformer distillation experiments and obtained encouraging preliminary results. Unfortunately, due to more time is needed to complete the remaining experiments and revise the paper, we have to decide to withdraw this submission. We appreciate the time and care you devoted to reviewing our work.

Best Regards

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.