Compressing Single-Cell Foundation Models via Sparse Multi-Level Distillation

Single-cell foundation models provide reusable representations for biological analysis, but their size and inference cost limit deployment. We introduce scPACK (Parameter-efficient Architecture Compression with Knowledge transfer), a frame-work for compressing these models while transferring their learned capabilities. The framework represents gene embeddings as sparse combinations of shared dictionary atoms and reduces backbone width and depth. Training combines ground-truth supervision and teacher-prediction matching for the same self-supervised tasks, alongside gene, cell, and internal feature matching; Nash multi-task learning adjusts the objective weights. Evaluated on scGPT, the 3.743M-parameter student retains 89.15% of the teacher’s fine-tuned Macro-F1 on the multi-study Broad collection and 99.49% on the Cardiac dataset, held blind during framework development, while using only 7.29% of the teacher’s deployed parameters. Under matched inference conditions, its forward throughput is 2.86× the teacher’s. Evaluation also covers few-shot annotation, clustering, donor mixing, and masked-expression reconstruction. Cell representations remain close to the teacher’s even though nearest-neighbor relationships among gene embeddings change substantially. At the same training budget, additional teacher supervision improves annotation over ground-truth-only training, and dynamic weighting improves matching to the teacher’s expression predictions during multi-objective training. With this framework, a compact model retains substantial teacher capability, supporting parameter-efficient single-cell deployment.