- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes TotiCell, a compact single-cell representation model that combines two design choices: (1) a block tokenizer that partitions all 21912 genes into 48 fixed, balanced, randomly-formed blocks; (2) GroupMask, a structured masking strategy that divides the 48 blocks into three disjoint, exhaustive target sets of 16 blocks each, masking one entire set per cell while preserving the same per-block masking probability and visible token budget as RandomMask. Trained on 50.5 million cells, GroupMask leads five of seven task means over RandomMask, most clearly in perturbation retrieval, where all three datasets favor it.
Critical strengths
- The 48-token fixed input yields concrete, measured efficiency gains on a single RTX 3090. The efficiency advantage is directly attributable to the block tokenizer design and is not confounded by architecture differences in the controlled comparison.
Critical weaknesses
- Random block partitioning plus three static fixed mask sets is simple, but the paper never establishes why random partitioning is preferable to biologically-informed grouping, nor why static three-set masking outperforms dynamic structured masking. Simplicity alone does not constitute novelty when the components and their combination have direct antecedents.
- TotiCell compresses 21912 genes into 48 tokens, which is the most aggressive ratio in the field. Therefore, how to deal with the potential information loss? There is zero evaluation on gene-level tasks where individual gene identity matters. It seems that the claim of "strong multi-task performance" holds only for the selected coarse tasks.
- GroupMask reduces mask support from C_48^16 to 3. At inference all 48 blocks are visible, but during training the model sees only 3 visibility configurations, leading to a severe train-inference distribution mismatch. The model pretraining may be overfitting to limited masking schemes. The paper never directly tests mask-pattern overfitting, and the validation is not enough to conclude the robustness of the proposed masking plan.
Questions for authors
Please refer to the Critical Weaknesses
6 comments
Response 2/3 (W2): Gene information, measured proteins and fine cell subtypes
We addressed the information-retention concern with individual-gene readouts, prediction of independently measured proteins, and finer cell-subtype tasks. Section 6.2 and Appendix F report the molecular experiments; Appendix D.2 reports the finer-label evaluations. G4/R4 denote the frozen 96M GroupMask/RandomMask models after four training passes.
What 48 tokens represent. At 96M, the encoder output is a 48 x 768 matrix, followed by a 768-dimensional mean-pooled cell vector. Each gene retains its fixed coordinate within a block, and reconstruction predicts individual gene positions. Thus 48 is the sequence length. The revised method distinguishes the block matrix, pooled cell vector, and coverage of the predefined 21,912-gene vocabulary.
Individual-gene expression is readable from the blocks. We evaluated two donor-held-out Human Immune Health Atlas (HIHA) splits and two technology-held-out Pancreas splits, with 768/192/384 training/validation/test cells per split. Training cells determine the gene panels (92/96 targets for HIHA, 96/96 for Pancreas). Each target has an independent validation-tuned ridge readout from its corresponding block, even when several targets share that block. The encoder is frozen and target RNA is present in the input, so this tests expression information retained after encoding.
| Dataset (G4) | Local Pearson r | Local NMSE | Pooled-cell NMSE |
|---|---|---|---|
| HIHA | 0.8371 | 0.2785 | 0.4156 |
| Pancreas | 0.8785 | 0.1669 | 0.6695 |
NMSE divides test MSE by target training variance; lower is better. The local-versus-pooled comparison uses the same 768-dimensional ridge protocol. It shows separately accessible gene signals in the block outputs and identifies information reduced by mean pooling. Table 25 includes RandomMask, CellVQ, scGPT and untrained-block results as well.
RNA predicts an independently measured molecular target. In PBMC CITE-seq, the inputs contain RNA only; targets are 24 prespecified antibody-measured proteins, expressed as log-transformed ADT relative abundances. For token encoders, each protein is predicted from the block/gene token corresponding to its RNA gene; RNA PCA64 uses 64 cell-level PCs fitted on training RNA. All methods use the same 14,554 common RNA genes. Each of eight donors is held out once, with another donor for validation and six for training; each contributes 256 test cells. Ridge penalties are selected on the validation donor, then readouts are refitted on training plus validation donors. Scores average proteins within each donor and then donors equally.
| Representation | Pearson r | NMSE | Within-cell-type r |
|---|---|---|---|
| GroupMask G4 | 0.7101 | 0.4465 | 0.2238 |
| RandomMask R4 | 0.7147 | 0.4395 | 0.2326 |
| CellVQ | 0.7119 | 0.4383 | 0.2177 |
| scGPT | 0.6886 | 0.4758 | 0.1858 |
| RNA PCA64 | 0.6724 | 0.5062 | 0.1453 |
| Untrained blocks | 0.5855 | 0.6319 | 0.0897 |
Within-cell-type correlation centers predictions and measurements by test-cell type before correlation. The results demonstrate molecular prediction across donors, including variation beyond broad cell-type mean differences. Tables 26, 27 and 28 provide the original P1 case, eight-donor results, and all protein/RNA target identities.
The cell-level tests also resolve finer distinctions. With G4, annotation macro-F1 is 0.6953/0.5221 for HYPOMAP C2/C3 (52/156 labels) and 0.6788/0.8350/0.7686/0.6449 for LBCL pDC/LEC/monocyte/CD8 T subtypes within the respective lineage. Few-shot AULC is 0.6724 on 43 ALS spinal-cord subtypes and 0.6784 on 59 obstructed-kidney subtypes (Tables 16 and 17).
Together, these additions test three distinct capabilities: gene-expression accessibility in local blocks, transfer to independently measured molecular targets, and separation of fine cell subtypes. The finer-label results also contribute to the revised main task summaries, with targets averaged within cohorts before equal cohort weighting.
Response 3/3 (W3): Training masks, unseen visibility patterns and full input
We added a direct visibility-pattern test on all original 22 settings, using the same frozen 96M GroupMask encoder at its three-pass checkpoint (G3). This fixed reference panel preserves the original tasks for the visibility control. It tests both transfer beyond the three training patterns and the effect of supplying all blocks at inference (Section 6.3; Figure 4; Appendix H).
The shared seven-family panel keeps the task profile fixed when changing the visible input, linking this inference test to the training-design controls.
The comparison. Let A/B/C be the disjoint 16-block targets used in pretraining:
| Condition | Visible input | Purpose |
|---|---|---|
| Full48 | All 48 blocks | Standard full-input inference |
| SeenA/B/C | B+C, A+C, A+B | The three training visibility configurations, each with 32 blocks |
| Unseen1/2/3 | Three fixed, uniformly drawn 32-block subsets | Unseen combinations at the same token budget |
Every random subset differs from all three training patterns and stays fixed across cells and tasks; 1/2/3 identify the fixed unseen subsets. Unseen32 denotes inputs to this GroupMask encoder, not a RandomMask model. The gene partition, preprocessing and block-position identities are unchanged. Visible tokens are mean-pooled into 768-dimensional vectors. Every block participates as a visible input in two training configurations and as a reconstruction target in the third; Full48 supplies those same blocks together.
Evaluation retains the original cells, labels, folds and scoring procedures. For tasks with supervised readouts, we fit and validate the readout separately for each input condition while keeping the encoder frozen. Thus this measures the downstream utility of the representation under changed visibility. Integration and retrieval retain their original procedures.
Results across all seven reference-panel task families. Full is Full48; S and U average the three Seen and Unseen patterns. Metrics are macro-F1 for annotation/mapping, the integration composite, few-shot AULC, disease/OOD AUROC, and perturbation MRR. Settings are averaged within each pattern before the pattern mean and sample SD are calculated; the SD therefore measures input-pattern variation. Differences use unrounded scores.
| Task | Full | S +/- SD | U +/- SD | S-U | Full-S | Full-U |
|---|---|---|---|---|---|---|
| Annotation | 0.8851 | 0.8807 +/- 0.0010 | 0.8778 +/- 0.0074 | +0.0029 | +0.0044 | +0.0073 |
| Integration | 0.5948 | 0.5932 +/- 0.0029 | 0.5887 +/- 0.0021 | +0.0044 | +0.0016 | +0.0061 |
| Few-shot | 0.8006 | 0.8046 +/- 0.0023 | 0.7959 +/- 0.0014 | +0.0087 | -0.0041 | +0.0046 |
| Disease | 0.8550 | 0.8434 +/- 0.0197 | 0.8431 +/- 0.0062 | +0.0003 | +0.0116 | +0.0119 |
| Mapping | 0.8160 | 0.7746 +/- 0.0132 | 0.8026 +/- 0.0134 | -0.0280 | +0.0415 | +0.0135 |
| Perturbation | 0.7909 | 0.7697 +/- 0.0035 | 0.7797 +/- 0.0074 | -0.0100 | +0.0212 | +0.0112 |
| OOD | 0.9348 | 0.9310 +/- 0.0012 | 0.9299 +/- 0.0021 | +0.0011 | +0.0038 | +0.0049 |
Seen and unseen combinations have absolute mean gaps below 0.009 in five of seven families. Annotation, disease and OOD are particularly close; Seen is higher on integration and few-shot, while unseen patterns are higher on mapping and perturbation retrieval. Full48 exceeds the Unseen32 mean in all seven families and the Seen32 mean in six, with few-shot the exception.
These results show that useful downstream information extends beyond the three recurring training contexts, and that full-input inference effectively combines the available blocks. They support robustness to the tested unseen visibility combinations. Appendix H reports all 22 setting-level Full48, SeenA/B/C and Unseen1/2/3 scores (Table 34), means/SDs and all three differences (Table 35), and exact block memberships (Table 36).
Response 1/3 (W1): Design rationale, partitions and static masking
Thank you for highlighting the need to justify both design choices. We added direct partition and dynamic-masking controls and clarified the contribution in Sections 2-3 and 6.1, with complete results in Appendix G.2.
Design rationale and contribution. Gene-to-block partitioning organizes the input; block-to-target partitioning defines the reconstruction tasks. Balanced random blocks provide a fixed organization without an ontology or expression graph. GroupMask repeatedly predicts A from B+C, B from A+C, and C from A+B, where A/B/C are complementary 16-block sets. Each block has masking probability 1/3 and 32 blocks remain visible, as in RandomMask. Our hypothesis is that recurring cross-group prediction helps the shared encoder learn reusable expression relationships under finite training exposure; cells and expression values vary within these recurring contexts.
We explicitly position this contribution relative to MAE, SAVE's gene-block attention in a conditional VAE/flow-matching framework, TxFM's gene-token MAE, and CorrMask's expression-graph-based joint masking. The contribution studied here is the complementary reconstruction-target policy and its controlled effect on frozen representations.
Random versus expression-informed blocks. We completed the 96M partition-by-masking 2x2 comparison at the same four-pass endpoint, fixing architecture, training corpus, optimizer and schedule. The prior uses a positive co-expression graph from 100,000 training cells, forms 48 balanced blocks before pretraining, and uses no downstream labels. G/R denote GroupMask/RandomMask. The table uses the fixed original 22-setting reference panel, so its seven family means are separate from the expanded source-balanced main tables:
Expression-informed blocks improve GroupMask in four families; random blocks retain higher few-shot, mapping and OOD means. The same directions hold at three passes. This supports random blocks as a practical default, with prior grouping offering complementary task-specific gains (Table 29).
Static versus dynamic target sets. A matched 30.65M experiment uses five million cells, two passes, 9,766 updates and three paired training runs. The dynamic policy refreshes the complementary partition every quarter pass, starting from the static partition. All policies retain the same masking marginal and token budget. Proxy averages linear-probe and kNN macro-F1 using frozen full-input representations:
Static GroupMask exceeds refresh by 0.016489 on average and in every paired run. At 96M, fixed-bank controls on the original 22 settings further show that the three-set bank leads each larger bank in six of seven family means at three passes; at four passes it leads Bank-6 in five and Bank-9 in six. Few-shot favors larger banks, and four-pass annotation favors Bank-6 (Figure 4; Tables 30 and 31). Thus both the dynamic control and main-scale downstream experiments support recurring complementary tasks under the tested training conditions.
The 96M downstream design controls share a fixed 22-setting panel spanning all seven task families, with common labels, splits and readouts across variants.