acceptodds
Desk rejected at ICLR 2027

TotiCell: Structured Masking over Full-Coverage Gene Blocks for Single-Cell Representation Learning

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

A masking ratio specifies how much expression is hidden during pretraining, but not which genes are hidden together. We introduce TotiCell, a compact single-cell representation model whose block tokenizer represents the full set of 21,912 genes—including genes with zero expression in a given cell—as a fixed sequence of 48 tokens, roughly 1/67 of the input length of the audited gene-level baselines. TotiCell is trained with GroupMask, a structured masking policy that groups the 48 blocks into three disjoint, exhaustive target sets and masks one entire set per cell. GroupMask preserves RandomMask's per-block masking probability and visible-token budget while replacing the combinatorial mask distribution with three recurring target sets. Trained on 50.5 million cells, GroupMask leads five of seven task means over RandomMask, most clearly in perturbation retrieval, where all three datasets favor it (mean MRR gain 0.0293). Scaling the recipe yields the highest annotation, perturbation-retrieval, and mapping means among the evaluated methods with complete task coverage, and exceeds CellVQ on five of seven task means with 55.5% fewer total parameters. The scaled model also achieves the best mean rank percentile among the nine complete-coverage methods. In a controlled single-RTX-3090 benchmark, the smaller architecture processes over 1,600 cells per second while using under 0.5 GiB of memory. Three-seed probes favor recurring mask sets and reduced zero-target loss weight, and characterize partition and block-count sensitivity.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
single-cell foundation model, masked autoencoder, self-supervised learning, representation learning, biological foundation model, efficient pretraining

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sLZzOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TotiCell, a compact single-cell representation model that combines two design choices: (1) a block tokenizer that partitions all 21912 genes into 48 fixed, balanced, randomly-formed blocks; (2) GroupMask, a structured masking strategy that divides the 48 blocks into three disjoint, exhaustive target sets of 16 blocks each, masking one entire set per cell while preserving the same per-block masking probability and visible token budget as RandomMask. Trained on 50.5 million cells, GroupMask leads five of seven task means over RandomMask, most clearly in perturbation retrieval, where all three datasets favor it.

Critical strengths
  1. The 48-token fixed input yields concrete, measured efficiency gains on a single RTX 3090. The efficiency advantage is directly attributable to the block tokenizer design and is not confounded by architecture differences in the controlled comparison.
Critical weaknesses
  1. Random block partitioning plus three static fixed mask sets is simple, but the paper never establishes why random partitioning is preferable to biologically-informed grouping, nor why static three-set masking outperforms dynamic structured masking. Simplicity alone does not constitute novelty when the components and their combination have direct antecedents.
  2. TotiCell compresses 21912 genes into 48 tokens, which is the most aggressive ratio in the field. Therefore, how to deal with the potential information loss? There is zero evaluation on gene-level tasks where individual gene identity matters. It seems that the claim of "strong multi-task performance" holds only for the selected coarse tasks.
  3. GroupMask reduces mask support from C_48^16 to 3. At inference all 48 blocks are visible, but during training the model sees only 3 visibility configurations, leading to a severe train-inference distribution mismatch. The model pretraining may be overfitting to limited masking schemes. The paper never directly tests mask-pattern overfitting, and the validation is not enough to conclude the robustness of the proposed masking plan.
Questions for authors

Please refer to the Critical Weaknesses

6 comments
AuthorsOpenReview ↗

Response 1/3 (W1): Design rationale, partitions and static masking

Thank you for highlighting the need to justify both design choices. We added direct partition and dynamic-masking controls and clarified the contribution in Sections 2-3 and 6.1, with complete results in Appendix G.2.

Design rationale and contribution. Gene-to-block partitioning organizes the input; block-to-target partitioning defines the reconstruction tasks. Balanced random blocks provide a fixed organization without an ontology or expression graph. GroupMask repeatedly predicts A from B+C, B from A+C, and C from A+B, where A/B/C are complementary 16-block sets. Each block has masking probability 1/3 and 32 blocks remain visible, as in RandomMask. Our hypothesis is that recurring cross-group prediction helps the shared encoder learn reusable expression relationships under finite training exposure; cells and expression values vary within these recurring contexts.

We explicitly position this contribution relative to MAE, SAVE's gene-block attention in a conditional VAE/flow-matching framework, TxFM's gene-token MAE, and CorrMask's expression-graph-based joint masking. The contribution studied here is the complementary reconstruction-target policy and its controlled effect on frozen representations.

Random versus expression-informed blocks. We completed the 96M partition-by-masking 2x2 comparison at the same four-pass endpoint, fixing architecture, training corpus, optimizer and schedule. The prior uses a positive co-expression graph from 100,000 training cells, forms 48 balanced blocks before pretraining, and uses no downstream labels. G/R denote GroupMask/RandomMask. The table uses the fixed original 22-setting reference panel, so its seven family means are separate from the expanded source-balanced main tables:

Task / metricRandom GPrior GRandom RPrior R
Annotation / macro-F10.88380.90110.88470.8905
Integration / scIB0.59370.59530.58910.5911
Few-shot / AULC0.79990.79590.80660.8064
Disease / AUROC0.85760.88550.85920.8422
Perturbation / MRR0.78490.79580.76160.7875
Mapping / macro-F10.80230.77720.81370.7956
OOD / AUROC0.92990.92370.93480.9295

Expression-informed blocks improve GroupMask in four families; random blocks retain higher few-shot, mapping and OOD means. The same directions hold at three passes. This supports random blocks as a practical default, with prior grouping offering complementary task-specific gains (Table 29).

Static versus dynamic target sets. A matched 30.65M experiment uses five million cells, two passes, 9,766 updates and three paired training runs. The dynamic policy refreshes the complementary partition every quarter pass, starting from the static partition. All policies retain the same masking marginal and token budget. Proxy averages linear-probe and kNN macro-F1 using frozen full-input representations:

PolicyProxy mean +/- sample SD
Static GroupMask, 3 sets0.471152 +/- 0.003042
Fixed Bank-60.456603 +/- 0.002991
Fixed Bank-90.450127 +/- 0.001272
Quarter-pass refresh0.454663 +/- 0.001441
RandomMask0.435037 +/- 0.005619

Static GroupMask exceeds refresh by 0.016489 on average and in every paired run. At 96M, fixed-bank controls on the original 22 settings further show that the three-set bank leads each larger bank in six of seven family means at three passes; at four passes it leads Bank-6 in five and Bank-9 in six. Few-shot favors larger banks, and four-pass annotation favors Bank-6 (Figure 4; Tables 30 and 31). Thus both the dynamic control and main-scale downstream experiments support recurring complementary tasks under the tested training conditions.

The 96M downstream design controls share a fixed 22-setting panel spanning all seven task families, with common labels, splits and readouts across variants.

AuthorsOpenReview ↗

Response 2/3 (W2): Gene information, measured proteins and fine cell subtypes

We addressed the information-retention concern with individual-gene readouts, prediction of independently measured proteins, and finer cell-subtype tasks. Section 6.2 and Appendix F report the molecular experiments; Appendix D.2 reports the finer-label evaluations. G4/R4 denote the frozen 96M GroupMask/RandomMask models after four training passes.

What 48 tokens represent. At 96M, the encoder output is a 48 x 768 matrix, followed by a 768-dimensional mean-pooled cell vector. Each gene retains its fixed coordinate within a block, and reconstruction predicts individual gene positions. Thus 48 is the sequence length. The revised method distinguishes the block matrix, pooled cell vector, and coverage of the predefined 21,912-gene vocabulary.

Individual-gene expression is readable from the blocks. We evaluated two donor-held-out Human Immune Health Atlas (HIHA) splits and two technology-held-out Pancreas splits, with 768/192/384 training/validation/test cells per split. Training cells determine the gene panels (92/96 targets for HIHA, 96/96 for Pancreas). Each target has an independent validation-tuned ridge readout from its corresponding block, even when several targets share that block. The encoder is frozen and target RNA is present in the input, so this tests expression information retained after encoding.

Dataset (G4)Local Pearson rLocal NMSEPooled-cell NMSE
HIHA0.83710.27850.4156
Pancreas0.87850.16690.6695

NMSE divides test MSE by target training variance; lower is better. The local-versus-pooled comparison uses the same 768-dimensional ridge protocol. It shows separately accessible gene signals in the block outputs and identifies information reduced by mean pooling. Table 25 includes RandomMask, CellVQ, scGPT and untrained-block results as well.

RNA predicts an independently measured molecular target. In PBMC CITE-seq, the inputs contain RNA only; targets are 24 prespecified antibody-measured proteins, expressed as log-transformed ADT relative abundances. For token encoders, each protein is predicted from the block/gene token corresponding to its RNA gene; RNA PCA64 uses 64 cell-level PCs fitted on training RNA. All methods use the same 14,554 common RNA genes. Each of eight donors is held out once, with another donor for validation and six for training; each contributes 256 test cells. Ridge penalties are selected on the validation donor, then readouts are refitted on training plus validation donors. Scores average proteins within each donor and then donors equally.

RepresentationPearson rNMSEWithin-cell-type r
GroupMask G40.71010.44650.2238
RandomMask R40.71470.43950.2326
CellVQ0.71190.43830.2177
scGPT0.68860.47580.1858
RNA PCA640.67240.50620.1453
Untrained blocks0.58550.63190.0897

Within-cell-type correlation centers predictions and measurements by test-cell type before correlation. The results demonstrate molecular prediction across donors, including variation beyond broad cell-type mean differences. Tables 26, 27 and 28 provide the original P1 case, eight-donor results, and all protein/RNA target identities.

The cell-level tests also resolve finer distinctions. With G4, annotation macro-F1 is 0.6953/0.5221 for HYPOMAP C2/C3 (52/156 labels) and 0.6788/0.8350/0.7686/0.6449 for LBCL pDC/LEC/monocyte/CD8 T subtypes within the respective lineage. Few-shot AULC is 0.6724 on 43 ALS spinal-cord subtypes and 0.6784 on 59 obstructed-kidney subtypes (Tables 16 and 17).

Together, these additions test three distinct capabilities: gene-expression accessibility in local blocks, transfer to independently measured molecular targets, and separation of fine cell subtypes. The finer-label results also contribute to the revised main task summaries, with targets averaged within cohorts before equal cohort weighting.

AuthorsOpenReview ↗

Response 3/3 (W3): Training masks, unseen visibility patterns and full input

We added a direct visibility-pattern test on all original 22 settings, using the same frozen 96M GroupMask encoder at its three-pass checkpoint (G3). This fixed reference panel preserves the original tasks for the visibility control. It tests both transfer beyond the three training patterns and the effect of supplying all blocks at inference (Section 6.3; Figure 4; Appendix H).

The shared seven-family panel keeps the task profile fixed when changing the visible input, linking this inference test to the training-design controls.

The comparison. Let A/B/C be the disjoint 16-block targets used in pretraining:

ConditionVisible inputPurpose
Full48All 48 blocksStandard full-input inference
SeenA/B/CB+C, A+C, A+BThe three training visibility configurations, each with 32 blocks
Unseen1/2/3Three fixed, uniformly drawn 32-block subsetsUnseen combinations at the same token budget

Every random subset differs from all three training patterns and stays fixed across cells and tasks; 1/2/3 identify the fixed unseen subsets. Unseen32 denotes inputs to this GroupMask encoder, not a RandomMask model. The gene partition, preprocessing and block-position identities are unchanged. Visible tokens are mean-pooled into 768-dimensional vectors. Every block participates as a visible input in two training configurations and as a reconstruction target in the third; Full48 supplies those same blocks together.

Evaluation retains the original cells, labels, folds and scoring procedures. For tasks with supervised readouts, we fit and validate the readout separately for each input condition while keeping the encoder frozen. Thus this measures the downstream utility of the representation under changed visibility. Integration and retrieval retain their original procedures.

Results across all seven reference-panel task families. Full is Full48; S and U average the three Seen and Unseen patterns. Metrics are macro-F1 for annotation/mapping, the integration composite, few-shot AULC, disease/OOD AUROC, and perturbation MRR. Settings are averaged within each pattern before the pattern mean and sample SD are calculated; the SD therefore measures input-pattern variation. Differences use unrounded scores.

TaskFullS +/- SDU +/- SDS-UFull-SFull-U
Annotation0.88510.8807 +/- 0.00100.8778 +/- 0.0074+0.0029+0.0044+0.0073
Integration0.59480.5932 +/- 0.00290.5887 +/- 0.0021+0.0044+0.0016+0.0061
Few-shot0.80060.8046 +/- 0.00230.7959 +/- 0.0014+0.0087-0.0041+0.0046
Disease0.85500.8434 +/- 0.01970.8431 +/- 0.0062+0.0003+0.0116+0.0119
Mapping0.81600.7746 +/- 0.01320.8026 +/- 0.0134-0.0280+0.0415+0.0135
Perturbation0.79090.7697 +/- 0.00350.7797 +/- 0.0074-0.0100+0.0212+0.0112
OOD0.93480.9310 +/- 0.00120.9299 +/- 0.0021+0.0011+0.0038+0.0049

Seen and unseen combinations have absolute mean gaps below 0.009 in five of seven families. Annotation, disease and OOD are particularly close; Seen is higher on integration and few-shot, while unseen patterns are higher on mapping and perturbation retrieval. Full48 exceeds the Unseen32 mean in all seven families and the Seen32 mean in six, with few-shot the exception.

These results show that useful downstream information extends beyond the three recurring training contexts, and that full-input inference effectively combines the available blocks. They support robustness to the tested unseen visibility combinations. Appendix H reports all 22 setting-level Full48, SeenA/B/C and Unseen1/2/3 scores (Table 34), means/SDs and all three differences (Table 35), and exact block memberships (Table 36).

Reviewer_yUKeOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents TotiCell, a single-cell representation learning framework that compresses 21,912 genes into 48 block tokens and introduces GroupMask, a structured masking strategy that selects one of three fixed gene-block groups for reconstruction. The method achieves competitive results across multiple downstream tasks and demonstrates notable computational efficiency.

Critical strengths
  1. The fixed-length block representation offers a simple approach to handling high-dimensional expression profiles, and the reported inference results suggest practical computational benefits.
  2. The study covers diverse downstream tasks and includes both pretrained models and a PCA-HVG baseline. Reporting task-level differences, including unfavorable results, is appreciated.
  3. The authors match the per-block masking probability and visible-token budget between GroupMask and RandomMask, and provide additional smaller-scale studies of mask support, block count, and zero-expression weighting.
Critical weaknesses
  1. Methodological novelty and motivation. The framework builds on the familiar masked-autoencoder formulation, while gene-block or patch-based representations have been explored in prior single-cell studies. The paper would benefit from a more explicit comparison with these approaches. In particular, GroupMask reduces the range of possible mask patterns to three recurring groups. The empirical finding is interesting, but the motivation for why reduced mask diversity should improve generalizable representations—as opposed to primarily changing reconstruction difficulty or optimization dynamics—could be developed further.
  2. Limited performance improvements. GroupMask achieves higher numerical task means than RandomMask on five of seven families, but the OOD difference is effectively negligible, and some gains are small. RandomMask performs better on few-shot and disease classification. For the scaled TotiCell model, performance relative to CellVQ also varies by task. Moreover, PCA-HVG outperforms TotiCell on two of the three perturbation-retrieval datasets. A more nuanced discussion of these mixed outcomes, along with uncertainty estimates where feasible, would help establish the scope of the claimed benefits.
  3. Claims require greater clarification. e.g. The term "Full Coverage" refers to a predefined set of 21,912 genes rather than necessarily covering the entire transcriptome. This distinction should be stated more clearly.
  4. Insufficient clarity and specificity in writing. "External-model donor and cell overlap remain unknown", "recorded dataset-specific transform", "lack trustworthy raw counts". These statements would benefit from further clarification. The lack of specific details may weaken the transparency and credibility of the manuscript.
Questions for authors
  1. How do the authors interpret the strong PCA-HVG results on Norman and Adamson and the weaker TotiCell integration result on Pancreas? What do these cases suggest about the strengths and limits of the learned representations?
  2. Were donor- or cell-level overlap checks attempted for any external baselines?
  3. What is the key advantage of GroupMask over conventional random masking, and why would reducing masking diversity improve representation learning?
  4. Would GroupMask still outperform RandomMask under matched training endpoints and at the larger 227M model scale?
4 comments
AuthorsOpenReview ↗

Response 1/3: Q1 and Q4 - task-specific behavior, matched endpoints, and scale

Thank you for identifying the task-specific contrasts. We address Q1 and Q4 together, distinguishing representation utility, masking effects and model scale.

Q1: PCA-HVG and Pancreas. The original perturbation protocol compares mean responses across artificial replicates within one source. PCA fits an unlabeled target-cohort subspace, which can preserve directions useful for retrieval. Restricting PCA to genes mapped into TotiCell, with the other settings unchanged, retains its advantage on Norman (MRR 0.8961 versus TotiCell's 0.8628) and Adamson (0.9345 versus 0.9222); TotiCell is stronger on Dixit (0.6454 versus 0.5711). Extra gene coverage alone therefore does not explain the contrast.

Pancreas involves both biological conservation (TotiCell/CellVQ: 0.3726/0.5605) and batch mixing (0.3295/0.3893). We checked cell/label/technology alignment and score arithmetic. The 96M embeddings' leading directions associate with technology and totals of the processed expression matrix; the vocabulary also omits INS and PRSS1. Useful supervised information need not yield equally effective unsupervised integration geometry. Section 5.1 and Appendix E now explain these cases.

Q4: matched endpoints and scaling. G3/G4 and R3/R4 are 96M GroupMask/RandomMask at three/four passes, sharing tokenizer, architecture, corpus and optimization schedule. Each selects its highest proxy among eight common half-pass milestones: G3 (0.710190) and R4 (0.713899). The proxy averages validation linear-probe and kNN macro-F1 (Table 5). At four passes both policies have 262,848 updates and 201,866,872 cell presentations. T4 is 227M GroupMask at the same exposure.

For the original 22 settings, four-pass perturbation/integration gains are +0.0233 MRR/+0.0046 scIB, while few-shot, disease and OOD favor R4. All original scores and differences remain in Tables 13 and 6. We also integrate the finer targets and added cohorts into revised Tables 1 and 2 using fixed source membership and weighting rules: targets are averaged within their source, then sources equally; partial coverage is marked. OOD Overall aliases are not double-counted; disease sensitivity protocols are specified separately, and perturbation uses distinct within-source and cross-unit columns. Appendix B.1 gives the full rules.

Source-balanced task / metricG4R4T4G4-R4T4-G4
Annotation / F10.85690.85450.8649+0.0024+0.0080
Integration / scIB0.59370.58910.5836+0.0046-0.0100
Few-shot / AULC0.72920.72470.7461+0.0045+0.0169
Disease / AUROC0.73160.72510.7327+0.0065+0.0011
Mapping / F10.80230.81370.8307-0.0113+0.0284
OOD / AUROC0.90050.89390.8986+0.0066-0.0019
Perturb. within / MRR0.72690.70580.7402+0.0211+0.0133
Perturb. cross / MRR0.53420.51690.5414+0.0173+0.0072

F1 is macro-F1; differences use unrounded scores. The expanded four-pass panel favors GroupMask in seven of eight columns, while mapping favors RandomMask. T4 improves six columns over G4; integration and OOD favor G4. Given the limited rebuttal time, training of the valid 227M RandomMask control has not yet completed. T4-G4 measures GroupMask scaling; the masking-policy control remains at 96M.

What the finer targets show. On the same HYPOMAP cells and donor folds, with per-class label budgets, the four-pass few-shot difference changes from -0.0074 AULC for eight labels to +0.0211/+0.0221 for C2/C3 (52/156 labels). LBCL pDC subtype F1 improves from 0.6566 to 0.6788; CF macrophage disease AUROC improves from 0.8471 to 0.8643. Figure 3 and Tables 16-18 in Appendix D.2 locate these benefits by target and study. PCA still leads both expanded perturbation means; the subtype and disease results identify settings where the learned representations are useful.

AuthorsOpenReview ↗

Response 2/3: Q2 - what the external-baseline overlap checks established

Yes. We attempted source-, donor- and cell-level checks for all six evaluated external releases wherever public records support them. The audit covers the 11 sources of the original 22-setting benchmark. Appendix C.2, Tables 10 and 11, replaces the blanket statement "external-model donor and cell overlap remain unknown" with model-specific findings and an audit-coverage matrix.

We linked DOI, GEO/ENA and CELLxGENE identifiers to official corpus inventories, paper supplements, code/release records and archived metadata, then compared sequencing runs, donors and full cell IDs where available. The concrete findings are:

Release / evaluation sourceFinding and evidence level
Geneformer V2 / MS, SLE, PancreasThe Genecorpus-104M inventory lists MS, SLE and the Baron, Segerstolpe and Wang studies retained in Pancreas: confirmed study-source correspondence.
scFoundation / MS21 inventory samples link through ENA SRX/SRR records to all 21 evaluation donors and the sample origins of 21,312 evaluation cells: sequencing-sample correspondence.
scFoundation / NormanGSE133344 and eight associated samples occur in the published inventories: study/sample correspondence.
scGPT / SLEAll 20,000 evaluation full-cell IDs match the archived source, including donor and disease fields. Of these, 7,596 normal cells from 99 donors meet the public pretraining filter.
scGPT / AIDAEarlier Phase 1 data share 503 donor IDs, covering 168,062 evaluation cells. Unique donor-plus-barcode matching yields 159,311 candidate cell correspondences.

These findings have distinct meanings. The scFoundation count describes evaluation cells originating from the matched samples. The scGPT SLE result establishes archived-source cell identity and filter eligibility. AIDA cell matches remain candidates because full IDs differ and library-name conversion is unresolved. Exact training membership of a released checkpoint additionally requires its post-filtering training/validation assignment. For scGPT, its paper specifies the 15 May 2023 Census snapshot, whereas the inspected construction script specifies 8 May; the revised text retains these source/version distinctions.

For the other releases, we also inspected the available documentation. CellVQ's reported 68.34M-cell/441-study corpus lacks a frozen study/donor/cell inventory in those records. For CellFM-80M, we did not establish a version-matched membership inventory. For CellPLM 20231027_85M, the earlier paper and evaluated release configuration describe different corpus scopes without a complete membership mapping. Appendix C.2 documents the inventory and version information available for each release and the remaining membership gaps.

The common evaluation roster is explicit. After the documented TotiCell overlap filtering, HIHA and Pancreas retain 17,089 and 14,097 cells. Their recorded source-cell IDs have no matches in TotiCell's actual 50,466,718-cell training split, checked across all 771 identity-metadata shards. Every compared model uses these same retained evaluation cells. The external-source findings above concern this common roster; the controlled GroupMask/RandomMask comparison additionally shares the same training split. Appendix C.2 now distinguishes what was checked, what correspondences were found, and the evidence level of each result.

AuthorsOpenReview ↗

Response 3/3: Q3 - recurring reconstruction tasks, design evidence, and coverage

GroupMask concentrates training on recurring complementary prediction tasks while preserving the masking budget. The 48 blocks are assigned to three fixed 16-block target sets A, B and C. The shared encoder repeatedly predicts A from B+C, B from A+C, or C from A+B. Each block is hidden with probability 1/3, and 32 tokens remain visible, exactly as under RandomMask.

The intervention changes joint visibility. When predicting a block, other blocks in its target group are always hidden; under RandomMask, another block is visible with conditional probability 32/47. The complementary groups are consistently available under GroupMask, while cells and expression values continue to vary. Our hypothesis is that these recurring context-target relationships help a shared encoder learn reusable expression signals distributed across groups under a finite training budget. Sections 2-3 now state this reconstruction-task rationale and position it relative to prior work.

We test this rationale with fixed-bank and dynamic controls. At 30.65M parameters, five million cells, two passes and 9,766 updates, all policies share architecture, gene blocks, per-block masking probability and 32 visible tokens. The proxy evaluates frozen, full-input representations using the mean of linear-probe and kNN macro-F1. SD is across three paired pretraining runs.

PolicyTarget constructionProxy mean +/- SD
Static GroupMask3 fixed complementary sets0.471152 +/- 0.003042
Bank-62 fixed partitions, 6 sets0.456603 +/- 0.002991
Bank-93 fixed partitions, 9 sets0.450127 +/- 0.001272
Periodic refreshNew partition every 0.25 pass0.454663 +/- 0.001441
RandomMaskUniform 16-of-48 subsets0.435037 +/- 0.005619

Static GroupMask exceeds periodic refresh by 0.016489, with positive paired differences in all three runs. Refresh starts from the same partition and uses eight partitions across training. Both expanding the fixed bank and refreshing the partition reduce the proxy in these controls, supporting recurring complementary tasks (Figure 4; Section 6.1; Appendix G.2, Table 30).

The evidence extends to the main 96M model. On the original 22-setting reference panel, at three passes, the three-pattern bank exceeds each expanded bank in six of seven task means; at four, it exceeds Bank-6 in five and Bank-9 in six. Few-shot favors the larger banks and four-pass annotation favors Bank-6 (Table 31). Together with the matched-endpoint gains and finer-target transfer described in our Q1/Q4 response, these results connect the design to frozen representation quality. The revised contribution focuses on recurring reconstruction targets and their measured effects under the controlled training recipe.

Relation to existing methods (W1). We now discuss the direct antecedents explicitly. MAE supplies visible-input encoding and hidden-target reconstruction. SAVE uses gene-block attention within a conditional VAE and flow-matching generation framework. TxFM applies transcriptome MAE with individual gene tokens and a cell-level reconstruction bottleneck. CorrMask uses an expression-derived dependency graph for joint masking. Our study examines complementary fixed targets in a block MAE through controlled masking, partition and exposure comparisons.

Coverage and input wording (W3-W4). "Full coverage" now explicitly means all positions of the predefined 21,912-gene vocabulary, including observed zeros. Genes outside it are omitted; unmeasured or unmapped positions are zero-filled. These numerical zeros do not distinguish missing measurements from observed zeros; padding is separately excluded from the loss. The 48-token count specifies sequence length. For the original evaluation sources, Appendix C.1, Table 9, reports mapped-gene counts and executed transformations: HIHA, HYPOMAP, SLE, LBCL and Sound Life use normalization to 10,000 followed by log1p; AIDA, Pancreas, MS, Dixit, Norman and Adamson use the supplied X without an additional normalization or log transform. This replaces the generic "recorded transform" and "lack trustworthy raw counts" wording with concrete input records.

The 96M downstream design controls share a fixed 22-setting panel spanning all seven task families, with common labels, splits and readouts across variants.

AuthorsOpenReview ↗

Rebuttal follow-up

Dear Reviewer yUKe,

We hope this message finds you well. As the discussion period is nearing its end, we would like to kindly check whether our responses have addressed your concerns. Your feedback is very important to us, and we are happy to clarify anything further.

Thank you again for your insightful comments, which have greatly improved our paper.

Best regards,

The Authors

Reviewer_rBsGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TotiCell, which maps 21,912 genes into 48 fixed blocks and trains a compact masked autoencoder with GroupMask. GroupMask chooses one of three fixed sets of 16 blocks, matching RandomMask in marginal masking probability and visible token budget while using a much smaller mask support. The paper evaluates 22 settings from seven task families and reports an efficiency advantage on one RTX 3090. The masking idea is concrete, but the paper needs to distinguish a genuine contribution from a well executed change in training recipe.

Critical strengths
  1. GroupMask is a clear and testable intervention. The paper gives its marginal mask probability, co masking probability, and entropy, so the reader can see exactly how it differs from RandomMask.

  2. The evaluation is broad and not limited to one downstream task. It covers annotation, integration, few shot transfer, disease classification, perturbation retrieval, cross atlas mapping, and OOD detection, with setting level results in the appendix.

  3. The fixed 48 token input has a real engineering motivation. The throughput section reports both model and end to end measurements and distinguishes the 96M GroupMask model from the 227M TotiCell model.

Critical weaknesses
  1. The paper presents a wide 22 setting table, but the main evidence is not equally strong across tasks. GroupMask is ahead of RandomMask in some task means, while annotation is only +0.049, mapping is +0.239, and OOD is essentially tied. The clearest gain is perturbation retrieval, so the broad claim should be narrowed if the authors do not add a more direct mechanism test.

  2. GroupMask and RandomMask are compared at different selected checkpoints. GroupMask is selected at three passes and RandomMask at four passes. That may indicate faster convergence, but it also mixes the masking policy with training exposure. A same endpoint comparison is needed before the difference can be attributed to GroupMask itself.

  3. The TotiCell versus CellVQ comparison is mainly an engineering comparison. The systems differ in tokenizer, architecture, parameter count, training corpus, number of training cells, input processing, pooling, and checkpoint source. It shows that this TotiCell system is competitive on the chosen roster, but it does not establish that GroupMask or the block tokenizer caused the gains.

  4. The novelty claim is stronger than the mechanism evidence. Fixed gene blocks and recurring mask sets are a sensible design, but the paper does not show that the result is more than a simple support restriction that happens to work with the chosen partition. The mechanism probes use a small model, expression informed partitions are not combined with GroupMask, and the perturbation result uses pseudo replicates from the same source. The paper should either add the missing matched experiments or present GroupMask as a useful engineering recipe rather than a general masking principle.

Questions for authors
  1. What are the 22 setting results for GroupMask and RandomMask at the same four pass endpoint and the same number of cells seen?

  2. How much of the TotiCell versus CellVQ gap remains when tokenizer, model size, training corpus, and training budget are matched?

  3. Does the GroupMask advantage remain on a donor or experiment level held out perturbation split, and what happens when expression informed partitions are combined with GroupMask?

4 comments
AuthorsOpenReview ↗

Response to rBsG (1/3): Q1 - matched endpoints and scope of the gains

Thank you for the concrete suggestions. This comment addresses Q1 and W1/W2: we first give the requested original 22-setting comparison, then explain how the broader evaluation changes the task summaries.

Checkpoint selection. G/R denote 96M GroupMask/RandomMask; 3/4 denote corpus passes. Both runs completed four passes. The proxy averages validation linear-probe and kNN macro-F1 from frozen, full-input, mean-pooled representations on a fixed split. Each policy selects its maximum among the same eight half-pass candidates:

PassG proxyR proxy
0.50.6846020.677570
1.00.6968950.696269
1.50.6924000.693509
2.00.7030590.706678
2.50.7092350.712917
3.00.7101900.710543
3.50.7089060.713490
4.00.7089910.713899

This selects G3/R4, shared across downstream tasks. Table 5 gives both proxy components.

The original 22 settings at matched exposure. Tokenizer, architecture, 96,450,121 parameters, the 50,466,718-cell corpus and optimization schedule are fixed. G4/R4 each have 262,848 updates and 201,866,872 cell presentations; G3/R3 each have 197,136 updates and 151,400,154 presentations. Below, D33=G3-R3, D34=G3-R4, D44=G4-R4. Differences use unrounded scores.

TaskG3G4R3R4D33D34D44
Annotation0.88510.88380.88390.8847+0.0012+0.0005-0.0009
Integration0.59480.59370.59230.5891+0.0025+0.0057+0.0046
Few-shot0.80060.79990.80560.8066-0.0050-0.0060-0.0067
Disease0.85500.85760.86810.8592-0.0131-0.0042-0.0017
Mapping0.81600.80230.79890.8137+0.0172+0.0024-0.0113
Perturbation0.79090.78490.76710.7616+0.0237+0.0293+0.0233
OOD0.93480.92990.93550.9348-0.0007+0.000002-0.0050

Metrics are macro-F1 for annotation/mapping, scIB composite for integration, AULC for few-shot, AUROC for disease/OOD, and MRR for perturbation. Tables 13 and 6 retain every original setting and paired difference. Perturbation and integration gains persist at both endpoints; all three original perturbation datasets favor G4 over R4.

Task evidence beyond the original label/split choices. The revised main tables incorporate the finer targets and added cohorts using one common rule: average eligible targets within each source, then weight sources equally. HYPOMAP's three label levels and LBCL's lineage targets therefore do not multiply their cohorts' weights. Disease averages five studies using the specified matched essential-tremor (ET) analyses, cystic-fibrosis (CF) subsets and vascular three-class target; OOD averages Easy/Hard once per source. Perturbation separates within-source retrieval from cross-unit all-pairs retrieval, with the two Dixit screens averaged within that study. Appendix B.1 records the membership, weights and sensitivity protocols. These exploratory analyses use fixed required targets and weighting rules; partial external-model coverage is marked.

Integrated taskG3-R3G3-R4G4-R4
Annotation+0.0018+0.0041+0.0024
Integration+0.0025+0.0057+0.0046
Few-shot+0.0049+0.0046+0.0045
Disease+0.0024+0.0046+0.0065
Mapping+0.0172+0.0024-0.0113
OOD+0.0113+0.0106+0.0066
Perturb. within+0.0225+0.0242+0.0211
Perturb. cross+0.0159+0.0161+0.0173

These are expanded source-balanced summaries, distinct from the original 22-setting means above. Table 2 also gives all four checkpoint scores. At four passes, gains occur in seven of eight columns across six of seven task families; mapping remains negative. Figure 3 shows the source-specific effects: on the same HYPOMAP cells and donor folds, with label budgets defined per class, the four-pass few-shot difference changes from -0.0074 AULC for eight labels to +0.0211/+0.0221 for 52/156 labels. LBCL pDC subtype F1 improves by 0.0222. These comparisons identify finer distinctions where GroupMask helps. Full target scores appear in Tables 16-19. The biological perturbation tests are addressed under Q3.

AuthorsOpenReview ↗

Response to rBsG (2/3): Q2 - controlled attribution and the CellVQ comparison

Q2: Matching TotiCell and CellVQ. We appreciate the aim of identifying each component's contribution. Matching tokenizer, model size, corpus and training budget requires adapting both systems and conducting new pretraining under common conditions. Further evaluation of released checkpoints cannot equalize these factors. The inspected CellVQ artifacts do not provide the evaluated checkpoint's exact training-cell assignment and cumulative exposure. A fully matched cross-system retraining study is beyond the time and resources available for this rebuttal, and the residual gap under all four matched conditions has not been quantified.

The revision makes the two comparisons' purposes explicit in Sections 4.2-5.2 and Appendix C.1.

1. Controlled masking contribution. The 96M comparison fixes tokenizer, architecture, parameter count, training corpus and optimization schedule, and includes equal-exposure endpoints. On the original 22 settings, the four-pass gains are 0.023345 MRR for perturbation and 0.004554 for integration. On the expanded source-balanced panel, the corresponding within-source/cross-unit perturbation gains are 0.021081/0.017316 MRR. Since the tokenizer is shared, these experiments measure GroupMask's effect conditional on that tokenizer. Our Q1 response gives both evaluation scopes separately.

2. Complete-system performance. TotiCell has 227.36M parameters and 50.47M pretraining cells; CellVQ reports approximately 511M parameters and 68.34M cells. Revised Table 1 uses the same cells, downstream splits, source membership and hierarchical weights for this complete-coverage pair, retaining each model's documented input adapter. The expanded panel gives:

TaskTotiCell-227MCellVQDifference
Annotation0.86490.8221+0.0427
Integration0.58360.5853-0.0017
Few-shot0.74610.6474+0.0988
Disease0.73270.7294+0.0033
Mapping0.83070.6848+0.1459
OOD0.89860.8653+0.0333
Perturb. within0.74020.6651+0.0750
Perturb. cross0.54140.4817+0.0597

Each source first averages its eligible targets, then receives equal weight within its task. Within/Cross separate artificial-replicate retrieval from all ordered experimental-unit pairs; Appendix B.1 specifies the source units and rules. Differences use unrounded scores.

TotiCell is higher in all columns except integration with 55.5% fewer total parameters. Figure 2 also reports the measured input length, throughput and memory costs. These are complete-system comparisons under documented conditions; the masking-policy attribution comes from the internal controls. The original 22-setting system scores remain in Table 12, and Appendices C.1 and C.2 give input processing and the available pretraining-source evidence.

AuthorsOpenReview ↗

Response to rBsG (3/3): Q3 - separated perturbation units and informed partitions

Q3(a): Perturbation retrieval across separated units. We added tests using actual donor, biological-replicate and acquisition-unit metadata. For each perturbation in each unit, the response is its mean embedding minus that unit's control mean. Cosine similarity ranks perturbation candidates across units. All encoders are frozen, with identical cells and candidates for each comparison. G/R denote 96M GroupMask/RandomMask after three/four passes; retrieval scores are MRR.

Shifrut uses TCR-stimulated primary T cells from two real donors, evaluated in both directions. Papalexi uses three author-labelled biological replicates under the same treatment. All-pairs evaluates each directed replicate pair; leave-one-unit-out (LOU) compares one query unit with the equal mean of the separately control-corrected responses from the other units. Differences below use unrounded scores.

ProtocolG3G4R3R4G3-R3G3-R4G4-R4
Shifrut, donor-to-donor0.34900.35660.33720.3333+0.0118+0.0157+0.0233
Papalexi, replicate pairs0.46100.46560.45220.4535+0.0088+0.0076+0.0122
Papalexi, replicate LOU0.47090.46570.47300.4700-0.0021+0.0009-0.0043

Shifrut directly tests donor-separated retrieval: the gain persists at both matched training endpoints. Papalexi shows a gain for replicate-pair retrieval and a smaller reversal for LOU, which changes the gallery construction.

For the revised cross-unit summary we use all ordered unit pairs throughout: eight screens, with the two Dixit screens averaged within that study, then seven studies weighted equally. G4-R4 is +0.017316 MRR; six study differences are positive and Adamson is -0.0062. Figure 3 distinguishes biological separation from technical libraries, capture-batch pools and lanes. Appendix D.2.2, Tables 20-22, reports all 20 protocol views, including LOU and matched-candidate pseudoreplicates. Equivalent two-unit LOU/pairs views for Shifrut and RPE1 count only once in the main summary.

Q3(b): Expression-informed blocks with GroupMask. We now combine them at 96M. A positive co-expression graph is estimated from 100,000 training cells and used to form 48 balanced blocks, frozen before pretraining without downstream labels. The comparison holds architecture, training data, optimizer, exposure and block-index target sets fixed; only gene-to-block assignment changes. The fixed original 22-setting reference panel gives these four-pass means, with GroupMask on both sides:

TaskRandom blocks + G4Prior blocks + G4Difference
Annotation0.88380.9011+0.0174
Integration0.59370.5953+0.0016
Few-shot0.79990.7959-0.0040
Disease0.85760.8855+0.0279
Perturbation0.78490.7958+0.0108
Mapping0.80230.7772-0.0251
OOD0.92990.9237-0.0062

The combination improves annotation, integration, disease classification and perturbation retrieval at both three and four passes. Random blocks remain stronger on the other three families. Table 29 also reports the full partition-by-masking 2x2 comparison. Thus expression-informed grouping is compatible with GroupMask and offers complementary, task-specific gains.

Design evidence underlying W4. The revision further tests recurring targets: at 30.65M and a fixed two-pass endpoint, static GroupMask's proxy is 0.471152 +/- 0.003042 versus 0.454663 +/- 0.001441 when the target partition is refreshed every quarter pass (mean +/- SD over three paired training runs); static is higher in all three. The 96M mask-bank study extends the support comparison to the original downstream tasks: at three passes, three recurring patterns exceed both six- and nine-pattern banks in six of seven task means. Section 6.1, Figure 4 and Appendix G.2 report these controls and their task-dependent outcomes. We frame the contribution as a recurring complementary reconstruction-task design supported by these controlled effects.

The 96M downstream design controls share a fixed 22-setting panel spanning all seven task families, with common labels, splits and readouts across variants.

I appreciate the substantial additional work in the rebuttal. The matched-training comparisons and the donor-to-donor evaluation address important concerns from my original review. I still find the motivation for the masking design and the significance of the gains limited, especially for practical single-cell analysis. On balance, the added evidence is enough for me to raise my rating from 2 to 3 (weak accept).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.