acceptodds
Under review as a conference paper at ICLR 2027

scWEAVE: An Efficient Multi-Species Single-Cell Foundation Model via Meta-Gene Compression and Contrastive Alignment

Abstract

Single-cell foundation models (scFMs) show great promise across various biological tasks. However, most existing models are computationally expensive and trained mainly on human data, limiting their applicability to multi-species analysis. Extending scFMs across species is non-trivial: gene sets differ substantially across species, and biological divergence together with batch effects induce significant cross-species distribution shift. We present scWEAVE, an efficient scFM that jointly reduces computational cost and generalizes across species. We introduce a gene compression module that uses knowledge from protein language models and gene ontology to softly compress genes from any species into a shared, compact set of meta-genes, making cross-species data directly comparable while shortening the token sequence the transformer must process. To address distribution shift, scWEAVE decomposes each cell's total variation into a shared cell factor encoding shared cell-state variation and a remnant factor that supplies species-specific information needed for accurate masked expression prediction – the main objective of scFMs. We further align the shared cell factor across species using a supervised contrastive regularizer based on cell-ontology identity, which accounts for inconsistent label granularity between species-specific annotations and corrects for residual batch-effect distortion during pretraining. We validate scWEAVE through extensive experiments on datasets spanning multiple species. scWEAVE matches or exceeds baseline performance on ortholog retrieval, gene expression imputation, batch effect removal, and novel cell type discovery, while cutting inference time up to 35x and memory up to 6x relative to TranscriptFormer.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
single-cell RNA-seq, multi-species data integration, single-cell foundation model, cell representation learning, computational biology

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tJyUOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

this paper propose a biological foundation model to reduce computation cost and generalize across species. its main innovation is the meta-gene, which bulids gene similarity graphs using the ESM2 embeddings and GO embeddings, and then performs soft-assignment and maps genes to meta-genes. in the way the gene sequences are shorten. the framework then trains the model with masked expression prediction task and contrastive regularizatoin. experiments on datasets covering multiple species demonstrate the effectiveness and efficiency of the proposed method.

Critical strengths

S1: compressing genes to meta-genes is quite inspiring. such a methodology sounds reasonable but never found in previous literatures on bioinformatics as far as i know. this could significantly reduce computation cost, which is especially critical given the wide application on gene expression data of transformer models, whose complexit is O(n^2) over the sequence length, and preserve the functional characteristics of the genes.

S2: the paper proposes a very promising method to eliminate or alleviate batch effect, i.e. through contrative learning via the cell type signal. moreover, it successfully addresss the problem of inconsistent naming conventions by using multi-granular labels.

Critical weaknesses

W1: introducing too much functions in the MEP step add the complexity of the framework. this is related to Q2 but different. in Q2 i question the evidence of gain, but it makes the method delicated and sensitive nevertheless. it is doubtable if this design could achieve consistent gain under varied scenarios.

W2: the authors claims a multi-species solution. however, in the methodology it does not explain how the designs overcome gene inconsistency between speicies.

Questions for authors

Q1: i am not sure about the purpose of the denoising autoencoder in Figure 1 (b). it is possible to get gene embeddings from the ESM2 model and GO annotations, which can then be concatted or added and used for graph construction in the next step. why do you introduce this additional embedding step? what is the motivation and purpose, and what is the performance gain and practical benefit?

Q2: how is x_ig recovered from x_img in the MEP step? there is no specification in the paper. the expression is the product of x^bl and x^context. what is its advantage against direct prediction? and the multiplication operation (instead of summation or function-based operation) reasonable? need theoretical justification or empirical validation.

Q3: how many genes do your dataset cover? are they all protein coding genes? there is around 20000 protein coding genes but far more non-coding genes. the latter is often neglected in current research. one of the constraint is the exceesive sequence length. can your method help with this problem? the ESM2 embeddings might be unavailable for these genes since they do not transcribe proteins. do you have and solutions?

6 comments

Summary: We thank the reviewer for the constructive feedback. Following the reviewer's suggestions, we conducted additional tests on scWEAVE. We highlight the following results that address the reviewer's major concerns:

  1. We conducted ablation studies on the designed components in the MEP step. We tested the necessity of using product of x^bl and x^context for expression prediction, and the necessity of using the binary detection head during training (Q1 and Q4). We trained ablated scWEAVE that predicts gene expression directly (x = x^context, remove x^bl) and ablated scWEAVE removed of the detection head, and compared the two ablated models with the original scWEAVE across 11 test datasets to make sure that the results are robust across various scenarios (Q1). scWEAVE shows consistent improvement with the inclusion of the detection head and of x^bl.

  2. We provided a detailed explanation on the motivation of MEP step designs (Q1), and on how scWEAVE generalizes across the genes of different species (Q2). We explained why we use the denoising autoencoder before gene compression (Q3), and provided a solution for future adaptation to non-protein-coding genes (Q5).

Q1. introducing too much functions in the MEP step add the complexity of the framework. this is related to Q2 but different. in Q2 i question the evidence of gain, but it makes the method delicated and sensitive nevertheless. it is doubtable if this design could achieve consistent gain under varied scenarios.

A1. We agree with the reviewer that a comprehensive test is needed to validate the effectiveness of each additional function/design in MEP step, and to show their robustness across various application scenarios.

The additional designs of MEP step mainly include: (1) decomposing the predicted expression into the product of x^bl and x^context; (2) including a binary detection head that predicts binary (zero vs. non-zero) gene expression as an auxiliary objective to the main masked prediction objective.

In the following, we explain the motivation for introducing these two components, and conduct ablation tests to evaluate their effectiveness.

Decomposition of expression (jointly addressing Q4): We model the predicted expression as the product of x^bl (the output of the baseline head) and x^context, instead of predicting x directly, to better capture cell-to-cell variation. The baseline head predicts x^bl, the background expression of each gene. x^bl depends only on the gene identity and is independent of cells, whereas x^context predicts the cell-specific deviation of gene expression from the background x^bl. This decomposition introduces an additional inductive bias that encourages the predictor to better explore cell-to-cell variation, and it improves rather than destabilizes the training of scWEAVE.

To evaluate the effectiveness of this design, we conducted an ablation study focusing only on the baseline head. We trained a vanilla scWEAVE with the baseline head (termed scWEAVE) and scWEAVE without the baseline head (no decomposition, termed scWEAVE (w/o decomposition)). We evaluate how well each model learns cell-to-cell variation through the masked prediction objective, which is the main training objective where the baseline head directly affect. To better measure the recovery of cell-to-cell variation in masked prediction, we introduce an additional metric, the residual Pearson correlation (Residual PCC). Compared with the Pearson correlation (PCC), Residual PCC removes the global expression level of each gene, and therefore focuses only on whether scWEAVE recovers cell-specific gene expression variation:

Residual PCC = PCC( prediction − gene mean ,  observed − gene mean )

To ensure that the result is robust across various scenarios, we evaluated the two models on the 11 evaluation datasets used in the manuscript. The results are shown in Table R1. Averaged over the 11 datasets, scWEAVE outperforms scWEAVE (w/o decomposition) on Pearson correlation (PCC), Spearman correlation (SCC), Residual PCC, and negative log-likelihood (NLL, the primary training loss). The gain especially in Residual PCC shows that the use of baseline head improves scWEAVE's capability of capturing cell-to-cell variation.

Use of the binary detection head: We first clarify that the binary detection head and its training loss (the binary detection loss) serve only as an auxiliary objective that improves the training of scWEAVE on real-valued gene expression prediction. The head is not used for any downstream application.

Single-cell gene expression is extremely sparse, and most masked entries are zero. Trained with the negative binomial loss alone, the predictor receives a learning signal dominated by zero entries and only a weak signal on which genes are actually expressed in a given cell. The detection head adds a class-balanced binary objective (whether each masked gene is expressed), computed from the same hidden representation that the expression predictor uses. This forces the shared representation to distinguish expressed from non-expressed genes in a cell-specific way, which in turn improves the real-valued expression prediction.

We conducted an ablation study to evaluate the effectiveness of the detection head. We trained a vanilla scWEAVE with the detection head (termed scWEAVE) and the ablated scWEAVE without the detection head (termed scWEAVE (w/o detection head)). We evaluate the effectiveness of the detection head through the masked prediction objective, which the detection head is designed to improve. As before, we evaluate both models using Pearson correlation (PCC), Spearman correlation (SCC), Residual Pearson correlation (Residual PCC), and negative log-likelihood (NLL, the primary training loss). The results (Table R2) show that scWEAVE with the detection head has a large performance gain over scWEAVE without the detection head across all evaluation metrics and various test datasets, shows the effectiveness and robustness of detection head design in scWEAVE.

Table R1. Ablation of the decomposition on gene expression imputation, measured using Pearson (PCC) and Spearman (SCC) correlation, residual Pearson correlation (Residual PCC), and negative binomial negative log-likelihood of the observed counts (NLL).

(a) Average over 11 datasets

MetricscWEAVEscWEAVE (w/o decomposition)
PCC ↑0.6050.604
SCC ↑0.4790.478
Residual PCC ↑0.1380.133
NLL ↓1.0011.008

(b) Pearson correlation (PCC ↑)

DatasetscWEAVEscWEAVE (w/o decomposition)
Immune0.7320.733
Pancreas0.5290.535
Covid-190.6700.669
PBMC0.7110.713
Dendritic0.6600.655
Cortex-3B0.5000.493
Retina0.5750.572
Hippo0.5050.506
Motor0.6680.667
AH-Outflow0.5840.583
Striatum0.5180.518

(c) Spearman correlation (SCC ↑)

DatasetscWEAVEscWEAVE (w/o decomposition)
Immune0.4750.473
Pancreas0.4670.471
Covid-190.4970.495
PBMC0.3990.398
Dendritic0.5100.506
Cortex-3B0.4650.461
Retina0.5160.515
Hippo0.4310.431
Motor0.6120.613
AH-Outflow0.4500.449
Striatum0.4480.449

(d) Residual Pearson correlation (Residual PCC ↑)

DatasetscWEAVEscWEAVE (w/o decomposition)
Immune0.1670.162
Pancreas0.0760.070
Covid-190.2200.219
PBMC0.1770.172
Dendritic0.0470.041
Cortex-3B0.1780.170
Retina0.1900.188
Hippo0.1480.144
Motor0.0930.086
AH-Outflow0.0830.080
Striatum0.1400.133

(e) Negative log-likelihood (NLL ↓)

DatasetscWEAVEscWEAVE (w/o decomposition)
Immune0.6010.602
Pancreas1.5401.540
Covid-190.5440.545
PBMC0.2870.288
Dendritic3.3213.358
Cortex-3B0.7180.720
Retina0.7090.710
Hippo0.6410.650
Motor1.2521.268
AH-Outflow0.7580.761
Striatum0.6400.647

Table R2. Ablation of the detection head on gene expression imputation, measured using Pearson (PCC) and Spearman (SCC) correlation, residual Pearson correlation (Residual PCC), and negative binomial negative log-likelihood of the observed counts (NLL).

(a) Average over 11 datasets

MetricscWEAVEscWEAVE (w/o detection head)
PCC ↑0.6050.598
SCC ↑0.4790.468
Residual PCC ↑0.1380.119
NLL ↓1.0011.010

(b) Pearson correlation (PCC ↑)

DatasetscWEAVEscWEAVE (w/o detection head)
Immune0.7320.723
Pancreas0.5290.519
Covid-190.6700.656
PBMC0.7110.704
Dendritic0.6600.647
Cortex-3B0.5000.488
Retina0.5750.570
Hippo0.5050.508
Motor0.6680.667
AH-Outflow0.5840.573
Striatum0.5180.521

(c) Spearman correlation (SCC ↑)

DatasetscWEAVEscWEAVE (w/o detection head)
Immune0.4750.457
Pancreas0.4670.461
Covid-190.4970.478
PBMC0.3990.387
Dendritic0.5100.485
Cortex-3B0.4650.451
Retina0.5160.508
Hippo0.4310.430
Motor0.6120.606
AH-Outflow0.4500.440
Striatum0.4480.447

(d) Residual Pearson correlation (Residual PCC ↑)

DatasetscWEAVEscWEAVE (w/o detection head)
Immune0.1670.141
Pancreas0.0760.065
Covid-190.2200.193
PBMC0.1770.145
Dendritic0.0470.039
Cortex-3B0.1780.150
Retina0.1900.173
Hippo0.1480.139
Motor0.0930.082
AH-Outflow0.0830.059
Striatum0.1400.127

(e) Negative log-likelihood (NLL ↓)

DatasetscWEAVEscWEAVE (w/o detection head)
Immune0.6010.608
Pancreas1.5401.540
Covid-190.5440.551
PBMC0.2870.293
Dendritic3.3213.374
Cortex-3B0.7180.724
Retina0.7090.706
Hippo0.6410.643
Motor1.2521.263
AH-Outflow0.7580.761
Striatum0.6400.642
Reviewer_iUcAOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary

scWEAVE proposes an efficient multi-species single-cell foundation model that compresses genes into 512 shared meta-genes using protein-language-model embeddings and Gene Ontology annotations. This shortens the transformer input and provides a common feature space across species. The model combines masked expression prediction with ontology-aware contrastive learning, separating its representation into a shared cell factor for cross-species alignment and a remnant factor intended to retain information needed for expression reconstruction.

The model is pretrained on 56 million healthy human and mouse cells and evaluated on 11 datasets covering expression imputation, ortholog retrieval, batch integration, and novel cell-type detection. It reports strong imputation performance and competitive integration results, while novel cell-type detection is more mixed. Its clearest practical contribution is computational efficiency: the reported inference is approximately 35 times faster and uses six times less GPU memory than TranscriptFormer under the tested configuration.

Critical strengths

The shared meta-gene representation addresses computational efficiency and cross-species compatibility within the same design. Compressing genes into 512 tokens substantially reduces inference costs while allowing species with different gene vocabularies to use a common representation space. The reported speed and memory improvements over TranscriptFormer are practically consequential, particularly for large-scale analyses with limited GPU resources.

The evaluation provides evidence that this compression retains useful biological information. scWEAVE achieves the highest imputation Pearson correlation on 10 of 11 datasets and competitive cell-clustering performance across human, mouse, and multi-species settings. Ortholog retrieval on species excluded from cell-level pretraining further supports the transferability of the gene representation. Together, these results make a credible case that the efficiency gains can coexist with useful predictive performance.

Critical weaknesses

The paper lacks controlled ablations establishing the contribution of its main components. Without comparisons removing GO information, the remnant factor, or ontology-aware contrastive learning, it is difficult to determine which design choices explain the results. In particular, the claimed separation of shared cell-state information from species- and batch-specific variation is not directly validated. This matters because improved alignment could also suppress meaningful biological differences, and the current evaluations do not establish that these differences are preserved.

The independence of the evaluation data from pretraining is insufficiently documented. Pretraining uses a large CELLxGENE human–mouse corpus, while several evaluation datasets are also available through CELLxGENE; the paper does not report a clear study-, donor-, or cell-overlap audit. Holding out other species provides useful evidence of transfer, but does not resolve potential overlap in the human and mouse evaluations. Consequently, it remains uncertain how much the reported performance demonstrates generalization to genuinely unseen datasets.

Questions for authors

This paper contains numerous issues—both in terms of its methodology and its experimental setup—that warrant further revision. Specifically:

  1. Meta-genes are intended to capture functional groupings of genes, yet these relationships can depend on cell identity, state, and microenvironment. However, scWEAVE uses a fixed, cell-independent assignment matrix to aggregate gene expression into meta-genes. It's rather like assuming that a group of genes are functionally interpreted in exactly the same way across completely different tissues. Correct me if I'm wrong.

  2. I find it hard to believe that introducing "meta-genes" is practically meaningful. Aggregating expression into a compressed meta-gene space sacrifices single-gene resolution, which is essential for tasks such as gene perturbation prediction: the model must distinguish the knockout of one gene from another and predict the coordinated responses of individual downstream genes. The authors should directly evaluate such tasks to determine whether its computational savings come at the expense of biological resolution.

  3. The cross-species motivation requires stronger validation against explicit ortholog mapping. Mapping query-species genes to human orthologs already provides a shared feature space without meta-gene compression. A controlled comparison on cross-species integration and transfer is needed to establish what the proposed mapping adds, particularly for genes lacking one-to-one orthologs.

  4. The contribution of each proposed component is unclear because the paper provides no controlled ablations. Comparisons removing GO information, ontology-aware contrastive learning, and the remnant factor are needed to establish which components drive the improvements. Varying the number of meta-genes would also reveal the trade-off between computational efficiency and biological information preservation.

  5. The CL-tree-based contrastive objective may distort continuous biological structure by imposing discrete annotation boundaries on the shared cell representation. In particular, sibling cell types receive no positive alignment weight, even when they represent adjacent states along a differentiation trajectory. Improved cell-type separation could therefore come at the expense of preserving developmental continuity and transitional populations. The current clustering and batch-mixing evaluations do not address this trade-off. Trajectory analyses before and after contrastive training are needed to establish whether the shared representation preserves these relationships.

9 comments

Summary: We thank the reviewer for the constructive feedback. We conduct additional tests on scWEAVE following the reviewer's suggestion, and we want to highlight the following results that address the reviewer's major concerns and strengthen our claim in the paper:

  1. We conducted a comprehensive ablation study on the remnant factor, contrastive regularization, meta-gene usage, and GO information. The results show that each component serves its intended function, and all components are important for scWEAVE performance.

  2. We conducted a comprehensive similarity audit between the training and test datasets. We removed the potentially overlapping parts of the test datasets and included an additional 4-species Striatum dataset (3 unseen species) in the evaluation, to address the reviewer's concern about train/test overlap and to strengthen the generalization claim of our model. scWEAVE still shows leading performance with the adjusted evaluation datasets.

  3. We evaluated scWEAVE on the perturbation prediction task and compared it with SOTA perturbation prediction methods, including GEARS, scGPT, and the Linear Predictor (shown to be highly accurate by Ahlmann-Eltze et al., Nature Methods 2025 [1]). The comparison shows that scWEAVE outperforms both scGPT and the Linear Predictor, and ties with GEARS.

  4. We evaluated scWEAVE on a dataset with a continuous cell trajectory. The evaluation shows that scWEAVE shows better results in trajectory preservation after contrastive regularization. Contrastive regularization helps separate cells of different stages, while the main objective, mask prediction, prevent it from overly separating cells in trajetory into distinct clusters.

Q1. The paper lacks controlled ablations establishing the contribution of its main components...

A1. We agree with the reviewer that a comprehensive ablation study is needed to evaluate the effectiveness of each key design component. Following the reviewer's suggestion, we conducted ablation studies on the remnant factor, contrastive regularization, GO information, and meta-gene compression. We test across all 4 tasks in the manuscript.

(1). Ablation on the remnant factor. We trained an ablated scWEAVE that uses only the shared cell factor for masked prediction (termed scWEAVE (w/o remnant)), and compared its performance with the original scWEAVE. The results show that the remnant factor provides a consistent performance gain to scWEAVE in data imputation, batch effect removal, and novel cell type detection (Tables R1, R2, R3). Ortholog retrieval performance remains similar (Table R4), which is expected because the remnant factor does not directly affect the gene alignment in the compression module.

(2). Ablation on the contrastive regularization. We trained an ablated scWEAVE without contrastive regularization (termed scWEAVE (w/o contra)), and compared it with the original scWEAVE. The results show that contrastive regularization improves scWEAVE's performance in data imputation, batch effect removal, and novel cell type detection (Tables R1, R2, R3). Ortholog retrieval performance also remains the same (Table R4), because contrastive regularization mainly affects the cell embedding.

(3). Ablation on the meta-genes. We trained an ablated scWEAVE that takes the top-512 highly expressed genes of each cell as input, instead of 512 meta-genes. For each species, the 512 top-expressed genes were selected by ranking their average expression. The ablated model is termed scWEAVE (w/o compression). We compared scWEAVE (w/o compression) with the original scWEAVE. The results show that the meta-gene representation of cells shows better performance than highly expressed genes (Tables R1, R2, R3). We did not evaluate ortholog retrieval for scWEAVE (w/o compression), because scWEAVE cannot retrieve orthologs without the gene compression module.

(4). Ablation on the GO information. We evaluated scWEAVE on the test datasets with protein-only and protein+GO input to test the effectiveness of the GO information (Tables R1, R2, R3, R5). GO information significantly improves ortholog retrieval (Table R5). It improves the quality of the gene representation and directly improves ortholog matching across species. GO also improves data imputation (Table R1). The improvement in batch effect removal and novel cell type detection is not apparent (Tables R2, R3). This result is expected because the gene representation mainly helps gene alignment across species and then the data imputation objective. Its influence on the cell representation is limited.

We only include average score in Tables R1 and R2 due to the space limit, but full score will be included in the manuscript. Bold and italic values indicate the best and second-best scores, respectively.

Table R1. Ablation of scWEAVE on gene expression imputation, measured using Pearson (PCC) and Spearman (SCC) correlation.

Average over 11 datasets

MetricscWEAVEscWEAVE (w/o remnant)scWEAVE (w/o compression)scWEAVE (w/o contra)scWEAVE (w/o GO)
PCC ↑0.5640.5560.4370.5600.555
SCC ↑0.4750.4740.3790.4690.469

Table R2. Ablation of scWEAVE on batch effect removal and cell type identification.

Average over 11 datasets

MetricscWEAVEscWEAVE (w/o remnant)scWEAVE (w/o compression)scWEAVE (w/o contra)scWEAVE (w/o GO)
ARI0.640.600.530.270.61
ASW0.600.590.600.510.60
ASW-B0.840.850.790.750.84
GC0.940.940.880.850.95

Table R3. Ablation of scWEAVE on novel cell type detection.

DatasetSpeciesscWEAVEscWEAVE (w/o remnant)scWEAVE (w/o compression)scWEAVE (w/o contra)scWEAVE (w/o GO)
HippoPig1.000.330.140.121.00
M. mulatta0.830.700.160.620.75
MotorC. jacchus0.390.580.090.380.45
Mouse1.001.000.440.591.00
AH-OutflowM. mulatta0.320.320.370.330.31
M. fascicularis0.430.350.280.630.58
Pig0.250.160.240.210.48
Mouse0.590.450.600.740.74
Average0.600.490.290.450.66

Table R4. Ablation of scWEAVE on top-1 human ortholog retrieval accuracy (%).

DatasetSpeciesscWEAVEscWEAVE (w/o remnant)scWEAVE (w/o contra)
AH-OutflowPig81.181.581.1
M. fascicularis81.681.781.7
M. mulatta79.679.979.8
HippoPig81.782.181.6
M. mulatta82.683.182.7
MotorM. mulatta83.383.383.3
C. jacchus83.884.184.1
StriatumChimpanzee85.585.785.5
M. mulatta81.381.581.2
P. discolor78.278.578.2

Table R5. Ablation of GO information on top-1 human ortholog retrieval accuracy (%).

DatasetSpeciesscWEAVEscWEAVE (w/o GO)
AH-OutflowPig81.177.8
M. fascicularis81.678.5
M. mulatta79.676.2
HippoPig81.778.7
M. mulatta82.680.2
MotorM. mulatta83.379.7
C. jacchus83.880.3
StriatumChimpanzee85.583.1
M. mulatta81.378.5
P. discolor78.275.1

Q2. The independence of the evaluation data from pretraining is insufficiently documented... Consequently, it remains uncertain how much the reported performance demonstrates generalization to genuinely unseen datasets.

A2. According to the reviewer's suggestion, we audited every evaluation dataset against the pretraining corpus. We looked up each test dataset in our training data, and confirmed that PBMC, Covid-19, Hippo, Motor, Cortex-3B, Retina, and AH-Outflow do not overlap with the training data. Immune, Lung Atlas, and Pancreas are more likely to overlap with, or be similar to, the training data. Since there are no directly comparable cell barcodes, we conducted a similarity search between the cells of these three datasets and the training data. We first selected the top 3,000 highly variable genes. For each evaluation cell, we computed the Jaccard similarity of expressed gene sets between the cell and every training cell in the same tissue pool to find the best match. For each best match, we then calculated the full-gene Jaccard similarity to verify the match, and flagged cells with Jaccard >= 0.7 as potentially overlapping.

The similarity check shows partial similarity in some batches of the Immune and Pancreas datasets, and a relatively larger potential overlap in the Lung Atlas dataset. We removed the potentially overlapping cells from the Immune and Pancreas datasets (keeping 15,289 Pancreas cells and 24,552 Immune cells), and dropped the Lung Atlas dataset from the evaluation. We also added a 4-species striatum dataset to strengthen the evaluation. The striatum dataset includes 3 species unseen in the training data, posing a larger generalization challenge to the models. We updated all benchmark results in the manuscript accordingly, and include the updated tables in this response (Tables R6, R7, R8, R9) for the reviewer's reference. After the update, scWEAVE still leads on the Immune and Pancreas datasets. More importantly, scWEAVE also leads on the Striatum dataset, showing its strong generalization ability in the multi-species setting.

Table R6. Imputation accuracy of scFMs, measured using Pearson correlation (PCC) and Spearman correlation (SCC).

DatasetscWEAVE PCC ↑scWEAVE SCC ↑scFoundation PCC ↑scFoundation SCC ↑scGPT PCC ↑scGPT SCC ↑scMulan PCC ↑scMulan SCC ↑
Immune0.7100.4870.4340.4210.7040.5550.4820.313
Pancreas0.4840.4590.4140.4080.4980.4220.2800.154
Covid-190.6240.5000.3810.4370.5760.4970.5970.375
PBMC0.6580.4030.4930.4520.6570.5760.5550.352
Dendritic0.6040.5200.4620.4340.5990.4900.5910.337
Cortex-3B0.4370.4250.2270.2310.2480.2160.2060.116
Retina0.5340.5100.2860.2760.3570.3090.2840.054
Hippo0.4860.4290.3190.2990.3430.3310.2420.140
Motor0.6420.6020.3560.3780.5020.4340.3620.205
AH-Outflow0.5230.4420.3520.3180.4790.3930.5110.292
Striatum0.5010.4470.3530.3450.3780.3640.3510.244
Reviewer_qvX9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents scWEAVE, a foundation model pretrained on single-cell gene expression data from human and mouse. The architecture claims three contributions: a gene compression module that uses protein-language-model embeddings and gene ontology annotations to softly map genes from any species onto 512 shared “meta-genes”; a decomposition of the encoder output into a shared cell factor, intended to be batch- and species-invariant, and a remnant factor that carries the species-specific information needed for masked expression prediction; and a supervised contrastive regularizer, weighted by Cell Ontology distance, that aligns the shared cell factor across datasets despite differences in label granularity. The authors then present experiments on inference cost, dropout imputation, ortholog retrieval, batch integration and novel cell type detection, reporting that the approach matches or outperforms existing scFMs on most of them.

Critical strengths
  1. scWeave is a comparably more efficient than other scFMs, with the latency being 90% lower than scGPT.
  2. The cross species gene compression module is inspired, and leverages pretrained PLMs and gene ontologies effectively. A new species with protein sequences can be easily without ortholog mapping.
  3. The ontology-weighted contrastive loss is a sensible fix for a real problem. Labels across atlases disagree in granularity (“T cell” in one, “CD4+ T cell” in another), and is a cleaner method rather than collapsing cell types.
Critical weaknesses

1.The primary weakness is the lack of useful ablations to show which of the architectural / training improvements contribute to the performance. Specifically, it would be great to have experiments showing the effects of the contrastive loss and the remnant factor.

  1. On the two multi-species brain datasets, scWEAVE is not in the top two on ARI (Hippo 0.67 vs 0.72; Motor 0.26 vs 0.28–0.34) and shows the largest seed variance in the table, which weakens the paper's claim that this approach generalizes to cross species. The ablation studies would further help here to narrow down if any of the design choices are actually reducing the effectiveness of the approach on cross species generalization.

  2. The paper is missing simple baselines. HVG+PCA is essentially free, and outperforms other scFMs. Kedzierska et al. also showed scGPT and Geneformer lose on three of the test sets. Separately, pretraining uses all of CELLxGENE healthy human and mouse, and the scIB sets are deposited there; the paper does not say evaluation cells or their labels were held out.

Questions for authors
  1. Can we see the results for the stage-2 checkpoint, i.e. the model before the contrastive loss is applied. How much of the ARI and GC gap to UCE and TranscriptFormer survives without label supervision?

  2. Train the same 6-layer transformer on 512 per-gene ESM-2 tokens (top-expressed or sampled, UCE-style) with the same objective and data, and report Tables 3 and 5. This will help isolate the meta-gene tokenization at matched cost.

6 comments

Summary: We thank the reviewer for the constructive feedback. We conduct additional tests in this response following the reviewer's suggestions, and we want to highlight the following results that address the reviewer's major concerns:

  1. We conducted comprehensive ablation study on remnant factor, contrastive regularization, and meta-genes vs. individual genes. The results show the improvement of different designing components scWEAVE performance, consistent with their intended roles.

  2. We conducted comprehensive data similarity check between train and test data. We introduced the Striatum 4-species dataset (3 unseen species) as an additional evaluation dataset to address the reviewer's concern about the cross-species generalization ability. We updated the table to reflect the changes: scWEAVE still shows leading performance with the adjusted evaluation datasets.

  3. We included HVG+PCA as an important baseline. HVG+PCA shows a competitive performance compared to baseline scFMs, consistent with what the reviewer suggested, but scWEAVE still shows better performance compared to HVG+PCA.

We will also update the manuscript to include the results above.

Q1. The primary weakness is the lack of useful ablations to show which of the architectural / training improvements contribute to the performance...

A1. We agree with the reviewer's comment on ablation test. We conduct comprehensive ablation evaluation regarding the key design components of scWEAVE, including cell remnant factors, contrastive regularization, and gene compression module. We compare the performance of scWEAVE with the ablated models across all 4 tasks in the manuscript.

(1). Ablation on the remnant factor. We train an ablated scWEAVE with only the shared cell factor being used for the masked prediction (termed scWEAVE (w/o remnant)), and compare its performance with original scWEAVE. The results (Tables R1, R2, R3, R4) show that the remnant factor provides a consistent performance gain in data imputation, batch effect removal, and novel cell type detection. Ortholog retrieval remains similar, which is expected because the remnant factor doesn't directly affect the compression module.

(2). Ablation on the contrastive regularization (answer Q4). We train an ablated scWEAVE without contrastive regularization (termed scWEAVE (w/o contra)), and compare it with original scWEAVE. The results (Tables R1, R2, R3, R4) show that contrastive regularization improves scWEAVE's performance in data imputation, batch effect removal, and novel cell type detection. Ortholog retrieval also remains the same because contrastive regularization mainly affects the cell embedding.

(3). Ablation on the meta-genes (answer Q5). We train an ablated scWEAVE from scratch that uses top-512 highly expressed genes as input for each cell, instead of 512 meta-genes. For each species, the 512 top-expressing genes are selected by ranking the averaged expression of genes across all cells within the species. This also follows the reviewer's suggested setting in Q5. The ablated model is termed scWEAVE (w/o compression). We compare scWEAVE (w/o compression) with original scWEAVE, and report the results in Tables R1, R2, R3, and R4. The results show that the meta-gene representation of cells is significantly better than highly expressed genes in scWEAVE's performance on downstream tasks. We do not evaluate the ortholog retrieval on scWEAVE (w/o compression) because scWEAVE cannot retrieve orthologs without the gene compression module.

We only include average score in R1, R2 due to the space limit, but full score will be included in the manuscript. Bold and underlined values indicate the best and second-best scores, respectively

Table R1. Ablation of scWEAVE on gene expression imputation, measured using Pearson (PCC) and Spearman (SCC) correlation.

Average over 11 datasets

MetricscWEAVEscWEAVE (w/o remnant)scWEAVE (w/o compression)scWEAVE (w/o contra)
PCC ↑0.5640.5560.4370.560‾\underline{0.560}
SCC ↑0.4750.474‾\underline{0.474}0.3790.469

Table R2. Ablation of scWEAVE on batch effect removal and cell type identification.

Average over 11 datasets

MetricscWEAVEscWEAVE (w/o remnant)scWEAVE (w/o compression)scWEAVE (w/o contra)
ARI0.640.60‾\underline{0.60}0.530.27
ASW0.600.59‾\underline{0.59}0.600.51
ASW-B0.84‾\underline{0.84}0.850.790.75
GC0.940.940.88‾\underline{0.88}0.85

Table R3. Ablation of scWEAVE on novel cell type detection.

DatasetSpeciesscWEAVEscWEAVE (w/o remnant)scWEAVE (w/o compression)scWEAVE (w/o contra)
HippoPig1.000.33‾\underline{0.33}0.140.12
M. mulatta0.830.70‾\underline{0.70}0.160.62
MotorC. jacchus0.39‾\underline{0.39}0.580.090.38
Mouse1.001.000.440.59‾\underline{0.59}
AH-OutflowM. mulatta0.320.320.370.33‾\underline{0.33}
M. fascicularis0.43‾\underline{0.43}0.350.280.63
Pig0.250.160.24‾\underline{0.24}0.21
Mouse0.590.450.60‾\underline{0.60}0.74
Average0.600.49‾\underline{0.49}0.290.45

Table R4. Ablation of scWEAVE on top-1 human ortholog retrieval accuracy (%).

DatasetSpeciesscWEAVEscWEAVE (w/o remnant)scWEAVE (w/o contra)
AH-OutflowPig81.181.581.1
M. fascicularis81.681.781.7
M. mulatta79.679.979.8
HippoPig81.782.181.6
M. mulatta82.683.182.7
MotorM. mulatta83.383.383.3
C. jacchus83.884.184.1
StriatumChimpanzee85.585.785.5
M. mulatta81.381.581.2
P. discolor78.278.578.2
Average81.982.181.9

Q2. On the two multi-species brain datasets, scWEAVE is not in the top two on ARI (Hippo 0.67 vs 0.72; Motor 0.26 vs 0.28–0.34) and shows the largest seed variance in the table, which weakens the paper's claim that this approach generalizes to cross species. The ablation studies would further help here to narrow down if any of the design choices are actually reducing the effectiveness of the approach on cross species generalization.

A2. We agree that ablation is necessary and have conducted ablation study to prove the effectiveness of each key model component in response A1. To better address the reviewer's concern on the multi-species generalization ability of scWEAVE, we further include a 4-species striatum dataset in our evaluation. The striatum dataset has 3 species unseen in the training data (Bat/P. discolor, Chimpanzee, and M. mulatta), posing a large challenge to the model generalization ability. We update the results of data imputation, batch effect removal, and ortholog retrieval to include the striatum dataset (striatum doesn't have novel cell type). We further incorporate the UMAP visualization of the striatum dataset cell embedding in Figure A4 of the manuscript. We also attach the updated scores of these tasks in Tables R5, R6, R7, R8 of the response. The figure cannot be attached in the response, but we will upload the updated manuscript in the final submission. scWEAVE leads in all tasks on the Striatum dataset, showing it generalizes well across 4 species. The UMAP visualization also shows a better distribution alignment of scWEAVE compared to the other methods.

In addition, the reviewer is right that scWEAVE's performance suffers especially on the Motor dataset. However, we notice that it is unusually low across all methods. We find that the labeling in Motor data is overly fine-grained (total of 23 classes), and some labels are incompatible across species. Clustering in this case is inaccurate, which affects the ARI scores. We re-run scWEAVE and the baselines after reorganizing the labels (10 classes), and update the batch effect removal scores on Motor dataset. The result is updated in the manuscript, and posted in Table R6. scWEAVE leads the baselines on evaluation of Motor dataset after label reorganization. In the Hippo dataset, even though scWEAVE does not rank the top among all methods, the methods scoring above scWEAVE in ARI do not have competitive batch mixing scores (ASW-B and GC). They separete cell type but do not fully remove the batch effect. This is also clearly reflected in the UMAP visualization (Figure A1): scWEAVE is the only method that aligns the cell distribution across batches.

Table R5. Imputation accuracy of scFMs, measured using Pearson correlation (PCC) and Spearman correlation (SCC). Bold and underlined values indicate the best and second-best scores, respectively.

DatasetscWEAVE PCC ↑scWEAVE SCC ↑scFoundation PCC ↑scFoundation SCC ↑scGPT PCC ↑scGPT SCC ↑scMulan PCC ↑scMulan SCC ↑
Immune0.7100.487‾\underline{0.487}0.4340.4210.704‾\underline{0.704}0.5550.4820.313
Pancreas0.484‾\underline{0.484}0.4590.4140.4080.4980.422‾\underline{0.422}0.2800.154
Covid-190.6240.5000.3810.4370.5760.497‾\underline{0.497}0.597‾\underline{0.597}0.375
PBMC0.6580.4030.4930.452‾\underline{0.452}0.657‾\underline{0.657}0.5760.5550.352
Dendritic0.6040.5200.4620.4340.599‾\underline{0.599}0.490‾\underline{0.490}0.5910.337
Cortex-3B0.4370.4250.2270.231‾\underline{0.231}0.248‾\underline{0.248}0.2160.2060.116
Retina0.5340.5100.2860.2760.357‾\underline{0.357}0.309‾\underline{0.309}0.2840.054
Hippo0.4860.4290.3190.2990.343‾\underline{0.343}0.331‾\underline{0.331}0.2420.140
Motor0.6420.6020.3560.3780.502‾\underline{0.502}0.434‾\underline{0.434}0.3620.205
AH-Outflow0.5230.4420.3520.3180.4790.393‾\underline{0.393}0.511‾\underline{0.511}0.292
Striatum0.5010.4470.3530.3450.378‾\underline{0.378}0.364‾\underline{0.364}0.3510.244

Comments on the paper

2 comments

Hi reviewers,

Thank you for your reviewing service. The authors have provided fairly extensive rebuttals. Please confirm that you have read them, state which of your concerns are or are not addressed, and whether that changes your recommendations.

AuthorsOpenReview ↗

Rebuttal Summary

We sincerely thank the reviewers for their thoughtful and constructive reviews, and we appreciate their suggestions for improving our manuscript. The reviewers' major concerns relate to the insufficiency of the experiments in the manuscript, including:

  • Insufficient ablation studies on each design component.
  • Missing important baseline methods.
  • The lack of a comprehensive audit showing that the training and test data are sufficiently independent.
  • The lack of a test on perturbation prediction to show that scWEAVE retains gene-level resolution.
  • The lack of an evaluation on continuous cell populations to show that scWEAVE preserves continuous developmental structure.

During the rebuttal period, we conducted additional experiments to address these concerns:

  • We conducted a comprehensive ablation study on the key design components, including the remnant factor, contrastive regularization, meta-genes vs. individual genes, GO information, and the MEP designs. The results show that each component improves scWEAVE's performance, consistent with its intended role.
  • We added HVG+PCA as a baseline for batch effect removal and novel cell type detection. We added GEARS, scGPT, and the Linear Predictor as baselines for perturbation prediction.
  • We conducted comprehensive data similarity check between train and test data. We introduced the Striatum 4-species dataset (3 unseen species) as an additional evaluation dataset to address the reviewer's concern about the cross-species generalization ability. scWEAVE still shows leading performance with the adjusted evaluation datasets.
  • We evaluated scWEAVE on the perturbation prediction task and compared it with SOTA perturbation prediction methods. The comparison shows that scWEAVE outperforms both scGPT and the Linear Predictor, and ties with GEARS.
  • We evaluated scWEAVE on a dataset with a continuous cell trajectory. scWEAVE preserves the trajectory better after contrastive regularization. Contrastive regularization helps separate cells at different stages, while the main objective, masked prediction, keeps cells along the trajectory from being split into distinct clusters.

These additional experiments address reviewers major concerns, and support our claim within the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.