- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
this paper propose a biological foundation model to reduce computation cost and generalize across species. its main innovation is the meta-gene, which bulids gene similarity graphs using the ESM2 embeddings and GO embeddings, and then performs soft-assignment and maps genes to meta-genes. in the way the gene sequences are shorten. the framework then trains the model with masked expression prediction task and contrastive regularizatoin. experiments on datasets covering multiple species demonstrate the effectiveness and efficiency of the proposed method.
Critical strengths
S1: compressing genes to meta-genes is quite inspiring. such a methodology sounds reasonable but never found in previous literatures on bioinformatics as far as i know. this could significantly reduce computation cost, which is especially critical given the wide application on gene expression data of transformer models, whose complexit is O(n^2) over the sequence length, and preserve the functional characteristics of the genes.
S2: the paper proposes a very promising method to eliminate or alleviate batch effect, i.e. through contrative learning via the cell type signal. moreover, it successfully addresss the problem of inconsistent naming conventions by using multi-granular labels.
Critical weaknesses
W1: introducing too much functions in the MEP step add the complexity of the framework. this is related to Q2 but different. in Q2 i question the evidence of gain, but it makes the method delicated and sensitive nevertheless. it is doubtable if this design could achieve consistent gain under varied scenarios.
W2: the authors claims a multi-species solution. however, in the methodology it does not explain how the designs overcome gene inconsistency between speicies.
Questions for authors
Q1: i am not sure about the purpose of the denoising autoencoder in Figure 1 (b). it is possible to get gene embeddings from the ESM2 model and GO annotations, which can then be concatted or added and used for graph construction in the next step. why do you introduce this additional embedding step? what is the motivation and purpose, and what is the performance gain and practical benefit?
Q2: how is x_ig recovered from x_img in the MEP step? there is no specification in the paper. the expression is the product of x^bl and x^context. what is its advantage against direct prediction? and the multiplication operation (instead of summation or function-based operation) reasonable? need theoretical justification or empirical validation.
Q3: how many genes do your dataset cover? are they all protein coding genes? there is around 20000 protein coding genes but far more non-coding genes. the latter is often neglected in current research. one of the constraint is the exceesive sequence length. can your method help with this problem? the ESM2 embeddings might be unavailable for these genes since they do not transcribe proteins. do you have and solutions?
6 comments
Q1. introducing too much functions in the MEP step add the complexity of the framework. this is related to Q2 but different. in Q2 i question the evidence of gain, but it makes the method delicated and sensitive nevertheless. it is doubtable if this design could achieve consistent gain under varied scenarios.
A1. We agree with the reviewer that a comprehensive test is needed to validate the effectiveness of each additional function/design in MEP step, and to show their robustness across various application scenarios.
The additional designs of MEP step mainly include: (1) decomposing the predicted expression into the product of x^bl and x^context; (2) including a binary detection head that predicts binary (zero vs. non-zero) gene expression as an auxiliary objective to the main masked prediction objective.
In the following, we explain the motivation for introducing these two components, and conduct ablation tests to evaluate their effectiveness.
Decomposition of expression (jointly addressing Q4): We model the predicted expression as the product of x^bl (the output of the baseline head) and x^context, instead of predicting x directly, to better capture cell-to-cell variation. The baseline head predicts x^bl, the background expression of each gene. x^bl depends only on the gene identity and is independent of cells, whereas x^context predicts the cell-specific deviation of gene expression from the background x^bl. This decomposition introduces an additional inductive bias that encourages the predictor to better explore cell-to-cell variation, and it improves rather than destabilizes the training of scWEAVE.
To evaluate the effectiveness of this design, we conducted an ablation study focusing only on the baseline head. We trained a vanilla scWEAVE with the baseline head (termed scWEAVE) and scWEAVE without the baseline head (no decomposition, termed scWEAVE (w/o decomposition)). We evaluate how well each model learns cell-to-cell variation through the masked prediction objective, which is the main training objective where the baseline head directly affect. To better measure the recovery of cell-to-cell variation in masked prediction, we introduce an additional metric, the residual Pearson correlation (Residual PCC). Compared with the Pearson correlation (PCC), Residual PCC removes the global expression level of each gene, and therefore focuses only on whether scWEAVE recovers cell-specific gene expression variation:
Residual PCC = PCC( prediction − gene mean , observed − gene mean )
To ensure that the result is robust across various scenarios, we evaluated the two models on the 11 evaluation datasets used in the manuscript. The results are shown in Table R1. Averaged over the 11 datasets, scWEAVE outperforms scWEAVE (w/o decomposition) on Pearson correlation (PCC), Spearman correlation (SCC), Residual PCC, and negative log-likelihood (NLL, the primary training loss). The gain especially in Residual PCC shows that the use of baseline head improves scWEAVE's capability of capturing cell-to-cell variation.
Use of the binary detection head: We first clarify that the binary detection head and its training loss (the binary detection loss) serve only as an auxiliary objective that improves the training of scWEAVE on real-valued gene expression prediction. The head is not used for any downstream application.
Single-cell gene expression is extremely sparse, and most masked entries are zero. Trained with the negative binomial loss alone, the predictor receives a learning signal dominated by zero entries and only a weak signal on which genes are actually expressed in a given cell. The detection head adds a class-balanced binary objective (whether each masked gene is expressed), computed from the same hidden representation that the expression predictor uses. This forces the shared representation to distinguish expressed from non-expressed genes in a cell-specific way, which in turn improves the real-valued expression prediction.
We conducted an ablation study to evaluate the effectiveness of the detection head. We trained a vanilla scWEAVE with the detection head (termed scWEAVE) and the ablated scWEAVE without the detection head (termed scWEAVE (w/o detection head)). We evaluate the effectiveness of the detection head through the masked prediction objective, which the detection head is designed to improve. As before, we evaluate both models using Pearson correlation (PCC), Spearman correlation (SCC), Residual Pearson correlation (Residual PCC), and negative log-likelihood (NLL, the primary training loss). The results (Table R2) show that scWEAVE with the detection head has a large performance gain over scWEAVE without the detection head across all evaluation metrics and various test datasets, shows the effectiveness and robustness of detection head design in scWEAVE.
Table R1. Ablation of the decomposition on gene expression imputation, measured using Pearson (PCC) and Spearman (SCC) correlation, residual Pearson correlation (Residual PCC), and negative binomial negative log-likelihood of the observed counts (NLL).
(a) Average over 11 datasets
| Metric | scWEAVE | scWEAVE (w/o decomposition) |
|---|---|---|
| PCC ↑ | 0.605 | 0.604 |
| SCC ↑ | 0.479 | 0.478 |
| Residual PCC ↑ | 0.138 | 0.133 |
| NLL ↓ | 1.001 | 1.008 |
(b) Pearson correlation (PCC ↑)
| Dataset | scWEAVE | scWEAVE (w/o decomposition) |
|---|---|---|
| Immune | 0.732 | 0.733 |
| Pancreas | 0.529 | 0.535 |
| Covid-19 | 0.670 | 0.669 |
| PBMC | 0.711 | 0.713 |
| Dendritic | 0.660 | 0.655 |
| Cortex-3B | 0.500 | 0.493 |
| Retina | 0.575 | 0.572 |
| Hippo | 0.505 | 0.506 |
| Motor | 0.668 | 0.667 |
| AH-Outflow | 0.584 | 0.583 |
| Striatum | 0.518 | 0.518 |
(c) Spearman correlation (SCC ↑)
| Dataset | scWEAVE | scWEAVE (w/o decomposition) |
|---|---|---|
| Immune | 0.475 | 0.473 |
| Pancreas | 0.467 | 0.471 |
| Covid-19 | 0.497 | 0.495 |
| PBMC | 0.399 | 0.398 |
| Dendritic | 0.510 | 0.506 |
| Cortex-3B | 0.465 | 0.461 |
| Retina | 0.516 | 0.515 |
| Hippo | 0.431 | 0.431 |
| Motor | 0.612 | 0.613 |
| AH-Outflow | 0.450 | 0.449 |
| Striatum | 0.448 | 0.449 |
(d) Residual Pearson correlation (Residual PCC ↑)
| Dataset | scWEAVE | scWEAVE (w/o decomposition) |
|---|---|---|
| Immune | 0.167 | 0.162 |
| Pancreas | 0.076 | 0.070 |
| Covid-19 | 0.220 | 0.219 |
| PBMC | 0.177 | 0.172 |
| Dendritic | 0.047 | 0.041 |
| Cortex-3B | 0.178 | 0.170 |
| Retina | 0.190 | 0.188 |
| Hippo | 0.148 | 0.144 |
| Motor | 0.093 | 0.086 |
| AH-Outflow | 0.083 | 0.080 |
| Striatum | 0.140 | 0.133 |
(e) Negative log-likelihood (NLL ↓)
| Dataset | scWEAVE | scWEAVE (w/o decomposition) |
|---|---|---|
| Immune | 0.601 | 0.602 |
| Pancreas | 1.540 | 1.540 |
| Covid-19 | 0.544 | 0.545 |
| PBMC | 0.287 | 0.288 |
| Dendritic | 3.321 | 3.358 |
| Cortex-3B | 0.718 | 0.720 |
| Retina | 0.709 | 0.710 |
| Hippo | 0.641 | 0.650 |
| Motor | 1.252 | 1.268 |
| AH-Outflow | 0.758 | 0.761 |
| Striatum | 0.640 | 0.647 |
Table R2. Ablation of the detection head on gene expression imputation, measured using Pearson (PCC) and Spearman (SCC) correlation, residual Pearson correlation (Residual PCC), and negative binomial negative log-likelihood of the observed counts (NLL).
(a) Average over 11 datasets
| Metric | scWEAVE | scWEAVE (w/o detection head) |
|---|---|---|
| PCC ↑ | 0.605 | 0.598 |
| SCC ↑ | 0.479 | 0.468 |
| Residual PCC ↑ | 0.138 | 0.119 |
| NLL ↓ | 1.001 | 1.010 |
(b) Pearson correlation (PCC ↑)
| Dataset | scWEAVE | scWEAVE (w/o detection head) |
|---|---|---|
| Immune | 0.732 | 0.723 |
| Pancreas | 0.529 | 0.519 |
| Covid-19 | 0.670 | 0.656 |
| PBMC | 0.711 | 0.704 |
| Dendritic | 0.660 | 0.647 |
| Cortex-3B | 0.500 | 0.488 |
| Retina | 0.575 | 0.570 |
| Hippo | 0.505 | 0.508 |
| Motor | 0.668 | 0.667 |
| AH-Outflow | 0.584 | 0.573 |
| Striatum | 0.518 | 0.521 |
(c) Spearman correlation (SCC ↑)
| Dataset | scWEAVE | scWEAVE (w/o detection head) |
|---|---|---|
| Immune | 0.475 | 0.457 |
| Pancreas | 0.467 | 0.461 |
| Covid-19 | 0.497 | 0.478 |
| PBMC | 0.399 | 0.387 |
| Dendritic | 0.510 | 0.485 |
| Cortex-3B | 0.465 | 0.451 |
| Retina | 0.516 | 0.508 |
| Hippo | 0.431 | 0.430 |
| Motor | 0.612 | 0.606 |
| AH-Outflow | 0.450 | 0.440 |
| Striatum | 0.448 | 0.447 |
(d) Residual Pearson correlation (Residual PCC ↑)
| Dataset | scWEAVE | scWEAVE (w/o detection head) |
|---|---|---|
| Immune | 0.167 | 0.141 |
| Pancreas | 0.076 | 0.065 |
| Covid-19 | 0.220 | 0.193 |
| PBMC | 0.177 | 0.145 |
| Dendritic | 0.047 | 0.039 |
| Cortex-3B | 0.178 | 0.150 |
| Retina | 0.190 | 0.173 |
| Hippo | 0.148 | 0.139 |
| Motor | 0.093 | 0.082 |
| AH-Outflow | 0.083 | 0.059 |
| Striatum | 0.140 | 0.127 |
(e) Negative log-likelihood (NLL ↓)
| Dataset | scWEAVE | scWEAVE (w/o detection head) |
|---|---|---|
| Immune | 0.601 | 0.608 |
| Pancreas | 1.540 | 1.540 |
| Covid-19 | 0.544 | 0.551 |
| PBMC | 0.287 | 0.293 |
| Dendritic | 3.321 | 3.374 |
| Cortex-3B | 0.718 | 0.724 |
| Retina | 0.709 | 0.706 |
| Hippo | 0.641 | 0.643 |
| Motor | 1.252 | 1.263 |
| AH-Outflow | 0.758 | 0.761 |
| Striatum | 0.640 | 0.642 |
Summary: We thank the reviewer for the constructive feedback. Following the reviewer's suggestions, we conducted additional tests on scWEAVE. We highlight the following results that address the reviewer's major concerns:
We conducted ablation studies on the designed components in the MEP step. We tested the necessity of using product of x^bl and x^context for expression prediction, and the necessity of using the binary detection head during training (Q1 and Q4). We trained ablated scWEAVE that predicts gene expression directly (x = x^context, remove x^bl) and ablated scWEAVE removed of the detection head, and compared the two ablated models with the original scWEAVE across 11 test datasets to make sure that the results are robust across various scenarios (Q1). scWEAVE shows consistent improvement with the inclusion of the detection head and of x^bl.
We provided a detailed explanation on the motivation of MEP step designs (Q1), and on how scWEAVE generalizes across the genes of different species (Q2). We explained why we use the denoising autoencoder before gene compression (Q3), and provided a solution for future adaptation to non-protein-coding genes (Q5).