acceptodds
Under review as a conference paper at ICLR 2027

GREmLN: A Cellular Graph Structure Aware Transcriptomics Foundation Model

Abstract

The ever-increasing availability of large-scale single-cell profiles presents an opportunity to develop foundation models to capture cell properties and behavior. However, standard language models such as transformers benefit from sequentially structured data with well defined absolute or relative positional relationships, while single cell RNA data have orderless gene features. Molecular-interaction graphs, such as protein-protein interaction (PPI) networks or gene regulatory networks (GRN), offer graph structure-based models that effectively encode both non-local gene token dependencies, as well as potential causal relationships. We introduce GREmLN (Gene Regulatory Embedding-based Large Neural model), a foundation model that leverages graph signal processing to embed gene token graph structure directly within its attention mechanism, producing biologically informed single cell specific gene embeddings. Our model faithfully captures transcriptomic landscapes and achieves superior performance relative to state-of-the-art baselines on cell type annotation, graph structure understanding, and fine-tuned reverse perturbation prediction tasks. It offers a unified and interpretable framework for learning high-capacity foundational representations that capture complex long-range regulatory dependencies from high-dimensional single-cell transcriptomic data. The quality of network inference is not a focus of this study. Our framework applies to all graph-structured priors.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Computational Biology, Foundation Model, Graph Deep Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hpuCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GREmLN, a foundation model for single-cell transcriptomics that uses gene interaction graphs to guide attention. The main idea is to apply a graph diffusion kernel to the query embeddings. The passthrough method preserves full-graph kernel entries when selecting genes, and a Chebyshev approximation reduces the computational cost. The authors pre-train the model on 11 million cell profiles. They evaluate cell type annotation, masked-edge recovery, and reverse perturbation prediction, reporting improvements over the tested foundation models with a 14.7M-parameter model.

Critical strengths
  1. Using molecular interaction graphs is a sensible choice for this problem. Diffusing the queries incorporates these relationships while retaining the original key and value representations. I also find the passthrough idea useful: selecting expressed genes should not necessarily discard paths through genes that were omitted (Sections 3.2-3.3, pp. 4-5).
  2. The frozen-embedding results are the strongest empirical part of the paper. In Table 3, GREmLN reaches 0.435 perturbation classification accuracy, compared with 0.162 for Geneformer and 0.281 for lasso logistic regression. Table 1 also shows an improvement in immune-cell macro F1 over scGPT, from 0.924 to 0.939. The tests with different graph sources and model depths are useful additions (pp. 7-9).
Critical weaknesses
  1. My main concern is that the equations do not give a consistent definition of GDKA. Section 3.2 (p. 4) treats positive semidefiniteness as sufficient for an orthogonal projection, whereas exp(-βL) is generally not idempotent. The attention formula uses ({\varPhi }{L}Q)K^{T}, but the pairwise score below it is {q{i}}^{T}{{\varPhi }{L}}^{1/2}k{j}. Since q_{i} and k_{j} are in {\mathbb{R}}^{d} while {\varPhi }{L} is in {\mathbb{R}}^{G\ \times \ G}, the pairwise score is dimensionally incompatible with the stated definitions. It is not equivalent to the preceding attention formula. There is a similar problem with the Chebyshev approximation. Appendix A.3 maps the spectrum to [-1,1], yet the recurrence uses L instead of L - I. The sum in Section 3.5 also omits the constant term, which should be \frac{c{0}Q}{2} under the stated coefficient convention. These are errors in the written formulation of the central operator. What remains uncertain is whether the implementation contains the same errors, since code is not included in the submission (p. 10).
  2. I am not yet convinced that the experiments establish the benefit of GDKA itself. For cell type annotation, the graph depends on a k-NN posterior learned from labeled training cells (Eqs. 6-8, pp. 6-7). This does not establish test-label leakage, but the contribution of that supervision needs to be measured. I would like to see k-NN-only and graph-only controls, along with the same backbone without diffusion or with a simpler use of the same graph. The zero-shot claim also conflicts with the described evaluation protocol. Section 4.2 explicitly trains a linear classifier on cell representations using cross-entropy and cell-type labels (p. 7). As described, this is supervised linear probing / representation transfer, not zero-shot annotation. Excluding cell types from foundation-model pre-training does not make downstream evaluation with their labels zero-shot. If Table 1 uses a separate protocol without using held-out-type labels for training or model selection, that protocol must be specified; none is described. For reverse perturbation prediction, only GREmLN is fine-tuned, so that comparison does not support superiority over fine-tuned baselines. Table 3 provides no uncertainty estimates or separate Norman results, despite describing that dataset in Appendix A.8. Finally, its random Top-10 AUC is listed as 0.50. Following the caption, the uniform-random expectation for 20 candidates is \mathrm{mean}(\frac{k}{20}), for k = 1,…,10, which is 0.275. This discrepancy needs to be resolved before interpreting the reported values.
Questions for authors
  1. Could you show the equations actually used in the implementation, including tensor dimensions and the complete Chebyshev expansion? A small-graph check against exact exp(-βL)Q would help. Do these corrections change any results? Please also clarify how the sparse recurrence in Appendix A.4 relates to the cached kernels in Appendix A.6.
  2. Which cells and labels are used for GRN inference, the k-NN graph selector, and the final classifier in Table 1? Please identify the classes used to train the linear classifier with cross-entropy. For the held-out types, describe a separate protocol that excludes their labels from training and model selection, or relabel the results as linear probing / representation transfer. Please specify the splits and compare k-NN alone and the same backbone without diffusion under the same data and supervision.
  3. Are the 15% held-out edges removed before computing the evaluation kernels and cached embeddings? Please explain negative sampling and the edge splits used to train and evaluate the decoder. A graph-only predictor using the same incomplete graph would help show what the expression embeddings add.
  4. How was Top-10 AUC calculated in Table 3, and why is its chance value 0.50? Please identify the dataset and report Adamson and Norman results separately, with uncertainty. Were the 20 perturbations selected from training data only, and how were cells split? A comparable fine-tuning baseline is needed for the fine-tuned superiority claim; otherwise, please limit that claim to the frozen comparison.
  5. Could you provide code, graph-construction settings, dataset and split identifiers, and downstream training details? For Tables 1-2, please specify the confidence level, interval calculation, and resampling unit. For Table 3, I would like to see repeated-run results to assess the stability of the gains.
4 comments
AuthorsOpenReview ↗

Response (1/2)

We thank the reviewer for an exceptionally careful and detailed review. The reviewer correctly identified several errors in the written formulation, and we address each point below. Shared items are covered in our General Response. We will make all corrections listed here in the revised paper.

  1. Consistent definition of GDKA The reviewer is right on all three points about the written formulation. The equations below are the ones used in the implementation. Tensor dimensions and attention. For a batch of BB cells with N≤4096N \le 4096 selected genes, hh heads and head dimension dhd_h, we have Q,K,V∈RB×h×N×dhQ, K, V \in \mathbb{R}^{B \times h \times N \times d_h} and a sparse sub-KGM Φ∈RB×N×N\Phi \in \mathbb{R}^{B \times N \times N}. The same Φ\Phi is shared across heads and layers. Diffusion acts on the gene dimension: Q~=ΦQ,Attn(Q,K,V)=softmax(Q~K⊤dh)V,Sij=∑m=1NΦimqm⊤kj\tilde{Q} = \Phi Q, \qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\left(\frac{\tilde{Q} K^\top}{\sqrt{d_h}}\right) V, \qquad S_{ij} = \sum_{m=1}^{N} \Phi_{im} q_m^\top k_j The expression qi⊤ΦL1/2kjq_i^\top \Phi_L^{1/2} k_j was incorrect and will be removed.

Projection claim. e−βLe^{-\beta L} has eigenvalues e−βλℓ∈(0,1]e^{-\beta\lambda_\ell} \in (0, 1] and is not idempotent, so it is not an orthogonal projection. It is symmetric positive definite, and is therefore a valid reproducing kernel on the gene set: the heat kernel of Kondor & Lafferty (2002). We will correct the statement. No part of the method relies on the projection claim.

Complete Chebyshev expansion. With L~=L−I\tilde{L} = L - I, whose spectrum lies in [−1,1][-1, 1], and the coefficients of Appendix A.3, we use a degree-KK expansion with K=8K = 8, i.e. nine terms including T0T_0:

e−βL≈c02I+∑k=1KckTk(L~),T0=I,T1=L~,Tk=2L~Tk−1−Tk−2e^{-\beta L} \approx \frac{c_0}{2} I + \sum_{k=1}^{K} c_k T_k(\tilde{L}), \qquad T_0 = I, \quad T_1 = \tilde{L}, \quad T_k = 2\tilde{L} T_{k-1} - T_{k-2}

Our implementation includes both the constant term and the L−IL - I rescaling; the omissions were transcription errors in the paper, so no results change.

Small-graph check. We verified the approximation on a GRN derived from the largest cell-type network in our corpus (neurons; 6.78M edges), keeping the 500 highest-degree genes (degree counted over the full network, as either regulator or target). On the resulting 500-node graph, we compared the Chebyshev approximation with the exact e−βLe^{-\beta L} (β = 0.1), computed by eigendecomposition, across expansion orders:

Chebyshev order KRelative Frobenius error
21.20 × 10⁻⁵
43.90 × 10⁻⁸
88.64 × 10⁻⁸
169.15 × 10⁻⁸
323.97 × 10⁻⁷

The approximation converges by K = 4. Beyond that point, the error stays at float32 machine precision, and its slight growth at larger K reflects rounding accumulated in the three-term recurrence rather than truncation error. At K = 8, the order we use, the maximum absolute error is 1.19 × 10⁻⁷, so the kernel is effectively exact.

Appendix A.4 vs A.6. In the implementation used for our experiments, we evaluate the matrix recurrence once per cell type per training job to construct a sparse master kernel Gram matrix (KGM) over the full gene set. For each cell, we extract the principal submatrix indexed by its input genes. We refer to this as the passthrough method. KGMs are not cached to disk. We report construction and extraction costs in the General Response.

  1. Controlled ablation of GDKA Please see our General Response for the controlled GDKA ablation.

  2. Cell-type annotation (Table 1) GRN inference is performed using train cells from each cell type. For example, the NK cells’ ARACNe graph is based on NK Cells within the training set. Likewise, the KNN classifier uses train cells to guess the cell type of each test cell, and then uses the graph of the guessed cell type during diffusion in the GDKA. All immune cell classes are used to fit the immune cell linear classifier, and all non-immune cell types are used to fit the non-immune cell linear classifier. We would like to further clarify that “held-out cell types” refers to cell types held out during pre-training. Such cells and their cell type labels are used to train the linear classifier whose purpose is to discriminate them. We apologize for this ambiguity and will rename this subset of data as “Cell types held-out during pre-training”.

AuthorsOpenReview ↗

Response (2/2)

  1. Edge recovery (Table 2) For each held-out graph, we randomly remove 15% of the true edges; these masked edges are the positives. We then add an equal number of false edges, sampled uniformly at random from gene pairs not connected in the full graph; these are the negatives. The model receives the corrupted graph (the remaining 85% of true edges plus the false edges), so it cannot recover the masked edges, or reject the false ones, from the input graph structure alone. Kernels and embeddings are computed from this corrupted graph, so the masked edges are never seen by the encoder.

  2. Reverse perturbation (Table 3)

Chance level. The reviewer is correct. For 20 candidates, the chance Top-10 AUC is 110∑k=110k20=0.275\frac{1}{10}\sum_{k=1}^{10} \frac{k}{20} = 0.275, not 0.50. We will correct Table 3. All reported values are unaffected; only the chance row was wrong.

Datasets and uncertainty. Table 3 reports results on the Adamson et al. dataset only, and we will state this in the caption. Although Section 4.1 and Appendix A.8 describe the Norman et al. dataset, the submission did not include results on it; we apologize for the confusion. We now report Norman results on held-out dual-perturbation combinations, with repeated runs, in our General Response, and will add them to the revised paper. Within the discussion period, we were not able to repeat the benchmarking for every model in Table 3. In the revised paper, we will report mean ± standard error for all models over repeated downstream runs (multiple seeds and data splits), using the public pre-trained checkpoints for the baseline foundation models.

Perturbation selection and splits. The 20 perturbations were selected as those with the strongest transcriptional effect relative to control cells. Cells from these perturbations were then split into training, validation and test sets, and the same splits were used for all models.

Fine-tuning claim. We agree with the reviewer. Since the foundation-model baselines were not fine-tuned, we will limit the claim of superiority over baselines to the frozen-embedding comparison. The fine-tuned results in our General Response compare GDKA only against the no-diffusion Transformer.

  1. Reproducibility We are unable to release the code at this time. To support reproducibility, the revised paper will give the full implementation details described in this response, including the exact GDKA equations with tensor dimensions, the complete Chebyshev expansion, the kernel computation pipeline, graph-construction settings and dataset splits. In Table 1, ± denotes the standard error across 3 random train/test splits. In Table 2, ± denotes the standard error of AUROC across 3 cross-validation folds; AP is computed on the pooled predictions. Repeated-run results for perturbation prediction are reported in the General Response. We will state these definitions explicitly in the table captions.

We thank the reviewer again. Their detailed reading has substantially improved the correctness of the paper.

Thank you for the detailed response and additional experiments. After considering the reviewers' comments and the rebuttal, I will maintain my score.

We thank the reviewer for their careful consideration of our response and for their detailed feedback throughout. We will incorporate their suggestions in the revised version of the paper.

Reviewer_jiDkOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents GREmLN, a foundation model for single-cell RNA sequencing representation learning using biological graph structure priors. The authors introduce Graph Diffusion Kernel Attention, which embeds structural relationships from Gene Regulatory Networks (GRN) and Protein-Protein Interaction (PPI) networks into attention mechanism of transformers. Furthermore, the paper introduces Passthrough method to maintain indirect connectivity among genes, and utilizes Chebyshev Polynomial Approximation to improve extend computation of graph diffusion. The authors pre-train their model on 11 million single-cell RNA-seq data, and demonstrate better performance compared to existing single-cell foundation models with excellent parameter efficiency.

Critical strengths
  1. The paper addresses a biologically meaningful research problem. It stresses that the gene tokens in single-cell transcriptomes are not naturally ordered but have a graph topological relationship in GRN and PPI networks. This work proposes to embed such relationship with graph diffusion priors in Transformers attention mechanism. The conceptualization is very interesting and has great potential value.

  2. The paper proposes a novel model architecture that applies graph diffusion kernel directly on Query embeddings in the self-attention. Besides, the Passthrough attempt maintains the complete structural information in GRN and PPI networks when handling different kinds of cells, providing a meaningful design for the fusion of graph structure and attention mechanism.

  3. The experiments cover several meaningful downstream tasks and show excellent performance compared to many prior foundation models. Moreover, the proposed model is only of 14.7M parameters, showing impressive parameter efficiency in representation learning.

Critical weaknesses
  1. The paper lacks controlled ablation studies to prove the independent contribution of the proposed Graph Diffusion Kernel Attention (GDKA). Despite presenting GDKA as the core methodological contribution of the paper, the proposed GREmLN differs significantly to scGPT, Geneformer and csFoundation in pretaining configuration and structural priors. It's unclear how much of the performance improvement comes from GDKA itself, but not the newly-introduced biological graph information. Especially, the paper lacks a No-Graph/No-Diffusion ablation under similar architecture, nor simpler established solutions including Graph Attention bias or GNN-based Feature Fusion. Similarly, the paper lacks an ablation study on the Passthrough method against direct computation of diffusion kernel on the induced subgraph.

  2. The experiment design in some downstream tasks is leading to unfair comparisons. For example, in the cell-type annotation task, the k-NN classifier is trained on a labeled training set, whereas the prediction probabilities are then used to compose the cell-type-specific GRNs. Such process introduces additional supervision signal that may also contribute to the performance on classification. We are not able to tell the exact contribution of GDKA.

  3. The paper lacks quantitative analysis on the effectiveness of graph structure priors and the graph diffusion mechanism. The paper uses diffusion rate β=0.1\beta=0.1, in which case the diffusion kernel is relatively close to an identity transformation and higher-order diffusion terms may not contribute that much. Thus, the conclusion on long-distance graph needs more direct experimental proof.

  4. Some mathematical equations and definitions in the paper are inconsistent and somewhat incorrect in my opinion. Please see my questions below.

Questions for authors
  1. In section 3.2, this work uses spectral decomposition of symmetric Laplacian L=UΛUTL=U \Lambda U^T to construct the diffusion kernel. However, the GRN can be a directed graph in most cases. How do authors handle this gap?

  2. In appendix A.3, the Chebyshev Expension should be ΦLQ≈c02Q+∑k=1KckTk(L~)Q\Phi_L Q\approx \frac{c_0}{2} Q+ \sum_{k=1}^{K} c_k T_k(\tilde L)Q, but then in section 3.5 the diffusion becomes ∑k=1KckTk(L)Q\sum_{k=1}^{K} c_k T_k(L)Q and the constant term is missing. Could the authors clarify the reason and the impact of dropping this constant term?

  3. In section 3.2 the authors write Sij=qi⊤ΦL1/2kjS_{ij}=q_i^\top\Phi_L^{1/2} k_j where $q_i, k_j \in \mathbb{R}^d, $ but ΦL∈RG×G\Phi_L \in \mathbb R^{G\times G}. There seems to be a dimension mismatch and the operation is invalid?

  4. In section 4.2, the loss computation maximizes probabilities on all the categories instead of optimizing the probability on the correct label only (as in standard cross-entropy). I believe this should be a typo, could the authors clarify?

  5. In section 3.2 the authors state that ΦL\Phi_L is "symmetric positive semidefinite and therefore constitutes the matrix representation of an orthogonal projection operator onto the RKHS". However, I am not convinced that this statement is mathematically true, since ΦL2=e−2βL≠e−βL\Phi_L^2 = e^{-2\beta L} \neq e^{-\beta L}? Could the authors clarity?

Overall I find this work very interesting and it has great potential. However, the missing ablation studies and those minor issues in mathematical formulations weaken the overall soundness of the work. I would be happy to reconsider my rating once the authors provide further justifications or corrections.

1 comment

We thank the reviewer for their careful and constructive review, and for the encouraging assessment of the problem and the architecture. We address their points below, and we will make all of the corrections described here in the revised paper.

0. Ablation study of GDKA

Please see our General Response for the controlled GDKA ablation and corrections to the pre-training description. We agree that comparisons with simpler uses of the same graph (graph attention bias, GNN-based feature fusion) and an ablation of the passthrough method against kernels recomputed on the induced subgraph would be informative. We were unable to complete these within the discussion period, and we will add them to the revised paper.

1. Directed GRNs and the symmetric Laplacian

ARACNe infers edges from mutual information, which is symmetric. Edge direction is assigned afterward from transcription-factor annotation. We symmetrize the adjacency matrix, A←max⁡(A,A⊤)A \leftarrow \max(A, A^\top), before computing LL. The PPI networks are undirected by construction. We will state this explicitly in Section 3.2. Extending GDKA to directed graphs is a natural direction for future work.

2. Chebyshev expansion and the constant term

The reviewer is correct, and the written formulation has two errors that we will fix. With the coefficient convention of Appendix A.3, the expansion is

e−βLQ≈c02Q+∑k=1KckTk(L~)Q,L~=L−Ie^{-\beta L} Q \approx \frac{c_0}{2} Q + \sum_{k=1}^{K} c_k T_k(\tilde{L}) Q, \qquad \tilde{L} = L - I

First, the c0/2c_0/2 term was omitted from Section 3.5. Second, the recurrence in Appendix A.3 must be applied to the rescaled Laplacian L~=L−I\tilde{L} = L - I, whose spectrum lies in [−1,1][-1, 1], not to LL.

3. Dimension mismatch in the pairwise score

The reviewer is correct. ΦL∈RG×G\Phi_L \in \mathbb{R}^{G \times G} acts on the gene (token) dimension, and Q,K∈RG×dQ, K \in \mathbb{R}^{G \times d}. The attention scores are S=ΦLQK⊤S = \Phi_L Q K^\top, which gives

Sij=∑m=1G(ΦL)imqm⊤kjS_{ij} = \sum_{m=1}^{G} (\Phi_L)_{im} q_m^\top k_j

The diffused query for gene ii is therefore a graph-weighted combination of the queries of related genes. The expression qi⊤ΦL1/2kjq_i^\top \Phi_L^{1/2} k_j was incorrect, and we will replace it with the equation above.

4. Cross-entropy loss in Section 4.2

This is a typo. The implementation uses standard cross-entropy on the true label only:

L(θ)=−∑i=1Nlog⁡p(yi∣hi,G^i,fθ)\mathcal{L}(\theta) = -\sum_{i=1}^{N} \log p\left(y_i \mid h_i, \hat{G}_i, f_\theta\right)

5. "Orthogonal projection onto the RKHS"

The reviewer is correct. ΦL=e−βL\Phi_L = e^{-\beta L} has eigenvalues e−βλℓ∈(0,1]e^{-\beta \lambda_\ell} \in (0, 1] and is not idempotent, so it is not an orthogonal projection. The correct statement is that ΦL\Phi_L is symmetric positive definite and therefore a valid reproducing kernel on the gene set: the heat (diffusion) kernel of Kondor & Lafferty (2002). This kernel defines an RKHS in which smoothness is measured with respect to the graph. We will replace the sentence accordingly. None of the method or results depend on the projection claim.

6. Diffusion rate β = 0.1 and the long-range claim

The reviewer is right that the diffusion is mild. Since L=I−A^L = I - \hat{A} with A^=D−1/2AD−1/2\hat{A} = D^{-1/2} A D^{-1/2},

e−βL=e−β(I+βA^+β22A^2+… )e^{-\beta L} = e^{-\beta}\left(I + \beta \hat{A} + \tfrac{\beta^2}{2}\hat{A}^2 + \dots\right)

so with β = 0.1, one-hop neighbors receive a weight of about 0.1 relative to self, and two-hop neighbors a weight of about 0.005. The kernel's eigenvalues lie in [e−0.2,1]≈[0.82,1][e^{-0.2}, 1] \approx [0.82, 1]. We will therefore soften the claims about long-range dependencies in the revised paper, and describe GDKA as a mild, mostly local graph smoothing of the queries. Notably, this mild diffusion is still enough to produce the gains in the ablation in our General Response. We will also include a sweep over the diffusion rate β in the revision.

7. k-NN supervision in cell-type annotation

The k-NN classifier is fitted only on training cells and their labels, and test-cell labels are never used to construct the graph.

We are grateful for the reviewer's careful reading of the formulation, which has made the paper more rigorous. In summary, we have (i) provided a controlled ablation showing that GDKA itself, with the backbone, data and supervision held fixed, improves OOD perturbation prediction by 28–54% relative; (ii) corrected the mathematical issues raised by the reviewer (the Chebyshev constant term and rescaling, the pairwise score, the cross-entropy loss and the RKHS statement); and (iii) clarified the handling of directed GRNs and tempered the long-range claims to match β = 0.1. Since the reviewer noted that the missing ablation and the issues in the formulation were the main factors limiting soundness, we hope these revisions address those concerns and that the reviewer will consider raising their score. We would be glad to answer any further questions.

Reviewer_yEJxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces GREmLN, a graph-aware single-cell RNA-sequencing foundation model that incorporates biological network priors into Transformer-based gene representation learning. Its core Graph Diffusion Kernel Attention mechanism applies a heat diffusion kernel derived from the normalized graph Laplacian to transform query representations while keeping keys and values unchanged, enabling the model to capture biologically informed relationships and long-range dependencies across unordered genes using gene regulatory networks or protein–protein interaction networks. To preserve global topology under gene subsetting and scale to large graphs, the authors employ a kernel submatrix passthrough strategy alongside a Chebyshev polynomial approximation. Pretrained on approximately 11 million single-cell transcriptomes, GREmLN outperforms evaluated baselines across cell type annotation, regulatory network structure prediction, and reverse perturbation prediction tasks, while demonstrating consistent scaling improvements with increased model depth.

Critical strengths

Well-Motivated Inductive Bias for Unordered Transcriptomes: Single-cell RNA-seq expression profiles lack the natural sequential structure found in language, making biological network information a well-motivated source of inductive bias for gene representation learning. By incorporating gene regulatory networks and protein–protein interaction networks, GREmLN provides a biologically grounded approach to modeling relationships among genes. Asymmetric Graph Diffusion Attention Design: Applying the diffusion kernel (\Phi_L(Q)=U\exp(-\beta\Lambda)U^\top Q) to queries while leaving keys and values untransformed is an interesting architectural choice. This design introduces graph-dependent contextual information into the attention mechanism without directly applying the same diffusion transformation to all attention components, offering a potentially useful balance between structural priors and flexible attention computation. Parameter Efficiency and Prior Flexibility: GREmLN uses 14.7M learnable parameters in its three-layer configuration, substantially fewer than the reported parameter counts of Geneformer, scGPT, and scFoundation. The experiments also investigate both context-specific inferred GRNs and curated PPI networks, demonstrating the architecture's applicability to different types of biological network priors. The reported results suggest that incorporating such priors can be beneficial for several evaluated downstream tasks.

Critical weaknesses

Insufficient Controlled Ablations to Isolate the Contribution of GDKA The paper lacks sufficient downstream performance comparisons against a controlled Transformer baseline without graph diffusion to isolate the contribution of the proposed Graph Diffusion Kernel Attention. Although Appendix A.6 mentions a same-sized Transformer without diffusion, its downstream performance is not reported in the main experimental tables. Consequently, it remains unclear whether the reported improvements arise specifically from GDKA or from other factors, such as the training objective, preprocessing pipeline, or use of biological network information. A controlled comparison under consistent training and evaluation settings would provide stronger evidence for the effectiveness of the proposed method. Ambiguity in the Implementation of the Graph Diffusion Kernel The paper does not sufficiently clarify the relationship between Chebyshev polynomial approximation, precomputation of the full-graph kernel Gram matrix, and the passthrough mechanism used to extract gene-specific submatrices. Although these operations are not inherently contradictory, the implementation details are insufficiently explicit to determine how they are combined in practice. In particular, it remains unclear how the kernel is approximated and cached, how gene subsets are handled during training, and what computational and memory costs are incurred by this procedure. Providing a more detailed description of the implementation pipeline and its associated resource requirements would improve the methodological clarity and reproducibility of the proposed approach.

Questions for authors

Contribution of GDKA. Appendix A.6 mentions a same-sized Transformer baseline without graph diffusion, but its downstream performance is not reported in the main experimental tables. Could the authors provide its results on the evaluated downstream tasks under the same training and evaluation settings as GREmLN? This comparison would help clarify whether the reported improvements can be attributed specifically to GDKA rather than other components of the model or training pipeline.

Implementation of the graph diffusion kernel. Could the authors clarify how Chebyshev polynomial approximation is used in conjunction with full-graph kernel Gram matrix precomputation and the passthrough mechanism for extracting gene-specific submatrices? In particular, is the kernel matrix approximated and cached during preprocessing, and how are gene subsets handled during training? Additional details on the associated computational and memory costs would help assess the implementation's reproducibility and scalability.

1 comment

We thank the reviewer for their careful reading and constructive comments. We address each concern below.

  1. Contribution of GDKA (ablation) Please see our General Response for the controlled GDKA ablation and corrections to the pre-training description.

  2. Implementation of the graph diffusion kernel We thank the reviewer for pointing out this ambiguity. Section 3.5 does not accurately describe the current pipeline, and this will be revised. The current implementation works as follows.

Master KGMs (computed once per training job). For each cell type, we compute one “master” kernel Gram matrix (KGM) over the full-genome GRN. These graphs can have several million edges, so we approximate the heat kernel with the Chebyshev expansion ∑kckTk(L)\sum_{k} c_k T_k(L) (Appendix A.3, K=8K = 8) instead of performing an expensive eigendecomposition. All master KGMs are computed at the start of each training job and held in memory. No KGMs are cached to disk. This also makes it cheap to experiment with different graphs and diffusion rates β\beta.

Per-cell subsetting (passthrough). For each cell, we select up to 4,096 genes: transcription factors and co-factors, plus highly variable genes for that cell type, after removing non-expressed genes. We then extract the corresponding principal submatrix of that cell type’s master KGM (Section 3.3). The resulting sub-KGM contains diffusion scores between the selected genes, computed on the full graph. By contrast, recomputing the kernel on the induced subgraph would discard all paths through unselected genes.

Forward pass. For each batch, the model receives a sparse sub-KGM of shape (B,N,N)(B, N, N). It computes ΦL(Q)=KGML Q\Phi_L(Q) = \mathrm{KGM}_L\, Q as a batched sparse–dense matrix multiplication. The forward pass costs the same as in the earlier subgraph-based implementation; only the values in the KGM differ.

Computational and memory costs.

  • Computing the master KGMs: about 2.5 hours for all 379 cell types on a single H100 GPU, once at the start of each training job.
  • Memory for the master KGMs: All master KGMs are under 5% dense, so we store them as sparse CSR matrices. With 19,245 genes, a master KGM has at most 19,245² × 0.05 ≈ 18.5M non-zero entries. Each entry takes 8 bytes (a float32 value and an int32 column index), so each master KGM needs at most about 148 MB. Across all 379 cell types, the total is at most about 56 GB.
  • Memory for the sub-KGMs: Sub-KGMs are passed to the forward pass as sparse COO tensors and are never converted to dense. If a sub-KGM is about as dense as its master KGM (5%), a cell with 4,096 genes has about 0.84M non-zero entries. Each entry takes 28 bytes (three int64 indices and a float32 value), so one sub-KGM needs about 23 MB. A batch of 256 cells needs about 6 GB, compared with about 17 GB if stored dense. The arithmetic checks out. Two suggestions:
  • Sub-KGM density. The selected genes, mostly transcription factors and highly variable genes, may be more densely connected than the genome as a whole, so 5% may underestimate their density. If you can measure the average density over a few batches, report that number instead of the assumption.
  • The Forward pass paragraph. Your draft called this a plain “batched matrix multiplication.” Since the sub-KGMs are sparse, I changed it to “sparse–dense” so it agrees with the memory bullet.

Earlier implementation. An earlier version of our pipeline precomputed a KGM for each cell from its induced subgraph and cached it to disk. This produced millions of files, slowed dataloading and lost information from the full graph structure.

Planned revisions. We will rewrite Section 3.5 to describe this pipeline. We will update Appendices A.2, A.4 and A.6 to match it, including the runtime and memory analysis.

We hope these additions address the reviewer’s concerns about soundness, and we would be grateful if the reviewer would consider updating their assessment.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers (1/2)

We thank all three reviewers for their thoughtful and constructive feedback. Several concerns were shared across reviews, so we address them here and respond to each reviewer's individual points in separate replies.

Contribution of GDKA (ablation)

All three reviewers (yEJx, jiDk, hpuC) asked for a controlled ablation isolating the contribution of GDKA, so we ran one. We compared GREmLN with a same-sized standard Transformer. The two models use identical pre-training data, objective, preprocessing and hyperparameters, and differ only in whether the query diffusion step ΦL(Q)\Phi_L(Q) is applied.

Task. We evaluate both models on a new and harder benchmark: out-of-distribution (OOD) reverse dual-perturbation prediction on the Norman et al. CRISPRa Perturb-seq dataset (Appendix A.8). The probe is trained on all single perturbations and a subset of dual perturbations. It is then evaluated on 10 held-out dual-perturbation combinations that it never saw during training. Because the model must recognize combinations it has never observed, this setting directly tests whether the learned representations generalize beyond the training distribution.

Multi-label inference. A single task head handles both single and dual perturbations. For each candidate gene i∈Gi \in G, where GG is [the set of candidate perturbation targets, e.g. the N genes targeted in the dataset], the model estimates the probability P(Yi=1)P(Y_i = 1) that gene ii was perturbed. The probability of a particular combination s⊆Gs \subseteq G is then given by a Bernoulli factorization: P(s)=P(⋂i∈sYi, ⋂j∉sYj‾)=∏i∈sP(Yi)∏j∉s(1−P(Yj))P(s) = P\left(\bigcap_{i \in s} Y_i,\ \bigcap_{j \notin s} \overline{Y_j}\right) = \prod_{i \in s} P(Y_i) \prod_{j \notin s} \bigl(1 - P(Y_j)\bigr) The predicted combination is the most likely one: s^=argmaxs∈CP(s)\hat{s} = \mathop{\mathrm{argmax}}_{s \in \mathcal{C}} P(s) where C={ s⊆G:1≤∣s∣≤2 }\mathcal{C} = \{\, s \subseteq G : 1 \le |s| \le 2 \,\} is the set of all possible 1-target and 2-target subsets of the full target set GG (∣C∣=210|\mathcal{C}| = 210). Combinations are ranked by P(s)P(s) to compute top-k accuracy.

Protocol. We pre-trained each model from four weight and dataloader initializations. Each initialization was then evaluated with two seeds, with both frozen embeddings and full fine-tuning, using identical downstream settings for both models. This gives up to 8 runs per setting. The candidate predictions are the 210 single-gene and two-gene combinations in C\mathcal{C}, ranked by P(s)P(s). We report three metrics:

  • Accuracy: the fraction of cells whose top-ranked combination is exactly correct. Chance is 1/210 ≈ 0.005.
  • Top-10 accuracy: the fraction of cells whose true combination is among the 10 highest-ranked candidates. Chance is 10/210 ≈ 0.048.
  • Top-10 AUC: as in Table 3, the mean top-k accuracy for k = 1, …, 10. Chance is ≈ 0.026.

Values are mean ± standard error across runs.

ModelNorman OOD AccuracyNorman OOD Top-10 AccuracyNorman OOD Top-10 AUC
Chance0.0050.0480.026
Transformer (no diffusion), frozen0.026 ± 0.0060.472 ± 0.0490.278 ± 0.037
GDKA (diffusion on GRN), frozen0.040 ± 0.0090.566 ± 0.0650.355 ± 0.050
Transformer (no diffusion), fine-tuned0.042 ± 0.0100.499 ± 0.0690.313 ± 0.051
GDKA (diffusion on GRN), fine-tuned0.057 ± 0.0080.619 ± 0.0230.403 ± 0.021

Adding query diffusion on the GRN improves all three metrics in both settings:

  • Frozen embeddings: accuracy rises from 0.026 to 0.040 (about 54% relative), Top-10 accuracy from 0.472 to 0.566 (about 20%), and Top-10 AUC from 0.278 to 0.355 (about 28%).
  • Fine-tuned: accuracy rises from 0.042 to 0.057 (about 36% relative), Top-10 accuracy from 0.499 to 0.619 (about 24%), and Top-10 AUC from 0.313 to 0.403 (about 29%).

The fine-tuned GDKA model also has the lowest variance across runs on both top-10 metrics. After fine-tuning, it places the true combination among its top 10 candidates for 62% of cells, compared with about 5% at chance.

Exact-match accuracy on unseen combinations is a strict criterion. By this measure, GDKA reaches about 8x chance with frozen embeddings and about 12x after fine-tuning, compared with about 5x and 9x for the Transformer without diffusion. Every other part of the model and training pipeline is held fixed, so these gains can be attributed to the GDKA mechanism. They also suggest that the graph prior is most useful when the model must generalize to unseen perturbation combinations.

AuthorsOpenReview ↗

General Response to All Reviewers (2/2)

Corrections to the description of pre-training

While preparing this response, we found two places where the paper's text was carried over from an earlier version and does not match the setup used for all reported results. We will correct both in the revision:

A. Pre-training corpus. All models were pre-trained on 24M single-cell profiles from CELLxGENE, covering 379 cell types, not 11M as stated in the abstract, Section 4 and Appendix A.2. The preprocessing pipeline is as described in Appendix A.2.

B. Objective. The masked-gene prediction head is trained with a mean squared error loss on the expression bin values, not the likelihood formulation in Section 3.4. For each masked gene ii in cell bb, the decoder dec\mathrm{dec} (a two-layer MLP, with no sigmoid) outputs logits over the K=100K = 100 expression bins. The prediction is the expected bin index under the softmax:

x^b,i=∑k=0K−1k⋅softmax(dec(Hb,i))k\hat{x}_ {b,i} = \sum_ {k=0}^{K-1} k \cdot \mathrm{softmax}\left( \mathrm{dec}(H_ {b,i}) \right)_ k

The loss is the mean squared error over all masked genes in a batch of BB cells:

L(θ)=1∑b=1B∣Mb∣∑b=1B∑i∈Mb(x^b,i−xb,ir)2\mathcal{L}(\theta) = \frac{1}{\sum_ {b=1}^{B} |M_ b|} \sum_ {b=1}^{B} \sum_ {i \in M_ b} \left( \hat{x}_ {b,i} - x^r_ {b,i} \right)^2

where Mb={i:mb,i=1}M_ b = \lbrace i : m_ {b,i} = 1 \rbrace is the set of masked genes in cell bb. Each gene is masked independently with probability 0.3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.