acceptodds
Under review as a conference paper at ICLR 2027

Inducing Functional Locality in Linear Attention for Whole-Transcriptome Generation

Abstract

Single-cell RNA-seq profiles are long, high-dimensional and sparse, imposing severe computational bottlenecks in generative modeling at the whole-transcriptome scale. Existing methods predominantly rely on Performer for linear approximation but neglect biological structural priors. We introduce PI2L, a generative framework operating across 19,295 protein-coding genes that explicitly leverages the geometry of the PPI network as a soft inductive bias. PI2L embeds the interaction confidence as a Gaussian distance decay on a unit sphere, which is reformulated into an exponential kernel of inner products. The concatenation adopts a norm-preserving mixture form, where content and prior terms are allocated one shared attention logit budget conditioned on the corruption level. Empirical results demonstrate that cells synthesized by PI2L outperform those from classical single-cell generative models in terms of gene expression correlation and distributional fidelity. Overall, PI2L provides a scalable optimization paradigm for single-cell generative modeling, exhibiting broad utility for in silico cell gen- eration and biological interpretability. Our code is anonymously available at https://anonymous.4open.science/r/pi2l-submission/.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Single-cell Generation, Masked Discrete Diffusion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_7qQ7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PI2L, a generative model for whole-transcriptome expression that incorporates PPI-derived gene relationships into linear attention. Spherical PPI embeddings modulate attention through a corruption-dependent gate and determine static expert routing, while a hurdle decoder models zero and positive expression. Experiments on PBMC, heart, and brain datasets demonstrate improved marginal expression fidelity and gains over a Performer baseline.

Critical strengths

Incorporating biological structure into scalable whole-transcriptome generation is an interesting and important direction.

Critical weaknesses
  1. The evaluation does not sufficiently evaluate the generation using biologically relevant metrics, eg the pearson and spearman compare population-average gene expression. Although the MMD and coverage has some evaluation on the population, it is still unclear whether the model can perserve bioological property such as cell-type proportion, pathway scores, and within-cell-type coexpression etc. This is particularly concerning given its relatively low coverage compared with existing baselines (Table 1).
  2. Related to last point, the parse PBMC dataset contains 12 donors, but the evaluation does not investigate whether the generated population captures donor level heterogeneity, an important source of biological variation in PBMCs.
  3. PI2L is partly motivated by the limitations of sequence compression in Lingshu-Cell, yet no direct comparison is provided.
  4. From the ablation experiments (page 9), removing MoE actually improves all five reported generation metrics. While I understand that MoE is introduced to reduce FLOPs rather than improve accuracy, theoretical FLOP reductions do not necessarily translate into faster inference, particularly for a relatively small model with only six layers and fixed expert routing. How does this impact the acutal inference time (ie token/cell per second)?
Questions for authors
  1. See the weakness
  2. How could the biological prior adapt to cell identity or experimental conditions, particularly when disease or perturbation changes gene relationships relative to the association network (ie preparing to model the perturbation part of the parse PBMC dataset, while cytokine perturbation can change the PPI substantially)?
2 comments
AuthorsOpenReview ↗

Rebuttal follow-up

Dear Reviewer 7qQ7,

We hope this message finds you well. As the discussion period is nearing its end, we would like to kindly check whether our responses have addressed your concerns. Your feedback is very important to us, and we are happy to clarify anything further.

Thank you again for your insightful comments, which have greatly improved our paper.

Best regards,

The Authors

Thanks for the detailed response and additional experiments. The biological validation still does not assess donor-level heterogeneity, and the MoE component does not show better generation quality or faster inference. I will maintain my original assessment.

Reviewer_wXUQOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes to incorporate protein-protein interaction (PPI) network into the self-attentions of Performer, which uses FAVOR+ method to estimate the softmax function via kernels, thereby achieving self-attention with linear computation/memory time, allowing for long token sequences. To do so, the PPI network is decomposed, leading to per-gene PPI embeddings. These per-gene PPI embeddings are concatenated to the key and query vectors of the transformer block. The experiments show consistent improvement over Performer.

Critical strengths
  • The paper is written clearly, and is easy to read.
  • The goal is valuable and is well motivated: avoiding the quadratic cost of applying a transformer module on gene expression vectors with many genes, while incorporating protein-protein interaction network (PPI).
  • The experiments show consistent improvements over Performer.
Critical weaknesses
  • The proposed method cannot consistently outperform the baselines in Tab. 1 (scVI, scGAN, scDiffusion). See my notes below.
  • The comparative results (Table 1, Figure 2) are only on a single dataset, PBMC. This is a shortcoming, because the improvement of incorporating PPI into self-attentions might vary from dataset to dataset.
Questions for authors
  • Regarding the derivations below Eq. 3, after concatenating the PPI embeddings z_1,...,z_G, isn’t it that the equations of Performer (or FAVOR+) are already applicable, which makes the derivations unnecessary?

  • In the paper it is mentioned that “Beyond modulating inter-gene attention, we further leverage PPI-derived functional locality to organize parameter sharing in the feed-forward layer…” Isn’t it that the MLPs of the transformer block are shared anyways?

  • The results in Tab. 1 show that the proposed method cannot outperform the baselines. Is it the case that the baselines (scGAN, etc.) are trained with the usual noising forward process, while the proposed method is trained with the [mask] absorbing state? Because that might explain the worse results of the proposed method, despite it containing the PPI information. Please refer to “Sahoo et al, Simple and Effective Masked Diffusion Language Models” for more info.

2 comments
AuthorsOpenReview ↗

Rebuttal follow-up

Dear Reviewer wXUQ,

We hope this message finds you well. As the discussion period is nearing its end, we would like to kindly check whether our responses have addressed your concerns. Your feedback is very important to us, and we are happy to clarify anything further.

Thank you again for your insightful comments, which have greatly improved our paper.

Best regards,

The Authors

AuthorsOpenReview ↗

Rebuttal follow-up

Dear Reviewer wXUQ,

We hope this message finds you well. As the discussion period is nearing its end, we would like to kindly check whether our responses have addressed your concerns. Your feedback is very important to us, and we are happy to clarify anything further.

Thank you again for your insightful comments, which have greatly improved our paper.

Best regards,

The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Rebuttal follow-up

Dear Reviewer pMDP,

We hope this message finds you well. As the discussion period is nearing its end, we would like to kindly check whether our responses have addressed your concerns. Your feedback is very important to us, and we are happy to clarify anything further.

Thank you again for your insightful comments, which have greatly improved our paper.

Best regards,

The Authors

Reviewer_pMDPOpenReview ↗

Re: Rebuttal follow-up

Thank you for the revisions. The PBMC lineage and T-cell co-expression analyses address my main concern about biological validation. Your explanations also clarify the shared training protocol, scVI preprocessing, and MMD evaluation. The matched controls support the value of aligned PPI information, though the advantage of dynamic gating over a learned constant remains unclear. I maintain my rating of 3 (weak acceptance).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.