acceptodds
Withdrawn from ICLR 2027

Contextual Differences Between Paired Molecular Representations Encode Cell State

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Different molecular measurements can provide complementary views of the same biological element and capture distinct aspects of cellular state. Existing representation learning approaches typically characterize such measurements through individual or integrated representations, while relationships between paired observations of the same biological element are less directly used as representations themselves. This raises the question of whether these relationships can themselves provide a representation of cellular state. We study this question in single cell transcriptomics, where unspliced and spliced RNA provide paired measurements of the same genes in different RNA processing states. We introduce scUS, a pretrained masked Transformer that jointly contextualizes the two RNA states within each cell and produces paired representations for each gene. Differences between these contextual representations define a gene resolved profile for every cell. These profiles retained cell state structure across biological samples, whereas reducing them to a single measure of overall discrepancy substantially weakened this structure. Representation differences also remained sensitive to cellular context even after matching gene identity and local unspliced and spliced input bins. Extending the same analysis to RNA and chromatin accessibility revealed similar organization associated with cell type and distinct patterns across different molecular pairs. Together, our results show that contextual differences between paired molecular representations can themselves form a structured representation of cellular state.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Representation differences, Multimodal learning, Single-cell transcriptomics, Cell state

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_RmcPOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents scUS, a framework that uses the differences between contextualized unspliced and spliced RNA representations to characterize cell states. Rather than combining the two representations into a single embedding, the authors compute gene-wise distances and use them as a cell-level profile. The paper evaluates this idea through cell clustering, context-dependent analyses, stimulus-response experiments, and an extension to ATAC-seq data.

Critical strengths
  1. Most existing methods focus on learning representations from individual modalities or integrating multiple modalities. This paper instead looks at the relationship between two representations of the same gene. The gene-wise distance profile is a simple but interesting way to capture this information, and I think this perspective is worth exploring.
  2. The experiments provide good support for the main idea. The authors evaluate the approach on several datasets and examine it from different angles. In particular, the distance profiles outperform pooled representations from the same encoder across all three clustering datasets. The matched-bin analysis is interesting, as it shows that the learned distances can vary across cellular contexts even when the local inputs are matched.
Critical weaknesses
  1. The method uses cosine distance to measure the differences between paired representations. It would be useful to know whether similar results can be obtained with other distance metrics, such as Euclidean distance. A comparison could help establish how sensitive the findings are to this choice.
  2. Some experimental settings are a little difficult to follow, particularly the preprocessing steps and differences between evaluation protocols. A table summarizing these settings would make the comparisons easier to understand.
Questions for authors
  1. Have the authors explored other distance metrics besides cosine distance, such as Euclidean distance? I wonder whether the observed cell-state structure is specific to cosine distance or whether similar patterns can be obtained with other measures.
  2. Could the authors clarify the main differences in preprocessing and feature selection across the evaluated methods? A brief summary table would be helpful for understanding the comparisons, particularly those involving scGPT and scFoundation.
  3. What are the relationship between gene-wise distances of scUS and gene expression levels? While Figure 5 provides some examples, it would be interesting to see whether there is a general relationship.
Reviewer_CcikOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors present scUS, a masked Transformer for single-cell transcriptomics that treats the relationship between paired unspliced (U) and spliced (S) RNA measurements as a cell representation in its own right. Each gene contributes interleaved U and S tokens formed from gene-identity, rank-binned value, and RNA-state embeddings, and the encoder is pretrained on the scBaseCount Velocyto collection with a masked-bin cross-entropy objective over both states. After joint contextualization, the cosine distance between each gene’s U and S hidden states defines a gene-resolved distance profile for every cell, accompanied by a pair-availability mask. Empirical results suggest higher clustering NMI than pooled U/S embeddings from the same encoder on forebrain, OSCC spatial regions, and mouse gastrulation, with mixed results against scGPT and scFoundation (higher in forebrain, lower than scFoundation in OSCC and mouse), complemented by matched-bin analyses showing that gene identity and local U/S bins do not fully determine the distance, a stimulus-response case study on an LPS time course, and a descriptive extension to U–ATAC and S–ATAC pairs in a multiomic dataset.

Critical strengths

Overall, I think the idea of exploiting unspliced and spliced information has some potential. Current exome-based encoding approaches toss away potential regulatory information that could be used to enrich functional embeddings.

Critical weaknesses

Overall, I believe the submission needs some extensive work and is not suitable for publication at this stage. I will give more inline technical feedback below. In general, I find the contribution quite incremental. Usually, I do not consider it a full blocker. But in this case, I think the paper does not offer crucial biological interpretation or analyses that compensate for the lack of novelty.

Personally, I struggled to find any evidence of why accounting for both spliced and unspliced counts could be beneficial in a foundation model. Most of the proposed experiments are rather diagnostic: comparing distances, reporting loss behavior. All these observations and the technical points below lead me to propose a rejection score, which I am not open to increasing at this stage. I hope the authors will find the feedback useful for future iterations of the paper.

In-depth feedback

  • L46-49: The need to account for unspliced transcripts as well needs to be better motivated. Why is this biologically interesting?
  • L51-52: At the introduction stage, I find the concept of "cases where local inputs do not determine the distance" a bit unclear. Could you paraphrase it more clearly at this stage?
  • The math is not presented very rigorously. What is bb in equation 1?
  • A lot of aspects are taken for granted. Why is the masking ratio 0.3 chosen?
  • A lot of emphasis is placed on aggregations of the distances. But I assume that this removes a lot of variance expected across different genes, some more expressed in specific cell states than others. An analysis of specific distances in specific cell types would make the paper more convincing.
  • L202-206: Example of what I meant with "reporting diagnostics". Lingering so much on loss decrease is not very scientific. That is the minimum for justifying the presentation of results.
  • L214-216: Also here, I think a notion of the distribution of distances would be important. A boxplot in the appendix would be enough to appreciate the quality of the separation between categories.
  • scFoundation outperforms the distance profile on two of three datasets (OSCC and mouse gastrulation), and simple baselines such as the availability mask also do better there. Could the authors report ARI and multiple clustering settings for all models, and discuss what the profile offers over scFoundation?
  • The matched-bin analysis (Section 4.3) shows that distances vary with cellular context, but this is expected from any attention-based encoder, and randomly initialized models show the same effect with similar direction agreement. Could the authors clarify what this experiment reveals beyond the architecture’s contextual design, and provide a control that isolates the contribution of pretraining?
  • Neighborhood moments (Mu, Ms) are central to the inputs and the LPS analysis, but the main text never defines them, and the notation is introduced only in the Figure 4 caption. Could the authors briefly define the moments in Section 3.1 and explain why Figure 5 uses smoothed moments for SKAP2 but raw CPM for RPL23? Also, it's not clear what each distance represents where a small sequence is provided (e.g., dist1/dist2...).
  • The LPS screen identifies only two ribosomal genes where distance changes without a change in total expression, and in neither are U and S separately stable. Could the authors show that the distance captures information beyond the U/S ratio, for example by comparing against a simple U/S ratio baseline?
  • The writing is often hard to follow, particularly in Section 4.4. Key procedures (e.g., the equivalence screen) are described only in the appendix, terms such as “saved screen” and “updated display” refer to material outside the paper, and dense caveats obscure the main findings.
  • The authors state that the checkpoint used to produce the multiomic distance matrices cannot be identified (Section 4.5), so these results cannot be reproduced or attributed to a specific model. Given this and the lack of quantitative evaluation, this exploratory analysis does not fit in the main paper. I would suggest either regenerating the results with a documented checkpoint or moving the RNA–chromatin extension to the appendix and removing it from the listed contributions.
  • The temporal heatmaps (Figure 6D–I) order cells by latent time and genes by expression peak, then smooth and standardize each row, which may produce banded patterns regardless of the underlying signal. As the authors acknowledge, no quantitative test is provided.
Questions for authors
  • L102-104: How is eligibility determined?
  • L106-107: Why discretize to 15 bins? Is this compatible with datasets with high inverse dispersion and spread counts? It sounds like a fixed value can become very coarse or very fine quickly.
  • Why do you refer to "positive inputs"? Since we are dealing with gene expression data, I can imagine the inputs can only be positive. Do you mean "positive nonzero"?
  • L192-193: Is there a reason why different versions are used?
  • How is the panel of 24 genes selected? It sounds a bit like a low number.
  • L222-223: How is the distance profile used as an embedding?
  • The matched-bin analysis (Section 4.3, Appendix C) shows context dependence for both pretrained and random encoders, with similar direction agreement, and uses a separate historical pipeline from the main benchmark. As the authors note, it does not isolate the contribution of pretraining. Could they clarify what conclusion this experiment supports, or replace it with a pretrained-versus-random comparison under the main benchmark protocol?
Reviewer_tgjTOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose a masked Transformer pretrained on paired unspliced (U) and spliced (S) RNA measurements. For each gene, the model jointly processes the two RNA states and computes the cosine distance between their contextualized hidden representations. These gene-wise distances form a cell-level profile used for clustering and biological analysis. The approach is evaluated on three datasets, examine whether the distances capture information beyond matched local RNA measurements, investigate changes following LPS stimulation, and explore extensions to RNA–chromatin accessibility pairs.

The main contribution is a new representation readout rather than a new model architecture. The paper aims to establish that relationships between contextualized molecular states are informative, even without explicitly modeling transcriptional dynamics.

Critical strengths

The paper explores whether relationships between contextualized molecular states can be useful representations. Some results show positive signals, such as figure 2B: U-S distances within correct pairs are smaller, although it looks less significantly to "same gene other type", but is significant to "different gene" (the authors are encourage to put in significance p values).

authors also explicitly acknowledge several limitations, including unmatched baselines, preprocessing effects, and incomplete provenance. This transparency makes the results easier to evaluate and is a strength of the work.

Critical weaknesses
  1. The central concern is that it is not clear theoretically nor intuitively, whey the authors use gene-wise distances form a cell-level profile for clustering and biological analysis, apart from this is an alternative than existing embeddings. In fact, looking at section 4.2 and figure 3, the scUS representation shows some capability in cell type differentiating, but no consistent advantage over existing methods.

  2. The RNA-ATAC experiments are qualitative. Again showing the distance embeddings can differentiate cell types (also without baseline comparison) in a qualitative way is trivial. One can think of PCA of the data or even random projection of the input should be able to capture such signal to some extent as well. Although the authors acknowledge some of these preliminary limitations. The sections still provides limited evidence to support the usefulness of the method.

  3. some of the content reads like AI written and quite difficult to follow. For example, this last paragraph on page 3, "The pair-availability mask accompanies Dc; a missing pair is not a measured zero distance. Downstream filling and feature selection are analysis-specific and must be recorded. Each coordinate retains a gene identity, while its value depends on the encoded cellular context. The construction is a profile of scalar distances, not a subtraction of hidden vectors." Words like construction, coordinates convey little information without giving enough context. I would encourage the authors to perform a more strident proofreading

Questions for authors
  1. Could the authors respond to the first question in the weakness by adding descriptions of theoretically and empirically what new signals can be revealed by the U-S embeddings and their distances.

  2. Given the method itself is not new, in the sense most of the training techniques are similar to existing works of scGPT and geneformer, with only the extension of adding unspliced and spliced input tokens together, then could the authors explain if any other significant novelty of their proposed approach?

  3. How do distance profiles of encoders perform on the three clustering benchmarks, if training from scratch? It would be important to see whether pretraining improves the biological usefulness of the distance representation.

Reviewer_zp6SOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors introduce scUS, a masked transformer that takes in both the unspliced and spliced measurements for a cell’s genes at the same time. The authors aim to solve the issue of direct U/S correlation depending heavily on preprocessing such as neighbourhood smoothing and nonzero filtering. The author's evaluation combines training diagnostic, three-set data clustering, locally matched gene examples and context controls.

Critical strengths

The problem being addressed is interesting where instead of just accounting for the expression level of the gene in the cell, it also accounts for the unspliced and spliced RNA measurements which correspond to different stages of the RNA processing.

Critical weaknesses
  • The results do not seem well presented at all and are quite hard to comprehend. Better formatting highlighting the different models and datasets would be useful in addition to description of the metrics.

  • It seems like on 2 out of the 3 datasets, the baselines outperform the proposed model. Is there an explanation to this?

  • The training loss in figure 2A does not seem to move at all.

  • A big issue of the paper is that in Figure 6 the paper says that “The model is target-adapted; the exact distance-export checkpoint remains unresolved.” which sounds AI generated and is a very big reproducibility issue if the authors cannot tell from which trained model the results actually come from. I recommend rejecting this paper.

Questions for authors
  • It seems like on 2 out of the 3 datasets, the baselines outperform the proposed model. Is there an explanation to this?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.