acceptodds
Withdrawn from ICLR 2027

Efficient Distributional Encoding for Multi-cellular RNA-seq Data

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Single-cell RNA-seq models typically tokenize cells independently as sequences of gene or cell-state tokens, causing sequence length to scale linearly with number of cells genes per cell . This scaling is poorly suited to tasks requiring multi-cellular context, such as patient-level disease diagnosis and spatial microenvironment niche mapping, where samples comprise dozens to thousands of cells. Here, we present a benchmarking study of single-cell tokenization strategies for multi-cellular prediction tasks. Holding a standard transformer backbone fixed, we evaluate gene- and cell-level tokenizers across three multi-cellular benchmarks, tracking predictive performance and token usage as the cellular cohort size varies from 1 to 1,000 cells. To overcome linear context expansion, we introduce module distribution encoding (MDE). MDE groups correlated genes into co-expression modules and summarizes the population-level expression profile of each module as a zero-adjusted distribution token. Total sequence length is therefore determined strictly by the number of gene modules rather than cell or gene count, decoupling context length from cohort size while preserving continuous population-level variance. MDE outperforms existing tokenization schemes on disease prediction and niche prediction, and cuts tokens by orders of magnitude, leading to significantly higher efficiency across all tasks. These results characterize the context-accuracy trade-offs of single-cell tokenization and establish sublinear, population-aware encodings as an effective basis for multi-cellular modeling.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Single-cell RNA sequencing, Tokenization, Deep Learning, Transcriptomics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jEdnOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates tokenization strategies for multicellular single-cell RNA-seq prediction tasks and introduces Module Distribution Encoding (MDE). Instead of representing individual genes or cells as separate tokens, MDE groups co-expressed genes into modules and summarizes each module's expression distribution across a population using zero-adjusted quantiles. This produces a fixed number of tokens independent of the number of input cells. The authors compare MDE against gene-level and cell-level tokenization approaches using a shared Transformer architecture across three tasks: patient-level disease classification, spatial niche composition prediction, and immune cell composition estimation. The results demonstrate substantial reductions in token usage, strong performance in disease prediction, and competitive performance in niche prediction, although cell-level tokenization performs better for cell composition estimation. Additional experiments examine module resolution and distribution encoding choices.

Critical strengths
  • Efficient population-level representation.
  • Useful comparative evaluation.
Critical weaknesses
  • Significant donor confounding in disease prediction. The disease benchmark uses cell-level splits within each donor because many disease categories occur in only one patient. Consequently, the training and test cohorts share donor identities, making it possible for models to exploit donor-specific transcriptional signatures rather than learn generalizable disease characteristics. This substantially weakens the interpretation of the nearly perfect disease classification results, which constitute one of the paper's main empirical findings.

  • Missing comparisons with simpler population-level aggregation methods. MDE is fundamentally a population-level summarization method, yet the main comparisons focus on gene-level tokenizers and per-cell Transformer inputs. Important alternatives, such as pseudobulk expression, mean/variance pooling, quantile aggregation without gene modules, and DeepSets-style pooling, are not evaluated. Without these baselines, it is difficult to determine whether the observed benefits arise from the proposed distributional encoding or from straightforward population aggregation.

  • Computational efficiency is not evaluated end to end. The claimed efficiency improvements are primarily measured through input token counts and an efficiency score based on performance divided by log token count. However, MDE still requires processing all cells, computing module scores, and estimating empirical quantiles before Transformer inference. The paper does not report wall-clock runtime, peak memory consumption, or the cost of token construction. Therefore, the practical computational benefits over simpler aggregation approaches are not sufficiently established.

  • Limited and task-dependent performance improvements. MDE substantially improves the reported disease classification results, but its advantage on spatial niche prediction is relatively small and inconsistent across cohort sizes. It also underperforms cell projection on cell composition estimation. Furthermore, the zero-adjustment ablation shows nearly identical performance without explicit zero adjustment for cohort sizes greater than one. These results suggest that the benefits of several proposed design choices are limited and that the broader advantages of distributional encoding require more careful qualification.

Questions for authors
  • Given that many disease labels occur in only one donor, could the authors provide the donor-to-disease distribution and evaluate whether a simple donor-identification baseline can explain the high disease prediction accuracy? This would help clarify the extent of donor confounding.

  • Could the authors compare MDE against simpler sample-level aggregation baselines, such as pseudobulk expression, mean/variance pooling, or distribution summaries without co-expression modules? These comparisons would help isolate the contribution of the proposed encoding.

  • Could the authors provide end-to-end runtime and memory comparisons that include module aggregation and quantile computation, rather than only Transformer token counts? This would strengthen the practical efficiency claims.

Reviewer_3GxxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MDE represents a group of cells with one distribution token for each co-expression module, so the transformer sequence length depends on the number of modules rather than the number of cells. The paper evaluates this design on disease prediction, spatial niche prediction, and cell composition estimation.

Critical strengths

The main idea is easy to understand and directly addresses the sequence-length problem in multi-cell RNA-seq prediction. Using a fixed number of module tokens can make it easier to process larger cell groups. The results also show that the representation behaves differently across tasks: it works well for disease and niche prediction, while cell projection performs better for composition estimation.

Critical weaknesses

The main concern is the split used for disease prediction. Cells from the same donor appear in both the training and test sets, so the model may learn donor-specific patterns. This split cannot show how well the method works on unseen patients. The donor-level split used for the composition task does not resolve this issue for disease prediction. Token count alone also does not fully describe the computational benefit. The paper does not clearly compare MDE with simpler group-aggregation or pseudobulk models under the same prediction setup, and it does not report enough runtime or memory measurements.

Questions for authors

Could the authors provide a donor-held-out disease evaluation for diseases with enough donors, or limit the disease-prediction claim to the current split?

Could the authors compare MDE with a simple pseudobulk or aggregation baseline using the same predictor, and report the corresponding runtime and memory usage?

Reviewer_7SF4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how reinforcement learning with verifiable rewards can improve single-sample accuracy while reducing access to correct solution strategies already supported by the base model. It introduces CAGE, an optimizer-conditioned attribution score that estimates how individual token updates affect entropy at held-out correct reasoning states. The score guides selective KL regularization of potentially harmful updates. Experiments examine whether CAGE predicts later loss of reasoning support and whether targeting these updates improves accuracy and retention.

Critical strengths
  1. The experimental question is well isolated. One backbone, four tokenizations, and a sweep in cohort size make the context-length trade-off visible. Figure 2 and the out-of-memory entries in Tables 2 and 3 are the clearest evidence in the paper: scGPT-style and Geneformer-style encodings exhaust memory by N=50 on niche prediction and by N=50 on composition, while MDE remains at M tokens and still produces a prediction at N=500–1,000. On niche proportions that constant-length model stays next to cell projection (Spearman 0.896 versus 0.899 at N=500).
  2. The composition experiment is the one task with a donor hold-out, and the paper reports a large loss there. At N=500, top-15 Spearman correlation is 0.734 for MDE and 0.897 for cell projection. Jensen–Shannon divergence is 0.083 versus 0.028. Stating a setting where the proposed inductive bias is the wrong one makes the “task-dependent tokenizer” conclusion credible.
Critical weaknesses
  1. Training and test sets contain cells from all 51 donors, and test cohorts are resampled from donor-specific distributions already seen during training. For diseases represented by a single donor, disease labels are confounded with donor identity. The near-perfect accuracy may therefore reflect donor recognition. A donor-held-out evaluation is needed to support claims about disease diagnosis.
  2. MDE does not consistently outperform other tokenizers on niche prediction. At N=50, its Spearman correlation is 0.817 versus 0.825 for scGPT, and at N=500 it scores 0.896 versus 0.899 for cell projection. Several statements also conflict with the tables, including the composition results and claims of monotonic improvement. The conclusions should reflect these mixed findings.
  3. The described pseudobulk and flattened-MDE baselines are not reported. Controls using mean-only summaries, mean plus zero fraction, random gene partitions, or matched-size PCA representations are missing. Comparing 100 modules with thousands of gene tokens also confounds representation quality with token budget.
  4. Dividing performance by log can favor MDE despite substantially worse predictive accuracy and may be undefined for a single token. Report accuracy, sequence length, runtime, memory, and preprocessing cost separately. Quantile construction still scales with cell count even if the transformer token budget is constant. Conflicting bin counts, unequal training schedules, and inconsistent backbone descriptions further hinder reproducibility. The scGPT and Geneformer comparisons should also be clearly identified as tokenization reimplementations trained from scratch, rather than evaluations of the pretrained foundation models.
Questions for authors
  1. Please evaluate disease classification with donors held out. Where a label has only one donor, restrict the reported metric to diseases with at least two donors and split by donor. On the current split, please also report a control that predicts donor identity rather than disease. If the MDE advantage in Table 1 disappears under a donor hold-out, or is explained by the donor-identity control, I would move from weak rejection to a clear rejection of the accuracy claim. A clear gain over a pseudobulk baseline under a donor hold-out would move me toward weak acceptance.
  2. On all three tasks, please report (i) a linear or MLP head on pseudobulk, and on mean module scores; (ii) an MDE ablation that keeps only the module mean and the zero fraction; (iii) random gene partitions and a PCA truncation at the same M as the co-expression modules. I need these to tell whether zero-adjusted quantiles and co-expression structure are doing the work, or whether any fixed-length summary of the cohort would match Table 1.
  3. Please reconcile the protocol with the tables. Which K was used in the reported disease, niche, and composition runs (50 or 100)? Why was MDE not run at niche N=5 and N=20? How many seeds are behind Table 2 and behind the MDE rows of Table 3, which have no standard deviation, and are the Figure 2 error bars across seeds or across cohorts? The two captions disagree. For the liver experiment, what is the spatial split (number of fields of view, and the gap between train and test regions), and were the six niche labels assigned independently of the genes used as input?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.