acceptodds
Withdrawn from ICLR 2027

From Histology to Cell Sentences: Visual Soft-Prompts for Long-Horizon Ranked Gene Expression

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Spatial transcriptomics (ST) measures gene expression in tissue but remains costly and sparse, whereas histology images (H&E) are abundant. Existing virtual-ST methods typically regress a small highly variable gene (HVG) panel or retrieve expression from a reference atlas. We instead cast H&Eexpression as ranked sequence generation in the spirit of LLaVA-style visual conditioning of an LLM, adapted to a domain where that recipe does not directly transfer. A lightweight adapter couples two domain-specific foundation models trained on massive corpora, a pathology expert and a single-cell language model, mapping frozen visual features into Visual Soft-Prompts that elicit cell sentences: gene symbols ranked by relative expression over a 27k-gene vocabulary. Prediction length is a central challenge: short-context models give the strongest early-rank fidelity but emit only a handful of genes, while naively extending the autoregressive horizon with a frozen decoder yields longer lists that eventually derail into duplicate loops and invalid tokens. A failure mode analysis shows that standard decoding fixes such as nucleus sampling or constrained decoding do not repair this. With attention LoRA and a larger training budget, derailing is largely prevented, producing near-complete cell sentences while retaining early-rank precision. Scored against the full ground-truth cell sentence at every depth, dense panel regressors and averaging retrieval methods (ST-Net, BLEEP) lead briefly but collapse once they exhaust the panel they were built and evaluated on: they cannot, by construction, name genes outside it, whereas our model continues informatively over the full expressed set. On a frozen 1k HVG panel used only as a secondary calibration, a mid-length LoRA model (2048 tokens) approaches ST-Net and BLEEP on panel correlation (within 0.03 PCC). The result is a generative path from H&E to ranked cell sentences that needs no pre-specified gene panel and can compose with external predictions rather than only replace them.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
spatial transcriptomics, multimodal foundation models, single-cell language models, LoRA, gene-expression prediction, histology

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GH4gOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a Visual Soft-Prompt framework for predicting ranked gene expression sequences from histology images. The approach connects a frozen pathology foundation model (GPFM) with a pretrained cell-sentence language model (C2S-Scale) through a trainable adapter, optionally incorporating attention LoRA for long-horizon generation. The paper investigates autoregressive degeneration in long gene-sequence generation and demonstrates that attention LoRA, combined with extended training, substantially improves generation stability and gene coverage. Experiments on the HEST dataset compare the proposed approach against conventional regression and retrieval methods, including ST-Net, BLEEP, and HisToGene. The results suggest that the method can generate longer gene lists beyond predefined gene panels, although several conventional baselines remain stronger on major shared-panel ranking metrics.

Critical strengths
  1. Meaningful formulation of gene expression prediction as sequence generation.

The paper explores an interesting alternative to fixed-panel gene expression regression by generating ranked gene lists over a substantially larger vocabulary.

This formulation provides greater flexibility in the output space and potentially enables the prediction of genes outside predefined panels.

  1. Thorough investigation of long-horizon generation failures.

The authors systematically investigate how sequence length, decoding constraints, attention LoRA, and training budgets affect generation quality.

In particular, the matched-length and matched-budget experiments provide convincing evidence that attention adaptation contributes to stable long-horizon generation beyond simply increasing the number of training steps. This analysis represents a valuable contribution of the paper.

Critical weaknesses
  1. Insufficient baseline coverage and inadequately controlled comparisons limit the evidence for the proposed architecture's independent contribution.

This is my primary concern regarding the paper.

First, the experimental comparisons mainly involve relatively established methods such as ST-Net, BLEEP, and HisToGene, while several more recent and potentially competitive approaches are not evaluated. Relevant examples include:

STFlow (Huang et al., ICML 2025): A whole-slide flow-matching approach for spatial gene expression generation, evaluated on datasets including HEST-1k.

OmiCLIP/Loki PredEx (Chen et al., Nature Methods 2025): A visual–omics foundation-model framework that supports cross-modal prediction of spatial gene expression from histology.

HECLIP (Bioinformatics 2025): A histology-enhanced contrastive learning approach for spatial transcriptomics imputation.

GenAR (Ouyang et al., Medical Image Analysis 2026): A multiscale autoregressive framework for predicting spatial gene expression from histology images, which is particularly relevant to the proposed generative formulation.

Notably, STFlow, OmiCLIP, and GenAR are already discussed in the Related Work section but are not included in the quantitative comparisons.

I acknowledge that these methods differ in their input requirements, prediction spaces, and output representations, so fully equivalent comparisons may not always be straightforward. Nevertheless, the authors should clarify why these recent approaches were excluded and, where feasible, provide comparisons using shared gene sets, consistent data splits, and common ranking metrics.

Second, the main long-horizon comparisons against ST-Net and BLEEP are affected by their fixed 1,000-gene prediction panels.

Once the evaluation depth exceeds the effective prediction range of these baselines, their performance degradation is partly an inherent consequence of their restricted output spaces. This does not necessarily demonstrate that the proposed generative model achieves superior gene-ranking accuracy.

Therefore, the reported long-horizon advantage may partly reflect differences in output coverage rather than improvements in predictive capability under equivalent conditions.

More importantly, the compared methods use different visual representations. The proposed approach benefits from a strong pretrained GPFM encoder and additional organ/technology metadata, whereas the conventional baselines primarily follow their original image-based configurations.

A direct comparison with a regression or gene-ranking model trained on the same GPFM features and metadata is missing. Without such a controlled baseline, it is difficult to determine whether the observed benefits arise from the visual foundation model, the pretrained cell-sentence language model, or the autoregressive generation architecture itself.

Furthermore, on the shared 1k-HVG panel, the proposed approach still trails ST-Net and BLEEP on major ranking metrics, including J@10, NDCG@50, and PCC(inv).

Overall, the experimental evaluation neither sufficiently covers recent competitive methods nor adequately isolates the contribution of the proposed architecture under matched input and output conditions. This substantially limits the strength of the paper's empirical claims and is a major factor in my recommendation.

  1. Long-horizon generation quality and statistical reliability require stronger validation.

Although the proposed model generates substantially longer gene sequences, the current evaluation does not sufficiently distinguish accurate gene ranking from expressed-gene set recovery.

As acknowledged by the authors, extensive expression-count ties in the long tail make windowed NDCG increasingly reflect gene-set membership rather than precise relative expression ordering.

Consequently, generating near-complete gene sequences does not necessarily establish faithful reconstruction of long-range gene expression rankings.

Additionally, the reported results are based on single training runs, without multi-seed evaluations of model stability.

The confidence intervals are computed over sampled spatial spots, while multiple spots originate from the same tissue slides and may exhibit substantial statistical dependence. Treating these spots as independent observations could underestimate the uncertainty associated with the reported results.

These limitations make it difficult to fully assess the robustness and statistical significance of the reported improvements.

Questions for authors
  1. Recent baselines and comparison fairness

The paper discusses recent methods such as STFlow, OmiCLIP, and GenAR but does not include them in the experimental comparisons. Could the authors clarify the specific reasons for their exclusion?

For methods with compatible prediction objectives, are comparisons available under consistent data splits, shared gene sets, and unified evaluation metrics? If direct comparisons are infeasible, please clarify the relevant methodological or protocol differences.

  1. Matched feature comparison

Could the authors clarify how much of the reported performance improvement can be attributed to the pretrained C2S language model rather than the GPFM visual representations?

Are results available for a regression model or gene-ranking head trained using the same GPFM features and organ/technology metadata?

  1. Long-horizon evaluation

Given that conventional baselines are restricted to fixed gene panels and that deep ranking metrics are affected by extensive expression ties, how should the reported long-horizon advantage be distinguished from increased gene coverage and expressed-gene set recovery?

What evidence supports improved ranking accuracy beyond the recovery of expressed genes?

  1. Statistical robustness

Were correlations between spots from the same slide or patient considered when computing confidence intervals?

Are results from additional training seeds available to assess the stability of the proposed approach?

Reviewer_zYKCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Visual Soft-Prompts to connects a pretrained pathology foundation model with a single-cell language model to predict ranked gene-expression sequences from histology images.

Critical strengths

The paper investigates an interesting perspective to conventional gene-expression regression by formulating the problem as open-vocabulary sequence generation.

The analysis of long-horizon generation failures is quiet thorough.

Critical weaknesses

Motst importantly, this work has very limited significance for the broader AI community. Although the paper is (barely) within scope, its problem formulation and findings are highly specific to bioinformatics people (and probably pathology). The potential readership is relatively limited, and the work does not demonstrate broader methodological insights applicable to other multimodal or structured sequence-generation tasks.

Also, the evidence for the effectiveness of the generative formulation is sounding, but not comphrehensive. As the author ackowledged, the experiments lack a regression baseline using the same pretrained pathology features, making it difficult to isolate the benefits of the generative formulation.

Questions for authors

Please add the missing experiements as ackowledged.

Beyond the spatial transcriptomics application, provide more insights into how the formulation of this work will be useful for other application, or be informative to general AI development.

Are there evidence that the additional genes predicted can provide meaningful biological information, rather than simply improving coverage-based ranking metrics?

Reviewer_QwLvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper aims to predict ranked list of genes from H&E, instead of the conventional approaches for predicting gene expression values.

The authors connect two frozen foundation models: a pathology image encoder and a single-cell LLM trained on cell-sentences. A trainable adapter turns each spot’s image embedding, together with organ and technology labels, into fixed Visual Soft-Prompt tokens, and the LLM is then conditioned on these soft-prompt tokens to output the ranked gene list.

The main finding is about output length. With the decoder frozen, longer outputs drift into repeated or invalid gene names, and the authors argue that decoding tricks such as nucleus sampling or constrained decoding do not fix this. Adding LoRA to the decoder’s attention layers, combined with longer training, lets the model produce much longer and stable lists.

The model is evaluated on the HEST dataset with a patient-level split, and is benchmarked against the full ground-truth gene ranking and, separately, compared with ST-Net, BLEEP, and HisToGene on a shared panel of HVGs. The baselines do better on the shared panel, while the proposed model keeps producing informative genes beyond it.

Critical strengths
  • The failure-mode analysis is honest and useful. The paper shows how a frozen cell-sentence decoder breaks down as the output gets longer, across position-resolved rate, gene validity, and novelty. Anyone trying to plug an LLM decoder into a biological prediction task will hit this problem, so the diagnostics are a real contribution.
  • The controls show that both the image and the cell-sentence prior matter. Shuffling the GPFM embedding while keeping metadata drops J@10, and swapping C2S-Scale for vanilla Gemma-2-2B drops it further. At the long-horizon checkpoint, the shuffled, zero, and noise arms stay below the real image deep into the list. This shows the image adds signal beyond the metadata-conditioned prior.
Critical weaknesses

The comparisons do not isolate the claims of the paper.

  • The baseline comparison is confounded. The abstract says panel regressors "cannot, by construction, name genes outside" their panel. But the 1K-HVG panel is the authors' choice, and only about 28% of it is expressed per spot (Sec. 4.4). The baselines also use their own image encoders, and do not use organ or technology labels, while the proposed model gets GPFM features and both labels (Sec. 3.2, 4.1). So, Fig. 2 and Table 1 mix the effect of the decoder with the encoder, the metadata, and the panel, which the Limitations section acknowledges. Please either soften the abstract or add a regressor on the same features and labels over the full gene set, scored with the same protocols.
  • The per-gene, across-spot correlation used in HEST-Benchmark is not reported. All the metrics score each spot's ranking separately, so they favour genes that are high in almost every spot. Please report the HEST-style per-gene correlation, and add a baseline that outputs the training-set average ranking per organ and technology, so the contribution of the image is clear.
  • Most of the gap to the baselines in Table 1 comes from one stratum. Fig. 6 breaks the Table 1 PCC(inv) down by technology and organ. The Spatial Transcriptomics spinal-cord stratum has 300/1310 test spots. There, Ours scores 0.50, against 0.79 for ST-Net and 0.82 for BLEEP. Elsewhere the model is much closer to the baselines. Without this stratum, the aggregate gap to ST-Net decreases significantly, and the gap to BLEEP almost disappears. Please report Table 1 with and without this stratum and explain what is different about it.
Questions for authors

Clarifications

  • Checkpoints are selected by "validation NLL" (Appendix A), but Sec. 3.2 describes only a 1014-slide train and 262-slide test split. Where does the validation set come from? Which choices (m, checkpoints, τ=0.5, K=100, decoding settings) were made on which data? If any used the test slides, the reported numbers are optimistic.

Suggested experiments

  • A regressor on GPFM features plus the same labels, over the full gene set. Tune its expression threshold on validation data to decide which genes are listed, and score it with the Fig. 2 and Table 1 protocols. This would separate what the LLM decoder adds from what the features and labels add.
  • An image-free baseline (the training-set mean cell sentence per organ and technology), plus the per-gene, across-spot correlation used in HEST-Benchmark for all methods, for example using 1/rank as the score. This would show how much of the signal comes from the image rather than from a generic prior.

Comments on the paper

1 comment

We thank the reviewers for their careful and constructive reviews. We ran the requested experiments and report the outcome briefly, because they change the paper's central claims.

Matched regressor (QwLv, zYKC, GH4g). A ridge/MLP regressor on the same frozen GPFM features plus organ/technology labels, over all 27k genes, with an expression cutoff tuned on validation data, outperforms our model on the spot-wise ranking metrics (Table 1 J@10 0.40 vs. 0.31 for our long-horizon model) and on the Fig. 2 windowed NDCG and precision at every depth (paired slide-level bootstrap, intervals well away from zero). It also selects expressed genes better (F1 0.49 vs. 0.28).

Image-free prior (QwLv). The training-set mean ranking per organ and technology reaches our model's J@10 and exceeds it on the Fig. 2 metrics at every depth. The reviewers are therefore right: the long-horizon advantage over ST-Net and BLEEP reflects output coverage and the encoder, not better ranking accuracy.

Per-gene correlation (QwLv). On 24 human slides, the HEST-style per-gene correlation of our model (r = 0.16 on the top-50 variable genes) is below ST-Net (0.22) and the GPFM regressors (0.26 to 0.32).

Stopping (zYKC, GH4g). Training sentences contained no end-of-sequence token at their true end, so the model cannot learn spot-specific list length. It almost always stops at the training budget.

What holds. Tracking of which genes are spatially structured (correlation of per-gene Moran's I with ground truth, 24 slides, slide-level bootstrap) is significantly better than ST-Net, BLEEP and HisToGene, and on par with the matched regressor (0.22 vs. 0.20, interval includes zero); per-gene variance and gene-gene correlation structure, however, are closer to ground truth for the regressors. The blind controls show that the image adds signal beyond the metadata, and the analysis of long-horizon degeneration in a frozen cell-sentence decoder, which all three reviewers named as a strength, is unaffected.

Clarification (QwLv). The validation set is 20% of the training shards (spots from training slides). Design comparisons (sequence length, LoRA, decoding) were evaluated on test spots, which are separated from training at the patient/study level. Given these results, the abstract's claims about the generative formulation are not supported in their current form. We are therefore withdrawing the paper and will rework it around the failure analysis and the matched comparisons. We thank the reviewers again for feedback that made this clear.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.