acceptodds
Withdrawn from ICLR 2027

Octave: Scale-Resolved Evaluation of Spatial Gene Expression Prediction from Histology

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Spatial gene expression prediction from histology is typically evaluated using the mean per-gene Pearson correlation (PCC). However, PCC does not distinguish spatial scales: high performance may reflect coarse tissue organisation rather than fine-scale expression patterns. Indeed, on a widely used benchmark with approximately 100-µm sampling, a simple predictor that captures image-derived tissue regions but no within-region spatial variation achieves 82% of the PCC of a trained model. This raises a fundamental question: at what spatial scales does a model actually predict gene expression accurately? Herein, we introduce OCTAVE, a scale-resolved evaluation framework that separates predicted and measured expression into spatial scales and evaluates their agreement at each scale. On higher-resolution Xenium data, differences that appear modest under PCC become substantially larger at finer scales: at approximately 20 µm, the domain oracle’s shortfall relative to a trained model is 2.4 times its shortfall under PCC, consistently across all 13 specimens. Across 57 image encoders, OCTAVE further reveals greater performance differences and changes the ordering of 261 of 1596 encoder pairs. The benchmark’s own finest band is 50 µm wide, coarser than the scale at which these differences appear, so no score on that pitch can directly validate finer structure. These results show that PCC alone can obscure the spatial structure underlying predictive performance. OCTAVE complements existing evaluation by revealing not only how well a model predicts spatial gene expression, but also at what spatial scales it succeeds.

Primary area:
datasets and benchmarks
Keywords:
spatial transcriptomics, histology, spatial gene expression prediction, evaluation metrics, benchmark, pathology foundation models, spatial scale, Pearson correlation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_AoLMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces OCTAVE, a scale-resolved evaluation framework for spatial gene expression prediction from histology images. The authors argue that conventional mean per-gene Pearson correlation (PCC) is insensitive to the spatial scales at which predictions are accurate, potentially rewarding coarse tissue organization while overlooking fine-scale expression patterns.

To address this limitation, OCTAVE uses graph-based diffusion operators to decompose predicted and measured gene expression into spatial frequency bands, evaluating their agreement at different physical scales. The paper also introduces a domain oracle to quantify how much conventional PCC can be explained by coarse tissue-domain information alone.

Experiments on HEST, Xenium, and Visium HD datasets show that coarse domain-level predictions can achieve relatively high PCC, while scale-resolved evaluation reveals larger discrepancies at finer spatial resolutions. Additional experiments across 57 image encoders demonstrate that model rankings can change when evaluated at different spatial scales.

Critical strengths
  1. Important observation regarding the limitations of conventional PCC-based evaluation.

The paper addresses a meaningful issue in spatial gene expression prediction: high PCC does not necessarily indicate accurate recovery of fine-scale spatial expression patterns.

The domain-oracle analysis provides useful evidence that coarse tissue organization can explain a substantial fraction of conventional prediction scores. This observation is relevant to the design and interpretation of future spatial transcriptomics benchmarks.

  1. Extensive quantitative evaluation and careful statistical analysis.

The experiments cover multiple spatial transcriptomics platforms, 13 independent Xenium specimens, and 57 image encoders. The authors also investigate sensitivity to graph construction, spatial resolution, clustering configurations, and other evaluation choices.

The observation that scale-resolved evaluation can substantially change relative model comparisons is interesting and highlights the importance of considering spatial structure when interpreting prediction performance.

Critical weaknesses
  1. The necessity and methodological novelty of introducing a new evaluation metric are insufficiently justified.

This is my primary concern.

Although the paper convincingly demonstrates that conventional PCC is insensitive to spatial scale, this observation does not necessarily establish the need for an entirely new evaluation framework.

The core OCTAVE formulation relies on graph diffusion, differences between successive smoothing operators, and correlations computed within the resulting spatial bands. These components are closely related to established techniques in multiscale signal processing, graph wavelets, spatial filtering, and scale-space analysis.

Indeed, the paper acknowledges related approaches such as diffusion wavelets, spectral graph wavelets, and multiscale image similarity in the Related Work section. However, it does not provide sufficiently direct empirical comparisons demonstrating why OCTAVE should be preferred over simpler alternatives.

For example, applying Gaussian smoothing or graph-based low-pass filtering at multiple scales, followed by high-pass residual correlation, could potentially reveal similar limitations of conventional PCC. Existing spatial diagnostics such as Moran's I may also provide complementary information about oversmoothing, although they do not measure exactly the same quantity.

The paper includes analyses involving spatial autocorrelation, but it does not establish whether the principal findings of OCTAVE are substantially different from those obtainable through these simpler multiscale evaluation strategies.

Furthermore, the reported changes in encoder rankings do not by themselves demonstrate that OCTAVE produces more biologically meaningful or practically useful model selections. A different ranking is informative only if the new criterion better captures spatial structures relevant to downstream scientific objectives.

I therefore remain unconvinced that introducing OCTAVE as a distinct evaluation metric is sufficiently necessary or offers a substantial methodological advance over established multiscale evaluation approaches.

A direct comparison with simpler scale-conditioned correlations or graph-wavelet-based evaluation, together with evidence of improved biological interpretability or practical model selection, would considerably strengthen the paper.

  1. Insufficient qualitative visualization limits the interpretability and practical validation of the proposed evaluation framework.

Despite the central emphasis on fine-scale spatial expression patterns, the paper provides surprisingly limited qualitative comparisons of actual spatial gene expression predictions.

Figure 1 presents only two selected Visium HD colon windows, chosen to highlight large finest-band discrepancies. The remaining figures primarily focus on quantitative curves, statistical summaries, and sensitivity analyses.

Several limitations arise from this presentation.

First, the paper does not systematically visualize ground-truth and predicted spatial expression patterns across the Xenium specimens that form the basis of its main fine-scale analysis. Representative examples from different tissues and individual genes would help demonstrate how the proposed metric relates to actual spatial prediction errors.

Second, the paper reports that OCTAVE changes the relative rankings of 261 out of 1,596 encoder pairs, yet it provides no direct qualitative comparisons illustrating why models with similar conventional PCC receive substantially different fine-scale scores. Side-by-side spatial predictions from representative encoder pairs would be particularly valuable.

Third, Figure 1 primarily visualizes the first principal component of a gene-expression panel rather than individual marker-gene expression maps. While this illustrates the overall spatial structure, it limits the biological interpretability of the observed differences.

Finally, selecting windows with particularly large discrepancies is useful for motivating the method but does not establish that the illustrated differences are representative of typical cases. Including randomly selected examples, representative cases, and failure cases would provide a more balanced assessment.

For a paper whose main contribution is an evaluation framework intended to reveal otherwise overlooked spatial structures, qualitative validation is particularly important. Without sufficiently representative spatial comparisons, it is difficult to assess whether differences in the proposed metric correspond to meaningful biological improvements rather than alternative mathematical characterizations of spatial variation.

Overall, the lack of systematic qualitative evidence weakens the practical justification for adopting OCTAVE as a new evaluation standard.

Questions for authors
  1. Necessity of a new evaluation metric

The proposed method builds upon established graph diffusion and multiscale signal-processing techniques. Could the authors clarify what OCTAVE provides beyond simpler approaches such as Gaussian high-pass correlation, scale-conditioned PCC, or graph-wavelet-based spatial evaluation?

Are direct comparisons with such alternatives available? In particular, would these simpler methods lead to similar conclusions about coarse-domain dominance and changes in encoder rankings?

  1. Biological and practical significance

The paper reports substantial differences in fine-scale correlations and changes in encoder rankings. However, how can we establish that these differences correspond to more biologically meaningful spatial expression predictions?

Is there evidence that selecting models according to OCTAVE leads to better preservation of marker-gene localization, tissue boundaries, or other downstream biological structures compared with selecting models using conventional PCC or existing spatial metrics?

  1. Qualitative visualization

Why are the qualitative spatial comparisons limited to two selected Visium HD windows?

Could the authors provide representative Xenium examples showing ground-truth expression, domain-oracle predictions, and trained-model predictions for individual marker genes across multiple tissues?

Additionally, could the authors visualize representative encoder pairs whose rankings change under OCTAVE, illustrating the actual spatial differences responsible for these ranking changes?

Reviewer_dXwiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Motivation:

Spatial gene expression prediction is typically evaluated using mean per-gene PCC, which may reward coarse tissue patterns without reflecting how accurately models recover the fine-grain expression variation. The paper constructs image-derived domain predictors and shows that a learnable variant achieves ~80% of a trained ridge model’s PCC on HEST. Then they proposed OCTAVE, a scale-resolved evaluation framework with graph diffusion to measure prediction accuracy at different spatial scales.

Experiments on Xenium show larger model differences at finer scales, and OCTAVE changes 261 of 1.5k pairwise rankings across multiple image encoders.

Critical strengths

1. The problem is well motivated and setting is reasonable

I think it is meaningful to distinguish between recovering broad expression patterns and recovering local spatial variation. A single aggregate correlation cannot identify which spatial components account for predictive agreement. This matters when models are described as recovering cellular organization or producing expression maps below the original measurement pitch.

The paper also presents OCTAVE as a complement to PCC, which is appropriate to me.

**2. The learnable controls is a good strengthen to the domain experiment **

I agree that a high score from an oracle that uses test expression would be limited evidence on its own. The authors address this directly with training-derived cluster means and a variant that also learns the partition from training data.

**3. Extensive evaluation **

The paper covers many encoders and examines sensitivity to variables like ridge penalty, gene count, cluster count, graph neighborhood, lazy factor, and clustering seed.

Critical weaknesses

1. Missing reference and weaken novelty

The most important missing reference is Haviv et al., “The covariance environment defines cellular niches for spatial inference”, Nature Biotechnology 2025. This work introduces the multiscale spectral similarity index, MSSI, for evaluating spatial-expression predictions. It constructs **progressively coarsened representations ** using a spatial cell graph, computes structural similarity at multiple scales, and aggregates the results. Its motivation explicitly includes the limitations of using correlation for assessing spatial patterns, since the evaluation metric operates on the predicted and measured expression maps, independently of how predictions were generated.

One possible way to address this concern is to compare the OCTAVE with MSSI with per-scale components on the controlled degradations and representative encoder comparison.

2. Interpretation of the shortfall ratio.

The reported ratio between fine-grain and overall PCC shortfalls is strongly influenced by the evaluation setup. Since the domain oracle assigns constant expression within each image cluster, it cannot capture within-cluster variation and has very low finest-band correlation. Therefore the ratio is highly sensitive to the scalar shortfall and choices such as cluster count and graph construction Tab. 6-7. More importantly, increasing ridge regularization raises the ratio from 3.90 to 5.14, even though the model's advantage over the oracle decreases in Tab. 6. So a larger ratio does not necessarily indicate better fine-scale prediction.

3. Few numerical inconsistencies

Firstly, Tab. 1 reports a mean model PCC of 0.2882 across 57 encoders. However, Fig. 5a labels the baseline as the HEST benchmark’s 72-sample mean, using log1p targets “as scored,” and reports a 0.2537.

Secondly, I notice the Tab. 1 gives encoder span 0.1177 while the Appendix K says 0.121.

4. It is not clear that counting-noise conclusions remain valid when reliability is handled gene by gene

Questions for authors
  1. Do the main results remain stable when isolated nodes preserve constants?
  2. What does OCTAVE reveal beyond MSSI and its per-scale scores?
Reviewer_4JXqOpenReview ↗
Rating
1
Confidence
2
Informativeness
3
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces OCTAVE, a scale-resolved evaluation framework for spatial gene-expression prediction from histology. It argues that conventional Pearson correlation (PCC) can obscure fine-scale prediction errors. OCTAVE applies graph-diffusion filters at different spatial scales and evaluates prediction–measurement agreement within the resulting bands. Experiments across Xenium, Visium HD, and HEST data show larger fine-scale performance differences and changes in the rankings of some pathology encoders.

Critical strengths

• Relevant problem: The paper identifies a meaningful limitation of conventional PCC-based evaluation. • Extensive analysis: Experiments span multiple datasets, spatial resolutions, and 57 image encoders, with useful robustness and measurement-noise controls.

Critical weaknesses
  1. Insufficient validation of the proposed evaluation framework. Although adapting established multiscale techniques to a new evaluation problem can be a valid contribution, the authors do not convincingly establish why OCTAVE should be preferred over simpler alternatives, such as Gaussian multiscale filtering or existing spatially aware similarity metrics. The experiments demonstrate that model rankings change, but different rankings are not necessarily better rankings. There is insufficient independent biological validation showing that OCTAVE better reflects biologically meaningful spatial expression recovery or downstream utility.
  2. Limited generality and unclear biological interpretation. The main ranking analysis uses frozen image encoders with a shared ridge-regression head, rather than systematically comparing contemporary end-to-end prediction models. Of the 261 reported pairwise ranking reversals, only 138 persist in at least 95% of specimen bootstrap resamples. The interpretation of the domain-oracle results also requires caution. Appendix K.4 shows that the image-derived clusters are highly spatially fragmented, meaning they do not necessarily represent contiguous tissue domains. The findings therefore support the importance of morphological cluster identity more directly than the claimed role of coarse tissue organization. Additionally, the spatial widths assigned to diffusion bands depend on the graph and smoothing operator, complicating their interpretation and comparison across platforms.
  3. Poor scientific writing obscures the contribution. The manuscript repeatedly substitutes vague and elaborate phrasing for clear technical explanations. The extensive theoretical framing makes a relatively simple evaluation procedure appear unnecessarily complicated and makes it harder to distinguish established mathematical results from genuinely new contributions.
Questions for authors
  1. What practical advantages does OCTAVE provide over existing multiscale filtering and spatial evaluation metrics?
  2. What evidence establishes that OCTAVE produces more biologically meaningful model rankings, rather than merely different rankings?
  3. How comparable are its spatial-scale scores across graph constructions and sampling resolutions?
  4. Given the highly fragmented image-derived clusters, how do the authors justify interpreting the domain-oracle results as evidence about coarse tissue organization?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.