- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces OCTAVE, a scale-resolved evaluation framework for spatial gene expression prediction from histology images. The authors argue that conventional mean per-gene Pearson correlation (PCC) is insensitive to the spatial scales at which predictions are accurate, potentially rewarding coarse tissue organization while overlooking fine-scale expression patterns.
To address this limitation, OCTAVE uses graph-based diffusion operators to decompose predicted and measured gene expression into spatial frequency bands, evaluating their agreement at different physical scales. The paper also introduces a domain oracle to quantify how much conventional PCC can be explained by coarse tissue-domain information alone.
Experiments on HEST, Xenium, and Visium HD datasets show that coarse domain-level predictions can achieve relatively high PCC, while scale-resolved evaluation reveals larger discrepancies at finer spatial resolutions. Additional experiments across 57 image encoders demonstrate that model rankings can change when evaluated at different spatial scales.
Critical strengths
- Important observation regarding the limitations of conventional PCC-based evaluation.
The paper addresses a meaningful issue in spatial gene expression prediction: high PCC does not necessarily indicate accurate recovery of fine-scale spatial expression patterns.
The domain-oracle analysis provides useful evidence that coarse tissue organization can explain a substantial fraction of conventional prediction scores. This observation is relevant to the design and interpretation of future spatial transcriptomics benchmarks.
- Extensive quantitative evaluation and careful statistical analysis.
The experiments cover multiple spatial transcriptomics platforms, 13 independent Xenium specimens, and 57 image encoders. The authors also investigate sensitivity to graph construction, spatial resolution, clustering configurations, and other evaluation choices.
The observation that scale-resolved evaluation can substantially change relative model comparisons is interesting and highlights the importance of considering spatial structure when interpreting prediction performance.
Critical weaknesses
- The necessity and methodological novelty of introducing a new evaluation metric are insufficiently justified.
This is my primary concern.
Although the paper convincingly demonstrates that conventional PCC is insensitive to spatial scale, this observation does not necessarily establish the need for an entirely new evaluation framework.
The core OCTAVE formulation relies on graph diffusion, differences between successive smoothing operators, and correlations computed within the resulting spatial bands. These components are closely related to established techniques in multiscale signal processing, graph wavelets, spatial filtering, and scale-space analysis.
Indeed, the paper acknowledges related approaches such as diffusion wavelets, spectral graph wavelets, and multiscale image similarity in the Related Work section. However, it does not provide sufficiently direct empirical comparisons demonstrating why OCTAVE should be preferred over simpler alternatives.
For example, applying Gaussian smoothing or graph-based low-pass filtering at multiple scales, followed by high-pass residual correlation, could potentially reveal similar limitations of conventional PCC. Existing spatial diagnostics such as Moran's I may also provide complementary information about oversmoothing, although they do not measure exactly the same quantity.
The paper includes analyses involving spatial autocorrelation, but it does not establish whether the principal findings of OCTAVE are substantially different from those obtainable through these simpler multiscale evaluation strategies.
Furthermore, the reported changes in encoder rankings do not by themselves demonstrate that OCTAVE produces more biologically meaningful or practically useful model selections. A different ranking is informative only if the new criterion better captures spatial structures relevant to downstream scientific objectives.
I therefore remain unconvinced that introducing OCTAVE as a distinct evaluation metric is sufficiently necessary or offers a substantial methodological advance over established multiscale evaluation approaches.
A direct comparison with simpler scale-conditioned correlations or graph-wavelet-based evaluation, together with evidence of improved biological interpretability or practical model selection, would considerably strengthen the paper.
- Insufficient qualitative visualization limits the interpretability and practical validation of the proposed evaluation framework.
Despite the central emphasis on fine-scale spatial expression patterns, the paper provides surprisingly limited qualitative comparisons of actual spatial gene expression predictions.
Figure 1 presents only two selected Visium HD colon windows, chosen to highlight large finest-band discrepancies. The remaining figures primarily focus on quantitative curves, statistical summaries, and sensitivity analyses.
Several limitations arise from this presentation.
First, the paper does not systematically visualize ground-truth and predicted spatial expression patterns across the Xenium specimens that form the basis of its main fine-scale analysis. Representative examples from different tissues and individual genes would help demonstrate how the proposed metric relates to actual spatial prediction errors.
Second, the paper reports that OCTAVE changes the relative rankings of 261 out of 1,596 encoder pairs, yet it provides no direct qualitative comparisons illustrating why models with similar conventional PCC receive substantially different fine-scale scores. Side-by-side spatial predictions from representative encoder pairs would be particularly valuable.
Third, Figure 1 primarily visualizes the first principal component of a gene-expression panel rather than individual marker-gene expression maps. While this illustrates the overall spatial structure, it limits the biological interpretability of the observed differences.
Finally, selecting windows with particularly large discrepancies is useful for motivating the method but does not establish that the illustrated differences are representative of typical cases. Including randomly selected examples, representative cases, and failure cases would provide a more balanced assessment.
For a paper whose main contribution is an evaluation framework intended to reveal otherwise overlooked spatial structures, qualitative validation is particularly important. Without sufficiently representative spatial comparisons, it is difficult to assess whether differences in the proposed metric correspond to meaningful biological improvements rather than alternative mathematical characterizations of spatial variation.
Overall, the lack of systematic qualitative evidence weakens the practical justification for adopting OCTAVE as a new evaluation standard.
Questions for authors
- Necessity of a new evaluation metric
The proposed method builds upon established graph diffusion and multiscale signal-processing techniques. Could the authors clarify what OCTAVE provides beyond simpler approaches such as Gaussian high-pass correlation, scale-conditioned PCC, or graph-wavelet-based spatial evaluation?
Are direct comparisons with such alternatives available? In particular, would these simpler methods lead to similar conclusions about coarse-domain dominance and changes in encoder rankings?
- Biological and practical significance
The paper reports substantial differences in fine-scale correlations and changes in encoder rankings. However, how can we establish that these differences correspond to more biologically meaningful spatial expression predictions?
Is there evidence that selecting models according to OCTAVE leads to better preservation of marker-gene localization, tissue boundaries, or other downstream biological structures compared with selecting models using conventional PCC or existing spatial metrics?
- Qualitative visualization
Why are the qualitative spatial comparisons limited to two selected Visium HD windows?
Could the authors provide representative Xenium examples showing ground-truth expression, domain-oracle predictions, and trained-model predictions for individual marker genes across multiple tissues?
Additionally, could the authors visualize representative encoder pairs whose rankings change under OCTAVE, illustrating the actual spatial differences responsible for these ranking changes?