- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper examines the reliability of spatial prediction evaluations in histology representations. The authors propose a diagnostic framework focusing on three issues: misleading relative performance gains, encoder-specific prediction targets, and overlap between visible tissue and prediction targets. Through experiments across multiple cancer cohorts and pathology encoders, the study shows that spatial prediction gains are sensitive to target construction, evaluation metrics, and tissue accessibility. The authors recommend more rigorous evaluation protocols to better assess spatial information in histology representations.
Critical strengths
- The paper provides a systematic evaluation of spatial prediction in histology representations, examining multiple factors that may influence the interpretation of prediction performance.
- The experiments are relatively comprehensive, covering multiple cancer cohorts, pathology foundation models, and spatial transcriptomics datasets. The analyses of target smoothing, shared prediction targets, and tissue accessibility provide useful empirical insights into existing evaluation practices.
Critical weaknesses
- Insufficient justification of unseen-tissue prediction. The paper does not clearly establish why predicting unseen surrounding tissue is a meaningful measure of spatial representation quality, particularly given the biological heterogeneity of tumor tissues.
- Mismatch with the objectives of prior work. Methods such as TRIPLEX and MERGE aim to improve gene expression prediction using visible spatial context, rather than predicting unseen tissue. The relevance of the proposed evaluation to their original claims is insufficiently justified.
- Questionable interpretation of reduced context gains. Spatial smoothing is a reasonable strategy for handling noisy spatial transcriptomics data. The reported 74–92% reduction in context gains may partly reflect changes in prediction targets and task difficulty, rather than misleading improvements in prior methods.
- Limited significance of the empirical findings. Several findings largely confirm expected effects, such as improved prediction when the target tissue is visible and changes in encoder comparisons when prediction targets differ. The additional scientific insights beyond these established principles remain unclear.
Questions for authors
- Why is predicting unseen surrounding tissue an important way to evaluate pathology representations? Could the authors explain what this task tells us about a model's capabilities and how it relates to practical pathology applications?
- TRIPLEX and MERGE mainly use visible neighboring tissue to improve gene expression prediction. Could the authors clarify which claims of these methods are challenged by evaluating prediction of unseen tissue?
- Spatial smoothing is a reasonable part of the original prediction tasks. Could the authors explain why the reduced gains on raw expression should be considered evidence of potentially misleading improvements, rather than simply reflecting a different prediction target?
- Several results seem expected, such as better prediction when the target tissue is visible. Could the authors explain what new insights their experiments provide beyond these observations, or give a concrete example where the proposed evaluation changes the interpretation or selection of a pathology model?
8 comments
Response to Reviewer qHYY (2/4): smoothed targets and task difficulty (W3/Q3)
We agree that smoothing is a reasonable way to handle noisy spatial transcriptomics, and we do not claim that the prior improvements are incorrect or misleading; Figure 1's "can mislead" refers to how a score is read, not to those results. The question is what a smoothed-target gain can be attributed to. Switching to raw targets changes two things at once, target noise and whether the input contains tissue that enters the target, so we test them separately.
Evidence. Keeping the smoothed target fixed, we mask from TRIPLEX's neighborhood view the adjacent capture areas that enter it. The task and its difficulty are unchanged, yet the gain in mean Pearson falls from 0.062 to 0.014 in ccRCC and from 0.075 to 0.015 in Xenium, reductions of 77% and 80%, to about the size of the raw-target gains (0.016 and 0.011; Table S16).
Scope. Masking applies to TRIPLEX; MERGE has no neighborhood view, and its spatial-edge gain shows the same smoothed-versus-raw pattern under coordinate permutation (Table S18). Target noise explains part of the difference: after reliability normalization, the target-by-view interaction falls from 0.046 to 0.037 in ccRCC and from 0.064 to 0.053 in Xenium, with both intervals above zero (Appendix F.9). A smoothed-target gain therefore measures better prediction of neighborhood-averaged expression, part of which the model can see. It supports a claim about each spot's own expression only together with a control such as masking.
Response to Reviewer qHYY (3/4): which claims of TRIPLEX and MERGE are tested (W2/Q2)
This test involves no unseen tissue. Both methods were retrained with their released code, hyperparameters and inputs, visible context included; we changed only the target, from smoothed to raw spot expression, and for TRIPLEX also masked the neighbors entering the smoothed target (W3/Q3). We do not question their use of visible context or their gains on smoothed targets, which we reproduce. The claim tested is narrower: that these gains show context improves prediction of each spot's own expression.
Evidence. On raw targets, no context gain in either method or cohort has a global interval above zero: TRIPLEX 0.009 [−0.005, 0.021] and 0.008 [−0.008, 0.023], MERGE 0.004 [−0.006, 0.013] and 0.004 [−0.007, 0.016] for ccRCC and Xenium (Table S15). On smoothed targets, all four intervals are above zero.
Scope. Two of the three published context claims replicate and are narrowed; the TRIPLEX whole-slide gain does not replicate on our renal cohort, so we draw no conclusion about it (Table S21). The results come from retraining on our cohorts, not the original data. These results narrow the interpretation of the published gains; they do not overturn the methods on the tasks they were designed for.
Response to Reviewer qHYY (1/4): why withhold the target tissue (W1/Q1)
Thank you for the review. We answer the four questions in turn.
Withholding is a control, not a proposed measure of representation quality. A spatial gain can come from recognizing the tissue in the input, from using additional tissue the model can see, or from inferring tissue outside the input. The first two are legitimate, and for prediction from an available slide they are the right design. Only the third requires the target to be withheld. It is also the claim made in practice when a gain is attributed to spatial context, or when an encoder is described as encoding its neighborhood. Withholding the target is how that claim is tested; failing the test does not mean a model lacks value.
Evidence that separates the two. On the same 320–640 µm ring targets, both local crop widths score below the training mean for every encoder, whereas an MLP on tiles inside the ring reaches R2 = 0.26–0.31 (Section 4.4, Table S23). The target is predictable from H&E; what fails is inference beyond the input. Heterogeneity is handled by design: each true ring is compared with a count-matched far ring from the same slide under the same eligibility rules, so slide-level heterogeneity affects both arms.
Scope. The direct-access result holds with a flexible readout; with ridge, ring tiles also stay below the training mean. A positive true-minus-far contrast shows a predictive association with the local neighborhood, not a mechanism.