acceptodds
Under review as a conference paper at ICLR 2027

Prototype-Driven Fusion of Pathology and Spatial Transcriptomics for Interpretable Multiple Instance Learning

Abstract

Spatial transcriptomics (ST) measures gene expression in situ, complementing tissue morphology with spatially resolved molecular information. Together, these modalities provide rich morphological and molecular information, but identifying and integrating signals relevant to diagnosis and prognosis remains challenging under limited sample-level supervision. We introduce PathoSpatial, a multimodal multiple instance learning (MIL) framework integrating co-registered H&E images and ST through task-guided prototype learning. By combining supervised adaptation and cohort-level updates, modality-specific prototype banks capture recurring morphological and molecular patterns, which are then fused through gated attention for prediction. The prototypes make each prediction interpretable: we characterize them biologically and use Owen values to attribute each prediction to prototypes grouped by modality, linking it to the histological patches and transcriptomic spots behind it. Across three cohort-level ST datasets spanning survival prediction, treatment-response classification, and disease-status classification, PathoSpatial provides this interpretability while performing comparably to the strongest multimodal baselines, supporting the investigation of patient-specific patterns for precision medicine.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
computational pathology, spatial transcriptomics, survival prediction, multimodal learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_d6GHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces PathoSpatial, a multimodal MIL framework for patient-level prediction from co-registered H&E slides and spatial transcriptomics. The model uses UNI2 for pathology slides and scVI features for ST data, incorporates spatial context, and aggregates each modality through task-guided prototypes. Then, a gated attention fuses the prototype representations for prediction. Biological characterisation and modality-grouped Owen values provide prototype-level interpretations of predictions. The framework is evaluated on three public cohorts covering treatment-response classification, survival prediction, and lung-disease classification, with great performance.

Critical strengths
  1. It is a well-motivated work for computational pathology, most of the previous papers focused on predicting ST expression from H&E, but only a few of them combined them together for patient-level prediction. The two modalities capture complementary morphological and molecular information, and the proposed framework provides a structured approach to integrating these signals while incorporating local tissue context.

  2. The interpretation part is also interesting. Modality-specific prototypes provide a compact intermediate representation that can be linked back to histological patches and transcriptomic spots. Their characterisation through tissue annotations, cell types, and pathway enrichment can give us biological context, while modality-grouped Owen values decompose predictions into prototype and modality contributions.

Critical weaknesses
  1. Most of the results in this paper are reported at the epoch with the highest AUC across pooled leave-one-out predictions. Which means the labels of the test set are being used to select the checkpoint, introducing optimistic model-selection bias. Applying this procedure to all methods are fair but still not a proper setting for independent evaluation. Same for the survival prediction, the paper similarly reports each fold’s best epoch without clearly specifying an independent validation set. Checkpoint selection should use training-side validation only, followed by evaluation on unseen test sets.

  2. The improvement evidence is weak for current datasets. The three cohorts contain only 44, 92, and 35 patients, with no external validation. The default GNN model improves TNBC Response AUC from 0.656 for ST alone to 0.680, but achieves the same accuracy. For survival, fusion exceeds the stronger single-modality model on only 2/5 endpoints. Lung Disease is particularly uninformative for assessing fusion: all methods achieve AUC 1.000, and ST alone achieves higher accuracy than the fused model. A better evaluation dataset, or harder tasks, are definitely needed to provide more convincing evidence.

  3. I am also concerned about whether paired H&E and ST are necessary for this proposed method. Although the framework is motivated by co-registered H&E and ST, it aggregates the modalities separately into prototype representations before fusion and does not explicitly model interactions between corresponding patches and ST spots. Spatial context within each modality does not establish the benefit of cross-modal alignment. Co-registration may support biological interpretation, but not really for prediction. The patient-level endpoints could potentially be addressed using patient-matched but unregistered modalities, which would be easier for data curation. Spatially resolved tasks such as multimodal niche identification using matched morphological and molecular cues would more directly test whether the framework benefits from local cross-modal correspondence.

Questions for authors

Can the authors clarify the patient-level training, validation, and test partitions, and report results with checkpoint selection performed exclusively on validation data rather than pooled held-out AUC? Please also clarify whether scVI fitting and data-dependent feature selection are only performed on the training folds.

Can the authors evaluate a spatially resolved task, such as multimodal niche identification with independent annotations, comparing H&E alone, ST alone, aligned fusion, and fusion with maybe disrupted cross-modal correspondence while keeping intra-modality spatial structure (basically simulate unpaired H&E, ST data)?

7 comments
AuthorsOpenReview ↗

Response 1/6: General

We thank the reviewer for the careful reading and constructive feedback. We appreciate the recognition that, whereas most prior work predicts ST expression from H&E, PathoSpatial combines the two complementary modalities for patient-level prediction while incorporating local tissue context. We are also grateful that the reviewer found the interpretation component interesting, in particular the modality-specific prototypes that link back to histological patches and transcriptomic spots, their biological characterization, and the modality-grouped Owen-value decomposition. We take the soundness concerns seriously and address each point below.

AuthorsOpenReview ↗

Response 2/6: W1

W1. Checkpoint selection. Thank you for this point. We agree that selecting the checkpoint on held-out performance makes the reported values optimistic, and that applying the same rule to all methods is consistent but does not amount to an independent evaluation. Below we clarify the protocol, explain why we used it, and describe how we have revised the paper.

Protocol. All methods (PathoSpatial and every baseline) use the same patient-level splits and the same budget of 30 training epochs. For TNBC Response and Lung Disease (leave-one-out cross-validation), we report the epoch with the highest AUC on the pooled held-out predictions. For TNBC Survival, each of the five outer folds is trained separately, the epoch with the highest C-index on that fold's held-out patients is selected, and the reported value is the mean ± SD over the five folds of these per-fold C-indices. There is no separate inner validation set (see also our answer to Q1).

Why we used this protocol. As noted in the Patho-Bench benchmark [14], some pathology tasks "have a small number of samples, which may make a validation set impractical." This applies to our cohorts (92, 44, and 35 patients): an inner 20% split would leave about 15 survival patients per validation set, spread over four survival-time bins with few events in each, and about 9 and 7 patients in the two classification cohorts, so the selection signal itself would be unstable [6]. Without a validation set, two conventions are common in multimodal survival prediction: reporting the best held-out epoch, as in PIBD [1], or reporting a fixed final epoch, as in MCAT [2], PORPOISE [3], SurvPath [4], and PANTHER [5]. The paper follows the first, because histology-only, expression-based, and multimodal models converge at different rates, so a single fixed epoch would favor some architectures over others. We now also report the second.

Last-epoch results. We also evaluated all methods at the final (30th) epoch, a fixed choice that uses no held-out information (new App. F.3, Table 9). PathoSpatial remains among the top methods: PathoSpatial-GNN has the second-highest mean survival C-index and the highest C-index on IDFS and DRFS, and PathoSpatial-Attn has the second-highest TNBC Response AUC. Neither protocol is ideal at this cohort size: best-epoch selection is optimistic, and a fixed final epoch is sensitive to differences in convergence, as some models may be heavily overfitted by the last epoch. Under both, PathoSpatial ranks first or second on mean survival C-index and TNBC Response AUC, and when second it trails the best method by 0.006, less than the seed-to-seed SD (App. F.1). This matched performance is what our interpretability claim requires.

Method (last epoch)Response AUCResponse ACCSurvival mean C-index
CLAM0.5870.5910.592
TransMIL0.6250.5910.613
Patch-GCN0.5600.5230.588
PANTHER0.5980.6360.536
PORPOISE0.5650.5230.567
MCAT0.4990.5680.560
SurvPath0.5120.5450.562
PIBD0.5420.5000.488
ProSurv0.5390.5680.569
PathoSpatial-Attn0.6190.5680.572
PathoSpatial-GNN0.5960.5680.607

Bold: best; italic: second best.

Limitation. As the reviewer notes, best-epoch selection is not an independent evaluation. Choosing the best of 30 epochs on the evaluation patients inflates the reported metrics, as shown for small-sample settings in [7, 8], and a shared rule does not guarantee unbiased rankings, because methods whose held-out performance varies more across epochs gain more from the selection. For this reason, our claims do not rest on performance margins. The paper claims performance comparable to the strongest baselines together with prototype- and modality-level interpretability, and our Conclusion already states that the cohorts are too small to distinguish methods. We do not claim superior generalization or transfer to unseen cohorts. Nested selection within each training fold, together with independent external cohorts, is the appropriate next step as larger paired ST–H&E cohorts become available.

Revisions.

  • Sec. 4 and App. B.1 now describe the selection rule explicitly as held-out best-epoch selection without an inner validation set (Sec. 4, lines 350–351; App. B.1, lines 1105–1113).
  • The Limitations now state that, because the cohorts are too small for an inner validation split, the epoch was selected on held-out patients for all methods, following prior work, and that nested selection is left to larger cohorts (Sec. 6, lines 536–538).
  • New App. F.3 (Table 9) reports all methods at the final epoch, without epoch selection; Sec. 4 points to it.
AuthorsOpenReview ↗

Response 3/6: W2

W2. Weak evidence of improvement. We thank the reviewer for this assessment and agree with the main point. We do not claim a performance gain: as stated in Sec. 6, with 92, 44, and 35 patients the cohorts are too small to distinguish methods. Our claim is interpretability at performance comparable to the strongest baselines. This ranking is robust: over five random seeds, PathoSpatial-GNN keeps the highest mean survival C-index (0.690 ± 0.009, versus 0.681 ± 0.014 for SurvPath) and the highest TNBC Response AUC (0.722 ± 0.063; Appendix F.1, Table 7). With CONCH v1.5 in place of UNI2, it again leads on both (mean C-index 0.678, AUC 0.672; Appendix F.2, Table 8). The survival margin is within about one seed SD, consistent with matched rather than superior performance. Likewise, in paired bootstrap comparisons with the strongest baselines, SurvPath and Patch-GCN (new App. F.5, Table 11), the differences are within the uncertainty: the 95% CIs of all survival C-index differences include zero.

Fusion across tasks. We agree that the benefit of fusion is not uniform: on TNBC Response, fusion improves AUC but not accuracy, and on TNBC Survival, it helps on some endpoints but not others (Table 2). We attribute this more to the tasks and data than to the model. The predictive information may be unevenly distributed between the two modalities and may differ across endpoints and patients; TNBC is notoriously heterogeneous; survival endpoints are censored and influenced by factors beyond the tumor sample (e.g., treatment and follow-up); and the cohorts are small, so per-endpoint differences are noisy.

Data availability. Better cohorts are indeed needed, but they are hard to obtain at present. ST profiling remains costly, about USD 5,000–10,000 per sample, and outcome labels require years of clinical follow-up: TNBC Survival comprises patients treated between 2000 and 2016, with about five years of follow-up. Patient backgrounds, such as disease stage and treatment, must also be well controlled or matched, which is itself difficult. Complete, high-quality cohorts that pair H&E, ST, and patient-level outcomes are rare. To our knowledge, the three cohorts used here are among the few currently accessible ones. To confirm this, we systematically evaluated more than 30 independent published studies with ST data and more than 20 samples, as well as the HEST-1k [12] and STimage-1K4M [13] collections, and could not identify further suitable training data: these data often lack high-resolution H&E images or clean clinical variables, or have poorly controlled confounding variables, such as patient background.

Role of Lung Disease. We agree that Lung Disease is an easy task and does not by itself demonstrate the benefit of fusion. We include it as a control for a different granularity of signal in comparative analysis. In Lung Disease, the signal is largely at the patch level: whether fibrotic tissue is present. PathoSpatial is primarily designed for sub-patch signals, such as subtle pathway changes within tumor- or immune-rich regions, as demonstrated in the new cohort-level analysis (Sec. 5.2, "Cohort-level prototypes and spatial biomarker discovery"; Fig. 4; App. I): in TNBC Response, such signals persist after adjusting for regional cell-type composition. Lung Disease checks that the framework also handles coarse, patch-level signals without loss: both PathoSpatial variants reach AUC 1.000.

Reviewer_mmywOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PathoSpatial, a multimodal multi-instance learning framework that integrates histopathological images and spatial transcriptomics for patient-level clinical outcome prediction. The approach uses a spatial encoder to extract contextual features from patterns and learning task-guided protocols through supervised training and EMA updates. The resulting prototype represents the use of gated attention for survival prediction and classification. The authors also use prototype representations and Irwin values to explain model predictions.

Critical strengths

The paper addresses an important problem in computational pathology by integrating histopathology and spatial transcriptomics for clinical outcome prediction. The use of spatial encoders and task-guided prototype learning is also well motivated.

Critical weaknesses
  1. The framework is largely built upon existing techniques. While their combination is reasonable, the main methodological contribution is not entirely clear.

  2. Several relevant methods are missing from the discussion, e.g. [1-3], etc. These studies are closely related to different aspects of PathoSpatial, but the paper does not clearly position its contributions against them.

[1] ProtoSurv: A Multimodal Survival Prediction Framework Based on Prototype Learning and Gated Hierarchical Attention. [2] STPath: A Generative Foundation Model for Integrating Spatial Transcriptomics and Whole-Slide Images. [3] HAGE: Hierarchical Alignment Gene-Enhanced Pathology Representation Learning with Spatial Transcriptomics.

  1. A key motivation for this work was to integrate spatially aligned histopathology and transcriptomics. However, both modes are minimally processed through separate encoders and prototype libraries before being combined with gated attention. This makes me question whether the model actually explains the spatial correspondence between image patches and ST points, or benefits primarily from two complementary sources of information. Current experiments do not seem to distinguish between these possibilities

  2. Interpretability: The prototype visualizations and pathway enrichment analyses are interesting, but I am not fully convinced that they establish the biological interpretability of the learned representations. Owen values explain how different prototypes contribute to a prediction; they do not necessarily demonstrate that those prototypes capture distinct or biologically meaningful tissue patterns. Most of the presented evidence also comes from selected patient examples.

Questions for authors

Same as above

4 comments
AuthorsOpenReview ↗

Response 1/2: W1 to W3

We thank the reviewer for the careful reading and constructive feedback, and for recognizing PathoSpatial's effort to bring emerging histology–omics data toward clinical translation and the motivation behind its spatial encoders and task-guided prototype learning. We address each concern below.

W1. Methodological contribution. Thank you for raising this. We agree that several components build on established techniques. Our main methodological contribution is a hybrid prototype learning strategy, in which supervised gradients and an epoch-level EMA jointly refine prototypes shared across patients, keeping them anchored to recurring data patterns while aligning them with the task. The prototypes form a compact, traceable intermediate representation that makes exact attribution tractable: with 24 prototypes in two modality groups, exact Owen values need only about 2142^{14} evaluations of the fusion head, whereas exact attribution over thousands of instances is intractable (Sec. 3.4).

W2. Missing related work. We thank the reviewer for these references and now cite them in Sec. 2. They belong to two lines of work that PathoSpatial builds on, not to the same task. STPath [2] and HAGE [3] predict spatial gene expression from H&E, by generative pretraining and contrastive image–expression alignment respectively, so that ST is not needed at inference; we add them to the histology–ST section. ProtoSurv [1] predicts survival from H&E and bulk genomics using prototypes obtained by unsupervised per-slide clustering (K-means/GMM); we add it to the unsupervised prototype methods. PathoSpatial differs from both: it uses measured, co-registered ST for patient-level prediction, and learns prototypes that are shared across slides, task-guided, and defined for both modalities. ProtoSurv has no public code release, so we do not include it as a baseline.

W3. Spatial correspondence. Spatial correspondence. We thank the reviewer for this question. The reviewer is correct: PathoSpatial benefits primarily from two complementary sources of information and does not model spatial correspondence between patches and spots. Each modality is encoded and aggregated into prototypes in its own branch, and the two branches meet only at the prototype level through gated attention. The forward pass never links a specific patch to its paired spot.

Architectural explanation. In PathoSpatial-GNN, the H&E branch receives patch features and H&E coordinates, and the ST branch receives spot features and ST coordinates. Each branch builds its own kNN graph and runs its own context encoder, and fusion operates on prototype-level representations, not on instance pairs. Because the two branches share no instance-level information, predictions do not depend on which H&E instance corresponds to which ST instance. PathoSpatial-Attn follows the same design, except that its global encoder arranges each modality's instances in a square grid for positional encoding, so each branch's layout depends on its slide orientation. We show this with a rotation test: we rotate only the histology branch coordinates by 90$^\circ$ and 180$^\circ$ while keeping the ST branch intact, so that the two modalities' layouts no longer agree (new App. F.4, Table 10):

PathoSpatial-AttnResponse AUCResponse ACCSurvival mean C-index
No rotation (Table 1)0.6910.6590.673
Histology 90$^\circ$0.6740.6590.646
Histology 180$^\circ$0.7180.6820.654

The effects are small and inconsistent: TNBC Response AUC decreases at 90$^\circ$ and increases at 180$^\circ$, and every survival C-index change is smaller than its fold-level SD. This is consistent with the architecture: the prediction does not rely on spatial correspondence between the two modalities.

AuthorsOpenReview ↗

Response 2/2: W4 to References

W4. Interpretability. We thank the reviewer for raising this. We agree that Owen values attribute a prediction to prototypes but do not by themselves show that the prototypes are distinct or biologically meaningful, and that selected patients cannot establish this.

To move beyond selected patients, we added a cohort-level analysis of the prototypes (Sec. 5.2, "Cohort-level prototypes and spatial biomarker discovery"; Fig. 4; Appendix I). On TNBC Response, we train one PathoSpatial-GNN on all 44 patients for 30 epochs (final checkpoint; training AUC 0.99) and, for each expression prototype, analyze its 100 most-attended patches in every patient, using the cohort's published cell-type annotation and the 50 Hallmark pathways. Beyond interpreting the model, this analysis demonstrates prototype-enabled spatial biomarker discovery, a step from explanation toward clinically actionable readouts.

Identity. Each prototype selects a characteristic tissue context relative to the patient's own slide, e.g., vascular/CAF-rich (P03; in 93%/80% of patients), CAF-rich (P01; 86%), and TAM-rich regions (P05; 75%) (Fig. 4A). Distinctness. Same-prototype regions from different patients are closer in composition than regions of different prototypes (pooled ratio 1.07, p<0.001 against within-patient label permutation); 6 of 12 prototypes are individually significant, and the rest form families of similar composition, suggesting that K=12 partly over-partitions (Fig. 4B). Composition is only an approximate test of distinctness: prototypes are defined in the learned feature space and may separate on expression programs or cell states that a 15-type cell count does not resolve. It is, however, the closest ground truth available, so we use it as a conservative check. Task signal. The pCR-versus-RD effect of pathway scores peaks in a prototype's top 50–200 patches and fades toward the whole-slide value as more patches are added, while random patches stay at the whole-slide value (Fig. 4C); the prototypes thus localize where response-related signal lies. Prototype-enabled spatial biomarker discovery. Measuring a pathway only within a prototype's region separates responders better than measuring it elsewhere: IL6–JAK–STAT3 in P08 reaches AUC 0.93 in the prototype's top-100 patches, versus 0.66 on the whole slide and 0.66 (range 0.63–0.71 over 20 draws) on 100 randomly selected patches. The same pattern holds for all four candidates (prototype region 0.86–0.93; random patches 0.66–0.74), so the gain comes from which patches are selected, not from using fewer of them. The association persists after adjusting for regional immune and tumor fractions (p=0.011; Fig. 4D–E). All four candidates are immune programs higher in responders, consistent with tumor-infiltrating lymphocytes as an established predictor of pCR. Clinical translation. Prototypes pinpoint the task-relevant patches within a slide, and a biomarker measured in those patches is more informative than the same biomarker averaged over the tissue. A prototype thus specifies where to measure and a pathway what to measure, which could support better-informed clinical decisions, such as selecting patients for neoadjuvant therapy, once validated in larger cohorts.

These analyses are in-sample, because prototype indices do not correspond across cross-validation folds: the AUCs describe what the model learned, not predictive performance (Sec. 5.2). In addition, we now show the held-out predictions for all 44 TNBC Response patients and all methods, including incorrect predictions (new App. H.5, Fig. 8).

References

[1] M. Wang, N. Wang, S. Han, Y. Xue, and C. Zhang. ProtoSurv: A multimodal survival prediction framework based on prototype learning and gated hierarchical attention. Expert Systems with Applications, 311:131391, 2026.

[2] T. Huang, T. Liu, M. Babadi, R. Ying, and W. Jin. STPath: A generative foundation model for integrating spatial transcriptomics and whole-slide images. npj Digital Medicine, 8:659, 2025.

[3] T. M. Dang, H. Li, Y. Guo, H. Ma, F. Jiang, Y. Miao, Q. Zhou, J. Gao, and J. Huang. HAGE: Hierarchical alignment gene-enhanced pathology representation learning with spatial transcriptomics. In MICCAI, LNCS 15960, pp. 228–238, 2025.

I appreciate the authors' detailed response. The additional experiments are robust and effectively address my concerns regarding interpretability.

However, I still have two reservations concerning the core design of the model:

Regarding spatial correspondence (W3): Thank you for confirming that the model does not leverage cross-modal spatial correspondence. Given that the fusion process essentially treats the data as non-registered, multi-modal input, the emphasis in the title and introduction on "integrating spatially aligned data" is somewhat misleading.

Regarding methodological innovation (W1): The use of gradients combined with EMA (Exponential Moving Average) to update prototypes is a fairly standard optimization technique. In the absence of cross-modal spatial interaction, the overall methodological innovation of the framework remains limited.

I will maintain my current score.

AuthorsOpenReview ↗

Thank you.

We thank the reviewer for their careful reading of our manuscript and for the constructive suggestions. We found the comments very helpful, and addressing them has substantially improved the clarity and quality of the paper.

Reviewer_bprROpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces PathoSpatial, a multiple instance learning framework for integrating HE and ST. Separate prototype banks summarize each modality, with prototypes updated through supervised gradients and an epoch-level EMA. The resulting representations are combined using gate attention. Experiments cover treatment response, survival, and pulmonary fibrosis classification (which is an interesting choice over other available benchmarks in the field). Predictive performance is broadly comparable to existing approaches, with interpretability presented as the main contribution. The application is relevant, but concerns about model selection and the limited validation of the explanations prevent me from supporting acceptance at this time.

Critical strengths

a, The paper addresses a relevant problem: learning from paired H&E and ST with limited patient-level supervision. This could also be a general framework for future H&E-ST-SP pairing which is much needed.

b, All methods use the same UNI2 features, and the spot-level baselines also share scVI features. Patient-level splitting appropriately keeps sections from the same patient within one fold.

c, The ablations examine several useful design choices. The supplement is also reasonably clear about limitations in the biological interpretation and attribution method.

Critical weaknesses

1, Model selection needs an independent validation step see App. B.1 selects the classification epoch using the highest pooled held-out AUC, which introduces selection bias into the reported performance. Survival checkpoint selection also needs clarification. Epoch selection should be confined to the training portion of each outer split, followed by evaluation on untouched patients.

2, Mean survival C-index is 0.693 vs 0.689 for SurvPath, and the best response AUC is 0.691 vs 0.686 for Patch-GCN i.e. added predictive value remains uncertain. All methods reach AUC 1.000 on lung disease (the task is likely quite easy and not the best benchmark?). These small differences, reported with one seed, provide limited evidence of improvement. Paired uncertainty estimates and repeated runs under the corrected evaluation protocol would make the comparisons more convincing.

3, The patient-level analyses in app. G use one selected successful prediction per cohort this interpretability needs systematic validation. This illustrates the method but does not establish the reliability of its explanations. Evaluation across heldout patients, including incorrect predictions and prototype stability across folds would strengthen the central contribution. The enrichment tests should also account for spatial auto correlation, which the current permutation procedure ignores.

4, App. G.2 includes eight hazard outputs although only four are supervised, with the remaining hazards fixed at 0.5. This adds a patient-dependent contribution that may change rankings. The single-patient attribution check does not resolve its effect on predictive performance. Reporting C-indices and patient-ranking agreement using only the supervised outputs would address this concern.

5, The gradient-only ablation supports adding EMA, but does not establish whether supervised adaptation improves on EMA-only updates or fixed prototypes. Adding these controls under the same architecture would directly test the motivation for the hybrid scheme and clarify its contribution beyond simpler prototype construction.

6, All histology experiments use UNI2, repeating key comparisons with CONCH v1.5, or other encoders including the strongest baselines and H&E-only models under matched splits, would establish whether the predictive differences and prototype interpretations persist across feature representations. This is particularly relevant because interpretability is the main claimed contribution.

Questions for authors

a, Was scVI fitted separately within each training fold or using the complete cohort? Which preprocessing and gene-selection steps had access to held-out samples?

b, How are all zero columns handled in the column normalization preceding eq15? top-k masking leaves instances selected by no prototype for which the stated normalization has a zero denominator.

c, What does the restriction in app G.3 to checkpoints that reproduce recorded predictions mean? Can all predictions underlying tab. 1 be reproduced from the saved models?

10 comments
AuthorsOpenReview ↗

Response 1/8: General

We thank the reviewer for the careful reading and constructive feedback. We appreciate the recognition that PathoSpatial could serve as a general framework for future H&E–ST–SP integration; as paired histology–omics datasets grow in scale, such frameworks hold considerable promise for clinical translation. We are also grateful that the reviewer acknowledged our controlled experimental design, and patient-level splitting, as well as the ablation studies and the discussion and limitations. We address each concern below.

AuthorsOpenReview ↗

Response 2/8: W1

W1. Model selection. Thank you for raising this. We agree that selecting the epoch on held-out performance makes the reported values optimistic. Below we clarify the protocol, explain why we used it, and describe how we have revised the paper.

Protocol. All methods use the same patient-level splits and the same budget of 30 training epochs. For TNBC Response and Lung Disease (leave-one-out cross-validation), we report the epoch with the highest AUC on the pooled held-out predictions. For TNBC Survival, each of the five outer folds is trained separately, the epoch with the highest C-index on that fold's held-out patients is selected, and the reported value is the mean ±\pm SD over the five folds of these per-fold C-indices. The four survival-time bins used by the loss are defined on each fold's training patients only. There is no separate inner validation set.

Why we used this protocol. As noted in the Patho-Bench benchmark [14], some pathology tasks "have a small number of samples, which may make a validation set impractical." This applies to our cohorts (92, 44, and 35 patients): an inner 20% split would leave about 15 survival patients per validation set, spread over four survival-time bins with few events in each, and about 9 and 7 patients in the two classification cohorts, so the selection signal itself would be unstable [6]. Without a validation set, two conventions are common in multimodal survival prediction: reporting the best held-out epoch, as in PIBD [1], or reporting a fixed final epoch, as in MCAT [2], PORPOISE [3], SurvPath [4], and PANTHER [5]. The paper follows the first, because histology-only, expression-based, and multimodal models converge at different rates, so a single fixed epoch would favor some architectures over others. We now also report the second.

Last-epoch results. We also evaluated all methods at the final (30th) epoch, a fixed choice that uses no held-out information (new App. F.3, Table 9). PathoSpatial remains among the top methods: PathoSpatial-GNN has the second-highest mean survival C-index and the highest C-index on IDFS and DRFS, and PathoSpatial-Attn has the second-highest TNBC Response AUC. Neither protocol is ideal at this cohort size: best-epoch selection is optimistic, and a fixed final epoch is sensitive to differences in convergence, as some models may be heavily overfitted by the last epoch. Under both, PathoSpatial ranks first or second on mean survival C-index and TNBC Response AUC, and when second it trails the best method by 0.006, less than the seed-to-seed SD (App. F.1). This matched performance is what our interpretability claim requires.

Method (last epoch)Response AUCResponse ACCSurvival mean C-index
CLAM0.5870.5910.592
TransMIL0.6250.5910.613
Patch-GCN0.5600.5230.588
PANTHER0.5980.6360.536
PORPOISE0.5650.5230.567
MCAT0.4990.5680.560
SurvPath0.5120.5450.562
PIBD0.5420.5000.488
ProSurv0.5390.5680.569
PathoSpatial-Attn0.6190.5680.572
PathoSpatial-GNN0.5960.5680.607

Bold: best; italic: second best.

Limitation. We nevertheless agree that best-epoch selection is not an independent evaluation. Choosing the best of 30 epochs on the evaluation patients inflates the reported metrics, as shown for small-sample settings in [7, 8]. Applying the same rule to all methods keeps the procedure consistent, but it does not guarantee unbiased rankings, because methods whose held-out performance varies more across epochs gain more from the selection. For this reason, our claims do not rest on performance margins. The paper claims performance comparable to the strongest baselines together with prototype- and modality-level interpretability, and our Conclusion already states that the cohorts are too small to distinguish methods. We do not claim superior generalization or transfer to unseen cohorts. Nested selection within each training fold, together with independent external cohorts, is the appropriate next step as larger paired ST–H&E cohorts become available.

Revisions.

  • Sec. 4 and App. B.1 now describe the selection rule explicitly as held-out best-epoch selection without an inner validation set, including the per-fold procedure for survival (Sec. 4, lines 350–351; App. B.1, lines 1105–1113).
  • The Limitations now state that, because the cohorts are too small for an inner validation split, the epoch was selected on held-out patients for all methods, following prior work, and that nested selection is left to larger cohorts (Sec. 6, lines 536–538).
  • New App. F.3 (Table 9) reports all methods at the final epoch, without epoch selection; Sec. 4 points to it.
AuthorsOpenReview ↗

Response 3/8: W2

W2. Performance differences and repeated runs. We thank the reviewer for this assessment and agree that the differences are small. We do not claim a performance gain: as stated in Sec. 6, with 92, 44, and 35 patients the cohorts are too small to distinguish methods, and our claim is interpretability at performance comparable to the strongest baselines.

Lung Disease. We agree that Lung Disease is an easy task and not a strong benchmark for comparing methods. We include it as a control for a different family of spatial tasks in comparative analysis, in which the signal lies at the patch level (whether fibrotic tissue is present). PathoSpatial is primarily designed for sub-patch signals, such as subtle pathway changes within tumor- or immune-rich regions, as demonstrated in the new cohort-level analysis (Sec. 5.2, "Cohort-level prototypes and spatial biomarker discovery"; Fig. 4; App. I), and Lung Disease checks that it also handles such coarse signals without loss.

Benchmark cohorts. Larger and harder benchmarks are indeed needed, but cohorts that pair H&E, ST, and patient-level outcomes are currently rare: ST profiling costs about USD 5,000–10,000 per sample, outcome labels require years of clinical follow-up, patient backgrounds (e.g., disease stage and treatment) must be well controlled or matched, which is itself difficult, and few cohorts are complete and of high quality. To our knowledge, the three cohorts used here are among the few currently accessible ones. To confirm this, we systematically evaluated more than 30 independent published studies with ST data and more than 20 samples, as well as the HEST-1k [12] and STimage-1K4M [13] collections, and could not identify further suitable training data: these data often lack high-resolution H&E images or clean clinical variables, or have poorly controlled confounding variables, such as patient background.

Repeated seeds. As requested, we retrained PathoSpatial-GNN and the strongest baselines, SurvPath and Patch-GCN (add), with five random seeds, keeping the data splits, hyperparameters, and evaluation protocol fixed (new App. F.1, Table 7). Results are mean ±\pm SD across seeds:

MethodResponse AUCResponse ACCSurvival mean C-index
PathoSpatial-GNN0.722 ±\pm 0.0630.665 ±\pm 0.0460.690 ±\pm 0.009
SurvPath0.555 ±\pm 0.0190.545 ±\pm 0.0360.681 ±\pm 0.014
Patch-GCN (add)0.675 ±\pm 0.0260.645 ±\pm 0.0590.664 ±\pm 0.009

The 5-seed means are close to the submitted single-seed values, and the seed-to-seed SD of each survival C-index (≤0.030\leq 0.030) is much smaller than the fold-level SD in Table 1. PathoSpatial-GNN retains the highest mean survival C-index, leading on IBCFS, DRFS, and OS, and the highest TNBC Response AUC and accuracy. Its survival margin over SurvPath is within about one seed SD, so we read it as matched survival performance rather than a significant gain, consistent with our framing of interpretability at matched performance.

Paired uncertainty. To compare PathoSpatial-GNN with the strongest baselines directly, we computed paired bootstrap 95% CIs of the performance difference over the five seeds, with 10,000 resamples of patients and both methods evaluated on the same resamples (new App. F.5, Table 11). Survival differences are not significant against either SurvPath or Patch-GCN (add) on any endpoint, consistent with matched performance.

MetricΔ\Delta [95% CI] (vs. SurvPath)pp (vs. SurvPath)Δ\Delta [95% CI] (vs. Patch-GCN (add))pp (vs. Patch-GCN (add))
Response AUC+0.167 [+0.039, +0.315]0.011+0.047 [$-$0.044, +0.158]0.29
Mean survival C-index+0.009 [$-$0.047, +0.065]0.80+0.026 [-\0.010, +0.046]0.21

Δ\Delta: PathoSpatial-GNN −- comparator, averaged over seeds; per-endpoint results are in Table 11.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.