- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces PathoSpatial, a multimodal MIL framework for patient-level prediction from co-registered H&E slides and spatial transcriptomics. The model uses UNI2 for pathology slides and scVI features for ST data, incorporates spatial context, and aggregates each modality through task-guided prototypes. Then, a gated attention fuses the prototype representations for prediction. Biological characterisation and modality-grouped Owen values provide prototype-level interpretations of predictions. The framework is evaluated on three public cohorts covering treatment-response classification, survival prediction, and lung-disease classification, with great performance.
Critical strengths
-
It is a well-motivated work for computational pathology, most of the previous papers focused on predicting ST expression from H&E, but only a few of them combined them together for patient-level prediction. The two modalities capture complementary morphological and molecular information, and the proposed framework provides a structured approach to integrating these signals while incorporating local tissue context.
-
The interpretation part is also interesting. Modality-specific prototypes provide a compact intermediate representation that can be linked back to histological patches and transcriptomic spots. Their characterisation through tissue annotations, cell types, and pathway enrichment can give us biological context, while modality-grouped Owen values decompose predictions into prototype and modality contributions.
Critical weaknesses
-
Most of the results in this paper are reported at the epoch with the highest AUC across pooled leave-one-out predictions. Which means the labels of the test set are being used to select the checkpoint, introducing optimistic model-selection bias. Applying this procedure to all methods are fair but still not a proper setting for independent evaluation. Same for the survival prediction, the paper similarly reports each fold’s best epoch without clearly specifying an independent validation set. Checkpoint selection should use training-side validation only, followed by evaluation on unseen test sets.
-
The improvement evidence is weak for current datasets. The three cohorts contain only 44, 92, and 35 patients, with no external validation. The default GNN model improves TNBC Response AUC from 0.656 for ST alone to 0.680, but achieves the same accuracy. For survival, fusion exceeds the stronger single-modality model on only 2/5 endpoints. Lung Disease is particularly uninformative for assessing fusion: all methods achieve AUC 1.000, and ST alone achieves higher accuracy than the fused model. A better evaluation dataset, or harder tasks, are definitely needed to provide more convincing evidence.
-
I am also concerned about whether paired H&E and ST are necessary for this proposed method. Although the framework is motivated by co-registered H&E and ST, it aggregates the modalities separately into prototype representations before fusion and does not explicitly model interactions between corresponding patches and ST spots. Spatial context within each modality does not establish the benefit of cross-modal alignment. Co-registration may support biological interpretation, but not really for prediction. The patient-level endpoints could potentially be addressed using patient-matched but unregistered modalities, which would be easier for data curation. Spatially resolved tasks such as multimodal niche identification using matched morphological and molecular cues would more directly test whether the framework benefits from local cross-modal correspondence.
Questions for authors
Can the authors clarify the patient-level training, validation, and test partitions, and report results with checkpoint selection performed exclusively on validation data rather than pooled held-out AUC? Please also clarify whether scVI fitting and data-dependent feature selection are only performed on the training folds.
Can the authors evaluate a spatially resolved task, such as multimodal niche identification with independent annotations, comparing H&E alone, ST alone, aligned fusion, and fusion with maybe disrupted cross-modal correspondence while keeping intra-modality spatial structure (basically simulate unpaired H&E, ST data)?
7 comments
Response 2/6: W1
W1. Checkpoint selection. Thank you for this point. We agree that selecting the checkpoint on held-out performance makes the reported values optimistic, and that applying the same rule to all methods is consistent but does not amount to an independent evaluation. Below we clarify the protocol, explain why we used it, and describe how we have revised the paper.
Protocol. All methods (PathoSpatial and every baseline) use the same patient-level splits and the same budget of 30 training epochs. For TNBC Response and Lung Disease (leave-one-out cross-validation), we report the epoch with the highest AUC on the pooled held-out predictions. For TNBC Survival, each of the five outer folds is trained separately, the epoch with the highest C-index on that fold's held-out patients is selected, and the reported value is the mean ± SD over the five folds of these per-fold C-indices. There is no separate inner validation set (see also our answer to Q1).
Why we used this protocol. As noted in the Patho-Bench benchmark [14], some pathology tasks "have a small number of samples, which may make a validation set impractical." This applies to our cohorts (92, 44, and 35 patients): an inner 20% split would leave about 15 survival patients per validation set, spread over four survival-time bins with few events in each, and about 9 and 7 patients in the two classification cohorts, so the selection signal itself would be unstable [6]. Without a validation set, two conventions are common in multimodal survival prediction: reporting the best held-out epoch, as in PIBD [1], or reporting a fixed final epoch, as in MCAT [2], PORPOISE [3], SurvPath [4], and PANTHER [5]. The paper follows the first, because histology-only, expression-based, and multimodal models converge at different rates, so a single fixed epoch would favor some architectures over others. We now also report the second.
Last-epoch results. We also evaluated all methods at the final (30th) epoch, a fixed choice that uses no held-out information (new App. F.3, Table 9). PathoSpatial remains among the top methods: PathoSpatial-GNN has the second-highest mean survival C-index and the highest C-index on IDFS and DRFS, and PathoSpatial-Attn has the second-highest TNBC Response AUC. Neither protocol is ideal at this cohort size: best-epoch selection is optimistic, and a fixed final epoch is sensitive to differences in convergence, as some models may be heavily overfitted by the last epoch. Under both, PathoSpatial ranks first or second on mean survival C-index and TNBC Response AUC, and when second it trails the best method by 0.006, less than the seed-to-seed SD (App. F.1). This matched performance is what our interpretability claim requires.
| Method (last epoch) | Response AUC | Response ACC | Survival mean C-index |
|---|---|---|---|
| CLAM | 0.587 | 0.591 | 0.592 |
| TransMIL | 0.625 | 0.591 | 0.613 |
| Patch-GCN | 0.560 | 0.523 | 0.588 |
| PANTHER | 0.598 | 0.636 | 0.536 |
| PORPOISE | 0.565 | 0.523 | 0.567 |
| MCAT | 0.499 | 0.568 | 0.560 |
| SurvPath | 0.512 | 0.545 | 0.562 |
| PIBD | 0.542 | 0.500 | 0.488 |
| ProSurv | 0.539 | 0.568 | 0.569 |
| PathoSpatial-Attn | 0.619 | 0.568 | 0.572 |
| PathoSpatial-GNN | 0.596 | 0.568 | 0.607 |
Bold: best; italic: second best.
Limitation. As the reviewer notes, best-epoch selection is not an independent evaluation. Choosing the best of 30 epochs on the evaluation patients inflates the reported metrics, as shown for small-sample settings in [7, 8], and a shared rule does not guarantee unbiased rankings, because methods whose held-out performance varies more across epochs gain more from the selection. For this reason, our claims do not rest on performance margins. The paper claims performance comparable to the strongest baselines together with prototype- and modality-level interpretability, and our Conclusion already states that the cohorts are too small to distinguish methods. We do not claim superior generalization or transfer to unseen cohorts. Nested selection within each training fold, together with independent external cohorts, is the appropriate next step as larger paired ST–H&E cohorts become available.
Revisions.
- Sec. 4 and App. B.1 now describe the selection rule explicitly as held-out best-epoch selection without an inner validation set (Sec. 4, lines 350–351; App. B.1, lines 1105–1113).
- The Limitations now state that, because the cohorts are too small for an inner validation split, the epoch was selected on held-out patients for all methods, following prior work, and that nested selection is left to larger cohorts (Sec. 6, lines 536–538).
- New App. F.3 (Table 9) reports all methods at the final epoch, without epoch selection; Sec. 4 points to it.
Response 3/6: W2
W2. Weak evidence of improvement. We thank the reviewer for this assessment and agree with the main point. We do not claim a performance gain: as stated in Sec. 6, with 92, 44, and 35 patients the cohorts are too small to distinguish methods. Our claim is interpretability at performance comparable to the strongest baselines. This ranking is robust: over five random seeds, PathoSpatial-GNN keeps the highest mean survival C-index (0.690 ± 0.009, versus 0.681 ± 0.014 for SurvPath) and the highest TNBC Response AUC (0.722 ± 0.063; Appendix F.1, Table 7). With CONCH v1.5 in place of UNI2, it again leads on both (mean C-index 0.678, AUC 0.672; Appendix F.2, Table 8). The survival margin is within about one seed SD, consistent with matched rather than superior performance. Likewise, in paired bootstrap comparisons with the strongest baselines, SurvPath and Patch-GCN (new App. F.5, Table 11), the differences are within the uncertainty: the 95% CIs of all survival C-index differences include zero.
Fusion across tasks. We agree that the benefit of fusion is not uniform: on TNBC Response, fusion improves AUC but not accuracy, and on TNBC Survival, it helps on some endpoints but not others (Table 2). We attribute this more to the tasks and data than to the model. The predictive information may be unevenly distributed between the two modalities and may differ across endpoints and patients; TNBC is notoriously heterogeneous; survival endpoints are censored and influenced by factors beyond the tumor sample (e.g., treatment and follow-up); and the cohorts are small, so per-endpoint differences are noisy.
Data availability. Better cohorts are indeed needed, but they are hard to obtain at present. ST profiling remains costly, about USD 5,000–10,000 per sample, and outcome labels require years of clinical follow-up: TNBC Survival comprises patients treated between 2000 and 2016, with about five years of follow-up. Patient backgrounds, such as disease stage and treatment, must also be well controlled or matched, which is itself difficult. Complete, high-quality cohorts that pair H&E, ST, and patient-level outcomes are rare. To our knowledge, the three cohorts used here are among the few currently accessible ones. To confirm this, we systematically evaluated more than 30 independent published studies with ST data and more than 20 samples, as well as the HEST-1k [12] and STimage-1K4M [13] collections, and could not identify further suitable training data: these data often lack high-resolution H&E images or clean clinical variables, or have poorly controlled confounding variables, such as patient background.
Role of Lung Disease. We agree that Lung Disease is an easy task and does not by itself demonstrate the benefit of fusion. We include it as a control for a different granularity of signal in comparative analysis. In Lung Disease, the signal is largely at the patch level: whether fibrotic tissue is present. PathoSpatial is primarily designed for sub-patch signals, such as subtle pathway changes within tumor- or immune-rich regions, as demonstrated in the new cohort-level analysis (Sec. 5.2, "Cohort-level prototypes and spatial biomarker discovery"; Fig. 4; App. I): in TNBC Response, such signals persist after adjusting for regional cell-type composition. Lung Disease checks that the framework also handles coarse, patch-level signals without loss: both PathoSpatial variants reach AUC 1.000.
Response 1/6: General
We thank the reviewer for the careful reading and constructive feedback. We appreciate the recognition that, whereas most prior work predicts ST expression from H&E, PathoSpatial combines the two complementary modalities for patient-level prediction while incorporating local tissue context. We are also grateful that the reviewer found the interpretation component interesting, in particular the modality-specific prototypes that link back to histological patches and transcriptomic spots, their biological characterization, and the modality-grouped Owen-value decomposition. We take the soundness concerns seriously and address each point below.