acceptodds
Under review as a conference paper at ICLR 2027

Stacked Multimodal Fusion of Genetic and Histopathology Features for Survival Prediction in Breast Cancer

Abstract

Accurate survival prediction in breast cancer requires capturing complementary information at both the molecular and tissue morphology levels, which single-modality models fail to represent effectively. We propose a multimodal survival prediction framework that integrates genetic expression features and whole-slide histopathology representations. Genetic embeddings are learned using a multi-head attention network designed to produce stable low-dimensional representations from high-dimensional transcriptomic profiles. Tissue morphology features are extracted from segmented whole-slide images using a fine-tuned ConvNeXt-Tiny backbone. Each modality is used to train an independent Cox proportional hazards model, producing unimodal risk estimates. These modality-specific risk scores are then combined through a stacked fusion strategy, where a meta-model learns the final survival prediction. Evaluated on the TCGA-BRCA dataset, the proposed approach achieves a concordance index of 0.666, outperforming both unimodal baselines and standard multimodal fusion strategies. To demonstrate generalizability, the framework was externally validated across three additional pan-cancer cohorts (LUAD, BLCA, KIRC). The stacked fusion consistently outperformed unimodal baselines across all cohorts with a mean cross-cancer Concordance Index (C-index) of 0.684. These results demonstrate that leveraging both molecular and histological signals jointly improves the robustness and discriminative power of patient-level survival prediction.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Survival Prediction, Multimodal Fusion, Computational Pathology, Cox Proportional Hazards

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_sz4nOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper combines histopathological and transcriptomic information for survival prediction. Modality-specific representations are mapped to risk scores using separate Cox models, and a second Cox model combines these scores.

Critical strengths

1. A compact and modular fusion stage. The final Cox model combines only two modality-specific risk scores, keeping the fusion stage low-dimensional and separating modality-specific prediction from risk aggregation. This simplicity makes the approach a useful reference for evaluating more complex fusion strategies. It is a practical strength, although its contribution to my overall assessment is modest because simplicity alone does not establish better generalization or reduced overfitting.

2. Controlled fusion ablations and consistently positive mean results. Section 4.5 states that the fusion variants use identical modality-specific representations, helping separate the choice of fusion strategy from encoder differences. The proposed model also achieves higher mean C-indices than the included unimodal models across all four cohorts. Together, these results provide preliminary evidence of practical value and justify examining the approach beyond a single dataset, although the reliability of the improvements requires further clarification.

Critical weaknesses

1. The methodological contribution is not established relative to direct prior work. The central procedure—independently predicting modality-specific risks and fitting another Cox model to combine them—already appears in the “Late fusion” approach of Steyaert et al., 2023. The remaining components, including ssGSEA, multi-head self-attention, ConvNeXt, PCA, and penalized Cox regression, are established techniques. The manuscript does not identify a substantive new mechanism or isolate the value of its particular encoder choices. This is consequential because the paper is presented as a methodological contribution, but its technical advance over an existing risk-level fusion pipeline remains unclear.

2. The comparisons do not substantiate competitiveness against recent methods. The evaluation omits directly relevant approaches such as LD-CVAE (CVPR 2025), SlotSPE (ICLR 2026), and MoHD (Medical Image Analysis 2026). SlotSPE is particularly relevant to the motivation of compressing high-dimensional multimodal inputs, while LD-CVAE includes complete-modality evaluation. Representative comparisons under matched inputs, endpoints, splits, and tuning conditions are needed. This omission is consequential because strong empirical evidence could establish the value of a simple existing strategy, but the current benchmark does not provide that evidence or support the SOTA claim.

Questions for authors
  1. Why is breast cancer the central application? The architecture and pathway representation appear broadly applicable across cancers. A disease-specific architecture is not necessary, but what unresolved breast cancer prognosis problem motivates this study? Can the authors establish disease-level value through relevant subgroup analyses or incremental prediction beyond clinical variables?

  2. How is independence maintained throughout the two-stage training procedure? Are the meta-Cox training inputs generated through out-of-fold predictions within each outer training set? Are encoder training/fine-tuning, normalization, PCA, and hyperparameter selection confined to the appropriate training partitions? Please provide the complete patient-level validation workflow.

  3. How reliable are the reported improvements and stability claims? On BRCA, the improvement over the WSI-only model is 0.005. Can the authors report paired confidence intervals for performance differences? Section 3.5 mentions bootstrap confidence intervals, but these are not presented in the main results. Moreover, the proposed model’s BRCA standard deviation is higher than those of concatenation and mean fusion. How does this support the claim of reduced variance? The log-rank tests establish risk-group separation, rather than superiority over competing models.

  4. What claim is the multi-cancer evaluation intended to validate?
    Survival prediction is defined with respect to a target population and endpoint. If models are fitted and evaluated separately within LUAD, BLCA, and KIRC, these experiments assess the reuse of the modeling pipeline for different cancer-specific prediction tasks. They do not establish generalization of the breast cancer prognostic model. What conclusion about the stated breast cancer task follows from these additional experiments, and on what basis are they described as “external validation”? Please distinguish applicability of the framework across tasks from generalization of a fitted prognostic model within its intended patient population.

  5. How are modality contributions interpreted? Section 4.4 infers greater histopathological contribution from larger meta-Cox coefficients. Were the input risk scores standardized before fusion? Please report the coefficients and their uncertainty, since coefficient magnitudes depend on input scale and do not directly measure modality importance.

  6. Can the experimental setup be made reproducible? Please specify the final matched sample and event counts, survival endpoint, patch extraction and aggregation settings, encoder pretraining/fine-tuning, MHA configuration, PCA dimensionality, and regularization/tuning procedure.

Reviewer_uP6YOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents a multimodal survival prediction pipeline for cancer prognosis that combines transcriptomic profiles and whole slide histopathology images. Gene expression profiles are compressed into 50 MSigDB Hallmark pathway enrichment scores via single-sample Gene Set Enrichment Analysis and processed through a Multi-Head Self-Attention encoder to produce a genomic risk score. Whole slide image patches are segmented via Otsu thresholding, encoded using a ConvNeXt-Tiny model, aggregated at the patient level, reduced via principal component analysis, and fed into a Cox proportional hazards model to produce a histopathological risk score. Finally, a meta-Cox model takes the two scalar unimodal risk scores to predict final patient survival risk. The pipeline is evaluated using 3-fold cross-validation on TCGA-BRCA, LUAD, BLCA, and KIRC.

Critical strengths
  1. Risk-level late fusion is a sensible answer to the small-n/high-dimensionality problem in TCGA-type data, and the paper is explicit about why naive concatenation tends to overfit.

  2. Testing on four cancer types, rather than BRCA alone, is more than many multimodal survival papers do.

Critical weaknesses
  1. The manuscript reads like a stitched-together, low-effort LLM report filled with generic textbook boilerplate and severe internal contradictions across sections. Although the AI Use Statement claims generative AI was only used for minor grammar checks and finding instructions to download TCGA data, the text is padded with elementary textbook explanations that have no place in an ICLR submission, such as writing out the generic Otsu thresholding formula in Equation 1 or explaining in Section 3.2 that the square-root scaling factor in standard dot-product attention prevents dot products from growing too large during backpropagation. More importantly, different sections of the paper read as if they were generated independently without checking whether the technical pipeline actually connects. Specifically, the Abstract claims on line 016 that tissue morphology features are extracted using a fine-tuned ConvNeXt-Tiny backbone. Yet Section 3.3 on lines 195 to 202 and Section 3.6 on line 231 state that patches are passed through ConvNeXt-Tiny, aggregated across the slide, standardized, reduced in dimension offline using PCA, and then fit using the CPU tabular survival library Lifelines. Because slide-level aggregation, standardization, PCA, and Lifelines Cox fitting are offline non-differentiable operations, fine-tuning ConvNeXt-Tiny end-to-end is impossible under the pipeline described in Section 3.3, and the paper never explains how ConvNeXt-Tiny was fine-tuned, at what magnification patches were extracted, how many patches were sampled per slide, what aggregation function was used across patches, or how many PCA components were retained. Similarly, Section 3.2 states in Equation 4 that the attention network outputs a risk score directly via a linear Cox layer trained with partial likelihood loss, whereas Figure 1c and Section 3.4 show the extracted attention features being fed into a separate Lifelines Cox model.

  2. The proposed stacked Cox fusion is mathematically trivial and suffers from severe in-sample stacking leakage. Substituting the two unimodal Cox hazard equations from Equation 5 into the meta-Cox model in Equation 6 simply multiplies each unimodal linear predictor by a scalar coefficient. Mathematically, a linear combination of two linear predictors is still just a two-weight linear rescaling of independent unimodal predictions, which is completely incapable of modeling any nonlinear cross-modal biological interactions between genomic pathways and tissue morphology. Worse still, in stacked ensemble learning, a meta-model must be trained strictly on out-of-fold predictions generated by the base models. Here, the authors fit the deep genomic attention network, the base Cox models, and the meta-Cox weights on the exact same training split without nested out-of-fold cross-validation. Because the base models are evaluated in-sample to train the meta-Cox layer, whichever unimodal branch overfits the training fold more severely will exhibit artificially higher concordance on the training data and receive a distorted weight in the meta-Cox model.

  3. The experimental evaluation misrepresents separate cross-validation runs as external validation, shows no statistically meaningful multimodal gain on the primary benchmark, and uses an invalid Kaplan-Meier pooling protocol. First, both the Abstract on line 023 and Section 4 on line 239 claim that the framework was externally validated across three additional pan-cancer cohorts, namely LUAD, BLCA, and KIRC. In survival analysis, external validation means evaluating a frozen model trained on a discovery cohort directly on an independent external cohort. However, Section 4.1 on line 244 reveals that the authors simply ran separate 3-fold cross-validation from scratch on each of the four TCGA datasets; training a new model on LUAD or KIRC is not external validation of a breast cancer model. Second, on the main TCGA-BRCA cohort in Table 1, unimodal ConvNeXt alone achieves a C-index of 0.661 with a standard deviation of 0.036, while the proposed Stacked Cox model achieves 0.666 with a standard deviation of 0.040. A gain of 0.005 is an order of magnitude smaller than the fold-to-fold standard deviation of 0.040, demonstrating that adding the genomic branch, which only achieves 0.576 unimodally, brings no statistically meaningful improvement on breast cancer. Third, Figure 2 plots single cohort-wide Kaplan-Meier curves and reports extreme log-rank p-values such as 3.01 times 10 to the minus 18 on KIRC from a 3-fold cross-validation setup without explaining how uncalibrated risk scores from three separate fold models were pooled or how the standard deviation bands were computed.

Questions for authors
  1. Clarification request on the training pipeline and stacking leakage: How was ConvNeXt-Tiny fine-tuned when the downstream pipeline relies on offline slide-level aggregation, PCA dimensionality reduction, and Lifelines Cox fitting? What magnification, patch count, patch aggregation operator, and PCA dimension were used? Furthermore, when fitting the meta-Cox model in Equation 6, were the unimodal risk scores generated via nested out-of-fold cross-validation within each training split, or were the meta-Cox weights fit directly on the in-sample training predictions of the base models?

  2. Clarification request on Kaplan-Meier evaluation: How were the single cohort-wide Kaplan-Meier curves, median risk cutoffs, shaded standard deviation bands, and log-rank p-values in Figure 2 computed across the three cross-validation folds?

Reviewer_EhUTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents a multimodal survival prediction framework combining transcriptomic and histopathology information. Gene expression is converted into 50 Hallmark pathway scores using ssGSEA and encoded with a multi-head attention network. Histopathology patches are encoded with ConvNeXt-Tiny and aggregated into patient-level features. Separate Cox models produce modality-specific risk scores, which are combined using a second, penalized Cox model. Experiments cover four TCGA cancer cohorts and report three-fold cross-validation results, comparisons with unimodal and multimodal baselines, fusion ablations, and Kaplan–Meier risk stratification.

Critical strengths

The framework is simple and easy to follow.

Critical weaknesses
  1. Novelty. The framework combines established components: ssGSEA pathway scoring, standard multi-head attention, ConvNeXt-Tiny feature extraction, Cox regression, and risk-level stacking. The second Cox model learns weights over two unimodal risk scores, but the paper does not establish a new fusion mechanism, survival objective, or estimation procedure.
  2. Insufficiently convincing and specified evaluation. On BRCA, the proposed model improves the C-index from 0.661 for the WSI-only model to 0.666, and from 0.655 for SurvPath to 0.666. The paper does not quantify uncertainty in these paired improvements, while significant log-rank tests for risk-group separation do not establish superiority over competing predictors. More importantly, the stacking protocol does not explain whether the meta-model is trained on out-of-fold unimodal predictions, or how preprocessing and hyperparameter selection are isolated from evaluation folds. The whole section seems unfinished.
Questions for authors
  1. Please clarify whether the meta-Cox model is trained on out-of-fold unimodal predictions, and how preprocessing, encoder training, and hyperparameter selection are isolated from the evaluation folds.

  2. For LUAD, BLCA, and KIRC, is the BRCA-trained model applied without retraining or tuning, or is a separate model fitted within each cohort?

  3. Please provide the result of controlled ablation comparing the MHA genomic branch against (i) penalized Cox regression directly on the same 50 ssGSEA pathway scores and (ii) a simple MLP followed by a Cox risk head. Does MHA provide a consistent benefit across the four cohorts?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.