acceptodds
Under review as a conference paper at ICLR 2027

M2U: MULTIMODAL-TO-UNIMODAL CROSS-MODAL GEOMETRY TRANSFER FOR CANCER PROGNOSIS

Abstract

Multimodal models typically assume that the same modalities are available during training and deployment. In practice, however, some modalities may be available in training cohorts but missing or prohibitively costly for new patients. We study this multimodal-to-unimodal setting in cancer prognosis, where paired histopathology and molecular omics are available during training, but predictions at deployment must be made from histopathology alone. Existing approaches typically distill predictions or features, reconstruct the missing modality, or align paired samples. Here, we introduce M2U, a framework that transfers patient-to-patient similarity structure from the privileged molecular space into the histopathology space and makes this structure queryable at deployment. Its Mapping Block introduces X-GeoSCL, a cross-modal objective that uses molecular representations to define omics neighborhoods and trains histopathology representations to reproduce these soft relational targets rather than merely matching individual pairs. Its Reasoning Block uses a query histopathology representation to retrieve patients from a train-only paired gallery, reads their stored molecular embeddings through cross-attention, and combines the recovered molecular evidence with direct histopathology predictions through a learned gate. We evaluate M2U on four TCGA cancer cohorts: kidney renal clear cell carcinoma, lung adenocarcinoma, brain lower-grade glioma, and pancreatic adenocarcinoma. The complete M2U framework achieves the highest mean AUC among all deployment-compatible methods on three of the four cohorts and outperforms both evaluated multimodal-to-unimodal baselines on all four. Within the Mapping Block, X-GeoSCL achieves the highest mean downstream AUC among the evaluated mapping objectives across all four cohorts. Ablations further show that the Mapping and Reasoning Blocks play complementary roles: transferring molecular neighborhood geometry enables cross-modal retrieval, but effectively exploiting this structure requires reading the retrieved molecular representations and integrating them with direct histopathology predictions. M2U can be applied to other settings in which a privileged modality is available during training but unavailable at deployment.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
multimodal learning, multimodal-to-unimodal learning, computational pathology, whole-slide imaging, molecular omics, privileged information, cross-modal representation learning, retrieval-augmented prediction, contrastive learning, cancer prognosis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fHHJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

M2U addresses cancer prognosis when histology images and omics are available for training, but only histology images are available for a new patient. Frozen omics representations define soft inter-patient neighborhoods, and X-GeoSCL trains a separate histology mapping encoder. At inference, the model uses the new patient's image to retrieve training patients and reads their omics representations through cross-attention. The resulting prediction is combined with a direct image-based prediction, without requiring the new patient's omics. Across four TCGA cancer cohorts, M2U achieves the highest mean AUC among the evaluated deployment-compatible methods in three cohorts and outperforms two multimodal-to-unimodal baselines in all four. Ablations with a fixed reader also support the incremental benefit of the mapping objective.

Critical strengths
  1. The deployment setting is practically relevant, and the method offers a distinctive way to use relationships derived from training-patient omics when predicting outcomes for new patients with histology alone.

  2. The fixed-reader ablation provides evidence for the incremental contribution of the mapping objective, while results across multiple cancer cohorts support the broader utility of the framework.

Critical weaknesses
  1. The geometry used for retrieval at deployment has not been clearly validated. Training optimizes image-to-omics relationships, whereas inference retrieves patients using image-to-image similarity. It is unclear whether the reported KL measure evaluates the neighborhoods actually used for retrieval. This gap limits the evidence that the learned alignment preserves the relationships needed by the deployed system.

  2. The reliability of the small AUC gains is not sufficiently established. Some key comparisons improve AUC by only 0.002, and paired uncertainty intervals for the AUC differences are not provided. The current evidence therefore does not establish how reliably the full framework improves over strong baselines.

Questions for authors
  1. Validation of deployment retrieval: Is the reported KL measure computed using the same image-to-image similarities and candidate patients used at inference? If not, what evidence shows that the actual retrieval neighborhoods preserve the intended omics-derived relationships?

  2. Uncertainty of the gains: Can the authors provide paired uncertainty intervals for AUC differences against the strongest deployment-compatible baselines using matched evaluation splits, particularly for comparisons with gains around 0.002? Do those estimates support a reliable improvement?

1 comment

We thank the reviewer for the clear and focused review, and for recognizing the deployment setting and the fixed-reader ablation. Both concerns are addressed with new analyses below.

Q1. Validation of deployment retrieval.

The KL in Table 2 measures slide-to-omics agreement, which is the quantity X-GeoSCL optimizes. To evaluate the retrieval geometry itself, we computed a second KL using the same image-to-image cosine similarity and the same candidate patients (the training-fold gallery) that are used at inference. It compares the molecular target distribution pp (Eq. 2) with the distribution rr induced by WSI-to-WSI similarity in the mapped space (τ=0.07\tau = 0.07, self-pairs excluded; mean ±\pm SEM over five folds, lower is better):

CohortX-GeoSCLInfoNCESoft-InfoNCE
KIRC0.301 ±\pm 0.0180.665 ±\pm 0.2690.394 ±\pm 0.014
LUAD0.426 ±\pm 0.1791.335 ±\pm 0.1960.518 ±\pm 0.144
LGG0.635 ±\pm 0.1080.658 ±\pm 0.0830.647 ±\pm 0.099
PAAD0.513 ±\pm 0.1310.835 ±\pm 0.1950.734 ±\pm 0.099

X-GeoSCL gives the closest match to the molecular targets in all four cohorts, so the alignment learned across modalities carries over to the image-to-image space used for retrieval.

A second, functional test uses held-out patients directly. We replaced the K=10K = 10 retrieved neighbors with random training patients and retrained the reader. AUC on held-out patients drops in all four cohorts, with paired 95% intervals excluding zero on KIRC (+0.025 [0.009, 0.043]) and PAAD (+0.037 [0.006, 0.070]). The neighborhoods actually retrieved at inference therefore carry predictive molecular information that random neighborhoods do not. We will add both analyses to the paper.

Q2. Uncertainty of the gains.

We performed a paired fold bootstrap over the five matched test folds, enumerating all 3,125 possible resamples. We report ΔAUC=AUC(M2U)−AUC(baseline)\Delta\mathrm{AUC}=\mathrm{AUC}(\mathrm{M2U})-\mathrm{AUC}(\mathrm{baseline}), with 95% CI (in bold).

Fold-bootstrap results:

Cohortvs. WSI retrievalvs. AMILvs. TransMILvs. MoMKDvs. M$^2$Surv
KIRC+0.115 [0.077, 0.151]+0.108 [0.077, 0.138]+0.034 [0.004, 0.064]+0.021 [−0.043, 0.079]+0.002 [−0.059, 0.052]
LUAD+0.064 [−0.028, 0.157]+0.113 [−0.004, 0.225]+0.020 [−0.044, 0.099]+0.104 [−0.006, 0.210]+0.054 [−0.017, 0.122]
LGG+0.012 [−0.019, 0.041]−0.016 [−0.041, 0.007]+0.158 [0.024, 0.279]+0.170 [0.112, 0.242]+0.186 [0.092, 0.297]
PAAD+0.090 [−0.005, 0.189]+0.014 [−0.023, 0.054]+0.204 [0.021, 0.349]+0.040 [−0.030, 0.128]+0.045 [−0.044, 0.125]

On KIRC, M2U significantly outperforms WSI retrieval (+0.115), AMIL (+0.108), and TransMIL (+0.034). On LGG, it significantly outperforms TransMIL (+0.158), MoMKD (+0.170), and M$^2$Surv (+0.186), where AMIL is strong. On PAAD, the improvement over TransMIL (+0.204) is significant, while LUAD shows positive but statistically uncertain gains. No significant degradation is detected against any evaluated baseline.

Regarding the reviewer's concern about the +0.002 margin over M$^2$Surv on KIRC, the confidence interval [−0.059, 0.052] does not establish superiority. Differences against M$^2$Surv on LUAD (+0.054) and PAAD (+0.045) also remain within uncertainty. Overall, M2U is significantly better in 7 of 20 comparisons and not significantly worse in any; the improvements are cohort-specific rather than uniform.

Summary.

We addressed the questions and comments with new analyses: the retrieval space is measured with the same similarity and gallery used at inference, and every AUC difference, including the 0.002 margin, has a paired confidence interval.

We thank the reviewer for the comments and believe that they improved the paper.

Reviewer_DsopOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

M2U learns a histology representation using soft molecular neighborhood targets, retrieves paired training patients through that representation, and reads their molecular embeddings with cross-attention. A validation-fitted scalar combines this prediction with an independent histology predictor. Five-fold evaluation on four TCGA cohorts reports the highest deployment-compatible mean AUC on three cohorts, alongside mapping ablations and censoring-aware Brier analyses.

Critical strengths

S1. The deployment setting is practically motivated: query patients need only histology, while paired training patients supply molecular context. The separate retrieval representation preserves a direct histology predictor when transferred information is less useful.

S2. Figure 2 and Appendix B report Brier scores that account for censoring. For KIRC and LUAD, the improvements over the strongest WSI-only baselines are supported by paired bootstrap confidence intervals that exclude zero. The authors transparently report findings such as the superior performance of AMIL on the LGG dataset and the reduction in AUC observed with gating in some cohorts; this level of detail makes the results far more informative than a single, aggregate performance table.

Critical weaknesses

W1. The main AUC advantage over M2Surv is only 0.002 on KIRC, while LUAD remains at 0.542 and LGG trails AMIL (Table 1). Paired uncertainty against the strongest deployment-compatible baselines is needed to establish superiority.

W2. Algorithm 1 builds the gallery before training the reader, but does not state whether training queries exclude their own gallery entries. Self-retrieval could expose query-specific molecular information during training that is unavailable at deployment.

W3. Equations 3-4 align WSI-to-omics similarities, whereas deployment retrieves using WSI-to-WSI similarities. These are different geometries; the current diagnostics do not isolate whether the optimized cross-modal neighborhoods preserve the retrieval relationships the reader actually needs.

Questions for authors

Q1. Are self-neighbors excluded when fitting the cross-attention reader? Please provide results using self-exclusion or out-of-fold training queries.

Q2. How closely do mapped WSI-to-WSI neighborhoods match the molecular targets?

Q3. How many patients remain in each fold after endpoint filtering, and are early-censored test patients included in IPCW-Brier calculations?

1 comment

We thank the reviewer for the careful review and for recognizing the deployment setting and the transparency of the reported results. We answer each question below, then address the remaining weakness.

Q1. Self-neighbors in reader training.

At inference, the gallery contains training patients only, so a test query can never retrieve itself. During reader training, self-neighbors were not excluded in the original runs. We therefore retrained the reader while removing each training query’s own gallery entry and keeping everything else unchanged:

CohortOriginal AUCSelf-excluded AUCΔ\DeltaAUC
KIRC0.6830.678−0.005
LUAD0.5420.546+0.004
LGG0.8020.806+0.004
PAAD0.7470.720−0.027

The change is at most 0.005 in three cohorts. PAAD, the smallest cohort, shows a larger decrease of 0.027, below its fold-level standard error (0.100). Overall, the results indicate that training-time self-retrieval does not primarily drive test performance.

The self-exclusion experiment provides a complementary robustness check, with the overall performance patterns remaining similar. We will add it to the revised manuscript.

Q2. WSI-to-WSI neighborhoods vs. molecular targets.

We measured this directly. For each training patient, we computed the KL divergence between the molecular target distribution pp (Eq. 2) and the distribution rr induced by WSI-to-WSI cosine similarity in the mapped space, averaged over patients and five folds (lower is better):

CohortX-GeoSCLInfoNCESoft-InfoNCE
KIRC0.301 ±\pm 0.0180.665 ±\pm 0.2690.394 ±\pm 0.014
LUAD0.426 ±\pm 0.1791.335 ±\pm 0.1960.518 ±\pm 0.144
LGG0.635 ±\pm 0.1080.658 ±\pm 0.0830.647 ±\pm 0.099
PAAD0.513 ±\pm 0.1310.835 ±\pm 0.1950.734 ±\pm 0.099

The additional KL analysis shows that X-GeoSCL better preserves molecular-neighborhood distributions in the mapped WSI space. Complementarily, the held-out random-neighbor ablation supports the predictive value of the retrieved molecular context. We will add these analyses to the paper.

Q3. Per-fold counts and IPCW-Brier.

Held-out test patients per fold after endpoint filtering:

CohortFold 0Fold 1Fold 2Fold 3Fold 4Retained
KIRC5853484850257 / 447
LUAD4845444342222 / 412
LGG2625253025131 / 445
PAAD2227282328128 / 165

The early-censored test patients are included in the IPCW-Brier calculation. We use the Graf estimator at the horizon τ\tau, with the censoring distribution estimated from each fold’s training patients only. This is why IPCW-Brier complements AUC: it uses the patients that the binary endpoint must exclude.

Summary and remaining weakness (W1).

The two questions on method validity are resolved by new experiments: self-exclusion leaves the results essentially unchanged (Q1), and the retrieval geometry follows the molecular targets (Q2). On paired uncertainty (W1), we ran a paired fold bootstrap against every baseline and a patient-level bootstrap against the WSI-only models (full tables in our response to Reviewer zdNf, Q1). M2U is significantly better in 7 of 20 comparisons and not significantly worse in any. Against M$^2$Surv on KIRC the difference is +0.002 [−0.059, 0.052]; we agree this does not establish superiority and will describe the two methods as comparable there. On LGG, M2U is significantly better than M$^2$Surv (+0.186 [0.092, 0.297]).

We thank the reviewer and believe these revisions have strengthened the paper.

Reviewer_zdNfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the issue of modal missingness during inference—specifically, the scenario where paired WSI and omics data are available during training, but only WSI data is available at deployment. The proposed M2U framework consists of two components: X-GeoSCL and the Reasoning Block.

During the training phase, X-GeoSCL accounts for the individual patient-to-patient similarities in both omics and WSI data. The goal is to train a WSI encoder that aligns the distribution of WSI patients with that of omics patients.

During inference, the Reasoning Block utilizes the similarity among WSI patients to retrieve the corresponding omics embeddings via cross-attention, and finally combines these with the predictions from a standard WSI model using a learned gate.

The authors validated the model on four TCGA cohorts (KIRC, LUAD, LGG, PAAD) and reported that M2U achieved the highest average AUC (deployment-compatible) in three cohorts and outperformed both multimodal-to-unimodal baselines (MoMKD and M²Surv) across all four cohorts.

Critical strengths

1: This paper presents the M2U, a clean and straightforward method that does not incur significant overhead during runtime or training—for example, through feature freezing, in-batch computation, and the use of a single GPU. This makes the method easier to reproduce and adopt.

2: The experimental setup described in the paper mitigates concerns about data leakage by defining the prediction horizons, all encoders, the gallery, and the censoring distribution exclusively within the training fold, thereby enhancing the credibility of the experimental results.

Critical weaknesses

1: The practical benefits are limited. In Table 1, compared to the strongest WSI-only method, M2U shows improvement beyond the standard error only on KIRC (+0.034). The performance differences on LUAD and PAAD fall within the standard error range (+0.020, +0.014). Furthermore, its performance on LGG lags behind that of AMIL (−0.017). Whether the proposed method’s performance can overcome the noise is crucial for real-world deployment scenarios.

2: Contribution of Omics Information is not clearly verified

The Purity@10 values in Table 2 range from approximately 0.50 to 0.54 for KIRC, LUAD, and PAAD. Furthermore, the AUC for K-NN label voting in Table 4 ranges from only 0.45 to 0.57 across all four cohorts. The retrieved neighbors yield results that are nearly random in terms of prognosis. However, under the same conditions, the CA reader in Table 4 achieves an AUC of 0.7 for KIRC. The authors explain this by noting that finer-grained information in the omics embedding is not reflected in the Purity score. However, this result could also stem from the fact that the reader’s input inherently includes the query’s own WSI embedding, making it difficult to distinguish whether the improvement comes from processing the omics data or simply from the addition of a WSI model. Conducting further experiments specifically on omics data would help clarify these contributions.

3: Lack of verification of key components which make the true contributions

The paper lacks ablation experiments on key components, such as the omics teacher and temperature, making it difficult to determine where the performance gains originate. In Table 2, the AUC difference between X-GeoSCL and InfoNCE falls within approximately one standard error; relevant ablation experiments would help clarify the differences between M2U and similar methods. Additionally, while the paper uses KL as an optimization objective, it acknowledges that KL “is not a standalone performance proxy” (L368–371); therefore, it remains to be seen whether using KL as a metric in Table 2 truly reflects the advantages of X-GeoSCL.

Minor:

  1. X-GeoSCL is similar to the Soft-InfoNCE method; when ℓ2 normalization is applied to the embeddings, the Gaussian kernel is effectively equivalent to a cosine softmax with a temperature of σ²/2. Consequently, the differences between X-GeoSCL and the original Soft-InfoNCE (Denize et al., 2023) are the choice of omics teacher and the temperature setting; however, the implementation details of Soft-InfoNCE are not specified.
  2. Neighborhoods are computed only within a single batch (batch size 64; PAAD uses 32). MoMKD uses cross-batch memory precisely because of the instability of alignment within a single batch. M2U, however, does not discuss this argument or the trade-offs involved.
  3. The differences from M²Surv are not explained. M²Surv also stores pathology–genomic paired features to compensate for missing modalities.
  4. The abstract states, “rather than merely matching individual pairs,” which suggests that paired alignment is performed; however, Eqs. 3–4 actually exclude the pairing itself, which could mislead readers.
  5. Cross-validation was performed only within TCGA; there was no external validation set. Deployment requires preserving the omics embeddings and labels of the training patients, and data governance was not discussed.
Questions for authors

1: Perfrom more experiments to better demonstrate the contribtuions of M2U:
(1) Paired tests: Combine the out-of-fold predictions from five folds and perform a bootstrap or DeLong test. (2) Explain how the peer methods were implemented, particularly how MoMKD (originally designed for classification) and M²Surv (originally using the C-index) were adapted to use binary AUC; explain why the AUC values for AMIL (0.429) and MoMKD (0.439) on the LUAD dataset are below 0.5; please include the C-index values to facilitate comparison with previously published figures.

2: Perform more experiments to better clarify the contribution of omics data: (1) Do not provide the neighbors' omics data to the reader; (2) Shuffle the neighbors' omics data (replace them with data from random patients); (3) Randomly select neighbors; (4) Directly combine two WSI encoders (AMIL + mapping encoder) using the same gate, without performing retrieval. Please evaluate the affect of the settings.

3: Please explain the calculation methods for KL and Purity in Table 2. Why is InfoNCE’s KL on LGG lower than that of X-GeoSCL, which directly optimizes KL?

  • How was the validation set for gate tuning partitioned, and does it overlap with the gallery? (On KIRC and LUAD, using CA alone [0.700, 0.550] performed better than adding the gate [0.683, 0.542]; the validation set may have been too small.)
  • The number of people excluded from each cohort due to loss to follow-up before the horizon, as well as the positive-to-negative case ratio (needed to interpret the random baseline for Purity).
  • What are the differences between M2U’s gallery, M²Surv’s memory, and MoMKD’s cross-batch memory (in terms of what is stored, how it is queried, and what is read)? What are the trade-offs of computing neighborhoods only within a single batch?
3 comments

We thank the reviewer for the careful and constructive review. We ran every analysis requested: paired significance tests, C-index, and the four omics-contribution controls. We answer each question below, then summarize how the new results address the weaknesses raised.

Q1(1). Paired tests.

We computed a paired fold bootstrap over the five matched test folds (all 3,125 resamples) against every baseline, and updated Table 1 to mark statistically significant differences.

Updated Table 1. Five-fold mean ROC-AUC ±\pm SEM. †^\dagger denotes a significant difference from M2U (p<0.05p < 0.05, paired fold bootstrap).

MethodKIRCLUADLGGPAAD
WSI retrieval0.568 ±\pm 0.028 †^\dagger0.479 ±\pm 0.0230.791 ±\pm 0.0260.657 ±\pm 0.114
AMIL0.576 ±\pm 0.021 †^\dagger0.429 ±\pm 0.0520.818 ±\pm 0.0320.733 ±\pm 0.093
TransMIL0.649 ±\pm 0.013 †^\dagger0.522 ±\pm 0.0420.645 ±\pm 0.081 †^\dagger0.543 ±\pm 0.086 †^\dagger
MoMKD0.663 ±\pm 0.0340.439 ±\pm 0.0460.633 ±\pm 0.054 †^\dagger0.707 ±\pm 0.088
M$^2$Surv0.681 ±\pm 0.0340.488 ±\pm 0.0540.617 ±\pm 0.076 †^\dagger0.702 ±\pm 0.101
M2U (ours)0.683 ±\pm 0.0130.542 ±\pm 0.0520.803 ±\pm 0.0220.747 ±\pm 0.100

Fold bootstrap. Δ\DeltaAUC = AUC(M2U) − AUC(baseline), with 95% CI; bold marks intervals that exclude zero.

Cohortvs. WSI retrievalvs. AMILvs. TransMILvs. MoMKDvs. M$^2$Surv
KIRC+0.115 [0.077, 0.151]+0.108 [0.077, 0.138]+0.034 [0.004, 0.064]+0.021 [−0.043, 0.079]+0.002 [−0.059, 0.052]
LUAD+0.064 [−0.028, 0.157]+0.113 [−0.004, 0.225]+0.020 [−0.044, 0.099]+0.104 [−0.006, 0.210]+0.054 [−0.017, 0.122]
LGG+0.012 [−0.019, 0.041]−0.016 [−0.041, 0.007]+0.158 [0.024, 0.279]+0.170 [0.112, 0.242]+0.186 [0.092, 0.297]
PAAD+0.090 [−0.005, 0.189]+0.014 [−0.023, 0.054]+0.204 [0.021, 0.349]+0.040 [−0.030, 0.128]+0.045 [−0.044, 0.125]

M2U is significantly better than WSI retrieval, AMIL and TransMIL on KIRC, than TransMIL, MoMKD and M$^2$Surv on LGG, and than TransMIL on PAAD.

Overall, across all four cohorts and all five baselines, M2U is either significantly better (7 of 20 comparisons) or statistically comparable (13 of 20); it is not significantly worse than any baseline in any cohort.

Patient-level bootstrap, WSI-only baselines. As the reviewer suggested, we also pooled the out-of-fold predictions and ran a paired patient-level bootstrap (10,000 resamples):

Cohortvs. WSI retrievalvs. AMILvs. TransMIL
KIRC+0.085 [0.035, 0.136]+0.070 [0.016, 0.124]+0.039 [−0.023, 0.100]
LUAD+0.056 [−0.034, 0.148]+0.104 [0.003, 0.201]−0.020 [−0.116, 0.077]
LGG−0.074 [−0.178, 0.024]−0.054 [−0.115, 0.001]+0.183 [0.052, 0.314]
PAAD+0.090 [0.001, 0.182]−0.031 [−0.067, 0.003]+0.269 [0.148, 0.389]

The two analyses give a consistent picture: significant gains over AMIL and WSI retrieval on KIRC and over TransMIL on LGG and PAAD in both, and no significant difference from AMIL on LGG and PAAD, where AMIL is already strong. The patient-level analysis additionally supports the gain over AMIL on LUAD. We will add both analyses to the paper.

Q1(2). Baseline adaptation

Both baselines use the same patch features, molecular embeddings, and splits as M2U.

  • MoMKD: MoMKD is a binary WSI classifier trained with genomic distillation, originally to predict whether a biomarker status is positive or negative; our task is also binary classification from a WSI, so the adaptation changes only the label, from biomarker status to the fixed-horizon prognosis label, and keeps its classification objective.

  • M$^2$Surv: M$^2$Surv is a multimodal survival model that predicts interval-specific hazards and is trained with a discrete-time survival NLL. To match our fixed-horizon task, we change its output head and its loss: the interval hazard logits are replaced with a single risk logit, and the survival NLL with binary cross-entropy on the same prognosis label used for all other methods. AUC is computed from this output, as for every method.

Q1(2). LUAD cohort performance

We verified the labels, prediction direction, patient-slide alignment and the AUC computation, and found no error. LUAD carries the weakest prognostic signal of the four cohorts: every histology-based method is near chance (0.43–0.52), and even the privileged omics models reach only about 0.61. The sub-0.5 means of AMIL and MoMKD come from a single fold (0.249 and 0.260). On the same patients and labels, TransMIL and WSI retrieval score 0.548 and 0.488, which rules out a label or evaluation error. On the other four folds both methods are at chance level (0.474 and 0.483).

Q1(3). C-index.

We computed Harrell’s C-index on held-out predictions, mean ±\pm SEM over five folds:

CohortM2UBest WSI-onlyBaseline
KIRC0.647 ±\pm 0.0120.620 ±\pm 0.010TransMIL
LUAD0.524 ±\pm 0.0300.513 ±\pm 0.027WSI retrieval
LGG0.642 ±\pm 0.0360.667 ±\pm 0.035AMIL
PAAD0.691 ±\pm 0.0470.694 ±\pm 0.043AMIL

The pattern matches the AUC results: M2U is higher on KIRC and LUAD, and comparable to AMIL on LGG and PAAD. Values are computed on the patients eligible for the fixed-horizon endpoint, which is the task all models were trained on. We will add them to the paper.

Q2. Contribution of omics.

We ran all four controls with the same splits, reader architecture and gate procedure. For (1)–(3) the reader was retrained under the intervention. Entries are Δ\DeltaAUC = AUC(M2U) − AUC(control), with paired-bootstrap 95% CI:

CohortFull M2U(1) No omics(2) Shuffled omics(3) Random neighbors(4) Two WSI + gate
KIRC0.683+0.021 [0.005, 0.039]+0.012 [−0.000, 0.026]+0.025 [0.009, 0.043]+0.026 [−0.007, 0.058]
LUAD0.542+0.015 [−0.044, 0.078]+0.055 [0.015, 0.098]+0.048 [−0.012, 0.106]+0.066 [−0.013, 0.143]
LGG0.802+0.018 [−0.035, 0.071]+0.030 [0.005, 0.062]+0.046 [−0.003, 0.095]+0.011 [−0.057, 0.082]
PAAD0.747+0.035 [0.006, 0.068]+0.020 [0.000, 0.040]+0.037 [0.006, 0.070]+0.045 [0.006, 0.090]

Full M2U is higher than every control in every cohort (16 of 16 comparisons). Removing the molecular content (1, 2) or the learned retrieval (3) each reduces AUC. For (4), we put a linear head on the frozen mapping encoder and fused it with AMIL through the same gate, with no gallery and no omics. The mapping encoder is itself trained with molecular supervision, so part of the transferred information is already present in its embedding; retrieving and reading the neighbors’ molecular embeddings adds to it in every cohort. We will add these controls to the paper.

Q3. KL and Purity

  • KL: the mean over patients of KL(pi∥qi)\mathrm{KL}(p_i \| q_i) between the molecular neighborhood distribution pp (Eq. 2) and the WSI-induced distribution qq (Eq. 3), computed over the full training-fold gallery with self-pairs excluded.
  • Purity@10: the fraction of a patient’s ten nearest neighbors in the mapped WSI space that share its label, averaged over held-out patients.
  • Both metrics are averaged over five folds. We will add the definitions to the paper.

Q3. LGG: On LGG, the difference (0.352 ±\pm 0.091 vs. 0.334 ±\pm 0.062) is well within fold-to-fold variability and reflects a single fold: X-GeoSCL has the lower KL in three of the five folds, so the two objectives are indistinguishable on this cohort.

Gate validation split.

Train, validation and test sets are disjoint within each fold (the validation set is 17–22% of the training fold, 20–42 patients depending on the cohort), and the gallery contains training patients only, so it does not overlap the gate-validation set.

The gate is a single scalar fitted once with the reader frozen. It costs little on KIRC and LUAD (0.017 and 0.008), and it is what lets one procedure handle both regimes: on LGG and PAAD it raises CA from 0.540 and 0.595 to 0.802 and 0.747.

Exclusions and class balance.

CohortTotalExcludedRetainedPositiveNegativeChance Purity@10
KIRC4471902571421150.499
LUAD412190222142800.533
LGG445314131101300.650
PAAD1653712860680.524

Purity@10 is close to chance, consistent with our statement that label purity does not explain the downstream ranking (L368–371). The reader uses the continuous molecular embeddings of the neighbors, not their labels, and Q2 tests that contribution directly.

Gallery vs. M$^2$Surv and MoMKD memories; in-batch neighborhoods.

MethodStoredQueried byRead
M2UMapped WSI and frozen molecular embedding (256-d each) for every training patientCosine similarity in the mapped WSI space, K = 10 patientsMolecular embeddings of the 10 neighbors, via cross-attention
M$^2$SurvPathology tokens and six genomic tokens per training patientPathology feature similarity, most similar patientThat patient’s genomic tokens
MoMKDTwo learned class-conditional prototype banks (16 each)Patch-to-prototype similarityPrototype weights guiding attention pooling; no patient-specific retrieval

M2U is the only one of the three whose query space is trained to follow molecular neighborhoods. In-batch neighborhoods cut cost from O(N2)O(N^2) to O(B2)O(B^2) at the price of each update seeing a subset of patients. They are used only for mapping training; inference retrieves from the full training-fold gallery.

Summary.

On practical benefit (W1), M2U is significantly better in 7 of 20 paired comparisons, including against the strongest WSI-only model on KIRC, and is not significantly worse in any; it gains where molecular transfer helps without a significant loss where it does not. On the contribution of omics (W2), full M2U is higher than every control in every cohort, including the two-WSI-encoder model without retrieval. On component contributions (W3), X-GeoSCL gives the highest mean AUC in all four cohorts against both alternative objectives with the reader, temperature and bandwidth held fixed.

We agree that, for ℓ2-normalized embeddings, X-GeoSCL’s Gaussian target is equivalent to a cosine softmax with temperature σ²/2; our InfoNCE and Soft-InfoNCE baselines are likewise our own cross-modal adaptations of contrastive objectives to WSI-omics mapping with frozen omics similarities as targets, so all three are equally novel in this respect. We adopted X-GeoSCL over Soft-InfoNCE because its bandwidth σ is more intuitive to infer from the data, as the median molecular distance, than a fixed teacher temperature.

We will specify the Soft-InfoNCE implementation in the appendix, reword the abstract to make clear that the self-pair is excluded, and add the lack of external validation and the governance of the stored gallery to the Limitations.

We thank the reviewer and believe that addressing these comments has made the paper stronger.

Comments on the paper

1 comment

Revised Manuscript Uploaded

We thank the reviewers and Area Chair for their constructive feedback. We have uploaded a revised manuscript incorporating additional analyses, experimental controls, and methodological clarifications addressing the concerns raised during the discussion.

The main revisions are:

  1. Predictive performance: Added paired fold-bootstrap significance analysis, uncertainty reporting, and additional evaluation metrics, including Harrell's C-index (Table 1, Appendices A-B).

  2. Contribution of molecular information: Added four controls evaluating molecular content, retrieval quality, and additional WSI capacity, alongside a reader self-exclusion experiment (Appendix C).

  3. Cross-modal geometry transfer: Expanded the neighborhood diagnostics to evaluate geometry preservation in the mapped WSI retrieval space (Section 5.2, Table 2).

  4. Methodological clarifications: Clarified the cross-modal SCE adaptation, geometry-calibrated teacher temperature, endpoint eligibility, validation separation, baseline adaptations, and implementation details (Section 4, Appendix A).

We also refined the discussion and limitations. The proposed framework, evaluation cohorts, and primary predictive results remain unchanged.

These revisions address the reviewers' concerns and improve the work's clarity and reproducibility.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.