acceptodds
Under review as a conference paper at ICLR 2027

Understanding and Mitigating Missing-Modality Effects in Medical MLLMs

Abstract

Missing modalities are common in medical settings and may hinder clinical decision-making. While prior work has explored ways to mitigate this issue, its impact on medical multimodal large language models (MLLMs) remains underexplored. This raises two questions: (1) how existing medical MLLMs perform under missing modalities; (2) how to improve their performance when modalities are missing. To address the first, we introduce MedGap, a MIMIC-IV-based benchmark with CXR, ECG, and laboratory data to systematically evaluate MLLMs under missing modalities. Unlike prior benchmarks that primarily focus on arbitrary modality combinations, MedGap further examines missingness within disease-relevant pairs, avoiding interference from irrelevant modalities. Our evaluation shows that more modalities do not automatically translate into better performance, while the paired diagnosis evaluation further shows that the impact of a missing modality depends on its relevance to the diagnosis target. Motivated by these findings, we address the second question with MIRI, a two-stage framework that combines probabilistic cross-modal recovery in latent space with teacher-guided integration of the recovered information into the LLM. Experiments on MedGap show that MIRI improves average performance across missing-modality settings and multiple model backbones. Further analysis shows that cross-modal recovery captures missing-modality information and downstream integration enables its effective use, with recovery becoming more beneficial as fewer modalities remain.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Medical MLLMs, Missing Modality

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UxxwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary

This paper studies the missing modality problems on medical multimodal large language models. The authors introduce the MedGap, a benchmark covering CXR, ECG, and laboratory data. It further proposes the MIRI, which combines probabilistic cross-modal recovery and teacher-guided integration to improve performance under missing modalities. Experiments across multiple datasets and medical MLLMs show good results.

Critical strengths

Strengths

  1. Overall, I think the motivation is meaningful, as missing modalities are common in real clinical settings and can affect the reliability of medical MLLMs.
  2. This multi-modal setting itself is valuable, it's good to see papers trying to bridge physiological signals to other medical modalities like medical imaging and EHR text.
  3. The proposed MedGap benchmark is useful.
  4. The proposed MIRI framework is interesting, and the experiments cover multiple models and missing-modality settings. The ablation studies also provide useful insights into the effectiveness of different components.
Critical weaknesses

Weaknesses

  1. I notice that MedGap is constructed from encounters with all three modalities available, and missing modalities are simulated by removing the inputs. However, I feel real-world missingness may not be random and could be related to patient conditions or clinical decisions. I wonder whether the conclusions can generalize to such realistic missingness patterns? Would it be possible to include some evaluation with naturally missing modalities?
  2. The ED / clinical settings have natural missing channel/missing modality scenarios. For example, in the MC-MED or MIMIC-III/MIMIC-IV datasets, there are numerics like HR, SPO2, and waveform modalities like ECG, PPG, they are often missing channels. It's also interesting to try or discuss if the proposed method can generalize to MLLMs in these datasets and method [1,2].
  3. The motivation for probabilistic recovery is reasonable, but I notice that only the predicted mean is used as the recovered representation during inference. I wonder how the uncertainty modeling actually benefits downstream prediction in this case? Would it be possible to compare with a simpler deterministic latent prediction method to better justify this design?
  4. I think the recovery evaluation in Table 7 is interesting, but I am not fully convinced that the improved linear probing performance necessarily indicates successful recovery of missing-modality information. Can you elaborate more?
  5. I'm also interested to dicuss: whether this paradigm generalize to a more practical settings that we have modality A in time A, and modality B in time B. It's realistic, because people may do lab test and have medication order, and have different medical imaging exams at different time points. Jointly modeling these modalities in a longitudinal scope might be a good future work.

[1] Dai D, She J, Cheong J, et al. QoQ-Med3: a multimodal reasoning foundation model for clinical analysis[J]. npj Digital Medicine, 2026.

[2] Xu Y, Shuai Z, Yang Y. Sensor-Language-Action Models[J]. arXiv preprint arXiv:2610.08244, 2026.

Questions for authors

See weakness

3 comments
AuthorsOpenReview ↗

Response [1/2]

We sincerely thank the reviewer for the positive assessment and constructive suggestions. We appreciate the recognition of our work and have carefully considered all the concerns. Below, we provide point-by-point responses and corresponding revisions to further clarify and strengthen our work.

W1: Generalizability to Real-World Missingness Patterns

To evaluate generalizability beyond simulated missingness, we randomly sampled 1,000 naturally incomplete ED encounters from MIMIC, preserving their original modality-availability patterns (EL: 547, L: 368, CL: 73, E: 8, C: 2, and CE: 2).

MethodDiagnosis MiF1 ↑Diagnosis MaF1 ↑LOS Acc ↑M30 Bal Acc ↑
Zero-shot27.1424.1030.9055.72
SFT49.4046.4437.4068.66
MIRI51.9350.7637.6068.68

MIRI outperforms SFT on all metrics (revised in Appendix Table 16 ), with notable improvements in diagnosis (+2.53% MiF1 and +4.32% MaF1). These results further support the effectiveness of MIRI under naturally occurring missingness, particularly for diagnosis.

W2: Generalizability to Clinical Signal Modalities We thank the reviewer for this valuable suggestion. Our current study focuses on modality-level missingness across CXR, ECG, and laboratory data. Since MIRI performs recovery in the latent token space rather than reconstructing raw inputs, its underlying recovery mechanism is not inherently tied to specific data modalities and could potentially be extended to finer-grained missingness scenarios, such as missing physiological signal channels, with appropriate modality representations and recovery objectives. Although such extensions are beyond the scope of the current study, we believe they offer a promising direction for future research. We have incorporated both references [1, 2] into the Related Work section and discussed this potential extension in Lines 119–120 of the revised manuscript.

AuthorsOpenReview ↗

Response [2/2]

W3: Effectiveness of Probabilistic Recovery and Uncertainty Modeling We thank the reviewer for raising this important question. Although only the predicted mean is used as the recovered representation during inference, probabilistic modeling provides distribution-level supervision during Stage 1 training. Specifically, by aligning the predicted and target latent distributions through KL divergence, MIRI learns cross-modal representations while explicitly accounting for uncertainty in cross-modal correspondence, rather than relying solely on point-wise feature matching.

To further validate the benefit of probabilistic modeling, we replace the probabilistic recovery objective with deterministic MSE prediction while keeping all other training conditions unchanged. As shown in the first table, MIRI achieves an average recovery probe AUROC of 67.02, outperforming deterministic MSE prediction (65.61) by 1.41%.

More importantly, we evaluate the impact on downstream tasks. Across three random seeds, MIRI achieves higher mean performance than deterministic MSE prediction on all six downstream metrics (revised in Appendix Table 12). For diagnosis, MIRI improves MiF1 from 43.45 ± 0.83 to 44.59 ± 0.42 and MaF1 from 32.75 ± 0.68 to 34.25 ± 0.46. Improvements are also observed across all LOS and M30 metrics. These results demonstrate that probabilistic recovery provides benefits beyond latent representation prediction, improving the quality of recovered representations for downstream clinical tasks, even when only the predicted mean is used during inference.

Recovery DirectionMIRIMSEΔ (MIRI − MSE)
ECG → CXR69.3967.02+2.37
Lab → CXR71.2971.04+0.25
ECG+Lab → CXR74.4273.00+1.42
CXR → ECG65.5164.39+1.12
Lab → ECG60.4061.47−1.07
CXR+Lab → ECG68.3365.15+3.18
CXR → Lab65.6263.46+2.16
ECG → Lab61.2559.65+1.60
CXR+ECG → Lab67.0165.31+1.70
Average67.0265.61+1.41
MethodDX MiF1 ↑DX MaF1 ↑LOS Acc ↑LOS MaF1 ↑M30 F1 ↑M30 BAcc ↑
MSE43.45 ± 0.8332.75 ± 0.6838.11 ± 0.2732.31 ± 0.3729.47 ± 0.2267.17 ± 0.47
MIRI44.59 ± 0.4234.25 ± 0.4638.63 ± 0.1632.58 ± 0.9430.07 ± 0.5367.66 ± 0.26

W4: Validity of Missing-Modality Recovery Evaluation

To further validate whether the recovered representations capture patient-specific missing-modality information beyond the disease-predictive information assessed by linear probing, we compare the KL divergence between predicted and true target distributions for the same patient (Matched) and different patients (Shuffled). As revised in Appendix Table 11, Matched KL is consistently lower than Shuffled KL across all nine recovery directions, with average values of 0.01456 and 0.01908, respectively (a 31.1% increase after shuffling). This indicates that the predicted distributions capture patient-specific cross-modal information aligned with the corresponding patient's true missing-modality representations, rather than merely learning generic modality representations. Together with Table 7, these results provide complementary evidence for meaningful missing-modality recovery.

Recovery DirectionMatched KL ↓Shuffled KL ↓Δ KL ↑
ECG → CXR0.026780.03649+0.00971
Lab → CXR0.025790.03723+0.01144
ECG+Lab → CXR0.023140.03905+0.01590
CXR → ECG0.004140.00496+0.00081
Lab → ECG0.004320.00485+0.00053
CXR+Lab → ECG0.004040.00512+0.00108
CXR → Lab0.014250.01472+0.00047
ECG → Lab0.014350.01453+0.00018
CXR+ECG → Lab0.014210.01480+0.00059

W5: Generalizability to Longitudinal Multimodal Settings

We thank the reviewer for this valuable suggestion. Our current study focuses on a relatively controlled clinical setting, where modalities are aligned within a constrained early-ED window to characterize the same disease episode while minimizing the confounding effects of disease progression and treatment. Extending MIRI to longitudinal settings, where modalities are acquired at different time points, would broaden the scope and clinical applicability of our approach. However, such an extension would require a newly curated dataset to account for temporal misalignment and evolving patient conditions. We consider this a promising direction for future work and have added a corresponding discussion to the Conclusion of the revised manuscript.

Thank you for your additional results, most of my concerns have been addressed, I will maintain my supportive scores.

Reviewer_jQzqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MedGap (a MIMIC-IV based benchmark) and MIRI (two-stage missing-modality information recovery and integration framework) for medical MLLMs. Evaluation across multiple MLLMs shows improved average performance with MIRI on the proposed benchmark across missing modality settings.

Critical strengths

The proposed MedGap benchmark would be a useful benchmark for the community to evaluate MLLMs using CXR, ECG and labs from early ED encounter. The incorporation of the missing evaluation and the proposed method is conceptually sound and easy to follow. The evaluation was well-conducted across a range of MLLMs for multiple combinations of missingness.

Critical weaknesses

Originality: The proposed MedGap benchmark is interesting but it lacks a positioning among the works that consider the impact of missing modalities or the contributions of a modality on the discriminative task [1,2,3]. Regarding the proposition of MIRI, it borrows fundamental ideas from conditional VAEs but the discussion with many of the existing works is missing [4,5]. It also remains unclear why the training has to follow a two-stage process and cannot be done in an end-to-end framework?

Significance and experimental evaluation: The paper mentions that more available modalities do not automatically translate into better performance but this really depends on a model. It'd have been useful to marginalize the effect of models by for instance taking an ensemble of predictions. For 5.3, while evaluating a single modality is useful, a better experiment is usually to use the same model but shuffle the input pair to remove the effect of that modality. The paper can also incorporate partial information decomposition [6] for a more rigorous analysis. I'd recommend to incorporate these suggestions into the analysis. For MIRI, its also unclear how the paper handles multiple missing modalities and evaluation for that setting given the focus of the paper seems important. It is also unclear why recovery is essential when we are retaining the context for downstream task and it'd be useful to incorporate a discussion regarding this.

Minor: The positioning of the figures in between the text disrupts the flow of text. I'd suggest to position the figure to the top of the page. Table 3 is also challenging to read because of a) too many numbers b) the meaning of bold is unclear, there are multiple bold numbers in a single row. The paper also uses excessive vspace at many places.

[1] Gu and Fu et al. Illusion of Readiness: Stress Testing Large Frontier Models on Multimodal Medical Benchmarks.
[2] Madaan et al. Characterizing the Predictive Impact of Modalities with Supervised Latent-Variable Modeling.
[3] Liang et al. Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework.
[4] Sohn et al. Learning Structured Output Representation using Deep Conditional Generative Models.
[5] Joy et al. Learning Multimodal VAEs through Mutual Supervision.
[6] Fang et al. Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition.

Questions for authors

Please refer to weaknesses above

5 comments
AuthorsOpenReview ↗

Response [1/2]

We sincerely thank the reviewer for the constructive and insightful comments. We have carefully considered each suggestion and revised the manuscript accordingly. Our point-by-point responses to the reviewer’s comments are provided below.

W1: Originality

1. Insufficient positioning of MedGap. Prior work has examined related problems from different perspectives: [1] stress-tests medical models by removing or perturbing inputs, [2] uses supervised latent-variable modeling to quantify the predictive impact of missing modalities at the instance level, and [3] applies information decomposition to quantify unique, redundant, and synergistic modality contributions. However, these studies do not systematically characterize how missing-modality effects vary across Medical MLLM backbones and disease-relevant diagnosis targets within a unified clinical setting. MedGap addresses this gap through a unified early-ED benchmark that evaluates model- and target-dependent effects across different modality-availability settings. We have added the corresponding discussions to Lines 130–132 of the revised manuscript.

2. Insufficient positioning of MIRI. Regarding MIRI, [4] introduces conditional variational generative modeling for structured output prediction, while [5] develops multimodal VAEs through mutual supervision to handle partially observed modalities. In contrast, MIRI is motivated by many-to-many clinical correspondence and inspired by JEPA-style predictive learning rather than VAE-based generation. It matches predicted and target latent distributions without reconstructing the missing modality, and integrates recovered representations into the LLM through teacher-guided learning. Its contribution lies in combining task-agnostic cross-modal latent recovery with downstream integration under incomplete inputs.

We have clarified the positioning of MedGap in Related Work and the methodological distinction between MIRI and VAE-based approaches in the Method section Lines 246-248.

3. Unclear training design Our two-stage design is motivated by the necessity of modality recovery and the potential to learn generalizable cross-modal knowledge from large-scale paired multimodal data. As demonstrated by our parameter-matched baseline comparison (reviewer JLcb: why recovery), directly processing observed modalities without explicit missing-modality recovery is insufficient to achieve comparable performance, highlighting the importance of Stage 1. Specifically, Stage 1 learns population-level cross-modal representations from abundant task-agnostic paired data, enabling the recovery of potentially informative missing modalities beyond the limited downstream task supervision. Stage 2 then adapts the LLM to integrate the recovered representations with observed inputs for clinical prediction. To further evaluate the effectiveness of this decoupled design, we compare MIRI with an end-to-end (E2E) variant that jointly trains the recovery and prediction components under downstream task supervision.

MethodDX MiF1 ↑DX MaF1 ↑LOS Acc ↑LOS MaF1 ↑M30 F1 ↑M30 BAcc ↑
E2E43.09 ± 0.6032.91 ± 0.3538.31 ± 0.5832.15 ± 0.8029.74 ± 0.3567.31 ± 0.40
MIRI44.59 ± 0.4234.25 ± 0.4638.63 ± 0.1632.58 ± 0.9430.07 ± 0.5367.66 ± 0.26

Across three random seeds, MIRI outperforms E2E on all six metrics in terms of mean performance (revised in Appendix Table 12). The largest gains occur in diagnosis, achieving 44.59 ± 0.42 MiF1 and 34.25 ± 0.46 MaF1, compared with 43.09 ± 0.60 and 32.91 ± 0.35 for E2E, respectively. These results support the benefit of learning task-agnostic cross-modal relationships before downstream integration, rather than relying solely on downstream task supervision for missing-modality recovery.

AuthorsOpenReview ↗

Response [2/2]

W2: Significance and Experimental Evaluation

1. Modality contribution vs. model effect. We thank the reviewer for this insightful suggestion. We would like to clarify that our goal is to evaluate how existing Medical MLLMs respond to missing modalities, rather than to quantify model-independent modality contributions. Therefore, model dependency is itself an important aspect of our investigation, and using an ensemble may obscure meaningful differences in how individual models respond to modality removal. To better characterize this phenomenon, we systematically evaluate nine single-modality removal settings across seven models (Table 3). Interestingly, removing a modality can improve performance in 4/7 models for Diagnosis MiF1, 7/7 for LOS Acc, and 6/7 for M30 BAcc, demonstrating that the effects of missing modalities are not necessarily monotonic.

Meanwhile, following the reviewer's suggestion, we further conduct a modality-shuffling experiment using Hulu-Med-7B in a zero-shot setting. Specifically, we shuffle each modality across patients while keeping the model and input configuration unchanged, thereby disrupting patient-specific cross-modal correspondence. Across three modality pairs (CE, CL, and EL), shuffling consistently reduces Diagnosis MiF1 and MaF1, with average drops of 7.20% and 9.51%, respectively (revised in Appendix Table 15). These results provide complementary evidence that patient-specific cross-modal correspondence contributes to prediction performance.

PairMatched (MiF1 / MaF1)Shuffled Δ (MiF1 / MaF1)
CE43.08 / 38.71−9.92 / −14.93
CL32.37 / 30.74−3.88 / −3.89
EL38.11 / 37.38−7.80 / −9.71

Overall, these experiments demonstrate both the model-dependent and non-monotonic effects of modality removal, as well as the importance of patient-specific cross-modal correspondence.

2. PID analysis. We thank the reviewer for suggesting PID to better understand cross-modal interactions. Fang et al. [6] perform decision-level PID using predictive distributions over a constrained answer space. However, MedGap uses generative multi-label diagnosis, where models output sets of diagnoses rather than explicit classification probabilities. Directly applying their framework would require redefining disease-specific decision distributions and validating their consistency with the original generative predictions. We also explored PID using binary diagnosis predictions, but nearly constant predictions under some unimodal settings led to degenerate and difficult-to-interpret decompositions. We therefore avoid reporting potentially misleading results. We appreciate this valuable suggestion, which provides a promising direction for extending MedGap beyond missing-modality evaluation toward more detailed cross-modal information analysis in future work.

3. Unclear handling and evaluation of multiple missing modalities. When two modalities are missing, MIRI independently recovers each missing modality conditioned on the single observed modality, and then integrates the recovered representations with the observed tokens. We would like to clarify that this scenario is included in our evaluation in Table 5, under the C, E, and L configurations, where only one modality is available.

4. Necessity of recovery. Our disease-relevant paired evaluation shows that removing clinically relevant modalities can lead to substantial performance degradation, suggesting that observed modalities alone may not provide sufficient disease-relevant information. Since clinical modalities capture complementary aspects of the same patient, MIRI leverages cross-modal relationships to recover potentially informative representations of missing modalities. Importantly, these recovered representations complement rather than replace the observed inputs, enriching the context available for downstream prediction.

This benefit is further supported by Table 8. With observed inputs and KD retained, adding recovered tokens improves Diagnosis MiF1 from 40.20 to 44.61 (+4.41), MaF1 from 27.78 to 34.46 (+6.68), and LOS Acc from 33.62 to 38.79 (+5.17). These results demonstrate that recovered representations provide complementary information beyond the original observed context, supporting the necessity of modality recovery.

Minor

We thank the reviewer for these helpful suggestions. We have revised the manuscript layout to improve readability, including adjusting figure placement and reducing excessive vertical spacing. For Table 3, we have also clarified the notation: bold font indicates the best result in each column, rather than in each row.

Reviewer_jQzqOpenReview ↗

Response to authors

While I appreciate the authors response, unfortunately, most of my concerns remain unaddressed in the rebuttal and thus I maintain my rating.

  • Originality: For insufficient positioning, the authors respond by highlighting what existing works that I mentioned do without really comparing how their work is different. For example, [1] did evaluate on medical MLLMs, [2] evaluated across multiple modality missing scenarios, remains unclear why information decomposition [3] cannot be used to analyze the achieved task.

Similarly, my question was about why modality recovery is essential, but the response mentions this as a necessary condition instead. The response also seems unsatisfying to answer the case for more than two modalities. Why is it essential to "recover each missing modality conditioned on the single observed modality" and why is this a computational feasible choice?

  • Modality contribution: While I understand the focus of the paper, it remains unclear why quantifying the modality contribution directly would not inform us "how existing Medical MLLMs respond to missing modalities?"
Reviewer_jLcbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MedGap, a benchmark built from 20,549 emergency-department encounters with CXR, ECG, and laboratory data. It evaluates diagnosis, length of stay, and 30-day mortality across modality-availability settings, with an additional disease-specific paired diagnosis evaluation. It also proposes MIRI, which learns Gaussian latent representations, predicts missing-modality representations from observed inputs, and integrates their means into an MLLM using bridges and full-modality teacher distillation. Experiments evaluate seven models in zero-shot settings and apply MIRI to three backbones. The reported results show that modality effects depend on the model and diagnosis target, while MIRI improves average missing-modality performance, particularly for diagnosis.

Critical strengths
  1. The benchmark offers a useful, target-aware evaluation of missing modalities. Patient-disjoint splits and comparisons on the same encounters help control patient and case differences. The disease-specific analysis further distinguishes which modality is missing and which diagnosis is being predicted, rather than treating modality count alone as the explanation for performance. This makes the benchmark informative for understanding model behavior beyond aggregate missing-modality scores.

  2. The experiments provide meaningful evidence for the recovery-and-integration pipeline. MIRI improves average diagnosis performance across three backbones, and the component ablations separate recovery from teacher distillation. On Hulu-Med-7B, removing recovery while retaining distillation reduces diagnosis MiF1 from 44.61 to 40.20. The patient-disjoint probing analysis provides complementary evidence that recovered representations make target-modality-associated findings more linearly accessible.

Critical weaknesses
  1. The advantage of probabilistic prediction is unclear. MIRI predicts a distribution for the missing features, but the experiments only compare using recovery with removing it. A simpler alternative is to predict the features directly, without modeling a distribution. Comparing these approaches under the same conditions is needed to show whether probabilistic modeling contributes to the gains.

  2. The need to predict the missing modality is unclear. When CXR is missing, MIRI uses ECG and Lab to predict CXR features. The paper does not compare this with adding a similarly sized network that processes ECG and Lab features directly for the same prediction task, without predicting CXR features. This control would clarify whether predicting the missing modality helps beyond adding extra feature processing.

Questions for authors
  1. Have you compared predicting a distribution for the missing features with predicting those features directly, without a distribution? Under otherwise matched training conditions, does probabilistic prediction provide an additional benefit?

  2. When CXR is missing, have you compared predicting CXR features from ECG and Lab with using a similarly sized network to process ECG and Lab features directly for clinical prediction? With distillation and training budgets matched, does predicting the missing modality perform better?

4 comments
AuthorsOpenReview ↗

Response [1/2]

We sincerely thank the reviewer for this valuable suggestion. We acknowledge the reviewer's concerns regarding the necessity of the recovery mechanism (W2 & Q2) and the advantages of the probabilistic design for recovery (W1 & Q1). In the following, we elaborate on these two aspects from the perspectives of design motivation and experimental evaluation.

Motivation.

Why recovery: Our motivation for modality recovery is that missing modalities may contain critical disease-relevant information that cannot be fully captured by the observed modalities alone. This is supported by our modality-paired evaluation, where removing clinically relevant modalities leads to substantial performance degradation (Table 4).

Why probabilistic: Nevertheless, missing-modality recovery is challenging due to the inherent many-to-many correspondence across clinical modalities, where the observed modalities may correspond to multiple plausible missing-modality representations (Figure 1). Consequently, deterministic point-to-point prediction may fail to adequately capture such cross-modal uncertainty. To address this challenge, MIRI aims to learn population-level cross-modal knowledge from a large number of paired patient observations, rather than relying solely on individual-level correspondence. Specifically, our probabilistic design captures statistical cross-modal relationships across patients and models the associated uncertainty, providing informative priors for missing-modality recovery rather than pursuing exact individual-level reconstruction.

ComparisonDeterministic point-to-point predictionMIRI's probabilistic design
Learning perspectiveIndividual-level predictionPopulation-level knowledge learning
Prediction outputSingle predicted representationDistribution (mean and variance)
Uncertainty modelingLimited explicit uncertainty modelingExplicitly models cross-modal uncertainty
Cross-modal correspondencePoint-to-point mappingProbabilistic many-to-many correspondence
Recovery objectivePredict a specific missing representationEstimate missing representations using population-level statistical priors
AuthorsOpenReview ↗

Response [2/2]

Experiments

Why recovery: Through probabilistic cross-modal prediction in Stage 1, MIRI learns statistical relationships between modalities across many patients. These learned relationships provide useful priors for estimating missing-modality representations from incomplete patient observations, beyond simply processing the available features.

To validate this, following the reviewer's suggestion, we introduce a parameter-matched (PM) baseline that directly processes the observed modalities without explicitly predicting missing modalities, while maintaining the same training budget and Stage 2 integration procedure.

Across three random seeds, MIRI outperforms PM on all six metrics (revised in Appendix Table 12), with notable improvements in diagnosis. MIRI achieves 44.59 ± 0.42 MiF1 and 34.25 ± 0.46 MaF1, compared with 43.00 ± 0.57 and 32.85 ± 0.08 for Direct, respectively. These results support that explicit cross-modal predictive supervision provides benefits beyond additional feature-processing capacity, enabling more effective use of population-level multimodal knowledge for downstream prediction.

MethodDX MiF1 ↑DX MaF1 ↑LOS Acc ↑LOS MaF1 ↑M30 F1 ↑M30 BAcc ↑
PM43.00 ± 0.5732.85 ± 0.0837.97 ± 0.4132.05 ± 0.8429.75 ± 0.3467.16 ± 0.58
MIRI44.59 ± 0.4234.25 ± 0.4638.63 ± 0.1632.58 ± 0.9430.07 ± 0.5367.66 ± 0.26

Why probabilistic: We introduce a deterministic recovery baseline that replaces Gaussian distribution prediction and KL matching in Stage 1 with direct latent feature prediction optimized by MSE. We retain the same latent target space, training data, predictor backbone, and Stage 2 teacher-guided integration procedure, isolating the contribution of probabilistic recovery. (1) Recovery quality. Following the same linear-probe protocol as Table 7, probabilistic prediction outperforms deterministic MSE prediction in 8 of 9 recovery directions (revised in Appendix Table 10), improving average AUROC from 65.61 to 67.02 (+1.41%). The improvements are particularly evident for CXR+Lab → ECG (+3.18%) and ECG → CXR (+2.37%). (2) Downstream effectiveness. Across three random seeds, probabilistic recovery outperforms deterministic MSE prediction on all six downstream metrics in terms of mean performance (revised in Appendix Table 12). The largest gains occur in diagnosis, achieving 44.59 ± 0.42 MiF1 and 34.25 ± 0.46 MaF1, compared with 43.45 ± 0.83 and 32.75 ± 0.68 for deterministic recovery, respectively. Gains on LOS and M30 are more modest, potentially because these prognostic outcomes depend on broader clinical factors beyond modality-specific findings. In contrast, the stronger improvements in diagnosis may reflect its more direct dependence on modality-specific clinical evidence, making it a central focus of our investigation into missing-modality effects.

Recovery DirectionMIRIMSEΔ
ECG → CXR69.3967.02−2.37
Lab → CXR71.2971.04−0.25
ECG+Lab → CXR74.4273.00−1.42
CXR → ECG65.5164.39−1.12
Lab → ECG60.4061.47+1.07
CXR+Lab → ECG68.3365.15−3.18
CXR → Lab65.6263.46−2.16
ECG → Lab61.2559.65−1.60
CXR+ECG → Lab67.0165.31−1.70
Average67.0265.61−1.41
MethodDX MiF1 ↑DX MaF1 ↑LOS Acc ↑LOS MaF1 ↑M30 F1 ↑M30 BAcc ↑
MSE43.45 ± 0.8332.75 ± 0.6838.11 ± 0.2732.31 ± 0.3729.47 ± 0.2267.17 ± 0.47
MIRI44.59 ± 0.4234.25 ± 0.4638.63 ± 0.1632.58 ± 0.9430.07 ± 0.5367.66 ± 0.26

These results show that the benefits of MIRI are not solely attributable to the presence of a recovery module. Under matched experimental conditions, probabilistic distribution matching yields more informative recovered representations and generally improves their downstream utility compared with deterministic feature regression.

Thank you for adding the parameter-matched and deterministic recovery baselines. These comparisons address my original experimental requests. My remaining reservation concerns the magnitude and scope of the additional contribution: improvements over deterministic recovery are modest, especially for LOS and mortality. I retain my rating of 2.

We appreciate the reviewer’s recognition that recovery itself is beneficial, and we agree that the remaining question concerns the additional benefit of probabilistic recovery over deterministic recovery. Table 8 shows that adding recovered representations consistently improves downstream performance, confirming the benefit of recovery. The MSE comparison further evaluates the incremental benefit of our probabilistic formulation. While this benefit is clear in recovery quality and diagnosis, we acknowledge that the gains are more modest for LOS and mortality.

We therefore report these task-dependent results transparently rather than selectively emphasizing tasks with larger improvements. In particular, the relatively modest gains over MSE on LOS and mortality also highlight the remaining challenges of missing-modality recovery for clinical prognostic tasks. More broadly, we hope that MedGap, as a curated benchmark, can provide a useful testbed for the community to further investigate and improve missing-modality learning for clinical prognosis.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.