- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Summary
This paper studies the missing modality problems on medical multimodal large language models. The authors introduce the MedGap, a benchmark covering CXR, ECG, and laboratory data. It further proposes the MIRI, which combines probabilistic cross-modal recovery and teacher-guided integration to improve performance under missing modalities. Experiments across multiple datasets and medical MLLMs show good results.
Critical strengths
Strengths
- Overall, I think the motivation is meaningful, as missing modalities are common in real clinical settings and can affect the reliability of medical MLLMs.
- This multi-modal setting itself is valuable, it's good to see papers trying to bridge physiological signals to other medical modalities like medical imaging and EHR text.
- The proposed MedGap benchmark is useful.
- The proposed MIRI framework is interesting, and the experiments cover multiple models and missing-modality settings. The ablation studies also provide useful insights into the effectiveness of different components.
Critical weaknesses
Weaknesses
- I notice that MedGap is constructed from encounters with all three modalities available, and missing modalities are simulated by removing the inputs. However, I feel real-world missingness may not be random and could be related to patient conditions or clinical decisions. I wonder whether the conclusions can generalize to such realistic missingness patterns? Would it be possible to include some evaluation with naturally missing modalities?
- The ED / clinical settings have natural missing channel/missing modality scenarios. For example, in the MC-MED or MIMIC-III/MIMIC-IV datasets, there are numerics like HR, SPO2, and waveform modalities like ECG, PPG, they are often missing channels. It's also interesting to try or discuss if the proposed method can generalize to MLLMs in these datasets and method [1,2].
- The motivation for probabilistic recovery is reasonable, but I notice that only the predicted mean is used as the recovered representation during inference. I wonder how the uncertainty modeling actually benefits downstream prediction in this case? Would it be possible to compare with a simpler deterministic latent prediction method to better justify this design?
- I think the recovery evaluation in Table 7 is interesting, but I am not fully convinced that the improved linear probing performance necessarily indicates successful recovery of missing-modality information. Can you elaborate more?
- I'm also interested to dicuss: whether this paradigm generalize to a more practical settings that we have modality A in time A, and modality B in time B. It's realistic, because people may do lab test and have medication order, and have different medical imaging exams at different time points. Jointly modeling these modalities in a longitudinal scope might be a good future work.
[1] Dai D, She J, Cheong J, et al. QoQ-Med3: a multimodal reasoning foundation model for clinical analysis[J]. npj Digital Medicine, 2026.
[2] Xu Y, Shuai Z, Yang Y. Sensor-Language-Action Models[J]. arXiv preprint arXiv:2610.08244, 2026.
Questions for authors
See weakness
3 comments
Response [2/2]
W3: Effectiveness of Probabilistic Recovery and Uncertainty Modeling We thank the reviewer for raising this important question. Although only the predicted mean is used as the recovered representation during inference, probabilistic modeling provides distribution-level supervision during Stage 1 training. Specifically, by aligning the predicted and target latent distributions through KL divergence, MIRI learns cross-modal representations while explicitly accounting for uncertainty in cross-modal correspondence, rather than relying solely on point-wise feature matching.
To further validate the benefit of probabilistic modeling, we replace the probabilistic recovery objective with deterministic MSE prediction while keeping all other training conditions unchanged. As shown in the first table, MIRI achieves an average recovery probe AUROC of 67.02, outperforming deterministic MSE prediction (65.61) by 1.41%.
More importantly, we evaluate the impact on downstream tasks. Across three random seeds, MIRI achieves higher mean performance than deterministic MSE prediction on all six downstream metrics (revised in Appendix Table 12). For diagnosis, MIRI improves MiF1 from 43.45 ± 0.83 to 44.59 ± 0.42 and MaF1 from 32.75 ± 0.68 to 34.25 ± 0.46. Improvements are also observed across all LOS and M30 metrics. These results demonstrate that probabilistic recovery provides benefits beyond latent representation prediction, improving the quality of recovered representations for downstream clinical tasks, even when only the predicted mean is used during inference.
| Recovery Direction | MIRI | MSE | Δ (MIRI − MSE) |
|---|---|---|---|
| ECG → CXR | 69.39 | 67.02 | +2.37 |
| Lab → CXR | 71.29 | 71.04 | +0.25 |
| ECG+Lab → CXR | 74.42 | 73.00 | +1.42 |
| CXR → ECG | 65.51 | 64.39 | +1.12 |
| Lab → ECG | 60.40 | 61.47 | −1.07 |
| CXR+Lab → ECG | 68.33 | 65.15 | +3.18 |
| CXR → Lab | 65.62 | 63.46 | +2.16 |
| ECG → Lab | 61.25 | 59.65 | +1.60 |
| CXR+ECG → Lab | 67.01 | 65.31 | +1.70 |
| Average | 67.02 | 65.61 | +1.41 |
| Method | DX MiF1 ↑ | DX MaF1 ↑ | LOS Acc ↑ | LOS MaF1 ↑ | M30 F1 ↑ | M30 BAcc ↑ |
|---|---|---|---|---|---|---|
| MSE | 43.45 ± 0.83 | 32.75 ± 0.68 | 38.11 ± 0.27 | 32.31 ± 0.37 | 29.47 ± 0.22 | 67.17 ± 0.47 |
| MIRI | 44.59 ± 0.42 | 34.25 ± 0.46 | 38.63 ± 0.16 | 32.58 ± 0.94 | 30.07 ± 0.53 | 67.66 ± 0.26 |
W4: Validity of Missing-Modality Recovery Evaluation
To further validate whether the recovered representations capture patient-specific missing-modality information beyond the disease-predictive information assessed by linear probing, we compare the KL divergence between predicted and true target distributions for the same patient (Matched) and different patients (Shuffled). As revised in Appendix Table 11, Matched KL is consistently lower than Shuffled KL across all nine recovery directions, with average values of 0.01456 and 0.01908, respectively (a 31.1% increase after shuffling). This indicates that the predicted distributions capture patient-specific cross-modal information aligned with the corresponding patient's true missing-modality representations, rather than merely learning generic modality representations. Together with Table 7, these results provide complementary evidence for meaningful missing-modality recovery.
| Recovery Direction | Matched KL ↓ | Shuffled KL ↓ | Δ KL ↑ |
|---|---|---|---|
| ECG → CXR | 0.02678 | 0.03649 | +0.00971 |
| Lab → CXR | 0.02579 | 0.03723 | +0.01144 |
| ECG+Lab → CXR | 0.02314 | 0.03905 | +0.01590 |
| CXR → ECG | 0.00414 | 0.00496 | +0.00081 |
| Lab → ECG | 0.00432 | 0.00485 | +0.00053 |
| CXR+Lab → ECG | 0.00404 | 0.00512 | +0.00108 |
| CXR → Lab | 0.01425 | 0.01472 | +0.00047 |
| ECG → Lab | 0.01435 | 0.01453 | +0.00018 |
| CXR+ECG → Lab | 0.01421 | 0.01480 | +0.00059 |
W5: Generalizability to Longitudinal Multimodal Settings
We thank the reviewer for this valuable suggestion. Our current study focuses on a relatively controlled clinical setting, where modalities are aligned within a constrained early-ED window to characterize the same disease episode while minimizing the confounding effects of disease progression and treatment. Extending MIRI to longitudinal settings, where modalities are acquired at different time points, would broaden the scope and clinical applicability of our approach. However, such an extension would require a newly curated dataset to account for temporal misalignment and evolving patient conditions. We consider this a promising direction for future work and have added a corresponding discussion to the Conclusion of the revised manuscript.
Thank you for your additional results, most of my concerns have been addressed, I will maintain my supportive scores.
Response [1/2]
We sincerely thank the reviewer for the positive assessment and constructive suggestions. We appreciate the recognition of our work and have carefully considered all the concerns. Below, we provide point-by-point responses and corresponding revisions to further clarify and strengthen our work.
W1: Generalizability to Real-World Missingness Patterns
To evaluate generalizability beyond simulated missingness, we randomly sampled 1,000 naturally incomplete ED encounters from MIMIC, preserving their original modality-availability patterns (EL: 547, L: 368, CL: 73, E: 8, C: 2, and CE: 2).
MIRI outperforms SFT on all metrics (revised in Appendix Table 16 ), with notable improvements in diagnosis (+2.53% MiF1 and +4.32% MaF1). These results further support the effectiveness of MIRI under naturally occurring missingness, particularly for diagnosis.
W2: Generalizability to Clinical Signal Modalities We thank the reviewer for this valuable suggestion. Our current study focuses on modality-level missingness across CXR, ECG, and laboratory data. Since MIRI performs recovery in the latent token space rather than reconstructing raw inputs, its underlying recovery mechanism is not inherently tied to specific data modalities and could potentially be extended to finer-grained missingness scenarios, such as missing physiological signal channels, with appropriate modality representations and recovery objectives. Although such extensions are beyond the scope of the current study, we believe they offer a promising direction for future research. We have incorporated both references [1, 2] into the Related Work section and discussed this potential extension in Lines 119–120 of the revised manuscript.