- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes LAMAE, a self-supervised framework that pretrains jointly on ECG, echocardiography, and chest X-ray. The model is pretrained on about 528k MIMIC-IV “hospital stays” split by patient. It is then fine-tuned on 50k labelled stays for six stay-level tasks: in-hospital mortality, ICD-10 Chapter IX coding, DRG code, DRG severity, DRG mortality risk, and 3-class length of stay. It is compared with two outside baselines, ProbMED and MedSigLip, and with an “Independent MAE” that uses the same encoders without the latent-attention module or entity masking. The results showed LAMAE scored best on five of the six tasks
Critical strengths
- The work targeted an Important, realistic problem. Combining multiple modalities in clinical data is a very active area of research. It is good to see a new attempt at a single model that can use several modalities (ECG, Echo and CXR) and handle whichever subset is available for a patient.
- The presented work is clear and easy to follow. Modality-specific encoders, one shared latent-attention module, and entity-level masking on top of a standard MAE. The study–view–entity formulation could extend to other multi-view clinical data.
- The work used broad evaluation with useful controls, and released code.
Critical weaknesses
The idea seems very promising and straightforward, but there are a few critical issues I have concerns about:
-
The paper proposed LAMAE by the joint structure of ECG, Echo and CXR, and Fig. 1 shows a stay with all three. By Fig. 4, however, no stay contains all three modalities, and no stay contains both CXR and Echo. The latent-attention module therefore never sees CXR and Echo tokens together during pretraining. The paper also states that the model can run on any subset of modalities, but it has never been trained on the CXR+Echo or three-modality combinations, so its performance under these settings is untested.
-
The paper did not explain in detail how different modality data are matched. From my understanding, a stay is an entire ED visit or an entire hospital admission, running from admission to discharge or death. The paper says ECG and Echo are “sampled at random from the stay”. So my concern is that the inputs for in-hospital mortality and length of stay can include scans taken shortly before discharge or death. For example, an admission ECG from a critically ill patient can be paired with its CXR taken just before discharge, by which point the patient has recovered. These tasks therefore measure retrospective recognition, not prediction. Under this setting, the multimodality model will accurately predict the survival of the patient compared to a single-modality ECG model that predicted death, but this might be due to potential temporal leakage. Please correct me if I'm wrong and clarify this in the main paper.
-
The paper doesn’t explain how a fully dropped lead, frame, or modality is represented. A dropped entity has no encoder output, so it is unclear where its output embedding z_out in Eqs. 4–5 comes from, or how the decoder reconstructs it.
-
One of the central claims is that LAMAE learns better stay- and patient-level representations. The only evidence, however, is downstream AUROC after full finetuning of the whole network, which mixes representation quality with finetuning capacity. Classification performance alone does not show that the representation is improved through multi-modal pretraining. The paper would be much stronger with direct analyses of the embedding space: such as qualitative t-SNE/UMAP plots of stay embeddings coloured by diagnosis, outcome, encounter type and available modalities. quantitative clustering metrics against diagnostic groups, linear probing or k-NN evaluation on frozen embeddings, reconstruction of a held-out modality from the others.
Questions for authors
- Reconstructing each modality with help from the others pulls a stay’s scans towards one shared representation, which is a form of implicit alignment. How do the authors see LAMAE relative to multimodal contrastive alignment such as CLIP or SigLIP?
- The multimodal pretraining assumes the paired inputs describe the same underlying state, but the patient's health condition is dynamic during the stay due to different medications and interventions. Does performance or cross-modal reconstruction change with the time gap between paired scans? How does the method address this issue?
4 comments
Dear Reviewer zBgo, thank you again for your feedback. We have now posted the remaining results (see our update above). They directly address your concerns: the temporal ablation shows that LAMAE's gains hold with only early observations (Tables 12–13), modality completion evaluates stays that contain all three modalities (Table 15), and linear probing now includes all baselines (Tables 9-10). We believe your suggestions have substantially strengthened the paper and would be grateful if you would consider raising your score.
Thank you for the detailed response and the revision. The revised version addressed several of my concerns. Three issues remain:
- Retrospective setting and clinical usefulness. Since the authors' reply mentioned that inputs are drawn from the entire stay and acquisition times are not used, so the model cannot be applied at a defined decision point. For in-hospital mortality and length of stay, the outcome is already known by the time all of a stay’s observations are available, so it is unclear when these predictions would be useful in practice. The temporal ablation in Appendix J is a useful step, but: Is the window applied only at test time, or were the models also fine-tuned on windowed inputs? If only at test time, all models are still trained with access to late observations. On mortality, LAMAE’s lead over Independent MAE falls from +1.76 on the whole stay to +0.97 in the first 24 h, and to about zero in the first 6 h after coverage adjustment. Modalities are acquired at different points in a stay (e.g., an ECG typically early, an echocardiogram often later), so restricting the window changes which modalities are available. Please also report the multimodal-vs-unimodal comparison (Table 2) under the same windows. I would suggest making a prediction-time protocol (e.g., inputs from the first 24 h after admission, used for both fine-tuning and testing) the main evaluation for mortality and length of stay, following other established works in this field such as ETHOS, so the performance are more comparable.
- Cohort definition is inconsistent. Section 4.1 and Appendix C define a stay as an ED, inpatient or outpatient encounter, but Appendix D says all hospital-stay tasks are “defined per MIMIC-IV admission”, and the rebuttal says a stay is “one MIMIC-IV admission”. This matters for the results. Please clarify the definition and report mortality and length of stay on inpatient admissions only.
- I found the results for LAMAE_CXR in Table 2 have changed since the first submission version. Please explain what was the reason that caused this change.
Dear reviewer, thank you for the discussion. Please find below our reply:
1. Retrospective setting. Our goal is representation learning, not clinical prediction. We test whether information exchange during pretraining yields better representations, and we measure their quality with downstream tasks, with identical inputs for all models. The tasks are probes, not deployment targets (Sections 4.1–4.2, Ethics Statement). We added the temporal ablation on request. Since this work is not a clinical prediction model, we see it as showing that the representations also yield useful predictions from early parts of the stay. As the reviewer observed, the mortality advantage over the Independent MAE shrinks toward 6 h. At 6 h, only 2.4% of stays have more than one modality, so there is little for a multimodal model to integrate and similar performance is expected. As the window grows, more stays contain several modalities, and the margin increases to +1.76 for the whole stay. That is what we expect from a model that gains by integrating observations. A clinical model with a prediction-time protocol, as in ETHOS, is future work; a windowed Table 2 is not feasible in the remaining time, nor would it significantly strengthen our evaluation.
2. Cohort definition. As specified in the manuscript Appendix Section C : Throughout the paper, a hospital stay is one MIMIC-IV encounter: an emergency department visit, an inpatient admission or an outpatient visit (as defined in Section 4.1, Appendix C). The term “admission” in Appendix D and in our earlier reply refers to a hospital stay in this sense. All models are evaluated on the same set of stays, so the definition does not favour any method. Stratifying mortality and length of stay by encounter type is an interesting analysis, which we leave to future work given the remaining time in the discussion period.
3. Table 2. The LAMAE$_{\text{CXR}}$ row was computed over all stays instead of the stays with at least one modality, as for all other rows. Only this row changed, in favor of the baseline, and the conclusions are unchanged.
We thank the reviewer for the thoughtful and constructive feedback, and for recognizing the importance of the problem, the clarity of the presentation and the breadth of our evaluation. We address each weakness and question below. New analyses are summarized in the general response; changes in the revised manuscript are marked in red.
(W1) Modality combinations. We agree that paired data is scarce (Fig. 4). However, this reflects an artifact of the data: MIMIC-CXR covers [2011–2016] and MIMIC-IV-Echo [2017–2019], so CXR and Echo never co-occur within a stay. Handling such incomplete data reflects the challenges of the medical domain, which is the setting LAMAE is designed for. Unlike contrastive methods, LAMAE learns from any observed subset. LAMAE can technically process unseen combinations such as CXR+Echo+ECG, but these cannot be evaluated in the current dataset. Nevertheless, joint pretraining improves representations even with largely unpaired data: in the unimodal results (Table 3), LAMAE$_\text{MM}$ outperforms the unimodal LAMAE models on CXR and ECG, and matches them on Echo.
(W2) Temporal leakage. A hospital stay is one MIMIC-IV admission, and observations are sampled from the whole stay without modeling acquisition time, i.e., the model has no information about the time of an acquisition. The hospital-stay tasks are therefore retrospective, stay-level benchmarks of representation quality. In Table 1, all models receive the same observations, so this setting does not favor LAMAE over the baselines. We have clarified this in the revised manuscript (Sections 4.1–4.2) and are running additional experiments restricting the input to early observations in the stay.
(W3) Dropped entities. We thank the reviewer for raising this point and added more details to Section 3.3. A dropped entity is represented by a learned mask token, which receives the same structural embeddings as visible tokens (position, entity, view, modality), so the model knows which lead, frame, or view it stands for. In the LA module, these tokens act only as queries: they attend to all visible tokens of the stay, but no token attends to them, since they carry no content. Their LA output is z_out in Eqs. 4–5, from which the modality-specific decoder reconstructs the dropped entity. Entire modalities are never dropped during pretraining. Modalities missing from a stay are padded and excluded from attention and the loss (Appendix A). Entire modalities are never dropped during pretraining. Previous work has highlighted the challenges of performing complete cross-modal reconstruction (Daunhawer et al., ICLR 2022, https://openreview.net/forum?id=w-CPUXXrAj). We're still investigating on how to overcome this in our setting, and it’s an important research question part of our future work.
(W4) Representation quality. We thank the reviewer for this suggestion and added analyses of the frozen embeddings with linear probes (Appendix F) and t-SNE (Appendix E) (general response, points 1–2). Linear probes on LAMAE MM outperform the Independent MAE, and the t-SNE shows a clear separation of frozen embeddings for several cardiac-related tasks (e.g., cardiomegaly and atrial fibrillation), confirming the quality of the representations both quantitatively and qualitatively. In addition, we ran a label ablation (Fig. 9) in which we progressively reduced the number of labeled training samples, down to only 9 positive cases in the training set. The gap between LAMAE MM and the Independent MAE widens as the number of labels decreases, showing that LAMAE learns representations that require fewer labeled samples to solve downstream tasks. We believe these analyses substantially strengthen the paper.
(Q1) Relation to contrastive alignment. We agree that LAMAE induces a form of implicit alignment. It differs from CLIP/SigLIP in two ways. First, contrastive learning requires paired samples, which are rare even in MIMIC-IV (Fig. 4), whereas LAMAE learns from any observed subset. Second, contrastive objectives force observations from the same stay to agree, even when acquired at different physiological states. LAMAE exchanges information without enforcing agreement, and a t-SNE of the stay embeddings shows that modality-specific structure is preserved (Appendix E).
(Q2) Dynamic patient state. We thank the reviewer for this important point: the patient's state can indeed change during a stay due to medications and interventions. LAMAE does not assume that observations within a stay describe the same state. Unlike contrastive objectives, it does not force them to agree (see Q1), and the LA module can learn how much to draw on other observations. Encoding acquisition time explicitly is a natural extension, which we leave to future work. We are also running additional experiments restricting the input to early observations in the stay (see W2).