acceptodds
Under review as a conference paper at ICLR 2027

Reading the Whole Heart: Latent-Attention Masked Autoencoders for Multimodal Cardiac Representation Learning

Abstract

Cardiovascular diagnosis and treatment rest on integrating complementary modalities, such as electrocardiogram, echocardiography, and chest X-rays, each capturing distinct but complementary aspects of cardiac pathophysiology. Yet most medical foundation models remain modality-specific, combining modalities only for finetuning or post-training. This discards the cross-modal evidence clinicians naturally integrate and ignores the structure within each modality. We introduce Latent-Attention Masked Autoencoder (LAMAE), a multimodal, structure-aware masked autoencoder that jointly learns patient-level representations during self-supervised pretraining. Instead of fusing modalities post hoc, LAMAE exchanges information directly in the latent space through a shared latent-attention module operating over a study–view–entity hierarchy, enabling aggregation of variable observations and handling of missing modalities. Pretrained on over MIMIC-IV hospital stays, LAMAE outperforms modality-specific pretraining and strong contrastive and vision–language baselines across multimodal hospital-stay tasks, such as in-hospital mortality, ICD-10 and DRG coding, and length of stay, while remaining competitive on unimodal tasks. Modeling this structure also pays off within a single modality: even without cross-modal information, the latent-attention module improves representations over modality-specific pretraining.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Learning, Cardiovascular AI, Self-Supervised Learning, Representation Learning, Masked Autoencoders, Medical Foundation Models, Medical Data Science

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_zBgoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LAMAE, a self-supervised framework that pretrains jointly on ECG, echocardiography, and chest X-ray. The model is pretrained on about 528k MIMIC-IV “hospital stays” split by patient. It is then fine-tuned on 50k labelled stays for six stay-level tasks: in-hospital mortality, ICD-10 Chapter IX coding, DRG code, DRG severity, DRG mortality risk, and 3-class length of stay. It is compared with two outside baselines, ProbMED and MedSigLip, and with an “Independent MAE” that uses the same encoders without the latent-attention module or entity masking. The results showed LAMAE scored best on five of the six tasks

Critical strengths
  1. The work targeted an Important, realistic problem. Combining multiple modalities in clinical data is a very active area of research. It is good to see a new attempt at a single model that can use several modalities (ECG, Echo and CXR) and handle whichever subset is available for a patient.
  2. The presented work is clear and easy to follow. Modality-specific encoders, one shared latent-attention module, and entity-level masking on top of a standard MAE. The study–view–entity formulation could extend to other multi-view clinical data.
  3. The work used broad evaluation with useful controls, and released code.
Critical weaknesses

The idea seems very promising and straightforward, but there are a few critical issues I have concerns about:

  1. The paper proposed LAMAE by the joint structure of ECG, Echo and CXR, and Fig. 1 shows a stay with all three. By Fig. 4, however, no stay contains all three modalities, and no stay contains both CXR and Echo. The latent-attention module therefore never sees CXR and Echo tokens together during pretraining. The paper also states that the model can run on any subset of modalities, but it has never been trained on the CXR+Echo or three-modality combinations, so its performance under these settings is untested.

  2. The paper did not explain in detail how different modality data are matched. From my understanding, a stay is an entire ED visit or an entire hospital admission, running from admission to discharge or death. The paper says ECG and Echo are “sampled at random from the stay”. So my concern is that the inputs for in-hospital mortality and length of stay can include scans taken shortly before discharge or death. For example, an admission ECG from a critically ill patient can be paired with its CXR taken just before discharge, by which point the patient has recovered. These tasks therefore measure retrospective recognition, not prediction. Under this setting, the multimodality model will accurately predict the survival of the patient compared to a single-modality ECG model that predicted death, but this might be due to potential temporal leakage. Please correct me if I'm wrong and clarify this in the main paper.

  3. The paper doesn’t explain how a fully dropped lead, frame, or modality is represented. A dropped entity has no encoder output, so it is unclear where its output embedding z_out in Eqs. 4–5 comes from, or how the decoder reconstructs it.

  4. One of the central claims is that LAMAE learns better stay- and patient-level representations. The only evidence, however, is downstream AUROC after full finetuning of the whole network, which mixes representation quality with finetuning capacity. Classification performance alone does not show that the representation is improved through multi-modal pretraining. The paper would be much stronger with direct analyses of the embedding space: such as qualitative t-SNE/UMAP plots of stay embeddings coloured by diagnosis, outcome, encounter type and available modalities. quantitative clustering metrics against diagnostic groups, linear probing or k-NN evaluation on frozen embeddings, reconstruction of a held-out modality from the others.

Questions for authors
  1. Reconstructing each modality with help from the others pulls a stay’s scans towards one shared representation, which is a form of implicit alignment. How do the authors see LAMAE relative to multimodal contrastive alignment such as CLIP or SigLIP?
  2. The multimodal pretraining assumes the paired inputs describe the same underlying state, but the patient's health condition is dynamic during the stay due to different medications and interventions. Does performance or cross-modal reconstruction change with the time gap between paired scans? How does the method address this issue?
4 comments

We thank the reviewer for the thoughtful and constructive feedback, and for recognizing the importance of the problem, the clarity of the presentation and the breadth of our evaluation. We address each weakness and question below. New analyses are summarized in the general response; changes in the revised manuscript are marked in red.

(W1) Modality combinations. We agree that paired data is scarce (Fig. 4). However, this reflects an artifact of the data: MIMIC-CXR covers [2011–2016] and MIMIC-IV-Echo [2017–2019], so CXR and Echo never co-occur within a stay. Handling such incomplete data reflects the challenges of the medical domain, which is the setting LAMAE is designed for. Unlike contrastive methods, LAMAE learns from any observed subset. LAMAE can technically process unseen combinations such as CXR+Echo+ECG, but these cannot be evaluated in the current dataset. Nevertheless, joint pretraining improves representations even with largely unpaired data: in the unimodal results (Table 3), LAMAE$_\text{MM}$ outperforms the unimodal LAMAE models on CXR and ECG, and matches them on Echo.

(W2) Temporal leakage. A hospital stay is one MIMIC-IV admission, and observations are sampled from the whole stay without modeling acquisition time, i.e., the model has no information about the time of an acquisition. The hospital-stay tasks are therefore retrospective, stay-level benchmarks of representation quality. In Table 1, all models receive the same observations, so this setting does not favor LAMAE over the baselines. We have clarified this in the revised manuscript (Sections 4.1–4.2) and are running additional experiments restricting the input to early observations in the stay.

(W3) Dropped entities. We thank the reviewer for raising this point and added more details to Section 3.3. A dropped entity is represented by a learned mask token, which receives the same structural embeddings as visible tokens (position, entity, view, modality), so the model knows which lead, frame, or view it stands for. In the LA module, these tokens act only as queries: they attend to all visible tokens of the stay, but no token attends to them, since they carry no content. Their LA output is z_out in Eqs. 4–5, from which the modality-specific decoder reconstructs the dropped entity. Entire modalities are never dropped during pretraining. Modalities missing from a stay are padded and excluded from attention and the loss (Appendix A). Entire modalities are never dropped during pretraining. Previous work has highlighted the challenges of performing complete cross-modal reconstruction (Daunhawer et al., ICLR 2022, https://openreview.net/forum?id=w-CPUXXrAj). We're still investigating on how to overcome this in our setting, and it’s an important research question part of our future work.

(W4) Representation quality. We thank the reviewer for this suggestion and added analyses of the frozen embeddings with linear probes (Appendix F) and t-SNE (Appendix E) (general response, points 1–2). Linear probes on LAMAE MM outperform the Independent MAE, and the t-SNE shows a clear separation of frozen embeddings for several cardiac-related tasks (e.g., cardiomegaly and atrial fibrillation), confirming the quality of the representations both quantitatively and qualitatively. In addition, we ran a label ablation (Fig. 9) in which we progressively reduced the number of labeled training samples, down to only 9 positive cases in the training set. The gap between LAMAE MM and the Independent MAE widens as the number of labels decreases, showing that LAMAE learns representations that require fewer labeled samples to solve downstream tasks. We believe these analyses substantially strengthen the paper.

(Q1) Relation to contrastive alignment. We agree that LAMAE induces a form of implicit alignment. It differs from CLIP/SigLIP in two ways. First, contrastive learning requires paired samples, which are rare even in MIMIC-IV (Fig. 4), whereas LAMAE learns from any observed subset. Second, contrastive objectives force observations from the same stay to agree, even when acquired at different physiological states. LAMAE exchanges information without enforcing agreement, and a t-SNE of the stay embeddings shows that modality-specific structure is preserved (Appendix E).

(Q2) Dynamic patient state. We thank the reviewer for this important point: the patient's state can indeed change during a stay due to medications and interventions. LAMAE does not assume that observations within a stay describe the same state. Unlike contrastive objectives, it does not force them to agree (see Q1), and the LA module can learn how much to draw on other observations. Encoding acquisition time explicitly is a natural extension, which we leave to future work. We are also running additional experiments restricting the input to early observations in the stay (see W2).

Dear Reviewer zBgo, thank you again for your feedback. We have now posted the remaining results (see our update above). They directly address your concerns: the temporal ablation shows that LAMAE's gains hold with only early observations (Tables 12–13), modality completion evaluates stays that contain all three modalities (Table 15), and linear probing now includes all baselines (Tables 9-10). We believe your suggestions have substantially strengthened the paper and would be grateful if you would consider raising your score.

Thank you for the detailed response and the revision. The revised version addressed several of my concerns. Three issues remain:

  1. Retrospective setting and clinical usefulness. Since the authors' reply mentioned that inputs are drawn from the entire stay and acquisition times are not used, so the model cannot be applied at a defined decision point. For in-hospital mortality and length of stay, the outcome is already known by the time all of a stay’s observations are available, so it is unclear when these predictions would be useful in practice. The temporal ablation in Appendix J is a useful step, but: Is the window applied only at test time, or were the models also fine-tuned on windowed inputs? If only at test time, all models are still trained with access to late observations. On mortality, LAMAE’s lead over Independent MAE falls from +1.76 on the whole stay to +0.97 in the first 24 h, and to about zero in the first 6 h after coverage adjustment. Modalities are acquired at different points in a stay (e.g., an ECG typically early, an echocardiogram often later), so restricting the window changes which modalities are available. Please also report the multimodal-vs-unimodal comparison (Table 2) under the same windows. I would suggest making a prediction-time protocol (e.g., inputs from the first 24 h after admission, used for both fine-tuning and testing) the main evaluation for mortality and length of stay, following other established works in this field such as ETHOS, so the performance are more comparable.
  2. Cohort definition is inconsistent. Section 4.1 and Appendix C define a stay as an ED, inpatient or outpatient encounter, but Appendix D says all hospital-stay tasks are “defined per MIMIC-IV admission”, and the rebuttal says a stay is “one MIMIC-IV admission”. This matters for the results. Please clarify the definition and report mortality and length of stay on inpatient admissions only.
  3. I found the results for LAMAE_CXR in Table 2 have changed since the first submission version. Please explain what was the reason that caused this change.

Dear reviewer, thank you for the discussion. Please find below our reply:

1. Retrospective setting. Our goal is representation learning, not clinical prediction. We test whether information exchange during pretraining yields better representations, and we measure their quality with downstream tasks, with identical inputs for all models. The tasks are probes, not deployment targets (Sections 4.1–4.2, Ethics Statement). We added the temporal ablation on request. Since this work is not a clinical prediction model, we see it as showing that the representations also yield useful predictions from early parts of the stay. As the reviewer observed, the mortality advantage over the Independent MAE shrinks toward 6 h. At 6 h, only 2.4% of stays have more than one modality, so there is little for a multimodal model to integrate and similar performance is expected. As the window grows, more stays contain several modalities, and the margin increases to +1.76 for the whole stay. That is what we expect from a model that gains by integrating observations. A clinical model with a prediction-time protocol, as in ETHOS, is future work; a windowed Table 2 is not feasible in the remaining time, nor would it significantly strengthen our evaluation.

2. Cohort definition. As specified in the manuscript Appendix Section C : Throughout the paper, a hospital stay is one MIMIC-IV encounter: an emergency department visit, an inpatient admission or an outpatient visit (as defined in Section 4.1, Appendix C). The term “admission” in Appendix D and in our earlier reply refers to a hospital stay in this sense. All models are evaluated on the same set of stays, so the definition does not favour any method. Stratifying mortality and length of stay by encounter type is an interesting analysis, which we leave to future work given the remaining time in the discussion period.

3. Table 2. The LAMAE$_{\text{CXR}}$ row was computed over all stays instead of the stays with at least one modality, as for all other rows. Only this row changed, in favor of the baseline, and the conclusions are unchanged.

Reviewer_1Fg3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript proposes a Latent-Attention masked autoencoder called LAMAE, which is used to learn cardiac characterizations for electrocardiogram (ECG), echocardiography (Echo), and chest radiographs (CXR). During self-supervised pretraining, the shared latent-attention (LA) module exchanges information in the latent space over a study-view-entity hierarchy. Additionally, entity-level masking encourages reconstruction between complementary observations, while the architecture accommodates a variable number of observations and missing modalities to jointly learn patient-level representations.

Critical strengths

The shared latent-attention (LA) module explicitly models the relationships between different modalities, views or entities in the study-view-entity hierarchy, achieving information integration during self-supervised pretraining. Additionally, entity-level masking encourages the model to use complementary observations rather than relying exclusively on local reconstruction. Furthermore, the architecture accommodates a variable number of observed and missing modalities without requiring complete multimodal input, while improving the quality of representations at all levels.

Critical weaknesses

(1) Table 5 reports a masking ratio of 95% for the multimodal LAMAE, compared to 90% for the independent MAE. Please explain the motivation behind this difference. Additionally, please conduct a comparison using the same masking ratio and perform a sensitivity analysis across a representative range of masking ratios, which would further establish whether the improvements are robust to the chosen configuration.

(2) The study-view-entity hierarchical structure is at the core of the proposed framework. Although structural embeddings can identify entities, views, and modalities, please explain how this hierarchy influences the computational process.

(3) The temporal relationships among the experimental data currently used are unclear, and it is unknown how the temporal information for predictions was obtained. Specifically, ECG, CXR, and Echo data collected during the same hospital stay may correspond to different physiological states. However, the manuscript lacks sufficient explanation of how the time of collection or time intervals were taken into account when grouping and fusing these observations.

(4) Since the pretraining objective is to reconstruct the input in both the signal space and the image space, the paper does not provide actual reconstruction examples or quantitative reconstruction results. Providing representative examples of the original input, masked input, and reconstructed output, supplemented by evaluation metrics suitable for this modality, would help deepen our understanding of the learned representations.

(5) As the manuscript notes, this evaluation covers several interrelated MIMIC resources but remains limited to a single institution. Perhaps an attempt at external validation would help strengthen the evidence supporting the generalizability of the methods described in this manuscript.

(6) As shown in Figure 4, there are significant differences among the modalities in terms of availability and quantity. These differences may simultaneously affect both shared attention learning and the final mean-pooled representations. Please explain whether modality-balanced sampling or weighting was employed; a comparison with modality-balanced pooling methods would also help assess the sensitivity of the system to imbalances in the number of entities.

Questions for authors

(1) Table 5 reports a masking ratio of 95% for the multimodal LAMAE, compared to 90% for the independent MAE. Please explain the motivation behind this difference. Additionally, please conduct a comparison using the same masking ratio and perform a sensitivity analysis across a representative range of masking ratios, which would further establish whether the improvements are robust to the chosen configuration.

(2) The study-view-entity hierarchical structure is at the core of the proposed framework. Although structural embeddings can identify entities, views, and modalities, please explain how this hierarchy influences the computational process.

(3) The temporal relationships among the experimental data currently used are unclear, and it is unknown how the temporal information for predictions was obtained. Specifically, ECG, CXR, and Echo data collected during the same hospital stay may correspond to different physiological states. However, the manuscript lacks sufficient explanation of how the time of collection or time intervals were taken into account when grouping and fusing these observations.

(4) Since the pretraining objective is to reconstruct the input in both the signal space and the image space, the paper does not provide actual reconstruction examples or quantitative reconstruction results. Providing representative examples of the original input, masked input, and reconstructed output, supplemented by evaluation metrics suitable for this modality, would help deepen our understanding of the learned representations.

(5) As the manuscript notes, this evaluation covers several interrelated MIMIC resources but remains limited to a single institution. Perhaps an attempt at external validation would help strengthen the evidence supporting the generalizability of the methods described in this manuscript.

(6) As shown in Figure 4, there are significant differences among the modalities in terms of availability and quantity. These differences may simultaneously affect both shared attention learning and the final mean-pooled representations. Please explain whether modality-balanced sampling or weighting was employed; a comparison with modality-balanced pooling methods would also help assess the sensitivity of the system to imbalances in the number of entities.

3 comments

We thank the reviewer for the detailed and constructive feedback. Since the questions overlap with the weaknesses, we address both together. New analyses are summarized in the general response; changes in the revised manuscript are marked in red.

(1) Masking ratio. The masking ratio of each model was selected based on preliminary ablations, choosing the value that maximized that model’s performance. To assess the effect of this choice, we are currently running ablation studies on different masking ratios. We will provide the results as soon as they become available.

(2) Role of the hierarchy. The hierarchy shapes the computation in three ways. (i) Encoders: each modality has its own encoder and decoder, which process each entity (lead, frame, CXR view) independently (Eqs. 3–4). Information is therefore exchanged across entities, views, and modalities only in the LA module. (ii) Hospital stays: LA attends jointly over all tokens of a single hospital stay, and never across stays. (iii) Objective: the loss is averaged over tokens, entities, and views before being summed over modalities (Eq. 9), so modalities with many entities (12 ECG leads) do not dominate those with few (one CXR image). As the reviewer notes, within LA the hierarchy enters only through learned embeddings. This is deliberate: we use full attention instead of fixing which levels may interact, because the useful interactions are not known in advance.

(3) Temporal relationships. LAMAE does not use acquisition times. Within a hospital stay, observations are treated as an unordered set, so the representation summarizes the stay as a whole rather than a single physiological state. Unlike contrastive objectives, LAMAE does not force observations from different times to agree, and the LA module can learn how much to draw on each.The hospital-stay tasks are therefore retrospective benchmarks of representation quality. We revised the manuscript to make this explicit. Section 4.1 now states that observations are sampled from the entire stay without using acquisition times, replacing the misleading phrase "collection of ordered modalities". Section 4.2 states that the hospital-stay tasks are retrospective, and the limitations list explicit temporal modeling as future work. To study whether the representations are also informative early in a stay, we are running additional experiments that restrict the input to observations acquired early in the hospital stay. We will provide the results as soon as they become available.

(4) Reconstructions. We added reconstruction examples showing the original, masked and reconstructed inputs for each modality (Appendix, Fig. 11 for CXR and Echo, Fig. 12 for ECG), and the per-modality validation reconstruction loss (MSE on masked patches, Fig. 13). To deepen the understanding of the learned representations, we additionally analyze the learned representations directly with linear probes (Appendix section F) and t-SNE (Appendix section E) (general response, points 1–2).

(5) External validation. We agree that external validation would strengthen the paper, we list the single institution setting as a limitation (Section 5). However, to our knowledge, MIMIC-IV is the only public dataset that pairs ECG, Echo and CXR from the same hospital stays with stay-level outcomes. Our main claims therefore rely on controlled comparisons: LAMAE and the Independent MAE share architecture, data, splits, and training budget, and differ only in the LA module and masking (see (1)). This isolates the relative improvement as the effect of LAMAE, while absolute performance may differ across institutions.

(6) Modality imbalance. We do not use modality-balanced sampling: each epoch iterates over all hospital stays once. The architecture limits the effect of imbalance in two ways. First, encoders and decoders are modality-specific, so only the single-layer LA module is shared across modalities, and frequent modalities do not take capacity from the encoders of rare ones. Second, the reconstruction loss is normalized per modality (Eq. 9), so within a stay each observed modality contributes equally regardless of its number of entities. Empirically, rare modalities are not degraded by sharing the LA module, as the comparison with the unimodal models in Tables 3a–b shows. Empirically, imbalance does not appear to hurt rare modalities. The unimodal models can be seen as an extreme case of balanced sampling, since each is trained only on its own modality. As shown in Table 3, LAMAE$_\text{MM}$ matches or outperforms the unimodal models on all modalities, so sharing the LA module with more frequent modalities does not degrade the rare ones. We agree that modality-balanced sampling during pretraining is an interesting direction that could improve results further. Since it requires new pretraining runs, we prioritized other requested ablations during the rebuttal period and now discuss balanced sampling as future work.

Thank you for the clarification. I will keep my score. Good luck.

Dear Reviewer 1Fg3, thank you again for your feedback. We have now posted the remaining results (see our update above). They directly address your concerns: the masking-ratio ablation shows that Independent MAE is evaluated at its best ratio (Fig. 14), the temporal ablation shows that the gains hold with only early observations (Tables 12–13), and modality-balanced pooling leaves performance unchanged (Table 14). We believe your suggestions have substantially strengthened the paper and would be grateful if you could reconsider raising your score in light of this evidence.

Comments on the paper

7 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank the reviewers for their constructive feedback. Reviewers recognize the importance of joint multimodal pretraining for medical data (zBgo, ixg7), LAMAE’s handling of variable and missing modalities (1Fg3, zBgo, ixg7), the improvement of the quality of representations at all levels (1Fg3), and a broad, well-controlled evaluation (zBgo, ixg7). Below, we address the remaining concerns and summarise new results and revisions to the manuscript, which we believe have improved the quality of the paper. We respond to every point individually and summarize the key changes here (marked in red in the revised manuscript).

New analyses in the revised manuscript

Does LAMAE learn better representations, or only finetune better? (zBgo, 1Fg3; App. F, Fig. 9)

  • Result: a linear probe on frozen LAMAE MM embeddings comes close to full finetuning and outperforms the Independent MAE and ProbMED probe on all six tasks (Appendix F). For example, on ICD-10, LAMAE$_\text{MM}$ drops from 83.02 to 81.74 AUROC under linear probing, but still matches the fully finetuned Independent MAE (81.66); on DRG code, its probe even exceeds it (58.10 vs. 57.46). We will add MedSigLip as it becomes available. Since the two models share architecture, data and training budget, the gain comes from the learned representation, not from finetuning. We additionally ran an ablation with limited labels: the gap between LAMAE and the Independent MAE widens as the number of labels decreases (Fig. 9), suggesting that LAMAE learns representations that require fewer labeled samples to solve downstream tasks.

2. What is captured in the embedding space? A t-SNE analysis (zBgo, 1Fg3; Appendix E)

  • Result: In-hospital deaths, cardiomegaly, and atrial fibrillation fall into distinct regions without any finetuning, while modality-specific structure is preserved. Pretraining alone yields clinically meaningful structure, consistent with the strong linear-probe results.

3. How many Echo views are needed? (ixg7; response to ixg7 [W4])

  • Result: pretraining with more Echo views per study brings no consistent gain. Finetuning with more views substantially improves LVEF estimation (R2R^2 0.41 → 0.66). Pretraining on one view per study therefore keeps pretraining cost low without limiting downstream performance, since in LAMAE multiple views can be used during finetuning.

Clarifications

We clarified that the hospital-stay tasks are retrospective, that all models receive the same observations, and added temporal modeling to the limitations (Sections 4.1, 4.2 and 5). We improve the entity masking description (Section 3.3). In the individual responses to the reviewers, we further explain how entity dropping interacts with the masking ratio, how the study–view–entity hierarchy shapes the computation, and how dropped entities are represented. Finally, we added reconstruction examples and losses (Appendix, Figs. 10–12) and a complete list of ICD-10 Chapter IX targets (Table 8).

Minor modification Updated Table 1 (MedSigLip, ICD10) with corrected results due to a reporting mistake (AUROC 82.06 → 82.01). Nothing changed in the considerations.

Ongoing work

We are also running further analyses, including a masking ratio ablation, an early-window evaluation, and bootstrap confidence intervals. These experiments are compute-intensive and take considerable time and resources. We will complete as many as possible within the rebuttal period and share results during the discussion as they become available.

AuthorsOpenReview ↗

Response 1/2

We thank the reviewer for the detailed and constructive feedback, and for recognizing the relevance of integrating ECG, Echo and CXR during pretraining and the breadth of our evaluation. Since several questions overlap with the weaknesses, we address them together. New analyses are summarized in the general response; changes in the revised manuscript are marked in red.

(W1) Paired multimodal data. We agree that paired data is scarce (Fig. 4). However, this reflects an artifact of the data: MIMIC-CXR covers [2011–2016] and MIMIC-IV-Echo [2017–2019], so CXR and Echo never co-occur within a stay. Handling such incomplete data reflects the challenges of the medical domain, which is the setting LAMAE is designed for. Unlike contrastive methods such as ProbMED, LAMAE learns from any observed subset, and it generally outperforms ProbMED (Table 1). LAMAE can technically process unseen combinations such as CXR+Echo+ECG, but these are not present in the current dataset. Nevertheless, joint pretraining improves representations even with largely unpaired data: in the unimodal results (Table 3), LAMAE$_\text{MM}$ outperforms the unimodal LAMAE models on CXR and most ECG tasks, and matches them on Echo.

(W2) Within- vs. cross-modal information. We agree that within-modality redundancy is highly informative, and LAMAE is designed to exploit it. Table 3 isolates this effect: LAMAE_k and the Independent MAE_k are pretrained on the same single modality k, so the gain of LAMAE stems from within-modality exchange through the LA module and entity masking. Empirically, cross-modal information also appears to be used, since LAMAE$_\text{MM}$ outperforms the unimodal LAMAE models on CXR and ECG with single-modality input (see W1).

(W3) Temporal relationships. As of now, LAMAE is time-agnostic, and modeling temporal relationships is an interesting direction for future work. The hospital-stay tasks are retrospective benchmarks. To ensure a fair and controlled comparison, all baselines receive the same observations. We thank the reviewer for pointing this out and have clarified the temporal setting in the revised manuscript (Sections 4.1–4.2). As suggested by the reviewer, we are running additional experiments restricting the input to early observations in the stay. We will provide the results as soon as they become available.

(W4) Hierarchy and novelty. Our contribution is not a new attention mechanism, but a pretraining framework that learns from incomplete, heterogeneous, and multimodal clinical data. The hierarchy shapes the computation in three ways. (i) Encoders: each modality has its own encoder and decoder, which process each entity (lead, frame, CXR view) independently (Eqs. 3–4). Information is therefore exchanged across entities, views, and modalities only in the LA module. (ii) Hospital stays: LA attends jointly over all tokens of a single hospital stay, and never across stays. (iii) Objective: the loss is averaged over tokens, entities, and views before being summed over modalities (Eq. 9), so modalities with many entities (12 ECG leads) do not dominate those with few (one CXR image). As the reviewer notes, within LA the hierarchy enters only through learned embeddings. This is deliberate: we use full attention instead of fixing which levels may interact, because the useful interactions are not known in advance. In a controlled comparison with identical encoders, adding the LA module and entity masking improves over or matches the Independent MAE in all cases, with improvements in most (Table 3). Our design restricts full-stay attention to a single layer, which keeps the cost manageable. Using one Echo video per study bounds pretraining cost, and the architecture supports multiple videos. An ablation over the number of Echo views supports this choice: pretraining with more views brings no consistent gain, whereas finetuning with more views improves LVEF estimation substantially (0.41 → 0.66 R2R^2).

PT / FT views1248
10.414 ± 0.0290.548 ± 0.0160.631 ± 0.0100.658 ± 0.017
20.406 ± 0.0280.561 ± 0.0170.601 ± 0.0180.614 ± 0.024
40.388 ± 0.0470.560 ± 0.0190.611 ± 0.0230.659 ± 0.018
80.429 ± 0.0300.575 ± 0.0070.630 ± 0.0130.659 ± 0.032

Test R2R^2 for LVEF regression with [LAMAE$_\text{Echo}$] (mean ± std over 3 seeds, n = 2606). Rows: Echo views per study during pretraining; columns: during finetuning. These runs use separate seeds from Table 3b, so the single-view standard deviation differs slightly.

AuthorsOpenReview ↗

Response 2/2

(W5) Masking formulation. We thank the reviewer for the careful check. When more entities are dropped than the target ratio allows, entity dropping takes precedence and β is set to zero, so the effective masking ratio is slightly higher (e.g., 91.7% instead of 90%). We have clarified this in the revised manuscript. Regarding CXR, Fig. 1 illustrates the general method, in which view dropping is possible. In preliminary experiments, not dropping CXR views performed equally or better, so it is not used in our configuration (as indicated in Table 5).

(Q1) Contribution of the LA module. The masking ratio of each model was selected based on preliminary ablations to maximize its performance. We are currently running masking-ratio ablations for Independent MAE$_\text{MM}$ and will share the results as soon as they become available.

(Q2) Cross-modal transfer. See W2. Entire modalities are never dropped during pretraining. Previous work has highlighted the challenges of performing complete cross-modal reconstruction (Daunhawer et al., ICLR 2022, https://openreview.net/forum?id=w-CPUXXrAj). We're still investigating on how to overcome this in our setting. Addressing this and shuffling modalities between patients are interesting directions that are part of our future work.

(Q3) Hierarchy and Echo views. See W4.

(Q4) Robustness and baselines. Confidence intervals: we appreciate the suggestion and are computing patient-level bootstrap confidence intervals, which we will share as soon as they become available. Baselines: the Independent MAE is our matched-budget baseline (256M vs. 263M parameters for LAMAE$_\text{MM}$). ProbMED has 251M and MedSigLip 428M parameters. Additionally, ProbMED includes all the datasets we’re also using, making it a very strong baseline (Appendix D). Cardiac evaluations: our evaluation includes cardiomegaly and enlarged cardiomediastinum derived from CXR reports (Fig. 7), LVEF (Table 3b), six ECG intervals (Table 3c), and ICD-10 Chapter IX (circulatory diseases), with per-code results in Fig. 6 (e.g., 93.0 AUROC for I50.2 - Systolic (congestive) heart failure). Additionally, we have now added Table 8, which lists all included diagnoses in text form.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.