acceptodds
Under review as a conference paper at ICLR 2027

HEART-JEPA: LEARNING CROSS-VIEW REPRESEN- TATIONS FOR CARDIAC MRI WITH JOINT-EMBEDDING PREDICTIVE ARCHITECTURES

Abstract

Cine cardiac magnetic resonance imaging (CMR) captures the same heart from complementary anatomical views. Learning representations that transfer between these views is a central challenge for reusable cardiac prediction. We introduce HEART-JEPA, a cross-view joint-embedding predictive architecture that predicts masked latent features across anatomical planes without spatially matched patches. Pretrained on 39,992 UK Biobank participants, HEART-JEPA achieves a crossview structural-transfer R2 of 0.48 in 15,935 participants absent from pretraining, without target-view refitting or recalibration. Matched ablation studies show substantial gains over same-view pretraining and stronger cardiac phenotype prediction and structural and functional transfer than cross-view contrastive learning. In a separate ventricular-function benchmark, supervised linear probes fitted to concatenated frozen features from all three views achieve the lowest mean LVEF MAE among the seven evaluated models (3.19 percentage points), with competitive RVEF performance. These findings establish cross-view latent prediction as an effective approach to cardiac representation learning and motivate its extension to videos acquired from complementary, spatially unaligned views

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Cardiac magnetic resonance imaging, Self-supervised learning, Cross-view representation learning, Joint-embedding predictive architecture, Masked latent prediction, Ventricular function estimation.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rvYKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors propose a JEPA-based pretraining framework for multi-view cardiac MRIs, focusing on feature consistency between long-axis views. Their evaluation focuses on cross-view phenotype prediction, while additional representation analyses (including probes and retrieval) are also provided.

Critical strengths

1. Multiple Evaluations and Analyses

The authors provide multiple evaluations of their representations, including probing, retrieval, and cross-view transfer. Additional ablation studies are provided in the appendix. Note that despite the abundance of evaluations and analyses, the resulting findings are hard to follow because many evaluations are not clearly motivated (see also weakness 2)

Critical weaknesses

1. Limited Contributions with unclear motivation

The proposed work is a minor adaptation of the JEPA framework to cardiac MRI representation learning. However, the authors’ method contributions are very limited, as similar JEPA-based frameworks have been widely studied:

  • Multi-modal JEPA frameworks have conceptual similarities with the proposed method. For example, M3-JEPA [1] uses a cross-model predictor, making it conceptually very similar to the cross-view predictor proposed in this work (while differing in implementation details). However, the authors did neither cite nor compare their work to any multimodal JEPA framework.
  • MR-JEPA (already cited by the authors) has already explored JEPA on cardiac MRIs, although they ignored multi-view relationships.

Furthermore, their contribution focuses solely on multi-view alignment and cross-view transfer of long-axis views. It is unclear or not thoroughly motivated:

  • Why focus on shared features (alignment) instead of learning to fuse information from multiple views (like many multimodal JEPA frameworks)
  • Why is cross-view transfer performance so relevant? If we want to focus on inference with only a subset of views (or just a specific view), why can’t we simply train on that specific view if training data is typically available anyway (as the common cardiac MRI acquisition strategy contains several long-axis views)
  • Why the authors focus on long-axis views while ignoring short-axis views

Overall, the chaotic structure of the paper makes it hard to understand the motivation of the proposed method, while the method contributions appear minor. Without clear motivation and significant contributions, the proposed work is of limited relevance to the machine learning community.

[1] Lei, Hongyang, et al. "M3-jepa: Multimodal alignment via multi-gate moe based on the joint-embedding predictive architecture." ICML 2025, arXiv:2409.05929 (2024).

2. Inconsistent Selection and Presentation of Evaluation Tasks with Sometimes Limited Performance Benefits

The selection of evaluation tasks (i.e., target classes/phenotypes) is not well motivated and is inconsistent between sections and with existing literature. Furthermore, the presentation of the experiments and results makes it hard to follow the paper and judge the performance benefits of the proposed method and understand the overall findings. Specifically:

  • In 5.1 (cross-view prediction) the authors evaluate on 17 image-derived phenotypes while in 5.5. (frozen representations), they consider only 2 (LVEF and RVEF)
  • Evaluation on disease labels, which is commonly done in previous cardiac MRI pretraining / foundation model works, is hidden in the appendix and only aggregated results are discussed. Notably, on several tasks, the proposed method underperforms the contrastive baseline (see Table S17 on p. 23). Furthermore, some baselines are excluded from this table
  • The target mixture experiment (Sec. 5.3) is not motivated and the resulting findings are unclear.

Overall, the many different cohorts and evaluations without clear motivation, consistent targets, and consistent baselines, while being scattered between the main paper and the appendix, make it hard to judge the actual benefits of the proposed method in comparison to existing works. The paper focuses heavily on cross-view prediction performance (without providing motivation for why it is relevant), while lacking a clear, coherent presentation of the typical downstream performance (e.g. diagnosis) that is common in similar works.

Questions for authors
  • Why focus on shared features (alignment) instead of learning to fuse information from multiple views (like many multimodal JEPA frameworks)
  • Why is cross-view transfer performance (contribution (ii)) so relevant?
  • What about short-axis views?
  • What is the purpose of the target mixtures in Sec. 5.3
Reviewer_eRaaOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes HEART-JEPA, an adaptation of the Joint-Embedding Predictive Architecture (JEPA) for learning transferable representations across different anatomical views of cardiac MRI (e.g., 2-chamber and 4-chamber views). The method aims to improve cross-view representation compatibility via pretraining to support downstream prediction tasks without requiring view-specific retraining. During self-supervised pretraining, a context encoder processes masked tokens from one view (e.g., the 2-chamber), while a predictor estimates selected latent features of another view (e.g., the 4-chamber) from the same participant. These predictions are matched using an MSE loss against features extracted by an EMA target encoder. The method was pretrained on ~40K UK Biobank participants and evaluated on ~16K participants, achieving a direct cross-view structural-transfer R^2 of 0.48, outperforming same-view JEPA (baseline method).

Critical strengths
  1. The motivation of the topic makes sense. The method may be able to address issues where some of the views were not acquired but are required for certain predictive tasks that require all or some views.
  2. The paper is generally easy to read. The methodology is straightforward.
  3. A large amount of experimental validation was conducted in the paper.
Critical weaknesses
  1. Revise the methodology section. This particularly refers to Eq. 2; many of the symbols were not introduced. Also, many acronyms are not spelled out for readers.
  2. Limited novelty. The methodology differs from the original JEPA by: 1) adapting it from 2D images to 3D video; 2) masking latent tokens instead of image regions; 3) predicting the latent representation of another view. In my opinion, all these changes are necessary problem-adaptation efforts instead of novelty.
  3. Disclose the CMR acquisition protocol. While it is understandable that some views may be missing during an MRI examination, it would be better to understand how severe the missingness is and therefore to assess whether the cross-view latent feature prediction is necessary - if the views are always acquired, I think that would undermine the significance of the approach as a direct transfer of view representations.
  4. Comparing to direction training of 17 IDP downstream tasks without HEART-JEPA pretraining or pretraining in general. Circling back to the missing view question, if the views are not missing anyway - which I think so in UKB, they'd acquire all three views as a standardized protocol. Direct training of the downstream 17 IDP predictors is possible and would be a meaningful performance ceiling baseline for HEART-JEPA to compare against.
  5. Clarify what the refitting and recalibration mean. What needs to be fitted and calibrated if HEART-JEPA is not used?
  6. Overwhelming results and appendix. I did not go through all these results, as I find some of the experiments have limited value, e.g., participant retrieval, temporal matching. The authors should consider pivoting the experiments to highlight the major contribution instead of try to covering all corners.
Questions for authors

Critical weaknesses 2.3.4.5

Reviewer_VHkMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes HEART-JEPA, a self-supervised method for cine cardiac MRI. It uses paired 2-, 3-, and 4-chamber long-axis views of the same participant. A masked context view is encoded, and a view-conditioned predictor predicts the latent features of a different view from an EMA target encoder. No patch correspondence, registration, or frame alignment is used. The architecture follows standard video JEPA. The main change is the prediction target, which is a different view rather than the same view. The model is pretrained on 39,992 UK Biobank participants.

The authors claim three contributions. The first is the cross-view predictive framework. The second is direct transfer, where a model fitted on one view’s features is applied unchanged to another view. The third is evidence that probing, retrieval, al and transfer capture different properties, and that mixing same-view and cross-view targets has a non-monotonic effect.

Validation centers on a matched experiment that varies only the share of same-view targets, with three seeds. It is supported by objective controls (independent queries, Global HEART-JEPA, contrastive learning) and by representation analyses. Downstream evaluation uses frozen features for LVEF and RVEF, 17 imaging-derived phenotypes, five UK Biobank disease labels, and external classification on M&Ms-2.

Critical strengths

Only the share of same-view targets changes. Everything else is fixed, and three seeds are used. Cross-view targets raise structural transfer R² from −1.04 to 0.48, and the gain holds on participants excluded from pretraining. This is the main reason I incline towards weak acceptance. The central claim rests on a fair causal comparison rather than on comparisons between different systems.

Global HEART-JEPA gives the best-fitted probe but negative transfer. The contrastive control gives better retrieval but weaker transfer. This matters because linear probing is often used as the default test of representation quality. The paper shows that this can be misleading, and direct transfer is a useful complementary test.

Critical weaknesses

Direct transfer R² decreases when target-view features have different means or scales, even if the information is present. Appendix F acknowledges this but reports neither a recalibration baseline nor a transferred Pearson correlation. This matters because the headline gap between cross-view and same-view training may partly reflect feature calibration rather than shared cardiac information. The size of the true gain is therefore unclear.

Table 1 describes C2 as participant-disjoint from pretraining. Appendix C states that C2 is disjoint only at the visit and instance levels, and that C3 is the participant-disjoint set. The two sections also define C3 differently (IDP availability versus pretraining exclusion). Many results, including the mixture sweep and the objective controls, are reported on C2. This matters because readers cannot tell which results are truly participant-disjoint.

Questions for authors
  1. Please report transferred Pearson correlation, or R² after a simple target-view mean and scale recalibration, for the cross-view and same-view endpoints. Does the gap remain large?

  2. Please confirm whether C2 is participant-disjoint or only study-disjoint from pretraining, and give one consistent definition of C3.

  3. The method keeps the standard video JEPA architecture and changes the prediction target. Please state clearly how the objective differs from MR-JEPA, which also applies JEPA to cine CMR, and from CroCo, which already performs cross-view completion. Your controls show that cross-view contrastive learning also transfers well (0.44), while Global HEART-JEPA does not (−0.55). What property of the token-level objective explains this difference?

  4. How do the three EF benchmark checkpoints relate to the three intervention runs? Are they the same runs?

  5. What fraction of C5 inputs contain repeated padding frames? Does temporal matching hold when these are excluded?

  6. The text highlights where CMR Transformer beats HEART-JEPA on M&Ms-2. CineMA also has a slightly higher macro AUC in both protocols. Please report this in the main text for balance.

  7. A test of direct transfer on an external multi-view dataset would show whether the property generalizes beyond UK Biobank.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.