acceptodds
Under review as a conference paper at ICLR 2027

Medical-4D: 4D Cardiac MRI Vision-Language Models for Cardiac Function Understanding

Abstract

Cardiac magnetic resonance (CMR) cine imaging provides rich 4D information for assessing cardiac function, underpinning clinical diagnosis. However, jointly modeling spatial, temporal, and inter-slice dependencies over full 4D CMR data is computationally prohibitive. Existing CMR-based diagnostic methods simplify the data via temporal downsampling or single-slice videos, discarding fine-grained cardiac motion and inter-slice anatomical dependencies essential for dynamic analysis. To address this, we propose factorized 4D representation learning with cardiac-phase and anatomical-slice positional encoding. Specifically, the factorized encoder separately performs spatial, temporal, and inter-slice modeling to enable joint spatial-temporal analysis of cardiac function. We further introduce continuous structured visual masking to capture dependencies across adjacent cardiac phases and neighboring slices. Together, these components align visual representations with clinical cardiac function assessment reports. To our knowledge, this is the first cardiac model to exploit the full 4D CMR data for continuous and complete vision-language alignment. Based on this model, we propose Medical-4D, the first cardiac function QA framework. Trained on CMR 4D clinical reports and QA pairs from three data centers, Medical-4D significantly outperforms previous state-of-the-art methods.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Medical Vision-Language Model, Factorized 4D Spatiotemporal Representation, Cardiac Function QA, 4D Cardiac Magnetic Resonance Imaging

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_F1EDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Medical-4D fully leverages the complete spatiotemporal information from cardiac cine MRI, significantly improving visual question answering performance for cardiac structure and motion function. This study holds substantial clinical value. Its factorized encoder separately models spatial, temporal, and inter-slice dependencies, enabling joint spatiotemporal analysis of cardiac function. Furthermore, the paper introduces continuous structured visual masking to capture dependencies between adjacent cardiac phases and neighboring slices. The authors constructed a three-center clinical dataset, and Medical-4D, trained on CMR 4D clinical reports and QA pairs, substantially outperforms previous state-of-the-art methods.

Critical strengths

On the technical level, the authors claim to be the first to achieve joint spatiotemporal modeling of complete 4D CMR data, avoiding information loss. In addition, they introduce continuous structured visual masking and bidirectional cross-modal recovery to achieve fine-grained phase-slice alignment.

Critical weaknesses

Although this paper attempts to fill the gap in 4D image-text pretraining within the medical domain, it does not introduce any fundamentally new learning mechanism from an innovation perspective. Moreover, the QA data were automatically generated by Qwen and only underwent rough manual filtering, making it difficult to adequately verify the accuracy and validity of such a data construction approach.

Questions for authors
  1. Performing Transformer attention sequentially along the temporal and slice dimensions is essentially an intuitive factorized spatiotemporal scheme. The paper fails to clearly articulate what new learning mechanism this approach introduces compared to existing methods such as axial attention, spatiotemporal decomposition, and medical vision-language pretraining.
  2. The QA data were automatically generated by Qwen from clinical reports, with answers derived from the same reports. The paper does not sufficiently demonstrate the clinical validity of the questions and answers, whether patient-level isolation is achieved in the independent test set, nor the relationship between open-ended QA metrics and true clinical correctness.
  3. The performance gains on some metrics are modest, and the proposed method does not lead across all external-center structure-related closed-ended QA tasks. No confidence intervals, repeated experiments, or statistical significance tests are reported. Therefore, the tabulated results alone are insufficient to claim that the method significantly outperforms all baselines.
2 comments

Q1

We thank the reviewer for asking us to clarify the learning mechanism. We agree that applying attention sequentially along separate dimensions, as in axial attention and divided space-time attention, is an established strategy. We do not claim that temporal attention or slice attention, by themselves, constitute a new attention algorithm.

Our contribution is to adapt factorized modeling to the distinct dimensions of full 4D short-axis cine CMR and the computational challenge of its large token volume. To the best of our knowledge, existing medical vision-language pretraining methods do not jointly process the complete multi-slice cine sequence; existing CMR approaches commonly reduce the input through temporal subsampling, slice selection, or use of only ED and ES frames. Medical-4D instead encodes each phase–slice image spatially, adds cardiac-phase and anatomical-slice identity embeddings together with coordinate encodings that preserve temporal order and anatomical position, and then applies temporal attention within each slice followed by slice attention within each phase. This factorization makes it possible to model the full phase–slice grid without discarding its phases or slices. It also preserves a contextual feature for each phase–slice pair before study-level pooling.

The learning objective then uses these phase–slice features for fine-grained cross-modal supervision. In report-assisted visual recovery, clinically structured masks cover contiguous slices or phases, and the paired report helps recover the masked visual features. In image-assisted text recovery, masked clinical phrases are recovered using the phase–slice visual features. These objectives add local report–image interaction to global image–report alignment, while retaining the full 4D representation.

Accordingly, our contribution is the CMR-specific application of full-4D representation learning and report-guided cross-modal recovery to cardiac-function understanding and QA, addressing both the large token volume and the need to preserve phase and slice information. We will revise the manuscript to clarify this scope and avoid implying that the factorized attention operation itself is a fundamentally new learning mechanism.

Q2

We thank the reviewer for this important question.

Clinical grounding and Qwen’s role. The questions are based on clinical questions that physicians address when assessing cardiac structure and function. Qwen was used to rephrase or vary the wording of these questions, rather than independently define new clinical questions from its prior knowledge. For answer construction, Qwen extracted the corresponding structural or functional information from each patient’s physician-authored report according to our predefined templates. Thus, the reference answers originate from the patient’s clinical report, rather than clinical findings independently generated by Qwen.

Throughout dataset construction, specialist physicians from two different hospitals provided ongoing training, guidance, and clarification to the three master’s/doctoral students involved in the process. The physicians also performed random spot checks of the QA pairs and corrected identified errors. We acknowledge that the phrase “rough manual screening” does not adequately describe this process. We will expand the manuscript to explain the physician-supervised construction and quality-control procedures, and provide a detailed breakdown of the question types and their corresponding cardiac-function categories.

Patient-level separation. We confirm that the training and test sets are separated at the patient level, not the QA-pair level. Specifically, the 80/20 split of the Center 1 and Center 2 datasets described in Section 5.3 was performed by patient: the images, reports, and QA pairs belonging to the same patient remain in the same partition. In addition, no patients from Center 3 participated in training; this center was used exclusively for external evaluation. Therefore, the training and test sets contain no overlapping patients. We will explicitly state this in the revised manuscript to remove the ambiguity in the current description.

Q3

We thank the reviewer for this important comment. To verify the reproducibility and validity of the experiments, in the revised paper we trained and tested each experimental setting five times using different random seeds, and we report the mean and standard deviation over the five runs. The updated results are shown below.

Comparative analysis on the center 1 and center 2 datasets. Token recall for open-ended questions; accuracy for close-ended questions.

MethodInputClose- StructureClose- Wall MotionOpen-StructureOpen-Wall Motion
Med-flamingo~\cite{pmlr-v225-moor23a}2D72.94$\pm$0.6435.90$\pm$0.7058.92$\pm$0.4556.60$\pm$1.12
Llava-med~\cite{li2023llava}2D79.49$\pm$0.6327.88$\pm$0.9859.87$\pm$0.5953.13$\pm$0.44
HealthGPT-Pro-8B3D77.57$\pm$0.7036.39$\pm$0.3341.45$\pm$1.0550.21$\pm$0.54
CineMA~\cite{fu2026development}3D91.61$\pm$0.6961.90$\pm$0.6365.25$\pm$1.8255.14$\pm$0.79
Qwen 3.8-27B~\cite{qwen38}2D+T54.91$\pm$0.3627.39$\pm$0.9750.49$\pm$2.9343.78$\pm$2.91
CMR-CLIP~\cite{nakashima2026contrastive}2D+T87.97$\pm$0.4184.86$\pm$0.5062.97$\pm$0.3464.86$\pm$0.45
CMR-Transformer~\cite{shad2026generalizable}2D+T89.82$\pm$0.4985.68$\pm$0.4764.77$\pm$0.6664.87$\pm$0.72
Ours (Medical-4D)4D91.93$\pm$0.2890.38$\pm$0.3066.40$\pm$0.3165.87$\pm$0.37

Evaluation on Center 3 (unseen during training). Token recall for open-ended questions; accuracy for close-ended questions.

MethodInputClose- StructureClose- Wall MotionOpen- StructureOpen- Wall Motion
Med-flamingo\cite{pmlr-v225-moor23a}2D73.36$\pm$0.5327.12$\pm$0.8060.95$\pm$0.3657.64$\pm$1.25
Llava-med \cite{li2023llava}2D80.47$\pm$0.7128.71$\pm$0.8861.19$\pm$0.6854.82$\pm$0.33
HealthGPT-Pro-8B3D78.11$\pm$0.5547.45$\pm$0.5042.05$\pm$1.2241.53$\pm$0.38
CineMA \cite{fu2026development}3D90.48$\pm$0.8463.37$\pm$0.5565.62$\pm$1.6557.31$\pm$0.89
Qwen 3.8-27B \cite{qwen38}2D+T55.34$\pm$1.2136.58$\pm$1.1445.97$\pm$2.7543.62$\pm$3.02
CMR-CLIP \cite{nakashima2026contrastive}2D+T88.39$\pm$0.5789.00$\pm$0.4364.27$\pm$0.4446.51$\pm$0.61
CMR-Transformer \cite{shad2026generalizable}2D+T91.30 $\pm$0.5188.72$\pm$0.5865.17$\pm$0.4963.34$\pm$0.83
Ours(Medical-4D)4D91.58$\pm$0.4490.11$\pm$0.3666.13$\pm$0.4764.92$\pm$0.22

We agree that the improvement in close-structure performance is modest. However, on the wall motion-related close and open tasks, our method achieves substantial improvements over existing CMR vision-language models. This demonstrates the effectiveness of using full CMR-4D data for cardiac functional motion diagnosis, outperforming existing CMR methods (which use only the two temporal frames ED/ES or extract only a subset of temporal frames). For example, the following diagnostic evidence for cardiac functional motion disease depends on continuous and fine-grained temporal frame change information: "reduced and dyssynchronous wall motion"; "reduced motion at the left ventricular apex"; "globally reduced systolic function"; "thinning of the left ventricular wall with regional dyssynchronous motion"; "markedly reduced left ventricular systolic function"; "impaired diastolic function"; and "reduced motion amplitude of the interventricular septum and the left ventricular anterior wall." These cardiac functional motion disease expressions all come directly from physician clinical reports and are also cardiac function phenomena that clinicians focus on in diagnosis; these cardiac function findings strictly depend on temporal changes across the entire cardiac cycle, and thus cannot be derived from subsampled frames or from ED/ES frames alone. In addition, we have revised the claim in the paper that the method significantly outperforms all baselines.

Reviewer_upg5OpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Medical-4D, a vision–language framework for cardiac function assessment and question answering from 4D cine MRI. The method first encodes individual images, then models relationships across cardiac phases and anatomical slices to preserve motion and structural information. Training begins with image–report alignment, followed by two complementary tasks: recovering masked image features using reports and recovering masked clinical phrases using images. The learned encoders are then frozen, and a decoder is trained for report generation and question answering.

The authors evaluate the framework using clinical images, physician reports, and automatically generated QA pairs from three centers, including one center excluded from training. The results show improvements on wall-motion questions, while ablation studies support the contribution of temporal and slice modeling.

Critical strengths
  • The main idea directly addresses information lost when cardiac MRI is reduced to selected frames or slices. This is a well-motivated approach to understanding cardiac function.
  • Table 3 supports the value of modeling temporal and slice relationships beyond independently encoding and pooling all images.
  • The improvements on wall-motion questions and evaluation on an unseen clinical center provide useful evidence for the approach.
Critical weaknesses

Presentation needs substantial improvement. Figure 1 is crowded and does not clearly explain the phase–slice organization, the two attention directions, or which components are used only during training. Figure 3 also needs correction: CMR-Transformer predicts reduced contractility, which agrees with the reference, although the text states that only Medical-4D is correct.

Dataset validation is insufficiently documented. QA pairs are generated from reports using Qwen, followed by “rough manual screening.” The reviewers’ expertise, review coverage, and error rate are not reported. Token recall alone also cannot establish the clinical correctness of open-ended answers.

Practical computational costs remain unclear. The appendix provides attention complexity and GPU configurations, but does not report measured memory use, training time, or inference latency. These measurements would help establish the practical benefit of the design.

Questions for authors
  1. Dataset quality checks: Please report the results of the manual checks on the generated QA pairs, including how many were reviewed, the reviewers’ clinical expertise, the observed error rate, and how errors were corrected. Please also confirm that all data from each patient remained in the same split across all training stages.

  2. Computational efficiency: Please report the measured computational costs, including training time for each stage, peak GPU memory, and inference time per study, with hardware, input size, and batch size specified. A comparison with a relevant baseline under the same conditions would help quantify the practical savings from the factorized encoder.

Reviewer_BaqjOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Medical-4D, a vision-language framework for question answering on cardiac function from 4D short-axis cine CMR. The paper proposes 2 contributions: 1) A factorized 4D cardiac representation that decomposes spatial, temporal, and inter-slice modeling, thus allowing the use of the full 4D CMR sequence (instead of just ED and ES). The usage of normalized position encodings aligned with anatomical structure and temporal order allows encoding slice structural and temporal information with local and global positional relationships. This allows alignment with cardiac function assessment reports. 2) The authors claim to present the first cardiac function Q&A framework. A third contribution refers to the experiments carried.

The paper relies on very rich data. Notably, Q&A pairs (~47k) were generated by an LLM from reports of ~6K subjects across three centers, with one center held out for evaluation.

Critical strengths

The paper relies on a impressively large multi-centric study that gives credibility to the conclusions it drives.

Critical weaknesses
  • The novelty of the core contributions is limited. Most of the elements (factorized attention, contrastive alignment with hard negatives, masked cross-modal recovery) that constitute the framework are well established. The two contributions that could be considered novel, the clinically structured masking stage and the use of the full cine sequence (30 phases per slice), are not supported by the results. For the first one, the improvements it brings are limited (up to 0.8 points). For the second one, while the standard approach in clinical practice is to use ED and ES, the paper fails to demonstrate that the extra phases bring added value.

  • Multiple aspects of the evaluation need to be revised. To start with, the method is tested on a single split and does not report standard deviations from multiple runs or statistical tests. As some differences are narrow, it is not possible to establish if those are simply due to noise. Second, Q&As are generated from reports by an LLM (Qwen) with only a rough manual screening. Therefore, the evaluation measures whether the model can reproduce the information in the report, not whether cardiac function is correctly assessed. Lastly, it seems that some baselines do not get a fair evaluation (see questions for a clarification).

Questions for authors
  1. How does Medical-4D perform when restricted to ED and ES frames, or to fewer phases (e.g., 10, 15)? This would show whether the additional phases bring added value over standard clinical practice.
  2. Can the authors report mean and standard deviation over multiple runs or cross-validation folds, with statistical tests for the key comparisons, particularly where differences are below one point?
  3. Was CineMA given both ED and ES frames, as in its original design, or a single time point? Were CMR-CLIP and CMR-Transformer evaluated with their original multi-slice inputs?
2 comments

W1: We thank the reviewer for the careful assessment. We agree that factorized attention, contrastive alignment with hard negatives, and masked cross-modal recovery are established techniques, and we do not claim these individual components as algorithmically novel. Our contribution lies in their cardiac-specific integration into a framework that models the full multi-slice cine CMR sequence across spatial, temporal, and inter-slice dimensions for report-grounded cardiac-function question answering. To evaluate the two novel contributions highlighted by the reviewer—the clinically structured masking stage and the use of the full cine sequence (30 phases per slice)—we ran the Q2 experiments over multiple runs on top of the original experiments, and we added new controlled ablation experiments in Q1.

W2: We thank the reviewer for the suggestion. Regarding the sufficiency of the experiments, for all experiments in the paper as well as the newly added experiments, we ran 5 multiple runs per setting and report the mean ± standard deviation over the 5 runs, to show that the performance differences are not merely due to run-to-run noise, thereby supporting the validity of our method and the constructed QA data.

Regarding the validity of the QA data, we would like to clarify that although the QA data were constructed with the help of an LLM (Qwen), the questions all come from the questions that physicians genuinely focus on when performing cardiac function diagnosis. Qwen's role in question construction was only to rephrase or vary the expression of these clinically relevant questions, and it did not create new questions that physicians do not focus on based on its own prior knowledge. In the answer construction stage, all answers come from the patient's actual clinical report; Qwen's role was to extract the corresponding structural or functional information from the physician's report according to the template we provided, and it did not create new answer content—all answer content still originates from the patient's corresponding clinical diagnostic report. In addition, in the revised manuscript we will add a detailed description of the manual screening procedure and quality-control measures.

Q1: We thank the reviewer for this important question.

We conducted a controlled phase-count ablation: fixing the backbone and all training settings, we varied the number of input cardiac phases over {2, 5, 10, 15, 30}, where 2 phases corresponds to ED and ES frames, and ran 5 random seeds per setting, reporting mean ± std.

PhasesClose-StructureClose-Wall MotionOpen-StructureOpen-Wall Motion
291.21±0.5387.97±0.2365.13±0.3463.31±0.29
591.18±0.6487.94±0.4965.15±0.5363.29±0.38
1091.41±0.5288.17±0.6165.38±0.4163.52±0.50
1591.56±0.3888.83±0.2266.38±0.3463.57±0.31
3091.93±0.2890.38±0.3066.40±0.3165.87±0.37

Additional phases do bring added value, and the gain is concentrated in dynamic (wall-motion) assessment. From ED/ES (2 phases) to the full 30 phases, wall-motion QA improves from 87.97 to 90.38 (+2.41) in the close-ended setting and from 63.31 to 65.87 (+2.56) in the open-ended setting. This is expected: wall motion is a temporal property, and denser phase sampling captures the cardiac cycle beyond the two extremes that ED/ES provide. This is further supported by the clinical reports themselves. Examples of findings we extracted from physician reports include: "reduced and dyssynchronous wall motion"; "reduced motion at the left ventricular apex"; "globally reduced systolic function"; "thinning of the left ventricular wall with regional dyssynchronous motion"; "markedly reduced left ventricular systolic function"; "impaired diastolic function"; and "reduced motion amplitude of the interventricular septum and the left ventricular anterior wall." These cardiac-function findings strictly depend on temporal change across the cardiac cycle, and thus cannot be derived from subsampled frames or from ED/ES alone.

Q2:

We thank the reviewer for this suggestion. Regarding the sufficiency of the experiments, for all experiments in the paper as well as the newly added experiments, we ran 5 multiple runs per setting and report the mean ± standard deviation over the 5 runs, thereby supporting the validity of our method and the constructed QA data.

Table: Comparative analysis on the center 1 and center 2 datasets. Token recall for open-ended questions; accuracy for close-ended questions.

MethodInputClose-StructureClose-Wall MotionOpen-StructureOpen-Wall Motion
Med-flamingo2D72.94±0.6435.90±0.7058.92±0.4556.60±1.12
Llava-med2D79.49±0.6327.88±0.9859.87±0.5953.13±0.44
HealthGPT-Pro-8B3D77.57±0.7036.39±0.3341.45±1.0550.21±0.54
CineMA3D91.61±0.6961.90±0.6365.25±1.8255.14±0.79
Qwen 3.8-27B2D+T54.91±0.3627.39±0.9750.49±2.9343.78±2.91
CMR-CLIP2D+T87.97±0.4184.86±0.5062.97±0.3464.86±0.45
CMR-Transformer2D+T89.82±0.4985.68±0.4764.77±0.6664.87±0.72
Ours (Medical-4D)4D91.93±0.2890.38±0.3066.40±0.3165.87±0.37

Table: Evaluation on Center 3 (unseen during training). Token recall for open-ended questions; accuracy for close-ended questions.

MethodInputClose-StructureClose-Wall MotionOpen-StructureOpen-Wall Motion
Med-flamingo2D73.36±0.5327.12±0.8060.95±0.3657.64±1.25
Llava-med2D80.47±0.7128.71±0.8861.19±0.6854.82±0.33
HealthGPT-Pro-8B3D78.11±0.5547.45±0.5042.05±1.2241.53±0.38
CineMA3D90.48±0.8463.37±0.5565.62±1.6557.31±0.89
Qwen 3.8-27B2D+T55.34±1.2136.58±1.1445.97±2.7543.62±3.02
CMR-CLIP2D+T88.39±0.5789.00±0.4364.27±0.4446.51±0.61
CMR-Transformer2D+T91.30±0.5188.72±0.5865.17±0.4963.34±0.83
Ours(Medical-4D)4D91.58±0.4490.11±0.3666.13±0.4764.92±0.22

Table: Ablation of bidirectional cross-modal recovery and clinically structured masking.

MethodClose-StructureClose-Wall MotionOpen-StructureOpen-Wall Motion
Stage I alignment91.13±0.5889.43±0.5565.65±0.4765.21±0.43
+ slice-mask91.83±0.3289.51±0.2866.29±0.2865.29±0.26
+ slice + temporal mask91.88±0.2190.33±0.3666.29±0.2565.84±0.39
+ Bidirectional Recovery91.93±0.2890.38±0.3066.40±0.3165.87±0.37

Q3: We thank the reviewer for this question.

We confirm that each baseline was evaluated under its original design, and we clarify the exact input configuration below: 1) CineMA: given both ED and ES frames (two time points), exactly as in its original design — not a single time point. 2) CMR-CLIP: given a single slice (one slice across the full cardiac cycle), following its original design. Specifically, we used the mid-ventricular slice. 3) CMR-Transformer: given a single slice's full temporal sequence, and, following its original design, 16 frames were sampled from this sequence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.