acceptodds
Under review as a conference paper at ICLR 2027

Learning Where to Look: Anatomical Grounding and Guided Attention for Cardiac MRI Vision–Language Models

Abstract

Cardiac magnetic resonance imaging (CMR) enables assessment of cardiac anatomy, ventricular function, and myocardial tissue characteristics. Clinicians interpret these images by identifying cardiac structures and focusing on the regions relevant to each clinical question, motivating anatomically guided vision–language models (VLMs). Yet CMR-specific supervision for anatomical localisation and clinical question answering remains limited. To address this gap, we investigate fine-grained CMR visual question answering through anatomical grounding and guided attention. We construct 128,915 anatomical-grounding and 42,799 clinical QA pairs across short-axis cine, late gadolinium enhancement, and long-axis cine. These datasets support anatomical recognition, localisation, and clinical assessment without requiring paired reports for individual training images. To help the model *learn where to look*, we introduce Cardiac Anatomy-Routed Attention (CARA), which selects predicted anatomical priors according to the question and guides decoder attention with learned task-specific strengths. Combining anatomical grounding pretraining with CARA yields our model, CARA-VL. Experiments demonstrate CARA-VL's strengths in clinical assessment and regional localisation across CMR imaging settings, with promising generalization to an external clinical cohort. Together, our data and method provide a practical framework for studying and advancing cardiac visual understanding in VLMs. We will release the QA data derived from public datasets upon publication.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Model, Medical Image, Cardiac MRI, Visual Question-Answering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JmFxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

They design an anatomically grounded VLM for CMR VQA task, and make anatomical-grounding VQA datsets from public CMR resources, which contains 128,915 grounding QA pairs, while quality-controlled image-derived measurements yield 42,799 clinical QA pairs across 14 tasks informed by clinical report analysis.

Critical strengths

They combine anatomical grounding pretraining with CARA to learnwhere to look for clinical question answering. Experiments across 14 clinical tasks show that CARA-VL outperforms the evaluated base-lines on all internal tasks and demonstrates promising external generalization. Ablations support the complementary benefits of anatomical grounding pretraining and CARA.

Critical weaknesses

The labels used for training contains segmentation masks, automatic measured results and predefined threshold generation, which may have human-rule bias. General medical VLM is evaluated by zero-shot way, but CARA-VL is trained on these method, which is not fair. CARA depends on 14 predefined tasks and fixed anatomical rules, which is hard to extend to other open-ended questions and compositional questions. The author uses some attention visualization to show that the model "learned where to look", but sometimes the attention does not provide a correct casual explanation. The author never report the means and standard deviations across several runs. Please add them in the experiments.

Questions for authors

Please see the weakness above to explain some details in your paper clearly. Moreover, some questions are listed as follows:

Why not use the same data in stage 2 to fine-tune the general medical VLM to obtain more fair comparison? I want to confirm whether the attention visualization is analysis in the whole test set, or you just show a few representative successful cases Please tell me whether the automatic segmentation results are evaluated by human experts. Could you please report the means and the standard deviations in multiple random seeds?

6 comments

Response to Weakness: “The labels used for training contains segmentation masks, automatic measured results and predefined threshold generation, which may have human-rule bias.”

We thank the reviewer for highlighting this concern. Our labelling criteria are informed by published clinical standards and quantitative measurement definitions, as described in Appendix C.1. For example, our EF grading is consistent with the 50/40/30% boundaries specified in the ACC/AHA clinical data standards. The task-specific labelling rules and exclusion intervals are detailed in Table 5 in Appendix C.1. Using explicit, shared criteria is intended to reduce subjective variability and makes our reference definitions transparent and reproducible. We nevertheless acknowledge that standardisation does not eliminate bias: the labels remain dependent on segmentation accuracy, measurement methods, and the chosen grading conventions, as discussed in our limitations.

Response to Question: Fine-Tuning Medical VLMs on the Same Stage-2 Data

We thank the reviewer for this constructive suggestion. We have now fine-tuned MedGemma-4B-it and Huatuo-Vision on the same 14-task Stage-2 training pool, using three training epochs and the same test sets and scoring protocol as CARA-VL (Tables 1–2; Appendix D.3). First, fine-tuning on our dataset substantially improves the original baselines. MedGemma-4B-it and Huatuo-Vision improve from 50.9% and 51.9% to 85.7% and 86.5% mean BA on internal SAX, respectively. These gains demonstrate that our dataset provides effective CMR-specific supervision for adapting existing medical VLMs, establishing its value beyond the proposed model. Second, CARA-VL remains stronger after these baselines receive the same Stage-2 supervision. CARA-VL achieves 90.7% mean BA on internal SAX, exceeding the fine-tuned MedGemma and Huatuo models by 5.0 and 4.2 percentage points, respectively, and also achieves higher mean SAX BA and macro-F1 on the external test sets. This supports the effectiveness of our proposed framework beyond clinical QA fine-tuning alone.

The newly added fine-tuned baseline results from Tables 1–2 are reproduced below. Classification entries report BA / macro-F1 (%); regional entries report micro-F1 (%). Avg. covers the six SAX classification tasks.

Table 1(a): ACDC test set (SAX).

ModelHyper.LV sizeRV sizeEFGRSGCSAvg.Reg. hyper.Reg. motion
MedGemma-4B-it (FT)85.3 / 82.475.2 / 76.874.6 / 76.794.6 / 94.590.1 / 90.194.1 / 94.385.7 / 85.857.154.9
Huatuo-Vision (FT)85.2 / 81.777.0 / 78.977.1 / 76.694.1 / 93.790.7 / 90.594.9 / 94.486.5 / 86.055.657.4

Table 1(b–c): the 50-patient LGE test set and M&Ms-2 test set (LAX).

ModelLGE: Scar presenceLGE: TransmuralityLGE: ExtentLAX: EFLAX: EDVLAX: GLS
MedGemma-4B-it (FT)77.1 / 77.564.8 / 65.978.7 / 75.081.3 / 81.074.5 / 76.073.5 / 75.4
Huatuo-Vision (FT)80.4 / 81.267.9 / 68.175.5 / 76.381.8 / 81.979.2 / 79.072.1 / 74.4

Table 2: In-house B external test set. Report-based GRS/GCS labels use WMS proxies.

ModelReferenceHyper.LV sizeRV sizeEFGRSGCSAvg.Reg. hyper.Reg. motion
MedGemma-4B-it (FT)Rule-based85.7 / 90.370.7 / 73.649.4 / 49.567.5 / 56.284.4 / 85.781.5 / 64.573.2 / 70.052.237.8
MedGemma-4B-it (FT)Report-based68.7 / 71.456.9 / 43.750.9 / 44.265.9 / 47.363.5 / 65.569.3 / 71.962.5 / 57.327.035.8
Huatuo-Vision (FT)Rule-based65.7 / 70.864.8 / 68.155.1 / 53.472.1 / 65.678.9 / 75.775.6 / 66.068.7 / 66.627.635.1
Huatuo-Vision (FT)Report-based58.7 / 57.354.6 / 40.151.6 / 45.771.8 / 58.466.0 / 68.568.2 / 71.161.8 / 56.814.339.9

Response to Question : Attention Visualization, Case Selection, and Causal Interpretation

We thank the reviewer for raising this point. Figure 3 serves an illustrative rather than statistical purpose: it is designed to make differences in spatial attention directly visible. Using the same SAX image for two clinical questions allows the attention patterns with and without CARA to be compared while keeping the visual input fixed. Accordingly, the figure presents case-level examples rather than a dataset-wide attention analysis. These visualizations illustrate the behavior of anatomical guidance and are not intended to establish a causal explanation of the predictions.

Response to Question “Please tell me whether the automatic segmentation results are evaluated by human experts”

Yes. All automatically generated masks were visually screened by a PhD student with four years of research experience in cardiac segmentation and a medical imaging researcher with over ten years of experience. The screening focused on obvious anatomical errors, such as missing structures or gross misalignment, rather than fine-grained boundary accuracy. We have added these details to the mask quality-control description in the revised manuscript in Section C.5.1.

Response to Question “Could you please report the means and the standard deviations in multiple random seeds?”

We thank the reviewer for this valuable suggestion. Although multi-seed experiments are informative, repeating complete training runs is particularly costly for large VLMs, and comprehensive reporting across independent training seeds is not universal in current large-model research. Nevertheless, to strengthen the statistical assessment, we have added 95% confidence intervals based on 10,000 patient-cluster bootstrap resamples across the internal and external test sets (Appendix E.1). All QA instances and repeat examinations from each sampled patient are retained together. These intervals quantify test-cohort sampling uncertainty for the fixed trained model, while variability across training seeds remains unassessed.

Table 16 (Appendix E.1): patient-cluster bootstrap confidence intervals for CARA-VL. Entries are point estimates [95% CI] in percent. F1 denotes macro-F1 for classification and micro-F1 for regional tasks (†).

SAX

TaskBA [95% CI]F1 [95% CI]
Hypertrophy86.7 [81.5, 91.2]84.3 [75.7, 90.0]
LV size89.9 [83.6, 95.5]88.4 [80.6, 94.2]
RV size86.6 [75.9, 94.2]88.8 [79.1, 94.8]
EF92.1 [88.0, 95.7]90.4 [85.6, 94.9]
GRS92.6 [85.2, 98.3]92.6 [85.5, 98.0]
GCS96.4 [93.0, 99.0]95.7 [91.3, 98.6]
Six-task average90.7 [87.5, 93.4]90.0 [86.2, 92.7]
Regional hypertrophy†—61.8 [51.0, 69.5]
Regional motion†—55.8 [46.3, 62.8]

Internal LGE (50 patients)

TaskBA [95% CI]F1 [95% CI]
Scar presence82.9 [78.7, 87.5]82.0 [77.6, 86.0]
Transmurality68.5 [57.5, 79.2]71.1 [58.6, 80.9]
Extent82.5 [75.9, 88.3]82.3 [74.8, 87.5]

M&Ms-2: LAX

TaskBA [95% CI]F1 [95% CI]
EF78.1 [72.5, 83.6]74.9 [68.0, 81.6]
EDV84.9 [78.8, 90.4]81.0 [74.4, 87.1]
GLS77.1 [68.4, 85.8]79.6 [70.5, 87.7]

In-house B: Rule-based

TaskBA [95% CI]F1 [95% CI]
Hypertrophy87.1 [83.1, 90.4]90.7 [87.6, 93.1]
LV size82.5 [78.9, 85.6]66.6 [61.9, 70.9]
RV size62.5 [54.4, 70.1]59.1 [52.4, 65.4]
EF73.5 [70.9, 76.1]66.3 [62.7, 69.8]
GRS87.3 [83.5, 90.9]86.0 [82.3, 89.2]
GCS84.1 [79.1, 88.5]65.8 [60.8, 70.7]
Six-task average79.5 [77.5, 81.4]72.4 [70.4, 74.2]
Regional hypertrophy†—55.1 [49.4, 60.1]
Regional motion†—36.7 [31.9, 41.2]

In-house B: Report-based

TaskBA [95% CI]F1 [95% CI]
Hypertrophy69.9 [65.5, 74.3]72.7 [67.3, 77.7]
LV size64.0 [60.6, 67.4]60.7 [56.5, 64.7]
RV size52.4 [51.1, 53.9]46.8 [44.0, 49.8]
EF72.6 [70.2, 75.1]58.2 [53.9, 62.3]
GRS62.7 [59.4, 66.2]64.5 [60.2, 68.9]
GCS75.0 [71.3, 78.6]77.2 [73.6, 80.7]
Six-task average66.1 [64.4, 67.8]63.4 [61.2, 65.4]
Regional hypertrophy†—30.9 [24.3, 37.3]
Regional motion†—34.5 [28.7, 40.2]

Thanks for your response. It would be nice if you could show me more experiments instead of explanations. Could you please conduct more experiments and visualizations to demonstrate the correctness of your explanations?

Reviewer_PBzROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates fine-grained cardiac MRI visual question answering (VQA) through anatomical grounding and guided attention. The authors construct 128,915 anatomical grounding and 42,799 clinical QA pairs across 14 clinical tasks. They propose CARA-VL, which combines anatomical grounding pretraining with task-specific anatomical attention guidance. Experiments demonstrate improvements over the evaluated baselines across 14 internal tasks.

Critical strengths
  1. Construction of fine-grained cardiac MRI VQA datasets.

The paper constructs anatomical grounding and clinical QA datasets covering cardiac structure, function, and myocardial scar. The use of segmentation annotations and image-derived measurements to generate QA supervision without paired clinical reports is practically valuable.

  1. Comprehensive experimental evaluation.

The paper evaluates multiple medical and cardiac-specific VLMs across 14 clinical tasks and three CMR imaging settings.

Critical weaknesses

W1. Limited motivation for the proposed design.

The motivation and novelty of the proposed CARA framework are not sufficiently established. Although the paper focuses on cardiac MRI, the proposed method appears largely modality-agnostic. In fact, apart from the proposed dataset and CMR-specific anatomical labels, I do not see any design in the proposed pretraining or CARA that specifically exploits the characteristics of CMR.

Specifically, CARA predicts anatomical occupancy maps from visual features, selects the relevant map using a predefined task-to-region mapping, and injects it into decoder attention through a learned scalar. These operations do not incorporate any distinctive characteristics of CMR. Similarly, anatomical grounding pretraining primarily consists of standard visual grounding tasks generated from segmentation annotations, with CMR-specific anatomical labels.

While these components are applicable to cardiac MRI, their designs appear equally applicable to other medical imaging or visual QA tasks with segmentation annotations. The paper does not adequately justify why the proposed architecture is particularly suitable for CMR or why this specific combination of anatomical grounding and attention guidance is necessary. Furthermore, the experiments do not compare CARA against simpler alternatives for incorporating anatomical information, such as region cropping or mask-guided visual inputs. Consequently, the methodological contribution and motivation for the proposed design remain unclear.

W2. The performance gains appear to come primarily from domain-specific training rather than the proposed methods.

The main comparisons in Tables 1–2 are primarily against medical or CMR-specific models evaluated without adaptation to the proposed training dataset (i.e., they are evaluated zero-shot), whereas CARA-VL receives extensive domain-specific anatomical and clinical supervision. Consequently, the large improvements over these baselines cannot be attributed solely to the proposed methods.

This concern is further supported by Table 3. The model without either Stage-1 pretraining or CARA already achieves 86.2% mean balanced accuracy on SAX, compared with 90.7% for the full model. While the paper reports approximately 40 percentage point improvements over general medical VLMs, the two proposed components together contribute only an additional 4.5 percentage points. This suggests that the majority of the reported performance improvements come from domain-specific clinical QA training rather than the proposed methods, raising concerns about their actual contribution.

W3. Overclaim of systematic analysis.

In line 72, the authors claim, "We systematically investigate cardiac visual understanding in vision–language models through fine-grained CMR VQA, examining how anatomical recognition and localisation support clinical assessment of cardiac structure, function, and myocardial scar." I believe the first half of this claim is supported. However, how anatomical recognition and localisation support clinical assessment is not explicitly examined in the paper. Although Table 3 demonstrates the benefits of anatomical grounding pretraining and CARA, it does not investigate how improvements in anatomical recognition or localisation relate to clinical QA performance. Therefore, claiming that this relationship has been systematically investigated appears overstated.

Questions for authors

Please see the weakness.

5 comments

Response to Question 1 (W1):

We thank the reviewer for the comprehensive and constructive comments on our design motivation and the need to consider simpler alternatives.

CMR is particularly suitable for this approach for two reasons. First, the questions studied here repeatedly involve a compact, consistent set of structures—the LV cavity, RV cavity, and myocardium—for which segmentation methods are well established. For example, Isensee et al. reported ACDC test-set Dice scores of 0.950, 0.923, and 0.911 for these structures, respectively. This makes anatomical supervision practical and motivates a shared representation that can support multiple clinical tasks. Second, the geometry and dynamics of these structures are directly related to the clinical quantities assessed in CMR: myocardial thickness informs hypertrophy, cavity volume informs ventricular enlargement, and end-diastolic–end-systolic volume changes determine ejection fraction. For several of our QA tasks, these anatomical measurements directly define the reference answers. For scar assessment, myocardial boundaries provide the spatial context for interpreting LGE enhancement relative to the myocardial wall. These relationships motivate learning both the shared anatomy and which structure is relevant to each question. Some existing medical VLMs incorporate segmentation or region guidance. For example, MedVP (Zhu et al., 2025) uses visual markers to guide attention, while LobA (Nguyen et al., 2025) uses anatomical masks to localize evidence for disease-related questions. In some of these settings, anatomical guidance primarily indicates where to inspect, with a less direct connection between anatomical geometry and the answer than in our measurement-derived CMR tasks. We will expand the Discussion to clarify this distinction and the CMR-specific motivation for our design.

In our early experiments, directly overlaying question-relevant masks on CMR images and adding prompts identifying the highlighted anatomy improved QA performance. However, we don't develop that method because many reference answers are derived from measurements computed using these same masks. Providing them at test time therefore exposes privileged, answer-related information and risks target leakage. This motivated CARA, which learns to predict latent anatomical guidance from images without requiring reference masks as model inputs. We agree that reporting the simpler baseline is nevertheless informative and have now added it in Section 4.3 and Table 4, with implementation details in Appendix D.3. With the specific mask prompt, the model get better performance. The mask-overlay baseline achieves 92.6% mean BA and 91.3% mean macro-F1, compared with CARA-VL's 90.7% and 90.0%, and also improves both regional scores. These results support the value of anatomical information for QA and motivate an implicit approach such as CARA. By learning to predict anatomical guidance directly from images, CARA can exploit this information without receiving the reference masks used to construct QA labels, allowing application to unseen cases where reference masks are unavailable.

We have now added the direct mask-prompting experiment in Section 4.3, under “Can anatomical mask prompts improve clinical QA?”, with results in Table 4 and implementation details in Appendix D.3. On ACDC SAX, mask overlays achieve 92.6% mean BA and 91.3% mean macro-F1, compared with 90.7% and 90.0% for CARA-VL. Regional hypertrophy and motion micro-F1 also increase from 61.8% to 71.8% and from 55.8% to 64.6%, respectively. These results support the value of anatomical information for QA and motivate an implicit approach such as CARA. By learning to predict anatomical guidance directly from images, CARA can exploit this information without receiving the reference masks used to construct QA labels, allowing application to unseen cases where reference masks are unavailable.

Reference-mask overlay comparison on ACDC SAX (Table 4). Classification entries report BA / macro-F1 (%); regional entries report micro-F1 (%). The overlay model receives the reference masks also used to construct QA labels; CARA-VL receives the original images. The average covers the six classification tasks.

ModelHypertrophyLV sizeRV sizeEFGRSGCSAverageRegional hypertrophyRegional motion
Mask overlay91.1 / 85.487.6 / 85.989.5 / 90.794.5 / 93.496.5 / 97.096.4 / 95.792.6 / 91.371.864.6
CARA-VL86.7 / 84.389.9 / 88.486.6 / 88.892.1 / 90.492.6 / 92.696.4 / 95.790.7 / 90.061.855.8

Response to Question 2 (W2):

We thank the reviewer for this constructive suggestion. We have now fine-tuned MedGemma-4B-it and Huatuo-Vision on the same 14-task Stage-2 training pool and evaluated them using the same test sets and scoring protocol (Tables 1–2; Appendix D.3). First, fine-tuning on our dataset substantially improves the original baselines. MedGemma-4B-it and Huatuo-Vision improve from 50.9% and 51.9% to 85.7% and 86.5% mean BA on internal SAX, respectively. These gains demonstrate that our dataset provides effective CMR-specific supervision for adapting existing medical VLMs, establishing its value beyond the proposed model. Second, CARA-VL remains stronger after these baselines receive the same Stage-2 supervision. CARA-VL achieves 90.7% mean BA on internal SAX, exceeding the fine-tuned MedGemma and Huatuo models by 5.0 and 4.2 percentage points, respectively, and also achieves higher mean SAX BA and macro-F1 on the external test sets. This supports the effectiveness of our framework beyond clinical QA fine-tuning alone. Table 3 further supports the contribution of our methods within the same backbone: the full framework improves mean BA from 86.2% to 90.7%, and adding CARA with Stage-1 retained improves it from 86.8% to 90.7%.

The newly added fine-tuned baseline results from Tables 1–2 are reproduced below.

Table 1(a): ACDC test set (SAX).

ModelHyper.LV sizeRV sizeEFGRSGCSAvg.Reg. hyper.Reg. motion
MedGemma-4B-it (FT)85.3 / 82.475.2 / 76.874.6 / 76.794.6 / 94.590.1 / 90.194.1 / 94.385.7 / 85.857.154.9
Huatuo-Vision (FT)85.2 / 81.777.0 / 78.977.1 / 76.694.1 / 93.790.7 / 90.594.9 / 94.486.5 / 86.055.657.4

Table 1(b–c): the 50-patient LGE test set and M&Ms-2 test set (LAX).

ModelLGE: Scar presenceLGE: TransmuralityLGE: ExtentLAX: EFLAX: EDVLAX: GLS
MedGemma-4B-it (FT)77.1 / 77.564.8 / 65.978.7 / 75.081.3 / 81.074.5 / 76.073.5 / 75.4
Huatuo-Vision (FT)80.4 / 81.267.9 / 68.175.5 / 76.381.8 / 81.979.2 / 79.072.1 / 74.4

Table 2: In-house B external test set.

ModelReferenceHyper.LV sizeRV sizeEFGRSGCSAvg.Reg. hyper.Reg. motion
MedGemma-4B-it (FT)Rule-based85.7 / 90.370.7 / 73.649.4 / 49.567.5 / 56.284.4 / 85.781.5 / 64.573.2 / 70.052.237.8
MedGemma-4B-it (FT)Report-based68.7 / 71.456.9 / 43.750.9 / 44.265.9 / 47.363.5 / 65.569.3 / 71.962.5 / 57.327.035.8
Huatuo-Vision (FT)Rule-based65.7 / 70.864.8 / 68.155.1 / 53.472.1 / 65.678.9 / 75.775.6 / 66.068.7 / 66.627.635.1
Huatuo-Vision (FT)Report-based58.7 / 57.354.6 / 40.151.6 / 45.771.8 / 58.466.0 / 68.568.2 / 71.161.8 / 56.814.339.9

Response to Question 3 (W3):

We thank the reviewer for pointing out this distinction. Table 5 and Appendix C.5 explain how anatomical measurements define the clinical reference answers, such as myocardial thickness for hypertrophy and ventricular volumes for size and function. They establish the anatomical basis of our tasks, but do not directly analyze the relationship between the model's anatomical recognition/localization accuracy and its clinical QA performance. Table 3 evaluates the contributions of anatomical pretraining and CARA at the component level. We agree that the original wording can imply a stronger analysis than the evidence presented, and will revise it to: “We systematically investigate cardiac visual understanding in vision–language models through fine-grained CMR VQA, evaluating the effects of anatomical grounding pretraining and anatomy-routed attention on clinical QA performance.”

We thank the reviewer again for the constructive feedback, which has helped us strengthen the manuscript. We hope that the additional experiments, analyses, and clarifications address your concerns and support a more favourable assessment of our work. We would be grateful for your reconsideration in light of these revisions.

Thank you again for your constructive feedback. As the discussion period is nearing its end, we would be grateful for any further thoughts on our updated response and whether any points would benefit from additional clarification. We hope the new evidence is helpful as you finalize and update your assessment. Thank you for your time and consideration!

Reviewer_BQ89OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses fine‑grained cardiac magnetic resonance visual question answering, where existing general‑purpose medical vision‑language models lack specialised anatomical awareness for CMR interpretation. The authors construct two complementary datasets without requiring paired image‑report samples: 128915 anatomical‑grounding QA pairs for anatomical recognition/localisation, and 42799 clinical QA pairs covering 14 clinical assessment tasks across short‑axis cine, LGE and long‑axis cine CMR. Its core module CARA predicts soft anatomical occupancy maps from visual features and routes task‑relevant anatomical priors to modulate decoder attention, guiding the model to attend to clinically meaningful cardiac structures. Training follows a two‑stage pipeline: stage‑1 anatomical grounding pretraining, then stage‑2 clinical QA fine‑tuning with LoRA. Evaluations are performed on internal held‑out test sets plus an external real‑world clinical cohort with dual reference standards (measurement‑derived rule‑based labels and human clinical report‑based labels). CARA‑VL outperforms general medical VLMs and existing CMR‑specific baselines across most internal metrics.

Critical strengths

1.The work builds large‑scale anatomical‑grounding and clinical QA datasets derived from public segmentation masks and quantitative image measurements, removing the dependency on scarce paired CMR image‑report training data. The 14 clinically‑motivated QA tasks are defined by analysing real clinical radiology reports, aligning with routine CMR assessment workflows. 2. The proposed CARA generates image‑wise soft anatomical occupancy maps and routes task‑specific anatomical priors to bias decoder attention, explicitly encouraging the VLM to focus on relevant cardiac structures instead of relying on implicit learned attention. Qualitative attention visualisations demonstrate its spatial guidance effect. 3.Stage‑1 anatomical grounding pretraining strengthens the model’s ability to recognise and localise cardiac anatomy before clinical QA fine‑tuning. Ablation experiments validate that anatomical pretraining and CARA bring complementary performance improvements. 4. Evaluations cover internal held‑out benchmarks and a fully separate external clinical cohort. Two distinct reference standards (rule‑based measurement labels and clinical‑report labels) are used for external testing, exposing performance gaps between measurement‑based ground‑truth and real‑world radiologist reporting practice. Multiple metrics including balanced accuracy, macro‑F1 and regional micro‑F1 are reported. 5. Extensive appendices detail dataset curation rules, segmentation quality‑control, measurement formulas, hyper‑parameters and evaluation metrics. The authors commit to releasing public‑dataset‑derived QA resources to boost CMR VQA research.

Critical weaknesses
  1. Nearly all training and internal‑test labels are generated algorithmically from segmentation masks and numeric measurements rather than direct radiologist annotations. These labels inherit segmentation errors and fixed threshold rules, and may diverge from actual clinical report conventions, as demonstrated by large disagreement rates between rule‑based and report‑based references on the external cohort.
  2. Although QA pair counts are large, many questions are augmented / derived from repeated images of the same patient examinations, so the number of unique patients is substantially smaller than QA sample counts. External evaluation is only available for short‑axis cine tasks and lacks LGE/long‑axis external test data.
  3. CARA‑VL achieves strong internal‑set results but has substantially degraded agreement with human clinical‑report‑based labels on the external cohort. Root‑cause analysis for this large performance drop is superficial; it remains unclear whether this arises from domain shift, label‑standard mismatch, or model deficits.
  4. There is no comparison against alternative anatomical‑grounded attention / prompting strategies (e.g. providing segmentation masks as input prompts, anatomy‑aware prompt engineering). It is hard to isolate how much gain is uniquely contributed by the CARA routing mechanism versus anatomical pretraining alone.
  5. All QA tasks are closed‑set, template‑based questions with pre‑defined answer categories. The model is not evaluated on free‑form open‑ended CMR report generation, which represents a major real‑world clinical use‑case.
  6. There is no formal human expert study to rate the clinical plausibility of model outputs, even though clinical interpretability and real‑world usability are core motivations of this work.
Questions for authors
  1. There is substantial disagreement between rule‑based measurement‑derived labels and clinical‑report‑based labels on the In‑house B external cohort. Could you quantitatively decompose how much CARA‑VL’s performance drop under report‑based references comes from (1) intrinsic discrepancies between two labelling standards versus (2) true model generalisation failure on real clinical cases? What concrete steps could mitigate this label‑standard mismatch for future CMR‑VQA benchmarking?
  2. Many QA samples are generated via augmentation from a limited pool of unique patient studies, inflating total QA counts without increasing patient diversity. Could you report your main ablation and primary test‑set results re‑aggregated at the patient‑level (instead of per‑QA‑instance level), to rule out performance over‑estimation caused by heavily augmented samples from the same patient?
  3. Why did you not compare CARA against simpler anatomy‑aware baselines, such as feeding segmentation masks/map overlays as explicit prompt inputs or anatomy‑focused prompt engineering? Can you disentangle how much performance improvement originates specifically from the CARA routing module, rather than purely from the heavy anatomical‑grounding stage‑1 pretraining?
  4. Your current evaluation is restricted to closed‑set template QA. What are the main obstacles preventing you from extending your framework to open‑ended CMR report generation? Could you provide pilot qualitative examples of free‑form outputs from CARA‑VL to illustrate its strengths / remaining failure modes for real‑world reporting?
6 comments

Response to Question 1

We thank the reviewer for this important suggestion. Our new paired analysis shows that, across hypertrophy, LV size, RV size, and EF, 65.7% of Report-based errors occur where the references disagree and CARA-VL matches the Rule-based label (95% CI: 62.6–68.7%). This fraction reaches 79.3–91.6% for the three anatomical tasks, highlighting substantial reference dependence in the external evaluation. Detailed results and interpretation are provided in Appendix E.3, Table 18. To mitigate this mismatch, we recommend first collecting sufficient paired CMR image-report data to better capture clinical reporting conventions, and then harmonising labelling criteria and using expert review to resolve label disagreements.

Results from Appendix E.3, Table 18. Conflict-associated errors are Report-based errors where the references disagree and CARA-VL matches the Rule-based label. Both references score identical QA instances; intervals are 95% patient-cluster bootstrap CIs.

TaskPaired QALabel disagreement (%)Report-based errors: conflicting / totalConflict share (%) [95% CI]
Hypertrophy1,93615.9286 / 35081.7 [76.7, 86.2]
LV size1,78046.9552 / 69679.3 [73.6, 84.4]
RV size1,77831.3515 / 56291.6 [88.1, 94.7]
EF1,79513.3182 / 72725.0 [19.3, 30.8]
GRS∗1,70821.6318 / 35090.9 [86.8, 94.4]
GCS∗1,68422.3195 / 25775.9 [68.0, 82.7]
First four tasks pooled7,28926.61,535 / 2,33565.7 [62.6, 68.7]

Response to Question 2

We thank the reviewer for this important suggestion. We have now added patient-level classification results for the internal and external test sets, using the same final model and evaluation archives as the main tables (Appendix E.2, Table 17). Predictions and reference labels are aggregated separately within each patient using task-specific rules, so patients with more QA instances no longer contribute multiple entries to each task's classification metric. The table reports patient-level BA and macro-F1 and includes the original QA-level SAX averages for comparison. CARA-VL maintains strong performance on internal SAX (94.2% mean BA), while external patient-level mean BA remains comparable to the QA-level results under both rule-based (77.9% vs. 79.5%) and report-based (69.3% vs. 66.1%) references.

SAX averages for the final CARA-VL model from Appendix E.2, Table 17. BA and macro-F1 are averaged across the six classification tasks (%). Task-specific results, including LGE and LAX, are reported in the full table.

Test set / referenceQA-level BAQA-level macro-F1Patient-level BAPatient-level macro-F1
Internal SAX90.790.094.292.1
External SAX / Rule-based79.572.477.971.1
External SAX / Report-based66.163.469.366.8

Response to Question 4

We thank the reviewer for raising this important question. Our initial goal was CMR report generation, and we chose closed-set QA as a first step for three reasons.

First, our report analysis identified recurring findings, such as normal versus reduced myocardial motion and normal versus enlarged ventricular size, that naturally translate into a common set of clinical questions. These findings informed our task definitions, while anatomical annotations and quantitative measurements supplied the training answers. Second, our available paired image–report data provided limited coverage of less frequent findings outside these categories. For example, pericardial effusion may be documented in only a dozen or so patients, leaving too few positive cases to support reliable training for this finding. Third, closed-set QA provides a structured foundation for report generation. Once the findings are predicted reliably, a language model could organize them into report text, with coverage determined by the assessed findings. Direct generation of comprehensive reports would require larger paired datasets and more complete imaging coverage across sequences, views, and cardiac phases than are currently available to us.

We have emphasized in the Discussion that our closed-set QA study is a preparatory step toward CMR report generation and have discussed its current limitations and potential extensions to open-ended clinical questions and comprehensive report generation.

We thank the reviewer again for the constructive feedback, which has helped us strengthen the manuscript. We hope that the additional experiments, analyses, and clarifications address your concerns and support a more favourable assessment of our work. We would be grateful for your reconsideration in light of these revisions.

Response to Question 3:

We thank the reviewer for the constructive suggestion to compare simpler anatomy-aware alternatives and distinguish the contribution of CARA from anatomical-grounding pretraining.

In our early experiments, directly overlaying question-relevant masks on CMR images and adding prompts identifying the highlighted anatomy improved QA performance. However, we don't develop that method because many reference answers are derived from measurements computed using these same masks. Providing them at test time therefore exposes privileged, answer-related information and risks target leakage. This motivated CARA, which learns to predict latent anatomical guidance from images without requiring reference masks as model inputs. We agree that reporting the simpler baseline is nevertheless informative and have now added it in Section 4.3 and Table 4, with implementation details in Appendix D.3. With the specific mask prompt, the model get better performance. The mask-overlay baseline achieves 92.6% mean BA and 91.3% mean macro-F1, compared with CARA-VL's 90.7% and 90.0%, and also improves both regional scores. These results support the value of anatomical information for QA and motivate an implicit approach such as CARA. By learning to predict anatomical guidance directly from images, CARA can exploit this information without receiving the reference masks used to construct QA labels, allowing application to unseen cases where reference masks are unavailable.

Reference-mask overlay comparison on ACDC SAX (Table 4). Classification entries report BA / macro-F1 (%); regional entries report micro-F1 (%). The overlay model receives the reference masks also used to construct QA labels; CARA-VL receives the original images. The average covers the six classification tasks.

ModelHypertrophyLV sizeRV sizeEFGRSGCSAverageRegional hypertrophyRegional motion
Mask overlay91.1 / 85.487.6 / 85.989.5 / 90.794.5 / 93.496.5 / 97.096.4 / 95.792.6 / 91.371.864.6
CARA-VL86.7 / 84.389.9 / 88.486.6 / 88.892.1 / 90.492.6 / 92.696.4 / 95.790.7 / 90.061.855.8

To distinguish CARA's contribution from Stage-1 pretraining, Table 3 compares all four combinations of the two components. With Stage-1 retained, adding CARA raises BA from 86.8% to 90.7% (+3.9 percentage points) and macro-F1 from 86.5% to 90.0% (+3.5 points). Without Stage-1, CARA still improves BA by 2.2 points and macro-F1 by 1.9 points. Conversely, Stage-1 provides gains of 0.6/0.5 points in BA/macro-F1 without CARA, and 2.3/2.1 points with CARA. These results show that the gains cannot be explained by Stage-1 pretraining alone and support complementary benefits from anatomical grounding and CARA. They quantify the contribution of CARA as a module, rather than isolating each operation within it.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary for the Area Chair

We thank the Area Chair and reviewers for their careful assessment. We summarize the main additions and clarifications below.

  1. Reviewers PBzR and JmFx requested fairer comparisons with medical VLMs trained on our data. We fine-tuned MedGemma-4B-it and Huatuo-Vision on the same 14-task Stage-2 training pool. Both improve substantially over their original baselines, demonstrating the value of CMR-specific supervision, while CARA-VL retains higher mean SAX BA and macro-F1 on the internal and external evaluations. The results are reported in Tables 1–2, with analysis in Section 4.2 and training details in Appendix D.3.
  2. Reviewers BQ89 and PBzR requested simpler anatomy-aware baselines and clearer attribution of the gains. We added a reference-mask overlay baseline in Section 4.3 and Table 4, with implementation details in Appendix D.3. This comparison supports the value of anatomical information while explicitly distinguishing reference-mask access from CARA’s image-predicted guidance. Our responses also clarify the CMR-specific motivation and explain how the existing four-way ablation in Table 3 separates the contributions of anatomical grounding and CARA.
  3. Reviewer BQ89 requested a quantitative analysis of the external performance gap between reference standards. We added a paired analysis using identical predictions and matched QA instances under both references. Across hypertrophy, LV size, RV size, and EF, 65.7% of Report-based errors occur where the references disagree and CARA-VL matches the Rule-based label. These results highlight substantial reference dependence in the evaluation. Details are provided in Appendix E.3 and Table 18.
  4. Reviewer BQ89 requested patient-level evaluation, and reviewer JmFx raised concerns about result stability. We added patient-level classification results for the internal and external test sets in Appendix E.2, Table 17. We also report 95% confidence intervals from 10,000 patient-cluster bootstrap resamples in Appendix E.1, Table 16, quantifying test-cohort sampling uncertainty for the fixed trained model.
  5. Reviewers BQ89 and JmFx questioned segmentation and label reliability. We clarified the human screening process, reviewer experience, screening scope, and handling of rejected masks in Appendix C.5.1, with accepted and rejected examples in Figure 8. Our responses also explain the task-specific labelling criteria and exclusion intervals documented in Appendix C.1, Table 5.
  6. The remaining concerns involve interpretation and scope. We clarified the illustrative purpose of the attention visualizations and revised the Introduction’s contribution statement to accurately describe the component-level evaluation. We also expanded the Discussion to explain closed-set QA as a preparatory step toward CMR report generation, including limitations in paired supervision and imaging coverage. Our rebuttal further discusses the fixed-routing limitation and extensions to open-ended questions.

We hope that these additional experiments, analyses, and clarifications address the reviewers’ concerns and strengthen the evidence for our contributions. We thank the reviewers and the Area Chair for their constructive feedback, which has helped us improve the quality and clarity of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.