acceptodds
Under review as a conference paper at ICLR 2027

Framing Abnormalities in Context: Reciprocal Grounding for Radiograph Vision–Language Pre-training

Abstract

Medical vision–language pre-training benefits from the rich supervision contained in radiology reports. However, existing methods often model focal clinical findings and their surrounding descriptions independently, without explicitly coordinating the complementary semantics they provide. We propose FRAME (Finding–context Reciprocal Alignment with Masked report reconstruction and Evidence-focused summary), a radiograph pre-training framework that couples finding-level and sentence-level semantics through shared visual evidence. Specifically, finding-conditioned visual representations are aligned with their corresponding sentence contexts, while context-conditioned visual representations are reciprocally aligned with the associated findings. This bidirectional constraint encourages the visual encoder to preserve both abnormality-specific evidence and its broader clinical context. Two complementary proxy tasks further strengthen the learned representation. Masked report reconstruction promotes the retention of dense diagnostic information, whereas evidence-focused summary emphasizes concise and clinically salient semantics. We evaluate FRAME across classification, segmentation, detection, visual question answering, and report generation under limited-label and cross-dataset settings. FRAME consistently outperforms strong Med-VLP baselines across these task families, especially with only 1% of the training labels. Ablation studies further demonstrate the advantage of reciprocal over unidirectional alignment and verify the complementary contributions of both proxy objectives. These results establish reciprocal finding–context grounding as an effective mechanism for learning transferable radiographic representations. Code will be available.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
medical vision-language pre-training, radiograph representation learning, vision-language alignment, multimodal representation learning, chest X-ray, medical image analysis

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_cTkZOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes frame, a visual language pre training framework for chest X-ray images. The author believes that although the existing methods will use the local disease concept and sentence level semantic information in radiology reports, they usually lack the explicit modeling of the relationship between them. In order to solve this problem, the author proposes finding – context reciprocal alignment (FRA), which uses the disease concept to guide the image feature to align with the sentence context, and uses the sentence context to guide the image feature to align with the corresponding disease concept through cross modal alignment in two directions. In addition, the author introduced masked report reconstruction (MRR) and evidence focused summary (EFS) to retain fine-grained clinical information and highlight important diagnostic semantics. The authors evaluated the tasks of classification, segmentation, target detection, medical visual question answering and report generation, and the results showed that frame achieved better performance than the comparison method on multiple datasets.

Critical strengths
  1. It is significant to study the problem. Now this paper models from the relationship between finding and context, and the overall motivation is relatively clear.
  2. The experiment is comprehensive and the overall performance is good. The authors evaluated five kinds of tasks: classification, segmentation, target detection, medical visual question answering and report generation, and considered different annotation data proportions and cross dataset migration scenarios. Frame has achieved good results in many tasks, especially in the case of less labeled data.
Critical weaknesses
  1. In this paper, the innovation is limited, and fine-grained alignment between local image regions and report words has been studied in a variety of methods.
  2. The author uses gpt-4o to generate brief abstracts from the original radiology report, and then uses these abstracts as the training objectives of the image generation task. However, the automatically generated summary may omit important exception information, or change the negative and uncertain expression in the original report. The statistical results in the appendix also show that the generated summary does not completely retain all the clinical concepts in the source report. Will it make the model learn the wrong representation to conduct supervised learning according to this?
  3. If we use a large language model which is more powerful than GPT, will it improve the performance of this method? However, this part of the experiment is not reflected in the article. It is suggested that the author add an ablation experiment using different large language models.
Questions for authors

Does masked report reconstruction really use image information. MRR reconstructs the original text through image information and part of the unobstructed report content. However, the article also said that the reconstruction loss affects all non padding tokens, not just the masked location, so whether some predictions may directly rely on existing text input. Even for occluded tokens, the model may recover through the language context, rather than making full use of the visual information in the image.

6 comments
AuthorsOpenReview ↗

Response to Reviewer cTkZ (Part 1/2)

Thank you for recognizing the method’s soundness and empirical performance. We focus on the specific contribution beyond existing fine-grained alignment, then address summary supervision and MRR’s use of images.

Common concern W1 What is new and why it matters

FRA makes the finding–sentence relation a supervision constraint on visual evidence. In Finding-Guided Context Reinforcement (FCR), a finding queries the image and the resulting visual response aligns with its containing sentence. In Context-Anchored Finding Alignment (CFA), that sentence queries the image and its response aligns separately with each linked finding. Contextualization enriches the query but does not determine the target used to train its visual response. FRA assigns that target at the linked textual level, without assuming independent text features.

ECAMP uses entity-centered context distillation and reconstruction, and ASG uses anatomical region–sentence alignment and image-tag supervision. RadZero compares a sentence-conditioned visual response with that sentence; CARZero learns cross-attention similarity in both modality directions. FRA exchanges the conditioning and target roles of two linked textual levels through visual evidence. The proposed element is this cross-level supervision relation, implemented with standard cross-attention.

Eq. (1) defines exact occurrence–sentence positives. Against Table R1’s first row, FRA changes only the alignment targets—from the querying text to the linked text at the other level—while retaining both query branches, MRR/EFS, training budget and downstream settings.

All variants in Table R1 are independently pretrained with matched budgets, MRR/EFS supervision and downstream settings. Scores are percentages. IU F1 denotes CheXbert-14 micro-F1. VQA-RAD closed accuracy in this table is the mean of five runs.

Table R1 Controlled comparison of alignment objectives

AlignmentCheXpert 1% AUCRSNA 1% mAPVQA-RAD closed accuracyIU X-Ray F1
Align to the querying text88.618.368.448.3
Global-pooling alignment87.412.366.143.2
FRA90.526.171.358.3

FRA gains 1.9 AUC, 7.8 mAP, 2.9 VQA accuracy and 10.0 IU F1 points. Both correlated queries and auxiliary objectives already exist in the control; these gains support the value of cross-level supervision beyond fine-grained querying alone.

MRR and EFS are supporting objectives. Table R2 shows their complementarity with FRA; Table R3 directly tests whether the alignment benefit extends to localization.

Table R2 reproduces selected results from original Table 4; MRR includes local and global visual conditioning. Scores are percentages.

Table R2 Component ablations reproduced from original Table 4

Pretraining objectivesCheXpert 1% AUCRSNA 1% mAPIU X-Ray F1
MRR + EFS without FRA88.49.854.6
CFA + MRR + EFS89.911.156.8
FCR + MRR + EFS90.119.557.1
FRA + EFS without MRR89.115.255.2
FRA + MRR without EFS88.225.354.2
FRAME with all objectives90.526.158.3

Table R3 uses all 880 images and 984 image–finding queries from the NIH ChestX-ray14 bounding-box cohort (eight findings), with frozen parameters and no spatially supervised adaptation. FCR uses finding queries; CFA uses “There is {finding}.” FRAME and its one-direction variants average all 12 attention heads. All methods use the same annotated image–finding pairs and scoring rules. Both metrics are query micro-averages: pointing game (PG) tests whether the peak patch center lies in a target box; mask IoU compares the min–max-normalized heatmap thresholded at 0.5 with GT rectangular masks.

Table R3 Grounding without spatially supervised adaptation

Model and branchPG (%)Mask IoU (%)
BioViL-T35.7714.15
MaCo25.0015.39
ECAMP28.8614.35
CFA-only model / CFA27.5414.73
FCR-only model / FCR22.3615.26
FRAME / CFA43.2916.85
FRAME / FCR38.7217.99

CFA’s finding-level target may favor finding-discriminative evidence, consistent with its higher PG (43.29 vs. 38.72); FCR aligns to the containing sentence and has higher mask IoU (17.99 vs. 16.85). This interpretation is not isolated from differences in query form. The controlled result is that reciprocal training improves PG/mask IoU over the respective one-direction models by 15.75/2.12 points for CFA and 16.36/2.73 for FCR, with both branches outperforming all three baselines.

AuthorsOpenReview ↗

Response to Reviewer cTkZ (Part 2/2)

W2 and W3 Summary fidelity and generator dependence

EFS uses offline GPT-4o summaries as auxiliary targets while MRR retains supervision from the original report. Appendix A.3 reports 58.5% source-concept retention, 99.4% target concepts present in the source and 89.3% shared-concept polarity agreement. Incomplete coverage and incorrect polarity are distinct concerns; these statistics do not establish error-free targets.

Table R2 measures utility despite imperfect targets: adding EFS to FRA + MRR improves CheXpert AUC from 88.2 to 90.5 and IU F1 from 54.2 to 58.3. Alternative summary generators yield downstream scores within 0.5 points of GPT-4o (Table R4), without a consistent GPT-4o advantage. This supports usefulness across the tested generators, not the fidelity of every summary.

Nemotron denotes Llama-3.1-Nemotron-70B-Instruct. Comparing normalized entity–attribute–certainty triples with GPT-4o summaries yields micro-F1 of 97.3% for Nemotron and 95.6% for Qwen2.5-7B. This measures agreement with GPT-4o summaries, not fidelity to the source reports, and complements the stable downstream results in Table R4. All table metrics are percentages. VQA-RAD closed accuracy in this table is the mean of five runs.

Table R4 Retraining with alternative summary generators

GeneratorCheXpert 1% AUCRSNA 1% mAPVQA-RAD closed accuracyIU X-Ray F1
GPT-4o90.526.171.358.3
Nemotron-70B90.725.671.257.8
Qwen2.5-7B-Instruct90.225.971.358.7

Question Does MRR use visual information

MRR trains on all non-padding positions; this diagnostic evaluates only masked positions. On 5,000 distinct patients in the patient-disjoint CheXpert Plus validation split, we hold reports, 75% masks and weights fixed and cyclically shift the images, verifying every replacement is cross-patient. Each patient supplies one report and one frontal image. Mismatches are not selected by disease label.

Table R5 MRR accuracy at masked positions

Visual inputAll masked tokensMasked finding tokens
Paired image30.42%41.51%
Mismatched image16.38%15.74%

With identical language context, wrong images reduce masked-finding accuracy by 25.77 points: MRR’s predictions are not determined by text alone. Separately, removing MRR lowers detection mAP from 26.1 to 15.2 and IU F1 from 58.3 to 55.2 (Table R2), demonstrating its benefit to the transferred encoder. Table R5 establishes image use; Table R3 evaluates FRAME’s localization.

Dear Reviewer cTkZ,

Thank you again for your feedback. We are following up on our rebuttal (Part 1, Part 2). To clarify the contribution of reciprocal finding–context supervision, we added a controlled comparison changing only the alignment targets, with the architecture, auxiliary objectives and training budget held fixed. We also report experiments with alternative summary generators and an image-mismatch control examining MRR’s use of visual information.

We would be grateful if you could review these additions and share your thoughts. Please let us know if any points would benefit from further clarification.

Best regards,
The authors

Reviewer_hVgqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper propose FRAME, a 2D radiograph vision-language pretrain framework. Its core idea is finding-context reciprocal alignment. One branch queries image features with a finding representation and aligns the resulting visual representation with its sentence context; the other queries the image with a sentence and aligns the output with a linked finding. Masked Report Reconstruction and image-conditioned Evidence-Focused Summary generation provide auxiliary supervision.

Critical strengths
  1. Evaluation spans five downstream task families, multiple annotation fractions, and cross-dataset settings.
  2. Component ablations support the empirical value of joint training.
  3. The paper provides details of finding–sentence associations, summary supervision, and downstream adaptation.
Critical weaknesses
  1. One key concern about finding-context reciprocal alignment is that even a successful alignment does not yet establish correct finding-lesion grounding. An unresolved alternative is therefore that both branches exploit global correlations, contextual associations, or non-lesion cues to achieve semantic matching without establishing correct local correspondence. Moreover, finding representations are contextualized using the report. In this case the finding representation is already similar to sentence representation.
  2. The auxiliary objectives have limited methodological novelty. Masked text recovery is a mature pre-training approach exemplified by BERT itself. Image-conditioned text generation (full report, report summary, specific content) is also a commonly used approach for vision-language pretraining.
Questions for authors
  1. Without training an additional head with spatial labels, how do both query branches localize findings on a representative test set? Does attention follow the appropriate lesion when the query changes within the same image?
  2. What happens when images are mismatched with fixed text, regional queries are replaced with global pooling, or lesions versus equal-area non-lesion regions are occluded? Do these controls rule out reliance on global correlations alone?
5 comments
AuthorsOpenReview ↗

Response to Reviewer hVgq (Part 1/2)

Thank you for distinguishing semantic matching from local correspondence. We address the common methodological concern first, then report the requested localization and visual-dependence controls.

Common concern W1 and W2 The contribution of reciprocal alignment

FRA makes the finding–sentence relation a supervision constraint on visual evidence. In Finding-Guided Context Reinforcement (FCR), a finding queries the image and the resulting visual response aligns with its containing sentence. In Context-Anchored Finding Alignment (CFA), that sentence queries the image and its response aligns separately with each linked finding. Contextualization enriches the query but does not determine the target used to train its visual response. FRA assigns that target at the linked textual level, without assuming independent text features.

Eq. (1) defines exact occurrence–sentence positives. Against Table R1’s first row, FRA changes only the alignment targets—from the querying text to the linked text at the other level—while retaining both query branches, MRR/EFS, training budget and downstream settings.

All variants in Table R1 are independently pretrained with matched budgets, MRR/EFS supervision and downstream settings. Scores are percentages. IU F1 denotes CheXbert-14 micro-F1. VQA-RAD closed accuracy in this table is the mean of five runs.

Table R1 Controlled comparison of alignment objectives

AlignmentCheXpert 1% AUCRSNA 1% mAPVQA-RAD closed accuracyIU X-Ray F1
Align to the querying text88.618.368.448.3
Global-pooling alignment87.412.366.143.2
FRA90.526.171.358.3

The querying-text control already contains both branches and auxiliary objectives. FRA’s gains of 1.9 AUC, 7.8 mAP, 2.9 VQA accuracy and 10.0 IU F1 points therefore support the cross-level supervision design. The grounding evaluations below separately test local correspondence.

MRR/EFS provide supporting reconstruction and generation objectives. Table R2 establishes their complementarity with FRA, while Table R3 tests localization directly.

Table R2 reproduces selected results from original Table 4; MRR includes local and global visual conditioning. Scores are percentages.

Table R2 Component ablations reproduced from original Table 4

Pretraining objectivesCheXpert 1% AUCRSNA 1% mAPIU X-Ray F1
MRR + EFS without FRA88.49.854.6
CFA + MRR + EFS89.911.156.8
FCR + MRR + EFS90.119.557.1
FRA + EFS without MRR89.115.255.2
FRA + MRR without EFS88.225.354.2
FRAME with all objectives90.526.158.3
AuthorsOpenReview ↗

Response to Reviewer hVgq (Part 2/2)

Q1 Localization without an additional spatial head

Table R3 uses all 880 images and 984 image–finding queries from the NIH ChestX-ray14 bounding-box cohort (eight findings), with frozen parameters and no spatially supervised adaptation. FCR uses finding queries; CFA uses “There is {finding}.” FRAME and its one-direction variants average all 12 attention heads. All methods use the same annotated image–finding pairs and scoring rules. Both metrics are query micro-averages: pointing game (PG) tests whether the peak patch center lies in a target box; mask IoU compares the min–max-normalized heatmap thresholded at 0.5 with GT rectangular masks.

Table R3 Grounding without spatially supervised adaptation

Model and branchPG (%)Mask IoU (%)
BioViL-T35.7714.15
MaCo25.0015.39
ECAMP28.8614.35
CFA-only model / CFA27.5414.73
FCR-only model / FCR22.3615.26
FRAME / CFA43.2916.85
FRAME / FCR38.7217.99

CFA’s finding-level target may favor finding-discriminative evidence, consistent with its higher PG (43.29 vs. 38.72); FCR aligns to the containing sentence and has higher mask IoU (17.99 vs. 16.85). This interpretation is not isolated from differences in query form. The controlled result is that reciprocal training improves PG/mask IoU over the respective one-direction models by 15.75/2.12 points for CFA and 16.36/2.73 for FCR, with both branches outperforming all three baselines.

We hold the image fixed and vary the query across annotated findings in 93 multi-finding images (197 queries). Both branches show higher target-box than other-box hit rates, providing aggregate evidence of query-conditioned localization within the same images. Rates are query micro-averages. Other-box hit means that the peak falls inside any annotated box for a different finding in the same image.

Table R4 Query changes within the same image

BranchTarget-box hit (%)Other-box hit (%)
FCR36.018.3
CFA45.722.3

Q2 Image mismatch and spatial controls

Table R5 evaluates frozen CFA on the same 880-image NIH cohort, using “There is {finding}.” and visual-response/finding-feature cosine scores. Replacement images are from other patients; view-matched controls share AP/PA and lack the target metadata label. Uniform weights (1/196) retain the value/output projections. AUROC uses image-level positive/negative metadata labels over all 880 images, macro-averaged across eight findings. Occlusion replaces the annotated box with the training-set grayscale mean and compares its score drop with the mean over five equal-sized outside regions; 896 queries qualify, with 88 excluded for insufficient control regions.

Table R5 Inference controls on frozen CFA

TestMetricReference conditionControl condition
Image replacementMean cosine similarity0.90 (paired)0.67 (random); 0.75 (view-matched)
Uniform weightsMacro AUROC (%)74.53 (learned)54.64 (uniform)
Regional occlusionMean cosine-score drop0.0025 (annotated box)0.0005 (outside box)

With text fixed, replacing the image lowers similarity; uniform weights reduce AUROC by 19.89 points. Annotated-box occlusion has the larger effect in 70.0% of eligible queries. These controls support visual dependence and preferential regional sensitivity, complementing the retrained alternatives in Table R1.

Uniform weighting can introduce inference-time distribution shift; Table R1 therefore evaluates a separately pretrained global-pooling alternative. Table R3 measures localization, while Table R5 measures preferential sensitivity to annotated regions. Together, these results support local visual use without assuming that global correlations play no role. Eq. (2) has no direct query residual.

Both text features use sentence-wise encoding; Sec. 3.2’s full-report notation is imprecise.

Dear Reviewer hVgq,

Thank you again for your feedback. We are following up on our rebuttal (Part 1, Part 2). We added grounding and query-switching evaluations on the frozen model, without spatially supervised adaptation, alongside image-replacement and regional-occlusion controls. These directly examine local correspondence and visual dependence. A controlled training comparison further isolates the contribution of reciprocal finding–context supervision.

We would appreciate your assessment of whether these results address your main concerns, and would be happy to clarify any remaining points during the discussion.

Best regards,
The authors

Reviewer_uuDqOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work proposes FRAME, a radiograph vision–language pre-training framework with three modules: (1) Finding–context Reciprocal Alignment (FRA), incorporating token-level findings and sentence-level context; (2) Masked Report Reconstruction (MRR), reconstructing masked report tokens from local and global visual features; (3) Evidence-Focused Summary (EFS), which trains the image encoder to preserve clinically salient evidence in a teacher-forcing way.

FRAME is pre-trained on MIMIC-CXR and evaluated on classification, segmentation, detection, VQA, and report generation under 1%, 10%, and 100% label settings, plus cross-dataset transfer to CheXpert and VinDr-CXR. It reports the best results among Med-VLP baselines in most settings, with the largest gains under limited labels.

Critical strengths

Reasonable intuition. The hierarchical, mismatched structure in radiology reports is real. Using the finding–sentence structure to build cross-level positive pairs is a clean way to operationalize it.

Careful loss formulation. The cardinality-normalized aggregation for negatives, exclusion of unlinked within-study pairs, and within-study averaging are well motivated and clearly specified (Eqs. 3–6).

Broad evaluation. Five task families, three label fractions and cross-dataset setting give a wide view of the FRAME framework.

Critical weaknesses

W1. Overly broad finding tokens. Several tokens in the finding vocabulary are too broad. Some are anatomical descriptors (silhouette, contour, vasculature), and others are non-specific terms (abnormality, opacity, thickening). These tokens may frequently occur in statements, but do not identify a meaningful focal clinical concept. They may therefore produce "noisy" positive pairs. Also, semantically equivalent sentences from other studies may become false negatives.

These spurious positives and negatives may introduce substantial training noise. The authors should provide:

  1. Occurrence statistics of each token in the finding vocabulary at the token, sentence, and study levels.
  2. A clearer justification for the selection of each token.
  3. Supplementary experiments, if necessary.
  4. The effective number of negatives per step. Please clarify whether negatives are gathered across GPUs or gradient-accumulation steps.

W2. Finding and context features are not independent. Finding features are contextualized CXR-BERT token features computed over the full report, so they already encode sentence-level context. Context features are mean-pooled over the same tokens, including the finding token itself. The two anchors are therefore highly correlated, which blurs the distinction between the “concept-selective” and “context-rich” views that motivates FRA and weakens the persuasiveness of the “reciprocal” claim.

W3. Limited comparison and technical novelty. Structured use of report text has been explored in ECAMP and ASG, but the Related Work section does not clearly discuss how FRAME differs from them. Sentence-query cross-attention over image patches has been used in RadZero (with cosine-similarity alignment) and CARZero [1] (with a learned similarity), both closely resembling CFA. The Related Work section does not discuss these designs. MRR and EFS offer limited technical novelty, largely following MRM, ECAMP, and BLIP-2. A more detailed discussion of these related works would help clarify the specific contributions of FRAME.

W4. VQA evaluation may not be fair. During pre-training, FRAME’s visual encoder is trained with Qwen3-4B through EFS, and the downstream VQA task uses the same Qwen3-4B as the decoder. Thus the performance advantage may conflate the encoder's inherent representation quality with the degree of alignment between the model and the decoder. The same concern applies to report generation.

Minor weaknesses:

Term inconsistency. In line 184, the vocabulary is defined as “abnormality vocabulary”, whereas Appendix A.1 and Table A.1 call it “finding vocabulary”. Since the vocabulary also includes non-abnormal terms, the authors may use the later one throughout.

Undefined abbreviations. In Sec. 3.2, The abbreviations FCR and CFA are used without being explicitly introduced; the paragraph headings give the full names, but the abbreviations are first defined only in Sec. 4.4 and Appendix.

[1] Lai, H., Yao, Q., Jiang, Z., Wang, R., He, Z., Tao, X., & Zhou, S. K. (2024, June). Carzero: Cross-attention alignment for radiology zero-shot classification. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 11137-11146). IEEE.

Questions for authors
  1. See W1.-W4.

  2. In Eq. (2), does MHAb\mathrm{MHA}_b include a residual connection from the query? If so, ui,kFu^{F}_{i,k} directly contains fi,kf_{i,k}, which is highly correlated with its sentence anchor ci,jc_{i,j}, so the alignment loss could be reduced without using the image.

  3. Please clarify whether ci,jc_{i,j} is pooled from the full-report encoding TiT_i (Sec. 3.2) or from encoding the sentence separately, as Algorithm A.1 (EncodeSentence) suggests.

6 comments
AuthorsOpenReview ↗

Response to Reviewer uuDq (Part 1/3)

Thank you for the detailed questions. We first address the shared concern about novelty and correlated textual views, then the vocabulary, visual dependence and decoder fairness.

Common concern W2 and W3 What is new and why it matters

FRA makes the finding–sentence relation a supervision constraint on visual evidence. In Finding-Guided Context Reinforcement (FCR), a finding queries the image and the resulting visual response aligns with its containing sentence. In Context-Anchored Finding Alignment (CFA), that sentence queries the image and its response aligns separately with each linked finding. Contextualization enriches the query but does not determine the target used to train its visual response. FRA assigns that target at the linked textual level, without assuming independent text features.

ECAMP uses entity-centered context distillation and reconstruction, and ASG uses anatomical region–sentence alignment and image-tag supervision. RadZero compares a sentence-conditioned visual response with that sentence; CARZero learns cross-attention similarity in both modality directions. FRA exchanges the conditioning and target roles of two linked textual levels through visual evidence. The proposed element is this cross-level supervision relation, implemented with standard cross-attention.

Eq. (1) defines exact occurrence–sentence positives. Against Table R1’s first row, FRA changes only the alignment targets—from the querying text to the linked text at the other level—while retaining both query branches, MRR/EFS, training budget and downstream settings.

All variants in Table R1 are independently pretrained with matched budgets, MRR/EFS supervision and downstream settings. Scores are percentages. IU F1 denotes CheXbert-14 micro-F1. VQA-RAD closed accuracy in this table is the mean of five runs.

Table R1 Controlled comparison of alignment objectives

AlignmentCheXpert 1% AUCRSNA 1% mAPVQA-RAD closed accuracyIU X-Ray F1
Align to the querying text88.618.368.448.3
Global-pooling alignment87.412.366.143.2
FRA90.526.171.358.3

FRA gains 1.9 AUC, 7.8 mAP, 2.9 VQA accuracy and 10.0 IU F1 points. Both correlated queries and auxiliary objectives already exist in the control; these gains support the value of cross-level supervision beyond fine-grained querying alone.

MRR and EFS support detail preservation and salient-evidence prediction in the shared encoder. With both fixed, FRA raises detection mAP from 9.8 to 26.1; joint FCR + CFA outperforms either direction alone (Table R2). The contribution is the validated alignment design, supported by complementary reconstruction and generation objectives.

Table R2 reproduces selected results from original Table 4; MRR includes local and global visual conditioning. Scores are percentages.

Table R2 Component ablations reproduced from original Table 4

Pretraining objectivesCheXpert 1% AUCRSNA 1% mAPIU X-Ray F1
MRR + EFS without FRA88.49.854.6
CFA + MRR + EFS89.911.156.8
FCR + MRR + EFS90.119.557.1
FRA + EFS without MRR89.115.255.2
FRA + MRR without EFS88.225.354.2
FRAME with all objectives90.526.158.3
AuthorsOpenReview ↗

Response to Reviewer uuDq (Part 2/3)

W1 Contextual anchors and negative candidates

Appendix A.1 defines 39 contextualized radiographic anchors selected for thoracic relevance and deterministic single-WordPiece matching. Opacity and thickening describe observable abnormalities without fixing etiology; generic terms signal unnamed abnormalities, while an anatomical term obtains its clinical meaning from its sentence, e.g., “enlarged cardiomediastinal silhouette.” The groups in Table R3 describe these roles and do not change training rules.

The accompanying statistical reply provides all 39 per-anchor occurrence, sentence and study counts before and after filtering. Across 216,141 unique studies, raw report coverage is 96.339%; positive/possible filtering retains 465,666 relations across 65.721% of studies. For example, 5,446 of 76,183 silhouette occurrences are retained (7.15%). These counts quantify filter selectivity; the localization tests below assess the resulting model.

Table R3 separately summarizes the 360,091 image-expanded rows, reproducing Fig. A.1. These are occurrence/edge counts, not independent studies or loss weights. The per-anchor table uses unique studies and the loader’s 256-token sequence for matching; raw counts precede polarity filtering.

Table R3 Vocabulary composition and retained relations

GroupTermsRaw occurrencesRetained edgesShare
Disease and imaging abnormality301,418,824684,01294.008%
Generic abnormality269,1297,0230.965%
Anatomical and appearance7398,01036,5775.027%
Total391,885,963727,612100.000%

Negatives are local to each 64-row GPU batch, grouped by study ID and excluding the query study: at most 63 other valid studies, without cross-GPU or accumulation-step gathering. Unlinked within-study pairs are excluded. Cardinality normalization controls candidate-set size; semantic false negatives across studies remain possible.

AuthorsOpenReview ↗

Response to Reviewer uuDq (Part 3/3)

Q2 Query residual and visual dependence

No query residual is used in our implementation of Eq. (2):

Gb(q,Vi)=MHA⁡b(gt(q),gv(Vi),gv(Vi)),b∈{F,C}.G_b(q,V_i)=\operatorname{MHA}_b(g_t(q),g_v(V_i),g_v(V_i)),\quad b\in\lbrace F,C\rbrace.

The query controls attention weights over visual values but is not added back to the aligned output. This excludes the direct query-residual shortcut; Table R4 and Table R5 separately test localization and visual dependence.

Table R4 uses all 880 images and 984 image–finding queries from the NIH ChestX-ray14 bounding-box cohort (eight findings), with frozen parameters and no spatially supervised adaptation. FCR uses finding queries; CFA uses “There is {finding}.” FRAME and its one-direction variants average all 12 attention heads. All methods use the same annotated image–finding pairs and scoring rules. Both metrics are query micro-averages: pointing game (PG) tests whether the peak patch center lies in a target box; mask IoU compares the min–max-normalized heatmap thresholded at 0.5 with GT rectangular masks.

Table R4 Grounding without spatially supervised adaptation

Model and branchPG (%)Mask IoU (%)
BioViL-T35.7714.15
MaCo25.0015.39
ECAMP28.8614.35
CFA-only model / CFA27.5414.73
FCR-only model / FCR22.3615.26
FRAME / CFA43.2916.85
FRAME / FCR38.7217.99

CFA’s finding-level target may favor finding-discriminative evidence, consistent with its higher PG (43.29 vs. 38.72); FCR aligns to the containing sentence and has higher mask IoU (17.99 vs. 16.85). This interpretation is not isolated from differences in query form. The controlled result is that reciprocal training improves PG/mask IoU over the respective one-direction models by 15.75/2.12 points for CFA and 16.36/2.73 for FCR, with both branches outperforming all three baselines.

Table R5 evaluates frozen CFA on the same 880-image NIH cohort, using “There is {finding}.” and visual-response/finding-feature cosine scores. Replacement images are from other patients; view-matched controls share AP/PA and lack the target metadata label. Uniform weights (1/196) retain the value/output projections. AUROC uses image-level positive/negative metadata labels over all 880 images, macro-averaged across eight findings. Occlusion replaces the annotated box with the training-set grayscale mean and compares its score drop with the mean over five equal-sized outside regions; 896 queries qualify, with 88 excluded for insufficient control regions.

Table R5 Inference controls on frozen CFA

TestMetricReference conditionControl condition
Image replacementMean cosine similarity0.90 (paired)0.67 (random); 0.75 (view-matched)
Uniform weightsMacro AUROC (%)74.53 (learned)54.64 (uniform)
Regional occlusionMean cosine-score drop0.0025 (annotated box)0.0005 (outside box)

With text fixed, replacing the image lowers similarity; uniform weights reduce AUROC by 19.89 points. Annotated-box occlusion has the larger effect in 70.0% of eligible queries. These controls support visual dependence and preferential regional sensitivity, complementing the retrained alternatives in Table R1.

W4 and Q3 Decoder transfer and implementation clarifications

For downstream evaluation, we replace Qwen3-4B, the decoder used during EFS pretraining, with Microsoft Phi-3.5-mini-instruct (Abdin et al., 2024). We reinitialize both Q-Former and projector and apply identical adaptation to all encoders; vision encoder and decoder remain frozen.

Scores in Table R6 are percentages. MIMIC-Ext retains pretraining overlap; external VQA-RAD and IU X-Ray provide the main decoder-transfer evidence. VQA-RAD closed accuracy in this table is the mean of five runs.

Table R6 Transfer with the Phi decoder

MethodVQA-RAD closed accuracyIU BLEU-4IU CheXbert F1MIMIC-Ext accuracy
MaCo62.715.455.777.1
RadZero64.217.753.279.4
FRAME70.519.657.880.6

FRAME retains gains of 6.3 VQA accuracy, 1.9 IU BLEU-4 and 2.1 IU F1, supporting transfer beyond the EFS decoder. Reports are tokenized and truncated before sentence parsing; finding and context features are then encoded sentence-wise. Sec. 3.2’s full-report encoding notation is imprecise. FCR/CFA and “finding vocabulary” are clarified above.

Reference. Abdin et al. (2024). Phi-3 Technical Report. arXiv:2404.14219v4.

Reviewer_CJvGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FRAME, a medical vision–language pre-training framework for chest radiographs that aims to strengthen the semantic connection between local clinical findings and their sentence-level contexts. The authors introduce Finding–Context Reciprocal Alignment (FRA), which bidirectionally aligns finding-conditioned visual representations with sentence contexts and context-conditioned visual representations with corresponding findings. Two auxiliary tasks, Masked Report Reconstruction (MRR) and Evidence-Focused Summary (EFS), further enhance the learning of fine-grained diagnostic information and clinically salient semantics. Experiments across disease classification, segmentation, detection, medical visual question answering, and radiology report generation demonstrate competitive performance, particularly in low-label settings. Ablation studies further validate the effectiveness of reciprocal alignment and the complementary auxiliary objectives.

Critical strengths
  1. Well-Motivated Reciprocal Alignment with Effective Experimental Support. Rather than simply increasing the granularity of vision–language alignment, the paper explicitly models the relationship between clinical findings and their sentence-level contexts through two complementary directions: Finding-Guided Context Reinforcement (FCR) and Context-Anchored Finding Alignment (CFA). The ablation results in Table 4 support the necessity of reciprocal alignment beyond simply adding another alignment branch.
  2. Consistent Performance Improvements Across Diverse Downstream Tasks. The paper provides extensive evaluations on disease classification, segmentation, detection, medical visual question answering, and radiology report generation, achieving superior performance over existing medical vision–language pre-training methods on most benchmarks, particularly in low-label settings. These results demonstrate that the learned representations support not only global disease recognition but also spatial understanding and cross-dataset transfer.
Critical weaknesses
  1. Insufficient Direct Validation of the Core Reciprocal Alignment Mechanism. FRAME aims to strengthen the semantic correspondence between clinical findings, visual regions, and sentence-level contexts. However, its effectiveness is primarily demonstrated through downstream performance improvements. Although Table 4 shows that reciprocal alignment outperforms unidirectional variants, quantitative evaluations of visual grounding or cross-modal correspondence remain limited. Moreover, the attention visualizations mainly present selected successful cases, making it difficult to determine whether the proposed mechanism genuinely improves fine-grained clinical visual grounding.
  2. Potential Concerns About Additional Supervision and Experimental Fairness. The paper uses GPT-4o to generate clinical summaries as additional supervision for EFS, but the contribution of this extra supervision relative to the core alignment mechanism is not fully isolated. Furthermore, FRAME is pre-trained on the full MIMIC-CXR corpus and evaluated on report generation using the same dataset, without clearly establishing whether the pre-training and downstream test sets are strictly separated. These concerns may affect the reliability and fairness of the experimental comparisons, particularly the attribution of improvements in report generation.
Questions for authors
  1. Effectiveness of Reciprocal Alignment. Could the authors provide quantitative visual grounding evaluations, such as lesion localization accuracy or attention overlap with ground-truth regions, to further validate the effectiveness of the proposed reciprocal alignment mechanism?
  2. Contribution and Fairness of Additional Summary Supervision. The paper uses GPT-4o-generated clinical summaries as supervision for EFS. Although the ablation studies demonstrate its effectiveness, to what extent do the improvements stem from this additional high-quality supervision rather than the core alignment mechanism? Could the authors clarify its implications for fair comparisons and provide evaluations under matched supervision settings?
  3. Separation Between Pre-training and Downstream Test Data. FRAME is pre-trained on the full MIMIC-CXR corpus and evaluated on report generation using the same dataset. Could the authors clarify whether there is any image-, patient-, or report-level overlap between pre-training and downstream test sets? If overlap exists, how do the authors rule out its influence on report generation performance?
2 comments
AuthorsOpenReview ↗

Response to Reviewer CJvG (Part 1/2)

Thank you for recognizing FRA’s motivation and empirical value. We address direct validation, supervision attribution and pretraining overlap with the following results.

Q1 Direct validation of reciprocal alignment

Table R1 uses all 880 images and 984 image–finding queries from the NIH ChestX-ray14 bounding-box cohort (eight findings), with frozen parameters and no spatially supervised adaptation. FCR uses finding queries; CFA uses “There is {finding}.” FRAME and its one-direction variants average all 12 attention heads. All methods use the same annotated image–finding pairs and scoring rules. Both metrics are query micro-averages: pointing game (PG) tests whether the peak patch center lies in a target box; mask IoU compares the min–max-normalized heatmap thresholded at 0.5 with GT rectangular masks.

Table R1 Grounding without spatially supervised adaptation

Model and branchPG (%)Mask IoU (%)
BioViL-T35.7714.15
MaCo25.0015.39
ECAMP28.8614.35
CFA-only model / CFA27.5414.73
FCR-only model / FCR22.3615.26
FRAME / CFA43.2916.85
FRAME / FCR38.7217.99

CFA’s finding-level target may favor finding-discriminative evidence, consistent with its higher PG (43.29 vs. 38.72); FCR aligns to the containing sentence and has higher mask IoU (17.99 vs. 16.85). This interpretation is not isolated from differences in query form. The controlled result is that reciprocal training improves PG/mask IoU over the respective one-direction models by 15.75/2.12 points for CFA and 16.36/2.73 for FCR, with both branches outperforming all three baselines.

Table R2 evaluates frozen CFA on the same 880-image NIH cohort, using “There is {finding}.” and visual-response/finding-feature cosine scores. Replacement images are from other patients; view-matched controls share AP/PA and lack the target metadata label. Uniform weights (1/196) retain the value/output projections. AUROC uses image-level positive/negative metadata labels over all 880 images, macro-averaged across eight findings. Occlusion replaces the annotated box with the training-set grayscale mean and compares its score drop with the mean over five equal-sized outside regions; 896 queries qualify, with 88 excluded for insufficient control regions.

Table R2 Inference controls on frozen CFA

TestMetricReference conditionControl condition
Image replacementMean cosine similarity0.90 (paired)0.67 (random); 0.75 (view-matched)
Uniform weightsMacro AUROC (%)74.53 (learned)54.64 (uniform)
Regional occlusionMean cosine-score drop0.0025 (annotated box)0.0005 (outside box)

With text fixed, replacing the image lowers similarity; uniform weights reduce AUROC by 19.89 points. Annotated-box occlusion has the larger effect in 70.0% of eligible queries. These controls support visual dependence and preferential regional sensitivity, complementing the retrained alternatives in Table R4.

AuthorsOpenReview ↗

Response to Reviewer CJvG (Part 2/2)

Q2 Separating reciprocal alignment from summary supervision

Holding MRR and EFS fixed, adding FRA improves RSNA 1% mAP from 9.8 to 26.1. FRA + MRR also reaches 25.3 without EFS (Table R3), showing that the detection benefit cannot be attributed simply to GPT-generated summaries.

Table R3 reproduces selected results from original Table 4; MRR includes local and global visual conditioning. Scores are percentages. IU F1 denotes CheXbert-14 micro-F1.

Table R3 Component ablations reproduced from original Table 4

Pretraining objectivesCheXpert 1% AUCRSNA 1% mAPIU X-Ray F1
MRR + EFS without FRA88.49.854.6
CFA + MRR + EFS89.911.156.8
FCR + MRR + EFS90.119.557.1
FRA + EFS without MRR89.115.255.2
FRA + MRR without EFS88.225.354.2
FRAME with all objectives90.526.158.3

Against the first row of Table R4, FRA retains both query branches and the same auxiliary supervision, changing only whether each visual response aligns with its querying text or the linked text at the other level. Reciprocal targets improve all four metrics, including +7.8 mAP and +10.0 IU F1. The global-pooling row is a separate retrained alternative.

All variants in Table R4 are independently pretrained with matched budgets, MRR/EFS supervision and downstream settings. Scores are percentages. VQA-RAD closed accuracy in this table is the mean of five runs.

Table R4 Controlled comparison of alignment objectives

AlignmentCheXpert 1% AUCRSNA 1% mAPVQA-RAD closed accuracyIU X-Ray F1
Align to the querying text88.618.368.448.3
Global-pooling alignment87.412.366.143.2
FRA90.526.171.358.3

EFS supplies additional report-derived supervision. The matched-EFS comparisons in Table R3 and Table R4 isolate FRA’s contribution; Table R5 tests generator dependence by regenerating targets with two open models and retraining FRAME. All downstream scores stay within 0.5 points of the GPT-4o variant, without a consistent GPT-4o advantage.

Nemotron denotes Llama-3.1-Nemotron-70B-Instruct. Comparing normalized entity–attribute–certainty triples with GPT-4o summaries yields micro-F1 of 97.3% for Nemotron and 95.6% for Qwen2.5-7B. This measures agreement with GPT-4o summaries, not fidelity to the source reports, and complements the stable downstream results in Table R5. All table metrics are percentages. VQA-RAD closed accuracy in this table is the mean of five runs.

Table R5 Retraining with alternative summary generators

GeneratorCheXpert 1% AUCRSNA 1% mAPVQA-RAD closed accuracyIU X-Ray F1
GPT-4o90.526.171.358.3
Nemotron-70B90.725.671.257.8
Qwen2.5-7B-Instruct90.225.971.358.7

Q3 External report generation and data separation

Original pretraining included MIMIC validation/test studies, so those results cannot establish held-out report-generation generalization. We provide the CheXpert Plus evaluation below, alongside the original IU X-Ray results, as external evidence and do not rely on the overlapping MIMIC results for that claim.

We split the original CheXpert Plus training pool by patient into 70%/10%/20%, with 37,810 test reports and zero patient/study/image intersections between the three CheXpert Plus subsets. All baselines use the same frozen-encoder/decoder adaptation protocol as IU X-Ray; only task adapters are trained. Targets use available Findings and Impression sections, with report-body fallback.

We use the df_chexpert_plus_240401.csv report snapshot and CheXpert-v1.0-small images. CheXbert treats positive/uncertain labels as present; RadGraph-XL partial F1 averages reports equally. Scores are percentages.

Table R6 Report generation on CheXpert Plus

MethodBLEU-4ROUGE-LCheXbert-14 micro-F1RadGraph-XL partial F1
REFERS11.0631.5048.7821.46
MGCA10.6431.5850.3722.45
MRM11.4132.0352.5423.43
MaCo11.3631.8950.4624.79
ECAMP12.1330.4651.3623.65
FRAME13.8733.0155.7826.17

FRAME leads all displayed baselines on all four metrics, including +3.24 CheXbert F1 and +1.38 RadGraph F1 over the strongest baseline in each column. Table R6 supports transfer from MIMIC pretraining to held-out CheXpert Plus patients under a shared downstream protocol; the overlapping MIMIC test results are excluded from this generalization claim.

Comments on the paper

1 comment
Area_Chair_TmDbOpenReview ↗

Dear Reviewer CJvG, Reviewer hVgq, Reviewer cTkZ, could you please indicate whether the authors' rebuttal addressed all of your suggestions or concerns and updated your final score? Thank you!

Dear ALL,

Dear Reviewer uuDq, thanks for your timely feedback. Dear Reviewer CJvG, Reviewer hVgq, Reviewer cTkZ, could you please indicate whether the authors' rebuttal addressed all of your suggestions or concerns and updated your final score? Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.