- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes frame, a visual language pre training framework for chest X-ray images. The author believes that although the existing methods will use the local disease concept and sentence level semantic information in radiology reports, they usually lack the explicit modeling of the relationship between them. In order to solve this problem, the author proposes finding – context reciprocal alignment (FRA), which uses the disease concept to guide the image feature to align with the sentence context, and uses the sentence context to guide the image feature to align with the corresponding disease concept through cross modal alignment in two directions. In addition, the author introduced masked report reconstruction (MRR) and evidence focused summary (EFS) to retain fine-grained clinical information and highlight important diagnostic semantics. The authors evaluated the tasks of classification, segmentation, target detection, medical visual question answering and report generation, and the results showed that frame achieved better performance than the comparison method on multiple datasets.
Critical strengths
- It is significant to study the problem. Now this paper models from the relationship between finding and context, and the overall motivation is relatively clear.
- The experiment is comprehensive and the overall performance is good. The authors evaluated five kinds of tasks: classification, segmentation, target detection, medical visual question answering and report generation, and considered different annotation data proportions and cross dataset migration scenarios. Frame has achieved good results in many tasks, especially in the case of less labeled data.
Critical weaknesses
- In this paper, the innovation is limited, and fine-grained alignment between local image regions and report words has been studied in a variety of methods.
- The author uses gpt-4o to generate brief abstracts from the original radiology report, and then uses these abstracts as the training objectives of the image generation task. However, the automatically generated summary may omit important exception information, or change the negative and uncertain expression in the original report. The statistical results in the appendix also show that the generated summary does not completely retain all the clinical concepts in the source report. Will it make the model learn the wrong representation to conduct supervised learning according to this?
- If we use a large language model which is more powerful than GPT, will it improve the performance of this method? However, this part of the experiment is not reflected in the article. It is suggested that the author add an ablation experiment using different large language models.
Questions for authors
Does masked report reconstruction really use image information. MRR reconstructs the original text through image information and part of the unobstructed report content. However, the article also said that the reconstruction loss affects all non padding tokens, not just the masked location, so whether some predictions may directly rely on existing text input. Even for occluded tokens, the model may recover through the language context, rather than making full use of the visual information in the image.
6 comments
Response to Reviewer cTkZ (Part 2/2)
W2 and W3 Summary fidelity and generator dependence
EFS uses offline GPT-4o summaries as auxiliary targets while MRR retains supervision from the original report. Appendix A.3 reports 58.5% source-concept retention, 99.4% target concepts present in the source and 89.3% shared-concept polarity agreement. Incomplete coverage and incorrect polarity are distinct concerns; these statistics do not establish error-free targets.
Table R2 measures utility despite imperfect targets: adding EFS to FRA + MRR improves CheXpert AUC from 88.2 to 90.5 and IU F1 from 54.2 to 58.3. Alternative summary generators yield downstream scores within 0.5 points of GPT-4o (Table R4), without a consistent GPT-4o advantage. This supports usefulness across the tested generators, not the fidelity of every summary.
Nemotron denotes Llama-3.1-Nemotron-70B-Instruct. Comparing normalized entity–attribute–certainty triples with GPT-4o summaries yields micro-F1 of 97.3% for Nemotron and 95.6% for Qwen2.5-7B. This measures agreement with GPT-4o summaries, not fidelity to the source reports, and complements the stable downstream results in Table R4. All table metrics are percentages. VQA-RAD closed accuracy in this table is the mean of five runs.
Table R4 Retraining with alternative summary generators
| Generator | CheXpert 1% AUC | RSNA 1% mAP | VQA-RAD closed accuracy | IU X-Ray F1 |
|---|---|---|---|---|
| GPT-4o | 90.5 | 26.1 | 71.3 | 58.3 |
| Nemotron-70B | 90.7 | 25.6 | 71.2 | 57.8 |
| Qwen2.5-7B-Instruct | 90.2 | 25.9 | 71.3 | 58.7 |
Question Does MRR use visual information
MRR trains on all non-padding positions; this diagnostic evaluates only masked positions. On 5,000 distinct patients in the patient-disjoint CheXpert Plus validation split, we hold reports, 75% masks and weights fixed and cyclically shift the images, verifying every replacement is cross-patient. Each patient supplies one report and one frontal image. Mismatches are not selected by disease label.
Table R5 MRR accuracy at masked positions
| Visual input | All masked tokens | Masked finding tokens |
|---|---|---|
| Paired image | 30.42% | 41.51% |
| Mismatched image | 16.38% | 15.74% |
With identical language context, wrong images reduce masked-finding accuracy by 25.77 points: MRR’s predictions are not determined by text alone. Separately, removing MRR lowers detection mAP from 26.1 to 15.2 and IU F1 from 58.3 to 55.2 (Table R2), demonstrating its benefit to the transferred encoder. Table R5 establishes image use; Table R3 evaluates FRAME’s localization.
Dear Reviewer cTkZ,
Thank you again for your feedback. We are following up on our rebuttal (Part 1, Part 2). To clarify the contribution of reciprocal finding–context supervision, we added a controlled comparison changing only the alignment targets, with the architecture, auxiliary objectives and training budget held fixed. We also report experiments with alternative summary generators and an image-mismatch control examining MRR’s use of visual information.
We would be grateful if you could review these additions and share your thoughts. Please let us know if any points would benefit from further clarification.
Best regards,
The authors
Response to Reviewer cTkZ (Part 1/2)
Thank you for recognizing the method’s soundness and empirical performance. We focus on the specific contribution beyond existing fine-grained alignment, then address summary supervision and MRR’s use of images.
Common concern W1 What is new and why it matters
FRA makes the finding–sentence relation a supervision constraint on visual evidence. In Finding-Guided Context Reinforcement (FCR), a finding queries the image and the resulting visual response aligns with its containing sentence. In Context-Anchored Finding Alignment (CFA), that sentence queries the image and its response aligns separately with each linked finding. Contextualization enriches the query but does not determine the target used to train its visual response. FRA assigns that target at the linked textual level, without assuming independent text features.
ECAMP uses entity-centered context distillation and reconstruction, and ASG uses anatomical region–sentence alignment and image-tag supervision. RadZero compares a sentence-conditioned visual response with that sentence; CARZero learns cross-attention similarity in both modality directions. FRA exchanges the conditioning and target roles of two linked textual levels through visual evidence. The proposed element is this cross-level supervision relation, implemented with standard cross-attention.
Eq. (1) defines exact occurrence–sentence positives. Against Table R1’s first row, FRA changes only the alignment targets—from the querying text to the linked text at the other level—while retaining both query branches, MRR/EFS, training budget and downstream settings.
All variants in Table R1 are independently pretrained with matched budgets, MRR/EFS supervision and downstream settings. Scores are percentages. IU F1 denotes CheXbert-14 micro-F1. VQA-RAD closed accuracy in this table is the mean of five runs.
Table R1 Controlled comparison of alignment objectives
FRA gains 1.9 AUC, 7.8 mAP, 2.9 VQA accuracy and 10.0 IU F1 points. Both correlated queries and auxiliary objectives already exist in the control; these gains support the value of cross-level supervision beyond fine-grained querying alone.
MRR and EFS are supporting objectives. Table R2 shows their complementarity with FRA; Table R3 directly tests whether the alignment benefit extends to localization.
Table R2 reproduces selected results from original Table 4; MRR includes local and global visual conditioning. Scores are percentages.
Table R2 Component ablations reproduced from original Table 4
Table R3 uses all 880 images and 984 image–finding queries from the NIH ChestX-ray14 bounding-box cohort (eight findings), with frozen parameters and no spatially supervised adaptation. FCR uses finding queries; CFA uses “There is {finding}.” FRAME and its one-direction variants average all 12 attention heads. All methods use the same annotated image–finding pairs and scoring rules. Both metrics are query micro-averages: pointing game (PG) tests whether the peak patch center lies in a target box; mask IoU compares the min–max-normalized heatmap thresholded at 0.5 with GT rectangular masks.
Table R3 Grounding without spatially supervised adaptation
CFA’s finding-level target may favor finding-discriminative evidence, consistent with its higher PG (43.29 vs. 38.72); FCR aligns to the containing sentence and has higher mask IoU (17.99 vs. 16.85). This interpretation is not isolated from differences in query form. The controlled result is that reciprocal training improves PG/mask IoU over the respective one-direction models by 15.75/2.12 points for CFA and 16.36/2.73 for FCR, with both branches outperforming all three baselines.