- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper introduces GazeBreast-FM, a unified foundation model for mammography, breast ultrasound, and breast MRI, pretrained on 302,047 images from 23 datasets. It combines masked image reconstruction and image–report alignment, using simulated radiologist-like gaze trajectories to guide diagnostic evidence allocation, a coverage-aware weighting mechanism, and a report-derived clinical graph for structured semantic supervision. The authors report improvements over general and modality-specific baselines across classification, lesion segmentation/localization, and clinical-attribute tasks.
Critical strengths
-
Unified representation learning across heterogeneous breast imaging modalities is valuable, given the annotation and model-development costs of separate systems.
-
Training on 23 datasets across three modalities is a meaningful step toward broader breast-imaging representations. The method connects masked reconstruction, image–report alignment, spatial attention, and structured clinical semantics in one framework. Consistent reported results: and ablations are present. The paper describes patient-level splitting, three downstream seeds, and dataset provenance.
Critical weaknesses
-
The framework combines established ingredients: MAE, CLIP-style alignment, attention priors, concept prediction, and graph-based supervision. The paper should better establish which component or interaction constitutes the main methodological advance.
-
The gaze prior is simulated, not directly measured. The claim that the method allocates evidence in a radiologist-like manner is not fully established by simulated gaze trajectories. The paper needs stronger validation against real radiologist eye-tracking data or expert-localized evidence.
-
The clinical graph and several semantic objectives derive from reports, and some reports are generated by a medical vision–language model when trustworthy official reports are unavailable. Errors or biases in these reports could propagate into pretraining; factual-consistency checks alone may not rule this out.
-
Evaluation spans three modalities, but results on a small set of downstream benchmarks do not by themselves establish robustness across institutions, scanners, populations, or clinical workflows. Independent external validation is especially important.
-
The model initializes from Mammo-FM and BiomedCLIP, and the pretraining corpus is large. It is important to distinguish gains from the proposed evidence-allocation mechanism from gains due to initialization, data scale, and additional training. Matched-compute and matched-data comparisons would strengthen the claims.
-
Classification and lesion-localization metrics are useful proxies, but there is no prospective evaluation or reader study establishing improved diagnostic decisions, calibration, or clinical utility.
-
A shared representation can benefit transfer but may also compromise modality-specific features. The paper acknowledges this limitation, yet the extent of the trade-off is not sufficiently characterized.
Questions for authors
-
How closely does the simulated gaze prior match real radiologist attention patterns, and does the method retain its advantage when trained with measured gaze or expert lesion annotations?
-
How much improvement comes from gaze guidance and evidence allocation versus larger-scale pretraining, initialization, and the clinical graph? Can these effects be isolated under matched-data and matched-compute settings?
-
What proportion of pretraining reports are generated, how are their errors audited, and how sensitive are results to excluding generated reports?
-
Are the downstream test sets institutionally independent of the pretraining sources, and can the model be evaluated on additional external cohorts?
-
How does performance vary by modality, lesion type, density, scanner/acquisition protocol, and demographic subgroup?
-
Does the model improve radiologist performance or diagnostic decisions compared with existing systems, rather than only improving benchmark metrics?
6 comments
Response to Q2
We thank the reviewer for the suggestion to distinguish the effects of continued pretraining from those of individual components. Under the same visual initialization and matched pretraining conditions, we compare image-only continued pretraining with random-mask feature prediction, complete GazeBreast-FM joint pretraining, and configurations removing Gaze Guidance, Clinical Graph, Evidence Allocation, or Salient-Evidence Preservation. All configurations are evaluated on the same four downstream tasks.
| Pretraining configuration | MG AUROC ↑ | MRI AUROC ↑ | US Dice ↑ | Attribute Macro-AUROC ↑ |
|---|---|---|---|---|
| GazeBreast-FM (full joint pretraining) | 0.8467 ± 0.0028 | 0.8674 ± 0.0056 | 0.8328 ± 0.0029 | 0.8876 ± 0.0016 |
| Image-only Random-mask Feature Prediction | 0.8091 ± 0.0045 | 0.8290 ± 0.0067 | 0.7994 ± 0.0048 | 0.8360 ± 0.0037 |
| w/o Gaze Guidance | 0.8238 ± 0.0042 | 0.8427 ± 0.0065 | 0.7986 ± 0.0051 | 0.8604 ± 0.0031 |
| w/o Clinical Graph | 0.8380 ± 0.0034 | 0.8591 ± 0.0060 | 0.8220 ± 0.0038 | 0.8653 ± 0.0027 |
| w/o Evidence Allocation | 0.8356 ± 0.0035 | 0.8558 ± 0.0060 | 0.8149 ± 0.0042 | 0.8752 ± 0.0025 |
| w/o Salient-Evidence Preservation | 0.8392 ± 0.0032 | 0.8601 ± 0.0059 | 0.8223 ± 0.0035 | 0.8780 ± 0.0022 |
Complete joint pretraining achieves the best performance on all four tasks, exceeding image-only continued pretraining. Removing Gaze Guidance, Clinical Graph, Evidence Allocation, or Salient-Evidence Preservation decreases downstream performance, supporting positive contributions from generated gaze guidance, structured clinical semantics, diagnostic evidence allocation, and preservation of salient diagnostic evidence.
Response to Q3
We thank the reviewer for asking about the proportion of generated reports, factual verification, and sensitivity to report-derived supervision. The pretraining inventory comprises 302,047 images from 23 datasets across MG, US, and MRI, corresponding to 62,840 case-level report units. Of these, 61,808 are model-generated and 1,032 originate from real data sources. The dataset-level breakdown is provided below.
| Mod. | Dataset | Images | Real source | Generated | Total report units |
|---|---|---|---|---|---|
| MG | EMBED | 100,217 | 0 | 35,034 | 35,034 |
| MG | RSNA | 36,185 | 0 | 10,955 | 10,955 |
| MG | Screening-DBT | 13,218 | 0 | 4,603 | 4,603 |
| MG | VinDr-Mammo | 11,558 | 0 | 4,780 | 4,780 |
| MG | CSAW-CC | 6,962 | 0 | 1,967 | 1,967 |
| MG | CBIS-DDSM | 2,254 | 0 | 1,458 | 1,458 |
| MG | CDD-CESM | 1,354 | 319 | 0 | 319 |
| MG | DMID | 396 | 396 | 0 | 396 |
| MG | MIAS | 304 | 0 | 304 | 304 |
| MG | INbreast | 293 | 0 | 293 | 293 |
| MG | BMCD | 234 | 0 | 163 | 163 |
| MG | CSAW-S | 174 | 0 | 112 | 112 |
| Subtotal MG | 12 datasets | 173,149 | 715 | 59,669 | 60,384 |
| US | Breast-USV | 64,739 | 0 | 418 | 418 |
| US | HiSBreast | 714 | 317 | 0 | 317 |
| US | BUS-UCLM | 644 | 0 | 644 | 644 |
| US | BUSI | 616 | 0 | 616 | 616 |
| US | BrEaST-Lesions-USG | 225 | 0 | 225 | 225 |
| US | OASBUD | 165 | 0 | 94 | 94 |
| Subtotal US | 6 datasets | 67,103 | 317 | 1,997 | 2,314 |
| MRI | Duke-Breast-Cancer-MRI | 58,711 | 0 | 72 | 72 |
| MRI | I-SPY1 | 3,014 | 0 | 18 | 18 |
| MRI | NACT | 39 | 0 | 32 | 32 |
| MRI | TCGA-BRCA-MRI | 25 | 0 | 18 | 18 |
| MRI | RIDER-Breast-MRI | 6 | 0 | 2 | 2 |
| Subtotal MRI | 5 datasets | 61,795 | 0 | 142 | 142 |
| TOTAL | 23 datasets | 302,047 | 1,032 | 61,808 | 62,840 |
For datasets without original clinical reports, MedGemma-27B-IT generated reports using source-dataset case-level clinical facts, including BI-RADS assessments, breast density, lesion descriptions, diagnoses, and pathology. All reports admitted to formal pretraining underwent case-level factual consistency checks; their diagnostic statements were consistent with the original clinical facts and pathology annotations.
When generated reports and their derived semantic supervision are removed, MG/MRI AUROC decreases from 0.8467/0.8674 to 0.8181/0.8369, US Dice decreases from 0.8328 to 0.8117, and attribute Macro-AUROC decreases from 0.8876 to 0.8486. These findings support the contribution of report-derived semantic supervision to tri-modal representation learning.
Response to Q1
We thank the reviewer for this question. The original appendix used AUC-Judd and NSS to evaluate spatial agreement between generated attention and expert fixation locations, and LD and DTW to assess scanpath differences. We additionally include Sequence Score (SS) and Semantic Sequence Score (Sem SS) to evaluate similarity in fixation order and attended semantic-region order, respectively. On held-out expert-gaze samples from MG, US, and MRI, these metrics jointly assess spatial distributions, scanpaths, and visual-search order.
After fine-tuning with expert eye-tracking data, DiffEye generates gaze trajectories that exhibit high similarity to radiologists’ eye movements across MG, US, and MRI. The spatial-attention, scanpath, and fixation-sequence measures indicate that the generated gaze captures salient aspects of expert fixation patterns and visual-search behavior.
Collecting real expert eye-movement data for all 302,047 pretraining images would incur substantial human annotation costs. Modality-adapted DiffEye enables scalable generation of gaze priors with high agreement with expert gaze. Although generated gaze still falls short of real expert gaze annotations, the ablation results demonstrate that gaze guidance improves MG/MRI classification and US segmentation performance, supporting its contribution to foundation-model representation learning.