acceptodds
Under review as a conference paper at ICLR 2027

GAZEBREAST-FM: AN OMNI BREAST IMAGING FOUNDATION MODEL WITH GAZE-GUIDED DIAGNOSTIC EVIDENCE ALLOCATION

Abstract

Breast cancer diagnosis relies on complementary information from different breast imaging modalities, yet developing separate models for individual modalities and tasks requires substantial manual annotation and limits representation sharing across imaging domains. Foundation models can reduce this dependence by learning transferable representations through large-scale pre-training, but existing foundation models for breast imaging remain largely modality-specific and are not designed to learn jointly from heterogeneous breast imaging data within a unified framework. GazeBreast-FM is an Omni breast imaging foundation model pre-trained on 302,047 images from 23 datasets. The framework jointly optimizes MAE-style masked reconstruction and CLIP-style image–report alignment, while introducing a spatial attention prior generated from simulated radiologist-like gaze trajectories to allocate sparse diagnostic evidence between masked reconstruction and visible-region semantic learning. A salient-evidence preservation constraint prevents diagnostically relevant information from being completely masked, and diagnostic-evidence coverage is used to dynamically coordinate the two learning objectives. Diagnostic evidence that remains visible after masking is further used for clinical concept learning, while a report-derived Clinical Graph captures continuous clinical relationships across cases and provides structured semantic supervision. Across multiple downstream tasks and breast imaging modalities, GazeBreast-FM achieves state-of-the-art performance, demonstrating transferable representations that support global diagnosis, lesion-level spatial understanding, and fine-grained clinical semantics.https://github.com/aaaaaaa12344/GazeBreast-FM

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
GazeBreast-FM: An Omini Breast Imaging Foundation Model with Gaze-Guided Diagnostic Evidence Allocation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XMrUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces GazeBreast-FM, a unified foundation model for mammography, breast ultrasound, and breast MRI, pretrained on 302,047 images from 23 datasets. It combines masked image reconstruction and image–report alignment, using simulated radiologist-like gaze trajectories to guide diagnostic evidence allocation, a coverage-aware weighting mechanism, and a report-derived clinical graph for structured semantic supervision. The authors report improvements over general and modality-specific baselines across classification, lesion segmentation/localization, and clinical-attribute tasks.

Critical strengths
  1. Unified representation learning across heterogeneous breast imaging modalities is valuable, given the annotation and model-development costs of separate systems.

  2. Training on 23 datasets across three modalities is a meaningful step toward broader breast-imaging representations. The method connects masked reconstruction, image–report alignment, spatial attention, and structured clinical semantics in one framework. Consistent reported results: and ablations are present. The paper describes patient-level splitting, three downstream seeds, and dataset provenance.

Critical weaknesses
  1. The framework combines established ingredients: MAE, CLIP-style alignment, attention priors, concept prediction, and graph-based supervision. The paper should better establish which component or interaction constitutes the main methodological advance.

  2. The gaze prior is simulated, not directly measured. The claim that the method allocates evidence in a radiologist-like manner is not fully established by simulated gaze trajectories. The paper needs stronger validation against real radiologist eye-tracking data or expert-localized evidence.

  3. The clinical graph and several semantic objectives derive from reports, and some reports are generated by a medical vision–language model when trustworthy official reports are unavailable. Errors or biases in these reports could propagate into pretraining; factual-consistency checks alone may not rule this out.

  4. Evaluation spans three modalities, but results on a small set of downstream benchmarks do not by themselves establish robustness across institutions, scanners, populations, or clinical workflows. Independent external validation is especially important.

  5. The model initializes from Mammo-FM and BiomedCLIP, and the pretraining corpus is large. It is important to distinguish gains from the proposed evidence-allocation mechanism from gains due to initialization, data scale, and additional training. Matched-compute and matched-data comparisons would strengthen the claims.

  6. Classification and lesion-localization metrics are useful proxies, but there is no prospective evaluation or reader study establishing improved diagnostic decisions, calibration, or clinical utility.

  7. A shared representation can benefit transfer but may also compromise modality-specific features. The paper acknowledges this limitation, yet the extent of the trade-off is not sufficiently characterized.

Questions for authors
  1. How closely does the simulated gaze prior match real radiologist attention patterns, and does the method retain its advantage when trained with measured gaze or expert lesion annotations?

  2. How much improvement comes from gaze guidance and evidence allocation versus larger-scale pretraining, initialization, and the clinical graph? Can these effects be isolated under matched-data and matched-compute settings?

  3. What proportion of pretraining reports are generated, how are their errors audited, and how sensitive are results to excluding generated reports?

  4. Are the downstream test sets institutionally independent of the pretraining sources, and can the model be evaluated on additional external cohorts?

  5. How does performance vary by modality, lesion type, density, scanner/acquisition protocol, and demographic subgroup?

  6. Does the model improve radiologist performance or diagnostic decisions compared with existing systems, rather than only improving benchmark metrics?

6 comments
AuthorsOpenReview ↗

Response to Q1

We thank the reviewer for this question. The original appendix used AUC-Judd and NSS to evaluate spatial agreement between generated attention and expert fixation locations, and LD and DTW to assess scanpath differences. We additionally include Sequence Score (SS) and Semantic Sequence Score (Sem SS) to evaluate similarity in fixation order and attended semantic-region order, respectively. On held-out expert-gaze samples from MG, US, and MRI, these metrics jointly assess spatial distributions, scanpaths, and visual-search order.

ModalityDiffEyeLD ↓DTW ↓AUC-Judd ↑NSS ↑SS ↑Sem SS ↑
MGOriginal17.42.320.6940.980.3730.518
MGAdapted13.61.690.8362.050.4980.657
USOriginal18.82.610.6720.840.3550.495
USAdapted14.51.850.8121.790.4700.624
MRIOriginal18.12.490.6810.900.3620.506
MRIAdapted14.11.780.8211.890.4830.638

After fine-tuning with expert eye-tracking data, DiffEye generates gaze trajectories that exhibit high similarity to radiologists’ eye movements across MG, US, and MRI. The spatial-attention, scanpath, and fixation-sequence measures indicate that the generated gaze captures salient aspects of expert fixation patterns and visual-search behavior.

Collecting real expert eye-movement data for all 302,047 pretraining images would incur substantial human annotation costs. Modality-adapted DiffEye enables scalable generation of gaze priors with high agreement with expert gaze. Although generated gaze still falls short of real expert gaze annotations, the ablation results demonstrate that gaze guidance improves MG/MRI classification and US segmentation performance, supporting its contribution to foundation-model representation learning.

AuthorsOpenReview ↗

Response to Q2

We thank the reviewer for the suggestion to distinguish the effects of continued pretraining from those of individual components. Under the same visual initialization and matched pretraining conditions, we compare image-only continued pretraining with random-mask feature prediction, complete GazeBreast-FM joint pretraining, and configurations removing Gaze Guidance, Clinical Graph, Evidence Allocation, or Salient-Evidence Preservation. All configurations are evaluated on the same four downstream tasks.

Pretraining configurationMG AUROC ↑MRI AUROC ↑US Dice ↑Attribute Macro-AUROC ↑
GazeBreast-FM (full joint pretraining)0.8467 ± 0.00280.8674 ± 0.00560.8328 ± 0.00290.8876 ± 0.0016
Image-only Random-mask Feature Prediction0.8091 ± 0.00450.8290 ± 0.00670.7994 ± 0.00480.8360 ± 0.0037
w/o Gaze Guidance0.8238 ± 0.00420.8427 ± 0.00650.7986 ± 0.00510.8604 ± 0.0031
w/o Clinical Graph0.8380 ± 0.00340.8591 ± 0.00600.8220 ± 0.00380.8653 ± 0.0027
w/o Evidence Allocation0.8356 ± 0.00350.8558 ± 0.00600.8149 ± 0.00420.8752 ± 0.0025
w/o Salient-Evidence Preservation0.8392 ± 0.00320.8601 ± 0.00590.8223 ± 0.00350.8780 ± 0.0022

Complete joint pretraining achieves the best performance on all four tasks, exceeding image-only continued pretraining. Removing Gaze Guidance, Clinical Graph, Evidence Allocation, or Salient-Evidence Preservation decreases downstream performance, supporting positive contributions from generated gaze guidance, structured clinical semantics, diagnostic evidence allocation, and preservation of salient diagnostic evidence.

AuthorsOpenReview ↗

Response to Q3

We thank the reviewer for asking about the proportion of generated reports, factual verification, and sensitivity to report-derived supervision. The pretraining inventory comprises 302,047 images from 23 datasets across MG, US, and MRI, corresponding to 62,840 case-level report units. Of these, 61,808 are model-generated and 1,032 originate from real data sources. The dataset-level breakdown is provided below.

Mod.DatasetImagesReal sourceGeneratedTotal report units
MGEMBED100,217035,03435,034
MGRSNA36,185010,95510,955
MGScreening-DBT13,21804,6034,603
MGVinDr-Mammo11,55804,7804,780
MGCSAW-CC6,96201,9671,967
MGCBIS-DDSM2,25401,4581,458
MGCDD-CESM1,3543190319
MGDMID3963960396
MGMIAS3040304304
MGINbreast2930293293
MGBMCD2340163163
MGCSAW-S1740112112
Subtotal MG12 datasets173,14971559,66960,384
USBreast-USV64,7390418418
USHiSBreast7143170317
USBUS-UCLM6440644644
USBUSI6160616616
USBrEaST-Lesions-USG2250225225
USOASBUD16509494
Subtotal US6 datasets67,1033171,9972,314
MRIDuke-Breast-Cancer-MRI58,71107272
MRII-SPY13,01401818
MRINACT3903232
MRITCGA-BRCA-MRI2501818
MRIRIDER-Breast-MRI6022
Subtotal MRI5 datasets61,7950142142
TOTAL23 datasets302,0471,03261,80862,840

For datasets without original clinical reports, MedGemma-27B-IT generated reports using source-dataset case-level clinical facts, including BI-RADS assessments, breast density, lesion descriptions, diagnoses, and pathology. All reports admitted to formal pretraining underwent case-level factual consistency checks; their diagnostic statements were consistent with the original clinical facts and pathology annotations.

When generated reports and their derived semantic supervision are removed, MG/MRI AUROC decreases from 0.8467/0.8674 to 0.8181/0.8369, US Dice decreases from 0.8328 to 0.8117, and attribute Macro-AUROC decreases from 0.8876 to 0.8486. These findings support the contribution of report-derived semantic supervision to tri-modal representation learning.

Reviewer_xK41OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

GazeBreast-FM is a unified breast imaging foundation model pretrained on 302,047 images from 23 datasets spanning mammography, ultrasound, and MRI. It combines MAE-style masked reconstruction with CLIP-style image–report alignment and introduces gaze-guided diagnostic-evidence allocation to distribute salient image regions between reconstruction and semantic learning. Additional components include salient-evidence preservation, coverage-aware dynamic loss weighting, and a report-derived Clinical Graph for structured clinical-semantic supervision. Experiments cover seven downstream tasks, including disease classification, lesion segmentation and localization, image–text retrieval, clinical attribute prediction, label efficiency, and external generalization. The reported results outperform several general-purpose and modality-specific baselines, with ablation studies supporting the contributions of the proposed components. However, inconsistencies in reported retrieval metrics and questions about dataset independence warrant clarification before the empirical claims can be fully assessed.

Critical strengths
  1. The framework addresses the fragmentation of existing modality-specific models and evaluates transfer across mammography, ultrasound, and MRI. This broad scope is relevant to developing reusable medical imaging representations.
  2. Gaze-guided evidence allocation, salient-evidence preservation, and coverage-aware optimization provide a coherent mechanism for combining local visual reconstruction with image–report alignment. The ablation studies offer preliminary evidence that these components contribute to downstream performance.
Critical weaknesses
  1. Table 6 (page 17) contains multiple Recall@K sequences in which recall decreases as K increases. For example, the MRI text-to-image results for MedSigLIP-448 report Recall@1 = 0.3526 and Recall@5 = 0.3158. Such values are inconsistent with standard Recall@K definitions and require correction or an explicit explanation. Given the paper’s extensive performance claims, this raises concerns about the reliability of the reported evaluation.
  2. The paper claims that downstream benchmarks are separated from pretraining data, yet the pretraining inventory includes a BUSI dataset while downstream experiments use BUSI-WHU. The authors should establish whether these datasets overlap at the patient or image level. In addition, the gaze generator is adapted using only 150 expert-tracked images per modality and validated on 15 images per modality. Although the appendix reports several gaze-similarity metrics, the limited validation leaves uncertainty about the reliability of synthetic gaze priors across the full pretraining corpus and their contribution beyond alternative spatial priors.
Questions for authors
  1. Can the authors verify and correct the non-monotonic Recall@K values in Table 6? Were the metrics computed using standard Recall@K definitions, and do the corrected results preserve the reported advantages over the baselines?
  2. What are the exact sources and patient-level overlap statistics for the pretraining BUSI dataset and the downstream BUSI-WHU benchmark? Were all pretraining images belonging to downstream test patients excluded, including across differently named datasets?
  3. How does the proposed method compare against random masking, conventional attention-based masking, and non-gaze spatial priors under otherwise identical pretraining conditions? Can the authors quantify the benefit attributable specifically to gaze guidance rather than the combined effects of the additional objectives?
  4. For cases without reliable official reports, how many generated reports were retained after quality control, and what independent validation was used to assess their clinical factuality? Could the authors clarify whether generated reports introduce diagnostic-label leakage into pretraining or downstream evaluation?
4 comments
AuthorsOpenReview ↗

Response to Q1

We thank the reviewer for identifying the anomalous Recall@K values in Table 6 of the original manuscript. We sincerely apologize for this error in result compilation.

Our investigation confirmed that image–report retrieval metrics from another dataset were mistakenly entered into the MRI retrieval results table during result compilation. We rechecked and corrected the affected metrics against the corresponding experimental records and evaluation protocol. The corrected results are shown below.

ModelI→T R@1 ↑I→T R@5 ↑I→T R@10 ↑T→I R@1 ↑T→I R@5 ↑T→I R@10 ↑
OpenAI CLIP0.05260.26320.47370.00000.21050.4737
MedSigLIP-4480.10530.31580.52630.05260.26320.5789
Pillar-0 BreastMRI0.05260.31580.52630.05260.21050.5263
GazeBreast-FM0.21050.47370.63160.15790.42110.6842

All corrected results satisfy the monotonicity requirement of standard Recall@K metrics: R@1 ≤ R@5 ≤ R@10.

In addition, we have checked all other experimental results and corresponding numerical references in the manuscript individually and confirmed the accuracy of the other reported results.

AuthorsOpenReview ↗

Response to Q2

We thank the reviewer for raising the issue of independence between pre-training and downstream evaluation data.

The BUSI dataset was collected at Baheya Hospital in Egypt. The BUSI-WHU dataset was collected at Renmin Hospital of Wuhan University in China. The two datasets were collected independently at different medical institutions in different countries and have distinct data provenance.

We also performed an SHA256 hash-matching audit between BUSI-WHU and the verified original-image pool of the pre-training datasets. No exactly matching images were identified (zero exact image matches), further supporting the use of BUSI-WHU as an independent downstream evaluation cohort.

The other downstream evaluation datasets are likewise absent from the set of 23 pre-training datasets. Pre-training and downstream evaluation use separate data-use and label-management procedures. Together, the dataset provenance records and image-level audit support the independence of the BUSI-WHU downstream evaluation cohort.

AuthorsOpenReview ↗

Response to Q3

We thank the reviewer for suggesting comparisons among masking strategies to evaluate the contribution of the gaze prior.

Under matched pretraining conditions and a fixed 40% mask ratio, we compare Random Masking, Attention-guided Masking, Gaze-guided Masking, and Gaze-guided Masking with Evidence Allocation. Attention-guided Masking uses spatial attention maps extracted from a pretrained DINOv2 model to guide masked-patch selection. The Random Masking configuration reuses the w/o Evidence Allocation results from Table 3 of the original manuscript.

Pre-training masking configurationMG AUROC ↑MRI AUROC ↑US Dice ↑Attribute Macro-AUROC ↑
Random Masking0.8356 ± 0.00350.8558 ± 0.00600.8149 ± 0.00420.8752 ± 0.0025
Attention-guided Masking0.8390 ± 0.00360.8593 ± 0.00620.8204 ± 0.00410.8792 ± 0.0027
Gaze-guided Masking0.8392 ± 0.00320.8601 ± 0.00590.8223 ± 0.00350.8780 ± 0.0022
Gaze-guided Masking with Evidence Allocation0.8467 ± 0.00280.8674 ± 0.00560.8328 ± 0.00290.8876 ± 0.0016

Gaze-guided Masking with Evidence Allocation achieves the best performance among the evaluated masking configurations across all four tasks. Gaze-guided Masking also outperforms Random Masking, demonstrating the benefit of incorporating gaze guidance. The further improvement with Evidence Allocation supports its contribution to model performance.

AuthorsOpenReview ↗

Response to Q4

We thank the reviewer for asking about the provenance and factual accuracy of generated reports and the independence of downstream evaluation labels.

The pre-training inventory contains 302,047 medical images from 23 MG, US, and MRI datasets, corresponding to 62,840 case-level report units. Of these, 61,808 units were model-generated and 1,032 units came from real data sources. Dataset-level counts are presented below.

ModalityDatasetImagesReal-sourceGeneratedTotal report units
MGEMBED100,217035,03435,034
MGRSNA36,185010,95510,955
MGScreening-DBT13,21804,6034,603
MGVinDr-Mammo11,55804,7804,780
MGCSAW-CC6,96201,9671,967
MGCBIS-DDSM2,25401,4581,458
MGCDD-CESM1,3543190319
MGDMID3963960396
MGMIAS3040304304
MGINbreast2930293293
MGBMCD2340163163
MGCSAW-S1740112112
MG total12 datasets173,14971559,66960,384
USBreast-USV64,7390418418
USHiSBreast7143170317
USBUS-UCLM6440644644
USBUSI6160616616
USBrEaST-Lesions-USG2250225225
USOASBUD16509494
US total6 datasets67,1033171,9972,314
MRIDuke-Breast-Cancer-MRI58,71107272
MRII-SPY13,01401818
MRINACT3903232
MRITCGA-BRCA-MRI2501818
MRIRIDER-Breast-MRI6022
MRI total5 datasets61,7950142142
Overall23 datasets302,0471,03261,80862,840

For datasets without original clinical reports, MedGemma-27B-IT generated reports using case-level clinical facts provided by the source datasets, including BI-RADS assessments, breast density, lesion descriptions, diagnoses, and pathology. All reports used for formal pre-training underwent case-level factual consistency checks, and their diagnostic statements were consistent with the original clinical facts and pathology annotations.

Generated reports and their derived semantic supervision were used only during pre-training. The downstream task datasets were kept fully separate from the pre-training datasets, and downstream evaluation used the original task labels and independent data splits. Removing generated reports and their derived supervision decreased performance on MG/MRI classification, US segmentation, and clinical attribute prediction, supporting the contribution of report-derived semantic supervision to tri-modal representation learning.

Reviewer_wUk6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

The paper proposes GazeBreast-FM, a unified breast imaging foundation model combining MAE-style masked reconstruction, CLIP-style image-report alignment aided by spatial “gaze prior”, and Clinical Graph supervision. The authors evaluate their framework across mammography, ultrasound, and breast MRI on downstream clinical tasks, including breast disease classification, lesion segmentation and localization. Experimental results show that the proposed unified model outperforms baseline methods across these modalities and tasks.unified model outperforms baseline methods across these modalities and tasks.

Critical strengths

Consistent improvements across imaging modalities. The method outperforms all included baselines on classification and segmentation across mammography, ultrasound, and MRI.

Broad downstream task evaluation. The paper evaluates disease classification, lesion segmentation and localization, clinical attribute prediction, image-text retrieval, label efficiency, and external generalization. This provides empirical evaluation of the learned representations across different clinical tasks.

Quantitative validation of synthetic gaze quality. The paper evaluates modality-adapted DiffEye against expert gaze using trajectory and spatial-attention metrics. The adapted model consistently outperforms the original DiffEye across all three modalities, supporting the quality of the generated spatial attention.

Critical weaknesses
  • GazeBreast-FM initializes from Mammo-FM and BiomedCLIP before further pretraining on 302,047 breast images. Although Table 1 includes Mammo-FM, comparable continued pretraining of this baseline is not reported. A control using the same initialization, data, and training budget with conventional objectives would help distinguish the proposed mechanisms’ benefit from additional domain training and changes in semantic supervision.

  • Missing ablation studies:

    • Missing dedicated Clinical Graph ablation. The paper introduces graph-derived semantic targets, clinical concept learning, concept consistency, and structural supervision, but does not independently evaluate their contributions. Consequently, the effectiveness of explicit graph supervision beyond conventional image–report alignment remains unverified.

    • Need more ablation of diagnostic-evidence allocation. Table 3 removes evidence allocation as a whole but does not separately evaluate gaze-guided masking, salient-evidence preservation, and diagnostic-weighted reconstruction.

    • Insufficient justification of coverage-aware dynamic weighting. Although dynamic weighting produces consistent improvements, Table 3 shows relatively modest gains. The paper does not compare this mechanism against tuned static or scheduled loss weights, making its advantage over simpler weighting strategies unclear.

Questions for authors

Please refer to the weaknesses mentioned above. I also have the following comments:

  • Figure 1 is difficult to follow. Section 3.1 states that all three imaging modalities share a single visual encoder, but this is not clearly illustrated in Figure 1. Additionally, the panels (c), (d), and (e) are arranged out of order, making the overall framework harder to understand. The authors should revise the figure to better illustrate the proposed architecture and improve its clarity.
  • The technical differences from existing work are not sufficiently discussed. The Introduction and Related Work sections would benefit from a more detailed comparison with existing pretraining and gaze-informed representation learning methods. In particular, the authors should clarify how GazeBreast-FM differs in its use of gaze information and masking strategy, and better justify the motivations behind these design choices.
6 comments
AuthorsOpenReview ↗

Response to Reviewer wUk6 Q1

We thank the reviewer for suggesting matched continued-pretraining controls. To distinguish the contributions of conventional continued pretraining, gaze guidance, and joint learning mechanisms, we compare five configurations under the same Mammo-FM initialization, pretraining data, and training budget. These configurations comprise conventional random-masked reconstruction pretraining with Mammo-FM without gaze guidance, gaze-guided masked reconstruction pretraining, standard image–report alignment pretraining with Mammo-FM without gaze guidance, gaze-guided clinical-semantic learning, and joint optimization of reconstruction and semantic learning objectives in the full model.

Pretraining configurationMG AUROC ↑MRI AUROC ↑US Dice ↑Attribute Macro-AUROC ↑
Random-Masked Reconstruction Only0.8091 ± 0.00450.8290 ± 0.00670.7994 ± 0.00480.8360 ± 0.0037
Gaze-guided Masked Reconstruction Only0.8142 ± 0.00470.8326 ± 0.00680.8097 ± 0.00450.8438 ± 0.0039
Conventional Image–Report Alignment Only (No Gaze)0.8243 ± 0.00420.8416 ± 0.00650.7798 ± 0.00540.8579 ± 0.0036
Gaze-guided Clinical-Semantic Alignment Only0.8315 ± 0.00390.8512 ± 0.00620.7894 ± 0.00570.8721 ± 0.0028
Full Gaze-guided Reconstruction–Semantic Joint Pretraining0.8467 ± 0.00280.8674 ± 0.00560.8328 ± 0.00290.8876 ± 0.0016

The original Mammo-FM achieves 0.8009 ± 0.0010 AUROC on OMAMA. Full joint pretraining achieves the best performance on all four downstream tasks. Gaze-guided clinical-semantic alignment improves upon the conventional image–report alignment configuration, and the results from image-only continued pretraining and gaze-guided reconstruction further support the contributions of gaze guidance, visual reconstruction, and clinical-semantic learning. Their joint optimization improves the transferability of tri-modal representations.

AuthorsOpenReview ↗

Response to Reviewer wUk6 Q2

We thank the reviewer for the suggestion. With other pretraining conditions held constant, we compare the complete model with configurations removing Clinical Graph-derived supervision as a whole or individually removing Graph-derived Soft Relations, Clinical Concept Learning, Concept Consistency, and Graph Structural Consistency.

Pretraining configurationMG AUROC ↑MRI AUROC ↑US Dice ↑Attribute Macro-AUROC ↑
GazeBreast-FM (Full)0.8467 ± 0.00280.8674 ± 0.00560.8328 ± 0.00290.8876 ± 0.0016
w/o Clinical Graph0.8380 ± 0.00340.8591 ± 0.00600.8220 ± 0.00380.8653 ± 0.0027
w/o Graph-derived Soft Relations0.8431 ± 0.00320.8640 ± 0.00600.8300 ± 0.00320.8816 ± 0.0022
w/o Clinical Concept Learning0.8426 ± 0.00350.8636 ± 0.00620.8277 ± 0.00360.8759 ± 0.0026
w/o Concept Consistency0.8452 ± 0.00300.8659 ± 0.00590.8308 ± 0.00320.8834 ± 0.0019
w/o Graph Structural Consistency0.8443 ± 0.00300.8648 ± 0.00590.8293 ± 0.00340.8811 ± 0.0021

The full model achieves the best performance across all four tasks. Removing Clinical Graph supervision or any of the four associated semantic-learning components reduces downstream performance, supporting the positive contributions of structured clinical relations, concept learning, and consistency constraints to tri-modal representation learning.

AuthorsOpenReview ↗

Response to Reviewer wUk6 Q3

We thank the reviewer for suggesting a more detailed examination of diagnostic evidence allocation. Under matched pretraining conditions, we compare random masking without evidence allocation with gaze-guided masking with and without salient-evidence preservation, and separately assess uniform masked-patch reconstruction weighting. Random Masking reuses the original manuscript's w/o Evidence Allocation results. Gaze-guided Masking with Evidence Allocation corresponds to the complete GazeBreast-FM configuration.

Pretraining configurationMG AUROC ↑MRI AUROC ↑US Dice ↑Attribute Macro-AUROC ↑
Random Masking (w/o Evidence Allocation)0.8356 ± 0.00350.8558 ± 0.00600.8149 ± 0.00420.8752 ± 0.0025
Gaze-guided Masking (w/o Salient-Evidence Preservation)0.8392 ± 0.00320.8601 ± 0.00590.8223 ± 0.00350.8780 ± 0.0022
Gaze-guided Masking (with Evidence Allocation)0.8467 ± 0.00280.8674 ± 0.00560.8328 ± 0.00290.8876 ± 0.0016
Uniform Masked-Patch Reconstruction Weighting0.8413 ± 0.00330.8628 ± 0.00600.8205 ± 0.00410.8851 ± 0.0024

Gaze-guided masking with Evidence Allocation achieves the best performance across all four tasks. Compared with random masking without Evidence Allocation, the gaze-guided configuration improves model performance. The comparison with gaze-guided masking without salient-evidence preservation further supports the contribution of retaining visible diagnostic evidence. Uniform reconstruction weighting also reduces performance relative to diagnostic-weighted reconstruction.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Response

We sincerely thank reviewers XMrU (R1), xK41 (R2), and wUk6 (R3) for the time, care, and expertise they devoted to reviewing our work. We deeply appreciate their thoughtful and constructive feedback on the methodology, experimental evidence, data reliability, and interpretation of our findings. Their careful assessment has been invaluable in improving the study. We have taken each concern seriously and are grateful for the reviewers' rigor and professional commitment.

We summarize the main contributions and revisions below. Core Contributions

  1. Unified breast imaging foundation model. GazeBreast-FM jointly pre-trains a shared visual encoder on 302,047 breast images across mammograms, ultrasound, and MRI. It demonstrates strong performance on multiple downstream tasks in all three modalities.
  2. Gaze-guided diagnostic evidence allocation. Modality-adapted DiffEye provides weak spatial-attention priors. Salient-evidence preservation and coverage-aware dynamic weighting coordinate masked reconstruction and image–report semantic learning.
  3. Structured clinical semantic learning. Clinical reports and a report-derived Clinical Graph provide inter-case semantic relationships and supervision for clinical concepts and semantic consistency. Experimental Updates During Rebuttal
  • Main text: We expanded the primary ablations to compare continued pre-training, reconstruction-only and alignment-only objectives, removal of gaze guidance, random masking, static objective weights, and removal of Clinical Graph or generated-report supervision. We also revised Figure 1 to clarify the framework.
  • Appendix: Additional ablations examine random, attention-guided, and gaze-guided masking; salient-evidence preservation; reconstruction weighting; alternative objective weights; and individual Clinical Graph components. Together, the experiments cover 19 distinct pre-training configurations. Eye-tracking metrics show improvements in visual-search trajectories and spatial-attention representations after modality-specific DiffEye adaptation, while ablations further assess the effect of gaze guidance on downstream performance. We also added external-cohort classification evaluations for all three modalities and further lesion-level spatial evaluation.
  • Reviewer responses: We provided targeted component comparisons, label-efficiency and cross-dataset evaluations, and corrected an MRI image–text retrieval result aggregation error. Further Clarifications and Discussion
  • Main text: We clarified how gaze guidance affects mask selection, visible diagnostic evidence, reconstruction and semantic learning, and downstream performance. We refined the explanation of the pre-training objectives, distinctions from prior gaze-guided methods, and the interpretation and limitations of the results.
  • Appendix: We documented the composition and provenance of 23 pre-training datasets and checked their dataset-level separation from all downstream evaluation datasets. For BUSI and BUSI-WHU, we additionally verified different acquisition sources and checked for exact image duplicates using SHA256. We expanded the descriptions of clinical report sources, generated-report consistency checks, data splits, and external evaluation protocols.
  • Reviewer responses: We clarified the distinctions among gaze guidance, salient-evidence preservation, reconstruction weighting, and dynamic objective weighting, as well as the roles of expert eye-tracking data and generated spatial priors. These additions strengthen the experimental evidence and make the evaluation and methodological explanations more transparent. Revisions are highlighted in blue in the manuscript for ease of review.

We again offer our sincere thanks to all reviewers for their careful judgment and generous investment of time. We greatly value their comments and hope the revised manuscript and supplementary material address their concerns and provide a clearer basis for assessing our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.