acceptodds
Under review as a conference paper at ICLR 2027

RetinaRobust: Benchmarking Artifact Robustness and Adaptation in Retinal Foundation Models

Abstract

Retinal foundation models support diverse diagnostic tasks through the analysis of routine, non-invasive images of the eye. Nevertheless, these images are subject to a range of imaging artifacts and variations across clinical settings. We benchmark six retinal foundation models on 17 diagnostic tasks to understand their robustness to artifacts and color removal. All models exhibit performance degradation with the loss of image quality and chromatic information, and substantial gaps remain even after end-to-end fine-tuning for downstream tasks. We examine whether downstream adaptations, including linear probing and end-to-end fine-tuning, can be still utilized to recover the lost performance. We find that training with clean and artifact-corrupted images improves performance not only on artifact-corrupted test images but also on clean test images under both linear probing and end-to-end fine-tuning. These improvements, however, transfer only partly to unseen artifacts. Similarly, transfer of robustness gained from artifacts to the color domain, and vice versa, is limited. Therefore, retinal foundation models must be rigorously evaluated to ensure that their performance generalizes to scenarios with possible corruptions observed beyond training. Our results also motivate more principled machine learning solutions to address the broad-spectrum resilience of these models that facilitate diverse ophthalmic research and clinical applications.

Primary area:
datasets and benchmarks
Keywords:
Retinal Foundation Models, Medical Image Robustness, Robustness Benchmarking, Ophthalmic Image Analysis, Foundation Model Adaptation, Out-of-Distribution Generalization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TFiHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper benchmarks six retinal foundation models on 17 diagnostic tasks, with a focus on quantifying robustness to artifacts and colour removal. Further observations on the utility of including artifact-corrupted training data on the classification of high-quality images, as well as on the transfer of robustness from the artifact to colour domain (and vice-versa), were made. The main contribution might be that a more-aggressive augmentation policy (including artifacts) in preprocessing, may be beneficial towards classification.

Critical strengths
  • Evaluation on the contribution of artifact presence and colour to classification performance, over a range of datasets and retinal image appearances
Critical weaknesses
  • Artifacts appear synthesized without validation of realism, possibly limiting clinical applicability on observations relating to artifacts
  • Limited technical novelty
Questions for authors
  1. In Section 3.2 (and Appendix B), 17 public datasets are stated to be used (after sampling). It might be clarified as to whether any assessment (including for natural artifacts) had been attempted for these retinal images, since the impact of later synthetic artifacts would be compounded.

  2. In Section 3.3 and Figure 2, cropping and rotation are further described as artifacts. However, cropping and rotation are often used in standard preprocessing pipelines for retinal image classification (as also presented in Table 8), and it is not clear as to what the weak and strong settings would imply for rotation in particular. This might be clarified.

  3. Related to the above, a fuller documentation of the weak and strong artifact settings for each artifact type, together with their justification(s) especially against real-world evidence, might be included.

  4. In Section 3.3, the grayscale conversion methodology (especially as to whether it was by green channel extraction - as often applied for fundus images - or otherwise) might be documented.

  5. In Section 5.1, additional experiments on the impact of artifact (and standard) augmentation magnitude towards classification performance would be enlightening.

6 comments

Response to Reviewer TFiH — Part 1 of 4

Reviewer TFiH

We thank the reviewer for the careful review and constructive feedback.

W1 / Q1. Artifact realism and quality of the source datasets

To measure how the synthetic artifacts changed image quality, we sampled original fundus images from all 17 benchmark tasks. For each image, we kept the original and generated ten views by applying each of the five artifact types at both weak and strong settings. We scored the original and each view with FIT, an independent fundus image-quality model. Lower scores indicate lower predicted quality.

The table groups these scores by condition: original images, weak artifact views, and strong artifact views. To put the scores on a real-image scale, we used 0.167, the 20th percentile of FIT scores in Kaggle DR (EyePACS), as a common reference. For example, 55% of the strong artifact views scored below that cutoff, compared with 14% of the weak views and 10% of the originals.

Image conditionMedian FIT scoreBelow EyePACS LQ threshold
Clean0.5510%
Weak artifacts0.3814%
Strong artifacts0.1555%

The clean row answers the source-quality question directly. Even before transformation, 10% of sampled source images scored below the EyePACS reference cutoff. Scoring transformed views of those same images lets us measure the additional change relative to that starting point.

To test transfer to naturally low-quality images, we evaluated DDR, Kaggle DR (EyePACS), and ODIR-5K. We used FIT, an independent fundus quality model, to select high-quality (HQ, top 40%) and low-quality (LQ, bottom 20%) cohorts, excluding FIT scores below 0.01. We matched DR-grade counts between HQ and LQ. Each cohort contained 968 DDR, 1,000 EyePACS, and 328 ODIR images.

We evaluated six encoders with linear probing and full fine-tuning under ten-fold cross-validation. EyePACS and ODIR use patient-grouped folds; DDR uses image-level folds because patient IDs are unavailable. Downstream training used HQ images only. We compared clean training with two augmentation policies: ten standard-augmentation views or ten synthetic artifact views per training image, in addition to the clean image.

We report real-LQ AUROC as mean ± SD across ten folds, averaging six encoders within each fold, and paired 95% confidence intervals for gains over standard augmentation. CIs use 5,000 bootstrap resamples within folds, grouped by patient on EyePACS and ODIR and by image on DDR, conditional on the fitted models.

DatasetAdaptationClean trainingStandard aug.Artifact trainingGain vs standard aug. [95% CI]
DDRLinear probe0.771±0.0320.771 \pm 0.0320.786±0.0320.786 \pm 0.0320.805±0.0250.805 \pm 0.025+0.0192 [0.0105, 0.0280]
EyePACSLinear probe0.715±0.0390.715 \pm 0.0390.731±0.0420.731 \pm 0.0420.740±0.0440.740 \pm 0.044+0.00872 [-0.00004, 0.01739]
ODIR-5KLinear probe0.766±0.0530.766 \pm 0.0530.766±0.0590.766 \pm 0.0590.792±0.0620.792 \pm 0.062+0.0266 [0.0066, 0.0463]
DDRFull fine-tuning0.793±0.0250.793 \pm 0.0250.827±0.0340.827 \pm 0.0340.834±0.0260.834 \pm 0.026+0.0070 [-0.0026, 0.0168]
EyePACSFull fine-tuning0.741±0.0430.741 \pm 0.0430.757±0.0470.757 \pm 0.0470.768±0.0340.768 \pm 0.034+0.0109 [0.0006, 0.0213]
ODIR-5KFull fine-tuning0.772±0.0690.772 \pm 0.0690.790±0.0690.790 \pm 0.0690.800±0.0630.800 \pm 0.063+0.0099 [-0.0102, 0.0291]

The training results followed the same pattern on naturally low-quality images across all three datasets. Artifact training achieved the highest mean AUROC with both linear probing and full fine-tuning. On ODIR, this training benefit was present even though clean-trained models had comparable mean AUROC on the HQ and LQ groups.

Response to Reviewer TFiH — Part 2 of 4

W2. Technical novelty

We submitted this work to ICLR under the primary area “Datasets and Benchmarks” with a focus on the robustness of retinal foundation models to changes in image quality. Retinal foundation models are typically developed and used with high-quality digital images of the eye. In clinical settings, however, acquisition and imaging variability may lead to artifacts arising from translation, rotation, scaling, and affine transformations as well as chromatic variation. Thus, we

ask a question about the generalization and robustness of their performance: how well do representations learned from high-quality images support disease detection when image quality changes? Our technical contribution is an evaluation protocol that tests two distinct abilities. Across 17 retinal tasks, we measure how six retinal image encoders respond to five artifact types at two severities and to color removal. We then hold out artifact families during training to measure how much robustness transfers beyond the degradations a model has seen.

The concurrent FOCUS benchmark (Restrepo et al.), accepted to the NeurIPS 2026 Evaluations & Datasets track, illustrates the relevance of retinal model evaluation to the broader machine learning community. FOCUS studies generalization across model families and datasets, including calibration, fairness, and performance across quality groups. They indicate that image quality labels were obtained from dataset-provided metadata when available and reflected the acquisition or grading quality. Nevertheless, the change in performance due to image quality, as reported in the supplementary Table 10, was marginal and not monotonically decreasing. Our benchmark complements that work by varying individual artifacts under controlled conditions and testing which training gains carry over to real images. Moreover, it highlights the vulnerabilities of retinal foundation models to changes in image quality more conclusively.

In follow-up work, we plan to investigate more sophisticated and systematic recipes for inducing robust retinal encoders. The benchmark sets the baselines and gives us a way to evaluate those on both held-out artifact families and naturally degraded images.

Q2. Rotation and crop settings

Regarding your question about rotation and cropping in standard preprocessing, we first compared artifact training with standard augmentation at matched view counts. Artifact training gave greater robustness. In the six-task linear-probe comparison, the mean artifact gap was 7.4 points with artifact training and 10.8 points with high-magnitude standard augmentation, with ten added views per image in both conditions (see Q5).

We follow the weak and strong settings released by Cheng et al. (2025). Below, we specify the rotation angles and crop sizes used in our benchmark and examine their effects.

Rotation uses fixed angles of 30° (weak) and 90° (strong). Cropping selects a random window retaining 80% (weak) or 40% (strong) of each side, corresponding to 64% or 16% of the image area. These settings test misalignment and restricted field of view. For context, Table 8 reports pretraining rotations up to ±5° and RETFound crops retaining 20–100% of the area.

For rotation, a 90° turn preserves the square frame. At 30°, the canvas expands from 512 × 512 to 699 × 699 pixels, and resizing makes the fundus about 27% smaller. Median FIT scores were 0.55 on clean images, 0.43 at 30°, and 0.50 at 90°. The corresponding 17-task accuracy losses were 14.0 and 10.0 points (Table 15).

We evaluated the angle and canvas controls with clean-trained linear probes on six datasets and six encoders. Mean clean accuracy was 82.9%.

Test view5°15°30°45°90°180°30°, same canvasPadding only, no rotation
Accuracy (%)76.771.768.466.674.178.277.570.1

Even a 5° rotation reduced accuracy by 6.2 points on average and by 20.3 points for RETFound-MAE. We will report the canvas policy alongside the angle so the scale change is reproducible.

Response to Reviewer TFiH — Part 3 of 4

Q3. Parameters of each artifact setting

We adopted the weak and strong presets released by Cheng et al. (2025, npj Digital Medicine). Their weak setting targets partially interpretable images, and their strong setting targets ungradable images. We apply the published parameters after resizing the long side of the RGB image to 512 pixels. The table gives the settings and their measured effects on 340 images (20 per benchmark task). Intensities are clipped to [0, 1].

ArtifactImplementationWeakStrongMeasured effect (weak / strong)
NoiseTorchIO RandomNoise, additive Gaussian, per pixel and channelσ ~ U(0, 0.10)σ ~ U(0, 0.35)PSNR 26.4 / 16.6 dB
Bias fieldTorchIO RandomBiasField, multiplicative exp(polynomial), order 3coefficients ~ U(−0.7, 0.7)coefficients ~ U(−1.8, 1.8)PSNR 16.0 / 10.5 dB
MotionTorchIO RandomMotion (k-space combination of moved copies)8 movements, rotation ±30°, translation ±30 px32 movements, rotation ±160°, translation ±120 pxpixels darker than 10/255: 34% / 93% (clean 22%)
Croprandom window, no padding80% of each side40% of each side64% / 16% of the area kept
Rotationfixed angle, canvas enlarged, zero fill30°90°original frame covers 54% / 100% of the canvas

Across the 17 tasks, median FIT scores fell from 0.55 on clean images to 0.38 on weak views and 0.15 on strong views. For strong intensity artifacts, 61–75% of views fell below the EyePACS low-quality cutoff. The real-image experiments above test whether training with these presets helps disease detection on naturally degraded images. The frequency of each artifact type in clinical practice remains an open question.

The measurements also clarify how to interpret individual transformations. Under strong motion, median brightness falls to 8% of its clean value, making this a test of severe signal loss. Rotation and cropping test orientation, scale, and field of view. We will document these effects alongside the parameters, example images, and per-artifact FIT scores.

Q4. Grayscale conversion

We compute luminance with PIL convert("L") using the BT.601 weights, Y = 0.299 R + 0.587 G + 0.114 B, then copy Y to all three input channels. We apply the conversion after resizing to the 512-pixel working scale and before encoder-specific preprocessing and normalization. Probing, fine-tuning, and grayscale training use the same converted images. In the artifact-plus-grayscale condition, we apply the artifact to the grayscale image.

We also tested green-channel extraction by copying the green channel to all three inputs and evaluating the same clean-trained linear probes. The table compares both conversions on the same 17 tasks.

Test view (17 tasks, 6 encoders)Accuracy (%)Gap (points)AUROC (%)
Clean80.1–86.2
Luminance grayscale (paper)62.118.074.1
Green channel60.919.273.8

Both grayscale conversions identify the same encoders at the extremes. RETFound-DINOv2 had the smallest accuracy gap under luminance and green-channel conversion (6.5 and 9.8 points), and VisionUnite had the largest (28.2 and 29.3 points). We will add the conversion formula to Sec. 3.3 and the green-channel comparison to the appendix.

Reviewer_pVrVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper benchmarks six retinal foundation models across 17 binary classification tasks to evaluate robustness against five types of synthetic image artifacts and grayscale conversion. It compares linear probing and end-to-end fine-tuning, and investigates whether artifact-aware training improves robustness and transfers to unseen artifacts. The results show substantial performance degradation under image corruption, partial robustness gains from targeted adaptation, and limited transfer across artifact types and color-removal conditions.

Critical strengths

The paper is generally clearly written, and the experimental evaluation is extensive, covering six retinal foundation models and 17 classification tasks. The comparison between linear probing and end-to-end fine-tuning, together with the analysis of cross-artifact transfer, provides a useful empirical assessment of robustness in retinal foundation models.

Critical weaknesses

1.Although the benchmark follows previous work, some synthetic transformations may not accurately reflect real retinal image degradation. To what extent do these perturbations represent artifacts encountered in actual clinical settings?

2.Some evaluation datasets may overlap with VisionUnite's pretraining data, and only two datasets use patient-level splits. This raises concerns about potential data leakage and the fairness of model comparisons.

3.Is the number of augmented views consistent across different experimental settings? If not, how much of the reported improvement comes from artifact diversity rather than simply increasing the number of training samples?

4.The benchmark converts all tasks into balanced binary classification, which may not reflect real-world disease prevalence. It remains unclear whether the conclusions generalize to practical screening scenarios.

5.The paper relies primarily on quantitative results and lacks qualitative visualizations that would help illustrate the impact of different perturbation severities. Visual examples for the main experiments would make the findings easier to interpret.

Questions for authors

Please refer to the weaknesses.

3 comments

Response to Reviewer pVrV — Part 1 of 3

Reviewer pVrV

We thank the reviewer for the careful review and constructive feedback.

W1. Whether synthetic artifacts reflect real degradation

Whether the synthetic transformations reflect real retinal image degradation is an important question. We added two complementary evaluations. Specifically, we tested whether training with synthetic artifacts improves disease detection on real low-quality images, and compared the quality scores of synthetic views with a real-image reference. To test transfer to naturally low-quality images, we used the DDR, Kaggle DR (EyePACS), and ODIR-5K datasets. We used FIT, an independent fundus quality model, to select high-quality (HQ, top 40%) and low-quality (LQ, bottom 20%) cohorts, excluding FIT scores (gradability probabilities) below 0.01. We matched DR-grade counts between HQ and LQ. Each cohort contained 968 DDR, 1,000 EyePACS, and 328 ODIR images.

We evaluated six encoders with linear probing and full fine-tuning under ten-fold cross-validation. EyePACS and ODIR use patient-grouped folds; DDR uses image-level folds because patient IDs are unavailable. Downstream training used HQ images only. We compared clean training with two augmentation policies: ten standard-augmentation views or ten synthetic artifact views per training image, in addition to the clean image.

The table reports mean AUROC [95% CI] on held-out real LQ images across six encoders and ten folds. CIs use 5,000 bootstrap resamples within folds, grouped by patient on EyePACS and ODIR and by image on DDR, conditional on the fitted models.

DatasetAdaptationClean trainingStandard aug.Artifact training
DDRLinear probe0.771 [0.755, 0.787]0.786 [0.768, 0.803]0.805 [0.788, 0.822]
EyePACSLinear probe0.715 [0.697, 0.733]0.731 [0.713, 0.749]0.740 [0.721, 0.759]
ODIR-5KLinear probe0.766 [0.741, 0.788]0.766 [0.738, 0.792]0.792 [0.765, 0.817]
DDRFull fine-tuning0.793 [0.776, 0.809]0.827 [0.810, 0.844]0.834 [0.817, 0.850]
EyePACSFull fine-tuning0.741 [0.723, 0.758]0.757 [0.738, 0.775]0.768 [0.749, 0.786]
ODIR-5KFull fine-tuning0.772 [0.745, 0.798]0.790 [0.761, 0.817]0.800 [0.771, 0.826]

With ten added views per image in both training arms, the advantage of synthetic artifact training carried over to real low-quality images on all three datasets. The mean AUROC gains over standard augmentation were 0.019 on DDR, 0.009 on EyePACS, and 0.027 on ODIR with linear probing. Full fine-tuning preserved that ordering between training methods on each dataset, with gains of 0.007, 0.011, and 0.010, respectively. Differences use the unrounded means.

To measure how the synthetic artifacts changed image quality, we sampled original fundus images from all 17 benchmark tasks. For each image, we kept the original and generated ten views by applying each of the five artifact types at both weak and strong settings. We scored the original and each view with FIT, an independent fundus image-quality model. Lower scores indicate lower predicted quality.

The table groups these scores by condition: original images, weak artifact views, and strong artifact views. To put the scores on a real-image scale, we used 0.167, the 20th percentile of FIT scores in Kaggle DR (EyePACS), as a common reference. For example, 55% of the strong artifact views scored below that cutoff, compared with 14% of the weak views and 10% of the originals.

Image conditionMedian FIT scoreBelow EyePACS LQ threshold
Clean0.5510%
Weak artifacts0.3814%
Strong artifacts0.1555%

These experiments address whether the synthetic benchmark captures degradation that matters for performance on real images. FIT scores declined from clean to weak to strong views, and artifact training achieved the highest mean AUROC on real low-quality images across all three datasets with both probing and full fine-tuning. Together, the results connect controlled synthetic degradation to robustness on naturally degraded retinal images.

Response to Reviewer pVrV — Part 2 of 3

W2. Pretraining overlap, patient-level splits, and potential information leakage

We examined two sources of potential information leakage: overlap between pretraining and evaluation datasets, and images from the same patient appearing in different evaluation folds. We audited the reported pretraining data for all six encoders and repeated the robustness evaluation on seven tasks whose source datasets were absent from those records. We also checked patient grouping and duplicate images, as described below.

Here, dataset overlap means that a downstream task’s source dataset appears in the encoder’s reported pretraining corpus. This includes self-supervised pretraining, where task-specific labels may never be used. Our audit identified seven tasks without reported overlap for any encoder: MMRDR, Yangxi, AnwaraHamida, AMDNet23, and the three EDDFS tasks. RETFound-MAE, RETFound-DINOv2, and EyeCLIP have no reported overlap with the original 17-task benchmark.

For each encoder, we grouped the downstream evaluations by whether their source datasets appeared in its reported pretraining corpus (“in”) or were absent from it (“out”). The table reports mean clean accuracy and the artifact-related accuracy drop for each group.

EncoderEvaluations in / outClean accuracy (%) in / outArtifact gap (points) in / out
VisionUnite10 / 786.8 / 77.720.1 / 13.7
FLAIR6 / 1180.6 / 84.313.9 / 14.0
RetiZero6 / 1181.3 / 84.515.1 / 14.0

All three encoders lost accuracy under artifacts in both groups. To check whether the robustness finding persists without reported pretraining overlap, we repeated the evaluation on the seven tasks whose source datasets were absent from every encoder’s reported pretraining corpus.

Results on the seven tasks outside all reported corpora.

EncoderClean (all 17)Artifact gap (all 17)Clean (7 tasks)Artifact gap (7 tasks)
RETFound-MAE77.917.578.018.3
RETFound-DINOv281.012.880.913.5
FLAIR83.014.083.313.6
EyeCLIP72.19.574.912.0
RetiZero83.314.383.914.5
VisionUnite83.017.577.713.7
Mean80.114.379.814.3

On these seven tasks, the mean linear-probe artifact gap was 14.3 points, matching the full benchmark. Accuracy fell for all 42 encoder–task pairs under both probing and fine-tuning. The mean fine-tuning gap was 13.1 points, compared with 12.6 across all 17 tasks. Clean-accuracy rankings varied with task selection, with VisionUnite second lowest on the seven-task subset.

Patient-level grouping and duplicates: We applied patient-level splits to PAPILA and AMDNet23 because these were the only two datasets with recorded patient identifiers available to us. The other 15 tasks lack this information, so we used image-level splits. As an additional check, we kept exact and near-duplicate images in the same fold and repeated linear probing. The mean artifact gap remained nearly unchanged, from 14.3 to 14.1 percentage points, showing that the robustness loss persists after accounting for duplicate images.

Response to Reviewer pVrV — Part 3 of 3

W3. Number of augmented views and the role of diversity

Artifact training and standard augmentation each use 11 training examples per source image: the clean image and ten augmented versions. Each version counts as one training example. The table lists the counts for the other settings.

SettingTraining examples per source image (LP and FT)
Clean only1
+ standard augmentation (control)11 (clean + 10)
+ artifacts11 (clean + 5 artifacts × 2 severities)
+ grayscale2
Transfer, one other artifact / seen3
Leave-one-artifact-out9

Both methods therefore use the same number of training examples per epoch. With early stopping, fine-tuning averaged 11.4 epochs for artifact training and 9.4 for standard augmentation, resulting in different total numbers of training steps.

To test the contribution of artifact diversity at a fixed sample count, we ran a new linear-probe experiment across all 17 tasks and six encoders. Each condition used 11 training examples per source image, counting repeated copies separately, and the same number of steps per epoch. The single-artifact result averages five runs, one per artifact type. Values are percentages, with gaps calculated before rounding.

Training condition (11 examples per source image)Clean acc.Artifact acc.Artifact gap
Clean image repeated 11 times80.966.514.5
Clean + 10 standard-augmentation views80.169.111.1
Clean + one artifact type (2 views, repeated 5 times)80.268.911.4
Clean + all five artifact types (10 views)82.175.46.7

With the number of training examples fixed, training on all five artifact types achieved 75.4% artifact accuracy, compared with 69.1% for standard augmentation and 68.9% for a single type. This was the strongest condition for every encoder. The gains therefore persist at equal sample counts, supporting the benefit of broader artifact coverage.

W4. Balanced tasks and real disease prevalence

Our benchmark uses multiple datasets and diagnostic tasks spanning diverse grading criteria and disease prevalence. We used the typical balanced binary classification to obtain a common ground and systematically evaluate the robustness of models. More specifically, the balanced binary classification scenario allowed us to compare the models across 17 tasks under a common class ratio and give sensitivity and specificity equal weight. On a balanced test set, accuracy equals balanced accuracy. AUROC, sensitivity, and specificity are unaffected by prevalence alone when the within-class score distributions remain fixed. Our robustness gaps compare clean and artifact conditions under this shared class balance.

In a concurrent work, the FOCUS benchmark (Restrepo et al.), accepted to the NeurIPS 2026 Evaluations & Datasets track, binary classification of eye diseases was also the choice for studying the generalization of retinal foundation model performance across model families and datasets, including calibration, fairness, and performance across quality groups. Our work complements theirs by focusing on robustness evaluation under controlled settings.

Regarding the question of whether our conclusions will generalize to practical screening scenarios with multi-class labels, we are optimistic, even though this would require further investigation to settle. A proper, multi-center clinical validation study would be more meaningful to this end.

W5. Qualitative examples of perturbation severity

In response to your request for qualitative visualizations, we will add more examples to the paper showing all five artifact types at both severities, together with the standard-augmentation control. We will show each transformation on the same source images and provide the corresponding parameters. This will let readers compare the visible effects of the severity settings alongside their measured impact on model performance.

Ethics flag

We used publicly released fundus image collections under their published access terms. We used the images and labels supplied by the curators and collected no new patient data. Any release of prepared data will follow the licenses and access conditions of the source datasets. We will add an ethics statement documenting the sources, terms of use, and privacy safeguards. We welcome further guidance on any specific concern.

We hope these clarifications and additional experiments address your concerns and strengthen the evidence for our findings. Thank you again for your time and constructive feedback.

Reviewer_C4MrOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper provides an extensive empirical evaluation of retinal foundation models. While the analysis may be valuable to the medical imaging community, its broader significance to the ICLR community remains unclear. The work may be better suited to a medical imaging venue, such as MICCAI.

Critical strengths
  1. The paper evaluates six retinal foundation models across 17 diagnostic tasks, involving a substantial amount of experimentation.

  2. The paper evaluates both linear probing and end-to-end fine-tuning using shared data splits.

Critical weaknesses

I have the following major concerns:

  1. The paper largely applies existing synthetic corruption techniques and standard adaptation strategies to retinal foundation models. Its methodological novelty is very limited, and its broader significance is not sufficient for ICLR.

  2. As the authors acknowledge, clinical gradability is not assessed. Strong corruptions may destroy essential diagnostic information, meaning that classification failures do not necessarily indicate poor robustness. Furthermore, the clinical realism of the synthetic artifacts remains unverified.

  3. The reported robustness improvements are mainly demonstrated on the same types of synthetic artifacts used during training. Although the authors conduct leave-one-artifact-out experiments, these evaluations remain restricted to a fixed set of synthetic corruptions. Comparisons with established robustness methods beyond standard augmentation, as well as evaluations on naturally degraded images, are needed to demonstrate broader generalizability.

Questions for authors

Please refer to the weaknesses section.

3 comments

Response to Reviewer C4Mr — Part 1 of 3

We thank the reviewer for the careful review and constructive feedback.

W1. Novelty and fit for ICLR

We submitted this work to ICLR under the primary area “Datasets and Benchmarks” with a focus on the robustness of retinal foundation models to changes in image quality. These models are typically developed and evaluated using high-quality retinal images. In clinical settings, acquisition variability can change image orientation, scale, field of view, illumination, and color.

We therefore ask how well representations learned from high-quality images support disease detection when image quality changes. Our technical contribution is an evaluation protocol that measures sensitivity to degradation and transfer of robustness. Across 17 retinal tasks, we measure how six retinal image encoders respond to five artifact types at two severities and to color removal. We then hold out artifact families during training to measure how much robustness transfers beyond the degradations a model has seen.

The concurrent FOCUS benchmark (Restrepo et al.), accepted to the NeurIPS 2026 Evaluations & Datasets track, illustrates the relevance of retinal model evaluation to the broader machine learning community. FOCUS studies generalization across model families and datasets, including calibration, fairness, and performance across quality groups. Its image-quality analysis uses dataset-provided quality metadata on BRSET and mBRSET. Our benchmark complements this analysis by varying individual artifacts under controlled conditions, measuring artifact-specific vulnerabilities, and testing which training gains carry over to real images. FOCUS

In follow-up work, we plan to investigate more sophisticated and systematic recipes for inducing robust retinal encoders. The benchmark sets the baselines and gives us a way to evaluate those on both held-out artifact families and naturally degraded images.

W2. Artifact realism and diagnostic gradability

To address your concern that synthetic corruptions may remove diagnostic information, we added two evaluations on real retinal images. First, we tested whether disease-detection performance declines on images that human graders still consider usable for diagnosis. Second, we tested whether training with our synthetic artifacts improves performance on real low-quality images. We also compared the quality scores of our synthetic views with those of real images.

EyeQ (Fu et al., 2019) provides human image-quality annotations for Kaggle DR (EyePACS), with three categories: Good, Usable, and Reject. Usable images have quality defects but remain suitable for diagnosis. We used these labels to evaluate robustness under degradation for which human graders judged that diagnostic information remained available.

For this comparison, we selected 1,000 Good and 1,000 Usable images, with 500 normal and 500 DR images in each group. We matched the DR-grade counts between groups, trained linear probes on Good images, and evaluated on held-out patients. The table reports mean AUROC and paired drops over six encoders and ten folds, calculated before rounding.

Test conditionMean AUROCMean drop from Good
Good (held out)0.799–
Usable (degraded but gradable)0.7220.076

All six encoders lost AUROC on real images rated Usable, with a mean drop of 0.076. The drop ranged from 0.035 for FLAIR to 0.117 for EyeCLIP. Thus, the performance loss also occurs on naturally degraded images that human graders consider suitable for diagnosis.

Response to Reviewer C4Mr — Part 2 of 3

W2. Artifact realism and diagnostic gradability (continued)

We next tested whether robustness learned from synthetic artifacts transfers to real low-quality images. To this end, we used DDR, Kaggle DR (EyePACS), and ODIR-5K. We selected high-quality (HQ, top 40%) and low-quality (LQ, bottom 20%) images using FIT, a fundus image-quality model. We excluded FIT scores below 0.01 and DDR’s ungradable class, and matched DR-grade counts across quality groups. Each quality group contained 968 DDR, 1,000 EyePACS, and 328 ODIR images.

All training used HQ images. We compared clean training, standard augmentation, and artifact training. The latter two each added ten views per source image, allowing us to test whether the choice of transformations improves transfer beyond adding the same number of standard-augmentation views. We used linear probing and full fine-tuning, with ten-fold cross-validation grouped by patient on EyePACS and ODIR and by image on DDR.

We report real-LQ AUROC as mean ± SD across ten folds, averaging six encoders within each fold, and paired 95% confidence intervals for gains over standard augmentation.

DatasetAdaptationClean trainingStandard aug.Artifact trainingGain vs standard aug. [95% CI]
DDRLinear probe0.771±0.0320.771 \pm 0.0320.786±0.0320.786 \pm 0.0320.805±0.0250.805 \pm 0.025+0.0192 [0.0105, 0.0280]
EyePACSLinear probe0.715±0.0390.715 \pm 0.0390.731±0.0420.731 \pm 0.0420.740±0.0440.740 \pm 0.044+0.00872 [-0.00004, 0.01739]
ODIR-5KLinear probe0.766±0.0530.766 \pm 0.0530.766±0.0590.766 \pm 0.0590.792±0.0620.792 \pm 0.062+0.0266 [0.0066, 0.0463]
DDRFull fine-tuning0.793±0.0250.793 \pm 0.0250.827±0.0340.827 \pm 0.0340.834±0.0260.834 \pm 0.026+0.0070 [-0.0026, 0.0168]
EyePACSFull fine-tuning0.741±0.0430.741 \pm 0.0430.757±0.0470.757 \pm 0.0470.768±0.0340.768 \pm 0.034+0.0109 [0.0006, 0.0213]
ODIR-5KFull fine-tuning0.772±0.0690.772 \pm 0.0690.790±0.0690.790 \pm 0.0690.800±0.0630.800 \pm 0.063+0.0099 [-0.0102, 0.0291]

Artifact training achieved the highest mean AUROC across all three datasets under both adaptation methods. The paired confidence interval excludes zero for DDR and ODIR with linear probing and for EyePACS with full fine-tuning. Intervals use 5,000 paired bootstrap resamples within folds, grouped by patient on EyePACS and ODIR and by image on DDR, conditional on the fitted models.

To relate the synthetic settings to real image quality, we used FIT, whose quality models were trained on real fundus images from DeepDRiD and DrimDB. On 28,790 EyePACS images with EyeQ labels, FIT distinguished Reject from Good or Usable with AUROC 0.96. We use FIT to compare image-quality distributions and EyeQ to evaluate performance on human-rated gradable images.

We sampled source images from all 17 benchmark tasks and generated ten views per image, covering five artifact types at weak and strong settings. We scored each original image and its transformed views with FIT. Lower scores indicate lower predicted quality.

The table compares these distributions using a common real-image reference. We set the LQ threshold to 0.167, the 20th percentile of FIT scores in Kaggle DR (EyePACS). The proportion below this threshold increased from 10% for original images to 14% for weak views and 55% for strong views.

Image conditionMedian FIT scoreBelow EyePACS LQ threshold
Clean0.5510%
Weak artifacts0.3814%
Strong artifacts0.1555%

We further tested whether performance losses persist at weak severity. We trained linear probes on HQ images, then evaluated them on real LQ images and on weakly or strongly transformed HQ test images. Each column reports the drop from the same held-out HQ baseline, averaged over six encoders and ten folds.

DatasetReal LQ dropWeak synthetic dropStrong synthetic drop
DDR0.1370.1090.241
EyePACS0.0680.0840.165
ODIR-5Knone (−0.027)0.0690.136

Weak artifacts reduced mean AUROC by 0.109 on DDR and 0.084 on EyePACS, compared with drops of 0.137 and 0.068 on real LQ images. ODIR showed a loss under synthetic artifacts and no corresponding real-LQ deficit. We will present the strong setting separately as a stress test of severe degradation.

Together, these evaluations address both parts of your concern. Performance declines on real images that human graders still consider gradable, and training with synthetic artifacts improves mean AUROC on real low-quality images across three datasets. These results show that the measured failures extend to gradable real images and that the training gains transfer beyond synthetic test conditions.

Response to Reviewer C4Mr — Part 3 of 3

W3. Comparison with AugMix and RandAugment on real images

Regarding comparisons with established robustness methods and evaluation beyond the synthetic artifacts used in training, we compared artifact training with the AugMix (Hendrycks et al., 2020) and RandAugment (Cubuk et al., 2020) augmentation policies on real low-quality images. We also included clean training and standard augmentation. All five conditions used the same frozen encoders, linear probes, folds, and validation rule. Each augmentation method added ten views per high-quality training image. We used standard torchvision settings for AugMix and RandAugment.

The table reports disease-detection AUROC on four cohorts from three datasets. The first three rows use low-quality images selected by FIT. The fourth uses human EyeQ labels and pools 1,000 Usable and 1,000 Reject images. We computed AUROC on the pooled predictions within each test fold, then averaged over folds and encoders.

Real-image test cohortCleanStandard aug.AugMixRandAugmentArtifact training
DDR0.7710.7860.7880.8010.805
EyePACS (FIT)0.7150.7310.7210.7230.740
ODIR-5K0.7660.7660.7690.7860.792
EyePACS (EyeQ Usable + Reject)0.6880.7040.6920.7000.712

Artifact training achieved the highest mean AUROC on all four real-image cohorts. Its gains ranged from 0.017 to 0.023 over AugMix and from 0.004 to 0.017 over RandAugment. These comparisons address whether the training gains generalize beyond familiar synthetic corruptions and remain competitive with established augmentation methods. With the same number of added views, the advantage of artifact training extended to naturally occurring low-quality images across three datasets.

We hope these clarifications and additional experiments address your concerns and strengthen the evidence for our findings. Thank you again for your time and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.