- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper benchmarks six retinal foundation models on 17 diagnostic tasks, with a focus on quantifying robustness to artifacts and colour removal. Further observations on the utility of including artifact-corrupted training data on the classification of high-quality images, as well as on the transfer of robustness from the artifact to colour domain (and vice-versa), were made. The main contribution might be that a more-aggressive augmentation policy (including artifacts) in preprocessing, may be beneficial towards classification.
Critical strengths
- Evaluation on the contribution of artifact presence and colour to classification performance, over a range of datasets and retinal image appearances
Critical weaknesses
- Artifacts appear synthesized without validation of realism, possibly limiting clinical applicability on observations relating to artifacts
- Limited technical novelty
Questions for authors
-
In Section 3.2 (and Appendix B), 17 public datasets are stated to be used (after sampling). It might be clarified as to whether any assessment (including for natural artifacts) had been attempted for these retinal images, since the impact of later synthetic artifacts would be compounded.
-
In Section 3.3 and Figure 2, cropping and rotation are further described as artifacts. However, cropping and rotation are often used in standard preprocessing pipelines for retinal image classification (as also presented in Table 8), and it is not clear as to what the weak and strong settings would imply for rotation in particular. This might be clarified.
-
Related to the above, a fuller documentation of the weak and strong artifact settings for each artifact type, together with their justification(s) especially against real-world evidence, might be included.
-
In Section 3.3, the grayscale conversion methodology (especially as to whether it was by green channel extraction - as often applied for fundus images - or otherwise) might be documented.
-
In Section 5.1, additional experiments on the impact of artifact (and standard) augmentation magnitude towards classification performance would be enlightening.
6 comments
Response to Reviewer TFiH — Part 2 of 4
W2. Technical novelty
We submitted this work to ICLR under the primary area “Datasets and Benchmarks” with a focus on the robustness of retinal foundation models to changes in image quality. Retinal foundation models are typically developed and used with high-quality digital images of the eye. In clinical settings, however, acquisition and imaging variability may lead to artifacts arising from translation, rotation, scaling, and affine transformations as well as chromatic variation. Thus, we
ask a question about the generalization and robustness of their performance: how well do representations learned from high-quality images support disease detection when image quality changes? Our technical contribution is an evaluation protocol that tests two distinct abilities. Across 17 retinal tasks, we measure how six retinal image encoders respond to five artifact types at two severities and to color removal. We then hold out artifact families during training to measure how much robustness transfers beyond the degradations a model has seen.
The concurrent FOCUS benchmark (Restrepo et al.), accepted to the NeurIPS 2026 Evaluations & Datasets track, illustrates the relevance of retinal model evaluation to the broader machine learning community. FOCUS studies generalization across model families and datasets, including calibration, fairness, and performance across quality groups. They indicate that image quality labels were obtained from dataset-provided metadata when available and reflected the acquisition or grading quality. Nevertheless, the change in performance due to image quality, as reported in the supplementary Table 10, was marginal and not monotonically decreasing. Our benchmark complements that work by varying individual artifacts under controlled conditions and testing which training gains carry over to real images. Moreover, it highlights the vulnerabilities of retinal foundation models to changes in image quality more conclusively.
In follow-up work, we plan to investigate more sophisticated and systematic recipes for inducing robust retinal encoders. The benchmark sets the baselines and gives us a way to evaluate those on both held-out artifact families and naturally degraded images.
Q2. Rotation and crop settings
Regarding your question about rotation and cropping in standard preprocessing, we first compared artifact training with standard augmentation at matched view counts. Artifact training gave greater robustness. In the six-task linear-probe comparison, the mean artifact gap was 7.4 points with artifact training and 10.8 points with high-magnitude standard augmentation, with ten added views per image in both conditions (see Q5).
We follow the weak and strong settings released by Cheng et al. (2025). Below, we specify the rotation angles and crop sizes used in our benchmark and examine their effects.
Rotation uses fixed angles of 30° (weak) and 90° (strong). Cropping selects a random window retaining 80% (weak) or 40% (strong) of each side, corresponding to 64% or 16% of the image area. These settings test misalignment and restricted field of view. For context, Table 8 reports pretraining rotations up to ±5° and RETFound crops retaining 20–100% of the area.
For rotation, a 90° turn preserves the square frame. At 30°, the canvas expands from 512 × 512 to 699 × 699 pixels, and resizing makes the fundus about 27% smaller. Median FIT scores were 0.55 on clean images, 0.43 at 30°, and 0.50 at 90°. The corresponding 17-task accuracy losses were 14.0 and 10.0 points (Table 15).
We evaluated the angle and canvas controls with clean-trained linear probes on six datasets and six encoders. Mean clean accuracy was 82.9%.
| Test view | 5° | 15° | 30° | 45° | 90° | 180° | 30°, same canvas | Padding only, no rotation |
|---|---|---|---|---|---|---|---|---|
| Accuracy (%) | 76.7 | 71.7 | 68.4 | 66.6 | 74.1 | 78.2 | 77.5 | 70.1 |
Even a 5° rotation reduced accuracy by 6.2 points on average and by 20.3 points for RETFound-MAE. We will report the canvas policy alongside the angle so the scale change is reproducible.
Response to Reviewer TFiH — Part 3 of 4
Q3. Parameters of each artifact setting
We adopted the weak and strong presets released by Cheng et al. (2025, npj Digital Medicine). Their weak setting targets partially interpretable images, and their strong setting targets ungradable images. We apply the published parameters after resizing the long side of the RGB image to 512 pixels. The table gives the settings and their measured effects on 340 images (20 per benchmark task). Intensities are clipped to [0, 1].
| Artifact | Implementation | Weak | Strong | Measured effect (weak / strong) |
|---|---|---|---|---|
| Noise | TorchIO RandomNoise, additive Gaussian, per pixel and channel | σ ~ U(0, 0.10) | σ ~ U(0, 0.35) | PSNR 26.4 / 16.6 dB |
| Bias field | TorchIO RandomBiasField, multiplicative exp(polynomial), order 3 | coefficients ~ U(−0.7, 0.7) | coefficients ~ U(−1.8, 1.8) | PSNR 16.0 / 10.5 dB |
| Motion | TorchIO RandomMotion (k-space combination of moved copies) | 8 movements, rotation ±30°, translation ±30 px | 32 movements, rotation ±160°, translation ±120 px | pixels darker than 10/255: 34% / 93% (clean 22%) |
| Crop | random window, no padding | 80% of each side | 40% of each side | 64% / 16% of the area kept |
| Rotation | fixed angle, canvas enlarged, zero fill | 30° | 90° | original frame covers 54% / 100% of the canvas |
Across the 17 tasks, median FIT scores fell from 0.55 on clean images to 0.38 on weak views and 0.15 on strong views. For strong intensity artifacts, 61–75% of views fell below the EyePACS low-quality cutoff. The real-image experiments above test whether training with these presets helps disease detection on naturally degraded images. The frequency of each artifact type in clinical practice remains an open question.
The measurements also clarify how to interpret individual transformations. Under strong motion, median brightness falls to 8% of its clean value, making this a test of severe signal loss. Rotation and cropping test orientation, scale, and field of view. We will document these effects alongside the parameters, example images, and per-artifact FIT scores.
Q4. Grayscale conversion
We compute luminance with PIL convert("L") using the BT.601 weights, Y = 0.299 R + 0.587 G + 0.114 B, then copy Y to all three input channels. We apply the conversion after resizing to the 512-pixel working scale and before encoder-specific preprocessing and normalization. Probing, fine-tuning, and grayscale training use the same converted images. In the artifact-plus-grayscale condition, we apply the artifact to the grayscale image.
We also tested green-channel extraction by copying the green channel to all three inputs and evaluating the same clean-trained linear probes. The table compares both conversions on the same 17 tasks.
| Test view (17 tasks, 6 encoders) | Accuracy (%) | Gap (points) | AUROC (%) |
|---|---|---|---|
| Clean | 80.1 | – | 86.2 |
| Luminance grayscale (paper) | 62.1 | 18.0 | 74.1 |
| Green channel | 60.9 | 19.2 | 73.8 |
Both grayscale conversions identify the same encoders at the extremes. RETFound-DINOv2 had the smallest accuracy gap under luminance and green-channel conversion (6.5 and 9.8 points), and VisionUnite had the largest (28.2 and 29.3 points). We will add the conversion formula to Sec. 3.3 and the green-channel comparison to the appendix.
Response to Reviewer TFiH — Part 1 of 4
Reviewer TFiH
We thank the reviewer for the careful review and constructive feedback.
W1 / Q1. Artifact realism and quality of the source datasets
To measure how the synthetic artifacts changed image quality, we sampled original fundus images from all 17 benchmark tasks. For each image, we kept the original and generated ten views by applying each of the five artifact types at both weak and strong settings. We scored the original and each view with FIT, an independent fundus image-quality model. Lower scores indicate lower predicted quality.
The table groups these scores by condition: original images, weak artifact views, and strong artifact views. To put the scores on a real-image scale, we used 0.167, the 20th percentile of FIT scores in Kaggle DR (EyePACS), as a common reference. For example, 55% of the strong artifact views scored below that cutoff, compared with 14% of the weak views and 10% of the originals.
The clean row answers the source-quality question directly. Even before transformation, 10% of sampled source images scored below the EyePACS reference cutoff. Scoring transformed views of those same images lets us measure the additional change relative to that starting point.
To test transfer to naturally low-quality images, we evaluated DDR, Kaggle DR (EyePACS), and ODIR-5K. We used FIT, an independent fundus quality model, to select high-quality (HQ, top 40%) and low-quality (LQ, bottom 20%) cohorts, excluding FIT scores below 0.01. We matched DR-grade counts between HQ and LQ. Each cohort contained 968 DDR, 1,000 EyePACS, and 328 ODIR images.
We evaluated six encoders with linear probing and full fine-tuning under ten-fold cross-validation. EyePACS and ODIR use patient-grouped folds; DDR uses image-level folds because patient IDs are unavailable. Downstream training used HQ images only. We compared clean training with two augmentation policies: ten standard-augmentation views or ten synthetic artifact views per training image, in addition to the clean image.
We report real-LQ AUROC as mean ± SD across ten folds, averaging six encoders within each fold, and paired 95% confidence intervals for gains over standard augmentation. CIs use 5,000 bootstrap resamples within folds, grouped by patient on EyePACS and ODIR and by image on DDR, conditional on the fitted models.
The training results followed the same pattern on naturally low-quality images across all three datasets. Artifact training achieved the highest mean AUROC with both linear probing and full fine-tuning. On ODIR, this training benefit was present even though clean-trained models had comparable mean AUROC on the HQ and LQ groups.