- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper looks at training one retinal classifier on many public fundus datasets, where a label that a dataset did not record is unknown and should not be treated as negative. It proposes AMA, which adds two things to a masked BCE loss. The first is an auxiliary head that predicts which labels were observed for each image. The second is a fixed set of source–label weights that balance supervision across dataset and label blocks. Neither needs source identity at inference. The method is tested on 252,116 images from 54 datasets mapped to 98 labels, using a single fixed split. It reports macro-AUPRC of 0.802 against 0.545 for a masked baseline, beats 17 foundation models and 7 multi-label losses, and shows gains on four backbones along with ablations and per-label analyses.
Critical strengths
-
The paper points out a real and common problem in pooled medical datasets, which is that "not recorded" gets silently turned into "negative".
-
The corpus is large and well documented, and Tables S1 and S2 list label coverage, split sizes and training support for every dataset and label.
-
The method is simple and cheap, and it needs no source ID or observation mask at test time, so it is easy to deploy.
-
The comparison set is broad, with 17 retinal foundation models and 7 multi-label losses for missing labels and class imbalance. The gain holds across four backbones from three pre-training families, which suggests it is not tied to one encoder. Results come with bootstrap confidence intervals, paired tests and Holm correction instead of bare point estimates.
-
The per-label analysis shows that gains are larger for labels with little training support, which is useful for rare findings. Figures 1 and 2 explain the observed, inferred and unobserved label logic and the training versus inference setup clearly.
Critical weaknesses
-
The masked Baseline does not seem right, since every small change beats it by about 0.25 macro-AUPRC, including U-as-negative (0.758), ASL (0.776) and AMA with γ=0 (0.792), and this goes against the paper's own motivation.
-
The real gain over strong losses is small, with AMA at 0.802 and LL-R at 0.780 and overlapping CIs, and the paper never tests AMA against LL-R or the other strong methods.
-
The ablation shows that each component alone gives almost the full gain (0.794 and 0.800 against 0.802), so the benefit of combining them is within noise.
-
Observability is almost fixed within each dataset, so the observation head mostly learns to recognize which dataset an image came from, and this shortcut may help on a test split drawn from the same datasets.
-
All results use a random split within the same 54 sources, so there is no held-out dataset or external test, even though cross-source generalization is the main point of a unified predictor.
-
The paper does not say whether the split is done at patient level, which matters for AIROGS and EyePACS where both eyes of one patient are included.
-
Every number comes from a single training run, and the bootstrap only covers test sampling, so differences like 0.800 against 0.802 or the choice of λ=γ=0.5 cannot be trusted without seed variance.
-
Table S5 shows that AMA makes calibration (ECE) worse on three of four backbones, for example RetFiner going from 0.014 to 0.058, but the main text does not discuss this.
Questions for authors
-
Why does the masked Baseline score lower than the U-as-negative variant, and could the positive weight ρc, which can be very large for rare labels, explain this?
-
Do ASL, DB, LL-R and the other compared losses also use the ρc positive weighting and the same masking of unobserved entries?
-
How does AMA compare with simpler source-aware baselines such as dataset-balanced sampling, per-dataset heads, or an auxiliary head that predicts the source ID directly?
-
How does AMA do when whole datasets are held out for testing, for example in leave-one-dataset-out runs?
-
Was the split done at the patient level, and how did you check for duplicate or augmented images across datasets like AOD and ODIR, or REFUGE and GNG?
-
How many of the 98 labels meet the five-positive and five-negative rule for macro metrics, and which labels are left out?
-
How was the hierarchical upward propagation in Figure 1 done, and did clinicians check the label mapping?