acceptodds
Under review as a conference paper at ICLR 2027

Learning from Heterogeneous Annotation Mechanisms for Unified Retinal Prediction

Abstract

Learning a unified retinal predictor from datasets collected for different clinical purposes is complicated by a fundamental ambiguity: an unrecorded label is not necessarily clinically negative. Because annotations follow heterogeneous, source-specific protocols, flattening such datasets into a common binary label matrix can introduce systematic false-negative supervision. Masking unobserved entries prevents this error, but uses label observability only to gate the loss and leaves both the structured observation process and uneven supervision across source–label blocks unmodeled. We introduce Annotation-Mechanism-Aware learning (AMA), which separates clinical prediction from label observability. AMA combines an auxiliary label-observability objective with source–label schema balancing, while requiring neither source identity nor observation masks at inference. We evaluate AMA on 252,116 fundus photographs from 54 public datasets whose annotations are harmonized into a 98-label clinical vocabulary. With the same pre-trained retinal encoder and training budget, AMA attains a macro-AUPRC of 0.802 (95% CI: 0.788–0.823), compared with 0.545 (95% CI: 0.538–0.572) for the matched masked-supervision Baseline. These gains persist across four retinal backbones spanning masked image modeling, self-distillation, and vision–language pre-training, with improved per-label AUPRC estimates for 96 labels. The results support treating label observability as part of the learning problem when pooling medical datasets, rather than assuming that the pooled label matrix is complete.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
retinal image analysis, multi-source learning, heterogeneous annotation mechanisms, partially observed labels, multi-label classification

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Ch9WOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper looks at training one retinal classifier on many public fundus datasets, where a label that a dataset did not record is unknown and should not be treated as negative. It proposes AMA, which adds two things to a masked BCE loss. The first is an auxiliary head that predicts which labels were observed for each image. The second is a fixed set of source–label weights that balance supervision across dataset and label blocks. Neither needs source identity at inference. The method is tested on 252,116 images from 54 datasets mapped to 98 labels, using a single fixed split. It reports macro-AUPRC of 0.802 against 0.545 for a masked baseline, beats 17 foundation models and 7 multi-label losses, and shows gains on four backbones along with ablations and per-label analyses.

Critical strengths
  1. The paper points out a real and common problem in pooled medical datasets, which is that "not recorded" gets silently turned into "negative".

  2. The corpus is large and well documented, and Tables S1 and S2 list label coverage, split sizes and training support for every dataset and label.

  3. The method is simple and cheap, and it needs no source ID or observation mask at test time, so it is easy to deploy.

  4. The comparison set is broad, with 17 retinal foundation models and 7 multi-label losses for missing labels and class imbalance. The gain holds across four backbones from three pre-training families, which suggests it is not tied to one encoder. Results come with bootstrap confidence intervals, paired tests and Holm correction instead of bare point estimates.

  5. The per-label analysis shows that gains are larger for labels with little training support, which is useful for rare findings. Figures 1 and 2 explain the observed, inferred and unobserved label logic and the training versus inference setup clearly.

Critical weaknesses
  1. The masked Baseline does not seem right, since every small change beats it by about 0.25 macro-AUPRC, including U-as-negative (0.758), ASL (0.776) and AMA with γ=0 (0.792), and this goes against the paper's own motivation.

  2. The real gain over strong losses is small, with AMA at 0.802 and LL-R at 0.780 and overlapping CIs, and the paper never tests AMA against LL-R or the other strong methods.

  3. The ablation shows that each component alone gives almost the full gain (0.794 and 0.800 against 0.802), so the benefit of combining them is within noise.

  4. Observability is almost fixed within each dataset, so the observation head mostly learns to recognize which dataset an image came from, and this shortcut may help on a test split drawn from the same datasets.

  5. All results use a random split within the same 54 sources, so there is no held-out dataset or external test, even though cross-source generalization is the main point of a unified predictor.

  6. The paper does not say whether the split is done at patient level, which matters for AIROGS and EyePACS where both eyes of one patient are included.

  7. Every number comes from a single training run, and the bootstrap only covers test sampling, so differences like 0.800 against 0.802 or the choice of λ=γ=0.5 cannot be trusted without seed variance.

  8. Table S5 shows that AMA makes calibration (ECE) worse on three of four backbones, for example RetFiner going from 0.014 to 0.058, but the main text does not discuss this.

Questions for authors
  1. Why does the masked Baseline score lower than the U-as-negative variant, and could the positive weight ρc, which can be very large for rare labels, explain this?

  2. Do ASL, DB, LL-R and the other compared losses also use the ρc positive weighting and the same masking of unobserved entries?

  3. How does AMA compare with simpler source-aware baselines such as dataset-balanced sampling, per-dataset heads, or an auxiliary head that predicts the source ID directly?

  4. How does AMA do when whole datasets are held out for testing, for example in leave-one-dataset-out runs?

  5. Was the split done at the patient level, and how did you check for duplicate or augmented images across datasets like AOD and ODIR, or REFUGE and GNG?

  6. How many of the 98 labels meet the five-positive and five-negative rule for macro metrics, and which labels are left out?

  7. How was the hierarchical upward propagation in Figure 1 done, and did clinicians check the label mapping?

Reviewer_7n4BOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces Annotation-Mechanism-Aware learning (AMA), a framework designed to tackle the problem of pooling multi-source, heterogeneous medical datasets without introducing systematic false-negative supervision. Medical image datasets often have partial annotations: an unobserved condition in a specific dataset is usually omitted because of protocol constraints rather than confirmed as clinically negative. To solve this, AMA decouples the latent clinical status (yicy_{ic}) from label observability (oico_{ic}): Auxiliary Observability Objective (LobsL_{\text{obs}}) uses binary observation masks during training to force the shared representation encoder to learn dataset-level annotation policies as a regularizing signal. Schema-Balanced Supervision (W=[wsc]W = [w_{sc}]) balances supervision density across different source datasets (ss) and clinical labels (cc) relative to median support. Inference efficiency requires neither source identity nor observation masks at deployment, retaining only the shared encoder and primary clinical prediction head. AMA is evaluated on a large-scale corpus of 252,116 fundus images across 54 public datasets harmonized into a 98-label clinical vocabulary. AMA attains a macro-AUPRC of0.802 compared with 0.545 for the matched masked-supervision Baseline.

Critical strengths

The paper is motivated by needing to distinguish between "unobserved" data and "confirmed clinical negatives" to directly address a limitation in pooling together diverse multi-source medical images with heterogeneous labels, in this case retinal images. The objective functions (LclinicalL_{\text{clinical}}, LobsL_{\text{obs}}, and schema weight matrix WW) are used to prevent false-negative label imputation. The paper builds a large evaluation dataset by pooling together 252,116 images from 54 datasets and uses 98 labels. Performance gains of the proposed approach persists across several different foundation model backbones (spanning masked image modeling, self-distillation, and vision–language pre-training). Component ablations isolate the contribution of the auxiliary observability loss versus schema-balanced clinical supervision. The experimental results use non-parametric bootstrap (500 resamples) for 95% confidence intervals and applies multiple-testing adjustments (Holm sequential step-down procedure). Demonstrates a clear performance improvement over baseline masked supervision using macro-AUPRC of 0.802 vs. 0.545. Per-label AUPRC improved for 96 out of 98 labels.

Critical weaknesses

While AMA eliminates the need for source masks at inference, the paper leaves open questions regarding prediction calibration across backbones and deployment to unseen clinical sources with novel annotation protocols. The paper frames the technique predominantly around retinal fundus photography. However, the core mathematical mechanism (Lobs+schema weightingL_{\text{obs}} + \text{schema weighting}) can be applied to other domains and this should be illustrated with another small example.

Mapping 54 heterogeneous datasets to a unified 98-label clinical vocabulary relies on manual curation and schema harmonization. It is not clear from the Reproducibility Statement (Code, experiment configurations, and evaluation scripts will be released upon publication.) whether the data the authors created with the 98-label vocabulary (37 disease labels, 18 grading labels, and 42 retinal finding labels) required any changes to the original labels and were identically used even in cases where the label may be ambiguous or known to be incorrect or incomplete. The data matrix D={(xi,si,yi)}i=1N\mathcal{D} = \{(x_i, s_i, \mathbf{y}_i)\}_{i=1}^N with the 98-element label vector, \mathbf{y}_i) (including U for Unobserved labels) should be made publicly available.

What does the label 89 Fundus Abnormality Status (48,005 samples from 51 observing sources) mean in the context of the overall pooled dataset since images are labeled with their disease state which is an abnormal retinal health finding from the viewpoint of the clinician. Does this mean that the authors added an additional label of their own (that is 89) in addition to the disease label like Diabetic Retinopathy (DR, 21) or Glaucoma Suspect (87) or Grading labels like Severe NDPR (7), Advanced PDR (7), and all of the Finding classes (46 to 88) would also receive the label 89. Disease, Grading and Finding categories the majority of labels would clinically relevant as an abnormal fundus status - this is not clear in Figure 1a and 1b (the Abnormal class can be shown as a dashed line around sample classes shown (ie Glaucoma, DR, Macular Edema, Hemorrhage, Advanced Disease). So it is not clear how the paper separately deals with Fundus Abnormality Status? In Table S2 the total number of images/samples should be added up for each category (label grouping) - Disease, Grading, Finding. Acronyms should be spelled out (like NDPR, PDR, etc).

Questions for authors

Demonstrating performance outside fundus photography would significantly strengthen the paper's representation-learning claims. This could be illustrated by testing the AMA approach on non-retinal multi-source benchmarks (e.g., multi-label chest X-ray databases like NIH ChestX-ray14 / CheXpert / Tuberculosis, or general multi-label computer vision datasets)?

A significant weakness is a lack of assessing the generalizability of AMA even for the ophthalmology domain by evaluating performance on unseen datasets. During inference, AMA discards the source mask and auxiliary observability head, running only the shared encoder and clinical prediction head. How does AMA perform when evaluated on a completely unseen 55th dataset whose collection protocol or device type was not part of the 54 training sources? Does the auxiliary observability loss LobsL_{\text{obs}} overfit the representation to the specific collection policies of the 54 known training sources?

The auxiliary observability objective LobsL_{\text{obs}} forces the shared feature encoder gϕg_\phi to encode dataset-level annotation policies. If the encoder heavily embeds features that signal which dataset an image originated from in order to minimize LobsL_{\text{obs}}, it may suffer from domain shortcut learning. If deployed in a real-world setting with images from an unobserved clinic or new camera model, these observability-dependent features may act as spurious correlations.

Was hyperparameter tuning performed (λobs\lambda_{\text{obs}}, γ\gamma, and α\alpha)? How sensitive is the final clinical performance to the choice of the observation loss weight λobs\lambda_{\text{obs}} and the schema-balancing power parameter γ\gamma? Was grid search performed on the validation set, and does a single fixed hyperparameter set hold across all four foundation-model backbones?

Thersholding and calibration across a large number of 98 clinical labels will be difficult to operationalize since labels will overlap - for example an observation of glaucoma along with grading of severity is across multiple label categories for the same image during inference even if the original dataset did not contain the combination of labels. Since macro-AUPRC and macro-AUROC evaluate ranking across decision thresholds, how are concrete clinical predictions operationalized? Does schema-balancing affect the logit calibration across labels with widely different sample densities (e.g., common abnormalities vs. rare disease findings)?

In clinical decision-support systems, raw probability scores dictate risk triage. Schema balancing alters positive/negative class loss weights during training, which can skew output confidence scores if not explicitly recalibrated (e.g., Expected Calibration Error / ECE or Brier Score).

In medical image analysis—particularly organ segmentation across heterogeneous CT/MRI datasets (e.g., combining a Liver-only dataset with a Kidney-only dataset)—the issue of "unannotated does not mean absent" is extensively studied under Partial Label Loss / Marginal Loss formulations and should be cited including Marginal loss and exclusion loss for partially supervised multi-organ segmentation (MIA, 2021) and Multi-Organ Segmentation Over Partially Labeled Datasets With Multi-Scale Feature Abstraction (IEEE TMI, 2020). Developing a single model gϕg_{\phi} shared encoder for the architecture may not be as robust as a multi-expert approach that has been shown to perform well on the RFMiD data (source 45 and 46); see Multi-Expert deep networks for multi-disease detection in retinal fundus images, IEEE EMBC 2022. Such alternative architectures should at least be referenced in the paper and ideally evaluated along with the foundational models.

The paper treats the 98-label mapping as ground truth, but clinical label definitions often suffer from inter-observer variability and semantic ambiguity across global health institutions. The paper does not thoroughly analyze how noisy or imperfect label harmonization impacts the schema-balancing weight matrix W=[wsc]W = [w_{sc}].

Reviewer_vWkVOpenReview ↗
Rating
4
Confidence
2
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

Learning from Heterogeneous Annotation Mechanisms for Unified Retinal Prediction

Critical strengths

The paper addresses the important challenge of heterogeneous and incomplete annotations in multi-source retinal datasets, proposing an effective framework that achieves substantial performance improvements across diverse retinal prediction tasks.

Critical weaknesses

As shown in Table 3, either component alone achieves nearly all the performance gains, while their combination brings only marginal improvements. Please further clarify the complementarity of the two components.

The significance markers (****) in Figure 4 are not explicitly explained in the figure caption. Please clarify their meaning for better readability.

The comparisons mainly involve retinal foundation models and general multi-label learning methods. Please consider comparing AMA with disease-specific models, such as Vision-Language Models Guided Graph Concept Reasoning for Interpretable Diabetic Retinopathy Diagnosis (AAAI 2026), or at least discuss their respective advantages and limitations.

Questions for authors

As shown in Table 3, either component alone achieves nearly all the performance gains, while their combination brings only marginal improvements. Please further clarify the complementarity of the two components.

The significance markers (****) in Figure 4 are not explicitly explained in the figure caption. Please clarify their meaning for better readability.

The comparisons mainly involve retinal foundation models and general multi-label learning methods. Please consider comparing AMA with disease-specific models, such as Vision-Language Models Guided Graph Concept Reasoning for Interpretable Diabetic Retinopathy Diagnosis (AAAI 2026), or at least discuss their respective advantages and limitations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.