acceptodds
Withdrawn from ICLR 2027

Multi-Label Class-Incremental Learning for Ophthalmic Diagnosis over a Recurring Population

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Many real-world problems require that each example be assigned to multiple non-exclusive labels, a task known as multi-label classification. However, most of the problems considered in the context of continual learning, including the class-incremental learning (CIL) scenario, assume a single-label classification task, which does not allow an object to be assigned multiple labels simultaneously. This paper addresses that gap by introducing a multi-label incremental learning scenario that removes the assumptions of disjoint tasks and mutually exclusive classes. This scenario is particularly relevant in oculomics, which studies the relationship between ophthalmic biomarkers and systemic health or diseases. In this field, new sets of diagnoses often coexist with existing classes rather than replacing them, and re-annotation for each new problem is rarely feasible, leaving most historical images only partially labeled. We identify and formalize this scenario as a multi-label class-incremental learning problem over a recurring population, with a dynamic label space and partial relabeling, in which each label may be positive, negative, or unobserved. The label space thus expands from task to task, and training examples recur across different tasks as new sets of labels are introduced; to our knowledge, this combination has not been studied before. We evaluate the setting experimentally on the ODIR-5K dataset, a real-world multi-label ophthalmology benchmark, by instantiating the case in which every introduced label is observed, so that missingness is driven solely by the schedule. Buffer-free continual learning baselines are compared against joint and naive bounds. About half of the gap between sequential and joint training is due to forgetting, and half to lost co-training, which no buffer-free method recovers. Refitting the decision layer on an unchanged backbone recovers three-quarters of the gap — placing the residual loss in the heads rather than in the shared representation. Our findings guide the implementation and maintenance of diagnostic models whose label sets evolve.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
class-incremental learning, multi-label classification, catastrophic forgetting, dynamic label space, oculomics, retinal fundus images

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_veGgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problems of traditional incremental learning, which mainly assumes mutually exclusive categories and non-overlapping samples from different tasks. It studies a multi-label incremental learning task for the diagnosis of ophthalmic diseases. This framework allows fundus images from the same patient to reappear across different incremental stages and enables the label space to expand as new disease categories are introduced, while explicitly distinguishing between unobserved labels and true negative labels.

Using a masked binary cross-entropy loss, the study evaluates several methods on the ODIR-5K dataset, including sequential fine-tuning, freezing the feature extractor, Elastic Weight Consolidation (EWC), output anchoring, and Knowledge Recovery and Transfer (KRT). The experiments reveal that the performance degradation observed in sequential learning—compared to joint training—stems from both catastrophic forgetting and the absence of joint supervision; furthermore, the order of tasks and the sample size per task significantly influence model performance. Additionally, by retraining the classification head, the authors demonstrate that a portion of the performance loss can be recovered without modifying the feature extractor, highlighting the critical role of the classification decision layer in the incremental learning process.

Critical strengths
  1. The task definition is well-defined. The paper provides a formal description of a dynamically expanding label space, distinguishing between labels that have not yet been introduced, unrecorded labels, and confirmed negative labels.

  2. The experimental analysis is thorough. Beyond comparing the final performance of various continual learning strategies, the paper decomposes the performance gap into two components: catastrophic forgetting and the absence of joint training.

  3. The causes of model degradation are analyzed in depth. Through experiments involving the retraining of classification heads—along with metrics such as ECE and FPR—the paper further examines the relationship between performance decline, the classification decision layer, and model calibration.

Critical weaknesses
  1. The technical innovation at the methodological level is relatively limited. The primary contribution of the paper lies in proposing a new task setting for multi-label class-incremental learning, along with the adaptation and comparison of existing continual learning methods. The techniques employed, such as masked binary cross-entropy and output distillation, are already well-established; the paper lacks novel learning algorithms or optimization mechanisms specifically designed for the proposed scenario. Consequently, this work is better characterized as a study on task formulation and empirical analysis rather than one featuring significant methodological innovation.

  2. The actual coverage of recurring samples is insufficient. Although the paper highlights "recurring populations" as a key feature distinguishing this approach from traditional class-incremental learning, the experimental setup ensures that a sample only enters a given task if it possesses a positive label for that task. Statistical data reveal that only 498 out of 7,000 images carry two or more positive labels, meaning at least 92.9% of the images appear in only one task. As a result, the experiments are still dominated by non-recurring samples, failing to adequately represent the "recurring population" learning scenario emphasized by the authors.

  3. The proposed mechanism for handling missing labels lacks direct experimental validation. In its problem definition, the paper allows for labels to be missing due to the unavailability of annotations, explicitly distinguishing between unobserved labels and negative labels. However, the ODIR-5K dataset used in the experiments is processed such that labels for all classes are available; the study only accounts for the temporary absence of labels caused by the sequential introduction of classes, failing to simulate the instance-level annotation incompleteness commonly encountered in real-world medical scenarios.

  4. The practical utility of the experiments involving the retraining of classification heads is limited. The paper demonstrates an increase in MAP from 0.278 to 0.489 by retraining the classification heads using fully labeled data, leading the authors to conclude that most performance degradation stems from the classification decision layer rather than the shared representation layer. However, this experiment relies on complete historical label information, data that would be unavailable during a standard incremental learning process. Therefore, the observed performance recovery may stem not only from the recalibration of the classification heads but also from additional joint supervision and a different distribution of training samples.

Questions for authors

Please see the weaknesses.

Reviewer_AATUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper defines "oculomics" as a problem of "multi-label, class-incremental learning based on repeated sample populations," and compares a "buffer-free" continual learning baseline against "joint training" and "naive" approaches. The experiments reveal that retraining the decision layer while keeping the backbone network fixed can recover three-quarters of the performance loss.

Critical strengths

S1: A reasonably sound problem definition.

S2: The illustration in Figure 1 is quite good; it’s very detailed.

Critical weaknesses

W1: This paper only has a method diagram and lacks a motivation diagram to highlight the difference between this method and other methods.

W2: This paper neither proposes a new method nor offers insightful conclusions regarding existing ones. It merely presents a numerical comparison of previously well-established techniques—specifically "joint training" and "naive bounds"—within the context of a new application. Furthermore, the paper fails to explain why the performance gap stems equally from "forgetting" and the "loss of co-training effects," nor does it address why this gap cannot be bridged by "cache-free" methods.

Questions for authors

Questions in W2.

Reviewer_pRKHOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies multi-label class-incremental learning in medical imaging, where new disease labels are added over time and the same patient images may appear in different tasks. The authors focus on a setting where images can have multiple diseases and some labels may be missing. They evaluate several continual learning methods on the ODIR-5K dataset, including fine-tuning, frozen backbones, EWC, anchoring, and KRT. The results show that both forgetting and the lack of joint training contribute to the performance gap. Anchoring and KRT reduce forgetting, while retraining only the classification heads recovers much of the lost performance. The paper also examines how task order and task size affect the results and finds that larger tasks introduced later can help preserve previously learned labels.

Critical strengths

** The paper addresses multi-label incremental learning with recurring patients and distinguishes missing labels from negative labels. ** The paper examines why sequential learning performs worse than joint training, rather than only reporting accuracy. ** The head-refitting experiment shows that much of the lost performance can be recovered without changing the backbone.

Critical weaknesses

** The paper only uses one dataset (ODIR-5K) and one backbone (ResNet-18), making it difficult to know whether the findings generalize. This is important because some conclusions, such as the frozen ImageNet backbone performing better than the adapted backbone and most of the performance loss coming from the classification heads, may not hold with other models or datasets. Testing at least one more backbone and dataset would make these findings more convincing. ** The introduction discusses partially labeled medical images, but in the experiments, all introduced labels are available whenever an image appears again (Section 4). So this part of the problem is not really tested. It would be useful to test the approach when some labels are still missing. ** The authors conclude that task size explains the effect of task order. The improvement from reversing the order drops from +0.120 to +0.034 mAP just by reducing the training data, even before making the tasks equal in size. This makes it hard to tell whether the change comes from task size or the smaller training budget. It would help to test different final-task sizes while keeping the earlier tasks unchanged. ** The authors report a gap of 0.283 mAP between joint and sequential training. They attribute 0.151 to forgetting and the remaining 0.132 to lost joint training. However, it is unclear whether the remaining gap is entirely due to the lack of joint training or whether other factors contribute. More analysis would help support this explanation.

Questions for authors

Please check the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.