- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the problems of traditional incremental learning, which mainly assumes mutually exclusive categories and non-overlapping samples from different tasks. It studies a multi-label incremental learning task for the diagnosis of ophthalmic diseases. This framework allows fundus images from the same patient to reappear across different incremental stages and enables the label space to expand as new disease categories are introduced, while explicitly distinguishing between unobserved labels and true negative labels.
Using a masked binary cross-entropy loss, the study evaluates several methods on the ODIR-5K dataset, including sequential fine-tuning, freezing the feature extractor, Elastic Weight Consolidation (EWC), output anchoring, and Knowledge Recovery and Transfer (KRT). The experiments reveal that the performance degradation observed in sequential learning—compared to joint training—stems from both catastrophic forgetting and the absence of joint supervision; furthermore, the order of tasks and the sample size per task significantly influence model performance. Additionally, by retraining the classification head, the authors demonstrate that a portion of the performance loss can be recovered without modifying the feature extractor, highlighting the critical role of the classification decision layer in the incremental learning process.
Critical strengths
-
The task definition is well-defined. The paper provides a formal description of a dynamically expanding label space, distinguishing between labels that have not yet been introduced, unrecorded labels, and confirmed negative labels.
-
The experimental analysis is thorough. Beyond comparing the final performance of various continual learning strategies, the paper decomposes the performance gap into two components: catastrophic forgetting and the absence of joint training.
-
The causes of model degradation are analyzed in depth. Through experiments involving the retraining of classification heads—along with metrics such as ECE and FPR—the paper further examines the relationship between performance decline, the classification decision layer, and model calibration.
Critical weaknesses
-
The technical innovation at the methodological level is relatively limited. The primary contribution of the paper lies in proposing a new task setting for multi-label class-incremental learning, along with the adaptation and comparison of existing continual learning methods. The techniques employed, such as masked binary cross-entropy and output distillation, are already well-established; the paper lacks novel learning algorithms or optimization mechanisms specifically designed for the proposed scenario. Consequently, this work is better characterized as a study on task formulation and empirical analysis rather than one featuring significant methodological innovation.
-
The actual coverage of recurring samples is insufficient. Although the paper highlights "recurring populations" as a key feature distinguishing this approach from traditional class-incremental learning, the experimental setup ensures that a sample only enters a given task if it possesses a positive label for that task. Statistical data reveal that only 498 out of 7,000 images carry two or more positive labels, meaning at least 92.9% of the images appear in only one task. As a result, the experiments are still dominated by non-recurring samples, failing to adequately represent the "recurring population" learning scenario emphasized by the authors.
-
The proposed mechanism for handling missing labels lacks direct experimental validation. In its problem definition, the paper allows for labels to be missing due to the unavailability of annotations, explicitly distinguishing between unobserved labels and negative labels. However, the ODIR-5K dataset used in the experiments is processed such that labels for all classes are available; the study only accounts for the temporary absence of labels caused by the sequential introduction of classes, failing to simulate the instance-level annotation incompleteness commonly encountered in real-world medical scenarios.
-
The practical utility of the experiments involving the retraining of classification heads is limited. The paper demonstrates an increase in MAP from 0.278 to 0.489 by retraining the classification heads using fully labeled data, leading the authors to conclude that most performance degradation stems from the classification decision layer rather than the shared representation layer. However, this experiment relies on complete historical label information, data that would be unavailable during a standard incremental learning process. Therefore, the observed performance recovery may stem not only from the recalibration of the classification heads but also from additional joint supervision and a different distribution of training samples.
Questions for authors
Please see the weaknesses.