- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses generalized category discovery (GCD) in the framework of biodiversity monitoring, especially under a limited expert annotation budget. To this end, the paper initially defines three application scenarios: discovering new categories, annotating samples from new categories, and training classifiers to recognize these new categories, which could benefit from GCD in biodiversity monitoring. Within this scope, the authors introduce Sciurus, a GCD-based sample selection framework to identify informative samples: Sciurus trains a GCD model for clustering, filters out seen category samples with a confidence threshold, and then repeatedly selects one sample per cluster with a query strategy. A sorting strategy ranks these candidates, and selection continues until the annotation budget is over. In addition, this paper also proposes the metric of homogeneity in place of clustering accuracy due to its limited representativeness for downstream performance. The experimental analysis includes three complementary datasets and backbones together with various GCD methods, showing the effectiveness of the best Sciurus configuration compared to active-learning baselines and random sampling.
Critical strengths
The GCD motivation for biodiversity monitoring, its formulation through well-defined applications, challenges, and the solutions with corresponding claims are all well explained in an easy-to-understand manner. In detail, the evaluation of GCD for downstream use under annotation budget constraints is realistic, which can open up new directions for the biodiversity monitoring community with significant operational impact potential.
The discovery rate results on the first application, which is the most informative test bed for the quality of sample selection with relatively less interference from sample order, features, etc., show the effectiveness of Sciurus across various levels of annotation budgets over three datasets compared to active learning baselines. This also supports one of the main claims of the paper: discovering all new categories does not require every observation to be assigned to the correct cluster.
Critical weaknesses
One of my main concerns about the paper is whether the main results in Figure 2 are fair across Sciurus and baselines. For each dataset and budget, the Sciurus performance is reported as the maximum metric of a quite large number of configurations over GCD methods, backbones, fine-tuning/query/sorting strategies, without using any cross-validation technique or a separate hold-out set, I assume. The reported baseline results are the maximum of five methods, each with three backbones, without considering the possible improvements of adapting query/sorting/filtering strategies, or even clustering used in Sciurus for these methods. To fairly assess the effectiveness of Sciurus, I would expect to see a single configuration chosen, for instance, on a separate set of one dataset, or via a cross-validation technique, and then always comparing this configuration with the baselines. Also, the quite low results of the baseline on Applications 2 and 3, even worse than random selection in some cases, seem to be questionable as well. Especially for Application 3, I can’t fully understand how active learning methods can underperform random selection on 2 out of 3 datasets. Also related to the comparison with other methods, I would expect to see a comparison with Adaptive-Novel due to its high relevance, as also stated by the authors.
My second main concern is that the homogeneity contribution of the paper seems to be overstated, while its novelty is limited. Eq. (4) is already well-known as cluster purity in the machine learning literature. Also, under different names and variants, this metric is widely used in active learning, information retrieval, and general machine learning. However, none of these studies are mentioned in the paper, while it is stated as one of the main contributions of the paper. In addition to the lack of novelty, its motivation over ACC is also questionable, as it can be trivially maximized if samples per class are distributed equally to all the clusters and the number of clusters is high. Thus, homogeneity as an alternative to ACC motivation of the paper requires further justification.
There are also other relatively minor weaknesses.
The related work section mainly focuses on GCD, but lacks discussion about general active learning studies, even though this line of research is highly relevant for budget-constrained applications like biodiversity monitoring and there are already some methods considered as baselines.
There are some typos and English abnormalities (e.g., whil in line 218, the the in line 129, as as in line 141, function function in line 316, etc.), requiring detailed proofreading.
Questions for authors
Can you please clarify whether Sciurus is fairly compared against active learning baselines? Would there be possible improvements by injecting query/sorting/filtering strategies, or even the clustering used in Sciurus, into these methods?
If a single configuration is set on a different dataset/split, etc., what would be the comparison of Sciurus against active learning baselines?
Can you please clarify how the baseline results on Applications 2 and 3 can sometimes be worse than random selection?
Can you please clarify the novelty of the homogeneity metric by stating its difference from class purity?
Can you please further elaborate on the effectiveness of homogeneity over ACC by referring to the trivial high score of homogeneity mentioned above?