- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies open-world active learning, where the unlabeled pool contains known classes, task-relevant novel classes, and irrelevant OOD samples. It addresses the problem that representation novelty alone cannot distinguish useful novel concepts from irrelevant distribution shifts. The proposed TRIG method combines a learned task-relevance score, predictive uncertainty, and a log-determinant batch-diversity objective to select samples and expand the task ontology. Experiments on ImageNet-200 and two secondary datasets show improved final classification accuracy and novel-class coverage in the authors' protocol.
Critical strengths
-
The paper highlights an important limitation of novelty-based active learning. A sample far from the labeled data may represent either a useful new concept or an irrelevant distribution shift.
-
The experimental study is broad, covering multiple datasets, two pretrained representation families, component ablations, sensitivity analyses, and comparisons with several active-learning approaches. The paper also provides an implementation and explicitly discuss some limitations of their evaluation.
Critical weaknesses
- The annotation protocol assumes strong feedback. After each query, the method needs to know whether a sample is task-relevant or irrelevant OOD. This moves a central difficulty of open-world learning to the annotator. While known classes can be identified from the active ontology, distinguishing a task-relevant novel class from irrelevant OOD requires knowledge of a broader task scope that a real annotator may not have. The paper does not explain how this feedback would be defined or collected in practice.
- The K/N/O benchmark may capture dataset provenance rather than task relevance. All known and relevant novel classes come from ImageNet-200, whereas irrelevant samples come from external datasets. The relevance model may therefore exploit dataset-specific visual patterns instead of learning semantic membership in the evolving task. The ImageNet-R, ImageNet-V2, and ImageNet-C diagnostics provide some evidence against this concern, but these samples remain outside the acquisition pool. The paper provides no qualitative examples, feature-space analysis, relevance error analysis or acquisition experiment that would more directly rule out this shortcut.
- The proposed method does not follow naturally from the motivating analysis. The unsupportedness argument motivates modeling task relevance, but it does not explain why annotation value should further include predictive uncertainty and batch diversity, or why these three components should be combined through a relevance-weighted log-determinant objective. Table 4 shows that entropy improves Global accuracy, AULC, and Discovery AUC, but it also increases the OOD query rate from 0.1298 to 0.6065. The paper does not provide a principled argument for why these gains justify the additional annotation cost or why the proposed combination is preferable to alternative formulations.
- The experiments do not clearly justify the claimed favorable tradeoff. On the main benchmark, TRIG achieves higher final Global accuracy than random sampling, but random sampling has better AULC, Discovery AUC, and OOD query rate, while both methods discover all 80 novel classes.
- Figure 1 is schematic, while the supporting quantitative evidence is deferred to Appendix Table 6 and does not directly reproduce the ranking depicted in the figure. Important details such as the rank used for the hard null-space score, the exact commands used for the reported experiments, and the frozen pool manifests are not provided.
- The presentation has recurring clarity problems. K/N/O is used before it is formally defined, and the same symbols are reused for data subsets and representation subspaces. Several metrics and protocol terms are also introduced only after they have already appeared in the discussion or results. These issues make the experimental logic and mathematical notation unnecessarily difficult to follow.
Questions for authors
-
In the main benchmark, all known and relevant novel classes come from ImageNet-200, while irrelevant samples come from external datasets. Do the authors have any acquisition results where relevant and irrelevant classes come from the same dataset to rule out this shortcut?
-
What principle motivates the specific combination of task relevance, predictive uncertainty, and batch diversity?