acceptodds
Withdrawn from ICLR 2027

Unsupportedness Is Not Usefulness: Task-Relevant Information Acquisition for Open-World Active Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Open-world active learning requires selecting annotations from pools that contain known classes, task-relevant novel classes, and irrelevant out-of-distribution (OOD) samples. Existing acquisition strategies often rely on representation novelty, uncertainty, or coverage as proxies for annotation value. We show that this assumption can fail: samples that are highly unsupported by the labeled set may correspond either to useful novel concepts or to irrelevant distribution shifts. We formalize this mismatch through unsupported directions and introduce Task-Relevance-conditioned Information Gain (TRIG), an acquisition framework that separates task relevance from information value. TRIG learns relevance from queried feedback and incorporates it as a continuous weight inside a batch log-determinant information objective, rather than using relevance as a hard filtering criterion. On a controlled ImageNet-200 open-world benchmark, TRIG achieves the strongest mean final in-task top-1 accuracy among comparable non-oracle selectors, discovers all 80 novel classes, and achieves a favorable Pareto trade-off among task performance, discovery, and irrelevant OOD annotation cost. The same behavior persists across DINOv2 and CLIP (Contrastive Language–Image Pre-training) representations and against recent open-set active-learning strategies. These results show that open-world acquisition should explicitly model task relevance instead of treating representation novelty as a surrogate for annotation value.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
open-world active learning, out-of-distribution detection, information acquisition, novel class discovery, submodular optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7KjSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies open-world active learning, where the unlabeled pool contains known classes, task-relevant novel classes, and irrelevant OOD samples. It addresses the problem that representation novelty alone cannot distinguish useful novel concepts from irrelevant distribution shifts. The proposed TRIG method combines a learned task-relevance score, predictive uncertainty, and a log-determinant batch-diversity objective to select samples and expand the task ontology. Experiments on ImageNet-200 and two secondary datasets show improved final classification accuracy and novel-class coverage in the authors' protocol.

Critical strengths
  • The paper highlights an important limitation of novelty-based active learning. A sample far from the labeled data may represent either a useful new concept or an irrelevant distribution shift.

  • The experimental study is broad, covering multiple datasets, two pretrained representation families, component ablations, sensitivity analyses, and comparisons with several active-learning approaches. The paper also provides an implementation and explicitly discuss some limitations of their evaluation.

Critical weaknesses
  • The annotation protocol assumes strong feedback. After each query, the method needs to know whether a sample is task-relevant or irrelevant OOD. This moves a central difficulty of open-world learning to the annotator. While known classes can be identified from the active ontology, distinguishing a task-relevant novel class from irrelevant OOD requires knowledge of a broader task scope that a real annotator may not have. The paper does not explain how this feedback would be defined or collected in practice.
  • The K/N/O benchmark may capture dataset provenance rather than task relevance. All known and relevant novel classes come from ImageNet-200, whereas irrelevant samples come from external datasets. The relevance model may therefore exploit dataset-specific visual patterns instead of learning semantic membership in the evolving task. The ImageNet-R, ImageNet-V2, and ImageNet-C diagnostics provide some evidence against this concern, but these samples remain outside the acquisition pool. The paper provides no qualitative examples, feature-space analysis, relevance error analysis or acquisition experiment that would more directly rule out this shortcut.
  • The proposed method does not follow naturally from the motivating analysis. The unsupportedness argument motivates modeling task relevance, but it does not explain why annotation value should further include predictive uncertainty and batch diversity, or why these three components should be combined through a relevance-weighted log-determinant objective. Table 4 shows that entropy improves Global accuracy, AULC, and Discovery AUC, but it also increases the OOD query rate from 0.1298 to 0.6065. The paper does not provide a principled argument for why these gains justify the additional annotation cost or why the proposed combination is preferable to alternative formulations.
  • The experiments do not clearly justify the claimed favorable tradeoff. On the main benchmark, TRIG achieves higher final Global accuracy than random sampling, but random sampling has better AULC, Discovery AUC, and OOD query rate, while both methods discover all 80 novel classes.
  • Figure 1 is schematic, while the supporting quantitative evidence is deferred to Appendix Table 6 and does not directly reproduce the ranking depicted in the figure. Important details such as the rank used for the hard null-space score, the exact commands used for the reported experiments, and the frozen pool manifests are not provided.
  • The presentation has recurring clarity problems. K/N/O is used before it is formally defined, and the same symbols are reused for data subsets and representation subspaces. Several metrics and protocol terms are also introduced only after they have already appeared in the discussion or results. These issues make the experimental logic and mathematical notation unnecessarily difficult to follow.
Questions for authors
  • In the main benchmark, all known and relevant novel classes come from ImageNet-200, while irrelevant samples come from external datasets. Do the authors have any acquisition results where relevant and irrelevant classes come from the same dataset to rule out this shortcut?

  • What principle motivates the specific combination of task relevance, predictive uncertainty, and batch diversity?

Reviewer_scp8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies open-world active learning with known classes, task-relevant novel classes, and irrelevant OOD samples. It proposes TRIG, which learns task relevance from queried feedback and combines it with predictive utility in a log-determinant batch objective that accounts for labeled feature coverage. Experiments demonstrate failures of deficiency-only acquisition and show strong final task accuracy, with trade-offs in early learning, discovery, and OOD annotation cost.

Critical strengths
  1. TRIG uses queried task-relevant novel samples to learn relevance before their classes enter the classifier. This allows relevance learning to account for useful novelty beyond the classifier’s current label space.

  2. The evaluation includes component ablations, secondary benchmarks, CLIP representations, and matched-cost analyses. These experiments help explain the roles of relevance, predictive utility, and batch diversity, while making the performance–cost trade-offs explicit.

Critical weaknesses
  1. The experiments do not isolate the benefit of continuous relevance weighting over relevance prescreening. In Table 2, rel-BADGE and TRIG differ in both their base selector and how they incorporate relevance. A comparison using the same log-determinant selector is needed to establish whether continuous weighting itself provides an advantage.

  2. Table 6 needs a clearer explanation of how its diagnostic relates to acquisition. Soft novel-versus-OOD AUROC is below 0.5 in every seed, but the top-250 sets contain only 0–18.8% OOD, including exclusively novel samples for seed 0. These results are compatible, but the table does not clearly identify the score used for the top-250 counts or show the composition at the actual query budget of 2,000. This limits its interpretation as evidence about which samples would be acquired.

Questions for authors
  1. Does continuous relevance weighting outperform hard relevance prescreening when both use the same log-determinant selector, labeled information matrix, and predictive utility?

  2. Which score and candidate population produce the top-250 counts in Table 6? How does the selected composition change at the main batch size of 2,000?

  3. How does TRIG perform when task-relevant shifted images are included in the acquisition pool? In particular, does the limited ImageNet-C recall at low OOD false-positive rates lead to useful shifted samples being overlooked?

Reviewer_JbP5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TRIG, which studies open-world active learning with known classes, task-relevant novel classes, and irrelevant OOD samples. TRIG learns task relevance from queried feedback and combines it with predictive utility in a labelled-set-relative log-determinant batch objective.

Critical strengths
  1. Better than JODA, this method uses queried feedback before novel-class promotion. These samples contribute to both relevance estimation and labelled-set coverage, allowing the acquisition process to use available annotations even when there are not enough examples to add a new class to the classifier.
  2. The method has a clear design that separates task relevance, predictive utility, and batch diversity. Combining these signals in one acquisition objective allows the selector to balance useful novel samples and informative known-class samples.
Critical weaknesses
  1. Innovation. Joda already studies known classes, relevant novel classes, and irrelevant OOD together. MQ-Net learns the purity–informativeness trade-off from queried feedback, and SIMILAR uses submodular objectives for relevance and diversity. TRIG mainly combines relevance weighting, entropy, and a standard log-det objective. The paper does not sufficiently establish the contribution beyond these existing ideas.
  2. The marginal gain depends on the product of relevance, entropy utility, and geometric unsupportedness, so large unsupportedness can outweigh low relevance. Queried OOD samples are excluded from the historical coverage matrix. As a result, irrelevant regions can remain geometrically unsupported across rounds, and the selector must repeatedly rely on the relevance head to reject them, potentially querying similar OOD samples again.
  3. Joda-Reimpl uses frozen features and does not retain the original method’s representation training mechanism. Several other baselines target fixed-class open-set learning rather than novel-class discovery. MQ-Net and SIMILAR are missing, and comparing TRIG with rel-BADGE changes both relevance handling and batch selection.
  4. On ImageNet, TRIG improves final accuracy over Random from 94.50% to 96.37%, but has lower mean AULC and discovery AUC, while the OOD query rate increases from 25.05% to 60.65%. The entropy-free variant still achieves 95.73% accuracy and discovers all 80 novel classes, with only 12.98% OOD queries. With CLIP features, TRIG reduces OOD queries compared with BADGE, but improves final accuracy by only 0.30 percentage points and has lower AULC and discovery AUC. These results show a trade-off, but do not establish a consistent advantage in learning and discovery efficiency.
Questions for authors

See weakness 3-4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.