Unsupportedness Is Not Usefulness: Task-Relevant Information Acquisition for Open-World Active Learning

Open-world active learning requires selecting annotations from pools that contain known classes, task-relevant novel classes, and irrelevant out-of-distribution (OOD) samples. Existing acquisition strategies often rely on representation novelty, uncertainty, or coverage as proxies for annotation value. We show that this assumption can fail: samples that are highly unsupported by the labeled set may correspond either to useful novel concepts or to irrelevant distribution shifts. We formalize this mismatch through unsupported directions and introduce Task-Relevance-conditioned Information Gain (TRIG), an acquisition framework that separates task relevance from information value. TRIG learns relevance from queried feedback and incorporates it as a continuous weight inside a batch log-determinant information objective, rather than using relevance as a hard filtering criterion. On a controlled ImageNet-200 open-world benchmark, TRIG achieves the strongest mean final in-task top-1 accuracy among comparable non-oracle selectors, discovers all 80 novel classes, and achieves a favorable Pareto trade-off among task performance, discovery, and irrelevant OOD annotation cost. The same behavior persists across DINOv2 and CLIP (Contrastive Language–Image Pre-training) representations and against recent open-set active-learning strategies. These results show that open-world acquisition should explicitly model task relevance instead of treating representation novelty as a surrogate for annotation value.