Open-World Zero-Shot Image Classification using an LLM-guided Discovery Loop

Contrastive Vision-Language models such as CLIP have enabled zero-shot image classification using only the pretrained model and a complete list of category names, vastly lowering labeling costs compared to traditional supervised approaches. However, in practice the full class list may not be available, creating an obstacle to applying CLIP-style models. Vocabulary-free image classification (VIC) approaches this problem by trying to infer the class list from just the images, however VIC is fundamentally ambiguous as images may be classified according to different attributes and granularities, each of them equally valid. We therefore propose Open-World Zero-Shot Image Classification (OWIC) as a well-defined alternative setting in which some category names are given, and the task is to discover the remaining ones based on the pattern defined by the existing classes. To tackle this task, we propose Multimodal Agentic Novelty Detection (MAND), an LLM-driven discovery loop in which a multimodal LLM identifies novel-class images and proposes category names for them that are then used by a contrastive image-text model such as CLIP. A similarity-based stopping criterion helps minimize LLM usage to lower compute cost. Once all category names are discovered the LLM can be discarded, maintaining deployment efficiency. Experimental results show MAND outperforming strong baselines based on vocabulary-free methods adapted to this new task. In addition to leveraging pretrained models, we also show that finetuning can further improve the LLM's performance within the discovery loop, generalizing even to domains beyond the training data. Code will be published.