- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Open-World Zero-Shot Image Classification (OWIC), a setting in which only a subset of category names is provided, while the remaining categories must be discovered without sample-level annotations. To address this task, the authors propose Multimodal Agentic Novelty Detection (MAND), which combines a multimodal large language model with a contrastive vision-language model. MAND iteratively identifies images from potentially novel categories, generates category names, and filters duplicate categories using CLIP-based similarity. The framework further incorporates prior correction and a similarity-gap-based stopping criterion to improve category discovery and reduce LLM inference costs. After discovery, the category prototypes are refined, enabling classification without the LLM at deployment. Experiments on ten image classification datasets compare MAND with adapted vocabulary-free classification methods, including CaSED and FiNDR. The authors also provide component ablations, comparisons of stopping criteria, sensitivity analyses, and an optional LLM finetuning pipeline.
Critical strengths
- A practically motivated problem formulation. The proposed OWIC setting addresses a meaningful limitation of conventional zero-shot classification, where a complete category vocabulary is typically assumed to be available. Providing partial category information without sample-level annotations is a reasonable setting that may facilitate category discovery in practical applications. The distinction from fully vocabulary-free classification is clearly motivated.
- Relatively comprehensive experimental evaluation. The paper evaluates the proposed framework on ten datasets spanning different visual domains and includes comparisons with adapted CaSED and FiNDR baselines. The component ablations and comparisons of stopping criteria provide useful insights into the framework. In particular, the similarity-based stopping criterion achieves competitive classification accuracy while requiring substantially fewer discovery steps than the RichnEst-based alternative.
Critical weaknesses
- Limited methodological novelty and insufficient justification of the proposed task formulation. Although OWIC is presented as a new classification setting, its relationship to existing vocabulary-free classification and generalized category discovery frameworks raises questions about the significance of the conceptual contribution. Furthermore, the claim that partial category names make the classification problem well-defined is not fully established, since a limited set of known names may still be consistent with multiple classification rules or levels of granularity. Methodologically, MAND mainly integrates existing techniques, including LLM-based category generation, CLIP-based similarity filtering, prior correction, and prototype refinement. While their integration is reasonable, the paper does not clearly establish a sufficiently distinctive algorithmic contribution beyond this combination of established components.
- Insufficient experimental evidence for the necessity and practical advantages of the proposed framework. Although the experiments cover multiple datasets, they do not sufficiently establish whether the proposed iterative discovery pipeline is necessary compared with simpler alternatives. In particular, a direct MLLM-based category generation baseline, followed by CLIP-based filtering and classification, would help clarify the benefits of the proposed discovery loop. Moreover, the evaluation primarily uses a fixed 50% known-category split, without systematically investigating the effects of different known-category proportions or class distributions. The additional benefit of finetuning is also limited: Table 2 reports 75.1% cAcc for the pretrained model and 75.0% for the finetuned model before reference images are introduced, suggesting that the improvement of the final finetuned configuration cannot be attributed to finetuning alone. Finally, the efficiency analysis mainly reports discovery steps rather than end-to-end runtime or computational cost, leaving the practical efficiency advantage insufficiently demonstrated.
Questions for authors
- Task formulation: How do the authors justify the claim that partial category names uniquely determine the intended classification rule? Could the authors clarify under what conditions the ambiguity is sufficiently reduced, particularly when only a small or semantically narrow subset of categories is known?
- Simpler baselines: How does MAND compare with a simpler approach that directly prompts an MLLM to infer the classification rule and generate candidate categories from a subset of images and the known category names, followed by CLIP-based filtering and classification? Such a comparison would help establish the necessity of the iterative discovery loop.
- Finetuning effectiveness: Table 2 shows that the finetuned model without reference images achieves 75.0% cAcc, compared with 75.1% for the pretrained model under the corresponding configuration. Could the authors clarify the independent contribution of finetuning and reference images, and explain why finetuning provides limited benefits without the additional visual context?
- Robustness and computational cost: How sensitive is MAND to the proportion of known categories, especially when substantially fewer than 50% are provided? Additionally, could the authors report the end-to-end computational cost, including total LLM calls and runtime, to better substantiate the claimed efficiency advantages?
2 comments
Here is the second part of our response:
Q2: We add a new method designated as MAND SP (single pass) based on your suggestions. MAND SP combines elements of FiNDR and MAND to create a method that processes the dataset in a single pass without a discovery loop, while keeping the filtering and refitting logic of MAND (see appendix C.2 for details). The results in appendix D.2 show that while MAND SP underperforms standard MAND on cluster accuracy, especially on novel classes (mean 65.3% for SP vs 70.2% for MAND pre), it achieves stronger results on semantic accuracy, about half of that driven by a 10-point improvement on EuroSAT. This seems to be a case of MAND SP not proposing many novel classes (1.6 novel classes detected on average for MAND SP vs 6.4 for MAND pre) for a dataset where other methods propose low-similarity names, resulting in MAND SP achieving better sAcc. Compared to FiNDR, MAND SP clearly outperforms on both cAcc and sAcc, showing the impact of our filtering pipeline. Note that we set the discovery set size based on MAND pre step counts to ensure equal budget, however in practice the budget would have to be set blindly, so MAND SP in this form is not practical.
Q4: We provide various performance metrics for MAND and FiNDR in appendix D.10 (CaSED uses no LLM and as a result has negligible cost). The results show that while MAND requires more wall-clock time than FiNDR, the time spent running LLM calls is very similar. A detailed inspection of LLM usage patterns reveals that MAND makes more LLM calls, but generates fewer tokens with them owing to the decision stage that discards known-class samples with a single-token response. Our efficiency claim concerns the stopping criterion relative to alternative criteria and to processing all images. MAND is not cheaper than FiNDR in wall-clock time, but uses comparable LLM time and about 30× fewer generated tokens. As the discovery stage is a one-time cost, the observed wall clock times are not problematic in practice as inference remains cheap regardless of dataset size.
Thank you for your review comments, we will give a general response in the main rebuttal comment, while addressing specific points here.
W1, Q1:
You are correct that our claim regarding task identifiability was too broad and not well-supported, we have revised the paper and included a detailed discussion on task identifiability in Section 3.1. The discussion shows that under some assumptions the known categories can substantially reduce ambiguity, and thus enable discovery. The degree to which ambiguity is reduced depends on the attribute variation among the known categories.
Regarding the conceptual significance of OWIC as a task, we believe that there are many practical tasks that fit OWIC, but not any existing zero-shot setting. Examples include surveying the species that occur in an area from a wildlife camera database, tracking failure types in a factory, or categorizing media libraries. In all these cases, we will usually have an initial category set that we are looking to extend with novel categories that are present in the data. Standard zero-shot classification methods will be unable to discover the novel categories, while VIC methods may not follow the classification scheme of the known classes. OWIC covers the requirements of these problems better than either setting, showing its real-world relevance. Compared to GCD, OWIC differs in that it requires to labeled data, and permits no per-dataset training.
Regarding the methodological novelty, there are several parts of MAND that are novel, including the CLIP-based prior estimation and recall gate, the granularity-adaptive duplicate filter, the similarity-based stopping criterion, and the hierarchical task sampling for the finetuning. As shown in Table 2, these components are critical to the method's accuracy, and especially important for preventing overprediction of novel categories, increasing cluster accuracy from 67.4% to 75.1% and decreasing the mean category error from 157.2% to 19.1%.
W2, Q2, Q3:
The finetuned model outperforms the pretrained one considerably in the baseline configuration (see Table 2). However, adding additional improvements such as the duplicate filter erodes this advantage, showing that these improvements improve the methods along similar lines as the finetuning. Concretely, the improvements mainly address the model's tendency to overpredict categories, whereas this problem is much less severe in the finetuned model, so it already performs well without them. In addition, the finetuned model can make use of the reference images, which the pretrained model cannot do. This enables the finetuned model to leverage more information to solve the task, achieving a novel-class accuracy gain of 3.9% in our main results. Also, we show in our revision that the gap between the pretrained and the finetuned model grows as the novel percentage increases, further strengthening the case for finetuning.
As for the additional simple baseline, we will provide the requested baseline in the second revision.
Q4: We have provided a study in which the known-class percentage is varied between 10% and 90% in the revised paper. The results in Figure 4 show that accuracy declines gracefully as the proportion of known classes decreases. The requested performance benchmarks will be provided in the second revision.