- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes SEGA, a framework for Test-Time Discovery (TTD) that replaces anonymous cluster IDs with interpretable semantic concepts.
Critical strengths
Addresses limitations of prior TTD work.
Extensive ablations.
Critical weaknesses
- In Fig. 1, the proposed scenario (c) is different with (b) predefined category vocabulary. However, I find this difference vague: the "predefined category vocabulary" in (c) is actually the vocabulary of the VLM (e.g., CLIP).
Moreover, the reranking step reintroduces a closed vocabulary. So the motivation of the paper is not so clear.
- There are risks of data leakage. CLIP is trained on a large corpus (WebImageText, ~400M image-text pairs) that is known to overlap substantially with standard vision benchmarks. CIFAR-100, CUB-200-2011, Tiny-ImageNet, and FGVC-Aircraft are all widely used datasets, and their class names and visual concepts are almost certainly represented in CLIP's training data.
The LLM (GPT-4o-mini) is also pretrained on web data and almost certainly knows CIFAR/CUB/Tiny-ImageNet/Aircraft class names. When asked to generate candidate names given "similar to tiger, lion; dissimilar to plane, car," it may simply retrieve the correct benchmark label from its own pretraining.
In conclusion, I found the boundary of unseen/seen, unknown/known became highly unclear by mixing pretrained VLM, closed LLM (GPT-4o), and the benchmark datasets by this paper. The paper should have more rigorous ablation analysis for the data leakage.
Questions for authors
The method relies on GPT-4o-mini at test time, which is a proprietary black-box model. Table 14 reports token/time costs, but there is no variance analysis across LLMs. How stable are the results across LLMs?
3 comments
2. Pretraining overlap and online discovery.
a) Pretrained concept knowledge does not by itself solve TTD. “Unknown” refers to classes excluded from the task-specific known-class training set, not necessarily concepts absent from foundation-model pretraining. CLIP may represent these concepts, and an LLM may know their names. SEGA studies how to convert these priors into online discovery and learning. This follows the broader direction of multimodal discovery methods such as TextGCD (Zheng et al., ECCV 2024) and GET (Wang et al., CVPR 2025), which introduce task-specific mechanisms to exploit pretrained knowledge.
b) Naming and online class learning have different responsibilities. In SEGA, the local visual rule first initiates a new class and maintains its prototype for subsequent assignments (Sec. 3.3). The LLM then proposes candidate names. Confidence-guided descriptions support local model updates through the objectives in Sec. 3.4. A plausible name provides a semantic reference; it does not perform these class-initiation, assignment, and learning operations.
c) Controlled experiments support this distinction. The frozen-CLIP and shared-representation controls in , discussed above, support SEGA’s additional contribution beyond the choice of pretrained backbone and the representation learned before deployment. further reports 79.19/74.81 real-time/post TA+CA without online candidate naming, compared with 82.55/77.63 for full SEGA. Thus, online naming is not a prerequisite for discovery in our pipeline, while generated semantics provide additional benefits. Exact naming also remains imperfect, with 22.85% exact Top-1 accuracy in Table 16. This motivates confidence-guided semantic feedback.
These results support SEGA’s contribution beyond access to pretrained knowledge. Familiarity with benchmark concepts should therefore be distinguished from solving online discovery.
3. Sensitivity to the choice of LLM
compares text-assisted naming with direct image-based VLM naming and reports their API costs. To directly address generator dependence, we additionally compare GPT-4o-mini, GPT-6.1 Sol, DeepSeek-V3, and locally deployed Qwen 7B:
| LLM | Exact Top-1 (%) | S² Top-1 | Real-time TA+CA | Post TA+CA |
|---|---|---|---|---|
| GPT-4o-mini API, original model | 22.85 | 55.66 | 82.55 | 77.63 |
| GPT-6.1 Sol API | 27.62 | 61.84 | 85.91 | 80.56 |
| DeepSeek-V3 API | 23.47 | 57.57 | 83.73 | 78.09 |
| Local Qwen2.5-7B-Instruct | 20.48 | 53.36 | 81.02 | 75.91 |
SEGA’s performance is not specific to GPT-4o-mini, although generator choice affects the results. GPT-6.1 Sol improves real-time/post TA+CA by 3.36/2.93 points over the original generator, while local Qwen2.5-7B-Instruct reduces them by 1.53/1.72 points. The latter demonstrates that online candidate naming can also use a locally hosted model.
Exact naming remains imperfect across all four generators, ranging from 20.48% to 27.62%. Together with the no-online-naming ablation in , these results support naming as useful auxiliary guidance whose quality influences TTD performance. This comparison quantifies sensitivity to generator choice; the distinction between pretrained benchmark knowledge and online discovery is addressed in Response 2.
We appreciate the reviewer's time and take the concerns raised in the updated review seriously, particularly those regarding research integrity and the AI Use Statement. However, the ethics comments do not identify specific passages or examples that would allow us to examine and address these concerns. We would therefore appreciate clarification on the following points.
1. Basis for the ethics flag. Our AI Use Statement explicitly discloses the role of GPT-4o-mini in our framework, including attribute description generation and test-time candidate naming. It also discloses the use of generative AI for manuscript language editing and specifies the authors' responsibility for the scientific content. Could you kindly identify which part of this disclosure is considered inaccurate or incomplete, and what specific evidence motivates the ethics concern? We take potential research integrity violations seriously and would like to understand the precise issue being raised.
2. Alleged misrepresentation of prior work. The updated review states that many cited papers do not match their descriptions, but does not identify any specific examples. Could you please point to representative references and the corresponding passages in our manuscript that you consider inaccurate? We would carefully examine and correct any substantiated errors. Without identifiable examples, it is difficult for us to determine what needs to be addressed.
3. Confusion about the theorem. The updated ethics comments criticize a theorem for lacking motivation, proper notation, and interpretation. However, after carefully checking the submitted manuscript, including the appendix, we could not locate any theorem or theorem-proof section. Could you kindly specify the page, equation, or passage to which this criticism refers? If the concern instead relates to a mathematical objective or methodological formulation, we would appreciate clarification so that we can address it appropriately.
4. Outstanding technical concerns and the current recommendation. Our previous rebuttal addressed the distinction between pretrained semantic knowledge and predefined target vocabularies, potential pretraining overlap, and sensitivity to LLM choice. We also provided additional controlled experiments to examine these issues. The motivations and roles of our components are described in Secs. 3.2–3.4, with supporting ablations in Tables 3–4.
We would appreciate knowing which specific technical concerns remain unresolved after our response and whether any of the additional evidence changes the reviewer's assessment.
We welcome rigorous scientific criticism and are willing to address concrete errors or shortcomings. At the same time, we respectfully request that potential research integrity concerns be distinguished from disagreements about technical novelty, methodological complexity, or presentation quality, and be supported by identifiable evidence so that they can be evaluated fairly.
Thank you for your consideration.
Response to Reviewer mTRd
Our objective is test-time discovery (TTD): discovering and learning new classes during testing from a continuous unlabeled stream, while retaining known-class knowledge. Text and LLMs provide auxiliary semantic guidance. Throughout the paper, “unknown” means absent from the task-specific known-class training set; it does not mean that the concept was necessarily absent from CLIP or LLM pretraining.
1. Predefined vocabulary versus generated candidates
We acknowledge that CLIP contains broad pretrained semantic knowledge. However, this is different from supplying an explicit target-category list before deployment. In SEGA, known-class names serve as semantic anchors, while the identities and names of future unknown classes are not provided in advance. This distinction is what Fig. 1 is intended to convey.
a) Access to pretrained CLIP does not by itself explain SEGA’s TTD performance. To illustrate this, Table 3 in the manuscript evaluates frozen CLIP with all proposed components disabled. Using frozen CLIP within the discovery pipeline (first row), its real-time/post TA+CA is 51.73/52.80, compared with 82.51/77.87 for full SEGA. Thus, access to CLIP's pretrained knowledge alone does not provide enough ability for the discovery and new-class learning achieved by SEGA. Attribute-based calibration and semantic feedback make that knowledge useful for TTD.
Appendix K.4/Table 15 provides a stronger control by running DAA’s online discovery policy on SEGA’s trained visual–semantic backbone. This gives DAA access to the representation learned through SEGA’s known-class training, including its visual–semantic alignment. On CIFAR100D, this variant achieves 80.11 real-time TA+CA, compared with 82.82 for full SEGA. Thus, even with the SEGA-trained representation available, the choice of online discovery and learning procedure still affects performance. Together with the frozen-CLIP comparison in Table 3, these results show that SEGA’s gains cannot be explained solely by access to CLIP or by its offline representation learning.
b) Reranking does not require a predefined vocabulary of unknown classes. SEGA reranks candidates generated from the relational and attribute evidence of each discovery event (Eqs. (9)–(11)), without receiving a fixed list of unknown-class names before deployment. Appendix K.5/Table 16 compares dynamically generated candidates with predefined vocabulary alternatives. Appendix K.5/Table 16 compares these different semantic interfaces under TTD. Appendix K.5, Table 16 compares the following semantic interfaces under TTD. The fixed vocabulary's higher exact accuracy (66.74%) is consistent with its easier label-space assumption: its candidate source explicitly contains the ground-truth names. TTD does not assume that future class identities are supplied before deployment. Without that list, SEGA still achieves 82.55/77.63 real-time/post TA+CA through relational inference, attribute-based calibration, and online learning. The external-lexicon control is another possible interface; LLM generation is our choice for producing candidates without precompiling the target-category list. Thus, SEGA does not require a predefined list of future unknown-class names.