acceptodds
Withdrawn from ICLR 2027

Relational Semantic Grounding for Test-Time Discovery

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Test-Time Discovery (TTD) aims to recognize known classes and detect novel ones from a continuous unlabeled test stream after deployment. Existing TTD methods mainly rely on visual clustering or prototype comparison, and can only assign anonymous labels to discovered classes, leaving them difficult to interpret in open-world applications. In this paper, we propose SEGA, a Semantic Grounding with Attributes framework for TTD. SEGA leverages large language models to generate attribute-rich descriptions for known classes and constructs an attribute-anchored visual-semantic space during training. At test time, we introduce Relational Semantic Grounding (RSG), which grounds unknown samples with similar and dissimilar known-class anchors, prompts the LLM to generate candidate concepts, and reranks them in the learned semantic space. The selected concept is further used for confidence-guided test-time semantic stabilization. By coupling attribute-level semantic alignment with relational concept grounding, SEGA transforms anonymous test-time clusters into interpretable semantic concepts. Experiments show that SEGA improves TTD baselines with the promise of semantic grounding for open-world. Our code will be available.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Test-Time Discovery, Novel Category Discovery, Online Learning, Test-Time Training

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_mTRdOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes SEGA, a framework for Test-Time Discovery (TTD) that replaces anonymous cluster IDs with interpretable semantic concepts.

Critical strengths

Addresses limitations of prior TTD work.

Extensive ablations.

Critical weaknesses
  1. In Fig. 1, the proposed scenario (c) is different with (b) predefined category vocabulary. However, I find this difference vague: the "predefined category vocabulary" in (c) is actually the vocabulary of the VLM (e.g., CLIP).

Moreover, the reranking step reintroduces a closed vocabulary. So the motivation of the paper is not so clear.

  1. There are risks of data leakage. CLIP is trained on a large corpus (WebImageText, ~400M image-text pairs) that is known to overlap substantially with standard vision benchmarks. CIFAR-100, CUB-200-2011, Tiny-ImageNet, and FGVC-Aircraft are all widely used datasets, and their class names and visual concepts are almost certainly represented in CLIP's training data.

The LLM (GPT-4o-mini) is also pretrained on web data and almost certainly knows CIFAR/CUB/Tiny-ImageNet/Aircraft class names. When asked to generate candidate names given "similar to tiger, lion; dissimilar to plane, car," it may simply retrieve the correct benchmark label from its own pretraining.

In conclusion, I found the boundary of unseen/seen, unknown/known became highly unclear by mixing pretrained VLM, closed LLM (GPT-4o), and the benchmark datasets by this paper. The paper should have more rigorous ablation analysis for the data leakage.

Questions for authors

The method relies on GPT-4o-mini at test time, which is a proprietary black-box model. Table 14 reports token/time costs, but there is no variance analysis across LLMs. How stable are the results across LLMs?

3 comments
AuthorsOpenReview ↗

Response to Reviewer mTRd

Thank you for your feedback. We have provided our point-by-point response below. We also noticed that your rating was revised from 2 to 1. To help us respond more precisely, could you clarify which concern was decisive in your updated assessment, and whether there are any additional concerns we should address?


Our objective is test-time discovery (TTD): discovering and learning new classes during testing from a continuous unlabeled stream, while retaining known-class knowledge. Text and LLMs provide auxiliary semantic guidance. Throughout the paper, “unknown” means absent from the task-specific known-class training set; it does not mean that the concept was necessarily absent from CLIP or LLM pretraining.

1. Predefined vocabulary versus generated candidates

We acknowledge that CLIP contains broad pretrained semantic knowledge. However, this is different from supplying an explicit target-category list before deployment. In SEGA, known-class names serve as semantic anchors, while the identities and names of future unknown classes are not provided in advance. This distinction is what Fig. 1 is intended to convey.

a) Access to pretrained CLIP does not by itself explain SEGA’s TTD performance. To illustrate this, Table 3 in the manuscript\textcolor{red}{\text{Table 3 in the manuscript}}​ evaluates frozen CLIP with all proposed components disabled. Using frozen CLIP within the discovery pipeline (first row), its real-time/post TA+CA is 51.73/52.80, compared with 82.51/77.87 for full SEGA. Thus, access to CLIP's pretrained knowledge alone does not provide enough ability for the discovery and new-class learning achieved by SEGA. Attribute-based calibration and semantic feedback make that knowledge useful for TTD.

Appendix K.4/Table 15\textcolor{red}{\text{Appendix K.4/Table 15}} provides a stronger control by running DAA’s online discovery policy on SEGA’s trained visual–semantic backbone. This gives DAA access to the representation learned through SEGA’s known-class training, including its visual–semantic alignment. On CIFAR100D, this variant achieves 80.11 real-time TA+CA, compared with 82.82 for full SEGA. Thus, even with the SEGA-trained representation available, the choice of online discovery and learning procedure still affects performance. Together with the frozen-CLIP comparison in Table 3, these results show that SEGA’s gains cannot be explained solely by access to CLIP or by its offline representation learning.

b) Reranking does not require a predefined vocabulary of unknown classes. SEGA reranks candidates generated from the relational and attribute evidence of each discovery event (Eqs. (9)–(11)), without receiving a fixed list of unknown-class names before deployment. Appendix K.5/Table 16 compares dynamically generated candidates with predefined vocabulary alternatives. Appendix K.5/Table 16 compares these different semantic interfaces under TTD. Appendix K.5, Table 16 \textcolor{red}{\text{Appendix K.5, Table 16 }} compares the following semantic interfaces under TTD. The fixed vocabulary's higher exact accuracy (66.74%) is consistent with its easier label-space assumption: its candidate source explicitly contains the ground-truth names. TTD does not assume that future class identities are supplied before deployment. Without that list, SEGA still achieves 82.55/77.63 real-time/post TA+CA through relational inference, attribute-based calibration, and online learning. The external-lexicon control is another possible interface; LLM generation is our choice for producing candidates without precompiling the target-category list. Thus, SEGA does not require a predefined list of future unknown-class names.

2. Pretraining overlap and online discovery.

a) Pretrained concept knowledge does not by itself solve TTD. “Unknown” refers to classes excluded from the task-specific known-class training set, not necessarily concepts absent from foundation-model pretraining. CLIP may represent these concepts, and an LLM may know their names. SEGA studies how to convert these priors into online discovery and learning. This follows the broader direction of multimodal discovery methods such as TextGCD (Zheng et al., ECCV 2024) and GET (Wang et al., CVPR 2025), which introduce task-specific mechanisms to exploit pretrained knowledge.

b) Naming and online class learning have different responsibilities. In SEGA, the local visual rule first initiates a new class and maintains its prototype for subsequent assignments (Sec. 3.3). The LLM then proposes candidate names. Confidence-guided descriptions support local model updates through the objectives in Sec. 3.4. A plausible name provides a semantic reference; it does not perform these class-initiation, assignment, and learning operations.

c) Controlled experiments support this distinction. The frozen-CLIP and shared-representation controls in Table 3 in main text and Appendix K.4/Table 15\textcolor{red}{\text{Table 3 in main text and Appendix K.4/Table 15}}, discussed above, support SEGA’s additional contribution beyond the choice of pretrained backbone and the representation learned before deployment. Appendix K.2/Table 13\textcolor{red}{\text{Appendix K.2/Table 13}} further reports 79.19/74.81 real-time/post TA+CA without online candidate naming, compared with 82.55/77.63 for full SEGA. Thus, online naming is not a prerequisite for discovery in our pipeline, while generated semantics provide additional benefits. Exact naming also remains imperfect, with 22.85% exact Top-1 accuracy in Table 16. This motivates confidence-guided semantic feedback.

These results support SEGA’s contribution beyond access to pretrained knowledge. Familiarity with benchmark concepts should therefore be distinguished from solving online discovery.

3. Sensitivity to the choice of LLM

Appendix K.3, Table 14\textcolor{red}{\text{Appendix K.3, Table 14}} compares text-assisted naming with direct image-based VLM naming and reports their API costs. To directly address generator dependence, we additionally compare GPT-4o-mini, GPT-6.1 Sol, DeepSeek-V3, and locally deployed Qwen 7B:

LLMExact Top-1 (%)S² Top-1Real-time TA+CAPost TA+CA
GPT-4o-mini API, original model22.8555.6682.5577.63
GPT-6.1 Sol API27.6261.8485.9180.56
DeepSeek-V3 API23.4757.5783.7378.09
Local Qwen2.5-7B-Instruct20.4853.3681.0275.91

SEGA’s performance is not specific to GPT-4o-mini, although generator choice affects the results. GPT-6.1 Sol improves real-time/post TA+CA by 3.36/2.93 points over the original generator, while local Qwen2.5-7B-Instruct reduces them by 1.53/1.72 points. The latter demonstrates that online candidate naming can also use a locally hosted model.

Exact naming remains imperfect across all four generators, ranging from 20.48% to 27.62%. Together with the no-online-naming ablation in Appendix K.2/Table 13\textcolor{red}{\text{Appendix K.2/Table 13}}, these results support naming as useful auxiliary guidance whose quality influences TTD performance. This comparison quantifies sensitivity to generator choice; the distinction between pretrained benchmark knowledge and online discovery is addressed in Response 2.

We appreciate the reviewer's time and take the concerns raised in the updated review seriously, particularly those regarding research integrity and the AI Use Statement. However, the ethics comments do not identify specific passages or examples that would allow us to examine and address these concerns. We would therefore appreciate clarification on the following points.

1. Basis for the ethics flag. Our AI Use Statement explicitly discloses the role of GPT-4o-mini in our framework, including attribute description generation and test-time candidate naming. It also discloses the use of generative AI for manuscript language editing and specifies the authors' responsibility for the scientific content. Could you kindly identify which part of this disclosure is considered inaccurate or incomplete, and what specific evidence motivates the ethics concern? We take potential research integrity violations seriously and would like to understand the precise issue being raised.

2. Alleged misrepresentation of prior work. The updated review states that many cited papers do not match their descriptions, but does not identify any specific examples. Could you please point to representative references and the corresponding passages in our manuscript that you consider inaccurate? We would carefully examine and correct any substantiated errors. Without identifiable examples, it is difficult for us to determine what needs to be addressed.

3. Confusion about the theorem. The updated ethics comments criticize a theorem for lacking motivation, proper notation, and interpretation. However, after carefully checking the submitted manuscript, including the appendix, we could not locate any theorem or theorem-proof section. Could you kindly specify the page, equation, or passage to which this criticism refers? If the concern instead relates to a mathematical objective or methodological formulation, we would appreciate clarification so that we can address it appropriately.

4. Outstanding technical concerns and the current recommendation. Our previous rebuttal addressed the distinction between pretrained semantic knowledge and predefined target vocabularies, potential pretraining overlap, and sensitivity to LLM choice. We also provided additional controlled experiments to examine these issues. The motivations and roles of our components are described in Secs. 3.2–3.4, with supporting ablations in Tables 3–4.

We would appreciate knowing which specific technical concerns remain unresolved after our response and whether any of the additional evidence changes the reviewer's assessment.

We welcome rigorous scientific criticism and are willing to address concrete errors or shortcomings. At the same time, we respectfully request that potential research integrity concerns be distinguished from disagreements about technical novelty, methodological complexity, or presentation quality, and be supported by identifiable evidence so that they can be evaluated fairly.

Thank you for your consideration.

Reviewer_eJHqOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  1. The paper addresses an important limitation of existing TTD methods, which primarily assign anonymous labels to discovered classes without providing semantic interpretations. Extending TTD toward interpretable category discovery is a valuable research direction that could improve the practical applicability of open-world recognition systems.

  2. The proposed framework combines attribute-enriched visual-semantic alignment, relational concept generation, and confidence-guided test-time adaptation. The use of similarities and dissimilarities to known classes provides a reasonable mechanism for inferring unknown concepts. Experiments across four benchmarks show improvements in several discovery metrics, providing empirical support for the overall approach.

Critical strengths
  1. The paper addresses an important limitation of existing TTD methods, which primarily assign anonymous labels to discovered classes without providing semantic interpretations. Extending TTD toward interpretable category discovery is a valuable research direction that could improve the practical applicability of open-world recognition systems.

  2. The proposed framework combines attribute-enriched visual-semantic alignment, relational concept generation, and confidence-guided test-time adaptation. The use of similarities and dissimilarities to known classes provides a reasonable mechanism for inferring unknown concepts. Experiments across four benchmarks show improvements in several discovery metrics, providing empirical support for the overall approach.

Critical weaknesses
  1. Limited methodological novelty. SEGA mainly combines existing techniques, including LLM-generated attributes, CLIP-based alignment, and pseudo-label learning. Moreover, its semantic reranking relies only on similarities to known classes, without directly considering unknown samples' visual features or dissimilar class anchors, potentially limiting its ability to identify novel categories accurately.

  2. Modest performance gains and unclear efficiency. SEGA achieves relatively small and inconsistent improvements over existing TTD methods, underperforming some baselines on certain metrics. Additionally, the computational cost and scalability of LLM-based generation and test-time adaptation are not sufficiently evaluated, raising concerns about its practicality for online discovery.

Questions for authors

Q1. Eq. (10) only considers similarities to known classes. Why are dissimilar class anchors and unknown samples' visual features excluded? Could the authors provide ablation studies to demonstrate the effectiveness of relational semantic grounding?

Q2. The Semantic Score relies on Sentence-BERT similarity, which may not reflect actual naming accuracy. Could the authors provide exact-match accuracy or human evaluations to better assess the reliability of generated labels?

Q3. What is the computational overhead of LLM-based semantic grounding during inference? Could the authors report inference latency, LLM call frequency, and computational costs compared with HM and DAA to demonstrate its practicality for real-time discovery?

7 comments

Thank you for the questions about the grounding mechanism, semantic evaluation, and online cost. SEGA contributes a way to convert semantic priors into online class discovery and learning: attribute-rich calibration prepares a discriminative reference space, relational inference grounds a newly discovered class, and confidence-guided feedback turns that grounding into local model updates.

W1. Methodological novelty.

SEGA contributes a mechanism for turning uncertain semantic hypotheses into guidance for online class discovery and learning. In TTD, future category identities are unavailable, and the model must establish new classes, learn to recognize subsequent instances, and retain known-class knowledge. Our contribution addresses how language can support these operations through three connected designs.

(1) Attribute-calibrated semantic coordinates for discovery. During known-class training, SEGA jointly optimizes visual discrimination, textual discrimination, and cross-modal alignment using attribute-rich descriptions. This makes known classes serve as semantic references: an unknown sample can be characterized through its position relative to anchors encoding attributes such as shape, texture, and structure. The purpose of this calibration is to prepare a shared coordinate system in which relational inference and subsequent semantic feedback are meaningful.

(2) Relational grounding after local class initiation. The visual rule first establishes a new class and initializes its prototype from the image. RSG then combines similar/dissimilar known-class anchors and appearance cues to generate candidate names, followed by semantic-consistency reranking. Its objective is to select a plausible, discriminative reference in the calibrated text space. This matters because directly selecting text to match an initially confused visual feature can preserve that confusion. A reasonable semantic reference can instead guide the feature toward a more appropriate region during subsequent learning. Naming therefore supports class learning without determining whether the class exists.

(3) Reliability-controlled semantic feedback into online learning. The selected semantics directly affect adaptation. Eq. (12) constructs an attribute-rich description when the candidate is sufficiently consistent with the anchors, and uses a relational description otherwise. Eq. (13) incorporates this reference into cross-modal learning, alongside visual adaptation, replay, and renewal. Consequently, uncertain naming does not require either forcing an unreliable name into training or abandoning new-class learning. This connects candidate generation and selection to the acquisition of local classification capability.

Existing controlled experiments substantiate these contributions. In Sec. 4.3, Table 3\textcolor{red}{\text{Sec. 4.3, Table 3}}, adding semantic reranking to semantic pre-construction and test-time training raises real-time TA+CA from 73.78 to 82.51, while reducing KF from 4.34 to 3.28. Moreover, Appendix K.4, Table 15\textcolor{red}{\text{Appendix K.4, Table 15}} holds the learned representation fixed: DAA operating on SEGA’s trained backbone obtains 80.11 real-time TA+CA on CIFAR100D, compared with 82.82 for full SEGA. These controls identify contributions from the semantic selection and online procedure beyond access to CLIP or the representation learned before deployment.

The methodological contribution is therefore the coupling of calibrated semantic references, relational grounding, and selective feedback to improve online discovery and subsequent class learning.

W1/Q1. Why does Eq. (10) exclude direct image matching and dissimilar-anchor penalties?

Images and dissimilar anchors already determine the evidence supplied to candidate generation (Sec. 3.3, Eqs. (7)–(9)\textcolor{red}{\text{Sec. 3.3, Eqs. (7)–(9)}}). The image retrieves similar/dissimilar known anchors, and its low-level statistics supply appearance cues. Their exclusion from the final scoring formula does not exclude them from RSG.

Eq. (10) checks candidate consistency with the positive-anchor neighborhood in the learned text space. Direct image–text matching instead selects text to fit the current unknown feature, which can be unreliable early in discovery. Our objective is to select a discriminative, semantically reasonable text reference that can help guide that feature during subsequent learning. Dissimilar anchors are useful as coarse exclusion evidence during candidate generation: they help filter out clearly incompatible candidates in Eqs. (7)–(9). Their value as a final additive penalty in Eq. (10) is, however, limited. The most dissimilar anchors are often semantically distant, heterogeneous, or already irrelevant categories that are not plausible candidates in the first place. Subtracting their similarities therefore mainly rewards distance from already irrelevant classes, rather than discriminating among plausible positive neighbors. Early in discovery, such negative similarities can also be noisy because the unknown feature is not yet reliable, so an explicit penalty can introduce interference rather than resolve the correct positive neighborhood. This is why Eq. (10) uses positive-anchor neighborhood consistency for final scoring, while dissimilar anchors remain part of candidate generation.

The local visual rule initiates the class before naming. For a reliable candidate, Eq. (12) constructs an attribute-rich description; otherwise it uses a relation-only description. Eq. (13) aligns the emerging visual class with this text reference, alongside visual learning, replay, and renewal. The LLM supplies a fallible semantic hypothesis; the deployed model performs discovery, assignment, and learning.

The evidence ablation is already in Appendix K.2, Table 13 \textcolor{red}{\text{Appendix K.2, Table 13 }}:

EvidenceS² Top-1S² Max Top-5S² Avg. Top-5Real-time TA+CAPost TA+CA
DAA———77.5472.10
Similar + dissimilar relations, no online LLM naming———79.1974.81
Similar + dissimilar relations, with LLM naming48.1851.1745.5680.1576.22
Relations + attributes, full SEGA55.2558.5751.2882.5577.63

Relation-only guidance supports discovery without online naming (79.19/74.81). Adding candidate names improves agreement, and appearance cues further raise S² Top-1 by 7.07 and real-time TA+CA by 2.40. These results show the complementary roles of relational and appearance evidence.

We also compare our method with direct image-text CLIP scoring and using anchors including negative categories

Candidate scoreExact Top-1 (%)S² Top-1Real-time TA+CAPost TA+CA
Direct image–text, SEGA-aligned encoders21.9254.8280.3575.10
Positive-negative text anchors22.1155.2081.5876.36
Positive-anchor consistency, our SEGA22.8555.6682.5577.63

The positive-anchor score provides the best downstream discovery–retention trade-off in this comparison. Direct visual matching and an explicit negative-anchor penalty do not improve on it. This supports the division of roles: images and negative anchors shape the candidate evidence, while the positive semantic neighborhood determines the reference used for learning.

W2. Modest performance gains

TA and CA capture complementary aspects of discovery: TA measures whether samples from a true novel class are consistently assigned to one discovered label, whereas CA measures the purity of each discovered cluster. Optimizing one alone can cause fragmentation or over-merging. Our claim is therefore a better joint discovery trade-off, measured by TA+CA, rather than a universal improvement on every individual metric.

To quantify whether the joint gains are robust, we additionally ran 10 paired full-stream runs per dataset and protocol. Each pair uses the same known/unknown split, pretrained checkpoint, stream order, and batch partition. TA+CA was pre-specified as the primary discovery metric; we report the paired mean difference, 95% confidence interval, two-sided paired t-test, Cohen's d_z, raw and Holm-corrected p-values, and a paired sign-flip p-value.

DatasetProtocolRunsDAA TA+CA (mean ± SD)SEGA TA+CA (mean ± SD)Delta [95% CI]Paired-difference SDt(9)raw p (two-sided)Holm pd_zsign-flip p (two-sided)
CIFAR100Dreal-time1076.55 ± 1.7683.24 ± 1.376.6899 [5.3804, 7.9994]1.830611.55640.000001060.000002123.65450.00195312
CIFAR100Dpost1072.06 ± 1.1577.74 ± 1.185.6800 [4.2865, 7.0735]1.94809.22040.000007000.000007002.91570.00195312
CUB200Dreal-time10112.75 ± 1.85116.13 ± 1.163.3800 [2.5570, 4.2030]1.15049.29090.000006580.000013152.93800.00195312
CUB200Dpost10108.63 ± 2.89110.60 ± 2.071.9700 [-0.1455, 4.0855]2.95732.10660.064429440.064429440.66620.05664062
Tiny-ImageNetDreal-time1095.14 ± 1.7398.98 ± 1.233.8400 [2.3454, 5.3346]2.08935.81210.000255570.000255571.83800.00195312
Tiny-ImageNetDpost1059.28 ± 1.5363.45 ± 1.134.1700 [3.6440, 4.6960]0.735317.93290.000000020.000000055.67090.00195312
AircraftDreal-time1068.64 ± 1.1270.73 ± 1.392.0900 [0.5021, 3.6779]2.21982.97740.015514400.015514400.94150.00390625
AircraftDpost1057.23 ± 1.2460.42 ± 1.643.1900 [2.4701, 3.9099]1.006410.02330.000003510.000007023.16960.00195312

Seven of the eight paired comparisons remain significant after Holm correction. The post-evaluation CUB200D result is positive but does not reach the 0.05 significance threshold.

Reviewer_35TvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SEGA for test-time discovery, aiming to recognize known classes, discover novel classes, and assign interpretable names to them in an unlabeled image stream. It constructs an aligned visual–semantic space using LLM-generated descriptions of known classes. At test time, relations to known classes and low-level visual attributes are used to generate candidate names, followed by semantic reranking and confidence-guided adaptation. Experiments on four benchmarks evaluate discovery performance, knowledge retention, and the benefits of the proposed components.

Critical strengths
  1. Semantic naming improves the interpretability of category discovery. SEGA integrates name generation, class maintenance, and online adaptation into a coherent pipeline.
  2. Experiments on four benchmarks cover both generic and fine-grained categories. Component ablations and threshold analyses support the overall method’s effectiveness.
Critical weaknesses
  1. The submission does not sufficiently establish its added methodological contribution. FineR already combines attribute-guided LLM naming with CLIP filtering, while TextGCD uses textual knowledge for category discovery. Existing ablations support component usefulness within SEGA, but do not clearly establish the added value of relational inputs and online feedback over comparable prior pipelines.
  2. The central relational design lacks decisive validation. Known-class anchors and HSV, texture, and edge statistics may miss key distinctions between unknown categories. Without a comparison against direct CLIP image–text scoring using the same candidate names, the necessity of indirect reranking through known classes remains unresolved.
Questions for authors
  1. What independent improvements do relational inputs and online feedback provide over a comparable attribute-based naming pipeline?
  2. How does RSG behave when unknown classes have weak semantic relationships with the available known-class anchors?
  3. Have you compared relational reranking with local CLIP image–text scoring using exactly the same candidate names?
  4. Is there evidence that higher reranking scores correspond to more accurate or semantically useful names?
3 comments

Thank you for the questions about the contribution of relational grounding and its role in online class learning.

W1/Q1. Methodological novelty and independent contributions of relational grounding and online feedback

Distinction from prior work. FineR (ICLR 2024) focuses on attribute-guided category naming, while TextGCD (ECCV 2024) performs offline category discovery using textual knowledge. In contrast, SEGA addresses online discovery and learning from a continuous unlabeled stream, where generated semantics not only interpret newly discovered classes but also guide their subsequent learning. Our contribution lies in integrating attribute-calibrated semantic anchors, relational concept grounding, and confidence-guided semantic feedback into a unified TTD framework.

Independent component contributions. The existing experiments provide complementary evidence:

  1. Table 3 (Sec. 4.3) isolates test-time training and semantic reranking. Adding test-time training to semantic pre-construction improves real-time/post TA+CA from 59.65/56.80 to 73.78/68.09. Further adding reranking improves performance to 82.51/77.87, while reducing known-class forgetting from 4.34 to 3.28.
  2. Table 13 (Appendix K.2) examines relational guidance and semantic evidence. Relations without online naming achieve 79.19/74.81. Adding LLM naming improves performance to 80.15/76.22, while incorporating appearance attributes further increases it to 82.55/77.63, with S² Top-1 improving from 48.18 to 55.25.
  3. Table 15 (Appendix K.4) controls for representation quality by applying DAA's online discovery policy to the same SEGA-trained backbone. On CIFAR100D, this achieves 80.11, compared with 82.82 for full SEGA, demonstrating additional gains beyond the learned representation.

Together, these controlled comparisons establish the contributions of online training, semantic reranking, and relationally informed naming, while showing that SEGA's improvements cannot be explained solely by pretrained knowledge or representation learning.

W2/Q2. Weak relationships to available anchors.

Discovery and semantic naming serve distinct roles in SEGA. Novel class detection is based on visual similarity to existing prototypes (Eq. 6), rather than requiring a correct semantic relationship with known classes. Thus, an unknown class that is visually distinct can still be discovered and assigned a new prototype even when its semantic relationships with known-class anchors are weak.

Weak semantic relationships primarily affect naming quality and subsequent semantic feedback. When informative anchors are unavailable, RSG may generate less accurate candidate names. To mitigate this uncertainty, our confidence-guided mechanism (Eq. 12) uses explicit predicted names only when the reranking score exceeds the reliability threshold. Otherwise, it falls back to relational descriptions without the predicted class name, while visual adaptation, replay, and renewal continue.

Existing experiments support the utility of this fallback mechanism. In Sec. 4.3 (Table 4), using only relational feedback (θ=1) achieves 79.05/75.39 real-time/post TA+CA, compared with 82.51/77.87 when reliable predicted names are selectively incorporated (θ=0.8). This shows that explicit semantic naming is not essential for the discovery and learning pipeline to operate, although reliable names provide additional benefits.

We acknowledge that weak relationships with all available anchors can limit semantic naming accuracy. Our approach therefore aims to reduce dependence on uncertain semantic predictions rather than guarantee correct naming in such cases. A dedicated evaluation conditioned on anchor similarity would further quantify this limitation.

W2/Q3. Direct local CLIP scoring with exactly the same candidate names.

We compare the two selection rules directly. Image–text scoring selects the candidate label closest to the current unknown-image representation. Eq. (10) instead measures a candidate's consistency with the mean positive-anchor embedding in the calibrated text space. The distinction is which representation guides the choice: matching text to the current image feature, or selecting a discriminative semantic reference to guide subsequent visual learning.

Known-class training aligns images with attribute-rich, class-discriminative descriptions (Sec. 3.2, Eqs. (2)–(4)\textcolor{red}{\text{Sec. 3.2, Eqs. (2)–(4)}}). An unknown class's early visual representation can still be uncertain. Selecting a name solely because it best matches that representation may carry its initial error into semantic feedback. Reranking uses the learned semantic neighborhood as an additional constraint, seeking a plausible text-space position that remains distinguishable from known classes. A reliable selected name then conditions the description used in Eq. (13). This is why the relevant comparison includes subsequent discovery and retention, alongside naming quality.

We compare our reranking strategy with direct image-text CLIP scoring as below, Exact Top-1 denotes the success rate of exact string match between Top1 candidate with the ground-truth class name:

Candidate selectorExact Top-1 (%)S² Top-1Real-time TA+CAPost TA+CA
Direct image–text scoring, SEGA-aligned encoders21.9254.8280.3575.10
Relational reranking, full SEGA22.8555.6682.5577.63

Direct image–text scoring selects text to fit a potentially confused visual representation. Our goal is to select a semantically plausible and discriminative reference in the calibrated text space, whose attribute-rich description then guides visual adaptation through the cross-modal objective. This encourages uncertain visual features to move toward a more appropriate semantic region and become better separated from other classes, improving subsequent classification. And relational reranking improves downstream agreement and retention relative to direct image–text matching. Its benefit is therefore not simply producing a name compatible with one image, but providing a more useful semantic reference for learning the discovered class. Exact naming remains desirable; it is not the sole criterion for the reference's usefulness in TTD.

Q4. Does a higher reranking score indicate a better name?

The purpose of Reranking Score R is to select a useful semantic reference for new-class learning. Eq. (10) measures a candidate’s consistency with the positive-anchor neighborhood in the calibrated text space. Within a candidate set, a higher R therefore indicates stronger agreement with these semantic references. Our objective is to select a candidate whose description provides a semantically reasonable and discriminative position in the shared space. Through Eq. (13), this reference can guide initially confused visual features toward a more appropriate region, improving subsequent classification. Exact naming is desirable, but the usefulness of this guidance is broader than recovering the benchmark label.

R and S² have different roles. R is used during inference and does not require the unknown class’s ground-truth name. S² is an evaluation metric that compares the generated and ground-truth names using Sentence-BERT. It measures graded semantic similarity and should be reported separately from exact naming accuracy.

We first compare the relationship between reranking score and candidate naming quality,

Candidate rank by RRAverage RRAverage Exact Top-1 (%)Average S² Top-1
Rank 1 (highest RR)0.8620.6654.02
Rank 20.8118.4553.14
Rank 30.7816.1551.74
Rank 40.7513.0849.80
Rank 50.7210.0446.95

As candidates are ranked from Rank 1 to Rank 5 by RR, the average reranking score decreases substantially, indicating that candidates with higher RR are overall more reasonable and semantically closer to the true class. However, some high-RR candidates are not exact names, and some low-RR candidates may occasionally be semantically close. This indicates that RR measures a candidate's consistency with the positive-anchor neighborhood, and its plausibility and usefulness as a semantic reference in the calibrated space rather than exact naming accuracy. The role of RR is to choose a semantically reasonable and discriminative reference for learning the discovered class, not to guarantee recovery of the benchmark's exact class name.

The bubble plots illustrate this distinction. In Sec. 4.3, Fig. 5\textcolor{red}{\text{Sec. 4.3, Fig. 5}}, bubble sizes and values represent S², not R. The examples include exact predictions such as Squirrel and Vase (S²=1.00), and a non-exact prediction, Falcon 900 → Dassault Falcon 2000 (S²=0.73). The accompanying text-space visualizations illustrate how a non-exact candidate can remain semantically related to the target while distinguishable from known-class references. Such a description can provide useful guidance for learning the discovered class. Additional examples in Appendix D, Fig. 7\textcolor{red}{\text{Appendix D, Fig. 7}} also show failures, such as Trout → Whale (S²=0.41), demonstrating that semantic grounding remains imperfect.

Complementary evaluations are already reported in Appendix K.6, Table 17\textcolor{red}{\text{Appendix K.6, Table 17}}. The blinded questionnaire involves 32 participants and distinguishes exact/synonym identity from semantic usefulness:

MethodCLIP Top-1CLIP Max Top-5Human exact/synonym (%)Human useful/non-confusing (%)
Fixed-prompt control52.9858.7236.5050.81
Full SEGA60.3165.8943.4758.81

Attribute-rich SEGA improves human exact/synonym judgments from 36.50% to 43.47%, and useful/non-confusing judgments from 50.81% to 58.81%. The CLIP columns measure semantic similarity rather than exact-match accuracy. These complementary evaluations support the semantic value of the generated references beyond Sentence-BERT similarity alone.

Their downstream value is examined in Appendix K.4, Table 15\textcolor{red}{\text{Appendix K.4, Table 15}}, which controls representation, online policy, and description source. These controls support the contribution of semantic references and their integration into online learning.

A higher R should therefore be interpreted as stronger anchor consistency, rather than a guarantee of exact naming. The qualitative examples and complementary evaluations support the usefulness of this selection criterion, while the system-level controls establish its role within the complete discovery-and-learning procedure.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.