acceptodds
Withdrawn from ICLR 2027

Beyond Pseudo-Labels: Vision-Language Guidance for Generalized Category Discovery

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Generalized category discovery (GCD) aims to recognise known categories while discovering and clustering novel ones in a pool of unlabelled data. Previous methods rely on the visual similarity between samples to form clusters; however, this fails in applications where the distinctions between categories are fine-grained. On the other hand, vision-language models (VLMs) carry rich semantic knowledge about how fine-grained categories differ from each other, which could complement visual category discovery. Yet existing approaches use this knowledge only through predicted labels or textual descriptions, and a VLM's words are least reliable exactly on the fine-grained novel classes. In this paper, we show that the internal semantic structure of a VLM provides useful supervision beyond its category predictions. We introduce BOLT, a framework that prompts a frozen VLM once per image for a coarse-to-fine taxonomy, reads its hidden state at every level, and uses this multi-level representation as a second view of the pseudo-label hierarchy of the vision encoder: the half of the contrastive embedding that GCD methods leave to an unsupervised loss is trained to the learner's hierarchical pseudo-labels from the image, from the VLM and from their average, and the VLM votes in the clustering that produces those pseudo-labels. BOLT never uses a VLM output as a label and leaves the baseline's objective and evaluation unchanged, so the discovery model keeps its visual discrimination while absorbing the VLM's knowledge. On three fine-grained benchmarks, added to five GCD methods (AL-GCD, SelEx, Hyp-SelEx, SimGCD and CMS) and four backbones (CLIP and DINOv1/v2/v3), BOLT improves AL-GCD by 5.7 points on average and by up to 9.3, and improves the DINOv1-based baselines by 11.5 on average and by up to 31 points, showing the benefit of the complementary semantic knowledge of VLMs.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Generalized Category Discovery, Vision Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_SWRzOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses Generalized Category Discovery, where an unlabelled dataset containing both known and novel categories must be partitioned and discovered. Existing visual GCD methods struggle to distinguish subtle, fine-grained look-alikes. Conversely, while Vision-Language Models possess rich semantic concepts, their generated words frequently hallucinate or misclassify novel fine-grained classes when used as explicit labels or descriptions.

To exploit VLM semantics without suffering from unreliable text outputs, the authors introduce BOLT:

  1. Multi-level Taxonomic Hidden States: A frozen VLM (e.g., Qwen3.5-9B) is queried once per image with a structured coarse-to-fine taxonomy prompt (e.g., domain → species). Its hidden states at each level are extracted prior to decoding, post-processed (mean and top principal directions removed via "all-but-the-top"), and compressed.

  2. Dual-View Clustering & Tri-View Pseudo-Label Supervision: GCD backbones typically supervise only the leading half (z_1) of their contrastive representation, leaving the spare half (z_2) shaped only by unsupervised instance contrast. BOLT lets the VLM bottleneck features participate in the clustering vote via concatenation [f ˆ;e ˆ ], and supervises the spare half z_2 using pseudo-labels across three views: the vision projection z_2, the VLM projection h, and their average 1/2 (z_2+h).

  3. Broad Empirical Validation: Adding BOLT across 5 GCD baselines (AL-GCD, SelEx, Hyp-SelEx, SimGCD, CMS) and 4 visual encoders (CLIP, DINOv1/v2/v3) on 3 fine-grained benchmarks (CUB, Stanford Cars, FGVC-Aircraft) yields substantial improvements (e.g., +5.7 points average on AL-GCD, and up to +31 points on DINOv1 baselines).

Critical strengths
  1. Novel Conceptual Positioning (Representational View vs. Textual Naming): Rather than treating VLM outputs as pseudo-labels, prompt-tuners, or textual retrievals (which frequently fail on novel classes), BOLT treats VLM internal hidden states from intermediate taxonomic levels as a continuous semantic view, effectively circumventing naming inaccuracies.

  2. Elegant Utilization of the "Spare Half" Embedding: Seamlessly integrating with hierarchical GCD frameworks (like SelEx/Matryoshka-style embeddings) by using the under-supervised second half of the contrastive feature space allows baseline objectives to remain completely intact while learning cross-modal alignments.

  3. Thorough Diagnostic and Failure Analysis: The paper provides transparent failure mode analysis, demonstrating that catastrophic class merging occurs specifically when both the visual backbone and the VLM conflate the same look-alike species, offering rare candor on failure mechanics.

  4. Zero Test-Time VLM Overhead: The learned semantic guidance transfers directly into the vision encoder; at test time, the VLM can be completely discarded while retaining most or all of the clustering gains.

Critical weaknesses
  1. Potential Dataset Leakage via Buggy Baseline Code: In Appendix I, the authors reveal a critical indexing bug in the released code of SelEx and AL-GCD: pseudo-labels for levels i≥1 were returned in concatenated order rather than dataloader order. On datasets ordered by class (CUB and FGVC-Aircraft), this accidentally leaks the true class label via file order! Table 10 shows that when this bug is fixed on FGVC-Aircraft, baseline AL-GCD accuracy drops by 9.7%, and BOLT’s novel class gain drops from +13.2% to -0.3%. Because the vast majority of tables were run under this protocol, the reported novel-class discovery gains on CUB and Aircraft are heavily confounded by dataset-order leakage. Please clarify.

  2. Diminishing Returns on Strong Backbones (DINOv2 & DINOv3): As shown in Table 2, BOLT provides marginal gains or even degrades performance on stronger, modern representations: on CUB, it drops by -1.3% with DINOv2 and gains only +0.1% with DINOv3; on Stanford Cars with Hyp-SelEx (DINOv2), it degrades by -0.8%. This suggests BOLT primarily compensates for visual under-capacity in older models (like DINOv1) rather than providing complementary semantics to strong visual representations.

  3. Manual, Dataset-Specific Taxonomy Engineering: Extracting hidden states requires handcrafted, dataset-specific taxonomy keys (e.g., domain, class, group, family, genus, species, subtype for CUB vs. make, model, generation, trim for Cars). In a true open-world discovery setup where unlabelled data may contain completely arbitrary or mixed objects, handcrafting rigid taxonomic keys is unrealistic.

  4. Single-Run Evaluations Despite Substantial Measurement Variance: The authors acknowledge in Section 5 (lines 476–478) and Appendix F that every result is based on a single run, and that run-to-run variation spans up to 2.2 points, while hardware changes cause shifts of up to 5.4 points. Given that several reported gains (especially on CUB and stronger backbones) are below 2 points, the lack of multi-seed error bars is prone to undermine the statistical significance of these results.

Questions for authors
  1. Given that fixing the pseudo-label ordering bug on FGVC-Aircraft turned a +13.2% novel class gain into a -0.3% loss (Table 10), what are the exact results of Table 1 and Table 2 across CUB and Aircraft when the corrected dataloader ordering is applied to all baselines and BOLT variants? Does BOLT still show statistically significant novel-class discovery without this leakage?

  2. How sensitive is BOLT if a domain-mismatched or generic prompt hierarchy is applied (e.g., using a single generic hierarchy across all datasets instead of specialized bird/car/aircraft keys)? Can the VLM discover the appropriate taxonomic keys automatically?

  3. Why does the VLM view actively degrade novel class clustering on CUB when paired with DINOv2 (-1.3%) and DINOv3? Is there a principled gating mechanism to prevent the VLM vote from corrupting strong visual features when the visual encoder is more reliable than the VLM?

  4. Can the authors provide mean and standard deviation over 3–5 random seeds for the primary comparisons (at least on AL-GCD and SelEx) to verify that marginal improvements (e.g., +1.7% on CUB) exceed the benchmark's intrinsic noise floor?

3 comments
AuthorsOpenReview ↗

Reply to Reviewer SWRz

We sincerely thank the reviewer for the valuable feedback and questions, which helped us improve the paper.

W1 / Q1. Pseudo-label misalignment, corrected tables, significance. We reran all affected results with the fix and three seeds. Here are the corrected Tables 1-2. Each cell gives All accuracy as baseline → +BOLT (mean ± sd of 3 seeds), and the brackets give the gain in All / Known / Novel accuracy.

baseline, backboneCUBFGVC-AircraftStanford Cars
SelEx, DINOv170.6 ± 0.6 → 77.0 ± 0.2 (+6.5 / +4.7 / +7.3)58.0 ± 1.4 → 65.0 ± 0.1 (+7.1 / +5.9 / +7.6)58.7 ± 1.1 → 74.8 ± 0.4 (+16.2 / +12.3 / +18.0)
SelEx, DINOv283.3 ± 0.2 → 84.1 ± 0.9 (+0.8 / +1.5 / +0.5)77.2 ± 0.1 → 77.8 ± 0.8 (+0.6 / 0.0 / +0.9)78.8 ± 0.9 → 83.5 ± 0.9 (+4.6 / +1.2 / +6.3)
SelEx, DINOv383.4 ± 0.2 → 83.8 ± 0.8 (+0.5 / -0.1 / +0.8)74.3 ± 0.8 → 74.4 ± 0.3 (0.0 / +0.3 / -0.1)76.8 ± 0.1 → 79.2 ± 0.7 (+2.4 / +2.3 / +2.5)
Hyp-SelEx, DINOv169.2 ± 0.2 → 75.1 ± 0.7 (+6.0 / +3.7 / +7.1)60.3 ± 2.4 → 63.6 ± 0.7 (+3.3 / +1.1 / +4.4)60.2 ± 0.5 → 71.1 ± 0.2 (+11.0 / +7.7 / +12.5)
Hyp-SelEx, DINOv283.7 ± 0.4 → 84.3 ± 0.2 (+0.6 / +0.5 / +0.6)77.6 ± 0.4 → 78.6 ± 1.2 (+1.1 / +1.1 / +1.0)80.8 ± 0.3 → 82.2 ± 1.5 (+1.4 / -0.6 / +2.4)
AL-GCD, CLIP74.3 ± 0.5 → 76.6 ± 1.1 (+2.3 / +0.4 / +3.2)60.2 ± 1.1 → 58.9 ± 1.0 (-1.2 / +2.4 / -3.1)77.6 ± 0.2 → 83.5 ± 1.0 (+6.0 / +2.5 / +7.6)

SimGCD and CMS do not use the affected code. With three seeds, their All gains are +7.1 / +4.4 / +10.7 and +10.3 / +5.6 / +29.5 (CUB / Aircraft / Cars). Regarding significance, for SelEx with DINOv1 the novel-class gains (+7.3 / +7.6 / +18.0) are far outside seed noise (the Novel-accuracy sd is at most 1.7). With DINOv2 and DINOv3, the Cars gains are also clearly outside the seed spread, while the CUB and Aircraft gains are not. For AL-GCD on Aircraft, the reviewer's reading of Table 10 was right. The gain disappears once the misalignment is fixed (-3.1 Novel).

W2 / Q3. Strong backbones; gating. On the corrected code, the negative cells raised in the review are no longer negative. DINOv2 CUB goes from -1.3 to +0.8 All, DINOv3 CUB from -1.5 to +0.8 Novel, and Hyp-SelEx DINOv2 Cars from -0.8 to +1.4 All (3 seeds each). We agree with the reviewer that with these backbones the gains are small on CUB and Aircraft, and for Hyp-SelEx also on Cars. BOLT helps most with a weaker backbone, or when the VLM knows the domain well. On Cars, for example, SelEx still gains +4.6 with DINOv2 and +2.4 with DINOv3. As for a gating mechanism, we tried gating the weight of the VLM view in the clustering by its reliability, and it did not consistently beat a fixed weight. The reason is that accuracy on held-out known classes saturates near 97%, so it cannot see what the VLM adds for novel classes (reply to Reviewer Lyid, Q3).

W3 / Q2. Hand-made keys. We ran the test the reviewer suggested and used one generic prompt for all three datasets. Its keys are domain, category, group, family, kind, type and variant, with no dataset or class names. On CUB / Aircraft / Cars it gains +6.3 / +6.8 / +16.4, compared with +6.5 / +7.1 / +16.2 for the dataset-specific keys (SelEx, DINOv1, 3 seeds), so the two agree within 0.3 points All. We did not try automatic key discovery.

W4 / Q4. Single runs. All corrected results now have three seeds (mean ± sd in the table above), and BOLT and its baseline share seeds and GPU type.

To sum up, the revised paper will fix the misalignment everywhere, report three seeds, discuss strong backbones, and add the generic prompt as a tested alternative to the hand-made keys. We hope this addresses the reviewer's concerns, and we would be glad to discuss any point further.

Reviewer_SWRzOpenReview ↗

Thanks for the rebuttal

Thank you for providing the additional experiments—they have effectively clarified my previous queries. However, given that the model remains affected by VLM bias and that the proposed method largely integrates existing concepts (as Reviewer WfsW also noted), I recommend restructuring your novelty claims to better reflect these nuances. I will consider re-adjusting my rating based on the responses and discussions from other reviewers.

AuthorsOpenReview ↗

Reply to Reviewer SWRz (follow-up)

We sincerely thank the reviewer for the follow-up and for considering our response. We are glad that the additional experiments clarified the earlier questions.

VLM bias. We tested whether BOLT's gain comes from what the VLM already knows about the classes. For every training image we compared the finest name that the VLM writes with the true class. The VLM names the class correctly for only 25% / 10% / 47% of the images on CUB / Aircraft / Cars. On the images that it names wrongly, BOLT still gains +6.4 / +7.0 / +14.9 points in All accuracy, compared with +6.6 / +7.4 / +17.6 on the images that it names correctly (SelEx, DINOv1, 3 seeds, corrected code). The classes for which the VLM never writes the correct name (66 / 65 / 49 classes) gain +6.3 / +7.8 / +13.8 points. So most of the gain does not depend on the VLM naming the class correctly. This agrees with two results of our first response. A generic prompt without dataset or class names gives the same gains, and with 50% wrong names BOLT keeps its gain on CUB (reply to Reviewer Lyid, W3). It also fits how BOLT uses the VLM. The VLM never labels an image. Its hidden states only vote, together with the image features, on which images are grouped, and the image encoder is then trained on these groups.

Novelty. We will restructure the novelty claims as the reviewer recommends, so that they state more precisely what is new. What is new is where the VLM enters. Earlier language-based GCD methods use text as labels, targets or features, and most of them keep a language component at test time. BOLT uses the hidden states of the VLM only as a second view in the clustering that makes the training pseudo-labels. The ablations show that this use is what matters. The same VLM features in the loss, without the vote, give no gain (-0.2 / -0.8 / -0.4), and distilling them into the image encoder gives about half (+3.1 / +3.1 / +8.4) of BOLT's +6.5 / +7.1 / +16.2 on CUB / Aircraft / Cars. With this one change, BOLT is above its baseline in 22 of 24 comparisons across five GCD methods, and their own objectives stay unchanged.

If there is an experiment that would help the reviewer judge either point, we would be glad to run it during the discussion.

Reviewer_WfsWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes BOLT, a framework that uses a frozen Vision-Language Model (VLM) to assist Generalized Category Discovery (GCD). Unlike methods that directly use VLM-predicted category labels or textual descriptions, BOLT extracts hidden states from coarse-to-fine category hierarchies generated by the VLM and incorporates semantic information into the visual model through joint feature clustering and three-view contrastive learning.

Critical strengths

The research motivation is reasonable. The experiments are relatively comprehensive. Some experimental results show significant improvements.

Critical weaknesses
  1. The main components of BOLT include VLM hidden-state extraction, feature concatenation, semi-supervised K-means, hierarchical pseudo-labeling, and supervised contrastive learning. These techniques have already been extensively studied. The paper mainly integrates them into existing GCD frameworks to achieve performance improvements.

  2. Performance improvements depend on the strength of the backbone, with limited gains or even negative transfer when using stronger visual models. This suggests that VLM semantic information is not always complementary and may even interfere with visual representations that already have strong discriminative capabilities. The paper lacks an adaptive mechanism to determine when VLM information should be introduced and when its influence should be reduced.

  3. The trade-off between additional computational costs and performance gains needs further discussion. The additional parameters and training latency introduced by BOLT do not appear proportional to the performance improvements.

Questions for authors
  1. Why does BOLT sometimes degrade performance with stronger visual backbones? Could an adaptive weighting mechanism prevent negative transfer when the VLM representation is less discriminative?

  2. Why is supervising the spare half of the embedding necessary? Figure 4(a) suggests that supervising the full embedding achieves comparable performance.

  3. Why do coarse-level VLM representations sometimes outperform the complete hierarchy? Does this suggest that fine-grained VLM representations introduce semantic noise?

  4. How much of the improvement can be achieved through simple VLM–visual feature fusion without the proposed three-view objective?

3 comments
AuthorsOpenReview ↗

Reply to Reviewer WfsW

We sincerely thank the reviewer for the valuable feedback and questions, which helped us improve the paper. In short, with the corrected code and three seeds, the gains are significant with DINOv1 and small with DINOv2 and DINOv3, no SelEx or Hyp-SelEx gain is negative, and no VLM is needed at test time (table in the general response).

W1. Integration of known techniques. We agree that the components are known. What the ablations support is a narrower claim. The VLM's taxonomy states, used as a second view inside the clustering that makes the training pseudo-labels (the vote), improve the image encoder, so no VLM is needed at test time. The parts do not work alone. On SelEx with DINOv1 (3 seeds, gain on CUB / Aircraft / Cars), the loss without the vote gives nothing (-0.2 / -0.8 / -0.4). The vote without the loss helps only where the VLM is strong (+4.1 / -6.0 / +17.0). Distillation without pseudo-labels gives about half (+3.1 / +3.1 / +8.4) of BOLT's +6.5 / +7.1 / +16.2. So these components together are the reason for the significant improvement.

W2 / Q1. Strong backbones and negative transfer. With DINOv2 and DINOv3, every SelEx and Hyp-SelEx gain in All accuracy is zero or positive. For SelEx on CUB / Aircraft / Cars, the gains are +0.8 / +0.6 / +4.6 with DINOv2 and +0.5 / 0.0 / +2.4 with DINOv3. The negative cell of the submission (DINOv2 CUB, -1.3) is +0.8 on the corrected SelEx code. The only negative comparison left is AL-GCD with CLIP on Aircraft (-1.2). With strong backbones, the gains on CUB and Aircraft are indeed small. The VLM adds most where the image encoder is weaker (DINOv1) or where the VLM knows the domain well (Stanford Cars, which still gains clearly with SelEx). The revised paper will discuss this. Note also that backbones such as DINOv2, DINOv3 and CLIP may have already seen images like those of these benchmarks in their pretraining. DINOv2, for example, built its pretraining set by retrieving web images similar to the training images of CUB, FGVC-Aircraft and Stanford Cars, among other datasets. This makes such backbones less suited to the GCD assumption that the novel classes are unseen. As for an adaptive mechanism, we tried five reliability-based weighting schemes, and none was consistently better than the fixed weight (reply to Reviewer Lyid, Q3).

W3. Cost. The VLM runs once per dataset before training, and its output is shared by all runs. This takes 24 to 86 minutes on one GPU, which is less than one AL-GCD training run (2.0 to 2.9 h). Training adds 9.96 M parameters and 6 to 44% epoch time (AL-GCD, Table 9). At test time nothing is added, because only the image encoder is used.

Q2. Why the spare half. It is not necessary. Supervising the full embedding gives the same gains (+6.3 / +6.9 / +16.0 on CUB / Aircraft / Cars). We keep it because it leaves the baseline's objective unchanged, and the revised paper will state that both work.

Q3. Coarse states vs the full hierarchy. On the corrected code the two are equal (CUB +6.2 vs. +6.5, Cars +16.7 vs. +16.3, with two seeds on CUB and one on Cars). The fine levels add noise on CUB. Alone they are the weakest (+4.0), and fine names are most often wrong there. The full hierarchy is robust because it keeps the coarse evidence.

Q4. How much simple fusion achieves. We tested two simple forms of fusion. The first is fusion inside the clustering without the three-view loss, that is, the vote alone. It gives +4.1 / -6.0 / +17.0, compared with +6.5 / +7.1 / +16.2 for BOLT on SelEx with DINOv1 (gain on CUB / Aircraft / Cars). This is enough on Cars, but the loss is needed on CUB and Aircraft. The second is late fusion at test time, where the baseline is trained unchanged and the raw VLM vector is added. With the weight tuned on the evaluation labels, which makes it an upper bound, it gives +2.8 / +1.2 / +22.7. It helps mainly on Cars. On CUB and Aircraft, BOLT without any VLM at test time is better or equal for every backbone (details in the reply to Reviewer Lyid, W2).

In the revised paper we will report the corrected three-seed results, discuss strong backbones, put the cost in proportion (Table 9) and add the ablations that show which parts carry the gain. We hope this answers the reviewer's questions, and we are happy to discuss any of them further.

Thank you for the clarifications and additional experiments. They alleviate my concerns about negative transfer and inference cost. However, the methodological novelty remains limited, gains with stronger backbones are small, and simple fusion already matches the full method on Cars. I therefore maintain my score.

Thank you for reading our response and for the follow-up. We are glad that the concerns about negative transfer and inference cost are resolved. We appreciate the three remaining points and hope the short clarifications below are helpful.

Novelty. We agree that the components are known. Our claim is the narrower one from our first reply: the VLM's taxonomy states vote inside the clustering that makes the training pseudo-labels, and the image encoder trained this way needs no VLM at test time.

Stronger backbones. We agree. With DINOv2 and DINOv3 the gains are clear on Cars (+4.6 and +2.4) and small on CUB and Aircraft (+0.8 / +0.6 with DINOv2 and +0.5 / 0.0 with DINOv3).

Simple fusion on Cars. We agree for Cars, where the VLM alone is already strong: the vote alone gives +17.0 and late fusion at most +22.7, against +16.2 for BOLT. The full method matters on CUB and Aircraft, where the vote alone gives +4.1 / -6.0 and late fusion at most +2.8 / +1.2, against +6.5 / +7.1 for BOLT, which also needs no VLM at test time.

Thank you again for the careful review; it has improved the paper.

Reviewer_LyidOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BOLT for generalized category discovery (GCD), which uses labeled examples from known categories to classify and cluster unlabeled images containing both known and novel categories. The authors argue that clustering based solely on visual similarity can confuse fine-grained categories, while directly using category names generated by vision–language models (VLMs) can introduce naming errors. BOLT therefore extracts hidden states at multiple levels of a coarse-to-fine taxonomy generated by a frozen VLM and uses them as a complementary view of the visual representation. It generates hierarchical pseudo-labels through joint clustering and applies hierarchical contrastive supervision to the second half of the image embedding through the image view, the VLM view, and their average. Experiments report performance improvements across multiple fine-grained datasets, GCD methods, and visual backbones.

Critical strengths
  • The paper distinguishes the accuracy of generated category names from the usefulness of internal representations, seeking complementary semantic information from multilevel hidden states without directly treating generated names as training labels. This perspective addresses the coexistence of visual ambiguity and naming errors in fine-grained category discovery.

  • The VLM view contributes to both clustering and visual representation learning through shared supervision of the image view, the VLM view, and their average. External knowledge can therefore influence both pseudo-label generation and feature updates, providing an explicit mechanism for integrating visual and semantic information.

  • BOLT can be integrated into multiple GCD frameworks. Evaluation across five methods and four visual backbone families provides experimental support for its applicability to different model configurations.

Critical weaknesses
  • Appendix I identifies a misalignment between higher-level pseudo-labels and sample indices, but the main experiments retain this implementation. In the Aircraft experiment at λ=0.5 in Table 10, correcting the implementation reduces the overall gain from 9.5 to 4.0 percentage points and changes the novel-class gain from +13.2 to −0.3 points. The affected central results therefore require revalidation under the corrected implementation.

  • ESMC [1] already uses token hidden states from multimodal large models for image clustering, together with a lightweight clustering head and pseudo-label learning; SelEx [2] already introduces hierarchical pseudo-label supervision into GCD. The contribution of this paper primarily lies in combining taxonomy-level hidden states as a complementary view with joint clustering and three-view supervision. However, the distinctive advantage of this combination over simpler fusion approaches has not been sufficiently established.

  • The hidden states are extracted by teacher-forced encoding of generated answers and therefore remain conditioned on generated text. The experiments support avoiding the direct use of generated names as labels, but comparisons with textual representations of the same content and incorrect-name perturbations are missing. Consequently, the evidence does not sufficiently explain how hidden states mitigate the effects of naming errors.

  • Multilevel states are concatenated and compressed for hierarchical supervision, leaving it unclear whether the gains arise from semantic hierarchical relationships or simply from aggregating features from multiple states. The ablations also do not fully disentangle the contributions of joint clustering, VLM-view supervision, and average-view supervision. Coarse-only states perform at least as well as the complete hierarchy on CUB and Cars, while second-half supervision does not show a consistent advantage over full-embedding supervision across datasets.

References

[1] Wang et al. ESMC: MLLM-Based Embedding Selection for Explainable Multiple Clustering. AAAI, 2026.

[2] Rastegar et al. SelEx: Self-Expertise in Fine-Grained Generalized Category Discovery. ECCV, 2024.

Questions for authors
  • Please rerun the affected main comparisons and key ablations after correcting the pseudo-label indexing misalignment, and include experiments with randomly permuted dataset storage orders.

  • The trainable VLM projection participates in generating pseudo-labels and is subsequently supervised by those same labels, potentially reinforcing errors. Please analyze this risk and explain how three-view consistency prevents the views from jointly fitting incorrect labels.

  • Joint clustering assigns fixed equal weights to the visual and VLM views, although their reliability may vary across categories and samples. Please justify this design and provide comparisons using different fusion weights or reliability-based weighting.

  • The second half of the embedding remains constrained by the instance-contrastive loss and shares an encoder with the first half. Please further justify its use as the space for additional supervision and explain the necessity of this partition in light of the full-embedding supervision results.

1 comment
AuthorsOpenReview ↗

Reply to Reviewer Lyid

We sincerely thank the reviewer for the valuable feedback and questions, which helped us improve the paper. With the corrected code and three seeds, BOLT improves SelEx with DINOv1 by +6.5 / +7.1 / +16.2 points in All accuracy on CUB / Aircraft / Cars, and less with DINOv2 and DINOv3 (see the general response). Unless noted, the numbers below use this setting and dataset order.

W1 / Q1. Rerun and storage order. We reran all affected results with the fix and three seeds, and all ablations below use the corrected code. We also randomly permuted the storage order (seed 1, baseline → +BOLT). With the permuted order, CUB goes 69.3 → 76.2 (+6.9) and Aircraft 59.3 → 65.6 (+6.3). With the original order, the same seed gives 70.9 → 77.1 (+6.2) and 56.7 → 65.1 (+8.4). The gain remains when the order is random.

W2. ESMC, SelEx and simpler fusion. ESMC clusters frozen features of a multimodal LLM, so it needs the LLM at test time, whereas BOLT is for semi-supervised GCD and drops the VLM after training. SelEx is one of the baselines that we add BOLT to. BOLT reuses the baseline's own pseudo-label levels and adds the VLM view to the clustering that builds them.

  • In late fusion, the baseline is trained unchanged and the raw VLM vector is added to its feature at test time. With the fusion weight tuned on the evaluation labels (an upper bound), it gains +2.8 / +1.2 / +22.7. With a weight chosen on held-out labelled images, the gain is 0.0 / 0.0 / +4.8.
  • BOLT, without any VLM at test time, gains +6.5 / +7.1 / +16.2. Late fusion is stronger only on Cars, where the VLM alone is strong. The two are complementary, since BOLT plus the same fusion gains +6.7 / +7.2 / +24.8.
  • Distillation, which aligns the embedding with the VLM vector without pseudo-labels, gains +3.1 / +3.1 / +8.4, about half of BOLT.

W3. Text vs hidden states, and wrong names. We encoded the same generated taxonomy without the image ("text only"), and we gave half of the images another image's finest names.

  • The image-conditioned states are stronger than the text alone. BOLT gains +6.5 vs. +3.5 on CUB and +7.1 vs. +2.9 on Aircraft. On Cars the two are similar (+16.2 vs. +15.9).
  • With 50% wrong names, BOLT keeps its gain when the states see the image (CUB +6.5, the same as with the right names). With text alone it does not (+0.5, seed 1). The hidden states keep the image evidence even when a name is wrong.

W4. Which parts matter (gains on CUB / Aircraft / Cars).

  • Without the vote (the VLM view joining the image feature in the training clustering), the VLM is only in the loss and there is no gain (-0.2 / -0.8 / -0.4), so the joint clustering carries the gain.
  • With the vote but without BOLT's loss, the gain is +4.1 / -6.0 / +17.0. The loss is what makes the vote work when the VLM is weak (Aircraft).
  • Full BOLT gives +6.5 / +7.1 / +16.2. Dropping the average view (+6.7 / +6.9 / +16.5) or supervising the full embedding instead of the spare half (+6.3 / +6.9 / +16.0) changes little.
  • At seed 1, removing the VLM projection as a view in the loss gives +6.0 / +6.2 / +16.5, compared with +6.2 / +8.4 / +14.8 for full BOLT.

Hierarchy or aggregation? Which levels are read matters more than how many states are aggregated. The coarse levels alone match the full hierarchy (+6.2 / +16.7 vs. +6.5 / +16.3 on CUB / Cars). The fine levels alone are weaker on CUB (+4.0 / +16.7), and a single state per image is the weakest (+3.7 / +15.2), with two seeds on CUB and one on Cars.

Q2. Does the trained projection reinforce its own errors? We tested this loop directly by letting the frozen raw VLM vector vote instead, so that no trained module enters the pseudo-labels. This gives +4.1 / -8.1 / +18.1, compared with +6.5 / +7.1 / +16.2 for BOLT. Where the VLM is weak, the trained projection reduces its errors instead of reinforcing them. On Aircraft it clusters the images better than the raw vector it is computed from (44.8 vs. 35.9 All). This is consistent with the loss tying the projection to the image view, which is trained on the same pseudo-labels.

Q3. Fixed weights vs reliability weighting. We tried five reliability-based schemes that choose the vote weight during training on held-out labelled images or held-out known classes. None was consistently better than the fixed weight. The best one is within noise of it with DINOv2 and about one point below it with DINOv1 (CUB and Aircraft, seed 1). This is because accuracy on held-out known classes saturates near 97%, so it cannot see what the VLM adds for novel classes.

Q4. Why the spare half. It leaves the baseline's own objective unchanged. It is not needed for accuracy (W4), so the revised paper will present it as a default, not a requirement.

All of this will be added to the revised paper. We hope it addresses the reviewer's concerns and are happy to clarify anything further.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response: summary of the changes

We thank all three reviewers for their feedback. All three asked for the main results to be rerun with the pseudo-label misalignment described in Appendix I fixed. This was the most important point, and we present the results accordingly.

What was done

  1. The misalignment is fixed using the corrected public release of the SelEx code (GitHub issue #11), in every baseline that uses this code (SelEx, Hyp-SelEx, AL-GCD), and all affected results are rerun with three seeds instead of one. SimGCD and CMS do not use this code and were not affected; they now also have three seeds.
  2. In every comparison, BOLT and its baseline use the same settings, seeds and GPU type. Our runs of the SelEx baseline come out within about one point of the numbers that the corrected release reports (DINOv1, CUB / Aircraft / Cars: 70.6 / 58.0 / 58.7 vs. 71.7 / 58.6 / 58.9).
  3. We ran the comparisons and ablations that the reviewers asked for: late fusion, distillation, text-only features, wrong names, a generic prompt, a storage-order test, and adaptive weighting.

Corrected results (All accuracy, baseline → +BOLT, mean of 3 seeds; gain in brackets)

baseline, backboneCUBFGVC-AircraftStanford Cars
SelEx, DINOv170.6 → 77.0 (+6.5)58.0 → 65.0 (+7.1)58.7 → 74.8 (+16.2)
SelEx, DINOv283.3 → 84.1 (+0.8)77.2 → 77.8 (+0.6)78.8 → 83.5 (+4.6)
SelEx, DINOv383.4 → 83.8 (+0.5)74.3 → 74.4 (0.0)76.8 → 79.2 (+2.4)
Hyp-SelEx, DINOv169.2 → 75.1 (+6.0)60.3 → 63.6 (+3.3)60.2 → 71.1 (+11.0)
Hyp-SelEx, DINOv283.7 → 84.3 (+0.6)77.6 → 78.6 (+1.1)80.8 → 82.2 (+1.4)
AL-GCD, CLIP74.3 → 76.6 (+2.3)60.2 → 58.9 (-1.2)77.6 → 83.5 (+6.0)
SimGCD, DINOv162.1 → 69.2 (+7.1)52.2 → 56.6 (+4.4)53.8 → 64.5 (+10.7)
CMS, DINOv167.4 → 77.7 (+10.3)49.4 → 55.0 (+5.6)52.1 → 81.6 (+29.5)

Summary of the results. BOLT is above its baseline in 22 of the 24 comparisons. For SelEx, the gains with DINOv1, and on Stanford Cars with every backbone, are well above seed noise. With DINOv2 and DINOv3, the gains on CUB and Aircraft are small and within seed noise. The corrected gains also remain when we randomly permute the storage order (Reviewer Lyid), and a single generic prompt gives the same gains as the dataset-specific keys (Reviewer SWRz). The component ablations show that the parts of BOLT do not work alone, and the VLM runs only once per dataset before training, in less time than one AL-GCD training run (Reviewer WfsW).

The revised paper will

  • use the corrected three-seed results in Tables 1-2, the abstract and the summary numbers, with mean ± sd for every comparison in an appendix table
  • include the new comparisons and ablations from this discussion
  • discuss when the VLM helps most (weaker backbones, Stanford Cars) and when its gains are small (strong backbones)
  • discuss the cost of BOLT in proportion to its gains (Table 9)

Dear Reviewers,

Thanks for your time and effort in reviewing this paper. The authors have responded to the comments by providing a rebuttal. Please read the rebuttal and indicate whether your concerns have been resolved. Thanks.

Regards from the AC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.