- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses Generalized Category Discovery, where an unlabelled dataset containing both known and novel categories must be partitioned and discovered. Existing visual GCD methods struggle to distinguish subtle, fine-grained look-alikes. Conversely, while Vision-Language Models possess rich semantic concepts, their generated words frequently hallucinate or misclassify novel fine-grained classes when used as explicit labels or descriptions.
To exploit VLM semantics without suffering from unreliable text outputs, the authors introduce BOLT:
-
Multi-level Taxonomic Hidden States: A frozen VLM (e.g., Qwen3.5-9B) is queried once per image with a structured coarse-to-fine taxonomy prompt (e.g., domain → species). Its hidden states at each level are extracted prior to decoding, post-processed (mean and top principal directions removed via "all-but-the-top"), and compressed.
-
Dual-View Clustering & Tri-View Pseudo-Label Supervision: GCD backbones typically supervise only the leading half (z_1) of their contrastive representation, leaving the spare half (z_2) shaped only by unsupervised instance contrast. BOLT lets the VLM bottleneck features participate in the clustering vote via concatenation [f ˆ;e ˆ ], and supervises the spare half z_2 using pseudo-labels across three views: the vision projection z_2, the VLM projection h, and their average 1/2 (z_2+h).
-
Broad Empirical Validation: Adding BOLT across 5 GCD baselines (AL-GCD, SelEx, Hyp-SelEx, SimGCD, CMS) and 4 visual encoders (CLIP, DINOv1/v2/v3) on 3 fine-grained benchmarks (CUB, Stanford Cars, FGVC-Aircraft) yields substantial improvements (e.g., +5.7 points average on AL-GCD, and up to +31 points on DINOv1 baselines).
Critical strengths
-
Novel Conceptual Positioning (Representational View vs. Textual Naming): Rather than treating VLM outputs as pseudo-labels, prompt-tuners, or textual retrievals (which frequently fail on novel classes), BOLT treats VLM internal hidden states from intermediate taxonomic levels as a continuous semantic view, effectively circumventing naming inaccuracies.
-
Elegant Utilization of the "Spare Half" Embedding: Seamlessly integrating with hierarchical GCD frameworks (like SelEx/Matryoshka-style embeddings) by using the under-supervised second half of the contrastive feature space allows baseline objectives to remain completely intact while learning cross-modal alignments.
-
Thorough Diagnostic and Failure Analysis: The paper provides transparent failure mode analysis, demonstrating that catastrophic class merging occurs specifically when both the visual backbone and the VLM conflate the same look-alike species, offering rare candor on failure mechanics.
-
Zero Test-Time VLM Overhead: The learned semantic guidance transfers directly into the vision encoder; at test time, the VLM can be completely discarded while retaining most or all of the clustering gains.
Critical weaknesses
-
Potential Dataset Leakage via Buggy Baseline Code: In Appendix I, the authors reveal a critical indexing bug in the released code of SelEx and AL-GCD: pseudo-labels for levels i≥1 were returned in concatenated order rather than dataloader order. On datasets ordered by class (CUB and FGVC-Aircraft), this accidentally leaks the true class label via file order! Table 10 shows that when this bug is fixed on FGVC-Aircraft, baseline AL-GCD accuracy drops by 9.7%, and BOLT’s novel class gain drops from +13.2% to -0.3%. Because the vast majority of tables were run under this protocol, the reported novel-class discovery gains on CUB and Aircraft are heavily confounded by dataset-order leakage. Please clarify.
-
Diminishing Returns on Strong Backbones (DINOv2 & DINOv3): As shown in Table 2, BOLT provides marginal gains or even degrades performance on stronger, modern representations: on CUB, it drops by -1.3% with DINOv2 and gains only +0.1% with DINOv3; on Stanford Cars with Hyp-SelEx (DINOv2), it degrades by -0.8%. This suggests BOLT primarily compensates for visual under-capacity in older models (like DINOv1) rather than providing complementary semantics to strong visual representations.
-
Manual, Dataset-Specific Taxonomy Engineering: Extracting hidden states requires handcrafted, dataset-specific taxonomy keys (e.g., domain, class, group, family, genus, species, subtype for CUB vs. make, model, generation, trim for Cars). In a true open-world discovery setup where unlabelled data may contain completely arbitrary or mixed objects, handcrafting rigid taxonomic keys is unrealistic.
-
Single-Run Evaluations Despite Substantial Measurement Variance: The authors acknowledge in Section 5 (lines 476–478) and Appendix F that every result is based on a single run, and that run-to-run variation spans up to 2.2 points, while hardware changes cause shifts of up to 5.4 points. Given that several reported gains (especially on CUB and stronger backbones) are below 2 points, the lack of multi-seed error bars is prone to undermine the statistical significance of these results.
Questions for authors
-
Given that fixing the pseudo-label ordering bug on FGVC-Aircraft turned a +13.2% novel class gain into a -0.3% loss (Table 10), what are the exact results of Table 1 and Table 2 across CUB and Aircraft when the corrected dataloader ordering is applied to all baselines and BOLT variants? Does BOLT still show statistically significant novel-class discovery without this leakage?
-
How sensitive is BOLT if a domain-mismatched or generic prompt hierarchy is applied (e.g., using a single generic hierarchy across all datasets instead of specialized bird/car/aircraft keys)? Can the VLM discover the appropriate taxonomic keys automatically?
-
Why does the VLM view actively degrade novel class clustering on CUB when paired with DINOv2 (-1.3%) and DINOv3? Is there a principled gating mechanism to prevent the VLM vote from corrupting strong visual features when the visual encoder is more reliable than the VLM?
-
Can the authors provide mean and standard deviation over 3–5 random seeds for the primary comparisons (at least on AL-GCD and SelEx) to verify that marginal improvements (e.g., +1.7% on CUB) exceed the benchmark's intrinsic noise floor?
3 comments
Thanks for the rebuttal
Thank you for providing the additional experiments—they have effectively clarified my previous queries. However, given that the model remains affected by VLM bias and that the proposed method largely integrates existing concepts (as Reviewer WfsW also noted), I recommend restructuring your novelty claims to better reflect these nuances. I will consider re-adjusting my rating based on the responses and discussions from other reviewers.
Reply to Reviewer SWRz (follow-up)
We sincerely thank the reviewer for the follow-up and for considering our response. We are glad that the additional experiments clarified the earlier questions.
VLM bias. We tested whether BOLT's gain comes from what the VLM already knows about the classes. For every training image we compared the finest name that the VLM writes with the true class. The VLM names the class correctly for only 25% / 10% / 47% of the images on CUB / Aircraft / Cars. On the images that it names wrongly, BOLT still gains +6.4 / +7.0 / +14.9 points in All accuracy, compared with +6.6 / +7.4 / +17.6 on the images that it names correctly (SelEx, DINOv1, 3 seeds, corrected code). The classes for which the VLM never writes the correct name (66 / 65 / 49 classes) gain +6.3 / +7.8 / +13.8 points. So most of the gain does not depend on the VLM naming the class correctly. This agrees with two results of our first response. A generic prompt without dataset or class names gives the same gains, and with 50% wrong names BOLT keeps its gain on CUB (reply to Reviewer Lyid, W3). It also fits how BOLT uses the VLM. The VLM never labels an image. Its hidden states only vote, together with the image features, on which images are grouped, and the image encoder is then trained on these groups.
Novelty. We will restructure the novelty claims as the reviewer recommends, so that they state more precisely what is new. What is new is where the VLM enters. Earlier language-based GCD methods use text as labels, targets or features, and most of them keep a language component at test time. BOLT uses the hidden states of the VLM only as a second view in the clustering that makes the training pseudo-labels. The ablations show that this use is what matters. The same VLM features in the loss, without the vote, give no gain (-0.2 / -0.8 / -0.4), and distilling them into the image encoder gives about half (+3.1 / +3.1 / +8.4) of BOLT's +6.5 / +7.1 / +16.2 on CUB / Aircraft / Cars. With this one change, BOLT is above its baseline in 22 of 24 comparisons across five GCD methods, and their own objectives stay unchanged.
If there is an experiment that would help the reviewer judge either point, we would be glad to run it during the discussion.
Reply to Reviewer SWRz
We sincerely thank the reviewer for the valuable feedback and questions, which helped us improve the paper.
W1 / Q1. Pseudo-label misalignment, corrected tables, significance. We reran all affected results with the fix and three seeds. Here are the corrected Tables 1-2. Each cell gives All accuracy as baseline → +BOLT (mean ± sd of 3 seeds), and the brackets give the gain in All / Known / Novel accuracy.
SimGCD and CMS do not use the affected code. With three seeds, their All gains are +7.1 / +4.4 / +10.7 and +10.3 / +5.6 / +29.5 (CUB / Aircraft / Cars). Regarding significance, for SelEx with DINOv1 the novel-class gains (+7.3 / +7.6 / +18.0) are far outside seed noise (the Novel-accuracy sd is at most 1.7). With DINOv2 and DINOv3, the Cars gains are also clearly outside the seed spread, while the CUB and Aircraft gains are not. For AL-GCD on Aircraft, the reviewer's reading of Table 10 was right. The gain disappears once the misalignment is fixed (-3.1 Novel).
W2 / Q3. Strong backbones; gating. On the corrected code, the negative cells raised in the review are no longer negative. DINOv2 CUB goes from -1.3 to +0.8 All, DINOv3 CUB from -1.5 to +0.8 Novel, and Hyp-SelEx DINOv2 Cars from -0.8 to +1.4 All (3 seeds each). We agree with the reviewer that with these backbones the gains are small on CUB and Aircraft, and for Hyp-SelEx also on Cars. BOLT helps most with a weaker backbone, or when the VLM knows the domain well. On Cars, for example, SelEx still gains +4.6 with DINOv2 and +2.4 with DINOv3. As for a gating mechanism, we tried gating the weight of the VLM view in the clustering by its reliability, and it did not consistently beat a fixed weight. The reason is that accuracy on held-out known classes saturates near 97%, so it cannot see what the VLM adds for novel classes (reply to Reviewer Lyid, Q3).
W3 / Q2. Hand-made keys. We ran the test the reviewer suggested and used one generic prompt for all three datasets. Its keys are domain, category, group, family, kind, type and variant, with no dataset or class names. On CUB / Aircraft / Cars it gains +6.3 / +6.8 / +16.4, compared with +6.5 / +7.1 / +16.2 for the dataset-specific keys (SelEx, DINOv1, 3 seeds), so the two agree within 0.3 points All. We did not try automatic key discovery.
W4 / Q4. Single runs. All corrected results now have three seeds (mean ± sd in the table above), and BOLT and its baseline share seeds and GPU type.
To sum up, the revised paper will fix the misalignment everywhere, report three seeds, discuss strong backbones, and add the generic prompt as a tested alternative to the hand-made keys. We hope this addresses the reviewer's concerns, and we would be glad to discuss any point further.