acceptodds
Under review as a conference paper at ICLR 2027

Learning Bidirectional Global-to-Local Reranking for Generalized Category Discovery

Abstract

Generalized Category Discovery (GCD) aims to identify both known and novel categories in an unlabeled image collection by exploiting supervision from a subset of known categories. Recent approaches improve representation learning by incorporating neighboring samples from a feature bank. However, these methods typically rely on predefined similarity measures for neighborhood construction, which may retrieve semantically unrelated instances and consequently provide unreliable learning signals. To address this limitation, we propose **G**lobal–**Lo**cal **Re**ranking Network (**GLoRe**), a learnable bidirectional ranking framework for GCD. Rather than treating feature similarity as a fixed indicator of semantic relevance, GLoRe learns to assess the relationship between a query and its retrieved candidates and uses this information to refine their ranking. By learning semantic relationships instead of fixed category predictions, GLoRe can transfer the learned ranking function from labeled examples to unlabeled instances belonging to both known and novel categories. Extensive experiments on fine-grained and coarse-grained benchmarks demonstrate that GLoRe achieves state-of-the-art performance on fine-grained datasets while remaining competitive with state-of-the-art methods on coarse-grained benchmarks. Our code will be made publicly available upon acceptance.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Generalized Category Discovery, Open-set Recognition, Semi-supervised Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZZX8OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes GLoRe, a bidirectional global-to-local reranking framework for Generalized Category Discovery (GCD). The method employs cross-attention to capture relationships between global representations and local visual features, aiming to improve the quality of retrieved neighbors. It also incorporates a momentum encoder and KoLeo regularization to enhance representation learning. The experiments show the effectiveness of their proposed method.

Critical strengths

The paper is well-structured and clearly presents its motivation and methodology. It also provides relatively detailed ablation studies, including analyses of different architectural designs, hyperparameter sensitivity,and retrieval quality.

Critical weaknesses
  1. Missing important baseline. The experimental comparisons omit several important recent GCD methods, particularly GET (CVPR 2025) and TAR (CVPR 2026). Notably, the reported results do not consistently outperform these methods. For example, GET and TAR achieve 77.0% and 78.4% on CUB, respectively, compared with 75.6% for GLoRe under its DINOv1 setting. Although these methods employ different pretrained backbones, their omission raises concerns about the completeness and fairness of the experimental evaluation. The authors should include these representative baselines, clarify the differences in experimental settings, and provide stronger evidence to support their state-of-the-art claims.

[1] Wang, E., Peng, Z., Xie, Z., Yang, F., Liu, X., & Cheng, M.-M. (2025). GET: Unlocking the Multi-modal Potential of CLIP for Generalized Category Discovery. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 20296–20306. [2] Yang, X., Zhang, Y., Mi, S., & Wei, X.-S. (2026). TAR: Token-Aware Refinement for Fine-grained Generalized Category Discovery. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 31995–32004.

  1. Limited Novelty and Incremental Contribution. The proposed method is largely an incremental extension of CMS, combining existing techniques such as cross-attention, momentum encoding, and KoLeo regularization, with limited methodological novelty.
Questions for authors

Could the authors explain why important recent GCD baselines are omitted from the experimental comparisons?

1 comment
AuthorsOpenReview ↗

Rebuttal by Authors for R3 (#ZZX8)

We sincerely thank R3 (#ZZX8) for the thoughtful feedback and for highlighting relevant recent work. Below, we clarify the differences in experimental settings and further explain the methodological contribution of GLoRe.

R3.1 – Comparison with GET and TAR

Response: We thank the reviewer for pointing us to these methods. We did not include GET and the GET-based TAR configuration originally because they train with text, while the methods in our original tables are vision-only. We have now added TAR to Tables 1–2.

GLoRe uses DINO-pretrained visual features throughout. GET [m] builds on CLIP and uses both visual and text representations during training, through a Text Embedding Synthesizer and a dual-branch architecture. Both the pretrained representation and the information available during training therefore differ, so the gap on CUB does not by itself isolate which discovery mechanism is more effective.

We note that TAR is evaluated on two base methods. In Table 1 of Yang et al. (2026) [e], 78.4% on CUB corresponds to TAR(GET), while the vision-only MOS+TAR configuration reaches 70.6%. The relevant comparisons are included in Tables 1–2.

In the vision-only DINOv1 setting, GLoRe significantly exceeds MOS+TAR on all three datasets (Table 1). With a stronger vision-only backbone (DINOv2, Table 8), GLoRe exceeds GET+TAR on all three datasets without using text. For reference, on CUB, GLoRe with DINOv2 achieves 83.7%, compared to 78.4% for the TAR(GET) combination [e].

R3.2 – Methodological novelty and relation to CMS

Response: We acknowledge that GLoRe builds upon CMS and incorporates established components, including cross-attention, momentum encoding, and KoLeo regularization. However, we respectfully disagree that the proposed contribution is merely an incremental combination of these techniques.

The central contribution of GLoRe is the introduction of learned, pair-dependent neighbor relevance estimation into retrieval-based GCD. While CMS relies on global embedding similarity to select neighbors for mean-shift representation learning, GLoRe investigates whether this selection can be improved by exploiting the relationship between global image representations and local patch-level information. This distinction is particularly important for fine-grained categories, where visually similar images may belong to different classes due to subtle local differences.

Our bidirectional reranker addresses this limitation by conditioning each image's local representations on the global descriptor of the other image, enabling complementary visual evidence to influence neighbor selection. Importantly, these comparisons are learned through category-agnostic pairwise supervision and applied to both known and novel categories. The resulting neighbor selection directly affects the mean-shifted representations used for contrastive learning, rather than serving merely as a retrieval post-processing operation.

The experimental evidence supports the relevance of this contribution. Table 12 demonstrates the importance of bidirectional comparison, while Figure 2 (Appendix C) shows improvements in retrieval precision on novel categories of +8.99 and +11.40 percentage points on CUB and Stanford Cars, respectively, despite the absence of direct reranker supervision for these categories.

Full references for [e, m] appear in our General Response.

Reviewer_9ioqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes GLoRe, a bidirectional global-to-local reranking framework for generalized category discovery, improving neighborhood selection through learned semantic relevance, momentum-based feature banks, and regularization, achieving strong fine-grained recognition performance.

Critical strengths

The proposed global-to-local reranking strategy addresses a relevant limitation of similarity-based neighborhood selection in fine-grained GCD.

Critical weaknesses
  1. Poor presentation quality. The framework figure is cluttered and difficult to follow. Inconsistent citation formatting, mathematical notation, and terminology further undermine readability and technical clarity.
  2. Insufficient comparisons with recent SOTA methods. The experimental comparisons rely heavily on older baselines, with limited coverage of recent 2025–2026 approaches, weakening the claimed state-of-the-art performance.
  3. Concerns about experimental fairness. GLoRe performs poorly on Herbarium19 (40.6%). The authors introduce an additional parametric classifier to improve performance to 47.3%, without consistently evaluating this modification across other datasets.
  4. Insufficient performance attribution. KoLeo alone improves Stanford Cars accuracy by 10.4 points, exceeding the reranker's 8.0-point gain. More comprehensive controlled ablations are needed to establish the contribution of the proposed reranking mechanism.
Questions for authors

please see weaknesses.

1 comment
AuthorsOpenReview ↗

Rebuttal by Authors for R2 (#9ioq)

We sincerely thank R2 (#9ioq) for the detailed assessment and constructive suggestions. The comments have helped us strengthen the presentation and experimental comparisons, and clarify the contribution of the individual components. We address each concern below.

R2.1 – Presentation and framework figure

Response: We revised Figure 1 and carefully reviewed the manuscript to identify and correct inconsistencies in citation formatting, mathematical notation, and terminology.

R2.2 – Comparison with recent methods

Response: We thank the reviewer for this observation. To address this concern, we have substantially expanded the experimental comparisons across all three evaluation groups—coarse-grained datasets, fine-grained datasets, and Herbarium19—incorporating a broad selection of recent methods published in 2025 and 2026.

Specifically, the updated comparisons also include FAIR [a], LEAH [b], BIA [c], EAGC [d], TAR [e], CoGe-GCD [f], CPF-GCD [g], RCA [h], FourEx [i], HVGCD [j], ClearGCD [k], and RPC [l], alongside established GCD baselines. Despite this substantially stronger comparison set, GLoRe shares the highest average All accuracy among the vision-only DINOv1 methods across the SSB datasets (Table 1), demonstrating that its performance remains competitive with the latest developments in the field.

A key consideration when interpreting these results is the extent of backbone fine-tuning. GLoRe follows the standard protocol of fine-tuning only the final transformer block of the DINO-pretrained backbone. In contrast, SelEx reports fine-tuning the final two blocks, while FourEx [i] fine-tunes the final three. These configurations therefore differ in the number of trainable backbone parameters and the scope of representation adaptation. Using the same pretrained backbone does not, by itself, establish an equivalent fine-tuning protocol. This distinction should be considered when assessing relative performance. Having this in mind, GLoRe’s leading average SSB performance is particularly compelling in this context, as it is achieved while retaining the more restrictive single-block fine-tuning protocol.

R2.3 – Parametric classifier on Herbarium19

Response: We would like to clarify that the addition of a parametric classifier on Herbarium19 is presented as an exploratory analysis rather than a modification to our proposed methodology or an additional claim of state-of-the-art performance. Our original GLoRe framework remains unchanged, with its performance reported as 40.6%.

Our motivation for this experiment stems from an observation in the Herbarium19 results (Table 2). Methods employing parametric classification objectives generally achieve stronger performance than non-parametric approaches, including GCD, PromptCAL, SelEx, CMS, and our original GLoRe. Given the severe class imbalance and fine-grained nature of Herbarium19, we sought to investigate whether this trend would also hold for our framework. We therefore incorporated the established classification head and objective from SimGCD without introducing any new architectural components beyond this adaptation.

The resulting improvement from 40.6% to 47.3% supports the observation that explicit parametric classification can substantially benefit performance on this benchmark. We do not present this configuration as a separate proposed method, nor claim that this experiment establishes superiority over other approaches under identical settings. Its purpose is to examine a dataset-specific observation and better understand the limitations of non-parametric approaches that might justify this performance discrepancy.

R2.4 – Contribution of KoLeo and the reranker

Response: We acknowledge the comment regarding the improved performance from the KoLeo regularizer. Table 4 is an incremental ablation for GLoRe to study each component’s cumulative contribution. Nevertheless, we have included various ablations in the appendix regarding the momentum coefficient, number of retrieved neighbors, neighborhood temperature scaling, number of patches, and reranker architecture (Appendices D–E and G–H).

Full references for [a–l] appear in our General Response.

Reviewer_iiRYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the Generalized Category Discovery (GCD) task, which aims to discover novel categories from unlabeled data while correctly classifying known categories. The authors propose GLoRe, a bidirectional global-to-local reranking framework to learn better representations from unlabeled data. Given candidates retrieved using global similarity, GLoRe uses cross-attention between each image’s global descriptor and the other image’s local patch descriptors to estimate pairwise relevance. The reranked neighbors are then used for neighborhood contrastive learning. The method also incorporates a momentum feature bank and KoLeo regularization. Experiments on six GCD benchmarks report strong results, particularly on fine-grained datasets.

Critical strengths
  1. The motivation that globally similar images may differ in category-defining local details, leading to noisy neighbors, is intuitive and clear.
  2. With DINOv1 features, GLoRe improves over the listed methods on all three fine-grained benchmarks in Table 1. With DINOv2, it also reports substantial gains on CUB and Stanford Cars.
  3. The ablation studies analyzed in the paper are useful and insightful.
Critical weaknesses
  1. Despite the improved performance, reranking with cross-attention is widely used in the retrieval area, so combining it with the existing neighborhood contrastive learning limits the technical contribution of the paper.
  2. The proposed method achieves encouraging performance improvement; however, this paper provides limited insight into how the reranker trained on labeled known categories transfers to novel categories and when this transfer succeeds or fails.
  3. The reproduced CMS baseline in Table 1 does not match the number reported in the ablation study baseline in Table 4.
  4. There is no comparison against other non-learning reranking methods such as k-reciprocal / Jaccard re-ranking.
  5. The t-SNE visualization should compare a stronger menthod rather than the baseline DINO.
Questions for authors
  1. How are the hyper-parameters selected?
2 comments
AuthorsOpenReview ↗

Rebuttal by Authors for R1 (#iiRY)

We sincerely thank R1 (#iiRY) for the careful reading and constructive feedback, and for recognizing the performance improvements achieved by GLoRe. Below, we address the concerns regarding the technical contribution, transfer to novel categories, and experimental validation.

R1.1 – Technical contribution and relation to retrieval reranking

Response: To the best of our knowledge, GLoRe is the first to introduce learned reranking for neighbor selection in GCD. Building on CMS, it uses bidirectional global-to-local comparisons to refine the neighbors that form mean-shifted training representations. Local patch information helps distinguish fine-grained classes with similar global representations, improving the learning signal for both known and novel categories. Table 12 shows that adding the reverse attention direction improves novel-class accuracy from 72.4% to 73.6% on CUB and from 62.9% to 68.0% on Stanford Cars.

R1.2 – Transfer to novel categories and its limitations

Response: The reranker learns a category-agnostic binary relation indicating whether two images belong to the same category. Bidirectional global-to-local attention encourages transferable visual matching by comparing each image’s global descriptor with the other image’s local features. Figure 2 (Appendix C) measures P@8 across labeled known, unlabeled known, and unlabeled novel queries. Comparable gains on unlabeled known and novel categories support transfer beyond supervised class identities. Ground-truth labels of unlabeled samples are used exclusively for this diagnostic.

Transfer depends on the candidate pool and discriminative local features. If the initial 128 candidates contain no same-category image, reranking cannot recover a correct neighbor. The bounded correction in Eq. 5 (β = 0.1) permits rank reversals only when the initial cosine-score gap is below β. At convergence, P@8 improves over global cosine retrieval by the following percentage points.

Query groupCUBStanford Cars
Labeled, known classes+0.02+0.06
Unlabeled, known classes+8.18+13.77
Unlabeled, unknown classes+8.99+11.40

R1.3 – CMS results in Tables 1 and 4

Response: Table 1 gives the overall clustering accuracy of the full CMS method, 68.2% on CUB and 56.9% on Stanford Cars, matching the results reported on CMS. The first row of Table 4 is a reimplementation of CMS. It retains the CMS contrastive mean-shift training objective and disables iterative mean-shift refinement (IMS) at inference, yielding 66.2% on CUB and 54.0% on Stanford Cars. Each subsequent row adds one component, with IMS introduced only in the final row.

R1.4 – Comparison with non-learning reranking methods

Response: An ablation study was added in Appendix L (Table 16), evaluating GLoRe against the baseline without the reranker on Stanford Cars and CUB. Specifically, we compare the baseline with reciprocal and Jaccard reranking methods using 64 and 128 neighbors.

R1.5 – t-SNE comparison with CMS

Response: A t-SNE visualization has been added to Appendix K.2 (Figure 5). The comparison now includes the original DINOv1, CMS, and GLoRe visualizations with 50 different classes from the Stanford Cars dataset.

R1.6 – Hyperparameter selection

Response: Hyperparameters were selected empirically through preliminary experiments, following standard practice in deep learning. We evaluated different configurations to assess their effects on training stability and clustering performance, and retained the settings that provided consistent performance across the considered benchmarks. There is an extensive analysis in the appendix of hyperparameters regarding the momentum coefficient, number of retrieved neighbors, neighborhood temperature scaling, number of patches, and reranker architecture (Appendices D–E and G–H).

Thanks for your responses. However, my concerns about the overall quality of this paper remain, so I decided to maintain my original score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response of Submission 33219 by Authors

We thank R1 (#iiRY), R2 (#9ioq), and R3 (#ZZX8) for their thoughtful and constructive feedback.

We have addressed the comments in the responses below, organized by reviewer and question index (for example, R1.1 and R3.2), and revised the paper accordingly. The supplementary material is included in the same PDF, following the References section.

The major updates are summarized below.

  • Main paper, Figure 1 and Sections 2.2 and 4.2–4.5
    Revised the framework figure and clarified the methodology, notation and terminology. (R1.1, R2.1, R3.2)

  • Main paper, Tables 1–3
    Expanded comparisons with recent 2025–2026 methods [a–l]. Table 1 includes both the vision-only MOS+TAR and multimodal GET+TAR configurations [e, m], with backbone fine-tuning settings indicated. Table 2 includes GET+TAR on Herbarium19. (R2.2, R3.1)

  • Appendix K.2, Figure 5 and Appendix L, Table 16
    Added t-SNE visualization for CMS on 50 Stanford Cars classes. (R1.5)

  • Appendix L, Table 16
    Added comparisons of our proposed reranker against reciprocal and Jaccard reranking. (R1.4)

  • Main paper, Table 2 and Section 5.2
    Clarified that adding the SimGCD classification head and objective on Herbarium19 is an exploratory analysis. Standard GLoRe achieves 40.6% All accuracy, while this additional experiment achieves 47.3%. (R2.3)

References

  • [a] Bian, Y., et al. “Foundation-Adaptive Integrated Refinement for Generalized Category Discovery.” AAAI, 2026.

  • [b] Duan, Y., et al. “Learning Intrinsic Hierarchy for Generalized Category Discovery.” AAAI, 2026.

  • [c] Tang, L., et al. “Bures-Isotropy Alignment: Manifold Learning of Generalized Category Discovery.” ICLR, 2026.

  • [d] Zheng, H., et al. “The Devil Is in Gradient Entanglement: Energy-Aware Gradient Coordinator for Robust Generalized Category Discovery.” CVPR, 2026.

  • [e] Yang, X., et al. “TAR: Token-Aware Refinement for Fine-grained Generalized Category Discovery.” CVPR, 2026.

  • [f] Tang, L., et al. “CoGe-GCD: Reframing Generalized Category Discovery with Compositional Generalization.” ICML, 2026.

  • [g] Dai, B., Chen, C., and Yu, Y. “Identifying Latent Concepts and Structures for Generalized Category Discovery.” ICML, 2026.

  • [h] Yu, J., et al. “Reliable Confidence Alignment for Generalized Category Discovery.” ICML, 2026.

  • [i] Rastegar, S., et al. “Fourier Self-Supervision for Fine-Grained Generalized Category Discovery.” ECCV, 2026.

  • [j] Zhang, B., et al. “HVGCD: Rethinking Generalized Category Discovery through Hypothesis–Verification.” ECCV, 2026.

  • [k] Lyu, K., et al. “ClearGCD: Mitigating Shortcut Learning for Robust Generalized Category Discovery.” ICASSP, 2026.

  • [l] Xu, Y., et al. “Relational Retrieval: Leveraging Known-Novel Interactions for Generalized Category Discovery.” ICMR, 2026.

  • [m] Wang, E., et al. “GET: Unlocking the Multi-modal Potential of CLIP for Generalized Category Discovery.” CVPR, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.