When [CLS] Falls Short: Object-Aware Feature Displacement for Robust Generalized Category Discovery
Desk rejected at ICLR 2027, so left out of the paper lists and the map.
Abstract
Generalized Category Discovery (GCD) aims to recognize known categories and discover novel ones from unlabeled data by leveraging knowledge from a partially labeled set. Most existing methods primarily rely on the global [CLS] token of Vision Transformers as the image representation. Although holistic and highly semantic, the view-invariant [CLS] feature may retain background and appearance cues that remain stable across augmentations, reducing the separability of novel categories. In this paper, we propose Object-Aware Feature Displacement (OADisp), a plug-and-play paradigm that improves the global representation without replacing [CLS] or retraining the backbone. OADisp exploits the rich local semantics encoded in patch features to form complementary guidance for adapting the original [CLS] toward an image-conditioned target, thereby preserving holistic global semantics while incorporating localized discriminative cues. Specifically, we instantiate this guidance with object-aware semantic descriptors constructed through mask-guided aggregation of fragmented patch features, and further regularize their extraction with frequency-domain perturbation consistency to improve guidance reliability. Without any domain-shift-specific design, we extensively evaluate OADisp on a wide range of 9 datasets under both standard and the more challenging Domain-Generalized settings, together with comprehensive ablation studies and quantitative analyses. OADisp consistently improves overall performance, demonstrating the effectiveness and robustness of the resulting representations for open-world category discovery.