When [CLS] Falls Short: Object-Aware Feature Displacement for Robust Generalized Category Discovery

Generalized Category Discovery (GCD) aims to recognize known categories and discover novel ones from unlabeled data by leveraging knowledge from a partially labeled set. Most existing methods primarily rely on the global [CLS] token of Vision Transformers as the image representation. Although holistic and highly semantic, the view-invariant [CLS] feature may retain background and appearance cues that remain stable across augmentations, reducing the separability of novel categories. In this paper, we propose Object-Aware Feature Displacement (OADisp), a plug-and-play paradigm that improves the global representation without replacing [CLS] or retraining the backbone. OADisp exploits the rich local semantics encoded in patch features to form complementary guidance for adapting the original [CLS] toward an image-conditioned target, thereby preserving holistic global semantics while incorporating localized discriminative cues. Specifically, we instantiate this guidance with object-aware semantic descriptors constructed through mask-guided aggregation of fragmented patch features, and further regularize their extraction with frequency-domain perturbation consistency to improve guidance reliability. Without any domain-shift-specific design, we extensively evaluate OADisp on a wide range of 9 datasets under both standard and the more challenging Domain-Generalized settings, together with comprehensive ablation studies and quantitative analyses. OADisp consistently improves overall performance, demonstrating the effectiveness and robustness of the resulting representations for open-world category discovery.