Familiarity-Aware Mixture of Energies for Multimodal Generalized Category Discovery

Generalized category discovery (GCD) seeks to recognize labeled old classes while discovering unlabeled novel classes from mixed data. Recent progress increasingly leverages foundation models to provide semantic evidence for category discovery. Yet novel categories are not equally familiar to such models: some are well represented in the pretrained CLIP space, whereas others remain long-tailed, fine-grained, or weakly grounded. Treating semantic evidence with a fixed or globally shared strategy can therefore lead to both semantic overtrust for unfamiliar clusters and semantic underuse for familiar ones. In this paper, we propose FAME, a Familiarity-Aware Mixture of Energies framework for multimodal GCD. FAME estimates cluster familiarity, an unsupervised reliability variable that measures whether CLIP image-space semantics provide stable and discriminative evidence for each nameless cluster, without using ground-truth novel class names or an external concept bank during discovery. It then performs risk-aware energy routing, adaptively combining CLIP-space semantic evidence, local visual cues, and self-supervised cluster structure for category assignment. This design strengthens semantic guidance for familiar and low-risk clusters while shifting uncertain or high-risk clusters toward visual-structural evidence. Assignments, prototypes, familiarity scores, and risk scores are jointly refined in an iterative process. Extensive experiments on standard GCD benchmarks demonstrate the effectiveness of FAME under semantically heterogeneous novel clusters.