GeoGCD: Geometry-Aware Prototype Learning for Histopathology Generalized Cancer Category Discovery

Generalized Category Discovery (GCD) aims to simultaneously recognize labeled categories and discover novel categories from partially annotated data. Histopathology generalized category discovery is particularly challenging due to the subtle morphological differences among cancer subtypes and the substantial appearance variations within the same subtype, which often lead to ambiguous decision boundaries and less discriminative feature representations. Existing GCD methods mainly focus on classifier optimization or clustering objectives while paying limited attention to preserving fine-grained histopathological structures and exploiting the geometric relationships among category prototypes.To address these challenges, we propose GeoGCD, a geometry-aware prototype learning framework for histopathology generalized category discovery. Specifically, we introduce a Feature Detail Refinement and Aggregation (FDRA) module that enhances discriminative local structural representations by preserving fine-grained pathological details while suppressing redundant feature responses. Furthermore, we develop an Adaptive Margin Prototype Adapter (AMPA) module that dynamically calibrates classification margins according to the geometric relationships among category prototypes, encouraging better discrimination between confusing cancer subtypes. By jointly improving local structural representations and prototype distributions, GeoGCD learns more compact intra-class representations and better-separated inter-class prototypes, leading to more reliable novel category discovery.Extensive experiments on multiple histopathology benchmark datasets demonstrate that GeoGCD consistently outperforms existing generalized category discovery methods under realistic semi-supervised settings, validating its effectiveness for discovering previously unseen cancer categories.