GeoHOC: Geometric Hierarchical Out-of-Distribution Classification via Nested Angular Caps

Many visual recognition tasks involve both hierarchical class structures and unknown classes in an open world. Hierarchical classification focuses on fine-grained recognition along the class hierarchy, while out-of-distribution recognition concerns when an input no longer belongs to the existing classes. Unifying the two requires jointly modeling the semantic direction and acceptance extent of each class, so that the model can both localize an appropriate semantic branch and determine the proper prediction depth. Based on this perspective, we propose GeoHOC, a geometric framework that jointly models semantic directions and acceptance extents. In a normalized vision–language space, each category node is represented by an angular cap with a learnable center and radius. A joint loss encourages correct branch selection and parent–child containment, while promoting image acceptance by correct caps and rejection by incorrect sibling caps. During inference, center similarity selects a candidate child branch, while a calibrated cap-based acceptance rule determines whether to descend or stop at the current node. We further introduce a known-only calibration scheme to select the stopping offset of this acceptance rule. Experiments on three benchmark datasets demonstrate that GeoHOC achieves the best overall performance among the compared methods, effectively balancing known class recognition and ancestor recovery for unseen classes.