On the Geometry of Hierarchical Concept Spaces in Vision Models: Testing Linear and Hyperbolic Representations

Semantic concepts provide a human-interpretable basis for understanding the representations learned by vision models. Under the Linear Representation Hypothesis (LRH), concepts are modeled as linear directions in a flat embedding space, where they can be composed and translated relative to one another. Meanwhile, prior work has shown that semantic concepts are often organized hierarchically, which raises a fundamental question: is such hierarchical organization geometrically compatible with LRH? We introduce *residual reconstruction*, a geometry-agnostic framework that formalizes geometric compatibility as a quantitative measure, allowing us to assess how well a given geometry captures the structure of hierarchical concepts. We apply it to the flat linear geometry underlying LRH, evaluating the compatibility across four distinct vision models on ImageNet and its associated WordNet hierarchy. Our results show that hierarchical concepts exhibit substantially larger residual reconstruction error, indicating a systematic mismatch between hierarchical organization and the flat additive geometry underlying the LRH formulation. We then investigate whether hyperbolic geometry better captures the structure of hierarchical concepts. Across the same models and hierarchies, hyperbolic representations yield substantially lower residual reconstruction error, providing empirical evidence that the hierarchical concept structures are better captured by hyperbolic geometry than by the corresponding flat geometry. Our work provides a general quantitative framework for studying the geometric compatibility of semantic structures and reveals a systematic tension between hierarchical concepts and linear representations in vision models.