Beyond Linearity: The Contextual Geometry of Concepts

Representations of concepts are commonly read as directions within neural activations, assuming that a concept is mediated by a single, context-free vector. We prove that this assumption fails in general because modern architectures inherently encode contextual interactions. Across fifteen models and three modalities, the directions corresponding to the same concept exhibit high variability across contexts. Furthermore, measuring their effective dimension reveals that concepts correspond to multidimensional structures in the activation space. This motivates our novel contextual representations of concepts, which are able to capture these structures across different contexts. Under causal intervention on LLMs, or steering, they allow consistent reductions in perplexity over single directions, highlighting more faithful representation of concepts, and improved control over the behavior of models.