Concept-Blobs: The Geometry of Concepts and Attribute Binding in Foundation Models

How are concepts geometrically organized within foundation-model representa- tion space? How can an attribute, such as color, be bound to a specific object? While existing literature often models concepts as one-dimensional linear direc- tions or as a decomposition into sparse features, we propose a paradigm shift by modeling the geometry as Concept-Blobs. A blob of a concept chas a com- pact effective support, parametrized by a mean vector and a covariance matrix (µc,Σc), inducing a Mahalanobis metric within the blob. Rather than collapsing a concept into a single vector or an unconstrained manifold, Concept-Blobs ex- plicitly capture intra-concept variance across attributes such as position, lighting, pose, and color. We show that this approach models multi-concept scenes sig- nificantly better. We model concept composition as a scaled Minkowski sum of individual blobs. Across various models in both visual and textual domains (CLIP, DINOv3, AIMv2, and SigLIP2 for images; CLIP, SigLIP2, Qwen3-Embedding, and Sentence-T5 for text), we reveal two key observations in a fully controlled multi-concept setting based on CLEVR: (1) the residual information left after composing individual concepts, which we call representation dark matter, can be largely explained by inter-concept relationships through a dedicated relational “and” blob; and (2) attribute binding emerges naturally from the geometry, associ- ating attributes with their corresponding objects without requiring explicit binding constraints. These results suggest that foundation-model representations encode concepts not as isolated directions, but as structured probabilistic regions whose geometry supports both relational composition and attribute binding.