acceptodds
Under review as a conference paper at ICLR 2027

Concept-Blobs: The Geometry of Concepts and Attribute Binding in Foundation Models

Abstract

How are concepts geometrically organized within foundation-model representa- tion space? How can an attribute, such as color, be bound to a specific object? While existing literature often models concepts as one-dimensional linear direc- tions or as a decomposition into sparse features, we propose a paradigm shift by modeling the geometry as Concept-Blobs. A blob of a concept chas a com- pact effective support, parametrized by a mean vector and a covariance matrix (µc,Σc), inducing a Mahalanobis metric within the blob. Rather than collapsing a concept into a single vector or an unconstrained manifold, Concept-Blobs ex- plicitly capture intra-concept variance across attributes such as position, lighting, pose, and color. We show that this approach models multi-concept scenes sig- nificantly better. We model concept composition as a scaled Minkowski sum of individual blobs. Across various models in both visual and textual domains (CLIP, DINOv3, AIMv2, and SigLIP2 for images; CLIP, SigLIP2, Qwen3-Embedding, and Sentence-T5 for text), we reveal two key observations in a fully controlled multi-concept setting based on CLEVR: (1) the residual information left after composing individual concepts, which we call representation dark matter, can be largely explained by inter-concept relationships through a dedicated relational “and” blob; and (2) attribute binding emerges naturally from the geometry, associ- ating attributes with their corresponding objects without requiring explicit binding constraints. These results suggest that foundation-model representations encode concepts not as isolated directions, but as structured probabilistic regions whose geometry supports both relational composition and attribute binding.

Primary area:
foundation or frontier models, including LLMs
Keywords:
concept geometry, attribute binding, compositionality, representation geometry, foundation models, CLIP, Minkowski sum, convex optimization

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_DXRMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces a geometric framework for modeling concepts as bounded regions in foundation-model embedding spaces rather than fixed vectors. Each concept is characterized by its mean, covariance, and a low-dimensional PCA subspace, allowing its representation to capture variations in attributes such as color and material. The authors formulate multi-concept composition as a scaled Minkowski sum and introduce an additional “and” blob to account for residual information that cannot be explained by the individual concepts alone. By reformulating the decomposition objective as a second-order cone program, the proposed method can recover concept-specific representations through constrained optimization without explicitly training an attribute-binding objective.

Critical strengths
  1. The paper introduces an interesting geometric perspective by representing concepts as bounded regions rather than fixed vectors, allowing the model to capture attribute variations within each concept.
  2. The proposed decomposition method provides a principled and tractable optimization framework for recovering concept-specific representations without explicit attribute-binding supervision.
  3. The experiments cover multiple foundation models and both image and text domains, with additional evaluations on three-object scenes and real images, providing encouraging evidence of the method's effectiveness and generality.
Critical weaknesses
  1. The construction of Concept-Blobs appears to rely heavily on additional annotated data, as the concept-specific statistics need to be estimated from labeled single-object samples, while the “and” blob requires paired composite scenes and their corresponding single-object constituents. This may limit the scalability of the proposed framework when extending to a large number of concepts or more complex real-world scenarios.
  2. In Eq. 2, the weights wcw_c are used to combine concept embeddings, but their interpretation remains somewhat unclear. In particular, it is not clear why the weights are not constrained to sum to one, as is common in many weighted-combination formulations.
  3. There exists a discrepancy between individual decompositions and composite reconstruction. For instance, in Figs. 3, 4, and 11, the individually recovered representations appear reasonably similar to the corresponding objects in the original scenes. However, the representation obtained by combining the recovered components seems to differ noticeably from the original composite representation.
  4. The proposed method is closely related to CRL (Conditional Representation Learning for Customized Tasks, NeurIPS 2025), which also investigates the relationship between foundation-model representations and corresponding attributes. A brief discussion of CRL in the Introduction or Related Work section would help better position the contribution of the proposed method.
  5. The authors should check and correct the formula for clarity and consistency with the method described in the main text. For instance, the composition formula for “Concept-Blobs w/o ‘and’” in Table 1 appears to be missing the term xcx_c.
Questions for authors

Please address my concerns mentioned in the weaknesses section.

3 comments

We thank the reviewer for the helpful comments. We uploaded a revised PDF with all changes highlighted in yellow.

1. Reliance on annotated data and scalability. We added a sample-size ablation (Appendix G, summarized in Section 5): concept blobs saturate at about 250 single-object samples per concept for images and 500 for text, and the "and" blob already reaches close to its full-data accuracy with a few dozen composites; adding more composites beyond that mainly gives a small but persistent improvement in position. Regarding the scalability of the "and" blob, it is a single blob shared by all concept pairs, not one per pair, and it is evaluated on objects that never appear in any composite. The method also applies to real images (ImageNet cats and dogs, Figure 4 and Appendix J), where the concept blobs need only class-labeled single-animal crops.

2. Why the weights wcw_c are not constrained to sum to one. The weights are free positive scales by design. Embedding norms are known to reflect frequency and uncertainty, so a rare composition may have a different norm than its parts (Section 3.2). A sum-to-one constraint would impose a fixed norm on the scene. In practice, however, the fitted object weights concentrate near 0.5 and their sum stays close to 1 in every encoder and both modalities (Appendix C, Table 4), so the model recovers this behavior without imposing it.

3. Discrepancy between the recovered components and the composite. The full-scene image in Figures 3, 4 and 11 is not decoded from a recombination of the recovered components; it is decoded directly from the original scene embedding ee. The difference therefore reflects the decoder (Stable unCLIP struggles to render several objects from one embedding), not a reconstruction error of the decomposition. We now state this explicitly in the caption of Figure 3. In fact, this is the point of the figure. Even when the full-scene decode mixes or loses attributes, each recovered component decodes to its own object.

4. Relation to CRL. Thank you for the pointer. We added CRL to Related Work (Section 2). CRL reads out a chosen attribute by a training-free re-representation along fixed directions shared by all objects, whereas in our model attribute displacement depends on the object (Section 4.3).

5. Formula of Concept-Blobs w/o "and" in Table 1. Thank you, this is fixed. The Composition column of Table 1 now uses one notation, defined in the caption, and the row reads e^=∑owoxo\hat e = \sum_o w_o x_o with xo∈Box_o \in \mathcal{B}_o, consistent with Eq. 3; in the full model xo∈Bo∪Bandx_o \in \mathcal{B}_o \cup \mathcal{B}_{\mathrm{and}}.

We hope these changes address the reviewer's concerns.

Regarding the reviewer's concern about more complex real-world scenarios, we extended the evaluation to PUG:SPAR (new Appendix K, Table 7, referenced in Section 5), a photorealistic dataset rendered with Unreal Engine. Each animal's concept blob is calibrated from only 80 single-animal renders (augmented by crops, flips and mild photometric jitter), and the "and" blob from composites of 8 training animals. We test on the 240 composites of 3 held-out animals (zebra, penguin, rabbit), none of which appears in any composite used for calibration. Probes for location and texture, trained only on single-animal renders, are applied to each recovered component, with the probe ceiling on genuine single renders in parentheses.

EncoderLocationTextureBoth
CLIP ViT-L/140.725 (0.896)0.925 (0.963)0.696 (0.867)
DINOv3 ViT-L/160.952 (0.983)0.979 (0.975)0.938 (0.958)

Even with this small amount of data per concept and with unseen animals, DINOv3 binds both attributes to the correct animal for 94% of the recovered components, 98% of its probe ceiling. This supports that the framework extends beyond synthetic CLEVR scenes and does not require large annotated datasets.

Thanks for the responses. My earlier concerns have been addressed, and I would like to maintain my original positive score.

Reviewer_JyrjOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Concept Blob and argues that a concept takes up a whole region in the embedding space, modeled by a mean vector and covariance, instead of a single fixed direction. The concept composition is modeled by Minkowski sum, and the unexplained residual is further modeled by a concept "and". The author claims this explains two puzzles: the issue of unexplained residual ("dark matter"), and attribute binding.

Critical strengths
  1. The perspective seems new and challenges the existing research.
  2. The examples favorably supports the new perspective.
  3. The paper is well written.
Critical weaknesses
  1. Most experiments are synthetic. Extensive real world examples/evaluations are still lacking.
  2. To my understanding, the explanation of "and" as a relationship is interesting but largely remains a plausible hypothesis. The claim that the "and" captures relationships, like "left of", is not directly examined. The evidence only shows "another object."
Questions for authors
  1. Can you add more real world examples to illustrate usefulness of Concept Blob?
  2. In the paper, binding largely relies on the fact that different concepts have different means and different directions of variation, which makes the Minkowski sum splittable. What if in the image there are two objects of the same concept, like "red ball" and "blue ball"?
  3. Can you add experiments to further validate the explanation of "and" as relationships like "in front of", "left of", etc.?
2 comments

We thank the reviewer for the positive assessment and the thoughtful questions. We uploaded a revised PDF with all changes highlighted in yellow.

1. Real-world examples. We deliberately rely mostly on synthetic data: to isolate two concepts and measure whether each attribute is bound to the correct object, we need exact per-object labels and full control over which objects and attributes co-occur, which real datasets rarely provide. Beyond this controlled setting, we already evaluate on real images (ImageNet cats and dogs, Figure 4 and Appendix J). In the revision we added PUG:SPAR (Appendix K, Table 7, referenced in Section 5), a photorealistic dataset rendered with Unreal Engine. It combines realistic scenes with exact per-object labels, so it allows a rigorous evaluation in a setting much closer to real images. The results are promising: with all test animals held out from every composite used for calibration, DINOv3 binds both location and texture to the correct animal for 94% of the recovered components, 98% of its probe ceiling.

2. Two objects of the same concept ("red ball" and "blue ball"). We thank the reviewer for this insightful question, which points to a real limitation of our current setup. With one blob per shape, both objects draw from the same blob, and the decomposition collapses: it cannot tell the two spheres apart, and we now discuss this in the Limitations. However, the blob does not have to be defined at the level of shape. Concepts are organized hierarchically (Figure 10), and we can choose the level at which blobs are built. Going one step down, for {sphere,metal,blue}\lbrace \mathrm{sphere}, \mathrm{metal}, \mathrm{blue} \rbrace and {sphere,rubber,red}\lbrace \mathrm{sphere}, \mathrm{rubber}, \mathrm{red} \rbrace we can build a "metal sphere" blob and a "rubber sphere" blob in the same way, including corresponding samples. The two objects then belong to different blobs, and the decomposition binds each color to the correct sphere. Conversely, going one step up, a single "geometric object" blob covering all shapes (sphere, cube, cylinder, ...) would make every pair in our dataset a same-concept pair, and the decomposition would fail on all of them. The level of the blobs therefore determines which objects can be separated.

We tested this on new CLEVR renders of two objects of the same shape (CLIP ViT-L/14). Color is the color accuracy of the recovered components:

ScenesBlobsStatusColor
Different shapes (reference)shape✅ Succeeded0.82
Same shapeshape❌ Collapsed0.51
Same shape, different materials(shape, material)✅ Succeeded0.90

With one blob per shape the two components are identical in every scene. With one step down in the hierarchy the collapse disappears and colors are bound to the correct object again. The same holds for text: on CLIP captions of same-shape pairs, (shape, material) blobs raise color accuracy from 0.57 to 0.97.

3. The "and" blob as relationships. We are very grateful for this question, which led us to a much better understanding of what the "and" blob actually encodes. Following it, we tested the relational interpretation directly. On CLEVR renders with random object positions (CLIP ViT-L/14), we trained linear probes on the recovered "and" component xand⋆x_{\mathrm{and}}^\star. We distinguish two kinds of information. Spatial relations such as "A is left of B" or "A is in front of B" depend on the order of the objects: swapping them flips the answer. Interactions such as occlusion (one object hides part of the other) and proximity (the objects are close) are properties of the pair and do not depend on the order. As a control we use the recovered object components. All values are AUC (chance 0.50).

FeaturesSpatial: Left ofSpatial: In front ofInteraction: OcclusionInteraction: Proximity
Object components x1⋆,x2⋆x_1^\star, x_2^\star0.570.690.730.58
"And" component xand⋆x_{\mathrm{and}}^\star0.540.580.960.81

The "and" component does not encode spatial relations such as "left of" or "in front of". It is near chance and no better than the object components. It does, however, strongly encode the interactions between the two objects, whether one occludes the other and how close they are, well above the object components. In other words, the "and" component tells how the two objects relate as a pair, not which one is where. The result replicates on DINOv3 ViT-L/16 (occlusion 0.95, proximity 0.85). These findings help us better understand what the "and" blob encodes, and we therefore revised the paper (Section 3.3 and Appendix B) to describe the "and" blob as capturing co-occurrence and interactions between objects (occlusion, proximity) rather than spatial relations such as "left of".

We thank the reviewer again for the encouraging review.

I thank the author for the further explanations. I maintain my positive assessment.

Reviewer_NHonOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a new geometric model of how concepts are organized in the embedding spaces of models. The main difference from prior work is that prior work modelled concepts as single vectors. In contrast, the authors introduce Concept-blobs, where a concept occupies a region in embedding space, and different variations of that concept are inside that region. So, attributes like colors or materials are not concepts by themselves; embeddings of 'blue ball' or 'fluffy ball' would be inside the concept blob of ball. The scene composed of multiple objects is modeled as a Minkowski sum of concept blobs. Importantly, the authors define an 'and' blob, which they hypothesize models the relations between objects and explains the residual from concept sums. The authors decompose embeddings via convex optimization. They test their framework across different image and text embedders and the CLEVR dataset.

Critical strengths

The ideas introduced in the paper are interesting and offer an alternative view of the geometry of the embedding spaces and how they represent binding. The paper follows a clear structure, and the authors explain the mathematical framework well.

Critical weaknesses

Limited validation of the decomposed concept representations. The qualitative results using unCLIP look interesting and promising. However, it is not clear whether the recovered concepts are correct beyond the unCLIP results. The results in Table 1 and Figure 5 operate on the reconstructed full scene embedding. However, it would be interesting if the authors performed further analysis on the recovered components. For example, if the decomposed term should represent 'yellow cube', can we confirm it's really that with perhaps retrieval to the 'yellow cube' embedding among other options or by applying probes on the components and not the full scene embedding? This could be useful for validating the textual results too, since there is no unCLIP equivalent for text. I believe this would strengthen the decomposition and framework's validity.

Limited comparison to prior work. The comparison in Table 1 focuses on the additive decomposition from [1]. However, [1] also investigates multiplicative interactions for concept binding. I think the authors should provide a quantitative comparison to that framework and discuss how these two frameworks contrast and compare to each other.

[1] Uselis, A., Koishigarina, D., & Oh, S.J. "How can embedding models bind concepts?" ICML 2026.

Questions for authors

Even though I put 'weak rejection' as my rating, I am open to increasing my score if the authors provide clarifications for the questions below.

Clarification questions:

  1. Could the authors explain Figure 2 in more detail? Currently, it is not clear how the illustration demonstrates the scaled Minkowski sum.
  2. Could the authors explain the equations in Table 1? In their own concept blob composition, what is the difference between the first and second equations? There is additional information about other methods in the Appendix. However, for a general reader, the differences between them should be clear without referring to the Appendix. I suggest using unified notation to make the distinctions between the methods more explicit.
  3. Could the authors explain in the main paper how they generate images for reconstructions? I think it would also help if the authors provide more clarity, either in the caption or text, on what they mean by 'full-scene decode ' and 'recovered object 1' in Figure 3.
  4. Could the authors clarify what is meant by "restricting evaluation to a single attribute value" in Section 4.3 (line 360)? How do they restrict the evaluation? Do they reuse the same probes from before, or do they train new probes on a restricted set?

Other questions:

  1. How sensitive is the method to the number of single-concept samples used to estimate the mean and covariance of concept blobs? It would be interesting to see how performance changes with scale and how much data is needed to obtain reliable concept blobs.

  2. Could the authors extend their findings beyond CLEVR, for example, to PUG:SPAR or PUG:SPARE?

3 comments

We thank the reviewer for the helpful comments. We uploaded a revised PDF with all changes highlighted in yellow.

1. Validation of the recovered components. The probes in Table 1 and Figure 5 are already applied to each recovered component, not to the full scene; the captions now say this. The probes are trained only on single-object embeddings, never on full scenes or on recovered components, so they test whether each recovered component looks like a real single object with the right attributes. Following your suggestion, we also added a probe-free retrieval test (Appendix E, Table 5, referenced in Section 5): the results agree with the probe evaluation, with over 80% of recovered components retrieving their own color and material first, in both images and text.

2. Comparison to the multiplicative model of Uselis et al.

What we added. The per-object multiplicative model (each object is the sum of its concept-value vectors plus their element-wise product) is now in Table 1 as "Concept-level add + mult" (details in Appendix H). Concept-Blobs outperform it on all attributes. The differentiation between the two frameworks is discussed in Related Work (Section 2).

A different protocol. Uselis et al. fit the multiplicative model on labeled two-object scenes. In Table 1, however, the test composites contain only shapes that never appear in any training scene, so a scene-fitted model has no learned parameters for the test objects. We therefore fit it on single-object embeddings, which is not the setting it was designed for.

Comparison under their protocol. To give the multiplicative model a fair test, we also ran their own setting (their Sec. 5.4): all three of their models (additive, per-object products, global product) are fitted on labeled two-object scenes of the seen shapes. The held-out scenes are built from the same 8 shapes, but in combinations (both objects' shape, color and material) that were not seen during fitting. Metric: mean of color and material accuracy over both objects.

EncoderUselis: AdditiveUselis: Per-obj. productsUselis: Global productConcept-Blobs w/o "and"Concept-Blobs full
ImageCLIP ViT-L/140.5760.8970.6300.8700.918
DINOv3 ViT-B/160.5330.8700.5220.8910.929
DINOv3 ViT-L/160.5160.8260.5380.8640.929
AIMv2 L/140.5000.8640.5760.7610.935
SigLIP2 B/160.5650.8750.6090.7770.891
TextCLIP ViT-L/140.5270.8150.6630.8750.951
SigLIP2 B/160.5380.8640.7770.6680.848
Qwen3-Emb 0.6B0.5870.9400.8210.6960.929
Sentence-T5 base0.4840.8640.7660.5760.870

3. Figure 2. We rewrote the caption and elaborated on how the scaled object embeddings and the "and" term add up to the scene embedding inside the Minkowski sum.

4. Equations in Table 1. Thank you for noticing this, we appreciate and agree with your suggestion to use unified notation. The Composition column now uses one notation, defined in the caption. The two Concept-Blobs rows differ only in the "and" blob term (without / with).

5. Image generation and Figure 3. All images are decoded with Stable unCLIP from an embedding alone. "Full-scene decode" is decoded from the scene embedding, and "recovered object" from the decomposed component. This is now stated in the caption of Figure 3. These generated images provide a better qualitative impression of what the embedding captures, complementing the quantitative measurements in the accuracy and retrieval tables.

6. "Restricting evaluation to a single attribute value" (Section 4.3). These are new shape probes, trained only on samples of one attribute value (e.g., only metal objects). We reworded the sentence to make this explicit.

7. Number of samples per blob. We added a sample-size ablation (Appendix G, summarized in Section 5). Concept blobs saturate at about 250 samples per concept for images and 500 for text, and the "and" blob is nearly insensitive to its sample size for color and material. Thanks to your critique here, we found another interesting observation: more "and" blob samples improve position (left/right), strengthening the relational information it is meant to capture.

8. Beyond CLEVR (PUG:SPAR). This experiment is in progress, and we will post the results here before the end of the discussion period.

We hope these additional experiments, together with our clarifications, address your concerns, and we would be grateful if you would consider raising the score accordingly.

I thank the authors for providing additional experiments, addressing my questions, and improving explanations. I increase my score.

We sincerely thank the reviewer for the constructive discussion and for increasing the score.

As promised, we also extended the evaluation beyond CLEVR to PUG:SPAR (new Appendix K, Table 7, referenced in Section 5), a photorealistic dataset rendered with Unreal Engine that still provides exact per-object labels. We follow the held-out protocol of our CLEVR experiments. The concept blobs of 11 animals are calibrated from single-animal renders, the "and" blob is built from composites of 8 training animals, and we test on the 240 composites of the 3 held-out animals (zebra, penguin, rabbit), none of which appears in any composite used for calibration. Probes for location (left, right, below, above) and texture, trained only on single-animal renders, are applied to each recovered component. Both means that location and texture are both correct, and the probe ceiling on genuine single renders is in parentheses.

EncoderLocationTextureBoth
CLIP ViT-L/140.725 (0.896)0.925 (0.963)0.696 (0.867)
DINOv3 ViT-L/160.952 (0.983)0.979 (0.975)0.938 (0.958)

The CLEVR findings carry over to photorealistic scenes. DINOv3 binds both attributes to the correct animal for 94% of the recovered components, 98% of its probe ceiling. CLIP binds texture well but location less so, in line with DINOv3 binding position better on CLEVR (Figure 5).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.