- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces a geometric framework for modeling concepts as bounded regions in foundation-model embedding spaces rather than fixed vectors. Each concept is characterized by its mean, covariance, and a low-dimensional PCA subspace, allowing its representation to capture variations in attributes such as color and material. The authors formulate multi-concept composition as a scaled Minkowski sum and introduce an additional “and” blob to account for residual information that cannot be explained by the individual concepts alone. By reformulating the decomposition objective as a second-order cone program, the proposed method can recover concept-specific representations through constrained optimization without explicitly training an attribute-binding objective.
Critical strengths
- The paper introduces an interesting geometric perspective by representing concepts as bounded regions rather than fixed vectors, allowing the model to capture attribute variations within each concept.
- The proposed decomposition method provides a principled and tractable optimization framework for recovering concept-specific representations without explicit attribute-binding supervision.
- The experiments cover multiple foundation models and both image and text domains, with additional evaluations on three-object scenes and real images, providing encouraging evidence of the method's effectiveness and generality.
Critical weaknesses
- The construction of Concept-Blobs appears to rely heavily on additional annotated data, as the concept-specific statistics need to be estimated from labeled single-object samples, while the “and” blob requires paired composite scenes and their corresponding single-object constituents. This may limit the scalability of the proposed framework when extending to a large number of concepts or more complex real-world scenarios.
- In Eq. 2, the weights are used to combine concept embeddings, but their interpretation remains somewhat unclear. In particular, it is not clear why the weights are not constrained to sum to one, as is common in many weighted-combination formulations.
- There exists a discrepancy between individual decompositions and composite reconstruction. For instance, in Figs. 3, 4, and 11, the individually recovered representations appear reasonably similar to the corresponding objects in the original scenes. However, the representation obtained by combining the recovered components seems to differ noticeably from the original composite representation.
- The proposed method is closely related to CRL (Conditional Representation Learning for Customized Tasks, NeurIPS 2025), which also investigates the relationship between foundation-model representations and corresponding attributes. A brief discussion of CRL in the Introduction or Related Work section would help better position the contribution of the proposed method.
- The authors should check and correct the formula for clarity and consistency with the method described in the main text. For instance, the composition formula for “Concept-Blobs w/o ‘and’” in Table 1 appears to be missing the term .
Questions for authors
Please address my concerns mentioned in the weaknesses section.
3 comments
Regarding the reviewer's concern about more complex real-world scenarios, we extended the evaluation to PUG:SPAR (new Appendix K, Table 7, referenced in Section 5), a photorealistic dataset rendered with Unreal Engine. Each animal's concept blob is calibrated from only 80 single-animal renders (augmented by crops, flips and mild photometric jitter), and the "and" blob from composites of 8 training animals. We test on the 240 composites of 3 held-out animals (zebra, penguin, rabbit), none of which appears in any composite used for calibration. Probes for location and texture, trained only on single-animal renders, are applied to each recovered component, with the probe ceiling on genuine single renders in parentheses.
| Encoder | Location | Texture | Both |
|---|---|---|---|
| CLIP ViT-L/14 | 0.725 (0.896) | 0.925 (0.963) | 0.696 (0.867) |
| DINOv3 ViT-L/16 | 0.952 (0.983) | 0.979 (0.975) | 0.938 (0.958) |
Even with this small amount of data per concept and with unseen animals, DINOv3 binds both attributes to the correct animal for 94% of the recovered components, 98% of its probe ceiling. This supports that the framework extends beyond synthetic CLEVR scenes and does not require large annotated datasets.
Thanks for the responses. My earlier concerns have been addressed, and I would like to maintain my original positive score.
We thank the reviewer for the helpful comments. We uploaded a revised PDF with all changes highlighted in yellow.
1. Reliance on annotated data and scalability. We added a sample-size ablation (Appendix G, summarized in Section 5): concept blobs saturate at about 250 single-object samples per concept for images and 500 for text, and the "and" blob already reaches close to its full-data accuracy with a few dozen composites; adding more composites beyond that mainly gives a small but persistent improvement in position. Regarding the scalability of the "and" blob, it is a single blob shared by all concept pairs, not one per pair, and it is evaluated on objects that never appear in any composite. The method also applies to real images (ImageNet cats and dogs, Figure 4 and Appendix J), where the concept blobs need only class-labeled single-animal crops.
2. Why the weights wc are not constrained to sum to one. The weights are free positive scales by design. Embedding norms are known to reflect frequency and uncertainty, so a rare composition may have a different norm than its parts (Section 3.2). A sum-to-one constraint would impose a fixed norm on the scene. In practice, however, the fitted object weights concentrate near 0.5 and their sum stays close to 1 in every encoder and both modalities (Appendix C, Table 4), so the model recovers this behavior without imposing it.
3. Discrepancy between the recovered components and the composite. The full-scene image in Figures 3, 4 and 11 is not decoded from a recombination of the recovered components; it is decoded directly from the original scene embedding e. The difference therefore reflects the decoder (Stable unCLIP struggles to render several objects from one embedding), not a reconstruction error of the decomposition. We now state this explicitly in the caption of Figure 3. In fact, this is the point of the figure. Even when the full-scene decode mixes or loses attributes, each recovered component decodes to its own object.
4. Relation to CRL. Thank you for the pointer. We added CRL to Related Work (Section 2). CRL reads out a chosen attribute by a training-free re-representation along fixed directions shared by all objects, whereas in our model attribute displacement depends on the object (Section 4.3).
5. Formula of Concept-Blobs w/o "and" in Table 1. Thank you, this is fixed. The Composition column of Table 1 now uses one notation, defined in the caption, and the row reads e^=∑owoxo with xo∈Bo, consistent with Eq. 3; in the full model xo∈Bo∪Band.
We hope these changes address the reviewer's concerns.