acceptodds
Under review as a conference paper at ICLR 2027

Disentangling Latent Embeddings with Sparse Linear Concept Cones (SLiCC)

Abstract

Pretrained vision models encode semantic information in latent embedding spaces that are useful for downstream tasks. We hypothesize that this information can be disentangled to separate the content of complex scenes by decomposing each embedding into multiple concept-specific components. We propose Sparse Linear Concept Cones (SLiCC), a supervised dictionary learning approach that estimates a linear synthesis model consisting of sparse, non-negative combinations of groups of dictionary atoms, whose group-wise activity is guided by multi-label information. Each concept-specific component is a non-negative combination of atoms associated with a label and lies in a corresponding concept cone. We introduce an alternating optimization algorithm and an autoencoder formulation of SLiCC. We apply SLiCC to three types of CLIP and DINOv2 embeddings across five datasets. For CLIP, we additionally exploit image–text alignment to enable a self-supervised SLiCC. We also introduce a fully unsupervised autoencoder that enforces group sparsity based on the norm of neuron activations. Quantitative results show that supervised SLiCC improves concept-filtered image retrieval over unfiltered embeddings, including retrieval of finer-grained categories not provided during training. Beyond retrieval of individual concepts, the disentangled components enable concept deletion and recombination across images. We further explore conditional image generation using Image-to-Prompt, with qualitative results illustrating retrieval and generation from manipulated concept representations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
subspace disentanglement, image retrieval, vision-language model, latent embeddings

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Drd8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a framework for disentangling pretrained visual embeddings into concept-specific components. The key idea is to model each semantic concept as a cone spanned by a group of dictionary atoms, such that an image embedding can be approximately decomposed into a sparse sum of non-negative concept components. The authors develop both a supervised dictionary-learning formulation and an autoencoder-based implementation, and further extend the framework to self-supervised and unsupervised settings. The learned concept-specific representations are mainly evaluated through concept-filtered image retrieval, including both coarse concept retrieval and finer-grained sub-label retrieval.

Critical strengths
  1. The paper is built upon an interesting and relatively novel motivation. Rather than treating pretrained visual embeddings as holistic representations, it explores whether their semantic information can be explicitly decomposed into concept-specific components, which provides an appealing perspective on representation disentanglement and controllable retrieval.
  2. The experimental evaluation is relatively comprehensive. The authors conduct experiments across multiple datasets and pretrained visual backbones, consider both general-label and fine-grained sub-label retrieval, and provide several ablation studies and qualitative analyses to examine different aspects of the proposed framework.
  3. The proposed method is technically well developed. The paper provides both a constrained dictionary-learning formulation and an autoencoder-based implementation, with carefully designed non-negativity and group-sparsity constraints.
Critical weaknesses
  1. The experimental evaluation could be strengthened by including additional relevant baselines. In particular, it would be helpful to compare with existing composed retrieval methods or other approaches designed to focus on user-specified semantic content. At present, a considerable portion of the experimental comparisons are among variants of the proposed framework itself.
  2. The evaluation protocol for the unsupervised variant could also be explained more clearly. Since the learned latent groups do not have predefined semantic labels, the paper associates active concepts with latent groups separately for each query using CLIP text embeddings and greedy exclusive assignment. As a result, the same latent group may correspond to different concepts for different queries.
  3. I would also appreciate some clarification on the hyperparameter-selection protocol. The paper emphasizes that SLiCC is fitted using only general concept labels, while the dictionary size d_0 is selected according to sub-label retrieval performance on the validation set. Therefore, fine-grained labels are not used in the training objective but are involved in model selection.
  4. The visual presentation of the figures could be improved. In addition, Figures 3, 4, and 5 contain several presentation details that could be refined, such as the bounding boxes, spacing, and occasional overlap between text and images.
  5. The related-work discussion could be expanded to include “Conditional Representation Learning for Custom Tasks” (NeurIPS 2025). This work is relevant in spirit, as it studies how to adapt or transform representations to focus on a specified concept or criterion from the perspective of coordinate transformation.
Questions for authors

See the above-mentioned weaknesses.

2 comments
AuthorsOpenReview ↗

Answers for the questions in **Critical Weaknesses**

C1.

We thank the reviewer for the concern. We have included more methods in the revised Table 1, including one projection-based method (CAV), one composed image retrieval method (CLAY), and one supervised metric learning method (CSN). The results show that SLiCC remains more effective in most of the cases, proving the strength of our proposed methods.

C2.

That is correct. For each query, the goal is to assign one unsupervised latent group to each active concept.

Based on the cosine similarity between the CLIP text embeddings and the latent group components, each concept claims the most similar free latent group. If several concepts claim the same group, the one with the highest similarity score wins; the losers try again on the remaining groups. The process stops when every concept is assigned. Since the same procedure is carried out every time a different query comes in, the same group can map to different concepts for different queries.

C3.

That is correct. Even though the fine-grained labels are not used in the training process, the hyperparameters are selected through the sub-label retrieval performance, which requires the information of the finer-grained labels on the validation set. This is necessary to find the size concept cones necessary to preserve meaningful variation within the concept. This is necessary as using only the coarse-grain labels for validation selects a lower-dimensional cone that fails to encode fine-grained information.

C4.

We apologize for the bad presentation of the figures. The figures have been fixed in the revised paper.

C5.

We thank the reviewer for bringing this work to our attention. It is indeed similar in spirit to our approach. Although its customized conditional retrieval takes into account the object factor and the scene factor at the same time, it inspires us to adopt the prompts to generate lists of expressions and use them for conditional visual similarity modulation (CLAY), which is also included in the revised paper as a baseline.

Thanks for the response. My concerns have been resolved, and I will keep my score.

Reviewer_Fw2QOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how to disentangle semantic concepts from pretrained visual embeddings. Instead of treating an image embedding as a single representation, the paper proposes to decompose it into multiple concept-specific components, allowing different semantic information to be accessed and manipulated separately. To this end, it introduces SLiCC, a dictionary learning approach that uses non-negative coefficients and group sparsity to learn concept-specific representations. The authors further develop an autoencoder formulation, along with self-supervised and unsupervised variants. The method is evaluated on five datasets using CLIP and DINOv2 embeddings. The results demonstrate improvements in concept-filtered image retrieval, including finer-grained retrieval beyond the coarse labels used for training. The paper also presents qualitative examples of concept deletion and recombination through image retrieval and generation.

Critical strengths
  1. This paper investigates an interesting problem of disentangling semantic concepts from pretrained visual embeddings. I find the idea of representing different concepts using separate linear cones reasonable, as it allows the model to focus on specific semantic information rather than the entire image representation. The learned components can also be manipulated through concept deletion and recombination, which makes the approach potentially useful for different applications.

  2. The evaluation is relatively comprehensive. The authors conduct experiments on five datasets using multiple CLIP and DINOv2 backbones. The results demonstrate improvements in both general-label and finer-grained retrieval. The paper also includes ablation studies and qualitative visualizations, which provide additional evidence for the effectiveness of the proposed method.

Critical weaknesses
  1. The comparison with existing methods is relatively limited. The paper mainly compares SLiCC with unfiltered retrieval, S-SVD, and SpLiCE. However, SLiCC uses concept labels during training, while some of the compared methods do not benefit from the same supervision. In addition, the paper does not include comparisons with other supervised concept learning approaches, such as concept-based projection or supervised metric learning. As a result, it remains unclear whether the observed improvements come from the proposed concept cone formulation or from the additional concept supervision.

  2. The evaluation of concept disentanglement is not sufficiently convincing. The paper mainly relies on retrieval performance to demonstrate the effectiveness of the learned concept representations. However, improved retrieval performance does not necessarily indicate that semantic concepts have been successfully disentangled. For example, the paper does not quantitatively assess whether removing one concept preserves the remaining concepts or whether different concept representations contain overlapping information. Although qualitative examples of concept deletion and recombination are provided, a more systematic evaluation is needed to support the disentanglement claims.

Questions for authors
  1. Why does the paper not compare SLiCC with other supervised concept learning methods? Can the improvements still be observed when all methods use the same concept labels?

  2. How can the authors verify that the learned representations are truly disentangled? Can they quantitatively evaluate concept deletion and whether information from other concepts is preserved?

  3. How is the number of active concepts determined in the self-supervised setting? Does estimating it from the whole dataset require ground-truth concept labels?

2 comments
AuthorsOpenReview ↗

Answers for questions in **Questions For Authors**

Q1.

We thank the reviewer for the good question. We include more established methods to compare against in the revised Table 1.

S-SVD is a supervised method based on concept-specific projection. For each concept jj, the atoms Bj\mathbf{B}_j are obtained by applying truncated SVD to the embeddings of images labeled as containing that concept. Embeddings are then decomposed by projection with non-negative coefficients.

We also include two more concept-based projection methods. One is concept activation vector (CAV). For each concept jj, we train a linear classifier, and take the normal vector of the decision boundary. We then rank the images by using the cosine similarity score between the image embeddings and the normal vector. The other is CLAY, a method that finds the textual subspace spanned by the expressions conditioned on the concept. After that, the query and the pool image embeddings are projected onto the textual subspace and similarity score is calculated.

We include a supervised metric learning method, conditional similarity network (CSN). Based on the ground truth labels, CSN learns a transformation for each concept to learn to capture the similarity.

The inclusion of these supervised methods in the revised paper makes it clear that SLiCC efficiently disentangles the image embeddings because of the proposed concept cone formulation instead of just the supervised learning.

Q2.

We thank the reviewer for the insightful question. To further analyze the effectiveness of disentanglement, we systematically evaluate the effect of concept deletion on retrieval. For each target concept jj, we modify the query embeddings (only the queries that contain jj and at least one other concept) by deleting either jj or an active non-target concept, and measure unfiltered retrieval performance for jj before and after deletion. The results are shown in Table 3 in A.3.2. The results show that there is a significant decrease when the target concept is deleted and the performances improve when removing a scaled non-target concept.

Similarly, we train a linear classifier for each concept and compute its predicted probability for each query before and after deleting each concept, averaging the predictions across queries. The results are shown in Figure 6 in A.3.2. The results show that the diagonal entries have substantial decreases across all datasets, indicating that deleting a concept reduces its corresponding classifier score.

Both of the experiments further prove the disentanglement of SLiCC.

Q3.

Yes. In self-supervised SLiCC, the number of active concepts is estimated by taking the average active concepts across the whole concepts. This is detailed in the Hyperparameter Selection paragraph.

Thanks for the response. My concerns have been resolved, so I will keep my score.

Reviewer_bCYKOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors propose a method, Sparse Linear Concept Cones (SLiCC), to decompose embeddings of complex scenes into concept components. The method is linear because the complex scene embedding is decomposed into sums of concept components. It's sparse because the concepts that are not present in the image are zeroed out. Concept components live in a cone and are described as a non-negative linear combination of (dictionary) atoms. Both the dictionary atoms and the coefficients of the linear combinations are learned through 1) a dictionary learning algorithm or 2) an autoencoder formulation when the number of concepts or the number of dictionary atoms per concept is large. The authors use different training methods: 1) supervised, when the concept labels (which concepts are present in an image) are available, 2) self-supervised, by using CLIP's zero-shot classification, or 3) unsupervised, when neither is available. The authors show improved concept-based retrieval. That is, if a query contains a cat and other objects in a complex scene and we want to retrieve images only with cats, this method can be used. The authors also show qualitative results for concept deletion and concept recombination using text-to-image generation.

Critical strengths

The idea of concept-based retrieval seems interesting and practical. The authors motivate it well. Images usually contain multiple objects, and a user of a retrieval system might want to retrieve similar images while emphasizing only a particular object. The authors introduce several variations of their concept disentanglement method depending on the nature of the available data (dictionary learning or autoencoder method; supervised, self-supervised, and unsupervised). This potentially increases the practicality of the proposed method.

Critical weaknesses

Insufficient details on evaluation. Although the proposed retrieval method outperforms the baselines, I found the evaluation setup insufficiently explained. In particular, it is unclear how retrieval is performed on the different datasets. The authors also introduce an unsupervised version of their method, but I could not find the corresponding results. It would be useful to evaluate this version on datasets where concept labels are available to measure the performance gap between supervised and unsupervised settings. This is particularly important for real-world applications, where the concept dictionary may not be known in advance. Similarly, the text-to-image generation procedure is not sufficiently explained.

Clarity and presentation. I found the paper difficult to follow due to unclear terminology, dense explanations, and the overall structure.

  • Key terms such as "concept components" and "dictionary atoms" should be clearly defined when introduced in Section 3.1.
  • The "Settings and Comparisons" paragraph on Page 6 introduces many methods and ablations at once, making them difficult to distinguish. Separating the main methods from the ablations, possibly in a table, would improve clarity.
  • Some terms are used without being defined. For example, Table 1 refers to SS-SLiCC, but "SS" has not been introduced.
  • The frequent use of words such as "naturally" (Lines 126, 127, 269) and "simply" (Lines 126, 315, 417) can be misleading when the underlying ideas require further explanation.
  • Many paragraphs are lengthy and could be divided to improve readability.

Overall, I believe these presentation issues substantially affect the paper's clarity and make it difficult to assess the proposed contributions. Given their extent, I am not confident they can be fully addressed through minor revisions, which contributes to my rating.

Questions for authors

Clarification questions:

  1. Does SS-SLiCC mean the self-supervised version of SLiCC? The SS terminology was not introduced but appears in the results tables.
  2. Can you explain further what you mean by synthetic CIFAR-100?
  3. Can you explain in detail how retrieval and image generation work for obtaining the results in Table 1 and Figures 4 and 5?

Other questions:

  1. The authors introduce the unsupervised version of the method, but I cannot find these results. Could the authors show how the unsupervised method performs compared to the supervised and self-supervised methods even when labels are available?
  2. How does the work relate to prior work on the Linear Representation Hypothesis [1]? The ideas of linear decomposition of embeddings seem similar, but this work does not cite it.
  3. How does this decomposition help when a user wants to retrieve more fine-grained concepts? For example, retrieving not just a bag of concepts but combinations of concepts (i.e., not just retrieving similar images with apples, but particularly red apples)?

[1] Park, K., Choe, Y. J., & Veitch, V. (2023). The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03658.

6 comments
AuthorsOpenReview ↗

Answers for questions in **Clarity and presentation**

P1.

We thank the reviewers for asking about the clarification of the fundamental concepts in this paper. “Concept component” refers to the disentangled component associated with a concept. Specifically, the jjth concept component is denoted as vj\mathbf{v}_j. “Dictionary atoms” as in the context of dictionary learning refers to the atoms that comprise the dictionary. The atoms are grouped into {Bj}j=1S\{\mathbf{B}_j\}_{j=1}^S. The atoms in Bj\mathbf{B}_j form the boundary of the jjth concept cone. We have updated the description in Sec. 3.1 in the revision.

P2.

We apologize for the lack of clarity in the introduction of the methods. We have put the methods presented in Table 1 into three categories: Proposed Methods, Baselines, and Ablations.

Proposed Methods includes all variants of SLiCC which include supervised version S-SLiCC (See Algorithm 1) and the sparse autoencoder (AE) S-SLiCC-AE, self-supervised (using zero-shot CLIP for pseudo labels) SS-SLiCC and the sparse autoencoder SS-SLiCC-AE, and the unsupervised sparse autoencoder version US-SLiCC-AE.

Baseline includes all methods we compare against: unfiltered baseline UF, supervised SVD version with no disentanglement S-SVD, SpLiCE, a concept-filtered version that uses group word embeddings F-SpLiCE, CAV, CSN, CLAY.

Ablations include S-SLiCC-FD, S-SLiCC-UC, S-SLiCC-AE-UC, S-SLiCC-AE-N.

P3.

We will remove the unnecessary or misleading cases of “naturally” and “simply”, but keep “simply” when appropriate as in the Ablations paragraph where it is used to describe the ablation that only removes the non-negativity constraint by removing the rectification unit.

AuthorsOpenReview ↗

Answers for questions in **Clarification questions**

C1.

We apologize for failing to state that SS-SLiCC refers to the self-supervised SLiCC introduced in Sec. 3.2.1. Similarly, SS-SLiCC-AE stands for the autoencoder formulation of self-supervised SLiCC. We have fixed it in the Proposed Methods paragraph in the revision.

C2.

In contrast to the CIFAR-100 dataset where each image contains a single concept, we create a synthetic dataset from CIFAR-100 such that images contain multiple concepts by placing two or three CIFAR-100 images into four squared quadrants (upper left, upper right, bottom left, bottom right) randomly, leaving the other one or two quadrants empty with black pixels. By design images from this “synthetic CIFAR-100” contain two or three concepts, each taking up a quadrant. We have clarified the description of the synthetic dataset in Sec. 4.1 in the revision.

C3.

For unfiltered retrieval, we compute the cosine similarity scores between the original query embedding and the image pool embeddings, rank the images in the pool by the score, and retrieve the highest-ranked images. For concept-filtered retrieval, we instead compute these scores using the concept component of the query embedding and rank the images accordingly. The procedure for obtaining the concept component varies across methods. The methods are detailed in Sec. 3.2.1 and Sec. 4.2.

We generate images from concept components using the Image-to-Prompt approach, which enables image generation from embeddings in the CLIP image embedding space. Since the concept components lie in this same space, they can be used as the conditions to generate images reflecting their associated visual concepts. We have clarified it in Sec. 4.2 in the revised paper.

AuthorsOpenReview ↗

Answers for questions in **Other questions**

O1.

We had only presented the qualitative results for the unsupervised SLiCC in the original paper. It was an oversight not to also include the quantitative results of unsupervised SLiCC. In the revision, unsupervised SLiCC (US-SLiCC-AE) appears in the revised Table 1. We see some improvements on general-label retrieval performances compared to unfiltered retrieval. But supervised and self-supervised SLiCC still outperform unsupervised SLiCC because the set of concepts are known to both methods during training.

O2.

We thank the reviewer for bringing this work to our attention. Park et al. (2023) formalizes the linear representation hypothesis for language models, while our paper focuses more on vision embedding like DINOv2 or vision-language co-embedding like CLIP. The paper also affirms our hypothesis that the orientation of a concept in the representation space matters, so the concept is represented in a “cone” instead of a subspace. We now include this paper as a reference and include a discussion of the paper for the similarity in the linear hypothesis and the geometrical structure in the latent space in the Related Work.

O3.

We thank the reviewer for the question about the combination of concepts and fine-grained concepts. In our paper, “concepts” are nouns and refer to a category of objects (“animal”, “vehicle”, etc.) or a location (“outside”, “inside”, “kitchen”, etc.) that have a hierarchy moving from a few coarse grained concepts to more fine-grained concepts. We do not consider concepts that refer to attributes or adjectives, such as the color or shape (for example, “red”) that may be shared across many different categories of objects or locations. Nonetheless, our proposed method disentangles the information of distinct concepts (a category of noun) from the embedding of a complex scene, preserving information about attributes (color, pose, etc.) within the concept cone. Therefore, in this particular example, if there is a red apple and a gray cat in the query, the concept-filtered retrieval for “food” or “fruit” or “apple” should rank images with similar red apples higher than other apple images and higher than images with cats. Likewise, concept-filtering to “animal” should prioritize images of gray cats above all else.

This is also shown in Figure 4, where a little girl in similar-colored clothes (the first row on the left) or a black cat (the second row on the left) is retrieved, instead of just any person or any cat. Figure 5 also shows the preservation of a white boat and a middle-aged man (the first row on the left).

Because we have only trained noun-based concepts, our disentanglement can be applied when the physical objects and locations of the scene could be separated. However, we have not tested whether training on both noun and attributes as concepts could enable concept-filtering with attributes. We have added this to a new limitation section (Sec. 4) before the conclusion as it is not what our paper aims to do.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.