acceptodds
Withdrawn from ICLR 2027

SCOPE: Self-Supervised Concept Discovery via Preference Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Self-supervised learning can learn useful representations from unlabeled images. Yet these representations remain largely opaque, making it difficult to understand what visual concepts they encode. Can self-supervision instead produce useful visual concepts that are directly understandable to people? We introduce SCOPE, a framework that represents images as sets of concepts expressed in natural language. A vision-language model proposes candidate concepts, and a self-supervised objective evaluates how well each concept describes the source image while distinguishing it from other images. This feedback is used to construct preferences and train the model to propose better concepts. Across eight diverse visual datasets, our self-supervised concept representations transfer without adaptation and improve downstream recognition accuracy over prior interpretable approaches. In medical imaging, our discovered concepts reveal spurious visual cues and enable a more robust classifier. These results show that self-supervision can discover useful, human-readable visual representations from unlabeled images.

Primary area:
interpretability and explainable AI
Keywords:
Interpretability, Self Supervised Learning, Preference Learning, Concept-Based Models.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_vZ7aOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

S2COPE is a framework for discovering concepts from unlabeled images, without classes/names/labels. A VLLM generates several descriptions for each image, which are then ranked using a contrastive model (such as an image-text model like CLIP) to further fine-tune the VLLM using Direct Preference Optimization (DPO) to better align a concept for images. Evaluation includes linear-probe benchmark and a human and VLLM preference study.

Critical strengths

Evaluation covers a variety of datasets from diverse categories (nature, medical, physics, and facial images). The evaluation shows the model’s ability of generalization to different types of datasets.
Extensive evaluation, reporting linear-probing accuracy across 8 datasets, concept preferences of VLLM and Human, ablations which highlight gains from different modules of model and show why such components/settings were chosen (reward, no augmentations, pair selection, model). Implementation details are reported (hyperparameters, pseudo-code of algorithm) – supporting a reproducibility

Critical weaknesses

Concepts should refer to high-level understandable properties/attributes that appear across many images, they should be shared across instances within same group/class/species [1,2]. For example: “soccer ball” and “basketball” might both share the concept “round”. Proposed method does not prioritize such concepts: they are not explicitly encouraged and are often omitted. Instead, model focuses on proposing concept i.e. descriptions that clearly identify the image. In this case, CLIP would favor the concept “basketball” over “round” since the latter is shared across many images. This results in an objective that identifies the image, not abstracting from it (Figure[3], baby girl, young adult female). On the other hand, concepts should not be too generic, which paper evaluates. How does the model ensure that discovered concepts are shared across instances of the same class?
Proposed framework is very similar to proposed framework in CLIP-DPO [3], the application is different, as S2COPE is used to create concepts for datasets, whereas CLIP-DPO is used to fix hallucinations in VLLMs. Could authors elaborate on specific methodological differences? Minor: Lack of statistical significance (results across different seeds). Methodology is not straightforward and is difficult to follow. It could be described more clearly.

[1] B Kim et al. Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV) [2] T Oikarinen et al. Label-Free Concept Bottleneck Models [3] Y Ouali et al. CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMs,

Questions for authors
  1. Why does the human study omit evaluation against baseline models?
  2. Could authors provide prompt used for concept generation? Is it post-processed?
  3. As the main contribution of paper is new method of concept discovering, paper would benefit from broader evaluation of concept quality (ex. How many concepts activate, is it dense?, sparse? What is the criteria of selecting top concepts? How concepts are shared across classes?). Authors presented only 3-5 concepts without enough context (Figure 3, Figure 8). On the other side Figure 9 shows 10 concepts in User study form. Could you specify the selection criteria?
3 comments

We thank the reviewer for recognizing S²COPE's generalization across diverse domains. We are glad that the reviewer finds that our extensive experiments provide strong support for our core idea.

Weakness 1: Concept abstraction and sharing across instances

We respectfully disagree that our objective necessarily favors instance-specific descriptions over reusable concepts. Our objective encourages discrimination through the composition of concepts, not through each concept individually identifying an image. For example, “round” is useful for distinguishing both soccer balls and basketballs from bananas, even though it cannot distinguish soccer balls from basketballs. Additional concepts can provide the remaining distinctions. Thus, different images can share individual concepts while being distinguished by their combinations.

This is analogous to SimCLR, where instance-level contrastive learning nevertheless produces reusable semantic features shared across images. Similarly, our instance-discriminative objective does not require each discovered concept to be unique to an instance. Therefore, the reviewer's conclusion that instance discrimination necessarily prevents concept sharing does not follow from the objective.

We agree that cross-instance concept sharing deserves more explicit empirical analysis. We have added a concept-frequency histogram and an analysis of cross-instance sharing in Appendix A.9 (Figure 7 and Table 11). On the iNaturalist training set (7,136 images, 26 classes), 96.7% of concept occurrences (image–concept pairs) come from concepts shared by multiple images, and 91.4% come from concepts shared by at least 10 images. For example, the shape description “round fruits” recurs in 83 images across 8 classes. Class labels are used only for this post-hoc sharing analysis. These results show that the instance-discriminative objective allows individual concepts to be reused across images and classes.

Weakness 2: Methodological differences from CLIP-DPO

We thank the reviewer for pointing out this related work. We will cite and discuss it in our revision. Our approach differs from CLIP-DPO in three key ways:

  1. Self-supervised contrastive learning. S²COPE discovers discriminative concepts directly from unlabeled images through cross-image comparisons, without predefined categories or concept annotations; CLIP-DPO optimizes image–caption alignment.

  2. Explicit concept representation. S²COPE constructs a shared concept vocabulary for representing images and making interpretable predictions, whereas CLIP-DPO directly improves the model's text responses.

  3. Iterative training. S²COPE alternates between concept generation and model optimization, regenerating its training preferences each round; CLIP-DPO fine-tunes on a fixed preference dataset.

Weakness 3: Statistical significance across seeds

The results in Table 1 are stable across four random seeds. As shown in Figure 4(a), accuracy also remains stable during the final optimization iterations. We report the mean, variance, and 95% confidence interval for each dataset in the table below.

DatasetMeanVariance95% CI
iNaturalist84.710.04[84.41, 85.02]
CUB85.030.01[84.88, 85.17]
HAM1000078.760.36[77.80, 79.71]
Gravity Spy82.860.64[81.59, 84.13]
Galaxy1066.390.02[66.16, 66.62]
OrganCMNIST91.330.01[91.13, 91.52]
OrganMNIST3D71.891.38[70.02, 73.77]
FairFace58.950.00[58.84, 59.06]

Weakness 4: Clarity of the methodology

We will clarify the methodology in the revision. S²COPE repeats three steps: generate candidate descriptions, rank them using the contrastive reward, and update the generator through DPO. After training, we freeze the generator and aggregate its concepts into a shared bank for linear-probe evaluation.

To be continued.

Question 1: Human evaluation against baselines

Conducting a large-scale human evaluation of all baselines is challenging. We therefore compared S²COPE with LF-CBM, the strongest baseline by average accuracy across the eight datasets. All four participants included in this analysis unanimously preferred S²COPE on every question (20/20 each), yielding a mean preference rate of 100.00% (standard deviation: 0.00 percentage points). This result supports the improved accuracy and specificity of our concepts relative to the strongest baseline in this study.

OpponentWin rate of S²COPE (%)
LF-CBM100.00 ± 0.00

Each participant answered 20 questions, choosing the concept list that more accurately described the object in each image. We report the mean win rate ± standard deviation across the four included participants.

Question 2: Prompt used for concept generation

Yes, we use the same prompt for concept generation across all datasets:

Describe the visual features in the image. Output only short visual feature phrases separated by commas, with no explanation.

It is used without post-processing.

Question 3: Broader evaluation of concept quality

We thank the reviewer for this suggestion. The table below reports the size of each concept bank.

iNaturalistCUBGalaxy10Gravity SpyHAM10000OrganCMNISTOrganMNIST3DFairFace
# Concepts8,5516,0792,9992,6133,1536,5511,9583,440

Concept selection criteria: Generated descriptions are parsed into phrases, pooled across training images, and deduplicated using CLIP text similarity. Each image is represented by its CLIP similarities to all concepts in the resulting bank for linear probing, without manually selecting a top-k subset.

Concept sharing across classes: The bank is constructed without class names or labels and is shared by all classes. Each concept supplies the same activation feature to every class, while the linear probe learns its weight separately for each class. A concept can thus support multiple classes.

Due to space constraints, Figures 3 and 9 show only a small selection of concepts with the highest CLIP text–image cosine similarity scores for each image. Figure 10 presents the responses from both models, allowing human evaluators to compare their outputs fairly.

AuthorsOpenReview ↗

Brief Follow-up on Our Response

We thank the reviewer for their time and consideration. We hope our responses have addressed their questions and concerns, and would appreciate any further feedback. We would be happy to clarify any remaining points.

Reviewer_CLRSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

S²COPE compares image description candidates generated by VLLM using the image-text contrast scores of the frozen CLIPs and iteratively trains VLLM with the preference pairs via DPO. In the concept discovery stage, the target class name or image label is not used. Then, create a concept bank with the target training images and evaluate classification performance using CLIP similarity and label-based linear probing. The paper reports an average improvement of +7.37 percentage points over the highest performance of each of the four concept-based baselines compared across the eight datasets.

Critical strengths
  • Meaningful problem framing and a clear approach. In situations where it is difficult to define classes or concepts in advance, the approach of improving a natural language concept generator with label-free images is useful. The proposed reward and learning procedure are also relatively clear.
  • Consistent results across various domains. In the settings reported in the paper, S²COPE demonstrates higher accuracy than the four comparison baselines and unoptimized VLLM across all eight datasets. However, these results are limited to the relevant control group and protocol.
  • I analyzed multiple components. Ablutions regarding reward composition, augmentation, VLLM size, and preference pair selection allow us to examine the operational factors of the method.
Critical weaknesses
  • It is difficult to separate the quality of the concept from the CLIP fit effect. The same OpenCLIP family is used for calculating learning rewards and final concept activations. Having an independent encoder evaluation or a fixed generator CLIP reranking baseline makes it clearer what DPO has improved.
  • Direct verification of interpretability is limited. It uses natural language concepts and linear classifiers, but the number, validity, and explanatory fidelity of concepts that actually contribute to individual predictions are not evaluated sufficiently. We cannot conclude that part by improving classification accuracy alone.
  • Careful attention is required when interpreting the shortcut results. SONIC group information is used to select the concept cluster to filter in ISIC. Additionally, the patch error rates for the two probes are calculated from the image that was correct before patch insertion, so the evaluation targets may differ. Therefore, it is difficult to conclude that a 15 percentage point decrease represents an improvement in the same sample.
  • Stability of the results and comparison conditions are needed. Variance or confidence intervals by seed were not reported in the main tables, and there is a lack of quantitative comparison with direct reranking baselines and similar automatic concept discovery methods
Questions for authors
  • CLIP dependencies: Does the improvement hold if evaluated with encoders other than OpenCLIP used for the training reward and final concept activation? Can we also compare the baseline, which reranks the same candidate with a CLIP reward, without training a generator?
  • Interpretability: What is the size of the concept bank for each dataset and how many concepts do they contribute meaningfully to individual predictions? Can we also evaluate accuracy and explanatory fidelity when using only a small number of key concepts?
  • Stability of results: Do the results in Table 1 hold up across multiple training and sampling seeds? Can I provide mean, variance, or confidence intervals for each dataset?
3 comments

We thank the reviewer for their constructive feedback. We are glad that the reviewer finds our problem framing meaningful and our approach clear, and recognizes the consistent gains across diverse domains.

Weakness 1: Additional ablations with different encoders

We thank the reviewer for these suggestions. Following the reviewer's suggestion, we evaluated S²COPE using three different encoders—OpenCLIP ViT-H/14, OpenAI CLIP ViT-L/14, and SigLIP2 SO400M—and compared it with the fixed-generator CLIP reranking baseline (Best-of-16 reranking). The results are shown in the table below.

With independent evaluation encoders, S²COPE outperforms reranking by 4.62–7.69 percentage points, supporting gains that generalize beyond the reward encoder. S²COPE achieves higher accuracy than fixed-generator reranking while internalizing concept discovery in the generator's parameters through DPO, enabling direct generation without inference-time reranking.

Across encoder architectures, S²COPE consistently outperforms the reranking baseline by 4.62–7.69 percentage points.

Evaluation encoderBaseReranking (Best-of-16)S²COPE
OpenCLIP ViT-H/1469.2377.6983.46
OpenAI CLIP ViT-L/1463.4673.0880.77
SigLIP2 SO400M64.2378.4683.08

Best-of-16 reranking selects the highest-reward description from 16 samples generated by the frozen generator. Independent evaluation encoders are used for concept deduplication and activation computation in place of OpenCLIP; the training reward remains unchanged.

Weakness 2: Direct evaluation of interpretability

We thank the reviewer for suggesting a more direct evaluation of interpretability. Following the reviewer's suggestion, we conducted experiments to further evaluate concept-bank size and concept validity:

  1. Concept-bank size: S²COPE achieves 76.92% accuracy and 85.00% fidelity with only 650 concepts (7.6% of the bank). We select the top-k concepts by their importance in the full probe and retrain the same probe on this subset. Accuracy decreases as fewer concepts are retained, from 85.00% with all 8,551 concepts to 58.85% with 100.
Concepts kept (k)Accuracy (%)Fidelity (%)
10058.8563.08
16064.6270.38
65076.9285.00
8,551 (full)85.00100.00
  1. Concept validity: A probe trained on the 650 concepts with the highest validity achieves 76.54% accuracy, compared with 69.23% for a probe trained on the 650 concepts with the lowest validity. This gap supports the benefit of visually grounded concepts for classification. This experiment selects concepts by CLIP validity, whereas the table above selects them by probe importance.
Concept group (650 each)Accuracy (%)
Highest validity76.54
Lowest validity69.23

Here, validity refers to whether a concept describes a visual feature actually present in the source images. We use the concept's mean CLIP similarity to the images that generated it as a proxy for validity: a high score suggests that the concept is actually visible in those images, while a low score suggests a hallucinated or off-object description. We rank all concepts by this score and retrain the same probe on the top or bottom 650 concepts.

Weakness 3: Clarification of supervision and evaluation in shortcut mitigation

We clarify that SONIC labels are used only to select discovered clusters for downstream filtering; concept discovery and generator training use no SONIC labels. The 15-percentage-point decrease (74.9% to 59.9%) compares flip rates on each probe's own initially correct examples, so it does not establish improved robustness on an identical sample set. We will make this explicit in the revision.

To be continued.

Weakness 4: Results across multiple random seeds

Our approach is stable, and our gain over the baseline is substantial. Following the reviewer's suggestion, we ran experiments with multiple random seeds and compared S²COPE with the direct reranking baseline. We will include these results in the revision.

Our results are stable across four random seeds, as shown in the table below. As shown in Figure 4(a), accuracy also remains stable during the final optimization iterations.

DatasetMeanVariance95% CI
iNaturalist84.710.04[84.41, 85.02]
CUB85.030.01[84.88, 85.17]
HAM1000078.760.36[77.80, 79.71]
Gravity Spy82.860.64[81.59, 84.13]
Galaxy1066.390.02[66.16, 66.62]
OrganCMNIST91.330.01[91.13, 91.52]
OrganMNIST3D71.891.38[70.02, 73.77]
FairFace58.950.00[58.84, 59.06]

Comparison with the direct reranking baseline: Direct reranking improves accuracy over the base VLLM but remains 5.77 percentage points below S²COPE. The results are shown in the table below.

Method (iNaturalist-mini)Accuracy (%)
Base VLLM69.23
Reranking (Best-of-16)77.69
S²COPE (Ours)83.46

For each image, the reranking baseline samples 16 descriptions from the unoptimized VLLM and keeps the one with the highest CLIP reward. The VLLM remains frozen and receives no DPO training.

Question 1: Additional ablations with different encoders and comparison with direct reranking

Yes. S²COPE retains its gains with independent evaluation encoders and outperforms a CLIP reranking baseline that uses a frozen generator. Following the reviewer's suggestion, we conducted both experiments on iNaturalist-mini.

  1. When evaluated with OpenCLIP ViT-H/14, OpenAI CLIP ViT-L/14, and SigLIP2 SO400M, S²COPE consistently and substantially outperforms the unoptimized baselines. The results are shown in the table below.

  2. Across encoder architectures, S²COPE consistently outperforms the reranking baseline by 4.62–7.69 percentage points.

Evaluation encoderBaseReranking (Best-of-16)S²COPE
OpenCLIP ViT-H/1469.2377.6983.46
OpenAI CLIP ViT-L/1463.4673.0880.77
SigLIP2 SO400M64.2378.4683.08

Best-of-16 reranking selects the highest-reward description from 16 samples generated by the frozen generator. Independent evaluation encoders are used for concept deduplication and activation computation in place of OpenCLIP; the training reward remains unchanged.

Question 2: Additional probing experiments with different concept-bank sizes

We thank the reviewer for this suggestion. The table below reports the size of each concept bank.

iNaturalistCUBGalaxy10Gravity SpyHAM10000OrganCMNISTOrganMNIST3DFairFace
# Concepts8,5516,0792,9992,6133,1536,5511,9583,440

On iNaturalist, S²COPE achieves 76.92% accuracy and 85.00% fidelity with only 650 concepts (7.6% of the bank). We select the top-k concepts by their importance in the full probe and retrain the same probe on this subset. Accuracy decreases as fewer concepts are retained, from 85.00% with all 8,551 concepts to 58.85% with 100. Fidelity is the fraction of images for which the reduced probe's prediction matches that of the full probe. The results are shown in the table below.

Concepts kept (k)Accuracy (%)Fidelity (%)
10058.8563.08
16064.6270.38
65076.9285.00
8,551 (full)85.00100.00

Question 3: Additional experiments with multiple random seeds

Yes, the results in Table 1 are stable across four random seeds. As shown in Figure 4(a), accuracy also remains stable during the final optimization iterations. We report the mean, variance, and 95% confidence interval for each dataset in the table below.

DatasetMeanVariance95% CI
iNaturalist84.710.04[84.41, 85.02]
CUB85.030.01[84.88, 85.17]
HAM1000078.760.36[77.80, 79.71]
Gravity Spy82.860.64[81.59, 84.13]
Galaxy1066.390.02[66.16, 66.62]
OrganCMNIST91.330.01[91.13, 91.52]
OrganMNIST3D71.891.38[70.02, 73.77]
FairFace58.950.00[58.84, 59.06]
AuthorsOpenReview ↗

Brief Follow-up on Our Response

We thank the reviewer for their time and consideration. We hope our responses have addressed their questions and concerns, and would appreciate any further feedback. We would be happy to clarify any remaining points.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.