- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
S2COPE is a framework for discovering concepts from unlabeled images, without classes/names/labels. A VLLM generates several descriptions for each image, which are then ranked using a contrastive model (such as an image-text model like CLIP) to further fine-tune the VLLM using Direct Preference Optimization (DPO) to better align a concept for images. Evaluation includes linear-probe benchmark and a human and VLLM preference study.
Critical strengths
Evaluation covers a variety of datasets from diverse categories (nature, medical, physics, and facial images). The evaluation shows the model’s ability of generalization to different types of datasets.
Extensive evaluation, reporting linear-probing accuracy across 8 datasets, concept preferences of VLLM and Human, ablations which highlight gains from different modules of model and show why such components/settings were chosen (reward, no augmentations, pair selection, model).
Implementation details are reported (hyperparameters, pseudo-code of algorithm) – supporting a reproducibility
Critical weaknesses
Concepts should refer to high-level understandable properties/attributes that appear across many images, they should be shared across instances within same group/class/species [1,2]. For example: “soccer ball” and “basketball” might both share the concept “round”. Proposed method does not prioritize such concepts: they are not explicitly encouraged and are often omitted. Instead, model focuses on proposing concept i.e. descriptions that clearly identify the image. In this case, CLIP would favor the concept “basketball” over “round” since the latter is shared across many images. This results in an objective that identifies the image, not abstracting from it (Figure[3], baby girl, young adult female). On the other hand, concepts should not be too generic, which paper evaluates. How does the model ensure that discovered concepts are shared across instances of the same class?
Proposed framework is very similar to proposed framework in CLIP-DPO [3], the application is different, as S2COPE is used to create concepts for datasets, whereas CLIP-DPO is used to fix hallucinations in VLLMs. Could authors elaborate on specific methodological differences?
Minor: Lack of statistical significance (results across different seeds). Methodology is not straightforward and is difficult to follow. It could be described more clearly.
[1] B Kim et al. Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV) [2] T Oikarinen et al. Label-Free Concept Bottleneck Models [3] Y Ouali et al. CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMs,
Questions for authors
- Why does the human study omit evaluation against baseline models?
- Could authors provide prompt used for concept generation? Is it post-processed?
- As the main contribution of paper is new method of concept discovering, paper would benefit from broader evaluation of concept quality (ex. How many concepts activate, is it dense?, sparse? What is the criteria of selecting top concepts? How concepts are shared across classes?). Authors presented only 3-5 concepts without enough context (Figure 3, Figure 8). On the other side Figure 9 shows 10 concepts in User study form. Could you specify the selection criteria?
3 comments
Part 2/2
Question 1: Human evaluation against baselines
Conducting a large-scale human evaluation of all baselines is challenging. We therefore compared S²COPE with LF-CBM, the strongest baseline by average accuracy across the eight datasets. All four participants included in this analysis unanimously preferred S²COPE on every question (20/20 each), yielding a mean preference rate of 100.00% (standard deviation: 0.00 percentage points). This result supports the improved accuracy and specificity of our concepts relative to the strongest baseline in this study.
| Opponent | Win rate of S²COPE (%) |
|---|---|
| LF-CBM | 100.00 ± 0.00 |
Each participant answered 20 questions, choosing the concept list that more accurately described the object in each image. We report the mean win rate ± standard deviation across the four included participants.
Question 2: Prompt used for concept generation
Yes, we use the same prompt for concept generation across all datasets:
Describe the visual features in the image. Output only short visual feature phrases separated by commas, with no explanation.
It is used without post-processing.
Question 3: Broader evaluation of concept quality
We thank the reviewer for this suggestion. The table below reports the size of each concept bank.
| iNaturalist | CUB | Galaxy10 | Gravity Spy | HAM10000 | OrganCMNIST | OrganMNIST3D | FairFace | |
|---|---|---|---|---|---|---|---|---|
| # Concepts | 8,551 | 6,079 | 2,999 | 2,613 | 3,153 | 6,551 | 1,958 | 3,440 |
Concept selection criteria: Generated descriptions are parsed into phrases, pooled across training images, and deduplicated using CLIP text similarity. Each image is represented by its CLIP similarities to all concepts in the resulting bank for linear probing, without manually selecting a top-k subset.
Concept sharing across classes: The bank is constructed without class names or labels and is shared by all classes. Each concept supplies the same activation feature to every class, while the linear probe learns its weight separately for each class. A concept can thus support multiple classes.
Due to space constraints, Figures 3 and 9 show only a small selection of concepts with the highest CLIP text–image cosine similarity scores for each image. Figure 10 presents the responses from both models, allowing human evaluators to compare their outputs fairly.
Brief Follow-up on Our Response
We thank the reviewer for their time and consideration. We hope our responses have addressed their questions and concerns, and would appreciate any further feedback. We would be happy to clarify any remaining points.
Part 1/2
We thank the reviewer for recognizing S²COPE's generalization across diverse domains. We are glad that the reviewer finds that our extensive experiments provide strong support for our core idea.
Weakness 1: Concept abstraction and sharing across instances
We respectfully disagree that our objective necessarily favors instance-specific descriptions over reusable concepts. Our objective encourages discrimination through the composition of concepts, not through each concept individually identifying an image. For example, “round” is useful for distinguishing both soccer balls and basketballs from bananas, even though it cannot distinguish soccer balls from basketballs. Additional concepts can provide the remaining distinctions. Thus, different images can share individual concepts while being distinguished by their combinations.
This is analogous to SimCLR, where instance-level contrastive learning nevertheless produces reusable semantic features shared across images. Similarly, our instance-discriminative objective does not require each discovered concept to be unique to an instance. Therefore, the reviewer's conclusion that instance discrimination necessarily prevents concept sharing does not follow from the objective.
We agree that cross-instance concept sharing deserves more explicit empirical analysis. We have added a concept-frequency histogram and an analysis of cross-instance sharing in Appendix A.9 (Figure 7 and Table 11). On the iNaturalist training set (7,136 images, 26 classes), 96.7% of concept occurrences (image–concept pairs) come from concepts shared by multiple images, and 91.4% come from concepts shared by at least 10 images. For example, the shape description “round fruits” recurs in 83 images across 8 classes. Class labels are used only for this post-hoc sharing analysis. These results show that the instance-discriminative objective allows individual concepts to be reused across images and classes.
Weakness 2: Methodological differences from CLIP-DPO
We thank the reviewer for pointing out this related work. We will cite and discuss it in our revision. Our approach differs from CLIP-DPO in three key ways:
Self-supervised contrastive learning. S²COPE discovers discriminative concepts directly from unlabeled images through cross-image comparisons, without predefined categories or concept annotations; CLIP-DPO optimizes image–caption alignment.
Explicit concept representation. S²COPE constructs a shared concept vocabulary for representing images and making interpretable predictions, whereas CLIP-DPO directly improves the model's text responses.
Iterative training. S²COPE alternates between concept generation and model optimization, regenerating its training preferences each round; CLIP-DPO fine-tunes on a fixed preference dataset.
Weakness 3: Statistical significance across seeds
The results in Table 1 are stable across four random seeds. As shown in Figure 4(a), accuracy also remains stable during the final optimization iterations. We report the mean, variance, and 95% confidence interval for each dataset in the table below.
Weakness 4: Clarity of the methodology
We will clarify the methodology in the revision. S²COPE repeats three steps: generate candidate descriptions, rank them using the contrastive reward, and update the generator through DPO. After training, we freeze the generator and aggregate its concepts into a shared bank for linear-probe evaluation.
To be continued.