acceptodds
Under review as a conference paper at ICLR 2027

Beyond Concept Generation: Interpretable Concept Scoring for Debiasing CLIP-based CBMs

Abstract

Concept Bottleneck Models (CBMs) explain predictions through human-readable concepts. However, label-free CBMs built on CLIP image-text similarities may inherit biases from CLIP representations, allowing visually or contextually associated concepts to serve as semantically unreliable shortcuts. Addressing such shortcuts is non-trivial: a concept that provides valid evidence for one class may form a misleading association with another. Therefore, globally removing potentially problematic concepts can be harmful, as it may discard useful information for other classes. To address this challenge, we introduce ScoreCBM, an interpretable concept-scoring framework for debiasing CLIP-based CBMs. Moving large language models (LLMs) beyond their conventional role of concept generation, we use them to explicitly score the semantic relevance of class-concept pairs. The resulting scores provide a class-specific semantic prior that guides two complementary interventions: score-guided activation attenuation suppresses low-relevance concept activations during projection learning, while adaptive regularization penalizes low-relevance concept-class connections in the final sparse classifier. Our framework requires neither group labels nor modifications to CLIP, while providing a traceable and editable interface for model intervention. To scale to large datasets, we further introduce CLIP-based top-\(r\) pre-filtering, reducing LLM scoring complexity from quadratic to linear in the number of classes. Experiments across biased and standard benchmarks demonstrate consistent improvements in predictive performance. On BAR, our method improves LF-CBM accuracy from 84.25% to 89.39%, surpassing the 88.91% non-interpretable ERM baseline. The framework further scales to the 1,000-class ImageNet, improving LF-CBM from 71.87% to 75.28%.

Primary area:
interpretability and explainable AI
Keywords:
Concept Bottleneck Models, Debiasing, CLIP

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dkvCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ScoreCBM, which uses a Label-Free CBM like approach, but with explicitly using LLMs to generate a semantic prior that guides the first process of the grounding of images to concepts. This prior is then later also used by the classifier, that uses this knowledge as a regularization. The motivation is to reduce concepts, that are semantically not needed in an explanation of a CBM, which has been proven in the past, that CBMs use concepts that are not necessarily useful.

Critical strengths
  • While LLMs have been widely used in the CBM literature for concept proposal [1], using their semantic knowledge to mitigate spurrious correlations is novel. The ablation shows, that all of the two stages actually improve the overall performance (Table 4) and the general idea of the CBM approach is sound.
  • You test a bunch of different datasets, explicitly separating in small, and large scale datasets. Also the fact that you not only compare your approach against LF-CBM but several different CBM works is nice.

[1] What’s in the Bottle? A Survey and Roadmap of Concept Bottleneck Models, Knab et. al., TMLR 2026

Critical weaknesses
  • Clarity of the method: If the reader does not really know the LF-CBM paper, your problem setup and preliminaries are not easy to follow. E.g., you never motivate ff, that Ai,jA_{i,j} is part of A bold, or generally why LF-CBM trains a projector with weights WgW_g and why this is particularly useful for your case. Also, all before of this, it would have helped me to know how you generate your mm concepts before you start the learning process, right now, you only mention that they exist.

  • Evaluation: The major issue I see right now is the evaluation. Your whole idea is motivated by using the semantic prior of LLMs. However, In the main paper, as well as in the appendix, you only sparsely discuss the characteristics of using different LLMs within your method. Your paper would greatly profit to analyze (i) How differs the semantic prior with different LLMs (in terms of concept agreement by different LLMs) (ii) a full performance evaluation with different LLMs on all datasets, not just one. (iii) Also, since you filter concept for deduplication before you feed them to the LLM, testing how different CLIP models result in deduplication removal would also be interesting. (iv) Next what is not mentioned in the paper, why did you choose different hyperparams for each dataset (Table 7), I would have expected to see a table with results where all share the same configuration. (v) A lot of ablations are focused on one dataset, more datasets would give the results more support. 
However, one hint, you have already some other relevant experiments in the paper, I would have liked it if you would have already referenced them in the main paper.

Questions for authors
  • You said that LLMScore for Sj,kS_{j,k} does not measure whether a concept is visually observable? I am not sure if this is problematic, since then it might be possible that ScoreCBM learns to assign concepts, the LLM has associated with the class, which are not even there, thus this might harm interpretability.
  • In your LLMScore prompt, what happens with the case if the LLM returns that a feature is unclear, are you simply treating the score as 0, or do you make sth with it during the process not mentioned in the text? Otherwise, an investigation of these features would have been insightful (see A.2)
  • in Appendix B.2, I think it is relevant to calculate an LLM agreement of Gemini and Qwen2.5, so if they have a similar prior, and if this can be used to explain the difference in accuracy. In addition, just the evaluation on just one dataset is in my opinion not enough. You can even think about moving this part later into the main paper with more experiments, since i think it is important to ablate your method design. 
- A nice baseline would have been using the binary states of the LLM produced prior to directly train a CBM, to see how large the gap between binary states, and clip refined states actually is.
  • Can you explain why with a corruption ratio of 80% the performance is still that high? You slightly discussed it in the appendix, but this raises questions if this behavior is even wanted. Could you also tell the performance with p=90p = 90, or even p=95p = 95?
5 comments

We thank the reviewer for the detailed and constructive feedback. We have organized our response around two main aspects: clarifying the role of LLM-derived semantic relevance in ScoreCBM and strengthening the evaluation of its reliability and robustness.

First, we clarify an important distinction underlying several questions: LLMScore provides a class–concept semantic prior rather than image-level concept annotations. CLIP image–concept similarities remain the source of image-dependent supervision, while relevance scores guide interventions during concept learning and classification.

Second, we provide additional experimental evidence, including cross-LLM consistency and multi-dataset performance comparisons, shared-hyperparameter evaluation across five datasets, and extended relevance-score corruption experiments. We have also improved methodological clarity and brought selected findings from the appendix into the main experimental discussion.

Detailed explanations and supporting evidence are provided in our responses to the corresponding comments below.

W1. Clarity of the method

We thank the reviewer for the constructive suggestions regarding the clarity of our method. We agree that the preliminaries should be more accessible to readers unfamiliar with LF-CBM.

To address these concerns, we have revised Section 3.1 to first introduce the basic CBM pipeline, which maps images to concept activations and then predicts classes from these activations. We have also clarified the distinct roles of the pretrained visual encoder ff and CLIP encoders, the notation for the CLIP activation matrix A\mathbf{A} and its entries AijA_{ij}, and how the concept projector learns from CLIP-derived supervision. The revised presentation further clarifies how the two learning stages relate to our relevance-guided interventions.

In addition, we have expanded Appendix A to provide a detailed description of the concept generation and preprocessing pipeline, including the prompts, filtering procedures, and dataset-specific settings. Section 3.1 now provides a concise overview and refers readers to the appendix for reproducibility details.

W2. Consistency and performance across different LLMs

We agree that the reliability of the semantic prior is an important aspect of the proposed framework. We have therefore conducted additional analyses of both the consistency of relevance scores across LLMs and their impact on downstream classification.

(i) Consistency of semantic priors across LLMs.

We thank the reviewer for this suggestion. To examine whether the class–concept relevance prior depends strongly on the choice of LLM, we compare relevance scores generated by Gemini and Qwen for identical class–concept pairs.

DatasetClass–concept pairsPearson correlationBinary agreement (%)
BAR1,3200.86791.29
Stanford408,0000.83190.66

For binary agreement, scores of 5–10 are considered relevant and scores of 0–4 irrelevant. The analysis covers all 1,320 pairs on BAR and the same 8,000 CLIP-retrieved pairs on Stanford40, excluding unscored pairs assigned zero by pre-filtering.

Both datasets exhibit high cross-LLM consistency, with Pearson correlations above 0.83 and binary agreement rates above 90%. These results suggest that the semantic relevance judgments used by ScoreCBM are reasonably stable across the two LLMs, rather than being specific to a single model. We have included this analysis in Appendix B.6 of the revised manuscript.

(ii) Performance across different LLMs.

We agree that extending the comparison across more datasets would strengthen the evaluation of ScoreCBM's robustness to the choice of LLM. We therefore report comparisons between Gemini and Qwen on four datasets, including the existing BAR results in Appendix B.2:

DatasetGemini ACC (%)Qwen ACC (%)
BAR89.39 ± 0.4087.65 ± 0.56
CIFAR-1088.55 ± 0.0987.93 ± 0.12
CIFAR-10069.21 ± 0.1069.22 ± 0.09
Waterbirds-Species62.05 ± 0.0861.26 ± 0.05

The two LLMs achieve broadly comparable performance across these datasets, with accuracy differences ranging from 0.01 to 1.74 percentage points. Together with the cross-LLM agreement analysis in (i), these results suggest that the proposed relevance-guided framework is not highly dependent on a particular LLM, although the choice of LLM can influence performance.

For larger datasets such as Places365 and ImageNet, extensive Gemini-based scoring is constrained by its API quota. Our evaluation on these datasets primarily aims to examine scalability. By using Qwen, a locally deployable LLM, we demonstrate that ScoreCBM can be applied to large-scale classification tasks without relying on a cloud-based LLM. We have incorporated the relevance-scorer comparison into Section 4.4 of the revised manuscript.

(iii) Sensitivity of concept deduplication to CLIP models.

We thank the reviewer for this suggestion. We conducted an additional analysis on BAR and Stanford40 to investigate how different CLIP text encoders affect concept filtering and deduplication.

Experimental setup. ScoreCBM follows the concept filtering and deduplication procedure of the original LF-CBM framework. The procedure combines MPNet and CLIP cosine similarities:

$ s(c_i,c_j)=0.25s_{\mathrm{MPNet}}(c_i,c_j)+0.75s_{\mathrm{CLIP}}(c_i,c_j). $

We fixed the original candidate concept lists, MPNet encoder, filtering rules, and similarity thresholds, varying only the CLIP text encoder among ViT-B/16 (default), RN50, and ViT-L/14. The class-name similarity threshold was 0.85 for both datasets, while the concept deduplication thresholds were 0.90 for BAR and 0.85 for Stanford40. The default ViT-B/16 setting exactly reproduced the concept vocabularies used in our main experiments.

Results. The table below reports the number of concepts retained at each stage of preprocessing.

DatasetCLIP encoderInitialAfter length filteringAfter class-name filteringRemoved by deduplicationFinal retained
BARViT-B/1630823423313220
BARRN5030823423310223
BARViT-L/143082342349225
Stanford40ViT-B/162,2562,2082,2025621,640
Stanford40RN502,2562,2082,2054531,752
Stanford40ViT-L/142,2562,2082,2073051,902

Discussion. On BAR, the retained vocabulary varies only slightly across encoders (220–225 concepts). On Stanford40, the differences are more pronounced, with RN50 and ViT-L/14 retaining 112 (+6.83%) and 262 (+15.98%) more concepts than the default ViT-B/16, respectively.

These results show that the sensitivity of concept deduplication to CLIP encoder choice varies across datasets. In particular, different encoders can produce different vocabulary sizes under fixed similarity thresholds. Since this preprocessing procedure is inherited from LF-CBM rather than introduced by ScoreCBM, the analysis characterizes an existing component of the concept vocabulary construction pipeline. We have included this analysis in Appendix C.5 of the revised manuscript.

(iv) Dataset-specific vs. Shared Hyperparameters

We thank the reviewer for this suggestion. The dataset-specific settings in Table 9 (Appendix A.4) of the revised manuscript reflect differences in dataset characteristics and training configurations. To examine whether these choices are essential to the reported performance, we additionally evaluate a shared set of core hyperparameters.

We transferred the attenuation and adaptive-regularization hyperparameters selected on BAR, including the regularization strength, to five other datasets without further tuning. Dataset-dependent backbones, concept vocabularies, and CLIP filtering cutoffs were retained, while the core method-specific hyperparameters were kept identical across datasets.

DatasetDataset-specific ACC (%)Shared ACC (%)ΔACC (pp)
CIFAR-1088.55 ± 0.0987.74 ± 0.02−0.81
CIFAR-10069.21 ± 0.1068.89 ± 0.10−0.32
Stanford4089.55 ± 0.1789.37 ± 0.10−0.18
CUB-20075.42 ± 0.0575.41 ± 0.20−0.01
Waterbirds-Species62.05 ± 0.0862.04 ± 0.11−0.01

All results are reported as mean ± standard deviation. ΔACC denotes the change from the dataset-specific configuration in percentage points.

The shared configuration yields accuracy drops of at most 0.81 percentage points across all five datasets. Notably, ScoreCBM retains exactly the same ranking among CBM baselines as under dataset-specific tuning, ranking first on CIFAR-10, CIFAR-100, and Waterbirds-Species, and second on Stanford40 and CUB-200. These results suggest that ScoreCBM's competitive performance does not rely heavily on dataset-specific tuning of its core hyperparameters.

We have included this shared-configuration evaluation in Appendix B.7 of the revised manuscript.

(v) Ablation studies across datasets.

We agree that evaluating key design choices across multiple datasets would strengthen the experimental evidence. In response, we have expanded our evaluation in several directions, including cross-LLM comparisons on four datasets (W2-ii), shared-hyperparameter evaluation on five datasets (W2-iv), and A-NEC evaluation on BAR and Stanford40 (Response to Reviewer iWmT, W3).

These additional experiments provide broader evidence for the robustness of our design choices across different datasets and evaluation settings. We have incorporated these results into the revised manuscript, with selected findings included in the main text.

Also, we appreciate the reviewer's observation that several relevant experiments are already included in the appendix but insufficiently highlighted in the main text. We have added explicit references to selected supplementary analyses and included key findings in the main experimental discussion.

Q1. Semantic relevance and visual observability

We apologize for the ambiguity in our wording in Section 3.2. Our intention was to distinguish class–concept semantic relevance from image-level concept presence, rather than suggest that ScoreCBM assigns concepts to images solely based on LLM knowledge.

Specifically, LLMScore estimates the semantic relevance of a concept to a class, not whether that concept is present in a particular image. Image-dependent concept supervision is still provided by CLIP image–concept similarities. A high relevance score does not itself cause a concept to be activated in an image. Instead, the relevance scores guide how CLIP supervision is learned: high-relevance associations retain their original supervision, while low-relevance supervision signals are attenuated rather than set to zero, preserving potentially useful visual information. In addition, adaptive ℓ1 regularization discourages the downstream classifier from relying on low-relevance concept–class connections.

In this way, LLM-derived semantic relevance complements rather than replaces CLIP-based visual supervision. While CLIP provides image-dependent information, relevance guidance helps mitigate its potentially problematic associations, including those arising from superficial visual resemblance.

We have revised the wording in Section 3.2, particularly the statement regarding visual observability, to eliminate this ambiguity.

Q2. Handling unclear features in LLM scoring

Thank you for this question. The scoring prompt in Appendix A.3 instructs the LLM to assign a score of 0 when a feature is unclear. Once returned, this score is handled in the same way as other zero relevance scores, without any additional uncertainty-specific processing. Specifically, it is subject to the configured activation attenuation and adaptive ℓ1 penalty, rather than automatically removing the concept or setting its activation to zero. The LLM-generated reasoning is not used during training.

We have clarified this handling procedure in Appendix A.3.

Q3. Cross-LLM agreement and evaluation

Thank you for this suggestion. We have addressed these concerns in our responses to W2-(i) and W2-(ii), where we provide quantitative agreement analysis between Gemini and Qwen and extend their downstream performance comparison to four datasets.

We have also incorporated these analyses into the revised manuscript and highlight the relevant findings in the main experimental discussion, as discussed in W2-(v).

Q4. Direct supervision using binary LLM relevance

We thank the reviewer for this suggestion. We would like to clarify that the LLM-derived relevance scores in ScoreCBM are class–concept semantic priors, rather than image-level concept annotations. They are used to guide concept learning and classifier regularization, while CLIP image–concept similarities provide image-dependent supervision.

Directly using binary LLM relevance scores as concept supervision would assign the same concept targets to all training images of a given class, regardless of whether those concepts are visually present. This would fundamentally change the role of the semantic prior and could introduce the very issue of assigning visually absent concepts that the reviewer raised in Q1.

Our approach instead preserves image-dependent CLIP supervision while using class–concept relevance to suppress potentially problematic associations. We hope this clarifies the distinction between the suggested alternative and the proposed framework.

Reviewer_iWmTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ScoreCBM, a CBM which uses class-concept relevance score as additional supervision to build a a CBM. An LLM is queried to provide a score between each class-concept pair, and these relevance scores are used as ground-truth in concept activation attenuation to suppress low-relevance concepts, and to penalize concept-class weights in the final classifier. Experiments conducted on small-scale and large-scale datasets demonstrate show that ScoreCBM achieves SoTA and the authors provide a method to reduce the complexity to O(K) when dealing with large datasets.

Critical strengths
  • In my opinion the direction of irrelevant concepts in CBMs is very important, and actually its the only thing that CBMs are missing. I've been working on CBMs since 2023, and many CBMs i've observed just spin out completely irrelevant concepts. CBMs just learn concept as opaque features. If they are non-opaque and readable to humans, it does mean the CBM uses them in the same way as humans do, in fact it just treats them as normal features and learns from them, and many of them make zero sense to us. So i think this direction is important and the authors tackle this problem well.
  • The method outperforms U-F2-CBM which is the first work that tackles CLIP biased concepts.
  • The paper overall is easy to follow and understand
Critical weaknesses
  • I am not sure why an LLM is necessary? Can't you use just CLIP text encoder text-text similarity between class and concepts? The authors performed this to filter concepts for the LLM stage, but i dont see why it can't be used as the main concept relevance method, without an LLM. For example, U-F2-CBM uses a sentence encoder to take similarities between concepts and classes. Unless the LLM provides better relevance scores, I dont see any need for an LLM for such a simple task. Please provide ablations on this.
  • Please provide quantitive experiments on concept interventions. The authors only provide a single qualitative example. Interventions is an integral part of CBMs and should not be evaluated only qualitatively. For example, U-F2-CBM provides quantitive interventions on both Waterbirds dataset by intervening in bird-related features, and also on a multi-class setup by intervening in class-related concepts (Table 13 in U-F2-CBM paper).
  • Please report the average A-NEC matrix [1]. This has become important in CBMs.
  • What is the motivation of using Rij? Why can't you just multiply the image-concept activations with the concept-relevance matrix ?

[1] VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance

Questions for authors

In general I am happy with the paper. However, there are some concerns that should be addressed. I am therefore providing a Weak reject as the initial assessment, but will raise my score to a Weak Accept if the concerns are addressed in the rebuttal/discussion phase.

5 comments

We sincerely thank the reviewer for recognizing the importance of addressing irrelevant concept dependencies in CBMs and for the constructive suggestions. We have conducted additional experiments to address all four concerns, including (1) comparisons with CLIP-based relevance scoring, (2) quantitative relevance-score interventions, (3) A-NEC evaluation, and (4) alternative activation attenuation strategies. We present the results and clarifications below, and have incorporated the related experiments into the revised manuscript.

W1. Why use LLMs instead of CLIP text–text similarity?

We agree that CLIP text–text similarity is a reasonable alternative and can capture class–concept relevance to some extent. However, prior studies have documented biases in CLIP representations and predictions [1], as well as their potential propagation into downstream CBMs [2]. Since our objective is to mitigate such inherited associations, relying solely on CLIP to construct the relevance prior may preserve some of the biases we seek to address. This motivates exploring LLM-based scoring as an alternative source of semantic knowledge.

We further evaluate this design choice empirically. We already included a CLIP-based scorer comparison in Appendix B.2 (Table 12). On BAR, Gemini achieves 89.39 ± 0.40%, compared with 87.05 ± 0.59% using CLIP text–text similarity. To further investigate this question, we conducted additional comparisons on three datasets:

DatasetGeminiQwenCLIP text–text
CIFAR-1088.55 ± 0.0987.93 ± 0.1287.42 ± 0.03
CIFAR-10069.21 ± 0.1069.22 ± 0.0962.63 ± 2.34
Waterbirds-Species62.05 ± 0.0861.26 ± 0.0560.31 ± 0.11

All values are classification accuracies (%). The results confirm that CLIP text–text similarity can serve as a useful relevance estimator, achieving performance comparable to LLM-based scoring on CIFAR-10. However, LLM-based scoring yields higher accuracy on BAR and Waterbirds-Species, with a larger observed difference on CIFAR-100.

These findings support our motivation for introducing an alternative semantic prior: although CLIP similarity can capture useful class–concept associations, relying exclusively on CLIP does not consistently achieve the performance of LLM-guided scoring. We therefore view LLM-based scoring as a complementary source of semantic knowledge for mitigating potentially biased associations in CLIP-based CBMs, rather than claiming that CLIP similarity is inherently ineffective.

We have added an analysis of different relevance scorers in the revision (Sec.4.4).

[1] Evaluating CLIP Towards Characterization of Broader Capabilities and Downstream Implications

[2] CLIP-Free, Label Free, Unsupervised Concept Bottleneck Models

W2. Quantitative concept intervention evaluation

We thank the reviewer for highlighting the importance of quantitative concept intervention evaluation. To extend the qualitative example in Figure 5, we conducted systematic interventions on Waterbirds-Species, a 200-class bird species classification benchmark with spurious background correlations. We divided the test images into four groups based on bird type (landbird or waterbird) and background (land or water). Landbird / Land and Waterbird / Water are background-aligned groups, whereas Landbird / Water and Waterbird / Land are background-conflicting groups.

Following the intervention mechanism illustrated in Figure 5, we modify class–concept relevance scores and retrain both the concept projector and sparse classifier accordingly. Specifically, we consider two interventions: (1) Suppressing bird-discriminative concepts, by setting their relevance scores to zero across all classes; and (2) Promoting background concepts, by setting their relevance scores to ten across all classes. Bird-discriminative and background concepts were identified by Qwen based on concept descriptions, independently of the original relevance scores.

GroupOriginal (%)Suppress bird (%)ΔACC (pp)Promote BG (%)ΔACC (pp)
Landbird / Land67.69 ± 0.0765.62 ± 0.37−2.0766.83 ± 0.23−0.86
Landbird / Water64.73 ± 0.1460.61 ± 0.47−4.1262.03 ± 0.40−2.70
Waterbird / Land43.20 ± 0.7039.77 ± 0.09−3.4341.85 ± 0.24−1.35
Waterbird / Water51.71 ± 0.4150.57 ± 1.65−1.1450.88 ± 0.18−0.83

All results report 200-class species classification accuracy (mean ± standard deviation). Changes are measured relative to the original ScoreCBM in percentage points. Bold group labels indicate background-conflicting groups.

Both interventions reduce classification accuracy, with larger performance drops in background-conflicting groups than in background-aligned groups. Suppressing bird-discriminative concepts has a greater overall impact, consistent with their direct importance for species recognition. The degradation in aligned groups is also expected, since the 200-class task requires fine-grained species discrimination, for which bird-related concepts remain important even when background cues are consistent with bird type. The moderate degradation is also consistent with our soft attenuation design, in which setting a relevance score to zero reduces the corresponding CLIP supervision rather than removing it entirely.

To distinguish the effects on fine-grained species recognition from those on bird-type classification, we additionally map the 200-class predictions to binary landbird/waterbird labels. Under this evaluation, accuracy in the two background-aligned groups remains largely unchanged across the three settings (maximum difference of 0.21 percentage points), whereas the background-conflicting groups exhibit more pronounced degradation. For example, for Landbird / Water, suppressing bird-discriminative concepts and promoting background concepts reduce accuracy by 4.92 and 4.80 points, respectively. These quantitative results extend the relevance-score intervention demonstrated in Figure 5 and support the importance of class–concept relevance guidance for fine-grained recognition and robustness to spurious background correlations.

We have incorporated this analysis in Sec 4.6, and provided details in Appendix C.6 in the revision.

W3. A-NEC evaluation

Following the A-NEC evaluation protocol introduced in VLG-CBM, we evaluate the learned concept representations by fitting sparse classifiers and measuring classification accuracy at NEC = {5, 10, 15, 20, 25, 30}. We apply the same procedure to LF-CBM and ScoreCBM.

DatasetMethodNEC=5NEC=10NEC=15NEC=20NEC=25NEC=30Average
BARLF-CBM77.29 ± 1.6983.12 ± 0.7484.39 ± 0.8684.85 ± 0.9285.12 ± 0.8285.14 ± 0.7683.32 ± 0.72
BARScoreCBM80.47 ± 0.9385.37 ± 0.6187.65 ± 0.2587.66 ± 0.6187.45 ± 0.5587.26 ± 0.5985.98 ± 0.41
Stanford40LF-CBM86.16 ± 0.2287.62 ± 0.2188.35 ± 0.0788.55 ± 0.0588.76 ± 0.0388.82 ± 0.1188.04 ± 0.04
Stanford40ScoreCBM88.89 ± 0.0788.91 ± 0.0788.78 ± 0.0688.65 ± 0.0988.55 ± 0.1288.36 ± 0.1488.69 ± 0.06

All values are classification accuracies (%). ScoreCBM achieves higher average A-NEC accuracy on both datasets, improving over LF-CBM by 2.66 percentage points on BAR and 0.65 points on Stanford40. On BAR, ScoreCBM consistently outperforms LF-CBM across all six NEC levels. On Stanford40, the improvement is most pronounced under stronger sparsity constraints (NEC=5 and 10), while LF-CBM performs slightly better at NEC=25 and 30.

These results are closely related to the motivation of ScoreCBM. By attenuating unreliable concept supervision according to class–concept relevance, ScoreCBM aims to reduce misleading associations in the learned concept representations. This may allow sparse classifiers to capture useful predictive information under tighter concept sparsity constraints, consistent with the improvements observed in A-NEC evaluation.

Since the sparse classifiers are refitted using the same evaluation protocol, these results primarily assess the learned concept representations rather than isolating the effect of our adaptive L1 regularization. They therefore provide complementary evidence to our component ablations and concept-relevance analyses.

We have incorporated the A-NEC evaluation into the main text (Section 4.5), including a summary table reporting the results at NEC=5 and the average accuracy across all six NEC settings. The complete results and additional analysis for all NEC levels are provided in Appendix C.4.

Reviewer_AG9cOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ScoreCBM to reduce misleading concept-class associations in CLIP-based CBMs. Instead of removing concepts entirely, the method uses LLM-generated relevance scores to adjust concept learning and classifier weights. The authors evaluate the method on several datasets, reporting improvements over LF-CBM.

Critical strengths

S1. The paper presents an interesting idea. S2. The proposed method does show improvements over LF.

Critical weaknesses

W1. Although the paper discusses ALBM and the method proposed by Kim et al. (ECCV 2024), neither is included in the quantitative comparisons. Adding these comparisons, or explaining why a direct comparison is not possible, would help clarify the contribution of ScoreCBM.

W2. The ablation study shows that combining the two modules works well, but it is unclear whether a simpler approach could achieve similar results. For example, directly masking the connections between low-relevance concepts and classes, without changing the concept learning process, could be a useful baseline.

W3. I am not fully convinced that semantic relevance is a reliable way to identify spurious correlations. For example, water may be highly relevant to waterbirds, but relying on the water background can still lead to incorrect predictions. W4. The method reduces concept activations based on their relevance to a class, even when those concepts are actually present in the image. Although the paper shows changes in concept activations, it does not directly evaluate whether the learned concepts still accurately reflect the visual content.

W5. The method relies heavily on relevance scores generated by LLMs, but the paper does not independently verify how reliable these scores are. The results also vary depending on which LLM is used for scoring.

W6. Table 5 uses the same relevance scores for both training and evaluation, which raises some concerns about circular validation.

Minor weaknesses

  1. The standard deviation for the attenuation-only result on BAR differs between Table 4 (0.33) and Tables 8–9 (0.35).

  2. The number of concept pairs reported in Tables 6 and 14 does not match. It would help to clarify whether different concept vocabularies were used.

  3. Appendix A.2 provides the relevance scoring prompt, but the modified concept generation prompt and the JSON output schema are not fully specified.

  4. The concept editing examples in Figures 5 and 8 are interesting, but the paper does not clearly explain whether editing a relevance score requires retraining the model.

Questions for authors

Please refer to the Weaknesses section.

5 comments

We thank the reviewer for the thoughtful and constructive feedback. In response, we have strengthened the evaluation with additional comparisons against ALBM and simpler hard-masking alternatives, and clarified the differences between ScoreCBM and Kim et al. (ECCV 2024).

We have also expanded our analyses of semantic relevance, visual faithfulness, and LLM scoring reliability, while clarifying the scope of our claims. In addition, we have corrected the reported inconsistencies and improved the documentation of concept generation and model editing.

Detailed explanations and supporting evidence are provided in our responses to the corresponding comments below.

W1. Comparisons with ALBM and Kim et al. (ECCV 2024)

We thank the reviewer for highlighting these relevant works. We agree that additional comparisons and a clearer discussion of their differences from ScoreCBM would strengthen our evaluation.

Comparison with ALBM. ALBM primarily investigates attribute-based class-specific concept spaces, with an emphasis on few-shot learning and base-to-novel generalization. Although its research focus differs from ours, we agree that a quantitative comparison would be informative. We therefore adapt ALBM to use the same backbone as ScoreCBM, keeping the backbone frozen and introducing a trainable feature projection where needed to match ALBM's input dimensionality.

The available results are reported below. ALBM performs substantially below ScoreCBM and other competitive baselines in our evaluation. We have incorporated the available ALBM results into the main experimental comparison and will update the manuscript with the remaining results once the ongoing experiments are completed.

MethodCUBBARStanford40
ALBM (adapted)69.55 ± 0.4378.85 ± 3.9777.69 ± 1.67

Regarding Kim et al. (ECCV 2024), we acknowledge its relevance to mitigating spurious correlations in CBMs. However, it operates under a different research setting from ScoreCBM and the other baselines considered in our evaluation, particularly in terms of supervision requirements, computational resources, and dataset-specific human guidance.

  1. Research setting and supervision requirements. Our evaluation focuses on CBM approaches that leverage existing pretrained visual models for concept construction and prediction, without requiring extensive MLLM-generated image-level concept annotations. Kim et al., in contrast, construct annotation-based CBMs by employing MLLMs to generate concept supervision for individual training images. This represents a substantially different computational setting, involving large-scale additional supervision construction. For example, their supplementary material reports approximately 295 hours on two RTX 3090 GPUs for image-level concept annotation on Waterbirds alone (Appendix C.2, Table G). ScoreCBM follows the setting of our existing evaluation and introduces only a lightweight class–concept semantic prior, whose LLM scoring cost does not scale with the number of training images.

  2. Intervention strategy and dataset-specific human guidance. Kim et al. identify potential spurious concepts through statistical correlations and filter them from the concept vocabulary. Their pipeline incorporates dataset-specific prompt design, including different in-context demonstrations for keyword extraction and a dedicated concept-expansion procedure for Waterbirds (Supplementary Appendix A.1). The paper does not specify an automated procedure for constructing these dataset-specific configurations. Furthermore, they introduce annotation refinement based on manual inspection of specific MLLM errors, explicitly demonstrated on ImageNet-Opener, while its application to other datasets is not clearly specified. In contrast, ScoreCBM uses a unified class–concept relevance scoring rubric and two complementary interventions across datasets, without requiring dataset-specific prompt demonstrations, manual identification of spurious concepts, or error-specific correction strategies.

These differences make Kim et al. a relevant but substantially different approach from the methods evaluated in our experimental setting. To our knowledge, no official public implementation is available. A faithful reproduction would require reconstructing its multi-stage pipeline, including dataset-specific configurations and extensive MLLM-based concept annotation. Given these substantial implementation and computational requirements, we are unable to complete a reliable quantitative comparison within the rebuttal period.

We have added a dedicated discussion of Kim et al. in the revised Appendix E.1, clarifying the differences in research settings, supervision requirements, computational costs, and intervention strategies.

W2. Comparison with simpler hard-masking alternatives

We thank the reviewer for this valuable suggestion. To investigate whether directly masking low-relevance concept–class connections could achieve comparable performance, we evaluated two simpler alternatives that retain the original LF-CBM concept projection without modifying concept learning:

  1. Direct masking: We set low-relevance concept–class weights to zero at inference time, without retraining the classifier.
  2. Masking with classifier retraining: We retrained the sparse classifier on the original LF-CBM concept representations, enforcing zero weights for low-relevance concept–class connections throughout training.

For both alternatives, connections with relevance scores Sjk≤tS_{jk}\leq t were masked, with thresholds t∈{3,4,5}t\in\{3,4,5\}. The masking decisions were based on class–concept relevance scores rather than the ground-truth labels of test images.

Experimental results. We evaluated both alternatives on BAR and Stanford40, reporting classification accuracy (%) as mean ± standard deviation.

MethodThreshold ttBARStanford40
LF-CBM (matched)—85.06 ± 0.3989.14 ± 0.17
Direct masking385.17 ± 0.6272.94 ± 0.75
Direct masking484.05 ± 0.4371.58 ± 0.66
Direct masking584.14 ± 0.5070.77 ± 0.54
Masking + retraining383.65 ± 0.4887.45 ± 0.23
Masking + retraining483.41 ± 0.4387.08 ± 0.40
Masking + retraining583.27 ± 0.4386.88 ± 0.42
ScoreCBM (full)—89.39 ± 0.4089.55 ± 0.17

Discussion. The results show that neither masking strategy achieves performance comparable to the full ScoreCBM under the evaluated thresholds. These results support our use of soft, relevance-guided interventions rather than hard removal of concept–class connections. In particular, ScoreCBM incorporates relevance information into both concept learning and classifier regularization, allowing low-relevance associations to be discouraged without imposing binary restrictions on their use.

We have included these additional comparisons in Appendix B.8 of the revised manuscript.

W3. Semantic relevance and spurious correlations

We thank the reviewer for raising this important concern. We would like to clarify two points: (1) our relevance scoring emphasizes class-discriminative semantic evidence rather than mere co-occurrence, allowing it to distinguish certain misleading contextual associations; and (2) ScoreCBM employs semantic relevance as a prior to mitigate problematic associations, rather than as a definitive criterion for identifying statistical spurious correlations.

First, our relevance scoring rubric explicitly emphasizes class-discriminative semantic evidence rather than contextual co-occurrence. This distinction is reflected in the actual LLM scores from our Waterbirds-Species experiments. For example, Gemini assigns a relevance score of only 1/10 to “rippled water surface” for Pacific Loon, compared with 9/10 for “slender pointed black bill”. Similarly, for Mallard, “deep water mid-lake” receives 2/10, whereas “yellow bill” receives 9/10. These examples illustrate that, even for bird species naturally associated with water, the LLM can distinguish general environmental associations from more discriminative morphological characteristics. Thus, the scoring mechanism can help suppress misleading contextual associations rather than simply reproducing co-occurrence patterns.

Second, our method targets problematic semantic associations rather than strictly identifying statistical spurious correlations. These two notions are related but not equivalent. While ScoreCBM can mitigate spurious correlations, it also addresses other potentially problematic associations, such as those arising from superficial visual resemblance. We acknowledge that semantically relevant concepts may still become unreliable shortcuts under particular data distributions, and that LLM relevance estimates can be imperfect. To accommodate such imperfections, ScoreCBM uses relevance scores as soft guidance rather than definitive judgments, allowing them to complement the image-dependent visual supervision provided by CLIP. Specifically, relevance-guided attenuation adjusts concept supervision without discarding it, while adaptive regularization discourages low-relevance concept–class connections without strictly eliminating them. Together, these interventions help mitigate biases inherited from CLIP, as supported by our empirical results.

We have included these illustrative relevance scores and a qualitative analysis of discriminative versus contextual associations in Appendix D.2 of the revised manuscript.

W4. Visual faithfulness of learned concepts

We thank the reviewer for raising this important concern. We agree that visual concept presence and class relevance are distinct properties. We would like to clarify two points: (1) our method uses class relevance to guide concept supervision during training, rather than directly determining concept presence at inference time; and (2) we provide an additional analysis of Figure 3 to examine how attenuation affects visually present and potentially misleading concepts.

First, class relevance guides concept learning rather than directly determining concept presence. Our method uses class–concept relevance to modulate CLIP similarity targets during training, rather than directly modifying concept activations at inference time. Specifically, low-relevance targets are attenuated rather than zeroed out. The concept projector learns a shared mapping across all training images and classes, allowing it to learn each concept from diverse visual evidence. At inference time, the projector operates without access to class labels or relevance scores. Therefore, reducing supervision for a particular class–concept pair does not necessarily prevent the learned projector from recognizing that concept when it is visually present.

Second, we further examine the concept activations underlying Figure 3. For the waterfall-climbing image, we extracted the activations and ranks of representative water-related concepts, together with their original Gemini relevance scores for the climbing class.

ConceptRelevanceLF-CBM activation (rank)ScoreCBM activation (rank)
water surface0/100.0717 (42)0.0717 (9)
water ripples0/100.0512 (68)0.0512 (28)
mountain stream2/100.1287 (1)0.0603 (21)
woodland creek0/100.1239 (2)−0.0214 (81)
sandy beach surf0/100.1234 (3)−0.0062 (72)

Several observations are worth noting. First, “water surface” and “water ripples”, both assigned relevance scores of 0, retain their original activations, while their relative ranks increase from 42nd to 9th and from 68th to 28th, respectively. Second, “woodland creek” and “sandy beach surf”, which do not accurately describe the image, show substantial reductions in activation and ranking. Finally, “mountain stream”, originally the highest-ranked concept in LF-CBM, becomes less prominent but retains a positive activation.

This example suggests that reducing supervision for low-relevance concepts does not necessarily eliminate their visual responses. Moreover, the reduced prominence of inaccurate scene descriptions indicates that attenuation may, in some cases, yield concept responses more consistent with the observed visual content.

We have added a brief discussion of these findings alongside Figure 3 in the main text and provided the detailed analysis in Appendix C.3 of the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.