- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper aims to reduce the global number of concepts in concept bottleneck models (CBMs), which output final predictions by using human-interpretable concepts as a bottleneck. Existing CBMs sparsely predict concepts for input images from a large concept pool. However, as the global number of concepts increases, human interpretability is compromised, which can hinder intervention and debuggability of concepts. This paper proposes a teacher-student-based recipe for constructing task-specific CBMs called AnyBottle. The goal of AnyBottle is to identify a subset of concepts required for a given task from an SAE with a large concept pool and achieve performance comparable to that of a black-box model. To this end, the paper introduces an algorithm based on orthogonal matching pursuit (OMP) that iteratively identifies important concepts. This algorithm focuses on samples where the predictions of the black-box teacher and the student currently being trained diverge, and extracts tokens that the teacher emphasizes but the current student fails to capture as "blind spots." From the set of concepts present in these blind spots, those with high activation scores are selected as a candidate set. The student is trained using nested dropout based on the selected set of concepts, and during inference, concepts are added sequentially to minimize the concept budget. In the experiments, AnyBottle is evaluated against existing CBMs on image and text datasets. As a result, AnyBottle achieves accuracy close to that of black-box models while maintaining a lower number of global concepts than existing CBMs. Additionally, analytical experiments were conducted to verify the effectiveness of each component.
Critical strengths
- The paper proposes an approach to concept selection based on teacher-student residuals.
- The paper proposes an inference method that dynamically adjusts the number of concepts for each sample using a student classifier trained with nested dropout.
- The paper confirms that these techniques allow us to maintain accuracy even with a small number of global concepts.
Critical weaknesses
- Although this paper proposes complex techniques for constructing CBMs with a small number of global concepts based on the blackbox teacher and SAE, the rationale for adopting these techniques is weak. In particular, if the goal is to achieve a small number of global concepts, reducing the number of intermediate dimensions in the SAE is the simplest baseline. Furthermore, constraining the overall budget using group lasso during task-specific fine-tuning of the SAE is also a promising option. The paper should provide evidence demonstrating why these baselines cannot achieve the goal, both theoretically and experimentally.
- Naming by MAIA relies heavily on external models, and its semantic interpretation does not reflect the internal mechanisms of the model. While MAIA extracts image regions from concept activation heatmaps and assigns names to the discovered concepts, the meanings of these named concepts have no causal relationship with the actual model outputs. The interpretations provided in this way are misleading, making it difficult to apply MAIA in critical fields such as healthcare, where CBMs are expected to be utilized.
- The relationship between compactness and interpretability has not been sufficiently verified. In particular, the paper lacks human intervention and concept debugging evaluations, making it impossible to determine whether the paper's objectives were achieved. The paper's primary motivation is the reduced interpretability for humans caused by a large number of global concepts. In this sense, it is essential to verify whether AnyBottle actually improved human interpretability.
- The proposed concept selection algorithm may overfit to a small number of samples. Although the paper introduces a mechanism to prevent overfitting by selecting concepts based on the validation score, preserving validation accuracy is unrelated to the validity of the selected concepts. In the algorithm described in the paper, residuals are calculated using weights obtained via least squares with respect to the teacher's predicted distribution, rather than the discrete ground-truth labels (Eq. (1)). This carries the risk of incorporating concepts that are inherently unnecessary for reproducing the distribution, as well as the risk of overfitting to the validation set due to hundreds of optimization iterations.
- There is a lack of evaluation regarding training time and computational resources. Since the concept selection process involves selecting one concept per round, this can result in a very large number of iterations in some cases. In particular, ISIC-7 requires as many as 447 concepts, raising concerns about the computational time required for optimization.
- The comparison conditions with competing methods are not standardized, making the comparison unfair. While the proposed method uses CLIP-DINOiser, other baselines utilize different backbones. The accuracy gap relative to the black-box teacher is not the result of a fully controlled experiment and is significantly influenced by differences in backbone performance. Furthermore, it would be desirable to conduct experiments with standardized concept dictionaries and concept budgets for SAE-based CBMs.
Questions for authors
Please see the weakness section and address the concerns.
6 comments
Individual Response 2
W3. Human interpretability and concept debugging
We agree that compactness alone does not establish improved human "interpretability". What we tackle in this work is to reduce the effort required to inspect, name, and verify a concept bottleneck, while retaining predictive performance and coherent concepts. AnyBottle addresses this through both a compact global vocabulary and sample-dependent concept budgets. Our motivation for compactness is supported by prior human studies [2] supporting the practical importance of reducing explanation size. We will clarify this distinction in the framing of the paper (e.g. introduction).
Notably, our evaluation goes beyond concept count. We jointly measure predictive performance, vocabulary size, and concept consistency through C² and AutoInterpScore. Figure 7 compares selection against the same SAE dictionary and shows comparable or higher patch-level consistency with substantially fewer concepts on most datasets. Thus, the result is not simply a smaller intermediate representation.
We have also added a direct concept-debugging experiment (App. D.15; see general response “Shortcut Debugging”). After introducing a border shortcut in Imagenette, AnyBottle selects the border-sensitive concept first in all three seeds (and its most likely meaning can be identified through MAIA). Removing it and refitting only the linear head reduces the student’s susceptibility to the shortcut from 9.5% to 0.6%, while retaining performance on samples without the shortcut. The teacher and SAE remain unchanged.
These results establish compactness, concept coherence, and a concrete debugging capability. We agree that interpretability claims beyond this aspect of compactness and consistency such as downstream user ability, require comprehensive studies. We have updated our framing on this point to be aligned with our intentions. Thank you for raising that point.
[2] Ramaswamy et al. “Overlooked Factors in Concept-Based Explanations: Dataset Choice, Concept Learnability, and Human Capability.” CVPR, 2023.
W4. Selection targets and overfitting
The focus examples guide candidate proposal, but candidates are scored against teacher residuals across the training dataset. Proposal also aggregates evidence across distinct examples and filters insufficiently supported candidates. Selection therefore does not rely only on the small focus set. Tables 1 and 3 evaluate the benefits of this guidance against simpler selection variants.
Equation (1) intentionally targets teacher outputs: a feature can help reproduce confidence differences even when unnecessary for matching discrete class decisions. Using one-hot labels would change that objective. Residual reduction is not the sole stopping criterion, however, and the final predictor is trained against ground-truth labels. We do not claim that every selected concept is indispensable.
Validation accuracy controls predictive model selection and stopping, not semantic validity. We assess concept coherence separately through C², AutoInterp, and qualitative inspection.
Repeated validation use can fit noise. Our generalization results therefore come from a separate test split, and the stopping-tolerance analysis examines sensitivity, particularly on ISIC. We clarified in the Discussion that validation-based stopping is not a guarantee against overfitting or a globally minimal vocabulary.
Individual Response 3
W5. Computational costs
We added a stage-by-stage runtime analysis on a single RTX 4500 Ada GPU (App. D.14; see “Computational Cost” in our general response).
Briefly, selection takes 17 seconds for 28 rounds on Imagenette and about 20 minutes for approximately 450 concepts on ISIC. The full Imagenette construction takes 21 minutes versus 2.1 minutes for DN-CBM, excluding naming and external pretraining; the additional cost mainly comes from patch-level features and SAE fine-tuning. These costs occur during construction, not inference. Importantly, the smaller vocabulary reduces estimated MAIA naming time from 269 to 6.4 GPU-hours under the same naming procedure.
W6. Controlled comparisons across methods
Please see “Same Dictionary Runs” in our general response and App. D.9. We compare the SAE-based DN-CBM and AnyBottle under the exact same settings.
Briefly, both methods use DN-CBM’s frozen backbone and released SAE, the same teacher and data splits, and no SAE fine-tuning. AnyBottle uses 2.5–20× fewer concepts, maintaings comparable accuracy and improves image-level consistency. The only exception is ISIC-7, which is most out of distribution from the pretrained SAEs.
Importantly, our original evaluations standardize datasets, splits, validation-based model selection, and seeds. Teacher-relative accuracy measures how much performance each bottleneck retains within its own representation, although it does not eliminate all representation effects. The new experiment additionally controls the backbone and dictionary, showing that the observed benefits do not depend on CLIP-DINOiser or our task-adapted SAE.
Individual Response 1
W1. Rationale for concept selection
We implemented both suggested alternatives. Please see “Smaller SAEs,” “Group-lasso SAEs,” and “Representation Fine-tuning and Concept Selection” in our general response for the protocols and results.
On Imagenette, smaller SAEs retain accuracy but yield substantially lower concept consistency: at comparable budgets, a 32-latent SAE achieves C²patch of 0.28–0.29, versus 0.46 for AnyBottle’s 30 selected concepts.
Group-lasso fine-tuning gives dataset-dependent trade-offs: it is more compact on Machines-15 and more accurate on Dogs-10, whereas AnyBottle has higher consistency on both and retains more accuracy with fewer concepts on Flowers-102 and ISIC.
These results explain why learning a compact representation and selecting from a richer pool are not interchangeable. Fine-tuning determines the available features, can change their decomposition and can be difficult to finetune satisfactorily due to multi-objective finetuning. Selection identifies a task-relevant subset while keeping those features fixed.
Notably, AnyBottle already combines task-aware fine-tuning with selection, with the former evaluated in Table 4. We do not claim an optimal concept pool in our work: selection applies to pools chosen for their coverage or granularity. The contribution is therefore a selection procedure with demonstrated benefits, not a claim that representation learning cannot sometimes aid or replace it.
W2. Concept naming and semantic faithfulness
We agree that semantic descriptions require verification, particularly in healthcare. However, the statement that the named features have “no causal relationship with the actual model outputs” overlooks their explicit role in prediction.
Specifically, only naming is post hoc; the bottleneck is part of the prediction pathway. AnyBottle first selects features and trains the predictor on their activations. For the linear head, each feature contributes its activation multiplied by its class weight to the corresponding logit. Naming subsequently describes these features; it does not create their relationship with the output. This follows the classic procedure of bottom-up approaches (see [1] for the distinction).
Next, MAIA tests interpretations rather than merely labeling heatmaps. It generates or edits inputs and measures feature responses to distinguish candidate interpretations. Our additional CLIP-based evaluation also finds higher mean image–description alignment for MAIA than decoder-based naming on all six datasets (App. D.13). We have updated our manuscript to make this more clear and refer to the original work for further details (Sec. 3.4).
Overall, the concepts in AnyBottle have a clear and observable connection to the output, and while AnyBottle requires proper concept naming in the same way as all bottom-up concept approaches, its compact concept space allows hypothesis-driven naming approaches such as MAIA which go beyond standard CLIP-based naming procedures often used in CBMs nowadays. Nevertheless, interactive human verification remains important (as recommended in Sec. 3.4 and discussed in Sec. 6).
[1] Knab, Patrick, et al. "What’s in the bottle? a survey and roadmap of concept bottleneck models." Transactions on Machine Learning Research (2026).