acceptodds
Under review as a conference paper at ICLR 2027

ViTLens: A Lightweight Interpretability Tool for Vision Transformers

Abstract

Do Vision Transformers (ViTs) represent human-interpretable semantic concepts, and if so, where do they emerge across modules and how do they influence downstream predictions? We introduce ViTLens, an efficient, non-invasive framework for arbitrary-location concept probing and attribution from forward activations using only per concept. We then estimate each concept’s influence on the prediction using our adapted SVARM-LOO and attribute that influence across ViT's modules with MC-Shapley. ViTLens achieves competitive concept identification on CIFAR-100, CUB-200-2011, and ImageNet-1k while fitting up to 1490 faster than concept bottleneck model variants per probing site, and achieves a Pearson correlation of up to 0.95 with exact Shapley value at substantially lower cost. We further demonstrate ViTLens's capabilities in selective concept suppression with minimal collateral damage and reveal that old semantic concepts remain recoverable despite severe catastrophic forgetting.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic interpretability, Subspaces, Vision Transformers

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_K327OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

model predictions. The method uses SVD to extract low-dimensional concept subspaces from a small number of positive examples, followed by a hierarchical Shapley-based approach to attribute prediction changes to concepts and individual modules. Experiments on CIFAR-100, CUB-200-2011, and ImageNet-1k show competitive concept identification performance with substantially lower fitting costs than concept bottleneck methods.

Critical strengths

S1. ViTLens achieves competitive concept identification performance with low computational cost and without modifying the original model.

S2. 实验比较全面,概念抑制和灾难性遗忘两个应用也为理解 ViT 的内部行为提供了有价值的发现。

Critical weaknesses

W1. The method extracts concept subspaces using SVD with only 10 positive examples. These examples may share backgrounds or class-related features. Could the authors further verify that the extracted subspaces capture the target concepts rather than other correlated features?

W2. The hierarchical Shapley approximation is validated only on the last 3 layers with a small number of concepts, leaving its reliability across the full ViT insufficiently established.

W3. The paper emphasizes computational efficiency, but ViTLens does not improve fitting speed over Linear Probe on some datasets. Its efficiency advantage is mainly demonstrated against CBM-based methods.

Questions for authors

Q1. Since the Z-score normalization depends on the concept set, would adding new concepts affect the detection scores of existing concepts? Would the thresholds need to be recalibrated?

Q2. Table F.2 shows that smaller subspace ranks sometimes perform better. Why is rank 8 used for all concepts and datasets? Could an adaptive rank selection improve performance?

4 comments
AuthorsOpenReview ↗

Response to Reviewer K327

We thank the reviewer for the thoughtful feedback, and for recognizing that ViTLens achieves competitive concept identification with low computational cost and without modifying the original model. For S2, "实验比较全面,概念抑制和灾难性遗忘两个应用也为理解 ViT 的内部行为提供了有价值的发现。", we translated it with an AI tool, and it means "The experimental comparison is comprehensive, and the two applications of concept suppression and catastrophic forgetting also provide valuable findings for understanding the internal behavior of ViTs." We are grateful for this positive assessment, and we will address each concern of the reviewer below.


W1: The method extracts concept subspaces using SVD with only 10 positive examples. These examples may share backgrounds or class-related features. Could the authors further verify that the extracted subspaces capture the target concepts rather than other correlated features?

For background-related features, we apply the subspaces fitted on the [CLS] token to the [PATCH] tokens as shown in Fig. B.1. If a subspace mainly captured the shared background, the energy would concentrate on the background patches. Instead, it concentrates on the object, e.g. "towering height" on the mountain peak and "tower" on the water tower. However, there are failure cases where the energy focuses on the wrong object, e.g. "crucifix" on the altarpiece which we show in Fig. B.2.

For class-related features, we further fit the subspace of concept cc only on positives of cc from other classes (Decorrelated), and score it on the test images of a class kk it has never seen. As a control, Class-only fits the subspace on 10 images of class kk with the concept labels ignored, which gives a rank-8 subspace spanned by the class features. The rank (r=8r=8), the number of fitting images (10) and the scoring are kept identical. We report the Lift metric [1, 2], the fraction of positives among the top-nc,kn_{c,k} scored images of class kk divided by the base rate, so chance is exactly 1 for every pair. We filter CIFAR-100 for concept–class pairs (c,k)(c,k) where class kk has at least 10 positives and 10 negatives of cc among its test images, and cc is shared by at least 2 such classes. The full details of this analysis are given in a separate reply below.

Lift (mean ± s.e.)PairsViTLensDecorrelatedVLG-CBM [3]Class-only
CIFAR-100491.210 ± 0.0571.358 ± 0.1031.279 ± 0.0610.963 ± 0.042

By fitting on other classes only, Decorrelated is +0.395 above Class-only and on par with VLG-CBM trained on the full labelled training set, while Class-only stays at chance. Thus, the subspace captures the concept shared across classes rather than the features of the class it is fitted on.


W2: The hierarchical Shapley approximation is validated only on the last 3 layers with a small number of concepts, leaving its reliability across the full ViT insufficiently established.

Validating the full hierarchy across all 12 layers is infeasible, as the exact Shapley value of the joint game over 5 concepts and 12 layers requires 260≈1.15×10182^{60}\approx1.15\times10^{18} coalition evaluations per image (Appendix D.3). The last 3 layers with 5 concepts already cost 215=32,7682^{15}=32{,}768 coalition evaluations per image, and each additional layer multiplies this by 25=322^5=32, so we restrict the test to this setting to keep exact enumeration tractable over the 400 test images. We choose the last 3 layers as they directly produce the output, and a removal at each of them propagates through the following ones.


W3: The paper emphasizes computational efficiency, but ViTLens does not improve fitting speed over Linear Probe on some datasets. Its efficiency advantage is mainly demonstrated against CBM-based methods.

We agree that ViTLens and Linear Probe have a similar fitting time (Table F.4), but at the same cost ViTLens outperforms Linear Probe by 20.01 macro-F1 over the FFN, Attn and Res sites and by 8.87 at Last Res (Table 1).

Moreover, Linear Probe is a single classifier shared by all concepts and must be retrained to add a new one, whereas ViTLens fits each concept independently and adds it without modifying the existing subspaces.

Finally, the cost of Linear Probe grows with the number of optimization steps, and it becomes hard to train with a large concept set such as the 4,167 concepts of ImageNet-1k, whereas ViTLens fits each concept with one SVD of its 10 exemplars at a fixed cost.

AuthorsOpenReview ↗

Response to Reviewer K327

Q1: Since the Z-score normalization depends on the concept set, would adding new concepts affect the detection scores of existing concepts? Would the thresholds need to be recalibrated?

We thank the reviewer for raising this. In short, adding new concepts leaves the detection of existing concepts essentially unchanged, and the threshold does not need to be recalibrated.

This is because each subspace Pc(l)P^{(l)}_c is fitted only on the positive exemplars of cc, so adding concepts never changes an existing subspace or its energy Ec(l)(x(l))E^{(l)}_c(\mathbf{x}^{(l)}) in Eq. (5). The concept set only enters through the per-image mean μE(l)\mu^{(l)}_E and standard deviation σE(l)\sigma^{(l)}_E, so adding concepts maps every existing score to E^′=aE^+b\hat{E}' = a\hat{E} + b with a=σE(l)/σE′(l)>0a=\sigma^{(l)}_E/\sigma'^{(l)}_E>0. Thus, at any single layer, the ranking of concepts within an image cannot change, and only the position of the scores relative to τ\tau can move.

To measure this, we start from a random 50% of the concepts on ViT-B/16 and add 25%, 50% and 100% of the remaining ones. τ\tau is selected once on the validation split with the initial set and then frozen, and we score only the evaluated concepts of the initial set (about 300, 350 and 2,080 of the 608, 698 and 4,167 in Table 1) at all four Table 1 sites.

Max |ΔF1| from re-selecting τ\tauΔF1 at frozen τ\tauPredictions changed
CUB-200-20110.30+0.05 to +0.47≤ 0.09%
CIFAR-1000.08−0.06 to +0.46≤ 0.10%
ImageNet-1k0.18−0.03 to +0.29≤ 0.01%

At the frozen τ\tau, macro-F1 changes by less than 0.5 for all 12 dataset × site combinations, and re-selecting τ\tau gains at most 0.30. Moreover, the ranking of concepts within each image is exactly preserved at Last Res, and almost exactly at Res (median per-image Spearman correlation above 0.999). Thus, no recalibration is needed.

Q2: Table F.2 shows that smaller subspace ranks sometimes perform better. Why is rank 8 used for all concepts and datasets? Could an adaptive rank selection improve performance?

We agree that a smaller rank sometimes performs better in Table F.2. However, the gain is marginal: selecting the best rank among {1,2,4,8}\{1,2,4,8\} for each site and dataset on the test set improves macro-F1 over r=8r=8 by only 0.20 on average, and r=8r=8 is already the best rank in 7 of the 12 cases. Thus, a fixed r=8r=8 is sufficient.

Nevertheless, adaptive rank selection is an interesting direction. Motivated by the hierarchical organization of concept subspaces in Appendix I.2, each concept could be modelled by a flag of nested subspaces, i.e. a point on the flag manifold [4], which comes at no extra cost as the singular vectors in Eq. (2) are already nested [5]. Weighting the energy of each nested block in Eq. (5) would then permit concept- and layer-dependent effective ranks, with the current formulation as a special case. We leave this as future work.

References

[1] Piatetsky-Shapiro, Gregory, and Sam Steingold. "Measuring lift quality in database marketing." ACM SIGKDD Explorations Newsletter 2(2), 2000.

[2] Postels, Janis, et al. "On the practicality of deterministic epistemic uncertainty." ICML 2022.

[3] Srivastava, Divyansh, Ge Yan, and Tsui-Wei Weng. "VLG-CBM: Training concept bottleneck models with vision-language guidance." NeurIPS 2024.

[4] Ye, Ke, Ken Sze-Wai Wong, and Lek-Heng Lim. "Optimization on flag manifolds." Mathematical Programming 194, 2022.

[5] Pennec, Xavier. "Barycentric subspace analysis on manifolds." Annals of Statistics 46(6A), 2018.

AuthorsOpenReview ↗

Details of the shared concept analysis (W1)

Below we give the full setup of the shared concept analysis on CIFAR-100 reported in W1.

Pair selection. Each CIFAR-100 class has 100 test images. We keep a concept–class pair (c,k)(c,k) when (i) class kk has at least 10 positives and at least 10 negatives of cc among its test images, (ii) cc satisfies (i) in at least 2 classes, (iii) the median detection confidence of the pseudo-labeler [3] on the pair is at least 0.20, and (iv) cc has positives in the training split (at least 2 images per class) in at least 2 classes other than kk, so that the subspace of cc can be fitted on other classes and tested on a class it has never seen. Condition (iv) reduces the 61 shared pairs to 49 (16 concepts, 36 classes), with 12 to 89 positives per pair. The subspaces are fitted on the training split, and the pairs are scored on the test split.

Metric. For each pair with test images Ic,k\mathcal{I}_{c,k} of class kk, labels yi∈0,1y_i\in\\{0,1\\}, nc,k=∑iyin_{c,k}=\sum_i y_i positives and base rate bc,k=nc,k/∣Ic,k∣b_{c,k}=n_{c,k}/|\mathcal{I}_{c,k}|, we report the Lift metric [1, 2]

Lc,k=fraction of positives among the top-nc,kbase rate=1bc,k⋅1nc,k∑i∈Top-nc,kyi,L_{c,k} = \frac{\text{fraction of positives among the top-}n_{c,k}}{\text{base rate}} = \frac{1}{b_{c,k}} \cdot \frac{1}{n_{c,k}}\sum_{i\in \text{Top-}n_{c,k}} y_i,

where Top-nc,kn_{c,k} denotes the nc,kn_{c,k} images of class kk with the highest Ec(x)E_c(\mathbf{x}). We use the Lift metric as it requires no threshold, and chance is exactly L=1L=1 for every pair regardless of its base rate. The mean and s.e. are taken over the 49 pairs.

Settings. All settings use ViT-B/16. The rank (r=8r=8), the number of fitting images (10) and the scoring are kept identical, and only the images used to fit the subspace are changed:

  • ViTLens: 10 positives of cc, as in Table 1.
  • ViTLens-Decorrelated: 10 positives of cc from other classes only, so class kk is never seen.
  • Class-only: 10 images of class kk with the concept labels ignored, which gives a rank-8 subspace spanned by the class.
  • VLG-CBM [3]: the supervised concept layer trained on the full labelled training set, with its scores z-scored across concepts as for ViTLens, scored on the same pairs.

[1] Piatetsky-Shapiro, Gregory, and Sam Steingold. "Measuring lift quality in database marketing." ACM SIGKDD Explorations Newsletter 2(2), 2000.

[2] Postels, Janis, et al. "On the practicality of deterministic epistemic uncertainty." ICML 2022.

[3] Srivastava, Divyansh, Ge Yan, and Tsui-Wei Weng. "VLG-CBM: Training concept bottleneck models with vision-language guidance." NeurIPS 2024.

AuthorsOpenReview ↗

Follow-up on Rebuttal

Dear Reviewer K327,

We would like to kindly follow up on our rebuttal and the additional experiments. We would appreciate any further feedback or remaining concerns we could address before the discussion closes.

Thank you again for your time and consideration.

Reviewer_aeLAOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ViTLens, a tool for finding human-understandable concepts (like "red head" or "striped coat") inside a trained Vision Transformer, and for measuring how much each concept affects the model's prediction. For each concept, the authors take 10 example images, collect the model's activations at different layers, and use SVD to get a small subspace that represents the concept. A concept is "detected" in a new image when a large share of the activation falls inside that subspace. To measure how important each concept is, they remove its subspace from the activations and see how much the output changes, then use Shapley-value approximations: first per concept (with a new estimator they call SVARM-LOO), then per layer (with Monte Carlo Shapley). Experiments on CIFAR-100, CUB-200-2011 and ImageNet-1k show concept detection close to concept bottleneck models while being much faster to fit, and the attribution estimates correlate well with exact Shapley values on small cases. The paper ends with two case studies: removing a class's concepts at test time with little damage to other classes, and studying catastrophic forgetting, where they argue that old concepts are still present but lose their influence on the output.

Critical strengths
  • Simple, cheap and non-invasive. The method only needs forward passes and an SVD. It does not change or retrain the model, so the model's accuracy is untouched. It is also much faster to fit than concept bottleneck models (up to 1490x per layer).
  • Works at any layer. ViTLens can probe FFN, attention and residual activations at every layer. This makes it useful for asking where concepts appear inside the network.
  • Careful Shapley estimation. The two-level attribution (concepts first, then layers) is a sensible way to make Shapley values affordable. The proposed SVARM-LOO estimator reuses leave-one-out results instead of throwing them away, and the authors benchmark it fairly against several other estimators at equal compute budgets.
  • Appendix is thorough with proofs, ablations, hyperparameters, visual outputs and failure cases, which makes the work easier to check and reproduce.
Critical weaknesses
  • It is not clear that the method finds concepts rather than classes. Each concept subspace is built from about 10 images, and in datasets like CUB these images mostly come from a few classes that share the concept. With rank 8 and only 10 images, the subspace is close to just the span of those images. So a high detection score may mean "this looks like one of those classes" rather than "this image has a red head". There is no control experiment to rule this out (maybe one can be done with "stripes" and "zebra"), and the ground-truth labels are themselves automatic (GroundingDINO) and noisy, even though CUB has human-labeled attributes that could have been used.
  • The detection results are less strong than the text says. ViTLens is the best method in only one of six settings in Table 1. It is clearly behind VLG-CBM on ImageNet, the largest dataset, and sometimes behind a simple linear probe. The claimed 7.46-point lead over kNN depends heavily on two kNN results that look broken (about 2% F1). Also, "10 images per concept" adds up to about two thirds of the CUB training set. And, I could not make sense of this statement: "ViTLens remains within 0.96 macro-F1 points of VLGCBM".
  • The attributions are shown to match Shapley values, but not shown to be meaningful. The paper checks that its estimator is close to exact Shapley values on small cases. It never checks whether the attributions themselves are faithful, for example with sanity checks, deletion tests, or comparisons with existing concept-importance methods. The headline 0.95 correlation is for each level separately; the full hierarchical scheme reaches about 0.86 to 0.88, tested only on the last 3 layers. The theory relies on strong assumptions that do not hold in the multi-layer setting.
  • The case studies do not fully support their conclusions. For suppression, the key design choice (the Rayleigh-quotient method versus simply projecting out the concept) is never compared, and the method is weak on ImageNet-100. Also, only class accuracy is measured, not whether the concept itself is removed. For forgetting, the claim that the relative geometry of concepts is preserved is never measured directly. The ViT-T versus ViT-B comparison mixes model size with different pretraining data, and the overlap measure is not compared to a random baseline.
Questions for authors
  1. How does ViTLens compare to a simple baseline that predicts the model's class and then outputs that class's concept list? If the scores are close, the method may be detecting classes rather than concepts.
  2. Can you report concept detection on CUB's human-annotated attributes, and on images where a concept appears outside the classes it was fitted on?
  3. How do the attribution scores perform on standard faithfulness checks (for example, deletion curves or randomization tests)? Matching Shapley values alone does not show the attributions are meaningful.
  4. Are the Shapley correlations in Fig. 2 computed per image or pooled across all images? Can you report per-image rank agreement?
  5. What happens to the kNN runs with about 2% F1, and how large is ViTLens's advantage when they are fixed or removed?
  6. How does the Rayleigh-quotient suppression compare to directly projecting out the concept subspaces? And after suppression, are the target concepts actually no longer detected?
12 comments
AuthorsOpenReview ↗

Response to Reviewer aeLA

We thank the reviewer for the constructive review, and for recognizing that ViTLens is cheap, non-invasive and works at any layer, with careful Shapley estimation. We are grateful for this positive assessment, and we will address each concern of the reviewer below.


W1, Q1 & Q2: It is not clear that the method finds concepts rather than classes. … The ground-truth labels are themselves automatic (GroundingDINO) and noisy, even though CUB has human-labeled attributes that could have been used. How does ViTLens compare to a simple baseline that predicts the model's class and then outputs that class's concept list? Can you report concept detection on CUB's human-annotated attributes, and on images where a concept appears outside the classes it was fitted on?

To address the reviewer's concern about concept-class bias, we have run the requested baseline and evaluated ViTLens when the class identity cannot tell whether the concept is present.

First, we compare the concepts of the model's predicted class against ViTLens and VLG-CBM on all concepts and on the concepts shared by at least 2 classes on CIFAR-100.

Macro-F1All concepts (%, ↑)Shared concepts (%, ↑)
Concepts of predicted class57.2553.90
VLG-CBM [1] (Last Res)54.6552.33
ViTLens (Last Res)57.0954.23

On all concepts, the concepts of the predicted class perform the best, as 77.2% of the CIFAR-100 concepts belong to a single class. This is not a limitation unique to ViTLens, as existing CBM-based works also face it [1, 2, 3]. However, on the shared concepts, ViTLens outperforms both the concepts of the predicted class and VLG-CBM, and thus the concept-class bias does not affect ViTLens significantly. Nevertheless, concepts and classes are inherently related, as humans may use concepts to tell different species apart, and CBMs are explicitly trained to predict the concepts and then use them to predict the class.

Furthermore, we evaluate ViTLens within a class, on concepts shared by multiple classes, where the class identity cannot tell whether the concept is present, and the results indicate that ViTLens detects the concept without relying on the class features.

We filter CIFAR-100 for concept–class pairs (c,k)(c,k) where class kk has at least 10 positives and 10 negatives of cc among its test images, and cc is shared by at least 2 such classes. With ViT-B/16, we rank the test images of class kk by the concept score Ec(x)E_c(\mathbf{x}). We then report the Lift metric [4, 5], which is the fraction of positives among the top-nc,kn_{c,k} images divided by the base rate of cc in class kk. It needs no threshold and is exactly 1 at chance. Only the images used to fit the subspace are changed:

  • ViTLens-Decorrelated: 10 positives of cc from other classes only, so class kk is never seen.
  • Class-only: 10 images of class kk with the concept labels ignored, which gives a rank-8 subspace spanned by the class, as hypothesized by the reviewer.

We also score VLG-CBM [1], trained on the full labelled training set, on the same pairs.

MethodLift (↑, mean ± s.e.)
Class-only0.963 ± 0.042
VLG-CBM [1]1.279 ± 0.061
ViTLens-Decorrelated1.358 ± 0.103
ViTLens1.210 ± 0.057

Class-only is at chance, so a subspace spanned by the class carries no concept signal within the class, while ViTLens retrieves 21% more positives than chance, close to VLG-CBM trained on the full training set (28%). ViTLens-Decorrelated, which never sees class kk, still retrieves 36% more positives than chance. Thus, ViTLens detects the concept rather than the class, showing that the ViT encodes the concepts as a by-product of learning to predict the class. The full details of this analysis are given in a separate reply below.

This also answers the second part of Q2, as ViTLens-Decorrelated is only scored on classes outside the ones it was fitted on.

As pointed out by Koh et al. [2], CUB's per-image attributes are too noisy to serve as ground truth, as each image is labelled by a single non-expert crowdworker and similar attributes are often confused (e.g., red vs. rufous belly). Thus, CBM [2] aggregates them by majority vote within each species and keeps only the 112 attributes present in at least 10 classes, so all images of the same class share the same attributes. These 112 concepts are therefore class-level by construction, and evaluating on them would measure class identity rather than concepts.

AuthorsOpenReview ↗

Response to Reviewer aeLA

W2a: The detection results are less strong than the text says. ViTLens is the best method in only one of six settings in Table 1. It is clearly behind VLG-CBM on ImageNet, the largest dataset, and sometimes behind a simple linear probe. … And, I could not make sense of this statement: "ViTLens remains within 0.96 macro-F1 points of VLG-CBM".

The goal of ViTLens is not to outperform CBMs, which are trained on the entire training set, but to show that a native ViT, without backpropagation or modification, already encodes concepts that can be read out at a similar level. On ImageNet-1k, ViTLens fits on only 3.2% of the training set (Table F.3), while VLG-CBM is trained with backpropagation on all of it. Among the methods that can probe every site, ViTLens is the best in 16 of the 24 settings of Table 1 (with the corrected kNN in W2b & Q5), and the linear probe is ahead of ViTLens only at Last Res of ViT-S/16 on CIFAR-100 and CUB-200-2011, by 0.64 and 1.31.

By "ViTLens remains within 0.96 macro-F1 points of VLG-CBM", we mean that, averaged over the six Last Res settings, ViTLens reaches 58.06 macro-F1 and VLG-CBM 59.02, so ViTLens is 0.96 lower. We will state this explicitly in the revision.


W2b & Q5: What happens to the kNN runs with about 2% F1, and how large is ViTLens's advantage when they are fixed or removed?

Upon a finer sweep of τ\tau, we found that kNN scores on the ViT-S/16 residual stream are strongly anisotropic, so kNN only operates in a very narrow window of τ\tau despite z-score normalization, and the τ\tau selected on the validation split falls just outside this window. We corrected the kNN values in the table below.

Concept macro-F1, kNN (ViT-S/16)Reported (%, ↑)Corrected (%, ↑)
CIFAR-100, Res1.80 ± 0.0134.09 ± 0.16
CIFAR-100, Last Res1.72 ± 0.0056.12 ± 0.15
CUB-200-2011, Res2.83 ± 0.0261.66 ± 0.23

Averaged over the FFN, Attn and Res sites, ViTLens remains the best method (47.80) when the kNN entries are fixed, 2.40 points above kNN (45.41), and 3.93 points above kNN when the two broken entries are removed instead (49.02 vs. 45.10). We apologize for this error and will update Table 1 and lines 212–213 accordingly.


W2c: "10 images per concept" adds up to about two thirds of the CUB training set.

CUB-200-2011 is small (5,994 training images) but has 608 concepts, so 10 images per concept covers 66.0% of the training set after deduplication (Table F.3). Even so, ViTLens is 381× and 756× faster to fit than VLG-CBM and CBM. On larger datasets, ViTLens uses only 13.3% (CIFAR-100) and 3.2% (ImageNet-1k) of the training set, and still outperforms VLG-CBM on CIFAR-100 (57.09 vs. 54.65 on B/16) at 1490× faster fitting.

AuthorsOpenReview ↗

Response to Reviewer aeLA

W3 & Q3: The attributions are shown to match Shapley values, but not shown to be meaningful. … for example with sanity checks, deletion tests, or comparisons with existing concept-importance methods.

We respectfully disagree, as these checks are proxies for how the model output changes when features are removed, which ViTLens measures exactly. Informally, if we remove the concepts one at a time in a random order and record how much the output changes at the step where a given concept is removed, the Shapley value of that concept is this change averaged over all possible orders (formal definition in Def. C.1). As it averages over every order, it accounts for every interaction between the concepts, and its scores add up exactly to the total output change and are equal for concepts with the same effect (stripes and horse shape below) [6], which makes it the gold standard for attribution.

The deletion curve [7] rewards the ranking that makes the output drop fastest, which can penalize a correct attribution when concepts interact. Take the reviewer's zebra example, where the model needs both "stripes" and "horse shape": removing either one or both changes the output by 1, and removing "grass" changes it by 0.2. Shapley splits their joint effect equally and gives stripes 0.5, horse shape 0.5 and grass 0.2, ranking stripes and horse shape above grass (Lemma C.3). The deletion curve instead prefers stripes, grass, then horse shape, as once stripes is removed, removing horse shape changes nothing. Thus, the deletion curve rewards ranking grass above horse shape for a zebra. Such metrics are also known to rank attribution methods inconsistently [8].

The randomization test [9] checks that an explanation changes when the model weights are randomized. On a Kaiming-initialized ViT-B/16, ViTLens no longer detects the concepts (3.63 macro-F1 at Last Res on CIFAR-100, our response to Reviewer aKXU, W1).

Existing well-known attribution methods such as EAP [10] and TCAV [11] use the first-order effect of a single component or concept. EAP further approximates the effect of removing a single component, similar to our LOO in Eq. (8), with a first-order Taylor expansion. Thus, both miss every effect that only appears when several components are removed together, which are the Harsanyi interactions of order ≥2\ge 2 (Corollary 3.1).

Thus, as ViTLens attributes the output change itself (Eq. (7)), the exact Shapley value needs no proxy check, and what remains to be validated is how close our estimate is to it, which Fig. 2 (each level) and Table D.1 (the composed hierarchy) measure.


W3 & Q4: Are the Shapley correlations in Fig. 2 computed per image or pooled across all images? Can you report per-image rank agreement?

Fig. 2 is computed per image, not pooled across images. For each image and probing site (FFN, Attn and Res), we compute the Pearson and Spearman correlations between the estimated and exact Shapley values of the concepts (concept level) or of the 12 layers (module level) of that image, and report the mean ± std over 400 images × 3 probing sites. Thus, the Spearman correlation in Fig. 2 is exactly the per-image rank agreement requested by the reviewer, which averages 0.85 over the concept and module levels on CUB-200-2011 and CIFAR-100.


W3: The full hierarchical scheme reaches about 0.86 to 0.88, tested only on the last 3 layers. The theory relies on strong assumptions that do not hold in the multi-layer setting.

The full hierarchical scheme in Table D.1 is tested only on the last 3 layers, as the exact Shapley value of the joint game over 5 concepts and 12 layers requires 260≈1.15×10182^{60}\approx1.15\times10^{18} coalition evaluations per image (Appendix D.3). The last 3 layers with 5 concepts already cost 215=32,7682^{15}=32{,}768 coalition evaluations per image, and each additional layer multiplies this by 25=322^5=32, so we restrict the test to this setting to keep exact enumeration tractable over the 400 test images. The last 3 layers are a reasonable test, as they directly produce the output, and a removal at each of them propagates through the following ones, so the test already includes cross-layer interactions.

We respectfully clarify that Assumption C.12 does not exclude multi-layer interactions. It only excludes interactions across different concepts, while arbitrary interactions among the layers of the same concept are permitted (Appendix C.4). These cross-layer interactions, induced by downstream propagation, are exactly what the module-level game captures with MC-Shapley (marginal-cancellation ratio of 1,193–1,794 vs. 1.56–1.59 at the concept level, Appendix C.2). Moreover, the stronger Assumptions C.5, C.8 and C.9 are used only in Prop. C.10, which explains when LOO is accurate, and are not required by ViTLens itself, as the hierarchical allocation in Eqs. (25) and (27) satisfies efficiency without any assumption (Appendix C.1).

Reviewer_aKXUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ViTLens, a post-hoc tool for probing and attributing concepts in ViTs. Specifically, each concept is a rank-8 subspace from an SVD of [CLS] activations of 10 positive exemplars at any FFN, Attn, or Res site, and presence is scored by z-normalized projection energy. Concept influence is estimated with SVARM-LOO and split across modules with MC-Shapley. On CIFAR-100, CUB-200-2011, and ImageNet-1k with ViT-S/16 and ViT-B/16, the paper reports concept macro-F1 within 0.96 points of VLG-CBM, up to 0.95 Pearson correlation with exact Shapley, and case studies on concept suppression and catastrophic forgetting.

Critical strengths
  1. Fitting is cheap and non-invasive: one SVD per concept and site, no backpropagation, and the backbone is left untouched. That is what makes it practical to probe every FFN, Attn, and Res site (4,167 concepts on ImageNet-1k), which CBMs cannot offer.
  2. The attribution design is principled. Corollary 3.1 gives the exact LOO-Shapley gap, and Fig. 2 compares six estimators at matched budgets, which justifies SVARM-LOO at the concept level and MC-Shapley at the module level.
Critical weaknesses
  1. The most important opportunity, I think, is to rule out that the subspaces mainly detect class identity. As I understand it, the VLG-CBM labels come from grounding each class's LLM-generated concept list. The probe also reads only [CLS], and lines 223 to 224 note that even Kaiming-initialized ViTs show non-trivial concept geometry. Three checks would settle this: a baseline that outputs the concept list of the model's predicted class, ViTLens F1 on a weight-randomized ViT [1], and evaluation on CUB's 312 human-annotated attributes.

  2. The probing-site advantage in Table 1 is smaller than stated. Roughly half of the 7.46-point margin over kNN (lines 212 to 213) comes from two degenerate kNN Res cells (1.80 on CIFAR-100 S/16, 2.83 on CUB S/16). Dropping those two settings for both methods, I get 49.0 vs 45.1. At Last Res, a linear probe beats ViTLens on CIFAR-100 and CUB with S/16. On ImageNet-1k B/16, ViTLens (44.54) trails VLG-CBM (52.41), KMeans (46.81), and CAV (45.76). It would help to explain the erratic baselines (e.g., the linear probe's 11.55 on ImageNet-1k B/16), state how τ is chosen for each method, and add a post-hoc CBM [2].

[1] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. "Sanity Checks for Saliency Maps." NeurIPS, 2018.
[2] Mert Yuksekgonul, Maggie Wang, and James Zou. "Post-hoc Concept Bottleneck Models." ICLR, 2023.

Questions for authors
  1. With Nc = 10 and r = 8, the subspace nearly spans the exemplars themselves, which resembles kNN. Does F1 keep rising with more exemplars at fixed r, and is X_c centered before the SVD?
  2. When a coalition removes overlapping subspaces at one module, is Eq. (6) applied as a sum of projections or as one projection onto their joint span? Relatedly, should line 289 say "interactions of order ≥ 2", as Eq. (9) implies, rather than "pairwise"?
  3. On ImageNet-100 (Table 2), Shapley-1 leaves SA at 34.12, above plain Diff-in-Mean at Last (27.11). How does the Rayleigh-quotient suppression compare with closed-form linear erasure such as LEACE [3]?

I'm willing to raise my score if the checks in Weakness 1 hold up and the Table 1 comparison is clarified.

[3] Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. "LEACE: Perfect Linear Concept Erasure in Closed Form." NeurIPS, 2023.

7 comments
AuthorsOpenReview ↗

Response to Reviewer aKXU

We thank the reviewer for the insightful review, and for acknowledging that ViTLens is cheap, non-invasive and practical to probe at every site, with a principled attribution design. We are grateful for this positive assessment, and we will address each concern of the reviewer below.


W1: The most important opportunity, I think, is to rule out that the subspaces mainly detect class identity. Three checks would settle this: a baseline that outputs the concept list of the model's predicted class, ViTLens F1 on a weight-randomized ViT, and evaluation on CUB's 312 human-annotated attributes.

To address the reviewer's concern about concept-class bias, we have run the requested checks and evaluated ViTLens when the class identity cannot tell whether the concept is present.

First, we compare the concepts of the model's predicted class against ViTLens and VLG-CBM on all concepts and on the concepts shared by at least 2 classes on CIFAR-100.

Macro-F1All concepts (%, ↑)Shared concepts (%, ↑)
Concepts of predicted class57.2553.90
VLG-CBM [1] (Last Res)54.6552.33
ViTLens (Last Res)57.0954.23

On all concepts, the concepts of the predicted class perform the best, as 77.2% of the CIFAR-100 concepts belong to a single class. This is not a limitation unique to ViTLens, as existing CBM-based works also face it [1, 2, 3]. However, on the shared concepts, ViTLens outperforms both the concepts of the predicted class and VLG-CBM, and thus the concept-class bias does not affect ViTLens significantly. Nevertheless, concepts and classes are inherently related, as humans may use concepts to tell different species apart, and CBMs are explicitly trained to predict the concepts and then use them to predict the class.

Furthermore, we evaluate ViTLens within a class, on concepts shared by multiple classes, where the class identity cannot tell whether the concept is present, and the results indicate that ViTLens detects the concept without relying on the class features.

We filter CIFAR-100 for concept–class pairs (c,k)(c,k) where class kk has at least 10 positives and 10 negatives of cc among its test images, and cc is shared by at least 2 such classes. With ViT-B/16, we rank the test images of class kk by the concept score Ec(x)E_c(\mathbf{x}). We then report the Lift metric [4, 5], which is the fraction of positives among the top-nc,kn_{c,k} images divided by the base rate of cc in class kk. It needs no threshold and is exactly 1 at chance. Only the images used to fit the subspace are changed:

  • ViTLens-Decorrelated: 10 positives of cc from other classes only, so class kk is never seen.
  • Class-only: 10 images of class kk with the concept labels ignored, which gives a rank-8 subspace spanned by the class, as hypothesized by the reviewer.

We also score VLG-CBM [1], trained on the full labelled training set, on the same pairs.

MethodLift (↑, mean ± s.e.)
Class-only0.963 ± 0.042
VLG-CBM [1]1.279 ± 0.061
ViTLens-Decorrelated1.358 ± 0.103
ViTLens1.210 ± 0.057

Class-only is at chance, so a subspace spanned by the class carries no concept signal within the class, while ViTLens retrieves 21% more positives than chance, close to VLG-CBM trained on the full training set (28%). ViTLens-Decorrelated, which never sees class kk, still retrieves 36% more positives than chance. Thus, ViTLens detects the concept rather than the class, showing that the ViT encodes the concepts as a by-product of learning to predict the class. The full details of this analysis are given in a separate reply below.

Regarding the weight-randomized ViT, ViTLens drops from 57.09 to 3.63 macro-F1 at Last Res on CIFAR-100 on a Kaiming-initialized ViT-B/16, which classifies at chance. Thus, ViTLens scores come from the learned features of the ViT, not from the subspace fitting or the label setup. We hypothesize that the non-trivial geometry in lines 223–224 is due to random projections in high-dimensional space mapping visually similar images into nearby subspaces, which carry no concept information as shown above and in Figs. I.4–I.6, where the subspace tree of the Kaiming-initialized ViT has no semantically meaningful structure.

As pointed out by Koh et al. [2], CUB's per-image attributes are too noisy to serve as ground truth, as each image is labelled by a single non-expert crowdworker and similar attributes are often confused (e.g., red vs. rufous belly). Thus, CBM [2] aggregates them by majority vote within each species and keeps only the 112 attributes present in at least 10 classes, so all images of the same class share the same attributes. These 112 concepts are therefore class-level by construction, and evaluating on them would measure class identity rather than concepts.

AuthorsOpenReview ↗

Response to Reviewer aKXU

W2a: The probing-site advantage in Table 1 is smaller than stated. It would help to explain the erratic baselines (e.g., the linear probe's 11.55 on ImageNet-1k B/16) and state how τ is chosen for each method.

We thank the reviewer for the careful check; the two kNN cells are indeed degenerate. All methods in Table 1 follow the same threshold protocol detailed in Appendix H. After z-score normalization, τ\tau is swept over the grid in Table H.1 on a validation split drawn from the training set, and the best value is selected for each method at each probing site, with the selected values listed in Table H.5. However, upon a finer sweep of τ\tau, we found that kNN scores on the ViT-S/16 residual stream are strongly anisotropic, so kNN only operates in a very narrow window of τ\tau, and the τ\tau selected on the validation split falls just outside this window. We corrected the kNN values in the table below.

Concept macro-F1, kNN (ViT-S/16)Reported (%, ↑)Corrected (%, ↑)
CIFAR-100, Res1.80 ± 0.0134.09 ± 0.16
CIFAR-100, Last Res1.72 ± 0.0056.12 ± 0.15
CUB-200-2011, Res2.83 ± 0.0261.66 ± 0.23

Averaged over the FFN, Attn and Res sites, ViTLens remains the best method (47.80), 2.40 points above kNN (45.41) with the corrected entries. We apologize for this error and will update Table 1 and lines 212–213 accordingly.

Unlike kNN, the low scores of the linear probe and CBM do not come from τ\tau, as re-sweeping τ\tau on a finer grid (221 values, step 0.05) improves the linear probe by only 0.41 macro-F1 on average and CBM on CUB-200-2011 by 0.46. For the linear probe, this is because a single classifier trained jointly over all concepts becomes harder to train as the number of concepts grows, which is why it performs worst on ImageNet-1k (4,167 concepts). For CBM, this is because its joint class and concept objective trades concept accuracy for class accuracy, which is a known property of CBM [2].


W2b: … and add a post-hoc CBM.

As suggested by the reviewer, we compare ViTLens with PCBM [6] on ViT-B/16. We use the CAV concept bank, where for each concept a linear SVM (C=0.1C=0.1) is fitted on the [CLS] embedding of 100 positive and 100 negative training images, using the same ground-truth concept labels [1].

Concept macro-F1 (%, ↑), ViT-B/16, Last ResBudgetCIFAR-100CUB-200-2011ImageNet-1k
VLG-CBM [1]Full training set, backprop54.65 ± 0.0569.76 ± 0.0852.41 ± 0.03
PCBM [6]100 pos. + 100 neg. per concept54.84 ± 0.0570.97 ± 0.2250.31 ± 0.02
ViTLens10 pos. per concept57.09 ± 0.1469.96 ± 0.0444.54 ± 0.05

By using only 10 positive images per concept, ViTLens outperforms PCBM on CIFAR-100 (+2.25) and is only 1.01 below it on CUB-200-2011, while PCBM fits on 12–20× more distinct images per concept, including negative samples. On ImageNet-1k, ViTLens trails PCBM at the last layer (44.54 vs. 50.31), but reaches 48.20 when averaged over all 12 layers (Res in Table 1).


AuthorsOpenReview ↗

Response to Reviewer aKXU

Q1: With Nc = 10 and r = 8, the subspace nearly spans the exemplars themselves, which resembles kNN. Does F1 keep rising with more exemplars at fixed r, and is X_c centered before the SVD?

At r=8r=8, performance largely plateaus at 10 images per concept, as doubling to 20 images improves macro-F1 by at most 3.09. We also do not center XcX_c before the SVD, as in the high-dimensional feature space, the mean of the positive exemplars carries a significant amount of concept information, which is captured by the rank-1 subspace. Both results are reported in Appendix F, where Table F.1 varies the number of images per concept at r=8r=8 and Table F.2 varies the rank.


Q2: When a coalition removes overlapping subspaces at one module, is Eq. (6) applied as a sum of projections or as one projection onto their joint span? Relatedly, should line 289 say "interactions of order ≥ 2", as Eq. (9) implies, rather than "pairwise"?

Eq. (6) is applied as a sum of projections, h(l)∗←h(l)−W(l)∑c∈SlPc(l)x(l)h^{(l)*} \leftarrow h^{(l)} - W^{(l)}\sum_{c\in S_l} P^{(l)}_c x^{(l)}, where SlS_l is the set of concepts in SS intervened at module ll.

We thank the reviewer for pointing out line 289, which should indeed read "interactions of order ≥2\ge 2", and we will correct it in the revision.


Q3: How does the Rayleigh-quotient suppression compare with closed-form linear erasure such as LEACE?

Following the reviewer's suggestion, we compare the Rayleigh-quotient suppression with LEACE [7]. For a fair comparison, LEACE erases the same concept set Csup\mathcal{C}_{\text{sup}} of the target class as ViTLens, and it is fitted on the same images as ViTLens, namely the union of the ≤10 positive images per concept, with all other images in this pool as negatives. As LEACE computes an affine projection of the activation that removes all linearly decodable information about the concept labels, we fit the projection per layer on the clean [CLS] activations of the Res output, and apply it to every image at all 12 layers (All) or at the last layer only (Last).

CIFAR-100│CUB-200│ImageNet-100
MethodSA (%, ↓)NOA (%, ↑)OA (%, ↑)│SA (%, ↓)NOA (%, ↑)OA (%, ↑)│SA (%, ↓)NOA (%, ↑)OA (%, ↑)
Base88.488.788.2│83.983.383.9│94.294.494.0
LEACE [7] (Last)16.788.887.9│13.783.382.9│11.294.493.8
LEACE [7] (All)8.988.787.5│9.983.282.1│33.294.593.9
ViTLens (Shapley-1)4.888.788.3│6.483.383.7│34.194.493.9
ViTLens (All)0.788.888.1│5.983.382.5│24.394.493.6

By using only positive images, ViTLens achieves a lower SA than LEACE on CIFAR-100 and CUB-200-2011 (0.7 vs. 8.9 and 5.9 vs. 9.9 with All) with a higher OA, while NOA stays within 0.2 of the base model for all methods.


References

[1] Srivastava, Divyansh, Ge Yan, and Tsui-Wei Weng. "VLG-CBM: Training concept bottleneck models with vision-language guidance." NeurIPS 2024.

[2] Koh, Pang Wei, et al. "Concept bottleneck models." ICML 2020.

[3] Margeloiu, Andrei, et al. "Do concept bottleneck models learn as intended?" ICLR 2021 Workshop on Responsible AI.

[4] Piatetsky-Shapiro, Gregory, and Sam Steingold. "Measuring lift quality in database marketing." ACM SIGKDD Explorations Newsletter 2(2), 2000.

[5] Postels, Janis, et al. "On the practicality of deterministic epistemic uncertainty." ICML 2022.

[6] Yuksekgonul, Mert, Maggie Wang, and James Zou. "Post-hoc Concept Bottleneck Models." ICLR 2023.

[7] Belrose, Nora, et al. "LEACE: Perfect Linear Concept Erasure in Closed Form." NeurIPS 2023.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of the discussion

We sincerely thank the PC, SAC and AC for facilitating the discussion and the reviewers for their time. Their feedback has improved the credibility of our work.

We really appreciate that the reviewers acknowledged the following strengths of ViTLens:

  1. Cheap and non-invasive. ViTLens only requires 10 exemplars per concept, is backpropagation-free and does not modify the ViT at all (aKXU, aeLA, K327).
  2. Concept detection is competitive with CBMs. ViTLens is competitive with CBMs while using far less data and being significantly faster to fit (aeLA, K327).
  3. Flexibility. ViTLens can probe every FFN, Attn and Res site, which CBMs cannot (aKXU, aeLA).
  4. Hierarchical Shapley attribution is principled and makes Shapley values affordable (aKXU, aeLA).

Throughout the discussion phase, we have addressed the following main concerns of the reviewers:

  1. Class leakage (aKXU, aeLA). In a new experiment where the training and test classes do not overlap, so the class identity cannot leak into the concept score, ViTLens outperforms the class-lookup baseline on every dataset and is competitive with CBMs, which are trained on all images. Reviewer aeLA confirmed that ViTLens transfers to unseen classes, which rules out class memorization. Moreover, while a class-lookup performs well in Table 1, ViTLens still reveals key information about the activation space of the ViT. In Sec. 4.2, after catastrophic forgetting the model can no longer classify the old classes, so a class-lookup would return the concepts of the wrong class, whereas ViTLens still detects the old concepts and reveals that they are not forgotten but lose their influence on the output.
  2. Table 1 (aKXU, aeLA). We corrected three degenerate kNN entries, and ViTLens remains the best method.
  3. Faithfulness of the attribution (aeLA). We clarified that the Shapley value is the gold standard for attribution, so the proxies suggested by the reviewer are not required.
  4. Additional experiments (aKXU, aeLA). As requested by the reviewers, we added PCBM for concept detection, and LEACE and direct projection for concept suppression, where an independent linear probe confirms that the suppressed concepts are no longer detected.
  5. Threshold recalibration (K327). We showed that adding new concepts leaves the existing ones essentially unchanged, so no recalibration is needed.

Our work shows that a ViT naturally encodes concepts, which ViTLens identifies from only 10 images per concept without backpropagation. Through hierarchical Shapley values over concepts and modules, ViTLens reveals not only what concepts are present but how much and where they contribute to the prediction. Finally, we show the applications of ViTLens to concept suppression and catastrophic forgetting.

We thank the PC, SAC, AC and the reviewers once again for their incredible efforts.

Best regards,
The Authors

AuthorsOpenReview ↗

Summary of the manuscript revision

We thank the reviewers for the feedback, and we have uploaded a revised manuscript based on the discussion, with all changes marked in blue. The main updates are listed below.

  1. Class leakage (aKXU, aeLA). Table 1 now includes the class-lookup baseline suggested by the reviewer. In Sec. 3.1, we added a paragraph on class leakage, with details in Appendix J.
  2. Corrected kNN entries (aKXU, aeLA). We corrected the three degenerate kNN entries in Table 1.
  3. Comparison with VLG-CBM (aeLA). In Sec. 3.1, we now state that the 0.96 macro-F1 gap to VLG-CBM is averaged over the six Last Res settings (58.06 vs. 59.02).
  4. LEACE (aKXU). Table 2 now includes LEACE, and Sec. 4.1 discusses the comparison.
  5. Wording of Eq. (9) (aKXU). In Sec. 3.2, the LOO-Shapley gap is now described as determined by the interactions of order ≥ 2, rather than the pairwise interactions.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.