- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose a test time interpretability method. For an intermediate activation the method learns a k-means initialised autoencoder, encouraged not to move too far away from that initialisation, applies post-hoc VLM derived descriptions of each learned autoencoder feature based on maximally activating examples, and then for a particular input selects features greedily until their covariance aware latent variable reconstruction (Eq 3) gives an activation reconstruction that meets some pre-determined fidelity level. To validate this they evaluate this method on a single layer from each of three models on three matched datasets, measuring reconstruction vs avg interpretability (mixture of three common VLM interpretability scores) showing their method KMeansAE to have higher interpretability for a given level of reconstruction than other methods. They also show they achieve higher for a given level of sparsity than other methods. Lastly there are some experiments showing the validity of steering using their reconstructed activations.
Critical strengths
-
I thought, for the most part, that this paper was well written and clearly laid out, aside from a few issues with how the method is described, I found the motivation, intuition behind the model and explanatory purpose of each experiment easy to understand. Given the recent interest in matched pursuit for interpretability and the limits of SAEs in identifying meaningful features this work seems timely and like it would be of interest to the field.
-
The breadth and quality of experiments is high, with a few caveats. Generally I thought the experiments were well thought out, informative and chosen thoughtfully.
Critical weaknesses
-
I would have liked a bit more qualitative evaluation of features, or some examples of features extracted from each method here since automated feature quality metrics can sometimes be deceiving. Figure 6 was a good start but some side by side comparisons might be enlightening here.
-
How fair is your metrics panel? You exclude the monosemanticity measures despite this being the very measure your method doesn't perform well on and also that I would expect a method which depends on feature covariance to struggle with. Isn't monosemanticity of features desirable? How sure are you of the validity of the other 3 metrics more generally, do they correlate with human judgement?
Questions for authors
- How are the ranks computed in equation 8 calculated?
- In the OpenImages/CLIP RN50 panel in figure 4 your method seems to plateau while Vanilla and top-k SAE are still improving. This seems fine because you achieve higher levels of interpretability (according to the metrics you select). Why then do you push so high in figure 12 and report vanilla SAE having higher sparsity?
- Could you explain the refitting step of your method in a bit more detail? It is mentioned at several points but doesn't appear to be explained.
2 comments
Thank you for taking the time to address my concerns. I believe the changes made improve the paper and as such I will raise my score.
We thank the reviewer for the suggestions on qualitative evaluation, metric choice, and the steering protocol.
W1 — Side-by-side feature examples. Following the reviewer's suggestion, we added three comparisons between TTS and Vanilla SAE on the same CLIP ViT-B/32 inputs, with spatial responses, labels, and top-activating images (Appendix H.4, Figures 16–18). We also ran a grounding experiment to test whether the positive concepts shown in an explanation describe objects or scenes that are present in the input image. At comparable reconstruction fidelity, Grounding DINO gives TTS the highest mean score across all three model–dataset pairs and both description methods (Section 4.2, Table 1, and Appendix H.3. See H7qq W2).
W2 — Monosemanticity and human agreement. We agree with the reviewer that monosemanticity belongs in the main comparison. We originally reported MS separately because it gave very similar scores to standard linear extractors such as PCA, NMF, and the identity baseline. The revised average includes MS and adds SemanticLens monosemanticity. KMeansAE still scores higher on the five-metric average than the evaluated Vanilla and TopK SAEs at high reconstruction fidelity, and some SAE configurations score higher on MS (Section 4.1, Figure 3, and Appendices G.3–G.4).
Clarity measures coherence among top-activating images, INVERT measures agreement with annotated classes, and CLIP-Dissect measures alignment with text labels. Each captures a different aspect of feature quality, and their scales differ. Prior studies compare MS and SemanticLens polysemanticity with human judgments in their own settings. How well the original three metrics and the five-metric average agree with human judgments on our dictionaries remains an open question. MS study · SemanticLens study
Q1 — Ranks in Eq. (8). We sort class scores from highest to lowest and record the target's position, starting at one. The scores are classifier logits for ResNet-18 and DenseNet-161, and image–text cosine similarities for CLIP RN50. Ties follow the sort order. Cases that start at rank one are excluded from the rank-gap metric, and none occurred in the reported sweep. Section 4.3 and Appendix I.1 give the candidate classes, CLIP prompt, and sorting details.
Q2 — The CLIP plateau and high-fidelity comparisons. TTS has a clear advantage when the goal is to show as few positive concepts as possible at comparable reconstruction fidelity. Figure 4 shows this advantage on CLIP RN50 in the regime with few positive activations. At higher fidelity, some SAE configurations continue to improve as the TTS curve levels off. We show that range to make clear where the advantage narrows. We read the reference to Figure 12 as referring to the layerwise analysis, now Figure 13 in Appendix G.9. That analysis uses CLIP ViT-B/32 and compares reconstruction and activation counts across layers. The KMeansAE counts there are for the dense encoder before test-time selection. Vanilla SAE is sparser at that stage. Figure 4 addresses the separate question of how many positive concepts TTS displays after selection.
Q3 — Refitting. We thank the reviewer for pointing out that our wording suggested a separate refitting step. We revised the description to clarify the reconstruction procedure. TTS holds the original selected activations fixed, including zeros, and updates the covariance reconstruction in Eq. (3) as it adds coordinates. Figure 2 displays two positive concepts and uses all 617 selected observations for reconstruction. We clarified this in Section 3, Figure 2, and the discussion of Algorithm 1.