- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces exemplar partitioning, which divides the latent space into regions using cosine similarity to exemplars, as an alternative to standard SAEs for interpreting activations. Unlike SAEs, EP does not decompose activation vectors into interpretable components, but simply associates an activation vector with some suitable exemplar vector. EP matches or beats SAEs at concept detection (AxBench), reveals how instruction tuning affects activations, and helps with finding hidden secret words in the Taboo models of Cywinski et al.
Critical strengths
The paper tries to recover SAE functionality (interpreting activation vectors) without actually needing to perform expensive SAE training, which is interesting and timely (and a direction that is gaining some traction recently -- see a few references below that may be cited).
It is also cheap to implement and execute, and can serve as a useful baseline for interpretability methods.
The work also illustrates that competitive performance can be obtained on many tasks without actually needing a full decomposition.
[1] Sriram Balasubramanian, Soheil Feizi "Retrieval is Enough: Training-Free Interpretability with a Tool-Using Agent" https://arxiv.org/abs/2607.16448
[2] Sida Liu, Feijiang Han, Sarah Wiegreffe, Furong Huang "Non-Gaussianity Reveals Interpretable Directions in Language Models" https://arxiv.org/abs/2606.11722v2
Critical weaknesses
-
The novelty of the paper is somewhat limited as clustering contextual representations to discover latent concepts goes back at least to Dalvi et al. (ICLR 2022, https://arxiv.org/abs/2205.07237). What is new here is the claim that these old methods can be competitive on many relevant "modern" interpretability tasks (axbench, taboo models, etc).
-
But with regards to this, it seems like a very great limitation that the method is only able to return the exemplars associated with the most dominant concept. In the tasks that were considered (finding secret, harmful vs benign, concept detection in axbench), there is only one relevant concept per input text, which makes matters easier. In my personal experience, it often happens that the nearest neighbours often concentrate on some concept which is not very helpful or not what you are looking for. Taking this into consideration, I can't come up with a clear use case for when I would prefer to use this method. Even in the absence of a trained SAE, I can do a quick PCA/ICALens[2] analysis, or come up with a potentially relevant concept set and get CAVs for each of them using mean diff. HARP [1] is more expensive and is also particularly close to EP in that it uses nearest neighbour databases, but has the option of projecting out dominant concepts so that weaker concepts can be surfaced.
[1] Sriram Balasubramanian, Soheil Feizi "Retrieval is Enough: Training-Free Interpretability with a Tool-Using Agent" https://arxiv.org/abs/2607.16448
[2] Sida Liu, Feijiang Han, Sarah Wiegreffe, Furong Huang "Non-Gaussianity Reveals Interpretable Directions in Language Models" https://arxiv.org/abs/2606.11722v2
Questions for authors
If you could convince me that W2 is not a big deal, or that this method can indeed return multiple relevant concepts, I can consider increasing the score.
Response to Reviewer Fx1p
Thank you for the detailed review and the pointers to related work.
W1 – novelty. We agree that clustering contextual representations to find concepts goes back at least to Dalvi et al. (2022), and we now cite it alongside ICA Lens and HARP. Our contribution is the dictionary: regions of activation space, each with a mean for detection and intervention and an exemplar that is a real activation from a specific prompt. Any clustering algorithm can produce it (Section 2 now says this, with a k-means variant). By default we use leader clustering, because it builds in one streaming pass with a calibrated distance threshold in place of a chosen number of clusters. Because we know which prompt each exemplar came from, and we can run the same prompts through two models, we can compare dictionaries across base and tuned models of two architectures (Section 4). We can also find the Taboo secrets as regions that only appear after fine-tuning (Section 5), without naming a concept in advance.
W2 – one concept per input. The inputs in these tasks contain many concepts, but each task labels one target concept per text, as in any detection benchmark. AxBench builds each positive text by answering an unrelated instruction while working the target concept in. For example, the first test text for "references to rental services and associated equipment" answers "What ancient organization invented the concept of Democracy?" with analogies to renting equipment, and the selected region scores highest at the position of the token equipment, not a word about Athens. Taboo transcripts are open-ended hint games, and each harmful prompt has a topic as well as being harmful. EP assigns a region to the activation at each position, not to each text, so a target concept can be detected when other concepts appear in the same input.
Within a single activation, you're right: a feature only gets its own region if it's the dominant feature in at least one activation. But with leader clustering, a single activation is enough to found a region, and we build the dictionary from every position of a large corpus, so a concept only has to be dominant once. On AxBench, 470 of the 500 concepts (94%) have a region with test AUROC ≥ 0.8, and 403 (81%) have one at ≥ 0.9.
Use case. Difference-of-means and probes need the concept named and labelled first, and directions from PCA, ICA or an SAE have to be aligned before two models can be compared. EP is for open-ended, comparative questions: which regions exist after fine-tuning that didn't exist before (Taboo), or how two models organise the same inputs (Section 4). Both questions need every region to have an exemplar that is a real activation from a known prompt.