acceptodds
Under review as a conference paper at ICLR 2027

Exemplar Partitioning for Mechanistic Interpretability

Abstract

We introduce Exemplar Partitioning (EP), an unsupervised method for building interpretable feature dictionaries from large language model activations. An EP dictionary is a Voronoi partition of activation space, built by leader-clustering streamed activations within a distance threshold. Each region is defined by an observed exemplar and an average of its member activations, which define region membership and provide directions for intervention. Dictionary size is determined by the activation stream at the chosen threshold rather than pre-selected. Exemplars link regions to observed inputs, allowing dictionaries built from the same input stream to be compared across layers, training checkpoints, and architectures. We demonstrate how EP can be used to interpret and intervene on model behaviour, track changes in activation space through training, and detect hidden concepts. Comparing EP dictionaries on base and instruction-tuned Gemma-2-2B and Llama-3.1-8B reveals that instruction tuning reorganises harmful prompt activations similarly across the two models, but at different granularities. Interventions on these regions make both models answer harmful requests they previously refused. In 19 of 21 Taboo models trained to hide a secret word, EP finds new regions that do not exist in the base model, whose decoded tokens relate to the known secret on inspection. Although EP assigns each token to a single region, linear probes built from EP regions achieve up to 90.5% of full-activation probe accuracy. On AxBench concept detection at Gemma-2-2B-it layer 20, EP achieves the highest mean AUROC of all unsupervised methods (), outperforming SAE-A () and approaching supervised probes (). Building EP dictionaries is fast and cheap: EP uses times fewer construction tokens than comparable SAEs.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic interpretability, unsupervised feature discovery, representation analysis, large language models, clustering, concept detection, cross-model comparison

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Fx1pOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces exemplar partitioning, which divides the latent space into regions using cosine similarity to exemplars, as an alternative to standard SAEs for interpreting activations. Unlike SAEs, EP does not decompose activation vectors into interpretable components, but simply associates an activation vector with some suitable exemplar vector. EP matches or beats SAEs at concept detection (AxBench), reveals how instruction tuning affects activations, and helps with finding hidden secret words in the Taboo models of Cywinski et al.

Critical strengths

The paper tries to recover SAE functionality (interpreting activation vectors) without actually needing to perform expensive SAE training, which is interesting and timely (and a direction that is gaining some traction recently -- see a few references below that may be cited).

It is also cheap to implement and execute, and can serve as a useful baseline for interpretability methods.

The work also illustrates that competitive performance can be obtained on many tasks without actually needing a full decomposition.

[1] Sriram Balasubramanian, Soheil Feizi "Retrieval is Enough: Training-Free Interpretability with a Tool-Using Agent" https://arxiv.org/abs/2607.16448

[2] Sida Liu, Feijiang Han, Sarah Wiegreffe, Furong Huang "Non-Gaussianity Reveals Interpretable Directions in Language Models" https://arxiv.org/abs/2606.11722v2

Critical weaknesses
  1. The novelty of the paper is somewhat limited as clustering contextual representations to discover latent concepts goes back at least to Dalvi et al. (ICLR 2022, https://arxiv.org/abs/2205.07237). What is new here is the claim that these old methods can be competitive on many relevant "modern" interpretability tasks (axbench, taboo models, etc).

  2. But with regards to this, it seems like a very great limitation that the method is only able to return the exemplars associated with the most dominant concept. In the tasks that were considered (finding secret, harmful vs benign, concept detection in axbench), there is only one relevant concept per input text, which makes matters easier. In my personal experience, it often happens that the nearest neighbours often concentrate on some concept which is not very helpful or not what you are looking for. Taking this into consideration, I can't come up with a clear use case for when I would prefer to use this method. Even in the absence of a trained SAE, I can do a quick PCA/ICALens[2] analysis, or come up with a potentially relevant concept set and get CAVs for each of them using mean diff. HARP [1] is more expensive and is also particularly close to EP in that it uses nearest neighbour databases, but has the option of projecting out dominant concepts so that weaker concepts can be surfaced.

[1] Sriram Balasubramanian, Soheil Feizi "Retrieval is Enough: Training-Free Interpretability with a Tool-Using Agent" https://arxiv.org/abs/2607.16448

[2] Sida Liu, Feijiang Han, Sarah Wiegreffe, Furong Huang "Non-Gaussianity Reveals Interpretable Directions in Language Models" https://arxiv.org/abs/2606.11722v2

Questions for authors

If you could convince me that W2 is not a big deal, or that this method can indeed return multiple relevant concepts, I can consider increasing the score.

1 comment
AuthorsOpenReview ↗

Response to Reviewer Fx1p

Thank you for the detailed review and the pointers to related work.

W1 – novelty. We agree that clustering contextual representations to find concepts goes back at least to Dalvi et al. (2022), and we now cite it alongside ICA Lens and HARP. Our contribution is the dictionary: regions of activation space, each with a mean for detection and intervention and an exemplar that is a real activation from a specific prompt. Any clustering algorithm can produce it (Section 2 now says this, with a k-means variant). By default we use leader clustering, because it builds in one streaming pass with a calibrated distance threshold in place of a chosen number of clusters. Because we know which prompt each exemplar came from, and we can run the same prompts through two models, we can compare dictionaries across base and tuned models of two architectures (Section 4). We can also find the Taboo secrets as regions that only appear after fine-tuning (Section 5), without naming a concept in advance.

W2 – one concept per input. The inputs in these tasks contain many concepts, but each task labels one target concept per text, as in any detection benchmark. AxBench builds each positive text by answering an unrelated instruction while working the target concept in. For example, the first test text for "references to rental services and associated equipment" answers "What ancient organization invented the concept of Democracy?" with analogies to renting equipment, and the selected region scores highest at the position of the token equipment, not a word about Athens. Taboo transcripts are open-ended hint games, and each harmful prompt has a topic as well as being harmful. EP assigns a region to the activation at each position, not to each text, so a target concept can be detected when other concepts appear in the same input.

Within a single activation, you're right: a feature only gets its own region if it's the dominant feature in at least one activation. But with leader clustering, a single activation is enough to found a region, and we build the dictionary from every position of a large corpus, so a concept only has to be dominant once. On AxBench, 470 of the 500 concepts (94%) have a region with test AUROC ≥ 0.8, and 403 (81%) have one at ≥ 0.9.

Use case. Difference-of-means and probes need the concept named and labelled first, and directions from PCA, ICA or an SAE have to be aligned before two models can be compared. EP is for open-ended, comparative questions: which regions exist after fine-tuning that didn't exist before (Taboo), or how two models organise the same inputs (Section 4). Both questions need every region to have an exemplar that is a real activation from a known prompt.

Reviewer_6QwqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EP is an unsupervised method for building interpretable dictionaries from LLM activations. It leader-clusters centred, normalised activations on the unit sphere: each activation joins the nearest exemplar within a percentile-calibrated threshold or becomes a new exemplar, yielding a Voronoi partition anchored by real activations with inspectable prompts.

Critical strengths

(1) No reconstruction objective, no gradient descent, and ~10³× fewer construction tokens than comparable SAEs; dictionary size K emerges from the stream rather than being pre-selected.

(2) Demonstrations: AxBench concept detection (0.937, beating SAE-A 0.911, near supervised probes 0.946); harmful/benign organisation before and after instruction tuning in Gemma and Llama, with a span swap at L14 overcoming refusal (0.00→0.88/0.91); Taboo secret-word detection (19/21 models); SAEBench sparse probing (90.5% of full-activation probe accuracy).

Critical weaknesses

(1) The paper involves a considerable number of hyperparameters, especially the thresholds p and K, which require careful tuning across different tasks and settings. Moreover, the exemplar appears to be rather unstable (Table 1).

(2) The best readout is not the exemplar. The core claim of the method is that "each region is anchored to a real activation and a real prompt". On AxBench, the region mean achieves 0.937 while the exemplar only reaches 0.832 (Table 1, p.6), with a 10-point gap. Meanwhile, the authors themselves acknowledge in §2 (p.4) that the mean "is not a real activation... may also lie outside the model's activation distribution". In other words, the representative vector with the strongest performance directly violates the interpretability principle of the method, as the mean has no retrievable corresponding prompt.

(3) I am concerned that the authors’ writing exhibits a defensive tone.

Questions for authors

The swap's peak at L14 (0.88/0.91) does not track separation: the best-separated layers (Gemma L18, Llama L17) yield only 0.22/0.16, and the authors concede "the degree of separation does not predict the effect of the intervention" (§4.1). Since only four layers per model were sampled, it is unclear whether L14 is special or merely sampled. I would appreciate further discussion of why the intervention is most effective at L14, ideally supplemented by a broader layer sweep.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 6Qwq

Thank you for the careful review.

Hyperparameters. We think there may be a misunderstanding here. EP has exactly one hyperparameter. For the leader-clustering build used throughout the paper, it's the percentile p, which sets the distance threshold, and K is the number of regions the build produces at that threshold, so it's never chosen or tuned (Section 2 now says this explicitly). For the k-means variant, the one hyperparameter is K. Everything else – the centring and the stopping rule – is fixed and the same in every experiment. We also didn't tune p per task. Every small-corpus experiment uses p = 12 (refusal at every layer of four checkpoints, and all 21 Taboo models), the Taboo result holds at p = 4 and p = 8 (Appendix F), and on AxBench every p we tested beats the canonical SAE row with either representative.

Stability. You're right that the exemplar itself depends on streaming order, since it's the first activation to arrive. We can tell from a single build which regions this affects. A region's member count and coherence predict how well it reproduces across streaming seeds (Spearman ρ ≈ 0.68 across a 16× range of dictionary sizes; matched cosine 0.80–0.83 for the top fifth of regions, against 0.59–0.62 for the bottom fifth, Section 2.2 and Appendix B). So unstable regions can be set aside before interpreting or intervening on them.

Exemplar vs region mean. Which representative to use depends on the task. When a task needs a real activation – interpreting a region from the prompt that produced it, or matching regions between models of different architectures (Sections 4 and 5) – we use the exemplar. When it doesn't, as in AxBench detection, the mean is the better choice, because it summarises all of a region's members, not just the one that arrived first. We've made this distinction explicit in Section 2.

The 10-point gap at p1 turned out to come from a single region. Selecting by exemplar picks the same low-coherence region (template-language code, 384 members) for 145 of the 500 concepts, and those concepts score 0.538. On the other 355 concepts, exemplars score 0.953 and region means 0.948.

Tone. Thank you – we've been through the paper and cut repeated qualifications.

Why L14. We've now run the swap at every layer of both models, with the same held-out prompts and judge (new table in Appendix E.1). The swap is effective over a band of middle layers – Gemma L11–16 and Llama L11–15, though only partly at Llama L12 (0.42) – and L14 is inside that band in both models. At every judged layer outside the band, compliance is at most 0.36. The layers where harmful and benign prompts separate most cleanly (Gemma L18, Llama L17) lie just past the band, so the intervention works best at different layers from the ones where harmfulness is most separable.

Reviewer_GYSNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes exemplar partitioning to cluster LLM activations into regions that can be labeled for interpretability. The authors use these partitioning regions to detect concepts, study refusal of harmful requests, and find secret words. The paper describes exemplar partitioning implementation for LLMs and evaluates it on several downstream tasks.

Critical strengths
  • The paper proposes a simple clustering method for studying representation interpretability in LLMs. Its applications to concept detection, safety guardrails, secret word finding, etc. are important.

  • The paper shows the effectiveness of the proposed exemplar partitioning on multiple tasks and benchmarks.

Critical weaknesses
  • The paper lacks comparisons with other clustering methods across multiple tasks. Simple baselines, such as (spherical) k-means and others in FAISS, would help establish the benefit of EP.

  • Some evaluations have limited coverage. Taboo only tests 21 secret words. The refusal experiment also uses only 32 harmful prompts per tested layer.

  • The experiments use Llama 3.1 and Gemma 2, with their model sizes ranging from 2B to 9B parameters. It remains unclear whether EP performs as well on newer and larger models.

  • EP requires a choice of layer and token positions. These choices vary across tasks. The paper provides no general rule for selecting the layer. It is unclear if there is a rule of token selection that can work across diverse tasks.

Questions for authors
  • EP requires input data samples for dictionary construction. How do the amount and choice of this data affect performance? Can a dictionary built from few examples work well on new inputs and tasks?

  • Could the authors test EP on newer and larger models (for example, Qwen 3.8, GLM 5.3 Flash), using broader benchmarks?

  • Could the authors test using the last input token activation (which aggregates information across the entire sequence) for every task? How does this compare with their current token choices?

  • Could the authors provide a general rule for selecting the input layer and token position for EP? If so, could the authors test this rule on new tasks?

1 comment
AuthorsOpenReview ↗

Response to Reviewer GYSN

Thank you for the review.

Clustering baselines. EP dictionaries can be constructed with any clustering algorithm. We primarily use leader clustering because it needs only one streaming pass at a fixed resolution, with no limit on the number of activations. A variant using k-means (EP-k-means, against the default EP-leader) scores slightly lower on the data tested. We fit k-means on a tractable subset and assign the rest, setting each region's exemplar by tracking the real activation closest to each centroid. We've added this to Section 2, with comparison rows on the same activations and protocol. On AxBench, spherical k-means fitted on 500k tokens and assigned over the full 3.62M reaches mean AUROC 0.922 at the same K as EP-leader (5,796), against 0.936 (paired difference 0.014, 95% bootstrap interval 0.007–0.023), and 0.910, 0.905 and 0.891 at K = 3,000, 1,000 and 500 (Table 2). Note that selection keeps the best of K candidates for each concept, so any dense enough set of directions does well there. On Taboo, EP-k-means matches EP-leader (Section 5, Appendix F). For the harmful/benign separation, Appendix E already includes k-means on the prompts' text (TF-IDF) instead of the model's activations: only 27% of harmful prompts land in wholly harmful clusters, against 99–100% for EP.

Coverage. The Taboo testbed contains 21 models, and we test all 21. The harmful/benign separation and cross-model results use all 2,352 prompts at every layer of all four checkpoints. Only the intervention experiment, where every generated response has to be judged, uses 32 harmful prompts per layer.

Newer and larger models. We test Gemma-2 (2B and 9B) and Llama-3.1 8B. We agree that larger models are the natural next step, but we couldn't run them within the discussion period.

Layer and position choice. As with probes, and with SAEs (which are also trained per layer), the right choice depends on the task. When the question is about the whole prompt (refusal, Taboo), we read a fixed position; when a concept appears at particular points in the text (AxBench, SAEBench), we read the activation at every position. We don't have a general rule for choosing the layer.

Amount and choice of data. We build the AxBench dictionary from Pile text, a different distribution from AxBench's instruction–response texts, in the same way that the Gemma Scope SAEs in AxBench's baselines are trained on general pretraining text. We use AxBench's training set only to select one region per concept, by the same rule as SAE-A. The Taboo dictionaries are built from 500 transcripts and the refusal dictionaries from 2,352 prompts, so small, task-specific builds work too. Because EP is a single streaming pass, we also saved the AxBench dictionary after 0.5M and 1M of its 3.62M tokens. Mean AUROC is 0.923 at 0.5M (K = 3,656) and 0.922 at 1M (K = 4,380), against 0.929 for the same local build at the full budget (K = 5,740; the dictionary reported in the paper, built on different hardware, gets 0.936). Appendix B.1 reports these numbers, along with how dictionary size saturates with data for each domain and layer.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of changes (to all reviewers and the AC)

Thank you all for the careful reviews. We've uploaded a revised PDF with the changed text in blue. The main changes:

  1. Only the dominant concept (Fx1p, W2). Fx1p said that each input in our tasks has only one relevant concept, which makes them easier. In fact each benchmark input already contains several concepts, and EP assigns a region to the activation at every position, so a target concept can be detected among the others (Sections 3 and 7).
  2. Clustering baselines (GYSN). GYSN asked for comparisons with other clustering methods, such as spherical k-means. EP dictionaries can be constructed with any clustering algorithm. We primarily use leader clustering because it needs only one streaming pass, and a k-means variant scores slightly lower on the data tested. We have included this comparison (EP-leader and EP-k-means). EP-k-means fits on a tractable subset, assigns the rest, and sets each region's exemplar to the real activation closest to its centroid. We've added it to Section 2 and compared it on AxBench (Section 3, Table 2) and Taboo (Section 5).
  3. Exemplar vs region mean (6Qwq, W2). 6Qwq pointed out that the region mean, which isn't a real activation, beats the exemplar by 10 points on AxBench. That gap comes from a single region, which exemplar selection picks for 145 concepts; on the other 355, exemplars score 0.953 and region means 0.948 (Section 3). Because each exemplar is an activation from a known prompt, we can compare dictionaries built from the same prompts in different architectures (Section 4).
  4. Why L14 (6Qwq) and layer choice (GYSN). 6Qwq asked whether L14 is special or was just one of four sampled layers, and suggested a broader sweep; GYSN asked for a rule for choosing layers. We've now run the region swap at every layer of both models. It's effective over a band of middle layers, and L14 is inside that band in both models (Appendix E.1).
  5. Amount of data (GYSN). GYSN asked how the amount of data affects performance. Mean AxBench AUROC is 0.923 from the first 0.5M of the 3.62M tokens, against 0.929 for the same build at the full budget (Appendix B.1). This is because AxBench concepts are relatively common in real data, and a single example of a rare concept is enough for EP-leader to assign it a region.
  6. Hyperparameters (6Qwq). 6Qwq noted a considerable number of hyperparameters, including p and K, that need tuning across tasks – this is not the case, EP has exactly one hyperparameter: the percentile p, which sets the distance threshold, for the leader-clustering build used throughout (or K for the k-means variant). With leader clustering, K is the number of regions the build produces at that threshold, so it is never chosen. Section 2 now says this explicitly. We also used one p for all the small-corpus experiments, with no per-task tuning.
  7. Tone (6Qwq). 6Qwq found the writing defensive. We've been through the paper and cut repeated qualifications.
  8. Related work (Fx1p). We now cite Dalvi et al. (2022), ICA Lens and HARP.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.