- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies how sparse autoencoder (SAE) features relate to each other, instead of looking at each feature on its own. The authors build a co-activation graph from SAE features. They then focus on one kind of relationship they call "mutual exclusivity" (ME): features that appear in similar contexts but never fire together. The authors study four models and three SAE architectures, and observe that ME features form interpretable group whose members are alternative values of the same type, such as digits, pronouns or months. Their decoder directions point in similar directions, forming narrow cones. The authors also run "intervention" experiments showing that these features have causal roles: swapping one ME feature for another from the same group shifts the model's prediction toward the swapped-in value.
Critical strengths
- Experimental breadth: the analysis covers four models, three SAE architectures and every layer. The steering experiments are compared against several sensible alternatives, so the effect is shown to be specific to ME features.
- New characterization of SAE features: most prior work, to the best of my knowledge, on feature interactions looks at features that fire together. This paper studies instead features that exclude each other, making a clear connection to distributional semantics.
Critical weaknesses
- Limited significance? The paper offers a careful and well-executed analysis that would likely interest the NLP and interpretability communities. In particular, the link between mutually exclusive features and linguistic categories (closed word classes, paradigmatic relations from distributional semantics) is a nice bridge between interpretability and linguistics, and it could be of interest in a venue like ACL or EMNLP. For ICLR, though, I'm less sure the findings tell us much that is new about how language models learn or compute. Most cliques are closed word categories (digits, pronouns, months, prepositions, modal verbs), for which it is fairly expected that features for different words don't fire together.
===
Minor:
- "Larger cliques are more interpretable" may be an artifact? A bigger set gives the LLM judge more examples to spot a pattern, so the trend in Fig. 2 may partly reflect how the judge works.
- fix citations format
Questions for authors
- IIUC, many ME features are highly token-specific (high token purity), and each position holds only one token, so features for "3" and "4" can hardly fire on the same token. How much of the ME signal do the authors think reflects structure the model has learned, as opposed to a consequence of tokenization? Would the findings hold or change if the analysis was about low-purity features?
3 comments
Dear reviewer, thank you again for your feedback. With the discussion period ending on November 18, we would appreciate hearing whether our response addresses your main concerns or whether you have any other thoughts. We would be happy to clarify any remaining questions.
I thank the authors for their response. As I think this can be a good interpretability contribution, I am raising my score.
Thank you for taking the time and reviewing our manuscript. We are glad that you find the breadth of our experiments a strength, and that you recognize the novelty in analyzing interactions between features. We first address your questions and return to the weakness in the end.
Could the clique-size trend come from the judge?
We applied the same prompt to ME cliques and several controls, including uncoupled groups matched for size and token purity. The YES rate consistently rises with group size only for ME cliques. We have revised Figure 2 and added Figure 15 to include these results, and added a description highlighted in blue in Section 4.2.The controls give the judge equally large groups, so having more features to look at alone does not explain the trend.
Token purity and tokenization
We follow your suggestion to look at low-purity cliques. We found several semantically coherent cliques among features with low token purity (Table 1, and additional highlighted text in 4.2). These include cliques whose members represent features that correspond to different programming languages, different regions of Canada, or different written language such as Spanish, Russian, etc.
These found cliques also support steering. For example, swapping the Russian feature for the Spanish feature on a Russian prompt changed the model’s continuation from Russian to Spanish (Figure 23). We will expand this evidence further in our next revision. Thank you for this suggestion! The low-purity examples strengthen our evidence that ME cliques capture meaningful relationships among features.
Citation formats
We fixed the issues in the citation formats. Thank you
Significance of findings
The paper is mainly an interpretability contribution, and the ideas described in it can be applied to any modality. We clarify it below and will revise the introduction and discussion to better clarify this point.
1. The contribution is a move from single features to structure among features. SAE-based interpretability today works feature by feature. Each feature is labeled and steered on its own, as if it were one entry in a flat dictionary. But how a model computes depends on how its representations relate to each other, not only on what each one means alone. Our paper takes a first step toward interpreting at that level. We show that one relationship, mutual exclusivity, recovers higher-level types on its own: a variable (e.g., “which direction”, “which pronoun”, “which language”, “which Canadian region”...) together with the values it can take. A single feature cannot show this, because a feature’s meaning as “one of the alternatives for X” exists only in relation to the other alternatives. This adds a layer of meaning that isolated feature labels miss. It is a step from reading a model one unit at a time toward reading the variables it computes with. We hope this is a first step towards interpretability that analyzes higher-level structure and more complicated mechanisms, beyond low level features.
2. Finding closed classes is not trivial, and it supports the method. Rarely firing together is not enough on its own: mutual exclusivity also requires shared context (Figure 5). ME-based swaps are causal, beating the controls and working in later layers where standard feature selection fails (Figure 8), and recovering known categories without supervision validates the method before it is used where we can’t check the results. As our results on impure-cliques highlight, this goes much beyond simple word features (Table 1).
We see ME features as the first evidence of the hypothesis in a broader modality-agnostic interpretability program, namely that analyzing how features co-occur can help to uncover the compositional structure of models.
Our contribution fits into the broader AI community and therefore to ICLR. We are happy to add this additional context and framing to the manuscript.
We hope that this addresses your concerns and are happy to continue the discussion.