acceptodds
Under review as a conference paper at ICLR 2027

Do Features in a Sparse Autoencoder Really Represent Clear Semantics?

10 views

Abstract

The sparse autoencoder (SAE) is a widely used mechanistic interpretability method that transforms dense LLM activations into sparse features. In this paper, we propose using interactions as a numerically verifiable metric to evaluate the semantic clarity of SAE features from four perspectives. (1) We observe that 19.2%–35.6% of the sampled SAE feature activations encode non-sparse interaction patterns, suggesting more diverse semantics. The average number of interactions per feature varies dramatically, and we find no clear correlation between the interaction number and the feature's activation frequency. (2) We find that SAE feature activations that encode fewer interactions tend to model simpler and more human-interpretable interactions. (3) 25.2%–41.9% of the feature activations encode interactions with severe positive-negative cancelling effects, indicating potentially less coherent semantics. (4) We find that 6.2%–19.5% of the SAE feature activations capture similar semantics across different contexts, and these activations also encode interactions with less positive-negative cancellation.

Primary area:
learning theory
Keywords:
Neural Symbolic, Explainable Artificial Intelligence, Mechanistic Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_U5gzOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

This paper uses the interaction idea of Ren et al. (2024) to explain the features learned by SAEs. Using this tool, the authors confirm that most SAE features remain polysemantic (i.e., they require multiple interactions to be explained) but a majority of features are in the "low interaction density" group, which suggests that these semantics may not be too complex.

Critical strengths
  • Interpreting SAE features through interactions is a potentially meaningful research directions. To my knowledge, the authors are the first to do it.
  • The findings starting on page 7 are interesting. I enjoyed reading them.
  • Even the negative results are interesting, e.g., "we find that the interaction number is positively correlated with interaction complexity" indicates that there remain complex SAE features.
Critical weaknesses
  • The scale of the experiment is limited, i.e., the authors experiment with 21 features and 10 input variables per feature. Given the scale of today's models, I am not sure this is enough to draw generalizable observations.
  • Some of the results (e.g., features with high PNCI score) suggest that interactions may not be a universally useful explainability tool.
Questions for authors
  • The scale of the experiment is limited. In Appendix I, the authors state "We select a total of 21 features per SAE". Further, "we only compute interactions among at most the 10 input variables closest to the token position where the SAE feature is triggered". Given the scale of today's models, I am not sure this is enough to draw generalizable observations. I recommend the authors choose one LLM and expand the experiment for that particular model.
  • Some of the results suggest that interactions may not be a universally useful explainability tool. For example, the authors report that multiple features have a high PNCI scores, which effectively makes interactions useless for those features. To counteract this observation, the authors should implement a controllability experiment to causally align features with their explanation. For example, the authors could steer features towards different semantics by aligning them with different interactions, and verify whether this change is reflected in the output.

Minor presentation issues:

  • The introduction mentions "using interactions as a numerically verifiable metric", without defining what an "interaction" means. Given that the term is overloaded, the authors should define it earlier in the paper.
  • "Conversely, encoding a large number of interactions indicates that the feature is likely activated by various phrase patterns representing diverse semantics." - Note that lexicalization != semantics. That is, the "various phrase patterns" may capture similar semantics.
  • Typo: "explanationion"
7 comments
AuthorsOpenReview ↗

This is the first part of Question A.

We thank you for your careful review and valuable suggestions. We have summarized the questions you raised into the following points and address them one by one.

Question A

Ask for scaling up the experiments to verify the generality of the conclusions.

Quoted from question 1: The scale of the experiment is limited ... Given the scale of today's models, I am not sure this is enough to draw generalizable observations. I recommend the authors choose one LLM and expand the experiment for that particular model.

Weakness 1: The scale of the experiment is limited, i.e., the authors experiment with 21 features and 10 input variables per feature.

Answer

It is a really good question. Following your suggestion, we have scaled up the experiments on the SAE built upon Gemma-2-9B and examined how the main conclusions of the paper change. Specifically, we reran the main experiments in the main text on approximately a thousand SAE feature activations, covering hundreds of thousands of distinct masked input samples in total, and updated the experimental results. All expanded experiments below use the same SAE built upon Gemma-2-9B and the dataset constructed in Appendix I.2 of the paper.

Finding 1 still holds after expanding SAE features, i.e., the number of interactions encoded by the same SAE feature still varies substantially across different positions.

Low-interaction-density groupMedium-interaction-density groupHigh-interaction-density group
Before expanding features80.8%13.3%5.9%
After expanding features75.4%14.6%10.0%

Interaction number measures the number of phrase patterns that a feature uses during inference.

According to the rules in Line 315 of the paper, we categorize these SAE feature activations into three groups based on the number of interactions they encode: (1) the low-interaction-density group, (2) the medium-interaction-density group, and (3) the high-interaction-density group.

As shown in the table, the proportion of SAE features with low interaction density changes from 80.8% before expanding features to 75.4% after expanding features. This indicates that, both before and after the expansion, SAE features mainly encode relatively sparse interactions. In addition, before expanding features, statistics over all activations of SAE features at different token positions show that the average number of encoded interactions is 6.90, with a large standard deviation (11.55); after expanding features, the average number of encoded interactions is 9.19, and the standard deviation remains large (16.27). This indicates that the number of interactions encoded by the same SAE feature still varies substantially across different positions. In summary, Finding 1 still holds.

AuthorsOpenReview ↗

This is the second part of Question A.

Finding 2 still holds after expanding SAE features, i.e., the average number of interactions differs markedly across different SAE features, and there is no clear relationship between the average number of interactions and activation frequency.

MetricBefore expanding featuresAfter expanding features
Average number of interactions encoded by different features (mean ± std)6.90 ± 3.719.19 ± 4.95
Pearson correlation coefficient between feature activation frequency and average number of interactions−0.161−0.207
Spearman correlation coefficient between feature activation frequency and average number of interactions0.067−0.047

As shown in the table, before and after expanding features, the average numbers of interactions across different features are 6.90 and 9.19, respectively, both with large standard deviations (3.71 and 4.95). This indicates that the average number of interactions differs markedly across different SAE features.

In addition, activation frequency refers to the proportion of tokens on which an SAE feature is activated. We find that, before and after expanding features, both the Pearson and Spearman correlation coefficients between the activation frequency of SAE features and the average interaction count are very low: the Pearson coefficients are −0.161 and −0.207 before and after expanding features, respectively, and the Spearman coefficients are 0.067 and −0.047. Therefore, we conclude that, both before and after expanding features, there is no clear relationship between the average number of interactions and activation frequency. In summary, Finding 2 still holds.

Finding 3 still holds after expanding SAE features, i.e., SAE feature activations encoding few interactions primarily model simpler interactions, whereas those encoding many interactions primarily model more complex interactions.

Low-interaction-density groupMedium-interaction-density groupHigh-interaction-density group
Proportion of low order interactions89.6%→86.1%76.7%→69.3%50.6%→44.7%

Note: In each cell, the value on the left of the → is the result before expanding features, and the value on the right is the result after expanding features.

The table shows that, among all interactions the SAE features in the low-interaction-density group (i.e., those encoding just a few interactions), 89.6% interactions are low-order (simple) interactions before expanding features. The ratio slightly changes to 86.1% after expanding features.

On the other hand, among all interactions the SAE features in the high-interaction-density group (i.e., those encoding many interactions), 50.6% interactions are low-order (simple) interactions before expanding features. The ratio slightly changes to 44.7% after expanding features. In summary, Finding 3 still holds.

Finding 4 still holds after expanding SAE features, Regardless of whether we expand the experiments, we still find that, although high activation strengths are often regarded as strong semantic signals, a small number of strongly activated SAE features still model severe positive-negative cancellation interactions (i.e., their phrase patterns are highly mutually conflicting).

MetricBefore expanding featuresAfter expanding features
Pearson correlation coefficient between activation strength and PNCI−0.044−0.001
Spearman correlation coefficient between activation strength and PNCI0.0520.087

This conclusion is supported by the fact that activation strength is almost unrelated to PNCI. Both the Pearson and Spearman correlation coefficients between activation strength and PNCI are very low: the Pearson coefficients are −0.044 and −0.001 before and after expanding features, respectively, and the Spearman coefficients are 0.052 and 0.087. In summary, Finding 4 still holds.

Overall, the scale of the experiments does not fundamentally affect the main findings of our work.

AuthorsOpenReview ↗

This is the first part of Question B.

Question B

Concerns about interactions with high PNCI values.

Weakness 2: Some of the results (e.g., features with high PNCI score) suggest that interactions may not be a universally useful explainability tool.

Quoted from question 2: The authors report that multiple features have a high PNCI scores, which effectively makes interactions useless for those features ... The authors should implement a controllability experiment to causally align features with their explanation ... The authors could steer features towards different semantics by aligning them with different interactions, and verify whether this change is reflected in the output...

Answer

First, interaction-based explanations have received increasing attention and have been widely used to explain various models [1-7]. According to Zhang et al. [8] and Ren et al. [9], a high PNCI value is a typical sign of overfitting, and almost all overfitted models exhibit high PNCI values across various applications [10]. Therefore, a high PNCI value does not indicate a failure of the interaction-based algorithm.

[1] Mingjie Li et al. Does a Neural Network Really Encode Symbolic Concepts? ICML 2023.

[2] Die Zhang et al. Building Interpretable Interaction Trees for Deep NLP Models. AAAI 2021.

[3] Wen Shen et al. Interpreting Representation Quality of DNNs for 3D Point Cloud Processing. NeurIPS 2021.

[4] Yingwei Ren et al. Lane-change risk prediction model based on traffic context and driving styles. Sci. Rep., 2025.

[5] Lu Chen et al. Can LLMs Reason Soundly in Law? Auditing Inference Patterns for Legal Judgment. ICLR 2026.

[6] Ruiyang Qin et al. Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions. ICML 2026.

[7] Hao Zhang et al. Interpreting and Boosting Dropout from a Game-Theoretic View. ICLR 2021.

[8] Junpeng Zhang et al. Two-Phase Dynamics of Interactions Explains the Starting Point of a DNN Learning Over-Fitted Features. arXiv:2405.10262, 2024.

[9] Qihan Ren et al. Towards the Dynamics of a DNN Learning Symbolic Interactions. NeurIPS 2024.

[10] Junpeng Zhang et al. Randomness of Low-Layer Parameters Determines Confusing Samples in Terms of Interaction Representations of a DNN. arXiv:2502.08625, 2025.

Second, unlike many explanation metrics, the existence of each interaction can be rigorously verified, regardless of whether their PNCI value is high or low. As stated in line 256 of the paper (Figure 2(a)), we have designed experiments to demonstrate that interactions indeed capture the actual patterns used by the SAE feature during inference. Specifically, given 1024 different masked states x′∈Ψ\mathbf{x}'\in\Psi of an input sample, we use the following metric ρ\rho (see Equation (3) in the paper) to measure the fidelity of using the interactions to match the SAE feature activations across all 1024 masked inputs:

\left[ \frac{\left|g(\mathbf{x}') - v(\mathbf{x}')\right|} {\left|v(\mathbf{x})\right|} \right],$$ where $v(\mathbf{x}')$ is the true activation value of the SAE feature under the masked input $\mathbf{x}'\in\Psi$, and $g(\mathbf{x}')$ is the activation value fitted using only the interactions we extracted. The experiments are conducted on the SAE built upon Gemma-2-9B using the SQuAD dataset. Figure 2 in the paper shows that, for each input sample, the interactions can always accurately match the the SAE feature's activation values under all masked inputs (with an average matching error $\rho=0.024$ ), and we find that the number of significant interactions typically does not exceed 20. Therefore, we have reason to believe that the interactions reflect the actual patterns used by the SAE feature during inference, rather than an artifact of manual construction. **Third, and more importantly, we have added new experiments following your suggestion.** The purpose of these experiments is to demonstrate that, on samples with severe positive-negative cancellation (PNCI > 0.7), interactions can still faithfully explain the activation values of SAE features. To this end, on the SAE built upon Gemma-2-9B, we use interaction-based explanations to precisely manipulate the activation values of SAE features. We present two examples below.
Reviewer_52qdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors argue that sparse activation alone doesn't show whether an SAE feature has a clear meaning, so they propose examining the AND-OR interactions among input words that drive each activation. They apply this to sampled activations from three open-source SAEs, and measure how many interactions each activation relies on, how complex and mutually cancelling those interactions are, and whether annotators judge the activation to reflect the feature's main meaning. The results show that many activations depend on numerous cancelling interactions and only a small fraction express their feature's primary semantics, and come up with the conclusion that SAE features are less semantically clean than their sparsity suggests.

Critical strengths
  1. The authors use AND-OR interaction decompositions to provide a set of numerically grounded diagnostics (interaction count, interaction order, positive-negative cancellation index, and cross-activation semantic consistency). By grounding the analysis in the universal matching and sparsity properties of interactions, the work offers a more rigorous and reproducible lens through which the semantic clarity of individual SAE features can be examined. This methodological shift is a clear contribution to the evaluation toolkit in mechanistic interpretability.
  2. The authors present quantitative evidence that a non-trivial fraction of feature activations encode non-sparse interaction patterns, higher-order interactions, substantial positive-negative cancellation, and limited cross-context consistency of primary semantics. What interesting is that neither activation strength nor activation frequency exhibits a clear positive correlation with these semantic metrics. These results provide measurable pressure on a core premise of the SAE literature and supply specific diagnostic signals that future work on feature quality or SAE training objectives can target.
Critical weaknesses
  1. SAEs are trained without supervision, and no prior work claims their features are guaranteed to be monosemantic. The main findings can plausibly be explained by known effects (A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders). Feature Absorption leads a general feature to learn context-dependent suppression, which would appear as negative, higher-order interactions and raise PNCI. Feature Splitting divides concepts at a granularity set by dictionary width, which naturally lowers SCATP. The paper does not vary width or L0, does not stratify by activation strength, and has no baselines or comparison with existing metrics (SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability). After accounting for these effects, it is unclear what the interaction framework adds, especially given its exponential cost.
  2. The author replace the JumpReLU/TopK activation with a ReLU on the pre-activation (Appendix I.4), which removes the learned thresholds and the TopK competition. SAEs are trained to approach the sparsity–reconstruction Pareto frontier, so the learned features depend heavily on the native activation function. Table 3 confirms that the change is substantial: on Gemma-2-9B, the share of high-density activations drops from 32.0% to 5.9%, which indicates the reconstruction performance might has been impacted significantly. Therefore, it is unclear whether the explained activations remain interpretable as SAE features, and the extracted interactions may describe ReLU projections onto encoder directions rather than the features themselves. This also makes the comparison across SAEs, which differ mainly in their activation functions, hard to interpret.
Questions for authors
  1. How would the conclusions of the authors’ interaction framework change when considering feature splitting and feature absorption?
  2. If the activation function is not replaced with ReLU, how would the conclusions of the interaction framework change?
10 comments

Thank you for your review! We will address your questions one by one.

Question A

Quoted from the first bullet in the weakness section: SAEs are trained without supervision, and no prior work claims their features are guaranteed to be monosemantic.

Answer

Thank you for your comment, although we are not entirely sure what specific concern lies behind this question. In fact, the pursuit of monosemanticity is the core motivation and ultimate goal driving the development of SAEs [1–5]. Various studies, including developing different SAE architectures [1–4] and setting the explanation target to more complex, better-trained commercial LLMs [5], all aim to improve the monosemanticity of SAE features. Thus, monosemanticity has consistently been the most central scientific problem in SAE research.

Actually, in the field of mechanistic interpretability, the evaluation, critique, and in-depth analysis of the monosemanticity and polysemanticity (i.e., semantic clarity) of SAE features have long been major research hotspots [6–9]. Compared to previous studies, our work advances traditional analysis methods, which rely on observing activating contexts and empirically summarizing semantics, into a new interaction-based paradigm. This paradigm provides a theoretical framework to clarify how SAE activation values mathematically connect to symbolic AND-OR patterns with verifiable error bounds. In this way, our work extends the frontier of SAE research and uncovers the inner connections between two prominent paradigms in XAI (SAEs and interaction-based explanations), which has been long awaited by the theoretical community.

[1] Trenton Bricken et al. Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Transformer Circuits Thread, 2023.

[2] Robert Huben et al. Sparse Autoencoders Find Highly Interpretable Features in Language Models. ICLR 2024.

[3] Senthooran Rajamanoharan et al. Improving Sparse Decomposition of Language Model Activations with Gated Sparse Autoencoders. NeurIPS 2024.

[4] Leo Gao et al. Scaling and Evaluating Sparse Autoencoders. ICLR 2025.

[5] Adly Templeton et al. Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. Transformer Circuits Thread, 2024.

[6] Aleksandar Makelov et al. Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control. ICLR 2025.

[7] David Chanin et al. A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders. NeurIPS 2025.

[8] Gonçalo Santos Paulo et al. Automatically Interpreting Millions of Features in Large Language Models. ICML 2025.

[9] Adam Karvonen et al. SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability. ICML 2025.

Question B

Question 1: How would the conclusions of the authors’ interaction framework change when considering feature splitting and feature absorption?

Quoted from the first bullet in the weakness section: ... Feature Absorption leads a general feature to learn context-dependent suppression, which would appear as negative, higher-order interactions and raise PNCI. Feature Splitting divides concepts at a granularity set by dictionary width, which naturally lowers SCATP... After accounting for these effects, it is unclear what the interaction framework adds, especially given its exponential cost.

Answer

Thank you very much for this insightful discussion. As you noted, phenomena such as feature splitting and feature absorption do, to some extent, render SAE features less monosemantic than desired. Consequently, these challenges have inspired a shared goal across the community (including ourselves): to continuously improve SAE reconstruction quality and monosemanticity through more advanced training algorithms and more rigorous evaluation methods.

To this end, we contribute a new analytical tool to investigate SAEs and their failure modes stemming from feature absorption and splitting. Specifically, unlike empirical semantic summarization based on activating contexts [1, 2], we strictly quantify which input patterns drive SAE feature activations and to what extent. In other words, while feature splitting and absorption represent phenomenon-level observations, our work leverages interactions as a rigorous metric to mathematically decompose polysemantic features with clear theoretical boundaries.

More importantly, by distinguishing between true and spurious monosemanticity, our interaction metric helps researchers better investigate phenomena such as feature splitting and absorption.

For example, let us focus on the SAE feature for "history-related concepts" in Figure 6, which activates on the tokens [of] and ['] in the following texts.

① Warsaw’s mixture of architectural styles reflects the turbulent history of

② The historical measure of a steam engine’

This feature appears to be polysemantic, as it fires on seemingly unrelated prepositions and punctuation marks.

However, through our proposed method, we find that the strongest interactions driving the activations in these two samples are I{“history”}AND=17.79I^{\mathrm{AND}}_{\lbrace \text{“history”} \rbrace}=17.79 and I{“historical”}AND=7.36I^{\mathrm{AND}}_{\lbrace \text{“historical”} \rbrace}=7.36, respectively. These clear numerical results reveal that the core contributions driving the above SAE feature still come from "history" and "historical"—the two words directly related to history. In other words, this is merely spurious polysemanticity.

Therefore, this example shows that our interaction metric can successfully filter out confusing surface phenomena and reveal deeper mechanistic insights.

[1] Steven Bills et al. Language Models Can Explain Neurons in Language Models. OpenAI, 2023.

[2]Trenton Bricken et al. Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Transformer Circuits Thread, 2023.

Question C

Question 2: If the activation function is not replaced with ReLU, how would the conclusions of the interaction framework change?

Quoted from the second bullet in the weakness section: The author replace the JumpReLU/TopK activation with a ReLU on the pre-activation (Appendix I.4), which removes the learned thresholds and the TopK competition ... Therefore, it is unclear whether the explained activations remain interpretable as SAE features, and the extracted interactions may describe ReLU projections onto encoder directions rather than the features themselves ...

Answer

A good question. We will clarify the concerns you raised from three aspects.

First, using ReLU merely simplifies the analysis of pre-activation values, it does not mean changing the SAE features used for later reconstruction. Using a simple (ReLU) activation as a surrogate explanation target is a common practice in both attribution methods [1, 2] and mechanistic interpretability studies [3–6]. This is because many activation functions (such as JumpReLU) involve hard-threshold truncation, which introduces high-frequency noise into the analysis.

For example, consider a simple linear function f(x)=xf(x)=x. Applying a hard threshold yields f~(x)=x⋅I(x>0.5)\tilde{f}(x)=x\cdot\mathbb{I}(x > 0.5), which introduces a sharp, non-differentiable jump discontinuity into an otherwise smooth function. Such discontinuities are known to introduce high-frequency components, making the truncated function considerably more complex to represent despite its simple underlying mechanism.

Therefore, lots of studies, including both theoretical studies [1-3] and practical interpretability studies [4-6] merely treat JumpReLU or TopK as a typical engineering trick for achieving sparsity. The complex function landscape behind JumpReLU is generally not regarded as an explanation of the essential nature of model representations. For example, Cammarata et al. [4] use pre-ReLU responses to analyze curve detector neurons, distinguishing strongly inhibited inputs from those right near the activation threshold, thus preserving information that becomes indistinguishable once ReLU zeroes them out. Similarly, Tillman et al. [5] directly use the pre-activations of TopK SAEs to train monitoring probes for language model outputs, thereby retaining information that might otherwise be lost during sparsification.

More critically, replacing the activation function with ReLU does not change the SAE features. Each SAE feature corresponds to a specific direction in the LLM's intermediate representation space (as formulated by Anthropic [7] , one of the pioneers and primary drivers of SAE research). The pre-activation of an SAE can be roughly understood as the projection of the LLM's representation vector onto that SAE feature direction. Activation functions like JumpReLU, TopK, and ReLU merely zero out certain pre-activations via thresholding or ranking; none of them alter the underlying SAE feature directions. Consequently, switching to ReLU does not change direction of the SAE features we are analyzing, nor does it alter the original activation values for features that are already active under JumpReLU. Using ReLU simply makes the interaction patterns cleaner and more concise without changing the underlying SAE features we aim to explain.

Finally, actually we have conducted your concerned experiment in Appendix I.4, in which we have extracted interactions by setting the target output vv as the output of the JumpReLU function. As shown in Figure 7, the hard truncation in JumpReLU introduces high-frequency noise, which is modeled as numerous medium-order and high-order interactions. These noisy interactions are merely mathematical artifacts of truncation rather than meaningful phrase patterns genuinely learned by the SAE feature, and should therefore not be used to interpret the SAE features' true semantics.

[1] Sebastian Bach et al. On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation. PLOS ONE, 2015.

[2] Emmanuel Ameisen et al. Circuit Tracing: Revealing Computational Graphs in Language Models. Transformer Circuits Thread, 2025.

[3] Karen Simonyan et al. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. ICLR Workshop 2014.

[4] Nick Cammarata et al. Curve Detectors. Distill, 2020.

[5] Henk Tillman et al. Investigating Task-Specific Prompts and Sparse Autoencoders for Activation Monitoring. arXiv:2504.20271, 2025.

[6] Boyi Deng et al. SASFT: Sparse Autoencoder-guided Supervised Finetuning to Mitigate Unexpected Code-Switching in LLMs. ICLR 2026.

[7] Nelson Elhage et al. Toy Models of Superposition. Transformer Circuits Thread, 2022.

Reviewer_fTyPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This work studies SAE features using interaction-based explanations. For an input that activates an SAE feature, the authors fit a logical function composed of AND and OR interactions over the content tokens present in the input, such that the function approximates the activation value. From this logical function, the authors compute several metrics, such as the number of interactions and their complexity (the number of tokens involved in each interaction), and provide analysis based on these metrics.

Critical strengths
  • This work connects SAE feature interpretation with interaction-based analysis, which is novel. That literature has some additional theoretical backing.
  • Interactions have the potential upside of offering more information per activating text example than standard max-activation displays.
  • Finding 5 is an initial step toward connecting interaction-based metrics to more traditional SAE interpretation from activating text examples.
Critical weaknesses
  • My primary concern is that most of the findings are presented as descriptive statistics of the interaction metrics rather than as conclusions about SAE features. Each reports a distribution or a (non-)correlation, but there is little follow-up analysis examining whether the pattern reflects a property of the SAE feature or an artifact of the interaction extraction procedure, which makes the claims difficult to falsify. For example, Finding 1 ("a single SAE feature can activate at multiple token positions, but the number of interactions it encodes varies widely") could equally be evidence that analyzing an SAE feature's interactions at individual activating examples is the wrong abstraction, rather than evidence that features play different roles per sample. Things that would help: comparing these metrics against those computed on control or random directions rather than SAE features, and additional experiments like Finding 5 that aim to connect interaction-based metrics to classic evaluations of SAE feature interpretability.

  • It seems very limiting to try to interpret SAE features on a single example. Additionally, many high-level features are better described by a shared high-level property of the text (e.g., sycophantic text) rather than by the presence of tokens in the input.

Questions for authors

Please see the weaknesses above, in particular comparisons to random/control directions, and trying to better connect interaction metrics to something independently provable about the SAE features.

5 comments

We thank you for your careful review and valuable suggestions. We have summarized your questions into the following points and will address them one by one.

Question A

Asking for evidence that interactions truly capture the internal inference patterns of SAE features.

Quoted from the first bullet in the weakness section: ... Each reports a distribution or a (non-)correlation, but there is little follow-up analysis examining whether the pattern reflects a property of the SAE feature or an artifact of the interaction extraction procedure, which makes the claims difficult to falsify ... could equally be evidence that analyzing an SAE feature's interactions at individual activating examples is the wrong abstraction, rather than evidence that features play different roles per sample...

Answer

Whether interaction serves as a proper abstraction of model representations is indeed a good question. In fact, the very purpose of introducing interaction patterns is to provide a numerically verifiable metric to validate whether the explained mechanism/pattern really exists within the neural network.

As detailed in Line 256 (Figure 2(a)) in the paper, we have designed an experiment to demonstrate that interactions indeed capture the real patterns used by SAE features during inference. Specifically, given 210=10242^{10}=1024 different masked states Ψ={xS∣S⊆N}\Psi=\{\mathbf{x_S}|S\subseteq N\} of an input sample x\mathbf{x}, the core of this experiment is to verify whether the interactions can well match the activation values of the SAE feature across all 1,024 masked states. If the matching error is sufficiently small and only a few interactions are salient, we can conclude that the interactions faithfully reflect the learned inference pattern of the SAE feature.

The metric ρ\rho for quantifying the matching error is defined as follows: ρ=Ex′∈Ψ[∣g(x′)−v(x′)∣∣v(x)∣].\rho = \mathbb{E}_{\mathbf{x}' \in \Psi}\left[\frac{\left|g(\mathbf{x}') - v(\mathbf{x}')\right|} {\left|v(\mathbf{x})\right|}\right]. where v(x′)v(\mathbf{x}') denotes the true activation value of the SAE feature under the masked input x′∈Ψ\mathbf{x}'\in \Psi, and g(x′)g(\mathbf{x}') is the activation value fitted solely using the extracted interactions. For each sample, we average the matching errors across all masked states in Ψ\Psi. Furthermore, considering the vast variations in activation values across different samples, we normalize this error by the true activation value ∣v(x)∣|v(\mathbf{x})| of the unmasked input x\mathbf{x}. Experiment is conducted on an SAE built upon Gemma-2-9B using the SQuAD dataset. Figure 2 shows that, for most input samples, typically no more than 20 salient interactions are extracted and the matching error ρ\rho remains below 0.05.

This demonstrates that interactions can precisely mimic the behavior of SAE features under all 2n2^n masked inputs. Therefore, there is reason to conclude that interactions reflect the true inference patterns used by SAE features.

AuthorsOpenReview ↗

This is the first part of Question B.

Question B

Ask for comparing our results against the interactions extracted from a random feature direction.

Quoted from the first bullet in the weakness section: ... there is little follow-up analysis examining whether the pattern reflects a property of the SAE feature... Things that would help: comparing these metrics against those computed on control or random directions rather than SAE features...

Quoted from question 1: ... in particular comparisons to random/control directions...

Answer

This is a good question. As you suggest, we add a new experiment to examine whether interactions reflect a distinct property of SAE features or merely intrinsic mathematical properties of the interaction-extraction algorithm.

Specifically, if interaction metrics capture distinct properties of SAE features, then the interaction metrics extracted from SAE features should differ significantly from those extracted from random feature directions.

Specifically, we conduct this experiment on the layer-20 residual stream of Gemma-2-9B. We compare SAE features with random feature directions in the residual stream using our proposed interaction metrics (interaction number, interaction order, PNCI, etc.).

Let ai(x)\mathbf{a}_i(\mathbf{x}) denote the intermediate output vector in the LLM's residual stream at the ii-th token of sample x\mathbf{x}, and let rj\mathbf{r}_j denote a random feature direction sampled from the representation space of ai(x)\mathbf{a}_i(\mathbf{x}). We define the target function to be explained as the rectified projection of ai(x)\mathbf{a}_i(\mathbf{x}) along the random feature direction rj\mathbf{r}_j:

vrandom=ReLU⁡!(ai(x)⊤rj).v_{\text{random}} = \operatorname{ReLU}\\!\left(\mathbf{a}_i(\mathbf{x})^\top\mathbf{r}_j\right).

For comparison, we compute the SAE feature by projecting ai(x)\mathbf{a}_i(\mathbf{x}) onto the learned encoder direction wenc,j\mathbf{w}_{\text{enc}, j} of the SAE:

vSAE=ReLU⁡!(ai(x)⊤wenc,j+benc).v_{\text{SAE}} = \operatorname{ReLU}\\!\left(\mathbf{a}_i(\mathbf{x})^\top\mathbf{w}_{\text{enc}, j}+b_{\text{enc}}\right).

For a fair comparison, we follow the experimental settings for SAE features in Appendix I.3; the number of SAE features and the number of random feature directions are both 21. The results are as follows.

(1) Interaction density

Feature TypeProportion of feature activations in low-interaction-density groupProportion of feature activations in medium-interaction-density groupProportion of feature activations in high-interaction-density group
SAE feature direction80.79%13.33%5.87%
Random feature direction35.08%14.76%50.16%

According to the rules in Line 315 of the paper, we categorize these SAE feature activations into three groups based on the number of interactions they encode: (1) the low-interaction-density group, (2) the medium-interaction-density group, and (3) the high-interaction-density group.

As shown in the table above, 80.79% of SAE features belong to the low-interaction-density group (i.e., they encode relatively simple interaction semantics), whereas only 35.08% of random feature directions belong to this group. This indicates that SAE feature directions encode noticeably fewer phrase/interaction patterns than random feature directions.

AuthorsOpenReview ↗

This is the second part of Question B.

(2) Interaction order

Feature TypeAverage interaction order of low-interaction-density groupAverage interaction order of medium-interaction-density groupAverage interaction order of high-interaction-density group
SAE feature direction2.1232.6033.274
Random feature direction2.5193.4224.375

Compared with random feature directions, SAE features encode lower-order (i.e., less complex) interactions in all three groups. This suggests that the phrase patterns (interactions) modeled by SAE features are relatively simple and thus easier for humans to interpret.

(3) Positive-Negative Cancellation Index (PNCI)

If a feature direction encodes interactions with severe mutually cancelling effects (i.e., PNCI > 0.5), we consider this feature direction to reflect potentially conflicting or less coherent semantics.

Feature TypeProportion of features with severe interaction cancellation
SAE feature direction25.16%
Random feature direction77.44%

The table above shows that the proportion of SAE feature directions with severe interaction cancellation is significantly lower than the corresponding proportion for random feature directions. This means that, compared with SAE feature directions, random feature directions encode more highly entangled and mutually conflicting interactions, making them more difficult for humans to interpret.

In summary, the interaction metrics (interaction number, order, and PNCI) are powerful enough to reveal the advantages of SAE feature directions over random feature directions. These advantages indicate that the findings based on interactions are unique to SAE features, rather than intrinsic mathematical properties of the interaction-extraction algorithm.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.