- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper uses the interaction idea of Ren et al. (2024) to explain the features learned by SAEs. Using this tool, the authors confirm that most SAE features remain polysemantic (i.e., they require multiple interactions to be explained) but a majority of features are in the "low interaction density" group, which suggests that these semantics may not be too complex.
Critical strengths
- Interpreting SAE features through interactions is a potentially meaningful research directions. To my knowledge, the authors are the first to do it.
- The findings starting on page 7 are interesting. I enjoyed reading them.
- Even the negative results are interesting, e.g., "we find that the interaction number is positively correlated with interaction complexity" indicates that there remain complex SAE features.
Critical weaknesses
- The scale of the experiment is limited, i.e., the authors experiment with 21 features and 10 input variables per feature. Given the scale of today's models, I am not sure this is enough to draw generalizable observations.
- Some of the results (e.g., features with high PNCI score) suggest that interactions may not be a universally useful explainability tool.
Questions for authors
- The scale of the experiment is limited. In Appendix I, the authors state "We select a total of 21 features per SAE". Further, "we only compute interactions among at most the 10 input variables closest to the token position where the SAE feature is triggered". Given the scale of today's models, I am not sure this is enough to draw generalizable observations. I recommend the authors choose one LLM and expand the experiment for that particular model.
- Some of the results suggest that interactions may not be a universally useful explainability tool. For example, the authors report that multiple features have a high PNCI scores, which effectively makes interactions useless for those features. To counteract this observation, the authors should implement a controllability experiment to causally align features with their explanation. For example, the authors could steer features towards different semantics by aligning them with different interactions, and verify whether this change is reflected in the output.
Minor presentation issues:
- The introduction mentions "using interactions as a numerically verifiable metric", without defining what an "interaction" means. Given that the term is overloaded, the authors should define it earlier in the paper.
- "Conversely, encoding a large number of interactions indicates that the feature is likely activated by various phrase patterns representing diverse semantics." - Note that lexicalization != semantics. That is, the "various phrase patterns" may capture similar semantics.
- Typo: "explanationion"
7 comments
This is the second part of Question A.
Finding 2 still holds after expanding SAE features, i.e., the average number of interactions differs markedly across different SAE features, and there is no clear relationship between the average number of interactions and activation frequency.
| Metric | Before expanding features | After expanding features |
|---|---|---|
| Average number of interactions encoded by different features (mean ± std) | 6.90 ± 3.71 | 9.19 ± 4.95 |
| Pearson correlation coefficient between feature activation frequency and average number of interactions | −0.161 | −0.207 |
| Spearman correlation coefficient between feature activation frequency and average number of interactions | 0.067 | −0.047 |
As shown in the table, before and after expanding features, the average numbers of interactions across different features are 6.90 and 9.19, respectively, both with large standard deviations (3.71 and 4.95). This indicates that the average number of interactions differs markedly across different SAE features.
In addition, activation frequency refers to the proportion of tokens on which an SAE feature is activated. We find that, before and after expanding features, both the Pearson and Spearman correlation coefficients between the activation frequency of SAE features and the average interaction count are very low: the Pearson coefficients are −0.161 and −0.207 before and after expanding features, respectively, and the Spearman coefficients are 0.067 and −0.047. Therefore, we conclude that, both before and after expanding features, there is no clear relationship between the average number of interactions and activation frequency. In summary, Finding 2 still holds.
Finding 3 still holds after expanding SAE features, i.e., SAE feature activations encoding few interactions primarily model simpler interactions, whereas those encoding many interactions primarily model more complex interactions.
| Low-interaction-density group | Medium-interaction-density group | High-interaction-density group | |
|---|---|---|---|
| Proportion of low order interactions | 89.6%→86.1% | 76.7%→69.3% | 50.6%→44.7% |
Note: In each cell, the value on the left of the → is the result before expanding features, and the value on the right is the result after expanding features.
The table shows that, among all interactions the SAE features in the low-interaction-density group (i.e., those encoding just a few interactions), 89.6% interactions are low-order (simple) interactions before expanding features. The ratio slightly changes to 86.1% after expanding features.
On the other hand, among all interactions the SAE features in the high-interaction-density group (i.e., those encoding many interactions), 50.6% interactions are low-order (simple) interactions before expanding features. The ratio slightly changes to 44.7% after expanding features. In summary, Finding 3 still holds.
Finding 4 still holds after expanding SAE features, Regardless of whether we expand the experiments, we still find that, although high activation strengths are often regarded as strong semantic signals, a small number of strongly activated SAE features still model severe positive-negative cancellation interactions (i.e., their phrase patterns are highly mutually conflicting).
| Metric | Before expanding features | After expanding features |
|---|---|---|
| Pearson correlation coefficient between activation strength and PNCI | −0.044 | −0.001 |
| Spearman correlation coefficient between activation strength and PNCI | 0.052 | 0.087 |
This conclusion is supported by the fact that activation strength is almost unrelated to PNCI. Both the Pearson and Spearman correlation coefficients between activation strength and PNCI are very low: the Pearson coefficients are −0.044 and −0.001 before and after expanding features, respectively, and the Spearman coefficients are 0.052 and 0.087. In summary, Finding 4 still holds.
Overall, the scale of the experiments does not fundamentally affect the main findings of our work.
This is the first part of Question B.
Question B
Concerns about interactions with high PNCI values.
Weakness 2: Some of the results (e.g., features with high PNCI score) suggest that interactions may not be a universally useful explainability tool.
Quoted from question 2: The authors report that multiple features have a high PNCI scores, which effectively makes interactions useless for those features ... The authors should implement a controllability experiment to causally align features with their explanation ... The authors could steer features towards different semantics by aligning them with different interactions, and verify whether this change is reflected in the output...
Answer
First, interaction-based explanations have received increasing attention and have been widely used to explain various models [1-7]. According to Zhang et al. [8] and Ren et al. [9], a high PNCI value is a typical sign of overfitting, and almost all overfitted models exhibit high PNCI values across various applications [10]. Therefore, a high PNCI value does not indicate a failure of the interaction-based algorithm.
[1] Mingjie Li et al. Does a Neural Network Really Encode Symbolic Concepts? ICML 2023.
[2] Die Zhang et al. Building Interpretable Interaction Trees for Deep NLP Models. AAAI 2021.
[3] Wen Shen et al. Interpreting Representation Quality of DNNs for 3D Point Cloud Processing. NeurIPS 2021.
[4] Yingwei Ren et al. Lane-change risk prediction model based on traffic context and driving styles. Sci. Rep., 2025.
[5] Lu Chen et al. Can LLMs Reason Soundly in Law? Auditing Inference Patterns for Legal Judgment. ICLR 2026.
[6] Ruiyang Qin et al. Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions. ICML 2026.
[7] Hao Zhang et al. Interpreting and Boosting Dropout from a Game-Theoretic View. ICLR 2021.
[8] Junpeng Zhang et al. Two-Phase Dynamics of Interactions Explains the Starting Point of a DNN Learning Over-Fitted Features. arXiv:2405.10262, 2024.
[9] Qihan Ren et al. Towards the Dynamics of a DNN Learning Symbolic Interactions. NeurIPS 2024.
[10] Junpeng Zhang et al. Randomness of Low-Layer Parameters Determines Confusing Samples in Terms of Interaction Representations of a DNN. arXiv:2502.08625, 2025.
Second, unlike many explanation metrics, the existence of each interaction can be rigorously verified, regardless of whether their PNCI value is high or low. As stated in line 256 of the paper (Figure 2(a)), we have designed experiments to demonstrate that interactions indeed capture the actual patterns used by the SAE feature during inference. Specifically, given 1024 different masked states of an input sample, we use the following metric (see Equation (3) in the paper) to measure the fidelity of using the interactions to match the SAE feature activations across all 1024 masked inputs:
\left[ \frac{\left|g(\mathbf{x}') - v(\mathbf{x}')\right|} {\left|v(\mathbf{x})\right|} \right],$$ where $v(\mathbf{x}')$ is the true activation value of the SAE feature under the masked input $\mathbf{x}'\in\Psi$, and $g(\mathbf{x}')$ is the activation value fitted using only the interactions we extracted. The experiments are conducted on the SAE built upon Gemma-2-9B using the SQuAD dataset. Figure 2 in the paper shows that, for each input sample, the interactions can always accurately match the the SAE feature's activation values under all masked inputs (with an average matching error $\rho=0.024$ ), and we find that the number of significant interactions typically does not exceed 20. Therefore, we have reason to believe that the interactions reflect the actual patterns used by the SAE feature during inference, rather than an artifact of manual construction. **Third, and more importantly, we have added new experiments following your suggestion.** The purpose of these experiments is to demonstrate that, on samples with severe positive-negative cancellation (PNCI > 0.7), interactions can still faithfully explain the activation values of SAE features. To this end, on the SAE built upon Gemma-2-9B, we use interaction-based explanations to precisely manipulate the activation values of SAE features. We present two examples below.
This is the first part of Question A.
We thank you for your careful review and valuable suggestions. We have summarized the questions you raised into the following points and address them one by one.
Question A
Ask for scaling up the experiments to verify the generality of the conclusions.
Answer
It is a really good question. Following your suggestion, we have scaled up the experiments on the SAE built upon Gemma-2-9B and examined how the main conclusions of the paper change. Specifically, we reran the main experiments in the main text on approximately a thousand SAE feature activations, covering hundreds of thousands of distinct masked input samples in total, and updated the experimental results. All expanded experiments below use the same SAE built upon Gemma-2-9B and the dataset constructed in Appendix I.2 of the paper.
Finding 1 still holds after expanding SAE features, i.e., the number of interactions encoded by the same SAE feature still varies substantially across different positions.
Interaction number measures the number of phrase patterns that a feature uses during inference.
According to the rules in Line 315 of the paper, we categorize these SAE feature activations into three groups based on the number of interactions they encode: (1) the low-interaction-density group, (2) the medium-interaction-density group, and (3) the high-interaction-density group.
As shown in the table, the proportion of SAE features with low interaction density changes from 80.8% before expanding features to 75.4% after expanding features. This indicates that, both before and after the expansion, SAE features mainly encode relatively sparse interactions. In addition, before expanding features, statistics over all activations of SAE features at different token positions show that the average number of encoded interactions is 6.90, with a large standard deviation (11.55); after expanding features, the average number of encoded interactions is 9.19, and the standard deviation remains large (16.27). This indicates that the number of interactions encoded by the same SAE feature still varies substantially across different positions. In summary, Finding 1 still holds.