- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes using non-gaussianity of the distribution produced by projecting residual-stream activations onto a direction as a signal for the direction’s interpretability. They show that this consistently identifies interpretable directions and suggest applying ICA to the residual stream in order to discover non-gaussian directions unsupervised. They demonstrate that a previous work which used ICA underestimated its interpretability due to sign ambiguity and once correcting for it, ICA produces more interpretable directions competitive with SAEs on multiple metrics.
Critical strengths
- The authors show that non-gaussianity is a strong signal for interpretable directions in activation space with a systematic evaluation including both human annotators and LLM based evaluation across 3 models. Their analysis goes beyond previously shown results for non-gaussianity [1] as a signal for interpretability, providing clear evidence of a correlation between the two.
- The utilized method (FastICA) is a simple and computationally efficient alternative to training SAEs for discovering a set of interpretable directions.
[1] Sparse Autoencoders Find Highly Interpretable Features in Language Models
Critical weaknesses
While the paper includes probe level interventions and some qualitative steering examples, the evaluations are mostly focused on showing that ICA directions are associated with interpretable concepts or contain concept-relevant information. While the TPP evaluation measures the effect of perturbations, it captures the effect on probes rather than directly on model behavior. In order to get a better sense of how well ICA holds in common featurizer applications, a more systematic causal evaluation such as MIB [2] or AxBench [3] would provide stronger evidence that ICA localizes features well. It would also provide a clearer comparison against commonly utilized featurizers and show what advantages the method has.
[2] MIB: A Mechanistic Interpretability Benchmark
[3] AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders
Questions for authors
- Figure 4, side B seems to contain only two curves: SAE and stronger-tail sign. Do you have the results for the arbitrary sign ICA as well? It would help make the claim that sign ambiguity was the problem. Similarly, do you have the results for the other models?
- How many annotators participated in the human eval and if multiple, could you report inter-annotator agreement ? It would help assess the reliability of the scores.
2 comments
Thank you for the detailed rebuttal! The authors made an effort to include additional causal results and clarify the missing details. Both of which address my concern. Accordingly, I have raised the score.
We thank the reviewer for recognizing our evidence across three models that non-Gaussianity is a useful signal for discovering interpretable directions, as well as the simplicity and computational efficiency of FastICA.
W. Causal evaluation on generated answers.
We added RAVEL [1] in Section 2.3.4, a task included in SAEBench [2] and MIB [3], to test whether ICA supports selective changes to model-generated answers. Following SAEBench’s RAVEL protocol, we learn a mask for each target attribute and replace the selected coordinates of a base entity with those of a source entity. For example, transferring Tokyo’s continent to Paris should change the continent answer from Europe to Asia while preserving France as the country. Cause measures successful transfer of the target attribute. Isolation measures preservation of other tested attributes. Disentangle averages the two.
On Gemma 2 2B layer 12, ICA achieves a Disentangle score of 0.7671, compared with 0.7208 for PCA and 0.6635 for the Gemma Scope 16K SAE, averaged over six city and Nobel laureate attribute tasks and two evaluation seeds. ICA also improves both mean Cause and mean Isolation over both baselines. We added the results to Section 2.3.4 and Figure 10, with the protocol in Appendix E.5.
[1] Huang et al. (2024). RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representations. [2] Karvonen et al. (2025). SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability. [3] Mueller et al. (2025). MIB: A Mechanistic Interpretability Benchmark.
Q1. Arbitrary-sign ICA in Figure 4B and results on other models.
We added the arbitrary-sign ICA curve for Qwen 3.5 9B Base to Figure 4B. Using the same fitted directions, stronger-tail orientation improves mean automated interpretability over arbitrary signs at all four evaluated layers.
We also compared stronger-tail ICA with the Gemma Scope 16K SAE at four Gemma 2 2B layers. Each method evaluates 150 features per layer.
Q2. Number of annotators and inter-annotator agreement.
The original submission reported ratings from one annotator. A second expert independently annotated the same 150 components using the same 0–5 rubric, during the rebuttal phrase. The second annotator was blinded to the original labels and ratings, the component statistics (kurtosis, skewness, and logcosh), and the component groupings.
Scores of 4 or 5 count as high confidence. The mean high-confidence rate across annotators is 85.7%, and both annotators rate 123/150 components (82.0%) as high confidence. Brackets give the 95% paired bootstrap confidence interval. The revised Appendix C.2 includes the protocol, both annotators’ labels and scores for all 150 components, and examples of disagreements.
We believe these additions address your concerns and questions.