acceptodds
Under review as a conference paper at ICLR 2027

Non-Gaussianity Reveals Interpretable Directions in Language Models

33 views

Abstract

Finding human-interpretable directions in language-model activations is important for understanding and controlling model behavior. These directions are usually discovered through indirect learning objectives such as sparse reconstruction, but it remains unclear what existing statistical structure distinguishes interpretable directions and whether that structure can be used directly for feature discovery. We find that non-Gaussianity provides such a signature: projection distributions of natural-text residual-stream activations along sparse autoencoder (SAE) decoder directions are substantially more non-Gaussian than those along random directions, with most contexts forming a concentrated background and recurring patterns often occupying one tail. This raises a natural question: if interpretable directions tend to have non-Gaussian projection distributions, can we discover them by searching for non-Gaussian directions directly in activation space, without training another sparse dictionary? To test this hypothesis, we introduce ICA Lens, a practical workflow for efficient discovery, reliable annotation, and faithful evaluation of non-Gaussian directions in LLM residual-stream activations. Across GPT-2 Small, Gemma 2 2B, and Qwen 3.5 9B Base, we find that: (1) the recovered non-Gaussian directions capture coherent lexical, syntactic, and semantic patterns under both human and LLM-based evaluation; (2) non-Gaussianity is positively associated with both human-label confidence and automated interpretability; and (3) the resulting coordinates are competitive with SAEs across interpretability, sparse reconstruction, concept prediction, and targeted probe perturbation, while fitting in minutes per layer and using 7–43× less memory. These results show that non-Gaussianity is not merely a statistical property of learned features, but a direct unsupervised signal for discovering interpretable and useful structure in activations. We release our full implementation, including code to reproduce all experiments and the complete ICA Lens workflow, in the supplementary material.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, large language models, sparse autoencoders, representation analysis, feature discovery

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_DSqtOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes using non-gaussianity of the distribution produced by projecting residual-stream activations onto a direction as a signal for the direction’s interpretability. They show that this consistently identifies interpretable directions and suggest applying ICA to the residual stream in order to discover non-gaussian directions unsupervised. They demonstrate that a previous work which used ICA underestimated its interpretability due to sign ambiguity and once correcting for it, ICA produces more interpretable directions competitive with SAEs on multiple metrics.

Critical strengths
  • The authors show that non-gaussianity is a strong signal for interpretable directions in activation space with a systematic evaluation including both human annotators and LLM based evaluation across 3 models. Their analysis goes beyond previously shown results for non-gaussianity [1] as a signal for interpretability, providing clear evidence of a correlation between the two.
  • The utilized method (FastICA) is a simple and computationally efficient alternative to training SAEs for discovering a set of interpretable directions.

[1] Sparse Autoencoders Find Highly Interpretable Features in Language Models

Critical weaknesses

While the paper includes probe level interventions and some qualitative steering examples, the evaluations are mostly focused on showing that ICA directions are associated with interpretable concepts or contain concept-relevant information. While the TPP evaluation measures the effect of perturbations, it captures the effect on probes rather than directly on model behavior. In order to get a better sense of how well ICA holds in common featurizer applications, a more systematic causal evaluation such as MIB [2] or AxBench [3] would provide stronger evidence that ICA localizes features well. It would also provide a clearer comparison against commonly utilized featurizers and show what advantages the method has.

[2] MIB: A Mechanistic Interpretability Benchmark

[3] AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders

Questions for authors
  • Figure 4, side B seems to contain only two curves: SAE and stronger-tail sign. Do you have the results for the arbitrary sign ICA as well? It would help make the claim that sign ambiguity was the problem. Similarly, do you have the results for the other models?
  • How many annotators participated in the human eval and if multiple, could you report inter-annotator agreement ? It would help assess the reliability of the scores.
2 comments

We thank the reviewer for recognizing our evidence across three models that non-Gaussianity is a useful signal for discovering interpretable directions, as well as the simplicity and computational efficiency of FastICA.

W. Causal evaluation on generated answers.

We added RAVEL [1] in Section 2.3.4, a task included in SAEBench [2] and MIB [3], to test whether ICA supports selective changes to model-generated answers. Following SAEBench’s RAVEL protocol, we learn a mask for each target attribute and replace the selected coordinates of a base entity with those of a source entity. For example, transferring Tokyo’s continent to Paris should change the continent answer from Europe to Asia while preserving France as the country. Cause measures successful transfer of the target attribute. Isolation measures preservation of other tested attributes. Disentangle averages the two.

On Gemma 2 2B layer 12, ICA achieves a Disentangle score of 0.7671, compared with 0.7208 for PCA and 0.6635 for the Gemma Scope 16K SAE, averaged over six city and Nobel laureate attribute tasks and two evaluation seeds. ICA also improves both mean Cause and mean Isolation over both baselines. We added the results to Section 2.3.4 and Figure 10, with the protocol in Appendix E.5.

[1] Huang et al. (2024). RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representations. [2] Karvonen et al. (2025). SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability. [3] Mueller et al. (2025). MIB: A Mechanistic Interpretability Benchmark.

Q1. Arbitrary-sign ICA in Figure 4B and results on other models.

We added the arbitrary-sign ICA curve for Qwen 3.5 9B Base to Figure 4B. Using the same fitted directions, stronger-tail orientation improves mean automated interpretability over arbitrary signs at all four evaluated layers.

We also compared stronger-tail ICA with the Gemma Scope 16K SAE at four Gemma 2 2B layers. Each method evaluates 150 features per layer.

LayerICA (stronger-tail sign)SAE (Gemma Scope 16K)
50.4820.510
120.3670.397
180.5000.406
250.3530.350

Q2. Number of annotators and inter-annotator agreement.

The original submission reported ratings from one annotator. A second expert independently annotated the same 150 components using the same 0–5 rubric, during the rebuttal phrase. The second annotator was blinded to the original labels and ratings, the component statistics (kurtosis, skewness, and logcosh), and the component groupings.

MetricGPT-2 SmallGemma 2 2BQwen 3.5 9B BaseOverall
Components annotated505050150
High confidence, A141/50 (82%)42/50 (84%)40/50 (80%)123/150 (82.0%)
High confidence, A246/50 (92%)43/50 (86%)45/50 (90%)134/150 (89.3%)
Exact score agreement76%82%60%72.7%
Scores within one point90%100%90%93.3%
Agreement on high-confidence status90%98%90%92.7%
Quadratically weighted Cohen’s κ0.6670.9580.7020.772 [0.635, 0.874]

Scores of 4 or 5 count as high confidence. The mean high-confidence rate across annotators is 85.7%, and both annotators rate 123/150 components (82.0%) as high confidence. Brackets give the 95% paired bootstrap confidence interval. The revised Appendix C.2 includes the protocol, both annotators’ labels and scores for all 150 components, and examples of disagreements.

We believe these additions address your concerns and questions.

Thank you for the detailed rebuttal! The authors made an effort to include additional causal results and clarify the missing details. Both of which address my concern. Accordingly, I have raised the score.

Reviewer_zuSCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the problem of finding interpretable directions in language models, which has been addressed in the past with several methods including sparse autoencoders (SAE), principal component analysis (PCA) and independent component analysis (ICA). Through analysis experiments, they find that the projection distributions of these interpretable directions from above methods are more non-Gaussian than random directions. Hence, they propose to discover these directions by directly searching for non-Gaussian directions using FastICA instead of training sparse dictionaries like in SAEs. They also find a correction is required to the FastICA components to make sure the more interpretable directions are chosen. Through extensive experiments, they show comparable performance to SAEs while being more efficient and using less memory.

Critical strengths
  • The paper is quite interesting and generally easy to follow. It focuses on ICA as a faster initial interpretability analysis step compared to more expensive SAE training, which will be valuable to the community.

  • Evaluating the non-Gaussianity of SAE directions as well as PCA on LLM activations is interesting and seems to be novel.

Critical weaknesses
  • The comparison between ICA and SAEs does not seem completely fair to me. Like ICA can only discover (upto) dd components while SAEs can discover k×dk \times d (with kk as the SAE expansion factor). Then, even if ICA and SAEs have comparable average performance on different metrics, the total number of useful components found in SAEs will be (roughly) kk times more than in ICA. In that sense, ICA is kk times less useful than an SAE but I do concede, also kk times more efficient than an SAE (which the paper already does claim and I agree with it).
    • Now, there are also other problems with SAEs like feature absorption [W1] and that not all SAE features are useful [W2, W3] which weakens my argument. But I feel that the current evaluation is on 150 features randomly sampled with matched eligibility filters, but it does not quantify the overall/total interpretability of SAE or ICA, it only quantifies the average on a small set.

References

[W1] Chanin et al., "A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders", NeurIPS 2025

[W2] Arad et al., "SAEs Are Good for Steering – If You Select the Right Features", EMNLP 2025

[W3] Kulkarni et al., "Interpretable and Steerable Concept Bottleneck Sparse Autoencoders", CVPR 2026

Questions for authors
  • Please address the weaknesses section.

Minor concerns

  • The full term "Independent Component Analysis" is first used on Page 9 but the acronym ICA is used from Page 1 itself. I suggest mentioning the full term on Page 1 so it is clear what ICA is.
2 comments

Thank you for recognizing the novelty of our non-Gaussianity analysis and the practical value of ICA as a lightweight first step for interpretability.

W1. ICA vs. SAE feature budgets

Our paper includes the following empirical comparisons:

  1. In Appendix E.3, we have an experiment controlling for SAE dictionary size. We included compact Matryoshka SAEs with dictionary sizes of 128 and 512, both smaller than ICA's basis size. Under the same feature budgets, these smaller SAEs still do not outperform ICA in sparse probing. This suggests that simply reducing SAE dictionary size does not close the performance gap under the same budgets.
  2. We also evaluated TPP on Gemma 2 2B (layer 12) by ablating the same target fraction of each dictionary. At the 1% budget, ICA achieved higher TPP (0.20718 vs. 0.18823) with 23 components compared with 164 SAE features. At the 0.5% and 5% budgets, ICA scored within 0.02 TPP points of SAE. ICA can approach SAE's intervention performance using substantially fewer components.
Target dictionary fractionAblated features (ICA / SAE)Intended effect ↑ (ICA / SAE)Unintended effect ↓ (ICA / SAE)TPP ↑ (ICA / SAE)
0.1%2 / 160.02020 / 0.041900.00475 / 0.004600.01545 / 0.03730
0.5%12 / 820.11520 / 0.126000.01535 / 0.007950.09985 / 0.11805
1.0%23 / 1640.23390 / 0.197100.02673 / 0.008880.20718 / 0.18823
5.0%115 / 8190.44190 / 0.380000.08955 / 0.012530.35235 / 0.36748

We also believe that comparisons should consider practical interpretability scenarios:

  1. Matching by proportion would reward dictionary size rather than explanation quality. A larger dictionary would receive a larger absolute feature budget, making it difficult to compare how much useful information each feature captures.
  2. Human attention is limited. In real interpretability debugging, if an SAE needs 100 features to capture concepts that ICA can capture with just 5 components, it becomes much harder for researchers to track and understand those features and new activations. This is why we believe matching the absolute feature budget is meaningful for practical interpretability.
  3. Finally, we want to emphasize the intended role of ICA Lens. SAEs can provide more numerous and fine-grained features, and we do not claim that ICA covers the same number of unique concepts. ICA Lens is meant to be a lightweight first lens, helping researchers quickly identify promising layers and directions before investing substantial time and engineering effort into training additional SAEs for more detailed analysis.

W2. Number of evaluated features

The original SAE paper [1] also relied on randomly sampled features rather than evaluating the entire dictionary. We reproduced and followed its evaluation protocol with 150 sampled features.

Moreover, evaluating interpretability at scale is expensive. Even with random sampling, we spent over $1,500 in Tinker API costs on our experiments, and each human annotator spent several hours labeling components. Across our randomly sampled components, an average of 85.7% received high-confidence interpretations from the two annotators. We believe this provides meaningful evidence that ICA recovers interpretable components, although it does not establish the total number of distinct interpretable concepts.

We agree that a full-dictionary evaluation would be valuable, and we hope to collaborate on scaling up the annotation effort in future work.

[1] Huben et al. (2024). Sparse Autoencoders Find Highly Interpretable Features in Language Models.

Q2. ICA acronym

Thank you for catching this. We have added independent component analysis (ICA) at its first mention on Page 1.

Reviewer_zuSCOpenReview ↗

Reply to rebuttal

I thank the authors for their efforts in the rebuttal. I think my concerns are resolved and I keep my already positive rating.

Reviewer_piFbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates non-Gaussianity as a statistical signature of interpretable directions in language model activations and introduces ICA Lens, a lightweight framework based on FastICA. It also identifies and corrects a sign-related bias in previous ICA interpretability evaluations. Experiments across three language models show competitive performance against SAEs in interpretability, sparse reconstruction, sparse probing, and targeted probe perturbation, with substantially lower parameter memory. However, the independent contribution of non-Gaussianity, generation-time steering effectiveness, and comparison fairness require further investigation.

Critical strengths
  1. The paper presents comprehensive experiments cover GPT-2 Small, Gemma 2 2B, and Qwen 3.5 9B Base, combining human annotation, automated interpretability evaluation, and statistical analysis. Among 150 randomly sampled components, 123 receive high-confidence human labels. Sparse reconstruction, sparse probing, and targeted probe perturbation further assess practical utility beyond selected high-activation examples.
  2. The proposed method provides a lightweight alternative to SAEs. ICA fits a complete basis in approximately 12–13 minutes per layer for the 9B model and uses less parameter memory than matched public SAEs, while remaining competitive in sparse reconstruction and concept prediction.
  3. The analysis shows that both SAE decoder directions and concept-derived directions exhibit substantial non-Gaussianity, motivating direct feature discovery through ICA. Identifying and correcting the sign-sensitive evaluation bias is also a valuable methodological contribution.
Critical weaknesses
  1. The contribution of non-Gaussianity is not fully isolated. High kurtosis may also arise from rare tokens, positional markers, code formatting, or a few extreme outliers. The experiments do not sufficiently distinguish non-Gaussianity from correlated factors such as activation sparsity and concept frequency. The component-level Spearman correlation of ρ = 0.45 supports an association but does not resolve these potential confounds.
  2. Evidence for generation-time control remains limited. TPP measures changes in target and non-target probe accuracy after ablation, directly supporting selective intervention at the probe level. It does not establish that the model selectively uses the corresponding concepts during generation.
  3. Matched feature counts may not ensure comparable interventions. ICA produces exactly dd components, whereas SAEs use larger, overcomplete dictionaries. Although the comparisons match the top-k feature budget, equal feature counts need not imply equal activation sparsity or perturbation magnitude. Additional comparisons controlling for these factors would strengthen the conclusions.
  4. I suggest moving some example component interpretations from appendix D to Section 2.3.
Questions for authors
  1. Does the correlation between non-Gaussianity and interpretability remain after controlling for activation sparsity and concept frequency in the text?
  2. Can the author reports intervention success rates and effects on unrelated capabilities, beyond probe-level TPP results?
  3. Do ICA’s advantages over SAEs persist when comparisons match perturbation magnitude or actual activation sparsity, rather than only the number of selected features?
2 comments

Thank you for highlighting the breadth of our evaluation across three models, the efficiency of ICA Lens, and the importance of correcting the sign bias in prior evaluations.

W1. The contribution of non-Gaussianity is not fully isolated

Our claim is that non-Gaussianity is associated with interpretability and is useful for feature discovery, not that it causes interpretability. The toy example in Appendix A.1 illustrates how extreme observations can drive non-Gaussianity. In this example, related tokens form a coherent tail. Removing those tail observations would leave a background distribution closer to Gaussian. The positional markers and code-formatting patterns you mention can themselves be interpretable if they hold up under counterfactual testing.

W1/Q1. controlling the input data

In Appendix B.5, we added a controlled fitting experiment with 1K, 5K, 10K, 100K, and 1M token activations. We expect larger fitting datasets to contain more occurrences of recurring concepts. With 1K activations, median excess kurtosis reached 514.94 on the fitting tokens but only 0.53 on held-out documents. With 100K and 1M activations, held-out medians rose to 11.04 and 11.83. The results suggest that ICA needs enough fitting activations to recover directions whose non-Gaussianity persists on unseen documents. One explanation is that larger samples cover more concepts and provide more examples of each concept across contexts, helping ICA recover recurring activation patterns.

As in SAE-based feature discovery, we use activations from pretraining text to discover interpretable features. Our experiments show that ICA works well for feature discovery in this setting. This experiment provides an initial test of how the amount of input data/concept affects the recovered non-Gaussian signal. We believe that fine-grained control of activation sparsity and concept frequency is beyond the scope of this paper and deserves future research, we have add this to our future work.

To examine whether this relationship generalizes beyond fitted ICA components, we also analyze native MLP neurons, which do not require a separate feature-fitting procedure. In Appendix A.2, we added an exploratory analysis of native MLP neurons learned during pretraining. We find that excess kurtosis is positively associated with lexical concentration in their activation tails. This extends our analysis beyond ICA and suggests that non-Gaussianity may also help identify native MLP neurons worth interpreting.

W2/Q2. Evidence for generation-time control remains limited.

We added RAVEL [1] in Section 2.3.4, a task included in SAEBench [2] and MIB [3], to test whether ICA supports selective changes to model-generated answers. Following SAEBench’s RAVEL protocol, we learn a mask for each target attribute and replace the selected coordinates of a base entity with those of a source entity. For example, transferring Tokyo’s continent to Paris should change the continent answer from Europe to Asia while preserving France as the country. Cause measures successful transfer of the target attribute. Isolation measures preservation of other tested attributes. Disentangle averages the two.

On Gemma 2 2B layer 12, ICA achieves a Disentangle score of 0.7671, compared with 0.7208 for PCA and 0.6635 for the Gemma Scope 16K SAE, averaged over six city and Nobel laureate attribute tasks and two evaluation seeds. ICA also improves both mean Cause and mean Isolation over both baselines. We added the results to Section 2.3.4 and Figure 10, with the protocol in Appendix E.5.

[1] Huang et al. (2024). RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representations.

[2] Karvonen et al. (2025). SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability.

[3] Mueller et al. (2025). MIB: A Mechanistic Interpretability Benchmark.

W3/Q3. Matched feature counts may not ensure comparable interventions.

Our paper includes the following empirical comparisons:

  1. In Appendix E.3, we have an experiment controlling for SAE dictionary size. We included compact Matryoshka SAEs with dictionary sizes of 128 and 512, both smaller than ICA's basis size. Under the same feature budgets, these smaller SAEs still do not outperform ICA in sparse probing. This suggests that simply reducing SAE dictionary size does not close the performance gap under the same budgets.
  2. We also evaluated TPP on Gemma 2 2B (layer 12) by ablating the same target fraction of each dictionary. At the 1% budget, ICA achieved higher TPP (0.20718 vs. 0.18823) with 23 components compared with 164 SAE features. At the 0.5% and 5% budgets, ICA scored within 0.02 TPP points of SAE. ICA can approach SAE's intervention performance using substantially fewer components.
Target dictionary fractionAblated features (ICA / SAE)Intended effect ↑ (ICA / SAE)Unintended effect ↓ (ICA / SAE)TPP ↑ (ICA / SAE)
0.1%2 / 160.02020 / 0.041900.00475 / 0.004600.01545 / 0.03730
0.5%12 / 820.11520 / 0.126000.01535 / 0.007950.09985 / 0.11805
1.0%23 / 1640.23390 / 0.197100.02673 / 0.008880.20718 / 0.18823
5.0%115 / 8190.44190 / 0.380000.08955 / 0.012530.35235 / 0.36748

We also believe that comparisons should consider practical interpretability scenarios:

  1. Matching by proportion would reward dictionary size rather than explanation quality. A larger dictionary would receive a larger absolute feature budget, making it difficult to compare how much useful information each feature captures.
  2. Human attention is limited. In real interpretability debugging, if an SAE needs 100 features to capture concepts that ICA can capture with just 5 components, it becomes much harder for researchers to track and understand those features and new activations. This is why we believe matching the absolute feature budget is meaningful for practical interpretability.
  3. Finally, we want to emphasize the intended role of ICA Lens. SAEs can provide more numerous and fine-grained features, and we do not claim that ICA covers the same number of unique concepts. ICA Lens is meant to be a lightweight first lens, helping researchers quickly identify promising layers and directions before investing substantial time and engineering effort into training additional SAEs for more detailed analysis.

Comments on the paper

1 comment
Area_Chair_CK4yOpenReview ↗

Discussion

Dear All,

Thank you for reviewing the paper. Please read the authors' response and leave a comment accordingly.

Yours, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.