acceptodds
Withdrawn from ICLR 2027

Shaped SAEs: Expanding the Toolkit for Mechanistic Interpretability

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Sparse autoencoders (SAEs) have become a widely used tool for decomposing the dense activations of large language models (LLMs) into sparse, potentially interpretable features using a learnt overcomplete dictionary. In this work, we introduce two complementary methods for shaping SAEs to enhance the semantic selectivity of the learnt dictionary atoms. The first, termed J-lens SAE, exploits the recently introduced Jacobian lens (J-lens), learning separate SAEs for the stronger and weaker eigenmodes of the averaged Jacobian between the current layer and the output layer. We expect the SAE learnt for the stronger J-lens modes to be more semantically selective, since they are more predictive of the LLM output. The second method, termed MAE-SAE, passes activations through a nonlinear masked autoencoder (MAE). The idea is that the MAE bottleneck layer will be forced to encode the relationships between the coordinates of the original token embedding, so that an SAE learnt for the MAE bottleneck layer should be more semantically selective than a standard SAE. We demonstrate that each shaping method outperforms standard SAEs on quantitative benchmarks such as SAE Bench, and discuss their qualitative attributes. Further experimentation is needed on how to combine these methods.

Primary area:
interpretability and explainable AI
Keywords:
Sparse autoencoders, mechanistic interpretability, large language models, Jacobian lens, masked autoencoder, semantic selectivity

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_A7ibOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how linearly shaping the dense activations before learning an SAE affects the semantic selectivity of the learnt dictionary atoms. It provides two approaches: (1) J-space shaping: project activation before SAE onto a "live" space and discard some low-gain quiet directions; (2) The Matched Filter SAE (MF-SAE): a TopK SAE without per-latent scales or biases, whose encoder and decoder share the same unit-norm atoms, wrapped between a learned affine encoder and decoder; encoder and decoder are thus tied up to a single shared linear map. These methods on Gemma-2-2B and GPT-2 Small are evaluated using only sparse-probing accuracy on SAEBench and SAE Probes, compared against dense activations and TopK SAE. The results show that MF-SAE outperforms the baselines on SAEBench when only a few latents are probed, while almost no advantage on SAE Probes. And J-space shaping adds little on top of MF-SAE cross two benchmarks.

Critical strengths

MF-SAE does show some real gains: when only probing a few latents, it consistently outperforms the available Gemma Scope and SAEBench TopK SAEs across both layers.

Critical weaknesses
  1. The motivation is relatively unclear: there are several changes to the original SAE, but there is no clear motivation for what specific problem these changes aim to solve. There is also no explanation of why these changes work, either intuitively or through theoretical proof.
  2. “Semantic selectivity” is undefined throughout the paper: the research question mentions “whether linearly shaping can enhance semantic selectivity,” but there is no definition of “semantic selectivity” nor a defined metric to measure it. Instead, the paper uses probe accuracy, which mainly measures whether a few latents can linearly separate predefined labeled concepts, rather than whether a latent responds selectively to a single concept. A more comprehensive set of metrics (e.g., aggregate autointerp detection scores) is needed to support the claim of improved semantic selectivity.
  3. The effectiveness of J-space shaping is unclear: adding a J-space projection on top of MF-SAE adds little gains. What is the relationship between the components? There is no dedicated analysis of this.
  4. Relevant work is missing: preprocessing activations before applying SAE has been explored in prior work, such as PCA whitening (Saraswatula, Ashwin, and David Klindt. "Data Whitening Improves Sparse Autoencoder Learning."). The paper neither cites these works nor compares against them.
  5. The paper is incomplete: some experiments are claimed but not included (e.g., PVE, alive-atom fraction and CE recovered in Sec. 5.3), and some references or links in the paper are missing.
Questions for authors
  1. Could you please propose a more clearly defined and more comprehensive metric to quantify “semantic selectivity”?
  2. Could you please improve the motivation and clarify what problem the proposed methods aims to solve?
  3. Could you please complete the related work and comparisons?
Reviewer_wJYGOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes shaping LLM activations before training an SAE, through J-space projection and a new Matched Filter SAE (MF-SAE). Experiments on Gemma-2-2B and GPT-2 Small show that MF-SAE outperforms existing SAEs on SAEBench sparse probing, but the gains on SAE Probes are marginal.

Critical strengths

None

Critical weaknesses

The paper does not look like a complete paper. There are many places showing “??”, where the citations or references are missing, and there are also quite a few typos. And the paper is not consistent about which GPT-2 model is used in the appendix. The introduction says GPT-2 Small at layers 7 and 8, but Section 5.1 says GPT-2 Large. Table 2 reports L0 ≈ 54 for Bloom’s SAE, but Appendix E says L0 ≈ 62. Similar inconsistencies appear in several other places in the paper. These inconsistencies raise doubts about the accuracy of the experimental results.

The paper studies only two models, and both of them are very very small. There are no models larger than 5B, not to mention models larger than 10B. Only two layers are used for each model, and there are only three baselines. I do not think this paper is ready for submission yet, and I hope the authors can further improve it or at least complete it before submitting.

The two main ideas of the paper, J-space shaping and MF-SAE, are largely independent. But the paper presents those two as one framework. The results show that most of the gains come from MF-SAE itself. Authors state that the raw and live MF-SAEs “perform similarly in most settings”, and the combination only helps at very tight feature budgets. The clear benefit of J-space shaping appears only when probing dense activations without an SAE. One possible reason is that MF-SAE already learns a linear transformation before encoding, which could learn a similar projection by itself. Also, J-space shaping is only tested with MF-SAE, not with standard SAEs such as TopK, JumpReLU and a lot of others, so it is unclear whether shaping is useful as a general tool.

Questions for authors

Please refer to Critical Weaknesses.

Reviewer_1fAROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes two more methods for training SAEs. One is to train on outputs of J-lens and the other is to train them to match a masked AE. The authors show improvement on some metrics of SAEbench as their main evaluation.

Critical strengths

None.

Critical weaknesses

The main weakness of this paper that the authors do not evaluate whether or not the improvements in in the performance of their proposed SAEs transfer to downstream utility such as causality or steering. most of the results are not very compelling to me as they don't show further utility of SAEs for interpretability.

There are also many legibility concerns, particularly that the images have been stretched such that the legends and text are completely unreadable, and there are many missing links in the paper.

Questions for authors

In the abstract you write "MAE bottleneck layer will be forced to encode the relationships between the coordinates of the original token embedding, so that an SAE learnt for the MAE bottleneck layer should be more semantically selective than a standard SAE". I I don't understand the intuition for why the MAE would be more semantically selective. Can you please explain this? in general, the motivation for use of the MAE is unclear to me.

Was the automated interpretability of your two SAEs greater than the automated interpretability of the baselines?

Were you able to reduce feature splitting and absorption? Can you please include results on all of the other aspects of SAE bench?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.