acceptodds
Under review as a conference paper at ICLR 2027

When Do Sparse Autoencoders Recover Features, and What Do They Learn Instead?

Abstract

Sparse autoencoders (SAEs) reconstruct activations as nonnegative sums of learned directions, and each latent's activation is read as the strength of one feature of the data. Training sees only reconstruction and sparsity, never the features, and learned latents are known to split, absorb and merge features and to change between runs. One explanation is limited inference: a one-layer encoder can provably fall short of the best sparse code chosen separately for each input, and more expressive encoders have been reported to recover such codes better and to yield more interpretable latents. We remove that limitation, building synthetic data whose features are identifiable and whose strengths a one-layer ReLU encoder computes exactly, and vary only how the strengths of co-occurring features depend on each other while directions, frequencies, marginal strength laws and covariance stay fixed. For independent exponential strengths, every sufficiently near-optimal affine-ReLU SAE of arbitrary finite width recovers each feature's additive contribution through fixed groups of latents as the penalty vanishes, whereas every such near-optimum retains nonvanishing contribution error when co-occurring strengths are often nearly equal. Adding a second encoder layer makes a cheaper non-recovering code available even under independence, while penalizing the number of active latents with one-layer responses restores individual recovery as its price vanishes. On a 16-dimensional synthetic source, SAEs trained on the coupled data reach a lower loss than any code on the generating directions. Recovery failures therefore need not come from limited inference, and when inference is already exact, a more expressive encoder can cause them.

Primary area:
interpretability and explainable AI
Keywords:
sparse autoencoders, interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NKabOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The authors conduct some intuitive case studies on sparse autoencoders, showing that there is a relationship between higher-order ground-truth feature magnitudes in the true data-generating process and the learned sparse codes. They generalize these case studies to some theoretical statements.

Critical strengths
  • The paper's claims are correct, as far as I can tell. There is indeed a connection between the co-occurence statistics of data-generating concepts and the recovered concepts in an SAE.
  • The conversational tone of the paper is nice when reading specific sections.
Critical weaknesses
  • The overall structure of the paper is fairly atypical. I would find it easier to parse a more conventionally structured paper (think: Intro/elated work/Theory/Experiments/Conclusions). The related work in the introduction would be best separated to its own section, in my opinion, to better understand the contrast. In my opinion, this is absolutely necessary in a research area as active as SAEs.
  • The main claim can be boiled down to: feature co-occurrence patterns (yes-no) are not the only aspect of importance when considering what SAEs learn relative to original representations, the higher-order co-activation statistics mattern, too. Unfortunately, this is not a new claim. For example, [1] says almost exactly this: "If co-occurring feature magnitudes vary relative to each other, we observe "partial absorption", where a latent tracking a main feature sometimes fires weakly instead of not firing at all, but sometimes does fully not fire." Additionally, [2] conducts experiments similar to the ones in this submission that manipulate these higher-order statistics to show that similar phenomena emerge. This submission cites a followup to [1] that unfortunately does not include the relevant commentary, likely simply an oversight. In addition there is substantial similarity to the empirical results of [2].
  • At first glance, oth Theorems 2 and 3 can be read as a fairly strong identifiability claim on ReLU SAEs, which is potentially of interest. However, I think this reading is perhaps too gracious. Neither theorem applies to the overcomplete setting, which unfortunately limit their applicability to SAEs. This is of particular note because I think Theorem 2 in particular says very little beyond typical statements about ℓ1\ell_1-penalized models. Theorem 2 as written applies to the 2-dimensional case in the main text; one has to turn to the appendix to see the generalization to multiple dimensions. Theorem 3 has no generalization beyond 2 dimensions, and again seems to be a standard statement about ℓ1\ell_1-penalized models.

[1] Chanin et al. "Toy Models of Feature Absorption in SAEs", LessWrong (2024).

[2] Anders et al. "Sparse Autoencoders Find Composed Features in Small Toy Models", LessWrong (2024).

Questions for authors
  1. Can you clarify whether Theorem 2 or 3, as written hold, in the overcomplete setting?
  2. Do your experiments extend to cases beyond [1]/[2] referenced above? Can you add these references?
Reviewer_9dhjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the problem of feature recovery in sparse autoencoders (SAEs). The authors demonstrate that dependencies between feature intensities can cause the SAE’s optimization objective to favor a non-ground-truth feature decomposition. Then analyzes how the depth of the encoder and the form of the sparsity penalty influence the feature recovery. Then validate findings through by synthetic experiments and experiments using SAEs trained on GPT-2.

Critical strengths

This paper constructs counterexamples where recovery fails and provides recovery guarantees and impossibility results applicable to SAEs of any finite width. The authors also demonstrate that enhancing the expressive power of the encoder can undermine feature recovery guarantees

Critical weaknesses

1: The main results rely on conditions like specific feature distributions, singleton events, and proximity to a global optimum.

2: The paper analyzes approximate global optima for the objective function; it does not prove that training can reach these solutions.

3: The paper does not provide a direct verification of the theoretical mechanisms behind the failure of feature recovery in real-world language models.

Questions for authors

Can the authors extend the recovery guarantees to more general feature distributions and overcomplete generating dictionaries, with stronger guarantees at finite sparsity penalties?

Reviewer_gHDGOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors question a common explanation to limitation of sparse autoencoders (SAE) “limited inference”: limited expression by using only 1 layer encoder. The authors do this by constructing simple synthetic dataset, and perform a theoretical framework to prove that the “limited inference” explanation is not the case. Instead, the authors explain the limitation by 4 cases which present the SAE to have perfect reconstruction but still choose wrong decomposition with higher L1. The authors provided 1 lemma and 2 theorems to justify this. Experiments and measurements are from several synthetic datasets generated depending on the settings.

Critical strengths
  • The authors focus on showing when “limited inference” is not the case, by constructing simple dataset and show try to solve that SAE still have recovery failure and try to explain why. This makes sense, but also leads to the first limitation.
Critical weaknesses
  • Limited inference can still occur when the dataset is sufficiently complex, which the paper does not address.
  • The paper works only with synthetic datasets, no noise, which are very simple and do not necessarily translate to real-world usage. The work reads more like a case study. As a result, the title may overclaim: the paper does not really establish when autoencoders recover features or what they learn in general.
  • There is no comparison with previous works, and no table reporting quantitative comparisons. The paper is really hard to read, and it feels largely AI-generated: Sentences are overly compressed (e.g., “Evaluations bear this out”). Some sentences lack a subject (e.g., Figure 1: “readable, not removable”). Word choice is sometimes poor. The organization is confusing. Section 2, which is supposed to describe the theoretical framework, also includes experimental setup details: “the generating quantities are retained only to evaluate the frozen model, to select hyperparameters on separate independent-strength populations, and to build ablation comparisons.”
  • Some simple ideas are explained in unnecessarily dense language, such as why the decoder d must be normalized (Eq. 5).
  • Almost every section heading is a question (2 “What,” 2.1 “What,” 2.2 “What,” 3 “What,” 3.1 “How,” 3.2 “What,” 3.3 “What,” 4 “Which,” 5 “What”). This does not read like standard academic writing.
  • There is a Theorem 2, but where is Theorem 1? In Equation (2), what are g and d? I assume g is the nonnegative activation of a code and d is a dictionary atom, but this is poorly defined.
Questions for authors
  • Why does theorem 2 only consider the specific set up of equation (6)? Does it apply to other cases/data? If it only applies to (6), then it is true that the applicability is very limited (data comes from 2 atoms) and this should only be a case study? Can the author discuss more about this for other cases/broader applications?
  • Same question for theorem 3, because 3 cites theorem 2.
  • The title of the paper “When Do Sparse Autoencoders Recover Features, and What Do They Learn Instead?”, so how can we answer these questions when in the more general case, and what do they learn?. In my opinion, the title is a bit overclaimed.
  • Is there any comparison to related works?
1 comment

Dear Authors,

Thank you for your paper, but I don't seem to find any rebuttal to my comments? At any rate, thank you for your effort in writing this paper. Much appreciated!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.