acceptodds
Under review as a conference paper at ICLR 2027

Sparse Autoencoders Do Not Guarantee Local Features

12 views

Abstract

Sparse autoencoders (SAEs) can recover sparse feature coordinates without identifying which inputs determine their activations. A feature associated with one input pattern may still depend on several controlled variables, and objectives based only on representation samples cannot distinguish input maps that induce the same representation distribution. We introduce the optimal input scope curve, which measures the least activation variance left unexplained when feature scopes are assigned under a shared dependency budget. For every interaction order , we construct systems with identical representation distributions and the same frozen SAE whose localization budgets differ by exactly a factor of at the same omission tolerance. This separation persists when an exact sparse coding objective uniquely selects the shared dictionary. We further derive compatibility and spectral lower bounds linking joint input effects, decoder directions, and required scope budgets, together with finite sample guarantees for paired intervention estimates after scope selection. In five learned overcomplete SAEs, two bijective input maps increase the mean budget needed to explain 90% of activation variance by 41% and 49%, despite leaving the representation and activation samples unchanged. These results show that sparsity and feature recovery do not by themselves establish which inputs determine a sparse feature.

Primary area:
interpretability and explainable AI
Keywords:
sparse autoencoders, mechanistic interpretability, feature localization, input dependence, sparse representations, functional ANOVA, intervention analysis

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_QePFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper argues that different relationships between inputs can produce the same internal model representations and, consequently, the same sparse codes learned by a sparse autoencoder (SAE), since the SAE observes only model activations. To address this limitation, the authors introduce the optimal input scope curve, which characterizes the trade-off between a feature-scope budget (the number of input variables a feature is allowed to depend on) and the activation variance that remains unexplained under that constraint. They show theoretically and empirically that, as the arity of input interactions increases, a larger scope budget is required to achieve the same approximation tolerance. Overall, the paper argues that evaluating SAE features requires examining not only the model’s internal activations, but also the relationship between those activations and the model’s inputs. For example, activation-based metrics alone cannot determine how localized or input-dependent a feature truly is.

Critical strengths
  • The paper raises an interesting point that, to my knowledge, has not been explored in the field: different relationships between inputs can produce the same internal model representations. This suggests that SAE evaluation should also consider the mapping from inputs to internal representations, rather than relying solely on activation-based metrics.

  • I value the authors’ theoretical approach, especially because much of the existing work in this area is primarily empirical. The theoretical analysis and carefully constructed toy setting clearly isolate the distinction between representation-level equivalence and input-level dependence, making the paper’s central argument precise.

Critical weaknesses
  • The paper shows that different input-to-representation mappings can produce the same distribution of internal activations, but it does not demonstrate that these mappings lead to different model behavior or materially different interpretations in realistic LLMs. In fact, I would be surprised if they did: for a fixed downstream network, the same activation distribution at a given layer should induce the same output distribution. If the model treats the inputs equivalently at the representation level, it is unclear why their input mappings should be distinguished for interpretability purposes. The authors should therefore establish that the proposed input-scope differences correspond to different input-conditional behavior or other practically meaningful distinctions in realistic LLMs.

  • The paper is difficult to follow: I had to read the abstract and introduction several times before understanding the main claim. It moves quickly into technical details without first giving a clear motivation or an intuitive example of why the proposed distinction matters, which is especially problematic given the practical-relevance concern raised above. In addition, the related-work discussion is somewhat limited for a paper proposing a new SAE evaluation metric. It would be helpful to explain more explicitly how input scope differs from or complements existing concepts and metrics, such as feature absorption and feature disentanglement in SAEBench, or how it could provide insight into phenomena such as feature instability for example.

Questions for authors
  • Can the authors provide evidence that the proposed input-scope differences occur in realistic LLMs, rather than only in the constructed and controlled settings?

  • If two input mappings produce the same activation distribution, what practical interpretability or behavioral consequence justifies treating them as different?

  • How does the input scope curve differ from or complement existing SAE evaluation methods beyond reconstruction and sparsity?

1 comment

Dear Reviewer QePF,

We would like to respectfully offer a reminder and invite you to join the further discussion. We believe that most concerns have been reasonably addressed or substantially resolved. If you have a different view, we can give further clarification. Alternatively, we would also welcome you to raise the score or adjust the confidence.

We would also like to share important new experimental results that may change your view of the paper. To address your central concern about practical interpretability and behavioral relevance, we conducted additional experiments connecting SAE input scopes to actual model output behavior.

  • Direct connection to model behavior. Using Gemma 3 1B IT and a pretrained SAE at layer 17, SAE derived scopes explained 97.93% of model output variance, compared with 95.43% for size matched random scopes. The improvement of 2.50 percentage points has a 95% bootstrap confidence interval of [2.29, 2.71].
  • Validation with pretrained language model SAEs. Across two language tasks and two SAE layers, interactions involving multiple input factors account for 53.37% to 62.63% of activation variance. The selected scopes explain approximately 89% to 90% of activation variance on held out language templates.
  • Practical interpretability significance. Our results demonstrate that input scopes identify which input factors and interactions contribute to SAE activations and downstream model responses, providing information beyond activation distributions alone.

We have also substantially revised the presentation and clarified how input scope complements existing SAE evaluation methods. Our responses have been reorganized to address your original questions directly.

Full experimental details are provided in our responses and revision. We look forward to your reply and will be happy to answer any further questions.

Reviewer_BpmsOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether sparse autoencoders (SAEs) guarantee that learned features depend on small subsets of input variables. It introduces an optimal input scope curve measuring unexplained activation variance under a dependency budget. The main theorem constructs systems with identical representation distributions and frozen SAEs but an exact factor-rr difference in localization requirements. Additional results establish sparse-coding uniqueness, spectral lower bounds, stability, and finite-sample estimation guarantees. Experiments with five learned SAEs show scope-budget increases of 41% and 49% under bijective input transformations.

Critical strengths

S1. Interesting experiment design. The authors use invertible XOR transformations to change input dependencies while preserving SAE activations and reconstruction quality. This provides a clean and controlled validation of their theoretical results.

S2. The compatibility and spectral bounds offer useful connections between input interactions, decoder subspaces, and localization requirements. The stability and finite-sample results are mathematically well motivated.

S3. The experiments carefully isolate input correspondence while preserving representation and SAE activation samples. Exact enumeration and paired-intervention estimation provide convincing controlled validation.

Critical weaknesses

W1. General limited novelty. The central motivation of this paper—that sparse SAE features can depend on multiple input tokens—appears well established and does not constitute a novel contribution. In transformer architectures, hidden representations at a given position can aggregate information from the entire preceding context through self-attention. Consequently, SAE features extracted from these representations can naturally depend on multiple tokens, regardless of activation sparsity.

This phenomenon has already been investigated in several works. Kissane et al. [1] identify short-range contextual, long-range contextual, and induction features using conventional attention-output SAEs. More directly, Lin et al. [2] explicitly quantify SAE feature nonlocality by measuring influence across input positions. Furthermore, Bhalla et al. [3] and Lubana et al. [4] investigate temporal dependencies in SAE representations and propose architectures designed to capture sequential semantic structure.

In my view, the question of whether SAE features exhibit nonlocal dependencies is already well studied.

W2. Limited novelty of the central impossibility observation. Lemma 1 follows directly from representation-distribution equivalence, closely related to established non-identifiability results in disentangled representation learning [5]. The main novelty lies in the quantitative scope formulation and lower bounds. This distinction should be emphasized more clearly.

W3. Limited empirical generalization. The experiments use only eight binary factors and deliberately constructed XOR transformations. Although the resulting changes in learned-feature dependence are nontrivial, the experiments do not establish whether comparable nonlocality occurs naturally in practical SAEs. Evaluation with meaningful input interventions would strengthen the practical implications.

W4. Weak spectral certificates. Theorem 3 produces empirical lower bounds of 4.2–20.6 against actual budgets of 136–203. Appendix J provides a stronger feature-order relaxation, but the representation-level bound remains too loose for practical estimation. Further analysis of the sources of looseness would clarify its usefulness.

[1] Kissane et al. (2024). Interpreting Attention Layer Outputs with Sparse Autoencoders. https://arxiv.org/abs/2406.17759

[2] Lin et al. (2026). Measuring Semantic Abstractness of SAE Features via Nonlocality. https://arxiv.org/abs/2608.10537

[3] Bhalla et al. (2026). Temporal Sparse Autoencoders: Leveraging the Sequential Nature of Language for Interpretability. https://arxiv.org/abs/2511.05541

[4] Lubana et al. (2026). Priors in Time: Missing Inductive Biases for Language Model Interpretability. https://arxiv.org/abs/2511.01836

[5] Locatello et al., Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations, ICML, 2019. https://proceedings.mlr.press/v97/locatello19a.html

Questions for authors

Please review the weaknesses, especially with the W1.

2 comments

Dear Reviewer Bpms,

We would like to respectfully offer a reminder and invite you to join the further discussion. We believe that most concerns have been reasonably addressed or substantially resolved. If you have a different view, we can give further clarification. Alternatively, we would also welcome you to raise the score or adjust the confidence.

We would also like to share important updates that may change your view of the paper. Following your suggestions, we have clarified our contributions relative to existing nonlocality research and conducted additional experiments with pretrained language model SAEs.

  • Quantitative contribution beyond existing nonlocality studies. Our optimal input scope curve measures the minimum input dependency budget required to explain fixed SAE features. Theorem 1 establishes an exact factor r separation in this budget under identical representation distributions and the same SAE, extending beyond the identification of nonlocal features.
  • Validation with pretrained SAEs. Experiments with Gemma 3 1B IT and Gemma Scope 2 SAEs show that higher order interactions account for 53.37% to 62.63% of activation variance. The selected scopes explain approximately 89% to 90% of activation variance on held out language templates.
  • Stronger localization certificates. The feature order relaxation substantially tightens the spectral lower bounds. At interaction arity 3, the lower bound increases from 12.6 to 145.0, compared with the exact budget of 192.4.

We have also expanded the related work and clarified the distinction between our quantitative framework and previous studies of SAE nonlocality.

Full experimental details are provided in our responses and revision. We look forward to your reply and will be happy to answer any further questions.

Thank you for the response. After reading it, I am keeping my score.

The distinction between “identifying” and “quantifying” nonlocality is not supported by the submission itself, which says Lin et al. already measured feature nonlocality. The rebuttal gives no comparison with that metric and no case where the scope curve reveals something positional-influence measures miss. The central finding follows from the transformer architecture itself. Attention means h already carries context from surrounding tokens, and SAEs see only h, so features can depend on several tokens regardless of sparsity (which is already clearly seen in many SAE interpretations, please see www.neuronpedia.org). The new Gemma experiments do not establish anything beyond this. Moreover, the conclusions are much weaker than those of the related papers cited in my review, which either use this property to improve existing methods or take it as a basic step to measure new properties. This paper does neither. It proposes a measurement framework but shows no demonstrated use for it. The novelty gap remains unaddressed.

Reviewer_gL9WOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

In AI interpretability, sparse autoencoders (SAEs) are used to identify features in models. One way of explaining a discovered feature is by relating it to the inputs that explain its activation variance. Identifying the best subset of inputs to explain a feature is something that has gone underexplored and leaves a gap in our knowledge about the quality of features SAEs can discover. The authors address this by defining the 'optimal scope' at some budget BB. This budget determines the number of input sets accessible to explain the variance the SAE features. The smaller BB required to reach some explained variance, the more local the SAE features. The authors show that even optimal dictionaries' locality can vary depending on the map of inputs to representations. The authors provide lower bounds for the budget under specified representations and reconstruction quality.

Critical strengths

The authors provide tight analytical bounds which are proved with sound mathematics. The authors make a clear conceptual distinction between two measures of SAE quality - reconstruction and input locality. They demonstrate that in a simple case with an optimal dictionary, the features learned need not be local to a small set of inputs. This points to a problem with SAEs discovering features which may be hard to interpret.

Critical weaknesses

The authors never cite prior literature that claims SAEs produce input-local features, so the position they are arguing against is not established. Much of the notation is hard to relate to a real setting. For example, "The variables xix_i are executable input choices, such as alternative tokens or controlled concept states." These variables are never instantiated as such and kept abstract. This brings me to the core weaknesses of this paper - nothing is tested on realistic SAEs at a scale which they are actually used. The architecture used in Theorem 1 is the identity map. The encoder zr(x)=Enc(hr(x))=ReLU(hr(x)+π1)=ReLU(qr(x))=qr(x)z^r(x) = Enc(h^r(x))=ReLU(h^r(x)+\pi \textbf{1})=ReLU(q^r(x)) = q^r(x), does nothing. The ReLU nonlinearity does not touch qr(x)q^r(x) since its values are nonnegative (eq. 10). The decoder D=ID=I with bias −π1-\pi\textbf{1}, giving us Dzr(x)−π1=qr(x)−π1=hr(x)Dz^r(x)-\pi \textbf{1}=q^r(x)-\pi \textbf{1}=h^r(x). This is akin to the example given is section F, which offers an affine transformation in the encoding step. The dictionary optimum used is trivial, and that dictionary is not overcomplete.

As a whole, the results do not seem to say anything about SAEs and only make weak points about the input map to hh, which is upstream of the SAE. Lemma 1 is a trivial statement. It would have been good to see the authors discuss nonlinear ICA identifiability and the lack of that literature lowers my confidence in the author's familiarity with this subject.

Questions for authors

What would this method look like if deployed at scale? Computing the optimal scope at fixed budget when the number of tokens or concepts becomes large is a hard problem because enumerating all possible candidate scopes grows exponentially. For nn input sets and mm features, here are 2n2^n possible scopes per feature and NN paired forward passes per feature, leading to 2n⋅m⋅N2^n\cdot m \cdot N forward passes. Do the authors have ideas for how to address this? Do the authors know how well they can estimate the optimal scope? The authors assume the probability distribution PP factorizes over the different input sets, which is not true when the inputs are tokens in language for instance. The representation distributions are very simple and not reflective of what we see when studying language models. How do the authors' findings translate to realistic representation distributions? Would the authors be able to test a small language model with an SAE trained over a large text corpus?

1 comment

Dear Reviewer gL9W,

We would like to respectfully offer a reminder and invite you to join the further discussion. We believe that most concerns have been reasonably addressed or substantially resolved. If you have a different view, we can give further clarification. Alternatively, we would also welcome you to raise the score or adjust the confidence.

We would also like to share important new experimental results that may change your view of the paper. To address your concerns about realistic SAEs, scalability, and estimation reliability, we extended our evaluation to learned overcomplete SAEs and pretrained language model SAEs.

  • Validation with learned overcomplete SAEs. Across five independently trained SAEs, two bijective input mappings increased the mean localization budget from 136.0 to 192.4 and 202.8, while preserving identical representation and SAE activation samples.
  • Evidence from pretrained language models. Experiments with Gemma 3 1B IT and Gemma Scope 2 SAEs show that interactions involving multiple input factors account for 53.37% to 62.63% of activation variance. The selected scopes explain approximately 89% to 90% of activation variance on held out language templates.
  • Reliable scope estimation. Increasing paired samples from 128 to 2,048 reduced scope curve estimation error from 0.0472 to 0.0109 and selection regret from 0.00996 to 0.00028.

We have also clarified the theoretical contributions, their relationship to nonlinear ICA, and practical approaches for evaluating candidate scopes.

Full experimental details are provided in our responses and revision. We look forward to your reply and will be happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.