acceptodds
Under review as a conference paper at ICLR 2027

A Geometric View for Understanding Concept Learning and Neuron Interpretation in Sparse Autoencoders

Abstract

We propose a unified mathematical framework for a geometric understanding of concept learning and neuron interpretation in sparse autoencoders (SAEs). While SAEs improve interpretability of neural networks by learning sparse feature representations, a principled definition of ”concept” and ”learning” remains unclear. We formalize concepts as sets of data points and cast concept learning as a set-alignment problem between human-defined and model-induced concepts. This formulation distinguishes three increasingly strong notions of learning – detection, separation, and approximation – and yields geometric conditions, error bounds, and capacity constraints for when concepts can be represented by individual neurons or multi-neuron units. It also provides a set-theoretic account for common SAE phenomena, including feature splitting, feature absorption, feature families, and hierarchical concepts. Finally, we connect concept learning and neuron interpretation through formal concept analysis, showing that the two directions need not agree and that their many-to-many structure can be organized by concept lattices. Experiments on synthetic data with ReLU and Top-K SAEs illustrate the theory and reveal the effects of SAE size and sparsity on concept learning.

Primary area:
interpretability and explainable AI
Keywords:
Sparse Autoencoders, Mechanistic Interpretability, Concept Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_BiBAOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper formulates concept learning in SAEs as alignment between human-defined sets and neuron activation regions. It gives geometric conditions for representing concepts with individual neurons or neuron intersections, and uses concept lattices to organize their relations. Experiments study how architecture, width, and sparsity affect alignment.

Critical strengths
  • The set formulation makes the difference between concept learning and neuron interpretation precise. This is a good contribution to formalizing the interpreability discourse.
  • Some follow-up theoretical results are straightforwardly derived from this foundation. This shows that the formulation is good to build on.
Critical weaknesses
  1. Limited novelty. Most results follow from standard convex separation, polytope approximation, and subset counting. The formalization is useful, but I do not see a substantial new insight about SAEs or concept learning in general.
  2. The theory concerns only representability of concept encoders, not but not whether the concepts can be recovered by SAE training. The experiments also select neurons by individual F1 before intersecting them. This can miss jointly useful neurons, making it unclear whether poor alignment reflects geometry, training, or selection.
Questions for authors
  1. What prediction about trained SAEs follows from the framework beyond existing geometric accounts?
  2. How do results change with selection based on joint F1 over multiple neurons?
  3. Why are detection, separation, and approximation called increasingly strong notions when approximation need not imply detection or separation-error control (due to the fact the approximation use both a different error notion and is defined on a different domain?
  4. Please discuss [1] on human-machine concept mismatch. I think it has a similar spark to your idea, but fully fleshed out in set theory. Figure 1b is also too small to read.

[1] John Hewitt, Robert Geirhos, and Been Kim. We Can't Understand AI Using our Existing Vocabulary. arXiv:2502.07586, 2025.

11 comments
AuthorsOpenReview ↗

**Summary**

We thank the reviewer for the careful assessment and for pointing us to Hewitt et al. (2025). We are glad that the reviewer finds that the set-based formulation makes the difference between concept learning and neuron interpretation precise, and that it is a good foundation to build on. The two weaknesses concern what our framework adds beyond standard mathematics and how its representability results relate to trained SAEs. We first provide a summary of our response, followed by a more detailed response. All new results are in the revised paper, where new text is marked in blue.

Summary of our response

  • W1 (novelty). The tools are standard. Our contribution is what they show once concepts and SAE units are both formalized as sets of inputs. To address the reviewer’s point, we provide (1) the differences from prior work, (2) new findings on unseen data and novel concept discovery from concept approximation, (3) new findings from formal concept analysis (FCA) on how SAE concepts differ from human ones, including feature absorption and hierarchy, and (4) how our framework accounts for the recent findings in Cammarata (2026), which appeared after our submission.

  • W2 & Q1 (representability versus training and predictions about trained SAEs). We agree with the reviewer’s characterization of our theoretical scope. However, (1) the necessary conditions still hold for every trained SAE, (2) the theoretical constructions give a reference against which training can be measured, and (3) our framework provides guidance on how SAE training parameters should be selected.

  • W2 & Q2 (joint selection over multiple neurons). Exhaustive search, forward selection, and sparse regression lead to the same conclusions.

  • Q3 ("increasingly strong"). The notions are ordered by the size of the region on which the unit must be correct, C⊂X⊂RnC \subset X \subset \mathbb{R}^n. We will add a remark to clarify this point.

  • Q4 (Hewitt et al., 2025; Fig. 1b). We now cite and discuss the paper in the Introduction and Appendix A.2. Our framework can be used to describe the mismatch between human and machine concepts generally discussed in Hewitt et al. 2025. In addition, our framework provides a formal version of the discourse at the level of inputs. We will enlarge Fig. 1(b).

AuthorsOpenReview ↗

**W1. Novelty**

We agree that the proofs use standard tools. Our novelty lies in the formulation and in what these tools reveal once human concepts and SAE units are both formalized as sets of inputs. We show the novelty of our work in four parts. (1) Differences from prior work. Prior work studies directions, single features or decoder groups. We study which inputs a unit covers and excludes. This shows that units can separate concepts that no single feature can, and that some concepts cannot be separated at any SAE width. (2) New findings from concept approximation. SAE units exclude unseen data, yet combinations of existing features can still detect a new concept, such as a held-out class. (3) New findings from FCA. The concepts of a trained SAE differ from human concepts: the SAE builds its own hierarchy and compositional novel concepts, absorption is an overlap rather than a hierarchy, and the human hierarchy is reorganized instead of copied. (4) Broader observations with fewer assumptions. The framework accounts for later findings such as Cammarata (2026). Following Occam's razor, our framework makes fewer assumptions than prior definitions yet explains more phenomena, which illustrate the fit between set-based formulation and our work.

1. Differences from prior work (Appendix A.2).

  • Directions and manifolds (Park et al., 2024, 2025; Engels et al., 2025). These papers describe a feature as a direction or a manifold in activation space. A direction does not directly determine which inputs activate a feature. The bias and the gate do. We directly study the set of inputs that activate a feature by formalizing whether a unit of features covers the target and excludes non-targets.
  • Receptive fields of single features (Hindupur et al., 2026). This prior work relates SAE architecture to the geometry of a feature's receptive field. Rather than studying each receptive field in isolation, we additionally investigate what can be represented by their intersections. Our answer improves upon the single-feature case studied in prior work: one neuron needs Conv(C)∩Conv(DC)‾=∅\mathrm{Conv}(C) \cap \overline{\mathrm{Conv}(D_C)} = \varnothing (Theorem 4.2), whereas a unit needs only Conv(C)∩DC‾=∅\mathrm{Conv}(C) \cap \overline{D_C} = \varnothing (Theorem 4.4). Since DC⊆Conv(DC)D_C \subseteq \mathrm{Conv}(D_C), the condition for a unit is weaker, so units can separate concepts that no single feature can, such as a concept surrounded by other concepts (F1 rises from 0.405 with one ReLU neuron to 0.887 with seven; Fig. 2). Corollary 4.5 is also a novel result that an analysis of single features cannot give: some concepts cannot be separated by any finite ReLU unit at any dictionary width, for example a child concept that must be separated from the remainder of its parent. Larger dictionaries and feature splitting therefore cannot resolve such cases.
  • Decoder-side constructions (Rocchi et al., 2025; Bhalla et al., 2026). In this prior work, concept cones and decoder groups are built from dictionary directions and are assessed by agreement in activation direction or by reconstruction. In our paper, units are intersections of encoder membership regions, and reconstruction alone does not establish coverage of the target or exclusion of non-targets.
  • FCA on neural networks (Endres & Földiák, 2008; Hirth & Hanika, 2022; Xiong & Staab, 2025; Xiong, 2026). This prior work and our work use the same algebraic tools. However, this line of prior work applies FCA in settings other than SAEs: to the neural code of the monkey visual cortex (Endres & Földiák, 2008), to global explanations of image classifiers (Hirth & Hanika, 2022), and to the conceptual knowledge of language models, read from masked-token predictions (Xiong & Staab, 2025) or from linear attribute directions in embeddings (Xiong, 2026). None of these works studies SAEs or asks whether the units of a learned sparse dictionary can represent a given human concept. In contrast, we connect concept lattice to the geometry of SAE gates, showing which alignments are feasible, at what approximation error, and with how many neurons, and use it to relate concept learning to neuron interpretation, so that the mismatch between human and model concepts, and the phenomena and failures of SAEs, can be studied systematically.
  • Separation versus approximation. To our knowledge, the distinction between alignment on the data support XX and alignment in the ambient space Rn\mathbb{R}^n has not been made for SAEs. This theoretical distinction leads to the new empirical findings below.

2. New findings from concept approximation (new Appendix A.10.3–A.10.4, summarized in Section 5.4). Evaluating units in the ambient space, with an explicit weight on unseen points, gives results about trained SAEs that evaluation on the data alone does not give.

  • An SAE can detect a concept it never saw. With tench held out of SAE training, combinations of features learned from the other nine classes still detect it with F1 = 0.80, while the approximation error of the other classes changes by only −0.009 to +0.012.
  • Implications. (i) SAE units can exclude unseen data. With tench held out, tench is unseen data for the units of the other nine classes, and these units still exclude the tench class about as well as when tench is in training: their approximation error changes by only −0.009 to +0.012. The same holds for sampled unseen points for both synthetic datasets, ImageNette, and Yelp. This is the exclusion that our theory describes: each added neuron cuts away a further non-target region, on the data (Theorem 4.4) and in the ambient space (Theorem 4.6). (ii) SAE features can be combined to discover new concepts. The SAE never saw tench in training, but the unit built from neurons of the other classes detects it with F1 = 0.80 (0.74 with ReLU SAEs). Together, (i) and (ii) suggest that SAEs have the potential for compositional novel concept discovery: a new concept is excluded by the units of the seen concepts, yet can still be detected by a combination of existing features, with possible use cases in anomaly detection and in the discovery of novel scientific concepts.
Reviewer_gWGAOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper distinguishes three notions of concept learning: concept detection, concept separation, and concept approximation. For ReLU SAEs, it derives geometric conditions for concept separability by individual neurons and multi-neuron units, establishes approximation error bounds, and characterizes combinatorial constraints on concept learning capacity. The authors further employ Formal Concept Analysis (FCA) to organize the many-to-many relationships between human concepts and learned neuron activation patterns. Experiments on synthetic two-dimensional data, Imagenette image embeddings, and Yelp text embeddings investigate how SAE architecture, dictionary width, sparsity, and multi-neuron combinations affect concept alignment. The paper compares ReLU and Top-K SAEs and analyzes the concept structures induced by their activation patterns. Overall, the work aims to provide a principled mathematical foundation for understanding concept representability and neuron interpretation in SAEs.

Critical strengths
  1. A coherent and interpretable geometric framework for SAE concept learning. A major strength of the paper is its unified formulation of human concepts and SAE activations as subsets of the input space. This perspective provides a principled distinction among concept detection, separation, and approximation, clarifying several notions of interpretability that are often conflated in empirical studies. The geometric characterization of ReLU neurons as halfspace constraints, together with the interpretation of multi-neuron units as intersections of activation regions, is particularly useful. It provides a formal explanation for how individually polysemantic neurons can jointly form more selective concept representations, while also revealing inherent geometric limitations.

  2. A meaningful connection between concept learning, neuron interpretation, and empirical SAE behavior. The connection to Formal Concept Analysis provides an interesting structural perspective on the many-to-many relationships between concepts and SAE neurons. Rather than assuming that each neuron corresponds to a unique semantic concept, the framework explicitly accommodates feature sharing and hierarchical relationships. The experiments also provide informative observations about the different behaviors of ReLU and Top-K SAEs. In particular, combining multiple ReLU neurons can substantially improve concept alignment, whereas individual Top-K neurons may already exhibit relatively complex activation regions due to competitive gating. These observations support the broader argument that SAE interpretability depends not only on individual features but also on activation geometry and the organization of neuron combinations.

Critical weaknesses

Limited consideration of alternative explanations for polysemanticity and the necessity of SAE-based concept decomposition. A central motivation of this work is that individual neurons may be polysemantic and that combining SAE neurons can yield more selective and interpretable concept representations. However, recent findings by Cammarata (2026), Neural Networks Are Already Clean, suggest that many apparently polysemantic neurons may instead reflect coherent generalization or limitations of existing interpretability tools, rather than arbitrary superposition of unrelated concepts.This raises a fundamental question about the interpretation of the paper's results: does improved concept-level F1 through multi-neuron intersections genuinely indicate better mechanistic interpretability, or merely stronger alignment with predefined human labels? The paper does not sufficiently distinguish these possibilities, nor does it compare its SAE-based framework against direct interpretation of native model neurons or alternative, mechanism-oriented explanations. Consequently, the practical necessity and added interpretability value of the proposed framework remain insufficiently established.

Questions for authors

Q1. Correctness and completeness of Theorem 4.4 In Appendix A.6.3, the proof appears to rely on the compactness of the non-concept region, although boundedness alone does not guarantee compactness.Could the authors provide a corrected finite-cover argument and clarify the precise role of the closure of the non-concept region? In particular, does the theorem remain valid under the general assumptions stated in Assumption 4.1, where concept complements need not be closed? A rigorous clarification would substantially improve my confidence in the theoretical results.

Q2. Direct empirical validation of concept approximation Could the authors provide an experiment that directly evaluates the ambient-space approximation error defined in Section 3.3.3? For example, on synthetic data with known convex concept regions, it would be valuable to specify the ambient probability measure explicitly and investigate how approximation error changes with the number of selected neurons. Comparing the resulting behavior with the theoretical rate in Proposition 4.7 would help establish whether the geometric approximation results offer meaningful insight into trained SAE representations. This would be more directly relevant to the theory than relying solely on held-out-class F1 evaluations.

Q3. Robustness to neuron selection and random seeds How sensitive are the reported multi-neuron improvements to the heuristic of ranking neurons by their individual F1 scores?Could the authors compare this approach with greedy marginal-gain selection, beam search, or exhaustive search on sufficiently small instances? Additionally, could the authors report mean performance and standard deviations across random seeds and clarify whether seed selection, hyperparameter tuning, and neuron subset selection were performed exclusively using validation data?These additional analyses would help distinguish representational gains from selection-related effects.

Q4. Interpretability and Evaluation Validity Recent work, Neural Networks Are Already Clean, suggests that apparent polysemanticity may sometimes reflect coherent generalization rather than unrelated feature superposition. How does the proposed framework distinguish genuine improvements in interpretability from finer-grained alignment with predefined concept labels? Could the authors clarify whether the observed F1 improvements from multi-neuron combinations provide additional mechanistic insights rather than merely improved label discrimination?

10 comments

We thank the reviewer for the careful reading and the constructive questions, which helped us tighten the theory and include additional direct empirical support for it. We are glad that the reviewer found our geometric framework coherent, and its connection to neuron interpretation and empirical SAE behavior meaningful. The relationship between our framework and Cammarata (2026) is an important question about the motivation of our paper, and we address it together with the stated weakness under Q4. Below we provide a summary of our response and then answer the questions in order. All new results are in the revised paper, where new text is marked in blue.

Summary of our response

  • Weakness and Q4 (Cammarata 2026; interpretability versus label alignment). Cammarata’s work was published on Oct 15th, while our paper was submitted on Sept 25th, but our framework can already accommodate such a contemporaneous work. In our framework a concept is a set of inputs, so the activation set of a neuron is a concept in its own right and need not align with a human one. The formal concept analysis (FCA) is built on this premise and directly answers Q4. The FCA in our work is label-free and does not assume superposition, so a neuron that appears polysemantic under human labels can be a single coherent node of the lattice, which corresponds to the cat-and-car example in Cammarata (2026). In the same spirit as Cammarata's observation that a neuron's assigned label need not be its best explanation, FCA shows empirically whether the unit selected for a concept is better described by another concept: this happens for 83% (ReLU) and 80% (Top-K) of parent-state concepts, against 2.5% and 4.8% of child-state concepts, and in 70-74% of these cases the lattice predicts which concept the unit will be read as (Section 5.5). Separately, Cammarata (2026) argues that native neurons can be studied directly, without SAEs. Conceptually, our lattice needs only an activation set per unit, so the same construction applies to native neurons. In addition, Cammarata's "recursive generalization" corresponds to parent-child relations in the lattice (Appendix A.2).
  • Q1 (Theorem 4.4). The theorem holds under Assumption 4.1 as stated. Compactness is used only for the closure of the non-concept region, which is compact by the Heine-Borel theorem.
  • Q2 (concept approximation). New experiments with an explicit ambient measure confirm the rate and the irreducible error of Proposition 4.7 by direct construction. Trained SAEs follow the same behavior as the constructed solution and the theoretical rate. The experiments yield new findings on novel concept discovery.
  • Q3 (selection and seeds). The multi-neuron gains persist under exhaustive search, under greedy selection, and when averaged over 10 seeds.
AuthorsOpenReview ↗

**Q1. Correctness and completeness of Theorem 4.4**

By Assumption 4.1, X=⋃C∈CCX = \bigcup_{C \in \mathcal{C}} C is a finite union of compact sets, so XX is compact, i.e., closed and bounded in the Euclidean space Rn\mathbb{R}^n by the Heine-Borel theorem. Hence DC=X∖CD_C = X \setminus C is bounded, and its closure DC‾\overline{D_C} is closed and bounded. Since we work in the Euclidean space, DC‾\overline{D_C} is again compact by the Heine-Borel theorem. Therefore, Theorem 4.4 remains valid under Assumption 4.1. The finite-cover argument in Appendix A.6.3 is applied to DC‾\overline{D_C}, not to DCD_C, so it does not require the non-concept region to be closed. The closure is needed because a finite ReLU unit is an open set and cannot exclude non-concept points that accumulate at a point inside it.

We acknowledge that working in the Euclidean space means compactness is implied by boundedness and closedness, and that this step is subtle. We have included this remark in the main text and appendix accordingly.

AuthorsOpenReview ↗

**Q2. Direct empirical validation of concept approximation**

The error bound in Prop 4.7 is empirically verified on constructed hyperplanes and trained SAEs, including the error rate (the m−2/(r−1)m^{-2/(r-1)} term) and the irreducible error.

We ran the suggested experiment (Appendix A.10.3, Figs. 11-18 in the revised paper). The setup is:

  • Ambient measure. ν=(1−w)μ+wνunseen\nu = (1-w) \mu + w \nu_{\mathrm{unseen}}, where μ\mu is the empirical measure of the seen concept data and νunseen\nu_{\mathrm{unseen}} is the empirical measure of points sampled from the blank region Conv(X)∖X\mathrm{Conv}(X) \setminus X. The unseen-data weight ww is 0%, 20%, or 80%, and all errors are computed on a held-out test split.
  • Training data. Mutually disjoint convex concepts in rr = 2, 4, 8, and 16 dimensions to test the rate m−2/(r−1)m^{-2/(r-1)}, and the 2-D partially overlapping dataset to test the irreducible term ν(CΔK)\nu(C \Delta K).
  • Methods. "Constructed hyperplanes" verifies the existence result directly by building mm halfspaces around each concept, without any SAE (this). "Best trained SAE" selects mm neurons by individual F1 from trained SAEs, taking the best result over configurations. "Bound of Prop. 4.7" is ν(CΔK)+Am−2/(r−1)\nu(C \Delta K) + A m^{-2/(r-1)}.

Rate term of Prop. 4.7. The two tables below show unseen-data weights of 20% and 80% for ReLU SAEs (the construction of hyperplanes needs m≥r+1m \ge r+1, hence the dashes). Results for the remaining settings are in Fig. 11 and Fig. 12. The empirical findings are summarized as below.

  1. The empirical error of the constructed hyperplanes follows the theoretical error. The empirical error decreases with mm and stays below the bound.
  2. The empirical error of trained SAEs shows the same trend as the theoretical error. The empirical error drops as mm increases and increases as rr goes up, and it decays slower for larger rr.

Approximation error at unseen-data weight 20%, ReLU SAEs:

DimensionNumber of Selected Neurons12481632641282565121024
r = 2Best trained SAE0.1080.01770.005690.004320.004320.004320.00432----
Constructed hyperplanes--0.001360.0002810.0001250.0001028.59e-58.59e-58.59e-58.59e-57.81e-5
Bound of Prop. 4.7--0.002190.0006070.000210.0001118.64e-58.02e-57.86e-57.83e-57.82e-5
r = 4Best trained SAE0.05110.01970.007980.007390.007390.007390.00739----
Constructed hyperplanes---0.002410.00120.0006170.0003520.0002190.0001250.0001250.000109
Bound of Prop. 4.7---0.004150.002660.001720.001130.0007520.0005170.0003690.000276
r = 8Best trained SAE0.05130.0230.0110.01040.01030.01030.0103----
Constructed hyperplanes----0.003160.002050.001360.0008130.0007810.0005230.000383
Bound of Prop. 4.7----0.004590.003780.003110.002560.002110.001740.00144
r = 16Best trained SAE0.05060.02510.01150.01080.01070.01070.0107----
Constructed hyperplanes-----0.004160.002950.002210.001770.00130.00102
Bound of Prop. 4.7-----0.004940.004510.004120.003760.003430.00313
Reviewer_8gy6OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2(+1)
Presentation
2
Soundness
3
Read review
Paper summary

This paper contributes a conceptual framework for the relationship between concepts and SAE features as sets of inputs that activate neurons via set theory. It studies when these sets align, deriving geometric conditions for representing concepts with individual neurons or combinations of neurons. Small scale experiments use synthetic clusters, image embeddings, and review embeddings.

Critical strengths

This paper takes an interesting philosophical stance on what the idea of a concept is in a model -- essentially arguing that any collection of instances is a concept, regardless of whether we can coherently name it. thus, interpretability via SAEs essentially boils down to two main issues: (1) have we learned a feature/neuron for every concept we care about and (2) does autointerp work well enough to reconstruct a concept from its instances. While this is well known, I find the new framing, particularly after the review/rebuttal, to be interesting.

Critical weaknesses

Many prior papers in interpretability and dictionary learning have attempted to define concepts rigorously already. It still remains unclear what the proposed novelty is of this paper, or what the actual downstream utility is, given that the same two issues (how to train an SAE well and how to label it well) remain completely unresolved. This paper mainly just formalizes that those two problems exist, but they were already well-known empirical problems.

Questions for authors

n/a

9 comments

We thank the reviewer for the assessment and for asking directly what the takeaway of the paper is and how the framework should be used. We answer each point below. All new results are in the revised paper, where new text is marked in blue.

Summary of our response

  • W1 (prior work). Prior work defines a concept as a direction, a manifold, the receptive field of one feature, or a group of decoder atoms. We define it as a set of inputs, which yields results these definitions do not give, in particular concept approximation and the concept lattice.

  • W2 (main takeaway). An SAE is better read as a system of reusable activation constraints than as a one-to-one dictionary of human concepts.

  • W3 and Q1 (utility). (1) Concept approximation shows that an SAE can detect a concept it never saw by combining existing features. (2) FCA gives practical checks for SAE features.

  • Q2 (use). Researchers can also use the framework to state and compare new positions and observations, such as those in Hewitt et al. (2025) and Cammarata (2026). Notably, Cammarata (2026) appeared after our submission, but our framework can already be used to explain the observations made in Cammarata (2026).

Please refer to the reply to Reviewer gWGA and Reviewer BiBA for full quantitative details for each point.

AuthorsOpenReview ↗

**W1. Overlap with prior work on defining concepts rigorously**

We agree that prior work has defined concepts rigorously, and we do not claim to be the first. Our novelty lies in the formulation and in what it reveals once human concepts and SAE units are both treated as sets of inputs, the object through which SAE features are interpreted and validated in practice (Bills et al., 2023). The differences from prior work are as follows (Appendix A.2).

  • Directions and manifolds (Park et al., 2024, 2025; Engels et al., 2025). A direction does not determine which inputs activate a feature; the bias and the gate do. We study this set directly: whether a unit covers the target and excludes non-targets.
  • Receptive fields of single features (Hindupur et al., 2026). We characterize what intersections of neurons can represent, which improves upon the single-feature case: one neuron needs Conv(C)∩Conv(DC)‾=∅\mathrm{Conv}(C) \cap \overline{\mathrm{Conv}(D_C)} = \varnothing (Theorem 4.2), whereas a unit needs only Conv(C)∩DC‾=∅\mathrm{Conv}(C) \cap \overline{D_C} = \varnothing (Theorem 4.4). Units can therefore separate concepts that no single feature can. Corollary 4.5 is also a novel result: some concepts, such as a child concept that must be separated from the remainder of its parent, cannot be separated by any finite ReLU unit at any dictionary width.
  • Decoder-side constructions (Rocchi et al., 2025; Bhalla et al., 2026). Concept cones and decoder groups are assessed by directional agreement or reconstruction. Our units are intersections of encoder membership regions, and reconstruction alone does not establish coverage of the target or exclusion of non-targets.
  • FCA on neural networks (Endres & Földiák, 2008; Hirth & Hanika, 2022; Xiong & Staab, 2025; Xiong, 2026). These works apply FCA to the monkey visual cortex, to image classifiers, and to the conceptual knowledge of language models. None studies SAEs or asks whether the units of a sparse dictionary can represent a given human concept. We connect the lattice to the geometry of SAE gates and use it to relate concept learning to neuron interpretation, so that the mismatch between human and model concepts and the failures of SAEs can be studied systematically (Theorem 4.9).
  • Concept learning versus neuron interpretation. Prior work studies the two directions separately: concept-based analysis finds the neurons for a given concept (Bau et al., 2017; Gurnee et al., 2023), and automated interpretation finds the concept for a given neuron (Bills et al., 2023; Paulo et al., 2025). To our knowledge, no prior work relates the two formally or shows when they disagree. With FCA, we show that they are the two maps of a Galois connection and need not agree (Theorem 4.9), and the lattice tells when a unit selected for a concept will be read as another concept, and as which one (see Utility 2 under W3 and Q1).
  • Separation versus approximation. To our knowledge, the distinction between alignment on the data XX and in the ambient space Rn\mathbb{R}^n has not been made for SAEs (Theorem 4.6, Prop. 4.7). It leads to the findings in our answer to W3 and Q1.

Beyond these differences, our formulation is self-consistent both philosophically and mathematically. Philosophically, it takes the class-nominalist view that a concept is its extension, the set of inputs to which it applies (Ayonrinde & Jaburi, 2025). This view is intuitive and fits the epistemology of interpretability, which knows a concept through the examples that activate it. It also follows Occam's razor: it assumes neither that concepts are directions nor that neurons are in superposition, yet our framework can explain more phenomena, including polysemanticity, feature splitting, feature absorption, and hierarchy (Appendix A.4 and A.5), as well as the later observations of Cammarata (2026) (Q2). Mathematically, one primitive (a set of inputs) and a few set operations (inclusion, intersection, and symmetric difference) define the three modes of concept learning, multi-neuron units, and the concept lattice, and the results follow naturally from classical tools: convex separation (Theorems 4.2 and 4.4), polytope approximation (Prop. 4.7), subset counting (Theorem 4.8), and Galois connections (Theorem 4.9). To our knowledge, no prior formulation of concepts for SAEs has both properties.

AuthorsOpenReview ↗

**W2. Main takeaway**

An SAE is better read as a system of reusable activation constraints than as a one-to-one dictionary of human concepts (Section 5.6). The geometric conditions state when these constraints can represent a concept, which an SAE needs in order to learn it (Theorems 4.2–4.8). Before using an SAE, researchers can therefore estimate from the geometry of the data which concepts it can learn, and they can understand why an SAE fails to learn a concept, for example because no finite ReLU unit at any width separates a concept whose convex hull meets the rest of the data (Theorem 4.4).

FCA connects this geometry to human semantics by linking concept learning and neuron interpretation (Theorem 4.9). It lets researchers study the misalignment between human and SAE concepts systematically (Section 5.6), and it changes how known SAE phenomena and failures are understood. Feature absorption can be an overlap of two model concepts instead of a child taking over its parent, and as L0L_0 grows the same pair of neurons moves from absorption to nesting to polysemanticity. The hierarchy of an SAE follows the geometry of the data rather than the human defined labels, and it reorganizes the human hierarchy into a lattice in which a model-induced concept can have more than parent model-induced concepts (Sections 5.5 and 5.6). FCA also finds new concepts without labels through combinatorial expressivity: the meet of two neurons can make explicit a concept that neither neuron shows alone, this phenomenon of compositionally expressing novel concepts is also verified by concept approximation (Appendix A.10.4). The same formulation also accounts for observations beyond our experiments, such as Cammarata (2026), which appeared after our submission (Q2).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.