- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper formulates concept learning in SAEs as alignment between human-defined sets and neuron activation regions. It gives geometric conditions for representing concepts with individual neurons or neuron intersections, and uses concept lattices to organize their relations. Experiments study how architecture, width, and sparsity affect alignment.
Critical strengths
- The set formulation makes the difference between concept learning and neuron interpretation precise. This is a good contribution to formalizing the interpreability discourse.
- Some follow-up theoretical results are straightforwardly derived from this foundation. This shows that the formulation is good to build on.
Critical weaknesses
- Limited novelty. Most results follow from standard convex separation, polytope approximation, and subset counting. The formalization is useful, but I do not see a substantial new insight about SAEs or concept learning in general.
- The theory concerns only representability of concept encoders, not but not whether the concepts can be recovered by SAE training. The experiments also select neurons by individual F1 before intersecting them. This can miss jointly useful neurons, making it unclear whether poor alignment reflects geometry, training, or selection.
Questions for authors
- What prediction about trained SAEs follows from the framework beyond existing geometric accounts?
- How do results change with selection based on joint F1 over multiple neurons?
- Why are detection, separation, and approximation called increasingly strong notions when approximation need not imply detection or separation-error control (due to the fact the approximation use both a different error notion and is defined on a different domain?
- Please discuss [1] on human-machine concept mismatch. I think it has a similar spark to your idea, but fully fleshed out in set theory. Figure 1b is also too small to read.
[1] John Hewitt, Robert Geirhos, and Been Kim. We Can't Understand AI Using our Existing Vocabulary. arXiv:2502.07586, 2025.
11 comments
**W1. Novelty**
We agree that the proofs use standard tools. Our novelty lies in the formulation and in what these tools reveal once human concepts and SAE units are both formalized as sets of inputs. We show the novelty of our work in four parts. (1) Differences from prior work. Prior work studies directions, single features or decoder groups. We study which inputs a unit covers and excludes. This shows that units can separate concepts that no single feature can, and that some concepts cannot be separated at any SAE width. (2) New findings from concept approximation. SAE units exclude unseen data, yet combinations of existing features can still detect a new concept, such as a held-out class. (3) New findings from FCA. The concepts of a trained SAE differ from human concepts: the SAE builds its own hierarchy and compositional novel concepts, absorption is an overlap rather than a hierarchy, and the human hierarchy is reorganized instead of copied. (4) Broader observations with fewer assumptions. The framework accounts for later findings such as Cammarata (2026). Following Occam's razor, our framework makes fewer assumptions than prior definitions yet explains more phenomena, which illustrate the fit between set-based formulation and our work.
1. Differences from prior work (Appendix A.2).
- Directions and manifolds (Park et al., 2024, 2025; Engels et al., 2025). These papers describe a feature as a direction or a manifold in activation space. A direction does not directly determine which inputs activate a feature. The bias and the gate do. We directly study the set of inputs that activate a feature by formalizing whether a unit of features covers the target and excludes non-targets.
- Receptive fields of single features (Hindupur et al., 2026). This prior work relates SAE architecture to the geometry of a feature's receptive field. Rather than studying each receptive field in isolation, we additionally investigate what can be represented by their intersections. Our answer improves upon the single-feature case studied in prior work: one neuron needs (Theorem 4.2), whereas a unit needs only (Theorem 4.4). Since , the condition for a unit is weaker, so units can separate concepts that no single feature can, such as a concept surrounded by other concepts (F1 rises from 0.405 with one ReLU neuron to 0.887 with seven; Fig. 2). Corollary 4.5 is also a novel result that an analysis of single features cannot give: some concepts cannot be separated by any finite ReLU unit at any dictionary width, for example a child concept that must be separated from the remainder of its parent. Larger dictionaries and feature splitting therefore cannot resolve such cases.
- Decoder-side constructions (Rocchi et al., 2025; Bhalla et al., 2026). In this prior work, concept cones and decoder groups are built from dictionary directions and are assessed by agreement in activation direction or by reconstruction. In our paper, units are intersections of encoder membership regions, and reconstruction alone does not establish coverage of the target or exclusion of non-targets.
- FCA on neural networks (Endres & Földiák, 2008; Hirth & Hanika, 2022; Xiong & Staab, 2025; Xiong, 2026). This prior work and our work use the same algebraic tools. However, this line of prior work applies FCA in settings other than SAEs: to the neural code of the monkey visual cortex (Endres & Földiák, 2008), to global explanations of image classifiers (Hirth & Hanika, 2022), and to the conceptual knowledge of language models, read from masked-token predictions (Xiong & Staab, 2025) or from linear attribute directions in embeddings (Xiong, 2026). None of these works studies SAEs or asks whether the units of a learned sparse dictionary can represent a given human concept. In contrast, we connect concept lattice to the geometry of SAE gates, showing which alignments are feasible, at what approximation error, and with how many neurons, and use it to relate concept learning to neuron interpretation, so that the mismatch between human and model concepts, and the phenomena and failures of SAEs, can be studied systematically.
- Separation versus approximation. To our knowledge, the distinction between alignment on the data support and alignment in the ambient space has not been made for SAEs. This theoretical distinction leads to the new empirical findings below.
2. New findings from concept approximation (new Appendix A.10.3–A.10.4, summarized in Section 5.4). Evaluating units in the ambient space, with an explicit weight on unseen points, gives results about trained SAEs that evaluation on the data alone does not give.
- An SAE can detect a concept it never saw. With tench held out of SAE training, combinations of features learned from the other nine classes still detect it with F1 = 0.80, while the approximation error of the other classes changes by only −0.009 to +0.012.
- Implications. (i) SAE units can exclude unseen data. With tench held out, tench is unseen data for the units of the other nine classes, and these units still exclude the tench class about as well as when tench is in training: their approximation error changes by only −0.009 to +0.012. The same holds for sampled unseen points for both synthetic datasets, ImageNette, and Yelp. This is the exclusion that our theory describes: each added neuron cuts away a further non-target region, on the data (Theorem 4.4) and in the ambient space (Theorem 4.6). (ii) SAE features can be combined to discover new concepts. The SAE never saw tench in training, but the unit built from neurons of the other classes detects it with F1 = 0.80 (0.74 with ReLU SAEs). Together, (i) and (ii) suggest that SAEs have the potential for compositional novel concept discovery: a new concept is excluded by the units of the seen concepts, yet can still be detected by a combination of existing features, with possible use cases in anomaly detection and in the discovery of novel scientific concepts.
**Summary**
We thank the reviewer for the careful assessment and for pointing us to Hewitt et al. (2025). We are glad that the reviewer finds that the set-based formulation makes the difference between concept learning and neuron interpretation precise, and that it is a good foundation to build on. The two weaknesses concern what our framework adds beyond standard mathematics and how its representability results relate to trained SAEs. We first provide a summary of our response, followed by a more detailed response. All new results are in the revised paper, where new text is marked in blue.
Summary of our response
W1 (novelty). The tools are standard. Our contribution is what they show once concepts and SAE units are both formalized as sets of inputs. To address the reviewer’s point, we provide (1) the differences from prior work, (2) new findings on unseen data and novel concept discovery from concept approximation, (3) new findings from formal concept analysis (FCA) on how SAE concepts differ from human ones, including feature absorption and hierarchy, and (4) how our framework accounts for the recent findings in Cammarata (2026), which appeared after our submission.
W2 & Q1 (representability versus training and predictions about trained SAEs). We agree with the reviewer’s characterization of our theoretical scope. However, (1) the necessary conditions still hold for every trained SAE, (2) the theoretical constructions give a reference against which training can be measured, and (3) our framework provides guidance on how SAE training parameters should be selected.
W2 & Q2 (joint selection over multiple neurons). Exhaustive search, forward selection, and sparse regression lead to the same conclusions.
Q3 ("increasingly strong"). The notions are ordered by the size of the region on which the unit must be correct, C⊂X⊂Rn. We will add a remark to clarify this point.
Q4 (Hewitt et al., 2025; Fig. 1b). We now cite and discuss the paper in the Introduction and Appendix A.2. Our framework can be used to describe the mismatch between human and machine concepts generally discussed in Hewitt et al. 2025. In addition, our framework provides a formal version of the discourse at the level of inputs. We will enlarge Fig. 1(b).