- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper unifies several concept-extraction methods used in mechanistic interpretability, including SAEs, transcoders, and crosscoders. By approximating the objectives optimized by these methods as biconvex problems, the authors theoretically demonstrate the existence of spurious partial minima, providing a theoretical explanation for the phenomenon of feature absorption.
Critical strengths
-
The paper tackles an important and timely question. Although recent empirical studies have highlighted limitations of sparse autoencoders, their theoretical analysis remains relatively under explored. By addressing this gap, the paper may contribute to the development of more principled approaches in a field that is still largely driven by empirical findings.
-
The paper is clearly organized and very easy to follow. The figures are well designed and helpful for illustrating the different scenarios and clarifying the paper’s main ideas.
Critical weaknesses
-
A very important conceptual issue concerns the paper’s definition of Sparse Dictionary Learning (SDL). The method described by the authors is an amortized version of SDL [1], rather than SDL itself. Definition 3.3 is currently inaccurate and imprecise. Standard sparse dictionary learning is formulated as a bilevel optimization problem, whose inner problem is constrained in the case of SAEs [2]. Since this paper aims to provide a theoretical analysis, the authors should refine this formulation and clearly distinguish between standard SDL and its amortized version.
-
Theorem 4.1 appears to be derived by dropping all terms corresponding to and putting them in the approximation bound. Thus, the analysis implicitly focuses on the case . This assumption should be stated explicitly rather than inferred from the proof. Related to the question raised below, I am also unsure whether the theoretical conclusions remain meaningful for larger sparsity levels. Even if a corresponding theory is not possible for larger , the authors should empirically evaluate whether remains a good approximation for higher sparsity.
-
The final contribution presented in the appendix (including the linear representation benchmark, feature anchoring, and the analysis of dead neurons) is not discussed in the main paper. These analyses and results should be brought into the main text as they are part of the contributions. I did not consider Appendices G and H in my evaluation because they are currently never referenced in the main paper and appear to introduce new analyses rather than supplementing the main results. Without these empirical results, the paper is currently weak, as it remains unclear whether the theoretical findings are applicable and scalable.
All together, these current issues prevent me from recommending the paper for acceptance in its current form. However, I consider this direction of developing a theoretical understanding of concept-extraction methods to be important. If the authors address the points above, I would be happy to increase my score.
[1] Charles O’Neill and David Klindt, “Compute Optimal Inference and Provable Amortisation Gap in Sparse Autoencoders”, ICML 2025.
[2] Sai Sumedh R. Hindupur, Ekdeep Singh Lubana, Thomas Fel, and Demba E. Ba, “Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry”, NeurIPS 2025.
Questions for authors
-
How large is the bound in Theorem 4.1 empirically? Can the resulting approximation be considered accurate when sparsity is typically set to at least approximately 20 in practice, which is far from the sparsity level of 1 implicitly assumed by the theory?
-
Assumption F.1 states that the concept activations are mutually independent. However, in a hierarchical structure, concept activations are generally dependent. Moreover, the implicit sparsity-1 assumption in Theorem 4.1 seems incompatible with hierarchical representations, where the sparsity is typically greater than or equal to 2. How do the authors justify this mismatch?
-
The bias term is omitted for mathematical simplicity. Could the authors provide intuition for how including a bias term would affect their theoretical findings?
-
How do different regularization choices fit into Equation (4) of Definition 3.3?
-
In Definition 4.2, the usual assumption is [3, 4]. Why should one assume that an active parent must always have an active child? This assumption does not appear to hold in the toy models considered in Matryoshka SAEs [3].
Minor Comments
-
The work appears to be related to instability in sparse autoencoders. If spurious minima exist, training with different random initializations might be expected to produce different solutions, but this connection is not discussed.
-
The notation is somewhat difficult to follow, with many variables of the form and . The authors might consider using for sparse codes and for the ground-truth dictionary.
-
Where is the factor used in Theorem 4.1?
-
There is an unresolved “??” in Appendix G.
[3] Bart Bussmann, Noa Nabeshima, Adam Karvonen, and Neel Nanda, “Learning Multi-Level Features with Matryoshka Sparse Autoencoders”, ICML 2025.
[4] Valérie Costa, Thomas Fel, Ekdeep Singh Lubana, Bahareh Tolooshams, and Demba Ba, “From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuit”, NeurIPS 2025.