acceptodds
Under review as a conference paper at ICLR 2027

An Identifiability Hierarchy for Mechanistic Interpretability: Interpretability Claims Live at Different Levels of Identifiability

Abstract

Interpretability methods such as sparse autoencoders (SAEs), probes, and steering vectors name units inside a network and claim that those units are useful abstractions. Every such claim presupposes that the unit is identifiable from the activation. We introduce a five-level identifiability framework: three structural levels defined on the code (coefficient, support, and subspace identifiability) and two functional levels defined on the downstream model (behavioral and causal identifiability). We test it on SAEs with one question: is the code the encoder produces the only good sparse code for that activation? Running classical sparse coders on the same dictionary, we find alternatives that overlap with the encoder's code at Jaccard —roughly a third of each code's atoms are absent from the other—while reconstructing the activation as well and the model's predictions slightly better. Support identifiability fails while subspace and behavioral identifiability hold. The codes agree on the atoms with large coefficients and disagree on the many small ones, and the disagreement grows as the SAE is made less sparse, in a regime where the classical conditions for unique sparse recovery do not hold. Coefficient magnitude is the best predictor of how much the model's output changes when an atom is removed; for small atoms it loses significance, and agreement among sparse coders predicts the effect instead. Identifiability, on this view, is not a property an interpretability method has or lacks. It is a level at which a given claim holds—one that a method reaches on a given model and data—and the level at which that claim should be tested.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, identifiability, sparse coding, dictionary learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ChNVOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper explores the question of identifiability in SAEs, presenting the idea that identifiability should be viewed as a 5-level spectrum rather than a binary question. The paper then explores how 3 different SAE training algorithms compare against the existing gemma-scope SAE on gemma-2-2b, at different layers and sparsity levels, and compares and contrasts them along these identifiability axes.

Critical strengths
  • The abstract and intro of the paper are mostly written well and present the motivation and research question in a way that made me interested and convinced this is an interesting research question to study.

  • Although I am not super familiar with the literature on identifiability, the claim that identifiability should be viewed as a spectrum with different levels of granularity, rather than a binary question, is interesting and brings nuance, which I think has potential to advance our understanding of model internals and SAEs specifically.

Critical weaknesses
  • No related work section in the main paper. The intro refers to the appendix for a related work section which is very limited - it mentions one study plus some concurrent studies. The authors provide no positioning of this work within existing literature on identifiability of model components / neurons, or other interpretability methods other than SAEs, which is critical for understanding the paper and its contributions. This is a major weakness which significantly lowered my score.

  • Overclaiming on applications to many interpretability methods, when the paper only discusses SAEs. The title and intro of the paper make it seem like the identifiability hierarchy is applicable to any (mechanistic) interpretability method, while the paper discusses SAEs alone. I would avoid this overclaiming, and a discussion on how this can be further applied/extended can be written in the discussion section.

  • The structure of the paper has major weaknesses that take away from its readability:

    • The intro does not provide any actual details on the levels proposed: coefficient, support, subspace, behavioral and causal. That makes the list of contributions difficult to read, as it lists insights before I understand (at least at an intuitive level) what each level means.
    • Moreover, one of the contributions refers to the appendix - contributions should all fit in the main text.
    • Section 2.1 is very dense with definitions that are critical for the paper, I would take more space to explain them in detail. Specifically, the definition of “identifiable up to a class T”.
    • Figure 1 is unclear (and AI generated?) - What are OMP/IHT/FISTA? Why is the encoder an arrow while the decoder is a box? The texts that accompany each level are unclear, and the bottom one is cut off.
    • 3.1 lists four activation datasets but lists MMLU twice.
    • In section 2, OMP and FISTA are again used but not defined/referenced, it's unclear what they are and very difficult to read. I later found the definitions and references in section 3.2, the definitions should come before the first mention of these methods.
    • Section 2 does not give technical details on how each level is evaluated and measured, while section 3 refers to these non-existent definitions.
  • As far as I understand, Table 1 presents results for a single dataset (the pile), but 4 (or 3) datasets where mentioned in the experimental setup. There is no mention of the rest of the results in the text.

The writing of this paper is very difficult to follow, and major revisions are needed in order for it to be accepted (some are listed above). For these reasons I give this paper a score of clear reject.

Questions for authors

No Questions

1 comment

We thank you for your time and help, especially when it comes to the detailed list of structural and legibility issues. We are glad to see that the introduction convinced you that this is a question worth studying, and that viewing identifiability in this way provides useful nuance to understanding model internals.

Before addressing the main comments, we do wish to start with an important clarification. OMP, IHT, and FISTA which are explained in section 3 (with added details in the appendix) are not SAE training methods. These are classic algorithms on sparse inference that we execute over an already trained dictionary (GemmaScope) that we freeze. To reiterate we do not ever train any new SAE. Our question is on whether with the same dictionary and the same activation, there exist other sparse coders as good as the encoder.

Introduction: We do not provide the full definitions and operationalizing details of each level in the hierarchy because we dedicate the whole subsequent section to this.

Related work: While our paper already integrated a lot of related work in the prose of sections 2 and 3 meant to function as background needed for the reader in order to understand the experimental sections, we have added a new related work subsection in section 2. We also have expanded the related work section in the appendix.

Scope: While the main experimental evidence in the main body of text concerns SAEs, we do still maintain that the framework is generalizable beyond these.The definitions in section 2 and proposition 1 in the appendix apply to any sparse code from a dictionary, such an SAE, a sparse probe, or selection of characteristics. While in the main body of text we provide experiments and results on SAEs only, Appendix G shows that the same dissociation we observed for SAEs appears in sparse probes with an external label, and even on decision trees.

Algorithm definition: We have revised the text so that OMP, IHT, and FISTA are presented on Section 2 with a reference to Section 3 for a more complete explanation.

How each level is measured: We have added a table on section 2 in addition to our original figure 2 which should make it easier to quickly grasp what is the test we use for each level and the verdict on our own experimental results.

Figure: We have fixed the cut-off text. And yes this was made with the assistance of AI tools, this was already explicitly disclosed in the required AI usage section. As for the visual representation of the encoder and decoder: What you see is two branching arrows, one for the encoder and one for the alternative sparse coding method, the decoder is shown in just one form because in our experimental protocol we freeze it.

Table 1: Table 1 shows the PILE results. The results on FACTUAL, MMLU, and CODE, and on a second model are provided in appendix D and show the same pattern (Jaccard 0.44–0.58, top-10 0.74–0.83). We have added a footnote on this.

Thank you again for taking the time to review the paper and providing helpful feedback!

Reviewer_ADGROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a hierarchy of identifiability levels for interpretability claims: three structural ones (coefficient, support, subspace) and two functional ones (behavioral, causal). It then tests it on GemmaScope by running classical sparse coders. The experiments show that support identifiability fails while subspace and behavioral identifiability hold.

Critical strengths

S1. The main strength of the paper is the presentation of the identifiability of a model in a hierarchical way. This maps quite neatly into the fact that, in some cases, the arguments about identifiability are made at different levels in the literature, without necessarily making it explicit at which level (subspace, support, etc.) they are discussing it. The experiments in Table 1 seem to show that there is a gap there, and there is potential to do something with it.

Critical weaknesses

W1. The main weakness is that, after proposing a way to look at the identifiability problem, the paper does not really provide a clear "recipe" to decide at what level of the hierarchy the identifiability check should be made, or which one is the useful one for, e.g., a given task or claim. It's also not clear what happens if some claim is made at a level that is not the right one (beyond the illustrative examples), and what to do if it fails. Moreover, no threshold for these is given explicitly (e.g., "up to a tolerance set by the reconstruction-error floor" or in the conclusion "... the level at which that claim should be tested"). There is a remark about a "practical content" at l.434, but the conclusion is vague "the features whose identity is ambiguous are the ones for which cross-algorithm agreement is informative about causal effect", which raises more questions than answers: what is the objective level to check this?, what does ambiguous mean quantitatively?, what is "informative". In short, the paper is missing either: (1) a conclusion that says "the levels exist" (and then it should be clearly stated in a self-contained way), (2) "this is what to do about it".

W2. Linked to W1, the experimental section is very dense and hard to read, but, surprisingly, it does not contain much information. Most of the results are relegated to the appendix (everything past section 3 refers to results not in the main paper). There's also a bunch of statistics all over the section (Jaccard, top-10 Jaccard, AUROC, coefficient correlations), that one has to dig for in the appendix to understand their relevance to the main message. More importantly, the results are not delineated level by level (Table 1 mixes all of them). Overall, as in W1, the experiments don't help with providing a conclusion.

Questions for authors

Q1 (main issue). What are the concrete and clear takeaways from this paper? Specifically, for a given claim and tasks, how does the hierarchical identifiability framework help to decide at which level the check should be made, and what is the threshold for failures (or how can it be derived)?

Clarifications

  • The sparsity levels are given as they are in the experiments. How were they decided?
  • How is causal identifiability different from the behavioral one in the context of the hierarchical framework? In other words, how are the "corresponding interventions" decided?
2 comments

We thank you for your time and help, especially with respect to the question on how to make this hierarchy more legible for concrete tests/criteria, which led us to add a new table on our revised version. We are glad that you found the presentation of identifiability as a hierarchy useful.

Question 1:

We added a new table at the end of section 2. The table assigns to each interpretability claim what level it requires, the test that would verify it, and verdict.

  • Coefficient: Required by classification, thresholds, and steering coefficients.
  • Support: Required by claims such as “characteristic j is active in x” and this is what the catalog-and-naming workflow tends to do.
  • Subspace: This is sufficient for probing, that is claiming that the information is present.
  • Behavioral: Required to be able to claim tha the SAE gives a satisfactory explanation of the model.
  • Causal: Required by ablation and steering on a concrete atom.

We do not provide absolute thresholds, because the point is that this would be dependent on a model and it use. Each threshold is fixed with respect to a reference measure in the paper:

  • Subspace: The R² between codes (0.96-0.98), compared to the R² between the raw activations (0.34–0.48).
  • Behavioral: The difference in loss between codes in comparison to the gap in reconstruction from the encoder.
  • Support: The disagreement between our sparse coding algorithms with respect to the stability the encoder shows with noise. Changing the algorithm perturbs the support as much as noise of σ ≈ 0.6–0.7.

It is against these references in which can make verdicts on whether each level passes or not,

On the definition of ambiguity:

We have defined this quantitatively in the paper. A code is ambiguous when not all alternative methods conserve it (n_agree < 3)- The accordance is informative when its partial Spearman correlation with ablation, controlling for magnitude, is significative.

On a practical conclusion:

Our conclusion boils down to (i) a claim must be evaluated at the level it requires and in GemmaScope the claims on a support or coefficient level are not guaranteed even if probing or reconstruction can show good results; (ii) the problem is concentrated on low magnitude characteristics); (iii) Using a classic sparse coder, such as OMP, over the same dictionary is a cheap verification to establish these characteristics.

Sparsit levels: We use the levels that GemmaScope provides, these are not chosen by ourselves.

Behavioral vs Causal identifiability:

Identifiability at the behavioral level compares the outputs of the model without an intervention. The causal level compares between two codes under the same intervention applied. Since both codes share the dictionary, we can have an index, j, that designates the same direction in both, and the corresponding intervention would be the same operation over that coefficient (ablating coefficient j). Two codes can indeed be behaviorally identical, but causally distinct. For example, if the encoder explains a component with atom “7493” and OMP with atom “3331”, ablating 7493 changes the output only with the first code. That is why we evaluated causal identifiability separately on shared atoms (ρ = 0.92–0.93), and in exclusive ones (ρ = 0.57)

Density: We agree that the text does read as quite dense in many sections. We added a new table in section 2 that organizes results level by level and points to the section where each is tested. We will further revise the experimental section for a clearer narrative flow.

Thank you again for taking the time to review the paper and providing helpful feedback!

Thank you for your answer and the efforts . Unfortunately, I didn't have much time to look at the revised paper in detail alongside your answer so I cannot engage in a substantive way before the rebuttal period ends. I will keep your answer in mind for the AC discussion period.

Reviewer_r6qzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a five-level identifiability framework covering coefficient, support, subspace, behavioral, and causal identifiability. Using OMP, IHT, and FISTA on fixed GemmaScope SAE dictionaries, the authors find alternative codes with only about 0.5 support overlap with the encoder, yet comparable reconstruction and downstream loss. They conclude that support identifiability fails while subspace and behavioral identifiability hold, with the disagreement concentrated in a low-magnitude tail.

Critical strengths

The hierarchy usefully reframes identifiability as level-dependent rather than binary and anchors the catalog-and-name workflow at the support level. Causal experiments further show that agreement supplies residual signal precisely where magnitude loses predictive power.

Critical weaknesses
  1. The major finding was already reported in (Which Sparse Code? Identifiability Failures in SAE Inference), and the paper does not discuss (Mechanistic Interpretability Should Prioritize Feature Consistency in Sparse Autoencoders). Feature naming relies on high-magnitude activations, exactly where all methods agree, so the reported disagreement may not affect the naming workflow, and the causal evidence is also weak.
  2. The alternative methods are never compared against one another, so the disagreement cannot be separated from the encoder’s own amortization error. The alternative codes also ignore JumpReLU’s thresholds, and the tail atoms contribute less than the reconstruction error, so the disagreement may be an artifact of the setup.
Questions for authors
  1. What is the pairwise support Jaccard among OMP, IHT, and FISTA themselves? If these methods agree with each other substantially more than any of them agrees with the encoder, how do you distinguish “the activation lacks a unique sparse code” from “the encoder incurs an amortization error relative to a well-defined consensus code”?
  2. What fraction of the alternative codes’ exclusive atoms have coefficients below their JumpReLU thresholds θ_j, and does the support disagreement persist when z_j ≥ θ_j is enforced? In addition, how do the results change under a column-wise notion of agreement, such as the correlation of an atom’s activations across the dataset under the two codes, or the overlap of its top-activating examples?
1 comment

We thank you for your time and help, especially in relation to questions about the comparison between OMP, IHT, and FISTA, the JumpReLU threshold. We are glad you found it useful that our work usefully reframes identifiability as level-dependent and not as something binary, and that it anchors the catalog-and-name workflow at the support level.

Novelty with respect to Carbo & Nalisnick (2026): This work was presented at a non-archival workshop, establishing that the inference of an SAE is not unique. This paper, however adds four things (i) A formal hierarchy of identifiability levels with stated implications; (ii) The localization of the disagreement in the low-maginitude tail and its growth with sparsity level (Fig 2); (iii) An analysis of the local geometry of the support that shows that the exact recovery conditions fails in all activations (Section 4); And (iv) Section 5. adds evidence for causal consequences

Related Work: Thank you for the reference to Mechanistic Interpretability Should Prioritize Feature Consistency in Sparse Autoencoders, which we added to our related work section.. While this work studies consistency of features across training iterations, we focus on a single published dictionary and study inference, which is a source of non-uniqueness that would persist even if the training conditions were exactly reproducible.

The catalog-and-name workflow is based on high magnitude activations: This is a fair observation that we can also partially agree on. Our results do agree with this: the methods agree in higher magnitude atoms (Jaccard top 10 from 0.76 to 0.83). However, a label obtained from the examples that most activate a characteristic is also applied in all cases in which this one activates, including low activation. In those cases too, the code is ambiguous. The same occurs with uses that read all active characteristics from one entry, such as attribution graphs.

Causal evidence is weak: We recognize that the increase is modest, and we do say so in the text. What we want to highlight though is that in the tercile with least magnitude, the agreement across methods is still significant, while magnitude loses predictive power in layer 12.

Jaccard between OMP, IHT, and FISTA: This is an excellent question, and we can add a complete matrix for this. We also wish to point out that there is already evidence in the text for this hypothesis on a consensus code. IHT initialized from the encoder stays there (0.99 to 0.996), which is indicative of the encoder being in a stationary point, and not being a bad approximation from another code.

JumpReLU thresholds: This is an important control, and we agree it might have added another threshold that could appear arbitrary at first-sight to a reader. There are two halves to the disagreement: the atoms that the alternative method adds, and those that the encoder omits. All the atoms from the encoder do exceed the threshold from construction, so this objection would only affect the first half. Second, is that the replication on the LLama-Scope topK SAE which does not have these threshold, does still show the same pattern as the JumpReLU (Jaccard from 0.43 to 0.52, top-10 from 0.74-0.76; Appendix D)

Thank you again for taking the time to review the paper and providing helpful feedback!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.