acceptodds
Withdrawn from ICLR 2027

Same Features, Opposite Conclusions: Concept Assignment in SAE Evaluation for Mechanistic Interpretability

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Sparse autoencoders (SAEs) help researchers study the features learned by neural networks, but only a small subset of these features can usually be inspected. These features may correspond to concepts which are human-interpretable properties or categories. Choosing that subset of features requires a way to score features and evaluate the selections of concepts. We investigated whether the relative performance of two feature-selection methods (top-480 and uniform-480 varies depending on the concepts used for evaluation. Across 163 trained SAE checkpoints, we compared feature scores calculated using either the 480 most strongly activating examples or 480 randomly sampled active examples. We represented each feature using its decoder direction and measured how well it separated active from inactive examples. We assigned each selected feature the concept that maximized either binary F1 or absolute cosine similarity between its decoder direction and a known concept direction. We then evaluated concept detection on held-out examples using precision, recall, and F1 between feature and concept presence. Across matching-pursuit and standard SAEs, changing the concept assignment rule reversed which selection method achieved higher mean F1, even though the feature scores and selections were unchanged. The reversals recurred with fresh scoring, assignment, and evaluation data and new candidate pools, although statistical support was weaker for matching-pursuit SAEs under a common-seed analysis. The original test across 105 held-out checkpoints did not support the hypothesis that uniform sampling selects better concept detectors overall. In a separate experiment, training SAEs on data where concepts occurred independently greatly reduced disagreement between the assignment rules, even when the models were evaluated on the same data. These results show that comparisons between feature-selection methods can depend on how concepts are assigned to features. Although our experiments do not establish generalization beyond one synthetic generator or identify a universally better method, they show that changing the evaluation target can reverse conclusions while feature scores and selections remain fixed.

Primary area:
interpretability and explainable AI
Keywords:
Sparse autoencoders, mechanistic interpretability, feature selection, concept assignment, evaluation robustness, activation sampling.

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_j5wpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In mechanistic interpretability, one can identify some concepts in large neural models by learning sparse representation out the activation of their layers with e.g sparse auto-encoders, the concepts then being the columns of the SAE decoder. However, since the approach is unsupervised, linking each concept identified as is, requires a supplementary step of assignement of each vector to a (annotated) label. The current work is interested in this specific question of concept assignment.

The paper mainly consists in an empirical study of concept assignment. It is conducted with data generated with SynthSAEBench, considering three type of SAEs and Matching Pursui and comparing the effect of behavioral versus geometric assignment. The principle of the study is finally applied to the activation of one layer of a Gemma2-2B model. The results of this study has three main claims:

  • if one compares several approaches of concept extraction (several flavor of SAEs) and estimate their performances, changing the assignement step only can change the relative merit of the approaches.
  • a quantitative estimation of this "assignment effect" in terms of F-1 score, which is significantly larger than changing the seed to train the SAE
  • the code to audit a large neural model with the proposed approach
Critical strengths

[S1] as a researcher in mechanistic interpretability, I consider that concept assignment (and more generaly the quantitative evaluation) is a question that is not addressed enough in the field. This paper has the merit to directly do so and, in fact, it shows that the issue is of importance. The first claim of the paper empiricaly demontrates that many previous paper previously published may have seen their claims ("my SAE is better than previous one") invalidated if they had better considered this assignment step. At least, it shows that these papers are incomplete and their experiments are only true for the particular assignment step they adopted.

Critical weaknesses

[W1] the experiments are half convincing, considering that the work is mainly an empirical study

  • most experiment are conducted on the synthetic data of SynthSAEBench. While it has the advantage to be well controlable, it remains synthetic and thus raise the question of the generalization of the claims with real data and LLM. Even the authors of SynthSAEBench "emphasize that SynthSAEBench is intended to complement, not replace, LLM benchmarks. Our synthetic model cannot capture all aspects of real neural network representations (see Appendix A for limitations)." as stated in their section 8. And beyond the "synthetic" aspect of data, they also make arbitrary choice on the a priori distribution of concepts, that could be arguable in a more real cases (with LLM trained on real data).
  • the experiment (audit) on more real data is very limited, with only 8 concepts and a model that is quite "old" and of "limited size" on nowadays. Using Gemma2 in its 2B version has been often done in previous works, but in 2026 Gemma4 is available and larger models (8B, 27B...) may seem of interest for an empirical study
  • many hyperparameters are fixed arbitrary (e.g lines 107-130) and there is no study on the sensitivity of the claims to these choices
  • among hyperparameters that are fixed arbitrary, there is in particular the metric considered in the experiments, as well as the assignment approaches. These two are
  • in particular, the second claim (quantitative estimation of "4.66 F1 percentage points (...) compared with 2.91 points...") is only valid for the particular data and hyperparameter considered and has no value inan other context.
  • other said, all the claims are limited in scope due to the experimental part, which is problematic for a paper that is mainly empirical.

As well (although my assessment is not based on this point), the main paper refers to many tables and figures in the appendix to support the claims (e.g lines 222, 232, 245... 333...). It is a clear infringment to the limit page, in the sense that without this material the main paper does not hold w.r.t exeperimental results.

  • As well Supplement S1-S8 were supposed to be included in the appendix. It is true that the manuscript would have become much too large.
  • it is the responsability of author to select some ``minimal'' useful information to support their claims, not that of the reviewers to seek for such information in >50 pages (in particular when they have few times to review several papers). Other papers submitted to ICLR try to respect these rules, it is not fair to not do so.

Minor remarks (no need to address them during discussion):

  • line 107: the "scoring set SS" should be defined. It is done in the caption of figure 2 and ±\pm in the abstract/intro but it would be better to (formally set or) diretly explain it on line 107.
  • same remark for the "active subset PiP_i on line 123.
  • the settings to train the SAE, and their exact list, should be reported in the main text
  • the data used is described in section 2.2, but a minimal information should be reminded in section 3.1, 3.2, 3.3... to explain the protocol and which data ("original pane", "data C", 'data E"...) is used.
  • in the same vein, the appendix should refer to the sections they relate to e.g section G.1 refers to section 3.2 and so on.
  • in equation (2), the terms ΔA\Delta_A and ΔD\Delta_D are not really defined, not to mention their difference II which is not used in the definition of Δl\Delta_l. Last, the notation Δl\Delta_l is no longer used later in the main manuscript (which could help understanding) while Table 5 reports a simple Δ\Delta not defined (with the caption mentioning AfitA_{fit} and ArefitA_{refit} without definition).
Questions for authors

My assessment could be affected by a discussion on the limits regarding:

  • [Q1] the usage of synthetic data and limited number of concepts considered
  • [Q2] the sensitivity of the choice of the (many) hyperparameters,
    • [Q3] in particular the metrics and assignment approaches considered in the paper

Minor questions (that could not "materially affect [my] assessment"):

  • [MQ1] what means "32 used as sensitives" on line 135 ?
  • [MQ2] why is it reported on line 184 "as we did not publicly preregister the studies"? What does it justify?
  • [MQ3] on e.g line 220, what is the meaning of the range in brackets? It could be max/min or a (95%) confidence interval
Reviewer_tcdbOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

In my understanding, this paper aims to compare two different ways of ranking sparse autoencoder (SAE) latents to find those that match well with human interpretable concepts. They show that the examples used to compare these methods (i.e. by using top activating examples, or sampling examples uniformly) can change which ranking method appears to perform better.

Critical strengths

None.

Critical weaknesses

The writing of this paper is extremely hard to follow, and in my view a significant revision is required before it can be published. In its current state is it very hard to understand what is being done. Some concrete examples:

  • The abstract mentions the details of the work in significant detail. For example, top-480 and uniform-480. It is not possible understand (or requires a lot of effort to guess or infer) what this means, but more critically, an abstract is supposed to focus on the big picture.
  • The introduction dives into the specific problem in the first paragraph without providing context and without citation. SAEs do not have any citation, nor do claims like "Investigators therefore rank features...". Who are these investigators?
  • The second paragraph (L048-050) is already describing a result, when the setup is not yet described. Terms like "concept's grandparent" are unclear.
  • The third paragraph makes a claim, then immediately states "These comparisons are secondary: our prespecified primary hypothesis, that uniform sampling improves selection on 105 checkpoints from families absent from calibration, is unsupported.". What primary hypothesis? Where is this pre-specified? Which 105 checkpoints, and why those? Why this specific hypothesis?
  • At the end of the introduction, I still do not understand what the high level takeaways from this paper are supposed to be. My impression is that the introduction is a highly compressed version of the results and has no "story" of what the paper aims to do, making it very hard to read and understand.
  • Many design choices seem to be assumed, e.g. why 480 specifically for selecting top or uniform samples, for the evaluations?
  • The experimental section seems to provide a series of procedures and experiments followed, without clear motivation for how one led to the other.

Overall, independent of the paper's merits in terms of content, I believe that the writing is not in a state where it can be accepted.

Questions for authors

None.

Reviewer_qWnNOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates how concept-assignment rules affect SAE feature-selection benchmarks. Comparing behavioral F1 and decoder–concept cosine assignment, the authors demonstrate that changing the assigned concepts can reverse the ranking of top-activation and uniform-sampling selectors while keeping selected features fixed. The reversal is observed in certain synthetic SAE families, but its generalization to independently trained models and natural language models remains inconclusive.

Critical strengths

S1. Controlled experimental design. The authors isolate concept assignment while holding feature selections and evaluation metrics fixed. S2. Extensive empirical analysis. The paper includes multiple SAE architectures, alternative uncertainty estimates, retraining experiments, and a natural-model evaluation. Negative results and limitations are transparently reported.

Critical weaknesses

W1. Limited novelty. Both behavioral and geometric concept assignment are established approaches. The paper demonstrates that changing evaluation targets can change relative method rankings, which is conceptually expected. I find the empirical observation interesting, but the work lacks a sufficiently novel methodological or theoretical contribution. Moreover, related studies on non-canonical SAE features [1], compositional feature representations [2], and supervised concept alignment [3] are insufficiently discussed.

W2. Limited practical contribution. The paper does not introduce a new feature-selection algorithm, concept-assignment method, or evaluation metric. Its main recommendation is to specify evaluation targets beforehand and report results under multiple assignments. While reasonable, this provides limited actionable guidance for improving mechanistic interpretability. In particular, it remains unclear how practitioners should choose between competing assignments or determine which features are genuinely useful.

W3. Limited generalization. The main reversal is concentrated in specific synthetic SAE families. The original confirmatory hypothesis is unsupported, and the fresh-sample replication reuses the same trained models. Independent retraining provides inconclusive evidence, while the natural-model experiment favors top-activation selection under both assignment rules.

W4. Unclear mechanism. The authors attribute assignment disagreement partly to hierarchical concept structure, but the experiments do not isolate this mechanism from changes in concept distributions and reconstruction quality.

References [1] Leask et al. Sparse Autoencoders Do Not Find Canonical Units of Analysis. ICLR 2025. [2] Kurochkin et al. Kronecker Factorization Improves Efficiency and Interpretability of Sparse Autoencoders. 2025. [3] Yang et al. AlignSAE: Concept-Aligned Sparse Autoencoders. TMLR 2026.

Questions for authors

Q1. What is the fundamentally new insight beyond the expected observation that evaluating features against different assigned concepts changes their scores? Can the authors identify a concrete contribution that existing SAE evaluation literature does not already imply?

Q2. How should practitioners actually use these findings? Can the authors propose and validate a principled procedure for selecting concept-assignment rules or feature-selection methods that improves downstream interpretability?

Q3. Can the authors demonstrate that the reversal generalizes across independently trained SAE models, different data distributions, or additional natural-model benchmarks?

Q4. Does the choice of concept assignment predict downstream feature utility, such as causal interventions, steering, or probing performance? Such evidence would substantially strengthen the paper's practical relevance.

Comments on the paper

1 comment

We thank the three reviewers for their time and their careful, constructive reviews. We agree with the main concerns and feel that addressing them requires a substantial rewrite and additional experiments, which go beyond what a rebuttal can deliver. We are therefore withdrawing this submission. We will use the feedback to rewrite the paper and submit it to a later venue. Thank you again.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.