- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In mechanistic interpretability, one can identify some concepts in large neural models by learning sparse representation out the activation of their layers with e.g sparse auto-encoders, the concepts then being the columns of the SAE decoder. However, since the approach is unsupervised, linking each concept identified as is, requires a supplementary step of assignement of each vector to a (annotated) label. The current work is interested in this specific question of concept assignment.
The paper mainly consists in an empirical study of concept assignment. It is conducted with data generated with SynthSAEBench, considering three type of SAEs and Matching Pursui and comparing the effect of behavioral versus geometric assignment. The principle of the study is finally applied to the activation of one layer of a Gemma2-2B model. The results of this study has three main claims:
- if one compares several approaches of concept extraction (several flavor of SAEs) and estimate their performances, changing the assignement step only can change the relative merit of the approaches.
- a quantitative estimation of this "assignment effect" in terms of F-1 score, which is significantly larger than changing the seed to train the SAE
- the code to audit a large neural model with the proposed approach
Critical strengths
[S1] as a researcher in mechanistic interpretability, I consider that concept assignment (and more generaly the quantitative evaluation) is a question that is not addressed enough in the field. This paper has the merit to directly do so and, in fact, it shows that the issue is of importance. The first claim of the paper empiricaly demontrates that many previous paper previously published may have seen their claims ("my SAE is better than previous one") invalidated if they had better considered this assignment step. At least, it shows that these papers are incomplete and their experiments are only true for the particular assignment step they adopted.
Critical weaknesses
[W1] the experiments are half convincing, considering that the work is mainly an empirical study
- most experiment are conducted on the synthetic data of SynthSAEBench. While it has the advantage to be well controlable, it remains synthetic and thus raise the question of the generalization of the claims with real data and LLM. Even the authors of SynthSAEBench "emphasize that SynthSAEBench is intended to complement, not replace, LLM benchmarks. Our synthetic model cannot capture all aspects of real neural network representations (see Appendix A for limitations)." as stated in their section 8. And beyond the "synthetic" aspect of data, they also make arbitrary choice on the a priori distribution of concepts, that could be arguable in a more real cases (with LLM trained on real data).
- the experiment (audit) on more real data is very limited, with only 8 concepts and a model that is quite "old" and of "limited size" on nowadays. Using Gemma2 in its 2B version has been often done in previous works, but in 2026 Gemma4 is available and larger models (8B, 27B...) may seem of interest for an empirical study
- many hyperparameters are fixed arbitrary (e.g lines 107-130) and there is no study on the sensitivity of the claims to these choices
- among hyperparameters that are fixed arbitrary, there is in particular the metric considered in the experiments, as well as the assignment approaches. These two are
- in particular, the second claim (quantitative estimation of "4.66 F1 percentage points (...) compared with 2.91 points...") is only valid for the particular data and hyperparameter considered and has no value inan other context.
- other said, all the claims are limited in scope due to the experimental part, which is problematic for a paper that is mainly empirical.
As well (although my assessment is not based on this point), the main paper refers to many tables and figures in the appendix to support the claims (e.g lines 222, 232, 245... 333...). It is a clear infringment to the limit page, in the sense that without this material the main paper does not hold w.r.t exeperimental results.
- As well Supplement S1-S8 were supposed to be included in the appendix. It is true that the manuscript would have become much too large.
- it is the responsability of author to select some ``minimal'' useful information to support their claims, not that of the reviewers to seek for such information in >50 pages (in particular when they have few times to review several papers). Other papers submitted to ICLR try to respect these rules, it is not fair to not do so.
Minor remarks (no need to address them during discussion):
- line 107: the "scoring set " should be defined. It is done in the caption of figure 2 and in the abstract/intro but it would be better to (formally set or) diretly explain it on line 107.
- same remark for the "active subset on line 123.
- the settings to train the SAE, and their exact list, should be reported in the main text
- the data used is described in section 2.2, but a minimal information should be reminded in section 3.1, 3.2, 3.3... to explain the protocol and which data ("original pane", "data C", 'data E"...) is used.
- in the same vein, the appendix should refer to the sections they relate to e.g section G.1 refers to section 3.2 and so on.
- in equation (2), the terms and are not really defined, not to mention their difference which is not used in the definition of . Last, the notation is no longer used later in the main manuscript (which could help understanding) while Table 5 reports a simple not defined (with the caption mentioning and without definition).
Questions for authors
My assessment could be affected by a discussion on the limits regarding:
- [Q1] the usage of synthetic data and limited number of concepts considered
- [Q2] the sensitivity of the choice of the (many) hyperparameters,
- [Q3] in particular the metrics and assignment approaches considered in the paper
Minor questions (that could not "materially affect [my] assessment"):
- [MQ1] what means "32 used as sensitives" on line 135 ?
- [MQ2] why is it reported on line 184 "as we did not publicly preregister the studies"? What does it justify?
- [MQ3] on e.g line 220, what is the meaning of the range in brackets? It could be max/min or a (95%) confidence interval
We thank the three reviewers for their time and their careful, constructive reviews. We agree with the main concerns and feel that addressing them requires a substantial rewrite and additional experiments, which go beyond what a rebuttal can deliver. We are therefore withdrawing this submission. We will use the feedback to rewrite the paper and submit it to a later venue. Thank you again.