- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper deals with mechasnitic interpretability, which aims to identify concepts within the activations of a layer of a large neural model, interpret them and understand how they combine to produce the final output. This paper specificaly focus on this last point, more precisely by studying the composationality of two concepts.
Given a SAE applied to the activation of a large model, a part of the top-K features is set to zero while the other () is kept. The authors define the SAE additivity residual , which reflects the part of the network output that is not explained by an approximation retaininig only features on the considered top-K activations of the layer. The case reflect individual features and pairs of such features. More generaly, each feature of S can be considered with a certain probability. The authors establish several theoretical properties of SAR. In particular, they show that if one approximate the layer with feature in S, then SAR is an upper bound of the average error made by this approximation.
Experiment are conducted with Gemma-3 4B, Mistral-7B, and Llama-3.1 8B on 6 types of tasks. These last relates to (quite) simple syntaxic issues, but also to entity tracking with reasoning (indirect object identification), model knowledge extraction from counter facts and numerical comprison. They use SAE released by previous authors. In addition to experiments aiming at asserting directly the theoretical derivation, the author conduct an experiment with steering with Llama-3.1 8B, leading to significant better results with pairwise modelling.
Critical strengths
[S1] the core quantity proposed, SAE additivity residual , is a useful tool in the context of mechanistic interpretability to analyse the behavior concept extraction approaches such as SAEs, and compare them. It allows to quantify the effect of the interaction of several features/concepts identified by such approaches and is thus a tool which opens the way to more rigorous analysis of this question. The experiments support the claim of the importance of this problem, and it meets my opinion as a researcher in mechanistic interpretability. Moreover, although the article provide a solid theoretical justification, SAR can be estimated easily (with equation (21)).
Critical weaknesses
I consider there are no "critical weaknesses of the paper that support rejection", in the sense of the review instructions. However, the experimental settings could/should be better described in the main manuscript. Minor remarks and questions below are, for most of them, related to this issue. As well, a section specificaly dedicated to discuss the limitation, or doing so in section 5, woud have been appreciable.
Minor weaknesses (low/marginal influence on my assessment, but should be considered)
- [MQ1] section 4.1 should refer to a result (Table, figure...). From the main manuscript only, it is difficult to understand the experiment conducted. For exemple what are the "57 model/layer/bahavior" in section 4.1? 3 models, tested on 6 tasks give 18 results. Such results are reported in Figure 2, as well as in Table 10 and 11 in the appendix. In addition, Table 8 indicates that for the three "main models" (three last lines), 4+3+4=10 layers are tested, leading to potential 60 results for 6 tasks, thus still not 57. One guesses that three of them have been discarded but it thus should be explained. And the full results reported in appendix with a reference in the main manuscript. See [Q1] as well.
Minor remarks (suggestions: no need to answer each in the rebuttal, it does not affect my global assessment)
- the explanation in the caption of Fig. 1 is clearer than in the introduction line 040. I suggest to replace "we retain different subsets..." by "we retain or remove subets of activation while other activation content is fixed" (as in caption 1)
- while extensively used in the early mechanistic interpretability literature, the ``reconstruction quality'' is an intrinsic metric of SAEs (or similar) that has no interest in terms of interpretability, and should be forgotten. In my opinion, this metric was used "by default" and mainly by a lack of more appropriate ones, because it was a classical one in the sparse coding literature (wherer it is relevant) but it should eb replaced by metrics more related to semantics or, other said, interpretability.
- line 137 reports thus assumes the output of the SAE is the same as its imput (perfect reconstruction). Maybe, at the beggining of section 3.1, this assumption should be explicit, such that "the full coalition recovers th eoriginal activation" (line 137).
- at the beggining of section 3.1, i tis not obvious that the reader has read th ecaption of Figure 1 in details, thus and should be formaly introduced. In my opinion, this point is important.
- still without being a major issue, explaining the index of in equation (4) i.e "when replacing layer by the output of the SAE with a part of the top-K activation being put to zero" would be nice. It nevertheless can be understood in its current form, I (really) let this suggestion to the opinion of the authors.
- line 168: "thus r=1" --> "thus contains...".
Questions for authors
[Q1] for Table 10, which layer is/are used? Same as Table 11? In Table 10 the SAE for Gemma-3 4B simple copying seems significantly out of the 95% confidence interval for the same settings in Table 11: why? [Q2] in equation (6) should be explained. One guesses they are the parameters of that compute while is the same with parameters of layer restricted to (among top-K activation only). A couple of details on would be nice as well. I think it is important since the definition of SAR (and further developement) relies on it.
- same issue for this model and IOI, "subject verb agreement" for Mistral-7B and others... [Q3] Other minor questions:
- [MQ1] in equation (17) the expectation is for all ?
- [MQ2] which layer(s) of the three models are used in the experiment? one giuess they are mainly those of the caption of Figure 2?
Dear Reviewer yeVG,
We would like to respectfully offer a reminder and invite you to join the further discussion. We believe that most concerns have been reasonably addressed or substantially resolved. If you have a different view, we can give further clarification. Alternatively, we would also welcome you to raise the score or adjust the confidence.
We would also like to share important new experimental results that may change your view of the paper. To address your concern about practical utility, we extended our evaluation to five challenging benchmarks, including BBH, DROP, GSM8K, HotpotQA, and MATH 500.
We have also clarified the methodology in Section 3 and explained the different sampling protocols underlying the reported steering residuals.
Full experimental details will be included in the revision. We look forward to your reply and will be happy to answer any further questions.