acceptodds
Under review as a conference paper at ICLR 2027

Behavioral Compositionality of Sparse Autoencoder Representations

13 views

Abstract

Sparse autoencoders (SAEs) represent model activations with sparse, linearly decoded features, but the downstream effects of those features need not compose additively. We introduce SAE Additivity Residuals (SAR), which measure the behavioral variation left unexplained above a chosen interaction order. Under independent feature subset sampling, population SAR is exactly the normalized energy of interactions above that order and bounds coalition ordering error for the optimal low order approximation. To distinguish compositionality associated with the SAE representation from that induced by downstream computation, we compare learned SAE coordinates with sum preserving random rotations of the same selected subspace and find lower first order residuals for the learned coordinates across all 18 model and behavior settings. The coordinates are nevertheless not fully additive: pairwise terms improve held out coalition ordering in every setting and outperform parameter matched models using randomly selected third order terms. In a combined steering experiment, pairwise modeling raises rank fidelity from 0.5874 to 0.7790 and reduces ordering error from 0.1444 to 0.0606. Together, these results show that SAE coordinates support comparatively compositional behavioral descriptions while faithful joint explanations and interventions still require interactions beyond individual feature effects.

Primary area:
interpretability and explainable AI
Keywords:
sparse autoencoders, mechanistic interpretability, behavioral compositionality, feature interactions, model steering, representation analysis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_S3DeOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper deals with mechasnitic interpretability, which aims to identify concepts within the activations of a layer of a large neural model, interpret them and understand how they combine to produce the final output. This paper specificaly focus on this last point, more precisely by studying the composationality of two concepts.

Given a SAE applied to the activation of a large model, a part of the top-K features is set to zero while the other (SS) is kept. The authors define the SAE additivity residual SARrSAR_r, which reflects the part of the network output that is not explained by an approximation retaininig only rr features on the considered top-K activations of the layer. The case r=1r=1 reflect individual features and r=2r=2 pairs of such features. More generaly, each feature of S can be considered with a certain probability. The authors establish several theoretical properties of SAR. In particular, they show that if one approximate the layer with rr feature in S, then SAR is an upper bound of the average error made by this approximation.

Experiment are conducted with Gemma-3 4B, Mistral-7B, and Llama-3.1 8B on 6 types of tasks. These last relates to (quite) simple syntaxic issues, but also to entity tracking with reasoning (indirect object identification), model knowledge extraction from counter facts and numerical comprison. They use SAE released by previous authors. In addition to experiments aiming at asserting directly the theoretical derivation, the author conduct an experiment with steering with Llama-3.1 8B, leading to significant better results with pairwise modelling.

Critical strengths

[S1] the core quantity proposed, SAE additivity residual SARrSAR_r, is a useful tool in the context of mechanistic interpretability to analyse the behavior concept extraction approaches such as SAEs, and compare them. It allows to quantify the effect of the interaction of several features/concepts identified by such approaches and is thus a tool which opens the way to more rigorous analysis of this question. The experiments support the claim of the importance of this problem, and it meets my opinion as a researcher in mechanistic interpretability. Moreover, although the article provide a solid theoretical justification, SAR can be estimated easily (with equation (21)).

Critical weaknesses

I consider there are no "critical weaknesses of the paper that support rejection", in the sense of the review instructions. However, the experimental settings could/should be better described in the main manuscript. Minor remarks and questions below are, for most of them, related to this issue. As well, a section specificaly dedicated to discuss the limitation, or doing so in section 5, woud have been appreciable.

Minor weaknesses (low/marginal influence on my assessment, but should be considered)

  • [MQ1] section 4.1 should refer to a result (Table, figure...). From the main manuscript only, it is difficult to understand the experiment conducted. For exemple what are the "57 model/layer/bahavior" in section 4.1? 3 models, tested on 6 tasks give 18 results. Such results are reported in Figure 2, as well as in Table 10 and 11 in the appendix. In addition, Table 8 indicates that for the three "main models" (three last lines), 4+3+4=10 layers are tested, leading to potential 60 results for 6 tasks, thus still not 57. One guesses that three of them have been discarded but it thus should be explained. And the full results reported in appendix with a reference in the main manuscript. See [Q1] as well.

Minor remarks (suggestions: no need to answer each in the rebuttal, it does not affect my global assessment)

  • the explanation in the caption of Fig. 1 is clearer than in the introduction line 040. I suggest to replace "we retain different subsets..." by "we retain or remove subets of activation while other activation content is fixed" (as in caption 1)
  • while extensively used in the early mechanistic interpretability literature, the ``reconstruction quality'' is an intrinsic metric of SAEs (or similar) that has no interest in terms of interpretability, and should be forgotten. In my opinion, this metric was used "by default" and mainly by a lack of more appropriate ones, because it was a classical one in the sparse coding literature (wherer it is relevant) but it should eb replaced by metrics more related to semantics or, other said, interpretability.
  • line 137 reports h~x(Nx)=hx\tilde{h}_x(N_x)=h_x thus assumes the output of the SAE is the same as its imput (perfect reconstruction). Maybe, at the beggining of section 3.1, this assumption should be explicit, such that "the full coalition recovers th eoriginal activation" (line 137).
  • at the beggining of section 3.1, i tis not obvious that the reader has read th ecaption of Figure 1 in details, thus gxg_x and ϕ\phi should be formaly introduced. In my opinion, this point is important.
  • still without being a major issue, explaining the index of FF in equation (4) i.e "when replacing layer ll by the output of the SAE with a part of the top-K activation being put to zero" would be nice. It nevertheless can be understood in its current form, I (really) let this suggestion to the opinion of the authors.
  • line 168: "thus r=1" --> "thus Ar\mathcal{A}_r contains...".
Questions for authors

[Q1] for Table 10, which layer is/are used? Same as Table 11? In Table 10 the SAE SAR^1\hat{SAR}_1 for Gemma-3 4B simple copying seems significantly out of the 95% confidence interval for the same settings in Table 11: why? [Q2] in equation (6) θ\theta should be explained. One guesses they are the parameters of FF that compute gxg_x while θB\theta_B is the same with parameters of layer ll restricted to BB (among top-K activation only). A couple of details on A\mathcal{A} would be nice as well. I think it is important since the definition of SAR (and further developement) relies on it.

  • same issue for this model and IOI, "subject verb agreement" for Mistral-7B and others... [Q3] Other minor questions:
  • [MQ1] in equation (17) the expectation is for all (S,T)∼μ\timeμ(S,T)\sim \mu\time\mu ?
  • [MQ2] which layer(s) of the three models are used in the experiment? one giuess they are mainly those of the caption of Figure 2?
Reviewer_Mk7uOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This work evaluates the compositionality of SAE features for LLMs.

The Authors randomly subtract a subset of the top-K SAE features from the LLM’s activations at layer ll, and then measure a scalar behaviour ψ\psi that is dependent on the current task. For a fixed input xx, they collect observations for random feature subsets in order to form training and validation sets. A linear probe is trained to predict the value of the scalar behaviour from indicators of feature subsets of rank at most rr. The reported SAE Additivity Residual of rank rr (SAR^r\widehat{\text{SAR}}_r) is the validation set MSE of the trained probe, normalized against the variance of the scalar behaviour.

SAR^r\widehat{\text{SAR}}_r estimates the fraction of variance unexplained of the best behavioural approximator trained on subset indicators of order at most rr. A SAR^1=0\widehat{\text{SAR}}_1 = 0 would imply that individual feature effects fully explain the observed behaviour, while positive values indicate that the features’ effects are not entirely additive.

The prompt median SAR^1\widehat{\text{SAR}}_1 is positive throughout all model, layer and behaviour combinations, indicating that first order additivity is not exact (Sec. 4.1). SAR^1\widehat{\text{SAR}}_1 deciles correlate well with rank fidelity (Eq. 23) and selection regret (Eq. ?, page 6) (Sec. 4.3): low SAR^1\widehat{\text{SAR}}_1 implies higher rank fidelity and lower selection regret. Pairwise models improve rank fidelity and selection regret over order-1 approximators (Sec. 4.4). The differences are more pronounced in a steering setting (Sec. 4.6). Lastly, the Authors find near-0 correlation between reconstruction quality and sign error or selection regret (Sec. 4.7).

Critical strengths

The methodology of this paper is technically sound and mostly clear. The results are consistent across models from different families and across different tasks.

Critical weaknesses

The paper mostly reports the composability metrics introduced in this work (SAR, selection regret, rank fidelity, sign error etc.), but omits any evaluation on a utility/feature-aligned axis. Most of the experiments in this work target simple tasks (e.g., subject-verb agreement, gendered pronoun preference) and intervene on prompt-specific, randomly selected features (among the top-K). The semantics or roles of these features are not considered, and they may be unrelated to the measured behaviours. This makes the observations less relevant to practical settings where axes with well-defined roles are selected. I think that a composability analysis should ideally measure behaviour function(s) aligned with the semantics of the feature(s) being intervened upon.

Questions for authors

Comment on Results Reporting

Exact first order additivity was not expected since the model class considered is non-linear after the intervention. As such, small positive SAR^1\widehat{\text{SAR}}_1 values are not necessarily significant evidence for a non-compositionality of SAE features. Similarly for selection regret.

The median SAR^1\widehat{\text{SAR}}_1 values reported by the Authors in Fig. 2 are quite small - most fit in the [0.000 - 0.08] range. The Authors do not comment on the values themselves in Sec. 4.1 and only mention that they are positive and prefer to make a relative comparison with the rotation variant in Sec. 4.2 and Fig.3.

The rank fidelity of the order 1 models is 0.97820.9782, which is close to perfect. For pairwise model the rank fidelity is 0.99590.9959, which does mark an improvement, as stated by the Authors even in the Abstract. However, this seems like a small and not very significant improvement on top of an already near-perfect score at rank 1.

In Fig. 4 (right), the selection regret across SAR^1\widehat{\text{SAR}}_1 deciles are all in the [0.00 - 0.06] range. These values again do not seem that significant, yet the Authors prefer a relative comparison, stating that "Selection regret approximately doubles in Gemma and Llama and increases by about fivefold in Mistral". Pairwise models improve selection regret from 0.02720.0272 to 0.00510.0051 on average.

I find this aggregation pattern a bit inadequate. Since SAR and selection regret are newly introduced, the Authors should also argue for the significance of the absolute numbers when using them as evidence for any claims. Exact additivity was not expected in the first place. I think that the strongest numerical evidence for this behaviour lies in Sec. 4.6, related to LLM steering. Unfortunately, this section in particular does not receive that much attention.

Prior numbers came from simple, controlled settings. Qualitatively, they show some clean, well-behaved patterns, but I think that they are not as relevant for practical applications of multi-feature editing. I think that the evidence from the controlled settings remains rather weak and it leads to weak claims (L480):

  • "Individual effects remain insufficient in many cases" - By design, individual effects should not, in general, be sufficient .
  • "pairwise terms improve coalition prediction" - The improvement of pairwise terms is small in most of the experiments. This is mainly because order-1 models already have near-perfect scores. This is most likely caused by the simplicity of the tasks (e.g., copying, subject-verb agreement, gendered-pronoun preference - these imply the generation of only a small number of tokens) and measured behaviours (e.g., logit difference). The effects of SAE-based interventions on the final output are not linear due to the network's architecture, but they may need longer completion horizons for the non-additive effect to become noticeable.

Another reason for these weak non-additive observations may be in the nature of interventions. Multi-feature interventions would usually be done on semantically meaningful axes, tied to the generation objective. In the controlled setting the Authors pick feature subsets at random. The features they intervene on could thus be unrelated to the considered tasks, leading to overall small effects.

Regarding my rating, I am oscillating between 2 and 3. The paper seems technically sound. Even if some of the experimental settings are not that practical, I don't think this alone should be grounds for rejection. However, I am also having some concerns regarding the way the Authors interpret their results. I will leave the score as 2 for now as a precaution and reconsider it after seeing the other reviewers' comments.

Questions:

  1. Did the Authors observe any utility shifts on the simple tasks while performing the random SAE feature subtraction? (E.g., increased failure rate)
  2. Are the features that the Authors randomly intervene upon tied to any semantic role/explanation?
  3. Is CoT used in the steering experiment in Sec. 4.6?

Recommendations:

  • please add a number to the equation defining selection regret such that it is easier to reference
Reviewer_yeVGOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The paper studies whether the effects of SAE features on model behavior can be predicted by adding their individual effects. It removes different subsets of selected feature contributions while keeping the rest of the activation fixed, then measures the model's response. SAR measures how much of this response is left unexplained by models using individual features or feature interactions. Across several language models and tasks, SAE features have more additive effects than the tested rotations, and adding pairwise terms improves prediction on held-out feature combinations. A steering experiment on MMLU-Pro also shows better prediction of combined feature effects when pairwise terms are included.

Critical strengths
  1. The experiments provide consistent evidence across models and tasks. SAE features have more additive effects than the tested rotated features, and including pairwise interactions improves prediction of combined feature effects. Evaluation on held-out feature combinations and comparisons with parameter-matched models help support these findings.
Critical weaknesses
  1. Despite the relative simplicity of the idea, I found Section 3 unnecessarily difficult to follow. The paper would benefit from explaining the procedure and the motivation behind the mathematical definitions more directly.

  2. The paper does not demonstrate a clear use for the analysis. The experiments show that including pairwise interactions helps predict the effects of feature combinations. However, they do not show how the analysis leads to a better feature-based explanation or more effective steering on new inputs. In the steering experiment, fitting and evaluation use the same 1,400 questions, with only the feature combinations held out. It remains unclear whether using the pairwise model to select interventions also improves performance on new questions.

Questions for authors
  1. Does selecting steering combinations with the pairwise model improve accuracy on questions not used to fit it?
  2. Why is the first-order steering residual 0.6424 in Tables 1 and 20, but 0.7768 in Table 23 at the same feature count?
3 comments

Dear Reviewer yeVG,

We would like to respectfully offer a reminder and invite you to join the further discussion. We believe that most concerns have been reasonably addressed or substantially resolved. If you have a different view, we can give further clarification. Alternatively, we would also welcome you to raise the score or adjust the confidence.

We would also like to share important new experimental results that may change your view of the paper. To address your concern about practical utility, we extended our evaluation to five challenging benchmarks, including BBH, DROP, GSM8K, HotpotQA, and MATH 500.

  • Better feature combination selection. Pairwise modeling outperforms First order selection across all five benchmarks, achieving selection win rates of 52.8% to 73.9%, compared with 17.4% to 42.0% for First order.
  • Better behavioral preservation and search. Pairwise modeling achieves lower deviations from the original behavioral response and higher top candidate hit rates on all five benchmarks.
  • Practical value of SAR₁. Questions with higher SAR₁ exhibit larger gains from pairwise selection on four of the five benchmarks, demonstrating that SAR₁ can help identify when feature interactions are particularly useful.

We have also clarified the methodology in Section 3 and explained the different sampling protocols underlying the reported steering residuals.

Full experimental details will be included in the revision. We look forward to your reply and will be happy to answer any further questions.

Dear Reviewer yeVG,

The rebuttal period ends in less than 24 hours, and we appreciate your follow up. We believe our responses and new experiments have addressed most of your concerns. We would really appreciate it if you could take another look, share your thoughts, or reconsider your score.

Thank you for the new results. They provide stronger evidence that modeling pairwise interactions helps select better feature combinations, so I’m raising my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.