acceptodds
Under review as a conference paper at ICLR 2027

Interference Beyond Geometry in Concept Extraction

Abstract

Interference is commonly treated as geometric overlap between learned features. We introduce *effective interference*, which combines feature geometry and code statistics to capture realized interactions, distinguishing constructive from destructive interference and frequent weak interactions from rare strong ones. Under local fixed-support assumptions, we characterize how architectural constraints shape interference through four mechanisms: feature orthogonalization, bias compensation, gain adaptation, and encoder-decoder separation. Experiments with sparse autoencoders show that constrained architectures selectively reduce overlap among co-active features, while bias, gain, and encoder freedom allow constructive cross-contributions to remain. Together, these results show that interference in learned representations depends not only on feature geometry, but also on how features are used and on the architecture that produces their codes.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic interpretability, interference, sparse autoencoders, concept extraction, superposition

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_b2U9OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces effective interference to capture feature interactions, which comes from both feature geometry and code statistics. Theoretically, under fixed support assumption, this paper presents code-optimality condition and encoder consistency identity. Also, this paper analyzes four controlled cases: constrained local orthogonalization, conditional bias compensation, encoder-gain adaptations, and encoder-decoder duality. Empirical results on controlled SAE training examines theoretical results.

Critical strengths

None.

Critical weaknesses
  1. The theoretical contributions are limited. Proposition 4.1 is a direct least-square optimality result. Proposition 4.2 is simply an algebraic identity. Proposition 4.3 requires very strong assumptions including only two active features, which significantly weakens the implications.

  2. The four mechanisms discussed in Section 4 are weakly connected to full training. Routes I, II, III, IV relies on different restricted setups. However, the paper does not provide how these mechanisms will take effect and interact with each other in full training.

  3. The empirical results are not convincing and requires further explanation. For instance:

    • In figure 2, the empirical result of ∑ρij2\sum \rho_{ij}^2 increases and ∑Iij2\sum I_{ij}^2 decreases from peak cannot directly reflects selective orthogonalization of co-active pairs, since the statistics is summed up over i,ji,j, and Iij=ρijπijMijI_{ij} = \rho_{ij} \pi_{ij} M_{ij} rely on multiple variables without explicit control.

    • In line 374, "large πijMij\pi_{ij}M_{ij}" and "near zero" are unspecified. Additionally, in figure 3, the vertical axis and color scale are both logarithmic, which makes it very difficult to assess the claims. More detailed criteria and quantitative results are necessary.

Questions for authors

Please refer to the weaknesses part.

  1. Could the authors explain the role of two active features in the proof of Proposition 4.3? Is it possible to generalize Proposition 4.3 to arbitrary size of SS?

  2. Could the authors further explain the implications from figure 2, and provide detailed quantitative analysis regarding figure 3?

6 comments
AuthorsOpenReview ↗

Addressing W1/Q1

We appreciate the time the reviewer spent on our paper and we address their concerns below.

[W1/Q1]

We agree that Proposition 4.1 is a standard fixed-support least-squares condition and Proposition 4.2 is an algebraic consistency identity. Indeed, their role is to establish the reference condition and common balance underlying the architectural analysis, rather than to constitute standalone theoretical novelties. We also agree that the original two-feature result alone does not characterize practical SAE training. Hence, we have added a new analyses (Propositions and Corollaries) on general-support Gram dynamics (<2<k<p<2<k<p, Appendix G) and bias/gain learning dynamics (Appendices H and I) that go beyond encoder consistency and do not assume per-input code optimality.

Clarifying the role of Propositions 4.1–4.2. Proposition 4.1 supplies an ideal code-optimality target, not a condition automatically satisfied by an amortized encoder. Proposition 4.2 identifies how encoder gain, incoming cross-talk, bias, and residual coupling jointly balance on an active region. Its value is in distinguishing which mechanisms different architectures can use; the identity alone does not establish that optimization learns a particular mechanism. The new results address this distinction directly. Beyond two active features: general-support Gram dynamics. Our analyses consider the full active-support Gram deviation ΔS=∥DS⊤DS−Ik∥F2,∣S∣=k.\Delta_S=\|\mathbf D_S^\top\mathbf D_S-\mathbf I_k\|_F^2, \qquad |S|=k. For tied, unit-norm, bias-free weights, let G=DS⊤DS\mathbf G=\mathbf D_S^\top\mathbf D_S, K=G−Ik\mathbf K=\mathbf G-\mathbf I_k, and C=1N∑nzSn(zSn)⊤\mathbf C=\frac{1}{N}\sum_n\mathbf z_S^n(\mathbf z_S^n)^\top for a batch sharing support SS. Our new fixed-support Gram-dynamics proposition gives, for a gradient step followed by column normalization,

ΔS+−ΔS=−8ηΦS+O(η2),ΦS=tr⁡(K2C)−∑i∈S(K2)ii(KC)ii.\Delta_S^+-\Delta_S =-8\eta\Phi_S+O(\eta^2), \qquad \Phi_S= \operatorname{tr}(\mathbf K^2\mathbf C) -\sum_{i\in S}(\mathbf K^2)_{ii}(\mathbf K\mathbf C)_{ii}.

This identity holds for k>2k>2 and accounts for interactions among all active features. The first term is nonnegative; the second is a signed correction from column normalization. Hence, ΦS>0\Phi_S>0 guarantees a decrease for sufficiently small, support-preserving steps. When ΔS>0\Delta_S>0, a sufficient condition is q<λq<\lambda, where λ=λmin⁡(CS)\lambda=\lambda_{\min} (\mathbf C_S) and q=∥diag⁡(KC)∥∞q=\|\operatorname{diag}(\mathbf K\mathbf C)\|_\infty. Intuitively, sufficiently varied code amplitudes provide excitation that dominates the normalization correction. These are sufficient conditions, not assumptions we claim every trained support satisfies (See Appendix G for formal proof).

Sensitivity to support changes. Fixing the support is a local condition for the analyzed update, not a claim that an input retains its support throughout training. For a finite batch away from selection boundaries, sufficiently small parameter changes preserve the active affine branches. As training progresses, supports may change between iterations and are therefore not restricted to a predefined set throughout training. Such local analyses are related to support-consistency arguments in sparse coding and autoencoder theory, including analyses of alternating-minimization schemes (Arora et al., 2015; Chatterji & Bartlett, 2017; Nguyen et al., 2019).

Fixing the support is therefore an analytical device for studying local gradient dynamics and stationarity conditions, rather than a constraint imposed on training. Across examples and iterations, overlapping and changing supports couple updates through shared dictionary atoms. Thus, features are not confined to permanently isolated subdictionaries: the local analysis holds the support fixed for the update being studied, while training remains free to select different supports subsequently. This distinction explains how local mechanisms can inform our interpretation of training without providing a guarantee across support transitions.

Deviations from code optimality. The encoder-consistency identity itself is exact on the active affine region; approximate code optimality enters when residual-coupling terms are dropped. The new Gram, bias, and gain gradient analyses are derived directly from reconstruction loss and do not assume DS⊤r=0\mathbf D_S^\top\mathbf r=0. Hence, our dynamic analyses do not use per-input code optimality.

It is important to note that sparse-coding analyses often separate inference with a fixed dictionary from dictionary learning, examining how inference accuracy affects learning. SAEs amortize inference through a shared encoder; Proposition 4.1 therefore provides an ideal fixed-support code-optimality reference, not a condition guaranteed by end-to-end training. Figure 2C shows that the gap, E∣DS⊤r∣2\mathbb E|\mathbf D_S^\top\mathbf r|_2, decreases but remains nonzero, helping assess approximations based on this condition without establishing exact per-input optimality or convergence of alternating minimization.

AuthorsOpenReview ↗

Addressing W2

[W2 - Role of the Four Routes]

Thank you for raising this point. The routes are controlled architectural limits of a common encoder-consistency balance, not mutually exclusive mechanisms or successive training phases. We analyze them separately to isolate their roles, while all permitted parameters can adapt jointly during training.

The settings are composable and partly nested. The tied baseline is a special case of both the bias and gain settings. The gain setting is an aligned-direction special case of our untied model, provided the bias constraints match. Thus, geometric adaptation can coexist with bias compensation or gain adaptation, and untied encoders permit both norm and directional changes. Bias remains an independent choice; the settings can be a nested sequence or an independent addition. The tied-with-bias setting therefore combines Routes I and II, and both can operate during training. In this setting, the bias is zero at initialization, so the local orthogonalization mechanism of Route I initially takes effect. As the bias adapts to the interference pattern, Route II allows more effective interference. This helps explain why the settings in Figure 6 show different levels of effective interference. Proposition F.1 in the appendix also combines Routes I and III. Route IV implicitly includes Route III because encoder weights are not normalized in the untied setting.

Evidence under full training. Our ablations jointly train all permitted parameters using minibatch Adam and changing supports. Within each SAE family, adding bias, gain, or independent encoder directions can retain positively aligned, co-active decoder features, reflected in the rightward extension of the inference heatmaps in Figure 3 and the aggregate differences in Figure 6. These comparisons test the effect of making mechanisms available, not uniquely attributing each interaction to one route. These choices also occur in practical SAEs: Cunningham et al. (2024) use tied weights with bias, while Gao et al. (2025) use separate encoder and decoder weights. We have also added an analysis of independently trained SAEBench models in the appendix to examine these mechanisms in SAEs used in practice, providing complementary evidence of substantial effective interference beyond our controlled recipe.

AuthorsOpenReview ↗

Addressing W3/Q2

Elaborating Figure 2

The reviewer’s comment is addressable by clarification. Figure 2 does not aim to directly provide numerical analyses for Proposition 4.3 (we have clarified it in the caption and color in the text). Figure 2 provides intuition on what happens to effective interference during training. This directly relates to the reconstruction analysis in Eq. (3.3):

=\sum_{i\ne j}(\rho_{ij}M_{ij}\pi_{ij})^2.$$ This is a weighted measure of orthogonalization: each pair’s correlation $\rho_{ij}$ is weighted by its co-activation magnitude $M_{ij}$ and co-activation probability $\pi_{ij}$. Equivalently, after normalization, the sum can be viewed as a weighted average of $\rho_{ij}^2$, with larger weights assigned to pairs that co-activate more strongly or more often. The decrease of effective interference, hence, relates to the active support in each iteration of the training (the setting used by Proposition 4.3 and Proposition G.1): the message of the proposition is that under some conditions (see the Propositions) the SAE pushes the __high co-active__ pairs to be more orthogonal (or similar statement on the Gram deviation error). This is a local orthogonalization on the active pairs, not a global orthogonalization. We have conveyed this in Figure 2 such that this means the rho becomes small on pairs that matter, and might be small on unrelated non co-activated pairs. So the unweighted correlation statistic, by contrast, treats all pairs equally may go up while what the network is concerned for is $I_{ij}$. We have included the above elaborations in the main text now, and also refer the reader to Figure 8 in the appendix. The figure illustrates why the weighted and unweighted statistics can move in different directions, and that the density of the point with high effective interference (high $\pi_{ij} M_{ij}$ and high $\rho_{ij}$ are those that move toward lower the interference. Particularly, those that keep their code overlap $\pi_{ij} M_{ij}$ high, end of having very low $\rho_{ij}$. We show that at initialization, the values of $\rho_{ij}$ are more concentrated around zero due to the random initialization. As training progresses, their overall range widens, explaining the increase in the unweighted statistic. This is what one sees by projecting all the points in Figure 8 onto the (x)-axis, which represents $\rho_{ij}$, while ignoring the (y)-axis, which represents the code statistics $M_{ij}\pi_{ij}$. Once the (y)-axis is taken into account, we observe that the correlations do not widen for all pairs. Pairs with higher $M_{ij}\pi_{ij}$, $\rho_{ij}$ instead shrinks relative to initialization while lower one widens. The pairs with higher $M_{ij}\pi_{ij}$ contribute most strongly to $\sum_{i\ne j}(\rho_{ij}M_{ij}\pi_{ij})^2$, so their behaviour captures the selective orthogonalization mechanism behind the decrease in this statistic which comes from the reconstruction (Eq 3.3). We hope this clarifies the distinction between global and selective orthogonalization. We now explain these intuitions from Figure 8 in the main manuscript. Finally, we would be happy to address the reviewer’s concern about multiple variables further if they specify which variables or controls they have in mind. **Elaborating Figure 3** Below, we elaborate how to interpret Figure 3. We have revised line 374 to describe the distributional trend more precisely: >“In the tied, normalized, bias-free model, the distribution of $\rho_{ij}$ narrows symmetrically around zero as $\pi_{ij}M_{ij}$ increases. Adding bias, encoder gain, or untying breaks this symmetry, shifting pairs toward positive $\rho_{ij}$ and retaining constructive interference.” Throughout, “large” and “small” describe relative values within the joint interference profile ($\rho_{ij}$ versus $\pi_{ij}M_{ij}$). Similarly, “near zero” describes concentration around $\rho_{ij}=0$. For example, “large $\rho_{ij}$” refers to pairs with relatively high positive geometric alignment among the displayed pairs. These are qualitative descriptions rather than fixed numerical thresholds. Figure 3 uses a logarithmic vertical axis because the co-activation-weighted magnitude $\pi_{ij} M_{ij}$ spans several orders of magnitude, and a logarithmic color scale because the bin counts do as well. It shows the distribution of feature pairs. To help the reader understand these colorbars, we have added the linear scale colormap in Figure 22. We note that beyond these distributional figures on the interference, we have quantitative metrics. Figure 6 shows quantitative aggregate summaries of constructive, destructive, and net interference; this figure is modified now with error bars on runs over three independent seeds. Moreover, Figure 13 contains quantitative results on the interference profile ($\pi_{ij}$, $M_{ij}$, $\rho_{ij}$); the figure illustrates the interference profile across support size $k$ and dictionary width $p$.
Reviewer_vcR8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper deals with mechinistic interpretability, which aims to identify the concepts that are encoded within LLM and other large neural networks, and how they possibly interact. More precisely, it is interested in the possible overlap of concepts, a phenomenon called interference. A usual framework to identify the concepts is that of sparse autoencoders (SAE), consisting to encode the output of a given layer of the large network into sparse codes, then reconstructing them with a the decoder of the SAE. The concepts are then assimilated to the columns of the SAE decoder, also seen as (linear) directions in the sparse latent space.

To study interferences between concepts/directions, the paper argue that, beyond directions themselves, one should also focus on the actual sparse codes and to which extend they actually co-occur for a given dataset. For this, they define two tools, based on the codes, the effective interference and the interference profile, the latter being a triplet which allows to identify various "regime" of interaction (in terms of strenght and actual frequency) and the former being in practice the product of the three terms. It allows to distinguish the interaction that are frequent but low to those that are seldom but large..

Through a constrained model that is progressively relaxed, the paper characterize how the interference contribute to a specific accomodation mechanism, identifying four of them: local orthogonalization, conditional bias compensation, encoder-gain adaptation and the diffrent roles of encoders as feature detectors and decoders as reconstruction components

Experiments are conducted with four types of SAEs trained on the activation coming from a layer of a Pythia-160M model.

Critical strengths

[S1] taking into account both the direction in the decoder and the effective co-occurence of codes from the actual data is original and interesting. This is done throught the definition of tools that make sense are properly defined and characterized. It allows to distinguish several types of interaction (see sumary) and the relaxation of the constrained models allows to identify precisely the role of 4 different (and clearly defined) mechanisms.

Critical weaknesses

[W1] the main experimental evaluation are conducted with SAEs trained from from the activations of a single layer of a single decoder model, namely Pythia-160M. Hyperparameter are also arbitrary fixed. I already met reviewers and AC who estimated that an evaluation with 6 models and 3 datasets from different modalities was not convincing in the field of mechanistic interpretability. I nevertheless do not approve such a position. Moreover other experiments are conducted with SAE directly trained on MNIST in the appendix, as well as with SAEs trained on SAEBench (by another team, who released them publicly) from activations of Gemma2-2B and Pythia-160M. However, it still remains limited, considering that the experiment on MNIST are roughtly a toy example, and that only a part of the experiments could have been conducted, with results that match partialy those of the main paper only. The effects of architecture, model scale, representation layer, pre-training objective, and dataset choice could have been investigated further.

[W2] another limit of the evaluation is that all the metric reported are based on the quatities defined in the article itself. While relevant to study the proposed work, it would have been more convincing to include extrinsic metrics, which are more universaly used and could be compared to other works. One must recognize the authors partially report such limitation in the conclusion " effective interference diagnoses energy allocation, not representa- tion quality: constructive and destructive describe cross-term signs without implying reconstruction quality, semantic interpretability, causal relevance, or downstream behaviour.". It nevertheless remains a weakness in the sense that it limits the impact of the proposed framework, making hard to see the point in practice.

Minor weakness (low impact on my assessment)

  • [MW1] all the theoretical framework relies on the assumption of fixed local support (line 218, propositin 4.1), which is quite restricted for a real model. However, I admit that such a consideration (non fixed SS) can be considered in future works.

Minor remarks (no need to be addressed in the rebutal)

  • line 154: "where $x\sim P is the input activations" or more precise explanation. While "obvious" it is better to explain what it is, possibly that they are the output of a layer of the large model studied.
  • line 178 "Proposition D.1" --> Proposition 4.1; same problem on line 301 with "Proposition F.1"
  • no need to put equation (3.5) in a box
  • line 252: the text should explicit what is intended by "tied weights"; it could be write at the end of line 236, after "asymetric".
  • line 260: after "γij=ρij\gamma_{ij} = \rho_{ij}" add "for i≠ji\neq j" as on line 283
  • line 281: "higher kk" --> more than two features
    • kk is not defined. One guesses it is ∣S∣|S| thus k=2k=2 in proposition (4.3)
Questions for authors

[Q1] do the constructive interaction can help to identify "better" concepts, in the sense they are more interpretable (or, possibly, being more usefull for a downstream task) ?

[Q2] what is the realism of the fixed local support assumption (see [MQ1]) for real models and data ?

[Q3] how the proposed metric relate to the more usual metrics in mechanistic interpretability?

9 comments
AuthorsOpenReview ↗

Addressing W1

We sincerely thank the reviewer for their thoughtful and constructive feedback. We are grateful for the time and attention they devoted to our paper, as reflected in their detailed comments and minor remarks. We were also pleased that they found our approach “original and interesting” and its concepts to “make sense [and be] properly defined and characterized.”

To address the reviewer’s concerns, we have uploaded a revised manuscript, with the main changes marked in blue. The revision includes:

  • An evaluation of effective interference in Pythia-160M at every even layer from 2 to 10 (Appendix O).
  • An evaluation of our trained SAEs with SAEBench and a discussion of how its metrics relate to effective interference (Appendix N).
  • Examples of feature pairs with different effective interference, annotated with Neuronpedia descriptions (Appendix P).
  • An extension of the local orthogonalization analysis beyond two active features** to a general fixed support size (Appendix G).
  • Going beyond encoder consistency mathematical analysis and providing new bias and encoder gradient dynamics without code-optimality assumption (Appendices H and I).

[W1]

We thank the reviewer for raising this point as it has motivated us to clarify our experimental choices and extend our analysis.

We have revised the manuscript to explain that the main hyperparameters follow those used by SAEBench for the same model and layer (k=40k=40, p=16,384p=16{,}384). Although we held them fixed for the comparisons in the main paper, we also tested multiple settings. The final paragraph of the Results section and Figure 12 in Appendix M examine k∈{10,40,160}k \in \{10, 40, 160\} and p∈{4,096,16,384,65,536}p \in \{4{,}096, 16{,}384, 65{,}536\}.

Since submission, we have extended our TopK analysis to layers 2, 4, 6, and 10 of Pythia-160M, in addition to layer 8 (Appendix O). At every tested layer, net effective interference follows the ordering shown in Figure 6: Tied < Tied + Bias < Tied + Gain < Untied. Layer 8 has the lowest net effective interference in every setting. The results are given below and in Figure 7. The “flame-shaped” distributions are also consistent across layers and follow the pattern in Figure 3 (new Figure 20). The results suggest that the effects of architectural choices on effective interference persist across layers despite differences in the input activation distributions: the tied baseline exhibits the lowest interference, which generally increases when bias or gain is introduced or the encoder and decoder are untied.

Together, these results show that the ordering is robust across the tested layers.

ConstraintL2L4L6L8L10
(i) Tied.073.053.043.035.077
(ii) Tied + Bias.261.176.119.114.192
(iii) Tied + Gain.399.324.255.229.355
(v) Untied.482.571.477.249.418

We appreciate that the reviewer acknowledges our MNIST and SAEBench evaluations. Our focus is on studying the inductive biases of SAEs as tools, rather than properties specific to language-model activations. To understand how architectural constraints shape interference, we tested four SAE families, five constraint configurations, and nine hyperparameter settings. We chose MNIST because its data distribution differs substantially from internal language-model activations. Observing similar patterns on MNIST provides further evidence that these inductive biases arise from SAE architectures rather than from language-model activations alone.

The same motivation led us to examine SAEs trained through a different pipeline. We used publicly released SAEBench SAEs trained by another team on activations from Gemma-2-2B and Pythia-160M. We observe similar interference patterns in these independently trained SAEs, despite differences from our setup, including auxiliary losses, a learned pre-encoder bias, and learning-rate scheduling.

Our goal is to highlight the different ways sparse autoencoders accommodate interference. We hope the new results across Pythia layers, together with the hyperparameter analysis and clarification of the paper’s focus, address the reviewer’s central concern and strengthen the evidence for our main claims.

AuthorsOpenReview ↗

Addressing W2/Q3

[W2/Q3] We appreciate this point, which was also raised by reviewer 5ZYE. In response, we evaluated our trained SAE configurations with SAEBench. This includes the new TopK SAEs trained at multiple layers. Appendices N and O compare these metrics with net effective interference (Eq. 3.6). We have added the following in our manuscript (and more details can be found in the appendices):

Relation of interference to other metrics We evaluated all trained SAEs on SAEBench. Across constraints, net effective interference increases with reconstruction quality (Table 2); across layers, however, lower-interference layers reconstruct better (Table 3). Thus, effective interference does not simply track reconstruction quality: it provides a distinct global measure of SAE behavior. Most other SAEBench metrics have weak relationships with net effective interference (Section N). This partly reflects a difference in scale: interference aggregates interactions across the full code and dictionary, whereas these metrics assess particular concepts. It also reflects a difference in what is measured: interference captures co-activation and geometry without identifying what a feature pair means. Consider the example of the absorption metric: zero interference between two features could indicate absorption, but it could also mean that the features are semantically unrelated. Targeted probe perturbation (TPP), however, is an exception: better TPP tends to accompany lower interference. We believe this is because TPP, like effective interference, aggregates information across many concepts.

We also analyse each metric bellow:

  • Absorption. When a parent feature is absorbed by a child, the pair does not co-activate, so its pairwise effective interference is zero. The reverse does not follow: unrelated features can also have zero interference, and frequent co-activation does not establish a hierarchy. Consistent with this distinction, we find no clear relationship between absorption and net effective interference.
  • Concept isolation. Sparse probing and SAE probes ask whether a single latent predicts a concept, whereas effective interference measures overlap and co-activation between pairs of latents. The two probing benchmarks show different trends in our results, so we find no consistent relationship with net effective interference.
  • Causal disentanglement. Lower net effective interference consistently accompanies better Targeted Probe Perturbation (TPP) performance in our experiments. This trend does not hold for Spurious Correlation Removal or RAVEL. One possible explanation is that TPP tests how selectively an ablation affects a target probe relative to other probes, while the other benchmarks also depend on which latents are selected and how the model responds to the intervention. The TPP association does not, by itself, establish a causal effect of interference.
  • Automated interpretability. Apart from the tied configuration, which has substantially lower scores, we find no consistent relationship between net effective interference and feature interpretability. The tied result suggests that this configuration may be too restrictive to learn readily describable features; it does not imply that reducing interference generally improves interpretability.

Regarding the practical value of our framework, we want to draw attention to the codes as well as the concept directions. Related work has used dictionary-level geometric quantities or orthogonality objectives [1, 2]. A measure based only on the dictionary, such as its Gram matrix DD, does not capture how often features activate together. Effective interference makes that distinction visible and helps characterize the interactions realized by a trained SAE. Our new preliminary exploration supports this view: we looked at a small number of feature pairs labelled on Neuronpedia, using the public Gemma Scope SAE (Gemma-2-2B, layer 12, 16k latents, 2M Pile tokens). In these examples, pairs with similar geometry often relate quite differently depending on how much they co-activate, which effective interference captures. We also note that Neuronpedia's "similar features" are ranked by decoder cosine only. For a typical feature, the 24 nearest neighbours by cosine include only about a third of its 24 strongest interference partners. Geometry alone therefore misses many of the features a given feature actually interacts with, and the interference profile helps identify them (See response to Q1 and Appendix P).

[1] Anton Korznikov, Andrey Galichin, Alexey Dontsov, Oleg Rogov, Elena Tutubalina, and Ivan Oseledets. “OrtSAE: Orthogonal Sparse Autoencoders Uncover Atomic Features.”

[2] Moritz Miller, Florent Draye, and Bernhard Schölkopf. “Superposition Without Interference? Towards Isolated Interventions via Almost Orthogonal Features in Language Models.” COLM 2026.

AuthorsOpenReview ↗

Addressing MW1/Q2

[MW1/Q2 - Fixed support]

We agree that the fixed-local-support assumption is a limitation of our theoretical analysis. However, we would like to emphasize that the support is held fixed only locally for the gradient update being analyzed, not throughout training. For a finite batch away from selection boundaries, sufficiently small parameter changes preserve the active affine regions, allowing us to characterize local gradient dynamics and stationarity conditions. Such local, support-conditional analyses are common in sparse coding and autoencoder theory, including support-consistency arguments and analyses of alternating-minimization schemes (Arora et al., 2015; Chatterji & Bartlett, 2017; Nguyen et al., 2019). Importantly, there is no constraint on support selection during training. Supports can differ across samples within a batch and change across training iterations. These varying and overlapping supports couple updates through shared dictionary atoms, so features are not confined to fixed or isolated subdictionaries. Thus, fixing the support is an analytical device for studying the local behavior of an update, rather than an assumption that supports remain unchanged throughout learning. While our analysis does not explicitly characterize transitions between supports, the empirical results in Figure 4 closely agree with its predictions, even though supports change during SAE training. This suggests that the identified local mechanisms remain informative in practice despite varying support. We agree that extending the theory to explicitly account for support transitions is an interesting direction for future work.

Minor remarks: We thank the reviewer again for their helpful formatting and notation suggestions. We have addressed them in the revised manuscript!

Reviewer_5ZYEOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates interference in learned concept representations beyond conventional geometric overlap. The authors introduce effective interference, which combines dictionary geometry with code co-activation statistics and activation magnitudes to characterize realized feature interactions. Under local fixed-support assumptions, the paper develops an encoder-consistency analysis and identifies four mechanisms through which sparse autoencoder (SAE) architectures accommodate interference: local orthogonalization, bias compensation, encoder-gain adaptation, and encoder-decoder separation. Experiments across multiple SAE families trained on Pythia-160M activations, together with additional evaluations on MNIST and pretrained language-model SAEs, demonstrate distinct interference patterns under different architectural constraints.

Critical strengths
  1. The paper highlights the limitations of measuring interference solely through feature geometry and introduces a data-dependent perspective that incorporates code co-activation and activation magnitudes. This provides a useful framework for distinguishing potential geometric overlap from realized feature interactions. 2.The encoder-consistency formulation offers a unified perspective on how architectural constraints influence feature interactions. The analysis of local orthogonalization, bias compensation, gain adaptation, and encoder-decoder separation provides interpretable explanations for different interference patterns in SAEs. 3.The authors investigate four SAE families and multiple controlled architectural variants, supported by analyses of interference dynamics, learned encoder parameters, and signed interaction contributions. Additional experiments on MNIST and independently trained language-model SAEs further support the generality of the observed patterns.
Critical weaknesses
  1. While the agreement between predicted and learned parameters in Figure 4 is encouraging, I am not fully convinced that it provides independent evidence for the proposed compensation mechanisms. Since Eqs. (4.8)–(4.9) are derived from the encoder-consistency identity using statistics from the same trained models, the results may primarily reflect the underlying mathematical relationships.
  2. The theoretical analysis relies on fixed-support conditions and approximate code optimality, with Proposition 4.3 further restricting the analysis to two active features. It remains unclear how well these results extend to practical SAE training, where activation supports change and optimization involves many interacting features.
  3. I find the architectural comparisons informative, but some of the observed differences may also be related to changes in reconstruction quality and feature utilization. For example, Table 2 shows a substantial increase in dead features for the untied JumpReLU model, which could affect the code statistics underlying effective interference.
  4. Although effective interference provides a useful characterization of reconstruction interactions, I would have liked to see a more concrete demonstration of its analytical value. The current experiments do not clearly show what meaningful feature relationships this measure reveals beyond conventional geometric overlap.
Questions for authors
  1. It would be helpful to understand whether the proposed bias and gain mechanisms can be validated beyond the parameter relationships shown in Figure 4. Have the authors considered controlled interventions to examine how these mechanisms affect interference?
  2. Could the authors comment on how sensitive the theoretical predictions are to support changes and deviations from code optimality? In particular, would similar orthogonalization trends be expected with larger active supports?
  3. Since reconstruction quality and dead-feature rates vary across architectural settings, would the main interference patterns persist if these factors were approximately matched?
  4. I would be interested in seeing a few representative feature pairs with similar geometric overlap but different effective interference. What additional insights about their interactions can be obtained from the proposed measure?
9 comments
AuthorsOpenReview ↗

Addressing W1/Q1

We thank the reviewer for the thoughtful feedback. The revised manuscript (changes in blue) addresses these concerns via:

  • New experiments to demonstrate the insights effective interference offer on Neuronpedia (see also our response to Reviewer vcR8 on relation of effective interference to other interpretability metrics).
  • New analyses on the gradient dynamics of bias, encoder gain, and the Gram matrix, extending beyond encoder consistency and generalizing the two-feature analysis to larger supports.

Please see the revised manuscript for the full new results, propositions, and figures.

[W1/Q1, encoder consistency, bias and gain mechanisms]

Thank you for this important distinction. We agree that Figure 4 alone does not provide independent causal evidence for compensation: it assesses agreement with balance predictions that neglect residual coupling, not how optimization learns that balance. We have clarified this diagnostic role and derived new bias and gain gradients directly from training reconstruction loss going beyond encoder consistency and without assuming per-input code optimality. Our new analyses establish how gradient descent learns mean bias compensation with a fixed dictionary and how compensation affects dictionary evolution. Gain dynamics connect parameter updates directly to signed effective interference.

Bias dynamics beyond consistency. For a tied, unit-norm model and a batch sharing support SS, define G=DS⊤DS\mathbf G=\mathbf D_S^\top\mathbf D_S, K=G−I\mathbf K=\mathbf G-\mathbf I, and s‾S=1N∑nDS⊤xn\overline{\mathbf s}_S=\frac{1}{N}\sum_n\mathbf D_S^\top\mathbf x^n. With mean code μS=s‾S+bS\boldsymbol\mu_S=\overline{\mathbf s}_S+\mathbf b_S, the reconstruction-loss gradient is

gb,S=bS+KμS\mathbf g_{b,S}=\mathbf b_S+\mathbf K\boldsymbol\mu_S.

Given the dictionary and inputs at the training iteration, the bias update is

bS+−bS⋆=(I−ηbG)(bS−bS⋆)\mathbf b_S^+-\mathbf b_S^{\star} =(\mathbf I-\eta_b\mathbf G)(\mathbf b_S-\mathbf b_S^{\star}), where bS⋆=−G−1Ks‾S\mathbf b_S^{\star}=-\mathbf G^{-1}\mathbf K\overline{\mathbf s}_S. This establishes a learning mechanism and a predictable recovery trajectory (For more and formal proof, see Proposition H1).

How bias changes dictionary learning. The decomposition C=Σ+μμ⊤\mathbf C=\boldsymbol\Sigma+\boldsymbol\mu\boldsymbol\mu^\top separates code fluctuations from mean activity. At bias stationarity, DS⊤(x^−x)=K(zS−μS)\mathbf D_S^\top(\widehat{\mathbf x}-\mathbf x)=\mathbf K(\mathbf z_S-\boldsymbol\mu_S), so centered covariance replaces the uncentered second moment in first-order Gram dynamics (Appendix H).

Gain dynamics beyond consistency. For tied encoder directions with positive gains, unit-norm decoder columns, and no bias, zi=αidi⊤xz_i=\alpha_i\mathbf d_i^\top\mathbf x on the active support. Direct differentiation gives

gα,i:=∂LS∂αi=αi−1αi2S^i+1αi∑j∈S,j≠iI^ij,g_{\alpha,i} :=\frac{\partial\mathcal L_S}{\partial\alpha_i} =\frac{\alpha_i-1}{\alpha_i^2} \widehat S_i+\frac1{\alpha_i} \sum_{\substack{j\in S, j\neq i}}\widehat I_{ij}, where first term is a self-gain correction and second term is a signed cross-feature contribution. Consequently, αi+=αi−ηααi2[(αi−1)S^i+αi∑j∈S,j≠iI^ij].\alpha_i^+ =\alpha_i-\frac{\eta_\alpha}{\alpha_i^2} \left[(\alpha_i-1)\widehat S_i +\alpha_i\sum_{\substack{j\in S, j\neq i}}\widehat I_{ij}\right].

At unit gain, a positive signed sum of interference involving feature i drives its gain downward, while a negative sum drives it upward. Away from unit gain, this contribution competes with the self-gain correction, so the update need not be monotonic. This provides an explicit optimization mechanism for adjusting amplitudes without changing decoder angles and requires neither per-input code optimality nor full column rank of DS\mathbf D_S (see Proposition I.1 and Corollary I.1).

Architectural ablations. Our existing experiments address this question through interventions on the availability of the proposed mechanisms. Within each SAE family, we compare the tied, unit-norm, bias-free baseline with tied + bias, tied + gain, and untied variants. These comparisons show that additional architectural freedom can preserve positively aligned, co-active decoder features and substantial constructive interference. Figure 8 illustrates this through distributions extending toward positive decoder alignment and, in some variants, larger co-activation-weighted magnitudes. Unlike Figure 4, these results concern changes in learned interference following architectural interventions, rather than agreement with a same-checkpoint parameter relationship. Importantly, Figures 8–11 examine pairwise distributions and do not rely on an aggregate statistic. Plotting decoder orientation ρij\rho_{ij} against co-activation and weighted magnitude ($\pi_{ij}, MijM_{ij}) reveals how geometric and statistical changes jointly shape interference. Complementarily, our net interference summary expresses the signed pairwise contribution relative to expected reconstruction energy. This normalization controls for overall reconstruction-energy scale.

AuthorsOpenReview ↗

Addressing W2/Q2

[W2/Q2 - k>2k>2]

Thank you for highlighting this distinction. We have now addressed the two-feature concern through a new general-support (2<k<m2<k<m) analysis. The results do not assume code optimality. We also note that our experiments were conducted with larger supports (k=10,40,160k=10,40,160) and ordinary training procedures, where the general trend we have reported still appears.

Our analyses consider the full active-support Gram deviation ΔS=∥DS⊤DS−Ik∥F2,∣S∣=k.\Delta_S=\|\mathbf D_S^\top\mathbf D_S-\mathbf I_k\|_F^2, \quad |S|=k. For tied, unit-norm, bias-free weights, let G=DS⊤DS\mathbf G=\mathbf D_S^\top\mathbf D_S, K=G−Ik\mathbf K=\mathbf G-\mathbf I_k, and C=1N∑nzSn(zSn)⊤\mathbf C=\frac{1}{N}\sum_n\mathbf z_S^n(\mathbf z_S^n)^\top for a batch sharing support SS. Our new fixed-support Gram-dynamics proposition gives, for a gradient step followed by column normalization,

ΔS+−ΔS=−8ηΦS+O(η2),ΦS=tr⁡(K2C)−∑i∈S(K2)ii(KC)ii.\Delta_S^+-\Delta_S =-8\eta\Phi_S+O(\eta^2), \qquad \Phi_S= \operatorname{tr}(\mathbf K^2\mathbf C) -\sum_{i\in S}(\mathbf K^2)_{ii}(\mathbf K\mathbf C)_{ii}.

This identity holds for k>2k>2 and accounts for interactions among all active features. The first term is nonnegative; the second is a signed correction from column normalization. Hence, ΦS>0\Phi_S>0 guarantees a decrease for sufficiently small, support-preserving steps. When ΔS>0\Delta_S>0, a sufficient condition is q<λq<\lambda, where λ=λmin⁡(CS)\lambda=\lambda_{\min} (\mathbf C_S) and q=∥diag⁡(KC)∥∞q=\|\operatorname{diag}(\mathbf K\mathbf C)\|_\infty. Intuitively, sufficiently varied code amplitudes provide excitation that dominates the normalization correction. These are sufficient conditions, not assumptions we claim every trained support satisfies (See Appendix G for formal proof).

Sensitivity to support changes. Fixing the support is a local condition for the analyzed update, not a claim that an input retains its support throughout training. For a finite batch away from selection boundaries, sufficiently small parameter changes preserve the active affine branches. As training progresses, supports may change between iterations and are therefore not restricted to a predefined set throughout training. Such local analyses are related to support-consistency arguments in sparse coding and autoencoder theory, including analyses of alternating-minimization schemes (Arora et al., 2015; Chatterji & Bartlett, 2017; Nguyen et al., 2019).

Fixing the support is therefore an analytical device for studying local gradient dynamics and stationarity conditions, rather than a constraint imposed on training. Across examples and iterations, overlapping and changing supports couple updates through shared dictionary atoms. Thus, features are not confined to permanently isolated subdictionaries: the local analysis holds the support fixed for the update being studied, while training remains free to select different supports subsequently. This distinction explains how local mechanisms can inform our interpretation of training without providing a guarantee across support transitions.

Deviations from code optimality. The encoder-consistency identity itself is exact on the active affine region; approximate code optimality enters when residual-coupling terms are dropped. The new Gram, bias, and gain gradient analyses are derived directly from reconstruction loss and do not assume DS⊤r=0\mathbf D_S^\top\mathbf r=0. Hence, our dynamic analyses do not use per-input code optimality.

It is important to note that sparse-coding analyses often separate inference with a fixed dictionary from dictionary learning, examining how inference accuracy affects learning. SAEs amortize inference through a shared encoder; Proposition 4.1 therefore provides an ideal fixed-support code-optimality reference, not a condition guaranteed by end-to-end training. Figure 2C shows that the gap, E∣DS⊤r∣2\mathbb E|\mathbf D_S^\top\mathbf r|_2, decreases but remains nonzero, helping assess approximations based on this condition without establishing exact per-input optimality or convergence of alternating minimization.

Evidence under practical training. Our main experiments already use minibatch Adam, with supports selected by the sparsifier nonlinearities rather than frozen for the analysis. Some of our main results are using sparsity of k=40k=40, but the evaluation is not restricted to that value: Figure 13 varies TopK support size over k∈{10,40,160}k\in\{10,40,160\} and dictionary width over the settings 4k4\mathrm{k}, 16k16\mathrm{k}, and 64k64\mathrm{k}. We have clarified this diverse range of sparsity, and that our results are not restricted to two active features or impose code optimality conditions.

Figures 2–3 illustrate the implications of our two-feature analysis: effective interference can decline without global dictionary orthogonalization, while the constrained model concentrates pairs with large co-activation-weighted magnitude πijMij\pi_{ij}M_{ij} near ρij=0\rho_{ij}=0. This pattern is consistent with selective pressure on jointly used features, not orthogonalization of every co-active pair.

AuthorsOpenReview ↗

Addressing W3/Q3

[W3/Q3, reconstruction quality and feature utilization]

Thank you for highlighting these factors. We agree that reconstruction quality and feature utilization matter when interpreting architectural comparisons. We therefore examine pairwise profiles of ρij\rho_{ij} versus πijMij\pi_{ij} M_{ij} , rather than only their product IijI_{ij}, alongside aggregate interference normalized by reconstruction energy.

Features that never activate have πij=Iij=0\pi_{ij} = I_{ij} = 0: they contribute neither to aggregate interference nor to the positive-πijMij\pi_{ij} M_{ij} heatmaps. The observed shift toward positively aligned, co-active pairs therefore concerns features actually used in reconstruction (e.g., in Figure 3). Moreover, the pronounced increase in dead features concerns the untied setting, particularly JumpReLU. Tied, tied + bias, and tied + gain models have similar dead-feature rates yet exhibit the same qualitative shift. Increased feature death is therefore not necessary for this pattern, although it can indirectly affect the statistics of surviving features.

We normalize aggregate interference by E∣x^∣22E|\widehat{\mathbf x}|_2^2 to control overall reconstruction scale, not reconstruction error. Interference contributes to reconstruction energy, whereas error also depends on input–reconstruction alignment. Our additional analyses show that their relationship is context-dependent (Table 3): relaxing architectural constraints increases interference while improving explained variance, but across layers, however, lower interference corresponds to better reconstruction. Additionally, Figure 19 also shows some models with similar interference but different explained variance. Thus, interference is not a one-to-one proxy for reconstruction quality.

These results support recurring architectural signatures among co-active features, including comparisons with similar dead-feature rates. We still have added a note in the paper to avoid claiming independence from these factors and interpret the untied JumpReLU comparison cautiously.

Comments on the paper

2 comments
AuthorsOpenReview ↗

(1/2) Summary of Our Rebuttal - See the Uploaded Paper for Full Details

We thank the reviewers for their constructive feedback. The revised manuscript strengthens the theoretical analysis, expands the experiments, and clarifies the interpretation and scope of effective interference. Below, we summarize the main concerns and the corresponding revisions and evidence. New manuscript content is colored in blue.

  1. Theoretical analysis beyond two active features.

We have generalized our two active feature dynamics to Gram deviation dynamics to 2<k<p2<k<p. Proposition G.1 contains the dynamics, and Corollary G.1 gives sufficient conditions for the Gram deviation from identity on the active support to decrease. Importantly, our results do not assume per-input code optimality. In these analyses, we provide intuition on the condition for decrease in Gram error deviation. Brief explanation is now included after Proposition 4.3 with full proof in Appendix G.

  1. Learning dynamics beyond encoder consistency identity and Figure 4.

To address the reviewers’ concerns on the encoder consistency identity, we have included two new gradient dynamics analysis on bias and encoder gain. Proposition H.1 provides the bias contraction toward mean cross-talk compensation, and Proposition I.1 and Corollary I.1 connect gain dynamics and stationarity to signed effective interference. Appendix H provides further intuition on how the bias compensation can change dictionary dynamics. Importantly, these analyses do not rely on code-optimality conditions and consider the gradient dynamics at a training iteration. The manuscript also reflects this in Routes II and III. Finally, we clarify that our architectural ablations goes beyond the consistency test in Figure 4 and evaluate the impact of the explained route mechanisms through changes in learned interference (Figures 3, 5–6, 9–10).

  1. Fixed support, code-optimality deviations, and practical training.

Section 4 states the local, support-preserving scope, and Figure 2C measures the nonzero code-optimality gap. The new gradient results in Appendices G–I do not require per-input code optimality. Experiments use larger supports and practical training procedures (Appendix L). Importantly, we now clarify that fixing the support is a common technique related to support-consistency arguments in sparse coding and autoencoder theory, including analyses of alternating-minimization schemes. Moreover, the fix is a local condition for the analyzed update, not a claim that an input retains its support throughout training. As training progresses, supports may change between iterations and are therefore not restricted to a predefined set throughout training.

  1. Clarifying how to interpret the orthogonalization evidence.

Our results distinguish declining effective interference from global dictionary orthogonalization. Figure 8 complements Figure 2 with joint-profile evolution during training; Figures 3 and 9 show architectural differences, while Figures 11–12 separate conditional magnitude from co-activation frequency. These distributions support a selective-pressure interpretation, not a claim that every co-active pair becomes more orthogonal.

  1. Reconstruction quality and feature-utilization.

We clarified that dead features contribute neither to aggregate interference nor to our heatmaps. Although feature death may indirectly affect the statistics of surviving features, we observe the same qualitative shift across models with different numbers of dead features. We also emphasize that effective interference captures something distinct from reconstruction loss (see point 7) and normalizes it by reconstruction energy to control for scale.

  1. Scaling experimental results beyond one layer and one hyperparameter setting.

We trained and evaluated TopK SAEs at every even layer of Pythia under each architectural constraint. The effect of architecture constraints on effective interference is persistent across layers despite differences in input activation distributions (Figure 7). We also highlight the results across hyperparameter settings in Appendix M.2.

AuthorsOpenReview ↗

(2/2) Summary of Our Rebuttal - See the Uploaded Paper for Full Details

  1. Effective interference’s relation to established interpretability metrics (Appendix N and Results section).

We evaluated all trained SAEs on SAEBench. Overall, net effective interference rises with reconstruction quality across constraints (Table 2), but across layers, lower-interference SAEs reconstruct better (Table 3). It therefore does not simply track reconstruction quality. We also note that net effective interference measures something different in both scale and content from most SAEBench metrics: it aggregates interactions across the full code and dictionary, whereas these metrics assess particular concepts; it captures co-activation and geometry without identifying what a feature pair means. Most SAEBench metrics (Figures 19, 21) thus correlate weakly with interference. Targeted probe perturbation is an exception: better performance tends to accompany lower interference. In point 8 below, we have demonstrated that effective interference is most useful when studied on pairs in its IijI_{ij} form, not the aggregate version.

  1. Preliminary semantic analysis beyond geometry: the value of effective interference (geometry and code statistics)

We added a preliminary analysis of labeled Gemma Scope feature pairs (Appendix P). The results suggest that interference profiles carry information that geometry alone misses. For example, at similar decoder cosine similarity, strongly co-activating pairs tend to resemble duplicate features, whereas pairs that rarely co-activate tend to be weakly related. This directly translates to high and low effective interference. We also find that a feature’s nearest cosine neighbors listed on Neuronpedia include only about a third of its strongest interference partners.


We hope the above summary eases the reviewers to navigate the comprehensive experiments and analyses added to the manuscript. We are happy to address any remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.