acceptodds
Under review as a conference paper at ICLR 2027

Beyond Coordinatewise Inference in Sparse Autoencoders

Abstract

Sparse autoencoders (SAEs) are often used as if each learned coefficient represented an independent, editable concept. We show this assumption is structurally false. Standard SAE encoders, including affine, ReLU, TopK, and Gated SAEs, compute each coefficient from a single affine score of the activation. The selection step in TopK compares coordinates, but each score remains a projection of the input alone. We call these coordinatewise-score encoders. Because SAE dictionaries are overcomplete, such scores cannot each respond to their own feature's decoder direction while ignoring all others, so coefficient errors persist even when the encoder is given the dictionary that generated the data. The errors have two consequences: (1) after a single-coordinate edit is decoded into activation space and the resulting activation is re-encoded, the inferred code change can involve unedited coordinates or even move the target coordinate in the wrong direction, creating attribution ambiguity. (2) During training, the same limitation lets the decoder adapt to compensate for the coefficient error instead of the encoder correcting it, which we call decoder absorption, a mechanism of which feature absorption—a well-known SAE phenomenon—is a special case. We show that a cross-coordinate encoder can overcome this failure, and introduce a local consistency loss that further improves edit consistency and reconstruction quality over standard affine and scalar-nonlinear baselines in the large majority of configurations tested.

Primary area:
interpretability and explainable AI
Keywords:
sparse autoencoders, mechanistic interpretability, feature steering, dictionary learning, feature absorption

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_x3zMOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The authors take a deep dive into the limitations of an affine encoder (linear + bias + sparsification activation) in SAEs. In particular, they show that for overcomplete dictionaries, where it is known that the cosine similarity between at least one pair of concepts will exceed zero, it is not possible to represent arbitrary sparse code distributions with such an encoder. This has some important consequences for downstream applications of SAEs. In particular, activation edits are not self-contained to a single concept. The authors propose a new encoder architecture combined with a regularization term to resolve this, and try to show that it has the desired effect in real language model activations.

Critical strengths

The authors’ core claims are correct and important. The limitations of an affine encoder are known, and yet their use seems to persist in the SAE community. I think a direct and clear exposition of these limitations like in this paper has real value, particularly with a new technique that helps address them.

The majority of the writing is extremely clear. I understood the core of the issue from the start, and each conclusion logically follows from the previous ones. It was, mostly, a pleasure to read.

The theoretical propositions are fairly mechanical and not too difficult; the proofs seem correct to me.

Critical weaknesses

The issues I have with recommending acceptance outright is (a) the originality of the idea, and more importantly (b) the limited experiments, particularly relative to these other works.

(a) The difficulties of affine + sparsifier encoders for amortized sparse coding were discussed as early as Gregor & LeCun (2010), which the authors appropriately cite. There are at least two additional recent papers which also address this problem specifically in SAEs but are not mentioned by the authors [1, 2] which I strongly recommend they cite. With that said, these papers are very recent, and touch on other topics as well, so I’m not sure that their existence alone should count against acceptance. I would recommend the authors review them and consider citing them.

(b) In an experiment on toy-sized SAEs, the authors show that their alternative encoder parameterization better recovers the ground-truth coordinate which would otherwise be mixed, thereby achieving lower reconstruction error as well. In LLM activations, they show that their models can achieve better reconstruction performance (EV) and better pre-post edit re-encoding error (RMS) a good fraction of the time, as opposed to techniques not using their encoder + regularizer.

In my opinion, these experiments are insufficient. For example, consider that [1] did much more extensive experiments, investigating both what they call the “amortization gap” using per-sample methods (which encompasses this affine limitation, but also others, which they also investigate experimentally). [2] also did extensive experiments, in both synthetic regimes of a more practical size than this submission’s and real LLM activation regimes. Importantly, they evaluated using a standard benchmark suite for SAEs, which I think is necessary here too [3], if only to show that the proposed models are reaching the concept-learning regimes relevant to real-world practice.

I think these experiments are necessary, because notably Gregor & Lecun (2010) settle on a fairly domain-specific architecture (namely, an iterated soft thresholding approach), so it would be somewhat surprising and very notable if the authors’ MLP + regularizer works for amortized sparse coding as a simpler architecture. On the other hand, these experiments would likely need to be somewhat more careful than those in the current text: SAEs have convergence which is notoriously difficult to assess, and there’s no evidence the authors did so (no loss curves, and the EV metrics feel surprisingly low for me on the real LM experiments). Furthermore, the main evaluation metric besides EV is the thing being regularized, which I think further supports that SAEBench or similar qualitative metrics would be useful here. Head-to-head comparisons with e.g. MP-SAE (cited in the submission but not compared to) or iSAE-ME [2] would be of substantial interest here as they both aim to tackle the same issue, at least in part.

[1] Barin-Pacela et al. Stop Probing, Start Coding: Why Linear Probes and Sparse Autoencoders Fail at Compositional Generalization. UAI 2026

[2] Nelson et al. Toward Identifiable Sparse Autoencoders. ICML 2026

[3] Karvonen et al. SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability. ICML 2025

Questions for authors
  1. Can you compare your work to [1] and [2] mentioned above to better position it in the literature, which discuss the affine encoder issue to varying degrees?
  2. Can you add more in-depth synthetic experiments of the kind in [1] or [2] or of your own design? If not, why?
  3. Can you add experiments which interrogate the “quality” of the SAE representations in the context of LLMs, such as with SAEBench [3]? If not, why?
  4. Can you add head-to-head experiments which compare to other methods aimed at tackling the same challenge?
  5. Can you add informal statements of your theoretical results to the main text in context so that the readers can better understand them?
3 comments
AuthorsOpenReview ↗

Response to Reviewer x3zM (Part 1/2)

We thank the reviewer for finding the core claims correct and important and the exposition clear. We answer the reviewer's questions and address the weaknesses below. The additions described here are in the revised submission, where changes are marked in red.

Q1. Compare with Barin-Pacela et al. and Nelson et al. to position the work.

Our paper makes three original contributions, none of which appears in Barin-Pacela et al. or Nelson et al.

  1. Where the coefficient error comes from (Section 3). Barin-Pacela et al. show that SAE encoders recover the latent factors less accurately than per-sample sparse inference. Nelson et al. state that a near-linear encoder is not expressive enough even with a fixed dictionary. O'Neill et al. (2025, Theorem 3.1) prove this limitation by a rank argument.

    We are instead the first to derive the error coefficient by coefficient. Each encoded coefficient contains, besides its own term, the contributions of the features active with it, and these contributions are the source of the coefficient error. We also give an exact condition for when a coefficient is recovered and a lower bound on its error.

  2. How the coefficient error affects editing and attribution (Sections 4, 6 and 7). To our knowledge, we are the first to show that the coefficient error compromises the attribution of edits to individual features. Feature labels and the effect attributed to an edited coordinate are derived from encoded coefficients, which contain the contributions of the other active features. Based on the SAE's encoded coefficients, the effect of an edit therefore cannot be attributed to the edited feature alone. We also introduce a check for this, edit consistency (RRMSR_{\mathrm{RMS}}), which can be computed for any trained SAE, and an encoder that improves it, Cross+LC.

  3. How the coefficient error affects the decoder (Section 5, Appendix G). We are the first to identify decoder absorption, which is caused by the coefficient error of point 1. During training, the decoder directions move away from the reference directions to compensate for the coefficient error, even when the active support is correct. This may explain part of the finding of Barin-Pacela et al. that SAE-learned dictionaries point in wrong directions. In addition, feature absorption, a well-known SAE phenomenon named by Chanin et al. (2025), is a special case of decoder absorption, so our analysis also explains one way in which it arises.

The revised submission discusses these works in Section 1 and Appendix A.

Q2. Larger synthetic experiments. We ran synthetic experiments in the setting of Nelson et al. (d = 768, m = 4096, 40 active coordinates per input; their i.i.d. and mixture dictionaries), now reported in Appendix I.4. With the decoder fixed to the true dictionary, the cross-coordinate encoder has a lower coefficient error than Affine. The iSAE-ME-style encoder of Q4, here without the aRIP regularizer and listed as Multistep in Appendix I.4, is nearly exact:

Encoderi.i.d.: relative coef. errori.i.d.: support F1mixture: relative coef. errormixture: support F1
Affine0.3670.8980.3760.891
Cross0.2300.9620.3250.919
iSAE-ME-style0.0031.0000.0051.000

In these experiments, the data are generated from a known sparse dictionary. On LLM activations the true features are unknown and the dictionary must be learned. However, as Barin-Pacela et al. find, SAE-learned dictionaries point in wrong directions, and per-sample inference on such a dictionary does not close the gap they measure. From the synthetic results, we therefore cannot tell whether an encoder with better inference also has higher edit consistency on LLM activations. Q4 tests this directly.

Q3. SAEBench. As the reviewer notes, edit consistency is the quantity that LC regularizes, so metrics not used in training are needed. SAEBench results for the Section 7 SAEs are added as Appendix I.1, with the number of settings in which the encoder is better than Affine in parentheses:

EncoderSparse probing, top-1 accuracy ↑CE increase ↓SCR ↑TPP ↑
Affine0.6900.0170.0850.065
Cross0.695 (8/12)0.015 (9/12)0.084 (7/12)0.061 (5/12)
Cross+LC (λ_LC = 0.1)0.694 (10/12)0.016 (6/12)0.090 (9/12)0.065 (6/12)

Both cross-coordinate encoders stay within 0.006 of Affine on each of these four metrics, so the gain in edit consistency does not come at a cost on metrics not used in training.

AuthorsOpenReview ↗

Response to Reviewer x3zM (Part 2/2)

Q4. Head-to-head comparison. MP-SAE and iSAE-ME address the encoder's limitation by improving sparse inference itself. Our paper studies how the coefficient error affects editing and attribution (Q1, point 2). The comparison therefore tests whether better sparse inference also gives higher edit consistency. Reconstruction and edit consistency are different requirements. Reconstruction requires that decoding the code returns the activation. Edit consistency requires that re-encoding the edited activation x′x' returns the edited code. Since an overcomplete decoder maps different codes to the same or nearly the same activation, an encoder can reconstruct well and still not return the edited code.

We trained an iSAE-ME-style multistep encoder and an MP-SAE (Costa et al., 2025) in the Section 7 TopK settings for 30K steps. The multistep encoder is z(t)=TopK(Sz(t−1)+W(x−b))z^{(t)} = \mathrm{TopK}(Sz^{(t-1)} + W(x - b)) with T=5T = 5 and the aRIP regularizer of Nelson et al. This encoder applies TopK at every step and MP-SAE runs KK matching-pursuit iterations per token, so both are compared in the four TopK settings. Appendix I.2 reports this table and lists the iSAE-ME-style encoder as Multistep+aRIP. Each cell gives EV ↑ / RRMSR_{\mathrm{RMS}} ↓:

EncoderGPT-2, K = 50GPT-2, K = 100OPT-125M, K = 50OPT-125M, K = 100
Affine0.643 / 0.03240.747 / 0.01730.799 / 0.03550.871 / 0.0220
Cross+LC (λ_LC = 0.1)0.643 / 0.02990.746 / 0.01710.815 / 0.03260.884 / 0.0208
iSAE-ME-style0.700 / 0.18270.806 / 0.38350.865 / 0.13860.945 / 0.0983
MP-SAE†0.719 / 2.3980.831 / 3.6780.863 / 1.4350.935 / 2.277

†MP-SAE has no candidate values separate from its code, so its error is computed on the encoded coefficients z^i\hat z_i after selection (RRMS,finalR_{\mathrm{RMS,final}}, Appendix H.2). On that quantity, in the four settings in table order, Affine has 0.231, 0.127, 0.181 and 0.110. The iSAE-ME-style encoder has 0.795, 1.060, 0.450 and 0.241.

Cross+LC is the most consistent encoder in all four settings (bold). In our runs, unlike in the synthetic setting of Q2, the iSAE-ME-style encoder and MP-SAE reconstruct best but are the least consistent. This is in line with Nelson et al.'s observation that iSAE-ME reconstructs best on LLM activations while identifiability worsens.

SAEBench results in the same four settings are in Appendix I.1, with the number of settings in which the encoder is better than Affine in parentheses:

EncoderSparse probing, top-1 accuracy ↑CE increase ↓SCR ↑TPP ↑
Affine0.6890.0130.0950.119
Cross+LC (λ_LC = 0.1)0.692 (4/4)0.012 (2/4)0.111 (4/4)0.122 (2/4)
iSAE-ME-style0.713 (4/4)0.009 (4/4)0.068 (2/4)0.099 (1/4)
MP-SAE0.706 (4/4)0.008 (4/4)0.094 (2/4)0.071 (0/4)

The iSAE-ME-style encoder and MP-SAE are the two best on sparse probing and CE, in line with their higher EV. Both are below Cross+LC on SCR and TPP, which measure the effect of ablating selected latents.

Q5. We thank the reviewer for this suggestion and have followed it. In the revised submission, each of Sections 3–6 opens with a Summary paragraph that states its result informally, before the formal statement.

Convergence and EV (weakness b). We evaluated held-out EV at all saved checkpoints of 84 Section 7 SAEs, one per setting and encoder (Appendix I.3). Mean EV over the 12 settings:

Step1K2K5K10K20K30K
Affine0.6740.6900.6950.6990.6990.707
Scalar0.6730.6880.6940.6990.7000.706
Cross0.6820.6940.7000.7110.7190.727
Cross+LC (λ_LC = 0.1)0.6790.6900.6990.7080.7140.724
Settings with Cross above Affine8/129/129/1212/1212/1212/12

EV still rises slowly at 30K steps, so we do not call the runs converged. However, from 10K steps on, the cross-coordinate encoder has higher EV than Affine in all 12 settings. The lowest EVs are in the sparsest GPT-2 settings (width 4096, about 123M training tokens).

I appreciate the authors’ efforts to address my concerns. Due to the length of the authors’ response, I will summarize their answers to my questions and ask them to please correct me if I am wrong.

Q1. Philosophical comparison

The main contributions over prior work is (1) an exact analysis of the per-coefficient affine encoder gap, (2) the round-trip edit error idea, and (3) the impact of the affine encoder gap on the decoder.

Q2. Synthetic experiments

The authors have added synthetic experiments which demonstrate that their proposed “cross” encoder exhibits better coefficient recovery than an affine encoder when the dictionary is fixed, but worse than iterative encoders (and presumably per-sample encoders like MP-SAE, based on the response to Q4). The benefit over an affine encoder declines somewhat when concepts have more coherence.

Q3/Q4. LLM activations

The authors have added experiments on real-world LLM activations which show that their method does not substantially improve explained variance (i.e. reconstruction MSE) in real-world applications over an affine encoder. One or two SAEbench metrics may improve despite this, which is potentially interesting.

Q5. Convergence

None of the SAEs in the study as presented are converged.

My takeaway

The authors have improved the positioning of their work relative to early literature, but the results are not particularly impactful, and so I cannot recommend acceptance. On soundness, I am also still not convinced that round-trip edit error is a meaningful metric in practice, which is the primary focus of the paper, nor am I confident that presenting results from unconverged SAEs is reasonable.

While the proposed encoder may slightly edge out the typical affine encoders in certain synthetic settings, its performance gains are extremely marginal and of unknown significance in real-world activations. In general the approach is outperformed by iterative and per-sample encoders, which is to be expected from the literature the paper cites. The one exception are two unlearning metrics where there is some evidence of marginal gain; based on my prior knowledge I would anticipate this is due to the MLP’s ability to learn a different feature scale.

I would focus on validating the meaningfulness of the round-trip edit metric (what consequences does a poor round-trip editor have on metrics that people care about when using SAEs?) and standardizing their experiments with the broader literature (non-convergence is not good, for example; 30k iterations seems awfully low). In my view, the theoretical claims (that affine encoders are insufficient) are perhaps unique in their presentation/precise results but not pedagogically novel in light of the extensive literature on this issue, nor do they motivate a particular solution compared to earlier results about the failures of affine encoding.

Reviewer_wwvZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that standard SAE encoders, which compute each latent from a single affine score of the input, cannot exactly recover the coefficients of an overcomplete dictionary, and that this leads to inconsistent edits when SAE latents are used for steering and to decoder absorption during training. The authors propose a nonlinear, cross-coordinate encoder together with a local consistency loss, and evaluate it on synthetic data and on SAEs trained on GPT-2 and OPT-125M.

Critical strengths

None. I could not identify a "substantive strength".

Critical weaknesses

W1. The central argument is not in line with how steering is used. The paper examines the code z′ = Enc(x + D·δe_i), obtained by decoding a latent edit, adding it to the activation and re-encoding, and argues that it does not change by the requested edit alone. What matters for steering, however, is not how the SAE re-reads the edited activation, but the fact that this activation is introduced in an intermediate layer of the model, so that the remaining layers and, in LLMs, the autoregressive generative process absorb the statistical inconsistency introduced by the edit. Given that the steering direction is relevant to a specific concept, a magnitude can be found at which adding it meaningfully changes the output of the whole model. Showing that the re-encoded edit differs from the requested one is therefore not in line with how steering is used in the literature. The same concern applies to the downstream check in Appendix H.5, where the activation is replaced by its SAE reconstruction plus the edit, since SAEs do not excel in reconstruction and there is no reason to expect the reconstructed signal to be of the same quality as the input.

W2. Attribution ambiguity is a misleading notion. Section 4 shows that editing one coordinate changes others upon re-encoding, which is in fact expected. The activations that the SAE receives are polysemantic by definition, with semantics entangled across several dimensions, and the SAE learns directions that disentangle them. When the edited latent is decoded back to activation space, the edit returns to the entangled space, where a linear read-out such as the SAE encoder also picks up the other directions, as the authors themselves prove in Section 3. The ambiguity therefore lies in the SAE's re-reading of the edited activation and not in the intervention, which acts along a single direction, so presenting it as a failure of SAE steering is misleading.

W3. The recovery criterion of Section 3 is out of scope with how SAEs work in practice. In their analysis of why affine encoders cannot recover sparse coefficients, the authors require every active coefficient to be recovered exactly, which forces each encoder row to respond only to its own decoder direction (w_i^T d_i = 1 and w_i^T d_j = 0 for j ≠ i, lines 181 to 184). This is not what SAEs aim for and not how they model concepts. A data point always contains a wide set of concepts that are projected onto a sparse code with several active latents, and the relevant question is whether a latent is active, not whether its value is recovered exactly or is the highest. For this reason many of the claims of the paper, for instance at line 187, are of limited relevance to how SAEs are used in practice.

W4. A nonlinear encoder compromises interpretability. The main contribution of the paper is to make the encoder nonlinear. A linear encoder is, however, vital for interpretability, since the aim is to find semantics that arise as linear combinations of the input activations [1, 2], in line with the success of linear probes in extracting semantic content [3, 4]. If the latent is a nonlinear function of the activations, it can no longer be interpreted in this way, and the paper neither discusses this trade-off nor evaluates interpretability.

W5. Limited evaluation that does not match the motivation. The experiments consist of a synthetic setting and two small-scale LLMs, and the main-text results rely on edit-consistency and reconstruction metrics computed inside the SAE, with small improvements. Since the motivation is steering, better reconstruction is nowhere near evidence of better steering, and the only steering experiment is a single-concept check in Appendix H.5. Several steering evaluations on pretrained LLMs are needed to support the motivation of the paper.

W6. Presentation. The abstract is difficult to read, and the paper lacks the usual organisation, with a terminology table in the main text on page 3, very short sections without subsections, and small unconnected statements one after the other. The related work in the main text is a single paragraph (lines 102 to 110), with the discussion of SAE architectures deferred to Appendix A. The notation also changes throughout the paper, for instance w_i is used at line 118 before it is defined, and a(x) appears only in Section 7.2.

References

[1] N. Elhage et al. Toy models of superposition. Transformer Circuits Thread, 2022.

[2] T. Bricken et al. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023.

[3] G. Alain and Y. Bengio. Understanding intermediate layers using linear classifier probes. arXiv:1610.01644, 2016.

[4] Y. Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207-219, 2022.

Questions for authors

I understand that the critical weaknesses identified in my review are difficult to address within a rebuttal period, and that, if addressed, the resulting paper would differ significantly from the initial submission. However, I write my questions and suggestions here in an attempt to help the authors improve their submission.

Q1 (W1, W5). It would strengthen the paper to evaluate steering on pretrained LLMs across several concepts and to show that the proposed encoder improves steering outcomes, rather than only edit consistency.

Q2 (W3). It would help to clarify how the analysis of Section 3 applies when the goal is to detect whether a latent is active rather than to recover its exact value.

Q3 (W4). It would strengthen the paper to justify the nonlinear encoder with respect to interpretability, for instance with an interpretability evaluation against the affine baseline.

4 comments
AuthorsOpenReview ↗

Response to Reviewer wwvZ (Part 1/3)

We thank the reviewer for the detailed review and the concrete suggestions. We answer the reviewer's questions and address the weaknesses below. The changes described here are in the revised manuscript, where changes from the submission are marked in red.

Q1 (W1, W5) and W2. Steering effects and why re-encoding matters.

When the edit is added to the original activation, as W1 describes, the output depends only on the edited activation x′=x+δd~ix' = x + \delta \tilde d_i and on the remaining layers. Here d~i\tilde d_i is the model direction of latent ii. The encoder is not involved in this process. Our results therefore do not predict that steering fails. They concern how the effect of steering is interpreted and attributed to feature ii.

How the encoder is involved in SAE steering. SAE steering uses the encoded coefficient z^i\hat z_i in three ways.

  • Interpreting the latent. The concept of latent ii is inferred from the inputs on which z^i\hat z_i is large (Bricken et al., 2023; Templeton et al., 2024).
  • Setting the steering strength. In prior work, δ\delta is often set to a multiple of the maximum of z^i\hat z_i (Templeton et al., 2024; Wu et al., 2025).
  • Attributing the output change. The change in the output is attributed to an increase of feature ii by δ\delta, with the other features unchanged.

Without z^i\hat z_i, the SAE would give only a direction, with no concept and no steering strength. A successful steering result does not confirm the attribution. Steering is judged successful when the output shows the concept. This does not check that only feature ii changed.

Why re-encode x′x'. The attribution is a statement about the encoded coefficients of x′x'. In an SAE, these coefficients are computed only by the encoder. Suppose the encoded coefficients of x′x' are not those of xx with feature ii increased by δ\delta.

  • If these coefficients are correct, the steering also changed other features, or feature ii did not change by δ\delta.
  • If they are wrong, the concept and the steering strength were obtained from an encoder that gives wrong coefficients.

Either way, the attribution is not supported. Re-encoding x′x' therefore checks the attribution with the same encoder that was used to find the concept and set the steering strength. The attribution holds when the returned edit Δiret(x,δ)=Enc(x′)−Enc(x)\Delta_i^{\mathrm{ret}}(x,\delta) = \mathrm{Enc}(x') - \mathrm{Enc}(x) equals the requested edit δei\delta e_i. It can fail in two ways.

  • (a) Other features change. Entries k≠ik \neq i of Δiret\Delta_i^{\mathrm{ret}} are not zero.
  • (b) The target changes too little. Entry ii is below δ\delta.
AuthorsOpenReview ↗

Response to Reviewer wwvZ (Part 2/3)

Q1 (W1, W5) and W2 (continued).

Both problems in public SAEs (Appendix I.5). We steer latents of Gemma Scope (Lieberum et al., 2024), the public JumpReLU SAEs of gemma-2-2b. As in AxBench, δ\delta is the latent's maximum activation times a factor, here 1 or 2. The same SAE then re-encodes x′x'. In example sentences, each latent below is active on its own word and not on the other two.

Steering "five" (16k SAE, layer 3), mean change of each latent over 512 tokens of ordinary text:

LatentFactor 1 (δ=28.9\delta = 28.9)Factor 2 (δ=57.7\delta = 57.7)
"five" (steered)+15.8 (54.6% of δ\delta)+43.8 (75.8% of δ\delta)
"six"+8.8 (30.5% of δ\delta)+28.8 (50.0% of δ\delta)
"four"+7.3 (25.4% of δ\delta)+31.7 (54.9% of δ\delta)

At factor 1, "five" increases by only 54.6% of δ\delta, which is problem (b). The latents "six" and "four" together increase by 16.1, more than the 15.8 of "five", which is problem (a). The edit raises the scalar score sk(x)=wk⊤x+bks_k(x) = w_k^\top x + b_k of "six" by exactly δ wk⊤d~i=28.9×0.69=20.1\delta\thinspace w_k^\top \tilde d_i = 28.9 \times 0.69 = 20.1 on every token. Suppose an interpretability study of mathematical reasoning steers "five" on prompts such as "(2 + 3) × 4 / 5 + 6 =". Here four, five and six all affect the answer, and five and four also appear as intermediate results (2 + 3 = 5 and 20 / 5 = 4). The edit also raises the scalar scores of "six" and "four". If the answer changes, the change can come from any of the three numbers, so the study cannot reliably credit it to "five" alone. A larger δ\delta reduces problem (b) but not problem (a). At factor 2, "six" reaches 28.8, above its value of about 21 on the word "six". The number latents of the 65k SAE at layers 6, 13 and 15 behave in the same way. There, another number latent increases by 22% to 28% of δ\delta at factor 1 and by 51% to 60% at factor 2. Numbers give the largest changes we found. When a city, country or color is steered, no other latent of the same kind increases by more than about 22% of δ\delta at factor 1 or about 30% at factor 2.

Steering also makes inactive latents active, which is relevant to Q2. Before the edit, "six" and "four" are active on 0.4% of the tokens. After the factor-1 edit, they are active on 95% and 75%. Steering "London" (16k SAE, layer 20, one of the two gemma-2-2b layers in AxBench) makes "New York City" active on 79.5% of the tokens, against 0.8% before. When "Seattle" (65k SAE, layer 25) is steered, "Seattle" itself increases by 89% of δ\delta. "Vancouver" becomes active on 97.7% of the tokens, against 0% before.

How the paper measures both problems. The edit-return error ϵi(x,δ)=∥Δiret(x,δ)−δei∥2/∣δ∣\epsilon_i(x,\delta) = \lVert \Delta_i^{\mathrm{ret}}(x,\delta) - \delta e_i \rVert_2 / |\delta| (Section 4.2) contains both. Its square is ([Δiret]i/δ−1)2([\Delta_i^{\mathrm{ret}}]_i/\delta - 1)^2, the term of (b), plus ∑k≠i([Δiret]k/δ)2\sum_{k \neq i} ([\Delta_i^{\mathrm{ret}}]_k/\delta)^2, the term of (a). The new Section 4.2, "What re-encoding tests", also states the two cases above.

We agree with W2 that the submission did not distinguish the intervention from how the SAE encodes it. We therefore changed two phrases. The introduction said that the coefficient errors "compromise the reliability of SAE steering, causing the attribution ambiguities". It now says that they "compromise the attribution of steering effects to individual features". Section 7.2 was titled "Edit consistency under steering" and is now titled "Edit consistency of single-coordinate edits". Section 1 now also states that the intervention itself only adds δd~i\delta \tilde d_i to the activation.

Steering outcomes across eight concepts (Appendix I.6). We also ran the evaluation the reviewer asks for, with the edit added to the original activation as W1 describes. We steered eight concepts in GPT-2 and OPT-125M with SAEs of Section 7. An LLM judge gave each generation the concept score and the fluency score of AxBench (Wu et al., 2025). The score of a generation is the harmonic mean of the two. The mean score is 0.350 without intervention. With the best strength for each concept, including no intervention, it is between 0.81 and 0.86 for each of Affine, Cross and Cross+LC. Cross has the highest mean score, 0.863 against 0.806 for Affine.

AuthorsOpenReview ↗

Response to Reviewer wwvZ (Part 3/3)

Q2 (W3). How does Section 3 apply when the goal is to detect whether a latent is active?

We agree that exact recovery of every coefficient is not the goal of SAE training. Section 3 also applies when the goal is only to decide whether a latent is active. The scalar score si(x)=wi⊤x+bis_i(x) = w_i^\top x + b_i decides whether latent ii is active. In terms of the reference code z⋆(x)z^\star(x) and with wi⊤di=1w_i^\top d_i = 1, it equals zi⋆+∑j≠i(wi⊤dj) zj⋆+biz_i^\star + \sum_{j\neq i} (w_i^\top d_j)\thinspace z_j^\star + b_i. The other active features therefore enter the decision as well as the value. Moreover, practice does not use only whether a latent is active:

  • Whether a latent is active is decided by comparing values. TopK keeps a latent only if its score is among the KK largest. JumpReLU and Gated compare values with thresholds.
  • Features are interpreted from the values of the encoded coefficients z^i\hat z_i. The examples used to explain and score a feature are chosen by the value of z^i\hat z_i (Bricken et al., 2023; Templeton et al., 2024; Paulo et al., 2025).
  • Steering, attribution and probing use the values of the encoded coefficients z^i\hat z_i. Steering clamps to multiples of the maximum activation (Templeton et al., 2024). Attribution uses gradient times activation (Marks et al., 2025). Probing uses coefficient values (Karvonen et al., 2025). Kantamneni et al. (2025) find that binarized latents perform worse.

The revised manuscript adds these points in the new paragraph "Uses of coefficient values" of Appendix A.

Q3 (W4). Justify the nonlinear encoder with respect to interpretability.

The linear representation in Elhage et al. (2022) and Bricken et al. (2023) means that an activation is a linear combination of feature directions, x≈∑iz^id~i+bdecx \approx \sum_i \hat z_i \tilde d_i + b_{\mathrm{dec}}. Our encoder keeps this unchanged. The decoder is still linear, each feature is still a model direction d~i\tilde d_i, and its value is still the coefficient of that direction. We change only how the encoder computes the encoded coefficients z^i\hat z_i, not what they mean. Requiring the encoder to be linear as well has a cost under superposition. Suppose dkd_k lies in the span of the directions of the features active together with kk. Then every linear function of xx that computes z^k\hat z_k includes contributions of those features (Section 3 and Proposition 1 in Appendix D). As Barin-Pacela et al. (2026) put it, concepts are "linearly represented in activations but not linearly accessible". Nonlinear encoders have also been used in SAEs before, for example matching pursuit (Costa et al., 2025) and multistep encoders (Nelson et al., 2026). The trade-off is computational cost. The cross-coordinate encoder has about 7.5M encoder parameters versus 3.1M for Affine. In a timing test, training steps of Cross+LC took 9% to 13% longer than those of Affine. Appendix H.1 now reports both numbers.

Weaknesses.

  • W1 to W4 are addressed in Q1 to Q3.
  • W1 (Appendix H.5). We agree. Replacing the activation with its SAE reconstruction does not isolate the edit. The revised manuscript removes this result from Section 7.2. Appendix H.5 keeps the experiment and states its limits, including this one. It no longer presents the result as evidence of better steering. The eight-concept experiment in Q1 (Appendix I.6) adds the edit to the original activation.
  • W5. The revised manuscript adds two evaluations beyond edit consistency and reconstruction. The first is the steering experiment of Q1. The second is SAEBench (Karvonen et al., 2025) in the 12 settings of Section 7 (Appendix I.1). On SAEBench the cross-coordinate encoders are comparable to Affine. Averaged over the 12 settings, Cross and Cross+LC are within 0.006 of Affine on sparse probing top-1 accuracy, spurious correlation removal (SCR), targeted probe perturbation (TPP) and the increase in cross-entropy loss.
  • W6. We agree. The revised manuscript makes these changes:
    • Abstract. It is reorganized and made clearer and easier to read.
    • Terminology table. It is moved from Section 2 to Appendix B, as suggested.
    • Related work. The related-work paragraph of Section 1 is expanded, and Appendix A is rewritten.
    • Notation. Section 2 now defines the encoder row wiw_i at its first use. It also introduces a(x)a(x) together with the candidate values v(x)v(x). We also checked the notation throughout the paper. Symbols that had two meanings in different parts of the paper were renamed, so each symbol now has one meaning. Appendix B lists the notation.
    • Organization. Sections 4, 5 and 6 are now divided into subsections. Each of Sections 3 to 6 opens with a "Summary" paragraph that states its result informally.

I thank the authors for their response and the revised manuscript. The inconsistency revealed by re-encoding is a property of the SAE encoder, and the paper does not show that improving it benefits steering as used in practice (W1). The paper still presents the re-encoding of the edited activation as evidence that steering effects cannot be attributed to individual features (W2). The analysis of Section 3 still relies on exact recovery of every coefficient, which the authors agree is not the objective of SAE training (W3), and the interpretability cost of a nonlinear encoder is not evaluated directly (W4). The evaluation remains limited to small models and does not show a meaningful improvement in steering (W5). I therefore maintain my rating.

Reviewer_s61GOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper challenges the notion of editing through SAE by providing spare changes. Simple and elegant counterexample is provided.

Critical strengths

The theoretical question is important, as well as a negative answer to the main question.

Critical weaknesses

There are weaknesses: if you raise a theoretical question, it needs to be studied deeper, both theoretically and experimentally. From theoretical part, once formulated, the result is trivial; the paper does not reference any classical papers on sparse recovery in compressed sensing, but they can be quite handy in my opinion, what are the conditions on exactness of the sparse recovery; moreover, the encoder can never be simple or affine in this case. This should be at least noted. On the other hand, there is a lot of empirical evidence that editing works, and there needs to be an explicit explanation for that. The numerical experiments in the paper are with old and small models only.

Questions for authors
  1. Can you provide a deeper theoretical study?
  2. Can you extend the experiments and explanation why SAE editing 'often works' in practice?
3 comments
AuthorsOpenReview ↗

Response to Reviewer s61G (Part 1/3)

We thank the reviewer for the positive assessment and for finding the question we raised important. All additions described below are in the revised manuscript, marked in red.

Q1. Can you provide a deeper theoretical study?

Section 3 shows that the error is caused by the encoder's function class. The error remains when the true dictionary is given.

  • Already in the submission: whether exact recovery is possible. Under the assumptions of Proposition 1 (Appendix D.1), the active coefficient zk⋆z_k^\star is recovered by an affine function of xx or a scalar function of one affine score iff dk∉span{dj:j∈Nk}d_k\notin\mathrm{span}\lbrace d_j:j\in N_k\rbrace. Here NkN_k are the coordinates that can be active together with kk.
  • New in the revised manuscript: how large the error must be otherwise. The results below are in the new Appendix D.3, with full proofs.

Setting. All quantities are conditional on kk being active. Write x=zk⋆dk+ux=z_k^\star d_k+u with u=∑j≠kzj⋆dju=\sum_{j\ne k}z_j^\star d_j, and assume:

  • zk⋆z_k^\star is independent of uu, with 0<σk2=Var(zk⋆)<∞0<\sigma_k^2=\mathrm{Var}(z_k^\star)<\infty.
  • ∥dk∥=1\Vert d_k\Vert=1 and Ck=Cov(u)≻0C_k=\mathrm{Cov}(u)\succ0. Let θk=dk⊤Ck−1dk\theta_k=d_k^\top C_k^{-1}d_k.

Proposition 2 (Affine functions). By the standard linear minimum mean squared error computation, min⁡w,b E[(w⊤x+b−zk⋆)2]/σk2=11+σk2θk∈[λmin⁡λmin⁡+σk2, λmax⁡λmax⁡+σk2],\min_{w,b}\ \mathbb E\big[(w^\top x+b-z_k^\star)^2\big]/\sigma_k^2=\frac{1}{1+\sigma_k^2\theta_k}\in\Big[\frac{\lambda_{\min}}{\lambda_{\min}+\sigma_k^2},\ \frac{\lambda_{\max}}{\lambda_{\max}+\sigma_k^2}\Big], with λmin⁡,λmax⁡\lambda_{\min},\lambda_{\max} the extreme eigenvalues of CkC_k.

Corollary 1 (Dimension, number of active coordinates, and coefficient spread). Assume:

  • KK coordinates are active, with 2≤K<m2\le K<m. They are kk and a uniformly random subset of K−1K-1 of the other m−1m-1.
  • All KK coefficients are i.i.d. with mean μ\mu and variance σ2\sigma^2, independent of the support.
  • The other m−1m-1 directions form a unit-norm tight frame with zero sum.

Then Ck=K−1d(σ2+μ2m−Km−2)IC_k=\frac{K-1}{d}\big(\sigma^2+\mu^2\frac{m-K}{m-2}\big)I, and the smallest error is φ1+φ,φ=K−1d(1+μ2σ2⋅m−Km−2).\frac{\varphi}{1+\varphi},\qquad \varphi=\frac{K-1}{d}\Big(1+\frac{\mu^2}{\sigma^2}\cdot\frac{m-K}{m-2}\Big). It increases with KK for K≤m/2K\le m/2 and with μ2/σ2\mu^2/\sigma^2. For d=768d=768, m=4096m=4096, K=50K=50 and μ=σ\mu=\sigma, it is 11% of the coefficient's variance.

Proposition 3 (Scalar functions of one affine score). Fix w≠0w\ne0 and bb, and let ν=w⊤u\nu=w^\top u. Assume that zk⋆z_k^\star and ν\nu have densities with finite differential entropies. Then for every measurable ff (e.g., ReLU, JumpReLU), E[(f(w⊤x+b)−zk⋆)2]σk2 ≥ N(zk⋆)σk2⋅N(ν)Var(ν)⋅11+σk2θk,\frac{\mathbb E\big[(f(w^\top x+b)-z_k^\star)^2\big]}{\sigma_k^2}\ \ge\ \frac{N(z_k^\star)}{\sigma_k^2}\cdot\frac{N(\nu)}{\mathrm{Var}(\nu)}\cdot\frac{1}{1+\sigma_k^2\theta_k}, where N(⋅)=e2h(⋅)/(2πe)N(\cdot)=e^{2h(\cdot)}/(2\pi e) is the entropy power. Each of the first two factors is at most 1, with equality for Gaussian variables.

Proof idea. With s=w⊤x+bs=w^\top x+b, the error of any f(s)f(s) is at least E Var(zk⋆∣s)≥e2h(zk⋆∣s)/(2πe)\mathbb E\thinspace \mathrm{Var}(z_k^\star\mid s)\ge e^{2h(z_k^\star|s)}/(2\pi e) (maximum-entropy bound and Jensen's inequality). Independence of zk⋆z_k^\star and ν\nu and the Cauchy-Schwarz inequality then give the three factors.

If uu is Gaussian, N(ν)=Var(ν)N(\nu)=\mathrm{Var}(\nu) for every ww, and the bound holds uniformly over ww, bb and ff. A scalar nonlinearity can then reduce the affine bound at most by the factor N(zk⋆)/σk2N(z_k^\star)/\sigma_k^2 (0.70 for a uniformly distributed coefficient).

Cross-coordinate encoders. For a function of one score, the smallest error is E Var(zk⋆∣w⊤x+b)/σk2\mathbb E\thinspace \mathrm{Var}(z_k^\star\mid w^\top x+b)/\sigma_k^2. For a function of all of xx, it is E Var(zk⋆∣x)/σk2\mathbb E\thinspace \mathrm{Var}(z_k^\star\mid x)/\sigma_k^2. This error is zero whenever xx determines the sparse code, e.g., when the sparse representation is unique (Donoho & Elad, 2003). Appendix D.2 gives weights of the cross-coordinate encoder of Section 6 that recover the code exactly on the controlled family there.

Relation to classical sparse recovery.

  • Compressed sensing studies when a sparse code can be recovered from xx at all. Its results hold under conditions on the dictionary, for example that no two directions are too similar (Donoho & Elad, 2003; Tropp, 2004; Candès & Tao, 2005). When they hold and the code is sparse enough, solvers such as basis pursuit or OMP recover the code exactly.
  • Propositions 2 and 3 need no condition on the dictionary beyond the Setting, so they also apply to these dictionaries. Exact recovery is possible there, yet every affine function of xx and every scalar function of one affine score covered by Proposition 3 has positive error.

The error is therefore caused by the encoder, not by the dictionary. This is the quantitative form of the reviewer's remark that the encoder cannot be affine. The revised manuscript discusses these works and makes this comparison in Appendix A and Appendix D.3.

AuthorsOpenReview ↗

Response to Reviewer s61G (Part 2/3)

Q2. Can you extend the experiments and explanation why SAE editing "often works" in practice?

Why editing often works. An edit is said to work when its observed effect matches the effect expected from the feature's label. The intervention adds δd~i\delta\tilde d_i to the activation xx, where d~i\tilde d_i is the model direction of coordinate ii. For a behavioral score FF and a small edit, ΔF≈δ ∇F(x)⊤d~i\Delta F\approx\delta\thinspace \nabla F(x)^\top\tilde d_i. The effect therefore depends on d~i\tilde d_i and on the model. It does not depend on whether the encoder recovers coefficients exactly. Our results do not predict that edits fail.

How a successful edit is interpreted. A successful edit is interpreted as follows. Coordinate ii increased by δ\delta, and the other coordinates stayed fixed. Observing the expected effect does not test this interpretation. The interpretation can be wrong in two ways.

  1. The edit also changes coordinates that should stay fixed.
  2. Coordinate ii increases by less than δ\delta.

Both can be measured by encoding the edited activation x′=x+δd~ix'=x+\delta\tilde d_i with the same SAE. The requested edit is δei\delta e_i. The returned edit is Δiret(x,δ)=Enc(x′)−Enc(x)\Delta_i^{\mathrm{ret}}(x,\delta)=\mathrm{Enc}(x')-\mathrm{Enc}(x) (Section 4). In problem 1, the entries k≠ik\ne i of the returned edit are not zero. In problem 2, its entry ii is smaller than δ\delta.

Why the returned edit can measure both problems. The encoder is already used to choose the edit. The label of feature ii is obtained from the inputs on which the encoded coefficient z^i\hat z_i is largest. The value of δ\delta is chosen relative to the values of z^i\hat z_i. Suppose the returned edit differs from the requested edit. There are two cases.

  • The SAE encodes xx and x′x' correctly. Then the intervention changed other features, or it did not change feature ii by δ\delta.
  • The SAE encodes xx or x′x' wrongly. Then encoded coefficients are wrong, and the label and δ\delta were obtained from encoded coefficients.

In both cases the SAE does not support the interpretation above. The label, δ\delta and the returned edit are all obtained with the same encoder. Disregarding the returned edit therefore means distrusting the encoded coefficients from which the edit was chosen.

Both problems in public SAEs (new experiment). We use Gemma Scope (Lieberum et al., 2024), the public JumpReLU SAEs of gemma-2-2b. We add δd~i\delta\tilde d_i to the residual-stream activations of 512 random tokens of ordinary text. As in AxBench (Wu et al., 2025), δ\delta is a factor times the maximum activation of coordinate ii. We use factors 1 and 2. AxBench uses factors up to 5. The same SAE then encodes x′x'. We use only coordinates whose label we checked on sentences. Each coordinate below is active on its own word and not on the other two words.

Editing "five" (16k SAE, layer 3), mean change of each encoded coefficient over the 512 tokens:

CoordinateFactor 1 (δ=28.9\delta=28.9)Factor 2 (δ=57.7\delta=57.7)
"five" (edited)+15.8 (54.6% of δ\delta)+43.8 (75.8% of δ\delta)
"six"+8.8 (30.5% of δ\delta)+28.8 (50.0% of δ\delta)
"four"+7.3 (25.4% of δ\delta)+31.7 (54.9% of δ\delta)
  • Example of problem 1. At factor 1, "six" and "four" together increase by 16.1, more than the 15.8 of "five". Before the edit they are active on 0.4% of the tokens. After it they are active on 95% and 75%. At factor 2, "six" increases by 28.8. Its encoded coefficient on the word "six" in ordinary sentences is about 21.
  • Example of problem 2. "five" increases by 54.6% of δ\delta at factor 1 and by 75.8% at factor 2.
  • Other SAEs and concepts. Number coordinates of the 65k SAE at layers 6, 13 and 15 behave in the same way. There, another number coordinate increases by 22% to 28% of δ\delta at factor 1 and by 51% to 60% at factor 2. When a city, country or color is edited, no other coordinate of the same kind increases by more than about 22% of δ\delta at factor 1 and about 30% at factor 2. For example, editing "Seattle" (65k SAE, layer 25) increases "Vancouver" by 15.3% of δ\delta at factor 1. "Vancouver" is then active on 97.7% of the tokens, compared with 0% before the edit.

In every coordinatewise-score encoder, including JumpReLU, the edit changes the scalar score sk(x)=wk⊤x+bks_k(x)=w_k^\top x+b_k of coordinate kk by exactly δ wk⊤d~i\delta\thinspace w_k^\top\tilde d_i on every input (Eq. (6) for the affine encoder). For "five" and "six", wk⊤d~i=0.69w_k^\top\tilde d_i=0.69.

The revised manuscript states the two cases above formally in the new Proposition 4 (Appendix D.4) and summarizes them in the new Section 4.2, "What re-encoding tests". The details of this experiment are in the new Appendix I.5.

AuthorsOpenReview ↗

Response to Reviewer s61G (Part 3/3)

Q2. Can you extend the experiments and explanation why SAE editing "often works" in practice? (continued)

Labels, attribution and ablation. These three uses of an SAE also depend on the values of the encoded coefficient z^i(x)\hat z_i(x). The scalar score si(x)s_i(x) from which z^i(x)\hat z_i(x) is computed contains the coefficients of the other active coordinates (Eq. (4), Section 3). Whether coordinate ii is active also depends on them, because ReLU thresholds and TopK selection are applied to the same scores.

  • Labels. The label of feature ii is inferred from the inputs on which z^i\hat z_i is large (Bricken et al., 2023; Templeton et al., 2024). An input can have a large z^i\hat z_i because other coordinates are active on it. The inputs with the largest z^i\hat z_i are then not always the inputs with the largest reference coefficient zi⋆z_i^\star. The next experiment measures how well the two rankings agree.
  • Attribution. The effect of coordinate ii on the output is estimated as a gradient multiplied by the difference of z^i\hat z_i between two inputs (Marks et al., 2025). This difference contains the coefficients of the other active coordinates, so they change the effect attributed to coordinate ii.
  • Ablation. Ablation subtracts z^i(x)d~i\hat z_i(x)\tilde d_i from xx to set feature ii to zero. Figure 1 does this with the true dictionary and a trained affine encoder (Appendix F.4). Each input has two active features, 0 and JJ, with coefficients between 1 and 2. On average over five seeds, z^0=−0.299+0.800 z0⋆+0.400 zJ⋆\hat z_0=-0.299+0.800\thinspace z_0^\star+0.400\thinspace z_J^\star. The coefficient of feature 0 that remains after the ablation is z0⋆−z^0z_0^\star-\hat z_0. It ranges over about ±0.3\pm0.3. It is negative when zJ⋆z_J^\star is large and positive when zJ⋆z_J^\star is small. The cross-coordinate encoder of Section 6 removes feature 0 exactly.

Error of the encoded coefficients (new experiment). Public SAEs have no known reference code, so the experiment on Gemma Scope cannot determine which of the two cases holds. This experiment uses synthetic activations with a known reference code z⋆(x)z^\star(x), where the error of the encoded coefficients z^i(x)\hat z_i(x) can be measured. The results are in the new Appendix I.7.

  • Setup. The activations are generated from a known reference decoder DD (d=768d=768, m=4096m=4096, 40 active coordinates per input, one seed). We train a standard TopK SAE (affine scores, K=40K=40) with its decoder fixed to DD, so all errors are caused by the encoder.
  • Result. Over the inputs with zi⋆(x)>0z_i^\star(x)>0, the rank correlation between z^i(x)\hat z_i(x) and zi⋆(x)z_i^\star(x) is only 0.68 on average over the coordinates.
  • Edit with a known answer. With the same generator and the same kind of SAE, we add δdi\delta d_i to xx for a coordinate ii with zi⋆(x)>0z_i^\star(x)>0, where ∣δ∣\vert\delta\vert is between 10% and 30% of zi⋆(x)z_i^\star(x). The reference code of x′x' is then exactly z⋆(x)+δeiz^\star(x)+\delta e_i. On average over three seeds, entry ii of the returned edit differs from δ\delta by 47% of ∣δ∣\vert\delta\vert, and the other entries have norm 1.46 ∣δ∣1.46\thinspace \vert\delta\vert. In 7% of the edits, entry ii does not have the sign of δ\delta (Appendix I.4).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.