acceptodds
Under review as a conference paper at ICLR 2027

Representation Redistribution in Sparse Autoencoders

Abstract

Sparse autoencoders (SAEs) decompose language model activations into sparse features, but related prompts can redistribute activity across these features even when the underlying request is preserved. Tracking isolated features can miss such distributed responses, while aggregate representation distances do not identify which changes recur or whether they matter downstream. We introduce the Redistribution Sensitivity Score (RSS), which identifies large feature responses that recur in a consistent direction, and evaluate whether the resulting supports persist on variants excluded from selection. Across 27 pretrained SAEs spanning eight language models, controlled prompt transformations produce a reproducible redistribution profile that persists on held out variants. We then connect these recurring representation changes to fixed linear readouts. An exact contribution factorization separates response magnitude, readout weight, directional alignment, and clean margin, showing why persistent redistribution need not imply decision failure. In two frozen safety readouts, combining RSS support erosion with the full clean margin yields recurrent failure AUROCs of 0.935 and 0.958 using median supports of 20 and 19 features, compared with 0.965 and 0.964 for the full readouts. These results provide a compact way to identify reproducible sparse representation changes and quantify their downstream consequences without equating representation change with failure.

Primary area:
interpretability and explainable AI
Keywords:
sparse autoencoders, mechanistic interpretability, representation robustness, feature attribution, prompt sensitivity, sparse representations, model auditing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vnCVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper establishes a new metric, the redistribution sensitivity score (RSS). The score tests/selects features whose activation changes relative to the clean prompt are large, recurrent, and directionally consistent, such as when adding unnecessary context. The MeanRSS summarizes the average magnitude of the absolute change in the selected features. The main experiment is conducted across 27 SAEs of 5 model families. They test if the selected support set still holds under a held-out validation variant. Finally, a linear probe contribution analysis demonstrates how response magnitude, readout coefficients, directional alignment, and clean margin affect a fixed readout.

Critical strengths

Very detailed method section

The paper provides a very detailed description of how their method works, with additional proofs in the Appendix. The downside is that, because of the extensive formalism, the section is sometimes hard to follow. However, I would not treat that as a weakness.

Extensive experiments

The experimental design covers up to 27 SAEs that are evaluated (Sec. 4.1) and more in-depth ablations on 2 SAEs (Sec. 4.2-4.4). Additional analysis is also included in the Appendix. The thorough testing and variety of SAEs strengthen the evidence that the reported patterns are robust across the evaluated configurations.

Critical weaknesses

Underdeveloped scientific interpretation and significance

Although the experiments and their results are extensive, their discussion and interpretation remain underdeveloped. Currently, Sec. 5 summarizes the results but does not sufficiently put them into context, e.g., what does RSS mean for SAEs, or how will it shape future work related to SAEs and/or mechanistic interpretability? This also ties back to the introduction, where, to me, it was unclear why we were establishing a new metric, which made the initial pass at the paper hard to understand. The discussion of limitations is currently insufficiently developed, a more explicit discussion of limitations and implications would be valuable. My central concern is therefore not the number of experiments, but the lack of a clear connection between the measured phenomenon and a clearly articulated scientific takeaway.

Reproducibility

In some parts, e.g., Sec. 4.1, details for reproducibility are missing, such as the dataset creation. Appendix S3 states that 1000 prompts are generated, but no system/user prompt is given. The paper describes the model used, temperature, acceptance criteria, and high-level transformation descriptions, however, how exactly the different variants of a prompt are created/transformed is not specified.

Questions for authors

Clarification

  • How does monosemanticity relate to representation redistribution, and how does RSS position itself relative to monosemanticity metrics, e.g., FMS [1]? Are those complementary measurements? What can RSS tell us about the quality of an SAE?
  • In Fig. 2, the Pythia SAEs seem to be outliers, especially for "Irrelevant context" and "Metalinguistic wrapper." Is there a reason why this is the case? How should this pattern be interpreted, and is there evidence explaining the difference?
  • Could the authors provide the creation details of the dataset used in Sec. 4.1 and also provide some samples for comparison? How were the acceptance criteria applied?

[1] Härle, Ruben, et al. "Measuring and guiding monosemanticity." NeurIPS (2025)

Reviewer_frGiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors propose a method (RSS) to understand which features behave consistently in a direction across perturbed prompts (whose meaning remains consistent), then keep a top K that are directionally self-consistent beyond some threshold. They characterize this through a series of theoretical propositions and empirically characterize its behavior, showing that (i) ~20 selected features explain ~60% of the variance of held-out logit changes (S4.4) and (ii) this on top of clean margin ranks recurrent probe failures nearly as well the full set (S4.3).

Critical strengths

[S1] Assuming experimental validity, they demonstrate that a small number of self-consistent features explain most of the variance in held-out logit changes in safety probes.

[S2] Representation-level evaluation is broad and experimentally strong, i.e. 27 pre-trained SAEs over 5 model families up to 8B parameters.

Critical weaknesses

[W1] There is a lot of theoretical machinery for what seems like a reasonably intuitive process: (i) select top-K features that move in the same direction in 4 of 5 prompt variants, (ii) re-sum weights for these top K, and (iii) rank the risk of margin dropping below a threshold. The paper would be better served by simplifying that machinery.

[W2] For all of this machinery, the gain of RSS-based methods over clean margin/magnitude-only baselines seems marginal and the functional study is quite narrow and weakly supported, e.g. Table S22 shows a gain of 6-8pp over magnitude-only, only two models (S18) only one frozen safety probe per model (S18). Even so, I have questions about the experimental validity of these results, e.g. why wasn't the functional study run on the other models, variability across different seeds, fits, λ\lambda, etc.?

Questions for authors

[Q1, selection] How were Gemma 3 4B L17 and Llama 3.1 8B L24 chosen, and from which candidates? Was any choice made after viewing transformed results? Please report functional results for all eight models, and variability across seeds, splits, and λ\lambda.

[Q2, payoff] Beyond mechanism, what does RSS enable? e.g. does masking or regularizing the recurring features during probe training improve worst-case robustness over readout-aware and random baselines?

3 comments

Dear Reviewer frGi,

We would like to respectfully offer a reminder and invite you to join the further discussion. We believe that most concerns have been reasonably addressed or substantially resolved. If you have a different view, we can give further clarification. Alternatively, we would also welcome you to raise the score or adjust the confidence.

We would also like to share important new experimental results that may change your view of the paper. Following your suggestions, we substantially extended the functional evaluation across models, readout configurations, and data splits to establish the broader significance and reliability of RSS.

  • Functional validation across 27 SAEs and eight models. RSS residual margins improve recurrent failure AUROC over clean margins in every evaluated unit. The median AUROC increases from 0.867 to 0.951, approaching 0.970 for the full readout.
  • Substantial contributions beyond simple baselines. RSS-selected features achieve mean held-out contributions of 0.240 versus 0.110 for Top-|w| on Gemma and 0.370 versus 0.188 on Llama. These results demonstrate that recurring activation changes identify functional information beyond static coefficient magnitudes.
  • Robustness and practical applications. Across 11 regularization settings, RSS residual-margin AUROC ranges from 0.915 to 0.952 for Gemma and 0.930 to 0.962 for Llama. RSS also supports functional localization, reversing 72.2% and 71.8% of transformed probe errors in the two models.

We have additionally clarified the model selection protocol, evaluated variability across converged data splits, and simplified the theoretical presentation.

Full experimental details are provided in our responses and revision. We look forward to your reply and will be happy to answer any further questions.

Thank you for the detailed rebuttal. Clarifying how layers were chosen helps mitigate my selection-bias concern, but my main concern is still how little RSS enables in practice. Masking or regularizing RSS features didn't improve robustness (your rebuttal), MeanRSS sampling doesn't give a consistent AUROC gain (Table S34), and for auditing a support selected for a different transformation captures nearly as much as RSS (Table S33). I'm keeping my score.

Dear Reviewer frGi,

Thank you for your follow up. We appreciate your careful consideration, but we would like to emphasize an important distinction regarding the practical value of RSS.

The central contribution of RSS is functional localization, not improving robustness through feature suppression. Our experiments directly establish this utility:

  • RSS identifies substantially more relevant features than simple baselines. Without using readout information for selection, RSS achieves approximately twice the mean held out functional contribution of Top-|w|, with 0.240 versus 0.110 for Gemma and 0.370 versus 0.188 for Llama.
  • These features explain actual downstream changes. Supports of approximately 20 features reconstruct held out logit changes with R² of 0.579 and 0.627. Restoring these features reverses 72.2% and 71.8% of transformed probe errors.

Regarding Table S33, the auditing metric combines support contributions with the same full clean margin. Similar failure ranking therefore does not imply equivalent feature-level explanations. Likewise, the mixed robustness results do not negate the demonstrated value of identifying and explaining functional changes.

We believe these results establish a meaningful practical contribution beyond merely observing representation changes. We would greatly appreciate it if you could reconsider your assessment in light of this distinction.

Thank you again for your time and consideration.

Reviewer_J3UbOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes the Redistribution Sensitivity Score (RSS) to identify SAE features with large, recurrent, and directionally consistent activation changes across related prompt transformations. Empirically, (1) RSS-selected features preserve the differences from prompt transformation on held-out variants better than baseline selection methods; (2) RSS-selected features can have some contribution to linear readout without using readout information; (3) RSS-selected features can be used to predict linear margin erosion and classification error; and (4) RSS-selected features can be used to reverse linear classification error and identify errors.

Critical strengths
  • The approach is simple and computationally efficient, with some theoretical justifications.
Critical weaknesses
  • The downstream utility of RSS on linear classifier does not seem strong compared to the simple approach of selection based on the coefficient magnitudes ∣w∣|w|. The false negative captured rates in Figures 6C-D show no substantial differences between top ∣w∣|w| vs. RSS mresSm_{\text{res}}^S. Top ∣w∣|w| should be included as a comparison in Figures 4B, 5A, 6A, and 6B. Furthermore, the sensitivity of the results should be evaluated with respect to varying classifier sparsity. Finally, the downstream utility is restricted to only linear classifier and not applicable to LLM generation.
  • The clarity of the paper needs improvement. Intuition needs to be provided for each design choice of the method and for each metric. Implications need to be discussed for each result in Section 4.
Questions for authors
  • How does RSS selection compared to top-∣w∣|w| selection for the settings in Figures 4B, 5A, 6A, and 6B?
  • Do the results on downstream utility hold with varying sparsity of the linear classifier?
  • Do the results hold for LLM generation instead of linear classification readout?
  • What are the motivations for the metrics discussed in this paper?
  • What are the implications for the results in Section 4?
2 comments

Dear Reviewer J3Ub,

We would like to respectfully offer a reminder and invite you to join the further discussion. We believe that most concerns have been reasonably addressed or substantially resolved. If you have a different view, we can give further clarification. Alternatively, we would also welcome you to raise the score or adjust the confidence.

We would also like to share important new experimental results that may change your view of the paper. To address your concerns about downstream utility and comparisons with simple coefficient-based selection, we conducted additional functional evaluations and interventions on the language model's own output.

  • Stronger functional contributions than Top-|w| selection. At matched support sizes, RSS-selected features achieve mean held-out contributions of 0.240 versus 0.110 for Gemma and 0.370 versus 0.188 for Llama. RSS identifies approximately twice the realized contribution without using readout coefficients for selection.
  • Consistent downstream relevance across classifier sparsity. Across 11 regularization settings, RSS residual-margin AUROC ranges from 0.915 to 0.952 for Gemma and 0.930 to 0.962 for Llama. RSS also reverses more transformed probe errors than Top-|w| selection in both models.
  • Direct influence on LLM outputs. Restoring RSS-selected features in Gemma 3 4B recovers 12.5% of the KL divergence between clean and transformed next-token distributions, compared with 6.0% for random active features, across 18,000 held-out variants.

We have also revised the presentation to clarify the motivation for each metric and the functional implications of RSS.

Full experimental details are provided in our responses and revision. We look forward to your reply and will be happy to answer any further questions.

The rebuttal addressed the concerns I raised, so I have raised my score from 1 to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.