acceptodds
Under review as a conference paper at ICLR 2027

Unstable Features, Reproducible Subspaces: Understanding Seed Dependence in Sparse Autoencoders

Abstract

Sparse autoencoders (SAEs) are widely used to interpret neural network representations, but their utility depends on whether the learned features are reproducible across training runs. We study this question through feature stability: for each SAE feature, we estimate the probability that a similar feature reappears in an independently trained SAE. This yields a scalable per-feature signal that separates stable from unstable features. In a large-scale study across seeds, models, layers, dictionary sizes, and SAE variants, we find a pronounced functional asymmetry: stable features carry most of the reconstruction- and prediction-relevant signal, while unstable features have weak marginal impact and are dominated by low-frequency surface-form triggers in both activation statistics and automatic explanations. Geometrically, unstable features are individually non-reproducible but concentrate in reproducible lower-rank subspaces, suggesting that seed dependence often reflects basis ambiguity within a shared region of activation space rather than pure noise. A controlled synthetic model makes this mechanism explicit, showing that low-rank ground-truth features can be recovered at the subspace level while remaining non-identifiable as individual SAE latents across seeds. Finally, by pooling unique cross-seed features, we construct SAEs from stable features and find no evidence of a trade-off between stability and explained variance in this setting. Together, these results support cross-seed stability as a practical faithfulness filter and null check for SAE-based interpretability.

Primary area:
interpretability and explainable AI
Keywords:
sparse autoencoders, mechanistic interpretability, feature stability, seed dependence, feature reproducibility, feature geometry, low-rank subspaces, feature identifiability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mXy5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors systematically study the cross-seed feature instability phenomenon in Sparse Autoencoders (SAEs), using reappearance probability as a metric to evaluate feature stability. They reveal that stable features primarily carry the vast majority of the reconstruction- and prediction-related signal, whereas unstable features correspond to low-frequency surface-form signals. Through further geometric analysis, they find that this instability mainly arises from basis vectors within a shared low-rank subspace, rather than from noise. Feature stability in SAEs has long been a relatively important problem in the field; the authors’ metric is simple and practical, the experimental setup is complete and convincing, and the work provides useful guidance for optimizing SAEs.

Critical strengths
  1. The authors cleverly use reappearance probability as a metric to evaluate cross-seed feature stability, clearly separating stable and unstable features. Combining masking experiments and geometric analysis, they demonstrate that stable features carry the majority of the signal, while unstable features correspond to low-frequency surface-form signals. This finding offers useful insights for subsequent research on knowledge editing with SAEs. The work forms a closed loop from hypotheses and experimental setup to conclusions, and is reasonably convincing.
  2. The authors propose a cross-seed feature-pooling construction method and show that stability and explained variance need not involve a significant trade-off. I believe this method and finding provide a promising direction for improving feature selection in future SAEs.
Critical weaknesses
  1. The major geometric analysis is mainly based on TopK SAEs trained on the residual stream of GPT-2 layer 7. I am curious about the systematic ranking of feature stability across different SAE variants. I also wonder whether there is some trade-off between feature stability and the sparsity–reconstruction Pareto frontier.
  2. Every latent in an SAE contributes to both sparsity and reconstruction, but the authors do not consider the contribution to sparsity. One concern I have is that stable features contribute more to reconstruction while unstable features contribute more to sparsity. When the sparsity constraint is strengthened (e.g., by reducing dictionary width or increasing the coefficient of the sparsity regularizer in the loss), the SAEs may tend to zero out latents that contribute less to reconstruction and more to sparsity (i.e., the unstable features), as discussed in “The Price of Amortized Inference in Sparse Autoencoders,” in order to get as close as possible to the sparsity–reconstruction Pareto frontier.
Questions for authors
  1. Could you provide a more systematic ranking of feature stability across different SAE variants (e.g., Vanilla, JumpReLU, HierarchicalTopK, BatchTopK)? Additionally, do you observe any systematic trade-off between feature stability and the sparsity–reconstruction Pareto frontier? That is, does SAE sacrifice feature stability in order to maximize the Pareto front of sparsity-reconstruction?
  2. Every SAE latent contributes to both sparsity and reconstruction, yet the current analysis focuses mainly on reconstruction impact. Have you examined the contribution of stable versus unstable features to the sparsity objective? In particular, when the sparsity constraint is strengthened (by reducing dictionary size or increasing the sparsity regularizer coefficient), do the unstable features tend to be preferentially zeroed out, consistent with the mechanism discussed in “The Price of Amortized Inference in Sparse Autoencoders”?
4 comments

Thank you for your thoughtful review and for recognizing the value of the stability analysis and feature-pooling construction. Your questions about architecture comparisons and sparsity motivated the additional experiments presented below.

Could you provide a more systematic ranking of feature stability across different SAE variants (e.g., Vanilla, JumpReLU, HierarchicalTopK, BatchTopK)? Additionally, do you observe any systematic trade-off between feature stability and the sparsity–reconstruction Pareto frontier? That is, does SAE sacrifice feature stability in order to maximize the Pareto front of sparsity-reconstruction?

We evaluate five SAE architectures on the same held-out GPT-2 L7 activations, with 16,384 features and eight seeds per configuration, each trained on 1B tokens. At comparable sparsity, the observed stability ranking is Vanilla > HierarchicalTopK > JumpReLU > BatchTopK ≈ TopK. Here, a feature counts as stable if its decoder has a match at cosine ≥0.7 in all seven other seeds.

SAEMean L0EVStable features
Vanilla60.183.31%84.8%
HierarchicalTopK64.087.49%71.1%
JumpReLU58.388.35%62.4%
BatchTopK64.089.25%52.2%
TopK64.089.03%52.1%

We observe a stability–reconstruction trade-off among these configurations: near L0=60–64, the more stable Vanilla and HierarchicalTopK configurations have lower EV. The revised Appendix C.1 plots reconstruction against sparsity for all 12 evaluated configurations, with stability indicated by color. The observed EV–L0 frontier contains TopK, BatchTopK and JumpReLU configurations. The more stable Vanilla and HierarchicalTopK configurations lie outside this frontier. In particular, JumpReLU achieves higher EV at lower L0 than both configurations shown above, while having lower feature stability. We restrict this conclusion to the evaluated architectures and operating points.

The submitted paper introduces and evaluates our pooling construction as a way to improve stability with near-baseline reconstruction (Section 5.3, Figure 3). The rebuttal strengthens that evaluation with two additional controls: disjoint source pools and matched training budgets. We compare two dictionaries constructed from disjoint pools of ten 1B-token SAEs with two ordinary TopK SAEs trained for 10B tokens each. All dictionaries have 16,384 features and k=64, and receive identical 2M-token post-training.

MethodAgreement between its two dictionariesMean EV
Pooling from disjoint groups of ten 1B-token SAEs87.5%0.9303
Ordinary 10B-token TopK78.8%0.9322

Agreement here counts one-to-one decoder matches at cosine ≥0.7. Pooling gains 8.7 percentage points of agreement at a cost of 0.19 percentage points of EV, at the same dictionary size, activity budget and aggregate source-training token budget (Appendix B.15). This quantifies the reconstruction cost of the stability improvement in our construction.

Every SAE latent contributes to both sparsity and reconstruction, yet the current analysis focuses mainly on reconstruction impact. Have you examined the contribution of stable versus unstable features to the sparsity objective? In particular, when the sparsity constraint is strengthened (by reducing dictionary size or increasing the sparsity regularizer coefficient), do the unstable features tend to be preferentially zeroed out, consistent with the mechanism discussed in “The Price of Amortized Inference in Sparse Autoencoders”?

Our additional experiments support the selection tendency suggested in your question: unstable features provide less marginal reconstruction benefit per unit of measured sparsity cost, and stronger sparsity constraints preferentially suppress their activity.

Reconstruction benefit per sparsity cost. Across all five architectures, masking stable features causes 2.1–5.8× greater EV loss per unit of removed cost, with the stable advantage in every evaluated seed. We pair stable and unstable features by their cost on calibration data, then draw three paired masks of 32 features per group per seed. On held-out text, we normalize each mask’s EV loss by its actual removed cost. Cost is summed activation magnitude with unit-normalized decoders for Vanilla, and nonzero activation count for the other architectures. In Vanilla, stable features account for 86.5% of total activation magnitude, versus 0.4% for unstable features. These ablations measure marginal reconstruction benefit under the existing codes, with the remaining activations held fixed (Appendix B.10).

Controlled increase in the L1 penalty. Four Vanilla SAEs trained without auxiliary loss are each continued with either their original coefficient, λ=0.0024, or λ=0.003, using identical fresh token streams. We match the original stable and unstable groups by calibration firing frequency and keep these groups fixed throughout continuation. After 100M additional tokens:

L1 coefficientStable firing retainedUnstable firing retained
Original: 0.0024100.1%99.9%
Increased: 0.00379.3%72.3%

The same ordering appears after 2M and 20M tokens. Thus stronger regularization preferentially reduces unstable-feature firing, even after matching initial firing frequency. Every tracked unstable feature still fires on the test sample after 100M tokens: at this penalty strength, we observe preferential attenuation without complete inactivity of the tracked unstable latents (Appendix C.3).

Hard activation budgets show the same tendency. Starting from eight TopK SAEs trained at k=64, we match the original stable and unstable groups by firing frequency and track them at lower k. At k=16, stable and unstable features retain 26.4% and 13.5% of their original firing counts immediately, and 29.9% and 15.0% after 2M tokens of adaptation. The larger reduction for unstable features holds in all eight seeds at k=16, 32 and 48. The immediate comparison shows that unstable activations more often fall below the stricter selection threshold. The difference persists after brief adaptation (Appendix C.2).

Dictionary capacity provides complementary evidence. For source dictionaries of width 32,768, recovery by independently trained target dictionaries falls from 95.1% to 62.2% for stable directions and 10.5% to 2.3% for unstable directions when target width is halved. All dictionaries retain k=64. This measures sensitivity to reduced dictionary capacity; the within-SAE interventions above directly measure changes in the activity of the original feature groups (Appendix G.1).

Thank you again for your constructive suggestions, which helped us examine more directly how sparsity constraints affect stable and unstable features.

Thank you for the detailed rebuttal and the additional experiments, which address my main interests. However, given that there have already been many studies addressing the issue of cross-seed consistency [1, 2] (even not SAE), and the conclusions drawn in this paper may not be particularly novel and appear intuitive. Identifiability theory shows that when latent components form irreducible low-dimensional groups, only the subspace they span is identifiable, not the basis within it [3]. In overcomplete sparse dictionary learning, standard uniqueness guarantees likewise fail when more atoms than dimensions share a low-rank subspace [4]. In SAEs, this would be expected to manifest as cross-seed inconsistency of individual latents alongside reproducible subspaces. Thanks to the author's hard work and endeavor, I decided to maintain the score.

[1] Convergent Learning: Do different neural networks learn the same representations? [2] SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability [3] Uniqueness of Linear Factorizations into Independent Subspaces, J. Multivariate Analysis [4] When Can Dictionary Learning Uniquely Recover Sparse Data from Subsamples?, IEEE Trans. Inf. Theory

Thank you for the follow-up. We will add an explicit discussion of all four papers to Related Work. They provide relevant connections, but their assumptions and claims differ substantially from those investigated here.

In particular, sparsity is central to the identification problem: a change of basis that preserves a subspace generally changes the sparsity of its codes. Consequently, shared representations across neural networks and identifiability under an independent-subspace model do not, by themselves, establish the behavior of sparsity-constrained SAE decompositions.

Prior workRelevant connectionDifference in assumptions and claims
[1] Convergent LearningReports individual-feature disagreement alongside shared representations across independently trained networks.Studies hidden representations of independently trained convolutional networks. Its sparse prediction method constrains the cross-network alignment map; it does not study identifiability under a sparse-coding objective. Our setting involves independently trained SAE dictionaries decomposing a fixed model’s activations, where code sparsity constrains admissible decompositions.
[2] SVCCAIntroduces an SVD–CCA method for comparing representations, with applications to learning dynamics and distributed representations.Provides a comparison method and empirical applications. It does not establish that individually unstable SAE features concentrate in reproducible subspaces, that this overlap exceeds appropriate dictionary and activation-space controls, or that recurrence predicts SAE feature utility.
[3] Uniqueness of Linear Factorizations into Independent SubspacesEstablishes uniqueness results for decompositions into statistically independent, irreducible random-vector blocks.Its identifying assumptions concern independence and irreducibility. These differ from sparsity constraints, and we have not assumed or established them for the unstable SAE population. The theorem therefore does not directly imply our empirical findings about sparse SAE decompositions.
[4] When Can Dictionary Learning Uniquely Recover Sparse Data from Subsamples?This is the closest theoretical connection: it gives conditions for unique recovery of sparse dictionaries and codes.Its claims concern uniqueness under spark and sampling conditions. It does not investigate cross-seed SAE training outcomes or establish shared subspaces specifically among unstable learned atoms. Moreover, overcompleteness alone does not invalidate its guarantees, and failure of sufficient conditions does not establish that only the subspace is recoverable.

Our geometric result is an empirical finding about the unstable population selected by cross-seed SAE recurrence. We test its subspace overlap against size-matched stable/random dictionary subsets and activation PCA, and examine partial functional correspondence through held-out cross-seed prediction. These findings require evidence beyond the general possibility of basis ambiguity.

We demonstrates practical consequences. At matched firing frequency, stable-feature removal causes 3.95–4.57× greater cross-entropy damage on GPT-2 L7. Separately, disjoint-pool construction decreases the fraction of unstable features from ~7% to ~0%, with explained variance changing from 0.9322 to 0.9303 at matched width, activity, and aggregate source-training tokens. These results quantify practical consequences that the general shared-subspace perspective leaves open.

We appreciate your engagement and your acknowledgment that the additional experiments address your main questions. If this clarification also addresses your remaining concern about novelty, we would be grateful if you would consider raising your score to reflect the concerns resolved during the discussion.

[1] Convergent Learning: Do different neural networks learn the same representations?

[2] SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability

[3] Uniqueness of Linear Factorizations into Independent Subspaces, J. Multivariate Analysis

[4] When Can Dictionary Learning Uniquely Recover Sparse Data from Subsamples?, IEEE Trans. Inf. Theory

Reviewer_xBq3OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper is an empirical study of feature stability in sparse autoencoders (SAEs). The authors divide SAE features into stable features, which are recovered across seeds, and unstable features, which are not recovered well across seeds. The paper mainly characterizes stable and unstable features in SAEs. It includes extensive analysis into what stable and unstable features capture in the data. The authors also provide geometric insights: they show that unstable features, even though they are individually not recovered across different random seeds, belong to a shared subspace. They explain this observation using a toy model which uses a product of matrices to model the observed non-identifiability of unstable features.

Critical strengths

Extensive analysis of stability: I like the multi-pronged and methodical analysis into how stable/unstable features contribute to reconstruction, what these features capture in data, how unstable features arrange geometrically, how the geometric observation can be modeled using a toy model, and finally effect of using different SAEs/ training on different model layers/ models. I am impressed by the thoroughness of the investigation.

Critical weaknesses

Studying SAE feature stability seems to be crowded, and the specific novelty of this paper is unclear. What does this paper's analysis teach us about SAE feature stability that was previously unknown? How is this useful to SAE research, and to interpretability more broadly? The authors do include a paragraph on related work and briefly mention other papers studying the same phenomenon. However, the specific novelty of this work that other papers lack is not made clear enough. The impact of this work can also be emphasized better: which aspects of the "Conclusion" section are the authors' specific novel recommendations that are not already being implemented? For instance, performing multiple runs of SAEs and using stable features, which the authors mention, is already used in the community.

While this may (or may not) be a relatively simple framing issue in specific sections, it matters significantly in determining whether this paper is useful to the broader community.

Questions for authors
  1. Specific novelty and impact of this work: What are the contributions in this paper that were not known in prior works, and what can we learn from them? How do they impact the way SAEs are used or designed?

  2. How do the findings of this paper relate to other known challenges with SAE features like feature splitting and absorption? Are unstable features usually split features corresponding to nonlinear manifold structure, or simply a reflection of linear features that occur less often in data?

  3. The authors use cosine similarity to search for recovered feature across different random initializations of SAEs. They use a threshold of 0.7 to determine that a feature is stable. Is it possible for semantically different features to be miscategorized as the same feature by this method? In general, it seems like there is no control: one could impose an arbitrary threshold and call a feature stable for that threshold. Ideally there should be some quantitative analysis of monosemanticity of these features, where the authors show that stable features are indeed monosemantic. This might be a critical issue with the method used in the paper.

  4. How do stable/unstable features differ across SAE architectures? This analysis, or a few observations from the results the authors already have, may be worth adding to the main paper since the authors claim to perform a cross-SAE study too: "large-scale study across seeds,.,SAE variants". While the authors report the fraction of stable/ unstable features across SAEs, are features themselves "stable" across different SAEs in what they capture about data? There are multiple papers that study this phenomenon, see for instance Bhalla et al 2026 (Temporal SAEs, https://arxiv.org/abs/2511.05541), Hindupur et al 2025 (Projecting Assumptions, https://arxiv.org/abs/2503.01822).

5 comments

Thank you for your thoughtful review and positive assessment of the thoroughness of our investigation. We appreciate the opportunity to clarify the scientific contribution and practical implications, and to address your questions below.

Section and figure numbers below refer to the submitted paper. New experiments are included in the revised appendices.

Specific novelty and impact of this work: What are the contributions in this paper that were not known in prior works, and what can we learn from them? How do they impact the way SAEs are used or designed?

Our paper establishes a systematic functional and geometric structure behind SAE seed dependence. Stable features carry most of the reconstruction- and prediction-relevant signal (Section 5.2, Figure 2). Unstable features have weaker marginal effects and predominantly surface-form interpretations, yet collectively span reproducible subspaces (Sections 6–7, Figures 4–5). Individual atoms can therefore change across seeds while shared geometric structure persists.

The distinctive contribution is connecting reproducibility, functional importance, and collective geometry—and turning that understanding into dictionary construction. Prior studies investigate inconsistency, noncanonical decompositions, and ensembling (Paulo and Belrose [5]; Song et al. [1]; Leask et al. [4]; Bhalla et al. [3]; Gadgil et al. [8]). We establish what distinguishes the stable and unstable populations and demonstrate a fixed-width construction from deduplicated cross-seed pools that achieves high stability with near-baseline reconstruction (Section 5.3, Figure 3). The random-model control further demonstrates the value of evaluating recurrence alongside automatic interpretation (Section 8, Figure 6b).

These results give concrete answers to what seed disagreement reveals about SAE representations and how to construct more reproducible dictionaries.

How do the findings of this paper relate to other known challenges with SAE features like feature splitting and absorption? Are unstable features usually split features corresponding to nonlinear manifold structure, or simply a reflection of linear features that occur less often in data?

We tested splitting and absorption on GPT-2 L7 and Gemma-2-2B L12, with four seeds at dictionary sizes 16,384 and 32,768. The first-letter task identifies splitting when several features jointly detect a word's first letter better than one alone. Absorption occurs when the selected letter detectors are silent but other features carry the letter's signal. We analysed 26 validated letter concepts on Gemma and 11 on GPT-2.

Gemma's added split features recur in 91–100% of 12 independent reference SAEs on average, depending on the setting. Among features matched for task firing frequency, features carrying the letter in full-absorption events comprise 17.2% of stable versus 3.6% of unstable Gemma features. GPT-2 gives 11.7% versus 3.6%, with the stable advantage in every setting. Thus these measured splitting and absorption events are not specific to unstable features.

For letters with a stable main detector, we also examined the additional detectors across letter, seed and width combinations: 1/15 on GPT-2 and 0/186 on Gemma were unstable. The single GPT-2 case had validation F1=0.27. In full-absorption cases with stable main detectors, the feature carrying the letter's signal was unstable in 3.33% of GPT-2 and 0.32% of Gemma cases. Stable and unstable features can thus participate in coding the same letter (revised Appendix E.10).

On the same 11 letters, both pooled GPT-2 L7 SAEs had less full absorption than ordinary TopK SAEs given the same 2M tokens of post-training (21.54% vs. 23.02% on average). The fraction of letters with splitting stayed at 13.6%, and the selected detectors' test F1 fell from 0.533 to 0.494 (Appendix B.16).

Following calendar-manifold analyses [2, 3], we selected SAE feature groups that reconstruct weekday differences on calibration text, independently of feature probability. On held-out text, we verified a circular weekday component in the groups' reconstructions and in the original activations projected into the same plane. This holds in all four seeds of each model. We then measured the selected features' probabilities:

ModelMean feature probabilityStable / intermediate / unstable features
GPT-2 L70.79021 / 23 / 1
Gemma-2-2B L70.83338 / 17 / 4

Means weight seeds equally. Counts sum features across their dictionaries. Most features representing this circular component have high or intermediate probabilities.

Our frequency-paired ablations also find greater stable-feature impact after matching firing frequency, so rarity alone does not explain the functional difference.

The authors use cosine similarity to search for recovered feature across different random initializations of SAEs. They use a threshold of 0.7 to determine that a feature is stable. Is it possible for semantically different features to be miscategorized as the same feature by this method?

Ideally there should be some quantitative analysis of monosemanticity of these features, where the authors show that stable features are indeed monosemantic.

Why 0.7? As stated in Section 4, our decoder cosine cutoff follows Leask et al. [4], who use 0.7 for GPT-2 SAEs. Paulo and Belrose [5] also use 0.7 in cross-seed comparisons, requiring the same pair to meet the cutoff in both encoder and decoder matchings. Feature probability is the fraction of reference SAEs containing a decoder vector that meets the cutoff. We call features stable at probability ≥0.95 and unstable at ≤0.05.

The paper already sweeps the cutoff across SAE variants (Appendix B.2, Figure 7a) and compares SAEs trained on pretrained and random GPT-2 across cutoffs (Section 8, Figure 6b). In the revised Appendix B.2, removing stable features causes more reconstruction damage per removed activation at 0.6, 0.7 and 0.8 in all four seeds.

Do explanations transfer near 0.7? We pair each source feature with the highest-cosine decoder vector in another GPT-2 layer-7 SAE, then sample pairs by cosine band. Qwen3-32B generates explanations on calibration text and uses either feature's explanation to predict whether the target fires at a marked token. Both explanations are scored on the same 16 held-out contexts (8 firing and 8 non-firing, so constant prediction scores 50%). Each band below averages 24 pairs, six per directed seed comparison.

Decoder cosineTarget's own explanationTransferred explanation
[0.6, 0.7)81.25%76.82%
[0.7, 0.8)84.11%83.85%
[0.8, 0.9)86.20%85.16%
[0.9, 1.0]82.81%81.25%

In [0.7, 0.8), transferred accuracy is 0.26 percentage points below the target's own explanation on average (95% paired CI [−4.43, +3.91]). Transfer stays at 81–85% in the higher bands. In the random-pair control, transfer for the same 12 source features is 47.40% when paired with targets in [0, 0.1) and 79.69% with targets in [0.7, 0.8).

The revised Appendix E.8 reports the protocol and full results by cosine band. Explanation transfer measures agreement in activation behavior on held-out text, which does not establish identical semantics for every pair. Feature probability measures reproducibility and does not require monosemanticity.

How do stable/unstable features differ across SAE architectures?

While the authors report the fraction of stable/ unstable features across SAEs, are features themselves "stable" across different SAEs in what they capture about data?

Temporal SAEs [6] change the training objective to encourage temporal consistency. Projecting Assumptions [7] studies how architectural constraints relate to concept geometry. Our question is whether feature probability across seeds within one architecture predicts recovery of the same features in others.

On GPT-2 L7, we searched for TopK features in four other architectures at the same dictionary size and similar activity. For this comparison, we counted a feature as recovered when its closest decoder match had cosine ≥0.7 and activation correlation ≥0.8 on unseen text, with at least 20 test firings per feature.

Target architectureOriginal stable TopK features recoveredOriginal unstable TopK features recovered
TopK, different seed88.7%0.0%
BatchTopK84.5%1.3%
JumpReLU72.3%0.8%
HierarchicalTopK68.2%1.0%
Vanilla56.7%0.8%

Percentages use the original stable or unstable TopK groups as denominators. The TopK reference retains 88.7% of stable features after the additional activation test. Stable TopK features also recover more often across all four other architectures. The revised Appendix G.6 reports the protocol and both geometric and functional recovery.

Thank you again for your careful reading and constructive suggestions. We appreciate your feedback on making the contribution and its implications clearer.

References

[1] Song, X., Muhamed, A., Zheng, Y., Kong, L., Tang, Z., Diab, M. T., Smith, V., and Zhang, K. (2026). Mechanistic Interpretability Should Prioritize Feature Consistency in Sparse Autoencoders. Proceedings of ACL, pp. 2172–2210.

[2] Engels, J., Michaud, E. J., Liao, I., Gurnee, W., and Tegmark, M. (2025). Not All Language Model Features Are One-Dimensionally Linear. International Conference on Learning Representations (ICLR).

[3] Bhalla, U., et al. (2026). Do Sparse Autoencoders Capture Concept Manifolds?. arXiv preprint.

[4] Leask, P., Bussmann, B., Pearce, M. T., Bloom, J. I., Tigges, C., Al Moubayed, N., Sharkey, L., and Nanda, N. (2025). Sparse Autoencoders Do Not Find Canonical Units of Analysis. International Conference on Learning Representations (ICLR).

[5] Paulo, G., and Belrose, N. (2026). Sparse Autoencoders Trained on the Same Data Learn Different Features. International Conference on Learning Representations (ICLR).

[6] Bhalla, U., Oesterling, A., Mayrink Verdun, C., Lakkaraju, H., and Calmon, F. P. (2026). Temporal Sparse Autoencoders: Leveraging the Sequential Nature of Language for Interpretability. International Conference on Learning Representations (ICLR).

[7] Hindupur, S. S. R., Lubana, E. S., Fel, T., and Ba, D. (2025). Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry. Advances in Neural Information Processing Systems (NeurIPS).

[8] Gadgil, S., Lin, C., and Lee, S.-I. (2025). Ensembling Sparse Autoencoders. arXiv preprint.

Reviewer_Jt3HOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies seed dependence in SAEs via a per-feature reappearance probability (fraction of N independently seeded SAEs containing a decoder vector with cosine >= 0.7). With 96 TopK SAEs on GPT-2 layer 7, the distribution is bimodal. Stable features account for most EV/CE impact under active-mass-matched masking, while unstable features are rarer and receive surface-form auto-interp explanations. Unstable decoder vectors have lower effective rank and their SVD subspaces transfer across seeds, which the authors interpret as basis ambiguity within a shared subspace, supported by a synthetic low-rank model. The paper also builds SAEs from deduplicated high-stability cross-seed feature pools, and reports results across SAE variants, training budgets, Pythia/Gemma-2/Qwen2.5, and a random-GPT-2 control.

Critical strengths
  1. very good scale and a practical null check. The 96 seeds per configuration is good, and the eval of per-feature stability is cheap and architecture-agnostic. The random-GPT-2 control is the most useful result: auto-interp scores stay high on random-model SAEs while the stable fraction collapses, so cross-seed recurrence complements plausibility-based evaluation.
  2. instability concentrates in an identifiable, low-frequency, surface-form subset, which is useful for filtering features before mechanistic claims. The planned checkpoint release would be a valuable resource.
Critical weaknesses
  1. The paper discussed very similar problem as ACL 2026 paper "Mechanistic Interpretability Should Prioritize Feature Consistency in Sparse Autoencoders." That work also studies run-to-run SAE consistency, proposes a consistency metric (PW-MCC) as a standard evaluation axis, grounds it in dictionary-learning identifiability with synthetic model organisms, and already reports that consistency increases with activation frequency, tracks explanation similarity, and shows no strong reconstruction–consistency trade-off; it also names subspace alignment as an open direction. Its architecture conclusion conflicts with this paper: TopK/BatchTopK are the most consistent there, but the least stable here (Table 1). The novelty and the discrepancy need to be addressed.

  2. The "reproducible low-rank subspace" claim lacks the right null and is overstated. Cross-seed transfer is compared only to a Haar-random baseline, but any decoder subset will concentrate in the high-variance subspace of anisotropic GPT-2 activations. Needed nulls: size-matched random/stable subsets, the activation PCA subspace, and random-GPT-2 SAEs. The rank gap is modest and confined to p=0 (Table 2: ER/d 0.541 vs 0.739–0.765 for all other bins; Table 9: ~450–500 of d=768). The mixing hypothesis is also never tested directly (e.g., linear predictability of one seed's unstable block from another's).

  3. Functional asymmetry is confounded with frequency. Unstable features fire ~2.4x less often. In the frequency-matched comparison (Table 4) CE ratios vary from 1.9x to 23.4x and the top-frequency EV ratio is 0.45x; in Table 3 only the [0.8, 1.0] bin stands out and the trend is non-monotone. The claim should be narrowed accordingly.

  4. the synthetic low-rank block is non-identifiable by construction and Fig. 12c (r=4: low-rank mean p=0.864) contradicts "the pattern persists"; the pooled construction is evaluated against the seeds it was selected from (two disjoint-pool constructions should be compared) and its EV (~0.926) vs. TopK (0.892) is not compute-matched; matching is decoder-only.

Questions for authors
  1. How does unstable-subspace transfer compare to size-matched random/stable subsets, the activation PCA subspace, and random-GPT-2 SAEs? (could change my score)
  2. Can one seed's unstable-feature activations be linearly predicted from another seed's unstable block, and with what R^2? (could change my score)
  3. How do you reconcile your SAE-type ranking and frequency findings with the ACL 2026 paper, and what does this paper add beyond them?
  4. Do two pooled constructions from disjoint seed pools agree with each other, and what is the compute-matched EV of a standard TopK SAE?
4 comments

Thank you for the detailed review and for highlighting the scale of the study and the value of the random-model control. We appreciate your constructive questions, which motivated the additional controls presented below.

Feature probability is the fraction of independently seeded SAEs containing a decoder match at cosine ≥0.7. We call features stable at probability ≥0.95 and unstable at probability ≤0.05.

Section and figure numbers below refer to the submitted paper. New experiments are included in the revised appendices.

How do you reconcile your SAE-type ranking and frequency findings with the ACL 2026 paper, and what does this paper add beyond them?

Song et al. [1] connect consistency to ground-truth recovery, explanation similarity and firing frequency; our frequency finding agrees. Our submitted contributions are functional tests of stable/unstable features (Section 5.2, Figure 2), shared geometry and its synthetic analogue (Sections 6–7, Figures 4–5), and construction of dictionaries from deduplicated feature pools (Section 5.3, Figure 3). The random-transformer control (Section 8) yields high automatic-interpretation scores but low reproducibility, motivating complementary checks. The experiments below strengthen these findings.

Whole-dictionary PW-MCC confounds active-feature reproducibility with dictionary utilization. In Song et al.'s Appendix F.2, 49% of JumpReLU matched pairs contain an endpoint that never fires on their evaluation text, versus 2% for TopK. These unused entries nevertheless contribute to the reported architecture ranking.

We isolate this effect in eight GPT-2 L7 JumpReLU checkpoints. Excluding their 6.97% nonfiring features and recomputing one-to-one matching raises PW-MCC from 0.775 to 0.825, reversing the comparison with our TopK reference (0.783) with no retraining or change in reconstruction on the evaluation text. Thus the treatment of unused dictionary entries alone can reverse this architecture comparison; the aggregate ranking in [1] does not establish lower reproducibility of active JumpReLU features. A separate eight-seed JumpReLU cohort with only 0.006% inactive features scores 0.822 even without filtering. ReLU/L1 outperforming TopK in stability is also reported by Paulo and Belrose [2] and Fel et al. [3] (Table 1).

How does unstable-subspace transfer compare to size-matched random/stable subsets, the activation PCA subspace, and random-GPT-2 SAEs?

To control Section 6's subspace result, we extract the leading 64 SVD directions from equally sized samples of unstable, stable or random whole-dictionary decoder vectors in SAE A. We project the same unstable vectors from independent SAE B into each subspace. The score is the fraction of their squared length preserved; decoder vectors have unit norm. Activation PCA uses 64 directions fitted on separate documents. The random-space baseline is 64 divided by the model's activation dimension.

SetupUnstable sourceStable sourceRandom dictionary sourceActivation PCARandom-space baseline
GPT-2 L7, main experiment44.6%23.1%24.7%9.6%8.3%
GPT-2 L444.6%21.5%33.3%16.9%8.3%
Gemma-2-2B L746.0%16.2%19.1%12.2%2.8%

Thus ordinary decoder subsets share geometry, but unstable subsets transfer substantially more. This ordering holds in all nine configuration means: GPT-2/Pythia-160M layers 4/7/10 and Gemma-2-2B layers 7/12/18. The table shows selected examples. Main GPT-2 uses six evaluation SAEs and 90 separate probability references; replications use four and 12, respectively (four references for Gemma L18). The revised Appendix F.1 reports all configurations and controls.

For SAEs trained on one fixed random GPT-2, we repeat the comparison with four probability references per model. Unstable transfer minus random-subset transfer is 31.8% − 21.3% = 10.5 percentage points for pretrained GPT-2, versus 31.8% − 30.2% = 1.6 points for random GPT-2. Generic dictionary geometry therefore explains much more of the random-model transfer.

The rank gap is modest and confined to p=0 (Table 2: ER/d 0.541 vs 0.739–0.765 for all other bins; Table 9: ~450–500 of d=768).

Effective rank summarizes the decoder spectrum's spread: 461 for unstable versus 575 for stable features, out of 768 dimensions. This indicates a relative reduction across hundreds of effective dimensions.

Can one seed's unstable-feature activations be linearly predicted from another seed's unstable block, and with what R^2?

For each token, independent SAEs A and B encode the same model activation. We fit a regularized linear map with A's unstable-feature activations as input and B's unstable-feature activations as output. Stable/random controls replace only the input with equally many features. A second target is B's unstable reconstruction contribution: the activation-weighted sum of its unstable decoder vectors.

SetupPredicted output in BUnstable inputRandom inputStable input
GPT-2 L7Unstable activations0.2310.1280.081
GPT-2 L7Unstable reconstruction contribution0.3340.1460.083
Gemma-2-2B L7Unstable activations0.1170.0240.015
Gemma-2-2B L7Unstable reconstruction contribution0.2120.0410.023

R² pools squared errors and coordinate-centered variance over test tokens. Entries average all 12 ordered source→target pairs among four SAEs. Training, validation and test documents are disjoint, and regularization is selected on validation only. Unstable inputs beat stable/random controls in all nine configuration means. The full GPT-2 L7 unstable block beats the best single feature from that block, selected per target on validation (0.231 versus 0.106): evidence for partial functional correspondence distributed across features. The revised Appendices F.2–F.3 give the regression protocol and full results.

Functional asymmetry is confounded with frequency. Unstable features fire ~2.4x less often.

The claim should be narrowed accordingly.

Section 5.2/Figure 2 approximately matches firing counts by masking N stable or 4N unstable features; Tables 3–4 add bin controls. Our new test instead pairs individual stable and unstable features with nearly equal calibration firing rates, then masks equal numbers from the paired sets.

Across four GPT-2 L7 SAEs we obtain 5,217 pairs without replacement. On separate test documents, we zero 32, 64 or 128 features from either side, reconstruct the activation and patch it into GPT-2. We measure the additional next-token CE increase and reconstruction EV loss relative to the unmasked SAE reconstruction. The masked sets' total test firing counts differ by less than 1% on average, yet stable-feature removal causes 3.95–4.57× greater CE damage and 3.65–3.88× greater EV loss. The direction holds in every seed and with the paper's control that restores the masked reconstruction's norm to its unmasked value at each token.

This supports an average stable–unstable functional difference at matched frequency. It does not require monotonicity across all probability bins or dominance in every frequency bin, consistent with the heterogeneity in Tables 3–4. The revised Appendix B.10 gives the paired protocol and results.

Do two pooled constructions from disjoint seed pools agree with each other, and what is the compute-matched EV of a standard TopK SAE?

The paper already reports pooling (Section 5.3/Figure 3), feature-probability distributions (Figure 9) and a post-trained single-SAE control (Figure 10). We add disjoint pools and a 10B-token baseline.

We take 20 independently trained 1B-token TopK SAEs, split them into two groups of ten, and separately pool, deduplicate and select the most probable features to construct two SAEs. Two independent standard TopK baselines each receive 10B training tokens, matching the source-training budget per pooled dictionary. All four dictionaries have 16,384 features and k=64, receive the same 2M-token post-training (Adam, lr=1e-3), and use the same test text.

MethodAgreement between its two dictionariesMean reconstruction EV
Pooling from disjoint groups of ten 1B SAEs87.5%0.9303
Standard 10B-token TopK78.8%0.9322

Agreement counts one-to-one decoder matches at cosine ≥0.7. Pooling gains 8.7 percentage points more agreement for 0.19 percentage points of EV loss. Against 76 further reference SAEs excluded from both source pools, unstable fractions are 0.27% versus 6.72%, respectively.

matching is decoder-only.

Section 4 defines probability using decoder matches. Repeating the calculation with normalized encoder-weight vectors, on the same 96 SAEs and cutoff, gives strongly correlated feature probabilities: Pearson 0.929, Spearman 0.932. Each calculation uses the other 95 SAEs as references (revised Appendix B.3).

the synthetic low-rank block is non-identifiable by construction and Fig. 12c (r=4: low-rank mean p=0.864) contradicts "the pattern persists"

For Section 7's rank-two synthetic data, we use generator labels to select examples containing exactly one planted low-rank feature. Subtracting the other group's SAE reconstruction and clustering the residual recovers planted directions at cosine 0.893–0.905 with ground truth, versus 0.246 for individual SAE atoms. This label-assisted recovery supports describing the original result as poor atom recovery by the tested SAE, without asserting fundamental non-identifiability.

The contrast is weaker at rank four (Figure 12), consistent with the low-rank block being less dimensionally constrained. Nevertheless, low-rank features remain less reproducible than full-rank features: mean feature probabilities are 0.864 versus 1.000, and mean atom-to-ground-truth cosines are 0.714 versus 0.966.

Thank you again for your careful reading and constructive feedback, which helped us strengthen the controls and clarify the interpretation of our results.

References

[1] Song, X., Muhamed, A., Zheng, Y., Kong, L., Tang, Z., Diab, M. T., Smith, V., and Zhang, K. (2026). Mechanistic Interpretability Should Prioritize Feature Consistency in Sparse Autoencoders. Proceedings of ACL, pp. 2172–2210.

[2] Paulo, G., and Belrose, N. (2026). Sparse Autoencoders Trained on the Same Data Learn Different Features. International Conference on Learning Representations (ICLR).

[3] Fel, T., Lubana, E. S., Prince, J. S., Kowal, M., Boutin, V., Papadimitriou, I., Wang, B., Wattenberg, M., Ba, D., and Konkle, T. (2025). Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Models. Proceedings of ICML, PMLR 267.

Dear Reviewer, thanks again for the constructive feedback! With less than 24 hours left in the discussion, we’d love to hear your thoughts if you’ve had a chance to read our reply. Happy to answer any questions you still have.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions and follow-up analyses

We thank all three reviewers for their feedback and discussion. The revised manuscript highlights additions from the initial rebuttal in red and subsequent follow-up analyses in blue.

(i) Contribution relative to prior work [Jt3H, xBq3]. Prior work showed that independently trained SAEs can learn different features. We show that removing features that recur across seeds has a larger average effect on reconstruction and next-token loss than removing unstable features matched for firing frequency. Unstable features collectively span similar subspaces across seeds even when their individual decoder directions rarely recur. We use recurrence to build a fixed-width SAE by pooling features across seeds, deduplicating by decoder cosine, and selecting the most recurrent features.

(ii) Functional impact beyond firing frequency [Jt3H, mXy5]. Appendix B.10 matches stable and unstable features by calibration firing frequency and masks equal numbers of each. On GPT-2 layer 7, masked test firing counts differ by less than 1% on average. Across mask sizes, removing stable features causes 3.95–4.57× the increase in next-token cross-entropy and 3.65–3.88× the loss in explained variance (EV). The effect holds in all four seeds. Across five SAE architectures, stable features also provide greater reconstruction benefit per unit of removed sparsity cost in every evaluated seed.

(iii) Geometry and cross-seed prediction [Jt3H, mXy5]. Appendices F.1–F.3 add size-matched stable and random subsets, activation PCA, a randomly initialized GPT-2 control, and held-out prediction across seeds. On GPT-2 layer 7, a rank-64 subspace fitted to unstable decoders captures 44.6% of another seed’s unstable decoder energy, versus 24.7% for random subsets and 9.6% for activation PCA. Cross-seed prediction reaches R² of 0.231 for unstable activations and 0.334 for their summed reconstruction contribution. Unstable inputs outperform stable and random controls on average in all nine configurations. Appendix D clarifies how planted rank affects recovery in the synthetic model.

(iv) Independent pooling and concept retention [Jt3H, xBq3, mXy5]. Appendix B.15 compares two disjoint pools of ten 1B-token SAEs with two ordinary 10B-token TopK SAEs at matched width, activity budget and post-training. Cross-seed agreement rises from 78.8% to 87.5%, with EV changing from 0.9322 to 0.9303. The unstable fraction falls from 6.7% to 0.27% against independent references. The pooled dictionaries detect 81 and 83 of 87 concepts, compared with 83.3 on average for one source SAE. Counting a concept if any of the ten source SAEs detects it gives 87 and 86 before pooling. Both pools retain all 63 tested object distinctions. The six and three losses all involve distinguishing meanings of the same word.

(v) Matching robustness and semantic agreement [Jt3H, xBq3]. Appendices B.2–B.3 test cosine cutoffs of 0.6, 0.7 and 0.8 and compare encoder and decoder matching. Appendix E.8 evaluates explanation transfer and word meaning on human-annotated SemCor text. Across 189 word-sense tests, mean class-balanced F1 is 0.741 for the stable source feature, 0.700 for its closest accepted match, and 0.030 for a frequency-matched low-cosine control. For the same source and meaning, partners in [0.7, 0.8) score 0.097 F1 higher on average than those in [0.6, 0.7) (95% CI [0.004, 0.210]). This measures agreement on the tested meaning.

(vi) Architecture and sparsity [mXy5, xBq3]. Appendices C.1–C.3 compare 12 configurations across five architectures. The more stable Vanilla and HierarchicalTopK configurations lie outside the observed EV–L0 frontier. Lowering the TopK budget or increasing the L1 penalty preferentially reduces unstable-feature firing after matching initial firing frequency. Appendix G.1 compares dictionary widths. Across four other architectures, stable TopK features are recovered more often than unstable ones when recovery requires both decoder similarity and held-out activation correlation (Appendix G.6).

(vii) Splitting, absorption and concept manifolds [xBq3]. Appendices E.9–E.10 show that most features in the selected weekday groups have stable or intermediate reappearance probabilities. Stable features also participate in first-letter splitting and absorption. On the 11 tested letters, pooled SAEs show less full absorption than equally post-trained TopK controls (21.54% vs. 23.02%), alongside lower detector F1 (0.494 vs. 0.533; Appendix B.16).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.