- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors systematically study the cross-seed feature instability phenomenon in Sparse Autoencoders (SAEs), using reappearance probability as a metric to evaluate feature stability. They reveal that stable features primarily carry the vast majority of the reconstruction- and prediction-related signal, whereas unstable features correspond to low-frequency surface-form signals. Through further geometric analysis, they find that this instability mainly arises from basis vectors within a shared low-rank subspace, rather than from noise. Feature stability in SAEs has long been a relatively important problem in the field; the authors’ metric is simple and practical, the experimental setup is complete and convincing, and the work provides useful guidance for optimizing SAEs.
Critical strengths
- The authors cleverly use reappearance probability as a metric to evaluate cross-seed feature stability, clearly separating stable and unstable features. Combining masking experiments and geometric analysis, they demonstrate that stable features carry the majority of the signal, while unstable features correspond to low-frequency surface-form signals. This finding offers useful insights for subsequent research on knowledge editing with SAEs. The work forms a closed loop from hypotheses and experimental setup to conclusions, and is reasonably convincing.
- The authors propose a cross-seed feature-pooling construction method and show that stability and explained variance need not involve a significant trade-off. I believe this method and finding provide a promising direction for improving feature selection in future SAEs.
Critical weaknesses
- The major geometric analysis is mainly based on TopK SAEs trained on the residual stream of GPT-2 layer 7. I am curious about the systematic ranking of feature stability across different SAE variants. I also wonder whether there is some trade-off between feature stability and the sparsity–reconstruction Pareto frontier.
- Every latent in an SAE contributes to both sparsity and reconstruction, but the authors do not consider the contribution to sparsity. One concern I have is that stable features contribute more to reconstruction while unstable features contribute more to sparsity. When the sparsity constraint is strengthened (e.g., by reducing dictionary width or increasing the coefficient of the sparsity regularizer in the loss), the SAEs may tend to zero out latents that contribute less to reconstruction and more to sparsity (i.e., the unstable features), as discussed in “The Price of Amortized Inference in Sparse Autoencoders,” in order to get as close as possible to the sparsity–reconstruction Pareto frontier.
Questions for authors
- Could you provide a more systematic ranking of feature stability across different SAE variants (e.g., Vanilla, JumpReLU, HierarchicalTopK, BatchTopK)? Additionally, do you observe any systematic trade-off between feature stability and the sparsity–reconstruction Pareto frontier? That is, does SAE sacrifice feature stability in order to maximize the Pareto front of sparsity-reconstruction?
- Every SAE latent contributes to both sparsity and reconstruction, yet the current analysis focuses mainly on reconstruction impact. Have you examined the contribution of stable versus unstable features to the sparsity objective? In particular, when the sparsity constraint is strengthened (by reducing dictionary size or increasing the sparsity regularizer coefficient), do the unstable features tend to be preferentially zeroed out, consistent with the mechanism discussed in “The Price of Amortized Inference in Sparse Autoencoders”?
4 comments
Every SAE latent contributes to both sparsity and reconstruction, yet the current analysis focuses mainly on reconstruction impact. Have you examined the contribution of stable versus unstable features to the sparsity objective? In particular, when the sparsity constraint is strengthened (by reducing dictionary size or increasing the sparsity regularizer coefficient), do the unstable features tend to be preferentially zeroed out, consistent with the mechanism discussed in “The Price of Amortized Inference in Sparse Autoencoders”?
Our additional experiments support the selection tendency suggested in your question: unstable features provide less marginal reconstruction benefit per unit of measured sparsity cost, and stronger sparsity constraints preferentially suppress their activity.
Reconstruction benefit per sparsity cost. Across all five architectures, masking stable features causes 2.1–5.8× greater EV loss per unit of removed cost, with the stable advantage in every evaluated seed. We pair stable and unstable features by their cost on calibration data, then draw three paired masks of 32 features per group per seed. On held-out text, we normalize each mask’s EV loss by its actual removed cost. Cost is summed activation magnitude with unit-normalized decoders for Vanilla, and nonzero activation count for the other architectures. In Vanilla, stable features account for 86.5% of total activation magnitude, versus 0.4% for unstable features. These ablations measure marginal reconstruction benefit under the existing codes, with the remaining activations held fixed (Appendix B.10).
Controlled increase in the L1 penalty. Four Vanilla SAEs trained without auxiliary loss are each continued with either their original coefficient, λ=0.0024, or λ=0.003, using identical fresh token streams. We match the original stable and unstable groups by calibration firing frequency and keep these groups fixed throughout continuation. After 100M additional tokens:
| L1 coefficient | Stable firing retained | Unstable firing retained |
|---|---|---|
| Original: 0.0024 | 100.1% | 99.9% |
| Increased: 0.003 | 79.3% | 72.3% |
The same ordering appears after 2M and 20M tokens. Thus stronger regularization preferentially reduces unstable-feature firing, even after matching initial firing frequency. Every tracked unstable feature still fires on the test sample after 100M tokens: at this penalty strength, we observe preferential attenuation without complete inactivity of the tracked unstable latents (Appendix C.3).
Hard activation budgets show the same tendency. Starting from eight TopK SAEs trained at k=64, we match the original stable and unstable groups by firing frequency and track them at lower k. At k=16, stable and unstable features retain 26.4% and 13.5% of their original firing counts immediately, and 29.9% and 15.0% after 2M tokens of adaptation. The larger reduction for unstable features holds in all eight seeds at k=16, 32 and 48. The immediate comparison shows that unstable activations more often fall below the stricter selection threshold. The difference persists after brief adaptation (Appendix C.2).
Dictionary capacity provides complementary evidence. For source dictionaries of width 32,768, recovery by independently trained target dictionaries falls from 95.1% to 62.2% for stable directions and 10.5% to 2.3% for unstable directions when target width is halved. All dictionaries retain k=64. This measures sensitivity to reduced dictionary capacity; the within-SAE interventions above directly measure changes in the activity of the original feature groups (Appendix G.1).
Thank you again for your constructive suggestions, which helped us examine more directly how sparsity constraints affect stable and unstable features.
Thank you for the detailed rebuttal and the additional experiments, which address my main interests. However, given that there have already been many studies addressing the issue of cross-seed consistency [1, 2] (even not SAE), and the conclusions drawn in this paper may not be particularly novel and appear intuitive. Identifiability theory shows that when latent components form irreducible low-dimensional groups, only the subspace they span is identifiable, not the basis within it [3]. In overcomplete sparse dictionary learning, standard uniqueness guarantees likewise fail when more atoms than dimensions share a low-rank subspace [4]. In SAEs, this would be expected to manifest as cross-seed inconsistency of individual latents alongside reproducible subspaces. Thanks to the author's hard work and endeavor, I decided to maintain the score.
[1] Convergent Learning: Do different neural networks learn the same representations? [2] SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability [3] Uniqueness of Linear Factorizations into Independent Subspaces, J. Multivariate Analysis [4] When Can Dictionary Learning Uniquely Recover Sparse Data from Subsamples?, IEEE Trans. Inf. Theory
Thank you for the follow-up. We will add an explicit discussion of all four papers to Related Work. They provide relevant connections, but their assumptions and claims differ substantially from those investigated here.
In particular, sparsity is central to the identification problem: a change of basis that preserves a subspace generally changes the sparsity of its codes. Consequently, shared representations across neural networks and identifiability under an independent-subspace model do not, by themselves, establish the behavior of sparsity-constrained SAE decompositions.
| Prior work | Relevant connection | Difference in assumptions and claims |
|---|---|---|
| [1] Convergent Learning | Reports individual-feature disagreement alongside shared representations across independently trained networks. | Studies hidden representations of independently trained convolutional networks. Its sparse prediction method constrains the cross-network alignment map; it does not study identifiability under a sparse-coding objective. Our setting involves independently trained SAE dictionaries decomposing a fixed model’s activations, where code sparsity constrains admissible decompositions. |
| [2] SVCCA | Introduces an SVD–CCA method for comparing representations, with applications to learning dynamics and distributed representations. | Provides a comparison method and empirical applications. It does not establish that individually unstable SAE features concentrate in reproducible subspaces, that this overlap exceeds appropriate dictionary and activation-space controls, or that recurrence predicts SAE feature utility. |
| [3] Uniqueness of Linear Factorizations into Independent Subspaces | Establishes uniqueness results for decompositions into statistically independent, irreducible random-vector blocks. | Its identifying assumptions concern independence and irreducibility. These differ from sparsity constraints, and we have not assumed or established them for the unstable SAE population. The theorem therefore does not directly imply our empirical findings about sparse SAE decompositions. |
| [4] When Can Dictionary Learning Uniquely Recover Sparse Data from Subsamples? | This is the closest theoretical connection: it gives conditions for unique recovery of sparse dictionaries and codes. | Its claims concern uniqueness under spark and sampling conditions. It does not investigate cross-seed SAE training outcomes or establish shared subspaces specifically among unstable learned atoms. Moreover, overcompleteness alone does not invalidate its guarantees, and failure of sufficient conditions does not establish that only the subspace is recoverable. |
Our geometric result is an empirical finding about the unstable population selected by cross-seed SAE recurrence. We test its subspace overlap against size-matched stable/random dictionary subsets and activation PCA, and examine partial functional correspondence through held-out cross-seed prediction. These findings require evidence beyond the general possibility of basis ambiguity.
We demonstrates practical consequences. At matched firing frequency, stable-feature removal causes 3.95–4.57× greater cross-entropy damage on GPT-2 L7. Separately, disjoint-pool construction decreases the fraction of unstable features from ~7% to ~0%, with explained variance changing from 0.9322 to 0.9303 at matched width, activity, and aggregate source-training tokens. These results quantify practical consequences that the general shared-subspace perspective leaves open.
We appreciate your engagement and your acknowledgment that the additional experiments address your main questions. If this clarification also addresses your remaining concern about novelty, we would be grateful if you would consider raising your score to reflect the concerns resolved during the discussion.
[1] Convergent Learning: Do different neural networks learn the same representations?
[2] SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
[3] Uniqueness of Linear Factorizations into Independent Subspaces, J. Multivariate Analysis
[4] When Can Dictionary Learning Uniquely Recover Sparse Data from Subsamples?, IEEE Trans. Inf. Theory
Thank you for your thoughtful review and for recognizing the value of the stability analysis and feature-pooling construction. Your questions about architecture comparisons and sparsity motivated the additional experiments presented below.
We evaluate five SAE architectures on the same held-out GPT-2 L7 activations, with 16,384 features and eight seeds per configuration, each trained on 1B tokens. At comparable sparsity, the observed stability ranking is Vanilla > HierarchicalTopK > JumpReLU > BatchTopK ≈ TopK. Here, a feature counts as stable if its decoder has a match at cosine ≥0.7 in all seven other seeds.
We observe a stability–reconstruction trade-off among these configurations: near L0=60–64, the more stable Vanilla and HierarchicalTopK configurations have lower EV. The revised Appendix C.1 plots reconstruction against sparsity for all 12 evaluated configurations, with stability indicated by color. The observed EV–L0 frontier contains TopK, BatchTopK and JumpReLU configurations. The more stable Vanilla and HierarchicalTopK configurations lie outside this frontier. In particular, JumpReLU achieves higher EV at lower L0 than both configurations shown above, while having lower feature stability. We restrict this conclusion to the evaluated architectures and operating points.
The submitted paper introduces and evaluates our pooling construction as a way to improve stability with near-baseline reconstruction (Section 5.3, Figure 3). The rebuttal strengthens that evaluation with two additional controls: disjoint source pools and matched training budgets. We compare two dictionaries constructed from disjoint pools of ten 1B-token SAEs with two ordinary TopK SAEs trained for 10B tokens each. All dictionaries have 16,384 features and k=64, and receive identical 2M-token post-training.
Agreement here counts one-to-one decoder matches at cosine ≥0.7. Pooling gains 8.7 percentage points of agreement at a cost of 0.19 percentage points of EV, at the same dictionary size, activity budget and aggregate source-training token budget (Appendix B.15). This quantifies the reconstruction cost of the stability improvement in our construction.