acceptodds
Under review as a conference paper at ICLR 2027

Decompose Sparsely Where You Should, Absorb Densely Where You Should Not

Abstract

Sparse autoencoders (SAEs) are typically trained to reconstruct the entire residual stream through a sparse dictionary, implicitly assuming that all activation content is amenable to sparse, monosemantic decomposition. We question this assumption and hypothesize that activations contain a low-rank, dense component that is computationally important to the model yet inherently unsuitable for sparse representation, which serves as a major source of the persistent dense latents widely observed in trained SAEs. To test this, we add a small rank-r linear bottleneck in parallel with standard SAEs (BatchTopK and Matryoshka), allowing dense structure to be absorbed before sparse reconstruction. On Gemma-2-2B layer 12, a rank-24 bottleneck reduces dense latent count by up to 84% while improving sparse probing and targeted probe perturbation on both architectures at matched sparsity. The absorbed component is (i) structurally identifiable as the top principal components and outlier dimensions; (ii) causally necessary, with removing it raising next-token cross-entropy by 7.5 , far exceeding the 2.8 from removing the geometrically near-identical top-24 PCA directions; and (iii) redundantly encoded by sparse dictionaries, with ablating 787 maximally aligned sparse features raising cross-entropy by only 2.9 and ablating 2,048 topic-aligned features leaving MMLU topic classification virtually unchanged, whereas removing the scaffold drops it from 98.7% to chance. Together, our findings identify a compact, semantically informative and causally important component of residual stream activations (which we term a computational scaffold) that standard sparse dictionaries represent inefficiently, suggesting that the scope of sparsity-based interpretability methods warrants careful re-examination.

Primary area:
interpretability and explainable AI
Keywords:
Sparse Autoencoders, Mechanistic Interpretability, Low-Rank Structure, Activation Decomposition

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_pgWdOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether some residual-stream activation content is represented inefficiently by standard sparse autoencoders. It introduces a small linear bottleneck alongside an SAE, with the SAE reconstructing the residual after subtracting the bottleneck contribution. Experiments with BatchTopK and Matryoshka show improvements in reconstruction, sparse probing, and targeted probe perturbation, together with fewer high-frequency latents. The authors characterize the absorbed component through its alignment with principal components and outlier dimensions, topic information, and effects on next-token prediction. They interpret this component as a computational scaffold that motivates separating dense and sparse reconstruction.

Critical strengths

S1. An interesting and simple architectural modification produces useful improvements. The idea of separating low-rank, dense information from sparse features is conceptually appealing and challenges the assumption that all residual-stream information should be sparsely decomposed. The bottleneck is small and compatible with existing SAE architectures. Improvements in reconstruction, sparse probing, and targeted probe perturbation suggest practical value beyond reducing dense latents. Replication across additional models and layers strengthens this contribution.

S2. The absorbed component is examined from several complementary perspectives. Architectural controls, geometric analyses, topic probes, and downstream interventions provide a richer characterization. The authors also report architecture-dependent regressions in absorption and intervention metrics, which helps identify the method’s practical trade-offs.

Critical weaknesses

W1. Limited evidence for generalization across independently trained models and different rr. Although the authors evaluate several bottleneck ranks and report multi-seed results for selected metrics, the rank sweep is restricted to one model, layer, and SAE architecture. A more systematic evaluation across independently initialized SAEs and different bottleneck ranks, tracking how interpretability and intervention metrics vary with (r), would strengthen the evidence for robustness and generalizability.

W2. On SAE architectures feature structure and concept alignment. Several relevant studies are missing. In [2] demonstrate that SAE architectures impose assumptions on concept geometry, potentially limiting feature recovery. In [1] introduce Kronecker-factorized SAEs with compositional 1-rank feature groups. Both provide relevant architectural context for the paper.

W3. Lack of guarantees on feature preservation. The proposed low-rank bottleneck has no theoretical guarantee that it will preserve useful, interpretable features in the sparse dictionary rather than absorbing them into the dense pathway. The observed degradation in some interpretability and intervention metrics raises concerns about this possibility. Although the authors discuss these metric regressions, they do not explicitly acknowledge the potential loss of interpretable sparse features as a limitation.

W4. Limited evaluation scope. The evaluation could benefit from additional benchmarks such as AxBench [3], experiments at other activation sites (e.g., after attention and MLP modules), and larger SAE dictionaries (e.g., 64K). Although not essential to the main claims, these experiments could strengthen the analysis and reveal interesting effects of activation location and dictionary capacity.

References:

[1] Kurochkin, V., Aksenov, Y., Laptev, D., Gavrilov, D., and Balagansky, N. Kronecker Factorization Improves Efficiency and Interpretability of Sparse Autoencoders. arXiv:2505.22255, 2025. https://arxiv.org/abs/2505.22255

[2] Hindupur, S. S. R., Lubana, E. S., Fel, T., and Ba, D. Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry. NeurIPS 2025. https://arxiv.org/abs/2503.01822

[3] Wu et al. (2025). AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders. ICML 2025. https://arxiv.org/abs/2501.17148

Questions for authors

Please see the weaknesses above for additional concerns. Also I would appreciate clarification on the following questions:

Q1. Why does ablating the learned bottleneck cause substantially greater degradation than removing the nearly identical PCA subspace? What explains this discrepancy?

Q2. What evidence distinguishes the proposed computational scaffold from ordinary high-variance activation structure?

Q3. How should the improvements in probing and reconstruction be interpreted alongside the degradation in some causal intervention metrics? Which metrics do the authors consider most indicative of improved interpretability?

8 comments
AuthorsOpenReview ↗

Robustness across seeds and ranks, and related work

We thank the reviewer for the thoughtful review and are encouraged that the reviewer finds the modification simple and conceptually appealing, its improvements of practical value, and our analyses complementary. The reviewer's concerns center on robustness and scope, and we have addressed them with new experiments, namely (i) additional training seeds, (ii) a step-matched rank sweep that tracks all interpretability and intervention metrics, (iii) a direct test of feature preservation, (iv) the mechanistic analyses at two additional sites, and (v) matched causal controls for the comparison raised in Q1. We summarize each below.

Independently trained SAEs (W1). We trained additional seeds of the BatchTopK pair in Table 1, with both models of each pair trained for the full 61k steps. Across the original seed and two additional seeds, the bottleneck improves explained variance by 0.8, 1.2, and 1.6 points and sparse-probing top-1 accuracy by 5.4, 5.8, and 3.2 points(single evaluation seed, whereas Table 1 averages sparse probing over three evaluation seeds and reports 4.0 points for the original seed). Among the intervention metrics, TPP@20 improves in both seeds for which we measured it (+.017 and +.012), while SCR direction-1 declines in both (from +.206 to −.295 and from +.221 to −.065). The learned subspace is also reproducible, as 22 to 23 of its 24 directions agree across seeds with a cosine above 0.99.

Bottleneck rank (W1). The table below extends the BatchTopK rank sweep at Gemma-2-2B layer 12 with all interpretability and intervention metrics. For a like-for-like comparison across ranks, every model in this table, including r=24r = 24, is trained for 30k steps, and Table 3 of the revised paper follows the same protocol.

rEVSP top-1SP top-5Absorption ↓TPP@20SCR@10RAVEL causeDense latents
0.793.749.851.058.023+.213.58225
8.797.759.855.064.081+.150.61420
16.797.742.858.059.111−.484.57514
24.801.750.874.060.044−.671.5805
48.805.771.863.092.063+.129.5454
96.816.768.886.096.038−.152.5483

Explained variance, top-5 probing accuracy, and TPP improve over r = 0 at every rank, and dense latents fall steadily up to r = 24 and then plateau. Absorption rises beyond r = 24, and RAVEL decreases slightly at r ≥ 48. SCR direction-1 changes sign across ranks, which together with its variation across seeds suggests that it is a noisy indicator in this setting. The gains in reconstruction, top-5 probing, and TPP therefore hold across ranks, and r = 24 lies where dense latents are largely removed before absorption begins to rise. At Llama-3-8B layer 16, where the dense structure extends beyond the top 24 principal components, increasing the rank from 24 to 96 further reduces the number of dense latents from 19 to 1 (34 without the bottleneck) and raises explained variance from .648 to .672 and sparse-probing top-1 accuracy from .807 to .817. A Matryoshka sweep was beyond the time available during the rebuttal period.

Related work (W2). We thank the reviewer for these references and have added both to the revised paper. Hindupur et al. (2025) show that each SAE architecture imposes assumptions on concept geometry that determine which concepts it can recover. Our work examines a related but distinct assumption, namely that all activation content is amenable to sparse decomposition. Kurochkin et al. (2026) propose a Kronecker-factorized SAE with compositional feature groups, which the revised paper includes among the architectural variants discussed in Section 2.

AuthorsOpenReview ↗

Feature preservation, scope, and metrics

Feature preservation (W3). We share the reviewer's concern and tested it directly. Structurally, the bottleneck is a linear map of rank 24, so it can capture at most 24 of the 2,304 directions of the residual stream, and the stop-gradient keeps the SAE objective from pushing content into it. Empirically, the sparse dictionary is preserved to the same degree as across random seeds. Matching each latent of the r=0 SAE to its nearest r=24 latent by decoder direction leaves 12.4%, 13.2%, and 13.7% unmatched across the three BatchTopK seeds, which is comparable to the 12.3% to 13.3% observed between r=0 SAEs that differ only in their random seed. On Matryoshka the corresponding fractions are 22.1% and 23.1%, again within the seed-to-seed range of 20.9% to 25.1%. The latents that do align with the bottleneck are few and mostly dense. Only 22 to 28 of the 16,384 latents have at least half of their decoder energy in the bottleneck subspace, and 67% to 87% of these are dense latents. For latents with less than a fifth of their energy in this subspace, the match rate equals the seed baseline. At the level of concepts, sparse-probing top-1 accuracy improves in all three seeds on six of the eight SAEBench datasets. The clearest exception is sentiment on Matryoshka (−.153 top-1 on Amazon sentiment), consistent with the partial absorption of sentiment discussed in Appendix H. Together, these results indicate that the bottleneck removes dense structure from the dictionary while leaving its sparse features intact. The revised paper reports this analysis in Appendix M, including the sentiment case.

Scope (W4). As the reviewer notes, these extensions lie outside the main claims of the paper. Our evaluation already covers 12 matched pairs across six sites, two model families, and two SAE architectures, with the full SAEBench suite at the main site. We focused on the residual stream because it aggregates the outputs of all components and is where low-rank broadcast structure is expected (Section 6.3). Attention and MLP output sites, 64K dictionaries, and steering evaluations on AxBench are natural extensions, which the revised paper mentions as future work, and the revision now includes a concept-detection evaluation on AxBench (Appendix M).

Interpreting the metrics (Q3). Apart from absorption on BatchTopK, which the paper reports alongside Table 4, the regressions concern two intervention metrics on one architecture. On BatchTopK, SCR direction-1 and RAVEL decrease (from +.206 to −.295 and from .569 to .558 against the baseline without the bottleneck), whereas on Matryoshka both improve (from +.263 to +.303 and from .575 to .583). TPP, which is also an intervention metric, improves on both architectures. One contributing factor is that SCR and RAVEL intervene on sparse latents only. When the content they target moves into the bottleneck, ablating sparse latents alone leaves it intact, and the metric registers this as a failure even if the combined system represents the content well. Appendix H illustrates this, as jointly ablating the top-8 bottleneck dimensions and the top-128 sparse atoms drives the gender probe to chance (0.503) while preserving profession accuracy (0.801). The regression also depends on how tightly the two pathways are coupled during training, and we ran two ablations of this coupling during the rebuttal period (61k steps, seed 0). Removing the full reconstruction objective (λ=0\lambda = 0), so that each pathway is trained on its own objective only, raises SCR direction-1 from −.295 to +.187, close to the +.206 of the baseline without the bottleneck, and RAVEL from .558 to .614, above the baseline's .569. TPP also rises from .054 to .062, and sparse-probing top-1 accuracy is preserved (.766 compared with .761). Removing the stop-gradient instead raises SCR direction-1 to +.044 and RAVEL to .590, while lowering sparse-probing top-1 accuracy to .748. Both variants preserve explained variance (.805) and most of the reduction in dense latents (from 23 to 8 and 6), and in exchange absorption rises from .084 to .107 and .105. The SCR and RAVEL regressions therefore arise largely from the coupling between the pathways, as opposed to the low-rank pathway itself, and the loss weighting offers a direct way to trade dictionary cleanliness against intervention metrics. The revised paper reports both ablations (Appendix C), and we will add further seeds in the camera-ready version. More broadly, we regard sparse probing and TPP as the most direct indicators of dictionary quality, and interventions on the combined bottleneck and SAE as the appropriate test of disentanglement. The revised Section 6.2 states this recommendation explicitly.

AuthorsOpenReview ↗

The causal comparison and the nature of the scaffold

Causal comparison with PCA (Q1). The difference stems from the mean component of the activation, which the two conditions treat differently. The bottleneck condition removes the full output of the dense pathway, including the large mean component that it reconstructs, and thus measures how much the model relies on the dense pathway as a whole. The PCA condition mean-ablates the principal subspace and retains the mean, and thus isolates the role of the centered directions. The difference between them therefore reflects the mean component to a large extent, as removing the mean alone already raises cross-entropy from 1.90 to 3.52, while removing the full output of the dense pathway raises it to 9.29. Under a common centered intervention, the two subspaces have effects of similar magnitude, and the remaining difference stems mainly from the first token. Projecting them out gives 2.98 for the bottleneck and 5.18 for PCA, mean-ablation gives 3.75 and 5.28, and with the first token left untouched the two are nearly equal (2.36 and 2.29). This agrees with the geometry, as the two subspaces share 21 of their 24 directions. The revised Section 5.2 and abstract report both quantities, namely the effect of mean-ablating the subspace (3.75) and that of removing the whole dense pathway (9.3), and Table 7 compares the two subspaces under matched interventions. The causal importance of the component itself holds under every intervention we tested, as removing random 24-dimensional subspaces leaves cross-entropy at 1.91, and this holds at all three sites. These matched controls further strengthen the causal analysis.

Scaffold versus high-variance structure (Q2). High variance alone is a property of the activation, whereas the scaffold is defined by how this structure interacts with sparse dictionaries and with the model. The learned subspace coincides with the dominant principal components, sharing 20 to 21 of its 24 directions with the top 24 at every site we examined, and this coincidence is central to the finding, since this structure dominates the reconstruction objective of every SAE. Three properties distinguish it. First, standard SAEs spend dense latents on it. At Gemma-2-2B layer 12, their dense latents place 0.66 of their decoder energy in the bottleneck subspace on average, compared with a median of 0.04 across all latents. Second, the SAE reconstructs it incompletely, with at most one of the top-24 principal components above 0.99 explained variance at any of the three sites, whereas adding the 24-dimensional bottleneck brings 19 to 20 of them to this level. Third, the model relies on it causally at all three sites (Q1). The revised Section 5.3 adds a direct comparison with directions of equal variance. Although the 24 scaffold dimensions hold only 30% of the activation variance, they recover 73% of the linearly decodable variance of both token frequency and output entropy, compared with 25% and 21% for the 323 principal components that follow them and hold the same total variance, and later attention layers read the scaffold mainly through their queries and keys. Separating this structure frees the dictionary, reducing dense latents from 25 to 5 in the rank sweep above and improving it across seeds and ranks (W1). The term scaffold names this combination of properties, and the revised Section 5 states this explicitly.

We hope these additions address the concerns that the reviewer described as minor or fixable. We also believe that the matched causal analysis strengthens the central claim of the paper, namely that part of the residual stream is better represented outside the sparse dictionary, which calls into question the standard practice of reconstructing the entire activation with an SAE. We would be grateful if the reviewer would consider raising the score in light of these results.

Reviewer_tkBKOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper challenges the notion of sparse decomposition of residual stream activations employed by tools like sparse auto encoders (SAEs). The authors identify multiple challenges of SAEs and posit that the challenges stem from the common underlying reason of sparse decompositions. They propose a sparse and dense decomposition instead, where the dense component acts as a linear bottleneck that captures information that is not amenable to sparse decomposition, and aligns with the top principal components. They claim that their method offers improvements on interpretability metrics.

Critical strengths

The paper tests sparse and dense decompositions on popular interpretability metrics.

Critical weaknesses
  1. The methodology used in this paper is suspiciously similar to the sparse and dense decomposition used in Lubana et al, "Priors in time ...", ICLR 2026 (https://openreview.net/forum?id=4J2e3nWiC8). The role of the dense component is also described similarly:

This paper Section 5: "characterize the absorbed content, ..., a component that carries coarse contextual information", and

Priors in time page 5: "contextual information present in the recent history that informs the next state—this part of the signal is deemed predictable"

Instead of citing the above paper and clearly describing how their method differs from the above, the authors completely ignore this prior work. There is also no mention about the literature on sparse and dense coding in signal processing (see Tasissa et al TMLR 2024, https://openreview.net/forum?id=FkgM06HEbk), which is the intellectual prior work. These concerns are substantial and may be considered plagiarism. Even otherwise, it is bad academic practice. Even if the authors came up with their method themselves, they above paper is very related prior work which appeared at the previous edition of the same conference ICLR, so it must have been reviewed and discussed in the paper!

  1. The claims do not seem to be substantiated by the results: The improvements shown by their method is marginal (Table 1), and due to the absence of error bars, it is unclear if the improvement is statistically significant. The autointerp score is also lower when they introduce the dense bottleneck (0.857 vs 0.869 without the dense bottleneck), further raising questions about their proposed method. Since they claim that their dense component aligns well with the top principal components (PCs), an important control is to compare results by replacing their dense component with the PCs: how would the metrics in Table 1 compare to this baseline?

Due to these weaknesses, the paper's positioning in existing work is unclear, and its contributions to the community are questionable. I recommend the paper be rejected.

Questions for authors
  1. Additional results to address the following concern (error bars, baselines) are needed to back up their claims: The claims do not seem to be substantiated by the results: The improvements shown by their method is marginal (Table 1), and due to the absence of error bars, it is unclear if the improvement is statistically significant. The autointerp score is also lower when they introduce the dense bottleneck (0.857 vs 0.869 without the dense bottleneck), further raising questions about their proposed method. Since they claim that their dense component aligns well with the top principal components (PCs), an important control is to compare results with replacing their dense component with the PCs: how would the metrics in Table 1 compare in that case?

  2. Proper citation, positioning with prior work and explaining novelty: Please review Lubana et al, and other related papers in interpretability and adjacent fields, and clearly position your work with respect to them.

6 comments
AuthorsOpenReview ↗

Clarifying the Fundamental Distinctions and Addressing the Plagiarism Allegation

We thank the reviewer for bringing these references to our attention. We first address the plagiarism concern directly, given the seriousness of such an allegation. Our work and Lubana et al. (2026, TFA) differ fundamentally in their motivations, methods, and research questions. Their similarity is limited to the general idea of decomposing an activation into dense and sparse components.

Lubana et al. (2026), TFAThis work
MotivationStandard SAEs overlook the temporal structure of language, treating tokens as independent observations and sparsity as time-invariant.Standard SAEs reconstruct entire activations using sparse dictionaries, implicitly assuming that all activation content admits a sparse decomposition. This assumption is challenged by persistent dense latents (Sun et al., 2025), the "dark matter" of SAE reconstruction error (Engels et al., 2025b), the anisotropic geometry of LM residual streams (Ethayarajh, 2019), and their outlier dimensions (Bondarenko et al., 2023).
MethodThe dense ("predictable") component of xtx_t is inferred from preceding context, while the sparse component captures what that context cannot explain.The dense component is obtained through a fixed low-rank map of the current activation xtx_t, trained jointly with a standard SAE, without access to preceding tokens.
QuestionHow do representations evolve across a sequence, particularly around garden-path sentences and event boundaries?Should SAEs reconstruct the entire activation when part of it is causally important yet poorly suited to sparse dictionaries?

On the quoted passages. Although both passages use the term "contextual," they refer to different notions of context. In TFA, it denotes information from preceding tokens that helps predict the current representation. In our §5, by contrast, it referred to information that a per-token linear probe can recover from (x_t) alone. Our method neither accesses preceding tokens nor makes claims about how representations evolve over time. To avoid ambiguity, the revised §5 no longer uses the term and describes this content as statistical properties of the text and the predictive state of the model.

We therefore maintain that the plagiarism concern is unfounded, as the similarity does not extend beyond a general decomposition shared by otherwise distinct approaches. Nevertheless, the revised paper cites TFA and clarifies this distinction in the related work.

Dense-and-sparse coding (Tasissa et al., 2024). This work studies a general signal-processing model in which a signal is expressed as the sum of dense and sparse components, y=Ax+Buy = Ax + Bu, and investigates the conditions under which these components can be identified and recovered, with applications to natural images.

Our work, however, starts from a different question: whether the standard practice of reconstructing the entire residual stream with SAEs is justified, given that it implicitly assumes all activation content is amenable to sparse, monosemantic decomposition. While the dense-and-sparse decomposition is not novel in itself, our contribution lies in using this architecture to test a specific assumption underlying standard SAEs and to uncover its consequences for representation geometry and interpretability.

Through this approach, we identify a component that is causally important to the model yet poorly represented by sparse dictionaries. Our findings further suggest that several SAE pathologies, which have previously been studied separately, may share a common underlying source. The central contribution of our paper therefore lies not in the dense-and-sparse decomposition itself, but in what it reveals about the structure of neural representations and the limitations of standard SAEs.

That a similar algebraic form arises from these different starting points reflects a convergence of ideas, as opposed to an overlap in research contributions. The revised paper cites Tasissa et al. (2024) and notes this connection in the related work.

AuthorsOpenReview ↗

Robustness, autointerp, and the PCA baseline

We also thank the reviewer for the suggestions regarding statistical robustness and the PCA baseline, which have helped us strengthen the empirical support for our claims.

Robustness of the gains. Table 1 reports one training run per configuration, as is common in prior work on SAE architectures. We nevertheless agree that training-seed variation deserves attention, and we address it in two ways. First, Table 4 repeats the comparison on 10 independent pairs, spanning two SAE architectures and five model-layer sites across Gemma-2-2B and Llama-3-8B. Across these pairs, the bottleneck improves explained variance in 9 of 10 cases (the remaining pair is tied), sparse-probing top-1 accuracy in 8, and top-5 accuracy in 9.

Additional seeds. Second, we trained additional seeds of the Table 1 BatchTopK pair, with both models of each pair trained for the full 61k steps. The table below reports r=0 → r=24 for the original seed and two additional seeds, all with a single evaluation seed, whereas Table 1 averages sparse probing over three evaluation seeds.

SeedEVSP top-1SP top-5
Original.797 → .805.707 → .761.844 → .863
New 1.793 → .805.753 → .811.837 → .882
New 2.789 → .805.725 → .758.854 → .869

The gains hold consistently across all three seeds. For the original seed, the bottleneck also reduces the number of dense latents from 23 to 4.

Autointerp. As discussed in §6.2, we view autointerp as a weak signal for the properties our method targets. It does not capture the benefit of removing persistent dense latents (Appendix I), and Heap et al. (2026) further show that it fails to distinguish trained transformers from randomly initialized ones. Moreover, the observed change on BatchTopK is small, from .865 to .857 in Table 1.As shown in Appendix I, the score within each populated fire-rate bin changes by at most 0.4 pp, and the decrease in the mean arises almost entirely from a shift in the fire-rate distribution, with fewer latents falling in the rarest-firing bin, which has the highest scores. We therefore do not consider this small difference to affect the main conclusions of the paper. As a more direct check, we matched each r=0 latent to its nearest r=24 latent by decoder direction. Across the three seeds, 12.4%, 13.2%, and 13.7% of r=0 latents have no counterpart with a cosine similarity of at least 0.5, which is comparable to the fraction observed between r=0 SAEs that differ only in their random seed (12.3% to 13.3%).

PCA baseline. We agree that this is an important control and have run it as requested. We replaced the learned bottleneck with a frozen projection onto the top-24 principal components of the training activations, keeping the SAE, data, and training schedule unchanged. All three models below are trained for 61k steps with seed 0.

BottleneckEVSP top-1SP top-5Dense latents
None (r=0).797.707.84423
Learned, r=24.805.761.8634
Frozen PCA, r=24.805.813.8885

The frozen PCA bottleneck matches the learned bottleneck in explained variance and in the reduction of dense latents, and it further improves sparse-probing accuracy. This is consistent with our central claim, which concerns the low-rank component of the activation itself, as opposed to any particular way of extracting it. In line with this, the learned bottleneck converges to nearly the same subspace during training (mean squared cosine of 0.908 with the top-24 principal components). The result also points to a simpler implementation, since the subspace can be computed once from activation statistics before training. The revised paper reports the frozen PCA bottleneck as a baseline in Appendix L (Table 10).

We hope that these additional results and clarifications address the reviewer's concerns.

Dear authors, Thank you for your clarifications and further experiments.

Tasissa et al 2024 is the methodological prior to your method, since they study how decomposing a representation into a sparse and a dense component works. An acknowledgement of this would be constructive and useful instead of a dismissal.

Thank you for clarifying how your work differs from Lubana et al. 2026 (TFA). I agree that the method for obtaining the dense component is different between these papers. However, from my understanding of the TFA paper, the underlying reason why this is done in TFA is that activations have a dense component and a sparse component: the context-dependence is the reason why the dense component is present in the activations. Therefore while the method they use to extract the dense component is different from yours, the dense component in your case (since you also study LLM activations) emerges for the same reason they identify. Please note that the sparse component is obtained in the SAME fashion in their paper (TFA) and in yours, using an SAE.

I appreciate the explicit distinction in the motivation and methodology between TFA and your paper added in the rebuttal. A similar distinction added to the paper would be useful for anyone reading your paper.

A good positioning of your work would be as using existing decomposition techniques from signal processing and applying them to interp using your specific method.

I am unconvinced about the performance gains with your method: the increase in scores appears minimal (Table 1), and even with multiple seeds, there are no error bars. Therefore, it is unclear if your method offers any advantage over existing SAEs.

Thank you for conducting the PCA baseline analysis. It is interesting that the PCA baseline is better than your method for obtaining the dense component. If the PCA baseline is better than your method, I am unable to understand the specific contribution of your paper. You claim that "our central claim, which concerns the low-rank component of the activation itself, as opposed to any particular way of extracting it."

If your main claim is that the sparse component from any sparse-and-dense decomposition method is more interpretable, then this does not appear to be novel. TFA sparse code interpretability has been analysed using GemmaScope.

I would appreciate a clarification of the above points, especially about what your paper contributes if a PCA baseline performs better and your results appear unconvincing.

Specifically, what is your contribution: is it empirical (gain in results appear weak), or methodological (sparse and dense has been done before, your distinction from TFA is in how you obtain the dense part but you show that PCA is better at this)? Can you state in simple words what your paper adds to the literature on interpretability?

I am happy to reconsider my score based on further discussion.

Thank you for engaging in discussion during the rebuttal period.

Reviewer_1r9WOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates a potential limitation of Sparse Autoencoders (SAEs) in language model interpretability research: conventional SAEs typically attempt to reconstruct the entire residual stream using a sparse dictionary, implicitly assuming that all activation information can be decomposed into sparse, approximately monosemantic features. The authors argue that activations also contain a type of low-rank, dense, and computationally important information that is not well suited for effective representation via a sparse dictionary.

To address this, the authors propose a parallel low-rank linear Bottleneck architecture. The module first extracts the low-rank dense component from activations, while the SAE is responsible for reconstructing the remaining part. Through linear constraints, low-rank constraints, and a gradient isolation mechanism, the authors prevent unnecessary representational competition between the Bottleneck and the SAE. In addition, through cross-model and cross-layer experiments, as well as PCA subspace analysis, causal intervention, and feature ablation experiments, the authors further investigate the information extracted by the Bottleneck and term it the Computational Scaffold.

The main contribution of the paper lies in proposing a perspective that re-examines the sparse decomposition assumption of SAEs, along with a corresponding architecture design and preliminary experimental evidence.

Critical strengths
  1. Interesting and valuable research question. The paper challenges the conventional assumption that all information in the residual stream should be represented sparsely. Investigating the role of dense, low-rank components provides a potentially valuable new perspective on SAE-based interpretability.

  2. Relatively comprehensive experimental evaluation. The paper evaluates the proposed method from multiple perspectives, including reconstruction performance, sparse probing, dense latent analysis, feature absorption, subject classification, and causal interventions. These experiments provide a reasonably broad assessment of the proposed approach.

Critical weaknesses
  1. Insufficient description of the proposed method. In particular, the paper does not adequately explain the motivations behind several key design choices—for example, why a low-rank bottleneck structure is adopted, and why this particular combination of reconstruction objectives and the stop-gradient mechanism is used. As a result, readers may find it difficult to fully understand the proposed method or to assess the necessity of its individual components.

  2. The generalizability of the paper remains limited. The proposed method was validated on only two models, and the key analyses supporting the existence and causal role of the computational scaffold are primarily conducted on the 12th layer of Gemma-2-2B. It is therefore unclear whether the identified properties generalize consistently across a wider range of models. The paper also does not adequately explain how the optimal bottleneck rank should vary with model size and layer depth.

Questions for authors
  1. How should the bottleneck rank be selected? Can the authors provide a principled guideline for choosing the rank based on model size or residual stream dimensionality?

  2. Does the proposed method remain effective when applied to substantially larger language models? Since the current evaluation is limited to Gemma-2-2B and Llama-3-8B, can the authors provide evidence that the proposed bottleneck architecture scales to larger models?

5 comments
AuthorsOpenReview ↗

Design choices

We thank the reviewer for recognizing both the importance of our research question and the breadth of our evaluation. As the reviewer notes, the central question of our work is whether SAEs should be trained to reconstruct the entire residual stream.

Motivation of the design (W1). Each design choice is motivated and ablated in Appendix C, and we summarize the rationale here together with new ablations. All three choices serve a single purpose, namely a stable division of labor in which the bottleneck absorbs a compact component of the activation and the SAE decomposes the remainder. All three choices serve a common purpose: to establish a stable division of labor in which the bottleneck captures a compact component of the activation, leaving the remaining content to be decomposed by the SAE.

Low rank. Our hypothesis is that the content resistant to sparse decomposition is low-dimensional, and the rank constraint ensures that the bottleneck is restricted to capturing such content. Without this constraint, the bottleneck can absorb the entire activation. Indeed, a full-rank bottleneck (r=2304r = 2304) achieves an explained variance of 1.00, but leaves little for the SAE to learn, reducing sparse-probing top-1 accuracy from .749 to .661 and the RAVEL cause score from .582 to .002.

Linearity. A nonlinear bottleneck can re-encode features that would otherwise be represented by the SAE, thereby undermining the intended separation between the two pathways. At the same training length of 30k steps, a GELU bottleneck achieves only the explained variance of the r=0r = 0 baseline (.793), reduces sparse-probing top-1 accuracy from .749 to .721, and decreases the number of dense latents only from 25 to 22, compared with a reduction to 5 under the linear bottleneck.

Gradient isolation and loss terms. The SAE is trained to reconstruct the residual x−sg[x^dense]x - \mathrm{sg}[\hat{x}_{\mathrm{dense}}], where the stop-gradient prevents its objective from updating the bottleneck. This discourages the bottleneck from absorbing content that the SAE could otherwise encode sparsely. The bottleneck, meanwhile, is trained through both its own low-rank reconstruction objective and the full reconstruction objective. The former encourages it to recover the minimum-error rank-rr subspace independently of the SAE, consistent with its strong alignment with the top-24 principal components (mean squared cosine of 0.908). The latter couples the two pathways, encouraging their combined output to reconstruct xx.

To examine the roles of these components, we conducted two additional ablations during the rebuttal period, each trained for 61k steps with seed 0 and evaluated using a single seed. The first removes the stop-gradient, allowing the SAE objective to update the bottleneck, while the second removes the full reconstruction objective (λ=0\lambda = 0), leaving each pathway to optimize only its own reconstruction objective.

EVSP top-1SP top-5Absorption ↓TPPSCR dir-1RAVELDense latents
With stop-gradient.805.761.863.084.054−.295.5584
Without stop-gradient.805.748.864.105.046+.044.5906
Without full objective (λ=0\lambda = 0).805.766.858.107.062+.187.6148

Both ablations preserve explained variance (.805), while the number of dense latents remains substantially below the r=0r = 0 baseline (23 at 61k steps), falling to 6 and 8, respectively. This suggests that much of the reduction in dense latents comes from the low-rank pathway itself, rather than from the specific coupling between the two pathways.

The coupling nevertheless affects the quality of the learned sparse dictionary. Removing either component increases absorption from .084 to .105 or .107 and leaves more dense latents, while removing the stop-gradient also reduces sparse-probing top-1 accuracy from .761 to .748. The intervention-based metrics, however, show the opposite trend: SCR direction-1 increases from −.295 to +.044 and +.187, while RAVEL rises from .558 to .590 and .614.

These results suggest a trade-off between the quality of the sparse dictionary and intervention performance. While gradient isolation and the full reconstruction objective help maintain a cleaner separation between dense and sparse content, relaxing these constraints can improve certain intervention metrics. The loss weighting therefore provides a means of controlling this trade-off. Given that these ablations were evaluated with a single seed, and that SCR in particular exhibits considerable variability across runs, we present these findings as a design trade-off in the revised paper (Appendix C), and we will add further seeds in the camera-ready version.

AuthorsOpenReview ↗

Generality and larger models

Scope of the current evaluation (W2). We would first like to clarify the scope of the evaluation already included in the paper, which may not be sufficiently apparent from its current presentation. Beyond the main setting, Section 4.1 and Table 4 (Appendix F) report 10 additional matched pairs (r=0r = 0 versus r=24r = 24) across Gemma-2-2B layers 6 and 18 and Llama-3-8B layers 8, 16, and 24, each evaluated with both BatchTopK and Matryoshka SAEs. Across these pairs, the bottleneck improves explained variance in 9 of 10 cases (with the remaining pair tied), sparse-probing top-1 accuracy in 8, and top-5 accuracy in 9. Our evaluation therefore covers two model families, six sites, two SAE architectures, and residual streams with dimensions of 2304 and 4096.

For context, the original papers introducing BatchTopK and Matryoshka SAEs (Bussmann et al., 2024, 2025), on which our experiments build, report their main results at Gemma-2-2B layer 12, with GPT-2 Small layer 8 serving as an additional setting for BatchTopK. Gemma-2-2B layer 12 is also the primary evaluation setting of SAEBench (Karvonen et al., 2025). The revised paper summarizes Table 4 in the main text (Section 4.1) to make the breadth of our evaluation more apparent.

Generality of the mechanistic analyses (W2). To further confirm the mechanistic analyses of Section 5 beyond Gemma-2-2B layer 12, we repeated the geometric, causal, and reconstruction analyses of Section 5 at Gemma-2-2B layer 18 and at Llama-3-8B layer 16. The main properties hold at both sites.

  • Geometry. The learned bottleneck again closely aligns with the top-24 principal subspace, with 20 of the 24 principal cosines reaching at least 0.95 at both sites, compared with 21 at Gemma-2-2B layer 12.
  • Causal role. Projecting out the bottleneck subspace raises next-token cross-entropy from 1.90 to 3.66 at Gemma-2-2B layer 18 and from 1.80 to 2.54 at Llama-3-8B layer 16. In contrast, removing random 24-dimensional subspaces yields cross-entropies of only 1.92 and 1.81, respectively.
  • Reconstruction of dominant directions. Without the bottleneck, the SAE reconstructs at most one of the top-24 principal components with explained variance above 0.99 (one at Gemma-2-2B layer 18 and none at Llama-3-8B layer 16). With the bottleneck, the combined reconstruction reaches this threshold for 20 and 19 components, respectively.
  • Dense latents. The bottleneck reduces the number of dense latents from 28 to 8 at Gemma-2-2B layer 18 and from 34 to 19 at Llama-3-8B layer 16.

These results provide further evidence that the geometric and causal properties identified in Section 5 are not specific to the original evaluation site. The revised paper includes these analyses in Appendix N and revises the limitations discussed in Section 6.3 accordingly.

Larger models (Q2). Our largest evaluated model, Llama-3-8B, has approximately three times as many parameters as Gemma-2-2B, which serves as the primary setting in the aforementioned SAE studies. Within this range, our findings remain consistent: on Llama-3-8B, the bottleneck improves explained variance in 5 of 6 matched pairs (with the remaining pair tied), while the mechanistic properties discussed above also hold at layer 16.

The diagnostic presented in our response to Q1 further suggests that scaling may primarily require adjusting the bottleneck rank, as opposed to changing the architecture, as the dense structure appears to extend across more principal components in the larger model. However, evaluating models beyond Llama-3-8B was not feasible within the rebuttal period. Scaling to even larger models is a natural next step, and the rank diagnostic in our answer to Q1 provides a principled starting point for it.

AuthorsOpenReview ↗

Rank Selection

Choice of rank (Q1). As stated in Section 6.3, we selected r=24r = 24 empirically for Gemma-2-2B layer 12 and kept it fixed across the additional settings, and how the optimal rank scales with model size or layer depth remains an open question. We share the reviewer's interest in this question, and our experience so far suggests three practical observations.

First, the gains in explained variance and the reduction in dense latents hold across a broad range of ranks. In a rank sweep at Gemma-2-2B layer 12 (30k steps, same seed), every rank from 8 to 96 improves explained variance over the standard SAE (.797 to .816, compared with .793), and the number of dense latents decreases steadily from 25 to 20, 14, 5, 4, and 3 as the rank grows from 0 to 96. The rank should nonetheless remain small relative to the residual stream. At full rank (r=2304r = 2304), the bottleneck absorbs the entire signal, and sparse-probing top-1 accuracy drops from .749 to .661.

Second, a useful diagnostic at a given site is the subspace occupied by the dense latents of a standard SAE. At Gemma-2-2B layer 12, these latents place 0.62 of their decoder mass on the top-24 principal components. At Llama-3-8B layer 16, they place 0.33 there, and 26 of 34 carry more mass in components 25 to 96 than in the top 24. The spectrum is also flatter, with components 25 to 96 holding 86% as much variance as the top 24, compared with 43% at Gemma-2-2B layer 12. Consistent with this, when dense latents are counted on the same Pile text at both sites, the rank-24 bottleneck reduces them from 26 to 8 at Gemma-2-2B layer 12 and from 34 to 19 at Llama-3-8B layer 16.

Third, these observations suggest that larger models may benefit from a larger rank. Increasing the rank from 24 to 96 at Llama-3-8B layer 16 supports this. The number of dense latents falls from 19 to 1 (34 without the bottleneck), explained variance rises from .648 to .672 (.637 without the bottleneck), sparse-probing top-1 accuracy rises from .807 to .817 (.805 without the bottleneck), and absorption decreases from .128 to .111. The Llama-3-8B results in Table 4, which already improve explained variance in 5 of 6 pairs at r=24r = 24, are therefore a conservative estimate of the method. In practice, we recommend choosing the smallest rank at which the dense latents of a standard SAE are largely removed, using the principal-component diagnostic above as a check. The revised paper adds these observations to Section 6.3 and the diagnostic to Appendix N as practical guidance for choosing the rank.

We hope that the clarified design rationale, the new ablations of the design, the scope of the existing evaluation across six sites, and the replication of our mechanistic analyses at two additional sites address the reviewer's concerns, and we would be grateful if the reviewer would consider raising the score in light of these additions.

Reviewer_XvHUOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a simple explanation for a variety of failure modes of sparse autoencoders (SAEs) -- i.e., that LLMs bifurcate semantic representation into two categories: a low-rank, dense "computational scaffold" (not suitable for interpretation by SAEs); and a sparse, monosemantic component (easily captured by SAEs). Across a comprehensive range of experiments, the authors show that this simple interpretation can indeed help explain a variety of SAE failures, including persistent dense latents and "dark matter" findings; and further experiments show both the causal necessity of the discovered "computational scaffold" and its resistance to SAE-style decomposition.

Critical strengths
  • The paper targets a range of important issues in the current mechanistic interpretability literature, proposing a single, simple answer to simultaneously explain convergent lines of evidence around failure modes of SAEs, including why certain structures encoded in foundation model representations are so resistant to SAE-style decomposition and interpretation. While I would not say that the current results yet conclusively resolve each issue (see questions to authors section, below), I find the evidence presented in this work sufficient to establish the "computational scaffold" hypothesis as the best available current answer to these pivotal questions. This is a clear and significant contribution to the modern study of mechanistic interpretability, and I expect the ideas introduced in this paper to help support a range of impactful future work (both towards better SAEs, and towards understanding the nature and cause of the so-called "computational scaffold").
  • Experiments are very comprehensive, considering a wide variety of potential causes of each key finding and eliminating most potential confounders. The additional evidence regarding the nature of the computational scaffold presented in sec 5 goes well above and beyond what is typically expected of most SAE papers, showing that the computational scaffold hypothesis is not only consistent with known SAE failure modes (as per the sec 4 results), but further showing (in sec 5) that the discovered subspace is both essential for next token prediction and cannot be easily decomposed into sparse, monosemantic latents.
    • (A brief note to the authors: from the abstract and intro, it initially appeared that you only experimented on a single model and layer; but of course it later becomes apparent in sec 4.2/appendix F that a subset of experiments were repeated across multiple models and a variety of residual stream layers, validating that several key findings generalize to these other settings. I would either briefly mention this up-front, or refrain from scoping results to a single model and layer combination in the abstract/intro -- otherwise, it would be easy for readers to prematurely dismiss these important findings amid the deluge of other work on closely related topics.)
  • Though it is not the main focus of this paper, I would like to briefly highlight the applied impact of this work: any future work seeking to operationalize potential applied benefits of SAEs (across safety, privacy, fairness/bias, and other applied areas where interpretability work is commonly leveraged) would benefit from the improved foundational understanding of LLM representation and the limits of SAEs, as explored in this work. For example, future applied work building on these findings will benefit from the linear bottlenecking procedure, which allows SAEs to dedicate their capacity towards the parts of model representations for which they are best suited.
Critical weaknesses

I do not see any critical weaknesses of the paper that would support rejection. While I do have several outstanding questions that I would like to see clarified in a future draft (see the questions for authors, below), I do not consider any of them to be substantive cause for rejection.

Questions for authors
  • The existing experiments (between both the main paper, and auxiliary experiments in the appendix) are, I believe, sufficient to validate the major contributions, and more than enough to cover the appropriate scope for a single paper on this topic. That said, I have a few questions for the authors about the existing results that could help motivate future work on the topic. (Please feel free to speculate!)
    • Do the authors have any hypothesis as to the SAEBench evaluation gap between BatchTopK and Matryoshka SAEs under the linear bottleneck in sec 4.1? Should this primarily be seen as a limitation of the BatchTopK methodology (compared to Matryoshka), a limitation of the generality of the computational scaffold hypothesis, or something else?
    • I am curious as to how the intervention-based metrics (SCR, RAVEL) and autointerp -- which, as observed in sec 4.1, can look worse under the linear bottleneck than baseline SAEs, at least for BatchTopK -- would fare for the other models and layers considered in appendix F. Again, I don't think this should be required within the scope of this work; but joint layerwise and cross-SAE-architectural analysis might be useful towards characterizing the nature of the computational scaffold.
  • A few nitpicks:
    • What dataset(s) are the experiments in sec 5 run over? Is it, e.g., the full ~500M-token training dataset reported in appendix G, a subset/held-out set, or something else? (Ideally, these should all be from a held-out test split.)
    • What are the criteria for ndeadn_{dead}? Is it the standard "zero activations per 10 million tokens" from Gao et al. (2024)?
    • In sec 5.1, it is stated that "a rank-r bottleneck explains >99% of the variance along each of the first ~r principal components" -- but unless I'm reading fig 3 wrong, this seems to actually be closer to 90-95% for most/all r? I suggest either clarifying where you're getting this number (and why it seems to conflict with the plot), or correct the number reported in the text.
    • Some of the effects in sec 5 (all three subsections + associated results) might be at least partially due to the even weighting of the loss function -- it would be interesting to see what happens given non-uniform weights (given that, e.g., the 24-rank bottleneck here is contributing the same to the loss function as sparse reconstruction across a 16K-feature dictionary, which might be especially important for some of the cross-entropy results). I've read appendix C, and I'm not sure that I completely buy that there is no reason to ablate the balance between the 3 loss terms -- but I do agree that the current setting makes sense as the default, and I don't see this as a major limitation.
  • Assuming the "computational scaffold" interpretation/argument in sec 6.1 is correct, do the authors have any hypotheses as to why this would be the case? That is: why would features represented by models be mostly sparse and monosemantic, encoded in superposition, and so on, while a low-rank subspace of activations seems to represent information in a completely different way (i.e., densely, in a way that seems to resist clean interpretation much more than the vast majority of other features)? Perhaps we should be thinking of this "dark matter" in a fundamentally different way, using distinct structures to encode different kinds of information, etc? Why might we expect such semantic bifurcation from models?
    • I see that Figure 2 (left) and associated discussion hints at what some of these features are -- but (a) this doesn't account for most dense features, even for r=0; and (b) even if we could cleanly decompose the "mystery" latents into straighforwardly-interpretable features, this still wouldn't explain why models do this. Again, this is arguably out of scope for the current paper; but even some basic "toy models of superposition"-level intuition on this point would be a valuable jumping-off point for future work to investigate in greater detail.
3 comments
AuthorsOpenReview ↗

Scope, a correction, and the questions on architectures and metrics

We sincerely thank the reviewer for the generous and insightful review, and for recognizing both the significance of the computational scaffold hypothesis and its potential applied impact. We are grateful for the careful reading, and we address the questions below.

Stating the scope up front. We agree, and the revised abstract and introduction state that the SAE improvements replicate across 10 further pairs of models, layers, and architectures. During the rebuttal period we also repeated the mechanistic analyses of Section 5 at Gemma-2-2B layer 18 and Llama-3-8B layer 16. At both sites the learned subspace again coincides with the top-24 principal subspace (20 of 24 principal cosines at or above 0.95), projecting it out raises cross-entropy far more than projecting out random subspaces (3.66 versus 1.92 and 2.54 versus 1.81), and the bottleneck reduces the number of dense latents (from 28 to 8 and from 34 to 19).

A clarification of Section 5.2. In response to Reviewer pgWd, we re-examined the comparison between removing the bottleneck and removing the top-24 PCA directions. The two conditions used different interventions, since the bottleneck condition removed the full output of the dense pathway, including its large mean component, while the PCA condition mean-ablated the subspace. Under matched interventions the two conditions have effects of similar magnitude, in line with the two subspaces sharing 21 of their 24 directions. The revised abstract and Section 5.2 report both the effect of mean-ablating the subspace and that of removing the whole dense pathway. The causal importance of the component relative to random subspaces, which the reviewer highlights, is unaffected.

BatchTopK versus Matryoshka. We view the gap mainly as a difference in how the two architectures allocate scaffold content, as opposed to a limitation of the hypothesis. A standard BatchTopK SAE spreads this content across many latents, as the 96% coverage criterion of Section 5.3 requires the 767 to 791 most aligned latents across seeds. Matryoshka concentrates it in 270 to 274 latents, consistent with its first group acting as an implicit channel for scaffold content (Appendix A). Removing the content therefore reorganizes more of the BatchTopK dictionary, and intervention metrics that ablate a small number of latents register this reorganization as a regression (Appendix H). In Matryoshka, the nested structure already isolates much of this content, so the bottleneck takes over part of an existing role and the rest of the dictionary changes less.

Intervention metrics and autointerp at other sites. These metrics are reported at Gemma-2-2B layer 12, the main setting of SAEBench. Within layer 12, the rank sweep in our response to Reviewer pgWd shows that SCR direction-1 on BatchTopK changes sign across ranks at matched training length (from −.671 at r = 24 to +.150 at r = 8), which suggests that a layerwise analysis would need several seeds per setting. The revised Section 6.3 notes this as a direction for future work.

AuthorsOpenReview ↗

Clarifications and the origin of the bifurcation

Data used in Section 5. The causal interventions use 200 sequences of 1024 tokens from monology/pile-uncopyrighted (Appendix K), Figure 3 and the dense-latent counts use OpenWebText. Following the reviewer's suggestion, we reran the causal interventions on 200 held-out sequences from the Pile test split. The baseline cross-entropy is 1.82 instead of 1.90, and every intervention changes by at most 0.14 nats. The revised paper reports the held-out results in Table 7 (Appendix K).

Dead and dense latents. A latent is counted as dead if it never activates on the 4,096,000 tokens of the SAEBench core evaluation (2,000 batches of two 1024-token OpenWebText sequences). This is fewer tokens than the 10M-token criterion of Gao et al. (2024), so our counts are an upper bound under that criterion. Dense latents are those with a firing rate above 0.1 on 407,600 OpenWebText tokens.

Per-component explained variance (Section 5.1). We thank the reviewer for the careful reading. Figure 3 plots the per-component explained variance of the combined reconstruction (bottleneck plus SAE), and the revised text and caption attribute it accordingly. At r = 24, the combined reconstruction exceeds 0.99 on 14 of the top 24 components for the OpenWebText data in the figure, and on 20 of 24 on Pile data, where the r = 0 SAE reaches this level on only one. The revised Section 5.1 states the count for the figure, and Appendix N gives the count on Pile data.

Loss weighting. In our implementation the reconstruction terms are per-token normalized squared errors, so they are on the same relative scale regardless of dictionary size. During the rebuttal period we ran two ablations of the coupling between the pathways at Gemma-2-2B layer 12 (r=24r = 24, 61k steps, seed 0). The first removes the joint term (λ=0\lambda = 0), so that each pathway is trained on its own reconstruction objective only. The second removes the stop-gradient, as reported in our response to Reviewer 1r9W.

EVSP top-1SP top-5Absorption ↓TPPSCR dir-1RAVELDense latents
Full method.805.761.863.084.054−.295.5584
Without joint term (λ=0\lambda = 0).805.766.858.107.062+.187.6148
Without stop-gradient.805.748.864.105.046+.044.5906

Both variants preserve explained variance (.805), and the number of dense latents still falls from 23 without the bottleneck to 8 and 6, respectively. Removing the joint term also preserves sparse-probing accuracy and improves all three intervention metrics, with SCR direction-1 rising from −.295 to +.187 and RAVEL from .558 to .614, above the .569 of the model without the bottleneck, while absorption rises from .084 to .107. The coupling between the two pathways thus trades intervention performance for a cleaner sparse dictionary with fewer dense latents and lower absorption, and the loss weighting offers a direct way to choose between these properties. Both ablations use a single seed. They are reported in the revised paper (Appendix C), and we will add further seeds and a sweep over the weights in the camera-ready version.

Why might models bifurcate? We are glad to speculate. One intuition comes from toy models of superposition (Elhage et al., 2022), where features that are both important and frequently active receive dedicated dimensions, while sparse features share dimensions in superposition.Content that is needed at nearly every token, such as the frequency of the current token or the confidence of the prediction, would then occupy a small number of dedicated high-variance directions outside the superposition regime that SAEs are designed for. A second intuition is that part of this structure may serve computational functions as opposed to semantic ones. At Gemma-2-2B layer 12 the scaffold aligns with the outlier dimensions documented by Bondarenko et al. (2023), with 47 of the top 50 neurons coinciding, and those authors link such outliers to attention heads that learn to perform no update. Functional directions of this kind would be dense by construction and would resist decomposition into monosemantic features. The revised Section 5.3 provides evidence for this second intuition, as the scaffold preferentially encodes token frequency and output entropy, and later attention layers read it mainly through their queries and keys. Testing these intuitions, for example in toy models that include a small set of dense and important features, is a direction we find exciting.

We thank the reviewer again for the encouraging assessment and the thought-provoking questions.

Reviewer_XvHUOpenReview ↗

Response to Authors

I would like to thank the authors for their detailed and thoughtful response. I appreciate the revisions in the updated draft, and the followup experiments -- I think adding something regarding your hypotheses around "why might models bifurcate" in a short future work paragraph in section 6 or 7 would be a good idea toward motivating future research to better understand this phenomenon, but I will leave that up to the authors.

My rating remains "clear accept". Well done!

Comments on the paper

2 comments
Area_Chair_EVjSOpenReview ↗

Discussion

Dear All,

Thank you for reviewing the paper. Please read the authors' response and leave a comment accordingly.

Yours,

AC

AuthorsOpenReview ↗

Revised Paper Uploaded and Summary of Changes

Dear Area Chair,

Thank you for coordinating the discussion. We have uploaded a revised version of the paper in which all changes are marked in blue, and we would like to summarize how it addresses the points raised by the reviewers.

Presentation. Following the reviewers' suggestions, we clarified that the paper examines whether all activation content is suited to sparse decomposition and that the bottleneck serves as a controlled test of this question, while we also expanded the discussion of related work, stated the training objective exactly as it is implemented, and moved the summary of the results across models and layers into the main text.

Experiments. The revision reports the experiments requested in the reviews, which comprise additional training seeds, a rank sweep that tracks all metrics at matched training length, a frozen PCA baseline, ablations of the coupling between the two pathways, matched causal controls for the comparison with PCA, a concept-detection evaluation on AxBench, and a replication of the mechanistic analyses at two additional sites.

A concrete account of what the scaffold does. In response to the question of how the scaffold differs from high-variance structure in general, the revised Section 5.3 characterizes its role directly. Although the 24 scaffold dimensions hold only 30% of the activation variance, they recover 73% of the linearly decodable variance of both token frequency and output entropy, compared with 25% and 21% for a control subspace of equal variance, and later attention layers read them mainly through their queries and keys, as removing the scaffold from this path changes the output distribution by 0.45 nats, compared with 0.08 nats on the value path. Both findings hold at three sites across two model families, and at Gemma-2-2B layer 12 we further show that a direction inside the scaffold has a graded causal effect on the confidence of the model. We believe that these results substantially strengthen the central claim of the paper, since they identify what this component carries and how the model uses it.

We have updated our responses to the reviewers so that they refer to the revised paper, and we would be glad to answer any further questions.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.