- Rating
- 4(+2)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates whether some residual-stream activation content is represented inefficiently by standard sparse autoencoders. It introduces a small linear bottleneck alongside an SAE, with the SAE reconstructing the residual after subtracting the bottleneck contribution. Experiments with BatchTopK and Matryoshka show improvements in reconstruction, sparse probing, and targeted probe perturbation, together with fewer high-frequency latents. The authors characterize the absorbed component through its alignment with principal components and outlier dimensions, topic information, and effects on next-token prediction. They interpret this component as a computational scaffold that motivates separating dense and sparse reconstruction.
Critical strengths
S1. An interesting and simple architectural modification produces useful improvements. The idea of separating low-rank, dense information from sparse features is conceptually appealing and challenges the assumption that all residual-stream information should be sparsely decomposed. The bottleneck is small and compatible with existing SAE architectures. Improvements in reconstruction, sparse probing, and targeted probe perturbation suggest practical value beyond reducing dense latents. Replication across additional models and layers strengthens this contribution.
S2. The absorbed component is examined from several complementary perspectives. Architectural controls, geometric analyses, topic probes, and downstream interventions provide a richer characterization. The authors also report architecture-dependent regressions in absorption and intervention metrics, which helps identify the method’s practical trade-offs.
Critical weaknesses
W1. Limited evidence for generalization across independently trained models and different . Although the authors evaluate several bottleneck ranks and report multi-seed results for selected metrics, the rank sweep is restricted to one model, layer, and SAE architecture. A more systematic evaluation across independently initialized SAEs and different bottleneck ranks, tracking how interpretability and intervention metrics vary with (r), would strengthen the evidence for robustness and generalizability.
W2. On SAE architectures feature structure and concept alignment. Several relevant studies are missing. In [2] demonstrate that SAE architectures impose assumptions on concept geometry, potentially limiting feature recovery. In [1] introduce Kronecker-factorized SAEs with compositional 1-rank feature groups. Both provide relevant architectural context for the paper.
W3. Lack of guarantees on feature preservation. The proposed low-rank bottleneck has no theoretical guarantee that it will preserve useful, interpretable features in the sparse dictionary rather than absorbing them into the dense pathway. The observed degradation in some interpretability and intervention metrics raises concerns about this possibility. Although the authors discuss these metric regressions, they do not explicitly acknowledge the potential loss of interpretable sparse features as a limitation.
W4. Limited evaluation scope. The evaluation could benefit from additional benchmarks such as AxBench [3], experiments at other activation sites (e.g., after attention and MLP modules), and larger SAE dictionaries (e.g., 64K). Although not essential to the main claims, these experiments could strengthen the analysis and reveal interesting effects of activation location and dictionary capacity.
References:
[1] Kurochkin, V., Aksenov, Y., Laptev, D., Gavrilov, D., and Balagansky, N. Kronecker Factorization Improves Efficiency and Interpretability of Sparse Autoencoders. arXiv:2505.22255, 2025. https://arxiv.org/abs/2505.22255
[2] Hindupur, S. S. R., Lubana, E. S., Fel, T., and Ba, D. Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry. NeurIPS 2025. https://arxiv.org/abs/2503.01822
[3] Wu et al. (2025). AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders. ICML 2025. https://arxiv.org/abs/2501.17148
Questions for authors
Please see the weaknesses above for additional concerns. Also I would appreciate clarification on the following questions:
Q1. Why does ablating the learned bottleneck cause substantially greater degradation than removing the nearly identical PCA subspace? What explains this discrepancy?
Q2. What evidence distinguishes the proposed computational scaffold from ordinary high-variance activation structure?
Q3. How should the improvements in probing and reconstruction be interpreted alongside the degradation in some causal intervention metrics? Which metrics do the authors consider most indicative of improved interpretability?
8 comments
Feature preservation, scope, and metrics
Feature preservation (W3). We share the reviewer's concern and tested it directly. Structurally, the bottleneck is a linear map of rank 24, so it can capture at most 24 of the 2,304 directions of the residual stream, and the stop-gradient keeps the SAE objective from pushing content into it. Empirically, the sparse dictionary is preserved to the same degree as across random seeds. Matching each latent of the r=0 SAE to its nearest r=24 latent by decoder direction leaves 12.4%, 13.2%, and 13.7% unmatched across the three BatchTopK seeds, which is comparable to the 12.3% to 13.3% observed between r=0 SAEs that differ only in their random seed. On Matryoshka the corresponding fractions are 22.1% and 23.1%, again within the seed-to-seed range of 20.9% to 25.1%. The latents that do align with the bottleneck are few and mostly dense. Only 22 to 28 of the 16,384 latents have at least half of their decoder energy in the bottleneck subspace, and 67% to 87% of these are dense latents. For latents with less than a fifth of their energy in this subspace, the match rate equals the seed baseline. At the level of concepts, sparse-probing top-1 accuracy improves in all three seeds on six of the eight SAEBench datasets. The clearest exception is sentiment on Matryoshka (−.153 top-1 on Amazon sentiment), consistent with the partial absorption of sentiment discussed in Appendix H. Together, these results indicate that the bottleneck removes dense structure from the dictionary while leaving its sparse features intact. The revised paper reports this analysis in Appendix M, including the sentiment case.
Scope (W4). As the reviewer notes, these extensions lie outside the main claims of the paper. Our evaluation already covers 12 matched pairs across six sites, two model families, and two SAE architectures, with the full SAEBench suite at the main site. We focused on the residual stream because it aggregates the outputs of all components and is where low-rank broadcast structure is expected (Section 6.3). Attention and MLP output sites, 64K dictionaries, and steering evaluations on AxBench are natural extensions, which the revised paper mentions as future work, and the revision now includes a concept-detection evaluation on AxBench (Appendix M).
Interpreting the metrics (Q3). Apart from absorption on BatchTopK, which the paper reports alongside Table 4, the regressions concern two intervention metrics on one architecture. On BatchTopK, SCR direction-1 and RAVEL decrease (from +.206 to −.295 and from .569 to .558 against the baseline without the bottleneck), whereas on Matryoshka both improve (from +.263 to +.303 and from .575 to .583). TPP, which is also an intervention metric, improves on both architectures. One contributing factor is that SCR and RAVEL intervene on sparse latents only. When the content they target moves into the bottleneck, ablating sparse latents alone leaves it intact, and the metric registers this as a failure even if the combined system represents the content well. Appendix H illustrates this, as jointly ablating the top-8 bottleneck dimensions and the top-128 sparse atoms drives the gender probe to chance (0.503) while preserving profession accuracy (0.801). The regression also depends on how tightly the two pathways are coupled during training, and we ran two ablations of this coupling during the rebuttal period (61k steps, seed 0). Removing the full reconstruction objective (), so that each pathway is trained on its own objective only, raises SCR direction-1 from −.295 to +.187, close to the +.206 of the baseline without the bottleneck, and RAVEL from .558 to .614, above the baseline's .569. TPP also rises from .054 to .062, and sparse-probing top-1 accuracy is preserved (.766 compared with .761). Removing the stop-gradient instead raises SCR direction-1 to +.044 and RAVEL to .590, while lowering sparse-probing top-1 accuracy to .748. Both variants preserve explained variance (.805) and most of the reduction in dense latents (from 23 to 8 and 6), and in exchange absorption rises from .084 to .107 and .105. The SCR and RAVEL regressions therefore arise largely from the coupling between the pathways, as opposed to the low-rank pathway itself, and the loss weighting offers a direct way to trade dictionary cleanliness against intervention metrics. The revised paper reports both ablations (Appendix C), and we will add further seeds in the camera-ready version. More broadly, we regard sparse probing and TPP as the most direct indicators of dictionary quality, and interventions on the combined bottleneck and SAE as the appropriate test of disentanglement. The revised Section 6.2 states this recommendation explicitly.
The causal comparison and the nature of the scaffold
Causal comparison with PCA (Q1). The difference stems from the mean component of the activation, which the two conditions treat differently. The bottleneck condition removes the full output of the dense pathway, including the large mean component that it reconstructs, and thus measures how much the model relies on the dense pathway as a whole. The PCA condition mean-ablates the principal subspace and retains the mean, and thus isolates the role of the centered directions. The difference between them therefore reflects the mean component to a large extent, as removing the mean alone already raises cross-entropy from 1.90 to 3.52, while removing the full output of the dense pathway raises it to 9.29. Under a common centered intervention, the two subspaces have effects of similar magnitude, and the remaining difference stems mainly from the first token. Projecting them out gives 2.98 for the bottleneck and 5.18 for PCA, mean-ablation gives 3.75 and 5.28, and with the first token left untouched the two are nearly equal (2.36 and 2.29). This agrees with the geometry, as the two subspaces share 21 of their 24 directions. The revised Section 5.2 and abstract report both quantities, namely the effect of mean-ablating the subspace (3.75) and that of removing the whole dense pathway (9.3), and Table 7 compares the two subspaces under matched interventions. The causal importance of the component itself holds under every intervention we tested, as removing random 24-dimensional subspaces leaves cross-entropy at 1.91, and this holds at all three sites. These matched controls further strengthen the causal analysis.
Scaffold versus high-variance structure (Q2). High variance alone is a property of the activation, whereas the scaffold is defined by how this structure interacts with sparse dictionaries and with the model. The learned subspace coincides with the dominant principal components, sharing 20 to 21 of its 24 directions with the top 24 at every site we examined, and this coincidence is central to the finding, since this structure dominates the reconstruction objective of every SAE. Three properties distinguish it. First, standard SAEs spend dense latents on it. At Gemma-2-2B layer 12, their dense latents place 0.66 of their decoder energy in the bottleneck subspace on average, compared with a median of 0.04 across all latents. Second, the SAE reconstructs it incompletely, with at most one of the top-24 principal components above 0.99 explained variance at any of the three sites, whereas adding the 24-dimensional bottleneck brings 19 to 20 of them to this level. Third, the model relies on it causally at all three sites (Q1). The revised Section 5.3 adds a direct comparison with directions of equal variance. Although the 24 scaffold dimensions hold only 30% of the activation variance, they recover 73% of the linearly decodable variance of both token frequency and output entropy, compared with 25% and 21% for the 323 principal components that follow them and hold the same total variance, and later attention layers read the scaffold mainly through their queries and keys. Separating this structure frees the dictionary, reducing dense latents from 25 to 5 in the rank sweep above and improving it across seeds and ranks (W1). The term scaffold names this combination of properties, and the revised Section 5 states this explicitly.
We hope these additions address the concerns that the reviewer described as minor or fixable. We also believe that the matched causal analysis strengthens the central claim of the paper, namely that part of the residual stream is better represented outside the sparse dictionary, which calls into question the standard practice of reconstructing the entire activation with an SAE. We would be grateful if the reviewer would consider raising the score in light of these results.
Robustness across seeds and ranks, and related work
We thank the reviewer for the thoughtful review and are encouraged that the reviewer finds the modification simple and conceptually appealing, its improvements of practical value, and our analyses complementary. The reviewer's concerns center on robustness and scope, and we have addressed them with new experiments, namely (i) additional training seeds, (ii) a step-matched rank sweep that tracks all interpretability and intervention metrics, (iii) a direct test of feature preservation, (iv) the mechanistic analyses at two additional sites, and (v) matched causal controls for the comparison raised in Q1. We summarize each below.
Independently trained SAEs (W1). We trained additional seeds of the BatchTopK pair in Table 1, with both models of each pair trained for the full 61k steps. Across the original seed and two additional seeds, the bottleneck improves explained variance by 0.8, 1.2, and 1.6 points and sparse-probing top-1 accuracy by 5.4, 5.8, and 3.2 points(single evaluation seed, whereas Table 1 averages sparse probing over three evaluation seeds and reports 4.0 points for the original seed). Among the intervention metrics, TPP@20 improves in both seeds for which we measured it (+.017 and +.012), while SCR direction-1 declines in both (from +.206 to −.295 and from +.221 to −.065). The learned subspace is also reproducible, as 22 to 23 of its 24 directions agree across seeds with a cosine above 0.99.
Bottleneck rank (W1). The table below extends the BatchTopK rank sweep at Gemma-2-2B layer 12 with all interpretability and intervention metrics. For a like-for-like comparison across ranks, every model in this table, including r=24, is trained for 30k steps, and Table 3 of the revised paper follows the same protocol.
Explained variance, top-5 probing accuracy, and TPP improve over r = 0 at every rank, and dense latents fall steadily up to r = 24 and then plateau. Absorption rises beyond r = 24, and RAVEL decreases slightly at r ≥ 48. SCR direction-1 changes sign across ranks, which together with its variation across seeds suggests that it is a noisy indicator in this setting. The gains in reconstruction, top-5 probing, and TPP therefore hold across ranks, and r = 24 lies where dense latents are largely removed before absorption begins to rise. At Llama-3-8B layer 16, where the dense structure extends beyond the top 24 principal components, increasing the rank from 24 to 96 further reduces the number of dense latents from 19 to 1 (34 without the bottleneck) and raises explained variance from .648 to .672 and sparse-probing top-1 accuracy from .807 to .817. A Matryoshka sweep was beyond the time available during the rebuttal period.
Related work (W2). We thank the reviewer for these references and have added both to the revised paper. Hindupur et al. (2025) show that each SAE architecture imposes assumptions on concept geometry that determine which concepts it can recover. Our work examines a related but distinct assumption, namely that all activation content is amenable to sparse decomposition. Kurochkin et al. (2026) propose a Kronecker-factorized SAE with compositional feature groups, which the revised paper includes among the architectural variants discussed in Section 2.