- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces a method called ResSAE, a two-stage sparse autoencoder. The first stage of the sparse autoencoder has two roles: firstly, it reconstructs the input as with a standard SAE; secondly, the active features in this stage selected a number of experts in the second stage. These experts are themselves SAEs, and act on the reconstruction residual from the first stage.
The authors report significantly reduced computational cost for a given MSE, at the expensive of an increased parameter and latent count.
Critical strengths
The paper is clearly positioned as a method for reducing the training cost of achieving a certain reconstruction quality at a fixed sparsity, at the expense of an increase parameter count. As with other gated SAEs, this is plausibly useful for certain tasks.
Critical weaknesses
The paper has limited novelty, as it combines feature-based routing similar to H-SAEs with residual encoding similar to matching-pursuit SAEs.
The main issue with the paper is that it does not robustly demonstrate an improvement over SwitchSAEs, which are the most relevant baseline for this method. The L_0=80 MSE reduction over SwitchSAEs in Table 2 is only 0.85%, which is a point estimate and confounded by the different tuning budgets for the two methods as reported in A.3. The comparisons also focus on FLOP budgets rather than token budgets - reporting both would improve the comparison.
Table 14 reports that, despite the considerably faster forward pass compared with flat TopK, the wall-time improvement is small.
Appendix C.2 only samples 24 experts in assessment of cross-expert similarity, and averages across features and pairs. Measuring the maximum cosine similarity across all other experts should be computationally inexpensive and provide a better indication of duplication.
Questions for authors
- How do SwitchSAEs and ResSAEs compare across a broader range of seeds, models, datasets, etc.?
- Have you observed larger wall-time improvements during training compared with other methods?
- How many features are duplicated across the entire dictionary, rather than across randomly sampled experts?
4 comments
Response to W3, Q2, W4 and Q3
Response to W3 and Q2: Training Speedup
Thank you. Forward speedups do not imply proportional training speedups. We added optimized comparisons and a training-time breakdown.
(1) Matched conditions. Both original implementations needed further optimization and alignment. We optimized both using Tensor Cores and Triton and benchmarked them on the same A100 80GB GPU, matching dictionary width, batch size (2048), and target sparsity (), with GPU-cached inputs. All ratios are TopK / ResSAE:
| Dictionary width | Forward FLOPs ratio | Forward speedup | Full training step speedup |
|---|---|---|---|
| 0.070M | 8.25× | 3.06× | 1.35× |
| 0.135M | 15.60× | 5.03× | 1.51× |
| 0.266M | 29.21× | 7.65× | 1.62× |
| 0.528M | 52.79× | 10.23× | 1.71× |
| 0.791M | 72.49× | 11.37× | 1.73× |
| 1.053M | 89.20× | 12.78× | 1.80× |
(2) What limits training speedup? FLOPs measure arithmetic work; forward runtime also depends on memory access, routing, TopK selection, and hardware utilization. Training adds backpropagation, gradient processing, and Adam updates, which do not accelerate proportionally. At the largest width, gradient processing and Adam take about 157 ms per method, or 72% of ResSAE’s summed stage times, limiting training speedup.
Training-step speedup increases from 1.35× to 1.80× with width. At the largest width, step time falls from 396.47 to 219.73 ms (44.6%).
The 3.06–12.78× full-SAE forward speedup also benefits repeated feature extraction, analysis, and interventions after training. Section 4.3 and Appendix B.2.1 report the FLOPs/runtime comparison, stage breakdown, and measurement protocol.
(3) Equal training time. We also compared ResSAE, Switch SAE, H-SAE, and KronSAE under an 80-minute single-GPU budget, with the same data, batch size 2048, and timing protocol, including data loading, transfers, and full training steps. ResSAE has the lowest MSE at and . At , its MSE is 6.96% lower than Switch SAE’s, with the highest CE/KL fidelity. These results measure reconstruction and downstream fidelity within the same GPU time (Section B.2.2, Table 15).
Response to W4 and Q3: Dictionary-Wide Feature Duplication
Thank you. Figure 9 in Appendix C.2 now covers all 2,048 experts, up from 24 sampled experts. For each of the 65,536 Level-2 features, we computed its maximum cosine similarity to any feature in another expert: 32 features (0.049%) have a neighbor with cosine > 0.9. Appendix C.2 now includes this statistic.
We also exhaustively searched the full dictionary of 2,048 Level-1 and 65,536 Level-2 features. For each feature, we computed its maximum cosine similarity to the other 67,583 features, excluding itself. Each feature is counted once per threshold; percentages use all 67,584 features as the denominator:
| Maximum cosine similarity to another feature in the full dictionary | Number of features above the threshold | Percentage |
|---|---|---|
| > 0.5 | 6,845 | 10.128% |
| > 0.7 | 1,072 | 1.586% |
| > 0.9 | 32 | 0.047% |
We define near-duplicates by cosine > 0.9. The checkpoint uses Gemma-2-2B layer-12 residual-stream activations, 500M training tokens, seed 42, and model-wide .
If these results address your main concerns, we would appreciate your reconsideration of the overall assessment and score.
Thanks for running these experiments, my concerns are largely addressed. I still believe the methodology to be an incremental improvement over existing work, but I'm more convinced of the usefulness of this method at improving reconstruction given fixed training budgets. As such, I've increased my score to a weak accept.
Thank you very much for your thoughtful reassessment and constructive feedback!
Response to W1, W2 and Q1
Thank you for your constructive feedback. We address your concerns with controlled ablations, matched-token comparisons, robustness checks, training-time benchmarks, and feature redundancy analysis.
Response to W1: Novelty
Thank you for pointing out this connection. We agree that both mechanisms have precedents; two ablations clarify their contributions in ResSAE.
(1) Coupling routing and residual encoding. Our “residual experts” couple feature-based routing with residual encoding. Table 3 (Section 4.4) gives the following absolute MSE reductions from the “original input + random routing” baseline at each sparsity budget:
At all three budgets, the combined gain exceeds the sum of the separate gains. This “1+1>2” effect supports a beneficial interaction between residual input and feature-based routing under the tested settings.
(2) Sharing the feature budget across experts. We compare ResSAE with H-SAE + residual encoding at L0=k1+k2∈{40,80,160}. All runs use 100M training tokens, the same data, batch size 2048, Level-1 width 4096, and per-expert Level-2 width 32. Paired runs share initialization, data order, and optimization settings:
ResSAE reduces MSE by 4.09%–4.86% (using unrounded values). Joint TopK lets each routed expert contribute multiple features or none, allocating the same total feature budget more flexibly to the current residual. These experiments and results for other allocation ratios appear in Section 4.4, “Why joint TopK?”, and Table 4.
The ablations support two design principles: features explaining part of the input can guide encoding of the remaining error; expert selection and feature-budget allocation can be decided separately. ResSAE implements these through Level-1 selection of residual experts and joint TopK allocation of their shared budget; the two ablations support these mechanisms, respectively.
Response to W2 and Q1: SwitchSAE Comparison and Robustness
Thank you. We added three controlled comparisons addressing training budgets and robustness.
(1) Matching training tokens. We trained an additional ResSAE checkpoint for approximately 100M tokens using the original L0≈40 configuration. All four methods now use approximately 100M Pile tokens. ResSAE’s MSE is 2.076, versus 2.172 for SwitchSAE: a 4.42% reduction.
(2) Matching encoding FLOPs. At L0≈80, SwitchSAE originally used 22.5M encoding FLOPs/token versus ResSAE’s 22.4M. We adjusted and retrained SwitchSAE so both use exactly 22,413,312 FLOPs/token on Gemma. Each newly added pair uses identical activation data batch by batch and exactly 100,003,840 training tokens. ResSAE’s MSE is 5.43% lower in the updated Table 2.
(3) Seeds, models, and corpora. With a common 100M-token budget, we tested two seeds on Gemma + Pile and added Gemma + FineWeb-Edu and Qwen + Pile. ResSAE has lower MSE in all four pairs (4.68%–37.50% reductions), along with lower feature absorption scores and fewer split features on average:
We updated Table 2 and the configurations in Appendix A.3, with revisions marked in red for ease of review.