acceptodds
Under review as a conference paper at ICLR 2027

Residual Experts for Sparse Autoencoding

Abstract

Sparse Autoencoders (SAEs) are widely used to extract interpretable features from language models, but scaling full-dictionary SAEs to capture diverse features in large models makes sparse autoencoding computationally challenging. We introduce ResSAE, a two-stage sparse autoencoder that combines conditional computation with residual refinement. A lightweight Level-1 SAE forms an initial reconstruction and uses its active features to select residual experts, small Level-2 SAEs that encode the reconstruction residual. A joint TopK operation allocates the Level-2 sparsity budget across the selected experts. Experiments on Gemma-2-2B residual stream activations show that ResSAE achieves lower reconstruction MSE and higher CE/KL fidelity than representative efficient SAE baselines at matched effective dictionary widths, sparsity levels, and encoder FLOPs. Compared with 16K-feature full-dictionary baselines, ResSAE accesses a 135K-feature space with 56—74% fewer encoder FLOPs while achieving up to 11% lower reconstruction MSE. Controlled ablations show that both residual input and Level-1 routing contribute to the reconstruction gains of residual experts. ResSAE further exhibits an emergent coarse-to-fine feature organization, with reduced feature splitting and absorption while maintaining auto-interpretability comparable to width-matched full-dictionary SAEs. Code and checkpoints will be released.

Primary area:
interpretability and explainable AI
Keywords:
Sparse Autoencoders, Mechanistic Interpretability, Language Model Interpretability, Conditional Computation, Residual Learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_EckxOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a method called ResSAE, a two-stage sparse autoencoder. The first stage of the sparse autoencoder has two roles: firstly, it reconstructs the input as with a standard SAE; secondly, the active features in this stage selected a number of experts in the second stage. These experts are themselves SAEs, and act on the reconstruction residual from the first stage.

The authors report significantly reduced computational cost for a given MSE, at the expensive of an increased parameter and latent count.

Critical strengths

The paper is clearly positioned as a method for reducing the training cost of achieving a certain reconstruction quality at a fixed sparsity, at the expense of an increase parameter count. As with other gated SAEs, this is plausibly useful for certain tasks.

Critical weaknesses

The paper has limited novelty, as it combines feature-based routing similar to H-SAEs with residual encoding similar to matching-pursuit SAEs.

The main issue with the paper is that it does not robustly demonstrate an improvement over SwitchSAEs, which are the most relevant baseline for this method. The L_0=80 MSE reduction over SwitchSAEs in Table 2 is only 0.85%, which is a point estimate and confounded by the different tuning budgets for the two methods as reported in A.3. The comparisons also focus on FLOP budgets rather than token budgets - reporting both would improve the comparison.

Table 14 reports that, despite the considerably faster forward pass compared with flat TopK, the wall-time improvement is small.

Appendix C.2 only samples 24 experts in assessment of cross-expert similarity, and averages across features and pairs. Measuring the maximum cosine similarity across all other experts should be computationally inexpensive and provide a better indication of duplication.

Questions for authors
  1. How do SwitchSAEs and ResSAEs compare across a broader range of seeds, models, datasets, etc.?
  2. Have you observed larger wall-time improvements during training compared with other methods?
  3. How many features are duplicated across the entire dictionary, rather than across randomly sampled experts?
4 comments
AuthorsOpenReview ↗

Response to W1, W2 and Q1

Thank you for your constructive feedback. We address your concerns with controlled ablations, matched-token comparisons, robustness checks, training-time benchmarks, and feature redundancy analysis.

Response to W1: Novelty

Thank you for pointing out this connection. We agree that both mechanisms have precedents; two ablations clarify their contributions in ResSAE.

(1) Coupling routing and residual encoding. Our “residual experts” couple feature-based routing with residual encoding. Table 3 (Section 4.4) gives the following absolute MSE reductions from the “original input + random routing” baseline at each sparsity budget:

ModificationL0=40L_0=40L0=80L_0=80L0=160L_0=160
Residual input only0.190.130.41
Level-1 routing only0.790.490.32
Sum of the two separate gains0.19+0.79=0.980.19+0.79=0.980.13+0.49=0.620.13+0.49=0.620.41+0.32=0.730.41+0.32=0.73
Gain from combining both1.080.961.08
Additional gain beyond the sum of separate gains0.100.340.35

At all three budgets, the combined gain exceeds the sum of the separate gains. This “1+1>21+1>2” effect supports a beneficial interaction between residual input and feature-based routing under the tested settings.

(2) Sharing the feature budget across experts. We compare ResSAE with H-SAE + residual encoding at L0=k1+k2∈{40,80,160}L_0=k_1+k_2\in\{40,80,160\}. All runs use 100M training tokens, the same data, batch size 2048, Level-1 width 4096, and per-expert Level-2 width 32. Paired runs share initialization, data order, and optimization settings:

Methodk1k_1k2k_2Level-2 feature selectionValidation MSE ↓
H-SAE + residual encoding2020Top1 per expert1.93
ResSAE2020Joint Top20 across experts1.85
H-SAE + residual encoding4040Top1 per expert1.61
ResSAE4040Joint Top40 across experts1.53
H-SAE + residual encoding8080Top1 per expert1.31
ResSAE8080Joint Top80 across experts1.24

ResSAE reduces MSE by 4.09%–4.86% (using unrounded values). Joint TopK lets each routed expert contribute multiple features or none, allocating the same total feature budget more flexibly to the current residual. These experiments and results for other allocation ratios appear in Section 4.4, “Why joint TopK?”, and Table 4.

The ablations support two design principles: features explaining part of the input can guide encoding of the remaining error; expert selection and feature-budget allocation can be decided separately. ResSAE implements these through Level-1 selection of residual experts and joint TopK allocation of their shared budget; the two ablations support these mechanisms, respectively.

Response to W2 and Q1: SwitchSAE Comparison and Robustness

Thank you. We added three controlled comparisons addressing training budgets and robustness.

(1) Matching training tokens. We trained an additional ResSAE checkpoint for approximately 100M tokens using the original L0≈40L_0\approx40 configuration. All four methods now use approximately 100M Pile tokens. ResSAE’s MSE is 2.076, versus 2.172 for SwitchSAE: a 4.42% reduction.

(2) Matching encoding FLOPs. At L0≈80L_0\approx80, SwitchSAE originally used 22.5M encoding FLOPs/token versus ResSAE’s 22.4M. We adjusted and retrained SwitchSAE so both use exactly 22,413,312 FLOPs/token on Gemma. Each newly added pair uses identical activation data batch by batch and exactly 100,003,840 training tokens. ResSAE’s MSE is 5.43% lower in the updated Table 2.

(3) Seeds, models, and corpora. With a common 100M-token budget, we tested two seeds on Gemma + Pile and added Gemma + FineWeb-Edu and Qwen + Pile. ResSAE has lower MSE in all four pairs (4.68%–37.50% reductions), along with lower feature absorption scores and fewer split features on average:

Model and training corpusSeedMethodEncoding FLOPs/tokenMSE ↓Absorption score ↓Average number of split features ↓
Gemma-2-2B L12 + Pile123SwitchSAE22.4M1.8830.3212.462
Gemma-2-2B L12 + Pile123ResSAE22.4M1.7500.0521.077
Gemma-2-2B L12 + Pile456SwitchSAE22.4M1.8750.3693.192
Gemma-2-2B L12 + Pile456ResSAE22.4M1.7500.0791.077
Gemma-2-2B L12 + FineWeb-Edu123SwitchSAE22.4M1.8360.4101.769
Gemma-2-2B L12 + FineWeb-Edu123ResSAE22.4M1.7500.0961.038
Qwen3-30B-A3B-Base L24 + Pile subset456SwitchSAE19.9M0.04980.6171.462
Qwen3-30B-A3B-Base L24 + Pile subset456ResSAE19.9M0.03110.0471.269

We updated Table 2 and the configurations in Appendix A.3, with revisions marked in red for ease of review.

AuthorsOpenReview ↗

Response to W3, Q2, W4 and Q3

Response to W3 and Q2: Training Speedup

Thank you. Forward speedups do not imply proportional training speedups. We added optimized comparisons and a training-time breakdown.

(1) Matched conditions. Both original implementations needed further optimization and alignment. We optimized both using Tensor Cores and Triton and benchmarked them on the same A100 80GB GPU, matching dictionary width, batch size (2048), and target sparsity (L0=40L_0=40), with GPU-cached inputs. All ratios are TopK / ResSAE:

Dictionary widthForward FLOPs ratioForward speedupFull training step speedup
0.070M8.25×3.06×1.35×
0.135M15.60×5.03×1.51×
0.266M29.21×7.65×1.62×
0.528M52.79×10.23×1.71×
0.791M72.49×11.37×1.73×
1.053M89.20×12.78×1.80×

(2) What limits training speedup? FLOPs measure arithmetic work; forward runtime also depends on memory access, routing, TopK selection, and hardware utilization. Training adds backpropagation, gradient processing, and Adam updates, which do not accelerate proportionally. At the largest width, gradient processing and Adam take about 157 ms per method, or 72% of ResSAE’s summed stage times, limiting training speedup.

Training-step speedup increases from 1.35× to 1.80× with width. At the largest width, step time falls from 396.47 to 219.73 ms (44.6%).

The 3.06–12.78× full-SAE forward speedup also benefits repeated feature extraction, analysis, and interventions after training. Section 4.3 and Appendix B.2.1 report the FLOPs/runtime comparison, stage breakdown, and measurement protocol.

(3) Equal training time. We also compared ResSAE, Switch SAE, H-SAE, and KronSAE under an 80-minute single-GPU budget, with the same data, batch size 2048, and timing protocol, including data loading, transfers, and full training steps. ResSAE has the lowest MSE at L0=40L_0=40 and 8080. At L0=80L_0=80, its MSE is 6.96% lower than Switch SAE’s, with the highest CE/KL fidelity. These results measure reconstruction and downstream fidelity within the same GPU time (Section B.2.2, Table 15).

Response to W4 and Q3: Dictionary-Wide Feature Duplication

Thank you. Figure 9 in Appendix C.2 now covers all 2,048 experts, up from 24 sampled experts. For each of the 65,536 Level-2 features, we computed its maximum cosine similarity to any feature in another expert: 32 features (0.049%) have a neighbor with cosine > 0.9. Appendix C.2 now includes this statistic.

We also exhaustively searched the full dictionary of 2,048 Level-1 and 65,536 Level-2 features. For each feature, we computed its maximum cosine similarity to the other 67,583 features, excluding itself. Each feature is counted once per threshold; percentages use all 67,584 features as the denominator:

Maximum cosine similarity to another feature in the full dictionaryNumber of features above the thresholdPercentage
> 0.56,84510.128%
> 0.71,0721.586%
> 0.9320.047%

We define near-duplicates by cosine > 0.9. The checkpoint uses Gemma-2-2B layer-12 residual-stream activations, 500M training tokens, seed 42, and model-wide L0≈40L_0\approx40.

If these results address your main concerns, we would appreciate your reconsideration of the overall assessment and score.

Thanks for running these experiments, my concerns are largely addressed. I still believe the methodology to be an incremental improvement over existing work, but I'm more convinced of the usefulness of this method at improving reconstruction given fixed training budgets. As such, I've increased my score to a weak accept.

Thank you very much for your thoughtful reassessment and constructive feedback!

Reviewer_YzfZOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Through this work, the authors present ResSAE, a two-stage SAE that combines residual reconstruction with conditional expert selection. The first level uses a smaller SAE to reconstruct the input and select which second-level experts to activate. These experts then work on the remaining reconstruction error. A shared TopK operation selects features over the active experts. The authors evaluate ResSAE mainly on Gemma-2-2B residual stream activations and compare it against several existing SAE architectures. The results show better reconstruction and downstream fidelity at comparable sparsity along with encoder FLOPs. Compared to 16K-feature flat SAEs, ResSAE uses a much larger dictionary while also reducing encoder FLOPs. The paper also investigates why residual input and feature-based routing help, and finds that both improve reconstruction quality. Finally, the authors show examples of coarse-to-fine feature organization and report lower feature splitting and absorption on their benchmark.

Critical strengths
  1. I really liked the idea of combining residual reconstruction with conditional computation. Instead of scoring a large dictionary for every token, you use the first-level features to decide which experts should process the remaining residual. I think this is a simple but effective idea, and it makes sense to use the information already learned by Level 1 for routing rather than having a separate router.

  2. The reconstruction and efficiency results are quite good. In particular, ResSAE achieves lower reconstruction error while using significantly fewer encoder FLOPs than flat SAEs. This matters because scaling SAE dictionaries is computationally expensive, and this approach lets you access much larger dictionaries without scoring every feature. I also liked the breadth of comparison (multiple SAE architectures).

  3. The ablation experiments are also comprehensive. The authors separately test the effects of residual input and Level-1 routing, and Table 3 shows that combining both works better than either alone. The additional analysis of residual subspace coverage is also useful.

Critical weaknesses
  1. I have a small concern about the training budget used in some baseline comparisons. In Table 2, the ResSAE checkpoint at L0 around 40 uses 500M training tokens, while the other baselines use 100M tokens. I understand that you mention this, and the other experiments provide additional evidence for ResSAE. But I think a comparison using the same number of training tokens would make the result more convincing.

  2. I would also like to better understand the practical computation tradeoff. Appendix B.2.1 reports a large improvement in forward latency, but the improvement in overall training time is much smaller. I think discussing this difference a bit more would help readers understand when ResSAE is most useful.

Questions for authors

Q1. In Table 2, the ResSAE checkpoint at L0 around 40 uses 500M tokens, while the baselines use 100M. Do you have results for ResSAE trained on 100M tokens under the same configuration? I would like to see how much of the improvement remains with exactly the same training budget. Q2. Appendix B.2.1 shows a large improvement in forward latency, but only a small improvement in actual training time. What is the main reason for this difference? Do you expect the training efficiency advantage to become larger as dictionary sizes increase? Q3. I found the coarse-to-fine feature organization interesting. Have you looked at how frequently this pattern appears across the dictionary? I think a quantitative analysis of how often Level-2 features represent meaningful refinements of their Level-1 anchors would make this result even more interesting.

3 comments
AuthorsOpenReview ↗

Response to Weaknesses and Questions

We sincerely thank the reviewer for the positive assessment of our work and the thoughtful, constructive suggestions. Your questions helped us strengthen the experimental comparisons, clarify the practical efficiency gains, and add quantitative evidence for coarse-to-fine feature organization. We address each point below with additional experiments and clarifications.

Response to W1 and Q1: Equal Token Budgets

Thank you. We made two corrections:

(1) Matching training tokens. We trained an additional ResSAE checkpoint for approximately 100M tokens using the original L0≈40L_0\approx40 configuration. All four methods now use approximately 100M Pile tokens. ResSAE’s MSE is 2.076, versus 2.172 for SwitchSAE: a 4.42% reduction.

(2) Matching encoding FLOPs. At L0≈80L_0\approx80, SwitchSAE originally used 22.5M encoding FLOPs/token versus ResSAE’s 22.4M. We adjusted and retrained SwitchSAE so both use exactly 22,413,312 FLOPs/token on Gemma. Each newly added pair uses identical activation data batch by batch and exactly 100,003,840 training tokens. ResSAE’s MSE is 5.43% lower in the updated Table 2.

The updated Table 2 is:

MethodNeffN_{\rm eff}L0L_0Enc. FLOPsMSE ↓CE fidelity ↑KL fidelity ↑
Switch SAE67,20040.011.2M2.1720.9700.970
H-SAE67,58439.711.2M2.4690.9520.954
KronSAE67,20040.011.2M2.4380.9540.955
ResSAE (ours)67,58440.011.2M2.0760.9720.972
Switch SAE135,40880.022.4M1.8590.9800.980
H-SAE135,16878.122.4M2.0310.9750.976
KronSAE131,04080.022.7M1.9300.9790.979
ResSAE (ours)135,16880.022.4M1.7580.9840.984

We updated Table 2 in the main text and the configurations in Appendix A.3.

Response to W2 and Q2: Speedup and Dictionary Scaling

Thank you. We added optimized comparisons at six dictionary widths and a training-time breakdown in Section 4.3 and Appendix B.2.1.

(1) Matched conditions. Both original implementations needed further optimization and alignment. Both now use Tensor Cores and Triton, with the same A100 80GB GPU, dictionary widths, batch size (2048), and target sparsity (L0=40L_0=40). Inputs are cached on the GPU; timing excludes data preparation and transfer. The speedups concern complete training steps, not end-to-end training runs. All ratios are TopK / ResSAE:

Dictionary sizeForward FLOPs ratioForward speedupFull training-step speedup
0.070M8.25×3.06×1.35×
0.135M15.60×5.03×1.51×
0.266M29.21×7.65×1.62×
0.528M52.79×10.23×1.71×
0.791M72.49×11.37×1.73×
1.053M89.20×12.78×1.80×

(2) Why the speedups differ. FLOPs measure arithmetic work, while forward latency also depends on memory access, routing, TopK selection, and hardware utilization. Training adds backpropagation, gradient processing, and Adam updates, which do not accelerate proportionally. At the largest width, gradient processing and Adam take about 156 ms for TopK and 158 ms for ResSAE, accounting for 72% of ResSAE’s summed stage times and limiting training speedup.

(3) How dictionary width affects training gains. Over the tested 0.070M–1.053M range, training-step speedup increases from 1.35× to 1.80×. At the largest width, step time falls from 396.47 to 219.73 ms (44.6%). Training gains over flat TopK therefore increase with width in this range, subject to gradient-processing and optimizer overhead.

The strongest gains are in forward computation with large dictionaries, making ResSAE particularly useful for feature extraction, analysis, and interventions that repeatedly apply an SAE.

Response to Q3: Coarse-to-Fine Feature Frequency

Thank you. Using the same 500M-token checkpoint as the main-text qualitative analysis, we automated the analysis of parent–child semantic relationships, with mismatched parents matched for activation frequency as controls.

Among 831 paired comparisons whose true and mismatched feature labels both passed validation, 31.65% (263/831) of true pairs were judged to show semantic refinement, versus 1.56% (13/831) of mismatched pairs. This supports an association between refinement and the actual parent–child relationships.

Across the dictionary, 6,638 parent–child pairs (5.06% of all pairs) passed automated label validation on held-out activating text. Of these, 1,977 (29.78%), spanning 396 Level-1 anchors, were judged meaningful semantic refinements. These results provide exploratory quantitative support for the coarse-to-fine organization illustrated in the main text.

I am satisfied with the response. My concerns have been addressed.

Thank you very much for your feedback and support !

Reviewer_V3GNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper presents ResSAE, a sparse autoencoder that combines feature-based routing with residual refinement. It reports improved reconstruction and downstream fidelity in comparisons based on encoder FLOPs, as well as reduced feature absorption.

Critical strengths
  • The paper is clearly written.

  • The idea is simple and combines established mechanisms.

  • The experiments are appropriate for evaluating performance in terms of encoder FLOPs.

Critical weaknesses

W1 Practical training efficiency

W1.1 Limited improvement in training time

My main concern is whether the reductions in encoder FLOPs translate into useful reductions in training time and GPU cost. Appendix B.2.1 reports a 5.1× forward-pass speedup, but cached-activation training takes 8.4 hours for ResSAE versus 9.0 hours for TopK, only a 1.07× speedup. Thus, the large forward-pass improvement translates into a much smaller improvement in training time, even with cached activations.

W1.2 Unequal implementation optimization

The implementation comparison also favors ResSAE: the TopK baseline uses dense PyTorch operations, while ResSAE uses custom Triton kernels. Optimized sparse kernels for TopK SAE training are publicly available (for example [1], [2], [3]). The authors should benchmark an optimized TopK implementation to establish the practical efficiency advantage.

W1.3 Unequal training token budgets

There is also a training-budget mismatch in the L0 ≈ 40 block of Table 2: ResSAE is evaluated after 500M tokens, while the baselines receive 100M tokens. Matching encoder FLOPs per token does not make this a comparison at equal total training compute or training time.

W1.4 Missing comparisons at equal training time

The authors should report reconstruction and functional metrics at equal wall-clock training budgets using optimized baselines, particularly for Table 2, Figure 3, and the other main performance comparisons. Without these results, I remain unconvinced that the reported FLOP advantages yield a meaningful benefit in practical settings where GPU time determines training cost.

W2 Incremental architectural contribution

I am also concerned about novelty. SAE Boost already uses residual correction without routing, while Switch SAE uses routing without residual refinement. H-SAE already uses high-level features to select associated low-level SAEs, although those SAEs encode projections of the original activation. MP-SAE already performs iterative residual refinement by repeatedly scoring a shared dictionary against the current residual. I understand that ResSAE combines feature-based routing with residual refinement and trains both stages jointly, but the resulting contribution still feels incremental.


Kernels:

[1] OpenAI, sparse_autoencoder: Triton kernels for sparse SAE operations.

[2] Meta, Memory Layers at Scale: weighted embedding-bag Triton kernels.

[3] Lindsey et al., The Circuits Research Landscape: Results and Perspectives, Appendix E, Sparse Kernels (EleutherAI), 2025.

Questions for authors

See Weaknesses

6 comments
AuthorsOpenReview ↗

Response to W1.1 and W1.2

Thank you for the careful assessment and constructive feedback. Below, we address your concerns with benchmarks using optimized implementations, comparisons at matched token and training-time budgets, and ablations clarifying the contributions of residual experts and joint TopK.

Response to W1.1: Training Time and GPU Cost

Thank you. We agree that forward speedup does not imply proportional training-time or GPU-cost savings; the original 8.4 versus 9.0 hours showed limited training gains. We therefore optimized both implementations and measured full training steps at six matched widths.

SAE cost includes both training and repeated use. Training efficiency affects construction cost; forward efficiency independently benefits feature extraction, activation analysis, and interventions. We assess these separately, using encoder FLOPs for per-encoding arithmetic cost and measured forward/training-step latencies for runtime (Section 4.3 and Appendix B.2.1).

(1) Matched optimized implementations. The original comparison did not sufficiently align implementation optimization. Both methods now use Tensor Cores and Triton on the same A100 80GB GPU, with matched dictionary widths, batch size (2048), and target sparsity (L0=40L_0=40). With inputs cached on the GPU, timings include forward computation, loss, backpropagation, gradient processing, and optimizer updates, but exclude data preparation and transfer. All ratios are TopK / ResSAE:

Dictionary sizeForward FLOPs ratioForward speedupFull training-step speedup
0.070M8.25×3.06×1.35×
0.135M15.60×5.03×1.51×
0.266M29.21×7.65×1.62×
0.528M52.79×10.23×1.71×
0.791M72.49×11.37×1.73×
1.053M89.20×12.78×1.80×

(2) What limits training speedup? Reducing FLOPs leaves memory-access, routing, and TopK-selection overhead. Training adds backpropagation, gradient processing, and Adam updates, which do not accelerate proportionally. At the largest width, gradient processing and Adam take about 156 ms for TopK and 158 ms for ResSAE, accounting for 72% of ResSAE’s summed stage times and limiting training speedup.

(3) Training compute saved. Training-step speedup increases from 1.35× to 1.80× across the tested widths. At the largest width, step time falls from 396.47 to 219.73 ms (44.6%). For the same GPU, batch size, and number of steps, this corresponds to 26%–45% less steady-state training compute time, with larger gains at larger widths.

Response to W1.2: Comparison with Optimized TopK

Thank you. The original standalone forward benchmark used dense decoding for TopK but custom Triton kernels for ResSAE; this optimization mismatch was our oversight. We optimized and re-benchmarked both methods.

(1) Forward benchmark versus training implementation. The TopK training implementation already used Triton sparse decoding and encoder weight-gradient computation, but the standalone forward benchmark omitted this sparse path:

# Sparse decoding
recons = TritonDecoderAutograd.apply(
    inds, vals, self.decoder.weight
)

# Sparse encoder-weight gradient
triton_sparse_transpose_dense_matmul(
    all_inds, all_grad_vals, x, N=self.n_dirs_local
)

We now use optimized TopK consistently for both forward and full training-step measurements.

(2) Optimization details. TopK uses BF16 Tensor Cores for dense encoding and Triton for reconstruction, AuxK decoding, and associated sparse gradients. ResSAE uses TF32x3 Tensor Cores for Level-2 encoding and its gradients, and shares Triton sparse operators with TopK. Both retain FP32 parameters and Adam states, use fused Adam, and passed output and gradient checks for the optimized operators.

The six-width comparisons and training-time breakdown in our W1.1 response show 3.06×–12.78× forward and 1.35×–1.80× training-step speedups over optimized TopK. Appendix B.2.1 details the implementations and measurement conditions, with results in Section 4.3.

AuthorsOpenReview ↗

Response to W1.3 and W1.4

Response to W1.3: Matching Training Token Budgets

Thank you. We made two corrections:

(1) Matching training tokens. We trained an additional ResSAE checkpoint for approximately 100M tokens using the original L0≈40L_0\approx40 configuration. All four methods now use approximately 100M Pile tokens. ResSAE’s MSE is 2.076, versus 2.172 for SwitchSAE: a 4.42% reduction.

(2) Matching encoding FLOPs. At L0≈80L_0\approx80, SwitchSAE originally used 22.5M encoding FLOPs/token versus ResSAE’s 22.4M. We adjusted and retrained SwitchSAE so both use exactly 22,413,312 FLOPs/token on Gemma. Each newly added pair uses identical activation data batch by batch and exactly 100,003,840 training tokens. ResSAE’s MSE is 5.43% lower in the updated Table 2.

The updated Table 2 is:

MethodNeffN_{\rm eff}L0L_0Enc. FLOPsMSE ↓CE fidelity ↑KL fidelity ↑
Switch SAE67,20040.011.2M2.1720.9700.970
H-SAE67,58439.711.2M2.4690.9520.954
KronSAE67,20040.011.2M2.4380.9540.955
ResSAE (ours)67,58440.011.2M2.0760.9720.972
Switch SAE135,40880.022.4M1.8590.9800.980
H-SAE135,16878.122.4M2.0310.9750.976
KronSAE131,04080.022.7M1.9300.9790.979
ResSAE (ours)135,16880.022.4M1.7580.9840.984

We updated Table 2 in the main text and the configurations in Appendix A.3.

Response to W1.4: Comparison at Equal Training Time

Thank you. Section B.2.2 now includes equal-time comparisons for the Table 2 and Figure 3 configurations. All methods use optimized implementations, a single A100 80GB GPU, batch size 2048, seed 42, and the same training data pool and shuffling rules. Timing includes data loading, transfers, and full training steps; it excludes one-time setup, checkpoint saving, and evaluation.

(1) Table 2 configurations at 80 minutes. Table 15 compares ResSAE, Switch SAE, H-SAE, and KronSAE at L0=40L_0=40 and 8080, reporting MSE, CE/KL fidelity, measured L0L_0, and cumulative training tokens. ResSAE has the lowest MSE at both sparsities. At L0=80L_0=80, its MSE is 6.96% lower than Switch SAE’s, with the highest CE/KL fidelity among all four methods.

(2) Figure 3 configurations over 120 minutes. Figure 8 plots reconstruction MSE against actual training time for ResSAE (135,168 features) and TopK (16,384 features) at L0=20,40,80,160L_0=20,40,80,160, following one training trajectory per method and sparsity. At 120 minutes, ResSAE’s MSE is 0.86%, 5.19%, 9.54%, and 11.34% lower, respectively. Its larger effective dictionary thus yields lower reconstruction error under the same GPU-time budget.

AuthorsOpenReview ↗

Response to W2

Response to the Novelty Concern Following W2

Thank you for highlighting these connections. We acknowledge these precedents in Table 1. Two ablations address the concern that the contribution is incremental.

(1) Coupling routing and residual encoding. Our “residual experts” couple feature-based routing with residual encoding. Table 3 (Section 4.4) gives the following absolute MSE reductions from the “original input + random routing” baseline at each sparsity budget:

ModificationL0=40L_0=40L0=80L_0=80L0=160L_0=160
Residual input only0.190.130.41
Level-1 routing only0.790.490.32
Sum of the two separate gains0.19+0.79=0.980.19+0.79=0.980.13+0.49=0.620.13+0.49=0.620.41+0.32=0.730.41+0.32=0.73
Gain from combining both1.080.961.08
Additional gain beyond the sum of separate gains0.100.340.35

At all three budgets, the combined gain exceeds the sum of the separate gains. This “1+1>21+1>2” effect supports a beneficial interaction between residual input and feature-based routing under the tested settings.

(2) Sharing the feature budget across experts. We compare ResSAE with H-SAE + residual encoding at L0=k1+k2∈{40,80,160}L_0=k_1+k_2\in\{40,80,160\}. All runs use 100M training tokens, the same data, batch size 2048, Level-1 width 4096, and per-expert Level-2 width 32. Paired runs share initialization, data order, and optimization settings:

Methodk1k_1k2k_2Level-2 feature selectionValidation MSE ↓
H-SAE + residual encoding2020Top1 per expert1.93
ResSAE2020Joint Top20 across experts1.85
H-SAE + residual encoding4040Top1 per expert1.61
ResSAE4040Joint Top40 across experts1.53
H-SAE + residual encoding8080Top1 per expert1.31
ResSAE8080Joint Top80 across experts1.24

ResSAE reduces MSE by 4.09%–4.86% (using unrounded values). Joint TopK lets each routed expert contribute multiple features or none, allocating the same total feature budget more flexibly to the current residual. These experiments and results for other allocation ratios appear in Section 4.4, “Why joint TopK?”, and Table 4.

The ablations support two design principles: features explaining part of the input can guide encoding of the remaining error; expert selection and feature-budget allocation can be decided separately. ResSAE implements these through Level-1 selection of residual experts and joint TopK allocation of their shared budget; the two ablations support these mechanisms, respectively.

We hope these results and clarifications address your concerns. If you find that they resolve the main issues underlying your original assessment, we would appreciate your reconsideration of the overall assessment and score.

Comments on the paper

2 comments
Area_Chair_6wSfOpenReview ↗

Discussion

Dear All,

Thank you for reviewing the paper. Please read the authors' response and leave a comment accordingly.

Yours,

AC

AuthorsOpenReview ↗

Summary of the Discussion and Revisions

Dear Area Chair, Senior Area Chairs, and Program Chairs,

For ease of reference, we briefly summarize the discussion. Reviewers Eckx and V3GN have both raised their scores to 3 (weak acceptance), recognizing the stronger evidence for ResSAE’s practical value. Reviewer YzfZ’s score is 4 (clear acceptance).

  • Eckx stated that the concerns were largely addressed and that the new experiments strengthened the evidence for improved reconstruction under fixed training budgets.

  • V3GN acknowledged the stronger efficiency evidence and raised the score on the condition that we extend the comparison with SwitchSAE. We have now added the requested learning curves to Figure 8, covering six hours of training and a common range of 750M tokens. ResSAE achieves lower MSE than SwitchSAE at both endpoints across all four sparsity settings.

  • YzfZ praised the combination of residual reconstruction and conditional computation as a “simple but effective” idea and confirmed, “I am satisfied with the response. My concerns have been addressed.”

The revised manuscript includes the additional comparisons and the results discussed in our responses.

Thank you for your time and for coordinating the review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.