- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this work, the authors study how much activation data is required to train SAEs. They mainly focus on 2 aspects of the central question: functional performance and feature reproducibility. They train 4 SAE architectures across 3 LLMs using activation budgets ranging from 5M to 500M tokens. They evaluate the resulting SAEs using the SAEBench benchmark. The main finding is that reconstruction metrics often approach the 500M-token reference with substantially less training data, while feature interpretability metrics continue to improve as data increases. The authors also compare random sampling against farthest-point sampling and find that random generally performs better on reconstruction, while farthest-point often gives better AutoInterp scores. Finally, the paper also presents a theoretical analysis identifying assumptions under which SAEs trained on subsampled and full datasets converge to the same features.
Critical strengths
-
Relevance to the SAE community: The SAE community is focused on efficient SAE training; how much training data is enough is always a useful question. Understanding the relationship between training data quantity and SAE quality would help allocate computational resources more efficiently.
-
Breadth of empirical evaluation: The authors evaluate multiple SAE architectures, models, training data budgets, and all SAEBench metrics. I appreciate that the evaluation goes beyond reconstruction metrics to include interpretability metrics, providing a full picture of how SAE behavior varies with the quantity of activation data.
Critical weaknesses
-
Limited theoretical contribution: I find the theoretical contribution limited. The main theorem states that SAEs trained on full and subsampled datasets will converge to the same feature system, given certain assumptions hold, including statistical consistency, successful optimization, and feature identifiability. However, in my view, feature identifiability is itself an open question in SAE research. By assuming it, I feel the theorem already assumes much of what it tries to establish. I am not sure what the theory adds to the experimental findings. The theorem does not really explain the trends observed at different data budgets.
-
Limited depth of analysis: While I appreciate the breadth of experiments, I find the overall empirical contribution is somewhat limited. Most experiments vary the training budget, run SAEBench, and report metric scores. Although this is useful, I feel that the paper focuses more on reporting these results than on understanding why they occur. Why do some metrics improve with more training data while others show non-monotonic trends? Could the observed differences in feature matching simply reflect the inherent instability of SAEs across independent training runs? The authors provide some analysis of the distribution shift caused by farthest-point sampling. However, I would have liked to see a similar level of investigation into the main findings. In its current form, the paper identifies some interesting observations but does not analyze them deeply enough.
-
Lack of a coherent connection between theory and experiments: I also struggled a bit to see how the different parts of the paper come together. The paper starts by asking how much training data SAEs actually need, introduces a theoretical result on feature convergence, and then presents experiments showing that different evaluation metrics behave differently as training data increases. Individually, these are reasonable investigations. However, I am not convinced that the theory and experiments tell a unified story. The paper feels more like a collection of related observations than a study that builds toward a clear conclusion.
Questions for authors
Q1. How reproducible are the features learned by two independently trained 500M-token SAEs? Suppose we train 2 SAEs with the same token budget but different random seeds; how many features would match? It would be great if the authors could include this comparison, as it would help us understand whether the low feature-matching scores are due to less training data or because SAEs are inherently unstable across runs.
Q2. The authors claim that functional performance approaches the 500M-token reference even with smaller training budgets. However, the results suggest this holds mainly for reconstruction. Several other SAEBench metrics vary substantially across training budgets. So, I am not entirely sure how the authors conclude that functional performance converges with less data. Could the authors clarify this?
Q3. The main theorem assumes feature identifiability, which is itself an open question in SAE research. Given this assumption, I am not sure what the theorem adds to the paper. Also, the theorem addresses asymptotic convergence, whereas the experiments examine finite training budgets. I would like to understand how the theoretical results relate to the experimental observations and what additional insight they provide.
8 comments
Response to Reviewer 3: Feature Reproducibility, Functional Data Requirements, and Theory–Experiment Connection
1. Independent-Seed Feature Reproducibility
*Reviewer: What fraction of features match between two independently trained 500M-token SAEs with different random seeds? *
To further investigate whether our feature-matching results reflect inherent SAE instability, we conducted additional independent-seed experiments at all four activation budgets, following the same Hungarian-matching protocol as Table 2.
Table R1: Feature correspondence across activation budgets for Gemma-2-2B BatchTopK (16K latents). In the same-seed experiment, SAEs trained with 5M, 25M, and 50M activations are compared against the 500M reference, with all models initialized using seed 0. In the different-seed experiment, two independently initialized SAEs (seeds 0 and 1) are trained at each identical activation budget (5M, 25M, 50M, and 500M) and compared to measure seed-dependent reproducibility. Both experiments use the same Hungarian-matching protocol. Counterpart agreement requires encoder- and decoder-based matchings to identify the same reference latent, while strict correspondence additionally requires both cosine similarities to be ≥ 0.7.
| Activation budget | Same-seed vs. 500M: strict overlap (%) | Different-seed: counterpart agreement (%) | Different-seed: strict overlap (%) |
|---|---|---|---|
| 5M | 2.20 | 54.79 | 1.97 |
| 25M | 16.94 | 75.28 | 10.85 |
| 50M | 28.86 | 80.30 | 19.40 |
| 500M | — | 89.14 | 50.71 |
The results reveal that feature instability persists even at 500M activations, with independently initialized SAEs achieving 50.71% strict correspondence. However, reproducibility improves substantially with activation exposure, increasing from 1.97% at 5M to 50.71% at 500M across independent seeds.
Our original cross-budget experiments hold initialization fixed, while the additional independent-seed analysis shows that feature reproducibility continues to improve with activation exposure even when initialization varies.
Together with our existing feature-importance and feature-matching analyses, these results strengthen our central finding: the activation budget required for near-reference functional performance can be substantially smaller than that required for reproducible individual features.
2. Functional performance at reduced activation budgets
Reviewer: The authors claim that functional performance approaches the 500M-token reference even with smaller training budgets. However, the results suggest this holds mainly for reconstruction. Several other SAEBench metrics vary substantially across training budgets. So, I am not entirely sure how the authors conclude that functional performance converges with less data. Could the authors clarify this?
Our claim is not that all functional metrics converge with less data, but that SAE activation-data requirements depend on the evaluation objective.
Reconstruction often approaches the 500M reference at substantially smaller budgets, whereas some functional metrics remain sensitive to increased activation exposure. Importantly, the paired statistical tests in Table 26 show that several functional metrics exhibit no statistically significant differences between reduced budgets and the 500M reference. Thus, even when additional activation exposure affects performance, the observed differences are not statistically significant in many of the evaluated comparisons.
We therefore do not claim universal functional convergence. Rather, our findings demonstrate that different evaluation objectives have different activation-data requirements, with some approaching reference performance at smaller budgets while others benefit from additional data. We will strengthen this distinction in the revised manuscript.
Response to Reviewer 3: Feature Reproducibility, Functional Data Requirements, and Theory–Experiment Connection
3. Theoretical contribution and its relationship to experiments
Reviewer: The theoretical contribution feels disconnected from the experiments. The main theorem assumes feature identifiability, an open problem in SAE research, and establishes asymptotic convergence rather than finite-budget behavior. It is unclear what additional insight the theorem provides or how it explains the empirical trends across training budgets.
We would like to clarify that our theoretical objective is to characterize sufficient conditions under which reduced-data and full-data SAEs asymptotically recover aligned features, rather than to establish feature identifiability for existing SAE architectures. Specifically, feature alignment follows under statistical consistency, optimization consistency, and feature identifiability.
This theoretical characterization provides a useful perspective for interpreting our empirical results. While random subsampling preserves the underlying activation distribution in expectation, this alone does not ensure alignment of the learned feature systems. Consistent with this distinction, our experiments show that reduced-data SAEs can approach reference reconstruction performance while exhibiting limited strict feature correspondence. Its contribution is to formalize the sufficient conditions for asymptotic feature alignment, providing a theoretical counterpart to our empirical observation that near-reference functional performance does not guarantee feature reproducibility.
3.2. Coherence between theory and experiments
The research question we tried to address in our study is how much activation data is required for different notions of SAE quality? Our functional evaluations characterize when near-reference performance can be achieved, while the theoretical analysis identifies the sufficient conditions required for asymptotic feature recovery. Together with our finite-budget matching experiments, these results distinguish functional data sufficiency from reproducible feature recovery.
Response to Reviewer 3: Feature Reproducibility, Functional Data Requirements, and Theory–Experiment Connection
We thank the reviewer for the productive assessment of our manuscript. The appreciation of the problem and the breadth of experiments are really encouraging.
Summary of the rebuttal response.