acceptodds
Under review as a conference paper at ICLR 2027

Diminishing Returns of Data in Sparse Autoencoders: Functional Behavior does not Guarantee Feature Reproducibility

Abstract

Sparse autoencoders (SAEs) are typically trained on hundreds of millions to billions of activation tokens, yet little work has tested whether this scale is necessary. We study how much activation data SAEs need by training four SAE architectures on three language models with budgets ranging from 5M tokens to a 500M-token reference. We measure two notions of convergence: functional behavior on standard evaluations and one-to-one reproducibility of learned features. Functional data requirements are strongly metric-dependent: reconstruction often approaches the full-data reference at substantially smaller budgets, whereas feature reproducibility continues to improve with data. For Gemma-2-2B BatchTopK, the fraction of strictly matched features rises from 2% at 5M tokens to 29% at 50M. Data selection also matters: random sampling better preserves the original activation distribution and generally favors reconstruction, whereas diversity-based sampling frequently improves AutoInterp. A theoretical analysis separates statistical consistency of subsampled training from the stronger identifiability and optimization conditions required for one-to-one feature recovery. Our results show that SAE training can often be made far more data-efficient when budgets are matched to the goal, and feature-level claims are backed by explicit reproducibility checks, paving the way toward cheaper and more reliable SAE research.

Primary area:
interpretability and explainable AI
Keywords:
sparse auto-encoder, large language model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_h2N4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this work, the authors study how much activation data is required to train SAEs. They mainly focus on 2 aspects of the central question: functional performance and feature reproducibility. They train 4 SAE architectures across 3 LLMs using activation budgets ranging from 5M to 500M tokens. They evaluate the resulting SAEs using the SAEBench benchmark. The main finding is that reconstruction metrics often approach the 500M-token reference with substantially less training data, while feature interpretability metrics continue to improve as data increases. The authors also compare random sampling against farthest-point sampling and find that random generally performs better on reconstruction, while farthest-point often gives better AutoInterp scores. Finally, the paper also presents a theoretical analysis identifying assumptions under which SAEs trained on subsampled and full datasets converge to the same features.

Critical strengths
  1. Relevance to the SAE community: The SAE community is focused on efficient SAE training; how much training data is enough is always a useful question. Understanding the relationship between training data quantity and SAE quality would help allocate computational resources more efficiently.

  2. Breadth of empirical evaluation: The authors evaluate multiple SAE architectures, models, training data budgets, and all SAEBench metrics. I appreciate that the evaluation goes beyond reconstruction metrics to include interpretability metrics, providing a full picture of how SAE behavior varies with the quantity of activation data.

Critical weaknesses
  1. Limited theoretical contribution: I find the theoretical contribution limited. The main theorem states that SAEs trained on full and subsampled datasets will converge to the same feature system, given certain assumptions hold, including statistical consistency, successful optimization, and feature identifiability. However, in my view, feature identifiability is itself an open question in SAE research. By assuming it, I feel the theorem already assumes much of what it tries to establish. I am not sure what the theory adds to the experimental findings. The theorem does not really explain the trends observed at different data budgets.

  2. Limited depth of analysis: While I appreciate the breadth of experiments, I find the overall empirical contribution is somewhat limited. Most experiments vary the training budget, run SAEBench, and report metric scores. Although this is useful, I feel that the paper focuses more on reporting these results than on understanding why they occur. Why do some metrics improve with more training data while others show non-monotonic trends? Could the observed differences in feature matching simply reflect the inherent instability of SAEs across independent training runs? The authors provide some analysis of the distribution shift caused by farthest-point sampling. However, I would have liked to see a similar level of investigation into the main findings. In its current form, the paper identifies some interesting observations but does not analyze them deeply enough.

  3. Lack of a coherent connection between theory and experiments: I also struggled a bit to see how the different parts of the paper come together. The paper starts by asking how much training data SAEs actually need, introduces a theoretical result on feature convergence, and then presents experiments showing that different evaluation metrics behave differently as training data increases. Individually, these are reasonable investigations. However, I am not convinced that the theory and experiments tell a unified story. The paper feels more like a collection of related observations than a study that builds toward a clear conclusion.

Questions for authors

Q1. How reproducible are the features learned by two independently trained 500M-token SAEs? Suppose we train 2 SAEs with the same token budget but different random seeds; how many features would match? It would be great if the authors could include this comparison, as it would help us understand whether the low feature-matching scores are due to less training data or because SAEs are inherently unstable across runs.

Q2. The authors claim that functional performance approaches the 500M-token reference even with smaller training budgets. However, the results suggest this holds mainly for reconstruction. Several other SAEBench metrics vary substantially across training budgets. So, I am not entirely sure how the authors conclude that functional performance converges with less data. Could the authors clarify this?

Q3. The main theorem assumes feature identifiability, which is itself an open question in SAE research. Given this assumption, I am not sure what the theorem adds to the paper. Also, the theorem addresses asymptotic convergence, whereas the experiments examine finite training budgets. I would like to understand how the theoretical results relate to the experimental observations and what additional insight they provide.

8 comments
AuthorsOpenReview ↗

Response to Reviewer 3: Feature Reproducibility, Functional Data Requirements, and Theory–Experiment Connection

We thank the reviewer for the productive assessment of our manuscript. The appreciation of the problem and the breadth of experiments are really encouraging.

Summary of the rebuttal response.

  • First, independent-seed experiments confirm that feature reproducibility improves with activation exposure.
  • Second, we clarify that functional data requirements are metric-dependent.
  • Third, we establish a unified connection between our theoretical characterization and empirical findings on SAE data requirements.
  • Four, we present a detailed analysis to explain the behavior of the functional metrics.
AuthorsOpenReview ↗

Response to Reviewer 3: Feature Reproducibility, Functional Data Requirements, and Theory–Experiment Connection

1. Independent-Seed Feature Reproducibility

*Reviewer: What fraction of features match between two independently trained 500M-token SAEs with different random seeds? *

To further investigate whether our feature-matching results reflect inherent SAE instability, we conducted additional independent-seed experiments at all four activation budgets, following the same Hungarian-matching protocol as Table 2.

Table R1: Feature correspondence across activation budgets for Gemma-2-2B BatchTopK (16K latents). In the same-seed experiment, SAEs trained with 5M, 25M, and 50M activations are compared against the 500M reference, with all models initialized using seed 0. In the different-seed experiment, two independently initialized SAEs (seeds 0 and 1) are trained at each identical activation budget (5M, 25M, 50M, and 500M) and compared to measure seed-dependent reproducibility. Both experiments use the same Hungarian-matching protocol. Counterpart agreement requires encoder- and decoder-based matchings to identify the same reference latent, while strict correspondence additionally requires both cosine similarities to be ≥ 0.7.

Activation budgetSame-seed vs. 500M: strict overlap (%)Different-seed: counterpart agreement (%)Different-seed: strict overlap (%)
5M2.2054.791.97
25M16.9475.2810.85
50M28.8680.3019.40
500M—89.1450.71

The results reveal that feature instability persists even at 500M activations, with independently initialized SAEs achieving 50.71% strict correspondence. However, reproducibility improves substantially with activation exposure, increasing from 1.97% at 5M to 50.71% at 500M across independent seeds.

Our original cross-budget experiments hold initialization fixed, while the additional independent-seed analysis shows that feature reproducibility continues to improve with activation exposure even when initialization varies.

Together with our existing feature-importance and feature-matching analyses, these results strengthen our central finding: the activation budget required for near-reference functional performance can be substantially smaller than that required for reproducible individual features.

2. Functional performance at reduced activation budgets

Reviewer: The authors claim that functional performance approaches the 500M-token reference even with smaller training budgets. However, the results suggest this holds mainly for reconstruction. Several other SAEBench metrics vary substantially across training budgets. So, I am not entirely sure how the authors conclude that functional performance converges with less data. Could the authors clarify this?

Our claim is not that all functional metrics converge with less data, but that SAE activation-data requirements depend on the evaluation objective.

Reconstruction often approaches the 500M reference at substantially smaller budgets, whereas some functional metrics remain sensitive to increased activation exposure. Importantly, the paired statistical tests in Table 26 show that several functional metrics exhibit no statistically significant differences between reduced budgets and the 500M reference. Thus, even when additional activation exposure affects performance, the observed differences are not statistically significant in many of the evaluated comparisons.

We therefore do not claim universal functional convergence. Rather, our findings demonstrate that different evaluation objectives have different activation-data requirements, with some approaching reference performance at smaller budgets while others benefit from additional data. We will strengthen this distinction in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 3: Feature Reproducibility, Functional Data Requirements, and Theory–Experiment Connection

3. Theoretical contribution and its relationship to experiments

Reviewer: The theoretical contribution feels disconnected from the experiments. The main theorem assumes feature identifiability, an open problem in SAE research, and establishes asymptotic convergence rather than finite-budget behavior. It is unclear what additional insight the theorem provides or how it explains the empirical trends across training budgets.

We would like to clarify that our theoretical objective is to characterize sufficient conditions under which reduced-data and full-data SAEs asymptotically recover aligned features, rather than to establish feature identifiability for existing SAE architectures. Specifically, feature alignment follows under statistical consistency, optimization consistency, and feature identifiability.

This theoretical characterization provides a useful perspective for interpreting our empirical results. While random subsampling preserves the underlying activation distribution in expectation, this alone does not ensure alignment of the learned feature systems. Consistent with this distinction, our experiments show that reduced-data SAEs can approach reference reconstruction performance while exhibiting limited strict feature correspondence. Its contribution is to formalize the sufficient conditions for asymptotic feature alignment, providing a theoretical counterpart to our empirical observation that near-reference functional performance does not guarantee feature reproducibility.

3.2. Coherence between theory and experiments

The research question we tried to address in our study is how much activation data is required for different notions of SAE quality? Our functional evaluations characterize when near-reference performance can be achieved, while the theoretical analysis identifies the sufficient conditions required for asymptotic feature recovery. Together with our finite-budget matching experiments, these results distinguish functional data sufficiency from reproducible feature recovery.

Reviewer_EMqpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how much activation data sparse autoencoders actually need. It trains four SAE architectures on three LLMs with 5M to 50M activation tokens and compares them with a 500M-token reference, using SAEBench for functional metrics and Hungarian matching for feature reproducibility. The main finding is that reconstruction and several SAEBench metrics are already close to the reference at small budgets, while the fraction of shared features keeps rising with more data. The paper also reports that farthest-point sampling (FPS) gives higher AutoInterp but worse reconstruction than random sampling, and gives an asymptotic theorem on when subsampled SAEs recover the same features.

Critical strengths
  1. The question is practical and the answer is useful: across three models and four architectures, 25–50M tokens bring reconstruction and several SAEBench metrics close to the 500M reference. That could save a lot of compute in SAE research.

  2. The distinction between counterpart agreement and actual feature alignment is a useful caution. At 5M tokens, encoder and decoder matching agree on 66.6% of latents, but only 2.2% are strongly aligned at τ = 0.7 (Table 2).

Critical weaknesses
  1. The reproducibility result, which is the paper's headline claim, lacks a seed baseline. Without knowing how many features two 500M runs with different seeds share, it is unclear whether 28.9% at 50M is still far from the ceiling or already close to it. It is also unclear whether smaller budgets get fewer optimizer steps, which would mix data quantity with training length.

  2. The FPS vs. random comparison is confounded by sparsity. With the same k, FPS SAEs end up at much lower L0 (e.g., 51 vs. 80 for BatchTopK at 5M, Table 6), and lower L0 by itself tends to raise AutoInterp and lower reconstruction. So the reported trade-off (FPS better for AutoInterp, random better for reconstruction) may simply reflect FPS producing sparser SAEs rather than a property of the selected data.

Questions for authors
  1. Could you report the shared-feature fraction between two 500M runs that differ only in seed? This would show how much of the gap in Table 2 is due to data.

  2. How many optimizer steps does each budget use, and is any data repeated? Sec. 4 seems to say both that the step count is fixed and that each activation is used once.

  3. Could you compare FPS and random sampling at matched L0, e.g., by plotting each metric against L0?

Minor: Theorem 1 is asymptotic and says little about finite budgets like 5M vs. 500M. There is also no budget between 50M and 500M, so it is hard to tell where reproducibility saturates.

8 comments
AuthorsOpenReview ↗

Response to Reviewer 2: Seed Reproducibility, Optimization Steps, Sparsity Confounding, and Scope

We thank the reviewer for the constructive assessment of our manuscript. The appreciation of the problem and experiments is really encouraging.

Summary of the Rebuttal Response

We address the reviewer's concerns through additional experiments and clarifications:

  1. Independent-seed baseline: Confirms that feature reproducibility continues to improve with activation exposure.
  2. Optimization horizon: Clarifies that all SAEs use the same optimization horizon without repeating activations.
  3. Matched sparsity: Matched-L0L_0 comparisons show that the FPS–random trade-off persists after sparsity adjustment.
  4. Scope: Clarifies the scope of our theoretical claims and evaluated activation budgets.
AuthorsOpenReview ↗

Response to Reviewer 2: Seed Reproducibility, Optimization Steps, Sparsity Confounding, and Scope

1. Full-data seed reproducibility

Reviewer: Could you report the shared-feature fraction between two 500M runs that differ only in seed? This would show how much of the gap in Table 2 is due to data.

We thank the reviewer for suggesting an independent-seed reproducibility baseline. We have extended our analysis to independently initialized SAEs at all four activation budgets, using the same Hungarian-matching protocol as Table 2.

Table R1: Feature correspondence across activation budgets for Gemma-2-2B BatchTopK (16K latents). In the same-seed experiment, SAEs trained with 5M, 25M, and 50M activations are compared against the 500M reference, with all models initialized using seed 0. In the different-seed experiment, two independently initialized SAEs (seeds 0 and 1) are trained at each identical activation budget (5M, 25M, 50M, and 500M) and compared to measure seed-dependent reproducibility. Both experiments use the same Hungarian-matching protocol. Counterpart agreement requires encoder- and decoder-based matchings to identify the same reference latent, while strict correspondence additionally requires both cosine similarities to be ≥ 0.7.

BudgetSame-seed vs. 500M: strict (%)Different-seed: agreement (%)Different-seed: strict (%)
5M2.2054.791.97
25M16.9475.2810.85
50M28.8680.3019.40
500M—89.1450.71

The new results directly address the reviewer's concern about the full-data reproducibility baseline. Independently initialized 500M SAEs achieve 50.71% strict correspondence, substantially higher than the 28.86% observed between the 50M and 500M SAEs with the same seed. Thus, the 50M result remains considerably below the observed full-data reproducibility baseline.

Furthermore, independent-seed strict correspondence increases from 1.97% at 5M to 10.85% at 25M, 19.40% at 50M, and 50.71% at 500M, consistent with the increasing feature overlap reported by Paulo and Belrose et al.

Our original cross-budget experiments hold initialization fixed, while the additional independent-seed analysis shows that feature reproducibility continues to improve with activation exposure even when initialization varies.

These findings confirm that inherent SAE instability remains substantial at 500M, but does not fully account for the limited feature correspondence observed at smaller activation budgets. Additional activation exposure continues to improve feature reproducibility beyond the budgets sufficient for near-reference functional performance.

2. Optimization steps and activation reuse

Reviewer: How many optimizer steps does each budget use, and is any data repeated?

Our experimental design is motivated by reducing the amount of activation data required to train SAEs while retaining their functional performance. We train separate SAEs using 5M, 25M, 50M, and 500M activations, maintaining the same optimization horizon of 244,140 updates across all budgets by selecting a subset of activations at each step.

Each selected activation is consumed exactly once, without repetition. The effective batch size therefore scales with the activation budget. This allows us to study SAE behavior with substantially fewer fresh activations without reducing the optimization-update count.

Unlike early-stopping experiments, our goal is to determine how much activation exposure is sufficient for different SAE evaluation objectives.

AuthorsOpenReview ↗

Response to Reviewer 2: Seed Reproducibility, Optimization Steps, Sparsity Confounding, and Scope

3. FPS versus random sampling: sparsity confounding

Reviewer: The FPS vs. random comparison is confounded by sparsity. With the same k, FPS SAEs end up at much lower L0 (e.g., 51 vs. 80 for BatchTopK at 5M, Table 6), and lower L0 by itself tends to raise AutoInterp and lower reconstruction. So the reported trade-off (FPS better for AutoInterp, random better for reconstruction) may simply reflect FPS producing sparser SAEs rather than a property of the selected data. Could you compare FPS and random sampling at matched L0, e.g., by plotting each metric against L0?

We thank the reviewer for highlighting the difference in achieved sparsity. We agree that comparing SAEs with identical target kk but different observed L0L_0 can confound the interpretation of AutoInterp and reconstruction performance.

To address this concern, we performed an additional matched-L0L_0 analysis. For each model, architecture, dictionary width, and activation budget, we identify the overlapping achieved-L0L_0 ranges of FPS and random sampling. We then use linear interpolation to compare their performance at common observed sparsity levels.

Table R2: FPS versus random sampling after achieved-L0L_0 adjustment.

ArchitectureComparisonsFPS AutoInterp winsMean Δ AutoInterpMean Δ Loss Recovered
BatchTopK139+0.0038−0.0076
JumpReLU156−0.0043−0.0079
Matryoshka BatchTopK1412+0.0211−0.0074
Standard ReLU1510+0.0021−0.0110
Overall5737 (64.9%)+0.0055−0.0085

Δ = FPS − Random. Comparisons are interpolated within overlapping observed-L0L_0 ranges.

After adjusting for L0L_0, FPS retains higher AutoInterp in 37/57 comparisons (64.9%), while random sampling retains higher reconstruction performance in 56/57 comparisons (98.2%). The AutoInterp advantage becomes less consistent and varies across architectures, whereas random sampling's reconstruction advantage remains robust.

These findings suggest that the observed trade-off is not entirely explained by differences in effective sparsity, although sparsity may contribute to the original comparison. We acknowledge that this analysis uses interpolation rather than retraining at exactly matched L0L_0.

Importantly, our objective is not to establish FPS as an optimal data-selection strategy, but to characterize how activation selection influences SAE behavior under restricted data budgets. The results demonstrate metric-dependent differences between FPS and random sampling that persist in the matched-L0L_0 analysis.

4. Asymptotic theory and data-budget limitations

Reviewer: Minor: Theorem 1 is asymptotic and says little about finite budgets like 5M vs. 500M. There is also no budget between 50M and 500M, so it is hard to tell where reproducibility saturates.

Our study focuses on whether substantially reduced activation budgets can achieve near-reference functional performance and how feature correspondence varies across those budgets. Theorem 1 provides a conditional asymptotic characterization of feature alignment, rather than predicting finite-budget behavior. Our empirical results demonstrate the discrepancy between functional performance and feature reproducibility at the evaluated data scales, without claiming to identify the exact saturation threshold. Determining where feature reproducibility saturates remains an important direction for future work.

Reviewer_HHqaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies SAE convergence across different data budgets, measuring different metrics at checkpoints saved during training. This work also studies the FPS data selection algorithm. FPS provides better AutoInterp scores, but worsens reconstruction quality and CE.

Critical strengths
  • Data selection for SAE training is interesting and, as far as I know, is an underexplored direction in the field.

  • The experimental design is good and covers many benchmarks.

Critical weaknesses

Most of the paper's results are already present in previous works:

  • The evaluation of SAEBench metrics across token budgets substantially overlaps with the original SAEBench study [1] (compare Figure 8 in SAEBench with Figures 1 and 2 in your work).

  • Feature matching versus token budget was also explored in [2] (see Figure 4, right).

The applicability of Theorem 1 to real SAEs is unclear. My main concern is assumption (iv), which requires all population minimizers to recover the same encoder and decoder feature directions up to a common permutation. The authors do not establish this condition for the SAE objectives studied. For example, consider counting manifolds and their associated features [3]. Several features (>3), each responding to a range of character counts after the '\n' token, jointly reconstruct a helix-like structure visible in the first three principal components. In principle, different feature tilings could reconstruct the same underlying structure. See also [4], which reports unstable individual features within reproducible low-rank subspaces. This evidence calls the identifiability assumption into question, although it does not establish the existence of multiple exact global population optima. Without justification of this assumption, Theorem 1 seems of little use for understanding feature stability in SAEs trained on real data.

FPS feels underexplored. What is the source of the higher AutoInterp scores? The paper already shows that FPS favors high-norm input activations. My concern is that the resulting SAE could learn more features with low token-distribution entropy that mostly respond to punctuation or high-frequency words, making them easier to explain and increasing the aggregate AutoInterp score. This is a hypothesis that needs testing. Reference [5] shows that simple, token-specific features can receive high AutoInterp scores, motivating this concern.

I also think ablations are needed to separate geometric diversity from activation-norm bias. I would like to see FPS using normalized vectors to compute selection distances, while training on the original activations, and random sampling matched to the norm distribution of the FPS-selected activations.

[1] Karvonen et al. SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability. ICML 2025.

[2] Paulo and Belrose. Sparse Autoencoders Trained on the Same Data Learn Different Features. ICLR 2026.

[3] Gurnee et al. When Models Manipulate Manifolds: The Geometry of a Counting Task. Transformer Circuits Thread, 2025.

[4] Gerasimov et al. Unstable Features, Reproducible Subspaces: Understanding Seed Dependence in Sparse Autoencoders.

[5] Heap et al. Automated Interpretability Metrics Do Not Distinguish Trained and Random Transformers. ICLR 2026.

Questions for authors

See Weaknesses

7 comments
AuthorsOpenReview ↗

Response to Reviewer 1: Novelty, Theoretical Assumptions, and FPS Analysis

We thank the reviewer for a careful assessment and recognition of our problem statement. We appreciate the reviewer's attention to our data selection for SAEs, particularly the experimental design and benchmarks. The concerns are addressed below.

Summary of the rebuttal response. We believe the review overlooks important distinctions in our contributions.

  • First, our fixed-update, reduced-activation experiments address a different question from SAEBench's checkpoint analysis and Paulo and Belrose's seed-reproducibility study.
  • Second, our theorem characterizes sufficient conditions for asymptotic feature alignment, complementing our empirical finding that near-reference reconstruction does not necessarily imply strong feature correspondence at finite data budgets.
  • Third, our additional feature-level analysis provides evidence that punctuation-specific features do not simply drive FPS's AutoInterp gains.
AuthorsOpenReview ↗

Response to Reviewer 1: Novelty, Theoretical Assumptions, and FPS Analysis

1. Comparison with prior work

Reviewer 1: Most of the paper's results are already present in previous works: evaluation of SAEBench metrics across token budgets [1] (compare Figure 8 in SAEBench with Figures 1 and 2 in your work) and feature matching versus token budget was also explored in [2].

1.1. SAEBench: training checkpoints versus reduced activation exposure

While SAEBench examines how SAE evaluation metrics evolve during training, in contrast, we investigate a separate question: how much fresh activation data is required to achieve near-reference functional performance? Specifically, we ask whether SAEs trained with substantially less activation exposure can approach the performance of a 500M-token reference when the number of optimization updates is held fixed.

Unlike Figure 8 of SAEBench, which evaluates intermediate checkpoints from a single training run, we train independent SAEs at activation budgets of 5M, 25M, 50M, and 500M using an online activation stream. All SAEs undergo the same 244,140 optimizer updates, while the number of fresh activations per update is scaled to (q \in {1%, 5%, 10%, 100%}) of the reference batch size, respectively. Each selected activation is used exactly once.

Thus, two studies examine two distinct settings. Karvonen et al. (2025) examine intermediate checkpoints where optimizer updates and cumulative activation exposure increase together. Our 5M-token SAE is therefore a separately trained model, not an early checkpoint of the 500M-token SAE. Our experiments isolate the effect of reducing fresh activation exposure while holding the number of optimization updates fixed, rather than measuring training progression.

1.2. Paulo and Belrose [2]: functional versus feature-level data requirements

We would like to clarify a misunderstanding as we address a completely different question as compared to [2]. While [2]  investigate seed-dependent feature reproducibility, showing that SAEs trained with different random initializations can learn different features, and that feature overlap improves with increasing training tokens, our study addresses a complementary question: How does reducing activation exposure affect feature recovery when initialization is held fixed? We train separate SAEs with the same initialization seed but different activation budgets (5M, 25M, 50M, and 500M), while maintaining a fixed optimization horizon. 

More specifically, we investigate whether activation budgets sufficient for near-reference functional performance also recover the same individual features. We systematically characterize this discrepancy across SAE architectures, functional evaluation metrics, and activation-selection strategies (random versus FPS). Thus, our contribution is not simply observing that feature overlap increases with training data but establishing that near-reference functional performance does not necessarily imply comparable feature-level reproducibility under reduced activation exposure.

2. Applicability of Theorem 1

Reviewer: The applicability of Theorem 1 to real SAEs is unclear. My main concern is assumption (iv), which requires all population minimizers to recover the same encoder and decoder feature directions up to a common permutation. The authors do not establish this condition for the SAE objectives studied. 

We would like to clarify that our theoretical objective is to characterize sufficient conditions under which reduced and full-data SAEs asymptotically recover aligned features, rather than to establish feature identifiability for existing SAE architectures. Specifically, feature alignment follows under statistical consistency, optimization consistency, and feature identifiability.

This theoretical characterization provides a useful perspective for interpreting our empirical results. While random subsampling preserves the underlying activation distribution in expectation, this alone does not ensure alignment of the learned feature systems. Consistent with this distinction, our experiments show that reduced-data SAEs can approach reference reconstruction performance while exhibiting limited strict feature correspondence. Thus, near-reference functional performance does not necessarily imply one-to-one feature recovery at finite data budgets.

We do not claim that practical SAEs satisfy these assumptions or that the theorem predicts the observed matching percentages. Rather, the theorem provides a conditional asymptotic characterization that complements our empirical investigation of finite-data feature reproducibility.

AuthorsOpenReview ↗

Response to Reviewer 1: Novelty, Theoretical Assumptions, and FPS Analysis

3. FPS and AutoInterp gains

Reviewer: FPS feels underexplored. What is the source of the higher AutoInterp scores? The paper already shows that FPS favors high-norm input activations. My concern is that the resulting SAE could learn more features with low token-distribution entropy that mostly respond to punctuation or high-frequency words, making them easier to explain and increasing the aggregate AutoInterp score. 

We thank the reviewer for this observation. FPS improves AutoInterp in 79% of matched comparisons, while random sampling generally performs better on reconstruction and intervention metrics (Figure 3).

To investigate whether these gains arise from trivially explainable features, we compared FPS and random sampling at matched activation budgets within the same model and SAE configuration. Specifically, we analyzed per-feature AutoInterp examples for Gemma-2-2B BatchTopK (64K latents) at 50M tokens.

  • The results do not support a simple punctuation-dominance explanation. For BatchTopK at 50M, FPS achieves higher AutoInterp (0.8114 vs. 0.7808), despite fewer punctuation-only highlighted activation spans (13.1% vs. 15.6%).

  • We further excluded features for which at least 50% of highlighted activation spans consist only of punctuation or symbols. This removes 47 of 990 FPS features and 70 of 994 random-sampling features. The AutoInterp advantage remains: 0.8150 for FPS versus 0.7856 for random sampling, a difference of +0.0294 compared with +0.0306 before exclusion.

  • Across all three ReLU budgets, FPS also produces fewer punctuation-only highlighted spans than random sampling.

These findings indicate that a greater prevalence of punctuation-specific features does not solely explain FPS's AutoInterp improvements. We will incorporate this analysis to characterize the underlying mechanism.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.