acceptodds
Under review as a conference paper at ICLR 2027

Active Budget Can Kill Sensitivity: Diagnosing and Repairing TopK Sparse Autoencoder Reliability

Abstract

Sparse autoencoders (SAEs) are increasingly scaled to wider dictionaries to recover fine-grained structure from large language model activations. However, a feature is useful for interpretation only if it remains a stable unit of analysis when the same meaning is expressed in different surface forms. We study this reliability question for TopK SAEs through feature sensitivity: the probability that a source-active feature remains active under meaning-preserving paraphrases. Experiments demonstrate that practical scaling selectively reduces the sensitivity of rare features, while common features remain comparatively stable. A controlled width×k factorial locates the cause in the active budget k: rare-feature sensitivity declines monotonically as k grows while reconstruction error improves over the same range, so the loss is a property of the selection boundary rather than of width alone. We trace this failure to the geometry of TopK selection: rare active features often lie close to the cutoff between selected and rejected features, so small paraphrase-induced shifts can reorder nearby competitors and remove them from the active set. The distance to that cutoff, the active margin, predicts which features are lost without any threshold, consistently across depths and model families. Guided by the margin diagnosis, we introduce pairwise rank stabilization, which targets the source-paraphrase ordering failure at the cutoff and raises rare-feature sensitivity by 8.83 percentage points on sources that played no role in selecting the objective or its hyperparameters, while reconstruction and alive-feature coverage stay close to the baseline. Overall, our results suggest that wide TopK SAEs should be evaluated not only by reconstruction, sparsity, and feature count, but also by feature reliability under semantic variation and the boundary geometry that determines whether features remain available as stable units of analysis.

Primary area:
interpretability and explainable AI
Keywords:
sparse autoencoders, mechanistic interpretability, TopK SAEs, feature sensitivity, representational stability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LP69OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

So this paper studies feature sensitivity on the top K SAEs. You'd have a discussion of the larger K cross on the top K cutoff will cause the active margin prediction paraphrase dropout.

Critical strengths

This paper have a clear and well posted question for the topic of discussion. It also has a good casual isolation of KBS width. It also provided an honest control and negative result that have enough validation and holdout split by source texts.

Critical weaknesses

This paper only included a very tiny and narrative evaluation data that should be expanded more. The repair is trained and evaluated on essentially the same distribution or metric. Therefore, the sensitivity is questionable, proxy for reliability and rarely features may be legitimately brittle.

Questions for authors

How does the sensitivity behave on token level activation and on other layers token position? Is the repair robust to a different paraphrase back translation type or other meaning preserving perturbations?

1 comment

Thank you for recognizing the clear question, the isolation of k from width, and the source-disjoint controls.

W1. Is the evaluation too small?

We agree that the original evaluation is limited in size and in rewrite style.

We have expanded it: four new rewrite types add 5,038 pairs, on top of the original held-out sources and cross-model checks. The original 856 test sources remain excluded from training and selection, and every method is evaluated on the same pairs. Confirmed details are in Apps. A.10–A.11.

W2. Are training and evaluation essentially the same distribution?

This is a fair question: if we only reuse the same rewrite generator, the test may look too close to training.

Training and testing do not share source texts, and the method still works when the rewrite procedure itself changes, including back-translation. No rewrite of an auxiliary-training or selection source enters the 856-source test set. To go beyond new sentences from the same generator, we change the rewriting procedure without retraining:

Q2. Does it work with another paraphraser or back-translation?

Yes. The submitted loss improves sensitivity on all four new rewrite types below, including back-translation.

Rewrite typePairsSensitivity gain: our loss+ other-source termAUC gain: + term
Another LLM (Qwen2.5-7B)1673+8.8±1.0+7.7±0.4+10.5±0.8
Back-translation via German840+4.9±0.9+5.3±0.9+9.6±0.6
Back-translation via Chinese837+6.6±1.1+7.0±1.4+8.8±0.6
Synonym substitution (~15%)1688+3.9±0.8+7.6±0.8+8.6±0.6
Reference: original GPT-4o, re-encoded1710+7.9±0.9+9.0±1.4+10.4±0.7

Changes in percentage points, paired means±SEM over five retrained seeds; p<0.01 for the submitted loss on all four new types. All five rows use the same encoding pipeline. The last row is a reference that re-encodes the original GPT-4o rewrites, not a fifth new method. AUC measures discrimination against other-source rewrites. Confirmed details are in App. A.10.

Is the evaluation just the training metric, and is sensitivity a reliability proxy?

Sensitivity is not just the training objective restated: we also check false alarms and rewrite types the method never trained on. The property at issue is whether an interpreted feature remains available after a meaning-preserving wording change. This complements reconstruction and coverage, rather than replacing them (Tian et al., 2025; Secs. 1, 6).

Could rare features be legitimately brittle?

Yes, some could be, and Sec. 3.1 already says so: a rare feature may encode a lexical or syntactic detail that a paraphrase can change. The original paraphrases were manually checked for meaning preservation.

Brittleness is possible, but it does not explain away the result: useful stability can be gained without firing indiscriminately. An optional discussion-period term improves per-feature sensitivity while reducing firing on other-source rewrites from 25.6% to 8.8% (Table 6).

Q1. What about token positions and other layers?

Other layers are already covered; token-level position is a different readout from the one this paper evaluates. Table 2 spans seven depths, and Table A.7 repeats the sensitivity check at layers 2, 8 and 10, with a +5.2pp gain at layer 10. Our readout is sentence-level mean pooling (Sec. 5.2; App. A).

We hope our responses address your concerns. If you have any further questions, please do not hesitate to let us know.

Reviewer_unmpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This papers studies the reliability of TopK SAEs under meaning-preserving paraphrases. It finds that rare-feature sensitivity drops disproportionately relative to active budget (compared to dictionary width), proposes pairwise rank stabilization and reports an 8.8pp gain on held-out pairs for rare-feature sensitivity with no cost reconstruction and alive coverage.

Critical strengths

[S1] Clear prose and a well-motivated diagnosis-to-repair arc, though tables are dense and hard to parse.

[S2] Rank stabilization closes ~29% (GPT-2) and ~24% (Qwen) of the gap to perfect rare-feature sensitivity at no reconstruction or coverage cost (Table 5)

Critical weaknesses

[W1] Specificity regression. Mismatch activation rises nearly as much as true retention (+5.5 vs. +6.5pt), leaving the retention gap flat within SEM (Table A.9). The repair may broaden firing rather than stabilize features.

[W2] Experimental validity. The W/k choices are unjustified (Table 5). Gemma seemingly doesn't transfer, but Gemma fixes k=64k=64, so it cannot test the kk mechanism. Table 5 shows +12.3 gain for GPT-2 but is misleading: that's the validation set, which drove selection of that result (Table 3 shows 8.8pp held-out); what are the held-out results for Gemma and Qwen? Table 1 reports only marginals despite claiming full factorial design, with no per-cell values or W×k interaction. Baselines appear only on validation and may not have had any selection benefit (unlike paper's method).

[W3] Narrow SAE design space. There are no ablations over activation function (ReLU+L1, JumpReLU, BatchTopK), Matryoshka, or expansion factor. Is the kk effect TopK-specific

Questions for authors

[Q1] Specificity. True retention and mismatch activation both rise, which is consistent with moving along the same frontier rather than improving it. Please report true retention at matched FPR.

3 comments

[1/2] Thank you for recognizing the diagnosis-to-repair arc and the absence of reconstruction or coverage cost. We answer specificity here and experimental validity and design choices in [2/2].

W1. Does the repair merely broaden firing?

This is a fair concern: if true retention and mismatch firing rise together, the method might only be moving along the same trade-off, not truly stabilizing the intended features.

The submitted method is not simply making features easier to fire. At the same false-alarm rate, it keeps more true rewrites, while firing on other-source rewrites does not rise in a meaningful way.

During the discussion period, we added a retrained comparison on the original 856 held-out sources, using rewrites of different sources as negatives. Under that protocol, the submitted loss keeps 11.9 percentage points more true rewrites, and firing on other-source rewrites changes by only +0.4pp (p=0.81, not significant). This 11.9pp figure uses the same retention accounting as Table A.9; the headline 8.8pp is per-feature sensitivity. The original Table A.9 comparison is kept in Appendix A.6, and the new held-out comparison is reported in Table 6.

Q1. True retention at the same false-alarm rate

Yes. With false alarms fixed at 20%, the submitted loss keeps 14.3 percentage points more true rewrites, so the gain is not just a looser firing threshold. Overall discrimination also improves (ROC-AUC +3.6pp, p=0.007). The confirmed numbers are in the table below, and in Table 6 and App. A.7 of the revised PDF.

Held-out rare featuresBaselineOur loss+ other-source penalty
Overall discrimination (ROC-AUC)0.798+3.6±0.7+12.5±0.7
True rewrites kept at 20% false alarms0.541+14.3±3.1+41.3±3.9
Firing on other-source rewrites0.256+0.4±1.5−16.8±1.8
Per-feature sensitivity0.450+9.6±1.0+10.4±1.2

Baseline entries are rates; changes are percentage points, mean±SEM over five paired seeds. Negatives are rewrites of different sources; rates condition on each model's source-active rare features.

What the added term changes.

The submitted loss is designed to stabilize ranking on true rewrites. To address specificity even more directly, we also tried, during discussion, an optional term that penalizes firing on other-source rewrites. This optional term is an extra check, not a replacement of the submitted method. It lowers other-source firing from 25.6% to 8.8%. The submitted loss remains the main result. This is not just a larger positive-loss weight: that control gains only 3.8pp AUC. Details are in Sec. 5.4 and App. A.8.

We hope our responses address your concerns. If you have any further questions, please do not hesitate to let us know.

[2/2] Experimental validity and design space

W2. Why choose W=12288 and k=128?

We understand the worry that this setting, and the +12.3pp in Table 5, could look selected after the fact.

We test the repair where the diagnosed failure is strongest, then keep the method fixed; the number to use is the independent +8.8pp, not the validation +12.3pp. In this setting, 54% of cases lie near the cutoff, versus 10% in the smaller setting. After selection, the hyperparameters stay fixed while we vary k. Confirmed details are in Sec. 5.3, Tables 3 and 5, and App. A.12.

Does Gemma test the k mechanism, and does the repair transfer?

You are right: Gemma with k fixed at 64 cannot test the k mechanism.

Gemma is a transfer check without retuning, not a test of k; the k test is in the GPT-2 experiments. The optional term improves Gemma's discrimination from 0.896 to 0.954, while sensitivity changes by −1.8pp (not significant). On Qwen, even the baseline barely discriminates (AUC 0.51); we report this as a readout limitation. Confirmed details are in Table 5 and App. A.11.

Is +12.3pp a selection-set result?

Yes. The headline result is the independent +8.8pp; +12.3pp is a validation number and is now labeled as such. The 856 test sources were excluded from auxiliary training and selection. Details are in Sec. 5.3 and Table 5.

Where are the per-cell results and interaction?

The full grid supports the same conclusion as the averages: raising k hurts sensitivity in every width, while changing width does little. Raising k lowers sensitivity inside every width (−12.9/−15.3/−16.2pp). Changing width at any fixed k produces a difference of about 3.6pp, with no consistent sign. The confirmed table is in App. A.14.

Did baselines receive the same selection benefit?

Yes. In the new comparison, every method gets the same validation-only selection; the test set is used only after that. The best-tuned hinge baseline does not improve significantly (+0.4pp sensitivity, +3.0pp AUC). Our loss gains 9.6pp sensitivity. The optional term is reported alongside, not in place of, the submitted loss. Details are in App. A.9.

W3. Is the k effect TopK-specific? What about other designs and expansion factors?

We agree that a broader design sweep would be valuable, and we have not ablated ReLU+L1, BatchTopK, or Matryoshka.

The k effect is about TopK's active budget, which is why we study TopK; the same boundary pattern already appears in a JumpReLU SAE. Expansion factor was already varied in the submission: 4×/8×/16× at fixed k (Table 1), and 1×–16× on the practical scaling path (Fig. 2a, Table A.4). In the submitted JumpReLU test, dropout is 61.8% nearest the threshold versus 1.5% farthest away (Table A.8). A repair across all those designs is not the focus here.

Table 3 distinguishes test from validation, Table 5 labels the GPT-2 selection results, and Sec. 5.5 clarifies Gemma's fixed k.

We hope our responses address your concerns. If you have any further questions, please do not hesitate to let us know.

Thank you for the detailed response and revisions. A.6, A.10 and A.17 help address some of my concerns. However, my main concerns remain, so I am keeping my score:

  1. Transfer. The submitted loss is −0.8pp on Gemma (Table 5), and Qwen’s baseline fires on 95% of other-source rewrites (A.14), so its sensitivity is at ceiling. The repair is effectively shown only on GPT-2, and §5.5’s claim that it “transfers beyond GPT-2” is not supported.
  2. Tuning budget. If I understand correctly, the hyperparameters for the submitted loss were selected from 16 configurations (Table 4) while the hinge control in A.12 was selected from 3. Paraphrase consistency, the strongest validation baseline (+7.1pp), is still not evaluated on held-out data. A fair comparison needs a matched search budget.
Reviewer_NNVFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper is focused on the reliability of features learned by TopK SAEs under meaning-preserving paraphrases. The authors observe a phenomenon that increasing the SAE dictionary width and active budget can reduce the sensitivity of rare features, even when reconstruction quality improves. Through controlled factorial experiments, they identify the active budget, rather than dictionary width alone, as the primary factor associated with this degradation. To explain this phenomenon, the authors analyze the geometry of TopK selection and show that rare features frequently lie close to the selection boundary, making them vulnerable to small changes in feature rankings induced by paraphrasing. Based on this observation, they propose Pairwise Rank Stabilization which is an auxiliary training objective that encourages source-active features to remain ranked above competing features in their paraphrases. Experiments across different models demonstrate that the proposed method improves rare-feature sensitivity in some settings while largely preserving reconstruction quality and alive-feature coverage.

Critical strengths
  1. The authors conduct controlled experiments to disentangle the effects of dictionary width and active budget.
  2. The proposed Pairwise Rank Stabilization objective directly addresses the identified ranking instability near the TopK selection boundary.
  3. The paper compares several regularization strategies, investigates different training budgets and model architectures, and evaluates the selected method on a source-disjoint held-out collection.
Critical weaknesses
  1. The experiments do not establish whether improvements in feature sensitivity correspond to better semantic coherence, feature interpretability, or causal faithfulness. As a result, the practical significance of the reported improvements remains unclear.

  2. The authors demonstrate that smaller active margins are associated with higher feature dropout rates. However, this relationship is partly expected from the definition of TopK selection, since features closer to the selection boundary are naturally more vulnerable to score perturbations.

  3. The experiments mainly rely on feature sensitivity, reconstruction error, and alive-feature coverage.These experiments do not directly demonstrate improvements in downstream applications such as concept detection, circuit discovery, or activation steering.

Questions for authors
  • Could the authors evaluate feature interpretability or semantic consistency directly to determine whether higher sensitivity corresponds to better semantic representations?
  • Since features close to the TopK cutoff are naturally more likely to change their activation status under perturbations, what evidence establishes that boundary crowding is the primary cause of the observed degradation?
  • Can the authors demonstrate that the improved feature sensitivity translates into better performance on practical interpretability tasks, such as concept detection, feature-based steering, or causal interventions?
1 comment

Thank you for recognizing the controlled isolation of k from width, the targeted repair, and the source-disjoint evaluation.

W1. Why is improved sensitivity practically meaningful?

We understand the concern: if higher sensitivity is not tied to clearer semantics or a downstream gain, its practical value can seem unclear.

The practical point is this: if you use an already-interpreted feature as a signal, that signal should not disappear just because the wording changed. SAE features are already used for model analysis and steering (Templeton et al., 2024), and as inputs to hallucination detectors (Chen et al., 2026). We do not claim that our repair has already improved those applications. Our contribution is to diagnose and repair a budget-driven reliability failure in TopK scaling (Secs. 4–6).

W2. Isn't vulnerability near a TopK boundary expected?

We agree: features close to the cutoff are naturally more vulnerable to small score changes, and Sec. 4.3 starts from this.

Our contribution is not the observation that nearby features can drop, but that this expected boundary effect drives the scaling failure, and that it can be repaired. The controlled comparison isolates a 14.8pp effect from k versus 1.9pp from width, and the repair improves held-out sensitivity by 8.8pp. Confirmed details are in Tables 1–3.

Q1. Can semantic consistency be tested more directly?

We take this as a request for a more direct consistency check than the original paraphrases.

Yes: the same method still improves sensitivity on four new rewrite types it never trained on, and this is not just easier firing. The gains are +3.9 to +8.8pp (all p<0.01). At a fixed 20% false-alarm rate, the submitted loss also keeps 14.3 percentage points more true rewrites. Confirmed results are in App. A.10 and Table 6.

Q2. What evidence connects boundary crowding to the degradation?

Three simple facts connect them: increasing k causes the drop, the features that drop are the ones near the cutoff, and stabilizing their ranking recovers sensitivity. First, holding width fixed while varying k isolates the budget effect (Table 1; full grid in App. A.14). Second, the near-cutoff share rises from 10% to 54%, and margins predict drops at all seven depths (Fig. 2b, Table 2). Third, stabilizing ordering across the cutoff recovers sensitivity while largely preserving reconstruction and coverage (Table 3). Further confirmation is in Sec. 4.3 and App. A.13.

W3/Q3. Does the improvement translate into practical tasks?

We agree this is an important question, and we have not claimed a gain on concept detection, steering, or circuit discovery.

The practical gain we do establish is that the feature remains available under wording changes, with false alarms held fixed. For analyses that use an interpreted feature as an indicator, a missing activation could otherwise reflect phrasing rather than absence of the property. We test this on source-disjoint text and four unseen rewrite types (Table 6; App. A.10). Improved concept detection, steering, or circuit discovery is a separate question already identified in Sec. 6.

We hope our responses address your concerns. If you have any further questions, please do not hesitate to let us know.

Comments on the paper

1 comment

Dear Reviewers and Area Chairs,

We thank all reviewers for their careful reading and constructive suggestions. We are glad that you recognized:

  • controlled isolation of k from width (NNVF, LP69);
  • source-disjoint evaluation with honest controls (NNVF, LP69);
  • a clear question and diagnosis-to-repair arc (LP69, unmp);
  • a repair targeting ranking instability without reconstruction or coverage cost (NNVF, unmp).

Motivation. An interpreted feature used to analyze a property should not lose its signal under meaning-preserving wording changes. Along practical TopK scaling paths, rare-feature sensitivity can fall while reconstruction improves. Sensitivity captures this distinct reliability property.

Contributions. The controlled comparison shows that the active budget, not width, drives the drop (k −14.8pp versus width 1.9pp; Table 1). Margins predict which features drop (Table 2), and a light rank-stabilizing loss improves held-out sensitivity by 8.8pp without reconstruction or coverage cost (Table 3).

Reviewers mainly asked whether the repair merely makes features fire more easily, and whether it still works away from the training rewrites. It is not just easier firing, and the same submitted method still helps under four new rewrite types it never trained on. The confirmed comparisons are in Table 6 and App. A.10; we answer each reviewer below.

The revised PDF marks these updates in blue:

  • Secs. 5.3–5.4, Table 6: held-out results, retention at the same false-alarm rate, and an optional term against firing on other-source rewrites (unmp, NNVF).
  • App. A.6–A.14: Table A.9's negative construction, a separately labeled retrained comparison, equal-budget baselines, new rewrites, cross-model tests, k-sweep and per-cell factorial results (all).

We hope our responses address your concerns. If you have any further questions, please do not hesitate to let us know.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.