acceptodds
Under review as a conference paper at ICLR 2027

Preserving Local Activation Structure for More Reliable Sparse Autoencoder Features

Abstract

Sparse autoencoders (SAEs) provide a way to interpret neural network representations in terms of human-understandable concepts. However, recent evidence of feature instability across training runs and sensitivity to input perturbations raises concerns about the reliability of these interpretations. We argue that preserving the local structure of model activations in their sparse representations allows SAE features to reflect similarities and differences captured by the model. We introduce kNN SAE, which adds a regularizer based on k-nearest neighbors among model activations to incorporate this structural constraint into training. This objective is designed to enable consistent descriptions across related inputs and training runs, making them easier to reproduce and compare. In our experiments, kNN SAE improves feature reproducibility and performance on concept detection and disentanglement evaluations at matched sparsity, with a modest cost in reconstruction and functional fidelity. These findings suggest that local geometric consistency provides a useful foundation for learning more stable and reliable concept representations.

Primary area:
interpretability and explainable AI
Keywords:
sparse autoencoders, mechanistic interpretability, feature stability, local structure preservation, nearest neighbors, feature geometry

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BneoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the instability of SAE dictionaries by preserving the local geometry of the original model activations. The authors propose kNN SAE, which introduces a regularizer that preserves neighborhood distances into SAE training. Experiments on language models demonstrate improvements in neighborhood preservation, cross-seed reproducibility, adversarial robustness, and downstream benchmarks.

Critical strengths

The method is simple and can be incorporated into standard SAE training without modifying the architecture.

Critical weaknesses

1. The mechanisms connecting local geometry preservation to improved reliability remain insufficiently understood.

  • While the method is simple and empirically effective, it remains unclear why preserving local distances should lead to more reproducible individual dictionary atoms across independently trained SAEs. The authors provide a plausible explanation based on reducing ambiguity in local feature selection, but this mechanism is not directly validated. Preserving the geometry of the sparse representation does not necessarily imply that individual latent directions are identifiable. A controlled synthetic experiment with known ground-truth concepts could help clarify when and why the proposed regularizer improves dictionary consistency.
  • Similarly, the connection between neighborhood preservation and improvements in SAEBench performance and adversarial robustness remains unclear. Although the paper reports improvements in these metrics under kNN regularization, it does not explain why the better neighborhood preservation leads to the downstream gains.

2. The experimental evaluation leaves important questions about generalizability and interpretability.

  • All experiments use TopK SAEs as the primary baseline. It remains unclear whether the proposed regularizer provides similar benefits for other widely used SAE variants, such as ReLU, Matryoshka, or MP-SAEs.
  • Although the paper includes geometric visualizations and steering case studies, it provides limited qualitative analysis of individual learned features. Examples comparing matched latents across training seeds or showing how learned latents changes with regularization, would help substantiate the claimed improvements in interpretability and reproducibility.
  • The optimal regularization strength varies substantially across evaluation metrics, including SAEBench performance, cross-seed reproducibility, and adversarial robustness. It remains unclear whether the method provides consistent benefits under a single hyperparameter setting or requires task-specific tuning.
Questions for authors
  1. Why should preserving local distances lead to more consistent individual decoder atoms across independent training runs? Could the authors provide a controlled experiment or a more direct analysis of the proposed mechanism?
  2. The optimal regularization strength appears to vary across downstream tasks and reliability metrics. Is there a single setting that provides consistent improvements across the different objectives?
  3. Could the authors provide examples of individual latents matched across independent training runs, comparing their top-activating contexts? Such examples would help illustrate what the increased reappearance probability means for practical interpretation.
  4. Could the proposed regularizer be applied to vision SAEs, and would it be expected to work with other SAE architectures, such as ReLU, Matryoshka, or MP-SAEs?
6 comments

We appreciate the care you put into this review and your questions about the mechanism, interpretability and generalization of kNN SAE. They motivated the additional experiments and controls presented below.

Q1. Toy-model evidence for feature reproducibility

Could the authors provide a controlled experiment or a more direct analysis of the proposed mechanism?

To investigate why preserving local distances can improve dictionary consistency across seeds, we compare TopK SAE with and without kNN regularization in a controlled toy model.

We use 100 input dimensions, 150 ground-truth (GT) directions, and 125 SAE latents with TopK (K = 4). Each input combines four uniformly sampled, distinct GT features with independent amplitudes between 0.5 and 1.5. Training uses 3,000 steps, batch size 512, kg=8k_g=8, 64 anchors, and four neighbors per anchor, with geometry enabled at step 900 and a 300-step regularization ramp.

LambdaEV (%)Reappearance (%)GT → decoder cosineDecoder → GT cosine
084.23 ± 0.2187.75 ± 0.750.8852 ± 0.00440.9649 ± 0.0029
0.0384.13 ± 0.2289.45 ± 0.750.8903 ± 0.00550.9619 ± 0.0027
0.383.88 ± 0.2191.93 ± 0.910.9098 ± 0.00230.9510 ± 0.0012
183.23 ± 0.3088.30 ± 1.400.9041 ± 0.00340.9473 ± 0.0016

Means and standard deviations are computed over five GT dictionaries with 11 SAE initializations each. Reappearance uses cosine similarity of at least 0.7 between runs within the same GT dictionary.

The toy model reproduces the LLM trade-off: increasing lambda lowers EV, while reappearance first rises and then falls. At lambda 0.3, the more reproducible dictionary also provides closer matches to GT directions on average. Decoder → GT cosine decreases gradually with lambda: improved coverage of the GT dictionary can coexist with slightly weaker alignment of individual decoders to their nearest GT feature.

We then freeze each encoder and refit its unit-norm decoder and reconstruction bias using reconstruction loss on 65,536 fresh samples, with an independent 16,384-sample holdout.

LambdaEV (%)Reappearance (%)GT → decoder cosineDecoder → GT cosine
084.26 ± 0.2387.95 ± 0.590.8850 ± 0.00430.9647 ± 0.0030
0.383.93 ± 0.2392.35 ± 0.890.9096 ± 0.00220.9501 ± 0.0011

The kNN codes themselves thus support more reproducible dictionaries with better average GT coverage at a small reconstruction cost.

Together, the results from joint SAE training from scratch and from decoder training with frozen encoders motivate a hypothesis about the codes: preserving neighbor distances may help retain useful distinctions while reducing arbitrary latent switching between similar inputs. If the relevant information remains accessible through small, selective groups of latents, this could benefit SAEBench probing and interventions (SCR/TPP), while more stable latent selection could help robustness to input perturbations.

These joint-training and frozen-encoder experiments are included in Section 5.3 (Tables 1–2).

Q2. Consistent benefits with a single regularization strength

Is there a single setting that provides consistent improvements across the different objectives?

A single fixed setting already provides simultaneous benefits across all three evaluation axes. For GPT-2 layer 7, λ = 8 improves all three reported SAEBench metrics averaged over three training runs relative to λ = 0 (Sparse Probing: 65.66% → 69.79%; SCR@20: 0.2586 → 0.3157; TPP@20: 0.0962 → 0.1661), mean cross-seed feature reappearance (74.76% → 79.97%), and retained feature overlap under both replacement and suffix attacks (67.63% → 73.80% and 56.20% → 62.69%, respectively). These gains occur with EV decreasing from 89.19% to 88.08%: a 1.11 percentage-point, or 1.24% relative, reduction. Thus, simultaneous improvements do not require selecting a different coefficient for each metric.

For Gemma-2-2B layer 7, λ = 8 also improves the three reported SAEBench metrics over λ = 0 (Sparse Probing Top-1: 75.88% → 76.65%, SCR@20: 0.3509 → 0.4152, TPP@20: 0.0639 → 0.0889) and replacement/suffix retained overlap (56.48/52.31% → 64.41/58.64%), with decoder reappearance increasing across the low-λ settings to 88.36% at λ = 8. The relative EV reductions from λ = 0 are 1.24% for GPT-2 and 1.90% for Gemma-2-2B (83.71% → 82.12%).

As expected, EV decreases with increasing λ, while reproducibility initially improves and then declines under stronger regularization. We suggest using the acceptable drop in validation EV relative to the unregularized SAE to guide the choice of λ, then keeping that value fixed across downstream evaluations. A related consideration appears in Chen et al. (2022) [1], who use a reconstruction-accuracy constraint when training CkNN autoencoders.

Q3. Interpreting matched features across training seeds

Could the authors provide examples of individual latents matched across independent training runs, comparing their top-activating contexts?

To probe the mechanism, we compared latents across runs, activation patterns and selectivity changes with regularization. We compared 16 seeds each of Baseline (λ=0) and batch-local kNN SAE (λ=8), on GPT-2 layer 7 (16,384 latents, top-64), using 2M FineWeb token positions excluded from SAE training.

Reappearance is the fraction of 15 other seeds whose nearest decoder has cosine ≥0.7 within the same method. Mean reappearance rises from 74.76% for Baseline to 79.97% for kNN. For features matched across seeds, Pearson correlation of activation values is 0.779/0.789 and Jaccard overlap of the positions where they fire is 0.527/0.483 (Baseline/kNN).

To assess semantic similarity, we randomly sampled 100 accepted cross-seed pairs per method. Qwen3-32B independently described each latent using the explanation prompt from SAEBench, then compared the descriptions with the exact Appendix C prompt from [2]. Its scale is 1 (incomparable), 2 (semi-comparable/neutral), 3 (comparable). Each pair was evaluated in both orders:

MethodMean comparabilityFrequency-matched random controls
Baseline2.66/31.40/3
kNN2.60/31.32/3

For each seed, Hungarian matching assigns each Baseline latent a unique kNN partner, maximizing total decoder cosine. 78.25% of pairs have cosine ≥0.7 (mean 0.781). In each method, stable features match in ≥14 of 15 other runs, unstable features in ≤1. Cross-method cosine is below 0.7 for 82.09% of unstable versus 5.22% of stable Baseline features, and 76.59% versus 8.01% for kNN.

Two examples illustrate these comparisons.

1. Gratitude: a shift in selectivity that repeats across runs. The seed-42 anchor matches seeds 43/44 within each method. Full independent Qwen descriptions are:

Seed, latentBaselinekNN
42, 11514expressions of gratitude and references to God or divine figures.expressions of gratitude and acknowledgment, especially phrases like “thank you to the” or “thanks to them.”
43, 2144expressions of gratitude and religious or spiritual references.expressions of gratitude or thanks, especially phrases like “thank you” and related sentiments.
44, 1885expressions of gratitude and phrases involving “thank” or “thanks,” often in a religious or formal context.expressions of gratitude and appreciation, especially phrases like “thank you” or “thanks to.”

All three Baseline runs peak on “…Glory Be [to] the Father…, in honor of Saint Anthony…”. All three kNN runs peak on “…We extend our [gratitude] to our departing colleagues for their important contribution…”. Moving from Baseline to kNN on fixed texts, all three seeds reduce their response to “Christ Jesus” while increasing responses to explicit gratitude and “glorify God”: religious praise remains, while the emphasis shifts towards explicit appreciation.

All six latents have decoder matches in all 15 other seeds. Within-method anchor-to-target cosines are 0.983–0.988 for Baseline and 0.965–0.967 for kNN. Same-seed cross-method cosines are 0.866–0.897.

2. Football/soccer: increased reappearance accompanied by stronger activation agreement. The seed-49 anchor changes from a broad acronym-related description to football/soccer-related terms, with decoder matches increasing from none to all 15 other seeds. Full independent Qwen descriptions are:

Seed, latentBaselinekNN
49, 12062abbreviations or acronyms that follow the pattern of two or more capitalized letters, often related to organizations or entities.words related to soccer and variations of team names like “FC” or “MLS”.
50, 8492abbreviations like “FC” and “Soccer” related to sports teams or clubs.words related to football/soccer and club-related terms.
51, 4395words related to soccer and football clubs, especially abbreviations like “FC.”the substring “FC” and related terms in the context of sports teams and clubs.

In kNN, seed 49 peaks on “…quit the [Soccer]oos later this year. ‘I have one kid…’”. Seeds 50 and 51 both peak on “…partnership with [FC] Barcelona… sponsoring the club…”. For seed 49 on fixed texts, moving from Baseline to kNN reduces activation on real-estate “MLS” (20.42→16.91) and increases it on “Upward Soccer” (15.80→26.49).

For the two targets, within-method cosines rise from 0.685–0.695 in Baseline (below threshold) to 0.925–0.943 in kNN, and Pearson from 0.178–0.199 to 0.632–0.684. Same-seed cross-method cosines are 0.707–0.927. Football/soccer responses already occur in Baseline, and acronym sensitivity remains in kNN. Increased reappearance does not mean a previously absent concept was created.

For details, see Appendix G (Tables 13–17) and Section 5.3.

Q4. Generalization to vision and other SAE architectures

Could the proposed regularizer be applied to vision SAEs, and would it be expected to work with other SAE architectures, such as ReLU, Matryoshka, or MP-SAEs?

We tested ReLU, Matryoshka and MP-SAEs against their own baselines without the kNN regularizer (λ=0). All models use GPT-2 layer 7, 16,384 features and 1B training tokens. ReLU uses fixed L1 α=0.0024. Matryoshka uses per-token TopK-64, reconstruction prefixes 4,096/8,192/16,384 and kNN loss on the full code. MP-SAE uses 64 pursuit steps.

SAEλEV (%)Recall@16 (%)Reappearance (%)Attack R/S (%)SP-1SCR@20TPP@20
ReLU083.3541.3894.3960.85/50.4068.180.10500.1730
ReLU883.4451.8892.3263.62/54.0667.720.14840.0582
ReLU3283.5860.4184.8475.17/67.1170.740.18500.1254
Matryoshka088.3162.7957.2572.41/64.2768.030.31190.1182
Matryoshka887.1576.5566.7075.15/64.7570.370.31240.1877
Matryoshka3285.2278.8357.9275.88/65.6372.490.35900.2786
MP-SAE092.1939.1874.3919.26/17.2863.560.26720.0761
MP-SAE891.0833.8771.3818.73/16.8767.000.24870.0622
MP-SAE3289.6252.0829.5619.71/18.2764.250.23740.0832

SAEBench averages three training seeds per setting.

Matryoshka gives the clearest positive answer to this question. At λ=8, better neighborhood preservation and more reproducible features come together with gains in sparse probing and TPP, at a modest reconstruction cost. ReLU's recall gains come with lower reappearance and mixed SAEBench results. MP-SAE shows a clearer limitation: its recall gain at λ=32 comes at a large cost to feature stability. The benefits extend beyond TopK, but they do not transfer equally to every architecture.

ReLU's mean L₀ rises from about 62 to 77 at λ=32, possibly because stronger kNN regularization favors denser codes while the L1 weight stays fixed.

Dead features at λ=0/8/32 (no activations on 200,000 held-out tokens): ReLU 0/89/829, Matryoshka 3/150/384, MP-SAE 0/0/0. Reappearance excludes dead features from both models and uses seven comparison runs. R/S denotes replacement/suffix attacks.

The ReLU, Matryoshka, and MP-SAE experiments are included in Appendix E (Table 12).

We also evaluated TopK SAEs on DINOv2-Base (ViT-B/14, layer 11). We used F=24,576, K=32 and 1B ImageNet-100 training patches. In ViSAEBench [3], sparse probing tests class prediction using a small set of SAE features, while spatial coherence measures clustering of feature activations on the patch grid. Results are mean ± std over four seeds.

λSparse probing AUC ↑Spatial coherence ↑
00.3177 ± 0.01170.2232 ± 0.0063
10.3002 ± 0.01890.2234 ± 0.0067
40.3080 ± 0.02100.2329 ± 0.0027

λ=4 improved spatial coherence in all four seeds, but did not improve mean sparse probing.

We thank the reviewer for suggesting this extension. Our paper focuses on LLMs, and we leave a systematic study of vision SAEs to future work.

Thank you again for your helpful comments. They led us to add more direct tests and explain our conclusions and limitations more clearly.

References

[1] Chen et al. Local Distance Preserving Auto-encoders using Continuous kNN Graphs. Topological, Algebraic, and Geometric Learning Workshops 2022.

[2] Balagansky et al. Mechanistic Permutability: Match Features Across Layers. ICLR 2025.

[3] ViSAEBench.

Reviewer_qno5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Sparse autoencoders (SAEs) provide a way to interpret neural networks by decomposing their internal activations into sparse, human-understandable features. However, these features are sensitive to small input perturbations and can vary across independent training runs. The paper proposes an extension of standard TopK SAE to kNN SAE, which is a geometric regularization approach aiming to improve the stability and robustness of sparse features. Specifically, the authors adapt a distance-preservation loss from Chen et al. (2022), using a continuous k-nearest-neighbour(CkNN) graph to preserve local activation geometry in sparse code. This method is evaluated across several language models, demonstrating improvements in feature stability, robustness and interpretability.

Critical strengths

1.Clear presentation.The paper is generally well structured and easy to follow. 2. Intuitive motivation. I think preserving local activation geometry is an intuitive idea for addressing SAE instability.

Critical weaknesses

1.Insuffisient experimental validation and analysis. Most comparisons are limited to vanilla TopK SAE , without competitive baselines, such as iSAE(Nelson et.al) or simpler graph-based regularizer), to establish the specific advantages of the method. The qualitative analysis is also very weak. So, it is difficult to assess how significant and reliable the improvements really are.

2.Limited methodological novelty and contribution.The central contribution appears to be adapting the CkNN distance-preservation objective from Chen et al. (2022) and only incorporating it into the standard TopK SAE training loss. While applying this constraint is a reasonable, I am not convinced that it is sufficient to call a methodological contribution. The authors are suggested to perform a clear methodological or mechanistic illustration to further explain how the proposed regularization changes feature learning compared with existing approaches. Currently, it remains unclear what fundamentally new insight the paper offers beyond applying an existing geometric regularizer to SAEs.

Questions for authors

1.Could the authors include comparisons with additional relevant baselines, such as iSAE or a simpler graph-based regularizer, to better establish the competitiveness of kNN SAE?

2.What is the key methodological insight beyond adapting the CkNN distance-preservation loss from Chen et al. (2022)? In particular, could the authors clarify why applying the regularization after TopK sparsification is important, ideally through a controlled ablation against pre-sparsification regularization?

3.The authors suggest that preserving local geometry reduces unstable feature selection across training runs and input perturbations. Could this mechanism be supported by more direct evidence, like feature-level analyses ?

4.Could the authors provide more intuitive illustration of the proposed method, highlighting how the CkNN regularization is incorporated into TopK SAE and how it differs from prior approaches? This could help readers better understand the method and identify its specific methodological contributions.

5 comments

We appreciate the reviewer’s thoughtful assessment and the recognition of the paper’s clear presentation and intuitive motivation. We added baseline comparisons and controlled analyses to address your questions.

Q1. Additional SAE baselines.

Could the authors include comparisons with additional relevant baselines, such as iSAE or a simpler graph-based regularizer, to better establish the competitiveness of kNN SAE?

We added graph-regularized SAE baselines, GeoSAE [1] and GSAE [2] (two variants), to test whether their objectives also preserve activation-space neighborhoods, and iSAE [3] as an additional baseline. Performance runs use GPT-2 layer 7, 1B target tokens, TopK-64 and 16,384 features. We choose regularization strengths to compare methods at roughly 85–89% EV.

GeoSAE encourages neighboring inputs to have similar encoder outputs before sparsification. We construct its neighborhood graph over token activations within each training batch.

GSAE (neuron graph) retains GSAE's cosine-weighted graph over LLM activation coordinates and its Laplacian penalty on decoder vectors. We rebuild the graph from coordinate profiles across tokens in each batch, replacing prompt-averaged profiles, and omit the supervised component of the original loss.

Sample-Laplacian SAE (ours) applies GSAE's Laplacian smoothing principle to SAE codes rather than decoder vectors. Within each batch, we connect nearby token activations and penalize the weighted sum of squared differences between their codes, after scaling each code by the inverse square root of its weighted graph degree. Gaussian edge weights determine each pair's contribution. Using the same input neighborhoods and pair budget as kNN SAE lets us compare geometric smoothing with local distance matching.

MethodλEVInactiveRecall@16ReappearanceSP-1SCR@20TPP@20
Baseline Topk089.19055.2474.8665.660.25860.0962
GeoSAE0.02588.271,10456.2374.9865.270.22740.1041
GeoSAE0.186.844,10857.6373.2564.580.29010.1137
GeoSAE0.2585.117,45660.3572.1267.890.23990.1201
Sample-Laplacian SAE (ours)0.000388.33063.8983.3967.720.21300.0503
Sample-Laplacian SAE (ours)0.00185.73365.8191.0769.960.17710.0264

EV, Recall@16, Reappearance and SP-1 are percentages, reappearance uses active features. SAEBench metrics are averaged over three training runs.

Where evaluated, robustness (replacement/suffix, %) exceeds Baseline Topk (67.63/56.20): GeoSAE reaches 71.68/61.83 at λ = 0.1 and 76.57/66.90 at λ = 0.25; Sample-Laplacian SAE reaches 72.21/62.20 at λ = 0.001.

Inactive features have zero firing on the same 200k held-out tokens. GeoSAE's many inactive features may reflect its preactivation-matching loss suppressing encoder variation before TopK.

For GSAE (neuron graph), decoder effective rank drops relative to Baseline Topk by roughly 24% at λ = 0.01, 97% at λ = 0.03 and 99% at λ = 0.1. The penalty favors the same smooth patterns across connected LLM coordinates for every decoder vector, allowing different features to concentrate in a small shared subspace. Before this concentration, λ = 0.003 gives EV 89.19%, Recall@16 55.14%, SP-1 65.20%, SCR@20 0.2607 and TPP@20 0.0883.

We added single-pass iSAE (L1=0) following the authors' implementation: GPT-2 L7, 16,384 features, K=64, 1B tokens and nine training runs per setting. The aRIP coefficient multiplies the approximate restricted isometry regularization loss; zero gives the AbsTopK control. EV is averaged over the nine training runs.

aRIP coefficientEV (%)InactiveSP Top-1 (%)SCR@20TPP@20Reappearance signed / abs. (%)Robustness (%)
0 (AbsTopK)87.08382369.790.23180.0202841.87 / 80.8265.01 / 55.87
0.186.3691,12070.700.25730.0273334.12 / 66.3762.21 / 53.38
0.2585.5781,87672.290.27800.0391527.72 / 55.1463.34 / 54.05

For iSAE, inactivity means neither sign activates on the same 200k FineWeb positions. We report signed and absolute cosine because flipping both decoder and activation signs leaves AbsTopK reconstruction unchanged. Stronger aRIP improves SP/SCR/TPP but lowers EV and reappearance; attack overlap stays below AbsTopK.

The baselines favor different properties: GeoSAE improves robustness, Sample-Laplacian SAE improves recurrence, and our iSAE sweep improves probing. Neuron-graph GSAE can concentrate decoder directions. Among the evaluated graph baselines near 88% EV, kNN SAE preserves neighborhoods best. Overall, at comparable EV, kNN SAE performs better on most of the reported metrics across these comparisons.

The graph-based and iSAE comparisons are included in Appendix D.1 and D.3 (Tables 9 and 11).

Q2. Methodological novelty and regularization after TopK.

In particular, could the authors clarify why applying the regularization after TopK sparsification is important, ideally through a controlled ablation against pre-sparsification regularization?

We agree that the distance-preservation objective itself is not new, as we adapt it from [4]. To test whether its placement matters for SAE training, we compare applying the same regularizer before and after TopK, keeping the rest of the training setup fixed. At comparable EV, applying it after TopK improves neighborhood preservation and attack retention while leaving fewer inactive features.

We use GPT-2 layer 7 (16,384 features, TopK-64, 1B tokens) with matched data, graph, optimizer, and regularization schedule. The table reports λ=4\lambda = 4 and λ=8\lambda = 8. Reappearance averages all pairs of 11 shared seeds at cosine 0.7. SAEBench metrics are averaged over three training runs.

MetricPre, λ=4\lambda = 4Post, λ=4\lambda = 4Pre, λ=8\lambda = 8Post, λ=8\lambda = 8
EV (%)88.3388.5687.8388.08
Recall@16 (%)66.2072.2469.3274.44
Sparse Probing Top-1 (%)66.4567.9367.8669.79
SCR@200.27830.26050.28790.3157
TPP@200.09280.13140.11850.1661
Replacement overlap (%)65.8973.2666.0373.80
Suffix overlap (%)54.2963.6053.1462.69
Decoder reappearance (%)78.9178.3480.9779.99

Inactive features have zero firing on the same 200k held-out FineWeb positions:

λPre-TopK inactivePost-TopK inactive
420
8201
641,77427
1283,25479

Post-TopK leaves fewer features inactive, with a substantially larger gap under strong regularization.

EV differs by only 0.23–0.25 percentage points, while Post-TopK achieves higher Recall, better Sparse Probing and TPP scores, and greater robustness under both attacks. SCR shows no consistent advantage for either variant.

The ablation shows why the placement of the loss matters for SAEs: preserving the dense encoder's geometry and preserving the geometry of its sparse features are different training goals. In this setting, our evidence supports the latter for neighborhood preservation, Sparse Probing, TPP, robustness, and retaining active features, while Pre-TopK is slightly higher on feature reproducibility and SCR remains mixed.

Appendix B provides the loss definitions, placement comparison, and inactive-feature counts (Tables 4–5).

Q3. Feature-level evidence for the stability mechanism.

Could this mechanism be supported by more direct evidence, like feature-level analyses ?

To probe the mechanism, we compared latents across runs, activation patterns and selectivity changes under regularization. We used 16 runs per method (Baseline λ=0, batch-local kNN SAE λ=8), GPT-2 layer 7 (16,384 latents, top-64) and 2M held-out FineWeb positions.

Within-method reappearance is the fraction of 15 other seeds with a nearest decoder cosine ≥0.7, averaging 74.76% for Baseline and 79.97% for kNN. Matched-feature activation Pearson is 0.779/0.789 and firing-position Jaccard is 0.527/0.483 (Baseline/kNN).

For 100 randomly sampled accepted cross-seed pairs per method, Qwen3-32B independently described each latent with SAEBench’s explanation prompt, then compared descriptions with the exact Appendix C prompt from [5] in both orders. Scale: 1 (incomparable), 2 (semi-comparable/neutral), 3 (comparable):

MethodMean comparabilityFrequency-matched random controls
Baseline2.66/31.40/3
kNN2.60/31.32/3

One-to-one Hungarian matching pairs all latents within each seed, maximizing total Baseline–kNN decoder cosine. 78.25% of pairs have cosine ≥0.7 (mean 0.781). Within methods, stable features match ≥14 of 15 other seeds and unstable features ≤1. Cross-method cosine is below 0.7 for 82.09% of unstable versus 5.22% of stable Baseline features, and 76.59% versus 8.01% for kNN.

We investigate the interpretations of matched features.

1. Gratitude: repeatable selectivity shift. The seed-42 anchor matches seeds 43/44 within each method. Full Qwen3-32 descriptions:

Seed, latentBaselinekNN
42, 11514expressions of gratitude and references to God or divine figures.expressions of gratitude and acknowledgment, especially phrases like “thank you to the” or “thanks to them.”
43, 2144expressions of gratitude and religious or spiritual references.expressions of gratitude or thanks, especially phrases like “thank you” and related sentiments.
44, 1885expressions of gratitude and phrases involving “thank” or “thanks,” often in a religious or formal context.expressions of gratitude and appreciation, especially phrases like “thank you” or “thanks to.”

All three Baseline runs peak on “…Glory Be [to] the Father…, in honor of Saint Anthony…”, all three kNN runs on “…We extend our [gratitude] to our departing colleagues for their important contribution…”. On fixed texts, all three seeds respond less to “Christ Jesus” and more to explicit gratitude and “glorify God” under kNN. Religious praise remains as emphasis shifts toward explicit appreciation.

All six latents match all 15 other seeds. Within-method anchor-to-target cosines are 0.983–0.988 (Baseline) and 0.965–0.967 (kNN). Same-seed cross-method cosines are 0.866–0.897.

2. Football/soccer: greater stability. The seed-49 anchor’s description shifts from broad acronyms to football/soccer, with decoder matches rising from 0 to all 15 other seeds:

Seed, latentBaselinekNN
49, 12062abbreviations or acronyms that follow the pattern of two or more capitalized letters, often related to organizations or entities.words related to soccer and variations of team names like “FC” or “MLS”.
50, 8492abbreviations like “FC” and “Soccer” related to sports teams or clubs.words related to football/soccer and club-related terms.
51, 4395words related to soccer and football clubs, especially abbreviations like “FC.”the substring “FC” and related terms in the context of sports teams and clubs.

Under kNN, seed 49 peaks on “…quit the [Soccer]oos later this year. ‘I have one kid…’”, seeds 50/51 on “…partnership with [FC] Barcelona… sponsoring the club…”. On fixed texts, seed 49’s Baseline→kNN activation is 20.42→16.91 for real-estate “MLS” and 15.80→26.49 for “Upward Soccer”.

For the two targets, within-method cosines rise from 0.685–0.695 (Baseline, below threshold) to 0.925–0.943 (kNN), and Pearson from 0.178–0.199 to 0.632–0.684. Same-seed cross-method cosines are 0.707–0.927. Football/soccer already occurs in Baseline and acronym sensitivity remains in kNN.

For details, see Appendix G (Tables 13–17) and Section 5.3.

Gaussian noise ε∼N(0,(rs)2I)\varepsilon\sim\mathcal{N}(0,(rs)^2I), s≈2.883s\approx2.883 (RMS standard deviation of clean coordinates), is tested on 2M held-out FineWeb positions with three runs, eight shared draws/position and ≈12.3k matched features/run on identical clean firing positions. Mean retention gains over baseline (pp):

λr=0.1r=0.4r=0.85
1+0.81+1.88+2.47
2+0.90+2.16+2.94
4+0.71+1.85+2.88
8−0.15−0.06+1.36

Moderate regularization improves retention at all three noise scales, with λ=2 best (Appendix H, Table 18).

Reviewer_6MonOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Motivated by the instability of sparse autoencoders (SAEs) across training runs and their sensitivity to input perturbations, this paper proposes kNN SAE, which adds regularization based on k-nearest neighbors based on the original model activations to preserve local structure. Empirical results show that (1) kNN SAE indeed preserves neighborhood structure better than vanilla SAE; (2) kNN SAE performs better than vanilla SAE on downstream tasks; (3) kNN SAE has better feature stability than vanilla SAE across training runs; and (4) kNN SAE is more robust to input perturbation than vanilla SAE; and (5) kNN SAE achieves a better steering-vs-fluency trade-off than vanilla SAE.

Critical strengths
  • The approach is simple and can be adopted to different SAE architecture.
  • The empirical experiments cover a wide range of SAE use cases including concept detection, removal, and steering.
Critical weaknesses
  • Other approaches of regularizing SAEs with respect to some local structure are not included as baselines for comparison. These include GeoSAE and GSAE as mentioned by the authors, as well as T-SAE [1]. These approaches are expected to also preserve local neighborhood to some extent, so it's important to consider them in the evaluation.
  • kNN SAE seems to be motivated by the problem of preserving local neighborhood, which is shown in Section 5.1 to work. Although kNN SAE performs better than vanilla SAE in other aspects, it remains unclear how preserving local neighborhood is theoretically related to those evaluations. I acknowledge that the authors offer some hypotheses, but stronger connections would (1) further justify the approach and (2) provide intuition for practitioners when applying kNN SAE.

References

[1] https://arxiv.org/abs/2511.05541

Questions for authors
  • How does the performance of kNN SAE compare to GeoSAE, GSAE, and T-SAE?
  • Theoretically, how does the kNN regularization relate to feature stability, concept detection, removal, and steering?
  • How does the performance of kNN SAE vary with respect to the number of neighbors used (kk)?
6 comments

Thank you for the thoughtful review and for recognizing the simplicity of our approach and the breadth of the evaluation. We added the comparisons, controlled experiments and sensitivity analyses below.

Q1 — Comparison with GeoSAE, GSAE and T-SAE.

How does the performance of kNN SAE compare to GeoSAE, GSAE, and T-SAE?

We added GeoSAE [1], GSAE [2] (two variants) and T-SAE [3] baselines to test activation-space neighborhood preservation. Performance runs use GPT-2 layer 7, 1B target tokens, TopK-64 and 16,384 features. We choose regularization strengths to compare methods at roughly 85–89% EV.

GeoSAE encourages nearby LLM activations to produce similar SAE encoder responses before sparsification. We adapt it to token-level training by selecting neighbors within each batch.

GSAE (neuron graph) retains GSAE's cosine-weighted graph over LLM activation coordinates and its Laplacian penalty on decoder vectors. We rebuild the graph from coordinate profiles across tokens in each batch, replacing prompt-averaged profiles, and omit the supervised component of the original loss.

Sample-Laplacian SAE (ours) applies GSAE's Laplacian smoothing principle to SAE codes rather than decoder vectors. Within each batch, we connect nearby token activations and penalize the weighted sum of squared differences between their codes, after scaling each code by the inverse square root of its weighted graph degree. Gaussian edge weights determine each pair's contribution. Using the same input neighborhoods and pair budget as kNN SAE lets us compare geometric smoothing with local distance matching.

MethodλEVInactiveRecall@16ReappearanceSP-1SCR@20TPP@20
Baseline Topk089.19055.2474.8665.660.25860.0962
GeoSAE0.02588.271,10456.2374.9865.270.22740.1041
GeoSAE0.186.844,10857.6373.2564.580.29010.1137
GeoSAE0.2585.117,45660.3572.1267.890.23990.1201
Sample-Laplacian SAE (ours)0.000388.33063.8983.3967.720.21300.0503
Sample-Laplacian SAE (ours)0.00185.73365.8191.0769.960.17710.0264

EV, Recall@16, Reappearance and SP-1 are percentages, reappearance uses active features. SAEBench metrics are averaged over three training runs.

Where evaluated, robustness (replacement/suffix, %) exceeds Baseline Topk (67.63/56.20): GeoSAE reaches 71.68/61.83 at λ = 0.1 and 76.57/66.90 at λ = 0.25; Sample-Laplacian SAE reaches 72.21/62.20 at λ = 0.001.

Inactive features have zero firing on the same 200k held-out tokens. GeoSAE's many inactive features may reflect its preactivation-matching loss suppressing encoder variation before TopK.

For GSAE (neuron graph), decoder effective rank drops relative to Baseline Topk by roughly 24% at λ = 0.01, 97% at λ = 0.03 and 99% at λ = 0.1. The penalty favors the same smooth patterns across connected LLM coordinates for every decoder vector, allowing different features to concentrate in a small shared subspace. Before this concentration, λ = 0.003 gives EV 89.19%, Recall@16 55.14%, SP-1 65.20%, SCR@20 0.2607 and TPP@20 0.0883.

T-SAE uses a 20%/80% feature split and reconstruction losses. A contrastive loss weighted by α aligns adjacent tokens' high-level codes, α = 0 gives the Matryoshka control. Reconstruction is trained on 1B tokens, their preceding tokens are also encoded to compute the temporal loss.

MethodαkNN λEVInactiveRecall@16ReappearanceSP-1SCR@20TPP@20
Matryoshka control0088.68057.9071.5770.350.33720.1183
T-SAE0.01087.70041.1877.6271.010.27880.1359
T-SAE0.2087.761,37262.4375.0268.850.27590.2402
T-SAE + kNN0.01886.54062.1378.6074.180.30480.1304

T-SAE improves robustness over its Matryoshka control (68.72/59.79): 71.26/61.02 at α = 0.01 and 73.46/62.50 at α = 0.2. At fixed α = 0.01, adding kNN (λ = 8) raises Recall@16 from 41.18% to 62.13% and improves reappearance, SP-1 and SCR@20, EV and TPP@20 decrease.

On GPT-2 layer 7, near 88% EV, kNN TopK SAE (λ = 8) reaches 74.44% Recall@16, versus 56.23% for GeoSAE, 63.89% for Sample-Laplacian SAE and 41.18–62.43% for T-SAE. It also exceeds GeoSAE and Sample-Laplacian SAE on all three SAEBench scores in these comparisons. GSAE (neuron graph) remains close to Baseline Topk before decoder concentration appears. The alternatives have different strengths: T-SAE can score higher on SP-1 or TPP@20, GeoSAE on robustness and Sample-Laplacian SAE on reappearance. Overall, at comparable EV, kNN SAE performs better on most of the reported metrics across these comparisons.

The baseline comparisons and implementation details are included in Appendix D.1–D.2 (Tables 9–10).

Q2 — Connection to feature stability and downstream performance.

Theoretically, how does the kNN regularization relate to feature stability, concept detection, removal, and steering?

To investigate why preserving local distances can improve dictionary consistency across seeds, we compare TopK SAE with and without kNN regularization in a controlled toy model.

We use 100 input dimensions, 150 ground-truth (GT) directions, and 125 SAE latents with TopK (K = 4). Each input combines four uniformly sampled, distinct GT features with independent amplitudes between 0.5 and 1.5. Training uses 3,000 steps, batch size 512, kg=8k_g=8, 64 anchors, and four neighbors per anchor, with geometry enabled at step 900 and a 300-step regularization ramp.

LambdaEV (%)Reappearance (%)GT → decoder cosineDecoder → GT cosine
084.23 ± 0.2187.75 ± 0.750.8852 ± 0.00440.9649 ± 0.0029
0.0384.13 ± 0.2289.45 ± 0.750.8903 ± 0.00550.9619 ± 0.0027
0.383.88 ± 0.2191.93 ± 0.910.9098 ± 0.00230.9510 ± 0.0012
183.23 ± 0.3088.30 ± 1.400.9041 ± 0.00340.9473 ± 0.0016

Means and standard deviations are computed over five GT dictionaries with 11 SAE initializations each. Reappearance uses cosine similarity of at least 0.7 between runs within the same GT dictionary.

The toy model reproduces the LLM trade-off: increasing lambda lowers EV, while reappearance first rises and then falls. At lambda 0.3, the more reproducible dictionary also provides closer matches to GT directions on average. Decoder → GT cosine decreases gradually with lambda: improved coverage of the GT dictionary can coexist with slightly weaker alignment of individual decoders to their nearest GT feature.

We then freeze each encoder and refit its unit-norm decoder and reconstruction bias using reconstruction loss on 65,536 new samples, with an independent 16,384-sample holdout.

LambdaEV (%)Reappearance (%)GT → decoder cosineDecoder → GT cosine
084.26 ± 0.2387.95 ± 0.590.8850 ± 0.00430.9647 ± 0.0030
0.383.93 ± 0.2392.35 ± 0.890.9096 ± 0.00220.9501 ± 0.0011

The kNN codes themselves thus support more reproducible dictionaries with better average GT coverage at a small reconstruction cost.

Together, the results from joint SAE training from scratch and from decoder training with frozen encoders motivate a hypothesis about the codes: preserving neighbor distances may help retain useful distinctions while reducing arbitrary latent switching between similar inputs. If the relevant information remains accessible through small, selective groups of latents, this could benefit SAEBench probing and interventions (SCR/TPP), while more stable latent selection could help robustness to input perturbations.

These joint-training and frozen-encoder experiments are included in Section 5.3 (Tables 1–2).

Q3 — Sensitivity to neighborhood and sampling parameters.

How does the performance of kNN SAE vary with respect to the number of neighbors used k?

Our standard training setup samples A=256A=256 anchor activations from each batch and q=4q=4 graph neighbors per anchor, giving Aq=1024Aq=1024 pair terms in the regularization loss. Anchors and neighbors are sampled with replacement. The CkNN graph uses kg=16k_g=16: each activation's distance to its 16th nearest neighbor sets its local distance scale, which determines graph connectivity. We vary these three parameters around their standard values below.

We trained TopK-64 SAEs with 16,384 features on GPT-2 layer 7 activations from 1B tokens. Each batch graph contains M=8192M=8192 activations, and we use λ=8\lambda=8 unless stated otherwise. Values are percentages except SCR/TPP and pair counts.

Sampled neighbors and anchors (kg=16k_g=16).

A/qPairs/stepEVRecall@16ReappearanceSPSCR@20TPP@20Robustness R/S
64/425687.6176.0478.3671.340.30750.207673.87 / 63.57
256/125687.8275.4579.4471.130.32820.186275.44 / 66.12
256/41,02488.0874.4479.8569.790.31570.166173.80 / 62.69
256/164,09688.1674.2879.5068.330.31020.154773.41 / 64.25
1024/44,09688.3873.4478.4968.810.29830.135074.06 / 61.70
4096/416,38488.5372.4876.7469.180.29010.120071.76 / 62.18

At fixed A=256A=256, increasing qq slightly improves EV but lowers Recall and all three mean SAEBench scores, while reappearance changes little. At fixed q=4q=4, increasing AA also improves EV and lowers Recall. Reappearance peaks at A=256A=256 and declines for larger AA, alongside SCR and TPP. The sampled pair budget grows as AqAq.

We also compared CkNN with union kNN at λ=8\lambda=8, keeping kg=16k_g=16, A=256A=256, and q=4q=4 fixed.

GraphEVSPSCR@20TPP@20
CkNN88.0869.790.31570.1661
Union kNN88.3269.840.30850.1497

Union kNN has slightly higher EV and similar mean SP, but lower mean SCR and TPP.

We also varied kgk_g, which sets the local distance scale used to build the CkNN graph. It refers to the kgk_g-th nearest neighbor, while the number of graph neighbors can differ. With A=256A=256 and q=4q=4, values of kg=8,16,32k_g=8,16,32 give EV 87.96/88.08/88.17% and Recall@16 74.82/74.44/74.35%, respectively. Without regularization, these scores are 89.19% and 55.24%. The differences across kgk_g are small: 0.21 percentage points in EV and 0.47 in Recall. Recall is evaluated at 16 neighbors throughout.

These graph and sampling sensitivity analyses are included in Appendix C (Tables 6–8).

Thank you again for the constructive feedback. It helped us make the comparisons more complete and clarify both the benefits and limitations of our approach.

References

[1] Nerrise et al. GeoSAE: Geometric Prior-Guided Layer-Wise Sparse Autoencoder Annotation of Brain MRI Foundation Models. CVPR Workshops 2026.

[2] Yeon et al. Graph-Regularized Sparse Autoencoders for LLM Safety Steering. NeurIPS 2026.

[3] Bhalla et al. Temporal Sparse Autoencoders: Leveraging the Sequential Nature of Language for Interpretability. ICLR 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and New Results

We thank all three reviewers for their constructive feedback and discussion. We expanded the comparisons, added controlled synthetic experiments and feature-level analyses, and incorporated the follow-up results into the revised manuscript. The initial rebuttal additions are highlighted in blue, and the subsequent updates in red.

(i) Structural and identifiability baselines [6Mon, qno5]. Appendix D.1–D.3 compares kNN SAE with token-level adaptations of GeoSAE, GSAE and T-SAE, single-pass iSAE, and our Sample-Laplacian control. Reconstruction, neighborhood preservation, reappearance and downstream utility are reported together to make their trade-offs explicit. On GPT-2 layer 7, a fixed λ=8 improves reappearance and all three reported SAEBench metrics over TopK, at a 1.11 percentage-point EV cost. Sparse Probing Top-1 rises from 65.66% to 69.79%, SCR@20 from 0.2586 to 0.3157, and TPP@20 from 0.0962 to 0.1661.

(ii) Ground-truth feature recovery [6Mon, qno5, Bneo]. To test how distance matching affects feature recovery, we trained eightfold-expanded SAEs on five known ground-truth (GT) dictionaries with 11 initializations each, then refitted the decoders with the encoders frozen. At λ=0.3, mean best-match GT-to-decoder cosine rises from 0.9019 to 0.9153 and reappearance from 89.63% to 94.47%. The reappearance gain persists after refitting and comes mainly from matches between GT features and mixtures containing them (Section 5.3, Tables 1–2 and Appendix I.1–I.3).

(iii) Connection to downstream scores [6Mon, Bneo]. The synthetic results suggest how grouping GT features into shared latents could improve SAEBench scores. A one-latent probe could detect the same concept across contexts (Sparse Probing Top-1). Within a fixed ablation budget, shared latents could also make it easier to suppress a spurious cue while preserving the task signal (SCR@20) or selectively weaken the target-class probe (TPP@20).

(iv) Regularization before versus after sparsification [qno5]. Appendix B compares the same distance-matching objective before and after TopK under matched training conditions. Post-TopK improves neighborhood recall and adversarial feature retention at comparable EV, while producing fewer inactive features. This supports applying the constraint to the final sparse representation.

(v) Individual features and perturbations [qno5, Bneo]. Appendix G examines matched features using activation agreement, independently generated descriptions, cross-method assignment and contextual examples. The GPT-2 analysis uses 16 runs per method and two million held-out token positions. Appendix H adds conditional feature-retention measurements under Gaussian activation noise.

(vi) Other SAE architectures [Bneo]. Appendix E (Table 12) shows that reproducibility and selected downstream gains extend to Matryoshka and, at smaller regularization strengths, MP-SAEs. The added MP-SAE results cover λ=1, 2 and 4. ReLU shows mixed effects, with no reappearance gain and varying sparsity under a fixed L1 penalty.

(vii) Neighborhood and sampling sensitivity [6Mon]. Appendix C varies the CkNN local-scale rank, anchor count, sampled neighbors and graph family. Reconstruction and Recall@16 change little across the tested local-scale ranks. The sampling comparisons document the associated performance trade-offs.

We appreciate the reviewers’ suggestions, which helped strengthen the comparisons and sharpen the interpretation of our findings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.