- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper addresses the instability of SAE dictionaries by preserving the local geometry of the original model activations. The authors propose kNN SAE, which introduces a regularizer that preserves neighborhood distances into SAE training. Experiments on language models demonstrate improvements in neighborhood preservation, cross-seed reproducibility, adversarial robustness, and downstream benchmarks.
Critical strengths
The method is simple and can be incorporated into standard SAE training without modifying the architecture.
Critical weaknesses
1. The mechanisms connecting local geometry preservation to improved reliability remain insufficiently understood.
- While the method is simple and empirically effective, it remains unclear why preserving local distances should lead to more reproducible individual dictionary atoms across independently trained SAEs. The authors provide a plausible explanation based on reducing ambiguity in local feature selection, but this mechanism is not directly validated. Preserving the geometry of the sparse representation does not necessarily imply that individual latent directions are identifiable. A controlled synthetic experiment with known ground-truth concepts could help clarify when and why the proposed regularizer improves dictionary consistency.
- Similarly, the connection between neighborhood preservation and improvements in SAEBench performance and adversarial robustness remains unclear. Although the paper reports improvements in these metrics under kNN regularization, it does not explain why the better neighborhood preservation leads to the downstream gains.
2. The experimental evaluation leaves important questions about generalizability and interpretability.
- All experiments use TopK SAEs as the primary baseline. It remains unclear whether the proposed regularizer provides similar benefits for other widely used SAE variants, such as ReLU, Matryoshka, or MP-SAEs.
- Although the paper includes geometric visualizations and steering case studies, it provides limited qualitative analysis of individual learned features. Examples comparing matched latents across training seeds or showing how learned latents changes with regularization, would help substantiate the claimed improvements in interpretability and reproducibility.
- The optimal regularization strength varies substantially across evaluation metrics, including SAEBench performance, cross-seed reproducibility, and adversarial robustness. It remains unclear whether the method provides consistent benefits under a single hyperparameter setting or requires task-specific tuning.
Questions for authors
- Why should preserving local distances lead to more consistent individual decoder atoms across independent training runs? Could the authors provide a controlled experiment or a more direct analysis of the proposed mechanism?
- The optimal regularization strength appears to vary across downstream tasks and reliability metrics. Is there a single setting that provides consistent improvements across the different objectives?
- Could the authors provide examples of individual latents matched across independent training runs, comparing their top-activating contexts? Such examples would help illustrate what the increased reappearance probability means for practical interpretation.
- Could the proposed regularizer be applied to vision SAEs, and would it be expected to work with other SAE architectures, such as ReLU, Matryoshka, or MP-SAEs?
6 comments
Q3. Interpreting matched features across training seeds
Could the authors provide examples of individual latents matched across independent training runs, comparing their top-activating contexts?
To probe the mechanism, we compared latents across runs, activation patterns and selectivity changes with regularization. We compared 16 seeds each of Baseline (λ=0) and batch-local kNN SAE (λ=8), on GPT-2 layer 7 (16,384 latents, top-64), using 2M FineWeb token positions excluded from SAE training.
Reappearance is the fraction of 15 other seeds whose nearest decoder has cosine ≥0.7 within the same method. Mean reappearance rises from 74.76% for Baseline to 79.97% for kNN. For features matched across seeds, Pearson correlation of activation values is 0.779/0.789 and Jaccard overlap of the positions where they fire is 0.527/0.483 (Baseline/kNN).
To assess semantic similarity, we randomly sampled 100 accepted cross-seed pairs per method. Qwen3-32B independently described each latent using the explanation prompt from SAEBench, then compared the descriptions with the exact Appendix C prompt from [2]. Its scale is 1 (incomparable), 2 (semi-comparable/neutral), 3 (comparable). Each pair was evaluated in both orders:
| Method | Mean comparability | Frequency-matched random controls |
|---|---|---|
| Baseline | 2.66/3 | 1.40/3 |
| kNN | 2.60/3 | 1.32/3 |
For each seed, Hungarian matching assigns each Baseline latent a unique kNN partner, maximizing total decoder cosine. 78.25% of pairs have cosine ≥0.7 (mean 0.781). In each method, stable features match in ≥14 of 15 other runs, unstable features in ≤1. Cross-method cosine is below 0.7 for 82.09% of unstable versus 5.22% of stable Baseline features, and 76.59% versus 8.01% for kNN.
Two examples illustrate these comparisons.
1. Gratitude: a shift in selectivity that repeats across runs. The seed-42 anchor matches seeds 43/44 within each method. Full independent Qwen descriptions are:
| Seed, latent | Baseline | kNN |
|---|---|---|
| 42, 11514 | expressions of gratitude and references to God or divine figures. | expressions of gratitude and acknowledgment, especially phrases like “thank you to the” or “thanks to them.” |
| 43, 2144 | expressions of gratitude and religious or spiritual references. | expressions of gratitude or thanks, especially phrases like “thank you” and related sentiments. |
| 44, 1885 | expressions of gratitude and phrases involving “thank” or “thanks,” often in a religious or formal context. | expressions of gratitude and appreciation, especially phrases like “thank you” or “thanks to.” |
All three Baseline runs peak on “…Glory Be [to] the Father…, in honor of Saint Anthony…”. All three kNN runs peak on “…We extend our [gratitude] to our departing colleagues for their important contribution…”. Moving from Baseline to kNN on fixed texts, all three seeds reduce their response to “Christ Jesus” while increasing responses to explicit gratitude and “glorify God”: religious praise remains, while the emphasis shifts towards explicit appreciation.
All six latents have decoder matches in all 15 other seeds. Within-method anchor-to-target cosines are 0.983–0.988 for Baseline and 0.965–0.967 for kNN. Same-seed cross-method cosines are 0.866–0.897.
2. Football/soccer: increased reappearance accompanied by stronger activation agreement. The seed-49 anchor changes from a broad acronym-related description to football/soccer-related terms, with decoder matches increasing from none to all 15 other seeds. Full independent Qwen descriptions are:
| Seed, latent | Baseline | kNN |
|---|---|---|
| 49, 12062 | abbreviations or acronyms that follow the pattern of two or more capitalized letters, often related to organizations or entities. | words related to soccer and variations of team names like “FC” or “MLS”. |
| 50, 8492 | abbreviations like “FC” and “Soccer” related to sports teams or clubs. | words related to football/soccer and club-related terms. |
| 51, 4395 | words related to soccer and football clubs, especially abbreviations like “FC.” | the substring “FC” and related terms in the context of sports teams and clubs. |
In kNN, seed 49 peaks on “…quit the [Soccer]oos later this year. ‘I have one kid…’”. Seeds 50 and 51 both peak on “…partnership with [FC] Barcelona… sponsoring the club…”. For seed 49 on fixed texts, moving from Baseline to kNN reduces activation on real-estate “MLS” (20.42→16.91) and increases it on “Upward Soccer” (15.80→26.49).
For the two targets, within-method cosines rise from 0.685–0.695 in Baseline (below threshold) to 0.925–0.943 in kNN, and Pearson from 0.178–0.199 to 0.632–0.684. Same-seed cross-method cosines are 0.707–0.927. Football/soccer responses already occur in Baseline, and acronym sensitivity remains in kNN. Increased reappearance does not mean a previously absent concept was created.
For details, see Appendix G (Tables 13–17) and Section 5.3.
Q4. Generalization to vision and other SAE architectures
Could the proposed regularizer be applied to vision SAEs, and would it be expected to work with other SAE architectures, such as ReLU, Matryoshka, or MP-SAEs?
We tested ReLU, Matryoshka and MP-SAEs against their own baselines without the kNN regularizer (λ=0). All models use GPT-2 layer 7, 16,384 features and 1B training tokens. ReLU uses fixed L1 α=0.0024. Matryoshka uses per-token TopK-64, reconstruction prefixes 4,096/8,192/16,384 and kNN loss on the full code. MP-SAE uses 64 pursuit steps.
| SAE | λ | EV (%) | Recall@16 (%) | Reappearance (%) | Attack R/S (%) | SP-1 | SCR@20 | TPP@20 |
|---|---|---|---|---|---|---|---|---|
| ReLU | 0 | 83.35 | 41.38 | 94.39 | 60.85/50.40 | 68.18 | 0.1050 | 0.1730 |
| ReLU | 8 | 83.44 | 51.88 | 92.32 | 63.62/54.06 | 67.72 | 0.1484 | 0.0582 |
| ReLU | 32 | 83.58 | 60.41 | 84.84 | 75.17/67.11 | 70.74 | 0.1850 | 0.1254 |
| Matryoshka | 0 | 88.31 | 62.79 | 57.25 | 72.41/64.27 | 68.03 | 0.3119 | 0.1182 |
| Matryoshka | 8 | 87.15 | 76.55 | 66.70 | 75.15/64.75 | 70.37 | 0.3124 | 0.1877 |
| Matryoshka | 32 | 85.22 | 78.83 | 57.92 | 75.88/65.63 | 72.49 | 0.3590 | 0.2786 |
| MP-SAE | 0 | 92.19 | 39.18 | 74.39 | 19.26/17.28 | 63.56 | 0.2672 | 0.0761 |
| MP-SAE | 8 | 91.08 | 33.87 | 71.38 | 18.73/16.87 | 67.00 | 0.2487 | 0.0622 |
| MP-SAE | 32 | 89.62 | 52.08 | 29.56 | 19.71/18.27 | 64.25 | 0.2374 | 0.0832 |
SAEBench averages three training seeds per setting.
Matryoshka gives the clearest positive answer to this question. At λ=8, better neighborhood preservation and more reproducible features come together with gains in sparse probing and TPP, at a modest reconstruction cost. ReLU's recall gains come with lower reappearance and mixed SAEBench results. MP-SAE shows a clearer limitation: its recall gain at λ=32 comes at a large cost to feature stability. The benefits extend beyond TopK, but they do not transfer equally to every architecture.
ReLU's mean L₀ rises from about 62 to 77 at λ=32, possibly because stronger kNN regularization favors denser codes while the L1 weight stays fixed.
Dead features at λ=0/8/32 (no activations on 200,000 held-out tokens): ReLU 0/89/829, Matryoshka 3/150/384, MP-SAE 0/0/0. Reappearance excludes dead features from both models and uses seven comparison runs. R/S denotes replacement/suffix attacks.
The ReLU, Matryoshka, and MP-SAE experiments are included in Appendix E (Table 12).
We also evaluated TopK SAEs on DINOv2-Base (ViT-B/14, layer 11). We used F=24,576, K=32 and 1B ImageNet-100 training patches. In ViSAEBench [3], sparse probing tests class prediction using a small set of SAE features, while spatial coherence measures clustering of feature activations on the patch grid. Results are mean ± std over four seeds.
| λ | Sparse probing AUC ↑ | Spatial coherence ↑ |
|---|---|---|
| 0 | 0.3177 ± 0.0117 | 0.2232 ± 0.0063 |
| 1 | 0.3002 ± 0.0189 | 0.2234 ± 0.0067 |
| 4 | 0.3080 ± 0.0210 | 0.2329 ± 0.0027 |
λ=4 improved spatial coherence in all four seeds, but did not improve mean sparse probing.
We thank the reviewer for suggesting this extension. Our paper focuses on LLMs, and we leave a systematic study of vision SAEs to future work.
Thank you again for your helpful comments. They led us to add more direct tests and explain our conclusions and limitations more clearly.
References
[1] Chen et al. Local Distance Preserving Auto-encoders using Continuous kNN Graphs. Topological, Algebraic, and Geometric Learning Workshops 2022.
[2] Balagansky et al. Mechanistic Permutability: Match Features Across Layers. ICLR 2025.
[3] ViSAEBench.
We appreciate the care you put into this review and your questions about the mechanism, interpretability and generalization of kNN SAE. They motivated the additional experiments and controls presented below.
Q1. Toy-model evidence for feature reproducibility
To investigate why preserving local distances can improve dictionary consistency across seeds, we compare TopK SAE with and without kNN regularization in a controlled toy model.
We use 100 input dimensions, 150 ground-truth (GT) directions, and 125 SAE latents with TopK (K = 4). Each input combines four uniformly sampled, distinct GT features with independent amplitudes between 0.5 and 1.5. Training uses 3,000 steps, batch size 512, kg=8, 64 anchors, and four neighbors per anchor, with geometry enabled at step 900 and a 300-step regularization ramp.
Means and standard deviations are computed over five GT dictionaries with 11 SAE initializations each. Reappearance uses cosine similarity of at least 0.7 between runs within the same GT dictionary.
The toy model reproduces the LLM trade-off: increasing lambda lowers EV, while reappearance first rises and then falls. At lambda 0.3, the more reproducible dictionary also provides closer matches to GT directions on average. Decoder → GT cosine decreases gradually with lambda: improved coverage of the GT dictionary can coexist with slightly weaker alignment of individual decoders to their nearest GT feature.
We then freeze each encoder and refit its unit-norm decoder and reconstruction bias using reconstruction loss on 65,536 fresh samples, with an independent 16,384-sample holdout.
The kNN codes themselves thus support more reproducible dictionaries with better average GT coverage at a small reconstruction cost.
Together, the results from joint SAE training from scratch and from decoder training with frozen encoders motivate a hypothesis about the codes: preserving neighbor distances may help retain useful distinctions while reducing arbitrary latent switching between similar inputs. If the relevant information remains accessible through small, selective groups of latents, this could benefit SAEBench probing and interventions (SCR/TPP), while more stable latent selection could help robustness to input perturbations.
These joint-training and frozen-encoder experiments are included in Section 5.3 (Tables 1–2).
Q2. Consistent benefits with a single regularization strength
A single fixed setting already provides simultaneous benefits across all three evaluation axes. For GPT-2 layer 7, λ = 8 improves all three reported SAEBench metrics averaged over three training runs relative to λ = 0 (Sparse Probing: 65.66% → 69.79%; SCR@20: 0.2586 → 0.3157; TPP@20: 0.0962 → 0.1661), mean cross-seed feature reappearance (74.76% → 79.97%), and retained feature overlap under both replacement and suffix attacks (67.63% → 73.80% and 56.20% → 62.69%, respectively). These gains occur with EV decreasing from 89.19% to 88.08%: a 1.11 percentage-point, or 1.24% relative, reduction. Thus, simultaneous improvements do not require selecting a different coefficient for each metric.
For Gemma-2-2B layer 7, λ = 8 also improves the three reported SAEBench metrics over λ = 0 (Sparse Probing Top-1: 75.88% → 76.65%, SCR@20: 0.3509 → 0.4152, TPP@20: 0.0639 → 0.0889) and replacement/suffix retained overlap (56.48/52.31% → 64.41/58.64%), with decoder reappearance increasing across the low-λ settings to 88.36% at λ = 8. The relative EV reductions from λ = 0 are 1.24% for GPT-2 and 1.90% for Gemma-2-2B (83.71% → 82.12%).
As expected, EV decreases with increasing λ, while reproducibility initially improves and then declines under stronger regularization. We suggest using the acceptable drop in validation EV relative to the unregularized SAE to guide the choice of λ, then keeping that value fixed across downstream evaluations. A related consideration appears in Chen et al. (2022) [1], who use a reconstruction-accuracy constraint when training CkNN autoencoders.