- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes a method for maintain a sparse autoencoder's (SAE) vocabulary across retraining, including across distributional shifts in its inputs. This work is motivated by the need for a shared vocabulary when SAEs are retrained due to shifts in the LLM input datasets (which is evaluated on a shift from wiki text to code), and when the LLM changes, for example during fine-tuning. Given a parent SAE and the goal of training a child SAE on a different data distribution, LineageSAE first classifies latents as inherited or released using a correct marginal-utility criterion, before retraining the released latents in the child SAE using a parent-child decoder handoff mechanism. To maintain retrained latent ordering between children, a temporary response-ordering mechanism is used. The method is evaluated on two language model distribution shifts, and on toy models where known dataset features are removed, modified, and added.
Critical strengths
The toy model and SynthSAEBench model interesting distributional shifts, where features are replaced, or have their activation frequencies changed. On these synthetic tasks, LineageSAE outperform baselines and other methods for maintaining a shared feature vocabulary.
Critical weaknesses
Confusingly, the paper seems to present results from two distinct methods as equivalent in the main body results tables. The method described in the main body applies the response ordering approach to all released addresses, whereas the residual-learning method described in the appendix only applies response ordering to zero-response addresses. However, this alternative method is not restricted to the appendix - Tables 1 and 2 in the main body differ in which method they use. Table 2 and 28 present very different results as a consequence of this method split on SynthSAEBench. I don't think I can currently determine whether each method outperforms baselines on both synthetic and LLM tasks, or whether this is only the case when choosing the best of the variants to display in the main body.
An additional basic control that needs including is using the parent SAE unchanged on the shifted distribution. My suspicion is that on the new-domain and updated-model experiments, the distribution shift is not so large that the parent SAE is invalidated - see results from Gemmascope on evaluating base model SAEs on fine-tuned instruct models. Table 25 has related (but not identical) baselines that empirically support this concern: parent-copy continuation dominates LineageSAE across similarity and reconstruction metrics, and even the frozen decoder baseline is competitive with much better similarity scores and only reducing CE recovery from 95.80% to 95.69% - I'm not sure whether that change is significant as only point estimates are provided.
Table 4 demonstrates better detection but not end-to-end utility for correct utility over uncorrected utility. Corrected utility is much better at detecting deleted-feature addresses, however uncorrected-utility performs better on every address-coordination metric, and all variants recover the 51 new features. The end-to-end benefits of the corrected utility are therefore not clear to me.
If LineageSAE is finding important new features in the LLM distributional shift experiments, case studies and qualitative examples of such would be very useful, otherwise I can't assess whether this method is actually useful for interpretability practitioners.
Questions for authors
- Clarify which configuration of the method is used for all the main body tables, and provide results for each variant on all the tables (or commit to a single variant).
- How does the basic parent-copy control I propose compare with the results present in tables 1 and 3?
- What is the end-to-end benefits of corrected utility?
3 comments
Thanks for the additional results, they represent substantial progress. Residual concerns:
- I think Figure 2 still uses the residual learning method: E.3 uses the A.5 config for Table 25 results, which Fig 2 summarise.
- The reported 74.9% SynthSAEBench rare-feature recovery result in the main paper doesn't seem to match the reported results in Table 17.
- I'm still not sure why the corrected utility is preferred to the uncorrected utility. The differences in some of those metrics are small and uncertainty estimates aren't provided.
- I'd like a more detailed example of a new feature that walks me through the parent address, reason for release, child feature examples, and a comparison with what ordinary continuation learns.
I'll maintain my score for now as I'm confused by the inconsistencies between the appendix and main body, but will bump it up if those can be addressed.
Thank you for reading the revised paper so closely. We updated Figure 2, Appendix E.3, and Appendix Table 17 in the revised PDF, and address all four points below.
1. Figure 2 and Appendix E.3 / Table 25 (four-generation Wiki trajectory)
In the last upload, Tables 1-5 and Figure 3 used the Section 2 method, whereas Figure 2 and Appendix E.3 (Tables 25-27) still plotted the earlier Appendix A.5 run. We updated Figure 2, Section 3.2, and Appendix E.3 to the Section 2 trajectory (3 chains, 12 children, zero post-hoc assignment; the A.5 run is kept alongside Table 28 controls):
| Gen | Adj. J / Cos | Root J / Cos | C-C J / Cos | NMSE | CE | Rel. |
|---|---|---|---|---|---|---|
| G1 | 0.6590 / 0.9683 | 0.6590 / 0.9683 | 0.8474 / 0.9976 | 0.01539 | 95.83% | 46.0 |
| G2 | 0.7479 / 0.9868 | 0.5686 / 0.9413 | 0.8092 / 0.9942 | 0.01539 | 95.83% | 55.7 |
| G3 | 0.7728 / 0.9918 | 0.5199 / 0.9220 | 0.7821 / 0.9910 | 0.01543 | 95.82% | 52.0 |
| G4 | 0.7411 / 0.9924 | 0.4790 / 0.9057 | 0.7175 / 0.9872 | 0.01553 | 95.74% | 52.7 |
Local parent-child J stays high (0.6590 to 0.7411) while root-relative J drifts (0.6590 to 0.4790) at 95.74%-95.83% CE recovery.
2. Match between 74.9% SynthSAEBench rare recovery and Appendix Table 17
Appendix Table 17 still held the pre-revision counts (0/153, 89/153, 153/153 = 242/459 = 52.7%). Updated to the Table 2 and Figure 3 checkpoints, Table 17 matches the main paper:
- SynthSAEBench: Rare recovery across the three reduced-frequency conditions is 54/153, 137/153, and 153/153, totaling 344/459 = 74.9% (Figure 3), with 100% new and ordinary recovery (918/918 and 7839/7839) and mean NMSE 0.111 (Table 2).
- Toy (roots 101-103): Rare, new, and ordinary recovery are 54/72 (75.0%), 65/72 (90.3%), and 876/936 (93.6%), with mean NMSE 0.097 (Table 2).
3. Why corrected utility is preferred and uncertainty estimates (Table 5, Appendix D.4, Table 44)
We agree that NMSE (0.1075 vs. 0.1077) and rare counts (125/204 vs. 120/204; diff +2.45%, 95% CI [-1.96%, +7.35%]) are close, and uncorrected covariance gives higher raw J (0.885 vs. 0.866). Corrected utility is preferred for two reasons (20k-draw paired feature bootstrap 95% CIs in Appendix D.4):
- Retiring dead features and cutting false positives: Because surviving features correlate with deleted directions, uncorrected covariance releases only 37/153 (24.2%) deleted features, vs. 130/153 (85.0%, +60.8 pp; 5/5 vs. 0/5 correlated deletions in Fig. 4a) for corrected utility. Leaving 116/153 dead features marked as inherited keeps monitors on obsolete slots and raises false firing on deleted slots (-0.155 pp, 95% CI [-0.255, -0.061] pp). Across the 102 rare survivors, corrected utility also raises TPR by +4.27 pp (86.63% vs. 82.37%, 95% CI [+1.02, +7.98] pp) and halves FPR (-0.110 pp, 0.113% vs. 0.223%, 95% CI [-0.211, -0.022] pp).
- Saturated LLM dictionaries: SynthSAEBench (512 features in 1,024 slots) leaves about 500 unused slots even if deleted slots stay frozen. On Pythia Wiki-to-Code (N = 4096), 99.4% (4,073/4,096) of parent slots fire on Wiki, and negative corrected utility frees 538 firing-but-obsolete Wiki slots (13.1% of width) plus 1,433 zero-firing slots.
4. Walkthrough of a newly learned LLM feature vs. ordinary continuation (Appendix H.3)
In the 1,000-step Pythia CodeSearchNet update (where yield statements are removed and comprehensions enter at 20%), we trace native slot 1768 across both child seeds:
- Parent slot and release reason: At step 0, parent slot 1768 had no comprehension role (AUPRC 0.243 vs. 0.225 prevalence) and had zero positive TopK activations across all 32,768 step-1k scoring tokens, so Eq. 4 releases slot 1768 in both seeds.
- Child feature at slot 1768: Guided by pre-ReLU ordering (Eq. 6), both LineageSAE seeds converge at slot 1768 to a detector for the token "for" in Python comprehensions, such as "[Asset(o) for o in resp]" (12.27 and 11.94 in seeds 1 and 2) and "[(rps, len(list(rpl))) for rps, rpl in rps_groups]" (11.42 and 12.41; C-C steering cosine 0.838). Across 569 held-out files, slot 1768 fires on comprehension "for" tokens but never on ordinary for-loops:
| Model at Slot 1768 | AUPRC | Comp. "for" (all / 50 shared) | Loop "for" |
|---|---|---|---|
| Parent (0 / 1k) | 0.243 / 0.234 | 3/188 / 1/72 (1k: 2/188) | 0/262 |
| LineageSAE Sec. 2 (s1 / s2) | 0.822 / 0.829 | 135/188, 136/188 (47/72, 48/72) | 0/262 |
| Stage-Wise (s1 / s2) | 0.225 / 0.225 | 0/188, 0/188 (0/72, 0/72) | 0/262 |
- Comparison with Stage-Wise: Without slot release, slot 1768 stays dead in both Stage-Wise seeds (0/188 firings, AUPRC 0.225), along with 38.9% of the dictionary (C-C J 0.376 vs. 0.716). Instead of reusing dead slots, Stage-Wise adapts active parent slots (e.g., slot 1597 shifts from parent AUPRC 0.331 to 0.940/0.942), which also overwrites suppressed with/async-with features (35.1%-43.4% silent vs. 4.9%-6.9% in LineageSAE, Table 42).
Response to Reviewer AfR2 (R3)
Thank you for cross-checking the main text against the appendix. Below we answer W1-W4 and Q1-Q3 directly.
1. Method clarification and unified main-body commitment (W1, Q1)
The table mismatch in our initial submission was our mistake. Initial Tables 2 and 4 ran Section 2 (Sections 2.1-2.5: corrected utility release in Eqs. 3-4, parent-guided decoder handoff in Eq. 5, and pre-activation response ordering on all released addresses in Eq. 6, zero post-training permutation), whereas initial Tables 1, 3, 25, and 28 mistakenly reported an early Appendix A.5-A.8 prototype (residual learning + zero-response ordering + post-hoc training-data assignment).
In the revision, all main-body tables (Tables 1-5) commit to the single Section 2 native method (r = 0 on single-step shifts, zero post-hoc assignment), while Appendix A.5-A.8 and Appendix F.2 (Tables 30-31) keep the residual prototype as a labeled ablation. Answering Q1, both variants independently outperform all 13 baselines across synthetic and LLM tasks:
2. Unchanged parent SAE control, Parent-copy continuation, and variance (W2, Q2)
Answering Q2 on Tables 1 and 3: initial Table 25 (Appendix Table 28) retrained on fixed static Wiki data with zero shift. Under shifts, LineageSAE beats Stage-Wise on Table 1 Wiki-to-Code (C-C J: 0.4766 vs. 0.1928; C-C cosine: 0.6905 vs. 0.3453) and Table 3 Pythia update (C-C J: 0.8013 vs. 0.6872; NMSE: 0.0154 vs. 0.0162; CE: 95.13% vs. 94.60%, a +0.53 pp gain vs. +/-0.01% 3-seed SD in Table 23), and cuts causal logit-shift error by 42% (Table 4, Figure 5: 0.1606 vs. 0.2782; silent 1.55% vs. 7.01%).
For the zero-shot unchanged parent control: on Wiki-to-Code (Section 3.1, Appendix H.2), zero-shot CE recovery on Code drops from 97.99% to 43.04%, as 1,433 / 4,096 (35.0%) Wiki features never fire on Code and 538 (13.1%) have negative marginal utility. On model updates, consistent with your GemmaScope point, a mild 300-step Pythia update induces a moderate shift (frozen parent old-vs-new J = 0.8318, steering cosine 0.9424), so Eq. 4 releases only 48 / 4,096 (1.2%) slots (Table 21). On larger shifts like Pythia 100k-to-143k (Table 43), unchanged-parent NMSE degrades 2.46x (0.0206 to 0.0507), and on step-900 unseen Wiki articles adaptation lowers NMSE from 0.03584 to 0.03390.
3. End-to-end benefits of corrected marginal utility (W3, Q3; revised Table 5, Appendix Table 44)
Uncorrected covariance had higher raw J (0.885 vs. 0.866) in Table 5 because it misses 116 / 153 deletions (37 / 153 vs. 130 / 153) and leaves dead slots frozen as inherited, causing correlated false-positive activations on neighbors that inflate raw overlap.
Corrected utility gives three end-to-end gains (Section 3.3, Table 5; Appendix H.4, Table 44): (1) higher active-set TPR (82.37% to 86.63%) with half the false-positive firing rate FPR (0.223% to 0.113%), better rare recovery (125 / 204 vs. 120 / 204), and lower NMSE (0.1075 vs. 0.1077); (2) an accurate retirement mask (85.0% vs. 24.2% obsolete detection); and (3) under matched-count controls (Table 44), TopK activation count falsely deletes 64 / 102 (62.7%) rare survivors vs. 3 / 102 (2.9%) for corrected utility (mean TopK amplitude catches only 26 / 153 deletions).
4. Qualitative case studies on LLM shifts (W4; Appendix H.1 Figure 9, Appendix H.3 Table 42)
We added steering transcripts (Appendix H.1, Figure 9f-g) and Python AST tracking (Appendix H.3, Table 42). On updated Pythia-1.4B, removing feature 1188 (QPR vs. Birmingham) drops clean P(" of") from 67.33% to 1.29% (parent) and 1.45% / 1.46% in both LineageSAE children at native index 1188 with identical 32-token continuations (native Stable Pool stays at 67.33%). Removing feature 3182 (St Ceidio's) drops P(" the") from 25.09% to 3.91% (parent) and 4.07% / 4.25% (LineageSAE), vs. 16.65% / 17.10% (Stage-Wise) and 20.47% / 20.51% (matched Stable Pool). On Python updates (Table 42), inherited "with" / "async with" features retain 0.766 / 0.868 causal cosine (6.94% silent vs. 43.40% Stage-Wise), while released slots learn new comprehension detectors at 0.680 AUPRC (+0.266 over parent).