acceptodds
Under review as a conference paper at ICLR 2027

The Model Already Has a Dictionary: Cumulative SVD Features as a Training-Free Alternative to Sparse Autoencoders

19 views

Abstract

Sparse autoencoders (SAEs) have become the prevailing paradigm for learning interpretable feature dictionaries from neural activations, but they require large activation corpora, introduce reconstruction errors, and vary across training seeds; consequently, public SAE releases remain largely confined to a few model families and activation sites. We ask how much of this feature dictionary can instead be recovered directly from the model's intrinsic weights. For each Transformer residual stream state, we construct an inherently overcomplete, model-intrinsic dictionary in a training-free manner by unifying two principles: accumulating historical module outputs across all upstream layers, and applying singular value decomposition (SVD) to attention head OV circuits and MLP down projections to extract compact, intra-module orthogonal read and write channels. The resulting dictionary reconstructs the accumulative residual activations exactly without training corpora or optimization. Across Gemma 3 (1B-27B) and Qwen3-8B, SAE features remain slightly easier to describe from activating contexts, while SVD attains full coarse topic coverage and captures complementary fine-grained semantics. Moreover, cumulative SVD features excel at sparse concept prediction and enable more selective attribute disentanglement alongside stronger factual relation steering under multi-feature composition. Extraction uses no corpus tokens and is about seven orders of magnitude more efficient than training a matched SAE pair. These findings position weight-derived feature dictionaries as a strong training-free baseline and a complementary source of concepts for SAE-based interpretability.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, sparse autoencoders, singular value decomposition, feature dictionaries, Transformer interpretability, residual stream, causal interventions

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wj9AOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

By applying the SVD to the model, This paper builds a training-free feature, dictionary from model weights.

Critical strengths

This paper shows a simple and well-multiple idea. It provides a cheap deterministic baseline which is valuable for the field. It also provided a useful application study for different comparisons.

Critical weaknesses

However, I think the comparison is structurally unfair to SAEs and the central claim is confounded. The dictionary size and selection give SVD an advantage in several benchmarks.

Questions for authors

How did you do the SVD features compared against SAE's training on the same cumulative module output? This is the cleanest test of whether SVD itself matters.

7 comments
AuthorsOpenReview ↗

Response to Reviewer wj9A (1/4)

We sincerely appreciate the reviewer’s time, thorough review, and very insightful feedback on our work. Below We detail our response to the weaknesses (i.e., W(1)) and questions (i.e., Q(1)) point by point, along with revisions to the manuscript, which are highlighted in blue for your convenience. Please kindly let us know if our response addresses the issues you raised in this paper.

Controlling for Candidate Dictionary Size, Selection Protocols, and Upstream Access

(W1) However, I think the comparison is structurally unfair to SAEs and the central claim is confounded. The dictionary size and selection give SVD an advantage in several benchmarks.

We sincerely appreciate the reviewer's careful scrutiny of our evaluation fairness. Most fairness controls were already reported in Sections 3.2–5.2 and Appendices C–G of the submitted manuscript. We have consolidated them and added further checks in Appendix B.4 of the revised manuscript to make our comparisons clearer and more rigorous.

We explicitly controlled for all three potential confounders: (1) dictionary size, (2) selection protocols, and (3) upstream module accumulation structure. Across all three dimensions, Cumulative SVD consistently maintains its performance advantage. This confirms our findings reflect intrinsic properties of the SVD basis rather than artifacts of unfair setups.

  • Controlling for dictionary size and width:

    • Residual SAE holds 1.2×–15.1× larger dictionary at most evaluation sites: Crucially, across all layers of Gemma 3 1B, 4B, and Qwen 3 8B, Cumulative SVD's dictionary size never exceeds the 262k Residual SAE pool. As noted in Lines 293–296 of the submitted manuscript and Table R1, despite fewer candidates, Cumulative SVD consistently outperforms Residual SAE across semantic coverage with ΔG16>0\Delta G_{16} > 0, sparse probing with AUC 0.866 vs. 0.819 at k=5k=5, RAVEL disentanglement 0.678 vs. 0.559, and relation steering gaining +10.1–18.6 pp.
    • Exhaustive width sweeps favor SAE and confirm non-monotonicity: In Figures 10, 13a, and 14 of the submitted manuscript, we evaluated nine Gemma Scope 2 configurations spanning widths 16k, 65k, and 262k across three sparsity levels, selecting the optimal width per task, namely 65k for probing and RAVEL, and 262k for coverage and steering. Consistent with SAEBench [1], increasing width does not monotonically improve performance, as on 4B layer 22 RAVEL disentanglement drops from 0.580 to 0.560 at width 262k. Cumulative SVD outperforms all nine SAE configurations.
    • Local One Attn+MLP SVD with a 57× smaller size still wins: Restricting SVD to single-block local writes yields a candidate pool 57× smaller than the 262k Residual SAE, yet One Attn+MLP SVD still beats 262k SAE in Gemma 4B relation steering, reaching 62.91% vs. 61.50% Acc@1 in Table 19 of the submitted manuscript.
    • Matched Cumulative SAE gives SAE 7.1×–15.1× larger dictionary: In response to Q1, we constructed Matched Cumulative SAE by pooling pretrained Gemma Scope 2 attention and MLP SAEs across blocks 0…l0 \dots l, as detailed in Section 2 and Appendix B.3 of the revised manuscript. Even with 7.1×–15.1× larger candidate pools under matched accumulation, Cumulative SVD consistently outperforms or matches Cumulative SAE across all five benchmarks in Sections 3.2–5.2.

    Table R1 -- SVD vs. Residual SAE across pool sizes and width sweeps in the revised manuscript

    Benchmark & Setting (Paper Ref.)Baseline (Pool Ratio vs. SVD)Baseline ScoreSVD Score
    Coverage ΔG16\Delta G_{16}: 1B / 4B, 7 sites; §3.2; App. D.3, Tab. 13262k Resid. SAE (1.2×–7.5×)Ref.+0.062 / +0.043
    Probing AUC: 1B 4 sites, k=5/128k=5 / 128; §4; App. E.265k Resid. SAE (1.3×–3.8×)0.819 / 0.8740.866 / 0.903 (wins 5/5 sites)
    Probing SAE Width Sweep: 4B L22, k=1/2/5k=1 / 2 / 5; App. E.3, Fig. 14a9 SAE configs (65k: 0.6×; 262k: 2.5×)0.792 / 0.826 / 0.8490.833 / 0.861 / 0.888
    RAVEL Disentangle: 1B 4 sites / 4B L9; §5.1; App. F.2, Tab. 2265k Resid. SAE (1.3×–3.8×)0.559 / 0.7270.678 / 0.816 (wins 5/5 sites)
    RAVEL SAE Width Sweep: 4B L22 Disentangle; App. F.5, Fig. 159 SAE configs (65k: 0.6×; 262k: 2.5×)0.580 (262k: 0.560)0.785
    Relation Steering: 1B / 4B Acc@1; §5.2; App. G.2, Tab. 28262k Resid. SAE (1.9×–15.1×)47.0%–63.9%60.5%–74.1% (+10.1–18.6 pp)
    Relation Steering SAE Width Sweep: 4B Acc@1; §5.2; App. B.4; App. G.3, Tab. 3016k (0.1×–0.4×) & 262k (1.9×–5.7×) SAE61.50% (16k: 59.10%)71.65%
    Relation Steering (Local SVD): 4B, Acc@1 / Acc@5; §5.2; App. G.3, Tab. 30262k Resid. SAE (57.0× vs. local SVD)61.50% / 77.40%62.91% / 81.93% (One Attn+MLP SVD)
AuthorsOpenReview ↗

Response to Reviewer wj9A (2/4)

Continued from our response to W1 in the preceding comment (1/4).

  • Controlling for feature selection protocols:

    • Strictly symmetric evaluation budgets and search algorithms: In Appendices D.1, E.1, F.1, and G.1 of the submitted manuscript, both dictionaries use identical budgets B∈16,32,64B \in \\{16, 32, 64\\} in coverage and identical k∈1,2,5,16,128k \in \\{1, 2, 5, 16, 128\\} across 113 sparse probing tasks on training splits. Both also train differentiable masks under identical schedules in RAVEL, and run identical Integrated Gradients ranking alongside a two-stage grid search spanning 420 coarse and 2,400 refine configurations over layer, kk, and scale α\alpha in relation steering.
    • Selecting more features confers no automatic metric advantage: In RAVEL, activating unneeded features causes collateral damage to non-target attributes, degrading Isolation\mathrm{Isolation} and lowering composite disentanglement D=(Cause+Isolation)/2D = (\mathrm{Cause} + \mathrm{Isolation})/2. In relation steering, the composed steering vector is normalized to unit norm before scaling by α∣hL∣2\alpha \\|h_L\\|_2 as formulated in Eq. 19 of the submitted manuscript; adding features refines steering direction without inflating steering magnitude.
  • Controlling for upstream module accumulation structure:

    • Cumulative SVD outperforms larger cumulative baselines under identical upstream access: Under identical depth accumulation across blocks 0…l0 \dots l, Cumulative SVD outperforms Cumulative Module Neurons with 1.1× larger size and Cumulative Weight Columns with 2.7× larger size, scoring 0.748 vs. 0.719 on RAVEL disentanglement and achieving 71.65% / 88.29% vs. 71.63% / 83.83% and 70.37% / 86.42% in relation steering, as reported in Figures 6b and 7c, Tables 15 and 19 of the submitted manuscript.
    • Matched Cumulative SAE confirms SVD advantage under identical accumulation: As introduced above, Cumulative SAE pools upstream attention and MLP SAEs to match Cumulative SVD's upstream accumulation, as shown in Table R2 and Sections 3.2–5.2 of the revised manuscript. Cumulative SVD consistently outperforms or matches Cumulative SAE across semantic coverage, concept complementarity, sparse probing, RAVEL disentanglement, and relation steering.
    • Local SVD beats Residual SAE without accumulation: Single-block One Attn+MLP SVD outperforming Residual SAE in relation steering in Table R1 confirms that SVD does not rely on depth accumulation to surpass residual-stream SAEs.

    Table R2 -- Cumulative SVD vs. baselines under matched upstream accumulation in the revised manuscript

    Benchmark & Setting (Paper Ref.)Baseline (Pool Ratio vs. SVD)Baseline ScoreSVD Score
    RAVEL Disentangle: 4B, 4-layer mean; §5.1; App. F.2, Tab. 23Matched Cumul. SAE (7.1×)0.6820.748
    RAVEL Disentangle: 4B, 4-layer mean; §5.1; App. F.3, Tab. 24Cumul. Module Neurons (1.1×)0.7190.748
    Relation Steering: 4B Pub, Acc@1 / Acc@5; §5.2; App. G.2, Tab. 29Matched Cumul. SAE (7.1×)70.10% / 85.04%71.65% / 88.29%
    Relation Steering: 4B, Acc@1 / Acc@5; §5.2; App. G.3, Tab. 30Cumul. Neurons (1.1×) | Weight Cols (2.7×)71.63/83.83% | 70.37/86.42%71.65% / 88.29%
  • Upstream accumulation reflects an intrinsic structural advantage of SVD rather than an evaluation artifact: Leveraging upstream module writes is an inherent structural property of SVD. Because each module's singular basis is closed-form and rank-bounded, SVD naturally spans residual accumulation across depth with zero training and lower inference cost. In contrast, stacking cumulative SAEs across depth is computationally prohibitive, causing parameter explosion along with 6×–12× higher latency and peak VRAM, as documented in Section 6 and Appendix H.3, Table 39 of the revised manuscript. Because single-layer Residual SAE remains the practical mainstream in mechanistic interpretability, evaluating Cumulative SVD against it reflects a realistic operational baseline, and SVD's ability to seamlessly leverage upstream accumulation is an intrinsic structural advantage of the formulation.

[1] Karvonen et al. SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability, ICML 2025.

AuthorsOpenReview ↗

Response to Reviewer wj9A (3/4)

Confirming SVD’s Advantages over SAE Under Matched Cumulative Information Access

(Q1) How did you do the SVD features compared against SAE's training on the same cumulative module output? This is the cleanest test of whether SVD itself matters.

We sincerely thank the reviewer for proposing this clean and decisive test, which directly motivated our new Matched Cumulative SAE baseline across all five evaluations and made the central claim of our work substantially more solid. Following the reviewer's suggestion, we constructed Matched Cumulative SAE on Gemma 3 1B and 4B by pooling upstream attention and MLP SAEs across identical blocks 0…l0 \dots l, and evaluated it across all five experimental sections and runtime scalability in §§3.2–5.2 and §6 of the revised manuscript. We found that Cumulative SVD consistently outperforms or matches Cumulative SAE across semantic coverage, concept complementarity, sparse probing, RAVEL disentanglement, and relation steering, while avoiding over an order of magnitude higher extra latency and extra peak VRAM from Cumulative SAE

By equalizing upstream access while giving SAE a 7.1×–15.1× larger candidate pool, this Cumulative SAE baseline strictly controls for dictionary size and upstream module accumulation. Under these controlled conditions, Cumulative SVD's advantages are intrinsically grounded in the effectiveness of SVD representation properties themselves, rather than being driven by dictionary size or upstream module accumulation.

  • Constructing Matched Cumulative SAE with equal upstream module accumulation and 7.1×–15.1× larger dictionary size: Because the residual stream after Transformer block ll accumulates additively as hl=e+∑i=0l(Δxattn(i)+Δxmlp(i)),h_l = e + \sum_{i=0}^l \left(\Delta x_{\mathrm{attn}}^{(i)} + \Delta x_{\mathrm{mlp}}^{(i)}\right), where hl∈Rdh_l \in \mathbb{R}^d is the residual state, e∈Rde \in \mathbb{R}^d is the token embedding, and Δxattn(i),Δxmlp(i)∈Rd\Delta x_{\mathrm{attn}}^{(i)}, \Delta x_{\mathrm{mlp}}^{(i)} \in \mathbb{R}^d are the attention and MLP writes from block ii, we pool pretrained Gemma Scope 2 attention- and MLP-output SAEs of width 16,384 across all upstream blocks i=0…li = 0 \dots l as described in Section 2 and Appendix B.3 of the revised manuscript. This preserves upstream module accumulation and yields 32,768 SAE features per block compared to 2,176 SVD features on Gemma 3 1B and 4,608 on Gemma 3 4B, making the Cumulative SAE pool 15.1× larger on 1B and 7.1× larger on 4B.

  • Semantic Coverage in §3.2: As shown in Figures 2b–c of the revised manuscript and summarized in Table R3, Cumulative SVD achieves higher pure-dictionary semantic coverage GG across all 24 layer-by-budget settings.

    Table R3 -- Semantic coverage density GG: Cumulative SVD / Cumulative SAE; Appendix D.2, Tables 6–7 of the revised manuscript

    Budget BBGemma 3 1B (4-layer mean)Gemma 3 4B (4-layer mean)
    160.212 / 0.139 (+52.9%)0.205 / 0.138 (+48.5%)
    320.341 / 0.269 (+26.8%)0.335 / 0.269 (+24.8%)
    640.539 / 0.521 (+3.4%)0.551 / 0.520 (+6.0%)
  • Topic Preference and Concept Specificity in §3.3: In Section 3.3 and Appendix D.3, Tables 8 and 17 of the revised manuscript, replacing Residual SAE with Cumulative SAE yields nearly identical 73-topic preferences compared with SVD across all layers with Spearman ρ∈[0.910,0.994]\rho \in [0.910, 0.994], while 47.5% of judged monosemantic SVD features on 1B and 49.6% on 4B still lack a semantic substitute among their top-3 nearest Cumulative SAE neighbors.

Reviewer_nhWVOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper constructs training-free feature dictionaries by applying SVD to attention OV and MLP down-projection matrices and accumulating their write directions across layers. Experiments on Gemma 3 and Qwen3 show complementary semantic coverage, generally stronger sparse concept prediction, and better attribute disentanglement and relation steering than the evaluated SAEs.

Critical strengths
  1. The method is simple and computationally inexpensive, requiring no auxiliary training to construct the dictionary.
  2. It integrates information across layers while preserving the source of each module’s contribution.
  3. It outperforms the evaluated SAEs on attribute disentanglement and factual relation steering.
  4. The evaluation covers multiple models, layers, and tasks, with relevant baselines and ablations.
Critical weaknesses
  1. Limited evidence for monosemanticity. Unipolar SVD uses post-hoc polarity selection, while raw SVD scores are weaker. The gains in Figures 2(b–c) could reflect broader semantic coverage OR greater polysemanticity of individual SVD features, and the paper does not adequately distinguish these explanations.
  2. Insufficient analysis of layer and component differences. The effects of layer depth and component type on SVD feature interpretability are not analyzed; Figures 2(a) and 9 only partially discuss these differences.
  3. Incomplete isolation of SVD’s contribution. The steering comparison gives cumulative SVD access to upstream module activations, whereas the SAE encodes only the target residual state. This makes it difficult to attribute the gains to SVD itself, and the single-layer SVD ablation is limited to Gemma 3 4B.
Questions for authors
  1. Appendix D.1 filters features by statistical significance of topic association, but does not establish their monosemanticity. Could the authors report monosemanticity scores for the features used in Figures 2(b–c), or clarify whether low-scoring features were excluded?
  2. Figure 9 suggests interesting variation across layers and components, but only a few layers are evaluated. I guess SVD does not inherently guarantee interpretable features at every layer, could the authors extend the analysis across more layers and examine differences between attention and MLP features?
  3. Table 4 shows that SVD and SAE candidate dictionary sizes differ substantially across models and layers. Could the authors compare cumulative SVD, single-layer SVD, and SAE steering across multiple models and layers? Given the substantial drop for single-layer SVD in the existing ablation, this would help distinguish gains from dictionary size, access to upstream module information, and the SVD directions themselves.
  4. The procedure described as “reranking the top 128 in full embedding space” on page 24 is unclear. Could the authors clarify how the initial 128 candidates are retrieved, which representations are used for reranking, and how the final three SAE features are selected?
10 comments
AuthorsOpenReview ↗

Response to Reviewer nhWV (1/5)

We sincerely appreciate the reviewer’s time, thorough review, and very insightful feedback on our work. Below We detail our response to the weaknesses (i.e., W(1) ~ W(3)) and questions (i.e., Q(1) ~ Q(4)) point by point, along with revisions to the manuscript, which are highlighted in blue for your convenience. Please kindly let us know if our response addresses the issues you raised in this paper.

Distinguishing Broad Semantic Coverage from Polysemanticity

(W1) Limited evidence for monosemanticity. Unipolar SVD uses post-hoc polarity selection, while raw SVD scores are weaker. The gains in Figures 2(b–c) could reflect broader semantic coverage OR greater polysemanticity of individual SVD features, and the paper does not adequately distinguish these explanations.
(Q1) Appendix D.1 filters features by statistical significance of topic association, but does not establish their monosemanticity. Could the authors report monosemanticity scores for the features used in Figures 2(b–c), or clarify whether low-scoring features were excluded?

We sincerely thank the reviewer for raising this crucial distinction between broad semantic coverage and polysemanticity, which makes Section 3.2 of the revised manuscript substantially more rigorous:

  1. Excluding low-monosemanticity features (Q1): We acknowledge that filtering solely by topic-association significance in Appendix D.1 of the submitted manuscript did not guarantee monosemanticity or exclude low-AutoInterp features. So we re-evaluated all 12 residual layer sites across Gemma 3 1B, 4B, and Qwen3 8B under a monosemanticity-constrained coverage protocol, where SVD's coverage gains persist with ~0.900.90 AutoInterp.

  2. Isolating broad coverage from polysemanticity (W1): To eliminate any remaining AutoInterp gap between admitted SVD and SAE features, we matched their exact held-out AutoInterp bins at each site. Under strictly identical AutoInterp score distributions, SVD features achieve higher average coverage across all 12 sites (12/1212/12, +11.50+11.50\\% overall gain), driven by coherent higher-level concepts rather than polysemanticity.

  • Strictly matching held-out AutoInterp score distributions confirms that SVD's coverage advantage is not driven by polysemanticity: Under our monosemanticity-constrained protocol (detailed below), admitted SAE features average slightly higher held-out AutoInterp (~0.940.94 vs. ~0.900.90), so we compare single-feature coverage after matching SVD and SAE features to the exact same held-out AutoInterp scores:

    • Exact bin-by-bin matching: At each site, we group features into exact held-out AutoInterp bins (0/400/40 to 40/4040/40 in 0.0250.025 steps). We weight each bin's coverage by min⁡(nSVD,nSAE)\min(n_{\text{SVD}},n_{\text{SAE}}), which is mathematically equivalent to comparing an equal number of SVD and SAE features at each exact score level—guaranteeing identical AutoInterp distributions.
    • Consistent gains at matched AutoInterp: As shown in Table R1, SVD achieves higher average coverage across the 73 topics at all 12 sites (12/1212/12) with a +11.50+11.50\\% overall gain, winning 80.880.8\\% (42/5242/52) of bins with ≥5\ge 5 features per dictionary.

    Table R1 -- Single-feature semantic coverage under matched held-out AutoInterp distributions

    Model (Residual Layers)SitesMatched CapacitySAE CoverageSVD CoverageRelative GainSVD > SAE Sites
    Gemma 3 1B (L7,13,17,22)41660.0083420.009844+18.00%4/4
    Gemma 3 4B (L9,17, 22, 29)41760.0084860.009494+11.87%4/4
    Qwen3 8B (L9, 18, 24, 31)41830.0082580.008633+4.54%4/4
    Overall125250.0083620.009323+11.50%12/12
  • Enforcing monosemanticity admission gates and verifying that SVD's broader coverage persists: On training fold A, candidate features are admitted only if an LLM explanation of their top-activating texts matches ≥80\ge 80\\% of validation activating texts and passes a specificity check against vague or multi-topic descriptions. As reported in Section 3.2, Figures 2(b–c) and Appendix D.2, Tables 6–7 of the revised manuscript, on test fold B, frozen SVD dictionaries improve pure coverage by up to +28.34+28.34\\% and SAE-core extensions by up to +15.89+15.89\\% across feature budgets at high AutoInterp (~0.900.90 for SVD vs. ~0.940.94 for SAE).

  • SVD's higher coverage stems from coherent broader concepts rather than polysemanticity: In case studies at matched 1.0001.000 (40/4040/40) AutoInterp in Appendix D.4, Table 19 of the revised manuscript, SAE often splits a domain into narrow sub-types (e.g., youth tournaments) whereas SVD captures a unified higher-level concept (e.g., general sports competitions); reverse cases also occur where SVD isolates a specific sub-concept while SAE spans a broader category as illustrated in Section 3.3 and Appendices D.3–D.4.

AuthorsOpenReview ↗

Response to Reviewer nhWV (2/5)

Analyzing SVD AutoInterp Patterns Across Models, Layers, and Modules

(W2) Insufficient analysis of layer and component differences. The effects of layer depth and component type on SVD feature interpretability are not analyzed; Figures 2(a) and 9 only partially discuss these differences.
(Q2) Figure 9 suggests interesting variation across layers and components, but only a few layers are evaluated. I guess SVD does not inherently guarantee interpretable features at every layer, could the authors extend the analysis across more layers and examine differences between attention and MLP features?

We thank the reviewer for this constructive suggestion, which motivated an all-layer, module-wise SVD AutoInterp evaluation in Section 3.1 of the revised manuscript. We agree that SVD does not guarantee uniformly interpretable features at every layer: evaluating across all 158 layers of Gemma 3 1B, 4B, 27B, and Qwen3 8B, SVD AutoInterp reveals clear drops and rebounds across layers for both attention and MLP features, with their count increasing with model scale, which we hypothesize reflects transitions across LLM inference stages; see Appendix C.3, Figure 10 of the revised manuscript for detailed curves.

  • Attention interpretability drops gradually in smaller models but rebounds after a mid-layer drop in larger models: As shown in Table R2 and Figure 10, attention AutoInterp accuracy drops steadily in Gemma 3 1B and 4B, whereas in Qwen3 8B and Gemma 3 27B it drops at middle layers before rebounding by 3.40–5.033.40\text{--}5.03 percentage points.

    Table R2 -- Attention SVD AutoInterp accuracy across depth ranges

    ModelEarly RangeMid-Layer DropDeep RangeRebound (pp)
    Gemma 3 1BL0–8: 72.69%L9–16: 70.76%L17–25: 69.13%-1.64
    Gemma 3 4BL0–10: 70.44%L11–21: 68.88%L22–33: 67.80%-1.08
    Qwen3 8BL0–11: 71.06%L12–21: 64.91%L24–34: 69.94%+5.03
    Gemma 3 27BL0–17: 71.13%L18–27: 66.26%L34–46: 69.66%+3.40
  • MLP interpretability exhibits repeated drops and rebounds whose count grows with model scale: As shown in Table R3, distinct drops and rebounds increase from 1 in Gemma 3 1B (late layers), to 2 in Gemma 3 4B and Qwen3 8B, and 3 across early, middle, and late layers in Gemma 3 27B.

    Table R3 -- MLP SVD AutoInterp drops and rebounds across model scales

    Model & Drop StageBefore DropDropRebound StageRebound (pp)
    Gemma 3 1B (Late)L9–16: 71.64%L21–24: 63.44%L25: 69.56%+6.12
    Gemma 3 4B (Early)L0: 78.16%L1–7: 73.05%L11–17: 77.47%+4.42
    Gemma 3 4B (Late)L11–17: 77.47%L21–30: 66.79%L31–33: 69.99%+3.20
    Qwen3 8B (Mid)L0–3: 78.63%L12–21: 65.19%L24–28: 74.47%+9.29
    Qwen3 8B (Late)L24–28: 74.47%L30–34: 64.13%L35: 66.97%+2.84
    Gemma 3 27B (Early)L0–8: 77.88%L9–13: 67.18%L14–20: 72.83%+5.65
    Gemma 3 27B (Mid)L14–20: 72.83%L21–28: 70.69%L34–40: 73.52%+2.83
    Gemma 3 27B (Late)L34–40: 73.52%L47–55: 66.65%L56–61: 71.04%+4.39
  • Explanatory hypothesis linking drops and rebounds to LLM inference stages: In Appendix C.3 of the revised manuscript, we hypothesize that SVD AutoInterp score variations reflect transitions across the LLM inference stages of Lad et al. [1], detailed in Table R4. This hypothesis is symmetrically supported by cross-layer SAE evidence: Sun et al. [2] similarly find that representations shift from shallow token signals to mid-layer mixed contextual concepts and late-layer output mechanisms. Furthermore, Geva et al. [3] show that MLP vectors promote vocabulary concepts to build predictions step by step; this suggests each MLP rebound we discovered indicates a convergence onto an intermediate prediction concept, with larger models with more MLP rebounds undergoing more rounds of contextualization and concept consolidation.

    Table R4 -- Hypothesized correspondence between inference stages and SVD AutoInterp patterns

    Inference Stage [1]Information Processing RoleHypothesized SVD AutoInterp Behavior
    DetokenizationForm local word and entity representations.Encodes single words/entities, aligning cleanly with SVD to yield high scores.
    Feature EngineeringContextualize and mix input concepts.Mixes prompt concepts which span multiple axes, weakening linear alignment with SVD, causing scores to drop.
    Concept ConsolidationConsolidate prediction concepts and outputs.Converges onto prediction concepts, restoring SVD alignment and causing scores to rebound.

[1] Lad et al. The Remarkable Robustness of LLMs: Stages of Inference? NeurIPS 2025.
[2] Sun et al. Dense SAE Latents Are Features, Not Bugs. NeurIPS 2025.
[3] Geva et al. Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space. EMNLP 2022.

AuthorsOpenReview ↗

Response to Reviewer nhWV (3/5)

Confirming the Persistence of SVD’s Advantages Beyond Upstream Information Access

(W3) Incomplete isolation of SVD’s contribution. The steering comparison gives cumulative SVD access to upstream module activations, whereas the SAE encodes only the target residual state. This makes it difficult to attribute the gains to SVD itself, and the single-layer SVD ablation is limited to Gemma 3 4B.

(Q3) Table 4 shows that SVD and SAE candidate dictionary sizes differ substantially across models and layers. Could the authors compare cumulative SVD, single-layer SVD, and SAE steering across multiple models and layers? Given the substantial drop for single-layer SVD in the existing ablation, this would help distinguish gains from dictionary size, access to upstream module information, and the SVD directions themselves.

We sincerely thank the reviewer for this constructive and insightful feedback. We deeply appreciate the reviewer highlighting this fundamental question of mechanistic attribution: distinguishing whether Cumulative SVD's steering and interpretability advantages over SAE originate from (1) dictionary size, (2) upstream module accumulation, or (3) SVD representation properties. To directly resolve this, we constructed a Matched Cumulative SAE baseline across all five evaluation benchmarks in §§3.2–5.2 of the revised manuscript. By equalizing upstream access while giving SAE a 7.1×–15.1× larger dictionary, Cumulative SAE strictly controls for dictionary size and upstream module accumulation. Under these controlled conditions, Cumulative SVD's advantages are intrinsically grounded in the effectiveness of SVD representation properties themselves, rather than being driven by dictionary size or upstream module accumulation.

  • Disentangling dictionary size, upstream module accumulation, and SVD representation properties via Matched Cumulative SAE baseline:

    • Control for candidate dictionary size: Pooling 32,768 SAE features per block gives Cumulative SAE a 15.1× larger dictionary size on Gemma 3 1B and 7.1× larger on Gemma 3 4B than Cumulative SVD, which has only 2,176 and 4,608 features per block. If dictionary size drove performance gains, Cumulative SAE would substantially outperform Cumulative SVD.
    • Equalize upstream module accumulation: In Section 2 and Appendix B.3 of the revised manuscript, Matched Cumulative SAE pools Gemma Scope 2 attention and MLP SAEs across upstream blocks i=0…li = 0 \dots l on Gemma 3 1B and 4B, with identical access to upstream module accumulation as Cumulative SVD.
    • Isolate SVD representation properties: With upstream module accumulation equalized and dictionary size favoring SAE, SVD's consistent lead or comparable performance isolates the intrinsic effectiveness of SVD representation properties, such as closed-form orthogonality and exact reconstruction.
  • Cumulative SVD consistently outperforms or matches Cumulative SAE, showing the intrinsic advantage of SVD representations:

    • Relation Steering in §5.2: In Figures 7c–d of the revised manuscript and Table R5, cumulative access improves SAE steering over Residual SAE, confirming upstream module accumulation benefits both paradigms. Under matched access, Cumulative SVD leads in 7 of 8 original template evaluations and matches Acc@1 with a consistent Acc@5 lead on 10 unseen templates across both models.

    Table R5 -- Relation steering accuracy: Cumulative SVD / Cumulative SAE; Appendix G.2, Table 29 of the revised manuscript

    Pool (Count)Original Acc@1 (Δ\Delta)Original Acc@5 (Δ\Delta)10-Template Acc@1 (Δ\Delta)10-Template Acc@5 (Δ\Delta)
    1B Pub (58)0.741 / 0.734 (+0.007)0.861 / 0.839 (+0.022)0.686 / 0.668 (+0.018)0.841 / 0.805 (+0.036)
    1B Syn (99)0.605 / 0.591 (+0.014)0.753 / 0.755 (-0.002)0.555 / 0.562 (-0.007)0.731 / 0.735 (-0.004)
    4B Pub (68)0.716 / 0.701 (+0.015)0.883 / 0.850 (+0.033)0.627 / 0.629 (-0.002)0.846 / 0.809 (+0.037)
    4B Syn (142)0.686 / 0.656 (+0.031)0.851 / 0.802 (+0.049)0.623 / 0.626 (-0.003)0.802 / 0.787 (+0.014)
    • Topic Preference and Concept Specificity in §3.3: In Section 3.3 and Appendix D.3, Tables 8 and 17 of the revised manuscript, replacing Residual SAE with Cumulative SAE yields nearly identical 73-topic preferences relative to SVD across all layers (Spearman ρ∈[0.910,0.994]\rho \in [0.910, 0.994]), while 47.5% of monosemantic SVD features on 1B and 49.6% on 4B still lack a semantic substitute among top-3 Cumulative SAE neighbors.
Reviewer_kP9oOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper addresses the task of learning a dictionary of interpretable features from LLM activations. The predominant paradigm relies on Sparse Autoencoders, but they have a number of drawbacks: they require a training corpora, computationally-intensive training, and produce inconsistent dictionaries across different initializations. Authors propose a training-free method, constructing an overcomplete feature basis from left singular vectors of W_OV attention circuits and “down” projections of MLP layers across all LLM layers. This approach is substantially more efficient than SAE training and does not depend on seed or training data. Resulting features are validated for interpretability, sparse concept prediction and causal control. Cumulative SVD features on average have lower AutoInterp detection score than SAE-features, but cover complementary semantic concepts, perform better for sparse concept prediction and have higher selectivity (per RAVEL benchmark). Overall, Cumulative SVD features provide a massively more efficient and stable alternative to SAEs, or can be used as a complementary tool.

Critical strengths

Massively more efficient approach for dictionary learning compared to SAE

Produced dictionary does not depend on training corpus (since no training data is used), random seed or optimization quirks.

Critical weaknesses

notation ‘x’ in equation (7) assumes discrete token inputs, while ‘x’ in (4) denotes a local vectorized input, which is inconsistent. Notation (7) seems wrong if discrete token inputs are used; I do not expect such a simple linear decomposition in this case.

Lower AutoInterp scores of raw SVD features on Figure 2(a) are concerning — could the results in Sections 4 & 5 be due to a trade-off between interpretability and downstream performance? It would be interesting to select a subset of SVD and SAE features with similar AutoInterp scores, and compare attribute disentanglement metrics for them.

Questions for authors

Section 3.3 — do you have hypotheses or evidence for why SAEs and SVD focus on distinct semantic areas?

Section 3.3, concept specificity — what are some examples of SAE features not representable by SVD?

7 comments
AuthorsOpenReview ↗

Response to Reviewer kP9o (1/4)

We sincerely appreciate the reviewer’s time, thorough review, and very insightful feedback on our work. Below We detail our response to the weaknesses (i.e., W(1) ~ W(2)) and questions (i.e., Q(1) ~ Q(2)) point by point, along with revisions to the manuscript, which are highlighted in blue for your convenience. Please kindly let us know if our response addresses the issues you raised in this paper.

Resolving Notation Conflict in Equation (7)

(W1) notation ‘x’ in equation (7) assumes discrete token inputs, while ‘x’ in (4) denotes a local vectorized input, which is inconsistent. Notation (7) seems wrong if discrete token inputs are used; I do not expect such a simple linear decomposition in this case.

We sincerely thank the reviewer for carefully catching this notation conflict, which has helped us make the mathematical formulation of our decomposition in Section 2.2 of the revised manuscript much clearer and more rigorous. We completely agree that using x\mathbf{x} for both local continuous vectors and discrete token sequences was confusing; we now reserve x∈Rdin\mathbf{x} \in \mathbb{R}^{d_{\text{in}}} strictly for continuous vectors in equation (4) and use ss for token sequences in equation (7).

  • Updated formulation in Section 2.2: We update Eqs. (5) and (7) of the revised manuscript on an input token sequence ss to:
where $\mathbf{h}_\ell(s) \in \mathbb{R}^{d_{\mathrm{model}}}$ is the residual state before layer $\ell$, $\mathbf{e}(s) \in \mathbb{R}^{d_{\mathrm{model}}}$ is the initial token embedding, and $\mathbf{F}_\ell^{\mathrm{SVD}}$ collects the upstream left singular write matrices $\mathbf{U}_{OV}^{(j,p)}$ and $\mathbf{U}_{\mathrm{down}}^{(j)}$ ($j < \ell, p \le H$). The coefficient vector $\mathbf{a}_\ell^{\mathrm{SVD}}(s)$ concatenates the upstream module responses $\boldsymbol{\Sigma}_{OV}^{(j,p)}\mathbf{V}_{OV}^{(j,p)\top}\mathbf{x}_{\mathrm{attn}}^{(j,p)}(s)$ and $\boldsymbol{\Sigma}_{\mathrm{down}}^{(j)}\mathbf{V}_{\mathrm{down}}^{(j)\top}\mathbf{z}^{(j)}(s)$, where $\boldsymbol{\Sigma}$ and $\mathbf{V}$ are the singular values and right singular vectors of each write matrix, and $\mathbf{x}_{\mathrm{attn}}^{(j,p)}(s)$ and $\mathbf{z}^{(j)}(s)$ are the contextual attention inputs and MLP intermediate states from the forward pass on $s$. - **Why the decomposition has a linear form**: We would like to respectfully clarify that **Eq. (7) is linear only in the realized SVD coefficients $\mathbf{a}_\ell^{\mathrm{SVD}}(s)$**, while all forward non-linearities on $s$ (such as attention softmax and MLP gating) are evaluated and absorbed into the intermediate activations $\mathbf{x}_{\mathrm{attn}}^{(j,p)}(s)$ and $\mathbf{z}^{(j)}(s)$ that form those coefficients.
AuthorsOpenReview ↗

Response to Reviewer kP9o (2/4)

Evaluating Correlations Between Feature AutoInterp and Causal Disentanglement

(W2) Lower AutoInterp scores of raw SVD features on Figure 2(a) are concerning — could the results in Sections 4 & 5 be due to a trade-off between interpretability and downstream performance? It would be interesting to select a subset of SVD and SAE features with similar AutoInterp scores, and compare attribute disentanglement metrics for them.

We sincerely thank the reviewer for this thought-provoking question and experimental suggestion, which motivated a new empirical investigation and yielded an insightful finding that significantly enriches our study. We show empirically that there is no trade-off between feature interpretability and downstream causal performance: a feature's AutoInterp score has near-zero correlation with its causal contribution to RAVEL attribute disentanglement across both SVD and SAEs, demonstrating that descriptive interpretability and causal utility are decoupled.

  • The reviewer's suggested experiment and why pre-filtering beforehand is infeasible: Pre-selecting SVD and SAE subsets with matched AutoInterp scores is confounded because RAVEL relies on a few task-specific features encoding a target attribute such as Country, whereas AutoInterp scores general-text predictability. Pre-filtering by AutoInterp alone inevitably discards many attribute-carrying features, confounding the omission of attribute-encoding features with interpretability.
  • Our alternative design via ablations within learned task masks: To control for AutoInterp without losing task-relevant features, we first learn the RAVEL mask on the full dictionary, then group the selected active features into AutoInterp bins and compare each bin's ablation drop against random mask ablations.
  • Relative causal importance metric II via controlled ablation: Across Gemma3-4B-PT at L9, L17, and L22 and Qwen3-8B-Base at L9, L18, and L24 on all 6 SAEBench RAVEL attributes, active mask features are grouped into 7 held-out AutoInterp accuracy bins from <9/14<9/14 to 14/1414/14 (averaging both polarities for SVD). Disabling a fraction p \in \\{20\\%, 40\\%, 60\\%, 80\\%, 100\\%\\} of features in each bin yields a disentanglement drop LgroupL_{\text{group}}, compared against disabling an equal number of random active mask features with drop LrandomL_{\text{random}}:
where $I > 0$ and $I < 0$ indicate above- and below-average causal contribution. A negative correlation $\rho < 0$ between AutoInterp bins and $I$ indicates an interpretability–performance trade-off, whereas $\rho > 0$ indicates synergy between interpretability and causal utility. - **Near-zero correlations confirm no interpretability–utility trade-off**: As shown in **Table R1** and reported in **Appendix F.6, Table 27 of the revised manuscript**, mean Spearman rank correlations $\rho$ remain near zero across all settings, from $-0.105$ to $+0.138$ without consistent sign. High- and low-auto-interp features contribute comparably per feature once selected for the task, confirming that SVD's gains in **Sections 4 & 5** do not stem from sacrificing interpretability. This aligns with prior SAE findings [1] showing that higher AutoInterp scores are largely uncorrelated with downstream steering utility. **Table R1 -- Mean Spearman rank correlation $\rho$ between AutoInterp score bins and causal importance $I$** |Model / Dictionary|20%|40%|60%|80%|100%|Across-Fraction Average| |:---|:---:|:---:|:---:|:---:|:---:|:---:| |Gemma3-4B / SAE|+0.250|+0.052|+0.111|+0.107|+0.168|**+0.138**| |Gemma3-4B / SVD|+0.014|+0.097|+0.049|+0.077|+0.167|**+0.081**| |Qwen3-8B / SAE|+0.165|−0.256|−0.129|−0.131|−0.176|**−0.105**| |Qwen3-8B / SVD|−0.002|−0.037|+0.001|−0.222|−0.104|**−0.073**| [1] Wang et al. *Does Higher Interpretability Imply Better Utility? A Pairwise Analysis on Sparse Autoencoders.* ICLR 2026.
AuthorsOpenReview ↗

Response to Reviewer kP9o (3/4)

Explaining the Mechanistic Origins of Topic Preferences in SVD and SAE

(Q1) Section 3.3 — do you have hypotheses or evidence for why SAEs and SVD focus on distinct semantic areas?

We sincerely thank the reviewer for this insightful question, which inspired us to uncover the geometric mechanism behind dictionary topic preferences and made our analysis in Section 3.3 of the revised manuscript much deeper and more solid. We hypothesize and show empirically that topic preference is driven by geometric dispersion: topics whose representations are widely dispersed in the representation space favor orthogonal SVD basis vectors, whereas topics whose representations are tightly clustered in a narrow region favor overcomplete SAEs that can pack many features locally.

  • Geometric mechanism of orthogonal SVD versus overcomplete SAE: Within each attention WOVW_{OV} or MLP WdownW_{\mathrm{down}} operator, SVD basis vectors are mutually orthogonal and bounded by matrix rank, which prevents SVD from placing multiple nearby directions inside a tightly clustered activation cone. In contrast, overcomplete SAEs have far more features than the residual dimension and can densely pack directions to subdivide narrow clusters. When a topic's representations spread broadly across the space, SAE's local subdivision advantage diminishes and activations align across more orthogonal singular axes, favoring SVD allocation.
  • Quantifying topic dispersion via angular breadth AtA_t: Across all 30 attention and 30 MLP modules of Gemma3-4B-PT on all 73 topics, we quantify each topic's dispersion in module output space using angular breadth AtA_t, computed over n=512n=512 documents:
where $v_{t,d}$ and $v_{t,e}$ are normalized document-level module output directions for distinct documents $d$ and $e$. Larger $A_t$ indicates wider angular dispersion across texts, so a positive correlation between $A_t$ and relative SVD topic allocation $P_{\mathrm{SVD}} - P_{\mathrm{SAE}}$ directly supports our hypothesis. - **Strong empirical correlation across attention and MLP modules**: As shown in **Table R2** and detailed in **Section 3.3 and Appendix D.3 of the revised manuscript**, angular breadth correlates strongly with SVD topic allocation at $\rho = 0.739$ but weakly with SAE allocation at $\rho = 0.205$, yielding a strong positive correlation of $\rho = 0.709$ with relative SVD preference and a paired difference of $\Delta\rho = +0.534$ across all 60 modules. This relationship holds across both attention and MLP modules and remains robust across document sample sizes $n \in \\{128, 256, 512\\}$. **Table R2 -- Spearman correlation $\rho$ between angular breadth and topic preference across 73 topics** | Module Scope | SVD $\rho$ | SAE $\rho$ | Relative $(P_\text{SVD}-P_\text{SAE})$ $\rho$ [95% CI] | Paired $\Delta\rho$ [95% CI] | |:---|:---:|:---:|:---:|:---:| | Attention (30) | 0.717 | 0.334 | 0.721 [0.573, 0.822] | +0.384 [0.226, 0.539] | | MLP (30) | 0.625 | 0.108 | 0.562 [0.405, 0.684] | +0.517 [0.291, 0.726] | | All Modules (60) | 0.739 | 0.205 | 0.709 [0.576, 0.803] | +0.534 [0.336, 0.703] |

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision Note

Dear Reviewers and Area Chairs,

We sincerely thank you for your valuable time, careful attention, and dedicated effort throughout the review process, which has helped us substantially strengthen the manuscript. All revisions relative to the original submission are highlighted in blue in the updated PDF and are indexed below by order of appearance for your convenience:

LocationAdditional Experiment / ClarificationAddresses
§2.2, Eqs. (4)-(7)Clarified s vs. x notation for dictionary constructionkP9o W1
§2, §§3.2-5.2; App. B.3, D-GMatched Cumulative SAE evaluated across semantic coverage, concept complementarity, sparse probing, RAVEL, and relation steeringnhWV W3/Q3, wj9A Q1
§3.1; App. C.3All-layer raw SVD AutoInterp analysis & inference-stage alignment mechanism hypothesisnhWV W2/Q2
§3.2, Fig. 2b-c; App. D.1-D.2, D.4Monosemanticity-constrained semantic coverage & matched-AutoInterp coverage comparisonnhWV W1/Q1
§3.3; App. D.3, Tab. 12Representation-geometry-based topic-preference hypothesis validated via angular breadthkP9o Q1
§3.3, Fig. 4; App. D.3, Tabs. 15-16Reverse SAE→SVD concept specificity & bidirectional complementary pattern analysiskP9o Q2
§6; App. H.3, Tab. 39Online extraction runtime & memorynhWV W3/Q3, wj9A Q1
App. B.4Fairness controls for dictionary size, evaluation protocol, and upstream accesswj9A W1
App. D.3, Eq. (21)Clarified semantic neighbor retrieval processnhWV Q4
App. F.6, Tab. 27AutoInterp vs. causal disentanglement correlation in RAVELkP9o W2
App. I.1-I.3Downstream performance SVD-SAE complementarity (probing, RAVEL, relation steering)nhWV follow-up

Best regards,

Submission#21577 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.