- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
By applying the SVD to the model, This paper builds a training-free feature, dictionary from model weights.
Critical strengths
This paper shows a simple and well-multiple idea. It provides a cheap deterministic baseline which is valuable for the field. It also provided a useful application study for different comparisons.
Critical weaknesses
However, I think the comparison is structurally unfair to SAEs and the central claim is confounded. The dictionary size and selection give SVD an advantage in several benchmarks.
Questions for authors
How did you do the SVD features compared against SAE's training on the same cumulative module output? This is the cleanest test of whether SVD itself matters.
7 comments
Response to Reviewer wj9A (2/4)
Continued from our response to W1 in the preceding comment (1/4).
Controlling for feature selection protocols:
- Strictly symmetric evaluation budgets and search algorithms: In Appendices D.1, E.1, F.1, and G.1 of the submitted manuscript, both dictionaries use identical budgets in coverage and identical across 113 sparse probing tasks on training splits. Both also train differentiable masks under identical schedules in RAVEL, and run identical Integrated Gradients ranking alongside a two-stage grid search spanning 420 coarse and 2,400 refine configurations over layer, , and scale in relation steering.
- Selecting more features confers no automatic metric advantage: In RAVEL, activating unneeded features causes collateral damage to non-target attributes, degrading and lowering composite disentanglement . In relation steering, the composed steering vector is normalized to unit norm before scaling by as formulated in Eq. 19 of the submitted manuscript; adding features refines steering direction without inflating steering magnitude.
Controlling for upstream module accumulation structure:
- Cumulative SVD outperforms larger cumulative baselines under identical upstream access: Under identical depth accumulation across blocks , Cumulative SVD outperforms Cumulative Module Neurons with 1.1× larger size and Cumulative Weight Columns with 2.7× larger size, scoring 0.748 vs. 0.719 on RAVEL disentanglement and achieving 71.65% / 88.29% vs. 71.63% / 83.83% and 70.37% / 86.42% in relation steering, as reported in Figures 6b and 7c, Tables 15 and 19 of the submitted manuscript.
- Matched Cumulative SAE confirms SVD advantage under identical accumulation: As introduced above, Cumulative SAE pools upstream attention and MLP SAEs to match Cumulative SVD's upstream accumulation, as shown in Table R2 and Sections 3.2–5.2 of the revised manuscript. Cumulative SVD consistently outperforms or matches Cumulative SAE across semantic coverage, concept complementarity, sparse probing, RAVEL disentanglement, and relation steering.
- Local SVD beats Residual SAE without accumulation: Single-block One Attn+MLP SVD outperforming Residual SAE in relation steering in Table R1 confirms that SVD does not rely on depth accumulation to surpass residual-stream SAEs.
Table R2 -- Cumulative SVD vs. baselines under matched upstream accumulation in the revised manuscript
Benchmark & Setting (Paper Ref.) Baseline (Pool Ratio vs. SVD) Baseline Score SVD Score RAVEL Disentangle: 4B, 4-layer mean; §5.1; App. F.2, Tab. 23 Matched Cumul. SAE (7.1×) 0.682 0.748 RAVEL Disentangle: 4B, 4-layer mean; §5.1; App. F.3, Tab. 24 Cumul. Module Neurons (1.1×) 0.719 0.748 Relation Steering: 4B Pub, Acc@1 / Acc@5; §5.2; App. G.2, Tab. 29 Matched Cumul. SAE (7.1×) 70.10% / 85.04% 71.65% / 88.29% Relation Steering: 4B, Acc@1 / Acc@5; §5.2; App. G.3, Tab. 30 Cumul. Neurons (1.1×) | Weight Cols (2.7×) 71.63/83.83% | 70.37/86.42% 71.65% / 88.29% Upstream accumulation reflects an intrinsic structural advantage of SVD rather than an evaluation artifact: Leveraging upstream module writes is an inherent structural property of SVD. Because each module's singular basis is closed-form and rank-bounded, SVD naturally spans residual accumulation across depth with zero training and lower inference cost. In contrast, stacking cumulative SAEs across depth is computationally prohibitive, causing parameter explosion along with 6×–12× higher latency and peak VRAM, as documented in Section 6 and Appendix H.3, Table 39 of the revised manuscript. Because single-layer Residual SAE remains the practical mainstream in mechanistic interpretability, evaluating Cumulative SVD against it reflects a realistic operational baseline, and SVD's ability to seamlessly leverage upstream accumulation is an intrinsic structural advantage of the formulation.
[1] Karvonen et al. SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability, ICML 2025.
Response to Reviewer wj9A (3/4)
Confirming SVD’s Advantages over SAE Under Matched Cumulative Information Access
(Q1) How did you do the SVD features compared against SAE's training on the same cumulative module output? This is the cleanest test of whether SVD itself matters.
We sincerely thank the reviewer for proposing this clean and decisive test, which directly motivated our new Matched Cumulative SAE baseline across all five evaluations and made the central claim of our work substantially more solid. Following the reviewer's suggestion, we constructed Matched Cumulative SAE on Gemma 3 1B and 4B by pooling upstream attention and MLP SAEs across identical blocks , and evaluated it across all five experimental sections and runtime scalability in §§3.2–5.2 and §6 of the revised manuscript. We found that Cumulative SVD consistently outperforms or matches Cumulative SAE across semantic coverage, concept complementarity, sparse probing, RAVEL disentanglement, and relation steering, while avoiding over an order of magnitude higher extra latency and extra peak VRAM from Cumulative SAE
By equalizing upstream access while giving SAE a 7.1×–15.1× larger candidate pool, this Cumulative SAE baseline strictly controls for dictionary size and upstream module accumulation. Under these controlled conditions, Cumulative SVD's advantages are intrinsically grounded in the effectiveness of SVD representation properties themselves, rather than being driven by dictionary size or upstream module accumulation.
Constructing Matched Cumulative SAE with equal upstream module accumulation and 7.1×–15.1× larger dictionary size: Because the residual stream after Transformer block accumulates additively as where is the residual state, is the token embedding, and are the attention and MLP writes from block , we pool pretrained Gemma Scope 2 attention- and MLP-output SAEs of width 16,384 across all upstream blocks as described in Section 2 and Appendix B.3 of the revised manuscript. This preserves upstream module accumulation and yields 32,768 SAE features per block compared to 2,176 SVD features on Gemma 3 1B and 4,608 on Gemma 3 4B, making the Cumulative SAE pool 15.1× larger on 1B and 7.1× larger on 4B.
Semantic Coverage in §3.2: As shown in Figures 2b–c of the revised manuscript and summarized in Table R3, Cumulative SVD achieves higher pure-dictionary semantic coverage across all 24 layer-by-budget settings.
Table R3 -- Semantic coverage density : Cumulative SVD / Cumulative SAE; Appendix D.2, Tables 6–7 of the revised manuscript
Budget Gemma 3 1B (4-layer mean) Gemma 3 4B (4-layer mean) 16 0.212 / 0.139 (+52.9%) 0.205 / 0.138 (+48.5%) 32 0.341 / 0.269 (+26.8%) 0.335 / 0.269 (+24.8%) 64 0.539 / 0.521 (+3.4%) 0.551 / 0.520 (+6.0%) Topic Preference and Concept Specificity in §3.3: In Section 3.3 and Appendix D.3, Tables 8 and 17 of the revised manuscript, replacing Residual SAE with Cumulative SAE yields nearly identical 73-topic preferences compared with SVD across all layers with Spearman , while 47.5% of judged monosemantic SVD features on 1B and 49.6% on 4B still lack a semantic substitute among their top-3 nearest Cumulative SAE neighbors.
Response to Reviewer wj9A (1/4)
We sincerely appreciate the reviewer’s time, thorough review, and very insightful feedback on our work. Below We detail our response to the weaknesses (i.e., W(1)) and questions (i.e., Q(1)) point by point, along with revisions to the manuscript, which are highlighted in blue for your convenience. Please kindly let us know if our response addresses the issues you raised in this paper.
Controlling for Candidate Dictionary Size, Selection Protocols, and Upstream Access
(W1) However, I think the comparison is structurally unfair to SAEs and the central claim is confounded. The dictionary size and selection give SVD an advantage in several benchmarks.