- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces Sparse Weight Decomposition (SWD), which factorizes a weight matrix using an existing solver. SWD decompose a weight matrix into two sparse matrices (W approx. AB). At training time, SWD introduces 'bottleneck units' which read and write only to a few dimensions. These bottleneck units' appear tractable and can be ablated for efficient circuit extraction. SWD requires a smaller training dataset, compared to other methods like transcoders, for the same sufficiency and necessity on GPT-2, and some Qwen models of different sizes. It seems to achieve this with fewer active edges. It also matches sparse-pretrained GPT-2 with fine-tuning, and offers a zero-data variant.
Critical strengths
-
A cheap, post-hoc route to weight-sparse circuits. Recent papers from openai and others (like Gao et al) demonstrate that weight sparsity can be a correctness concern for large models even at small replacement ratios. In this case, weight sparsity introduces interpretability concerns. The paper asks whether the interpretability benefits of weight-sparse models can be obtained using an existing dense checkpoint, as opposed to training a new model at 1000times the compute. The authors argue that SWD reaches matched replacement fidelity using a fraction of the data that Transcoders and others use, and requires little to no training/calibration data. This suggests that sparse circuit analysis may be practical for a wide range of large models that already exist, particularly for circuit replacements with a small replacement ratio.
-
A carefully attempt to have a controlled comparison. Circuits are compared only between replacements matched to within small CE. Exact SVD/Random-B controls and a zero ablation robustness check are performed. Based on the findings, I believe they meant that the differences are due to the type of representation, however, I have concerns about the metric (see Weaknesses).
Critical weaknesses
-
No head-to-head against models trained from scratch. The main motivation of this paper is from Gao et al. (2025), but the only comparison with that approach is confounded: a sparse GPT-2 model with different GELU activations, no terminal checkpoints, and activation sparsity disabled, is trained from scratch. Using SWD to train a dense model with Gao et al.’s recipe (for example their dense baselines) and comparing circuits on the same tasks, would isolate SWD’s effects from post-hoc sparsity. To me, this feels important as one of the main goal should be to investigate whether post-hoc SWD sparsity can replace sparse pretraining. Even if SWD’s results go against them, it will be beneficial given that SWD is much less costly.
-
Related work feels weird/incomplete, and just links an appendix. I personally don't think makes sense to have the "full" (or we could say "more complete") related work section in appendix, but even if that was ok, it feels incomplete. Specifically the "parameter interpretability" is missing very relevant work, especially old, recent and follow-ups to Gao et al. (2025), which you cite to motivate the introduction quite extensively. Some works that feels like they should be there (among others)
- https://arxiv.org/abs/2607.02964 (interpretable weight-sparse parametesr),
- https://arxiv.org/abs/2508.00161 (SVD-based weight monitoring) and
- https://arxiv.org/abs/2410.08417 (weight-based circuit analysis).
Minor: Generally, text of the figures is very small (especially Fig 7)
Questions for authors
-
To me, If non-zero sparse SVD or random factors are not used, then it is not clear what SWD adds over sparsiyfing SVD. This is especially true if the non-zero budget is much smaller than what is done here.Then, the case for a new decomposition is not fully made. The edge advantage over Transcoders and the SVD/Random-B controls is built into the evaluation method?
-
Feels there's a lot of information in the appendix (over 30 pages!). Is there a way the authors could condense it or perhaps make a summary in the first page of the appendix? It's hard to grasp every experiment made, otherwise.
-
What would be define as an SWD unit? Units are active on every token, unlike Transcoder features or Gao et al.'s sparse activations. Would be lovely to see how the authors think about always-on unit should be interpreted, and ideally report how stable selected units are across seeds and calibration data
3 comments
Thank you for the thorough response and revised manuscript; I appreciate the additional work, and the same-recipe comparison with sparse pretraining (Appendix F) addresses one of my concern. Still, the new equal-budget controls (Table 11) show that pruned SVD and Random-B give comparably compact circuits, so the edge is due to sparsity, and what SWD adds is replacement fidelity. (This should be expressed/highlighted in the abstract). I am also concerned that, in the case of 13-21 of the top-32 task units, the survival of circuits depends on the calibration data (tested on a single matrix), which means that the extracted circuits are fairly sensitive to the calibration set, what matches my intuition and limits the scope of the method.
Taking this into account, I will keep my score.
Thank you for reading the revision carefully and for the follow-up. We are glad the same-recipe comparison with sparse pretraining (Appendix F) addressed one of your concerns.
Sparsity and replacement fidelity. We agree with your reading of Table 11 and will state this distinction explicitly in the Abstract: sparse read/write supports are what bound the active edges per selected unit, while SWD's alternating optimization is what finds sparse supports that remain faithful to the original model (2.6–7.5× lower CE increase than equal-budget pruned and refitted SVD or Random-B, Appendix H.2). Both properties are essential for circuit extraction, as a compact subgraph is only meaningful when the underlying replacement faithfully preserves the original model's computation.
Calibration sensitivity. We also appreciate your point on cross-calibration variation (Appendix H.4). While the hard top-32 cutoff preserves 13–21/32 individual units across disjoint 16k-token calibration splits, the matched units maintain high functional alignment (0.82–0.83 activation correlation); moreover, our zero-data variant (, Section 3.7 and Appendix I) requires no calibration data at all and still yields compact, causally faithful task circuits. We have noted the role of calibration sampling in the limitations (Appendix C).
Thank you again for your time and constructive feedback throughout the review.
Thank you very much for your insightful review, appreciation of our post-hoc motivation, and concrete experimental suggestions.
Below, we use W1 for Weakness 1, Q1 for Question 1, etc.
W1 — Controlled comparison with same-recipe sparse pretraining. Following your suggestion to remove the confounds in our original comparison, we pretrained a dense GPT-2 from scratch using the same corpus, tokenizer, architecture (
nn.GELU), and 2.884B-token step as our sparse-pretrained reference, then applied post-hoc SWD (4.19M calibration tokens) and SWD-FT (+16.38M fine-tuning tokens) across all 48 transformer-body matrices at the matched budget (26.77M nonzeros; Appendix F):W2 — Expanding Related Work. We expanded the main Related Work (§4) to discuss parameter-space interpretability, including Marin-Llobet & Heimersheim [4], Zhong & Raghunathan [3], and Pearce et al. [5].
Q1 — Equal-budget Sparse SVD / Random-B controls and the edge metric. Thank you for raising this point. To directly test what SWD adds over sparsifying SVD, we pruned exact SVD and Random-B (3 seeds) to SWD's exact nonzero budgets on GPT-2 layer-8
mlp.c_projand refitted surviving entries on the same calibration data (§3.6; Appendix H.2, Table 11). At shared sparsity, all three yield competitive within-model circuit curves, while differing substantially in replacement fidelity (ΔCE over dense):Without refitting, pruned SVD degrades further (0.0189 / 0.0428). This controlled experiment further clarifies SWD's contribution: sparse connectivity yields compact subgraphs, while SWD's alternating optimization lowers replacement CE loss by 2.6–7.5$\times$ at the same edge budget so those subgraphs stay faithful to the dense model.
Q2 & Minor — Appendix roadmap and figure readability. We added a 1-page summary table at the start of the Appendix and enlarged figure text throughout (especially original Figure 7, now Figure 34 in Appendix J).
Q3 — Interpreting "always-on" SWD units and stability.
Thank you again—your control designs and framing suggestions substantially sharpened the paper, and we welcome any further questions.
References:
[1] Conmy et al. Towards Automated Circuit Discovery for Mechanistic Interpretability. NeurIPS 2023.
[2] Rimsky et al. Steering Llama 2 via Contrastive Activation Addition. ACL 2024.
[3] Zhong & Raghunathan. Watch the Weights: Unsupervised monitoring and control of fine-tuned LLMs. ICLR 2026.
[4] Marin-Llobet & Heimersheim. Individual Parameters in Weight-Sparse Transformers Appear Interpretable. arXiv:2607.02964, 2026.
[5] Pearce et al. Bilinear MLPs enable weight-based mechanistic interpretability. ICLR 2025.