acceptodds
Under review as a conference paper at ICLR 2027

Sparse Weight Decomposition for Efficient Circuit Extraction

12 views

Abstract

Dense pretrained transformers do not naturally expose interpretable units for circuit extraction. Existing approaches obtain such units by learning auxiliary sparse representations or training sparse models, incurring substantial additional computation while potentially introducing a fidelity gap between the representation being analyzed and the original pretrained model. We propose Sparse Weight Decomposition (SWD), which reparameterizes pretrained linear projections by factorizing each weight matrix into two sparse factors whose shared intermediate coordinates serve as individually addressable circuit units. Without training a separate replacement network, this parametric representation supports the same scoring, selection, and ablation circuit extraction workflow used for methods that learn sparse features. Across single-matrix replacements, SWD matches the held-out fidelity achieved by Transcoder and other strong baselines while using less than 1% of the data that those baselines use to train their replacements. At matched replacement fidelity, SWD reaches the same circuit sufficiency and necessity targets with fewer active read/write edges and selected units across tasks on GPT-2, Qwen2.5, and Qwen3.5-27B. We further show that SWD remains effective for full-model replacement of all attention and MLP weight matrices after fine-tuning the nonzero factor values. Finally, SWD also features a zero-data variant, allowing broader use of mechanistic interpretability analysis (, per-step analysis).

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic interpretability, circuit extraction, sparse factorization, language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1nfSOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Sparse Weight Decomposition (SWD), which factorizes a weight matrix using an existing solver. SWD decompose a weight matrix into two sparse matrices (W approx. AB). At training time, SWD introduces 'bottleneck units' which read and write only to a few dimensions. These bottleneck units' appear tractable and can be ablated for efficient circuit extraction. SWD requires a smaller training dataset, compared to other methods like transcoders, for the same sufficiency and necessity on GPT-2, and some Qwen models of different sizes. It seems to achieve this with fewer active edges. It also matches sparse-pretrained GPT-2 with fine-tuning, and offers a zero-data variant.

Critical strengths
  1. A cheap, post-hoc route to weight-sparse circuits. Recent papers from openai and others (like Gao et al) demonstrate that weight sparsity can be a correctness concern for large models even at small replacement ratios. In this case, weight sparsity introduces interpretability concerns. The paper asks whether the interpretability benefits of weight-sparse models can be obtained using an existing dense checkpoint, as opposed to training a new model at 1000times the compute. The authors argue that SWD reaches matched replacement fidelity using a fraction of the data that Transcoders and others use, and requires little to no training/calibration data. This suggests that sparse circuit analysis may be practical for a wide range of large models that already exist, particularly for circuit replacements with a small replacement ratio.

  2. A carefully attempt to have a controlled comparison. Circuits are compared only between replacements matched to within small CE. Exact SVD/Random-B controls and a zero ablation robustness check are performed. Based on the findings, I believe they meant that the differences are due to the type of representation, however, I have concerns about the metric (see Weaknesses).

Critical weaknesses
  1. No head-to-head against models trained from scratch. The main motivation of this paper is from Gao et al. (2025), but the only comparison with that approach is confounded: a sparse GPT-2 model with different GELU activations, no terminal checkpoints, and activation sparsity disabled, is trained from scratch. Using SWD to train a dense model with Gao et al.’s recipe (for example their dense baselines) and comparing circuits on the same tasks, would isolate SWD’s effects from post-hoc sparsity. To me, this feels important as one of the main goal should be to investigate whether post-hoc SWD sparsity can replace sparse pretraining. Even if SWD’s results go against them, it will be beneficial given that SWD is much less costly.

  2. Related work feels weird/incomplete, and just links an appendix. I personally don't think makes sense to have the "full" (or we could say "more complete") related work section in appendix, but even if that was ok, it feels incomplete. Specifically the "parameter interpretability" is missing very relevant work, especially old, recent and follow-ups to Gao et al. (2025), which you cite to motivate the introduction quite extensively. Some works that feels like they should be there (among others)

Minor: Generally, text of the figures is very small (especially Fig 7)

Questions for authors
  • To me, If non-zero sparse SVD or random factors are not used, then it is not clear what SWD adds over sparsiyfing SVD. This is especially true if the non-zero budget is much smaller than what is done here.Then, the case for a new decomposition is not fully made. The edge advantage over Transcoders and the SVD/Random-B controls is built into the evaluation method?

  • Feels there's a lot of information in the appendix (over 30 pages!). Is there a way the authors could condense it or perhaps make a summary in the first page of the appendix? It's hard to grasp every experiment made, otherwise.

  • What would be define as an SWD unit? Units are active on every token, unlike Transcoder features or Gao et al.'s sparse activations. Would be lovely to see how the authors think about always-on unit should be interpreted, and ideally report how stable selected units are across seeds and calibration data

3 comments

Thank you very much for your insightful review, appreciation of our post-hoc motivation, and concrete experimental suggestions.

Below, we use W1 for Weakness 1, Q1 for Question 1, etc.

W1 — Controlled comparison with same-recipe sparse pretraining. Following your suggestion to remove the confounds in our original comparison, we pretrained a dense GPT-2 from scratch using the same corpus, tokenizer, architecture (nn.GELU), and 2.884B-token step as our sparse-pretrained reference, then applied post-hoc SWD (4.19M calibration tokens) and SWD-FT (+16.38M fine-tuning tokens) across all 48 transformer-body matrices at the matched budget (26.77M nonzeros; Appendix F):

  • CE is 3.450 (sparse pretraining) vs. 3.459 (SWD) and 3.478 (SWD-FT), where fixed-support fine-tuning in SWD-FT helps preserve downstream task behavior across the 48-matrix replacement, and circuit curves are broadly comparable. Activation sparsity was disabled in the reference by design so both methods isolate weight sparsity.
  • As you highlighted—avoiding 1000×1000\times pretraining compute to analyze large models that already exist—this controlled result supports SWD as a practical post-hoc complement to sparse pretraining: it requires <1%<1\% adaptation tokens, lets practitioners choose target layers and sparsity after pretraining, and scales directly to existing large checkpoints such as Qwen3.5-27B (§3.3, Appendix E).

W2 — Expanding Related Work. We expanded the main Related Work (§4) to discuss parameter-space interpretability, including Marin-Llobet & Heimersheim [4], Zhong & Raghunathan [3], and Pearce et al. [5].

Q1 — Equal-budget Sparse SVD / Random-B controls and the edge metric. Thank you for raising this point. To directly test what SWD adds over sparsifying SVD, we pruned exact SVD and Random-B (3 seeds) to SWD's exact nonzero budgets on GPT-2 layer-8 mlp.c_proj and refitted surviving entries on the same calibration data (§3.6; Appendix H.2, Table 11). At shared sparsity, all three yield competitive within-model circuit curves, while differing substantially in replacement fidelity (ΔCE\Delta\text{CE} over dense):

Sparsity (Nonzeros)SWDSparse SVD (refit)Sparse Random-B (refit, 3 seeds)
50% (1.18M)0.00120.00630.0086–0.0093
75% (0.59M)0.00680.01750.0204–0.0214

Without refitting, pruned SVD degrades further (0.01890.0189 / 0.04280.0428). This controlled experiment further clarifies SWD's contribution: sparse connectivity yields compact subgraphs, while SWD's alternating optimization lowers replacement CE loss by 2.6–7.5$\times$ at the same edge budget so those subgraphs stay faithful to the dense model.

Q2 & Minor — Appendix roadmap and figure readability. We added a 1-page summary table at the start of the Appendix and enlarged figure text throughout (especially original Figure 7, now Figure 34 in Appendix J).

Q3 — Interpreting "always-on" SWD units and stability.

  • Always-on units: An SWD unit is a rank-one read/write path whose scalar activation varies continuously across tokens (formalized in §2.1–2.2 and Appendix A, Table 2). Whereas activation sparsity selects features per token, SWD selects a fixed unit subset per task via attribution and tests it causally on held-out data (as in ACDC [1]). Continuous linear directions also support activation steering and localized weight editing without per-token zeros (e.g., CAA [2], Watch the Weights [3], and our single-unit edit in Appendix K).
  • Stability: Factor non-uniqueness is already discussed in §5 and Appendix C. To test empirical stability directly (Appendix H.4), we evaluated SWD across seeds and disjoint calibration splits: given the calibration Gram matrix, our initialization and solver are deterministic (identical factors across seeds), and across disjoint calibration splits matched units maintain 0.82–0.83 mean activation correlation while preserving 13–21 of top-32 task units.

Thank you again—your control designs and framing suggestions substantially sharpened the paper, and we welcome any further questions.

References:

[1] Conmy et al. Towards Automated Circuit Discovery for Mechanistic Interpretability. NeurIPS 2023.

[2] Rimsky et al. Steering Llama 2 via Contrastive Activation Addition. ACL 2024.

[3] Zhong & Raghunathan. Watch the Weights: Unsupervised monitoring and control of fine-tuned LLMs. ICLR 2026.

[4] Marin-Llobet & Heimersheim. Individual Parameters in Weight-Sparse Transformers Appear Interpretable. arXiv:2607.02964, 2026.

[5] Pearce et al. Bilinear MLPs enable weight-based mechanistic interpretability. ICLR 2025.

Thank you for the thorough response and revised manuscript; I appreciate the additional work, and the same-recipe comparison with sparse pretraining (Appendix F) addresses one of my concern. Still, the new equal-budget controls (Table 11) show that pruned SVD and Random-B give comparably compact circuits, so the edge is due to sparsity, and what SWD adds is replacement fidelity. (This should be expressed/highlighted in the abstract). I am also concerned that, in the case of 13-21 of the top-32 task units, the survival of circuits depends on the calibration data (tested on a single matrix), which means that the extracted circuits are fairly sensitive to the calibration set, what matches my intuition and limits the scope of the method.

Taking this into account, I will keep my score.

Thank you for reading the revision carefully and for the follow-up. We are glad the same-recipe comparison with sparse pretraining (Appendix F) addressed one of your concerns.

Sparsity and replacement fidelity. We agree with your reading of Table 11 and will state this distinction explicitly in the Abstract: sparse read/write supports are what bound the active edges per selected unit, while SWD's alternating optimization is what finds sparse supports that remain faithful to the original model (2.6–7.5× lower CE increase than equal-budget pruned and refitted SVD or Random-B, Appendix H.2). Both properties are essential for circuit extraction, as a compact subgraph is only meaningful when the underlying replacement faithfully preserves the original model's computation.

Calibration sensitivity. We also appreciate your point on cross-calibration variation (Appendix H.4). While the hard top-32 cutoff preserves 13–21/32 individual units across disjoint 16k-token calibration splits, the matched units maintain high functional alignment (0.82–0.83 activation correlation); moreover, our zero-data variant (G=I\mathbf{G}=\mathbf{I}, Section 3.7 and Appendix I) requires no calibration data at all and still yields compact, causally faithful task circuits. We have noted the role of calibration sampling in the limitations (Appendix C).

Thank you again for your time and constructive feedback throughout the review.

Reviewer_1smLOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Sparse Weight Decomposition (SWD) for efficient circuit extraction. It decomposes pretrained weight matrices into sparse factors and uses the intermediate bottleneck units as components for attribution and ablation. The authors evaluate the method on GPT-2 and Qwen models, comparing replacement fidelity, circuit size, and training cost with Transcoder and other baselines. The approach offers a potentially inexpensive way to obtain units for circuit analysis, although the extent to which these units provide interpretable accounts of model computation remains unclear.

Critical strengths

The fidelity is higher than that of the Transcoder approach, and I believe the resulting mechanistic explanations are also more likely to reflect the model’s actual internal mechanisms.

The training cost is also lower than that of SAEs.

Critical weaknesses

Sparse connectivity alone does not imply mechanistic interpretability, and the work lacks validation of complete circuit-level explanations.

The scale of interpretability validation is too small to assess how much mechanistic understanding the method actually provides.

The method does not explicitly model interactions between features; instead, it ranks features by their individual first-order contributions within the full model. This feels closer to a search-based interpretability approach, which I find less compelling. The authors might consider gradient-based circuit tracing methods instead.

Questions for authors

Do the low-rank components in Table 9 actually have interpretability? They do not seem to demonstrate interpretability at the level of computation.

The paper highlights one compelling case from 2,160 candidate evaluations on three prompts, yet lacks systematic reporting on how many units or unit pairs yield clear, verifiable functional interpretations, how robust these interpretations are across inputs, and how much of the task-relevant computation they explain. This is admittedly challenging, but I would have liked to see richer and more varied evidence.

The main experiments replace only mlp.c_proj (h↦hWout), which matches SWD’s native target but asks Transcoder to approximate a linear map rather than the full nonlinear MLP (x↦GELU(xWin)Wout). This comparison is informative but favors SWD’s design; it does not establish broad superiority in the standard Transcoder regime. Appendix G’s full-MLP results are more mixed, with standard Transcoder competitive on several metrics—worth emphasizing in the main narrative.

4 comments

Thank you very much for your thoughtful review and constructive feedback.

Below, we use W1 for Weakness 1, Q1 for Question 1, etc.

W1 & Q1 — Unit selectivity and circuit-level computation.

  • Unit selectivity (Q1): In original Table 9 and Figure 7 (now Table 12 and Figure 34), unit selection uses only task gradients (Appendix D.3), while labelling uses only task-agnostic WikiText-2 activations (Appendix J). Their independent agreement on numeric/quantity contexts (5/9 units in Table 12, 4/6 in Figure 34) provides an operational check of unit selectivity [3, 4].
  • Circuit-level validation (W1, Q1; Figure 7 in §3.8, Appendix M): To further validate circuit-level computation, we traced an end-to-end Greater-Than circuit on SWD-FT GPT-2 across attributed c_proj SWD units in MLP8–11, attention heads, and earlier MLPs via gradient-based EAP-IG [1, 2]. The resulting 377-edge circuit includes 16 individual SWD units across L9–L11 (5 in L9, 6 in L10, and 5 in L11) that connect upstream attention heads and earlier MLPs to downstream MLPs and the readout.

W2 & Q2 — Validation scale and cross-input robustness. First, the 2,160 evaluations on three prompts in Appendix L were an automated numerical screen to pick one illustrative attention example, alongside a separate single-unit weight-editing study already in Appendix K; we apologize for the unclear scope and have clarified this in Appendix L. Second, our quantitative circuit results in §3.3–3.5 are already evaluated systematically across full held-out test splits. To further test cross-input robustness at the full-circuit level, we evaluated the 377-edge Greater-Than circuit above on 512 held-out prompts spanning 8 unseen templates, achieving 0.87 causal faithfulness overall (0.85–0.89 per template).

W3 — Gradient-based attribution and component interactions. We completely agree with your intuition favoring gradient-based circuit tracing—in fact, our pipeline already follows the exact gradient-based approach you suggest: every unit is scored in a single backward pass via first-order attribution patching [1] (§3.2, Appendix D.3), with no subset search. We apologize that our main-text wording caused this ambiguity and have clarified §3.2 accordingly.

Regarding interactions, our initial submission already analyzes within-layer unit interactions in Appendix N (N.1–N.2; Appendix M in the original submission), measuring output correlations in c_proj and pairwise non-additivity across GELU in c_fc. To further examine cross-layer interactions on a full task circuit as you suggested, we ran single, pairwise, and group ablations across the 20 key components (attention heads and MLPs) of the Greater-Than circuit on the 512 held-out prompts (§3.8; Appendix N.3, Figure 44). Ablating all 20 together lowers the task score by 1.21 versus 4.23 when summing their individual drops (3.5×3.5\times first-order over-counting), while pairwise and group ablations cleanly separate backup attention-head pairs from sequential late-MLP pairs.

Q3 — Standard MLP-level Transcoder comparison in the main narrative. We agree and have moved the Appendix G comparison (replacing the entire nonlinear MLP layer, with an updated and longer-trained standard Transcoder baseline) into the main text (§3.4). As shown there, Transcoders use fewer units on some sufficiency curves, whereas SWD reaches larger necessity drops across all four tasks. Beyond these empirical tradeoffs, SWD also has clear methodological advantages: it requires no auxiliary network training, uses far fewer tokens, and applies directly to any linear projection (including attention matrices).

Thank you again for your helpful suggestions, and we welcome any further questions.

References:

[1] Syed et al. Attribution Patching Outperforms Automated Circuit Discovery. BlackboxNLP 2024.

[2] Hanna et al. Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms. COLM 2024.

[3] Bills et al. Language models can explain neurons in language models. OpenAI, 2023.

[4] Marks et al. Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models. ICLR 2025.

Thank you for your detailed response, and especially the case in Appendix M and the feature interactions in Figure 44. But I still concerns that it seems the interpretation complexity is weaker than transcoder features, but I know its hard. Could you evaluate the interpretability and complexity scores of your sparse units [1]? Also, Figure 39 does not seem to reveal any specific reasoning patterns of the model. Can you translate this circuit to a text interpretation or summarize some of the computational patterns of the model's internal behavior?

[1] H. Cunningham and T. Conerly, “Circuits Updates — June 2024,” Transformer Circuits Thread, Jun. 28, 2024. [Online]. Available: https://transformer-circuits.pub/2024/june-update/index.html

Thank you for your constructive follow-up and for your positive feedback on Appendix M and Figure 44. We have revised our paper (Appendices P & Q, Figures 45–46) following your suggestions.

1. Interpretability and Complexity Scores Following [1]

Adapting the blinded evaluation of [1], an LLM rater (gpt-6-astra) scored anonymized activation cards (method and unit identities hidden) on a shared 131k-token WikiText-2 corpus along two 1–5 scales: Activation Consistency and Complexity. Because SWD units are signed (no ReLU), we score positive and negative tails separately (weighted 0.5 each). At Layer 8 we compare 32 random units per method against the paper's two 24,576-feature Transcoders (TC high, CE-matched to SWD; TC low, lower CE), and also score all 16 explicit SWD circuit units of Figure 39 under the same corpus and rubric (Appendix P, Table 14):

Evaluated Unit SetWidthEval CEActive TokensConsistency (1–5)Complexity (1–5)
SWD, L8 (32 random units)7683.258850.0%2.792.81
Transcoder (TC high), L8 (CE-matched)24,5763.25913.7%3.562.87
Transcoder (TC low), L8 (lower CE)24,5763.25626.6%3.472.87
SWD, Figure 39 units (16 units, L9–L11)7683.478*50.0%2.942.84

(*Full-model 48-matrix replacement CE vs. 3.272 dense; rows 1–3 replace only the L8 MLP).

  • Complexity is comparable (~2.8–2.9/5 for both SWD and Transcoders): The difference is +0.06 for both Transcoders. Nearly all units across all methods—and 28 of 31 scored directions in Figure 39—sit at Level 3 (phrases, semantic categories, or grammatical patterns, e.g., dates/durations in L10.c92), matching the ~2.6–2.9 range reported for SAEs in [1, Fig. 2].
  • Where SWD is weaker is Activation Consistency, reflecting an architectural trade-off: Transcoder features are more consistent (+0.77 at matched CE). A Transcoder expands MLP width by 32× (24,576 ReLU features, firing on 3.7%–6.6% of tokens), whereas SWD decomposes the existing weights without expanding width (768 signed units, active per sign on ~50% of tokens)—capturing Level-3 patterns of similar abstraction while more often multiplexing a primary pattern with secondary clusters.

(Continued in the next comment.)

(Continued from the previous comment.)

2. Text Interpretation of Figure 39 and Computational Patterns

In words, the circuit implements "copy the start year, then favor later years": mid-layer heads copy the start year YY to the final token; the late MLPs raise years above YY relative to years at or below it; within them, the explicit SWD units act as start-year-dependent "later-year" biases, while the exact cut at YY is written by the grouped units.

On Greater-Than prompts such as "The conflict began around 1551 and ended around 15" (P(year > 51) = 0.70 vs. P(≤ 51) = 0.28), Figure 39 (Lℓ.Hh = attention head, Lℓ.ck = individual SWD c_proj unit scaling a fixed rank-1 write vector, others (752) = remaining grouped units) unfolds from bottom to top in three steps (Appendix Q):

  1. Attention heads copy the start year (51) to the final token. In the example, at the prediction token " 15", heads L8.H11 and L9.H1 attend directly back to "51" (weights 0.67 and 0.93), and reading L8.H11's output through the unembedding (logit lens) ranks "51" #1 among all 100 year tokens (00–99). Across all 512 held-out prompts, the circuit contains all 7/7 start-year heads that Hanna et al. [2, §3.2] identified in dense GPT-2, although we did not use [2] as a prior (2.4 expected by chance, hypergeometric p = 4×10⁻⁴). Each of the seven places 0.43–0.92 of the final position's attention on YY (above the 90th percentile 0.14 of non-circuit heads; Figure 45), and L7.H10 / L8.H11 rank YY #1 in 99% / 94% of prompts.
  2. Explicit SWD units (c_k) apply start-year-dependent year-slope shifts. In the example, the copied year signal reaches the explicit units directly from L9.H1 (→ L9.c406, c445, c650, c691) or via earlier MLPs (MLP8 → L9.c85; L9 others → L10.c564, c363). Changing the start year from 01 to 51 moves 13 of the 16 units toward later years (e.g., L10.c564 rises from −3.19 to +0.55 along a later-years direction); resetting just the 16 explicit units to their 01 activations lowers P(>51) − P(≤51) from 0.42 to 0.33. Across prompts, projecting each unit's rank-1 write vector onto the 100 year tokens shows that 15 of the 16 units order the years almost monotonically from 00 to 99 (median |Spearman ρ| = 0.74, vs. 0.27 across all 2,304 L9–L11 units). The wiring matches [2]: all six heads that [2] finds feeding MLP8 keep their edges into MLP8, and L9.H1 has the largest head→logit attribution in the circuit (0.161).
  3. MLP9–10 make the cut at YY, with a division of labor. In the example, the model's two most likely years are "53" and "52", just above the start year. Across prompts, MLP9 and MLP10 write an upper-triangular logit pattern in (YY, Y') that boosts years > YY (+0.60 and +1.18 logits; Figure 46), as Hanna et al. [2] find in dense GPT-2. Within these layers the work is split: the 16 explicit units (0.7% of L9–L11 units) supply a start-year-dependent tilt toward later years and carry 13% of the late MLPs' clean-vs-corrupt shift (+0.28 vs. −0.01 for 16 random units), while the grouped L9/L10 others (752) nodes write the sharp cut at YY (+0.66 and +0.71). This is why Figure 39 keeps both the explicit units and the grouped nodes.

What the circuit shows.

  • The model computes greater-than by first copying the start year and then favoring later years; the explicit SWD units supply a start-year-dependent tilt toward later years, and the grouped late-layer units make the cut at YY.
  • Traced without any prior, the circuit recovers all seven start-year heads of Hanna et al. [2].

Thank you again for the detailed follow-up. We hope these additions address your remaining concerns, and we are happy to clarify anything further.

References

[1] H. Cunningham and T. Conerly, "Circuits Updates — June 2024," Transformer Circuits Thread, 2024.

[2] M. Hanna, O. Liu, and A. Variengien, "How does GPT-2 compute greater-than?," NeurIPS, 2023.

Reviewer_ECzmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose Sparse Weight Decomposition (SWD), a method to factorize LLM weight matrices as a product of two sparse factors (W≈ABW \approx AB). Since this problem is NP-hard in general, they employ the double-sparse factorization heuristic to obtain an acceptable factorization. This factorization has a nice interpretability property: each computation is a rank-one read-write path that can be manipulated independently.

The authors compare SWD to Transcoders, sparse pretrained models, and adVersarial Parameter Decomposition (VPD), measuring (a) change in cross-entropy and (b) circuit reconstruction cost-quality tradeoffs. They measure this in single-weight and full-model settings, and carry out ablations wherein they (a) use exact dense factorizations, and (b) carry out a data-free sparse factorization of model weights alone.

SWD matches baselines on reconstruction fidelity using <1% of the data. Under a fidelity-matched comparison, SWD can use fewer edges to hit the same sufficiency/necessity levels as the other baselines.

Critical strengths
  • As evaluated, SWD is competitive with baselines using less compute
  • I like that the authors provide and evaluate both data-free and calibrated variants of the work
  • The authors are honest and rigorous in their evaluations. They do not try to bury or sugarcoat results when they are unfavorable, and they set tough standards for their experiments
Critical weaknesses
  • The Transcoder baseline is misspecified and undertrained. As I understand it, a Transcoder is supposed to replace MLP(x)=σ(zWin+b)Wout→TC(x)=ReLU(zWenc+b)Wdec.MLP(x) = \sigma(z W_{in} + b) W_{out} \quad \to \quad TC(x) = ReLU(z W_{enc} + b) W_{dec}. This really only makes sense when replacing the entire MLP layer, not a single weight matrix. My read of Section 3.3 "each method replaces only one MLP matrix," plus the supplementary material code, suggests that the transcoder replaces just MLP-out, creating something like TCpaper(z)=ReLU(σ(zWin+b)Wenc)Wdec.TC_{paper}(z) = ReLU( \sigma(z W_{in} + b) W_{enc}) W_{dec}. This is not the typical way that transcoders are used! The Appendix G comparison is vastly more natural, and it shows more mixed results.
    • Additionally, the source code suggests the transcoder baseline is severely undertuned: for instance, the decoder norm isn't fixed, allowing the model to game the L1 penalty by shrinking activations rather than correctly sparsifying. The GPT-2 circuit checkpoint at 2,000 steps ≈\approx 2M tokens is severely undertrained.
  • Residual stream sparsity is a strong, undefendended assumption. See question 1 below.
  • I'm not convinced that Figure 7 fully spells out the greater-than SWD circuit: can you use attribution-like methods to understand the interaction between these different components outside of just "they each read and write to the residual stream"? The general standard is for circuit diagrams to do this, e.g. in [1]. Appendices L and M study inter-unit structure; this should be incorporated into the figure.
  • Figure 3 should be rearranged: necessity retention / sufficiency retention should be their own rows, with one column for (greater-than, IOI, docstring, gendered pronoun). This would significantly improve readability.

[1] https://arxiv.org/abs/2304.14997

Questions for authors
  • Please justify the use of representations that are sparse in the residual stream basis (e.g. an MLP-out decomposition writes to a subset of residual stream dimension; an MLP-in reads from a subset of residual stream dimensions, etc). The standard assumption is that the residual stream does not have a generally privileged basis.
    • Is it possible to relax your decomposition so that bottleneck units are sparse, but they still have dense units in the residual stream path? Naively, this would seem to be like (for W≈ABW \approx AB) having BB dense for MLP-out and AA dense for MLP-in layers.
    • What happens if you apply a random rotation to the residual stream before/after decomposition? If the residual stream really lacks a privileged basis, I would expect that you get similar sparsity:reconstruction tradeoffs using any rotation of the residual stream.
    • If helpful, you can see [1] for a sketch of why residual stream bases aren't privileged, and [2] for a discussion of some privileged bases that we do actually observe empirically in the residual stream
  • Please comment on the relevance of [3] to your work. Do you believe a direct comparison is feasible, like with VPD?

[1] https://transformer-circuits.pub/2022/toy_model/index.html#motivation-privileged

[2] https://www.anthropic.com/research/privileged-bases-in-the-transformer-residual-stream

[3] https://arxiv.org/pdf/2410.08417

3 comments

Thank you very much for your thoughtful review, encouraging feedback on our evaluation standards, and concrete suggestions.

Below, we use W1 for Weakness 1, Q1 for Question 1, etc.

W1 — Transcoder baseline specification and training.

  • Replacing the entire MLP layer: We agree that replacing the entire nonlinear MLP layer (instead of a single matrix) is the standard Transcoder setting, so we moved the Appendix G comparison into the main text (§3.4).
  • Training setup and results: Originally, we selected Transcoder checkpoints at the point where held-out replacement CE first matched SWD (Figure 2). Following your suggestion, we re-ran this entire-MLP Transcoder baseline with unit-norm decoder columns, three L1L_1 values, and longer training (33.6M tokens). Smaller L1L_1 lowers replacement ΔCE\Delta\text{CE} (with denser activations), while larger L1L_1 matches SWD-50% in held-out CE (ΔCE\Delta\text{CE} gap 0.00030.0003). Across all three L1L_1 values, Transcoders use fewer units on some sufficiency curves (such as Greater-Than), whereas SWD reaches larger necessity drops across all four tasks. Beyond these empirical results, SWD also has clear methodological advantages: it requires no auxiliary network training, uses far fewer tokens (16K calibration vs. 33.6M training tokens), and applies directly to any linear projection (including attention matrices).

W2 & Q1 — Justifying residual-stream sparsity, one-sided relaxation, and rotations. While we do not assume individual residual coordinates are monosemantic (§2.1, §5), using native residual coordinates is natural for an in-place factorization: every existing projection and LayerNorm is already parameterized in these coordinates. Sharing the intuition of weight-sparse pretraining (Gao et al., 2025 [5])—which sparsifies weights directly in the native residual basis—SWD factors W≈ABW\approx AB in place and shares the sparsity budget across both factors. Testing your two controls at 50% sparsity (Appendix H.3) supports this:

  • Random rotations: Three random orthogonal residual rotations on layer-8 mlp.c_proj give similar, slightly higher ΔCE\Delta\text{CE} (0.00189–0.002020.00189\text{--}0.00202 vs. 0.001590.00159 native). Thus SWD achieves comparable tradeoffs across bases without needing a strong semantic basis, while native coordinates remain slightly more amenable (consistent with your second reference).
  • One-sided dense relaxation: SWD naturally supports this variant by fixing the residual-side mask to all ones. Under a matched total nonzero budget, however, keeping the residual factor dense crowds all zeros onto the neuron-side factor, increasing layer-8 MLP replacement error (ΔCE=0.0070\Delta\text{CE}=0.0070 one-sided vs. 0.00420.0042 two-sided) while losing sparse residual edges.

W3 — Attribution-based circuit diagram for Figure 7. Following your pointer to attribution-based circuit diagrams [1], we replaced Figure 7 with an end-to-end Greater-Than circuit on SWD-FT GPT-2 traced via gradient-based EAP-IG [2, 3] across attributed c_proj SWD units in MLP8–11, attention heads, and earlier MLPs (Figure 7 in §3.8; full version in Appendix M). The resulting 377-edge circuit includes 16 individual SWD units across L9–L11 (5 in L9, 6 in L10, and 5 in L11) that connect upstream attention heads and earlier MLPs to downstream MLPs and the readout, achieving 0.87 faithfulness across 512 held-out prompts on 8 unseen templates.

Q2 — Relevance and comparison to Pearce et al. [4]. We appreciate the pointer to [4], now added to the main Related Work (§4). Methodologically, [4] requires bilinear MLPs without elementwise activations, whereas SWD applies directly to standard GELU transformers. Empirically, we compared SWD against an extension of their §3.2 eigendecomposition on their TinyStories bilinear checkpoint (layer 4) at matched stored values (Appendix O): SWD stays much closer to the original model's output distribution (KL 0.003–0.1410.003\text{--}0.141 vs. 0.42–0.480.42\text{--}0.48), faithfully preserving the original computation.

W4 — Figure 3 layout. We rearranged Figure 3 into 2 rows (sufficiency/necessity) ×\times 4 columns (tasks) and enlarged figure text throughout.

Thank you again—your suggestions substantially strengthened both our baselines and our analysis, and we welcome any further questions.

References:

[1] Conmy et al. Towards Automated Circuit Discovery for Mechanistic Interpretability. NeurIPS 2023.

[2] Syed et al. Attribution Patching Outperforms Automated Circuit Discovery. BlackboxNLP 2024.

[3] Hanna et al. Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms. COLM 2024.

[4] Pearce et al. Bilinear MLPs enable weight-based mechanistic interpretability. ICLR 2025.

[5] Gao et al. Weight-sparse transformers have interpretable circuits. arXiv:2511.13653, 2025.

I am satisfied with this rebuttal and will raise my score to a 3. My rationale:

  • The transcoder comparison, which was my main objection, has been fixed. While I would argue this does not demonstrate the superiority of SWD over transcoders, I appreciate the authors reporting the mixed outcome and see the value of this method.
  • The authors have convinced me that they have thought about residual stream sparsity, and have made me aware of a valuable connection to weight-sparse pretraining. I did not know this method was sparse in the residual stream basis. While I have some concerns that the residual stream dimensions will be substantially more privileged/monosemantic under sparse pretraining versus sparse post-hoc factorization, their random rotation and one-sided sparse relaxation experiments are convincing (and surprising!) evidence that (a) SWD works in all kinds of bases, but (b) the native basis is still modestly better. As such, I now consider my objection a limitation of the paper, not a fundamental blocker.
  • I am satisfied with the paper edits: Figures 3 and 7 look good, and the transcoder discussion feels much more grounded in the way transcoders are actually used.

Thank you very much for carefully reviewing our updates. We truly appreciate your constructive feedback and guidance, which helped us significantly strengthen the paper!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Discussion and Manuscript Revisions

We thank the reviewers for their engagement and summarize the discussion and revisions below.

Strengths noted by the reviewers. SWD offers "a cheap, post-hoc route to weight-sparse circuits" from existing dense checkpoints (1nfS), with higher replacement fidelity than Transcoders so explanations are "more likely to reflect the model's actual internal mechanisms" (1smL) at a fraction of the compute and data (ECzm, 1smL, 1nfS). Reviewers also highlighted the controlled comparisons at matched CE (1nfS), the zero-data variant (ECzm), and that the evaluations are "honest and rigorous" (ECzm).

Main concerns, new results, and current status:

  • Full-MLP Transcoder baseline (ECzm, 1smL): Moved to §3.4 and added 33.6M-token unit-norm Transcoders across three ℓ1\ell_1 penalties at matched CE (Appendix G). Using <1% of the tokens (16K calibration tokens), SWD remains competitive on active edges and achieves larger necessity drops on all four tasks, while Transcoders use fewer units on several sufficiency curves. ECzm noted that their main objection "has been fixed" and that they "appreciate the authors reporting the mixed outcome and see the value of this method."
  • Residual-basis sparsity (ECzm): Tested random orthogonal residual rotations and a one-sided dense relaxation (Appendix H.3): SWD maintains comparable fidelity across rotated bases, while two-sided sparsity lowers CE increase by 40% relative to one-sided sparsity (0.0042 vs. 0.0070). ECzm found this "convincing (and surprising!) evidence" and now considers the issue "a limitation of the paper, not a fundamental blocker."
  • Unit-level circuit interactions and presentation (ECzm, 1smL): Traced a unit-level Greater-Than circuit (0.87 held-out faithfulness with 16 explicit SWD units), quantified cross-component interactions via joint ablations (§3.8, Appendices M, N.3, Figures 7, 39, 44), and redrew Figures 3 and 7. ECzm noted "Figures 3 and 7 look good," and 1smL appreciated Appendix M and Figure 44.
  • Same-recipe sparse pretraining comparison and related work (1nfS): Pretrained a dense GPT-2 with the exact same corpus, architecture, and token budget as the sparse-pretrained reference and applied SWD post hoc (Appendix F), achieving comparable circuit sufficiency and necessity—which 1nfS confirmed "addresses one of my concerns." Related Work (§4) now covers further parameter-space methods, with a bilinear-MLP comparison (Appendix O).
  • Unit complexity scores and mechanistic reading of Figure 39 (1smL, Round 2): Added (1) a blinded evaluation following Cunningham & Conerly [1] against Transcoders (Appendix P, Table 14), showing SWD matches Transcoders in semantic Complexity (2.81 vs. 2.87 on a 1–5 scale, ~90% at Level 3, matching the 2.6–2.9 SAE range) without expanding model width (768 in-place units vs. 24,576 Transcoder features); and (2) a step-by-step walkthrough of Figure 39 (Appendix Q, Figures 45–46), showing that, without using Hanna et al. [2] as a prior, the circuit recovers all seven of their start-year heads (p=3.9×10−4p = 3.9\times10^{-4}), with the 16 explicit units acting as start-year-dependent biases toward later years (median ∣ρ∣=0.74|\rho| = 0.74, carrying 13% of the late-MLP shift from 0.7% of units) while the sharp cutoff at the start year is distributed over many units.
  • Sparsity vs. fidelity and calibration stability (1nfS): Equal-budget sparse SVD/Random-B controls (§3.6, Appendix H.2, Table 11) show that while sparse supports bound active edges per unit, SWD's optimization keeps those supports faithful to the original model (2.62.6–7.5×7.5\times lower CE increase than equal-budget controls). Both are essential, since a compact circuit is only meaningful on a faithful replacement; we will make this explicit in the Abstract. Across disjoint calibration splits, matched task units keep 0.82–0.83 activation correlation (Appendix H.4), and zero-data SWD (G=I\mathbf{G}=\mathbf{I}, §3.7, Appendix I) requires no calibration data.

All revisions are highlighted in the updated PDF (blue: Round 1; yellow: Round 2).

References

[1] H. Cunningham and T. Conerly, "Circuits Updates — June 2024," Transformer Circuits Thread, 2024.

[2] M. Hanna, O. Liu, and A. Variengien, "How does GPT-2 compute greater-than?," NeurIPS, 2023.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.