acceptodds
Under review as a conference paper at ICLR 2027

Weights Read and Write Features: Scalable Parameter Decomposition Grounded in Activation Space

Abstract

Activation space and parameter space provide complementary views of model computation. Activations represent information, while weights read, transform, and write that information. Yet existing interpretability methods largely study the two spaces separately, leaving the connection between represented information and parameter-level computation underexplored. We introduce Activation-Supported Parameter Decomposition (ASPD), which jointly decomposes activation and parameter spaces and grounds each learned weight component in the activation features it reads or writes. This grounding constrains otherwise non-unique parameter decompositions using the model's internal activations, while an internal reconstruction objective provides a local learning signal at the weight matrix being analyzed. Together, these properties enable scalable, interpretable, and causally editable parameter decomposition in pretrained large language models, demonstrated on Qwen-3-8B. The learned read–write components can also be composed into parameter-level mechanism circuits. We use ASPD to recover mechanisms underlying the classic IOI circuit and trace semantic transformations through model weights.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic Interpretability, Parameter Decomposition, Sparse Autoencoder, Language Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_22HwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to bridge activation-space and parameter-space interpretability, which is indeed a major challenge currently facing mechanistic interpretability. The proposed approach represents weight matrices as a collection of rank-1 matrices, with the number of components set to equal the number of SAE features, thereby enforcing a one-to-one correspondence between weight functions and individual features

Critical strengths
  1. The proposed method offers valuable insights into the problem of combining activations and weights.

  2. The interaction formulation, Interact(c1,c2)=EX,t[gt,c2(X)⋅et,c1]⋅⟨uc1,vc2⟩\text{Interact}(c_1, c_2) = \mathbb{E}_{X,t} [g_{t,c_2}(X) \cdot e_{t,c_1}] \cdot \langle u_{c_1}, v_{c_2} \rangle, is quite interesting and nicely captures the intersection of semantic relevance and algebraic alignment.

  3. The empirical evaluation is extensive and genuinely showcases the performance of the method.

Critical weaknesses
  1. While I acknowledge that this work targets a central challenge in mechanistic interpretability and brings useful insights, methods relying on SAEs and VPD suffer from too many practical limitations: for example, SAEs neglect the geometric structure among features, are highly sensitive to feature count and sparsity hyperparameters, treat features as an independent "bag of words", and produce many weak, long-tail features. On top of that, the authors force the number of components to match the number of features. Most of these assumptions lean heavily on the strong linear representation hypothesis, which has already been challenged or falsified, leaving notable theoretical gaps[1, 2].

  2. The reason current mechanistic interpretability seeks to link weights and activations is to better pinpoint which weights are critical. However, the proposed method does not directly answer this question: mapping weights into a low-rank space still lacks a direct mapping back to the native parameter matrices themselves.

That said, the paper still holds undeniable merit and provides a solid reference for exploring the activation-weight interface. It is just that the two weaknesses above make its real contribution somewhat narrow, largely serving experimental research and exploration only. Therefore, I lean toward a Weak Accept and believe this kind of effort deserves encouragement.

Ref: [1]. Interpreting neural networks through the polytope lens [2]. Recurrent neural networks learn to store and generate sequences using non-linear representations

Questions for authors
  1. Could you show how much task performance can be recovered using only the components identified by this method? This would further substantiate the practical contribution of the paper, though it might require designing more task datasets.

  2. Could you provide an analysis of the relationship across different total numbers of components and different sparsity levels (total number of features)? This directly relates to how severely the paper's underlying assumptions affect the results.

Reviewer_SyVXOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ASPD, which links features in model activations to rank-one weight components. A shared set of features selects these components and helps reconstruct the matrix output. Experiments test selected matrices in GPT-2, Gemma-2-2B, and Qwen-3-8B. A decomposition of all 72 GPT-2 attention and MLP matrices supports examples of indirect object identification (IOI) and semantic transformations.

Critical strengths
  1. The method connects parameter decomposition with feature-circuit analysis. It gives a practical way to link an interpretable feature to a weight edit. The tests cover both attention and MLP matrices.
  2. The evaluation uses a separate output feature dictionary, rather than judging edits only with ASPD's own features. Direct weight edits and the VPD controls in Appendices D–F add support to the interpretability results.
Critical weaknesses
  1. The paper does not report held-out reconstruction error or the change in model loss after replacement. ASPD removes constraints used by VPD to preserve the original computation. Better interpretability could therefore come at the cost of a less accurate model approximation. The controls in Appendix D do not answer this question.
  2. Equation (30) connects MLP input and output components through an inner product, despite the nonlinearity between them. Appendix G.1 also ignores LayerNorm in cross-layer scores. The IOI edits test selected components, but do not directly test the inferred connections or the semantic chain in Figure 3.
Questions for authors
  1. At matched sparsity, what are the reconstruction errors and model-loss changes, including when all GPT-2 decompositions are used together?
  2. Do the interaction rankings agree with direct interventions that include nonlinearities and LayerNorm?
Reviewer_z243OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors propose Activation-Supported Parameter Decomposition (ASPD): a method to jointly decompose model weights and activations. Similar to Transcoders, ASPD can be thought of as a sum of many rank-1 "read-write" outer products, sparsified by a participation score. The authors evaluate the interpretability, diversity, and meaning localization of ASPD against Transcoders and VPD, with somewhat favorable results. They also demonstrate that weight editing specific features has good localization, as measured by attribution patching on independently-trained SAE features. Finally, the authors demonstrate ASPD to analyze the IOI circuit.

Critical strengths
  • This problem is nicely grounded in model weights. I think understanding the interplay between weights and activations is valuable, and showing how Transcoders fit into this framework is nice. This formulation enables a repertoire of rank-1 edits to model weights a la [1], which I like.
    • I also suspect that framing Transcoders as parameter decomposition may open the gate for SAE-style techniques like batch-top-K, etc., to be applied to parameter decomposition. This would be an interesting consequence of this work in its own right.
  • The experiments are carried out carefully and rigorously

[1] https://arxiv.org/abs/2406.11717

Critical weaknesses
  • As written, this isn't a real parameter decomposition. Since LASPD\mathcal{L}_\text{ASPD} does not include Lparam\mathcal{L}_\text{param}, and the reconstruction loss of ∑PC\sum P_C seems to not be reported, it's not really clear to me that we get a good parameter decomposition here.
  • It's not clear to me how this is better than a regular Transcoder. Most of the results are fairly inconclusive re: ASPD vs PD Transcoders, and there is no comparison to vanilla Transcoders.
  • Basic fidelity metrics are not reported, e.g. ΔCE\Delta CE or KL divergence when using the top-K components of the decomposition
  • The activation-weights distinction feels oversold: in paragraph 2 of the intro, each class of methods not looking at the other is stated as a key limitation, which this paper claims to solve. This is basically the same point made 3 times, and given my first bullet I'm not convinced this successfully bridges the weights-activations gap either.
  • Some of the experiments seem questionable from a design standpoint. I would like to be convinced of the soundness of a few of these choices:
    • The use of Jaccard similarity as a diversity metric in Table 1, particularly since the values are so low.
    • The use of SAEs + attribution patching. What does this buy us over simply checking the dot-product of a feature's output with WencW_{enc} of the SAE?
    • The choice of IOI seems to restrict results to attention; it would be good to see this extended to more MLP-heavy tasks like greater-than [1]

[1] https://arxiv.org/abs/2305.00586

Questions for authors

Do you have any theories as to why the results between ASPD / PD Transcoders are so inconsistent in Table 3?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.