- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper aims to bridge activation-space and parameter-space interpretability, which is indeed a major challenge currently facing mechanistic interpretability. The proposed approach represents weight matrices as a collection of rank-1 matrices, with the number of components set to equal the number of SAE features, thereby enforcing a one-to-one correspondence between weight functions and individual features
Critical strengths
-
The proposed method offers valuable insights into the problem of combining activations and weights.
-
The interaction formulation, , is quite interesting and nicely captures the intersection of semantic relevance and algebraic alignment.
-
The empirical evaluation is extensive and genuinely showcases the performance of the method.
Critical weaknesses
-
While I acknowledge that this work targets a central challenge in mechanistic interpretability and brings useful insights, methods relying on SAEs and VPD suffer from too many practical limitations: for example, SAEs neglect the geometric structure among features, are highly sensitive to feature count and sparsity hyperparameters, treat features as an independent "bag of words", and produce many weak, long-tail features. On top of that, the authors force the number of components to match the number of features. Most of these assumptions lean heavily on the strong linear representation hypothesis, which has already been challenged or falsified, leaving notable theoretical gaps[1, 2].
-
The reason current mechanistic interpretability seeks to link weights and activations is to better pinpoint which weights are critical. However, the proposed method does not directly answer this question: mapping weights into a low-rank space still lacks a direct mapping back to the native parameter matrices themselves.
That said, the paper still holds undeniable merit and provides a solid reference for exploring the activation-weight interface. It is just that the two weaknesses above make its real contribution somewhat narrow, largely serving experimental research and exploration only. Therefore, I lean toward a Weak Accept and believe this kind of effort deserves encouragement.
Ref: [1]. Interpreting neural networks through the polytope lens [2]. Recurrent neural networks learn to store and generate sequences using non-linear representations
Questions for authors
-
Could you show how much task performance can be recovered using only the components identified by this method? This would further substantiate the practical contribution of the paper, though it might require designing more task datasets.
-
Could you provide an analysis of the relationship across different total numbers of components and different sparsity levels (total number of features)? This directly relates to how severely the paper's underlying assumptions affect the results.