- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Problem: This paper identifies a challenge of TopK SAE training where the sparse features learned from such a model are not rotation invariant. The paper identifies the parameter-wise preconditioning of optimizers such (e.g. AdamW) as the cause for breaking this invariance. The paper begins by discussing simple cases such as signed permutations which TopK SAE learning is invariant to before expanding to dense rotations and non-orthogonal transformations.
Theory: Definition 1 introduces the invariance condition that the paper argues is desirable. The propositions in the paper highlight the main discussion surrounding when an update rule / feature transform is invariant against the other. They build cleanly towards Proposition 5 which introduces a variant of Adam satisfying the invariance described in Definition 1.
Experiment: Using a frozen dataset of 55,000 activations from GPT-2, a set of transformations ranging from signed permutation to the DCT-II transform.
Takeaway:
Critical strengths
Well-written paper, with that introduces and works on a minimal version of a problem. Many machine learning papers are written in a manner that makes it hard to read from beginning to end. I enjoyed that I did not have to strategically read sections out of order to understand the main idea of the paper. Further, I appreciate that focus of the paper was reduced to studying the geometry of the updates on the TopK SAEs without attempting to also build a full interpretability suite on top - this made it easy to focus on what was important. That being said, in the questions section below, I do request some more discussion about what the Author imagines the impact of this work on existing interpretability pipelines may be.
An principled angle to improving a simple challenge of TopK SAEs. From my cursory read of the field, it seems that invariance under simple (orthogonal) transformations of the activations is clearly something you would want out of an interpretability method - otherwise it is hard to argue that your features are a only property of the model being interpreted and not the SAE. That being said I am not sure about the related work, novelty of the problem, and other approaches that may already exist in the literature.
Critical weaknesses
I am not an expert in this field, but the results in this paper are quite simple, and it is not clear what their impact is. It is also not clear from reading the main body alone that this problem is unaddressed in the literature. Interpretability is topical both because of the (supposed) importance for AI safety and the funding being allocated to it. It would be valuable to connect the insight from this paper to the broader state of interpretability research here. It would also be nice to see a larger result suite with a more intensive tweezing apart of when the feature / decoder / reconstruction error misalignment impacts common interpretability performance metrics.
Questions for authors
- What is the empirical takeaway having run experiments that is not obvious from the propositions alone? It would be nice to highlight this. In this same question, I'd request that you clean up the Results section or even create a separate Experimental Setup and Results section, so that it is clear to the reader what information you are seeking with the experiments and why they may meaningfully impacts the takeaway of this paper.
- The lack of communication of the significance of the claim / relationship to the rest of the field leads me to lean weak reject, but I am happy to amend if I see more justification for why this is an important result is presented.
- I think it is fine if this problem has been introduced / approached in the literature before, but I'd then want to know what makes the result here important to publish.
- I understand that this paper is focused on the impact of update geometry on the learned sparse features, but have you run anything to see if features align beyond the Hungarian matching? What do you hypothesize the effect would be? As a reader not familiar with the area deeply, this would be good to know.