- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper is broadly concerned with whether sparse autoencoders (SAEs) learn a canonical set of dictionary atoms when there exist feature manifolds in activations, under different distributions that activations may be sampled from on those manifolds. The authors mostly consider activations in a 2d plane, and are interested in the conditions under which there is a unique optimal set of dictionary directions under different distributions that activations are sampled from within that plane . In particular though, they study a more narrow question, which is: for a fixed set of dictionary directions, whether rotating those directions by changes the dictionary learning loss. They find a particular criterion for when rotations by won't change the loss, which is defined in terms of both and the arrangement of the dictionary atoms. Interestingly, this criterion says that the activations need not be isotropic ( constant) for the dictionary learning loss to not depend on the angle of the atoms. They do some experiments training SAEs on synthetic data to show this analysis applies to real SAEs. My overall takeaway from the paper is that the manner in which sparse autoencoders (or dictionary learning algorithms generally) "tile" feature manifolds depends on the particular way that activation vectors are distributed on that manifold in a somewhat nuanced manner.
Critical strengths
- The broader question motivating this work, of how sparse autoencoders behave when there exist feature manifolds in activations, is interesting and topical.
- I appreciate the mathematical approach that the authors take to this question. They define a clear setup and prove a result that was cool/surprising to me about how dictionary learning behaves in the presence of manifolds (though see caveats below).
Critical weaknesses
- Overall, I found the paper to be very difficult to read in a manner that I think is characteristic of AI systems being used extensively in the writing process. The abstract and intro are quite dense. The whole paper is also extremely long. I know that these are not grounds for rejection since AI use was properly disclosed by the authors, but I feel I must point out that this has substantially degraded my experience reading the paper. I had to rely extensively on AI systems myself in order to understand any of the paper's results and contributions. I really wish that the authors had spent more time on the writing/presentation of their results, and overall I recommend clear rejection on this basis. I will not describe all places where the paper is unclear, but I'll highlight some phrases that are blant, below. These all evoke jargon that is not defined yet and so are meaningless to readers when they are read:
- "agreement at an installed harmonic (strongest in 28 of 35 Gemma-2-2B tilts at q≥2 no symmetry explains; chance 8.8"
- "cross-seed jitter tracks the density’s circular moment over an 8.6× jitter range"
- "has its objective deepen from flat to pinned as that symmetry is broken, in the representation (5/5 defect levels) and by dose (3/3; seeds unresolved)"
- "Trained seeds lock when the density says."
- The authors prove a relatively narrow result about whether dictionaries are canonical in the presence of feature manifolds. In particular, they show that for a fixed dictionary (in 2d), rotating those atoms doesn't change the dictionary learning loss exactly when a certain criterion is met that has to do with the relationship between the geometry of the dictionary atoms and the probability density of the activations . Their result implies that there are some dictionaries where even nonisotropic distributions over activations achieve the same dictionary learning loss (reconstruction + sparsity) no matter the orientation of those dictionary atoms. This is cool! However, if I understand correctly, in general the dictionaries that have this property are not the dictionaries that actually minimize the dictionary learning loss when activations are distributed in this way. So an optimal solution from something like an SAE might actually be canonical, even though this symmetry does indeed exist in the space of dictionaries. The authors acknowledge this on line 059, although I also find the language they use for this caveat to be inscrutable "a flat rotation at an optimum is a continuum of optima, and with free atoms we found one only under isotropy (for L1) and for TopK at k ≥ 2..." In simple terms, this means that the only optimal solutions for a dictionary learner that are genuinely rotation-invariant (and thus non-canonical) are when the activation distribution is indeed isotropic, which I think makes the paper's claims a bit less interesting.
Questions for authors
- Have I correctly understood the nature of your most important contributions, as you see them? There are other results in the paper that are more empirical, including ablations on the days of the week representation. I'd be happy to have some discussion of these if you think they should change my outlook on the paper.
- Related to the above, in your own words, what do you think are the most important takeaways from the paper? What should the interpretability community learn from your work? What have you learned from doing this work? Please write this yourself in plain language.