- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies the decoder geometry of SAEs trained on the 12 layers of GPT-2 small through a PCA analysis of the decoder. It constructs a 2-D plane from a layer pairs' principal components and defines a rotation on that plane. The authors claim that they demonstrate a clear rotational structure through the layers of the transformer, and that this can reduce L0 and dead features when training SAEs on consecutive layers.
Critical strengths
None.
Critical weaknesses
The claim of a shared rotation mechanism is unsubstantiated by the evidence presented. Figure 2(a) demonstrates a gradual change in dictionary covariance, without demonstrating that this is due to a common rotation. Furthermore, similarity after projection onto a common plane does not demonstrate that the original vectors lie close to the plane. A clearly defined angular trajectory is required, with reconstruction errors under that proposed model.
The SAE training sparsity results in 4(a) demonstrate the value of transfer learning, not of rotation. It's possible that the number of dead latents is genuinely reduced, however I can't determine this from a single run, and reconstruction errors are omitted despite this being a critical metric for assessing SAE quality. This evaluation needs considerably more detail - see the papers on TopK / BatchTopK / Matryoshka SAEs for guidance.
Questions for authors
- Is there a clear angular trajectory and what are the reconstruction errors under that predictive model?
- Are there further evaluation results on the transferred and rotated SAEs?
- Can you expand on the connection to working memory? This seems entirely speculative without evidence of content and persistence across layers?