acceptodds
Under review as a conference paper at ICLR 2027

The Shape of Depth: Consistent Decoder Geometry in Sparse Autoencoders Across LLM Layers

Abstract

To disentangle the superposed semantic units inside large language models (LLMs), sparse autoencoders (SAEs) have become a go-to choice. Yet, they remain layer-wise snapshots that are blind to how representations evolve throughout the full forward pass to the next token. In this work, we interrogate the geometrical structure of SAE decoders across layers of GPT-2 and find a kernel of information that undergoes a consistent rotation from layer to layer – perhaps a form of LLM working memory. We thus show that the top activation directions of different layers exhibit a clear rotational trend in cosine similarity, and canonical correlation analysis of decoder structure further reveals that residual activation dimensions map into a persistent structure. To show practical benefit, we construct a plane spanned by the leading activation directions of two layers via Gram–Schmidt orthogonalization and derive a rotation operator on this plane. We control this rotation to applied activation-concept spaces during cross-layer transfer learning, which improves SAE training, and yields fewer dead SAE features by treating rotation angle as a tunable hyperparameter. These insights suggest that transformations of a geometric object persistent across layers can be monitored and leveraged, such as to enhance SAE training and performance.

Primary area:
interpretability and explainable AI
Keywords:
Explainable AI, Sparse Autoencoder, LLM, Mechanistic Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_oXgZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the decoder geometry of SAEs trained on the 12 layers of GPT-2 small through a PCA analysis of the decoder. It constructs a 2-D plane from a layer pairs' principal components and defines a rotation on that plane. The authors claim that they demonstrate a clear rotational structure through the layers of the transformer, and that this can reduce L0 and dead features when training SAEs on consecutive layers.

Critical strengths

None.

Critical weaknesses

The claim of a shared rotation mechanism is unsubstantiated by the evidence presented. Figure 2(a) demonstrates a gradual change in dictionary covariance, without demonstrating that this is due to a common rotation. Furthermore, similarity after projection onto a common plane does not demonstrate that the original vectors lie close to the plane. A clearly defined angular trajectory is required, with reconstruction errors under that proposed model.

The SAE training sparsity results in 4(a) demonstrate the value of transfer learning, not of rotation. It's possible that the number of dead latents is genuinely reduced, however I can't determine this from a single run, and reconstruction errors are omitted despite this being a critical metric for assessing SAE quality. This evaluation needs considerably more detail - see the papers on TopK / BatchTopK / Matryoshka SAEs for guidance.

Questions for authors
  1. Is there a clear angular trajectory and what are the reconstruction errors under that predictive model?
  2. Are there further evaluation results on the transferred and rotated SAEs?
  3. Can you expand on the connection to working memory? This seems entirely speculative without evidence of content and persistence across layers?
Reviewer_AFsSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the evolution of representations across layers of large language models (LLMs). The authors investigate whether the principal directions of SAE decoder weight matrices across GPT-2 layers exhibit a consistent rotational trend. They aim to identify persistent geometric structure across layers and use it to improve SAE training through layer-wise transfer learning. The paper constructs a plane spanned by the leading principal-component directions of SAE decoders from two different layers and defines a rotation operator on this plane. The authors claim that treating the rotation angle as a tunable hyperparameter during cross-layer SAE weight transfer can improve training and reduce the number of dead SAE features.The authors further employ Canonical Correlation Analysis (CCA) on SAE decoder PCA scores across layers and report high correlations, suggesting that a common representational structure persists across layers.

Critical strengths

None

Critical weaknesses
  1. Limited experimental validation: Figure 4a suggests that the proposed rotation does not improve L0 sparsity over ordinary transfer learning, as the two curves almost completely overlap. While Figures 4b and 5b show improvements in the number of dead SAE features, these results are not sufficient to establish a general improvement in SAE performance. Moreover, the experiments are not exhaustive. The authors only experiment with GPT-2-small, which is relatively small compared to modern LLMs. The authors should consider experimenting with larger models, additional datasets, and multiple random seeds, and provide systematic ablations across rotation angles and other relevant hyperparameters. 2)The results are being overclaimed as a solution to the problem proposed , cosine similarity across principal components is not an accurate measure of rotational relation between layers. The authors neither quantify how much of the original directional information is preserved under projection nor demonstrate that their proposed rotation predicts transformations between consecutive layers
Questions for authors
  1. I would love to see results on bigger models and hope that authors show better ablations, as mentioned in the weaknesses
  2. I request the authors to show experiments that better supports the claim of there being a rotational relation between SAE decoder layers
  3. I would like to know what emperical evidence according to the authors leads to the speculation of a form of working memory and how according to them there tool can lead to circuit discovery
Reviewer_RZSnOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Through this paper, the authors study how SAE decoder geometry changes across layers of GPT-2-small. They do this by performing PCA on decoder weights from different layers and then comparing the cosine similarity of the leading principal components. They observe a pattern in these similarities and interpret it as evidence of consistent rotations across layers. They also use CCA to study similarities between decoder representations. Based on these observations, they construct a 2d plane using principal components from different layers and define a rotation operator. They then apply this rotation to SAE weights during cross-layer transfer learning. The main practical result is that certain rotation angles reduce the number of dead SAE features compared to random initialization and standard transfer learning.

Critical strengths
  1. I think studying SAE decoder geometry across layers is an interesting question. Most SAE analysis looks at individual layers, and understanding how representations change across layers is important. I liked that the authors tried to look at this from a geometric perspective rather than just matching individual features.

  2. I acknowledge that the paper tries to use the geometric observations for something practical. The idea of using cross-layer information to initialize SAEs is reasonable. The random-plane comparison in Figure 5 is also useful, although I have concerns about how the results are interpreted.

Critical weaknesses
  1. I am not convinced that the experiments actually establish consistent rotations across layers. This is my biggest concern with the paper. The authors compute the cosine similarity between principal components of SAE decoders and observe a pattern across layers. But I don’t understand why this is enough to conclude that there is a uniform rotation of information. For example, Figure 2 compares principal components from different layers against layer 6. Since layer 6 is the reference, we naturally expect the similarity to be highest there. The observed pattern could simply indicate that nearby layers have more similar decoder directions. Also, PCA directions are only defined up to sign, and the paper uses absolute cosine similarities. So I am not sure how the authors infer a particular rotation direction from these measurements. The CCA results in Figure 3 also show that the decoder representations share some structure, but I don’t think this indicates that information is being preserved or rotated across layers. Since the residual stream uses the same coordinate space across layers, some correspondence is expected.

  2. I would have liked to see stronger baselines and an analysis that actually distinguishes consistent rotations from general representational similarity. The connection to working memory also seems quite speculative. I have serious concerns about the experimental evaluation and the claimed actual benefits. The paper claims that applying these rotations improves SAE training. But in Section 4, the authors explicitly state that they do not report reconstruction errors because they did not find significant differences. Instead, the evaluation mainly focuses on dead features and L0 sparsity. I don’t think these metrics alone are enough to establish better SAE performance. Fewer dead features do not necessarily mean better representations, and lower L0 is not always better unless we also consider reconstruction quality. Another major concern is that the authors use the same data for training and evaluation. There is no held-out evaluation.

  3. The experiments are also quite limited. Most of the analysis uses GPT-2-small, and I don’t see sufficient experiments over independently trained SAEs or multiple random seeds to establish that the observed rotation patterns are stable. I think the paper needs a much stronger experimental evaluation before making claims about improved SAE training or persistent geometrical structure across layers. Overall, I don’t think the paper is ready for ICLR in its current form.

Questions for authors

See weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.