acceptodds
Under review as a conference paper at ICLR 2027

How Vision Transformers Represent Position: A Mechanistic Comparison of Absolute and Rotary Positional Encodings

Abstract

Positional embeddings encode the spatial position of each image patch in a Vision Transformer (ViT), allowing the model to distinguish different spatial arrangements of visual content. However, we know little about how the choice of embedding shapes a vision transformer's internal representation. In this work, we show that Absolute Positional Embeddings (APE) and Rotary Positional Embeddings (RoPE) encode spatial position through different internal mechanisms: APE stores position in explicit features, while RoPE builds it inside attention and rebuilds it when it is withheld. Specifically, in APE models, sparse autoencoders recover causal row- and column-specific features, whereas interventions on RoPE reveal distinct rotary components that separately encode row and column information. Additionally, causal interventions selectively disabling RoPE's row or column rotations show that decodable row and column information can be independently removed and, once rotations are restored, partly reconstructed, in part by axis-specialized attention heads. Both mechanisms are causally important for classification: permuting APE's positional features across patches reduces top-1 accuracy by 24–31 points, and ablating the rotations of RoPE's primary rebuilding head reduces it more than ablating those of any other head. Our findings show that the choice of positional encoding fundamentally changes how spatial information is organized and processed inside Vision Transformers. More broadly, our work demonstrates how mechanistic analysis can reveal how models construct, reconstruct, and maintain structured representations, providing a basis for targeted interventions in vision models.

Primary area:
interpretability and explainable AI
Keywords:
Vision Transformers, Mechanistic Interpretability, Representation Learning, Sparse Autoencoders, Positional Encodings, Causal Interventions

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_91UEOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper investigates how Vision Transformers represent spatial position under learned absolute positional embeddings (APE) and 2D rotary positional encodings (RoPE). The authors use similarity–distance correlation (SSDC) to quantify the relationship between hidden-state similarity and spatial distance, along with a random-permutation protocol (RPI) that shuffles patch content while preserving positional information. These analyses are complemented by linear probes, sparse autoencoders (SAEs), and reversible interventions that selectively remove RoPE rotations across axes, blocks, and heads. The results suggest that APE encodes positional information through a small number of row and column coordinate features in the residual stream, which cannot be recovered by later layers once removed. In contrast, RoPE models reconstruct positional information within attention layers, recover spatial structure after early rotations are disabled, and exhibit heads associated with axis-specific position reconstruction. The authors further demonstrate that these positional mechanisms have causal effects on ImageNet classification beyond matched control interventions.

Critical strengths
  1. The paper addresses an interesting and well-motivated question. Although ViTs have increasingly adopted RoPE over APE for accuracy and extrapolation benefits, the underlying differences in how these positional encoding schemes represent spatial information remain insufficiently understood.

  2. The analysis goes beyond correlational probing by introducing causal interventions. Selective rotation masking provides an exact and reversible way to manipulate positional information in RoPE models, while SAE feature ablations allow the authors to directly investigate positional representations in APE models.

  3. The experimental controls are carefully designed. RPI reduces content-related confounding, random SAE features are matched by activation frequency and magnitude, and patching experiments include appropriate alignment controls. Adding back SAE reconstruction errors also ensures that the unmodified intervention pipeline reproduces the original model.

  4. The main findings are supported by several compelling quantitative results. Ablating approximately 20 APE features reduces exact-position probe accuracy from 99.7% to 22.9%, while RoPE models recover normalized SSDC to 0.50–0.71 after early rotations are removed. Relocating APE features among patches within the same image also causes a 24–31 point reduction in top-1 accuracy.

  5. The experiments examine multiple models and training settings, including supervised, DINO, AugReg, SAM, naver, DINOv3, and a hybrid architecture. The paper also provides detailed implementation and reproducibility information.

Critical weaknesses
  1. The comparison between APE and RoPE is complicated by the use of different intervention mechanisms. APE is analyzed through SAE feature ablation, whereas RoPE is studied through rotation masking, making it difficult to determine whether the observed "stored versus recomputed" distinction arises entirely from the positional encodings rather than the intervention methods. The authors acknowledge that these interventions allow only qualitative comparisons, not direct comparisons of effect sizes. Furthermore, the inability of APE models to reconstruct position after feature removal is somewhat expected, since removing these features at block 2 eliminates the remaining positional information available to subsequent layers. Most comparisons are also confounded by differences in training configurations, with only DeiT-III and naver ViT-B sharing the same training recipe.

  2. The incremental novelty relative to prior work needs clearer discussion. The two primary measurement tools, SSDC and RPI, are adopted from Mannes (2026), which already investigated how positional encodings establish spatial structure in ViTs. The main new contributions appear to be the causal localization of positional mechanisms and their effects on downstream classification. The paper would benefit from a more explicit distinction between these new findings and the earlier analysis, as well as an examination of whether the conclusions depend on the particular SSDC/RPI measurements.

  3. The SAE-based findings rely on a relatively narrow experimental configuration. The analysis uses a single SAE with one layer, one width, and a heuristic procedure for selecting positional features. Moreover, the selected features do not capture all recoverable positional information: when SAE reconstruction errors are added back, exact-position probe accuracy still recovers to 57–93% after the selected features are removed. For RoPE models, simply passing representations through the SAE can change SSDC by as much as 0.214, suggesting that the absence of coordinate-like features in RoPE should be interpreted with greater caution.

  4. The evidence supporting axis-specialized rebuilder heads is not entirely convincing. The strongest head exceeds the second-ranked head by only around 0.8–1.1 accuracy points, while the three highest-ranked DINOv3 heads have attribution scores differing by less than approximately 0.004. The top head in naver ViT-S also shows no clear axis preference, and paired standard errors are not consistently provided across models. Additionally, a "head" in this analysis refers to the same head index across six blocks rather than an individual attention head in a specific block. These limitations make the identification and claimed axis specialization of rebuilder heads less conclusive.

  5. The experimental scope is relatively limited. The analysis focuses on ViT-B/S models at 224×224 resolution and evaluates downstream effects only through ImageNet classification, without experiments on dense prediction, larger architectures, or substantially higher resolutions. SSDC also becomes less sensitive at 320×320. Consequently, the connection between the observed positional mechanisms and the broader spatial-reasoning limitations motivating the study remains insufficiently established.

Questions for authors
  1. Could you investigate APE and RoPE using a shared intervention protocol that allows their causal effects to be compared quantitatively?

  2. What are the main findings that go beyond the earlier SSDC/RPI analysis? Do the conclusions remain consistent when using alternative measures of spatial structure?

  3. How robust are the identified APE positional features and the associated conclusions across different SAE widths, sparsity settings, layers, and random seeds?

  4. Do the identified rebuilder heads maintain their importance and axis specialization at higher image resolutions or on dense prediction tasks?

3 comments

Thank you for your detailed and careful review, and we are glad you found the research question well motivated. We addressed every weakness and question you raised, almost all with new experiments, and the revised PDF states what we did not test.

[W1/Q1] A shared intervention with effect sizes on one scale. With both encodings left on, we apply the same edit to both families: we subtract the residual stream's per-slot positional component (Appendix G.3). Erased at block 4, SSDC comes back by block 11 in all four RoPE models and in none of the five APE models, even though APE's embedding stays on, so APE's failure to rebuild no longer follows from removing its source. The block-4 edit costs the RoPE classifiers 0.06–0.30 top-1 points and the APE models 3.1–11.3. Applied before the attention and the MLP of every block, it costs RoPE 0.3–0.6 points (about 1% of removing every rotation) and APE 37.3–57.7 (89–95% of removing the embedding). Removing the encoding entirely costs both families about the same (45.8–57.3 and 42.1–61.1 points): both need position, but only APE depends on the per-slot copy in the residual stream. The recipe-matched pair splits the same way (naver ViT-B 0.20 ± 0.18 points, DeiT-III 11.32 ± 0.54). As the paper states, probes still read some position after the edit, and the block-4 comparison across models was chosen after the run.

[W2/Q2] Clarifying the novelty beyond SSDC and RPI. We reuse SSDC and RPI from Mannes (2026), and the contributions now say so. What this paper adds is causal. On the RoPE side, we remove rotations by axis, block and single head, show that later blocks rebuild position the early blocks never received, identify the heads that do it, and show that their rotations matter for classification beyond matched and block-matched random controls. On the APE side, we go beyond observing position-selective features by intervening on them and measuring their effect on spatial structure, on each axis and on classification. Both families are now compared under one intervention. To our knowledge, these interventions on RoPE ViTs have not been reported before. The conclusions do not rest on SSDC alone: probes give the axis results, top-1 the functional ones, and a new SAE-free positional-variance measure separates three of the four RoPE models from APE and shows that DINOv3 stores about as much position as APE. Section 3.2 now states which measure supports which claim.

[W3/Q3] Robustness of the SAE results. We trained three more APE SAEs (Top96-8×, Top96-5× and Top64-8×, alongside the original Top64-5×) and repeated the feature ablation: the identified features and their contribution to spatial structure hold in all four (Appendix D.8). We did not rerun the classification and control experiments on the new SAEs, which we state, and the sweep varies only width and k, at block 2. For RoPE we agree: the paper now claims only that these models show no comparably selective features, and that the SAE is less reliable there. The SAE-free measure gives 1.4–4.3% positional variance at block 4 in three RoPE models and 12.3% in DINOv3, against 9.4–18.0% in APE.

[W4] Re-examining the rebuilder heads. We agree, and the analysis now uses single heads (block, head) throughout. Since the top head is within error of the next on two classifiers (0.12 ± 0.27 and 0.26 ± 0.29 points), we now claim a few rebuilding heads rather than one. That claim does not rest on ties: SSDC cost predicts top-1 cost across heads (ρ = 0.55–0.67) and within blocks (0.35–0.50), and the six top heads cost 10.1–17.6 points against 4.1–6.8 on average for 48 random six-head sets from the same blocks, above all 48 on two models and 47 on reg1-gap. DINOv3's "within 0.004" came from grouping heads by index: as single heads, (6,2) leads by 0.013 SSDC in every bootstrap resample. On naver ViT-S, (7,4) is a clear row head and the costliest (6.08 ± 0.40 points, 2.48 ± 0.38 ahead of the next). Paired standard errors are now given for every classifier.

[W5/Q4] Extending the scope. We added 320×320 input for three RoPE models, naver ViT-L and Qwen2-VL's vision encoder (Appendices H.3–H.5), and the axis split and the rebuilding appear in all of them. At 320×320 the heads named in Section 4.5 stay the costliest and stay row heads, while reg1-gap rebuilds less (0.43). Single-head SSDC still tracks top-1 at 320 on reg1-gap, trained at 256, but much less on the two models trained at 224, so the head analysis ranks heads by accuracy. We have not tested dense prediction and say so.

We hope this addresses your concerns. Thank you again for the detailed review.

Thank you for your detailed response.

Some of the weaknessed I raised has been addressed. However, here are some remianing concerns:

  1. The erasure criterion is mostly not met and the block-4 comparison was chosen post hoc, so the small drop for RoPE may partly reflect incomplete erasure. Please report results where the criterion is met.

  2. The error-add-back test was not rerun on the new SAEs, so the 57–97% residual decodable position is still unexplained.

  3. Effects are weaker on DINOv3, Qwen2-VL and ViT-L, which suggests the mechanism depends on the training objective as well as the encoding type.

I am raising my soundness score to 2 and my rating to 2 (weak reject).

Thank you for the follow-up and for raising your scores. We answer the three remaining concerns below.

1. Results where the erasure criterion is met.

We agree that the block-4 check was not always met and that the block-4 comparison was chosen after the run, as the paper states. However, on net, the intervention substantially reduces the measured positional information even in cases where it does not meet the predefined erasure criterion: at the edited block, SSDC falls to the floor in all four RoPE models and to −0.01–0.28 in the APE models, while exact-position probe accuracy falls from 27–45% intact to 3.3–6.4% on RoPE and from 95.9–100% to 5.1–9.8% on APE (Table 26). Thus, although the criterion is not met, the intervention removes most of the positional information captured by our proxies at the edited state. The table lists the cases where the criterion is met, next to the every-block result on RoPE, where it mostly is not (Tables 26–28).

EditModelsCriterion metNormalised SSDC, block 11Top-1 cost (points)
Mean erasure, every block5 APE25 of 25 pointsAt most 0.0437.3–57.7
Subspace erasure, block 4naver ViT-B, ViT-SYes1.23, 1.089.86, 71.14
Mean erasure, every block4 RoPE2–5 of 24 points0.69–0.92 (DINOv3: 0.18)0.30–0.64

The criterion-met results support APE's dependence on stored position and show RoPE rebuilding after subspace erasure. Subspace erasure costs far more than a random subspace of the same size (0.32 and 1.28 points), so it removes more than position and we do not read its cost as the cost of position. The small RoPE cost does not have the same support. It comes from the row where the criterion is mostly unmet, so we agree that the 0.3–0.6 points may partly reflect incomplete erasure. What we can say is that this edit removes most decodable position on RoPE (exact probe 0.7–4.3%, against up to 27–45% intact) and takes SSDC to the floor at that cost, while the same edit costs APE 37.3–57.7 points.

2. The 57–97% that remains decodable.

You are right that we did not rerun the error-added-back test on the new SAEs, and Appendix D.8 reports as such. However, the existing SAEs provide some insight into the source of this residual decodability (Table 7). After the selected features are zeroed, the exact probe reads 5.9–45.3% without the SAE error and 57–97% with it, a gain of 48–79 points in every model. A large part of it therefore comes from what the SAE does not reconstruct, and the rest is carried by features outside the selected set. Due to time constraints, we have not established whether the same decomposition holds for the new SAE configurations.

3. Dependence on the training objective.

We agree that the strength of these effects depends on the training objective and model scale, not only on the positional encoding, and the paper discusses/reports this nuance (Section 5, Appendices H.3–H.5). However, although the mechanism manifests differently across DINOv3, Qwen2-VL and ViT-L, its qualitative features persist. Specifically, naver ViT-L shares naver ViT-B's training recipe, and its difference is one of size: its axes still separate and its later blocks still rebuild (normalised SSDC 0.54), with the rebuilding spread over many heads (illustrating the fundamental mechanism). Similarly, although Qwen2-VL and DINOv3 exhibit weaker effects, we still observe the same qualitative RoPE mechanism in both: positional information separates by axis when the rotations are decomposed, and later blocks rebuild spatial structure when the rotations are restored. Additionally, the pair that shares a recipe isolates the effect of the encoding. On net, the fundamental mechanism holds in spite of the results being somewhat affected by encoding and training, as is to be expected.

We hope this addresses the remaining concerns. Thank you again for your time.

Reviewer_NpkGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper compares how Vision Transformers represent patch position under absolute positional embeddings (APE) and rotary positional embeddings (RoPE), measuring spatial structure with SSDC under random permutation at inference and with linear probes. For APE, an SAE isolates a small set of row- and column-selective features. Removing them eliminates spatial structure without later recovery, and swapping them between patches degrades ImageNet classification. For RoPE, the authors replace rotation planes with the identity by axis, block or head, and show that each axis is carried by its own rotations and that later blocks rebuild much of the structure when rotations are withheld early, partly through axis-specialized heads. The authors conclude that APE stores position once in residual-stream features, whereas RoPE recomputes it inside attention at every block.

Critical strengths
  • [S1] Masking RoPE rotation planes with the identity is an exact and reversible intervention that requires no retraining. It allows position to be removed selectively by axis, block or head while leaving the rest of each head's computation intact.

  • [S2] The paper connects the identified mechanisms to downstream function. In particular, the swap experiment shows that classification in APE models depends on the position the features report, not merely on their presence.

  • [S3] The study covers a range of public checkpoints and is transparent about its protocol. It releases code and result files, reports failed implementation checks, and discusses its limitations openly.

Critical weaknesses

I have devoted considerable time to reviewing this paper carefully. My main concerns regarding the paper are as follows:

  • [W1] The central conclusion is largely predictable from the architectural definitions. Since APE adds a position vector to the residual stream once at the input, it is expected by design that positional information is carried in the residual stream. Also, since RoPE injects position only through query/key rotations at every block, it also follows directly from the definition that position is computed inside attention and reappears once the rotations are restored. The row/column separation in RoPE likewise results from axial RoPE assigning rotation planes to each axis, and the fact that the SSDC of RoPE models equals the floor at block 0 holds by definition, because no rotation has been applied yet. The paper presents this contrast as a new finding, but it does not specify which outcomes would have falsified its hypothesis. For example, RoPE models storing position in the residual stream during the early blocks and no longer relying on later rotations would be such a counterexample. I believe the contribution would become clearer if the authors stated such alternative hypotheses explicitly and brought to the fore the less predictable findings, such as the anchoring of absolute position at the image border and the dissociation between linear decodability and functional use.

  • [W2] Positional SAE features in APE ViTs have been reported in prior work, so the scope of the novelty should be adjusted. As far as I know, prior work has already reported that SAEs trained on the residual stream of supervised ViTs recover position-selective features that tile the image grid along rows and columns [1]. Moreover, that work also reported positional features with shapes other than rows and columns, and traced how they are composed across layers. SAE positional features corresponding to absolute positional embeddings have also been analyzed in language models. Therefore, the observational part of the first contribution ("a sparse autoencoder isolates a small set of row- and column-selective features that tile the image grid") is close to a confirmation of existing results. What appears new in this paper is the per-axis ablation and the finding that classification depends on the position these features report (Section 4.5). The paper should discuss its relation to this prior work and restate the contribution accordingly.

  • [W3] The axis separation in APE largely follows from how the features are selected. The screen selects, for each grid position, the feature with the highest row selectivity and the feature with the highest column selectivity. Therefore, if I understand correctly, a row feature is by definition one that varies only along rows and is nearly uniform across columns, so it carries little column information. That the column probe is preserved after zeroing such features would hold regardless of how the model stores position. For the claim that "rows and columns are stored separately" to be non-trivial, one would need to show that position is not also stored in conjunctive (row, column) features or in positional features of other shapes. However, since the screen only considers row and column selectivity, such features are excluded from the test. Given that positional features with shapes other than rows and columns have already been reported in APE ViTs [1], I consider this omission substantive.

  • [W4] The comparison between APE and RoPE is asymmetric in design. In the key comparison that "APE does not rebuild position whereas RoPE does," the two interventions differ in nature. In APE, the source of position (the positional features) is permanently removed at block 2, whereas in RoPE the source (the rotations) is removed temporarily and then restored. Since it is natural for position to return on the side where its source is restored, this comparison may reflect a difference in the interventions rather than in the two mechanisms. Two experiments could make the comparison symmetric. One is to keep the rotations in RoPE while erasing or scrambling the positional component of the residual stream at block 4, and test whether it reappears. The other is to foreground the result in APE that SSDC does not recover even when the SAE error is added back and the exact-position probe remains at 57–93% (Appendix D.5). The latter shows that APE does not rebuild the grid structure even from the positional information that remains, and in my view it is more convincing evidence than the comparison currently in the main text. In addition, Section 4.4 states that "APE writes position into the residual stream once and the network carries it forward," but Figure 2 and Table 6 show that the positional structure in APE decreases in later blocks (in supervised ViT-B, the exact-position probe drops from 100% to 8.1% at block 11, and SSDC drops to about 0.2). Since the model gradually discards position rather than maintaining it, this statement needs to be revised.

  • [W5] The evidence for the claim that RoPE models contain no coordinate-like features is weak. I judge that this claim rests on two pieces of evidence, neither of which is sufficient to support the conclusion. First, the authors themselves state that the SAE is not a reliable tool for RoPE models. In naver ViT-B, reconstruction alone shifts SSDC by up to 0.214, compared with 0.062–0.065 in APE models (Appendix D.7). Using an unreliable tool to show that features are absent is self-contradictory. The training configuration of the SAEs for RoPE models is also missing from Table 4. Second, the selectivity metric (the mean activation of the most active row divided by the mean over the grid) is high only for band-shaped features, so positional features that are periodic or based on relative position structurally receive low scores. Given that positional features with shapes other than bands have been reported even in APE ViTs [1], low scores under a band-based criterion (at most 3.4 in RoPE models and 1.8 in DINOv3) do not demonstrate the absence of coordinate-like features. The conclusion should be weakened, or supplemented with a shape-agnostic measure of positional selectivity (e.g., the fraction of activation variance explained by position) or with SAEs at sites with better reconstruction quality.

  • [W6] Key analyses are missing for the only recipe-matched pair. Most APE and RoPE models differ in training recipe and objective, so the effect of the encoding is confounded with the effect of training. In Figure 2, the variation within an encoding is as large as the difference between encodings. For example, DINOv3 (RoPE) has higher SSDC than every APE model at most blocks. The only pair that can control this confound is DeiT-III ViT-B and naver ViT-B, but for DeiT-III only the depth profile and the embedding zeroing/permutation were performed. The SAE screen and the feature-editing classification experiments, which are central to the paper, were not (there is no DeiT-III SAE in Table 4). In addition, in Figure 2a the SSDC of DeiT-III drops sharply from about 0.82 at block 1 to about 0.45 at block 2, which suggests that block 2 may not be an appropriate SAE site for DeiT-III. Performing the SAE analysis and the classification experiments of Section 4.5 on DeiT-III at an appropriate site would show more clearly whether the observed differences come from the encoding.

  • [W7] (Relatively minor) "Removing one axis's features removes only that axis" overstates the result. In Figure 3c and Table 8, the probe for the ablated axis still shows substantial values except for DINO. For the row axis, R2R^2 = 0.43–0.52 on supervised, AugReg and SAM, and for the column axis, R2R^2 = 0.56–0.71 on supervised and AugReg. That is, roughly half of the variance of the coordinate is still linearly recoverable. The ablation substantially reduces the targeted axis but does not remove it. Moreover, since ridge R2R^2 on a continuous coordinate is a lenient measure, reporting the 14-class row/column probe results for the same edits would show more directly how much of each axis remains. The contribution statement and Section 4.2 should be revised accordingly.

  • [W8] (Relatively minor) Figure captions are too long and contain content absent from the main text. Several captions (especially those of Figures 1, 3 and 4) span ten lines or more and include definitions and experimental details that are not in the main text. Examples include the definition that a "head" refers to the same head index across blocks 6–11, the noise criterion in Figure 4d, and what each condition in Figure 4b keeps or removes. As a result, the experiments cannot be understood from the main text alone, and readers have to move back and forth between the text and the captions excessively. Furthermore, the bar outlines are unconventional and mean different things in different figures. They show the SAE reconstruction in Figure 3c, the intact model in Figure 4a, and an experimental condition (the same-patch, other-image swap) in Figure 5a. This makes it difficult for readers to understand the paper properly.

  • [W9] (Relatively minor) Section 3.2 lacks guiding sentences and does not explain when each metric is used. In my view, Section 3.2 introduces SSDC and the linear probes one after another but does not explain why both metrics are needed or which one to trust when they disagree, which makes the main text harder to follow. Briefly summarizing in Section 3.2 what each metric measures, which claims it is used for, and how to interpret cases where the two disagree would make the main text much easier to follow.

References

[1] Jinyeong Kim et al., Interpreting Vision Transformers via Residual Replacement Model, NeurIPS'25

Questions for authors
  • [Q1] In Figure 3c, the direction of the asymmetry between the two axes differs across models. In supervised ViT-B and AugReg, the column axis drops less than the row axis (R2R^2 0.70 and 0.56), whereas in SAM the column axis drops more (0.27). Does this reflect a difference in how the models store the two axes, or is it a result of the screen not capturing the two axes evenly? If it is the latter, I am curious how it affects the reliability of the axis-separation result.

  • [Q2] In RoPE models, removing the rotation planes of one axis repeatedly raises the probe for the other axis above its intact value. When the column planes are removed, the row probe rises from 65.5% to 71.7% in naver ViT-B, from 69.7% to 74.2% in reg1-gap, and from 75.4% to 81.7% in DINOv3. In DINOv3, even removing 16 random planes raises the column probe from 77.2% to 85.0% (Table 9, Appendix E.1). Figure 4c also shows cases where the untouched axis exceeds its intact value (the column probe reaches 1.17 when the row planes are removed in naver ViT-S). Does this mean that the two axes compete for attention capacity? I am curious about the authors' interpretation of this phenomenon.

5 comments

Thank you for the time and care you put into this review. Many of your suggestions were specific enough to run as experiments, and the new results, all in the revised PDF, follow them closely.

[W1] Stating what is definitional and what could have failed. We agree that part of the contrast follows from the definitions (for example, RoPE's SSDC at the floor entering block 0), and Section 4.4 now says which differences are definitional. The central claim could have failed: it would have been falsified if, after we erase the stored position with the rotations on, the later blocks did not rebuild it or the classifiers lost as much as APE does. Neither happens (W4). DINOv3 partly fits your counterexample (position explains 12.3% of its patch-state variance, within the APE range, and it keeps 0.64 of its SSDC without later rotations), so the outcome is not fixed by the architecture. Nor is the axis split fixed by axial RoPE: in a mixed-frequency RoPE ViT-B, where every angle combines both coordinates, removing one coordinate still spares the other (Appendix F.4). The two findings you named, the anchoring at the image border and the dissociation between decodability and use, are now stated as results.

[W2] Restating the APE contribution relative to Kim et al. (2025). We now cite and discuss Kim et al. (NeurIPS 2025), and the first contribution now credits that work for the features and restates ours as the interventions on them (Sections 4.2 and 4.5).

[W3] Screening for conjunctive and other-shaped features. We agree that the column probe surviving a row-feature ablation partly follows from the screen. We therefore dropped the row and column selectivity filter and kept, for each grid position, the feature most active there, a screen that favours conjunctive features. It returns almost the same features in all five APE models (89–100% overlap, at most three new features per model; new appendix, cited in Section 4.2). The screen finds no conjunctive features beyond ours, although some position remains outside the selected features (Appendix D.5).

[W4] A shared intervention. We ran the experiment you proposed. With the rotations and the embedding on, we subtract the residual stream's positional component at block 4: all four RoPE models rebuild by block 11 (normalised SSDC 0.75–1.10), none of the five APE models do, and after a scramble the RoPE models rebuild the true layout (Section 4.4, Appendix G.3). Applied at every block, the edit costs the RoPE classifiers 0.3–0.6 top-1 points and APE 37.3–57.7. Probes still read some position after the edit, and the block-4 comparison across models was chosen after the run, as the paper states. Section 4.4 now includes the SAE-error result and reads "APE writes position into the residual stream once, after which later blocks only carry or discard it."

[W5] Weakening the RoPE claim and adding a shape-agnostic measure. The claim is now "RoPE models show no comparably selective features", and we state that the SAE is less reliable on RoPE. We added the measure you suggested, the share of patch-state variance explained by position: 1.4–4.3% in three RoPE models and 12.3% in DINOv3, against 9.4–18.0% in APE (Appendix G.3). The RoPE SAE settings are now in Table 4.

[W6] Completing the recipe-matched pair. We ran the full SAE and classification analyses on DeiT-III (Appendix G.1). Block 2 remains a suitable site: the SAE finds 36 row and column features, zeroing them keeps SSDC at or below 0.25 in every later block and costs 41.2 more top-1 points than matched random features. Under the shared edit, DeiT-III loses 11.32 ± 0.54 points and naver ViT-B 0.20 ± 0.18, and only naver rebuilds.

[W7–W9] Tightening claims and presentation. The contribution statement and Section 4.2 now say that zeroing one axis's features lowers that axis while largely sparing the other, rather than removing it. The captions of Figures 1, 4 and 5 are shorter, with their definitions moved into the main text. Section 3.2 now says what each measure is for: SSDC ranks components, the probes give the axis and top-1 settles disagreements. Across single heads, SSDC cost predicts top-1 cost better than probe cost (ρ = 0.55–0.67 against 0.19–0.33).

[Q1] The axis asymmetry. Its direction varies by model, while the separation holds in every SAE of our sweep (untouched axis R² ≥ 0.91; Appendix D.8). Because we cannot cleanly separate a model effect from uneven screening, we do not read the direction as a property of the model.

[Q2] Competition between the axes. The data do not support it: removing an axis widens attention along it in all eight cases, but random planes widen it too and the widening does not track the rise (ρ = −0.45). The rise also appears in mixed-frequency RoPE, so we leave its cause open (Appendix E.1).

We hope this addresses your concerns.

Thank you for your responses. Most of my concerns have been addressed, particularly through the impressive additional experiments for W6. I also appreciate your answers to my questions.

However, I still have some remaining concerns regarding W3 and W4.

[W3] I think the new feature screening experiment is useful for assessing how the original row/column selectivity criterion may have influenced the results. However, since the new screening still selects the most strongly activated feature at each position, I wonder whether it might miss features that encode positional information in a distributed manner across multiple positions, if I understand the procedure correctly. While I agree that this experiment mitigates some of the bias in the original screening, I remain uncertain whether it sufficiently addresses the possibility that other forms of positional representations exist or play an important role.

[W4] I think applying the same residual-stream intervention to both APE and RoPE is an important step toward addressing the asymmetry in the original comparison. However, according to the additional experiments in Appendix G.3, the predefined success criterion for positional erasure was not met in most cases. I therefore remain uncertain how strongly the observed differences support a fundamental difference between the two mechanisms, given that positional information may not have been sufficiently removed.

Could the authors provide further clarification on these points? Please also let me know if I have misunderstood any part of your responses or additional experiments.

Thank you for the close reading of the new appendices and for the follow-up; you have understood both experiments correctly. We answer the two points below.

[W3] Position outside the screened features.

We agree that the new screen keeps, for each position, only the feature most active there relative to elsewhere, so it cannot see position spread over many features that are each weakly tied to a position: decodable position does remain outside the selected features. After they are zeroed, a probe still reads exact position at 57–97% once the SAE reconstruction error is added back (Appendix D.5; the upper end is DeiT-III, added in the revision). This establishes that positional information remains decodable in the reconstruction error, but does not establish how that information is organized or whether it corresponds to a distributed representation across other SAE features. Specifically, Section 4.2 and Appendix D.7 note that the screen also leaves room for position in less selective features.

Two results help assess the functional significance of positional information that may remain outside the selected features. First, adding back the SAE reconstruction error does not restore the grid structure. With the selected features zeroed and the error added back, SSDC stays at 0.07–0.25, and at or below 0.25 at every later block. Second, the selected features make a large contribution to classification as zeroing them costs 15.1, 13.9 and 41.2 top-1 points more than matched random features on supervised ViT-B, SAM and DeiT-III (Appendix H.1). For comparison, subtracting the per-slot positional component at block 2, which uses no SAE and no screen, costs 17.7, 17.5 and 44.7 points on the same models (Table 27). The two edits use different baselines, and for that reason their effects cannot be directly compared to quantify the relative contribution of the selected features.

We therefore conclude that the selected features are causally important to the spatial structure measured by SSDC and contribute substantially to classification. Their removal disrupts both, and the disruption to spatial structure persists even when the SAE reconstruction error is added back. We do not claim that these features exhaust the model's positional representation, and we cannot rule out contributions from other, potentially distributed, representations.

[W4] The erasure check.

You are right that the check was rarely met at block 4, which is why the paper presents the block-4 comparison as chosen after the run. Three results suggest that the position left behind is unlikely to explain the difference on its own.

  • On APE the every-block edit meets the check. It does so at all 25 points in all five APE models (normalised SSDC at most 0.04, exact probe at most 1.71% against 0.51% chance) and costs 37.3-57.7 top-1 points, 89-95% of removing the embedding (Table 28).

  • On RoPE the rebuilding depends on the rotations. With the rotations also off from block 4, SSDC after the block-4 erasure stays at or below 0.10 in every RoPE model (Table 26). Subspace erasure did pass the check on naver ViT-B and ViT-S, and both models still rebuild (1.23 and 1.08 at block 11), although that edit removes more than position, as the paper notes.

  • APE keeps a comparable or larger leftover and still does not rebuild. After the block-4 erasure the exact probe reads 6.3-9.2% on the APE models against 3.3-6.4% on the RoPE models (Table 26). Separately, when the selected SAE features are zeroed with the error added back, 57-97% stays decodable and SSDC does not return (Appendix D.5), which is the result you suggested we foreground.

One limitation remains for RoPE. There the every-block erasure meets the check at only 2–5 of 24 points per model (exact probe 0.7–4.3%, against up to 27–45% intact; Table 28), and a linear probe does not measure everything that is left. We therefore cannot rule out that a more complete erasure would cost the RoPE classifiers more than 0.3–0.6 points, and the claim is limited to position stored as a per-slot mean while the rotations are on (Appendix G.3).

We hope this resolves the two remaining concerns, and we would be glad to clarify anything further.

Reviewer_RxJ2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks where a ViT internally stores position information and how that depends on the positional encoding used. Comparing four APE and four RoPE pretrained ViTs, the authors find two different mechanisms. In APE models, position is written into the residual stream once at the start. A sparse autoencoder can find about 20 features that act like row and column coordinates, and zeroing them removes spatial structure for the rest of the network, with no recovery downstream. In RoPE models, they do not find these clean features. Here, position is computed inside attention by the rotations, separately for rows and columns, at every layer. Because it is recomputed at each block, position removed from early blocks gets rebuilt by later ones, mostly by a few heads that each restore one axis. Both mechanisms matter for classification. Scrambling APE’s positional features across patches costs top-1 accuracy, and removing RoPE’s main rebuilding head costs more than removing any other head. The authors conclude that APE stores position once and it propagates further in the residual stream while RoPE recomputes it throughout. This way the encoding method determines how position information is internally represented and used.

Critical strengths
  1. I like the swap control experiment. Moving a patch’s positional features to the same patch of a different image costs almost nothing, while moving them to a different patch of the same image costs the top-1 points.
  2. Rotation masking is a clean intervention. Replacing a rotation with the identity removes position exactly and reversibly, per axis, per block, or per head, with no reconstruction step.
  3. The writing is clear, and the stored once vs. recomputed picture is a real mechanistic insight, especially the rebuilding experiment, where later blocks restore position the early blocks never received.
Critical weaknesses
  1. The 20 SAE features explain a grid-like organisation of position, not the position information itself. When the reconstruction error is added back, position is still 57–93% decodable even after zeroing these features. So “position is stored in these features” could be overstating it?
  2. The paper is motivated by spatial reasoning in VLMs, but no VLM encoder (CLIP, SigLIP, EVA-02, Qwen2-VL) is tested. Every model is an ImageNet-trained or self-supervised ViT-B/S at 224×224, and the only downstream task is ImageNet classification.
Questions for authors
  1. Where is the position that the 20 SAE features do not account for? Appendix D.5 shows it sits in the reconstruction error, but it is unclear whether that is a limit of the SAE’s size or of sparse dictionaries in general. Could you retrain the SAE with more latents or a larger k, re-run the screen, and repeat the zero-with-error-added-back test? If the 57–93% recovery drops, the original SAE was too small to isolate that part of position and if it does not, the remaining position is not something a sparse dictionary can isolate cleanly, whether it stays in the error term or is splitting across many features.
  2. You mention in the limitations that a “head” spans six blocks. Since head indices are arbitrary across layers, could you mask the top rebuilder’s index one block at a time and report the rebuilding cost per block?
  3. In the hybrid model, position at block 4 is at least as decodable with the rotations off (88.6% vs 85.1%) yet top-1 falls from 80.5% to 37.4%. What is the classifier getting from the rotations that it cannot get from the embedding?
3 comments

Thank you for your careful and generous review. We are glad you found the stored-once versus recomputed picture and the rebuilding experiment insightful, and your questions led directly to new analyses, all now in the revised PDF.

[W1/Q1] Narrowing the claim about the selected SAE features. We agree that "position is stored in these features" overstated the result, and we have revised it in the introduction, the Figure 1 caption and Section 4.2. The selected features are still what the model uses: with the reconstruction error added back, zeroing them keeps SSDC low (0.07–0.25) and costs 13.9–41.2 more top-1 points than zeroing matched random features, even though a probe can still read the leftover position. Decodable position is not the same as used position, and the text now says so. Following Q1, we retrained the SAE with more latents and a larger k (Appendix D.8): the identified features and their effect on spatial structure hold in every configuration. We did not rerun the control experiments, including the error-added-back test, on the larger SAEs, so whether the 57–93% drops remains open, and we say so.

[W2] Reframing the introduction and adding a VLM encoder, a larger model and a higher resolution. We removed the vision–language framing from the introduction (lines 45–46). We also added Qwen2-VL's vision encoder from your list, naver ViT-L and 320×320 input for three RoPE models (Appendices H.3–H.5). In all of them, removing one axis's rotations removes that axis and spares the other, and later blocks rebuild the spatial structure (normalised SSDC 0.43–0.62). We also report where they differ: in Qwen2-VL the row and column probes recover only 11–12% of the way back to intact, and in ViT-L no single head stands out. Classification is still our only downstream task, and the limitations say so.

[Q2] Reporting the rebuilding cost block by block. We did this, and extended it to every single head (block, head) in blocks 6–11 (Appendix F.1); Sections 4.3 and 4.5 now report single heads. The cost sits in one or two blocks right after the window: on naver ViT-B, (6,6) carries 0.042 of the index's 0.055 SSDC cost and 3.04 of its 4.30 top-1 points. As single heads, the top head is within error of the next on two classifiers, so we now claim a few rebuilding heads rather than one. That claim does not rest on ties: a head's SSDC cost predicts its top-1 cost (Spearman ρ = 0.55–0.67), and the six top heads cost 10.1–17.6 points against 4.1–6.8 on average for random six-head sets from the same blocks (Appendix H.2).

[Q3] What the hybrid's classifier gets from the rotations. The rotations keep the late blocks' attention local, which the embedding does not do even though it keeps position decodable. Removing the rotations costs 42.2 top-1 points and removing the embedding 0.9, and without the rotations attention in blocks 4–11 is 1.39 patches wider on average, against 0.15 without the embedding (Section 4.4, Appendix G.2). The decodable position is still used once the rotations are gone: erasing it then costs 5.34 more points (Appendix G.3).

Beyond your questions, we added one intervention applied identically to both encodings, subtracting the residual stream's per-slot positional component at every block (Appendix G.3). It costs the RoPE classifiers 0.3–0.6 top-1 points and the APE classifiers 37.3–57.7, which supports the stored-versus-recomputed picture with effect sizes on one scale.

We hope this addresses your concerns.

Thank you for your response. I appreciate the additional analyses and will maintain my original positive score.

Thank you for taking the time to review our response and the additional analyses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.