- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper investigates how Vision Transformers represent spatial position under learned absolute positional embeddings (APE) and 2D rotary positional encodings (RoPE). The authors use similarity–distance correlation (SSDC) to quantify the relationship between hidden-state similarity and spatial distance, along with a random-permutation protocol (RPI) that shuffles patch content while preserving positional information. These analyses are complemented by linear probes, sparse autoencoders (SAEs), and reversible interventions that selectively remove RoPE rotations across axes, blocks, and heads. The results suggest that APE encodes positional information through a small number of row and column coordinate features in the residual stream, which cannot be recovered by later layers once removed. In contrast, RoPE models reconstruct positional information within attention layers, recover spatial structure after early rotations are disabled, and exhibit heads associated with axis-specific position reconstruction. The authors further demonstrate that these positional mechanisms have causal effects on ImageNet classification beyond matched control interventions.
Critical strengths
-
The paper addresses an interesting and well-motivated question. Although ViTs have increasingly adopted RoPE over APE for accuracy and extrapolation benefits, the underlying differences in how these positional encoding schemes represent spatial information remain insufficiently understood.
-
The analysis goes beyond correlational probing by introducing causal interventions. Selective rotation masking provides an exact and reversible way to manipulate positional information in RoPE models, while SAE feature ablations allow the authors to directly investigate positional representations in APE models.
-
The experimental controls are carefully designed. RPI reduces content-related confounding, random SAE features are matched by activation frequency and magnitude, and patching experiments include appropriate alignment controls. Adding back SAE reconstruction errors also ensures that the unmodified intervention pipeline reproduces the original model.
-
The main findings are supported by several compelling quantitative results. Ablating approximately 20 APE features reduces exact-position probe accuracy from 99.7% to 22.9%, while RoPE models recover normalized SSDC to 0.50–0.71 after early rotations are removed. Relocating APE features among patches within the same image also causes a 24–31 point reduction in top-1 accuracy.
-
The experiments examine multiple models and training settings, including supervised, DINO, AugReg, SAM, naver, DINOv3, and a hybrid architecture. The paper also provides detailed implementation and reproducibility information.
Critical weaknesses
-
The comparison between APE and RoPE is complicated by the use of different intervention mechanisms. APE is analyzed through SAE feature ablation, whereas RoPE is studied through rotation masking, making it difficult to determine whether the observed "stored versus recomputed" distinction arises entirely from the positional encodings rather than the intervention methods. The authors acknowledge that these interventions allow only qualitative comparisons, not direct comparisons of effect sizes. Furthermore, the inability of APE models to reconstruct position after feature removal is somewhat expected, since removing these features at block 2 eliminates the remaining positional information available to subsequent layers. Most comparisons are also confounded by differences in training configurations, with only DeiT-III and naver ViT-B sharing the same training recipe.
-
The incremental novelty relative to prior work needs clearer discussion. The two primary measurement tools, SSDC and RPI, are adopted from Mannes (2026), which already investigated how positional encodings establish spatial structure in ViTs. The main new contributions appear to be the causal localization of positional mechanisms and their effects on downstream classification. The paper would benefit from a more explicit distinction between these new findings and the earlier analysis, as well as an examination of whether the conclusions depend on the particular SSDC/RPI measurements.
-
The SAE-based findings rely on a relatively narrow experimental configuration. The analysis uses a single SAE with one layer, one width, and a heuristic procedure for selecting positional features. Moreover, the selected features do not capture all recoverable positional information: when SAE reconstruction errors are added back, exact-position probe accuracy still recovers to 57–93% after the selected features are removed. For RoPE models, simply passing representations through the SAE can change SSDC by as much as 0.214, suggesting that the absence of coordinate-like features in RoPE should be interpreted with greater caution.
-
The evidence supporting axis-specialized rebuilder heads is not entirely convincing. The strongest head exceeds the second-ranked head by only around 0.8–1.1 accuracy points, while the three highest-ranked DINOv3 heads have attribution scores differing by less than approximately 0.004. The top head in naver ViT-S also shows no clear axis preference, and paired standard errors are not consistently provided across models. Additionally, a "head" in this analysis refers to the same head index across six blocks rather than an individual attention head in a specific block. These limitations make the identification and claimed axis specialization of rebuilder heads less conclusive.
-
The experimental scope is relatively limited. The analysis focuses on ViT-B/S models at 224×224 resolution and evaluates downstream effects only through ImageNet classification, without experiments on dense prediction, larger architectures, or substantially higher resolutions. SSDC also becomes less sensitive at 320×320. Consequently, the connection between the observed positional mechanisms and the broader spatial-reasoning limitations motivating the study remains insufficiently established.
Questions for authors
-
Could you investigate APE and RoPE using a shared intervention protocol that allows their causal effects to be compared quantitatively?
-
What are the main findings that go beyond the earlier SSDC/RPI analysis? Do the conclusions remain consistent when using alternative measures of spatial structure?
-
How robust are the identified APE positional features and the associated conclusions across different SAE widths, sparsity settings, layers, and random seeds?
-
Do the identified rebuilder heads maintain their importance and axis specialization at higher image resolutions or on dense prediction tasks?
3 comments
Thank you for your detailed response.
Some of the weaknessed I raised has been addressed. However, here are some remianing concerns:
-
The erasure criterion is mostly not met and the block-4 comparison was chosen post hoc, so the small drop for RoPE may partly reflect incomplete erasure. Please report results where the criterion is met.
-
The error-add-back test was not rerun on the new SAEs, so the 57–97% residual decodable position is still unexplained.
-
Effects are weaker on DINOv3, Qwen2-VL and ViT-L, which suggests the mechanism depends on the training objective as well as the encoding type.
I am raising my soundness score to 2 and my rating to 2 (weak reject).
Thank you for the follow-up and for raising your scores. We answer the three remaining concerns below.
1. Results where the erasure criterion is met.
We agree that the block-4 check was not always met and that the block-4 comparison was chosen after the run, as the paper states. However, on net, the intervention substantially reduces the measured positional information even in cases where it does not meet the predefined erasure criterion: at the edited block, SSDC falls to the floor in all four RoPE models and to −0.01–0.28 in the APE models, while exact-position probe accuracy falls from 27–45% intact to 3.3–6.4% on RoPE and from 95.9–100% to 5.1–9.8% on APE (Table 26). Thus, although the criterion is not met, the intervention removes most of the positional information captured by our proxies at the edited state. The table lists the cases where the criterion is met, next to the every-block result on RoPE, where it mostly is not (Tables 26–28).
| Edit | Models | Criterion met | Normalised SSDC, block 11 | Top-1 cost (points) |
|---|---|---|---|---|
| Mean erasure, every block | 5 APE | 25 of 25 points | At most 0.04 | 37.3–57.7 |
| Subspace erasure, block 4 | naver ViT-B, ViT-S | Yes | 1.23, 1.08 | 9.86, 71.14 |
| Mean erasure, every block | 4 RoPE | 2–5 of 24 points | 0.69–0.92 (DINOv3: 0.18) | 0.30–0.64 |
The criterion-met results support APE's dependence on stored position and show RoPE rebuilding after subspace erasure. Subspace erasure costs far more than a random subspace of the same size (0.32 and 1.28 points), so it removes more than position and we do not read its cost as the cost of position. The small RoPE cost does not have the same support. It comes from the row where the criterion is mostly unmet, so we agree that the 0.3–0.6 points may partly reflect incomplete erasure. What we can say is that this edit removes most decodable position on RoPE (exact probe 0.7–4.3%, against up to 27–45% intact) and takes SSDC to the floor at that cost, while the same edit costs APE 37.3–57.7 points.
2. The 57–97% that remains decodable.
You are right that we did not rerun the error-added-back test on the new SAEs, and Appendix D.8 reports as such. However, the existing SAEs provide some insight into the source of this residual decodability (Table 7). After the selected features are zeroed, the exact probe reads 5.9–45.3% without the SAE error and 57–97% with it, a gain of 48–79 points in every model. A large part of it therefore comes from what the SAE does not reconstruct, and the rest is carried by features outside the selected set. Due to time constraints, we have not established whether the same decomposition holds for the new SAE configurations.
3. Dependence on the training objective.
We agree that the strength of these effects depends on the training objective and model scale, not only on the positional encoding, and the paper discusses/reports this nuance (Section 5, Appendices H.3–H.5). However, although the mechanism manifests differently across DINOv3, Qwen2-VL and ViT-L, its qualitative features persist. Specifically, naver ViT-L shares naver ViT-B's training recipe, and its difference is one of size: its axes still separate and its later blocks still rebuild (normalised SSDC 0.54), with the rebuilding spread over many heads (illustrating the fundamental mechanism). Similarly, although Qwen2-VL and DINOv3 exhibit weaker effects, we still observe the same qualitative RoPE mechanism in both: positional information separates by axis when the rotations are decomposed, and later blocks rebuild spatial structure when the rotations are restored. Additionally, the pair that shares a recipe isolates the effect of the encoding. On net, the fundamental mechanism holds in spite of the results being somewhat affected by encoding and training, as is to be expected.
We hope this addresses the remaining concerns. Thank you again for your time.
Thank you for your detailed and careful review, and we are glad you found the research question well motivated. We addressed every weakness and question you raised, almost all with new experiments, and the revised PDF states what we did not test.
[W1/Q1] A shared intervention with effect sizes on one scale. With both encodings left on, we apply the same edit to both families: we subtract the residual stream's per-slot positional component (Appendix G.3). Erased at block 4, SSDC comes back by block 11 in all four RoPE models and in none of the five APE models, even though APE's embedding stays on, so APE's failure to rebuild no longer follows from removing its source. The block-4 edit costs the RoPE classifiers 0.06–0.30 top-1 points and the APE models 3.1–11.3. Applied before the attention and the MLP of every block, it costs RoPE 0.3–0.6 points (about 1% of removing every rotation) and APE 37.3–57.7 (89–95% of removing the embedding). Removing the encoding entirely costs both families about the same (45.8–57.3 and 42.1–61.1 points): both need position, but only APE depends on the per-slot copy in the residual stream. The recipe-matched pair splits the same way (naver ViT-B 0.20 ± 0.18 points, DeiT-III 11.32 ± 0.54). As the paper states, probes still read some position after the edit, and the block-4 comparison across models was chosen after the run.
[W2/Q2] Clarifying the novelty beyond SSDC and RPI. We reuse SSDC and RPI from Mannes (2026), and the contributions now say so. What this paper adds is causal. On the RoPE side, we remove rotations by axis, block and single head, show that later blocks rebuild position the early blocks never received, identify the heads that do it, and show that their rotations matter for classification beyond matched and block-matched random controls. On the APE side, we go beyond observing position-selective features by intervening on them and measuring their effect on spatial structure, on each axis and on classification. Both families are now compared under one intervention. To our knowledge, these interventions on RoPE ViTs have not been reported before. The conclusions do not rest on SSDC alone: probes give the axis results, top-1 the functional ones, and a new SAE-free positional-variance measure separates three of the four RoPE models from APE and shows that DINOv3 stores about as much position as APE. Section 3.2 now states which measure supports which claim.
[W3/Q3] Robustness of the SAE results. We trained three more APE SAEs (Top96-8×, Top96-5× and Top64-8×, alongside the original Top64-5×) and repeated the feature ablation: the identified features and their contribution to spatial structure hold in all four (Appendix D.8). We did not rerun the classification and control experiments on the new SAEs, which we state, and the sweep varies only width and k, at block 2. For RoPE we agree: the paper now claims only that these models show no comparably selective features, and that the SAE is less reliable there. The SAE-free measure gives 1.4–4.3% positional variance at block 4 in three RoPE models and 12.3% in DINOv3, against 9.4–18.0% in APE.
[W4] Re-examining the rebuilder heads. We agree, and the analysis now uses single heads (block, head) throughout. Since the top head is within error of the next on two classifiers (0.12 ± 0.27 and 0.26 ± 0.29 points), we now claim a few rebuilding heads rather than one. That claim does not rest on ties: SSDC cost predicts top-1 cost across heads (ρ = 0.55–0.67) and within blocks (0.35–0.50), and the six top heads cost 10.1–17.6 points against 4.1–6.8 on average for 48 random six-head sets from the same blocks, above all 48 on two models and 47 on reg1-gap. DINOv3's "within 0.004" came from grouping heads by index: as single heads, (6,2) leads by 0.013 SSDC in every bootstrap resample. On naver ViT-S, (7,4) is a clear row head and the costliest (6.08 ± 0.40 points, 2.48 ± 0.38 ahead of the next). Paired standard errors are now given for every classifier.
[W5/Q4] Extending the scope. We added 320×320 input for three RoPE models, naver ViT-L and Qwen2-VL's vision encoder (Appendices H.3–H.5), and the axis split and the rebuilding appear in all of them. At 320×320 the heads named in Section 4.5 stay the costliest and stay row heads, while reg1-gap rebuilds less (0.43). Single-head SSDC still tracks top-1 at 320 on reg1-gap, trained at 256, but much less on the two models trained at 224, so the head analysis ranks heads by accuracy. We have not tested dense prediction and say so.
We hope this addresses your concerns. Thank you again for the detailed review.