acceptodds
Withdrawn from ICLR 2027

Understanding RoPE through Head-wise Retrieval Distance Profiling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Rotary Position Embedding (RoPE) is widely used in large language models (LLMs) and is central to context-length extension. However, existing analyses mainly infer the importance of RoPE frequencies from magnitude-based statistics such as the norms of query and key representations, which do not reveal how attention heads function across different query-key relative positions. We introduce Head-wise Retrieval Distance Profiling (HRDP), which explicitly controls the query-key relative offset and measures, for each attention head, whether the target key achieves the highest query-key score among all preceding keys. HRDP provides an offset-dependent retrieval profile for each attention head. Our analysis reveals that approximately 20% of heads successfully retrieve targets at short offsets, whereas long-range retrieval relies on only a small subset of heads. We further find that better-performing positional interpolation more faithfully preserves the original head-wise retrieval structure, whereas poorer interpolation substantially disrupts it. These results provide a functional view of RoPE beyond frequency magnitude and offer a new basis for analyzing and designing context-extension methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Rotary Position Embedding, Head-Wise Retrieval Distance Profiling, Position Interpolation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tfzoOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes HRDP, a probe that places the same marker token at the query position and at a controlled offset dd earlier, and scores each head by the fraction of samples in which the target key wins the pre-softmax argmax over all preceding keys. On Llama-3-8B it reports that the share of heads with at least one success falls from ~20% at d=2d=2 to ~5% at d=2,048d=2{,}048, collapses beyond the 8K training length even at d=2d=2, that ablating the high or mid-low frequency band hurts at both offsets while the low band barely matters, and that ℓ2\ell_2 norms do not track these effects. HRDP is then used to compare Linear PI and YaRN at 32K and to pick the YaRN factor (s∗=6.3s^*=6.3) whose 32K profile is closest to the 8K profile; after QLoRA fine-tuning this beats s=4s=4 on RULER@32K (62.99 vs 47.94). Additional default-RoPE profiles for Qwen3 and Gemma models are in the appendix.

Critical strengths
  1. The probe. HRDP fixes the exact query–key offset, works at the logit level, and is cheap and reproducible. The head-level facts it produces (12 of 1,024 heads reach 0.8; distance-specialized heads such as (0, 9) for d≤32d\le32 and (10, 2) for d≥512d\ge512, App. D) are new and checkable. This is the part of the paper I would want to see published.
  2. HRDP as a selection signal for context extension. Choosing the YaRN factor by profile preservation (Eqs. 7–8) is a simple idea, and the RULER@32K gain is large. If it holds under proper comparison it is a useful practical result.
Critical weaknesses
  1. The RoPE “understanding” claims are largely predicted by the frequency schedule and by prior work, and the paper frames them as a challenge to intuition. With θj=(5×105)−j/64\theta_j = (5\times10^5)^{-j/64}, the low band (blocks 48–63) rotates at most 0.11 rad over d=2,048d=2{,}048, and most of it under 0.05 rad. Replacing it with NoPE is close to a no-op at every tested offset, so the small effect of the low-band ablation follows from the grid d≤2,048d\le2{,}048 and says nothing about long-range retrieval, which for this band begins around 10510^5 tokens. Barbero et al. (2025) already showed that the lowest frequencies act as near-NoPE semantic channels and can be removed (p-RoPE) without loss. The mid-low band (wavelengths ≈4K–90K tokens) is the one that resolves the tested offsets, and the high band is needed to separate the target from its immediate neighbours under a Top-1 criterion; both are the standard multi-scale argument, relabeled. The paper argues against the coarse “higher base = longer range” reading of Xiong et al. (2024) while not citing Hong et al. (Findings of EMNLP 2024), whose dimension-level analysis and ablation find positional heads shifting to lower-frequency dimensions with distance, which is the direct prior claim and points the other way. The ℓ2\ell_2-norm section is qualitative, uses query norms only, and the norm-dominant mid-low region is also among the most damaging to ablate, so the data support at most “norms do not flag the high band”. Two further confounds limit what §4.3 can show: the NoPE ablation is applied to all keys (and, as far as I can tell, all layers), so a drop can come from distractors gaining score or from upstream changes rather than from the target’s own score, and the probe cannot separate content matching from purely positional attention at short offsets, since a previous-token head scores 1 at d=2d=2 with no control (e.g., a random token at the target position). The one result the geometry does not predict, mid-low ablation hurting at d=2d=2 where the target block rotates ~0.003 rad, is the one that is left unexplained.

  2. The practical claim rests on two fine-tuned configurations. s=4s=4 vs s∗=6.3s^*=6.3, one run each, no variance, 8K differences within noise (90.10 vs 90.45). “Best 32K RULER performance” is the best of two. The paper does not show that the HRDP distance tracks RULER across several values of ss, nor compare against a cheap baseline such as selecting ss by 32K validation perplexity. The selection also uses the WikiText-103 test split that is later used for perplexity evaluation. Related, the “~20% of heads successfully retrieve” headline counts one success in 100 trials; only 12 heads reach 0.8.

Weakness 1 drives my rating: the paper’s title and most of its first Key Findings box describe contributions that are either predictable from the schedule or already in the literature. The probe and the selection idea are worth publishing once the frequency claims are rescoped and the selection result is backed by a sweep and a baseline.

Questions for authors
  1. Is the band ablation in §4.3 applied in all layers and to all keys? For the mid-low ablation at d=2d=2, is the drop driven by the target’s score, by distractor scores, or by upstream changes? A per-band decomposition of the target-vs-best-distractor margin would answer this.
  2. Did you run a positional control (random token at the target, $ at the query)? What fraction of the d=2d=2 “active” heads survive it?
  3. Were the Fig. 8 maps and the grid over ss computed training-free (App. A.4) or after fine-tuning? §5.1 says s=8s=8; the setup and figure say s=4s=4.
  4. Does the HRDP distance (Eq. 7) track RULER@32K over several fine-tuned values of ss, and how does s∗s^* compare with ss chosen by validation perplexity? Are Tables 1–2 single runs, and how many RULER samples per task?
  5. At L=16L=16K and 32K, which keys win the argmax instead of the target and at what offsets? This separates the absolute-position explanation in §4.2 from distractors at unseen relative distances.
Reviewer_eVHPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces HRDP, a probe for how individual attention heads use RoPE across relative distances. The same marker token (a dollar sign) is placed at the query position n=L−1n = L - 1 and at a target position n−dn - d, and for each (layer, head) the authors check whether the target key gets the highest QK score among all preceding keys. Averaging over 100 WikiText samples gives each head a retrieval profile as a function of the offset dd.

On Llama-3-8B, about 20% of heads "retrieve" at d=2d = 2, dropping to about 5% at d=2048d = 2048, and beyond the 8K training length almost no head retrieves, even at d=2d = 2. A band-wise RoPE ablation suggests that the high and mid-low bands matter while the low band doesn't, which the authors read as contradicting both the "low frequency = long range" intuition and ℓ2\ell_2-norm-based importance analyses. Finally, they use HRDP to analyze positional interpolation: YaRN preserves the pretrained profile while Linear PI and extreme YaRN don't, and choosing the YaRN scale by profile matching (s⋆=6.3s^{\star} = 6.3) beats the usual s=4s = 4 on 32K RULER after QLoRA fine-tuning (63.0 vs 47.9).

Critical strengths
  • A simple, cheap, interpretable probe. Controlling the exact query–key offset makes sense, since RoPE logits depend on relative position.
  • Using profile preservation to pick YaRN's scale is a cute, actionable idea, and the 32K RULER gain is sizable.
Critical weaknesses

I have some concerns:

  • HRDP measures something narrower than "retrieval". Query and target are the same token, so it mainly picks up duplicate/same-token matching heads. Retrieval heads in the Wu et al. sense attend to the token after the earlier occurrence, and real retrieval is semantic. Also, at small dd, a purely positional head (one that always attends to n−2n - 2) gets a perfect score regardless of content, so a control with a random token at position mm is needed. The metric is also loose: "active head = retrieves at least once in 100 samples" is a very low bar, and top-1 over all keys can be dominated by attention sinks / BOS.
  • The "collapse beyond the training length" is probably misinterpreted. RoPE scores are purely relative, so for a layer-0 head the query–target score at d=2d = 2 is identical at L=8KL = 8\text{K} and L=32KL = 32\text{K}. What changes is that distractor keys now sit at relative offsets beyond 8K that were never seen in training, and they can win the argmax. So the claim that "retrieval depends on absolute position" doesn't follow. Showing where the argmax lands on failures, or restricting candidates to keys within 8K, would clarify this. Deeper layers also suffer cascading corruption from earlier layers.
  • The frequency ablation removes each band's rotation (NoPE), not its content:
    • For Llama-3 (base 5×1055 \times 10^5), the lowest band rotates at most ~0.11 rad over d=2048d = 2048, and at most ~0.44 rad over the whole 8K.
    • So removing its rotation is nearly a no-op, and "the low band doesn't matter" is expected rather than evidence of functional unimportance.
    • It's also apples-to-oranges with the ℓ2\ell_2-norm literature, which is about content magnitude (Barbero et al. even argue that low frequencies act as semantic channels). On top of that, the ℓ2\ell_2-norm comparison itself is only qualitative, with no per-head correlation between norm profiles and HRDP profiles.
    • Zeroing out each band's qq/kk components would be the comparable experiment, ideally with finer bands and more offsets.
  • The HRDP-based choice of ss rests on a single comparison (s=4s = 4 vs s=6.3s = 6.3), one model, one length, and one fine-tuning run. You'd need to show RULER across the whole ss grid to see whether the HRDP-MSE minimum lines up with the RULER peak, and compare against cheap alternatives (pick ss by validation PPL, or just use a somewhat larger ss). Also, s⋆s^{\star} is chosen with training-free YaRN, but the evaluated models are QLoRA-fine-tuned.
Questions for authors
  1. If the target marker is replaced with a random token (keeping the dollar-sign marker at the query), how many heads still "retrieve" position mm? How many heads are positional vs content-matching?
  2. When retrieval fails beyond 8K, where does the argmax land? Does restricting candidates to keys within 8K restore the profile? What happens at layer 0?
Reviewer_kt43OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Head-wise Retrieval Distance Profiling (HRDP), which measures how each attention head matches a repeated marker at different query–target distances. A trial succeeds when the target receives the highest attention logit among preceding keys. The experiments mainly use Llama-3-8B, with additional profiles from Qwen and Gemma models. The authors examine how these profiles change with sequence length, frequency-band interventions, and positional interpolation. They report distance-dependent head behavior and argue that query norms alone do not capture frequency importance. They also use the difference between the original 8K and extended 32K profiles to select a YaRN scale. Under matched fine-tuning settings, the selected scale of 6.3 improves 32K RULER from 47.94 to 62.99 compared with the default scale of 4.

Critical strengths
  1. Making target distance an explicit variable is useful. The profiles show how the same head’s matching behavior changes across distances, providing information that is lost in an aggregate retrieval score.
  2. The paper applies the diagnostic to several concrete settings, including frequency-band NoPE, Linear PI, and YaRN. The additional Qwen and Gemma profiles also extend the descriptive analysis beyond one model family.
  3. The parameter-selection experiment provides a meaningful positive result. The comparison uses matched fine-tuning settings, and the selected scale improves 32K RULER by 15.05 points while maintaining comparable 8K performance. This is a promising initial application of the diagnostic.
Critical weaknesses
  1. The frequency intervention does not support the full interpretation. Section 4.3 removes a band’s rotation but leaves its Q/K content contribution in place. Little change after low-frequency NoPE therefore shows limited sensitivity to that rotation; it does not establish that the channels contribute little to retrieval. The further conclusion that large norms misrepresent functional importance needs a comparison that measures or removes the channels’ actual score contributions.
  2. The experiments do not isolate the proposed positional explanation. At fixed target distance, increasing sequence length also changes the prefix, contextual representations, and competing keys. Figure 4 therefore does not isolate an absolute-position effect. The behavioral check also uses different distances: Appendix A.3 adds 67 question tokens, so the actual QA offset is d + 67. The shared decline with sequence length is useful evidence, but it does not validate the same-offset failure mechanism.
  3. The parameter-selection result is stronger than the evidence for the criterion itself. The selected scale of 6.3 outperforms 4, but this is the only downstream scale comparison reported after searching 61 candidates. It remains unclear whether HRDP MSE predicts performance across candidate scales. The separation between calibration and perplexity evaluation also needs clarification, since both refer to WikiText-103 test data.
Questions for authors
  1. Using the same pre-RoPE Q/K representations, how does removing the low-band score contributions compare with replacing their rotations by the identity? Does removing those contributions also preserve target rank? This would directly test the limited-contribution interpretation in Section 4.3.
  2. Does HRDP degrade if all position IDs are shifted beyond 8K while the tokens, causal mask, and RoPE frequencies are held fixed? If not, what supports attributing Figure 4’s failure to the absolute-position range rather than the changed prefix and competing keys?
  3. Appendix A.3 makes the actual QA input length L + 67 and target offset d + 67. Does the correspondence with QA still hold when HRDP is evaluated at those actual lengths and offsets—for example, at distance 69 for the setting labeled d = 2?
  4. For several representative YaRN scales, such as 4, 6, 6.3, and 8, does lower pre-fine-tuning HRDP MSE predict higher post-fine-tuning RULER scores under matched training budgets? A comparison across these settings would help distinguish a useful ranking criterion from one successful choice.
  5. Are the underlying WikiText passages used for scale selection disjoint from those used for Table 2 perplexity evaluation? If they overlap, does the perplexity advantage of 6.3 over 4 remain on a separate held-out set?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.