Understanding RoPE through Head-wise Retrieval Distance Profiling

Rotary Position Embedding (RoPE) is widely used in large language models (LLMs) and is central to context-length extension. However, existing analyses mainly infer the importance of RoPE frequencies from magnitude-based statistics such as the ${\ell }_2$ norms of query and key representations, which do not reveal how attention heads function across different query-key relative positions. We introduce Head-wise Retrieval Distance Profiling (HRDP), which explicitly controls the query-key relative offset and measures, for each attention head, whether the target key achieves the highest query-key score among all preceding keys. HRDP provides an offset-dependent retrieval profile for each attention head. Our analysis reveals that approximately 20% of heads successfully retrieve targets at short offsets, whereas long-range retrieval relies on only a small subset of heads. We further find that better-performing positional interpolation more faithfully preserves the original head-wise retrieval structure, whereas poorer interpolation substantially disrupts it. These results provide a functional view of RoPE beyond frequency magnitude and offer a new basis for analyzing and designing context-extension methods.