HARP: Hierarchically Anchored Rotary Phases for Long and Structured Contexts

Rotary Position Embedding (RoPE) derives one fixed global phase schedule from the raw token index. The raw index must therefore encode local order, long-range distance, and structured layout at once. Recent analyses reveal the overload to long-context retrieval failures, to frequency crowding, and to fragility under the bfloat16 (BF16) format. We propose Hierarchically Anchored Rotary Phases (HARP), which factors the raw index into a coarse chunk index, a within-chunk offset, and optional ordered structure coordinates. Each coordinate governs its own band of learnable rotary phases, while a fraction of channels stay unrotated as a content bypass. The encoding comes in a general form (HARP-core) and a separated-band form (HARP-sb). We initialize both exactly on the RoPE submanifold, so training departs from RoPE only when the data rewards the departure. The factorization yields an attention kernel over relative hierarchical coordinates rather than the raw gap alone. The resulting kernel recovers RoPE exactly and strictly enlarges the positional kernel class. Separating the bands further bounds the phase budget at $O(\sqrt{L})$ rather than $O(L)$ in context length $L$. We evaluate HARP through deterministic experiments, synthetic probes, from-scratch pretraining at $160$M and $420$M parameters, a BF16 analysis, and ablations. On structural tasks HARP separates equal-distance token pairs that every purely relative encoding maps to identical logits. Beyond that, the trained model concentrates attention inside the current chunk, carries non-redundant information in all three bands, and HARP-core raises scope resolution from $4.7\%$ to $42.5\%$. On length and precision at scale, by contrast, the advantage narrows from a clear win at $160$M to parity at $420$M. One reason is that training specializes the global band alone, which puts the bounded phase budget out of reach. HARP therefore encodes hierarchy where RoPE encodes distance alone.