- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Hierarchically Anchored Rotary Phases (HARP), which replaces the single token-position coordinate in RoPE with a chunk index, a within-chunk offset, and optional structural coordinates. It introduces two variants, HARP-Core and HARP-SB, with learnable phase coefficients or gates, aiming to distinguish token pairs that share the same relative distance but differ in their hierarchical relationships. The paper derives the resulting attention kernel, characterizes its relationship to RoPE, and provides a conditional bound on the phase budget. Experiments include synthetic tasks, small-scale language-model pretraining, length extrapolation, and numerical precision analyses.
Critical strengths
- Clear and interpretable formulation. The coordinate decomposition provides a straightforward way to incorporate hierarchical relationships into rotary position encoding. The distinction between within-chunk and cross-chunk interactions is intuitive, and the mathematical formulation makes the intended inductive bias explicit.
- Useful characterization of the positional kernel. The analysis explains how the proposed encoding can distinguish equal-distance token pairs and identifies conditions under which its rotated components recover RoPE. These results help clarify the behavior of the parameterization.
- Transparent reporting of limitations and negative results. The paper acknowledges that training does not achieve the specialization required by the favorable phase-budget bound, and that the proposed method does not consistently improve length extrapolation or full-network BF16 stability. This transparency helps readers assess the scope of the findings.
Critical weaknesses
Limited novelty relative to closely related positional encodings. The central idea of representing hierarchical coordinates through different rotary dimensions substantially overlaps with prior work such as HiRoPE. Learnable combinations of multiple coordinates also have close precedents in multidimensional RoPE. The paper would benefit from a more precise account of which limitation of these existing methods motivates its design and why the proposed gates, initialization, and regularizers address that limitation. As presented, the contribution appears primarily to be a combination of existing ideas, with limited additional insight into when or why this combination should be preferable.
Missing comparisons prevent attribution of the proposed benefits. The experiments do not directly compare HARP with HiRoPE or with a learnable multidimensional RoPE using the same chunk and within-chunk coordinates. Consequently, the reported improvements over standard RoPE do not establish the independent value of the proposed components. In particular, equal-distance discrimination demonstrates a departure from a raw-distance-only positional kernel, but does not establish an advantage over existing hierarchical encodings.
The theoretical contribution is relatively modest and its practical implications are limited. The main results largely follow from phase-difference expansion, an equal-distance counterexample, and optimization of a simple upper bound. These observations are useful, but provide limited guidance beyond the immediate consequences of the parameterization. Moreover, the phase-budget bound requires a specialized configuration and appropriate choices of chunk size and frequency bounds; it is not a general guarantee for the default trained model. The paper reports that this configuration is not reached in practice, leaving a substantial gap between the theoretical motivation and the demonstrated behavior.
Evidence of practical task-level benefits remains insufficient. Much of the evaluation centers on perplexity, attention statistics, and numerical error measures. The synthetic tasks provide some supporting evidence, but their benefits are inconsistent across the two variants. Furthermore, the pretraining experiments do not activate the optional structural coordinates. The results therefore do not yet establish that the proposed encoding improves the ability to use meaningful hierarchy in realistic tasks, such as code understanding, scope resolution, or structured-document reasoning.
Questions for authors
-
What specific limitation of HiRoPE or existing multidimensional RoPE motivates HARP? Which new insight leads to the proposed parameterization and training objective, beyond combining hierarchical coordinates with learnable coefficients?
-
Can the authors provide comparisons with HiRoPE and a learnable multidimensional RoPE operating on the same coordinates? Controlled comparisons should match the training budget, coordinate definitions, and unrotated-channel allocation where applicable.
-
Which component accounts for the observed improvements: the coordinate decomposition, learnable phase mixing, RoPE-based initialization, specialization regularization, or the unrotated channels? Can targeted ablations isolate their individual contributions?
-
What useful prediction does the phase-budget analysis make for the trained models, given that the required specialization is not achieved? How should the claim be interpreted for a model with fixed chunk size and retained mixed bands?
-
Can the authors evaluate HARP on realistic tasks with explicit structural coordinates and compare fixed chunk boundaries against semantic boundaries? Such experiments would help determine whether the method learns useful hierarchical relationships rather than sensitivity to an arbitrary chunk grid.
-
How should the claim of exact RoPE initialization be reconciled with the default allocation of 12.5% unrotated channels, given that the stated full-head equivalence requires ?
Thank you for the careful review, and for recognizing the clarity of the formulation, the kernel analysis and our transparent reporting. We address each point below with new experiments.
W1 and Q1 (novelty relative to HiRoPE and multidimensional RoPE): HARP builds on HiRoPE, RoPE-Mixed, LieRE and ComRoPE. We see its contribution as:
We will position HARP in related work as a RoPE-anchored, learnable generalization of HiRoPE-style coordinates, which we hope addresses your concern.
W2, Q2 and Q3 (comparisons and ablations): We implemented HiRoPE (high-frequency blocks take r_i, low-frequency blocks take c_i) and 2D-Mixed RoPE (learnable per-block mixing with random init, i.e. RoPE-Mixed) on the same (c_i, r_i) coordinates with B = 256, plus single-component ablations. Synthetic probes, accuracy %, 3 seeds:
HARP-SB is above both HiRoPE and 2D-Mixed RoPE on both tasks, which resolves this point. Among the components, the specialization regularizer has the largest effect (scope 4.3 → 46.1, with wide seed variation). LF-RoPE reaches similar scope accuracy, indicating that learnable frequencies contribute substantially, since scope programs fit inside one chunk. On scope-straddle, where programs cross a chunk boundary, all encodings tested are near ceiling (95.6 to 98.9%). We will replace the single-seed 42.5% with these three-seed means. We also note that HiRoPE separates equal-distance pairs too, so that probe distinguishes HARP from raw-distance encodings specifically.
At 160M (three seeds, matched schedule), HARP-SB reaches 42.81 ± 1.86 vs RoPE 42.93 ± 1.85, lower on every seed. RoPE-skip4 (42.90) and LF-RoPE (42.89) fall in between on average but exceed RoPE on seed 456. Due to the limited time of the rebuttal period, we could not run the 160M comparison with HiRoPE and 2D-Mixed RoPE, or scope-straddle for these baselines, and will report them in the final version.
W3 and Q4 (theory and the O(√L) budget): We agree the results are compact, but they are precise: they state exactly when HARP is RoPE, when it is translation-invariant, and what separated bands can achieve. For trained models the analysis makes a conditional prediction that we verified: kernel-level BF16 drift tracks the local-band gate g_{m,g}. With specialized gates, drift is 2 to 9.7× below RoPE. With the trained gates (g_{m,g} ≈ 1.03) it matches RoPE (1.059 vs 1.070), and per-band phases grow linearly in L (Table 14). We will present O(√L) as the budget of a reachable configuration that standard training does not yet select, answering your question.
W4 and Q5 (realistic tasks with structure coordinates): All pretraining runs use S = 0, so we do not yet claim gains on structured inputs. A fair test needs a pipeline that extracts ordered coordinates (e.g. indentation or AST depth) and longer pretraining. Due to the limited rebuttal time, we could not complete this and will report it in the final version. We did test sensitivity to the grid: shifting the chunk origin by o ∈ {64, 128, 192} at inference changes no HARP-SB checkpoint's perplexity by more than 0.02% (at 4096 and 16384 tokens), so HARP is robust to where boundaries fall, easing this concern.
Q6 (RoPE initialization with unrotated channels): Thank you for pointing this out. With d_skip > 0, the initialization equals RoPE on the rotated blocks, i.e. partial RoPE at step 0. HARP-SB leaves the four lowest-frequency blocks unrotated (rotary fraction 87.5%). These rotate by at most 1.3 rad within 4096 tokens, so the effect inside the training context is small. We will restate Corollary 1 and the initialization claim for the rotated subspace, clarifying this point. The matching RoPE-skip4 control behaves close to RoPE (scope 5.1 vs 6.5, two-hop 6.0 vs 6.3, 160M PPL 42.90 ± 1.88 vs 42.93 ± 1.85).
We hope these new results and clarifications address your concerns, and we would be grateful if you would consider increasing your score.