acceptodds
Under review as a conference paper at ICLR 2027

HARP: Hierarchically Anchored Rotary Phases for Long and Structured Contexts

11 views

Abstract

Rotary Position Embedding (RoPE) derives one fixed global phase schedule from the raw token index. The raw index must therefore encode local order, long-range distance, and structured layout at once. Recent analyses reveal the overload to long-context retrieval failures, to frequency crowding, and to fragility under the bfloat16 (BF16) format. We propose Hierarchically Anchored Rotary Phases (HARP), which factors the raw index into a coarse chunk index, a within-chunk offset, and optional ordered structure coordinates. Each coordinate governs its own band of learnable rotary phases, while a fraction of channels stay unrotated as a content bypass. The encoding comes in a general form (HARP-core) and a separated-band form (HARP-sb). We initialize both exactly on the RoPE submanifold, so training departs from RoPE only when the data rewards the departure. The factorization yields an attention kernel over relative hierarchical coordinates rather than the raw gap alone. The resulting kernel recovers RoPE exactly and strictly enlarges the positional kernel class. Separating the bands further bounds the phase budget at rather than in context length . We evaluate HARP through deterministic experiments, synthetic probes, from-scratch pretraining at M and M parameters, a BF16 analysis, and ablations. On structural tasks HARP separates equal-distance token pairs that every purely relative encoding maps to identical logits. Beyond that, the trained model concentrates attention inside the current chunk, carries non-redundant information in all three bands, and HARP-core raises scope resolution from to . On length and precision at scale, by contrast, the advantage narrows from a clear win at M to parity at M. One reason is that training specializes the global band alone, which puts the bounded phase budget out of reach. HARP therefore encodes hierarchy where RoPE encodes distance alone.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Positional Encoding, Large Language Model, Pre-training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MdSoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Hierarchically Anchored Rotary Phases (HARP), which replaces the single token-position coordinate in RoPE with a chunk index, a within-chunk offset, and optional structural coordinates. It introduces two variants, HARP-Core and HARP-SB, with learnable phase coefficients or gates, aiming to distinguish token pairs that share the same relative distance but differ in their hierarchical relationships. The paper derives the resulting attention kernel, characterizes its relationship to RoPE, and provides a conditional bound on the phase budget. Experiments include synthetic tasks, small-scale language-model pretraining, length extrapolation, and numerical precision analyses.

Critical strengths
  1. Clear and interpretable formulation. The coordinate decomposition provides a straightforward way to incorporate hierarchical relationships into rotary position encoding. The distinction between within-chunk and cross-chunk interactions is intuitive, and the mathematical formulation makes the intended inductive bias explicit.
  2. Useful characterization of the positional kernel. The analysis explains how the proposed encoding can distinguish equal-distance token pairs and identifies conditions under which its rotated components recover RoPE. These results help clarify the behavior of the parameterization.
  3. Transparent reporting of limitations and negative results. The paper acknowledges that training does not achieve the specialization required by the favorable phase-budget bound, and that the proposed method does not consistently improve length extrapolation or full-network BF16 stability. This transparency helps readers assess the scope of the findings.
Critical weaknesses

Limited novelty relative to closely related positional encodings. The central idea of representing hierarchical coordinates through different rotary dimensions substantially overlaps with prior work such as HiRoPE. Learnable combinations of multiple coordinates also have close precedents in multidimensional RoPE. The paper would benefit from a more precise account of which limitation of these existing methods motivates its design and why the proposed gates, initialization, and regularizers address that limitation. As presented, the contribution appears primarily to be a combination of existing ideas, with limited additional insight into when or why this combination should be preferable.

Missing comparisons prevent attribution of the proposed benefits. The experiments do not directly compare HARP with HiRoPE or with a learnable multidimensional RoPE using the same chunk and within-chunk coordinates. Consequently, the reported improvements over standard RoPE do not establish the independent value of the proposed components. In particular, equal-distance discrimination demonstrates a departure from a raw-distance-only positional kernel, but does not establish an advantage over existing hierarchical encodings.

The theoretical contribution is relatively modest and its practical implications are limited. The main results largely follow from phase-difference expansion, an equal-distance counterexample, and optimization of a simple upper bound. These observations are useful, but provide limited guidance beyond the immediate consequences of the parameterization. Moreover, the O(L)O(\sqrt{L}) phase-budget bound requires a specialized configuration and appropriate choices of chunk size and frequency bounds; it is not a general guarantee for the default trained model. The paper reports that this configuration is not reached in practice, leaving a substantial gap between the theoretical motivation and the demonstrated behavior.

Evidence of practical task-level benefits remains insufficient. Much of the evaluation centers on perplexity, attention statistics, and numerical error measures. The synthetic tasks provide some supporting evidence, but their benefits are inconsistent across the two variants. Furthermore, the pretraining experiments do not activate the optional structural coordinates. The results therefore do not yet establish that the proposed encoding improves the ability to use meaningful hierarchy in realistic tasks, such as code understanding, scope resolution, or structured-document reasoning.

Questions for authors
  1. What specific limitation of HiRoPE or existing multidimensional RoPE motivates HARP? Which new insight leads to the proposed parameterization and training objective, beyond combining hierarchical coordinates with learnable coefficients?

  2. Can the authors provide comparisons with HiRoPE and a learnable multidimensional RoPE operating on the same (ci,ri)(c_i,r_i) coordinates? Controlled comparisons should match the training budget, coordinate definitions, and unrotated-channel allocation where applicable.

  3. Which component accounts for the observed improvements: the coordinate decomposition, learnable phase mixing, RoPE-based initialization, specialization regularization, or the unrotated channels? Can targeted ablations isolate their individual contributions?

  4. What useful prediction does the phase-budget analysis make for the trained models, given that the required specialization is not achieved? How should the O(L)O(\sqrt{L}) claim be interpreted for a model with fixed chunk size and retained mixed bands?

  5. Can the authors evaluate HARP on realistic tasks with explicit structural coordinates and compare fixed chunk boundaries against semantic boundaries? Such experiments would help determine whether the method learns useful hierarchical relationships rather than sensitivity to an arbitrary chunk grid.

  6. How should the claim of exact RoPE initialization be reconciled with the default allocation of 12.5% unrotated channels, given that the stated full-head equivalence requires dskip=0d_{\mathrm{skip}}=0?

1 comment

Thank you for the careful review, and for recognizing the clarity of the formulation, the kernel analysis and our transparent reporting. We address each point below with new experiments.

W1 and Q1 (novelty relative to HiRoPE and multidimensional RoPE): HARP builds on HiRoPE, RoPE-Mixed, LieRE and ComRoPE. We see its contribution as:

  1. RoPE is contained exactly. HiRoPE's fixed split is not RoPE, whereas HARP starts on the RoPE submanifold of its rotated subspace, and any departure is learned and measurable (Table 2).
  2. A precise account of when it departs from RoPE: shift-equivariance holds iff α_m ≡ Bβ_m (mod 2π) (Prop. 3), the kernel class strictly contains the raw-gap class (Prop. 1), and trained models occupy the larger class (Table 8).
  3. An analysis of what training does with this freedom: the global specialist emerges, while the bounded phase budget is not reached.

We will position HARP in related work as a RoPE-anchored, learnable generalization of HiRoPE-style coordinates, which we hope addresses your concern.

W2, Q2 and Q3 (comparisons and ablations): We implemented HiRoPE (high-frequency blocks take r_i, low-frequency blocks take c_i) and 2D-Mixed RoPE (learnable per-block mixing with random init, i.e. RoPE-Mixed) on the same (c_i, r_i) coordinates with B = 256, plus single-component ablations. Synthetic probes, accuracy %, 3 seeds:

scopetwo-hop
RoPE6.5 ± 1.56.3 ± 1.0
HiRoPE5.5 ± 2.34.7 ± 0.6
2D-Mixed RoPE (= random init)5.9 ± 1.41.3 ± 0.5
LF-RoPE (no factorization)22.8 ± 17.34.8 ± 0.4
RoPE-skip4 (skip channels only)5.1 ± 2.26.0 ± 2.2
HARP-Core, skip = 03.8 ± 0.22.0 ± 0.4
HARP-Core4.3 ± 1.92.2 ± 1.1
HARP-Core + warm-anchor reg.46.1 ± 33.25.7 ± 3.4
HARP-SB18.9 ± 15.17.0 ± 0.5

HARP-SB is above both HiRoPE and 2D-Mixed RoPE on both tasks, which resolves this point. Among the components, the specialization regularizer has the largest effect (scope 4.3 → 46.1, with wide seed variation). LF-RoPE reaches similar scope accuracy, indicating that learnable frequencies contribute substantially, since scope programs fit inside one chunk. On scope-straddle, where programs cross a chunk boundary, all encodings tested are near ceiling (95.6 to 98.9%). We will replace the single-seed 42.5% with these three-seed means. We also note that HiRoPE separates equal-distance pairs too, so that probe distinguishes HARP from raw-distance encodings specifically.

At 160M (three seeds, matched schedule), HARP-SB reaches 42.81 ± 1.86 vs RoPE 42.93 ± 1.85, lower on every seed. RoPE-skip4 (42.90) and LF-RoPE (42.89) fall in between on average but exceed RoPE on seed 456. Due to the limited time of the rebuttal period, we could not run the 160M comparison with HiRoPE and 2D-Mixed RoPE, or scope-straddle for these baselines, and will report them in the final version.

W3 and Q4 (theory and the O(√L) budget): We agree the results are compact, but they are precise: they state exactly when HARP is RoPE, when it is translation-invariant, and what separated bands can achieve. For trained models the analysis makes a conditional prediction that we verified: kernel-level BF16 drift tracks the local-band gate g_{m,g}. With specialized gates, drift is 2 to 9.7× below RoPE. With the trained gates (g_{m,g} ≈ 1.03) it matches RoPE (1.059 vs 1.070), and per-band phases grow linearly in L (Table 14). We will present O(√L) as the budget of a reachable configuration that standard training does not yet select, answering your question.

W4 and Q5 (realistic tasks with structure coordinates): All pretraining runs use S = 0, so we do not yet claim gains on structured inputs. A fair test needs a pipeline that extracts ordered coordinates (e.g. indentation or AST depth) and longer pretraining. Due to the limited rebuttal time, we could not complete this and will report it in the final version. We did test sensitivity to the grid: shifting the chunk origin by o ∈ {64, 128, 192} at inference changes no HARP-SB checkpoint's perplexity by more than 0.02% (at 4096 and 16384 tokens), so HARP is robust to where boundaries fall, easing this concern.

Q6 (RoPE initialization with unrotated channels): Thank you for pointing this out. With d_skip > 0, the initialization equals RoPE on the rotated blocks, i.e. partial RoPE at step 0. HARP-SB leaves the four lowest-frequency blocks unrotated (rotary fraction 87.5%). These rotate by at most 1.3 rad within 4096 tokens, so the effect inside the training context is small. We will restate Corollary 1 and the initialization claim for the rotated subspace, clarifying this point. The matching RoPE-skip4 control behaves close to RoPE (scope 5.1 vs 6.5, two-hop 6.0 vs 6.3, 160M PPL 42.90 ± 1.88 vs 42.93 ± 1.85).

We hope these new results and clarifications address your concerns, and we would be grateful if you would consider increasing your score.

Reviewer_ZjoQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes HARP, a positional encoding that decomposes token position into a chunk index, a within-chunk offset, and optional structure coordinates. Different rotary bands are intended to capture different positional relationships. The paper develops the mathematical formulation and evaluates the method through synthetic tasks and small-scale language-model pretraining.

Critical strengths
  1. The central idea is intuitive. Separating local order from cross-chunk relationships could provide a useful inductive bias for structured inputs.
  2. The mathematical derivation is clearly presented and appears rigorous, although I have not independently verified all proofs.
  3. The paper is well structured and clearly written.
Critical weaknesses
  1. The empirical results do not establish a compelling practical benefit. The paper does not clearly demonstrate which important limitation of existing LLMs HARP resolves. Given the long-context capabilities of current RoPE-based models, the proposed method needs stronger evidence of improvements on meaningful downstream tasks.
  2. The motivation for fixed-size chunks is insufficiently justified. Dividing text every 256 tokens does not necessarily capture meaningful hierarchy. A boundary may fall within a sentence, while closely related content may span multiple chunks. HARP consequently treats some neighboring token pairs differently because they cross an arbitrary boundary. Since the natural-text pretraining experiments do not use explicit structure coordinates, it remains unclear whether HARP captures useful hierarchy or merely adapts to the imposed chunk grid.
  3. The claimed limitations of RoPE need stronger practical justification. Linear growth of rotary phase does not, by itself, imply that RoPE becomes ineffective at long contexts. Comparisons against RoPE with FP32 phase computation and appropriate context-extension methods would help establish whether HARP addresses a limitation that remains under practical implementations. This is particularly important because the trained HARP models do not demonstrate improved numerical stability or length generalization.
  4. The reported improvements are inconsistent across scales and tasks. HARP-SB improves perplexity at 160M, but the methods are approximately tied at 420M, and HARP performs worse at several lengths beyond the training window. The synthetic gains are also variant-dependent: HARP-Core improves scope resolution but underperforms RoPE on two-hop retrieval, whereas HARP-SB shows the opposite pattern. Same-gap discrimination confirms the intended positional behavior, but does not establish that this behavior translates into useful downstream improvements.
Questions for authors
  1. Do meaningful boundaries, such as paragraph or document boundaries, outperform fixed-size chunks? How sensitive is HARP to shifting the chunk origin?
  2. Does HARP offer an advantage over RoPE with FP32 phase computation and matched context-extension techniques?
  3. Can the proposed local/global frequency specialization be achieved during training without degrading language-model performance?
  4. How do the authors reconcile the motivation with the long-context capabilities of RoPE-based models such as DeepSeek-V4.1-Flash? Is there evidence that the targeted limitation remains practically significant at larger model scales?
1 comment

Thank you for the thoughtful review and constructive questions. We ran several new experiments in response. We hope they address your concerns and that you will consider raising your score.

W1 and Q4 (practical benefit and long-context RoPE models): We see HARP as complementary to, not a replacement for, how production models reach long context (FP32 phase, base scaling, YaRN and long-context training). HARP keeps the rotary form, so these apply to it directly. Uniquely, its kernel can distinguish equal-distance token pairs that any raw-distance encoding must treat identically (Table 8), at a cost of 112 scalars and an unchanged KV cache. We will revise the title and introduction to reflect this focus. We also evaluated the 160M checkpoints zero-shot (accuracy %, mean ± std, seeds in brackets):

160MHellaSwagPIQAARC-eWinoGrande
RoPE (3)26.8 ± 0.155.4 ± 0.836.2 ± 0.649.2 ± 0.6
RoPE, FP32 phase (1)26.756.036.049.7
RoPE-skip4 (2)26.7 ± 0.255.3 ± 0.436.8 ± 0.050.4 ± 0.3
LF-RoPE (1)26.654.636.251.2
HARP-SB (3)26.4 ± 0.255.4 ± 0.636.7 ± 0.350.6 ± 2.4
HARP-SB, FP32 phase (1)26.956.737.551.8

With FP32 phase, HARP-SB is best on all four tasks and beats RoPE on each (+0.2 to +2.1), and in BF16 it matches or exceeds RoPE on three of four. Most gaps are within seed variation (FP32 rows: one seed), so HARP adds expressivity at no downstream cost, addressing your concern.

W2 and Q1 (fixed-size chunks): We shifted the chunk grid at inference by an offset o (160M, mean of 3 seeds, FP32 eval):

o = 0o = 64o = 128o = 192
HARP-SB @ 409642.8042.8042.8042.80
HARP-SB @ 1638494.3494.3494.3394.33

No checkpoint changes by more than 0.02%, so HARP is robust to where the boundaries fall and does not overfit to the grid position, which resolves this concern. On synthetic probes where structure is independent of the grid (accuracy %, 3 seeds):

scopescope-straddletwo-hop
RoPE6.5 ± 1.597.4 ± 0.96.3 ± 1.0
LF-RoPE22.8 ± 17.395.6 ± 2.64.8 ± 0.4
HARP-SB18.9 ± 15.198.9 ± 1.17.0 ± 0.5

HARP-SB improves scope on average over RoPE (with high variance) and is slightly above it on the other two tasks. Regarding meaningful boundaries, Theorem 1 only needs the phase to be linear in (c, r), so it extends to paragraph or document segments (exact RoPE recovery requires i = Bc + r). This needs a new data pipeline, and due to the limited time of the rebuttal period we could not complete it. We will report it in the final version and discuss the fixed grid as a limitation.

W3 and Q2 (FP32 phase and context extension): Thank you for this suggestion. In our code every rotary method, RoPE included, computes the phase in activation precision (BF16 under autocast), so the comparisons are matched, ruling out a precision bias. Following your suggestion, we retrained at 160M with FP32 phase (val PPL, seed 42):

160MPPL @ 4096PPL @ 16384
RoPE, BF16 phase43.2281.4
RoPE, FP32 phase42.9175.3
HARP-SB, BF16 phase43.0580.8
HARP-SB, FP32 phase42.0075.1

FP32 phase benefits both methods, and HARP-SB is lower than RoPE in every setting: by 0.17 / 0.91 PPL (BF16 / FP32) within the training length and 0.6 / 0.2 beyond it, so it extrapolates at least as well. We will adopt FP32 phase as the main setting. Due to limited rebuttal time, we could not run matched context extension on HARP checkpoints and will report it in the final version.

W4 (consistency across scales and tasks): We reran the 160M comparison with three seeds under a matched schedule (the Table 7 entry was read at step 3500 of a 6000-step schedule): RoPE 42.93 ± 1.85 and HARP-SB 42.81 ± 1.86, with HARP-SB lower on every seed (−0.11 ± 0.04). The 420M models saw 4× fewer tokens (0.23B vs 0.92B), and there HARP-SB is also lower than RoPE on every seed (57.03 / 62.18 / 65.00 vs 57.61 / 63.26 / 65.30), and we will describe them as a lower-token regime. On the synthetic tasks we now report HARP-SB throughout, with three seeds. This answers your question.

Q3 (specialization without a perplexity cost): With the current objective, specialization comes at some cost (59.2 vs 58.1 PPL at 4096, 92.1 vs 85.4 at 16384), since the sparsity penalty mainly drives the global specialist. Objectives that reach the specialized configuration without this cost are a promising direction. We will present the O(√L) budget as a property of a reachable configuration.

We hope these new results and clarifications address your concerns, and we would be grateful if you would consider increasing your score.

Reviewer_xQgaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RoPE applies one phase schedule to the raw token index, which must then encode local order, long-range distance, and document structure at once. HARP factors each index into a chunk index ci=⌊i/B⌋c_i = \lfloor i/B \rfloor, a within-chunk offset rir_i, and optional structure coordinates uiu_i, each with its own learnable rotary frequencies, while some channels stay unrotated. HARP-Core learns free per-block coefficients αm,βm\alpha_m, \beta_m. HARP-SB gates the coarse and fine RoPE frequencies per block and splits blocks into local, global, mixed, and skip bands under a sparsity regularizer. Both start exactly on the RoPE submanifold and train with a decaying anchor regularizer

Critical strengths
  1. The theory pins down exactly what HARP adds to RoPE. Thm. 1 and Prop. 3 are correct as proved in App. A.1. With uiu_i empty, as in every pretraining run, Eqs. 3 and 7 give each block phase as βm(i−j)+δmΔcij\beta_m(i-j) + \delta_m \Delta c_{ij} with δm=αm−Bβm\delta_m = \alpha_m - B\beta_m: the RoPE phase at frequency βm\beta_m plus a fixed offset added each time a pair crosses a chunk boundary. Prop. 3 shows that shift invariance is lost exactly when some δm≢0(mod2π)\delta_m \not\equiv 0 \pmod{2\pi}. This makes the method well defined and reduces the empirical question to one term: does δmΔcij\delta_m \Delta c_{ij} help? The theory does not answer it.

  2. Transparent reporting of where the design fails. After training, the local band keeps its chunk term, so its phase grows linearly with length and the bounded-band assumption of Prop. 2 fails (Table 14). Forcing the specialized gates costs 1.1 to 11.6 perplexity (Table 15), network-level BF16 logit error is about 6% higher than RoPE's (Table 13), and perplexity degrades faster beyond the training length (Table 11). These negative results are useful to anyone working on hierarchical rotary encodings.

Critical weaknesses
  1. The paper does not show that the factorization helps. The 160M perplexity gain (39.2 vs 41.3, Table 7) comes from one seed; at 420M with three seeds HARP and RoPE are level (Fig. 3). The comparison with RoPE also changes more than the factorization: HARP leaves 4 of 32 blocks unrotated, and HARP-SB's gates rescale each block's frequency, so with tied gates it is already learnable-frequency RoPE (Cor. 1). p-RoPE controls for the first only if it leaves the same blocks unrotated, which the paper does not state, and no run controls for the second.

Two further results point against the factorization. The chunk-size sweep (Table 3) improves monotonically up to B=1024B = 1024; as B→LB \to L every token has ci=0c_i = 0 and HARP-SB collapses to RoPE with a learned local gate, so the ablation rewards less hierarchy. And each headline uses whichever variant wins: HARP-Core for scope (42.5%; HARP-SB 6.0% vs RoPE 4.7%), HARP-SB for two-hop (18.6%; HARP-Core 4.5% vs RoPE 9.9%) and for 160M perplexity (HARP-Core 41.1 ≈\approx RoPE 41.3). The comparison of record, HARP-SB, wins only one of the two hierarchical probes and loses two-hop at 420M (Table 16, 0.0 for all).

The synthetic probes do not fill this gap. The tasks in Table 1 are described only by name, use one seed, and App. D indicates that HARP receives the task structure through uiu_i while RoPE cannot. Since the perplexity and synthetic results are the only evidence that the factorization helps, that claim is not yet supported.

  1. The two settings in the title are not supported.

Long contexts: HARP-SB degrades faster than RoPE beyond the training length (85.4 vs 72.7 at 16k, Table 11). The BF16 gain requires gates that training does not reach, and forcing them costs perplexity. Structured contexts: the structure channel is empty in every pretraining run. On natural text the only hierarchy HARP encodes is a fixed 256-token grid, which ignores document structure.

The paper's framing is therefore contradicted for long contexts and, for structured ones, rests on the synthetic probes of weakness 1.

Questions for authors
  1. Please define the four synthetic tasks of Table 1 (input format, sequence length, model size, training steps, seeds) and state the units of the raw-gap column (1.0 / 0.9 vs percentages elsewhere). Where do the scope and boundary positions fall relative to the chunk grid, and does HARP receive the structure through uiu_i while RoPE does not? If the task structure is independent of the grid, RoPE gets the same information, and the gains hold over three seeds, I would consider the structural claim supported.
  2. Please report the 160M HARP-SB vs RoPE comparison over three seeds, and the per-seed RoPE values at 420M (App. C.1 lists them only for HARP-SB). If HARP-SB beats RoPE on every seed by more than the seed-to-seed spread of the differences, the single-seed concern is resolved.
  3. Which frequencies do the four skip blocks replace, and what unrotated fraction does p-RoPE use? Eq. 8 and Table 6 list two gates per block, while App. C.5 counts four learned scalars per block (112 total); which is correct? Two 160M controls would isolate the factorization: RoPE with the same four blocks unrotated, and HARP-SB with tied gates. If HARP-SB beats both across seeds, together with question 2 this resolves the perplexity part of weakness 1.
  4. What batch size and how many training tokens do the 160M and 420M runs use (Table 4)? The 420M models end at higher validation perplexity than the 160M models after the same 3500 steps (Fig. 3 vs Table 7). If the 420M runs saw fewer tokens, the 420M parity may reflect under-training rather than scale; if they saw the same number, the 420M training needs an explanation before any 420M comparison can be trusted.
  5. Why does perplexity improve monotonically with BB in Table 3, and what does HARP-SB give at B=L=4096B = L = 4096? If the single-chunk setting matches B=1024B = 1024, the sweep shows that the chunk term does not help at this scale.
1 comment

Thank you for such a careful review. The additional experiments below, prompted by your questions, have made the paper more rigorous. We hope they address your concerns and that you will consider raising your score.

Q1 (synthetic tasks): All tasks use 6 layers, d=256, 4 heads, length 512, chunk size B=256, 20k steps at batch 16, LR 1e-3, and 3200 test examples. In raw-gap, a KEY and value appear at a random position, followed by a QUERY. In boundary, a SEP is followed by a QUERY labeled SAME or CROSS by the grid. Scope has nested OPEN/CLOSE blocks with a shadowed variable. Two-hop has 3 to 6 sections of key=value pairs, and the query names a section and a key. Numbers are percentages (chance ≈ 0.5%). Raw-gap stays near chance for all methods, so we will list it as a control.

To clarify, HARP does not receive any task structure through the structure channel. All synthetic runs use S=0 with identical inputs for every method, and we will correct the appendix wording. Since scope programs (≤ ~100 tokens) fit inside one chunk, we added scope-straddle, where each program crosses a chunk boundary. We also added your two Q3 controls: RoPE-skip4 (RoPE with the same four blocks unrotated as HARP-SB) and LF-RoPE (HARP-SB with α_m = Bβ_m and tied gates, i.e. learnable frequencies without chunk factorization). Three seeds, mean ± std:

boundaryscopestraddletwo-hop
RoPE100.0 ± 0.06.5 ± 1.597.4 ± 0.96.3 ± 1.0
RoPE-skip4100.0 ± 0.05.1 ± 2.296.2 ± 1.46.0 ± 2.2
LF-RoPE100.0 ± 0.022.8 ± 17.395.6 ± 2.64.8 ± 0.4
HARP-Core100.0 ± 0.04.3 ± 1.998.9 ± 0.72.2 ± 1.1
HARP-SB100.0 ± 0.018.9 ± 15.198.9 ± 1.17.0 ± 0.5

Over three seeds, HARP-SB is well above RoPE on scope (18.9 vs 6.5) and shows better results in straddle and two-hop, though variance is high and LF-RoPE behaves similarly. We will replace the single-seed 42.5% with these averages and use HARP-SB as the reference variant. The two-hop trace in Table 16 does not separate methods. This answers your question.

Q2 (160M over three seeds, 420M per seed): On re-examination, the 39.2 in Table 7 was taken at step 3500 of a 6000-step schedule on an earlier pipeline, so we reran all models with matched schedule, pipeline and evaluation (val PPL at step 3500), including both controls:

160Ms42s123s456mean ± std
RoPE43.2240.5345.0342.93 ± 1.85
RoPE-skip443.1040.5045.1042.90 ± 1.88
LF-RoPE43.0840.5045.0842.89 ± 1.87
HARP-SB43.0540.4244.9742.81 ± 1.86
Δ (HARP-SB vs RoPE)−0.17−0.11−0.06−0.11 ± 0.04

HARP-SB beats RoPE on every seed by more than the spread, meeting your criterion. The matched margin is 0.11 PPL, and on seed 42 the controls account for much of it, but only HARP-SB improves on RoPE on every seed, so we will present it as a consistent but modest gain rather than attribute it to the factorization alone.

420M (best val PPL)s42s123s456
RoPE57.6163.2665.30
HARP-SB57.0362.1865.00
HARP-Core56.4562.1865.22
p-RoPE65.8965.7867.68

HARP-SB beats RoPE on every seed at this scale, resolving your concern.

Q3 (skip blocks, p-RoPE, gates): HARP-SB keeps the four lowest-frequency RoPE blocks unrotated (wavelengths 19.9k–47k). p-RoPE uses a rotary fraction of 0.5, so RoPE-skip4 is the more direct control. App. C.5 gives the correct count: each block learns α_m, β_m, g_{m,g} and g_{m,l} (28×4 = 112 scalars), and we will update Eq. 8 and Table 6. Control results are in Q1 and Q2: RoPE-skip4 stays close to RoPE, so skip blocks do not explain the gains, addressing your concern.

Q4 (tokens): Thank you for raising this. The 160M runs use 64×4096 tokens per step (0.92B total) and the 420M runs 16×4096 (0.23B), which accounts for the higher 420M perplexity. We will state the budgets in Table 4 and describe the 420M results as a lower-token regime rather than a scaling study, settling this point.

Q5 (chunk size): As B approaches L, HARP-SB approaches RoPE with a learned local gate. We ran larger chunk sizes under the 500-step protocol, with B=1024 rerun as an anchor:

B102420484096
train-stream PPL173.8172.9173.2
held-out PPL182.5181.8182.2

Perplexity is stable for B ≥ 1024, so at this scale the chunk size does not change perplexity, and we will state this in the revision, answering your question.

W2 (title): We appreciate this point. We will revise the title and framing to focus on the factorized rotary kernel itself: what it can express, how training uses it, and the settings where its benefits appear, in line with your suggestion.

We hope these clarifications and new results address your concerns, and we would be grateful if you would consider increasing your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.