- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes HRDP, a probe that places the same marker token at the query position and at a controlled offset earlier, and scores each head by the fraction of samples in which the target key wins the pre-softmax argmax over all preceding keys. On Llama-3-8B it reports that the share of heads with at least one success falls from ~20% at to ~5% at , collapses beyond the 8K training length even at , that ablating the high or mid-low frequency band hurts at both offsets while the low band barely matters, and that norms do not track these effects. HRDP is then used to compare Linear PI and YaRN at 32K and to pick the YaRN factor () whose 32K profile is closest to the 8K profile; after QLoRA fine-tuning this beats on RULER@32K (62.99 vs 47.94). Additional default-RoPE profiles for Qwen3 and Gemma models are in the appendix.
Critical strengths
- The probe. HRDP fixes the exact query–key offset, works at the logit level, and is cheap and reproducible. The head-level facts it produces (12 of 1,024 heads reach 0.8; distance-specialized heads such as (0, 9) for and (10, 2) for , App. D) are new and checkable. This is the part of the paper I would want to see published.
- HRDP as a selection signal for context extension. Choosing the YaRN factor by profile preservation (Eqs. 7–8) is a simple idea, and the RULER@32K gain is large. If it holds under proper comparison it is a useful practical result.
Critical weaknesses
-
The RoPE “understanding” claims are largely predicted by the frequency schedule and by prior work, and the paper frames them as a challenge to intuition. With , the low band (blocks 48–63) rotates at most 0.11 rad over , and most of it under 0.05 rad. Replacing it with NoPE is close to a no-op at every tested offset, so the small effect of the low-band ablation follows from the grid and says nothing about long-range retrieval, which for this band begins around tokens. Barbero et al. (2025) already showed that the lowest frequencies act as near-NoPE semantic channels and can be removed (p-RoPE) without loss. The mid-low band (wavelengths ≈4K–90K tokens) is the one that resolves the tested offsets, and the high band is needed to separate the target from its immediate neighbours under a Top-1 criterion; both are the standard multi-scale argument, relabeled. The paper argues against the coarse “higher base = longer range” reading of Xiong et al. (2024) while not citing Hong et al. (Findings of EMNLP 2024), whose dimension-level analysis and ablation find positional heads shifting to lower-frequency dimensions with distance, which is the direct prior claim and points the other way. The -norm section is qualitative, uses query norms only, and the norm-dominant mid-low region is also among the most damaging to ablate, so the data support at most “norms do not flag the high band”. Two further confounds limit what §4.3 can show: the NoPE ablation is applied to all keys (and, as far as I can tell, all layers), so a drop can come from distractors gaining score or from upstream changes rather than from the target’s own score, and the probe cannot separate content matching from purely positional attention at short offsets, since a previous-token head scores 1 at with no control (e.g., a random token at the target position). The one result the geometry does not predict, mid-low ablation hurting at where the target block rotates ~0.003 rad, is the one that is left unexplained.
-
The practical claim rests on two fine-tuned configurations. vs , one run each, no variance, 8K differences within noise (90.10 vs 90.45). “Best 32K RULER performance” is the best of two. The paper does not show that the HRDP distance tracks RULER across several values of , nor compare against a cheap baseline such as selecting by 32K validation perplexity. The selection also uses the WikiText-103 test split that is later used for perplexity evaluation. Related, the “~20% of heads successfully retrieve” headline counts one success in 100 trials; only 12 heads reach 0.8.
Weakness 1 drives my rating: the paper’s title and most of its first Key Findings box describe contributions that are either predictable from the schedule or already in the literature. The probe and the selection idea are worth publishing once the frequency claims are rescoped and the selection result is backed by a sweep and a baseline.
Questions for authors
- Is the band ablation in §4.3 applied in all layers and to all keys? For the mid-low ablation at , is the drop driven by the target’s score, by distractor scores, or by upstream changes? A per-band decomposition of the target-vs-best-distractor margin would answer this.
- Did you run a positional control (random token at the target, $ at the query)? What fraction of the “active” heads survive it?
- Were the Fig. 8 maps and the grid over computed training-free (App. A.4) or after fine-tuning? §5.1 says ; the setup and figure say .
- Does the HRDP distance (Eq. 7) track RULER@32K over several fine-tuned values of , and how does compare with chosen by validation perplexity? Are Tables 1–2 single runs, and how many RULER samples per task?
- At K and 32K, which keys win the argmax instead of the target and at what offsets? This separates the absolute-position explanation in §4.2 from distractors at unseen relative distances.