W1. The method is an additive attention-logit bias on top of unchanged RoPE, not a repair of RoPE; the title and framing overstate what is modified.
The intervention leaves every rotary frequency untouched and adds a content-independent linear distance penalty −τ(i−j)/H_h to the attention logits (Eq. 2; Sec. 2.2, “RoPE frequencies remain unchanged”). The authors describe the operator themselves as “an additive linear bias from this existing family” of ALiBi/KERPLE/FIRE (Sec. 6), and the one variant that did touch RoPE — replacing each head’s rotary base by H_h/(2π) — was abandoned because it caused large phase changes (Sec. 2.3). What is evaluated is therefore ALiBi with per-head calibrated slopes, served through FlashAttention’s ALiBi interface, trained for 30 updates on a frozen-geometry RoPE model.
Calling this a “RoPE extrapolation repair” (title, abstract, Sec. 1) is misleading: nothing about RoPE’s extrapolation behaviour is repaired. The rotary scores are simply overridden at long distance by a recency prior, and the diagnostic B, as the authors concede, “detects need rather than anything repair-specific” (App. K). The paper should (i) reframe the contribution as a horizon-calibrated ALiBi bias for RoPE models, and (ii) justify why a bias on top of unchanged RoPE should be preferred over methods that do modify the positional geometry, given that on the only public checkpoint where they are compared the RoPE-modifying baseline is far better at range (Table 2, Pythia-1b 16K late-token PPL: 17.3 for continued YaRN and 27.1 for training-free SelfExtend vs. 144.0 for the proposed repair).
W2. The repair does not improve the model. It trades a late-token perplexity reduction on PG19 for degradation on every other measurement, and the cross-domain cost is large and unstable. (Main concern.)
The only quantity that improves anywhere in the paper is late-token perplexity on PG19 books — the training corpus of the development family (Sec. 3) and the only long-document corpus used for the public panel (Table 1, “eight PG19 books per cell”). Every in-window measurement gets worse. On the three parents, PG19 validation PPL rises by +0.12/+0.11/+0.12 (Table 4), and WikiText-103 PPL rises by +17.8/+1.4/+5.7 points relative to the matched control (Fig. 2b) — a cost that varies by more than an order of magnitude across seeds of the same recipe, which the authors themselves call “much less stable than the in-domain cost” (App. K). The same pattern holds on the public checkpoint with the full comparison: on Pythia-1b the repair raises both PG19 (14.648 → 14.746) and WT-103 (12.731 → 13.059) in-window PPL over the matched continuation (Table 2).
The 300-update study (Fig. 4, App. C) shows that the WT-103 cost can be trained down to +0.30 at τ = 1, but this is a single seed-0 dose selected from a ladder after the fact (App. K), and even then it only returns the model to the control’s level. No configuration anywhere in the paper improves in-window or cross-domain perplexity, and no late-token result is reported on any corpus other than PG19, so the one benefit that is claimed has not been shown to transfer across domains at all. Taken together, the evidence supports “a recency prior suppresses the perplexity blow-up beyond 2,048 tokens on PG19,” not “the repair helps.” To support the latter the paper needs (a) late-token PPL on at least one non-PG19 long-document corpus for both the development family and the public panel, (b) the in-window and WT-103 cost reported for every public checkpoint, not only Pythia-1b, and (c) multi-seed results for the 300-update configuration before its cross-domain cost can be called small.
W3. The paper does not sufficiently justify why distance penalties calibrated to each head’s attention horizon should improve long-context performance.
The penalty tau*(i-j)/H_h in Figure 1 suppresses distant attention more strongly for heads with smaller H_h and less strongly for heads with larger H_h, at the same query–key distance. This design can be interpreted as reinforcing local attention in heads that already attend locally while allowing relatively greater access to distant information in heads with broader attention spans. However, it remains unclear why reinforcing these existing distance preferences should benefit long-context processing. Attention patterns observed on short calibration sequences do not necessarily reflect a head’s appropriate role in longer contexts, and a predominantly local head may still need to access distant information for certain inputs. Further justification is therefore needed for using the measured horizon to determine penalty strength, along with evidence that this design improves long-context performance while preserving access to necessary distant information.
W4. Perplexity is the only outcome. No standard long-context benchmark is reported, and the paper’s own task-level evidence is at floor or negative.
The authors cite RULER and LongBench in Related Work (Sec. 6) but evaluate neither, although the fired public checkpoints (OLMo-2-7B, Pythia-1b/2.8b, TinyLlama) are exactly the models on which these benchmarks are routinely run. The Discussion itself warns that “lower loss does not establish decoded retrieval” (Sec. 7). The retrieval evidence that is offered instead does not fill the gap: the needle “margin” is a teacher-forced answer-NLL contrast, not accuracy (Sec. 3); passkey exact-match is at floor for every arm, repair included (App. G); and in the 16,380-token key-swap experiment the content-selective part of the gain is ≤ 1.1 nats at 2,048 filler tokens and ≈ 0 from 4,096 onward on all three seeds, so what survives at range is, in the authors’ words, “symmetric scene-level suppression, not recovered content-addressing” (App. I).
Where decoding is measured, the repair loses or harms. On Pythia-1b it decodes 0.00 of passkeys and 0.00 of word keys, while training-free SelfExtend reaches 0.91/0.98 and continued YaRN 0.77/0.78 (Table 7b). On Qwen3-4B all nine retrieval/tracking cells are unchanged and the 32K reasoning cell is harmed by about −27 points, a harm that persisted under a halved cap and lower τ (App. F). The appendix states plainly that “nothing in this paper claims demonstrated long-context retrieval” (App. I), yet the abstract, Fig. 1, and the contribution list present the key-swap selectivity as evidence that the repair works. Without accuracy on RULER, LongBench, or a comparably broad benchmark for the fired public checkpoints, a late-token PPL reduction on PG19 cannot be accepted as evidence of a useful long-context model.