acceptodds
Under review as a conference paper at ICLR 2027

When Does a RoPE Extrapolation Repair Help?

Abstract

Context-extension repairs for rotary position embeddings (RoPE) can rescue a model whose predictive performance collapses on long documents, yet harm one that already handles long inputs, so practitioners must know in advance which case applies. We study a diagnostic that predicts when one low-budget repair helps. The repair retains RoPE, adds to each attention head a distance penalty scaled to how far back that head attends, and trains for 30 updates. The diagnostic is the matched no-penalty continuation's ratio of perplexity after a document's first 2,048 tokens (late-token perplexity) to perplexity on those initial tokens. At a ratio of at least 2, a fixed rule fires and predicts at least 5% lower late-token perplexity than that continuation; otherwise, it abstains. On 30 checkpoint–length cells from ten public checkpoints, including five named before evaluation, the rule fires on 20 and all 20 improve; it abstains on exactly the 10 cells without a reduction, including native long-context Qwen3 checkpoints, where the repair raises perplexity. On three independently trained 124M parents, the repair lowers 16K late-token perplexity from 168–182 to about 25 in 74 GPU-seconds, at a cost of about 0.1 in in-window (PG19) perplexity; measured slopes beat three calibration-free schedules on late-token perplexity for every parent. In a key-swap control, a correct key just before the question raises the repaired models' answer log-likelihood 7.3–8.2 nats more than a wrong key. On Pythia-1b, the repair has the lowest in-window perplexity of the compared extensions. A single paired run thus makes the decision to apply this repair testable.

Primary area:
generative models
Keywords:
rotary position embeddings

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_SPPDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how to determine whether a RoPE-based model can benefit from long-context extrapolation repair. The authors propose a Horizon-calibrated Repair that preserves the original RoPE positional encoding while estimating each attention head's attention horizon from its attention distance distribution. Based on these estimates, the method introduces a distance-dependent attention bias and improves long-context performance through a small number of continuation training updates.

Critical strengths
  1. Existing studies on long-context extension typically focus on improving model performance beyond the training context window, while this paper further considers when such a repair is actually necessary. For models that already handle long contexts well, additional modifications to positional mechanisms may degrade their original performance. Therefore, developing a diagnostic metric to guide repair decisions has clear practical value.

  2. The authors compare different attention bias configurations as well as established methods such as SelfExtend and YaRN. They also validate the proposed diagnostic rule across multiple public checkpoints, with some checkpoints designated prospectively before evaluation, which further strengthens the credibility of the experimental findings.

Critical weaknesses

The paper demonstrates substantial improvements in late-token perplexity, but these gains do not fully translate into effective long-range information retrieval. For example, Table 2 shows that on Pythia-1b, the proposed method reduces 16K late-token perplexity from 712.3 to 144.0, whereas SelfExtend and YaRN with continuation training achieve 27.1 and 17.3, respectively. Although the proposed method incurs a smaller perplexity cost within the original context window, it still lags considerably behind these methods in long-context modeling performance. Furthermore, Table 7 shows that its decoding-based retrieval accuracy remains at zero, while the other extension methods achieve substantially higher retrieval accuracy.

Questions for authors
  1. Although the experiments show that the proposed method significantly reduces late-token perplexity, its content selectivity and decoding-based retrieval ability remain limited at longer distances. Could the authors further clarify whether these improvements mainly come from suppressing irrelevant long-range attention, or whether the model actually gains a stronger ability to utilize information from long contexts?

  2. The advantage of attention horizon calibration over fixed slope configurations has primarily been demonstrated on the 124M development family. Could the authors further clarify whether this horizon-based calibration strategy consistently provides similar benefits across different architectures and larger public models?

Reviewer_kDoPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how brief adaptation can improve RoPE models on long inputs and when this repair should be applied. The authors retain the existing positional encoding, set a linear distance penalty based on how far each attention head typically attends, and continue training for 30 updates. They use a diagnostic, B, to measure prediction deterioration on later tokens of long documents and predict whether the repair will help. Experiments cover custom-trained models and several public models, with comparisons against SelfExtend and YaRN.

Critical strengths
  • The method is simple and consistently effective on the custom-trained models. It retains the existing RoPE and only adds a distance penalty followed by brief continued training. All three independently trained small models show substantial improvements in long-document prediction, with little change in short-context performance on the same corpus. These results suggest some practical potential.
Critical weaknesses
  • Long-context performance is not strong enough in the main public-model comparison. On Pythia-1b at 16K, the method has substantially higher long-document perplexity than SelfExtend and adapted YaRN. Its decoded retrieval performance on this model is also weaker. SelfExtend requires no additional training, while YaRN receives the same number of adaptation updates as the proposed method. Although the custom-trained small models show good results, the direct comparison on a public model does not demonstrate sufficiently competitive long-context performance.

  • The main gains occur within context lengths already supported by existing long-context models. Substantial improvements mainly occur when short-context models process 8K to 32K tokens. Qwen3-8B already natively supports 32K, and Llama 3.1 supports 128K, covering or exceeding this range. The paper also evaluates native long-context Qwen models at lengths up to 128K, but the repair does not help them. Evidence that the repair can further extend the usable context window of existing long-context models therefore remains limited.

Questions for authors
  • Are there results on other public models showing better long-context performance than SelfExtend and a YaRN baseline adapted with the same training budget as the proposed method?

  • Beyond the reported Qwen results, are there experiments showing practical gains from further extending the context window of models that already natively support long contexts?

Reviewer_7o17OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a criterion for deciding whether to apply RoPE repair when a model’s predictive performance deteriorates on long documents. Since the repair can harm models that already handle long documents well, the authors compare perplexity over the first 2,048 tokens with perplexity over the subsequent tokens. When late-token perplexity is at least twice the initial-token perplexity, the criterion predicts that the RoPE repair will reduce late-token perplexity by at least 5%; otherwise, it abstains. The proposed method retains the existing RoPE, adds a distance penalty based on each head’s attention horizon, and performs 30 adaptation updates. The paper shows that this repair improves predictive performance on long documents.

Critical strengths

S1. The paper addresses a clearly defined question with well-controlled continuation experiments. The paper clearly formulates the question of when a specified RoPE repair improves late-token perplexity. The matched continuation experiments use the same initialization, training data, and update count for the repair and control arms. This design helps distinguish the combined effect of the intervention and its subsequent adaptation from the effect of additional training alone.

Critical weaknesses

W1. The method is an additive attention-logit bias on top of unchanged RoPE, not a repair of RoPE; the title and framing overstate what is modified. The intervention leaves every rotary frequency untouched and adds a content-independent linear distance penalty −τ(i−j)/H_h to the attention logits (Eq. 2; Sec. 2.2, “RoPE frequencies remain unchanged”). The authors describe the operator themselves as “an additive linear bias from this existing family” of ALiBi/KERPLE/FIRE (Sec. 6), and the one variant that did touch RoPE — replacing each head’s rotary base by H_h/(2π) — was abandoned because it caused large phase changes (Sec. 2.3). What is evaluated is therefore ALiBi with per-head calibrated slopes, served through FlashAttention’s ALiBi interface, trained for 30 updates on a frozen-geometry RoPE model. Calling this a “RoPE extrapolation repair” (title, abstract, Sec. 1) is misleading: nothing about RoPE’s extrapolation behaviour is repaired. The rotary scores are simply overridden at long distance by a recency prior, and the diagnostic B, as the authors concede, “detects need rather than anything repair-specific” (App. K). The paper should (i) reframe the contribution as a horizon-calibrated ALiBi bias for RoPE models, and (ii) justify why a bias on top of unchanged RoPE should be preferred over methods that do modify the positional geometry, given that on the only public checkpoint where they are compared the RoPE-modifying baseline is far better at range (Table 2, Pythia-1b 16K late-token PPL: 17.3 for continued YaRN and 27.1 for training-free SelfExtend vs. 144.0 for the proposed repair).

W2. The repair does not improve the model. It trades a late-token perplexity reduction on PG19 for degradation on every other measurement, and the cross-domain cost is large and unstable. (Main concern.) The only quantity that improves anywhere in the paper is late-token perplexity on PG19 books — the training corpus of the development family (Sec. 3) and the only long-document corpus used for the public panel (Table 1, “eight PG19 books per cell”). Every in-window measurement gets worse. On the three parents, PG19 validation PPL rises by +0.12/+0.11/+0.12 (Table 4), and WikiText-103 PPL rises by +17.8/+1.4/+5.7 points relative to the matched control (Fig. 2b) — a cost that varies by more than an order of magnitude across seeds of the same recipe, which the authors themselves call “much less stable than the in-domain cost” (App. K). The same pattern holds on the public checkpoint with the full comparison: on Pythia-1b the repair raises both PG19 (14.648 → 14.746) and WT-103 (12.731 → 13.059) in-window PPL over the matched continuation (Table 2). The 300-update study (Fig. 4, App. C) shows that the WT-103 cost can be trained down to +0.30 at τ = 1, but this is a single seed-0 dose selected from a ladder after the fact (App. K), and even then it only returns the model to the control’s level. No configuration anywhere in the paper improves in-window or cross-domain perplexity, and no late-token result is reported on any corpus other than PG19, so the one benefit that is claimed has not been shown to transfer across domains at all. Taken together, the evidence supports “a recency prior suppresses the perplexity blow-up beyond 2,048 tokens on PG19,” not “the repair helps.” To support the latter the paper needs (a) late-token PPL on at least one non-PG19 long-document corpus for both the development family and the public panel, (b) the in-window and WT-103 cost reported for every public checkpoint, not only Pythia-1b, and (c) multi-seed results for the 300-update configuration before its cross-domain cost can be called small.

W3. The paper does not sufficiently justify why distance penalties calibrated to each head’s attention horizon should improve long-context performance. The penalty tau*(i-j)/H_h in Figure 1 suppresses distant attention more strongly for heads with smaller H_h and less strongly for heads with larger H_h, at the same query–key distance. This design can be interpreted as reinforcing local attention in heads that already attend locally while allowing relatively greater access to distant information in heads with broader attention spans. However, it remains unclear why reinforcing these existing distance preferences should benefit long-context processing. Attention patterns observed on short calibration sequences do not necessarily reflect a head’s appropriate role in longer contexts, and a predominantly local head may still need to access distant information for certain inputs. Further justification is therefore needed for using the measured horizon to determine penalty strength, along with evidence that this design improves long-context performance while preserving access to necessary distant information.

W4. Perplexity is the only outcome. No standard long-context benchmark is reported, and the paper’s own task-level evidence is at floor or negative. The authors cite RULER and LongBench in Related Work (Sec. 6) but evaluate neither, although the fired public checkpoints (OLMo-2-7B, Pythia-1b/2.8b, TinyLlama) are exactly the models on which these benchmarks are routinely run. The Discussion itself warns that “lower loss does not establish decoded retrieval” (Sec. 7). The retrieval evidence that is offered instead does not fill the gap: the needle “margin” is a teacher-forced answer-NLL contrast, not accuracy (Sec. 3); passkey exact-match is at floor for every arm, repair included (App. G); and in the 16,380-token key-swap experiment the content-selective part of the gain is ≤ 1.1 nats at 2,048 filler tokens and ≈ 0 from 4,096 onward on all three seeds, so what survives at range is, in the authors’ words, “symmetric scene-level suppression, not recovered content-addressing” (App. I). Where decoding is measured, the repair loses or harms. On Pythia-1b it decodes 0.00 of passkeys and 0.00 of word keys, while training-free SelfExtend reaches 0.91/0.98 and continued YaRN 0.77/0.78 (Table 7b). On Qwen3-4B all nine retrieval/tracking cells are unchanged and the 32K reasoning cell is harmed by about −27 points, a harm that persisted under a halved cap and lower τ (App. F). The appendix states plainly that “nothing in this paper claims demonstrated long-context retrieval” (App. I), yet the abstract, Fig. 1, and the contribution list present the key-swap selectivity as evidence that the repair works. Without accuracy on RULER, LongBench, or a comparably broad benchmark for the fired public checkpoints, a late-token PPL reduction on PG19 cannot be accepted as evidence of a useful long-context model.

Questions for authors

Q1. Why are horizons measured on 48 held-out windows for the development family but only 24 for public models (line 140), and how were those windows chosen? Section 2.2 states the asymmetry without justification. It matters because the two groups measured under different protocols are exactly the two groups that the coverage gate separates: C = 0.708–0.771 for the family versus ≤ 0.087 for every probed ≥ 1B public map (Sec. 5, App. C). The long-document evaluation is asymmetric in the same direction — up to 24 PG19 test books for the family versus up to eight per public cell (Sec. 3). As written, a reader cannot rule out that part of the coverage separation, and part of the gap between the family’s 168 → 25 result and the public outcomes, is a protocol artifact rather than a property of the checkpoints, and the asymmetry invites the reading that the family protocol was tuned. Please (i) explain the choice, (ii) report the development family’s H_h, coverage C, and resulting late-token PPL when measured with 24 windows (and, ideally, at least one public map with 48), and (iii) state how the 24 windows and the eight books were selected.

Q2. Why is the needle margin reported at requested depth 0.9, and what does it look like across depths? In the fixed-grid battery, requested depth 0.9 realizes zero filler between key and question — the key sits 30 tokens from the scene end (Sec. 3; App. G). Because the intervention is a penalty that grows linearly with i − j, a key adjacent to the question is the one placement where the recency prior cannot hurt and can only help. The headline needle numbers (+15.4/+17.3/+17.5 nats in Table 4; all nine needle cells in Table 3) are therefore measured at the most favourable setting for the method. The paper’s own appendix shows the advantage shrinking with distance: about +6.7 nats at requested depth 0.5 (≈ 1.07K-token separation) and −2.03 to +0.27 at depth 0.1 (≈ 7.6K tokens), with content selectivity ≈ 0 beyond 2,048 filler tokens (App. G, App. I). I am not asking for depth 0 only. I am asking that the main text report the needle margin across a range of depths (e.g., 0.1/0.3/0.5/0.7/0.9) with the realized key–question distance stated for each, and that the depth-0.9 numbers be labelled as the zero-filler case wherever they appear. If, as App. G suggests, the positive addressing evidence reaches only about the first ninth of the scene, that should be the claim.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.