acceptodds
Under review as a conference paper at ICLR 2027

Training-Free Long-Context Extension by Warping Rotary Relative Distance

Abstract

Rotary position embeddings (ROPE) are the default positional mechanism in modern language models, but training-free context extension remains brittle: global position or frequency rescaling can bring distant tokens into the trained range while distorting the local geometry the model learned during pretraining. We introduce WARP (Warped Rotary Positions), a training-free method that keeps the pretrained rotary spectrum fixed and instead applies a monotone nonlinear warp to the relative query–key distance used inside rotary attention. WARP is exactly ROPE at native context length, approximately preserves local distances, and compresses only the far field into the model’s trained phase budget. Because the warp is a nonlinear function of the pairwise distance i − j, it cannot be represented by a standard per-token position cache, which makes WARP a distinct class of rotary extension from position or frequency rescaling. The evaluated configuration adds a small frequency, temperature, and density calibration and applies the warp through a block-anchored approximation. In ten-seed exact-retrieval stress tests, WARP reaches 50/50 with zero observed variance on Qwen2.5-3B-Instruct and Mistral-7B-v0.3 at both 32k and 64k and on Qwen2.5-0.5B-Instruct at 64k, and reaches 49.20/50 on Llama-2-13B at 16k; Llama-2-13B at 32k remains weak. On official RULER v1 at 64k with Qwen2.5-3B-Instruct, WARP achieves 70.82% macro accuracy, compared with 62.92% for unmodified ROPE, 71.22% for NTK, 73.31% for YaRN, and 73.44% for MrRoPE-Pro. WARP is therefore competitive on broad RULER rather than dominant; its advantage is concentrated in retrieval-heavy behavior, where it is best or tied-best among the strongest methods on several RULER retrieval and QA components. The hardest setting we observe is Llama-2-7B exact retrieval at 32k, where base WARP and every baseline score 0/50; an exploratory variant that adds head routing and a small trained readout adapter recovers up to 31/50. These results suggest that relative-distance warping is a promising, model-adaptable route to long-context extension without finetuning, while exposing architecture-specific failure modes that we report explicitly.

Primary area:
foundation or frontier models, including LLMs
Keywords:
long-context extension, rotary position embeddings, RoPE, training-free context extension

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_rXYzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces WaRP, a training-free long-context extension method that applies a monotone nonlinear transformation to pairwise query–key distances in Rotary Position Embeddings (RoPE), rather than relying exclusively on position or frequency rescaling. The proposed framework theoretically distinguishes nonlinear relative-distance warping from conventional per-token positional transformations, while its practical implementation incorporates frequency calibration, attention-temperature and density corrections, and block-anchored approximation. Experiments across six language models, including exact retrieval, perplexity, and RULER evaluations, demonstrate competitive performance in selected settings but also reveal inconsistent generalization across architectures and substantial inference overhead.

Critical strengths
  1. Interesting formulation of relative-distance warping.

The paper presents a mathematically well-motivated alternative to conventional RoPE scaling by applying a nonlinear transformation directly to pairwise relative distances, with a clear argument establishing why such transformations cannot generally be represented through standard per-token positional caches. This provides an interesting design direction for training-free context extension and highlights a potentially useful distinction between relative-distance transformations and conventional position or frequency rescaling.

  1. Comprehensive evaluation and transparent discussion of limitations.

The paper evaluates WaRP across multiple model architectures and context lengths using complementary metrics, including exact retrieval, perplexity, and RULER, while additionally examining seed variance and computational efficiency. The explicit reporting of unsuccessful configurations, particularly the Llama-2 retrieval failures, strengthens the empirical assessment and provides useful insights into the architecture-dependent behavior of context-extension techniques.

Critical weaknesses
  1. Limited empirical advantage over existing methods and insufficient comparison with the closest related approaches.

Despite its interesting formulation, WaRP achieves 70.82% accuracy on the primary 64k RULER benchmark, below YaRN (73.31%) and MrRoPE-Pro (73.44%), while several successful exact-retrieval configurations already reach the performance ceiling with existing methods. Furthermore, the absence of direct comparisons against more closely related pairwise relative-position approaches, particularly ReRoPE/Leaky ReRoPE and SelfExtend, makes it difficult to establish whether the proposed nonlinear warping formulation offers a meaningful advantage over existing alternatives.

  1. Insufficient isolation of the proposed mechanism and questionable practical trade-offs.

The evaluated implementation combines nonlinear distance warping with frequency calibration, attention-temperature adjustments, density corrections, and block-anchored approximation, but lacks systematic ablations establishing which components account for the reported improvements; moreover, the main cross-method retrieval comparison in Figure 2 predates implementation corrections and is based on a single seed. These limitations, together with the approximately threefold increase in attention-prefill latency, make it difficult to determine whether relative-distance warping itself provides a sufficiently robust and practically meaningful improvement.

Questions for authors
  1. Component Contributions and Approximation Effects

Could the authors provide systematic ablation experiments isolating the contributions of nonlinear distance warping, frequency calibration, temperature scaling, and density correction, ideally including a comparison between exact pairwise warping and the block-anchored approximation? These results are important for determining whether the proposed relative-distance transformation is the primary source of improvement or whether the gains mainly arise from auxiliary calibration mechanisms.

  1. Comparisons with Closely Related Methods

How does WaRP compare with ReRoPE/Leaky ReRoPE and SelfExtend under identical evaluation protocols, particularly on RULER and the more challenging Llama-2 configurations? Since these methods also distinguish local and distant relative positions, direct comparisons would substantially strengthen the argument for the novelty and practical significance of the proposed approach.

2 comments

Thank you.

Q2 (ReRoPE / Leaky ReRoPE / SelfExtend under identical protocols).

  • Llama-2 at 16k, held out, preregistered: Leaky ReRoPE 59.3 / 59.5 vs calibrated WaRP 31.0 / 28.4 (13B / 7B).
  • Llama-2 at 16k, development, all four pair-dependent methods: listed in our reply to 7nZg.
  • Llama-2-7B at 32k, passkey: Leaky ReRoPE and ReRoPE 50/50, DCA 46/50, SelfExtend 17/50 (16/50 under strict matching), WaRP 0/50.
  • Qwen2.5-3B at 64k:
    • fresh preregistered test: WaRP 73.07 vs Leaky ReRoPE 74.27, which met the 5-point non-inferiority margin;
    • matched retrospective panel: WaRP +2.46 vs DCA;
    • first 20 instances per task (260; retrospective): Leaky ReRoPE 77.44, SelfExtend 76.47, WaRP 72.88, DCA 68.81 (WaRP − SelfExtend −3.60 [−7.53, +0.24]); ReRoPE did not finish (15/260).

Q1 (ablations).

  • Warp vs calibration (held out): pure 15.1 / 19.4 → calibrated 31.0 / 28.4. On the length-matched 7B@16k development panel, calibration lowers perplexity from 8.62 to 7.23 but reduces retrieval from 10.0 to 5.2/50.
  • Shoulder, calibration off (development): W = 1 → N/8 lowers RULER on all three models (reply to ECbz).
  • Tail shape (exploratory development, 65 prompts): at W = 512 with native-base query scaling and calibration off, linear scores 61.7 versus 28.4 for logarithmic on 13B. Its preregistered H2 test failed overall.
  • Temperature and density: inactive at the Llama-2 16k settings. On Qwen 64k, frequency calibration adds +0.33 and temperature +1.56 RULER points (both intervals include zero); a temperature-only arm did not finish.
  • Exact vs block: 25/26 identical scores; maximum supported NLL discrepancy about 0.0033 nats/token; H3 passed at Leaky ReRoPE's geometry; one documented failure configuration.

Figure 2. We removed the single-seed, pre-correction Figure 2. The revision reports corrected five-seed WaRP retrieval separately from the historical baseline checks, and adds a length-matched five-seed comparison of all methods on Llama-2-7B. We do not present the historical checks as a corrected multi-seed comparison.

Latency. End to end in the 128-token setting, calibrated WaRP takes 1.61× (Qwen 64k) and 1.02× (Llama-2-13B 16k) the time of RoPE in the same stack. Its prefill is 5.6–5.7× / 3.0× that of FlashAttention-2 RoPE across stacks; these measurements do not isolate method overhead. Full numbers are in the appendix.

Thank you for the detailed rebuttal and additional experiments. My main concern is still the limited advantage over existing methods. While WaRP achieves comparable performance to Leaky ReRoPE on Qwen2.5-3B, it performs substantially worse on Llama-2. The authors also show that their fused kernel can be applied to Leaky ReRoPE, achieving similar inference efficiency with better performance on Llama-2. Given these results, I am still not convinced what the proposed logarithmic warping offers over existing methods in practice. Therefore, I will maintain my rating of 2.

Reviewer_7nZgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes WARP, a training-free method for extending the context window of RoPE-based language models. The method applies a nonlinear logarithmic transformation to query–key relative distances, approximately preserving nearby distances while compressing distant positions into the native context range. The evaluated configuration additionally includes frequency and temperature calibration, a density bias, and a block-anchored approximation. Experiments assess retrieval, perplexity, and RULER performance across several model families.

Critical strengths

The distance transformation is explicitly formulated. The paper specifies its endpoint constraint and explains why a nonlinear relative-distance transformation cannot generally be implemented through a single per-token absolute-position mapping.

Critical weaknesses
  1. Novelty relative to closely related methods is insufficiently established. Preserving local positional structure while modifying long-range relationships for training-free context extension is an established direction. Dual Chunk Attention (DCA) [1] is directly relevant but is neither cited nor evaluated. ReRoPE, Leaky ReRoPE, and SelfExtend are discussed but also absent from the experimental comparisons. Consequently, the results do not establish the benefit of the proposed logarithmic transformation over the closest existing approaches. The nonseparability argument distinguishes WARP from simple absolute-position remapping, but does not by itself establish novelty over prior pair-dependent methods.
  2. The evaluation provides limited evidence of relevance to contemporary long-context settings. Experiments focus on older checkpoints and context lengths up to 64k, with the main broad-task evaluation conducted on Qwen2.5-3B-Instruct at 64k. While these models remain useful diagnostic testbeds, this scope does not establish whether WARP addresses the challenges encountered when extending contemporary long-context models beyond their substantially larger native windows. The paper needs either stronger evidence in such settings or a more compelling practical use case for its current evaluation regime.
  3. The demonstrated benefits do not establish a compelling accuracy–efficiency trade-off. On the main 64k RULER evaluation, WARP achieves 70.82%, below NTK at 71.22%, YaRN at 73.31%, and MrRoPE-Pro at 73.44%. Several highlighted exact-retrieval results are already saturated by unmodified RoPE and competing methods. Although WARP improves selected settings, its gains are inconsistent across models and lengths. Meanwhile, the reported attention microbenchmark shows approximately threefold higher prefill latency than RoPE, with additional decoding overhead. These results leave the practical advantage of adopting WARP unclear.

Reference

[1] Training-Free Long-Context Scaling of Large Language Models. https://arxiv.org/abs/2402.17463

Questions for authors
1 comment

Thank you.

Novelty. We agree that preserving local structure while compressing long-range relations predates WaRP (ReRoPE, Leaky ReRoPE, SelfExtend, DCA and later work). Non-separability distinguishes WaRP from per-token position and frequency rescaling, not from these pair-dependent methods. The specific contribution is the endpoint-constrained logarithmic distance map, measured against both families. DCA is now cited and evaluated.

The requested comparisons. Qwen2.5-3B@64k: on fresh preregistered instances, calibrated WaRP met a 5-point non-inferiority margin against Leaky ReRoPE (−1.20 points) and is 1.17 below YaRN; on the matched retrospective panel it is +2.46 above DCA and −3.14 below Leaky ReRoPE, and on 260 of those prompts −3.60 below SelfExtend ([−7.53, +0.24]). Llama-2 at 16k (held out): Leaky ReRoPE beats WaRP by 28–31 points. Development RULER (13B / 7B): ReRoPE 50.4 / 45.9, Leaky ReRoPE 57.8 / 55.3, SelfExtend 43.7 / 48.9, DCA 44.6 / 39.9, calibrated WaRP 33.6 / 27.3. Against the authors' released SelfExtend and DCA code on the same development prompts (86 pairs on Llama-2-7B/13B and Qwen-3B), official RULER scores agree on 84 and the generated text is identical on 52; the appendix lists the divergences, which run in both directions.

Accuracy–efficiency trade-off. The revision narrows the empirical claim. Logarithmic WaRP improves over RoPE and the selected rescaling comparator in the reported settings and stays within a 5-point margin of Leaky ReRoPE on fresh Qwen data. It does not outperform the strongest pair-dependent methods. We now separate geometry from implementation. At Leaky ReRoPE's geometry, H3 met the preregistered accuracy margins on both Llama-2 models, and in separate controlled timings the fused kernel is 2.63× (13B/16k) and 2.30× (Qwen-3B/64k) faster than our pairwise Leaky reference, above the 1.5× threshold recorded before timing; its latency is 1.03× / 1.70× RoPE in the same stack. Calibrated WaRP itself costs 1.02× / 1.61× RoPE end to end, against 1.19× / 1.12× for DCA in its authors' code. This is an implementation result for the tested maps, not a novelty claim for Leaky geometry or an accuracy advantage of logarithmic WaRP. The WaRP–DCA comparison is confounded by the software stacks.

Contemporary relevance. Our evidence covers the evaluated checkpoints and contexts up to 64k. It does not establish performance when extending contemporary models with much longer native windows. We narrowed the paper accordingly; the added Llama experiments clarify operating regimes and do not fill this gap.

Reviewer_ECbzOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes WARP (Warped Rotary Positions), a training-free method for extending the context length of RoPE-based language models. Instead of globally rescaling absolute positions or rotary frequencies, WARP applies a monotone nonlinear warp to the pairwise relative distance d=i−jd = i - j inside rotary attention. The warp is the identity at native length, approximately preserves local distances, and compresses only the far field into the model’s pretrained phase budget via a logarithmic shoulder. The authors show that a genuinely nonlinear function of i−ji - j cannot be reduced to a standard per-token absolute-position cache. The evaluated configuration adds frequency, temperature, and density calibration and implements the warp through a block-anchored approximation. Experiments cover exact retrieval, perplexity, and official RULER across several model families.

Critical strengths

The core idea is conceptually clean and well-motivated. By warping the relative distance supplied to a fixed pretrained spectrum rather than altering the spectrum itself, WARP aims to preserve local geometry while still bringing distant pairs into the trained phase range. Figure 1 effectively contrasts this geometry with PI, NTK, YaRN, and MrRoPE-Pro. The non-separability argument (Section 4) is a useful distinction: a nonlinear f(i−j)f(i - j) cannot satisfy the Cauchy functional equation required for representation by absolute positions g(i)−g(j)g(i) - g(j). This cleanly separates WARP from methods that only modify a position or frequency cache. Evaluation practice is careful. The authors use a locked protocol, report ten-seed exact-retrieval results with zero observed variance on several modern checkpoints, include official RULER v1, and explicitly document failure regimes (especially Llama-2-7B at 32k). On Qwen2.5-3B and Mistral-7B, WARP reaches the 50/50 retrieval ceiling at 32k–64k; on RULER it is competitive (70.82% macro) and strongest on several retrieval and QA components. The paper is refreshingly honest about limitations: it does not claim RULER dominance, reports the Llama-2 cliffs, notes that the evaluated kernel is a block-anchored approximation rather than exact pairwise application, and treats the exploratory head-routing + trained adapter as non-training-free.

Critical weaknesses

Empirical gains are modest and model-dependent. On the main RULER result (Qwen2.5-3B-Instruct at 64k), WARP is essentially tied with NTK and 2.5–2.6 points behind YaRN and MrRoPE-Pro. Its advantage is concentrated in retrieval-heavy tasks; it loses on variable tracking and common-word extraction. On models that already extrapolate reasonably well, the practical benefit over strong frequency-rescaling baselines is limited. The gap between the ideal construction and the evaluated method is significant. The default sets the identity shoulder W=1W = 1 (so it does not preserve an extended local window), adds non-trivial frequency calibration that modifies the spectrum for 1<s<81 < s < 8, and evaluates a block-anchored approximation whose distance error is bounded by block size plus rounding. Consequently, the reported numbers do not isolate the pure relative-distance warp. Sensitivity to these design choices is not analyzed in the main text. Calibration is heavily empirical. The frequency schedule involves multiple hand-tuned constants (0.15 peak, 8× cutoff, 2048-token reference, head-dimension gates, sinusoidal envelope). While the authors note that these should be assessed separately, the absence of such analysis leaves open the question of how much performance depends on model-specific tuning versus the warp itself. Efficiency is a practical concern. Prefill is approximately 3× slower than standard RoPE because pairwise geometry cannot reuse a conventional position cache, and decode latency is also higher. For a training-free inference method, this overhead is non-negligible and is only partially mitigated by the claim that fused kernels could close the gap. Failure modes on older architectures (Llama-2-7B/13B) remain largely unexplained beyond diagnostic experiments. The fact that every training-free baseline also scores 0/50 on Llama-2-7B at 32k suggests a deeper architectural incompatibility that relative-distance warping alone does not resolve.

Questions for authors

How large is the numerical discrepancy between the block-anchored approximation and exact pairwise application of Eq. (1) on the reported tasks? Does exact pairwise warping change any of the headline numbers? Can you provide an ablation that isolates the pure logarithmic warp (rm=1r_m = 1, λ=0\lambda = 0, larger WW) from the frequency/temperature/density calibration on both modern and Llama-2 models? Given the 3× prefill slowdown, what is the end-to-end latency impact on a realistic generation workload, and how much of the gap is recoverable with a custom fused kernel? On RULER, the macro gap is driven by variable tracking and extraction tasks. Is there a systematic reason the nonlinear warp hurts these more than frequency-rescaling methods?

1 comment

Thank you for the precise questions.

Q1 (exact vs block-anchored). On 26 development RULER examples (Llama-2-13B@16k, calibrated), exact pairwise attention changes 1 of 26 scores (29.2% vs 28.8%). Full-model NLL controls show a maximum observed discrepancy of about 0.0033 nats/token in supported configurations. The preregistered H3 test at Leaky ReRoPE's geometry passed on both Llama-2 models (general comment). On Qwen2.5-3B@64k (pure geometry, sampled activations) the relative output error of block 16 is 0.6–1.3%. We have not run the calibrated Qwen headline with full exact attention, so these results do not fully answer whether it would change; we found and report one configuration where block anchoring fails.

Q2 (pure warp; larger shoulder). Held-out Llama-2 at 16k: the pure logarithmic warp (r_m = 1, λ = 0, t = 1) scores 15.1 / 19.4 (13B / 7B) and calibration raises it to 31.0 / 28.4 (+15.9 [14.0, 17.8] and +9.0 [7.4, 10.7]; post hoc contrasts of preregistered arms). Shoulder, with all calibration off (development, five instances per task): widening W from 1 to N/8 changes RULER from 18.9 to 10.6 on Llama-2-13B, 16.0 to 9.2 on Llama-2-7B, and 67.8 to 66.7 on Qwen2.5-3B@64k. With the endpoint fixed, a wider shoulder forces stronger far-tail compression; these scores do not establish the cause. Qwen2.5-3B@64k, matched 650 prompts (retrospective): frequency calibration alone adds +0.33 [−0.56, +1.25] points to pure WaRP (69.47 → 69.80) and the temperature term +1.56 [−0.47, +3.61] more (71.36). Neither interval excludes zero; calibration matters far less on Qwen than on Llama-2. The temperature-only arm did not finish (430/650).

Q3 (latency). We withdrew the first full-model timings (they computed full-prompt logits) and reran with a corrected harness on an idle H100. These timings use random prompts and repeated-token decode forwards. Calibrated WaRP's prefill is 20.8 / 5.0 s versus RoPE's 8.4 / 2.6 s in our stack (Qwen 64k / Llama-2-13B 16k); its 128-token total-time ratios are 1.61 / 1.02. Decode profiles show different implementation paths: at Qwen 64k, RoPE's largest elementwise-copy kernel takes 22.9 ms/step and its attention kernel 6.5 ms/step; at Llama-2-13B 16k, the largest concatenation kernel takes 36.6 ms/step for RoPE and 24.8 for WaRP. These profiles do not isolate an effect of the distance map. FlashAttention-2 RoPE decodes at 47 / 45 ms/token, calibrated WaRP at 27 / 51. The reference/fused ratios are 2.30 / 2.63, meeting the rule fixed before timing (≥ 1.5 in both cells); H3's accuracy validation covers Llama-2 only. The appendix reports all timings and the two excluded, rerun rows.

Q4 (why VT and CWE). We found a consistent error signature, though not a causal explanation. Every official VT and CWE prompt opens with a worked example. On the matched Qwen panel, WaRP's first answer line contains items found only in that example in 56% of VT and 92% of CWE instances, against 16% / 60% for YaRN and 2% / 2% for Leaky ReRoPE. Using the whole generation preserves the ordering (WaRP 58% / 96%, YaRN 18% / 72%, Leaky 2% / 20%). Pure WaRP shows the same pattern (56% / 84%), so calibration does not cause it alone. In VT, WaRP recovers the 2nd and 3rd chain variables 56% of the time vs 82% (YaRN) and 91% (Leaky). VT and CWE account for 1.66 of the 2.49-point macro gap to YaRN, but the deficit remains on instances without copying. The analysis is exploratory and post hoc (appendix).

W3 (calibration constants). The constants are the submission's, identical across models and never retuned. At the evaluated settings they reduce to one value: the head-dimension gate is 1 for 128-dimensional heads; the native-length factor clip(log₂(N/2048), 0, 1) is 1 for every evaluated model (N ≥ 4096); and the envelope is √3/2 at both evaluated ratios (2× and 4×). The largest frequency exponent is therefore 0.130 on Llama-2, Qwen2.5-3B and Mistral-7B. This describes the realized settings; it does not derive the constants. A ±50% peak sweep on Llama-2 development data (exploratory: same 20 RULER instances per task and NLL panel as comparator selection) finds accuracy and NLL improving as the peak grows. RULER macro is 25.2 / 27.3 / 30.3 on 7B and 30.0 / 33.6 / 34.8 on 13B at 0.075 / 0.15 / 0.225. So 0.15 is not the best of the three values on development data. At every peak, calibrated WaRP scores above pure WaRP and every frequency-scaling method there, and below every prior-based method, so its position among the methods does not change. All other calibrated WaRP results retain peak 0.15 (appendix table).

W5 (Llama-2 failures). You were right. Pair-dependent methods succeed at 7B@32k (Leaky ReRoPE and ReRoPE 50/50, DCA 46/50), so we withdraw the capacity explanation; WaRP's Llama-2 failures depend on its configuration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision results: fresh preregistered test, direct comparisons with ReRoPE-family methods and DCA, ablations, latency

We thank the reviewers. The revision adds the comparisons you asked for, separates the logarithmic warp from its calibration and from the fused implementation, and tests the main claim on fresh data. Several outcomes go against the submitted method; we report them as they came out.

1. Main setting, fresh preregistered test (passed). Qwen2.5-3B-Instruct extended from 32k to 64k; 780 newly generated official RULER instances (60 per task, no prompt or QA question shared with any earlier panel). The design, 5-point margin and analysis were frozen before any output existed, and the analysis was run once. Disclosures: one generator seed; QA questions are a contiguous source block (indices 1020–1079); while monitoring progress we twice saw a few per-instance needle scores in run logs (recorded in the preregistration file), with no aggregate computed and nothing changed.

Fresh RULER, Qwen2.5-3B @64kMacro (%)Calibrated WaRP minus comparator
Calibrated WaRP (submitted configuration)73.07
Leaky ReRoPE (primary; model-native query scaling)74.27−1.20 (one-sided 95% bound −3.20; two-sided [−3.62, +1.20]; p = 0.001 for the 5-point margin)
YaRN (secondary)74.24−1.17 (bound −2.83; two-sided [−3.16, +0.80])

This establishes non-inferiority within five points on this setting, not equivalence: both point estimates favour the comparator. The deficit is again in variable tracking and common-word extraction.

2. Short-native Llama-2 at 16k, preregistered held-out comparison (H1). Calibrated WaRP beats the development-selected frequency method (PI) by 23.6–23.8 RULER points, but loses to the development-selected pair-dependent method, Leaky ReRoPE, by 28.3–31.1 (13B / 7B: 31.0 / 28.4 vs 59.3 / 59.5). A preregistered linear-tail variant (H2) failed overall (it passed on 13B, not on 7B). The pinned RULER generator selects QA questions by index, so 40 of the 1,300 held-out instances reuse a development question with new distractors; excluding them changes no contrast by more than 0.43 points and no decision.

3. The fused kernel, preregistered accuracy test (H3, passed). With Leaky ReRoPE's geometry held fixed, the fused block-anchored kernel met its accuracy margins against the reference implementation on both Llama-2 models (−0.61 and −0.31 RULER points; under 0.001 nats/token). H3 reused H1's evaluation set, so it is a preregistered follow-up, not an independent confirmation. One out-of-memory restart was an unchanged, post-freeze exception to its no-rerun rule (only the traceback was inspected before restarting); details are in the appendix. This validates block anchoring at that geometry on those models; the calibrated logarithmic Qwen headline has not been evaluated with full exact attention. Smaller exact-versus-block NLL controls have a maximum observed discrepancy of about 0.0033 nats/token in supported configurations and expose one configuration where block anchoring fails (linear tail without a shoulder), which we report.

4. Latency (exploratory; 128-token setting, warm medians of 5 runs). In our stack, calibrated WaRP/RoPE total-time ratios are 1.61 at Qwen 64k and 1.02 at Llama-2-13B 16k; DCA/RoPE in the DCA authors' stack is 1.12 / 1.19, and SelfExtend/RoPE in the SelfExtend authors' stack is 1.72 / 2.27. WaRP/RoPE prefill ratios are 2.5 / 1.9 in our stack and, across stacks, 5.6–5.7 / 3.0 against FlashAttention-2 RoPE; cross-stack measurements do not isolate method overhead. Reference/fused Leaky ratios are 2.30 / 2.63 (Qwen / Llama), exceeding the 1.5 rule fixed before timing; fused/RoPE ratios are 1.70 / 1.03 in our stack. H3 validated accuracy at that geometry on Llama-2 only.

5. Retrospective comparisons on the original Qwen panel (used during development; descriptive). Full 1,300 instances: WaRP 70.82%, RoPE 62.92, NTK 71.22, YaRN 73.31, MrRoPE-Pro 73.44. Matched 650 instances: Leaky ReRoPE 74.49, WaRP 71.36, DCA 68.89 (WaRP − DCA +2.46 [+0.02, +4.94]; WaRP − Leaky −3.14 [−5.63, −0.67]). First 20 instances per task (260): SelfExtend 76.47 vs WaRP 72.88 (−3.60 [−7.53, +0.24]).

6. Corrections. We withdraw the architectural-capacity explanation (Leaky/ReRoPE 50/50 vs WaRP 0/50 at 7B@32k), no-degradation wording (Qwen perplexity +0.2%/+1.8% at 32k/64k), pre-correction Figure 2 and 31/50 adapter headline (corrected: 17.8/50 vs 12.4/50). We correct Llama perplexity length matching and limit frequency preservation to pure WaRP. Corrected Qwen-0.5B retrieval is 49.4/50, versus historical 50/50. We omit TinyLlama (six checkpoints become five), replace NTK parity by descriptive estimates, and withdraw the claim that global tuning did not recover 7B@32k.

Not finished: ReRoPE on the 260-prompt subset (15/260) and a temperature-only Qwen arm (430/650); we report no partial scores.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.