- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces WaRP, a training-free long-context extension method that applies a monotone nonlinear transformation to pairwise query–key distances in Rotary Position Embeddings (RoPE), rather than relying exclusively on position or frequency rescaling. The proposed framework theoretically distinguishes nonlinear relative-distance warping from conventional per-token positional transformations, while its practical implementation incorporates frequency calibration, attention-temperature and density corrections, and block-anchored approximation. Experiments across six language models, including exact retrieval, perplexity, and RULER evaluations, demonstrate competitive performance in selected settings but also reveal inconsistent generalization across architectures and substantial inference overhead.
Critical strengths
- Interesting formulation of relative-distance warping.
The paper presents a mathematically well-motivated alternative to conventional RoPE scaling by applying a nonlinear transformation directly to pairwise relative distances, with a clear argument establishing why such transformations cannot generally be represented through standard per-token positional caches. This provides an interesting design direction for training-free context extension and highlights a potentially useful distinction between relative-distance transformations and conventional position or frequency rescaling.
- Comprehensive evaluation and transparent discussion of limitations.
The paper evaluates WaRP across multiple model architectures and context lengths using complementary metrics, including exact retrieval, perplexity, and RULER, while additionally examining seed variance and computational efficiency. The explicit reporting of unsuccessful configurations, particularly the Llama-2 retrieval failures, strengthens the empirical assessment and provides useful insights into the architecture-dependent behavior of context-extension techniques.
Critical weaknesses
- Limited empirical advantage over existing methods and insufficient comparison with the closest related approaches.
Despite its interesting formulation, WaRP achieves 70.82% accuracy on the primary 64k RULER benchmark, below YaRN (73.31%) and MrRoPE-Pro (73.44%), while several successful exact-retrieval configurations already reach the performance ceiling with existing methods. Furthermore, the absence of direct comparisons against more closely related pairwise relative-position approaches, particularly ReRoPE/Leaky ReRoPE and SelfExtend, makes it difficult to establish whether the proposed nonlinear warping formulation offers a meaningful advantage over existing alternatives.
- Insufficient isolation of the proposed mechanism and questionable practical trade-offs.
The evaluated implementation combines nonlinear distance warping with frequency calibration, attention-temperature adjustments, density corrections, and block-anchored approximation, but lacks systematic ablations establishing which components account for the reported improvements; moreover, the main cross-method retrieval comparison in Figure 2 predates implementation corrections and is based on a single seed. These limitations, together with the approximately threefold increase in attention-prefill latency, make it difficult to determine whether relative-distance warping itself provides a sufficiently robust and practically meaningful improvement.
Questions for authors
- Component Contributions and Approximation Effects
Could the authors provide systematic ablation experiments isolating the contributions of nonlinear distance warping, frequency calibration, temperature scaling, and density correction, ideally including a comparison between exact pairwise warping and the block-anchored approximation? These results are important for determining whether the proposed relative-distance transformation is the primary source of improvement or whether the gains mainly arise from auxiliary calibration mechanisms.
- Comparisons with Closely Related Methods
How does WaRP compare with ReRoPE/Leaky ReRoPE and SelfExtend under identical evaluation protocols, particularly on RULER and the more challenging Llama-2 configurations? Since these methods also distinguish local and distant relative positions, direct comparisons would substantially strengthen the argument for the novelty and practical significance of the proposed approach.
2 comments
Thank you for the detailed rebuttal and additional experiments. My main concern is still the limited advantage over existing methods. While WaRP achieves comparable performance to Leaky ReRoPE on Qwen2.5-3B, it performs substantially worse on Llama-2. The authors also show that their fused kernel can be applied to Leaky ReRoPE, achieving similar inference efficiency with better performance on Llama-2. Given these results, I am still not convinced what the proposed logarithmic warping offers over existing methods in practice. Therefore, I will maintain my rating of 2.
Thank you.
Q2 (ReRoPE / Leaky ReRoPE / SelfExtend under identical protocols).
Q1 (ablations).
Figure 2. We removed the single-seed, pre-correction Figure 2. The revision reports corrected five-seed WaRP retrieval separately from the historical baseline checks, and adds a length-matched five-seed comparison of all methods on Llama-2-7B. We do not present the historical checks as a corrected multi-seed comparison.
Latency. End to end in the 128-token setting, calibrated WaRP takes 1.61× (Qwen 64k) and 1.02× (Llama-2-13B 16k) the time of RoPE in the same stack. Its prefill is 5.6–5.7× / 3.0× that of FlashAttention-2 RoPE across stacks; these measurements do not isolate method overhead. Full numbers are in the appendix.