- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The manuscript proposes WavePrune, a training-free modification to Rotary Position Embeddings (RoPE) that aims to improve long-context performance by addressing positional aliasing caused by RoPE's periodic rotations. The authors argue that repeated rotational phases can cause distant tokens to exhibit similar positional alignments, producing spurious attention patterns termed ghost sub-diagonals, which may degrade long-context understanding. To address this, WavePrune truncates the attention computation of each RoPE channel to its first rotation period, eliminating ghost sub-diagonals caused by positional aliasing through a channel-wise sliding window mask, while allowing low-frequency channels to capture long-range dependencies.
Critical strengths
1.) Simple and training-free approach: WavePrune introduces a straightforward modification to RoPE by restricting each channel's attention computation to its first rotation period. The method can be applied to existing pretrained LLMs without additional training or architectural changes, making it practical and easy to adopt.
2.) Efficient implementation: The authors propose FlashWavePrune, an optimized implementation that exploits the channel-wise sliding window structure to reduce computation and KV-cache memory traffic. This provides practical efficiency benefits, including improved decoding speed compared to FlashAttention-2 in the evaluated setting.
Critical weaknesses
1.) Insufficient motivation and justification for pruning beyond the first RoPE period: The motivation remains unclear, as the authors do not convincingly establish why RoPE-induced positional aliasing and ghost sub-diagonals are harmful to downstream performance. Furthermore, truncating each channel's attention contribution after its first rotation period lacks sufficient theoretical justification, and the connection between positional aliasing and actual model failures is not adequately demonstrated.
2.) Limited and inconsistent empirical improvements across pretrained models: The improvements reported in Table 1 on HELMET are modest for most models, with a notable gain only for Qwen3-8B, while Llama3.1-8B experiences a performance degradation. Many improvements are also small relative to the reported cross-task standard deviations, raising concerns about their practical significance. Overall, the results do not convincingly demonstrate consistent improvements across model families.
3.) Limited relevance of the training-from-scratch experiments to modern long-context settings: The training-from-scratch experiments are interesting but limited to small GPT-2 models trained at 2K context length and evaluated primarily at 4K. Given the manuscript's focus on long-context modeling, these experiments do not sufficiently demonstrate whether the observed benefits generalize to modern LLMs operating at substantially longer contexts (32K-128K).
4.) Redundant and insufficiently focused presentation of RoPE and the underlying problem: The presentation could be improved, as RoPE and its periodic properties are discussed redundantly in the Introduction and Preliminaries without clearly motivating the underlying problem. The Introduction should focus on why positional aliasing is harmful to long-context modeling, while the mathematical details of RoPE could be consolidated in the Preliminaries.
5.) Insufficient ablations to isolate the benefits of RoPE-period-based truncation: It remains unclear whether WavePrune's improvements result specifically from eliminating positional aliasing or simply from suppressing long-range attention contributions. Comparisons with alternative channel-wise truncation schemes, such as fixed or random thresholds and smooth decay, would help establish whether restricting each channel to its first rotation period is essential.
Questions for authors
1.) Refer to Weaknesses.
2.) The efficiency comparisons are primarily conducted against FlashAttention-2 on A100 GPUs. How does FlashWavePrune compare against FlashAttention-3 on H100 GPUs? Given the architectural differences and optimizations in FA3, would the reported speedups remain significant?
3.) Using HELMET might not be sufficient, can authors provide some results on RULER-32K and RULER-64K (HARD subset)?
I am open to revisiting my evaluation if the authors thoroughly address the questions raised and provide convincing supporting evidence.
4 comments
Response to Reviewer 5JFM
W4: Presentation of RoPE and the underlying problem
We thank the reviewer for this suggestion and will restructure the presentation accordingly. Currently, RoPE is described in both the Introduction (its rotation mechanism) and Section 2 (its definition). In the revision, each concept will be introduced exactly once:
- Introduction. We will focus on the motivation and the underlying issue: why positional aliasing harms long-context modeling, instead of RoPE mechanics.
- Preliminaries. All RoPE material (e.g., channel rotations and the frequency schedule) will be consolidated here.
W5: Comparison with other channel-wise truncation schemes
We compare WavePrune (the window of channel equals its wavelength ) with alternative schedules on Qwen3-8B at 32K. The fixed threshold gives all channels the same window (Sliding Window Attention), set so that its attention computation matches WavePrune’s. The other schedules give each channel a different window; they remain monotone (i.e., slower channels attend farther) and keep the same total window length.
| Schedule | Window of channel | HELMET vs. WavePrune (90% CI) |
|---|---|---|
| WavePrune | (its wavelength) | — |
| Fixed threshold | 9,167 for all channels | −42.87 [−44.42, −41.31] |
| Faster half shorter | for the faster half; for the slower half | −2.02 [−3.17, −0.90] |
| Faster half longer | for the faster half; for the slower half | −6.05 [−7.27, −4.86] |
Conclusion. Restricting each channel to its first rotation period is optimal: every alternative loses 2–43 points. The gain therefore does not come from suppressing long-range attention in general.
Q2: Comparison with FlashAttention-3 on Hopper GPUs
We optimize our FlashWavePrune kernel for the Hopper architecture, following the same principle as on Ampere (Section 3.3). We compare it with FlashAttention-3 (FA3) on an H200 GPU, in the same setting as the paper's Table 4.
Speedup over FA3. The speedups remain significant against FA3: up to 1.17× in prefill and 1.30× in decoding, comparable to those against FA2 on A100 (up to 1.19× and 1.28×, Table 4).
| Setting | 8K | 16K | 32K | 64K | 128K |
|---|---|---|---|---|---|
| Prefill, head dim 64 | 1.03× | 1.07× | 1.10× | 1.12× | 1.12× |
| Prefill, head dim 128 | 1.03× | 1.08× | 1.11× | 1.15× | 1.17× |
| Decode, head dim 64 | 1.03× | 1.09× | 1.13× | 1.15× | 1.14× |
| Decode, head dim 128 | 1.14× | 1.22× | 1.27× | 1.30× | 1.29× |
We note that FA3 is a strong baseline: on H200, it runs 1.6–1.9× faster than FA2 in prefill, and FA4's Hopper path does not outperform it. This demonstrates that WavePrune's efficiency gains are not limited to Ampere GPUs.
Q3: RULER scores (HARD subset)
We run the official RULER benchmark (13 tasks, 100 questions each) on four of the paper's five models. As the HARD subset, we use the 9 tasks that are not saturated, excluding the 4 easiest tasks (three single-needle tasks and the simplest multi-key task) where every model averages 96–100 at 32K and 64K. Change in the HARD score with WavePrune:
| Model | 32K | 64K | 128K | Average |
|---|---|---|---|---|
| Ministral3-3B | −0.26 | +0.51 | −0.14 | +0.04 |
| Qwen3-8B | −0.03 | +0.54 | +2.84 | +1.12 |
| Gemma3-12B | +0.34 | −0.01 | +0.12 | +0.15 |
| Llama3.1-8B | +1.35 | −2.24 | −8.79 | −3.23 |
Conclusion on RULER. At 32K and 64K, WavePrune does not significantly change any model's RULER score except the outlier Llama3.1-8B. The same holds at 128K for Ministral3-3B and Gemma3-12B, while Qwen3-8B improves (+2.84, 95% CI [+0.85, +4.84]).
Relation to HELMET. We emphasize that HELMET [1] is larger and more realistic than RULER; RULER mainly consists of synthetic tasks, whereas HELMET centers on real-world applications such as many-shot in-context learning and document re-ranking, and is also widely used (e.g., [2, 3, 4]).
Since synthetic tasks such as NIAH often do not reliably predict downstream performance [1], small or no change on RULER does not mean there is no impact on real-world tasks, whose benefits the paper has already demonstrated.
References
[1] H. Yen, T. Gao, M. Hou, K. Ding, D. Fleischer, P. Izsak, M. Wasserblat, D. Chen. HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly. ICLR 2025 (arXiv:2410.02694).
[2] GLM-5 Team. GLM-5: from Vibe Coding to Agentic Engineering. 2026 (arXiv:2602.15763).
[3] M. Abdin, J. Aneja, H. Behl, et al. Phi-4 Technical Report. 2024 (arXiv:2412.08905).
[4] Kimi Team. Kimi Linear: An Expressive, Efficient Attention Architecture. 2025 (arXiv:2510.26692).
Response R1 to Authors
I thank the authors for their detailed and thorough rebuttal. I appreciate the substantial experimental effort undertaken to address my concerns, particularly the causal intervention study, statistical confidence intervals, longer-context training experiments, alternative truncation ablations, FlashAttention-3 comparisons, and RULER evaluations.
Some of my original concerns have been substantially addressed, and the additional evidence considerably strengthens the manuscript. However, I believe a few concerns remain:
-
Causal justification: The intervention study provides encouraging evidence that ghost sub-diagonals can contribute to model failures. However, the experiment is conducted on a selected subset of examples, and masking periodic attention positions does not fully disentangle the effects of positional aliasing from those of suppressing potentially distracting attention contributions. Therefore, while the results support the authors' hypothesis, I encourage them to moderate their claim that the experiment conclusively identifies ghost sub-diagonals as the root cause of these failures.
-
Generalization across models: The newly reported confidence intervals are helpful in establishing the statistical reliability of the observed improvements. However, they also indicate that statistically significant gains are limited to a subset of the evaluated models, while Llama3.1-8B experiences a substantial performance regression. Furthermore, the RULER results demonstrate limited improvements for most models, suggesting that the benefits are model- and task-dependent. I encourage the authors to explicitly acknowledge these limitations and avoid overstating the general applicability of WavePrune.
-
Optimality of the truncation schedule: I appreciate the additional ablations, which provide stronger evidence that the first-period truncation schedule is more effective than the evaluated alternatives. However, these experiments are limited to a single model and context length and do not establish that the proposed schedule is universally optimal. I encourage the authors to qualify this conclusion accordingly.
Overall, I appreciate the authors' responsiveness and the substantial additional empirical evidence. I will raise my score accordingly.
Response to Reviewer 5JFM
We thank the reviewer for the careful reading of our rebuttal and for raising the score. We resolve each remaining concern below.
1. Causal justification
To fully convince the reviewer that position aliasing is the root cause of these model failures, we provide a direct visualization, a clarification of what our intervention experiment implies, and note that the examples are not cherry-picked.
- Evidence 1 (a direct visualization): We have uploaded a figure to the supplementary material that plots the 2D vector for channel 8 at each query-key distance. It shows position aliasing clearly: keys about one wavelength apart point in similar directions.
- Evidence 2 (the intervention): The intervention targets a single channel: channel 8 of head L14H3. The masked distances, d ≈ 7.1 + 35.33k, repeat with a period that exactly matches this channel's wavelength (35.33). Masking only this periodic set (and its neighboring positions) makes the model answer correctly. Simply suppressing distracting attention cannot at all explain why the fix follows this channel's rotation period, but position aliasing does.
- Not cherry-picked: The question samples were not chosen by hand. They are all the HELMET ICL questions that applying WavePrune to head L14H3 alone fixes. Head L14H3 was chosen because applying WavePrune to it alone gives the largest improvement of any head.
2. Generalization across models
We agree that WavePrune's long-context benefits depend on the model and the task; our Limitations section already notes the variation across models. In the revision, we will make this more explicit:
- Moderate the claims: we will state that the statistically significant HELMET gains hold for Qwen3-8B and Qwen3-32B, and that WavePrune has little effect on the RULER HARD subset. Thus, its benefit is model- and task-dependent.
- Add supporting analysis: we will add the paired 90% confidence intervals, the RULER results.
- Reframe the practical use of training-free WavePrune: we will present it as a training-free option to evaluate on the target model, not as a universal improvement.
Crucially, we note that the model- and task-dependence applies only to the training-free setting. Models pretrained with WavePrune extrapolate much better than the baseline in every tested configuration (model scale, training set, and context length). WavePrune also speeds up inference in prefill and decoding. Finally, WavePrune shows how much position aliasing, an overlooked issue in RoPE, can hurt a model. We hope the reviewer will take these contributions into account.
3. Optimality of the truncation schedule
Following the reviewer's advice, we repeat the ablation on a second model and a second context length. The schedules are still those of our earlier response (monotone & compute-matched to WavePrune). Each entry is the change in the HELMET average score when WavePrune's wavelength windows are replaced by that schedule, so a negative value means the alternative is worse.
| Setting | Fixed threshold | Faster half longer | Faster half shorter |
|---|---|---|---|
| Qwen3-8B, 32K | −42.9 | −6.1 | −2.0 |
| Qwen3-32B, 32K | −64.1 | −5.8 | −0.6 |
| Qwen3-8B + YaRN, 64K | −48.0 | −1.2 | −4.9 |
No alternative beats WavePrune in any setting, so the gain cannot be attributed to generic locality.
We clarify that the paper never claims the first-period schedule is universally optimal; the word "optimal" appeared only in our earlier response. We apologize if this phrasing overstated our position. What the paper claims is that one rotation period per channel is often enough.
Response to Reviewer 5JFM
We are grateful to Reviewer 5JFM for highlighting WavePrune's simplicity, practical training-free approach, and efficient kernel implementation. We directly address each concern below.
W1: Why ghost sub-diagonals hurt, and theoretical justification
We first intuitively explain how ghost sub-diagonals harm performance, then empirically show a direct causal link to model failures, and finally point to the existing theoretical justification.
How ghost sub-diagonals harm performance. Ghost sub-diagonals divert attention from the relevant tokens to irrelevant ones at periodic distances, diluting the signal a head should normally attend to. As the context grows, more such sub-diagonals fit in, so the dilution worsens and the tasks that need precise attention (e.g., retrieval and in-context learning) fail.
Experiment: direct causal link from ghost sub-diagonals to model failures. On Qwen3-8B at 32K, we take HELMET ICL questions where standard RoPE gives wrong answers while WavePrune answers correctly. Applying WavePrune to a single head (L14H3) fixes 8 of them. Among the 8, we find that masking only one periodic set of the head's sub-diagonals fixes 7/8:
Determining and masking the ghost sub-diagonal of L14H3. We find head L14H3 attends to keys at regularly spaced distances from each query, d≈7.1+35.33k (k=1,2,…), where 35.33 tokens is the wavelength (one full rotation) of channel 8 and 7.1 is an offset estimated from the attention map. We then mask the head's attention logits to −∞ for keys within 8 tokens of these distances, i.e., ∣d−(7.1+35.33k)∣≤8 with k≥1. Removing only this one periodic set of ghost sub-diagonals fixes 7/8 of the errors, which identifies the ghost sub-diagonals as the root cause of these model failures.
Theory. We respectfully note that this is already theoretically justified in the paper (Section 3; proofs in Appendix A): Theorem 1 proves that RoPE attention logits are near-periodic in the query–key distance, Corollary 1 shows that attention is therefore increasingly diluted as the context grows, and Appendix A.3 shows that WavePrune removes this periodicity.
W2: Inconsistency of improvements across models
We acknowledge that the training-free benefit of WavePrune is not uniform across models. To clarify which changes in Table 1 are statistically significant, we measure the uncertainty of WavePrune's improvement directly: we resample each dataset's questions with replacement, and recompute the HELMET score improvement. Repeating this 10,000 times, the middle 90% of the results gives the 90% confidence interval.
†Over 5 of 6 tasks (Cite, 2.2 → 2.2, lacks stored per-question baseline scores; the 6-task Δ is −3.0).
Llama3.1-8B is a clear outlier. We suspect the cause lies in how it was trained. However, its training recipe is not open-sourced, so we are unable to determine why WavePrune fails on it.
In practice, WavePrune has near-zero cost to try. Being fully training-free, it can simply be evaluated on a validation set of the target model, and kept only if it improves accuracy (or improves speed while preserving accuracy).
W3: Training from scratch at long context
Following the reviewer's advice, we repeat the experiments at longer context length: GPT2-large trained at 64K tokens and evaluated at 64K / 128K:
WavePrune outperforms the baseline on both tasks, most clearly at 128K. Given that this benefit holds consistently across every model size and training length we have tested, we expect it to carry over to larger LLMs and training datasets.