acceptodds
Under review as a conference paper at ICLR 2027

WavePrune: One period is often enough for RoPE

Abstract

Rotary Position Embedding (RoPE) encodes token positions by rotating each two-dimensional channel of the query and key vectors at a channel-specific frequency, making the attention logits invariant to a common shift of positions. However, this rotation is periodic, and it leads to position aliasing where relative positions separated by a full rotation period become hard to tell apart. To address this, we propose WavePrune, which restricts each channel to its first rotation period. We show that it removes the distractions in attention maps created by position aliasing and improves overall long-context performance. Specifically, WavePrune raises the HELMET score on four of five models we test without any extra tuning (e.g., 35.7 → 40.0 on Qwen3-8B). When pretraining models from scratch, WavePrune also achieves lower validation loss at extrapolated lengths than pretraining without it. Because WavePrune restricts each channel to a sliding window, it induces a fine-grained sparsity that our hardware-aligned CUDA kernels exploit for 1.16× prefill and 1.24× decoding speedups over FlashAttention-2 at 32K context. Together, these results show that RoPE's periodic structure, widely regarded as essential, is largely redundant beyond the first rotation period.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Transformers, Long Context, Efficient Attention, CUDA kernels, Rotary Position Embeddings

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5JFMOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript proposes WavePrune, a training-free modification to Rotary Position Embeddings (RoPE) that aims to improve long-context performance by addressing positional aliasing caused by RoPE's periodic rotations. The authors argue that repeated rotational phases can cause distant tokens to exhibit similar positional alignments, producing spurious attention patterns termed ghost sub-diagonals, which may degrade long-context understanding. To address this, WavePrune truncates the attention computation of each RoPE channel to its first rotation period, eliminating ghost sub-diagonals caused by positional aliasing through a channel-wise sliding window mask, while allowing low-frequency channels to capture long-range dependencies.

Critical strengths

1.) Simple and training-free approach: WavePrune introduces a straightforward modification to RoPE by restricting each channel's attention computation to its first rotation period. The method can be applied to existing pretrained LLMs without additional training or architectural changes, making it practical and easy to adopt.

2.) Efficient implementation: The authors propose FlashWavePrune, an optimized implementation that exploits the channel-wise sliding window structure to reduce computation and KV-cache memory traffic. This provides practical efficiency benefits, including improved decoding speed compared to FlashAttention-2 in the evaluated setting.

Critical weaknesses

1.) Insufficient motivation and justification for pruning beyond the first RoPE period: The motivation remains unclear, as the authors do not convincingly establish why RoPE-induced positional aliasing and ghost sub-diagonals are harmful to downstream performance. Furthermore, truncating each channel's attention contribution after its first rotation period lacks sufficient theoretical justification, and the connection between positional aliasing and actual model failures is not adequately demonstrated.

2.) Limited and inconsistent empirical improvements across pretrained models: The improvements reported in Table 1 on HELMET are modest for most models, with a notable gain only for Qwen3-8B, while Llama3.1-8B experiences a performance degradation. Many improvements are also small relative to the reported cross-task standard deviations, raising concerns about their practical significance. Overall, the results do not convincingly demonstrate consistent improvements across model families.

3.) Limited relevance of the training-from-scratch experiments to modern long-context settings: The training-from-scratch experiments are interesting but limited to small GPT-2 models trained at 2K context length and evaluated primarily at 4K. Given the manuscript's focus on long-context modeling, these experiments do not sufficiently demonstrate whether the observed benefits generalize to modern LLMs operating at substantially longer contexts (32K-128K).

4.) Redundant and insufficiently focused presentation of RoPE and the underlying problem: The presentation could be improved, as RoPE and its periodic properties are discussed redundantly in the Introduction and Preliminaries without clearly motivating the underlying problem. The Introduction should focus on why positional aliasing is harmful to long-context modeling, while the mathematical details of RoPE could be consolidated in the Preliminaries.

5.) Insufficient ablations to isolate the benefits of RoPE-period-based truncation: It remains unclear whether WavePrune's improvements result specifically from eliminating positional aliasing or simply from suppressing long-range attention contributions. Comparisons with alternative channel-wise truncation schemes, such as fixed or random thresholds and smooth decay, would help establish whether restricting each channel to its first rotation period is essential.

Questions for authors

1.) Refer to Weaknesses.

2.) The efficiency comparisons are primarily conducted against FlashAttention-2 on A100 GPUs. How does FlashWavePrune compare against FlashAttention-3 on H100 GPUs? Given the architectural differences and optimizations in FA3, would the reported speedups remain significant?

3.) Using HELMET might not be sufficient, can authors provide some results on RULER-32K and RULER-64K (HARD subset)?

I am open to revisiting my evaluation if the authors thoroughly address the questions raised and provide convincing supporting evidence.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 5JFM

We are grateful to Reviewer 5JFM for highlighting WavePrune's simplicity, practical training-free approach, and efficient kernel implementation. We directly address each concern below.

W1: Why ghost sub-diagonals hurt, and theoretical justification

We first intuitively explain how ghost sub-diagonals harm performance, then empirically show a direct causal link to model failures, and finally point to the existing theoretical justification.

How ghost sub-diagonals harm performance. Ghost sub-diagonals divert attention from the relevant tokens to irrelevant ones at periodic distances, diluting the signal a head should normally attend to. As the context grows, more such sub-diagonals fit in, so the dilution worsens and the tasks that need precise attention (e.g., retrieval and in-context learning) fail.

Experiment: direct causal link from ghost sub-diagonals to model failures. On Qwen3-8B at 32K, we take HELMET ICL questions where standard RoPE gives wrong answers while WavePrune answers correctly. Applying WavePrune to a single head (L14H3) fixes 8 of them. Among the 8, we find that masking only one periodic set of the head's sub-diagonals fixes 7/8:

Intervention on head L14H3Answers fixed
Standard RoPE0/8
WavePrune8/8
Mask attention at the ghost sub-diagonals7/8

Determining and masking the ghost sub-diagonal of L14H3. We find head L14H3 attends to keys at regularly spaced distances from each query, d≈7.1+35.33kd \approx 7.1 + 35.33k (k=1,2,…k = 1, 2, \dots), where 35.33 tokens is the wavelength (one full rotation) of channel 8 and 7.1 is an offset estimated from the attention map. We then mask the head's attention logits to −∞-\infty for keys within 8 tokens of these distances, i.e., ∣d−(7.1+35.33k)∣≤8|d - (7.1 + 35.33k)| \le 8 with k≥1k \ge 1. Removing only this one periodic set of ghost sub-diagonals fixes 7/8 of the errors, which identifies the ghost sub-diagonals as the root cause of these model failures.

Theory. We respectfully note that this is already theoretically justified in the paper (Section 3; proofs in Appendix A): Theorem 1 proves that RoPE attention logits are near-periodic in the query–key distance, Corollary 1 shows that attention is therefore increasingly diluted as the context grows, and Appendix A.3 shows that WavePrune removes this periodicity.

W2: Inconsistency of improvements across models

We acknowledge that the training-free benefit of WavePrune is not uniform across models. To clarify which changes in Table 1 are statistically significant, we measure the uncertainty of WavePrune's improvement directly: we resample each dataset's questions with replacement, and recompute the HELMET score improvement. Repeating this 10,000 times, the middle 90% of the results gives the 90% confidence interval.

ModelHELMET Δ\Delta90% CI
Qwen3-8B+4.32[+3.40, +5.22]
Qwen3-32B+1.21[+0.40, +2.03]
Ministral3-3B+0.64[−0.03, +1.31]
Gemma3-12B+0.46[−0.04, +0.97]
Llama3.1-8B†−3.60[−4.34, −2.87]

†Over 5 of 6 tasks (Cite, 2.2 → 2.2, lacks stored per-question baseline scores; the 6-task Δ\Delta is −3.0).

Llama3.1-8B is a clear outlier. We suspect the cause lies in how it was trained. However, its training recipe is not open-sourced, so we are unable to determine why WavePrune fails on it.

In practice, WavePrune has near-zero cost to try. Being fully training-free, it can simply be evaluated on a validation set of the target model, and kept only if it improves accuracy (or improves speed while preserving accuracy).

W3: Training from scratch at long context

Following the reviewer's advice, we repeat the experiments at longer context length: GPT2-large trained at 64K tokens and evaluated at 64K / 128K:

Task (metric)64K (training length)128K (2× training length)
Tetranacci (value accuracy, %)2.97 → 11.961.99 → 10.30
Multi-key NIAH, 1–16 keys (exact match, %)90.4 → 91.286.6 → 89.6
Multi-key NIAH, 16–256 keys (exact match, %)75.8 → 78.670.2 → 78.4

WavePrune outperforms the baseline on both tasks, most clearly at 128K. Given that this benefit holds consistently across every model size and training length we have tested, we expect it to carry over to larger LLMs and training datasets.

AuthorsOpenReview ↗

Response to Reviewer 5JFM

W4: Presentation of RoPE and the underlying problem

We thank the reviewer for this suggestion and will restructure the presentation accordingly. Currently, RoPE is described in both the Introduction (its rotation mechanism) and Section 2 (its definition). In the revision, each concept will be introduced exactly once:

  • Introduction. We will focus on the motivation and the underlying issue: why positional aliasing harms long-context modeling, instead of RoPE mechanics.
  • Preliminaries. All RoPE material (e.g., channel rotations and the frequency schedule) will be consolidated here.

W5: Comparison with other channel-wise truncation schemes

We compare WavePrune (the window of channel cc equals its wavelength λc\lambda_c) with alternative schedules on Qwen3-8B at 32K. The fixed threshold gives all channels the same window (Sliding Window Attention), set so that its attention computation matches WavePrune’s. The other schedules give each channel a different window; they remain monotone (i.e., slower channels attend farther) and keep the same total window length.

ScheduleWindow of channel ccHELMET Δ\Delta vs. WavePrune (90% CI)
WavePruneλc\lambda_c (its wavelength)—
Fixed threshold9,167 for all channels−42.87 [−44.42, −41.31]
Faster half shorter0.25λc0.25\lambda_c for the faster half; 1.17λc1.17\lambda_c for the slower half−2.02 [−3.17, −0.90]
Faster half longermin⁡(16λc,11.1K)\min(16\lambda_c, 11.1\text{K}) for the faster half; max⁡(0.24λc,11.1K)\max(0.24\lambda_c, 11.1\text{K}) for the slower half−6.05 [−7.27, −4.86]

Conclusion. Restricting each channel to its first rotation period is optimal: every alternative loses 2–43 points. The gain therefore does not come from suppressing long-range attention in general.

Q2: Comparison with FlashAttention-3 on Hopper GPUs

We optimize our FlashWavePrune kernel for the Hopper architecture, following the same principle as on Ampere (Section 3.3). We compare it with FlashAttention-3 (FA3) on an H200 GPU, in the same setting as the paper's Table 4.

Speedup over FA3. The speedups remain significant against FA3: up to 1.17× in prefill and 1.30× in decoding, comparable to those against FA2 on A100 (up to 1.19× and 1.28×, Table 4).

Setting8K16K32K64K128K
Prefill, head dim 641.03×1.07×1.10×1.12×1.12×
Prefill, head dim 1281.03×1.08×1.11×1.15×1.17×
Decode, head dim 641.03×1.09×1.13×1.15×1.14×
Decode, head dim 1281.14×1.22×1.27×1.30×1.29×

We note that FA3 is a strong baseline: on H200, it runs 1.6–1.9× faster than FA2 in prefill, and FA4's Hopper path does not outperform it. This demonstrates that WavePrune's efficiency gains are not limited to Ampere GPUs.

Q3: RULER scores (HARD subset)

We run the official RULER benchmark (13 tasks, 100 questions each) on four of the paper's five models. As the HARD subset, we use the 9 tasks that are not saturated, excluding the 4 easiest tasks (three single-needle tasks and the simplest multi-key task) where every model averages 96–100 at 32K and 64K. Change in the HARD score with WavePrune:

Model32K64K128KAverage
Ministral3-3B−0.26+0.51−0.14+0.04
Qwen3-8B−0.03+0.54+2.84+1.12
Gemma3-12B+0.34−0.01+0.12+0.15
Llama3.1-8B+1.35−2.24−8.79−3.23

Conclusion on RULER. At 32K and 64K, WavePrune does not significantly change any model's RULER score except the outlier Llama3.1-8B. The same holds at 128K for Ministral3-3B and Gemma3-12B, while Qwen3-8B improves (+2.84, 95% CI [+0.85, +4.84]).

Relation to HELMET. We emphasize that HELMET [1] is larger and more realistic than RULER; RULER mainly consists of synthetic tasks, whereas HELMET centers on real-world applications such as many-shot in-context learning and document re-ranking, and is also widely used (e.g., [2, 3, 4]).

Since synthetic tasks such as NIAH often do not reliably predict downstream performance [1], small or no change on RULER does not mean there is no impact on real-world tasks, whose benefits the paper has already demonstrated.

References

[1] H. Yen, T. Gao, M. Hou, K. Ding, D. Fleischer, P. Izsak, M. Wasserblat, D. Chen. HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly. ICLR 2025 (arXiv:2410.02694).

[2] GLM-5 Team. GLM-5: from Vibe Coding to Agentic Engineering. 2026 (arXiv:2602.15763).

[3] M. Abdin, J. Aneja, H. Behl, et al. Phi-4 Technical Report. 2024 (arXiv:2412.08905).

[4] Kimi Team. Kimi Linear: An Expressive, Efficient Attention Architecture. 2025 (arXiv:2510.26692).

Reviewer_5JFMOpenReview ↗

Response R1 to Authors

I thank the authors for their detailed and thorough rebuttal. I appreciate the substantial experimental effort undertaken to address my concerns, particularly the causal intervention study, statistical confidence intervals, longer-context training experiments, alternative truncation ablations, FlashAttention-3 comparisons, and RULER evaluations.

Some of my original concerns have been substantially addressed, and the additional evidence considerably strengthens the manuscript. However, I believe a few concerns remain:

  1. Causal justification: The intervention study provides encouraging evidence that ghost sub-diagonals can contribute to model failures. However, the experiment is conducted on a selected subset of examples, and masking periodic attention positions does not fully disentangle the effects of positional aliasing from those of suppressing potentially distracting attention contributions. Therefore, while the results support the authors' hypothesis, I encourage them to moderate their claim that the experiment conclusively identifies ghost sub-diagonals as the root cause of these failures.

  2. Generalization across models: The newly reported confidence intervals are helpful in establishing the statistical reliability of the observed improvements. However, they also indicate that statistically significant gains are limited to a subset of the evaluated models, while Llama3.1-8B experiences a substantial performance regression. Furthermore, the RULER results demonstrate limited improvements for most models, suggesting that the benefits are model- and task-dependent. I encourage the authors to explicitly acknowledge these limitations and avoid overstating the general applicability of WavePrune.

  3. Optimality of the truncation schedule: I appreciate the additional ablations, which provide stronger evidence that the first-period truncation schedule is more effective than the evaluated alternatives. However, these experiments are limited to a single model and context length and do not establish that the proposed schedule is universally optimal. I encourage the authors to qualify this conclusion accordingly.

Overall, I appreciate the authors' responsiveness and the substantial additional empirical evidence. I will raise my score accordingly.

AuthorsOpenReview ↗

Response to Reviewer 5JFM

We thank the reviewer for the careful reading of our rebuttal and for raising the score. We resolve each remaining concern below.

1. Causal justification

To fully convince the reviewer that position aliasing is the root cause of these model failures, we provide a direct visualization, a clarification of what our intervention experiment implies, and note that the examples are not cherry-picked.

  • Evidence 1 (a direct visualization): We have uploaded a figure to the supplementary material that plots the 2D vector for channel 8 at each query-key distance. It shows position aliasing clearly: keys about one wavelength apart point in similar directions.
  • Evidence 2 (the intervention): The intervention targets a single channel: channel 8 of head L14H3. The masked distances, d ≈ 7.1 + 35.33k, repeat with a period that exactly matches this channel's wavelength (35.33). Masking only this periodic set (and its neighboring positions) makes the model answer correctly. Simply suppressing distracting attention cannot at all explain why the fix follows this channel's rotation period, but position aliasing does.
  • Not cherry-picked: The question samples were not chosen by hand. They are all the HELMET ICL questions that applying WavePrune to head L14H3 alone fixes. Head L14H3 was chosen because applying WavePrune to it alone gives the largest improvement of any head.

2. Generalization across models

We agree that WavePrune's long-context benefits depend on the model and the task; our Limitations section already notes the variation across models. In the revision, we will make this more explicit:

  • Moderate the claims: we will state that the statistically significant HELMET gains hold for Qwen3-8B and Qwen3-32B, and that WavePrune has little effect on the RULER HARD subset. Thus, its benefit is model- and task-dependent.
  • Add supporting analysis: we will add the paired 90% confidence intervals, the RULER results.
  • Reframe the practical use of training-free WavePrune: we will present it as a training-free option to evaluate on the target model, not as a universal improvement.

Crucially, we note that the model- and task-dependence applies only to the training-free setting. Models pretrained with WavePrune extrapolate much better than the baseline in every tested configuration (model scale, training set, and context length). WavePrune also speeds up inference in prefill and decoding. Finally, WavePrune shows how much position aliasing, an overlooked issue in RoPE, can hurt a model. We hope the reviewer will take these contributions into account.

3. Optimality of the truncation schedule

Following the reviewer's advice, we repeat the ablation on a second model and a second context length. The schedules are still those of our earlier response (monotone & compute-matched to WavePrune). Each entry is the change in the HELMET average score when WavePrune's wavelength windows are replaced by that schedule, so a negative value means the alternative is worse.

SettingFixed thresholdFaster half longerFaster half shorter
Qwen3-8B, 32K−42.9−6.1−2.0
Qwen3-32B, 32K−64.1−5.8−0.6
Qwen3-8B + YaRN, 64K−48.0−1.2−4.9

No alternative beats WavePrune in any setting, so the gain cannot be attributed to generic locality.

We clarify that the paper never claims the first-period schedule is universally optimal; the word "optimal" appeared only in our earlier response. We apologize if this phrasing overstated our position. What the paper claims is that one rotation period per channel is often enough.

Reviewer_qbf3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

WavePrune restricts each RoPE channel’s attention to its first rotation period. Its contributions are a training-free pruning rule, theoretical and empirical analysis of repeated attention patterns, and FlashWavePrune, a GPU implementation using grouped channel masks. Validation includes five pretrained LLMs, three pretraining scales, synthetic tasks, and kernel benchmarks. Training-free use improves HELMET on four of five LLMs, with large gains on Qwen3-8B, but degrades Llama3.1-8B. Pretraining GPT-2 models from scratch with WavePrune improves length extrapolation from 2048 to 4096 tokens. The kernel achieves approximately 1.16× prefill and 1.24× decoding speedups over FlashAttention2 at 32K.

Critical strengths
  1. WavePrune is a simple and training-free change that improves models and runs faster. On Qwen3-8B the HELMET gains are large and spread across tasks.
  2. The pretraining results make WavePrune more than a post-hoc patch. The paper runs paired from-scratch runs at three GPT-2 scales. Models pretrained with WavePrune do much worse when it is removed, which shows the model could adapt to the channel-wise mask.
Critical weaknesses
  1. The reliability of the training-free benefit across models remains insufficiently established. The Llama3.1-8B result is a counterexample to the central claim and no remedy fixes it, including continued long-context training with WavePrune.
  2. Both the training-free and pretraining experiments compare only against unmodified RoPE. One established alternative in each setting would show whether WavePrune beats existing methods or only a weak default.
Questions for authors
  1. The reported speedups measure the attention kernel alone. What end-to-end improvements does FlashWavePrune achieve in prefill latency and decoding throughput? This would establish whether the kernel gains translate into meaningful practical benefits.
  2. How does WavePrune do within each model’s native context (8K/16K/32K, with no YaRN) for all five models? If the Qwen3 gains appear mainly in the YaRN regime, that changes how the result should be interpreted.
  3. What exactly do the ± values in Table 1 represent? I interpret them as "uncertainties" but I'm not sure of the exact meaning.
8 comments
AuthorsOpenReview ↗

Response to Reviewer qbf3

We thank Reviewer qbf3's for the positive feedback on the clear presentation, training-free gains, and promising pretraining results. Below, we respond to each concern.

W1: Reliability across models

We acknowledge that the training-free benefit of WavePrune is not uniform across models. We have added experiments on three more models (Qwen3-14B with YaRN, Ministral-3-8B, and GLM-4.7-Flash which has MLA). Except for Llama3.1-8B, WavePrune does not hurt and often improves the performance on the models tested (HELMET average):

ModelBaselineWavePruneΔ\Delta
Qwen3-8B35.640.0+4.32
Qwen3-32B55.156.3+1.21
Ministral3-3B39.139.7+0.64
Gemma3-12B38.338.7+0.46
Qwen3-14B (new)61.562.5+0.98
Ministral-3-8B (new)50.951.2+0.22
GLM-4.7-Flash (new)56.056.5+0.45
Llama3.1-8B38.235.2−3.0

Llama3.1-8B is a clear outlier. We suspect the cause lies in how it was trained. However, its training data and long-context training recipe are not open-sourced, so we are unable to determine why WavePrune fails on it.

In practice, WavePrune has near-zero cost to try. WavePrune is training-free and requires no retraining or tuning. Before applying it to a target model, practitioners can simply evaluate it on a validation set: keep it if it improves accuracy or speed, and leave it off otherwise.

W2: Comparison with established methods

Training-free. We already compare with an established method: every Qwen3 result in Table 1 runs on top of YaRN ×4 following the HF model card. Those gains are already measured against YaRN, not unmodified RoPE.

Pretraining. Following the reviewer's advice, we compare against the two methods that production LLMs rely on for length extrapolation: a larger RoPE base (e.g., Llama-3, Qwen3-2507, MiniMax-M2) and YaRN (e.g., DeepSeek-V3, Kimi-K2).

We pretrain GPT2-medium from scratch with each, with and without WavePrune, and report validation loss at 4K as in the paper:

MethodVal loss @4K
Larger RoPE base (θ=106\theta = 10^6)3.845
+ WavePrune3.020 (−0.83)
YaRN ×43.897
+ WavePrune2.987 (−0.91)

All four runs reach the same final training loss (2.69). Thus, WavePrune brings gains on top of both methods, improving length extrapolation without hurting the fit within the training length.

AuthorsOpenReview ↗

Response to Reviewer qbf3

Q1: End-to-end speedup

We measure the end-to-end inference latency of Qwen3-8B on a single A100 GPU with FlashWavePrune versus FlashAttention-2 (FA2), for both prefill (TTFT) and decoding (TPOT):

ContextFA2 TTFT (s)WavePrune TTFT (s)SpeedupFA2 TPOT (ms)WavePrune TPOT (ms)Speedup
8K0.6400.6311.015×13.42313.3191.008×
16K1.5041.4421.043×14.32613.8691.033×
32K3.8713.5861.080×15.04314.9831.004×
64K11.1709.9651.121×19.38217.0671.136×
128K35.82930.8301.162×23.38521.4611.090×

The kernel gains translate into end-to-end speedups: WavePrune is faster than FA2 at every context length, and the advantage grows as attention takes up more of the runtime at longer contexts, reaching 1.16× in prefill (128K) and 1.14× in decoding (64K).

Q2: Native context without YaRN

Averaged over 8K/16K/32K (no YaRN), WavePrune changes the HELMET average only slightly for the four models it improves at 64K-128K (Qwen3-8B, Qwen3-32B, Ministral3-3B, Gemma3-12B):

ModelBaselineWavePruneΔ\Delta
Qwen3-8B52.553.4+0.88
Qwen3-32B65.865.6−0.19
Ministral3-3B47.647.7+0.17
Gemma3-12B56.455.8−0.55
Llama3.1-8B44.843.9−0.93

Interpretation of all training-free results: WavePrune has little effect on relatively short contexts, and its gain become visible only on sufficiently long contexts.

We note that it is neither dependent on YaRN nor limited to lengths beyond the native window:

  • Without YaRN. Some evaluated models do not use YaRN at all, e.g., Gemma3-12B and GLM-4.7-Flash, which WavePrune still improves (W1). At 64K, we also find WavePrune improves Qwen3-8B with YaRN disabled (32.6 → 34.0, +1.42). Hence, WavePrune does not rely on YaRN.
  • Within the native window. 128K is still within the native window of most models we evaluated (e.g., Gemma3-12B: 128K, GLM-4.7-Flash: 200K, Ministral3-3B: 256K), and WavePrune improves them at 64K–128K (Table 1). This means WavePrune can improve performance within the native window, not only beyond it.

Q3: Meaning of ±\pm

The ±\pm is the statistical uncertainty of each score, indicating how much the entry would change if the test questions were resampled. Each entry in Table 1 averages a task's questions at 64K and 128K, so its ±\pm is s/ns/\sqrt{n}, where nn is the total number of questions (from 200 for Re-rank to 1,000 for ICL) and ss is the standard deviation of the scores.

Thank you for the detailed responses and additional experiments. For Q1, how long is the decode length?

Reviewer_PLihOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes WavePrune, which truncates each RoPE channel's contribution to attention logits beyond its first rotation period. FlashWavePrune approximates these channel-dependent windows using hardware-aligned groups. The authors report improved average scores on a modified six-category HELMET evaluation for four of five pretrained models, improved length extrapolation in GPT-2-scale pretraining, and attention-kernel speedups on A100. The paper combines attention visualizations, theoretical analysis, controlled tasks, and kernel ablations, and documents a substantial performance decrease on Llama3.1-8B.

Critical strengths

S1. A simple intervention supported by complementary experiments. The channel-dependent mask differs from a conventional token-level sliding window. The evaluation covers multiple pretrained model families, 64K and 128K contexts, and training/inference combinations at GPT-2 scale. The reported Qwen3-8B improvement from 35.7 to 40.0 is substantial enough to motivate further study.

S2. Concrete implementation and informative negative evidence. FlashWavePrune provides a practical grouped approximation and detailed kernel ablations. The Llama3.1-8B failure analysis is particularly valuable: it shows that high-frequency channels can retain useful long-distance retrieval information, helping delimit when the intervention may be appropriate.

Critical weaknesses

W1. The first-period explanation is not sufficiently isolated from alternative explanations. The threshold and channel-count sweeps are informative, but do not compare the proposed schedule against similarly sparse alternative distance/frequency schedules. The visualizations and synthetic tasks are also consistent with a beneficial locality or frequency-selection bias. The recurrence analysis uses idealized query/key assumptions and does not establish the magnitude of the effect in real models at the evaluated lengths. The evidence therefore supports the intervention more strongly than the general claim that rotations beyond the first period are redundant.

W2. Reliability and applicability need tighter characterization. Two improved models gain less than one average point, while Llama3.1-8B loses approximately three points. The reported cross-task standard deviations do not establish uncertainty in paired improvements. In addition, the evaluation modifies the HELMET protocol, and the speed measurements concern particular attention-kernel configurations rather than end-to-end model performance. These limitations are partly disclosed, but should constrain the breadth of the practical claims.

Questions for authors

Q1. At a matched amount of retained computation, how does the wavelength-based schedule compare with an alternative monotone channel-window schedule? Which result most directly distinguishes the first-period explanation from a general frequency-dependent sparsity benefit?

Q2. Can the authors report paired uncertainty for the average HELMET improvements, especially for Ministral3-3B and Gemma3-12B, and clarify how sensitive the conclusions are to the modified evaluation protocol?

Q3. What measurable property distinguishes the Llama failure from the successful models? Does the long-distance content signal in the removed channels provide evidence that could guide when truncation should be applied?

2 comments
AuthorsOpenReview ↗

Response to Reviewer PLih

We thank Reviewer PLih for recognizing WavePrune's simple and effective design, practical kernel, informative failure analysis, and for the constructive questions. We address each concern below.

W1 & Q1: First-period rule vs. other monotone schedules

To separate the first-period pruning from a generic locality or frequency-selection bias, we give each channel a different window than WavePrune's wavelength windows. The new windows remain monotone (i.e., slower channels attend farther) and have the same total window length as WavePrune’s.

At 32K on Qwen3-8B, they stay within 1% of the unpruned model's language-modeling loss. However, their HELMET evaluation results differ substantially:

Monotone scheduleWindow of channel ccHELMET Δ\Delta vs. WavePrune (90% CI)
WavePruneλc\lambda_c (its wavelength)—
Faster half shorter0.25λc0.25\lambda_c for the faster half; 1.17λc1.17\lambda_c for the slower half−2.02 [−3.17, −0.90]
Faster half longermin⁡(16λc,11.1K)\min(16\lambda_c, 11.1\text{K}) for the faster half; max⁡(0.24λc,11.1K)\max(0.24\lambda_c, 11.1\text{K}) for the slower half−6.05 [−7.27, −4.86]

All three are worse than WavePrune, whether the faster channels get more or less than one period, so a generic locality or frequency-selection bias cannot explain the benefit of WavePrune.

W2 & Q2: Paired uncertainty, HELMET changes, and end-to-end speed

Paired uncertainty. We measure the uncertainty of the improvement directly: we resample each dataset's questions with replacement, and recompute the HELMET improvement as in the paper. Repeating this 10,000 times, the middle 90% of the results gives the 90% confidence interval.

ModelHELMET Δ\Delta90% CI
Qwen3-8B+4.32[+3.40, +5.22]
Qwen3-32B+1.21[+0.40, +2.03]
Ministral3-3B+0.64[−0.03, +1.31]
Gemma3-12B+0.46[−0.04, +0.97]
Llama3.1-8B†−3.60[−4.34, −2.87]

†Over 5 of 6 tasks (Cite, 2.2 → 2.2, lacks stored per-question baseline scores; the 6-task Δ\Delta is −3.0).

The HELMET modifications are legitimate reliability fixes. They are carefully documented in Appendix C and applied identically to the baseline and WavePrune: we raise output limits that cut off answers before completion (RULER MV, JSON KV), and replace the costly and unreliable GPT-4o judge with ROUGE (NarrativeQA), dropping the one task that cannot be scored without it (Summarization). These are corrective adjustments to the benchmark, not a change that could favor either method.

End-to-end speed. We measure the end-to-end inference latency of Qwen3-8B on a single A100 GPU with FlashWavePrune versus FlashAttention-2 (FA2), for both prefill (TTFT) and decoding (TPOT):

ContextFA2 TTFT (s)WavePrune TTFT (s)SpeedupFA2 TPOT (ms)WavePrune TPOT (ms)Speedup
8K0.6400.6311.015×13.42313.3191.008×
16K1.5041.4421.043×14.32613.8691.033×
32K3.8713.5861.080×15.04314.9831.004×
64K11.1709.9651.121×19.38217.0671.136×
128K35.82930.8301.162×23.38521.4611.090×

The kernel gains translate into end-to-end speedups: WavePrune is faster than FA2 at every context length, and the advantage grows as attention takes up more of the runtime at longer contexts, reaching 1.16× in prefill (128K) and 1.14× in decoding (64K).

Q3: What distinguishes the Llama failure, and can it guide truncation?

We thank the reviewer for this insightful question. Following it, we identified the property behind the Llama failure and a simple way to fix its multi-key NIAH retrieval score.

Measurable property. For each RoPE channel, we measure how much attention to the answer WavePrune removes, on 40 synthetic retrieval prompts (no benchmark data). In Llama3.1-8B, this removed signal is large and concentrated in a single channel (channel 22, 97% of the total removed). It is small in other models we evaluated.

Guidance for truncation. We find exempting only channel 22 from truncation restores Llama3.1-8B's performance on a held-out RULER task (niah_multikey_3, 100 questions):

Llama3.1-8B64K128K
No pruning10073
WavePrune (exact)7929
WavePrune (exact), channel 22 exempted10071

At 64K, it resolves all 21 of WavePrune's failures, whereas exempting a neighboring or random channel instead fixes only 3–4. This suggests a simple rule: exempt the channels whose removed signal is large. We will further investigate this rule in the revised version.

I appreciate the authors' response to my concerns. I believe the current score accurately reflects the paper's contribution and its current state. I will maintain my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision Summary

We sincerely thank all the reviewers for their valuable feedback and constructive suggestions. We have revised the manuscript as follows (all changes are marked in blue):

  • Throughout the manuscript (training-free results): We moderated the claims about training-free WavePrune in the abstract, introduction, Sections 3.2 and 4.1, and the conclusion. We now state that the HELMET gains are statistically significant for Qwen3-8B and Qwen3-32B, are smaller on RULER and at 8K–32K context, and are thus model- and task-dependent. The limitations section presents training-free WavePrune as an option to evaluate on the target model. (5JFM; qbf3; PLih)
  • Appendix D: We added paired 90% confidence intervals, RULER hard-subset results, and results within the native context window, and corrected the definition of "±" in Table 1. (5JFM; qbf3; PLih)
  • Appendix B: We added a visualization of position aliasing, the attention head intervention, and extended the schedule ablation to a second model and context length. (5JFM; PLih)
  • Introduction & Section 2: RoPE mechanics now appear once, in the Preliminaries, while the introduction focuses on the core motivation: why position aliasing hurts long-context modeling. (5JFM)
  • Appendix C.3: We added a comparison with FlashAttention-3 on H200 and end-to-end latency results, including long generation and concurrent serving (1.09–1.24× end-to-end speedups). (5JFM; qbf3; PLih)
  • Appendices F.3 & G.1: Pretraining gains hold for GPT2-large trained at 64K, and WavePrune is complementary to a larger RoPE base and YaRN. (5JFM; qbf3)

We believe these revisions address the reviewers' concerns and clarify the scope, robustness, and practical benefits of WavePrune.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.