- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper proposes RoPE-Filter, a training-free method for selecting long-context training data based on RoPE frequency characteristics. The authors observe that low-frequency RoPE components contribute substantially to long-range attention and propose selectively attenuating these components for distant token interactions. Documents are then ranked according to the relative increase in perplexity caused by this intervention, with the aim of identifying samples containing stronger long-range dependencies. Experimental results show improvements over existing long-context data selection methods.
Critical strengths
-
Simple and interpretable methodology. The proposed method is straightforward and requires no additional model training. By leveraging the characteristics of different RoPE frequency components, RoPE-Filter provides an interpretable mechanism for identifying long-range dependencies and selecting long-context training data.
-
Comprehensive experiments and promising results. The authors evaluate RoPE-Filter across multiple long-context benchmarks and compare it with several existing data selection methods. The results demonstrate consistent improvements across different data domains and models, highlighting the practical effectiveness of the proposed approach.
Critical weaknesses
-
Limited novelty and insufficient justification of the proposed design. The roles of different RoPE frequency components have already been investigated in previous studies. The authors should better clarify the novelty of their approach. More importantly, I suggest adding a direct ablation that uniformly attenuates all long-range attention logits without distinguishing RoPE frequency bands, while keeping the same perplexity-based selection criterion. This would help determine whether the improvements come from the proposed low-frequency intervention or simply from weakening long-range attention.
-
Limited generalizability of Figure 1. Figure 1 only presents head-level frequency contribution results for Llama-3.1-8B. Although Figure 2 includes additional models, it remains unclear whether the head-level observations in Figure 1 generalize across different LLMs. The authors should provide similar analyses for additional models.
-
Insufficient case studies. The paper aims to identify documents with stronger long-range dependencies, but it lacks concrete examples demonstrating that the selected samples indeed exhibit such characteristics. I suggest presenting representative high- and low-scoring documents and highlighting their long-range dependencies to provide more direct and interpretable evidence for the effectiveness of RoPE-Filter.
Questions for authors
Please see the weaknesses listed above.
5 comments
(2/4) Response to Reviewer eKyh
Weakness 2: Adding a direct ablation that uniformly attenuates all long-range attention logits without distinguishing RoPE frequency bands, while keeping the same perplexity-based selection criterion.
Response: Thank you for the suggestion. We have added this all-band ablation to the revised paper and evaluate it from two perspectives.
1. All-band intervention: NIAH heatmaps. We applied the decay in Eq. (13) (Section 3.3) to all RoPE blocks , i.e., we decay all attention beyond regardless of the frequency band. We added this intervention to the NIAH heatmaps in Figure 5 (Section 4.3) of the revised paper. Decaying all blocks also largely destroys retrieval at short context lengths (5–10K). As a result, the relative PPL difference would mix in the effect of short-range (5–10K) dependencies, and the score in Eq. (15) (Section 3.3) would no longer reflect the density of long-range dependencies in the original data. In contrast, decaying only impairs long-range retrieval while largely preserving short-range ability, so it identifies long-range dependencies more accurately.
2. All-band intervention: downstream training. We further trained on the data selected with the all-band intervention and added the results to Table 3 (Section 4.3) of the revised paper:
| Block set | HELMET | LongBench v2 | RULER | MRCR v2 | AA-LCR | LongProc | Overall |
|---|---|---|---|---|---|---|---|
| 64.74 | 27.40 | 75.52 | 16.21 | 6.67 | 11.10 | 33.61 | |
| 64.71 | 27.23 | 75.66 | 20.26 | 8.33 | 10.24 | 34.41 | |
| 64.37 | 25.90 | 76.20 | 16.89 | 10.33 | 11.72 | 34.24 | |
| 65.36 | 27.90 | 74.77 | 17.47 | 7.00 | 9.22 | 33.62 | |
| (all-band) | 63.46 | 26.20 | 74.60 | 14.94 | 9.33 | 9.26 | 32.97 |
The all-band intervention gives the lowest Overall score of 32.97, -1.44 below the predicted band (34.41). This matches the qualitative analysis in Figure 5 (Section 4.3). Therefore, the frequency-band mechanism is necessary: RoPE-Filter's ability to select data with long-range dependencies comes from selectively decaying the low-frequency band defined in Eq. (10) (Section 3.2).
(3/4) Response to Reviewer eKyh
Weakness 3: It remains unclear whether the head-level observations in Figure 1 generalize across different LLMs. The authors should provide similar analyses for additional models.
Response: Thank you for the suggestion. We have added this analysis in Appendix A.1 (Head-level FSG on Additional Models) of the revised paper. We apply the per-head analysis of Figure 1 (Section 3.2) to Qwen3-0.6B-Base and Qwen3-8B-Base under their native RoPE configurations (without YaRN). As shown in Figure 6 in Appendix A.1, over 90% of the attention heads in both models concentrate their FSG in low-frequency blocks, consistent with Llama-3.1-8B. The active region begins at approximately block 47, which agrees with the boundary predicted by Eq. (10). Both models share rope_theta and head_dim , so Eq. (10) gives . Substituting K gives . These results show that the head-level FSG mechanism holds across model families (Llama and Qwen) and model sizes (0.6B and 8B), and that the half-rotation criterion predicts the active region without model-specific probing.
Furthermore, in Appendix A.2 (Per-Head FSG Before and After YaRN) of the revised paper, we find that when Qwen3-8B-Base is extended from 32K to 64K with training-free YaRN, its head-level FSG values remain nearly unchanged (Figure 7 in Appendix A.2). This indicates that the low-frequency distribution of FSG is a property formed during training.
(1/4) Response to Reviewer eKyh
We sincerely appreciate your thoughtful and insightful review. We have uploaded a revised version of the paper, with all revisions highlighted in blue. Below, we address each of your comments point by point. All references to sections, tables, figures, and appendices refer to the revised paper. We hope that our responses and revisions help clarify the points raised in your review, and we would be grateful for any further feedback.
Response: The novelty of our work lies in the following three aspects.1. A relative measure of frequency-band contribution that reveals a mechanism shared by most heads. We propose FSG (Eqs. (8)–(9) in Section 3.2), which measures the relative attention preference among keys within the same distance range, rather than only the absolute attention magnitude. Repetitive data can produce consistently high attention scores over consecutive local positions, and FSG can distinguish this pattern from genuine selection of distant information. Using FSG, we find that over 90% of the attention heads exhibit low-frequency concentration across model families and sizes, including Llama-3.1-8B, Qwen3-0.6B-Base, and Qwen3-8B-Base (Figure 1 (Section 3.2) and Appendix A.1 (Head-level FSG on Additional Models)). This mechanism exists in the vast majority of heads and is not limited to specialized long-context heads such as retrieval heads [1].
2. Long-context analysis with a theoretical boundary estimate. We study this RoPE mechanism in long contexts and how it changes with distance (Figure 2 (Section 3.2)). We further derive a theoretical estimate of the low-frequency boundary, r∗(T)=⌈2d⋅lnbln(T/π)⌉ (Eq. (10) in Section 3.2), which can be computed directly from the RoPE configuration.
3. The first use of this mechanism for long-range dependency data selection. To the best of our knowledge, RoPE-Filter is the first method to connect this mechanism with long-context data selection. It introduces a new scoring perspective based on the relative PPL difference (Section 3.3): we remove the model's use of long-range dependencies in a document and measure the relative increase in PPL. A larger relative increase means that more long-range dependency information has been removed, which indicates stronger long-range dependency in the original document.
References
[1] Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. Retrieval Head Mechanistically Explains Long-Context Factuality. In The Thirteenth International Conference on Learning Representations (ICLR), 2025. https://openreview.net/forum?id=EytBpUGB1Z