acceptodds
Under review as a conference paper at ICLR 2027

RoPE-Filter: Identifying Long-Range Dependencies for Long-Context Data Selection

Abstract

Long-context modeling is a core capability of large language models. Continued pretraining on long-context data is a standard approach for extending model context windows. Effective long-context extension requires data with strong long-range dependencies, but simply increasing the length of training sequences does not reliably improve long-context capabilities. In this work, we propose RoPE-Filter, a training-free method that probes each document's reliance on distant context by selectively attenuating the contributions of low-frequency RoPE blocks to attention logits in long-distance interactions. The resulting relative increase in perplexity is used to select documents with strong long-range dependencies. Experiments across widely used long-context benchmarks show that models trained on data selected by RoPE-Filter achieve better performance than those trained on data selected by existing open-source methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Data Curation, Long-range Dependencies

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_eKyhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes RoPE-Filter, a training-free method for selecting long-context training data based on RoPE frequency characteristics. The authors observe that low-frequency RoPE components contribute substantially to long-range attention and propose selectively attenuating these components for distant token interactions. Documents are then ranked according to the relative increase in perplexity caused by this intervention, with the aim of identifying samples containing stronger long-range dependencies. Experimental results show improvements over existing long-context data selection methods.

Critical strengths
  1. Simple and interpretable methodology. The proposed method is straightforward and requires no additional model training. By leveraging the characteristics of different RoPE frequency components, RoPE-Filter provides an interpretable mechanism for identifying long-range dependencies and selecting long-context training data.

  2. Comprehensive experiments and promising results. The authors evaluate RoPE-Filter across multiple long-context benchmarks and compare it with several existing data selection methods. The results demonstrate consistent improvements across different data domains and models, highlighting the practical effectiveness of the proposed approach.

Critical weaknesses
  1. Limited novelty and insufficient justification of the proposed design. The roles of different RoPE frequency components have already been investigated in previous studies. The authors should better clarify the novelty of their approach. More importantly, I suggest adding a direct ablation that uniformly attenuates all long-range attention logits without distinguishing RoPE frequency bands, while keeping the same perplexity-based selection criterion. This would help determine whether the improvements come from the proposed low-frequency intervention or simply from weakening long-range attention.

  2. Limited generalizability of Figure 1. Figure 1 only presents head-level frequency contribution results for Llama-3.1-8B. Although Figure 2 includes additional models, it remains unclear whether the head-level observations in Figure 1 generalize across different LLMs. The authors should provide similar analyses for additional models.

  3. Insufficient case studies. The paper aims to identify documents with stronger long-range dependencies, but it lacks concrete examples demonstrating that the selected samples indeed exhibit such characteristics. I suggest presenting representative high- and low-scoring documents and highlighting their long-range dependencies to provide more direct and interpretable evidence for the effectiveness of RoPE-Filter.

Questions for authors

Please see the weaknesses listed above.

5 comments
AuthorsOpenReview ↗

(1/4) Response to Reviewer eKyh

We sincerely appreciate your thoughtful and insightful review. We have uploaded a revised version of the paper, with all revisions highlighted in blue. Below, we address each of your comments point by point. All references to sections, tables, figures, and appendices refer to the revised paper. We hope that our responses and revisions help clarify the points raised in your review, and we would be grateful for any further feedback.


Weakness 1: The roles of different RoPE frequency components have already been investigated in previous studies. The authors should better clarify the novelty of their approach.

Response: The novelty of our work lies in the following three aspects.

1. A relative measure of frequency-band contribution that reveals a mechanism shared by most heads. We propose FSG (Eqs. (8)–(9) in Section 3.2), which measures the relative attention preference among keys within the same distance range, rather than only the absolute attention magnitude. Repetitive data can produce consistently high attention scores over consecutive local positions, and FSG can distinguish this pattern from genuine selection of distant information. Using FSG, we find that over 90% of the attention heads exhibit low-frequency concentration across model families and sizes, including Llama-3.1-8B, Qwen3-0.6B-Base, and Qwen3-8B-Base (Figure 1 (Section 3.2) and Appendix A.1 (Head-level FSG on Additional Models)). This mechanism exists in the vast majority of heads and is not limited to specialized long-context heads such as retrieval heads [1].

2. Long-context analysis with a theoretical boundary estimate. We study this RoPE mechanism in long contexts and how it changes with distance (Figure 2 (Section 3.2)). We further derive a theoretical estimate of the low-frequency boundary, r∗(T)=⌈d2⋅ln⁡(T/π)ln⁡b⌉r^{\ast}(T)=\left\lceil \frac{d}{2}\cdot\frac{\ln(T/\pi)}{\ln b}\right\rceil (Eq. (10) in Section 3.2), which can be computed directly from the RoPE configuration.

3. The first use of this mechanism for long-range dependency data selection. To the best of our knowledge, RoPE-Filter is the first method to connect this mechanism with long-context data selection. It introduces a new scoring perspective based on the relative PPL difference (Section 3.3): we remove the model's use of long-range dependencies in a document and measure the relative increase in PPL. A larger relative increase means that more long-range dependency information has been removed, which indicates stronger long-range dependency in the original document.

References

[1] Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. Retrieval Head Mechanistically Explains Long-Context Factuality. In The Thirteenth International Conference on Learning Representations (ICLR), 2025. https://openreview.net/forum?id=EytBpUGB1Z

AuthorsOpenReview ↗

(2/4) Response to Reviewer eKyh

Weakness 2: Adding a direct ablation that uniformly attenuates all long-range attention logits without distinguishing RoPE frequency bands, while keeping the same perplexity-based selection criterion.

Response: Thank you for the suggestion. We have added this all-band ablation to the revised paper and evaluate it from two perspectives.

1. All-band intervention: NIAH heatmaps. We applied the decay in Eq. (13) (Section 3.3) to all RoPE blocks [0,63][0,63], i.e., we decay all attention beyond Δ>D\Delta > D regardless of the frequency band. We added this intervention to the NIAH heatmaps in Figure 5 (Section 4.3) of the revised paper. Decaying all blocks also largely destroys retrieval at short context lengths (5–10K). As a result, the relative PPL difference would mix in the effect of short-range (5–10K) dependencies, and the score in Eq. (15) (Section 3.3) would no longer reflect the density of long-range dependencies in the original data. In contrast, decaying only [47,63][47,63] impairs long-range retrieval while largely preserving short-range ability, so it identifies long-range dependencies more accurately.

2. All-band intervention: downstream training. We further trained on the data selected with the all-band intervention [0,63][0,63] and added the results to Table 3 (Section 4.3) of the revised paper:

Block setHELMETLongBench v2RULERMRCR v2AA-LCRLongProcOverall
[45,63][45,63]64.7427.4075.5216.216.6711.1033.61
[47,63]⋆[47,63]^{\star}64.7127.2375.6620.268.3310.2434.41
[50,63][50,63]64.3725.9076.2016.8910.3311.7234.24
[30,46][30,46]65.3627.9074.7717.477.009.2233.62
[0,63][0,63] (all-band)63.4626.2074.6014.949.339.2632.97

The all-band intervention gives the lowest Overall score of 32.97, -1.44 below the predicted band [47,63][47,63] (34.41). This matches the qualitative analysis in Figure 5 (Section 4.3). Therefore, the frequency-band mechanism is necessary: RoPE-Filter's ability to select data with long-range dependencies comes from selectively decaying the low-frequency band B∗(T)\mathcal{B}^{\ast}(T) defined in Eq. (10) (Section 3.2).

AuthorsOpenReview ↗

(3/4) Response to Reviewer eKyh

Weakness 3: It remains unclear whether the head-level observations in Figure 1 generalize across different LLMs. The authors should provide similar analyses for additional models.

Response: Thank you for the suggestion. We have added this analysis in Appendix A.1 (Head-level FSG on Additional Models) of the revised paper. We apply the per-head analysis of Figure 1 (Section 3.2) to Qwen3-0.6B-Base and Qwen3-8B-Base under their native RoPE configurations (without YaRN). As shown in Figure 6 in Appendix A.1, over 90% of the attention heads in both models concentrate their FSG in low-frequency blocks, consistent with Llama-3.1-8B. The active region begins at approximately block 47, which agrees with the boundary predicted by Eq. (10). Both models share rope_theta =106=10^{6} and head_dim d=128d=128, so Eq. (10) gives r∗(T)=⌈64⋅ln⁡(T/π)ln⁡(106)⌉r^{\ast}(T)=\left\lceil 64\cdot\frac{\ln(T/\pi)}{\ln(10^{6})}\right\rceil. Substituting T=64T=64K gives r∗=47r^{\ast}=47. These results show that the head-level FSG mechanism holds across model families (Llama and Qwen) and model sizes (0.6B and 8B), and that the half-rotation criterion predicts the active region without model-specific probing.

Furthermore, in Appendix A.2 (Per-Head FSG Before and After YaRN) of the revised paper, we find that when Qwen3-8B-Base is extended from 32K to 64K with training-free YaRN, its head-level FSG values remain nearly unchanged (Figure 7 in Appendix A.2). This indicates that the low-frequency distribution of FSG is a property formed during training.

Reviewer_bFsXOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper defines Frequency-band Selection Gain (FSG): within the same query–key distance bin, it contrasts a RoPE frequency band's contribution to attention-preferred keys with its contribution to uniformly sampled keys in that bin, producing a distance-resolved frequency-band selection spectrum. The authors observe that long-distance selection spectra are generally concentrated in the low-frequency region and propose predicting the band boundary using the half-rotation condition θ_r T ≤ π. They then attenuate the corresponding low-frequency bands in long-range attention interactions, while preserving a local 4K window and the first eight keys, and rank documents by the relative change in perplexity (PPL) before and after the intervention. In the main experiments, Qwen3-8B-Base with YaRN serves as the scoring model. For each of ArXiv, Book, and CommonCrawl, 1.6B long-document tokens are selected from a 10B-token candidate pool and mixed with a shared 0.4B-token short-document set. Llama-3-8B then undergoes continued pretraining at a 128K context length followed by the same supervised fine-tuning (SFT) procedure, and is evaluated on six groups of long-context tasks.

Critical strengths

The experimental comparison is fairly thorough and fair: All methods share the same 10B candidate pool, the same selection budget, the same short-document set, and the same packing and training configuration; the only variable is which long documents are selected (App. B.1). This is the right experimental design.

Critical weaknesses

A random-selection baseline is also missing. Another important ablation would be to replace Eq. 13 with a simple rule: decay all attention beyond Δ>D\Delta>D, regardless of the frequency band—in other words, compare sliding-window PPL with full-context PPL. This would directly test whether the frequency-band mechanism is necessary at all. If plain truncation achieves comparable gains, the central claim of Contribution 2—the role of the RoPE frequency band—would not be supported. The [29,45][29,45] row in Table 3 only swaps one band for another and cannot substitute for this control.

Questions for authors
  1. How do you explain LongProc 21.51 → 33.13 in the Qwen3-14B results of Table 2?

  2. Could the authors report the length distribution of the 1.6B selected tokens for each method ?

9 comments
AuthorsOpenReview ↗

(1/3) Response to Reviewer bFsX

Thank you for your careful and insightful review. We have uploaded a revised version of the paper, with all revisions highlighted in blue. All references to sections, tables, figures, and appendices below refer to the revised paper.

We would also like to point out that the paragraph Scoring Model and Frequency-Band Ablations in Section 4.3 (Ablation Studies) of the original submission contained a few inadvertent typos in the description of the frequency bands. The frequency-band ablations we actually ran correspond to the results shown in Figure 5 (Section 4.3). We have corrected all of these typos in the revised paper and highlighted the changes in blue. We apologize for any confusion they may have caused.

Below, we address each of your comments point by point. We hope our responses address your concerns, and we would be grateful for any further feedback.


Weakness 1: A random-selection baseline is also missing. Another important ablation would be to replace Eq. 13 with a simple rule: decay all attention beyond Δ>D\Delta > D, regardless of the frequency band.

Response:

1. Random-selection baseline under a length-matched control. We added a Random baseline in the first block (Length distribution ablation) of Table 2 (Section 4.3) of the revised paper. To fully remove the effect of document length, this block strictly fixes the length distribution for every method: the 10B Book pool is split into five length bins (64K–80K, 80K–96K, 96K–112K, 112K–128K, >128K), and every method selects 320M tokens from each bin (1:1:1:1:1). Random samples uniformly within each bin, while the other methods use Qwen3-8B-Base extended to 64K with YaRN for scoring. Llama-3-8B is used for continual pretraining. The best result in each column is shown in bold.

MethodHELMETLongBench v2RULERMRCR v2AA-LCRLongProcOverall
Random64.4325.1075.4516.748.0010.2933.34
LongAttn65.0227.1075.3418.8810.339.4334.35
LADM64.2124.3074.2619.289.3312.8134.03
LongFilter65.4627.2074.2720.289.0011.7334.66
RoPE-Filter65.4726.8075.5721.899.3311.8235.15

Random gives the lowest Overall score (33.34) and is clearly worse than all data-selection methods. RoPE-Filter outperforms Random by +1.81 and still leads the second-best method by +0.49. Random is therefore a weak baseline, which agrees with the original LongAttn and LADM papers, where both methods were also compared against random selection and outperformed it.

2. All-band intervention: NIAH heatmaps. We applied the decay in Eq. (13) (Section 3.3) to all RoPE blocks [0,63][0,63], i.e., we decay all attention beyond Δ>D\Delta > D regardless of the frequency band. We added this intervention to the NIAH heatmaps in Figure 5 (Section 4.3) of the revised paper. Decaying all blocks also largely destroys retrieval at short context lengths (5–10K). As a result, the relative PPL difference would mix in the effect of short-range (5–10K) dependencies, and the score in Eq. (15) (Section 3.3) would no longer reflect the density of long-range dependencies in the original data. In contrast, decaying only [47,63][47,63] impairs long-range retrieval while largely preserving short-range ability, so it identifies long-range dependencies more accurately.

3. All-band intervention: downstream training. We further trained on the data selected with the all-band intervention [0,63][0,63] and added the results to Table 3 (Section 4.3) of the revised paper:

Block setHELMETLongBench v2RULERMRCR v2AA-LCRLongProcOverall
[45,63][45,63]64.7427.4075.5216.216.6711.1033.61
[47,63]⋆[47,63]^{\star}64.7127.2375.6620.268.3310.2434.41
[50,63][50,63]64.3725.9076.2016.8910.3311.7234.24
[30,46][30,46]65.3627.9074.7717.477.009.2233.62
[0,63][0,63] (all-band)63.4626.2074.6014.949.339.2632.97

The all-band intervention gives the lowest Overall score of 32.97, -1.44 below the predicted band [47,63][47,63] (34.41). This matches the qualitative analysis in Figure 5 (Section 4.3). Therefore, the frequency-band mechanism is necessary: RoPE-Filter's ability to select data with long-range dependencies comes from selectively decaying the low-frequency band B∗(T)\mathcal{B}^{\ast}(T) defined in Eq. (10) (Section 3.2).

AuthorsOpenReview ↗

(2/3) Response to Reviewer bFsX

Question 1: How do you explain LongProc 21.51 → 33.13 in the Qwen3-14B results of Table 2?

Response: As shown in Table 7 in Appendix E (Per-Task Results on LongProc) of the revised paper, the gain is particularly large at the 8K output length, where the average score rises from at most 3.41 for the baselines to 16.35. This improvement is concentrated on HTML to TSV and Countdown. Both tasks require the model to maintain and update intermediate states over a long generated sequence while integrating information dispersed across the input, which is exactly long-range state tracking.

This capability matches the data that RoPE-Filter selects. RoPE-Filter favors texts with complex narrative structures, multiple turning points, and explicit state dependencies, such as Darksong Rising (Figure 8 in Appendix D (Case Study: High- and Low-Scoring Documents)), whose later passages rely on characters and events introduced far earlier. RoPE-Filter also has the lowest overlap with highly repetitive data in all three domains (Table 8 in Appendix E (Per-Task Results on LongProc), reproduced below). The table reports the overlap (B tokens) between the 1.6B tokens selected by each method and the 1B tokens with the highest 8-gram repetition rate in the same 10B pool. The best result in each column is shown in bold.

Method (1.6B selected)Book ↓\downarrowArXiv ↓\downarrowCommonCrawl ↓\downarrow
LongAttn0.5050.3690.312
LADM0.4180.2750.143
LongFilter0.4510.5300.426
RoPE-Filter0.0540.0820.137

Training on such low-repetition data with genuine long-range dependencies strengthens long-range state tracking, which explains the large LongProc gain on long-output reasoning tasks.

AuthorsOpenReview ↗

(3/3) Response to Reviewer bFsX

Question 2: Could the authors report the length distribution of the 1.6B selected tokens for each method?

Response: The length distributions of the 1.6B tokens selected by each method and of the 10B candidate pool are reported in Figure 4 (Section 4.3) and Table 10 in Appendix F.2 (Length Distribution of Selected Data) of the revised paper. Table 10 is reproduced below. Each entry is the token share (%) of a document-length bin, computed as 100×Nbin/Ntotal100 \times N_{\text{bin}} / N_{\text{total}}, where NbinN_{\text{bin}} is the number of tokens from documents in the bin and NtotalN_{\text{total}} is the total number of tokens in the set.

DomainMethod16–32K32–48K48–64K64–80K80–96K96–112K112–128K$ > $128K
ArXivLongAttn12.9542.8618.337.674.022.701.789.69
ArXivLADM65.0713.927.884.192.261.461.084.14
ArXivLongFilter15.9823.3622.2014.687.374.463.278.67
ArXivRoPE-Filter28.6920.8916.1614.896.373.062.237.71
ArXiv10B Pool43.0922.7812.626.833.682.501.726.78
BookLongAttn0.000.000.007.7511.1514.0315.4051.67
BookLADM0.000.000.006.348.369.6510.1265.53
BookLongFilter0.000.000.007.857.768.668.2067.53
BookRoPE-Filter0.000.000.005.809.0112.7914.8357.57
Book10B Pool0.000.000.006.328.4510.0410.6464.55
CommonCrawlLongAttn0.0033.1618.0411.267.496.654.7518.65
CommonCrawlLADM0.0030.1416.7611.127.926.886.8420.35
CommonCrawlLongFilter0.0029.8121.4814.309.256.764.9413.47
CommonCrawlRoPE-Filter0.0030.0715.6912.178.688.625.0319.74
CommonCrawl10B Pool0.0031.0617.4111.387.866.745.8619.70

We further ablate the effect of length distribution in Table 2 (Section 4.3), first block, where the length distribution is fixed for all methods by selecting 320M tokens from each of five length bins on Book. RoPE-Filter still achieves the highest Overall score of 35.15 (+0.49 over the second-best method). In addition, on Book, LongFilter and LADM select more tokens from documents longer than 128K (67.53% and 65.53%) than RoPE-Filter (57.57%), yet RoPE-Filter achieves the highest Overall score in Table 1 (Section 4.2). Together, these results show that although length distribution is a relevant variable, training on more long documents does not necessarily yield better performance. When length distributions are similar, long-range dependency is the dominant factor.

Reviewer_B2b5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RoPE-Filter, a training-free method for selecting long-context pretraining documents. The authors introduce Frequency-band Selection Gain (FSG), which measures individual RoPE blocks’ contributions to attention preferences within distance bins. They observe low-frequency dominance at long distances and an approximate boundary following a half-rotation criterion. RoPE-Filter attenuates the selected blocks’ contributions to long-range attention logits and ranks documents by the resulting relative perplexity increase. Experiments across three domains report aggregate improvements over LongAttn, LADM, and LongFilter, with additional scoring-model and training-model ablations.

Critical strengths
  1. An interesting connection between mechanistic analysis and practical data selection. Distance-conditioned FSG offers a useful perspective on how RoPE blocks contribute to attention preferences. Figure 4 includes adjacent-band and random higher-frequency controls, supporting the importance of the identified low-frequency region for long-range retrieval. Using this observation to construct a document-selection score is a potentially useful contribution.

  2. A substantial downstream evaluation with controlled training conditions. The methods share candidate pools, selected-token budgets, short-document mixtures, packing procedures, and training configurations. Table 1 reports the highest aggregate score in each domain, averaging 33.72 versus 32.94 for LongFilter. Three-seed means, an alternative scoring model, a larger training model, and short-context evaluations strengthen the practical evidence.

Critical weaknesses
  1. The frequency-based justification and interpretation of the selection score remain insufficiently established. There is a concrete question about the boundary used after YaRN. With the released Qwen3-8B-Base configuration (�0�)(�0�) and standard YaRN settings for extending 32K to 64K—effective factor 2, beta_fast=32, and beta_slow=1—applying the half-rotation criterion to the effective frequencies gives a boundary of block 43, whereas block 47 corresponds to the unscaled frequencies. The paper reports 47 after YaRN without providing sufficient configuration details to reproduce this calculation. This could be a configuration or reporting issue, but it directly concerns the claim that the intervention region is predictable from the effective RoPE configuration. See the released model configuration and reference YaRN implementation. More generally, showing that the intervention damages NIAH retrieval does not establish that its document-level perplexity damage reliably ranks useful long-range dependencies. Length is a particularly relevant potential confound: if Equation 12 uses a fixed W=64KW=64K, shorter documents experience systematically weaker attenuation. Table 3 provides a useful higher-frequency-band control, but a length-matched selection comparison and a simpler all-band long-range intervention would more directly establish the value of the proposed frequency-specific mechanism.
  2. The robustness and breadth of the empirical advantage are not sufficiently characterized. The paper reports three-seed means, which is a strength, but does not report variability for the actual method comparisons. Several margins are modest: the Book improvement over the strongest baseline is 0.38 in Table 1, and the alternative-scoring-model improvement is 0.32 in Table 2. Appendix B.3 estimates variability using a separate fixed 1B-token configuration; this does not establish uncertainty for the main 2B-token comparisons. Per-seed results and paired evaluation uncertainty are needed to assess these margins. The larger-model result also supports a narrower conclusion than the discussion suggests. Against LongAttn, LongProc improves from 21.51 to 33.13, accounting for the entire aggregate advantage. Across the remaining five benchmarks, RoPE-Filter averages approximately 44.03 versus 44.57 for LongAttn. This is a valuable task-specific improvement, but does not establish a broad increase in advantage with model size. Moreover, this comparison changes both model size and model family.
Questions for authors
  1. Frequency configuration: Please provide the exact scoring-model YaRN configuration and effective inverse frequencies. How does applying Equation 10 to these frequencies produce boundary 47 rather than 43 under standard factor-2 scaling?

  2. Score interpretation: How are documents shorter than 64K handled in Equations 12–14? What are the candidate and selected length distributions? Within the existing Book setup, can a length-matched selection control and an all-band long-range intervention clarify whether frequency specificity explains the downstream benefit?

  3. Empirical uncertainty: Can you provide the existing per-seed results for the principal comparisons and the 14B LongProc result? Do the aggregate advantages persist across seeds, and what evidence supports interpreting the larger-model result as a general scaling benefit?

6 comments
AuthorsOpenReview ↗

(1/6) Response to Reviewer B2b5

Thank you for your careful and insightful review. We have uploaded a revised version of the paper, with all revisions highlighted in blue. Below, we address each of your comments point by point. All references to sections, tables, figures, and appendices refer to the revised paper. We hope our responses address your concerns, and we would be grateful for any further feedback.


Weakness 1: The paper reports 47 after YaRN without providing sufficient configuration details to reproduce this calculation. This could be a configuration or reporting issue.

Question 1: Please provide the exact scoring-model YaRN configuration and effective inverse frequencies. How does applying Equation 10 to these frequencies produce boundary 47 rather than 43 under standard factor-2 scaling?

Response: The low-frequency distribution of FSG is a property formed during training. When Qwen3-8B-Base is extended from 32K to 64K with training-free YaRN, its head-level FSG values remain nearly unchanged. Therefore, when Qwen3-8B-Base with YaRN (64K) is used as the scoring model, we compute the FSG high/low-frequency boundary using the native configuration of Qwen3-8B-Base. The detailed calculation is as follows.

  • Qwen3-8B-Base (Qwen3-8B-Base with YaRN-64K shares the same boundary): rope_theta =106=10^{6}, head_dim d=128d=128, and rope_scaling is null, so θr=(106)−2r/d\theta_r=(10^{6})^{-2r/d} and Eq. (10) applies directly: r∗(T)=⌈64⋅ln⁡(T/π)ln⁡(106)⌉r^{\ast}(T)=\left\lceil 64\cdot\frac{\ln(T/\pi)}{\ln(10^{6})}\right\rceil. Substituting T=64T=64K gives r∗=47r^{\ast}=47.
  • Llama-3.1-8B: rope_theta =5×105=5\times10^{5}, head_dim d=128d=128, and rope_scaling of type llama3 (factor =8=8, low_freq_factor =1=1, high_freq_factor =4=4, original_max_position_embeddings =8192=8192). Under this scaling, every block whose wavelength exceeds 8192 has its frequency divided by 8, i.e., θreffective=(5×105)−2r/d/8\theta_r^{\mathrm{effective}}=(5\times10^{5})^{-2r/d}/8. The boundary block at T=64T=64K falls in this regime, so Eq. (10) becomes r∗(T)=⌈64⋅ln⁡(T/(8π))ln⁡(5×105)⌉r^{\ast}(T)=\left\lceil 64\cdot\frac{\ln\big(T/(8\pi)\big)}{\ln(5\times10^{5})}\right\rceil. Substituting T=64T=64K gives r∗=39r^{\ast}=39.

In both cases, Eq. (10) is applied to the RoPE frequencies under which the model was trained. Llama-3.1-8B was pretrained with its llama3 rope_scaling, so its scaled frequencies are the trained frequencies. In contrast, the YaRN extension of Qwen3-8B-Base is training-free and does not update the model parameters, so its trained frequencies remain the native ones. This is why the boundary for Qwen3-8B-Base with YaRN (64K) is computed from the native configuration. Further visualizations and discussion are provided in Appendix A.2 (Per-Head FSG Before and After YaRN) of the revised paper.

AuthorsOpenReview ↗

(2/6) Response to Reviewer B2b5

Weakness 2: More generally, showing that the intervention damages NIAH retrieval does not establish that its document-level perplexity damage reliably ranks useful long-range dependencies.

Response: We address this concern from three perspectives.

1. Direct evidence from training outcomes. Long-range dependency in data is not a well-defined quantity in itself, so the most convincing measure is the performance of a model trained on that data. We conducted an additional experiment: we ranked a 10B-token Book pool with RoPE-Filter and trained on three 1.6B-token segments taken from different positions in the ranking, namely Top (0–1.6B), Middle (4.2–5.8B), and Bottom (8.4–10B). The results are as follows:

SegmentHELMETLongBench v2RULERMRCR v2AA_LCRLongProcOverall
Top 1.6B64.7127.2375.6620.268.3310.2434.41
Middle 1.6B63.7826.2074.7119.548.3311.3733.99
Bottom 1.6B63.7925.2074.3518.896.3310.4033.16

The Overall score decreases monotonically from Top to Middle to Bottom, which matches the RoPE-Filter ranking. In addition, Table 1 (Section 4.2) of the paper shows that RoPE-Filter outperforms recent open-source data selection methods. Together, these results show that RoPE-Filter is an effective proxy for ranking long-range dependencies.

2. Mechanistic evidence from NIAH. NIAH is one type of long-context task. The NIAH heatmaps in Figure 5 show that the intervention selectively damages long-distance retrieval, while shorter ranges (5–10K) are largely preserved. This specific case indicates that the attenuation targets long-range capability. Because the intervention impairs mainly the model's long-range capability, the relative PPL difference before and after attenuation reflects how strongly a document depends on long-range context. This gives a mechanistic explanation of why RoPE-Filter is an effective proxy for ranking long-range dependencies.

3. Evidence from repetitive data. Highly repetitive data contains few long-range dependencies. RoPE-Filter filters out such data more effectively than other methods (Table 8 in Appendix E (Per-Task Results on LongProc) of the revised paper, reproduced below). The table reports the overlap (B tokens) between the 1.6B tokens selected by each method and the 1B tokens with the highest 8-gram repetition rate in the same 10B pool. In other words, RoPE-Filter reliably excludes data with clearly weak long-range dependencies, which further supports the reliability of its ranking.

Method (1.6B selected)Book ↓\downarrowArXiv ↓\downarrowCommonCrawl ↓\downarrow
LongAttn0.5050.3690.312
LADM0.4180.2750.143
LongFilter0.4510.5300.426
RoPE-Filter0.0540.0820.137
AuthorsOpenReview ↗

(3/6) Response to Reviewer B2b5

Weakness 3: Length is a particularly relevant potential confound: if Equation 12 uses a fixed W=64W=64K, shorter documents experience systematically weaker attenuation.

Question 2: How are documents shorter than 64K handled in Equations 12–14? What are the candidate and selected length distributions?

Response:

1. Handling of documents shorter than 64K. The scoring window of the ii-th document is Wi=min⁡(ni,64K)W_i=\min(n_i, 64\text{K}), where nin_i is its length in tokens, and WW in Eq. (12) (Section 3.3) refers to WiW_i. The decay schedule therefore always spans the full scoring window of each document, so shorter documents are not systematically under-attenuated. RoPE-Filter scores the original candidate pools with Qwen3-8B-Base extended to 64K via YaRN. The Spearman correlations between the RoPE-Filter score ranking and the length ranking are as follows, showing that RoPE-Filter has no length preference in data selection:

ArXivBookCommonCrawl
Spearman correlation+0.027+0.032−0.060

2. Candidate and selected length distributions. Figure 4 (Section 4.3) and Table 10 in Appendix F.2 (Length Distribution of Selected Data) of the revised paper report the length distributions of the data selected by each method and of the 10B candidate pool. Together with the near-zero Spearman correlations, these results show that the RoPE-Filter score has no preference for document length.

3. Long-range dependency, not length, drives the gains. As shown in Table 1 (Section 4.2) and Figure 4 (Section 4.3), although length distribution is a relevant variable, training on more long documents does not necessarily yield better performance. When length distributions are similar, long-range dependency is the dominant factor. Specifically, on Book, LongFilter and LADM select more tokens from documents longer than 128K (67.53% and 65.53%) than RoPE-Filter (57.57%) (Table 10 in Appendix F.2 (Length Distribution of Selected Data)), yet RoPE-Filter achieves the highest Overall score (Table 1 (Section 4.2)). Moreover, when the length distribution is fixed for all methods by selecting 320M tokens from each of five length bins, RoPE-Filter still achieves the highest Overall score of 35.15 (+0.49 over the second-best method, Table 2 (Section 4.3), first block).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary for Area Chair

Dear Reviewers, ACs, and SACs,

We sincerely thank the Reviewers, ACs, and SACs for the time and effort you have devoted to our submission. We are truly grateful for the reviewers' careful reading and valuable comments, from which we have learned a great deal and which have greatly improved our paper. Below, we would like to respectfully provide a brief summary of our work and the rebuttal process.


Core Value of Our Work

1. From mechanism discovery to practical application. Based on FSG (Eqs. (8)–(9)), our proposed measure of relative attention preference, we discover a general mechanism of long-range attention: across model families and sizes, the vast majority of attention heads (>90%) rely on low-frequency RoPE bands to select distant information (Figure 1, Appendix A.1). We further provide a theoretical boundary (Eq. (10)) that agrees well with the measured results (Figure 2, Appendix A.2). Building on this mechanism, we propose RoPE-Filter, a simple, training-free method with a precisely targeted intervention that effectively measures the strength of long-range dependency in a document. Frequency-band ablations and comprehensive benchmark evaluations (Figure 5, Tables 1–3) show that the score is a reliable ranking criterion for long-range dependency, and that the identified frequency band is both necessary and central to the method. The work thus closes the loop from a mechanistic finding in LLMs to a practical application.

2. A new scoring perspective with a clear mechanistic explanation. RoPE-Filter scores a document from a new angle: we remove the model's use of long-range dependencies in a document and measure the relative increase in PPL. A larger relative increase means that more long-range dependency information has been removed, which indicates stronger long-range dependency in the original document. We explain in detail, from a mechanistic perspective, why this scoring approach leads to better downstream training results (see Response to Reviewer bFsX).

3. Strong generality, robustness, and scalability. RoPE-Filter achieves the best Overall performance on six long-context benchmarks in all three data domains, and it does so on every individual seed (Table 1, Table 5). On the larger Qwen3-14B-Base, the model trained on RoPE-Filter data achieves both the best Overall benchmark score (Table 2) and the lowest LongPPL (see Response to Reviewer B2b5). This shows that the model trained on RoPE-Filter data indeed has the strongest long-context ability among all compared methods.


Rebuttal Summary

We are grateful for the reviewers' careful and constructive comments, and we have taken every one of them seriously. We went through each weakness and each question raised by every reviewer sentence by sentence, and made sure that none was left out. For each point, we provided a detailed point-by-point response, conducted the corresponding additional experiments, and updated the paper accordingly, with all revisions highlighted in blue. We believe that all concerns raised by the reviewers have now been fully answered and resolved.


Current Status of the Discussion

  • Reviewer bFsX kindly gave a positive recommendation after our rebuttal: "I believe this is a paper worthy of acceptance."
  • Reviewer eKyh kindly raised the score: "Given their excellent and sincere response, I will increase my score."
  • Reviewer B2b5 has not replied so far. We have responded to every question raised in this review, one by one and without omission, with solid answers supported by the corresponding experimental results. We believe that all of the concerns raised by Reviewer B2b5 have been resolved.

Once again, we sincerely thank the Reviewers, ACs, and SACs. Long-context modeling is an important capability of LLMs, and we hope that our work can contribute to the progress of this field. Finally, we are deeply grateful for all the guidance we have received on this work, which has been truly inspiring for our future research.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.