acceptodds
Desk rejected at ICLR 2027

From Frequency to Function: Component-Specific RoPE Extrapolation for Long-Context Language Models

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Rotary Position Embedding (RoPE)-based language models form position-sensitive attention by superimposing contributions from dimension pairs associated with different rotation frequencies. Long-context extrapolation therefore requires rescaling rotation frequencies. However, common frequency-based methods partition dimensions by the relative ordering of their associated RoPE frequencies and scale attention logits with a single global factor. This design overlooks the distinct functional roles that dimension pairs acquire during pretraining. We group dimension pairs into three classes: positional pairs have concentrated pre-RoPE activation phases and produce stable relative-position periodicity, semantic pairs are dominated by input content variation, and sink pairs make exceptionally large positive contributions to sink-token attention logits. Based on this decomposition, we propose FARoPE (Function-Aware RoPE), a training-free RoPE extrapolation method that first identifies the three pair classes and then applies component-specific transformations. Specifically, FARoPE applies a shared frequency scale to positional pairs to prevent spurious shifts in position-sensitive attention patterns. Sink pairs receive the same frequency reduction to maintain sink alignment, while semantic pairs retain the YaRN ramp. FARoPE separately calibrates the additional attention-logit gain for sink pairs without masking sink tokens. Our evaluation covers two model families and three long-context benchmarks with context windows up to 128K.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-context language models, Attention sinks

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_uPmtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FARoPE explores a function-aware approach to training-free context extension for RoPE-based language models. Instead of relying solely on frequency order, it uses phase concentration and sink bias to classify RoPE dimension pairs into positional, semantic, and sink components. Different frequency scaling rules are applied to these components, together with a separate attention-logit calibration for sink pairs. The method is evaluated on Qwen3-8B and Llama 3 8B Instruct across RULER, Infinite-Bench, and LongBench v2, with additional ablations and calibration studies.

Critical strengths

At 128K context length, FARoPE improves the RULER score from 83.2 to 85.5 on Qwen3-8B and from 58.2 to 63.4 on Llama 3 8B compared with the strongest baselines. Appendix C examines the contributions of the two main modules, and the calibration results remain similar when switching between PG-19 and C4. These experiments give some support to the effectiveness and stability of the proposed method.

Instead of assigning RoPE scaling rules based only on frequency, FARoPE distinguishes dimension pairs according to their observed behavior. Positional and sink pairs use a shared frequency reduction, while semantic pairs follow the YaRN scaling rule. The separate treatment of sink-related logits is also useful, as it limits additional sink amplification without modifying the pretrained model weights.

Critical weaknesses

One question left open by the ablation studies is how important the proposed functional classification actually is. In Appendix C, the pair assignments remain fixed while the frequency scaling and logit calibration modules are removed separately. It is therefore difficult to separate the effect of the classification from that of the scaling rules. For example, would grouping dimensions by frequency or randomly assigning pairs to groups of similar sizes give comparable results under the same transformations? Without such comparisons, the benefit of identifying functional roles remains unclear.

The results on Infinite-Bench and LongBench v2 are less convincing than those on RULER. The improvements over the strongest baselines are mostly around one percentage point, and no confidence intervals are provided for these main comparisons. The evaluation also covers only two 8B models. In particular, Llama 3 achieves 18.1 on LongBench v2, below the nominal random-choice reference of 25%. While the controlled long-context results are encouraging, I am less certain how much the method improves more challenging long-context understanding and reasoning.

Questions for authors
  1. Have the authors tried alternative ways of grouping RoPE dimension pairs, such as frequency-based partitioning or random assignments with matched group sizes? Applying the same scaling rules to these groups would help isolate the contribution of the functional classification.
  2. Could the authors report confidence intervals for the main Infinite-Bench and LongBench v2 comparisons? Given the small differences between methods, it would be useful to know whether the improvements are statistically reliable.
4 comments
AuthorsOpenReview ↗

(1/2) Author Rebuttal

Thank you for your careful and balanced review. We appreciate your recognition of the benefit of selecting RoPE pairs by their observed behavior. Your suggestions have helped us demonstrate this contribution more directly.

W1(Q1) Effectiveness of Functional Classification

Following your suggestion, we added frequency-based and random assignments to separate the effect of classification from that of the scaling rules, while keeping FARoPE's transformations unchanged (Table 1).

To isolate pair selection, we match class counts in every layer and KV group. Spectrum-matched random grouping also preserves these counts within each YaRN region. We replace semantic/positional assignments, sink assignments, or both, and refit sink calibration for each alternative.

FARoPE exceeds every control on both models across all three benchmarks, including each random control's best seed. Replacing either set of assignments reduces performance, with larger drops for semantic/positional labels. These results support the contribution of the functional classification under the same transformations.

Table 1. Functional Classification Versus Alternative Groupings

RULER Avg. averages six input lengths (4K–128K). Random controls report mean ± sample standard deviation over five seeds, with the best score in parentheses.

Grouping strategyReplacement scopeQwen3-8B RULER Avg.Qwen3-8B Infinite-BenchQwen3-8B LongBench v2Llama 3 8B Instruct RULER Avg.Llama 3 8B Instruct Infinite-BenchLlama 3 8B Instruct LongBench v2
FARoPEOriginal functional assignments95.2325.3435.2079.5431.2418.07
Frequency-basedSemantic/positional87.3822.9432.2461.4221.4716.82
Frequency-basedSink93.2924.2732.8977.9630.3817.45
Frequency-basedBoth84.5922.4631.9160.6220.1316.82
Random, size-matchedSemantic/positional78.65 ± 1.60 (79.97)21.20 ± 0.79 (21.79)29.47 ± 1.13 (30.59)58.43 ± 1.26 (59.81)21.43 ± 0.37 (21.91)15.70 ± 0.36 (16.20)
Random, size-matchedSink91.09 ± 1.30 (92.37)22.54 ± 0.91 (23.36)32.43 ± 1.08 (33.55)75.45 ± 0.58 (76.23)29.04 ± 0.41 (29.63)17.38 ± 0.26 (17.76)
Random, size-matchedBoth77.71 ± 1.75 (79.35)20.50 ± 1.03 (21.36)28.75 ± 1.18 (29.93)54.36 ± 1.22 (55.79)19.68 ± 0.54 (20.51)14.64 ± 0.49 (15.26)
Random, spectrum-matchedSemantic/positional79.66 ± 0.85 (80.08)21.64 ± 0.41 (21.85)30.26 ± 0.74 (30.59)60.57 ± 0.62 (61.41)22.05 ± 0.27 (22.42)16.57 ± 0.26 (16.82)
Random, spectrum-matchedSink92.84 ± 1.30 (93.74)23.06 ± 0.93 (23.81)32.76 ± 0.86 (33.55)76.66 ± 0.60 (77.32)29.73 ± 0.39 (30.31)17.51 ± 0.26 (17.76)
Random, spectrum-matchedBoth79.18 ± 1.20 (80.03)21.40 ± 0.73 (21.89)29.93 ± 1.01 (30.59)57.16 ± 0.92 (58.26)21.15 ± 0.48 (21.85)15.70 ± 0.36 (16.20)
AuthorsOpenReview ↗

(2/2) Author Rebuttal

W2(Q2) Statistical Support for the Main Results

We added 95% paired bootstrap intervals to assess the reliability of the improvements over the strongest baseline for each model and benchmark (Table 2). For each replicate, we resample paired examples with replacement within each task and recompute the original benchmark aggregates and their difference.

On Infinite-Bench, FARoPE gains 0.98 and 1.17 points for Qwen3-8B and Llama 3 8B Instruct, respectively, representing relative improvements of about 4.0% and 3.9%. Both intervals lie above zero. LongBench v2 scores also improve on both models.

For context, LaMPE [1] and AdaGroPE [2] report LongBench gains of 1.09 and 0.99 points for Llama 3, respectively, over their best baselines. These reports use different evaluation settings but put the size of our gains in context.

Table 2. Confidence Intervals for the Main Results

Score differences are FARoPE minus the strongest baseline for each model and benchmark, in percentage points.

ModelDatasetScore Difference95% Paired Bootstrap CI
Qwen3-8BRULER Avg.+1.64[1.37, 1.92]
Qwen3-8BInfinite-Bench+0.98[0.31, 1.81]
Qwen3-8BLongBench v2+0.66[0.00, 1.64]
Llama 3 8B InstructRULER Avg.+3.59[3.18, 4.00]
Llama 3 8B InstructInfinite-Bench+1.17[0.37, 2.07]
Llama 3 8B InstructLongBench v2+0.93[0.00, 2.18]

[1] LaMPE: Length-aware Multi-grained Positional Encoding for Adaptive Long-context Scaling Without Training. Zhang et al., Findings of ACL'26.

[2] Extending LLM Context Window with Adaptive Grouped Positional Encoding: A Training-Free Method. Xu et al., ACL'25.

We have added the new controls and statistical analyses suggested by your review to the paper and submitted the revised version. These additions are highlighted in blue. We hope the new evidence addresses your concerns about functional classification and the reliability of the reported gains. Thank you again for your careful assessment and helpful suggestions.

Thank you for the detailed responses and additional experiments. These additional results have strengthened the empirical evidence and improved the clarity of the paper. I consider my concerns to be sufficiently addressed and will maintain my original rating.

Thank you for your careful review and for taking the time to consider our response and additional experiments. We are glad to hear that the new results addressed your concerns. We sincerely appreciate your constructive feedback, which helped us improve the paper.

Reviewer_z6ocOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes FaRoPE, a novel training-free positional interpolation scheme. This approach uses a calibration dataset to annotate each 2D RoPE band per head per layer as either a sink, positional or semantic band based off its role. It then proposes separate interpolation rules per role and attenuate YaRN's temperature scaling for the sink bands. They demonstrate this approach outperforms existing PI techniques on RULER, LongBenchv2, and Infinite-Bench. Since they are operating on 2D bands they can directly visualize their distributions and provide some interesting plots showing how these 3 roles differ.

Critical strengths
  • This is a practical and intuitive method which seems to deliver consistent gains without needing any extra fine-tuning or training.
  • The visualizations and analysis for the roles of the 2D RoPE bands are informative and differs from existing work.
Critical weaknesses
  • There is no direct ablation that uses the frequencies as a proxy for roles. Experiments directly showing this would strengthen the claim that per-band roles are important when considering interpolation.
  • The non-RULER eval gains are marginal and potentially within noise.
Questions for authors
  • How are the 3 roles distributed across band index? Some understanding of how close or different this is to YaRN's partitioning would more clearly show what this method is doing.
  • Is dynamic-NTK equivalent to the unmodified base model for context lengths less than or equal L0L_0? If so it seems like FaRoPE outperforms the base model within training length. Could the authors clear up this detail and provide an explanation as to why this is the case if so?
4 comments
AuthorsOpenReview ↗

(1/2) Author Rebuttal

Thank you for your thoughtful and constructive review. We appreciate your recognition of FARoPE's practical value for training-free context extension. Your questions have helped us explain the method and its results more clearly.

W1 Effectiveness of Functional Classification

We added the frequency-based alternative you suggested, together with random controls, to test the contribution of the functional labels under the same transformations (Table 1).

To isolate pair selection, we match class counts per layer and KV group. The spectrum-matched random control also matches counts within each YaRN region. The frequency-based control selects sink pairs from the lowest original frequencies and positional pairs from the low-frequency end of YaRN's transition region. We replace semantic/positional labels and sink labels separately or jointly, and refit sink calibration for each assignment.

FARoPE outperforms every alternative on both models and all three benchmarks, including the best seed of each random control. This shows that measured pair behavior provides useful information beyond frequency rank, class size, and class counts within YaRN regions.

Table 1. Frequency-Based and Random Alternatives

RULER Avg. averages six input lengths (4K–128K). Random controls report mean ± sample standard deviation over five seeds, with the best score in parentheses.

Grouping strategyReplacement scopeQwen3-8B RULER Avg.Qwen3-8B Infinite-BenchQwen3-8B LongBench v2Llama 3 8B Instruct RULER Avg.Llama 3 8B Instruct Infinite-BenchLlama 3 8B Instruct LongBench v2
FARoPEOriginal functional assignments95.2325.3435.2079.5431.2418.07
Frequency-basedSemantic/positional87.3822.9432.2461.4221.4716.82
Frequency-basedSink93.2924.2732.8977.9630.3817.45
Frequency-basedBoth84.5922.4631.9160.6220.1316.82
Random, size-matchedSemantic/positional78.65 ± 1.60 (79.97)21.20 ± 0.79 (21.79)29.47 ± 1.13 (30.59)58.43 ± 1.26 (59.81)21.43 ± 0.37 (21.91)15.70 ± 0.36 (16.20)
Random, size-matchedSink91.09 ± 1.30 (92.37)22.54 ± 0.91 (23.36)32.43 ± 1.08 (33.55)75.45 ± 0.58 (76.23)29.04 ± 0.41 (29.63)17.38 ± 0.26 (17.76)
Random, size-matchedBoth77.71 ± 1.75 (79.35)20.50 ± 1.03 (21.36)28.75 ± 1.18 (29.93)54.36 ± 1.22 (55.79)19.68 ± 0.54 (20.51)14.64 ± 0.49 (15.26)
Random, spectrum-matchedSemantic/positional79.66 ± 0.85 (80.08)21.64 ± 0.41 (21.85)30.26 ± 0.74 (30.59)60.57 ± 0.62 (61.41)22.05 ± 0.27 (22.42)16.57 ± 0.26 (16.82)
Random, spectrum-matchedSink92.84 ± 1.30 (93.74)23.06 ± 0.93 (23.81)32.76 ± 0.86 (33.55)76.66 ± 0.60 (77.32)29.73 ± 0.39 (30.31)17.51 ± 0.26 (17.76)
Random, spectrum-matchedBoth79.18 ± 1.20 (80.03)21.40 ± 0.73 (21.89)29.93 ± 1.01 (30.59)57.16 ± 0.92 (58.26)21.15 ± 0.48 (21.85)15.70 ± 0.36 (16.20)

W2 Statistical Support for the Main Results

To address your concern about the gains beyond RULER, we report 95% paired bootstrap intervals for comparisons with the strongest baseline for each model and benchmark (Table 2). For each replicate, we resample paired examples with replacement within each task and recompute the original benchmark aggregates and their difference.

On Infinite-Bench, FARoPE gains 0.98 and 1.17 points for Qwen3-8B and Llama 3 8B Instruct, respectively, representing relative improvements of about 4.0% and 3.9%. Both intervals lie above zero. LongBench v2 scores also improve on both models.

For context, LaMPE [1] and AdaGroPE [2] report LongBench gains of 1.09 and 0.99 points for Llama 3, respectively, over their best baselines. These reports use different evaluation settings but put the size of our gains in context.

Table 2. Confidence Intervals for the Main Results

Score differences are FARoPE minus the strongest baseline for each model and benchmark, in percentage points.

ModelDatasetScore Difference95% Paired Bootstrap CI
Qwen3-8BRULER Avg.+1.64[1.37, 1.92]
Qwen3-8BInfinite-Bench+0.98[0.31, 1.81]
Qwen3-8BLongBench v2+0.66[0.00, 1.64]
Llama 3 8B InstructRULER Avg.+3.59[3.18, 4.00]
Llama 3 8B InstructInfinite-Bench+1.17[0.37, 2.07]
Llama 3 8B InstructLongBench v2+0.93[0.00, 2.18]
AuthorsOpenReview ↗

(2/2) Author Rebuttal

Q1 Distribution Across Pair Indices and YaRN Regions

Following your suggestion, we added a distribution plot in response to your question about how the functional assignments relate to YaRN's frequency regions. It appears as Figure 4 in Section 4.3 ("Evaluating Function-Aware Pair Classification") of the revised paper. The plot shows Qwen3-8B layers 0, 17, and 35, with pair indices ordered from high to low frequency.

Semantic pairs dominate every displayed layer and span all three YaRN regions. Positional pairs are fewer and occur at separated indices across these regions rather than forming a continuous frequency band. The selected indices vary across layers and KV groups, although some are shared by several groups within a layer. Layer 35 contains more positional pairs than the other two examples.

Sink pairs are rare and confined to the low-frequency region in these plots. None appear in layer 0, while layers 17 and 35 each contain only a few. A single YaRN region can therefore contain different functional classes. The functional labels describe this variation within the fixed frequency regions.

Q2 Dynamic NTK and Performance Within the Native Window

Your interpretation is correct, within the native window, our dynamic NTK baseline uses original RoPE frequencies and unit gain, making it equivalent to the base model. FARoPE improves RULER at every reported native-window length, by 0.4–1.0 points for Qwen3-8B and by 5.0 and 15.1 points for Llama 3 8B Instruct at 4K and 8K, respectively.

These gains come from the frequency changes. FARoPE still uses frequencies set for the 128K target at native-window lengths, while both global and sink gains remain one. The common scale preserves frequency ratios among positional pairs, and the YaRN ramp reduces rotation in some semantic pairs. This explanation is consistent with native-window benefits reported by p-RoPE [3] and HoPE [4].

[1] LaMPE: Length-aware Multi-grained Positional Encoding for Adaptive Long-context Scaling Without Training. Zhang et al., Findings of ACL'26.

[2] Extending LLM Context Window with Adaptive Grouped Positional Encoding: A Training-Free Method. Xu et al., ACL'25.

[3] Round and Round We Go! What makes Rotary Positional Encodings useful? Barbero et al., ICLR'25.

[4] HoPE: A Novel Positional Encoding Without Long-Term Decay for Enhanced Context Awareness and Extrapolation. Chen et al., ACL'25.

Following your comments, we have updated the paper with the additional analyses and explanations and submitted the revised version. The new content is marked in blue. We hope these revisions address your concerns and make the method and its results clearer. We sincerely appreciate your thoughtful feedback and the time you devoted to reviewing our work.

Thanks for providing the extra experiments. The distribution of roles is indeed an interesting addition. With the inclusion of that as well as the confidence intervals for the downstream metrics I will raise my soundness from a 2 to a 3. Given the smaller gains on the non-synthetic benchmarks such as LongBenchv2 I maintain my recommendation of 3 rather than 4. Nevertheless I did enjoy the analysis of the frequency roles and do think it's an intuitive grouping.

Thank you very much for reviewing our additional experiments and revisiting your soundness assessment. Your constructive suggestions helped us make a stronger empirical case for our conclusions, and we sincerely appreciate your guidance. Thank you again for the care and attention you devoted to reviewing our work.

Reviewer_ioD9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FARoPE extends the context of a pretrained RoPE model without updating its weights. It classifies query/key dimension pairs using pre-RoPE phase concentration and their contribution to preference for the first token. Positional and sink pairs receive uniform frequency scaling; the remaining pairs retain the YaRN ramp. Sink-pair logit gain is calibrated separately. The contribution is this joint selection and calibration rule. Evaluation extends Qwen3-8B and Llama 3 8B Instruct to 128K and covers RULER, Infinite-Bench, LongBench v2, module ablations, and calibration-corpus sensitivity.

Critical strengths
  1. Meaningful improvements on controlled long-context tasks. At 128K, FARoPE scores 85.5 and 63.4 on RULER for Qwen and Llama, versus 83.2 and 58.2 for the best reported baselines at that length. These results support the method's usefulness at both 4× and 16× context extension. Ablations support both transformations in the full configuration.

  2. A well-defined calibration procedure with evidence of corpus stability. The calibration fit has a closed-form solution, and Appendix F expresses the intervention through standard dot-product attention. Small numerical checks of the supplied components agree with the main formulas. Switching from PG-19 to C4 while controlling for packing preserves most labels and gives similar scores, supporting stability across the tested calibration sources.

Critical weaknesses
  1. The benefit of functional pair selection is not isolated. All ablations keep the same labels and change only the applied transformations (Appendix C.1, Tables 2–3). Yet the central hypothesis is that measured behavior provides useful information beyond frequency rank. Positional and sink pairs receive the same frequency rule, which changes nothing where YaRN already uses ωp/f\omega_p/f. The results also fit a simpler explanation: stronger interpolation of selected channels and reduced logit amplification help, while the advantage of the specific classification rule remains unclear. Comparing the same interventions under comparable masks selected by functional statistics and a simpler criterion would directly test the main new principle.

  2. The evidence for comparative advantage remains incomplete. Section 4.1 and Appendices D.1–D.2 do not fully specify baseline grouping/chunking, frequency-ramp and temperature settings, or the standalone dynamic NTK policy. The ProofPile threshold search lacks its grid, exact objective, and comparable baseline tuning budgets. The supplementary material provides FARoPE components and defaults, but no model adapters, baseline configurations, or search and evaluation scripts. These choices matter: changing from standalone YaRN to “w/o both modules” changes Llama's RULER score from 60.7 to 62.2 and Infinite-Bench from 20.7 to 22.7. C.1 identifies the different gain policies, but complete experimental configurations are needed to assess the size of the advantage.

Evidence beyond RULER is also limited. The LongBench v2 gains correspond to only two additional correct answers for Qwen (107 versus 105 out of 304) and three for Llama (58 versus 55 out of 321). Even the most favorable paired error counts consistent with these totals give exact two-sided McNemar pp-values of 0.50.5 and 0.250.25. Infinite-Bench gains of 0.9/1.1 points lack paired uncertainty estimates for the original macro-average; D.3 addresses calibration-corpus changes instead. Llama remains below the nominal 25% accuracy of uniform random choice. A broader advantage on realistic understanding and reasoning tasks is therefore not established.

Questions for authors
  1. Validation of pair selection: Do any existing experiments compare functional masks with masks selected by frequency rank or another simple criterion, while matching mask size and spectral composition and keeping the transformations fixed? This would separate the value of the selection rule from the value of the transformations.

  2. Experimental configuration: What exact baseline settings and ProofPile search grid, objective, and tuning budgets produced Tables 1–3? Which model integration was used for the reported runs, given that the supplied runtime provides the diagnostic pair-logit path rather than the query-only path in Appendix F? Are prefill/decoding overhead measurements available for that implementation?

  3. Uncertainty of realistic-benchmark gains: What do paired per-example results show for FARoPE versus the strongest baselines on Infinite-Bench and LongBench v2, with uncertainty computed under each benchmark's original aggregation?

5 comments
AuthorsOpenReview ↗

(1/3) Author Rebuttal

Thank you for your careful reading and constructive review. We particularly appreciate your detailed examination of the calibration procedure. Your feedback has helped us present the experimental evidence more clearly.

W1(Q1) Effectiveness of Functional Classification

Following your suggestion, we added frequency-based and random assignments to test whether the functional labels improve performance under the same frequency and gain rules (Table 1).

To isolate pair selection, we match class counts within each layer and KV group. The frequency-based control selects sink pairs from the lowest original frequencies and positional pairs from the low-frequency end of YaRN's transition region. The random controls match class sizes, with the spectrum-matched version also preserving counts within each of YaRN's three regions.

We replace semantic/positional labels, sink labels, or both, keeping the other original set in single-part replacements. Every variant uses the same transformations, with sink calibration refitted for each assignment.

As you noted, YaRN already uses the same scaled frequencies in the fully interpolated region. Sink labels still determine which contributions receive calibrated gain. The spectrum-matched control therefore tests whether functional labels add value within the same frequency regions.

FARoPE outperforms every control on both models across all three benchmarks, including the best seed of each random control. These results support the value of measured pair behavior beyond class counts and their distribution across YaRN regions.

Table 1. Pair-Selection Controls

RULER Avg. averages six input lengths (4K–128K). Random controls report mean ± sample standard deviation over five seeds, with the best score in parentheses.

Grouping strategyReplacement scopeQwen3-8B RULER Avg.Qwen3-8B Infinite-BenchQwen3-8B LongBench v2Llama 3 8B Instruct RULER Avg.Llama 3 8B Instruct Infinite-BenchLlama 3 8B Instruct LongBench v2
FARoPEOriginal functional assignments95.2325.3435.2079.5431.2418.07
Frequency-basedSemantic/positional87.3822.9432.2461.4221.4716.82
Frequency-basedSink93.2924.2732.8977.9630.3817.45
Frequency-basedBoth84.5922.4631.9160.6220.1316.82
Random, size-matchedSemantic/positional78.65 ± 1.60 (79.97)21.20 ± 0.79 (21.79)29.47 ± 1.13 (30.59)58.43 ± 1.26 (59.81)21.43 ± 0.37 (21.91)15.70 ± 0.36 (16.20)
Random, size-matchedSink91.09 ± 1.30 (92.37)22.54 ± 0.91 (23.36)32.43 ± 1.08 (33.55)75.45 ± 0.58 (76.23)29.04 ± 0.41 (29.63)17.38 ± 0.26 (17.76)
Random, size-matchedBoth77.71 ± 1.75 (79.35)20.50 ± 1.03 (21.36)28.75 ± 1.18 (29.93)54.36 ± 1.22 (55.79)19.68 ± 0.54 (20.51)14.64 ± 0.49 (15.26)
Random, spectrum-matchedSemantic/positional79.66 ± 0.85 (80.08)21.64 ± 0.41 (21.85)30.26 ± 0.74 (30.59)60.57 ± 0.62 (61.41)22.05 ± 0.27 (22.42)16.57 ± 0.26 (16.82)
Random, spectrum-matchedSink92.84 ± 1.30 (93.74)23.06 ± 0.93 (23.81)32.76 ± 0.86 (33.55)76.66 ± 0.60 (77.32)29.73 ± 0.39 (30.31)17.51 ± 0.26 (17.76)
Random, spectrum-matchedBoth79.18 ± 1.20 (80.03)21.40 ± 0.73 (21.89)29.93 ± 1.01 (30.59)57.16 ± 0.92 (58.26)21.15 ± 0.48 (21.85)15.70 ± 0.36 (16.20)
AuthorsOpenReview ↗

(2/3) Author Rebuttal

W2(Q2) Experimental Settings and Inference Efficiency

To clarify the settings you asked about, we provide the baseline implementation details for the reported comparisons below.

Both models target 131,072 tokens and use rotary dimension 128, bfloat16 inference, and greedy decoding. Qwen3-8B uses a 32,768-token native window, extension factor 4, and RoPE base 1,000,000. Llama 3 8B Instruct uses an 8,192-token native window, factor 16, and base 500,000. YaRN and MrROPE use the logit gains below, while other baselines use unit gain. Attention temperature is the reciprocal of logit gain.

Dynamic NTK. Frequencies are computed from the tokenized prompt length before prefill, held fixed through cached decoding, and reset for each request. Prompts within the native window use original RoPE frequencies. YaRN. The linear ramp uses fast and slow beta parameters of 32 and 1, respectively. We round the fast boundary down and the slow boundary up. Zero-based pair-index endpoints are (23, 40) for Qwen3-8B and (18, 35) for Llama 3 8B Instruct. The target factor fixes the schedule and logit gain, with gains of 1.296 and 1.631, respectively. MrROPE. We use the PRO variant with the same beta parameters, band boundaries, rounding convention, and fixed logit gains as YaRN. SelfExtend. Both models use group size 32 and neighbor window 1024, with additional query scaling disabled. LaMPE. We use LaMPE's multi-branch attention implementation. Mapping parameters are determined from the prompt length and kept fixed throughout each request. Qwen3-8B uses head/tail lengths 2048/200 and generic mapping length 24576. Llama 3 8B Instruct uses head/tail lengths 512/200 and the Llama 3 mapping profile. DCA. Chunk size, local window, and inter-query position are (24576, 8192, 32767) for Qwen3-8B and (6144, 2048, 8191) for Llama 3 8B Instruct. AdaGroPE. We use adaptive relative-position reuse with ratio 0.25. Position budgets are 16384 for Qwen3-8B and 4096 for Llama 3 8B Instruct, with neighbor windows 4096 and 1024, respectively.

The score difference you highlighted comes from the attention gain, even though both configurations use the same YaRN frequencies. Standalone YaRN uses a fixed gain determined by the target context length. The w/o both modules ablation retains FARoPE's gain, which equals one within the native window and increases with the actual input length. This reduces extra logit amplification on shorter inputs. Full FARoPE uses the same global gain as this ablation, so their comparison measures the benefit of function-aware scaling and sink calibration.

We use ProofPile to select FARoPE's parameters. We test phase concentration thresholds of 0.80, 0.90, 0.92, 0.94, 0.95, 0.96, 0.97, and 0.98, MAD multipliers of 2, 3, and 4, and sink floors of 0.10, 0.20, and 0.30. Each configuration uses 40 sequences per model at each of 4K, 8K, 16K, 32K, 64K, and 128K (240 per model). We minimize perplexity averaged equally across lengths and then equally across models. The selected threshold, MAD multiplier, and sink floor are 0.96, 3, and 0.10, respectively. Both models use these values for all downstream benchmarks. Baselines use their papers' recommended settings without additional search.

For the reported runs, both models use the query-only implementation in Appendix F. It is algebraically equivalent to the diagnostic pair-logit calculation and supports standard attention kernels without materializing pair-wise attention matrices. Table 2 reports combined prefill and decoding runtime. FARoPE is comparable to YaRN overall and faster on Qwen3-8B for Infinite-Bench and LongBench v2.

Table 2. Average Runtime per Example

Seconds per example, combining prefill and decoding.

MethodQwen3-8B RULERQwen3-8B Infinite-BenchQwen3-8B LongBench v2Llama 3 8B Instruct RULERLlama 3 8B Instruct Infinite-BenchLlama 3 8B Instruct LongBench v2
Dynamic NTK10.86068.10014.9818.32953.04212.201
YaRN12.358106.52417.93610.18683.54414.036
MrROPE12.50170.64415.12810.32082.96314.099
SelfExtend35.841176.30323.90710.46574.32114.881
LaMPE13.658143.92717.70812.462131.80616.184
DCA11.624103.58216.32713.216112.74317.492
AdaGroPE76.594483.294106.03665.821376.20194.094
FARoPE12.44992.18316.36711.24184.50614.091
AuthorsOpenReview ↗

(3/3) Author Rebuttal

W3(Q3) Statistical Support for the Main Results

We assess the reliability of the gains using 95% paired bootstrap intervals for comparisons with the strongest baseline for each model and benchmark (Table 3). For each replicate, we resample paired examples with replacement within each task and recompute the original benchmark aggregates and their difference.

On Infinite-Bench, FARoPE gains 0.98 and 1.17 points for Qwen3-8B and Llama 3 8B Instruct, respectively, representing relative improvements of about 4.0% and 3.9%. Both intervals lie above zero. LongBench v2 scores also improve on both models.

For context, LaMPE [1] and AdaGroPE [2] report LongBench gains of 1.09 and 0.99 points for Llama 3, respectively, over their best baselines. These reports use different evaluation settings but put the size of our gains in context.

Table 3. Confidence Intervals for the Main Results

Score differences are FARoPE minus the strongest baseline for each model and benchmark, in percentage points.

ModelBenchmarkScore difference95% paired bootstrap CI
Qwen3-8BRULER Avg.+1.64[1.37, 1.92]
Qwen3-8BInfinite-Bench+0.98[0.31, 1.81]
Qwen3-8BLongBench v2+0.66[0.00, 1.64]
Llama 3 8B InstructRULER Avg.+3.59[3.18, 4.00]
Llama 3 8B InstructInfinite-Bench+1.17[0.37, 2.07]
Llama 3 8B InstructLongBench v2+0.93[0.00, 2.18]

[1] LaMPE: Length-aware Multi-grained Positional Encoding for Adaptive Long-context Scaling Without Training. Zhang et al., Findings of ACL'26.

[2] Extending LLM Context Window with Adaptive Grouped Positional Encoding: A Training-Free Method. Xu et al., ACL'25.

We have incorporated the additional experiments and clarifications prompted by your suggestions into the paper and submitted the revised version. The new material is shown in blue. We hope these additions address your concerns and clarify the experimental evidence. Thank you again for your careful review and constructive suggestions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chair and Program Chairs,

We sincerely thank you for the time and effort you’ve put into the review process. We appreciate the care you’ve taken in overseeing our submission and supporting a constructive discussion.

We are grateful to all three reviewers for carefully evaluating the paper and engaging thoughtfully with our responses.

Below is a brief summary of our rebuttal and the reviewers’ follow-up.

Acknowledged Strengths

FARoPE extends pretrained language models to 128K without fine-tuning. It uses the measured behavior of individual RoPE dimension pairs to guide frequency scaling and calibrates the logit gains for sink pairs separately.

  1. Practical context extension with strong results at 128K. Reviewer z6oc highlighted FARoPE’s intuitive design and its gains without additional training. Reviewers ioD9 and uPmt noted the RULER gains over the strongest reported baseline on both Qwen3-8B and Llama 3 8B Instruct.

  2. Informative analysis and stable calibration. Reviewer z6oc highlighted the visualizations of positional, semantic, and sink roles. Reviewer uPmt recognized the value of separate sink-logit calibration and, along with reviewer ioD9, noted its stability across PG-19 and C4.

Reviewer Concerns and Our Responses

  1. Does functional pair selection improve performance? All three reviewers asked for a direct comparison with simpler assignments. Reviewer z6oc also asked how functional roles are distributed across frequencies. In Qwen3-8B, we found that pairs with different functional roles can fall within the same YaRN frequency region. On both models, FARoPE outperformed all tested frequency-based and random controls on RULER, Infinite-Bench, and LongBench v2.

  2. How reliable are the benchmark gains? We used paired bootstrap resampling to estimate 95% confidence intervals for FARoPE’s gains over the strongest baselines. For both models, the estimated gains were positive across all three benchmarks, and the intervals for length-averaged RULER and Infinite-Bench lay entirely above zero.

Reviewer Feedback After Rebuttal

Reviewer ioD9 raised the overall rating from 2 to 3 after finding that the new pair-selection controls substantially addressed the main concern. After reviewing the role analysis and confidence intervals, reviewer z6oc raised the soundness score from 2 to 3 and kept the overall rating at 3. Reviewer uPmt found the concerns sufficiently addressed and kept the original overall rating, noting that the additional results strengthened the empirical evidence. All three posted overall ratings are 3.

We hope the above summary helps streamline your decision process. Thank you again for guiding our submission through this review cycle.

Warm regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.