- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
FARoPE explores a function-aware approach to training-free context extension for RoPE-based language models. Instead of relying solely on frequency order, it uses phase concentration and sink bias to classify RoPE dimension pairs into positional, semantic, and sink components. Different frequency scaling rules are applied to these components, together with a separate attention-logit calibration for sink pairs. The method is evaluated on Qwen3-8B and Llama 3 8B Instruct across RULER, Infinite-Bench, and LongBench v2, with additional ablations and calibration studies.
Critical strengths
At 128K context length, FARoPE improves the RULER score from 83.2 to 85.5 on Qwen3-8B and from 58.2 to 63.4 on Llama 3 8B compared with the strongest baselines. Appendix C examines the contributions of the two main modules, and the calibration results remain similar when switching between PG-19 and C4. These experiments give some support to the effectiveness and stability of the proposed method.
Instead of assigning RoPE scaling rules based only on frequency, FARoPE distinguishes dimension pairs according to their observed behavior. Positional and sink pairs use a shared frequency reduction, while semantic pairs follow the YaRN scaling rule. The separate treatment of sink-related logits is also useful, as it limits additional sink amplification without modifying the pretrained model weights.
Critical weaknesses
One question left open by the ablation studies is how important the proposed functional classification actually is. In Appendix C, the pair assignments remain fixed while the frequency scaling and logit calibration modules are removed separately. It is therefore difficult to separate the effect of the classification from that of the scaling rules. For example, would grouping dimensions by frequency or randomly assigning pairs to groups of similar sizes give comparable results under the same transformations? Without such comparisons, the benefit of identifying functional roles remains unclear.
The results on Infinite-Bench and LongBench v2 are less convincing than those on RULER. The improvements over the strongest baselines are mostly around one percentage point, and no confidence intervals are provided for these main comparisons. The evaluation also covers only two 8B models. In particular, Llama 3 achieves 18.1 on LongBench v2, below the nominal random-choice reference of 25%. While the controlled long-context results are encouraging, I am less certain how much the method improves more challenging long-context understanding and reasoning.
Questions for authors
- Have the authors tried alternative ways of grouping RoPE dimension pairs, such as frequency-based partitioning or random assignments with matched group sizes? Applying the same scaling rules to these groups would help isolate the contribution of the functional classification.
- Could the authors report confidence intervals for the main Infinite-Bench and LongBench v2 comparisons? Given the small differences between methods, it would be useful to know whether the improvements are statistically reliable.
4 comments
(2/2) Author Rebuttal
W2(Q2) Statistical Support for the Main Results
We added 95% paired bootstrap intervals to assess the reliability of the improvements over the strongest baseline for each model and benchmark (Table 2). For each replicate, we resample paired examples with replacement within each task and recompute the original benchmark aggregates and their difference.
On Infinite-Bench, FARoPE gains 0.98 and 1.17 points for Qwen3-8B and Llama 3 8B Instruct, respectively, representing relative improvements of about 4.0% and 3.9%. Both intervals lie above zero. LongBench v2 scores also improve on both models.
For context, LaMPE [1] and AdaGroPE [2] report LongBench gains of 1.09 and 0.99 points for Llama 3, respectively, over their best baselines. These reports use different evaluation settings but put the size of our gains in context.
Table 2. Confidence Intervals for the Main Results
Score differences are FARoPE minus the strongest baseline for each model and benchmark, in percentage points.
| Model | Dataset | Score Difference | 95% Paired Bootstrap CI |
|---|---|---|---|
| Qwen3-8B | RULER Avg. | +1.64 | [1.37, 1.92] |
| Qwen3-8B | Infinite-Bench | +0.98 | [0.31, 1.81] |
| Qwen3-8B | LongBench v2 | +0.66 | [0.00, 1.64] |
| Llama 3 8B Instruct | RULER Avg. | +3.59 | [3.18, 4.00] |
| Llama 3 8B Instruct | Infinite-Bench | +1.17 | [0.37, 2.07] |
| Llama 3 8B Instruct | LongBench v2 | +0.93 | [0.00, 2.18] |
[1] LaMPE: Length-aware Multi-grained Positional Encoding for Adaptive Long-context Scaling Without Training. Zhang et al., Findings of ACL'26.
[2] Extending LLM Context Window with Adaptive Grouped Positional Encoding: A Training-Free Method. Xu et al., ACL'25.
We have added the new controls and statistical analyses suggested by your review to the paper and submitted the revised version. These additions are highlighted in blue. We hope the new evidence addresses your concerns about functional classification and the reliability of the reported gains. Thank you again for your careful assessment and helpful suggestions.
Thank you for the detailed responses and additional experiments. These additional results have strengthened the empirical evidence and improved the clarity of the paper. I consider my concerns to be sufficiently addressed and will maintain my original rating.
Thank you for your careful review and for taking the time to consider our response and additional experiments. We are glad to hear that the new results addressed your concerns. We sincerely appreciate your constructive feedback, which helped us improve the paper.
(1/2) Author Rebuttal
Thank you for your careful and balanced review. We appreciate your recognition of the benefit of selecting RoPE pairs by their observed behavior. Your suggestions have helped us demonstrate this contribution more directly.
W1(Q1) Effectiveness of Functional Classification
Following your suggestion, we added frequency-based and random assignments to separate the effect of classification from that of the scaling rules, while keeping FARoPE's transformations unchanged (Table 1).
To isolate pair selection, we match class counts in every layer and KV group. Spectrum-matched random grouping also preserves these counts within each YaRN region. We replace semantic/positional assignments, sink assignments, or both, and refit sink calibration for each alternative.
FARoPE exceeds every control on both models across all three benchmarks, including each random control's best seed. Replacing either set of assignments reduces performance, with larger drops for semantic/positional labels. These results support the contribution of the functional classification under the same transformations.
Table 1. Functional Classification Versus Alternative Groupings
RULER Avg. averages six input lengths (4K–128K). Random controls report mean ± sample standard deviation over five seeds, with the best score in parentheses.