acceptodds
Under review as a conference paper at ICLR 2027

RoPE Knows Its Limits: Frequency Bands Show Effective Context Length in LLMs

Abstract

Recent large language models claim to support extremely long contexts. However, as demonstrated by the RULER benchmark, the maximum sequence length a model claims to accept can diverge substantially from its effective context length, namely the length over which the model can actually process contextual information. This observation raises a natural question: is effective context length merely an empirical property that can only be measured by running long-context benchmarks, or can it be predicted from a model’s internal structure? In this work, we focus on the frequency bands induced by Rotary Position Embeddings (RoPE), which are widely used in modern LLMs. Through our analysis, we find that the trends in effective context length observed in RULER are proportional to the base frequencies of the dimensions in which these frequency bands appear. This suggests that the context length a RoPE-based LLM can truly exploit may be determined by the scale of the frequency bands it actually uses. In hybrid attention architectures that combine Full Attention and Sliding Window Attention, the frequency bands differ depending on the attention type, and these bands indicate the effective context length available to each layer. Our findings recast effective context length as a model-specific spectral quantity, enabling its dataset-independent estimation in RoPE-based LLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Rotary Position Embedding, LLM, RULER

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_hv9oOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces RoPE-RULER, a diagnostic method that estimates effective context length from a model’s internal RoPE representations. Given an unlabeled text input, the method computes the average query-vector norm for each two-dimensional RoPE component in every layer and attention head. It selects the component with the largest norm for each head, then takes the most frequently selected component across layers and heads as the representative frequency. The positional distance corresponding to a quarter rotation of this component is used as the estimated context length.

The authors compare these estimates with RULER-measured effective context lengths across Mistral, Llama, Qwen, and Phi models. They also examine the stability of frequency selection across input corpora and lengths for Llama-3.1-8B. An additional analysis of GPT-OSS-120B identifies different frequency scales for full-attention and sliding-window layers, with the latter producing an estimate close to the configured window size. The method is positioned as a lightweight internal diagnostic that requires neither training nor supervised labels.

Critical strengths
  1. The problem is practically relevant, and the proposed method is simple. Estimating a model’s context scale from a short forward pass could reduce the cost of preliminary long-context analysis. The method requires no training or supervised labels, and its computational steps are clearly specified. It has potential as a model diagnostic.

  2. The observations about frequency stability and differences between attention types are informative. Llama-3.1-8B exhibits a relatively stable representative frequency across different corpora and input lengths. GPT-OSS-120B’s full-attention and sliding-window layers exhibit different frequency scales, with the sliding-window estimate of approximately 135 closely matching the configured window size of 128. These observations provide useful evidence for studying the relationship between RoPE representations and the distance scales available to a model.

Critical weaknesses
  1. The central quantitative agreement needs clarification. The reported Qwen3-8B positional scale matches the unscaled RoPE value despite the stated 4× YaRN configuration. Additionally, the 61,641-token predictions for Qwen3-4B/8B are marked at 32K, although their nearest bucket is 64K. These discrepancies affect the claimed agreement with RULER and require reconciliation, together with reporting prediction errors before bucket mapping.

  2. The frequency-to-length conversion lacks sufficient validation. Crossing 90 degrees does not itself create positional ambiguity, and selecting the most common high-norm component does not establish a model’s behavioral context limit. The rule could still be useful as a heuristic, but stronger comparisons with simple baselines and validation on independent models are needed. For GPT-OSS, proximity to the advertised context length does not provide independent behavioral validation.

Questions for authors
  1. Frequency-scaling configuration: Which RoPE configurations are used for Qwen3-8B during diagnosis and RULER evaluation? Why is the effective positional scale in Table 2 almost identical to the unscaled value? Please provide the actual inv_freq value for the representative component, together with the scaling parameters and implementation version.

  2. Bucket mapping and prediction error: Does Table 1 use nearest-bucket mapping, rounding down, or another rule? How should the marker positions for Qwen3-4B/8B and Phi-3-mini be interpreted? After applying a consistent rule, what are the continuous prediction errors and bucket-matching results for each model?

  3. Validation of the conversion rule: Were the quarter-period coefficient and the mode-based aggregation rule fixed before examining the RULER results? Are there results on independent models showing that these choices outperform other simple coefficients, aggregation rules, or context-configuration baselines?

  4. Prediction target for GPT-OSS: When the model does not meet the RULER threshold, how is its effective context length independently defined and validated? Beyond proximity to the advertised context length or configured sliding-window size, what behavioral evidence supports the predictive meaning of the approximately 182K estimate?

5 comments
AuthorsOpenReview ↗

Author Response (1/3)

Thank you for taking the time to review our paper. We appreciate your recognition of the practical importance of estimating effective context length, the simplicity of RoPE-RULER, and its potential as a diagnostic tool for language models.

We respond to your comments and questions below. We have also revised the paper in light of your feedback. All changes are shown in red in the manuscript itself.

W1. Mismatch between the YaRN settings used for RULER evaluation and RoPE-RULER prediction in Qwen models

Thank you for raising this point.

(Reviewer comment) Additionally, the 61,641-token predictions for Qwen3-4B/8B are marked at 32K, although their nearest bucket is 64K.

We map each continuous RoPE-RULER estimate to the largest evaluated length at or below it. For bucket assignment, we treat the estimate as an upper limit on the context length predicted to be usable. Rounding to the nearest evaluated length could therefore classify a length above that estimate as usable, which is why we adopted the more conservative rule. The word “nearest” in the Table 1 caption misrepresents the rule we used. We will revise the caption to read as follows.

The red downward triangle marks the RULER-measured effective length, while the green downward triangle marks the largest evaluated length not exceeding the continuous RoPE-RULER estimate, as defined in Section 4.1.

We have also corrected the corresponding wording on line 366 and in Table 3.


(Revieweer comment) The reported Qwen3-8B positional scale matches the unscaled RoPE value despite the stated 4× YaRN configuration.

We apologize for the confusion caused by our incomplete description of the settings used for Qwen3-4B and Qwen3-8B. For Qwen3 models, the standard practice is to use default, unscaled RoPE for inputs within the native context window of 32K tokens [1]. Accordingly, in our RULER evaluations, we used unscaled RoPE at evaluated lengths up to and including 32K tokens and YaRN with a scaling factor of s=4s=4 at evaluated lengths of 64K tokens and above. The following statement already appeared in the appendix, and we have now added it to Section 4.1 of the main text.

In our long-context setting, we use a scaling factor of 4 from an original 32,768-token range, yielding a target context length of 131,072 tokens.

RoPE-RULER uses each pretrained checkpoint’s default RoPE configuration as the reference for identifying its learned frequency band. Users can change the YaRN scaling factor at inference time. Substituting the YaRN-adjusted frequencies into the diagnostic could therefore change the estimated length solely because a different scaling factor was chosen, even if the learned band itself remained the same. We omit this inference-time transformation from the diagnostic so that we measure the bands formed during training against a consistent reference.

For the behavioral RULER evaluation, we applied the same length-dependent protocol to all Qwen3 models because it follows the recommended Qwen3 inference settings. We make no claim about how the effective context length varies with the YaRN scaling factor or whether the same effective length would be obtained without YaRN. We have added this clarification to Section 3.2 (L207–210) of the main text.

AuthorsOpenReview ↗

Author Response (2/3)

W2. Insufficient validation of the rule for converting frequency to context length

Thank you for raising this point. We do not claim that the frequency chunk that most often has the largest query norm is theoretically guaranteed to determine a model’s effective context length.

Building on the relationship between query norms and attention scores established in prior work [2, 3], we select a representative frequency and derive a distance scale from it. Whether this distance scale predicts a model’s behaviorally measured effective context length is precisely the question our experiments address.

We compared RoPE-RULER estimates with RULER measurements across four model families (Mistral, Llama, Qwen, and Phi). Our empirical contribution is to show that the distance scale derived from the selected chunk predicts the RULER-measured length bucket for all seven models evaluated.


(Reviewer Comment) This rule may be useful as a heuristic. However, demonstrating its usefulness requires stronger comparisons with simple baselines and validation on independent models.

Thank you for raising this point. RoPE-RULER selects a representative frequency chunk and estimates context length from the distance at which its relative rotation reaches π/2\pi/2. We compared these estimates with the effective context lengths measured by RULER across four model families (Mistral, Llama, Qwen, and Phi). We also found that frequency selection was stable across the input corpora and sequence lengths we tested.

Our goal is to estimate the effective context length measured by RULER, so we use RULER as the behavioral reference for evaluating our predictions.

Could you clarify which simple baseline estimator you have in mind?


(Reviewer Comment) For GPT-OSS, an estimate close to the nominal context length does not constitute independent behavioral validation.

The GPT-OSS analysis examines whether RoPE-RULER can distinguish the positional scales associated with different attention mechanisms within the same model. We observed distinct frequency bands in the full-attention and sliding-window layers. For the sliding-window layers, the estimated scale was 135 tokens, close to the configured window size of 128 tokens. This finding suggests that the diagnostic captures differences in positional scale across attention mechanisms. We make no claim that this experiment behaviorally validates the effective context length of GPT-OSS.


Q1. Frequency-scaling configuration: inv_freq settings for the Qwen3 models

Our response to W1 explains when we use standard RoPE and when we apply YaRN. Here we provide the numerical details for Qwen3-8B. Its RoPE base is θ=106\theta=10^6, its head dimension is d=128d=128, and its representative frequency-band index is iband=49i_{\mathrm{band}}=49.

Our experiments used Hugging Face Transformers v4.57.6. For YaRN, we used a scaling factor of s=4.0s=4.0, with βfast=32\beta_{\mathrm{fast}}=32 and βslow=1\beta_{\mathrm{slow}}=1, extending the original context length of 32,768 tokens to a target of 131,072 tokens. The following values were calculated from these parameters.

Quantity at i=49i=49Without YaRNWith YaRN (s=4s=4)
inv_freq[49] (radians per token)2.548297×10−52.548297 \times 10^{-5}6.370742×10−66.370742 \times 10^{-6}
Distance corresponding to one radian, 1/ω491/\omega_{49} (tokens)39,241.9156,967.6

For the RoPE-RULER estimate reported in Table 1, we use the checkpoint’s unscaled frequency. Applying the π/2\pi/2 conversion gives (π/2)/ω49≈61,641(\pi/2)/\omega_{49} \approx 61{,}641 tokens for Qwen3-8B.


Q2. Bucket mapping and prediction error: How should the marker positions for Qwen3-4B, Qwen3-8B, and Phi-3-mini in Table 1 be interpreted?

For all three models, both markers correspond to 32K. The red triangle marks the effective context length reported by RULER. The green triangle marks the largest evaluated length at or below the continuous RoPE-RULER estimate. The estimates are 61,641 tokens for both Qwen3 models and 52,675 tokens for Phi-3-mini. Each falls between 32K and 64K, so the green marker is placed at 32K.

For bucket assignment, we treat the estimate as the upper end of the predicted usable context range. Rounding to the nearest evaluated length would place all three green markers at 64K, labeling a length above each estimate as usable. The word “nearest” in the Table 1 caption therefore misdescribed our mapping rule. We explain the correction in our response to W1.

Because RULER evaluates performance only at discrete sequence lengths, it does not pinpoint where performance crosses the threshold between two tested lengths. We can therefore assess agreement at the level of length buckets, but an exact prediction error in tokens cannot be determined from these measurements.

AuthorsOpenReview ↗

Author Response (3/3)

Q3. Validation of the conversion rule

We fixed the phase coefficient at π/2\pi/2, corresponding to one quarter of a rotation, before examining the RULER results. We chose this value on theoretical grounds and did not tune it to the benchmark. As shown in our response to W2, π/2\pi/2 is the first rotation angle at which perturbing the key along the pre-RoPE query direction ceases to increase that chunk’s contribution to the attention score.

Our use of query norms is motivated by the evidence in Section 3 of Oka et al. that high-norm frequency bands contribute substantially to attention scores. We discuss this finding in the “Frequency bands in RoPE” paragraph of Section 2. Our aggregation rule takes the frequency chunk that most often has the largest query norm as the representative band.

To test whether a different coefficient would yield better agreement with RULER, we varied the phase coefficient while keeping the frequency-band selection, aggregation, and bucket-mapping rules fixed.

ModelRULER00π/4\pi/4π/2\pi/23π/43\pi/4π\pi3π/23\pi/22π2\pi
Mistral-7B16K—8K ✗16K ✓16K ✓32K ✗32K ✗64K ✗
Llama-3.1-8B32K—16K ✗32K ✓32K ✓64K ✗64K ✗128K ✗
Qwen3-4B32K—16K ✗32K ✓64K ✗64K ✗128K ✗128K ✗
Qwen3-8B32K—16K ✗32K ✓64K ✗64K ✗128K ✗128K ✗
Qwen3-14B64K—32K ✗64K ✓64K ✓128K ✗128K ✗128K ✗
Qwen3-32B128K—32K ✗64K ✗64K ✗128K ✓128K ✓128K ✓
Phi-3-mini32K—16K ✗32K ✓64K ✗64K ✗128K ✗128K ✗
Matches with RULER0/70/76/73/71/71/71/7

A check mark (✓) indicates an exact match with the RULER length bucket. A dash (—) indicates that the zero coefficient yields an estimate below the shortest evaluated length.

Among the coefficients tested, the prespecified π/2\pi/2 was the only one to match the RULER bucket for six models. The next-best coefficient, 3π/43\pi/4, matched three of seven. We will include these results in Appendix E. Thank you for suggesting this comparison, which makes the sensitivity of our predictions to the choice of coefficient explicit.


Q4. Prediction target for GPT-OSS

Thank you for raising this important question about what RoPE-RULER predicts for GPT-OSS. We define effective context length as the longest evaluated sequence length at which a model meets the RULER performance threshold. However, GPT-OSS fell below this threshold at every evaluated length, so its effective context length could not be determined under this criterion. Our experiments therefore provide no independent behavioral evidence that approximately 182K tokens is the effective context length of GPT-OSS.

Our complementary analysis examines whether RoPE-RULER can distinguish the positional scales of full-attention (FA) and sliding-window attention (SWA) layers within the same model. The estimated scales were 181,899 tokens for FA layers and 135 tokens for SWA layers. The proximity of the SWA estimate to the configured window of 128 tokens is consistent with the hypothesis that a layer’s dominant RoPE frequency band reflects its characteristic distance scale.

To clarify what this structural finding establishes and how it relates to model-level effective context length, we have added the following passage to Section 5.3.

These results provide complementary structural evidence that RoPE-RULER identifies distinct positional scales for FA and SWA layers. However, agreement with the configured SWA window does not independently validate behavioral effective context length. In particular, the FA estimate of approximately 182K tokens has not been independently validated by behavioral experiments and should therefore be interpreted as a spectral estimate rather than a verified effective context length.

Moreover, we have already stated in Section 7 (Limitations, lines 489–495) that independent behavioral evaluation is needed before the FA estimate can be interpreted as the model’s overall effective context length.


We believe these revisions strengthen the paper. Thank you again for your constructive feedback.

[1] Qwen-3 Model card https://huggingface.co/Qwen/Qwen3-8B

[2]Round and Round We Go! What makes Rotary Positional Encodings useful?, ICLR2025

[3]Frequency Bands in RoPE: Base Frequency and Context Length Shape the Interpolation–Extrapolation Trade-off, ICLR2026

Reviewer_gWkkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes RoPE-RULER, a method for estimating the effective context length of a language model from its RoPE frequency bands. The authors examine the norms of query representations, identify the dominant frequency band across layers and heads, and convert its RoPE scale into an estimated context length. They compare the estimates with RULER results on several language models and also examine the frequency bands of full-attention and sliding-window attention layers in GPT-OSS-120B. The method requires only a short forward pass with unlabeled text.

Critical strengths

I find the idea of estimating usable context length from internal model representations interesting. If reliable, it could be a much cheaper alternative to running long-context benchmarks at many different sequence lengths. The method itself is also simple and does not require any model training.

The comparison between full attention and sliding-window attention is a useful experiment. In particular, the estimated length of 135 for SWA layers is close to the actual window size of 128. This suggests that the frequency-band statistics can reflect differences in the positional scales used by different attention layers.

Critical weaknesses

My main concern is the agreement between RoPE-RULER and RULER reported in Table 1. The paper states that the estimates fall into the same length buckets as the RULER results, but this does not seem to hold for all models. For Qwen3-4B, Qwen3-8B, and Phi-3-mini, the predicted lengths are closer to 64K, while RULER gives 32K. This leaves only four of the seven models with matching length buckets. Since predicting effective context length is the main contribution, I think this discrepancy needs a clearer discussion.

I am also not fully convinced by the justification for Eq. (8). The choice of a phase range of pi/2 seems heuristic. A RoPE component can still distinguish positions after the rotation exceeds pi/2, and a large query norm does not necessarily mean that the corresponding frequency determines long-range attention behavior. The paper shows an interesting correlation, but I do not think the current analysis is enough to explain why this particular frequency scale should predict effective context length.

Questions for authors
  1. Could the authors clarify the length-bucket comparison in Table 1? In particular, how should we interpret the mismatches for Qwen3-4B, Qwen3-8B, and Phi-3-mini? It would help to report the prediction error across all evaluated models.
  2. Why is pi/2 used in Eq. (8) rather than another phase threshold? Have the authors compared different conversion factors or simpler frequency-based estimates?
  3. Is there evidence that the selected high-norm band contributes more to long-distance attention than other bands? The norm statistics alone do not seem sufficient to establish this connection.
2 comments
AuthorsOpenReview ↗

Author Response (1/2)

Thank you for taking the time to review our paper. We are pleased that you found it clear, easy to follow, and interesting. We also appreciate your observation that RoPE-RULER could offer a substantially less costly alternative to existing benchmarks for estimating effective context length.

We have revised the paper in response to your comments and questions. All changes are shown in red in the manuscript itself. We hope these revisions address your concerns.

W1. Concern about the reported agreement between the predictions in Table 1 and the RULER results

Thank you for pointing this out. In Table 1, we mapped each continuous RoPE-RULER estimate to the largest evaluated length at or below that estimate. The word “nearest” in the caption incorrectly described the rule we used. We revised the Table 1 caption to read as follows.

The red downward triangle marks the RULER-measured effective length, while the green downward triangle marks the largest evaluated length not exceeding the continuous RoPE-RULER estimate, as defined in Section 4.1.

We have also corrected the corresponding wording on line 366 and in Table 3.

W2. Theoretical justification for the π2\frac{\pi}{2} threshold in Eq.(8)

Thank you for raising this point. We provide the following analysis to clarify the theoretical basis for the π/2\pi/2 threshold and our use of query norms.

1. Mathematical basis for the π/2\pi/2 threshold

We derive the threshold from the effect of perturbing the key along the query direction, independently of any self-intersection argument. This direction is the direction of steepest increase in the pre-RoPE query–key dot product.

Consider a frequency chunk ii with an arbitrary query qi≠0q_i \neq 0, key kik_i, and effective angular frequency ωi>0\omega_i > 0. Let R(ϕ)R(\phi) denote a two-dimensional rotation by angle ϕ\phi. At relative distance r≥0r \geq 0, the chunk contributes the following term to the unscaled attention score:

si(r;ki)=qi⊤R(−ωir)ki.s_i(r;k_i) = q_i^\top R(-\omega_i r)k_i.

Perturbing the key by εui\varepsilon u_i, where ui=qi/∥qi∥2u_i = q_i/\|q_i\|_2 and ε>0\varepsilon > 0, increases the pre-RoPE dot product by ε∥qi∥2\varepsilon\|q_i\|_2. The corresponding change in the post-RoPE score is exactly

\Delta s_i &= s_i(r;k_i+\varepsilon u_i)-s_i(r;k_i) \\ &= \varepsilon q_i^\top R(-\omega_i r)u_i \\ &= \varepsilon\|q_i\|_2\cos(\omega_i r). \end{aligned}$$ Thus, $\Delta s_i$ is positive for $0 \leq r < \pi/(2\omega_i)$, equals zero at $r = \pi/(2\omega_i)$, and becomes negative immediately afterward. **The relative phase $\pi/2$ is therefore the first point at which a key perturbation along the query direction ceases to increase the post-RoPE score.** At this point, the rotated perturbation is orthogonal to the query. The identity holds for every $k_i$ and thus allows for arbitrary content-dependent alignment between the original query and key. **2. Why we use the query norm** The expression above shows that $\|q_i\|_2$ sets the amplitude of this directional score response for a fixed perturbation magnitude. **Prior work [1, 2] has also shown that frequency chunks with the largest query norms can substantially influence attention scores.** These observations motivate our use of query norms to select frequency chunks. **We have already discussed this point in Section 2 (“Background and Related Work”).** **3. Connection to effective context length** Our analysis does not imply that all attention contributions vanish once the relative rotation angle exceeds $\pi/2$. For a frequency chunk with effective angular frequency $\omega_i$, it identifies the first relative distance, $r_i=\pi/(2\omega_i)$, at which perturbing the key in the direction of the pre-RoPE query ceases to increase that chunk’s contribution to the attention score. And, RoPE-RULER applies this boundary to each frequency chunk, selects chunks based on their query norms, and aggregates these selections to estimate the model’s effective context length. **The per-chunk $\pi/2$ boundary follows mathematically from the score response to this specific perturbation. Whether that boundary, together with our selection and aggregation rule, predicts the model’s effective context length is an empirical question.** In Section 3, we evaluated this relationship by comparing the resulting predictions with the effective context lengths measured by RULER.
AuthorsOpenReview ↗

Author Response (2/2)

Q1. Could the authors clarify the length-bucket comparison in Table 1?

Thank you for raising this important question. In our response to W1, we explain how we compare length buckets and address the discrepancies noted for Qwen3-4B, Qwen3-8B, and Phi-3-mini.

RULER evaluates models at a discrete set of sequence lengths and reports the longest tested length at which a model meets the performance threshold as its effective context length. For example, if a model meets the threshold at 32K tokens but falls below it at 64K, those two measurements cannot pinpoint where performance crosses the threshold within that interval. A continuous estimate can therefore differ from the length reported by RULER.

When we map each continuous RoPE-RULER estimate to the largest tested length at or below it, the predicted bucket matches the one reported by RULER for all seven models. This agreement suggests that RoPE-RULER captures the scale of a model’s effective context length at the resolution of RULER’s length buckets. Assessing the accuracy of the continuous estimates themselves would require finer-grained measurements.

Q2. Rationale for choosing π2\frac{\pi}{2} as the threshold

Thank you for raising this point. As explained in our response to W2, π/2\pi/2 is the first relative rotation angle at which perturbing the key along the pre-RoPE query direction ceases to increase that frequency chunk’s contribution to the attention score. Could you clarify which simpler frequency-based estimator you had in mind? A concrete example would help us address the comparison directly.

We also tested whether alternative phase thresholds yield better agreement with RULER. We varied the threshold while keeping the frequency-band selection and bucket-mapping rules fixed. Among the thresholds tested, π/2\pi/2 was the only one that matched the RULER length bucket for six models. The next-best threshold, 3π/43\pi/4, matched three of seven. We will include these results in Appendix E.

ModelRULER00π/4\pi/4π/2\pi/23π/43\pi/4π\pi3π/23\pi/22π2\pi
Mistral-7B16K—8K ✗16K ✓16K ✓32K ✗32K ✗64K ✗
Llama-3.1-8B32K—16K ✗32K ✓32K ✓64K ✗64K ✗128K ✗
Qwen3-4B32K—16K ✗32K ✓64K ✗64K ✗128K ✗128K ✗
Qwen3-8B32K—16K ✗32K ✓64K ✗64K ✗128K ✗128K ✗
Qwen3-14B64K—32K ✗64K ✓64K ✓128K ✗128K ✗128K ✗
Qwen3-32B128K—32K ✗64K ✗64K ✗128K ✓128K ✓128K ✓
Phi-3-mini32K—16K ✗32K ✓64K ✗64K ✗128K ✗128K ✗
Matches with RULER0/70/76/73/71/71/71/7

A check mark (✓) indicates an exact bucket match. A dash (—) indicates that the estimated length is zero and therefore falls below the shortest evaluated length.

Q3. Evidence that high-norm frequency bands contribute to long-range attention

Oka et al. showed that high-norm frequency bands contribute substantially to attention scores [2]. Section 3 of their paper presents the supporting evidence in detail, and we discuss this finding in the “Frequency bands in RoPE” paragraph of Section 2 (“Background and Related Work”).

Our study builds on this finding to ask whether these bands can be used to estimate a model’s effective context length. Independently validating their contribution to attention scores is therefore beyond the scope of the present work.


We believe these revisions strengthen the paper. Thank you again for your constructive feedback.

[1]Round and Round We Go! What makes Rotary Positional Encodings useful?, ICLR2025

[2]Frequency Bands in RoPE: Base Frequency and Context Length Shape the Interpolation–Extrapolation Trade-off, ICLR2026

Comments on the paper

5 comments
AuthorsOpenReview ↗

Author Response (1/3)

Thank you for taking the time to review our paper. We are pleased that you found our research question worth pursuing, our approach simple, clear, and reproducible, and our structural observations interesting. We have revised the manuscript in response to your comments. All changes are shown in red.


W1. Concern about the reported agreement between the predictions in Table 1 and the RULER results

Thank you for pointing this out. In Table 1, we mapped each continuous RoPE-RULER estimate to the largest evaluated length at or below that estimate. The word “nearest” in the caption incorrectly described the rule we used. We revised the Table 1 caption to read as follows.

The red downward triangle marks the RULER-measured effective length, while the green downward triangle marks the largest evaluated length not exceeding the continuous RoPE-RULER estimate, as defined in Section 4.1.

We have also corrected the corresponding wording on line 366 and in Table 3.


W2. Theoretical justification for the π2\frac{\pi}{2} threshold

Thank you for raising this point. We provide the following analysis to clarify the theoretical basis for the π/2\pi/2 threshold and our use of query norms.

1. Mathematical basis for the π/2\pi/2 threshold

We derive the threshold from the effect of perturbing the key along the query direction, independently of any self-intersection argument. This direction is the direction of steepest increase in the pre-RoPE query–key dot product.

Consider a frequency chunk ii with an arbitrary query qi≠0q_i \neq 0, key kik_i, and effective angular frequency ωi>0\omega_i > 0. Let R(ϕ)R(\phi) denote a two-dimensional rotation by angle ϕ\phi. At relative distance r≥0r \geq 0, the chunk contributes the following term to the unscaled attention score:

si(r;ki)=qi⊤R(−ωir)ki.s_i(r;k_i) = q_i^\top R(-\omega_i r)k_i.

Perturbing the key by εui\varepsilon u_i, where ui=qi/∥qi∥2u_i = q_i/\|q_i\|_2 and ε>0\varepsilon > 0, increases the pre-RoPE dot product by ε∥qi∥2\varepsilon\|q_i\|_2. The corresponding change in the post-RoPE score is exactly

\Delta s_i &= s_i(r;k_i+\varepsilon u_i)-s_i(r;k_i) \\ &= \varepsilon q_i^\top R(-\omega_i r)u_i \\ &= \varepsilon\|q_i\|_2\cos(\omega_i r). \end{aligned}$$ Thus, $\Delta s_i$ is positive for $0 \leq r < \pi/(2\omega_i)$, equals zero at $r = \pi/(2\omega_i)$, and becomes negative immediately afterward. **The relative phase $\pi/2$ is therefore the first point at which a key perturbation along the query direction ceases to increase the post-RoPE score.** At this point, the rotated perturbation is orthogonal to the query. The identity holds for every $k_i$ and thus allows for arbitrary content-dependent alignment between the original query and key. **2. Why we use the query norm** The expression above shows that $\|q_i\|_2$ sets the amplitude of this directional score response for a fixed perturbation magnitude. Prior work [1, 2] has also shown that frequency chunks with the largest query norms can substantially influence attention scores. These observations motivate our use of query norms to select frequency chunks. **We have already discussed this point in Section 2 (“Background and Related Work”).** **3. Connection to effective context length** Our analysis does not imply that all attention contributions vanish once the relative rotation angle exceeds $\pi/2$. For a frequency chunk with effective angular frequency $\omega_i$, it identifies the first relative distance, $r_i=\pi/(2\omega_i)$, at which perturbing the key in the direction of the pre-RoPE query ceases to increase that chunk’s contribution to the attention score. And, RoPE-RULER applies this boundary to each frequency chunk, selects chunks based on their query norms, and aggregates these selections to estimate the model’s effective context length. **The per-chunk $\pi/2$ boundary follows mathematically from the score response to this specific perturbation. Whether that boundary, together with our selection and aggregation rule, predicts the model’s effective context length is an empirical question.** In Section 3, we evaluated this relationship by comparing the resulting predictions with the effective context lengths measured by RULER. ---- [1]Round and Round We Go! What makes Rotary Positional Encodings useful?, ICLR2025 [2]Frequency Bands in RoPE: Base Frequency and Context Length Shape the Interpolation–Extrapolation Trade-off, ICLR2026
AuthorsOpenReview ↗

Author Response (2/3)

W3. Mismatch between the RoPE settings used for prediction and evaluation

Thank you for raising this important point. We sincerely apologize for the confusion caused by our incomplete description of the RoPE settings used for the Qwen3 models. For Qwen3 models, the standard practice is to use the default, unscaled RoPE configuration for contexts within the native context window of 32K tokens. Accordingly, in our RULER evaluations, we used standard RoPE at evaluated lengths of 32K tokens or fewer and YaRN with a scaling factor of s=4s=4 at evaluated lengths of 64K tokens or more. The appendix already described the long-context scaling configuration as follows:

In our long-context setting, we use a scaling factor of 4 from an original 32,768-token range, yielding a target context length of 131,072 tokens.

We have now added this same statement to Section 4.1 of the main text. RoPE-RULER uses each pretrained checkpoint’s default RoPE configuration as the reference for identifying its learned frequency band. Because users can change the YaRN scaling factor at inference time, substituting the YaRN-adjusted frequencies into the diagnostic could change the estimated length solely because a different scaling factor was chosen, even if the learned band remained the same. We therefore compute the diagnostic estimate without this additional transformation so that the frequency bands formed during training are measured against a consistent reference.

For the RULER evaluation, we applied the same protocol to all Qwen3 models, using standard RoPE at 32K tokens and below and YaRN (s=4s=4) at 64K tokens and above. We adopted this protocol because it follows the recommended usage for Qwen3. We make no claim about how the effective context length changes when the YaRN scaling factor is varied or whether the same effective length would be obtained without YaRN. We have added this clarification to Section 3.2 (lines 207–210) of the main text.

AuthorsOpenReview ↗

Author Response (3/3)

W4. Lack of Verification

(Reviewer Comment)The main comparison contains seven models, four from one family, without held-out-family evaluation, uncertainty estimates, threshold sensitivity, or comparisons against simple configuration-based predictors.

We selected four Qwen3 model sizes (4B, 8B, 14B, and 32B) to examine how the dominant frequency band and the resulting context-length estimate vary with model size within a single family. We have added a discussion of this comparison to lines 292–300 of the main text. Our main evaluation covers four representative RoPE-based model families (Mistral, Llama, Qwen, and Phi). Our complementary analysis of GPT-OSS examines whether the method also applies to a hybrid attention architecture.

  1. Validation across model families. We evaluated seven models from four distinct families (Mistral, Llama, Qwen, and Phi). The four Qwen3 models provide a within-family comparison of model sizes.

  2. Uncertainty. RoPE-RULER computes a deterministic estimate from the frequency band identified in the model’s internal representations. An empirical question is how sensitive that estimate is to the inputs used to identify the band. Section 4.3 examines this sensitivity across input corpora and sequence lengths. We also observed similar stability in the other models we examined.

  3. Sensitivity to the RULER threshold. The RULER-measured effective context length depends on the chosen performance threshold. The RoPE-RULER estimate is independent of this threshold because it is computed from the model’s internal frequency structure. Changing the threshold could therefore change the agreement reported in Table 1. Such an analysis would assess sensitivity to the behavioral evaluation criterion. Our objective is to estimate a characteristic context length from the learned frequency band.

  4. Baseline based on model configuration. Table 1 reports both the nominal context length and the RULER-measured effective length for each model. These two lengths differ for all seven models. For those same seven models, mapping each RoPE-RULER estimate to the largest evaluated length at or below it yields the RULER-measured bucket. This comparison shows that the learned frequency band provides information about effective context length beyond the nominal setting alone.


(Reviewer Comment) Corpus robustness is examined primarily for one model.

We also tested other models with different input corpora and observed the same pattern. Prior work has likewise shown, across multiple models, that the dominant frequency band within a given model remains stable across inputs. We refer the reviewer to those analyses for further cross-corpus comparisons.


(Reviewer Comment) For GPT-OSS, obtaining an estimate near its advertised context window does not validate effective context length when the behavioral criterion is unmet; similarly, recovering a configured sliding window establishes architectural correspondence rather than demonstrated information use.

GPT-OSS did not meet the RULER performance threshold at any evaluated length, so there is no RULER-measured effective length against which to validate the approximately 182K-token estimate. Our objective was to test whether the model’s internal frequency bands capture distinct positional scales for different attention mechanisms within the same model.

We obtained estimates of 181,899 tokens for full-attention (FA) layers and 135 tokens for sliding-window attention (SWA) layers. The SWA estimate is close to the configured window size of 128 tokens. We obtained this estimate from the frequency band in the internal representations without supplying the window size to the estimator.

These results support the interpretation that RoPE-RULER captures positional scales specific to each attention mechanism, even within a model with a single nominal context length. Whether the model can use information at the estimated distances remains a behavioral question. We therefore present the GPT-OSS experiment as a complementary analysis of hybrid attention. Section 4 evaluates predictive accuracy against RULER.


(Reviewer Comment) Earlier work already connects frequency bands with training context length, making comparative predictive evaluation particularly important.

Prior work characterized how frequency bands form in RoPE-based models. Our study builds on that finding to address a different question: can the learned frequency band be used to estimate a model’s effective context length? Our contribution is the rule that maps the observed band to a length estimate and the evaluation of that estimate against RULER.


We believe these revisions strengthen the paper. Thank you again for your constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.