acceptodds
Under review as a conference paper at ICLR 2027

Science of Long Context

Abstract

Language models increasingly operate over long inputs, such as long documents, code repositories, and their own reasoning traces, and must retrieve information from any position in them. However, language model performance degrades as input length increases, and the extent of the degradation depends on the architecture. Every architectural choice is an inductive bias with its own limitations, and these limitations are poorly understood. We study them by training many architectures from scratch in a controlled setting, where the models differ only in architecture and serve as controls for each other. We find that removing explicit positional encoding alone does not improve length generalization, and that RoPE's retrieval beyond the training length degrades due to keys at offsets never observed during training. Hybrid models that restrict most layers to a sliding window still read far keys at these offsets in their global attention layers. We then show that in a relative positional encoding, keeping positions within the trained range comes at the cost of resolving nearby positions. Building on this insight, we design Log-Ratio RoPE, which rotates queries and keys by their log-position to keep offsets close to the trained range. We use it in the global attention layers of a hybrid model, while sliding-window layers retain RoPE to resolve nearby positions within the trained range. In 0.5B models trained at 8K, this design improves accuracy on the RULER long-context benchmark at 64K, eight times the training length, by 18.7 percentage points over RoPE and 8.1 percentage points over the RoPE–NoPE hybrid.

Primary area:
foundation or frontier models, including LLMs
Keywords:
long context, length generalization, positional encoding, local-global hybrid models, language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.80 over 5 reviews, above 85% of the others.
Reviewer_iq3QOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an improvement to LLMs' performance on long context (i.e. >> training length), namely a log-ratio version of RoPE that in hybrid (local, vs. global attention) that outperforms the RoPE-NoPE hybrid on long-context tasks. The paper also proves a theorem that shows an inverse relation between length generalization and positional resolution, such that the number of keys the model keeps in the trained range is inversely related to how well it generalizes beyond that range.

Critical strengths
  • Provides a new hybrid model (log-ratio RoPE hybrid) that outperforms other hybrid or non-hybrid RoPE/NoPE-based models on extending to long context on the RULER task.

  • Proves a theorem about the relation between positional encoding resolution and length generation that can inform future work on improving long context generalization.

Critical weaknesses
  • The title seems a bit hyperbolic compared to the actual contribution of the paper. One could argue that Theorem 1 is a contribution to a theory of the "Science of Long Context", but apart from that, the paper seems to be an instance of a genre that has seen a lot of work recently, namely variations on position encoding methods, or lack thereof, in order to try to improve LLMs' performance on long context.

  • As far as I can tell, the method is only evaluated on one set of long-context tasks, namely RULER. In past work even within this task, various subtasks (e.g. NIAH, QA), can be expected to show a wide range of performance.

  • The method only compares against PE approaches that are static. What about methods that learn to reorder positional indices to optimize for specific tasks, such as RePO (Li et al, 2026, ICML)?

Questions for authors
  • Which subsets of RULER are you reporting? Is this an aggregate score?

  • It seems odd to have the related work section be all the way back in Section 7.

  • Not a question, but it looks as if the AI use statement could use some editing.

2 comments

We thank the reviewer for the feedback and for the pointer to RePo, which we have now trained in our setup and compared against.

W1. Is the title too broad for the paper's contribution, which looks like one more positional-encoding variant?

Thank you for raising this. Log-Ratio RoPE is not the main contribution of the paper. Most of the paper (Sec. 2–5) is a controlled study of why two architectural choices, the positional encoding and how local and global attention layers are combined, help or hurt performance beyond the training length. Log-Ratio RoPE is the design that follows from this study (Theorem 1). We agree that the original title did not say which part of long context we study. We have changed it to "Science of Long Context: How Positional Encoding and Hybrid Attention Shape Extrapolation". The new title names the two choices we study and the property we measure.

Paper Update: Title.

W2. Is RULER alone enough, given that its subtasks can vary widely in performance?

We agree that our main long-context results use only RULER. Our reason is that at academic pretraining scale (0.5B and 1.5B parameters), with generic pretraining data such as FineWeb-Edu, benchmarks that need reasoning or stored knowledge do not give a reliable signal for comparing architectures [1]. We chose RULER because it covers a wide variety of tasks (13 tasks) and is largely decoupled from other model capabilities, such as reasoning and knowledge storage. We also agree that performance varies widely across its subtasks. This was our original reason for reporting the average over all 13 tasks, which accounts for the variance across subtasks. To show the full picture, we now also include per-task results for all 13 RULER tasks at both scales in Appendix H.1 (Figs. 20 and 21).

Paper Update: Appendix H.1 (Figs. 20 and 21).

W3. How does Log-Ratio RoPE compare with learned positional methods such as RePo?

Thank you for pointing out this recent work, which is relevant to ours. We have added RePo to the related work (Appendix I). We also trained RePo in our small-model setup at an 8K training length, both with full attention and as the global layers of a hybrid with sliding-window RoPE layers, with the same learning-rate sweep as our other models (Appendix H.4, Fig. 23, left). With full attention, RePo falls to chance by 16K (2.0%). In the hybrid, RePo does much better than the RoPE–RoPE hybrid: 52.0% against 18.4% at 32K, and 33.6% against 7.3% at 64K. It stays below the Log-Ratio hybrid (61.3% and 50.4%).

Paper Update: Appendix H.4 (Fig. 23, left), RePo in the extended related work (Appendix I).

8-pair kv recall (%), 500 sequences per length. Errors are 95% confidence intervals.

Model1K2K4K8K16K32K64K
RoPE, Full Attention75.6±1.375.6 \pm 1.375.2±1.475.2 \pm 1.473.2±1.473.2 \pm 1.463.8±1.663.8 \pm 1.632.0±1.432.0 \pm 1.410.8±0.910.8 \pm 0.92.1±0.42.1 \pm 0.4
RePo, Full Attention72.2±1.472.2 \pm 1.470.7±1.470.7 \pm 1.465.5±1.565.5 \pm 1.538.0±2.338.0 \pm 2.32.0±0.42.0 \pm 0.41.7±0.41.7 \pm 0.41.3±0.31.3 \pm 0.3
RoPE Hybrid73.9±1.473.9 \pm 1.472.7±1.472.7 \pm 1.471.6±1.471.6 \pm 1.465.7±1.665.7 \pm 1.640.3±1.440.3 \pm 1.418.4±0.918.4 \pm 0.97.3±0.77.3 \pm 0.7
RePo Hybrid75.8±1.375.8 \pm 1.375.1±1.375.1 \pm 1.374.1±1.474.1 \pm 1.471.2±1.571.2 \pm 1.564.0±1.664.0 \pm 1.652.0±1.752.0 \pm 1.733.6±1.733.6 \pm 1.7
NoPE Hybrid75.1±1.375.1 \pm 1.374.4±1.474.4 \pm 1.474.0±1.474.0 \pm 1.471.6±1.571.6 \pm 1.566.2±1.666.2 \pm 1.657.6±1.757.6 \pm 1.740.6±1.840.6 \pm 1.8
Log-Ratio Hybrid76.1±1.476.1 \pm 1.474.9±1.474.9 \pm 1.474.2±1.474.2 \pm 1.471.7±1.571.7 \pm 1.568.4±1.668.4 \pm 1.661.3±1.661.3 \pm 1.650.4±1.750.4 \pm 1.7

Questions

Q1. Is the reported RULER score an aggregate over tasks?

Yes, it is the average over all 13 RULER tasks, as stated in the figure caption. Per-task scores are now in Appendix H.1 (Figs. 20 and 21).

Q2. Why is the related work in Sec. 7?

We placed the related work in Sec. 7 for clarity and flow, since each section introduces the methods it studies at its start.

Q3. Not a question, but it looks as if the AI use statement could use some editing.

We have updated this. Thank you for bringing this to our attention.

Summary of paper changes:

  • Title now names the two architectural choices we study: "Science of Long Context: How Positional Encoding and Hybrid Attention Shape Extrapolation".
  • Per-task RULER at 0.5B and 1.5B: Appendix H.1 (Figs. 20 and 21).
  • RePo with full attention and as the global layers of the hybrid: Appendix H.4 (Fig. 23, left); RePo in the extended related work (Appendix I).
  • Revised AI use statement.

Thank you for taking the time to review our paper and helping us further improve it.

[1] https://arxiv.org/abs/2512.17351

Reviewer_iq3QOpenReview ↗

Response

Thank you for your updates, and for the additional experiments. I am happy to raise my score.

Reviewer_AXUROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how positional encoding and local–global attention affect long-context performance. The authors train models from scratch under controlled settings and examine why retrieval degrades beyond the training length. Based on a trade-off between positional resolution and length generalization, they propose Log-Ratio RoPE for global attention while keeping standard RoPE in local layers. The resulting model improves RULER performance beyond the training length without a clear loss in short-context performance.

Critical strengths

The controlled experimental setup is a strength. Training different architectures with the same data and token budget makes the comparisons more informative than comparing existing models with different training histories.

The paper connects its analysis to a concrete design. The proposed hybrid improves RULER accuracy at 64K by 18.7 points over RoPE and 8.1 points over the RoPE–NoPE hybrid, despite being trained at 8K. The appendices also examine several design choices.

Critical weaknesses

The experimental scope is limited relative to the broad conclusions. Most analysis uses 64M models, and the final evaluation uses 0.5B models trained at one context length. Long-context evaluation mainly relies on synthetic retrieval tasks and RULER. It remains unclear whether the findings hold for larger models and tasks requiring reasoning over long documents.

The connection between the theoretical result and practical performance needs more discussion. The theorem concerns a specific class of positional encodings and a worst-case resolution measure for keys with identical content. Keeping positional values within the trained range does not itself guarantee retrieval accuracy. Some conclusions, particularly that RoPE should be restricted to local layers, are stronger than the evidence currently supports.

Questions for authors

Do the improvements hold on long-document question answering or other tasks requiring more than retrieval?

How sensitive are the 0.5B results to the training seed and learning rate? Do the reported confidence intervals include training variability?

How does Log-Ratio RoPE behave when the same query and evidence are shifted to later positions while their token distance remains unchanged?

Does the advantage persist beyond the 128K design horizon or when the models are trained at longer context lengths?

2 comments

We thank the reviewer for the detailed questions on model scale, training seeds and extrapolation. We ran new experiments for each, which we describe below.

W1. Do the findings hold for larger models and for tasks that need reasoning over long documents?

We agree that our final evaluation used a single scale and a single training length. To address this, we trained three 1.5B models (RoPE, the RoPE–NoPE hybrid and the Log-Ratio hybrid) at a 4K training length, with the same hyperparameters as the 0.5B models (Sec. 8, Fig. 6). The result holds: past the training length, the Log-Ratio hybrid again has the highest RULER accuracy from 16K to 64K. At 32K, eight times the training length, it improves on RoPE by 28.7 points and on the RoPE–NoPE hybrid by 10.2 points, close to the gain at 0.5B (8.1 points over the RoPE–NoPE hybrid at 64K). So the findings hold at two scales and two training lengths. Short-context accuracy does not drop at either scale (Appendix H.2, Table 2).

On tasks that need reasoning over long documents: at academic pretraining scale, with generic pretraining data such as FineWeb-Edu, benchmarks that need reasoning or stored knowledge do not give a reliable signal for comparing architectures [1]. We therefore use RULER, which covers 13 tasks, including variable tracking and aggregation in addition to retrieval, and is largely decoupled from reasoning and knowledge storage.

Paper Update: Abstract and Sec. 1 (1.5B results), Sec. 8 (Fig. 6, bottom row), Appendix A (Table 1), Appendix H.1 (Figs. 20 and 21), Appendix H.2 (Table 2).

W2a. How does the theorem, with its worst-case resolution measure, connect to practical performance?

We agree that keeping keys in the trained range does not by itself guarantee retrieval. Theorem 1 covers only the positional input to attention, and we now say so in the limitations (Appendix J). We also revised the resolution measure (Sec. 5.3). A key is now unresolved if it is closer in Δ\Delta to another key than any two distinct values of Δ\Delta seen in training, and we measure the share of a query's keys that are unresolved. This mirrors length generalization, which measures the share of keys whose Δ\Delta lies outside the range seen in training. Both properties are now measured against what the model saw during training, so the theorem can compare the two shares directly. The measure still assumes keys with the same content, which is the worst case. In practice, both sides of the trade-off appear. Log-Ratio RoPE in all layers (in the trained range, low resolution) and NoPE fall below RoPE already within the training length (66.1% and 67.5% against 73.2% at 4K; Appendix G.3), while RoPE fails past it. The hybrid, which splits the two across layers, does best both within and past the training length.

Paper Update: Sec. 5.3 (revised resolution measure), Appendix J (limitations).

W2b. Is the conclusion that RoPE should be used only in the local layers too strong?

We believe the conclusion on RoPE in the local layers is supported. In Sec. 3, we added an artificial-gap experiment (Fig. 3, Appendix C.2). We shift every token after a key-value pair, the query included, by a gap, so the pair moves to unseen offsets while the number of keys stays at 8K. Recall falls almost as fast as in a real longer context: at a query position of 16K, it is 18% with the gap and 14% in the real context, down from 62% at 8K. So unseen offsets alone, without additional keys, cause most of RoPE's drop. In Sec. 4, the RoPE–RoPE and RoPE–NoPE hybrids differ only in their global layers, and RoPE there costs 26 to 39 points of 8-pair recall past the training length (7% against 41% at 64K). This loss is at unseen offsets: single-key recall there is 3% against 31% at 64K. The same holds at 0.5B on RULER: the RoPE–RoPE hybrid drops sharply past the training length, while the RoPE–NoPE and Log-Ratio hybrids, which remove RoPE from the global layers, stay well above it (Sec. 8, Fig. 6).

A related question is why we do not use NoPE or Log-Ratio RoPE in the local layers. RoPE has positional resolution, and the other two do not. Without it, the model is already weaker within the training length, and this weakness carries over past it. The NoPE–NoPE hybrid, with NoPE in the local layers as well, still fails at the training length: its loss rises by 1.1 nats from 8K to 16K and its 8-pair recall falls to chance by 16K, while the RoPE–RoPE hybrid does not fail there (Sec. 4). With Log-Ratio RoPE in every layer, 8-pair recall is below all-layer RoPE already within the training length (66.1% against 73.2% at 4K) and falls to 1.6% by 32K (Appendix G.3, Fig. 14). RoPE in the local layers gives the resolution, and Log-Ratio RoPE in the global layers keeps keys in the trained range.

Questions

Q2. Do the results depend on the training seed and learning rate at 0.5B scale, and do the confidence intervals include training variability?

The confidence intervals in our figures are 95% intervals over evaluation samples (500 per task and length), so they measure evaluation noise, not training variability. We also added a new experiment in which we trained the small RoPE–RoPE, RoPE–NoPE and Log-Ratio hybrids with three seeds. In this figure, the error bars are one standard deviation across training seeds (Appendix H.5, Fig. 23, right). The ordering of the three hybrids holds past the training length: at 64K, the Log-Ratio hybrid reaches 47.7±4.647.7 \pm 4.6% recall, against 34.1±6.934.1 \pm 6.9% for the RoPE–NoPE hybrid and 5.3±1.85.3 \pm 1.8% for the RoPE–RoPE hybrid.

We also check training variability in two other ways. First, we show the training loss of all 0.5B and 1.5B models (Appendix H.3, Fig. 22). The models at each scale share the same seed for weight initialization and data order, and their loss curves are nearly identical, so the gaps past the training length do not come from tuning the hyperparameters of a specific model. Second, we trained 1.5B models with the same hyperparameters as the 0.5B models, without retuning, and the Log-Ratio hybrid again has the highest RULER accuracy past the training length (Sec. 8, Fig. 6). Pre-training the 0.5B models with several seeds or learning rates was too costly, so these checks are at the small scale and at a second model scale.

Paper Update: Appendix H.5 (Fig. 23, right), Appendix H.3 (Fig. 22), Sec. 8 (Fig. 6).

Q3. How does Log-Ratio RoPE behave when the same query and evidence are shifted to later positions while their token distance remains unchanged?

We thank the reviewer for this suggestion. We added this test (Appendix G.2, Fig. 13): we hold the distance between the key-value pair and the query fixed and move both to later positions, from 8K to 64K. For Log-Ratio RoPE, a fixed distance dd gives the coordinate ln⁡(m/(m−d))\ln(m/(m-d)), which decreases as the query position mm grows. At a fixed distance, the recall of the Log-Ratio hybrid falls slowly, by 8 to 14 points between 8K and 64K (for example, from 66.2% to 52.2% at a distance of 7.5K). Over the same range, the RoPE–NoPE hybrid falls to 31–41% and RoPE full attention to 7–30%. The RoPE–RoPE hybrid holds up at short distances, since its pair stays at a trained offset, but at a distance of 7.5K it falls to 31% at 64K.

Paper Update: Appendix G.2 (Fig. 13).

Q4. Does the advantage persist beyond the 128K design horizon or when the models are trained at longer context lengths?

We now report recall up to 128K for the small hybrids trained at 8K, averaged over three training seeds (Appendix H.5, Fig. 23, right). At 128K, the Log-Ratio hybrid reaches 32.6±2.832.6 \pm 2.8% recall, against 18.8±4.718.8 \pm 4.7% for the RoPE–NoPE hybrid and 1.7±0.31.7 \pm 0.3% for the RoPE–RoPE hybrid.

Paper Update: Appendix H.5 (Fig. 23, right).

8-pair kv recall (%), mean ±\pm sd over seeds 0, 4916, 5328 (chance 1.6):

Model64K128K
Log-Ratio Hybrid47.7±4.647.7 \pm 4.632.6±2.832.6 \pm 2.8
NoPE Hybrid34.1±6.934.1 \pm 6.918.8±4.718.8 \pm 4.7
RoPE Hybrid5.3±1.85.3 \pm 1.81.7±0.31.7 \pm 0.3
Log-Ratio − NoPE Hybrid (paired)+13.6±3.3+13.6 \pm 3.3+13.8±2.9+13.8 \pm 2.9

Summary of paper changes:

  • 1.5B models at a 4K training length: Abstract, Sec. 1, Sec. 8 (Fig. 6), Appendix A (Table 1), Appendix H.1 (Figs. 20 and 21), Appendix H.2 (Table 2).
  • Resolution measure revised to the share of unresolved keys: Sec. 5.3.
  • Limitations (Appendix J): Theorem 1 covers only the positional input to attention, while retrieval also depends on the content of the keys.
  • Gap test (unseen offsets at a fixed number of keys): Sec. 3 (Fig. 3), Appendix C.2.
  • Log-Ratio RoPE on all full-attention layers: Appendix G.3 (Fig. 14).
  • Recall at a fixed distance between the pair and the query: Appendix G.2 (Fig. 13).
  • Training loss of the 0.5B and 1.5B models: Appendix H.3 (Fig. 22).
  • Seed replicates of the small hybrids and recall up to 128K: Appendix H.5 (Fig. 23, right).

Thank you for taking the time to review our paper and helping us further improve it.

References

[1] https://arxiv.org/abs/2512.17351

Reviewer_yV4vOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper uses controlled from-scratch training, where models differ only in architecture, to show that RoPE’s failure beyond the training length mainly comes from offsets never seen in training. Building on this, it proves an inverse relation between length generalization and positional resolution, and proposes a hybrid with RoPE in the local layers and Log-Ratio RoPE in the global layers.

Critical strengths

The problem is clearly framed, the controlled analysis is convincing, and the logic from analysis to theory to design is natural.

Critical weaknesses
  1. The 8.1-point gain over the RoPE–NoPE hybrid at 64K comes from a single scale (0.5B), one learning rate, and one training length. Yet the appendix shows that changing only the frequency base moves RULER by 11.3 points, which is of the same order as this margin. Was the RoPE–NoPE hybrid tuned to the same degree?
  2. Lack of direct validation of Log-Ratio RoPE itself. The single-key analysis in Sec. 3 and 4 nicely separates OOD offsets from the number of keys, but the same analysis is not done for the Log-Ratio hybrid. Is its recall consistent between trained and OOD positions? In log space, the resolution between distant keys drops, so what is the practical cost for far-range retrieval?
Questions for authors

See weaknesses above.

2 comments

Thank you for taking the time to review our paper and helping us further improve it.

W1. Is the 8.1-point gain over the RoPE–NoPE hybrid robust, given one scale, one learning rate and one training length, and was the RoPE–NoPE hybrid tuned as much as Log-Ratio?

The appendix ablations are a sensitivity analysis of the hyperparameters of Log-Ratio RoPE. The 11.3-point drop appears only at frequency base b=2b=2, a setting in which all frequencies lie within a factor of 2 (standard RoPE uses b≈106b \approx 10^6). Moving the base from b=32b = 32 to b=256b = 256 changes RULER by at most 2.8 points up to 32K. Log-Ratio RoPE needs a smaller frequency base bb than RoPE because it rotates by ln⁡(t/s)\ln(t/s) rather than by the offset t−st - s, so its range is much smaller. A large frequency base bb would therefore mean that most pairs barely rotate and carry almost no position information. The RoPE–NoPE hybrid shares every hyperparameter with the other hybrids, including the Log-Ratio hybrid (local RoPE base, window, layer ratio, learning rate), and its NoPE layers have none to tune.

Additionally, with the same hyperparameters used to train our 0.5B models, we trained 1.5B models at a 4K training length for three variants: RoPE, the RoPE–NoPE hybrid, and the Log-Ratio hybrid. We trained the 1.5B models at 4K instead of 8K because attention on long sequences adds training cost.

1.5B RULER average over 13 tasks (%), 500 samples per task and length. Errors are 95% confidence intervals.

Model1K2K4K8K16K32K64K
RoPE66.8±0.966.8 \pm 0.957.6±0.857.6 \pm 0.849.4±0.949.4 \pm 0.922.0±0.922.0 \pm 0.97.8±0.67.8 \pm 0.62.5±0.42.5 \pm 0.40.9±0.20.9 \pm 0.2
RoPE + YaRN67.6±0.967.6 \pm 0.958.0±0.958.0 \pm 0.952.6±0.952.6 \pm 0.949.4±0.849.4 \pm 0.841.9±0.841.9 \pm 0.814.8±0.814.8 \pm 0.87.1±0.67.1 \pm 0.6
NoPE Hybrid69.8±0.969.8 \pm 0.962.4±0.862.4 \pm 0.859.1±0.959.1 \pm 0.940.5±1.040.5 \pm 1.035.1±0.935.1 \pm 0.921.0±0.821.0 \pm 0.812.4±0.512.4 \pm 0.5
Log-Ratio Hybrid67.6±0.967.6 \pm 0.960.1±0.960.1 \pm 0.957.4±1.057.4 \pm 1.047.9±0.947.9 \pm 0.943.8±0.943.8 \pm 0.931.2±0.931.2 \pm 0.917.1±0.717.1 \pm 0.7

Past the training length, the Log-Ratio hybrid again has the highest RULER accuracy from 16K to 64K, above both RoPE and the RoPE–NoPE hybrid. Its largest gain over the RoPE–NoPE hybrid, +10.2 points, is at eight times the training length (32K), as at 0.5B (+8.1 at 64K). Within the training length, the RoPE–NoPE hybrid is slightly higher, and RoPE + YaRN is higher at 8K. This shows that the hyperparameters were not tuned for a single 0.5B run. Our RoPE base b=106b = 10^6 is also standard: Qwen2 [1] uses 10610^6, and Llama 3 [2], Olmo 3 [3] and Muse-Glimmer-30B [4] use 0.5×1060.5 \times 10^6.

We also added the training loss of all 0.5B and 1.5B models (Appendix H.3, Fig. 22). All models at each scale use the same seed for weight initialization and data order, so their training loss curves are nearly identical. The models differ only in the architecture, and the difference in long-context performance past the training length comes from the positional encoding.

Paper Update: Abstract and Sec. 1 (1.5B results), Sec. 8 (Fig. 6, bottom row), 1.5B training details in Appendix A (Table 1), per-task RULER in Appendix H.1 (Fig. 21), short-context benchmarks in Appendix H.2 (Table 2), training loss in Appendix H.3 (Fig. 22).

2a. Is the Log-Ratio hybrid's recall the same at trained and OOD offsets (the single-key test of Sec. 3–4)?

Thank you for suggesting this analysis. We added the single-key test of Sec. 3–4 for the Log-Ratio hybrid (Sec. 6, Fig. 5; additional details in Appendix G.1). Unlike RoPE, its recall does not drop sharply at OOD offsets: at 10K it is 62.8% at OOD offsets against 65.0% at trained offsets. It then falls with length, to 39.8% at OOD offsets at 64K, 14.7 points below its trained-offset recall but above the RoPE–NoPE hybrid (31.2%). We also trained our small hybrid models with 3 seeds (Appendix H.5, Fig. 23 right). Past the training length, the Log-Ratio hybrid stays above the RoPE–NoPE hybrid in 8-pair recall in all 3 seeds: at 64K, 47.7±4.647.7 \pm 4.6% against 34.1±6.934.1 \pm 6.9% (mean ±\pm standard deviation over seeds).

Paper Update: Sec. 6 (Fig. 5), details in Appendix G.1.

2b. What does the lower far-range resolution of the log map cost for retrieval?

From the results above, we do not find that the lower resolution in log space harms retrieval of distant keys. However, when every layer is a full-attention layer with Log-Ratio RoPE, without any sliding-window RoPE layers, 8-pair recall falls below that of all-layer RoPE, already within the training length (66.1% against 73.2% at 4K). The NoPE model, which has no explicit positional encoding, shows a similar drop (67.5% at 4K). For nearby tokens late in the sequence, Log-Ratio RoPE gives almost the same position, since ln⁡m−ln⁡n≈(m−n)/m\ln m - \ln n \approx (m-n)/m. So neither model has an explicit signal (positional resolution) that tells nearby tokens apart. In the hybrid, the sliding-window RoPE layers provide this signal, and the drop does not appear (74.2% at 4K).

Paper Update: Appendix G.3 (Fig. 14).

Summary of paper changes:

  • 1.5B models (RoPE, RoPE–NoPE hybrid, Log-Ratio hybrid) at a 4K training length: Abstract, Sec. 1, Sec. 8 (Fig. 6), Appendix A (Table 1), Appendix H.1–H.3 (Figs. 21 and 22, Table 2).
  • Single-key recall at trained vs OOD offsets for the Log-Ratio hybrid: Sec. 6 (Fig. 5), Appendix G.1.
  • Log-Ratio RoPE on all full-attention layers: Appendix G.3 (Fig. 14).
  • Three training seeds of the small hybrids: Appendix H.5 (Fig. 23, right).

We thank the reviewer for the careful reading and for the questions on tuning and on validating Log-Ratio directly. Both led to new experiments (1.5B scale, Sec. 6 Fig. 5).

References

[1] https://arxiv.org/abs/2407.10671

[2] https://arxiv.org/abs/2407.21783

[3] https://arxiv.org/abs/2512.13961

[4] https://huggingface.co/meta-models/Muse-Glimmer-30B

Reviewer_Cw7QOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how positional encoding and local–global attention composition affect context utilization and length extrapolation. It compares models trained from scratch under a shared training setup, using language-modeling loss, associative recall, and graph-walk tasks. It then formalizes a tradeoff between positional resolution and keeping positional relations within the training range, and proposes Log-Ratio RoPE for the global layers of a hybrid model while retaining ordinary RoPE in local layers. In 0.5B models trained at 8K, the proposed hybrid improves average RULER accuracy at 64K by 18.7 percentage points over full-attention RoPE and 8.1 points over the RoPE–NoPE hybrid, with similar average short-context performance.

Critical strengths
  1. The controlled comparisons provide useful insight into the different roles of positional information in local processing and long-range retrieval. In particular, the contrast between fully position-free models and hybrids with position-free global layers helps explain why removing positional encoding can have different effects depending on where it is applied. Evaluating recall alongside language-modeling loss makes this analysis substantially more informative.

  2. The analysis leads to a simple, computationally compatible architectural modification with promising empirical benefits. Log-Ratio RoPE preserves separate query/key transformations and improves over the relevant RoPE–NoPE hybrid, rather than only over standard RoPE. The combination of a clear design rationale and the reported 8.1-point gain at 64K supports a weak acceptance recommendation despite the remaining limitations in mechanistic attribution.

Critical weaknesses
  1. The causal interpretation in Section 3 is stronger than the experimental controls establish. The additional 19-point near-versus-far recall gap is attributed to unseen positional offsets by comparing models trained at different context lengths. However, changing training length also changes the representations and attention strategies the model can learn. Holding evaluation length fixed controls the number of competing keys, but does not isolate the effect of the positional rotations themselves. The evidence supports the proposed explanation, but the quantitative causal attribution requires either a more direct intervention or more qualified wording.

  2. The contribution of the log-ratio position map is not fully separated from the accompanying frequency design. The proposed method also changes the frequency ladder and introduces trainable per-head scaling. Appendix F usefully studies these choices within the log-ratio family, but does not establish how much of the benefit requires the logarithmic map rather than a suitably configured linear-position rotary encoding. This distinction matters because the paper presents the empirical improvement as evidence for its specific theoretical design principle.

Questions for authors
  1. What evidence beyond the cross-training-length comparison supports attributing the additional 19-point recall gap specifically to unseen RoPE offsets? Can the authors provide a controlled positional intervention or a more direct analysis of the relevant attention logits? Alternatively, would they qualify this result as evidence consistent with the proposed mechanism?

  2. How much of the improvement depends specifically on the log-ratio map? A focused comparison using the same hybrid layout, frequency ladder, and per-head scaling, with a linear position map matched to the training-range phase span, would help distinguish the proposed mapping from frequency-design effects.

  3. Could the authors provide per-task RULER results for Log-Ratio versus the RoPE–NoPE hybrid at 32K and 64K? I would particularly like to understand whether the aggregate improvement extends to tasks requiring tracking or composition, in addition to direct retrieval.

1 comment

We thank the reviewer for the precise suggestions. The gap test (Sec. 3) and the linear-map control (Appendix G.7) both follow the reviewer's proposals and are now in the paper.

W1. The causal interpretation in Sec. 3 is stronger than the experimental controls establish. Can the authors present a better causal experiment?

Thank you for suggesting this causal experiment. We agree that training at a longer length also changes the model's representations. To address this, we added a new causal experiment that moves a key to unseen offsets without adding keys, by directly changing the position indices before the RoPE rotation (Sec. 3, Fig. 3; details in Appendix C.2). We use the RoPE model trained at 8K and an 8K-token input with one key-value pair and the query at the end. We insert a gap in the positions right after the pair: the tokens up to the pair keep their positions, and every later token, including the query, is shifted by the gap. The pair is then at an offset never seen in training, while the input still has 8K tokens. We compare this with a real context of length 8K plus the gap, which has the pair at the same offset but more keys. With only 8K keys, recall falls almost as fast as in the real context: at a query position of 16K, it is 18% with the gap and 14% in the real context, down from 62% at 8K. Removing the additional keys therefore recovers only a few points, and most of the drop is due to the unseen offsets. Since this experiment changes only the position indices, it provides causal evidence that unseen offsets are the primary cause of the performance loss in RoPE models.

Paper Update: Sec. 3 (Fig. 3), details in Appendix C.2.

8K RoPE model, single-pair kv recall (%), pair before the gap (right panel). Errors are 95% confidence intervals.

Query positionGapReal context (8K + gap keys)Position gap (8K keys)
8K062.2±1.562.2 \pm 1.562.3±1.562.3 \pm 1.5
10K2K44.2±1.544.2 \pm 1.544.2±1.544.2 \pm 1.5
12K4K29.4±1.429.4 \pm 1.432.1±1.432.1 \pm 1.4
14K6K21.6±1.321.6 \pm 1.324.7±1.324.7 \pm 1.3
16K8K13.9±1.113.9 \pm 1.118.1±1.218.1 \pm 1.2
20K12K5.2±0.75.2 \pm 0.710.1±0.910.1 \pm 0.9
24K16K4.1±0.64.1 \pm 0.66.4±0.86.4 \pm 0.8
32K24K2.9±0.52.9 \pm 0.54.1±0.64.1 \pm 0.6
48K40K2.1±0.42.1 \pm 0.42.6±0.52.6 \pm 0.5
64K56K2.6±0.52.6 \pm 0.52.4±0.52.4 \pm 0.5

W2. Does the gain come from the log map, or from the frequency ladder and per-head scale?

We agree with your concern about whether the gain comes from the log-ratio position map (Δln⁡=ln⁡m−ln⁡n\Delta_{\ln} = \ln m - \ln n) rather than from a linear map (Δ=m−n\Delta = m - n), as in RoPE. To isolate this effect, we trained the control you describe (Appendix G.7, Fig. 19): a small hybrid with the same layout, frequency ladder, and per-head scale as the Log-Ratio hybrid, but with the linear map Δ=m−n\Delta = m - n in its global layers. We scale every frequency by ln⁡(S)/S\ln(S)/S, so that over the training length the angles cover the same range as in Log-Ratio RoPE. Within the training length, the two models are within 4 points in recall of 8 key-value pairs (68.2% against 71.7% at 8K). Beyond that, the linear map fails like RoPE: its recall drops to 20.5% at 16K and 3.2% at 64K, while the Log-Ratio hybrid maintains 68.4% and 50.4%. At OOD offsets, its single-key recall is 4.1% at 16K, against 58.9% for the Log-Ratio hybrid. The gain therefore comes from the log map, not from the frequency ladder or the per-head scale.

Paper Update: Appendix G.7 (Fig. 19).

Questions

Q3. Could the authors provide per-task RULER results?

Yes. We now report all 13 RULER tasks at every length (Appendix H.1, Figs. 20 and 21).

Paper Update: Appendix H.1 (Figs. 20 and 21).

Summary of paper changes:

  • Gap test (unseen offsets at a fixed number of keys): Sec. 3 (Fig. 3), Appendix C.2.
  • Linear-map control for Log-Ratio RoPE: Appendix G.7 (Fig. 19).
  • Per-task RULER at 0.5B and 1.5B: Appendix H.1 (Figs. 20 and 21).

Thank you for taking the time to review our paper and helping us further improve it.

Reviewer_PpGjOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies positional encoding in Transformer architectures. Through experiments, the authors show that removing explicit positional encoding does not improve length generalization, and that the retrieval ability of RoPE degrades when extrapolating beyond the training length. The paper then discusses these observations and provides some theoretical analyses. The authors propose Log-RoPE, which applies a logarithmic transformation to the relative positional differences in RoPE to achieve better generalization.

Critical strengths

The authors study an important problem: the expressiveness and generalization ability of positional encoding in long-context settings. Some of the controlled experimental analyses are interesting.

Critical weaknesses

The title of the paper is confusing. Even after reading the title and the entire paper, I still do not understand why the authors chose this title.

The paper is very difficult to follow. Much of the writing is confusing, and at times it is difficult to understand what the authors are trying to convey. Although the Introduction includes a dedicated Contributions section, it is still very difficult to identify the main contributions of the paper. I believe the writing needs to be significantly improved.

There are several overclaims in the paper. For example, in the Abstract and Introduction, the authors claim to study model architectures, but the investigation is actually limited to hybrid sliding-window attention and positional encoding. To my understanding, most other components of model architectures are not discussed, such as residual connections, normalization, FFNs, MoE, and activation functions. A broader architectural exploration should also consider non-Transformer architectures.

Many aspects of the paper are not novel, such as hybrid attention structures, sliding-window attention, and studies on the generalization ability of positional encoding. The authors' observations and findings do not seem substantially different from those of prior works. However, these discussions occupy a large portion of the paper. The proposed theoretical analyses also do not seem to directly support the proposed method. The connection between worst-case positional resolution and actual long-context capabilities remains unclear. For example, it is unclear why a model must maintain precise positional resolution at arbitrarily large distances.

Regarding the experiments, although the paper includes some interesting analytical experiments, the main experimental results are not sufficiently convincing. The choice of baseline architecture seems somewhat outdated. Qwen2 is a relatively old architecture, and some components commonly used in more recent architectures, such as QK-Norm, are absent from Qwen2. Since the results on short-context benchmarks are nearly identical across different methods, and short-context performance is not the main focus of this paper, the proposed method is effectively evaluated on only one long-context benchmark, RULER. I suggest that the authors refer to the original RoPE paper and include additional experiments, such as reporting training loss.

Questions for authors

See weaknesses.

2 comments

We thank the reviewer for feedback regarding clarity.

  1. The title of the paper is confusing. Even after reading the title and the entire paper, I still do not understand why the authors chose this title.

    We have changed the title to "Science of Long Context: How Positional Encoding and Hybrid Attention Shape Extrapolation". By "science" we mean a controlled study that explains why models behave as they do, not only how well they perform. We train models from scratch that differ only in the positional encoding and in how local and global attention layers are combined. Each model serves as a control for the others, which lets us find why performance degrades beyond the training length. The new subtitle names the two architectural choices we study and the property we measure. This matches the narrower scope of the abstract and Sec. 1 (see our response to Weakness 3).

    Paper Update: Title.

  2. The paper is very difficult to follow. Much of the writing is confusing, and at times it is difficult to understand what the authors are trying to convey. Although the Introduction includes a dedicated Contributions section, it is still very difficult to identify the main contributions of the paper. I believe the writing needs to be significantly improved.

    We have updated our abstract and introduction to make the paper easier to follow.

  3. There are several overclaims in the paper. For example, in the Abstract and Introduction, the authors claim to study model architectures, but the investigation is actually limited to hybrid sliding-window attention and positional encoding. To my understanding, most other components of model architectures are not discussed, such as residual connections, normalization, FFNs, MoE, and activation functions. A broader architectural exploration should also consider non-Transformer architectures.

    In our original submission, in the second line of the second paragraph of our introduction, we state: "Throughout this work, we use architecture to refer to the choice of positional encoding and to how global and local attention are composed across the model." In our revised version, we have updated the paper to state this more strongly, i.e., "Throughout this work, we explore the role of two architectural choices which are most relevant to long-context performance: the positional encoding, and how global and local attention layers are composed across the model.", to leave minimal room for ambiguity.

4a. Many aspects of the paper are not novel, such as hybrid attention structures, sliding-window attention, and studies on the generalization ability of positional encoding. The authors' observations and findings do not seem substantially different from those of prior works. However, these discussions occupy a large portion of the paper.

The reviewer is correct to point out that many of the architectures we study in the paper are not novel, and we do not claim novelty for prior work. Our only novel architectural contribution is Log-Ratio RoPE. The primary reason we have spent more than two-thirds of our paper studying prior architectures is to understand better why and how architectural choices affect long-context performance, as we have limited insight into why certain modifications to the model enable or impair long-context performance. This is the main question we aim to answer in this paper, and we do so through controlled experiments with theoretical analysis. Our core contribution is a better understanding of this specific model capability.

4b. The proposed theoretical analyses also do not seem to directly support the proposed method. The connection between worst-case positional resolution and actual long-context capabilities remains unclear. For example, it is unclear why a model must maintain precise positional resolution at arbitrarily large distances.

We do not require precise positional resolution at large distances. Theorem 1 shows that no relative encoding can keep almost all keys both resolved and within the trained range, so resolution must be sacrificed to achieve long-context capability. Log-Ratio RoPE gives up at large distances, where keys remain within the trained range but are resolved less finely, while the sliding-window RoPE layers maintain fine resolution for nearby tokens. The experiments support both parts of this design. In Sec. 4, the NoPE–NoPE hybrid, without RoPE in its sliding-window layers, fails at the training length, while the RoPE–NoPE hybrid, which removes RoPE only from the global layers, recalls far more than the RoPE–RoPE hybrid past it (41% against 7% at 64K). New runs also show why positional resolution matters. With Log-Ratio RoPE in every layer and no sliding-window RoPE layers, 8-pair recall is lower than all-layer RoPE already within the training length (66.1% against 73.2% at 4K; Appendix G.3, Fig. 14), and the NoPE model shows a similar drop (67.5%).

5a. The choice of baseline architecture seems somewhat outdated. Qwen2 is a relatively old architecture, and some components commonly used in more recent architectures, such as QK-Norm, are absent from Qwen2.

We primarily use Qwen2 because it is a widely recognized transformer model architecture; more formally, in other similar works, such a model architecture would be called "Transformer++". In our original submission, we explicitly state in our training details that we add QK-Norm to our Qwen2 model architecture. In our revised version, we added the phrase "including additional changes to Qwen2's architecture" in Sec. 8 to clarify for the reader.

5b. Since the results on short-context benchmarks are nearly identical across different methods, and short-context performance is not the main focus of this paper, the proposed method is effectively evaluated on only one long-context benchmark, RULER. I suggest that the authors refer to the original RoPE paper and include additional experiments, such as reporting training loss.

We thank the reviewer for suggesting that we refer to the original RoPE paper for additional experiments. Upon your request, we have added the training loss of the Qwen2 model runs in Appendix H.3 (Fig. 22). Additionally, we would like to point out that the original RoPE paper does not directly measure any long-context capability. Our training loss curves for both model sizes show that when the weight initialization and data order seeds are the same across different model runs, we get almost identical training loss curves, which does not help us measure long-context performance, since identical training loss curves can have different long-context performance.

Paper Update: Appendix H.3 (Fig. 22).

Summary of paper changes:

  • Title now names the two architectural choices we study: "Science of Long Context: How Positional Encoding and Hybrid Attention Shape Extrapolation".
  • Abstract and Sec. 1 narrowed to two architectural choices: positional encoding and how local and global attention layers are combined.
  • Log-Ratio RoPE on all full-attention layers: Appendix G.3 (Fig. 14).
  • Sec. 8 now points to the changes to Qwen2's architecture in Appendix A.
  • Training loss of the 0.5B and 1.5B models: Appendix H.3 (Fig. 22).

Thank you for taking the time to review our paper and helping us further improve it.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

We thank all reviewers for their feedback. Below we summarize the changes to the paper. Section and figure numbers refer to the revised version.

New experiments

  • 1.5B models. We trained RoPE, the RoPE–NoPE hybrid and the Log-Ratio hybrid at 1.5B parameters and a 4K training length, with the same hyperparameters as the 0.5B models and no retuning. We also post-trained the 1.5B RoPE model with YaRN. Past the training length, the Log-Ratio hybrid again has the highest RULER accuracy from 16K to 64K. At 32K, eight times the training length, it improves on RoPE by 28.7 points and on the RoPE–NoPE hybrid by 10.2 points (0.5B: 18.7 and 8.1 points at 64K). Short-context accuracy does not drop at either scale. (Abstract, Sec. 1, Sec. 8, Fig. 6, Appendix A, Table 1, Appendix H.2, Table 2.)
  • Causal test of unseen offsets in RoPE. We shift the positions of every token after a key-value pair, the query included, so the pair moves to unseen offsets while the number of keys stays at 8K. Recall falls almost as fast as in a real longer context. Most of RoPE's drop past the training length therefore comes from unseen offsets, not from additional keys. (Sec. 3, Fig. 3, Appendix C.2.)
  • Log-Ratio hybrid at trained and OOD offsets. The single-key test of Sec. 3–4 for the Log-Ratio hybrid, and the share of keys outside the trained range for RoPE and Log-Ratio RoPE. (Sec. 6, Fig. 5, Appendix G.1.)
  • Linear-map control. The Log-Ratio hybrid with the same frequency ladder and per-head scale but a linear position map fails past the training length like RoPE. The gain comes from the log map. (Appendix G.7, Fig. 19.)
  • Log-Ratio RoPE in every layer. Without sliding-window RoPE layers, recall is lower than RoPE already within the training length, which shows why the local layers need RoPE's positional resolution. (Appendix G.3, Fig. 14.)
  • Fixed distance at later positions. Recall when the pair and the query keep the same distance but move to later positions. (Appendix G.2, Fig. 13.)
  • Training seeds and 128K. The three small hybrids trained with three seeds and evaluated up to 128K. The Log-Ratio hybrid stays above the RoPE–NoPE hybrid past the training length in all seeds. (Appendix H.5, Fig. 23, right.)
  • RePo baseline. RePo trained with full attention and as the global layers of a hybrid. (Appendix H.4, Fig. 23, left; related work in Appendix I.)
  • Per-task RULER. All 13 RULER tasks at 0.5B and 1.5B. (Appendix H.1, Figs. 20 and 21.)
  • Training loss. Training loss of all 0.5B and 1.5B models. (Appendix H.3, Fig. 22.)
  • RoPE at other training lengths. The trained vs OOD offset gap appears for RoPE models trained at 1K, 8K, 16K and 32K. (Appendix C.1, Fig. 8.)

Theory

  • Positional resolution (Sec. 5.3). The worst-case logit gap rmr_m is replaced by the share of unresolved keys ψS(m)\psi_S(m). A key is unresolved if it is closer in Δ\Delta to another key than any two distinct values of Δ\Delta seen in training. Both properties are now measured against what the model saw in training: length generalization by the share of keys outside the trained range (ϕS\phi_S), and resolution by the share of unresolved keys (ψS\psi_S).
  • Limitations (Appendix J). Theorem 1 covers only the positional input to attention; retrieval also depends on the content of the keys.

Title

  • We have updated the title to "Science of Long Context: How Positional Encoding and Hybrid Attention Shape Extrapolation". The subtitle names the two architectural choices we study: the positional encoding and how local and global attention layers are combined. It also names the property we measure, extrapolation beyond the training length.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.