acceptodds
Under review as a conference paper at ICLR 2027

MoPE: Learning One Positional Structure for Short and Long Contexts

Abstract

Positional encoding is central to Transformer language models, yet explicit positional bias is not equally beneficial across attention components. Recent hybrid designs combine rotary position embeddings (RoPE) with no positional encoding (NoPE), but typically prescribe their allocation through fixed rotary fractions or layer schedules. We introduce Mixture of Position Embeddings (MoPE), a simple framework that learns this allocation from data. MoPE jointly optimizes the model backbone and binary gates that select RoPE or NoPE for each layer and rotary frequency. At inference, these gates define a fixed positional structure without input-dependent routing. The resulting gate configurations exhibit distinct layer- and frequency-dependent patterns, revealing heterogeneous positional requirements within Transformers. Experiments on models ranging from 70M to 1B parameters show that MoPE maintains competitive short-context performance while achieving strong long-context retrieval. After 4K pretraining and context extension to 64K, the 410M and 1B models achieve 94.9% and 98.8% NIAH accuracy at 128K, respectively. Combined with YaRN, the 410M model further achieves 99.1% NIAH accuracy at 256K.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLMs, Positional Encoding, Rotary Position Embedding, Long-Context Language Modeling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_XVfvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MoPE, which learns whether to use RoPE or NoPE for each layer and rotary frequency pair. The gates and backbone are trained together during short-context pretraining. The gates are then fixed during context extension and inference. The paper evaluates models from 70M to 1B on short-context tasks, long-context perplexity, and NIAH retrieval. It also studies the learned gate patterns and how different training choices affect the results.

Critical strengths
  1. The method provides a simple way to learn where RoPE is needed instead of choosing the allocation manually. The gates are fixed after training, so there is no input-dependent routing during inference. Each gate selects either a RoPE rotation or the identity operation, avoiding the magnitude changes caused by continuously mixing the two. I find this a useful extension of fixed partial-RoPE designs.

  2. Figure 1 shows a clear improvement in long-context retrieval for the 410M model. Although the model is trained up to 64K, MoPE keeps high NIAH accuracy at longer contexts, where the other methods show larger drops. At 128K, MoPE reaches 94.9%, while the best baseline reaches 85.2%. This is a clear benefit when the input is longer than the training context.

Critical weaknesses
  1. The NIAH results are good, but it is still unclear how much MoPE helps on other long-context tasks. The short-context scores and perplexity results are mostly close to the stronger baselines, and MoPE is not always better. For example, at 1B and 128K, FoPE has higher NIAH accuracy and lower PG19 perplexity than MoPE. The YaRN experiments also show perplexity improvements, so the benefit is not limited to NIAH. However, the long-context evaluation mainly covers perplexity and finding a single piece of information in a long input. These results do not yet show whether the improvement carries over to more demanding long-context tasks.
Questions for authors
  1. Could you compare MoPE with FoPE or other strong baselines when several facts need to be retrieved from the same context? This would help check whether the improvement extends beyond single-needle retrieval.

  2. Do the RoPE-ID results with and without temperature scaling use the same trained checkpoint? Please explain how scaling is applied to short inputs and why the scaled 1B model has 0% NIAH accuracy at 4K in Table 2.

6 comments

Dear reviewer XVfv:

Thank you for your valuable feedback on multi-needle reasoning and the RoPE-ID comparison. We provide additional results and clarify the experimental settings below and will revise the manuscript accordingly.

1. Could you compare MoPE with FoPE or other strong baselines when several facts need to be retrieved from the same context? This would help check whether the improvement extends beyond single-needle retrieval.

We have added a multi-needle reasoning evaluation for the 64K-trained 1B models, adapted from NeedleBench (Li et al., 2025). The task requires models to retrieve supporting statements from different positions in a long context and combine them to answer a question. We evaluate nine nominal context lengths from 4K to 128K tokens, using the same questions across models and lengths. The first supporting statement is inserted at approximately 25% depth, with subsequent statements spaced by approximately 10% of the context length. We evaluate 30 samples at each context length and report keyword-based scores.

Below is an actual example from our evaluation set, using a Paul Graham essay as the haystack. Unrelated essay text is omitted for clarity.

Haystack: Paul Graham essay text ... Needle 1, at 24.9% depth: “Jessica Alba is associated with The Honest Company.” ... more essay text ... Needle 2, at 35.4% depth: “The Honest Company was valued at $1.7 billion as of August 2015.” ... more essay text ... Question: “How much is the company that Jessica Alba is associated with worth as of 2015?”

MoPE achieves the highest score at seven of the nine evaluated context lengths, including 80K, 96K, and 112K. At 112K, MoPE scores 7.59, compared with 4.44 for the strongest baseline. These results suggest that MoPE's benefits extend beyond single-needle retrieval to tasks requiring the integration of multiple pieces of evidence. However, MoPE does not lead at 64K or 128K.

All models achieve low absolute scores on this task. This may partly be because they are base models pretrained on limited data without supervised fine-tuning (SFT), which could limit their ability to follow instructions and reason across multiple supporting statements.

Table R.1. Multi-needle reasoning results for 64K-trained 1B models, averaged over three runs.

Method4K8K16K32K64K80K96K112K128K
Full RoPE9.077.044.264.443.983.152.222.781.67
Partial RoPE 50%8.068.244.815.654.545.652.871.942.04
Partial RoPE 25%8.528.066.115.935.743.702.693.334.54
CoPE (RoPE init)9.447.044.633.331.392.502.221.481.67
HoPE8.703.895.093.433.333.152.044.441.94
FoPE8.436.394.265.932.592.963.433.432.04
RoPE-ID (scaled)1.112.044.074.265.652.593.703.805.00
RoPE-ID (no scale)10.935.836.945.656.392.314.813.151.39
MoPE12.4119.6311.579.175.938.336.947.594.26

Li, M., Zhang, S., Zhang, T., Duan, H., Liu, Y., & Chen, K. (2025). NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densities. Trans. Mach. Learn. Res., 2025.

2. Do the RoPE-ID results with and without temperature scaling use the same trained checkpoint? Please explain how scaling is applied to short inputs and why the scaled 1B model has 0% NIAH accuracy at 4K in Table 2.

The RoPE-ID (scaled) and RoPE-ID (no scale) results reported in Table 2 of the submitted manuscript use different final trained checkpoints. For each run, we initialized both variants from the same 4K checkpoint and continued pretraining them separately at 32K and 64K. We enabled temperature scaling during both continued pretraining and evaluation for the scaled variant and used an attention-logit multiplier of 1 throughout for the no-scale variant.

We followed Codeblock 1 in Appendix A.4 of the original RoPE-ID paper and kept the reference context length fixed at 4096 tokens during both continued pretraining and evaluation. We computed the attention-logit multiplier as

s(L)=[1+0.1ln⁡(max⁡(L,4096)4096)]2,s(L)=\left[1+0.1\ln\left(\frac{\max(L,4096)}{4096}\right)\right]^2,

where LL is the current key-sequence length. This gives multipliers of approximately 1.459 and 1.631 for 32K and 64K training sequences, respectively. During evaluation, the multiplier depends on the current key-sequence length and equals 1 when L≤4096L\le4096.

We hypothesize that the scaled model co-adapts to the larger attention-logit multiplier used during long-context training, potentially through changes in query/key norms and attention entropy. At 4K evaluation, the multiplier returns to 1, removing the additional logit sharpening used during training. This training–inference mismatch may impair short-input retrieval and explain the observed collapse to 0% NIAH accuracy.

To address your question about the 0% NIAH accuracy at 4K, we additionally evaluated the same 64K-trained 1B checkpoint from the scaled setting with the inference multiplier fixed at 1.63139, matching its value during the 64K training stage. The 4K NIAH accuracy increased from 0% to 100%. Below, we provide the settings and full results of this additional evaluation, together with a second setting that tests scaling only beyond the 64K training window.

RoPE-ID (scaled training, fixed inference scaling). This is the setting used in the recovery experiment above. We reused the scaled-training checkpoints and fixed the inference multiplier at 1.63139 for all evaluation lengths to examine its effect on both short-input retrieval and length extrapolation.

RoPE-ID (no scale + scaling beyond 64K). We also tested a modified temperature-scaling rule by replacing the original 4096-token reference with the final training context length of 65,536 tokens. We reused the checkpoint trained without scaling and applied the following multiplier during inference:

s64K(L)=[1+0.1ln⁡(max⁡(L,65536)65536)]2.s_{\mathrm{64K}}(L)=\left[1+0.1\ln\left(\frac{\max(L,65536)}{65536}\right)\right]^2.

Tables R.2 and R.3 below report these additional results alongside RoPE-ID (scaled), RoPE-ID (no scale), and MoPE for both the 410M and 1B models.

Fixing the inference multiplier restores 100% NIAH accuracy at 1K and 4K using the same scaled-training checkpoints, but lowers 128K accuracy to 21.23% for 410M and 77.87% for 1B.

RoPE-ID (no scale + scaling beyond 64K), listed as method 4 in the tables, provides a stronger baseline for comparison with MoPE by preserving within-window performance while improving 128K accuracy from 2.13% to 91.53% for 410M and from 79.70% to 97.27% for 1B. MoPE still outperforms all four RoPE-ID settings at every tested length beyond 64K for both model sizes, reaching 94.87% and 98.80% at 128K for 410M and 1B, respectively.

Table R.2. Single-needle NIAH accuracy (%) for 64K-trained 410M models, averaged over three runs.

Method1K4K8K16K24K32K48K64K80K96K112K128K
1RoPE-ID (scaled)20.307.5724.2346.9761.5078.5093.9396.3795.7792.4084.8385.17
2RoPE-ID (scaled training, fixed inference scaling)100.00100.0097.8799.4097.9095.1397.9096.3790.3375.4350.0021.23
3RoPE-ID (no scale)100.00100.0099.1099.4098.5097.5798.2095.4391.2075.7729.072.13
4RoPE-ID (no scale + scaling beyond 64K)100.00100.0099.1099.4098.5097.5798.2095.4397.6097.3094.8791.53
5MoPE100.00100.00100.00100.00100.00100.00100.00100.00100.00100.0099.1094.87

Table R.3. Single-needle NIAH accuracy (%) for 64K-trained 1B models, averaged over three runs.

Method1K4K8K16K24K32K48K64K80K96K112K128K
1RoPE-ID (scaled)0.300.003.6349.1075.1788.1796.9793.9797.8794.8794.5396.97
2RoPE-ID (scaled training, fixed inference scaling)100.00100.00100.00100.0099.7099.7098.2093.9793.9388.4779.7077.87
3RoPE-ID (no scale)100.00100.00100.0099.1098.8099.1098.2098.5096.0792.1383.6379.70
4RoPE-ID (no scale + scaling beyond 64K)100.00100.00100.0099.1098.8099.1098.2098.5097.5798.2097.2797.27
5MoPE100.00100.00100.0099.70100.00100.00100.0099.70100.0099.70100.0098.80
Reviewer_D6wGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MoPE, which learns layer- and frequency-wise RoPE/NoPE allocation through trainable gates. Experiments up to 1B parameters show competitive short-context performance and strong long-context retrieval, with analysis of the learned routing patterns.

Critical strengths
  1. Simple and practical design. Learns positional encoding allocation with negligible inference overhead.

  2. Strong empirical results. Consistent improvements in long-context retrieval across model scales, with interesting learned layer/frequency patterns.

Critical weaknesses
  1. Limited long-context evaluation. Mainly single-needle NIAH; lacks more challenging retrieval and reasoning tasks.

  2. Missing baselines. No Full NoPE or random fixed masks to isolate the benefit of learned allocation.

Questions for authors
  1. Can the authors add multi-needle retrieval or multi-hop QA evaluations?

  2. Can the authors add Full NoPE and random fixed-mask baselines with matched RoPE/NoPE ratios?

3 comments

Dear reviewer D6wG:

Thank you for your valuable suggestions on additional evaluations and baselines. We present the new results and address your questions below and will revise the manuscript accordingly.

1. Can the authors add multi-needle retrieval or multi-hop QA evaluations?

We have added a multi-needle reasoning evaluation for the 64K-trained 1B models, adapted from NeedleBench (Li et al., 2025). The task requires models to retrieve supporting statements from different positions in a long context and combine them to answer a question. We evaluate nine nominal context lengths from 4K to 128K tokens, using the same questions across models and lengths. The first supporting statement is inserted at approximately 25% depth, with subsequent statements spaced by approximately 10% of the context length. We evaluate 30 samples at each context length and report keyword-based scores.

Below is an actual example from our evaluation set, using a Paul Graham essay as the haystack. Unrelated essay text is omitted for clarity.

Haystack: Paul Graham essay text ... Needle 1, at 24.9% depth: “Jessica Alba is associated with The Honest Company.” ... more essay text ... Needle 2, at 35.4% depth: “The Honest Company was valued at $1.7 billion as of August 2015.” ... more essay text ... Question: “How much is the company that Jessica Alba is associated with worth as of 2015?”

MoPE achieves the highest score at seven of the nine evaluated context lengths, including 80K, 96K, and 112K. At 112K, MoPE scores 7.59, compared with 4.44 for the strongest baseline. These results suggest that MoPE's benefits extend beyond single-needle retrieval to tasks requiring the integration of multiple pieces of evidence. However, MoPE does not lead at 64K or 128K.

All models achieve low absolute scores on this task. This may partly be because they are base models pretrained on limited data without supervised fine-tuning (SFT), which could limit their ability to follow instructions and reason across multiple supporting statements.

Table R.1. Multi-needle reasoning results for 64K-trained 1B models, averaged over three runs.

Method4K8K16K32K64K80K96K112K128K
Full RoPE9.077.044.264.443.983.152.222.781.67
Partial RoPE 50%8.068.244.815.654.545.652.871.942.04
Partial RoPE 25%8.528.066.115.935.743.702.693.334.54
CoPE (RoPE init)9.447.044.633.331.392.502.221.481.67
HoPE8.703.895.093.433.333.152.044.441.94
FoPE8.436.394.265.932.592.963.433.432.04
RoPE-ID (scaled)1.112.044.074.265.652.593.703.805.00
RoPE-ID (no scale)10.935.836.945.656.392.314.813.151.39
MoPE12.4119.6311.579.175.938.336.947.594.26

Li, M., Zhang, S., Zhang, T., Duan, H., Liu, Y., & Chen, K. (2025). NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densities. Trans. Mach. Learn. Res., 2025.

2. Can the authors add Full NoPE and random fixed-mask baselines with matched RoPE/NoPE ratios?

We have added Full NoPE and random fixed-mask baselines for both model sizes. Alongside 25% and 50% RoPE, the random-mask baselines include Random74% for 410M and Random60% for 1B, matching MoPE's global RoPE/NoPE ratios. The percentages denote the model-wide fraction assigned to RoPE. Each random mask is sampled once before training and kept fixed throughout training and evaluation. The results below are averaged over three runs using 64K-trained checkpoints.

At 128K, Full NoPE reaches 0.00% and 21.83% NIAH accuracy for 410M and 1B, respectively, and all random fixed-mask baselines reach 0%, while MoPE achieves 94.87% and 98.80%. For the matched-ratio random baselines, PG-19 perplexity at 128K is 21.15 for 410M and 20.32 for 1B, compared with 16.37 and 16.22 for MoPE.

Among random fixed masks, increasing the RoPE fraction improves NIAH accuracy at 112K for both model sizes, although trends vary across evaluation lengths and metrics. One possible explanation is that higher RoPE fractions preserve useful positional signals during training that random masks with lower fractions may remove, helping the backbone learn relationships needed for retrieval. Even with global RoPE fractions matched to MoPE, random fixed masks fail to reproduce its length-extrapolation performance.

Table R.2. Short-context evaluation and single-needle NIAH accuracy (%) for 64K-trained 410M models, averaged over three runs.

MethodVal. lossWikiText PPLAverageNIAH 4KNIAH 8KNIAH 16KNIAH 32KNIAH 64KNIAH 80KNIAH 96KNIAH 112KNIAH 128K
Full NoPE3.09238.1839.9195.1792.7383.3771.5058.2032.433.630.000.00
Partial RoPE 25%2.81729.4142.0299.1098.50100.0099.7095.7380.2714.530.000.00
Partial RoPE 50%2.80328.7642.6197.8798.5097.5797.8793.9384.8358.173.030.00
Full RoPE2.81529.8442.2597.2796.0795.1793.3754.8342.1033.039.100.60
Random25%2.81729.4642.57100.00100.0096.9795.4390.6379.7013.070.000.00
Random50%2.80528.9543.0799.7099.4099.1094.8391.8370.9050.630.300.00
Random74%2.80129.3242.31100.00100.00100.0095.0084.1070.0052.756.800.00
MoPE2.80329.2043.42100.00100.00100.00100.00100.00100.00100.0099.1094.87

Table R.3. PG-19 perplexity for 64K-trained 410M models, averaged over three runs.

MethodPG-19 2KPG-19 4KPG-19 8KPG-19 16KPG-19 32KPG-19 64KPG-19 128K
Full NoPE21.3421.6121.7221.4321.0221.1545.02
Partial RoPE 25%15.4916.0516.3216.1615.8915.9541.46
Partial RoPE 50%15.2415.8016.0615.9115.6615.7221.87
Full RoPE15.2715.8516.1616.0515.8516.0019.92
Random25%15.5216.0716.3216.1515.8815.9429.59
Random50%15.2915.8216.1015.9615.7015.7624.79
Random74%15.2115.7616.0415.8915.6615.7421.15
MoPE15.2115.7816.0615.9215.6715.7216.37

Table R.4. Short-context evaluation and single-needle NIAH accuracy (%) for 64K-trained 1B models, averaged over three runs.

MethodVal. lossWikiText PPLAverageNIAH 4KNIAH 8KNIAH 16KNIAH 32KNIAH 64KNIAH 80KNIAH 96KNIAH 112KNIAH 128K
Full NoPE3.21342.0739.0276.9375.1766.9757.9045.4335.1732.4323.6321.83
Partial RoPE 25%2.77427.5943.96100.00100.00100.00100.00100.00100.00100.0018.800.00
Partial RoPE 50%2.77027.6143.68100.00100.00100.00100.00100.00100.00100.0018.770.00
Full RoPE2.77027.6243.73100.00100.00100.00100.0099.7097.6085.4312.400.00
Random25%2.76827.3043.63100.00100.00100.00100.00100.00100.0098.500.300.00
Random50%2.76427.3343.77100.00100.00100.00100.00100.00100.0097.9012.730.00
Random60%2.76927.2543.78100.00100.00100.00100.00100.00100.0099.5529.100.00
MoPE2.76527.3644.15100.00100.0099.70100.0099.70100.0099.70100.0098.80

Table R.5. PG-19 perplexity for 64K-trained 1B models, averaged over three runs.

MethodPG-19 2KPG-19 4KPG-19 8KPG-19 16KPG-19 32KPG-19 64KPG-19 128K
Full NoPE25.3625.2425.0724.5924.0924.4228.51
Partial RoPE 25%15.1115.6515.9015.7515.4915.5421.55
Partial RoPE 50%15.0415.5715.8315.6815.4215.4719.51
Full RoPE14.9515.5015.7915.6615.4315.5118.75
Random25%15.0015.5515.8115.6615.4015.4422.92
Random50%14.9115.4615.7315.5815.3315.3819.92
Random60%14.9915.5315.8115.6715.4115.4720.32
MoPE14.9315.4715.7515.6115.3515.3916.22
Reviewer_Lv96OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a mechanism to let the model learn whether to use rope or nope instead of fixing it at the beginning of training. To prevent overfitting to initial choices of the gates, they use STE gumbel-softmax to decide between rope or nope but apply a higher learning rate to let the gates stabilizie quickly. The result suggest better performance across different context lengths, showing improvement both at train length and at extrapolation. This is demonstrated across model scales and datasets and compared with several other existing methods proposed for improving length extrapolation.

Critical strengths

The idea is interesting, the experimentation setting is sound. There is a good set of baselines and comparison is done across scales and several tasks.

Critical weaknesses

One concern is that Full NoPE as a baseline is currently missing. From the Partial RoPE results 25% > 50% > 100% so it's very reasonable to also consider 0%. Given that MoPE is doing an interpolation between RoPE and NoPE, this can potentially explain its ´better performance though optimally MoPE should be better than NoPE.

It is unclear how the gate learning rate multiplier is chosen. Also, while there is the risk that if the gates stabilize too late, it hurts the training, there is also another concern that if they stabilize too quickly, this happens before the model learns strong attention parameters.

Questions for authors
  1. Figure 2 suggests the pattern of gates is very similar across scales. Can the pattern be learned once and then fixed to be used across models?

  2. What is the intuition that this helps extrapolation? What incentivizes the model to learn a pattern that is extrapolatable? Is this just luck? If so, is it possible that it doesn't extend to other variants of Transformer (not Pythia)?

  3. For evals and training of RoPE-ID in table 2, what base was used during training and eval?

6 comments

Dear reviewer Lv96:

Thank you for your valuable comments on MoPE's design and extrapolation behavior. We address your questions below and will revise the manuscript accordingly.

1. Figure 2 suggests the pattern of gates is very similar across scales. Can the pattern be learned once and then fixed to be used across models?

The shared patterns in Figure 2 can guide the design of fixed positional structures, such as retaining RoPE in the first layer, allocating a higher proportion of NoPE to some of the subsequent layers, and selecting which rotary dimensions should preserve positional information. However, similar overall patterns do not imply identical gate configurations. Models can differ in both layer count and head dimension, so transferring a learned mask requires an appropriate mapping across depth and frequency dimensions. Direct reuse across model scales has not been evaluated in our current experiments. Our MoPE-Fixed experiment provides evidence that the learned structure is useful even when fixed from the beginning of training. Within the same 410M architecture, training a fresh backbone with the learned mask achieves a NIAH OOD Avg. of 72.58, exceeding Full RoPE’s 21.21 but remaining below jointly trained MoPE’s 98.49. This suggests that both the final allocation and the training process through which the gates and backbone co-adapt contribute to extrapolation. Thus, the observed patterns offer useful guidance for fixed designs, but learning a pattern once and reusing it across models cannot yet be assumed to preserve the full benefits of MoPE.

2. What is the intuition that this helps extrapolation? What incentivizes the model to learn a pattern that is extrapolatable? Is this just luck? If so, is it possible that it doesn't extend to other variants of Transformer (not Pythia)?

Our intuition is that MoPE’s extrapolation benefits come from both the learned gate configuration and the training process:

  1. MoPE learns to allocate NoPE and RoPE across layers and rotary dimensions to accommodate both positional and semantic information, an allocation that remains useful as the context grows. RoPE provides explicit positional information, while NoPE provides a semantic channel without explicit rotary positional encoding. Even within a 4K window, next-token prediction requires both position-sensitive interactions and content-based associations. MoPE learns a routing mechanism that allows different layers to play distinct roles in modeling positional relationships and integrating semantic information. During long-context continued pretraining, the gates remain fixed, while the remaining model weights continue adapting to longer sequences. This adaptation refines how each layer represents, matches, and integrates information within the learned RoPE and NoPE allocation, allowing the same positional structure to support dependencies over greater distances.

  2. Initializing the MoPE gate with NoPE encourages the backbone to capture global dependencies and integrate information across the available context early in training, without an explicit rotary positional prior. Our co-adaptation ablations support the importance of this process: using the final mask from the beginning of training does not reproduce the full extrapolation benefit.

We have extended MoPE to SmolLM-360M, a model from a distinct Llama-family architecture. The SmolLM authors describe the 360M model as a depth-oriented design with grouped-query attention (GQA). Its published configuration has 32 layers, 15 query heads and 5 key/value heads, a 960-dimensional hidden state, SiLU activation, and tied input/output embeddings. By comparison, Pythia-410M is an MHA model using GPT-NeoX with 24 layers, 16 attention heads, a 1,024-dimensional hidden state, GELU activation, and untied embeddings. Thus, the experiment changes the backbone and attention-head organization, as well as the native tokenizer. We compare MoPE with a Full RoPE baseline on single-needle NIAH using SmolLM-360M. We use the same training data and training setup as in our Pythia-410M experiments, progressively increasing the training context length from 4K to 32K and then to 64K. Tables R.1 and R.2 report mean NIAH accuracy over three independent runs. MoPE maintains strong retrieval accuracy beyond the training context length, providing evidence that its extrapolation gains on single-needle NIAH extend to a different Transformer architecture.

Table R.1. Single-needle NIAH accuracy (%) on SmolLM-360M trained with a 32K context window, averaged over three runs.

Method4K8K16K24K32K48K64K
Full RoPE100.0100.098.296.794.875.558.5
MoPE100.0100.0100.099.799.499.798.5

Table R.2. Single-needle NIAH accuracy (%) on SmolLM-360M trained with a 64K context window, averaged over three runs.

Method4K8K16K32K48K64K80K96K112K128K
Full RoPE100.0100.099.797.996.193.087.680.368.856.4
MoPE100.0100.0100.099.4100.096.498.296.194.295.2

3. For evals and training of RoPE-ID in table 2, what base was used during training and eval?

Following the original paper, we apply RoPE to 50% of the channels in each attention head, with rotary wavelengths geometrically spaced between 32 and 2048 tokens. These frequencies are determined directly by the wavelength endpoints, so the conventional RoPE base θ\theta is not used in their construction. In the original RoPE-ID parameterization, base = 32 specifies the shortest rotary wavelength. We keep this frequency configuration fixed throughout the 4K, 32K, and 64K training stages and use the same frequencies during evaluation.

If by “base” you mean the reference context length used for temperature scaling, we used 4096 tokens for the RoPE-ID (scaled) results in Table 2 of the submitted manuscript. This choice follows Codeblock 1 in Appendix A.4 of the original RoPE-ID paper, which uses a fixed reference length of 4096 tokens. We retained this setting during both continued pretraining and evaluation, including when the training context was extended to 32K and 64K. The scaling multiplier was computed according to the current key-sequence length relative to this fixed reference.

s(L)=[1+0.1ln⁡(max⁡(L,4096)4096)]2,s(L)=\left[1+0.1\ln\left(\frac{\max(L,4096)}{4096}\right)\right]^2,

Here, LL denotes the current key-sequence length, and s(L)s(L) is the multiplier applied to the attention logits.

To examine how inference-time scaling affects short-context retrieval and length extrapolation, we evaluate two additional settings using the existing 64K-trained RoPE-ID checkpoints.

RoPE-ID (scaled training, fixed inference scaling). We reuse the checkpoint trained with scaling and fix the attention-logit multiplier to s(65536)≈1.63139s(65536)\approx1.63139 at every evaluation length. This matches the multiplier used during the 64K training stage. Comparing this setting with the original length-dependent scaling tests whether maintaining the training-time multiplier improves short-context retrieval.

RoPE-ID (no scale + scaling beyond 64K). We further test a modified temperature-scaling rule by replacing the original 4096-token reference with the final training context length of 65,536 tokens. We reuse the checkpoint trained without scaling and apply the following multiplier during inference:

s64K(L)=[1+0.1ln⁡(max⁡(L,65536)65536)]2.s_{\mathrm{64K}}(L)=\left[1+0.1\ln\left(\frac{\max(L,65536)}{65536}\right)\right]^2.

Tables R.3 and R.4 below compare these two settings with RoPE-ID (scaled), RoPE-ID (no scale), and MoPE for both the 410M and 1B models.

Fixing the inference multiplier restores 100% NIAH accuracy at 1K and 4K using the same scaled-training checkpoints, but lowers 128K accuracy to 21.23% for 410M and 77.87% for 1B.

RoPE-ID (no scale + scaling beyond 64K), listed as method 4 in the tables, provides a stronger baseline for comparison with MoPE by preserving within-window performance while improving 128K accuracy from 2.13% to 91.53% for 410M and from 79.70% to 97.27% for 1B. MoPE still outperforms all four RoPE-ID settings at every tested length beyond 64K for both model sizes, reaching 94.87% and 98.80% at 128K for 410M and 1B, respectively.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.