- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MoPE, which learns whether to use RoPE or NoPE for each layer and rotary frequency pair. The gates and backbone are trained together during short-context pretraining. The gates are then fixed during context extension and inference. The paper evaluates models from 70M to 1B on short-context tasks, long-context perplexity, and NIAH retrieval. It also studies the learned gate patterns and how different training choices affect the results.
Critical strengths
-
The method provides a simple way to learn where RoPE is needed instead of choosing the allocation manually. The gates are fixed after training, so there is no input-dependent routing during inference. Each gate selects either a RoPE rotation or the identity operation, avoiding the magnitude changes caused by continuously mixing the two. I find this a useful extension of fixed partial-RoPE designs.
-
Figure 1 shows a clear improvement in long-context retrieval for the 410M model. Although the model is trained up to 64K, MoPE keeps high NIAH accuracy at longer contexts, where the other methods show larger drops. At 128K, MoPE reaches 94.9%, while the best baseline reaches 85.2%. This is a clear benefit when the input is longer than the training context.
Critical weaknesses
- The NIAH results are good, but it is still unclear how much MoPE helps on other long-context tasks. The short-context scores and perplexity results are mostly close to the stronger baselines, and MoPE is not always better. For example, at 1B and 128K, FoPE has higher NIAH accuracy and lower PG19 perplexity than MoPE. The YaRN experiments also show perplexity improvements, so the benefit is not limited to NIAH. However, the long-context evaluation mainly covers perplexity and finding a single piece of information in a long input. These results do not yet show whether the improvement carries over to more demanding long-context tasks.
Questions for authors
-
Could you compare MoPE with FoPE or other strong baselines when several facts need to be retrieved from the same context? This would help check whether the improvement extends beyond single-needle retrieval.
-
Do the RoPE-ID results with and without temperature scaling use the same trained checkpoint? Please explain how scaling is applied to short inputs and why the scaled 1B model has 0% NIAH accuracy at 4K in Table 2.
6 comments
2. Do the RoPE-ID results with and without temperature scaling use the same trained checkpoint? Please explain how scaling is applied to short inputs and why the scaled 1B model has 0% NIAH accuracy at 4K in Table 2.
The RoPE-ID (scaled) and RoPE-ID (no scale) results reported in Table 2 of the submitted manuscript use different final trained checkpoints. For each run, we initialized both variants from the same 4K checkpoint and continued pretraining them separately at 32K and 64K. We enabled temperature scaling during both continued pretraining and evaluation for the scaled variant and used an attention-logit multiplier of 1 throughout for the no-scale variant.
We followed Codeblock 1 in Appendix A.4 of the original RoPE-ID paper and kept the reference context length fixed at 4096 tokens during both continued pretraining and evaluation. We computed the attention-logit multiplier as
where is the current key-sequence length. This gives multipliers of approximately 1.459 and 1.631 for 32K and 64K training sequences, respectively. During evaluation, the multiplier depends on the current key-sequence length and equals 1 when .
We hypothesize that the scaled model co-adapts to the larger attention-logit multiplier used during long-context training, potentially through changes in query/key norms and attention entropy. At 4K evaluation, the multiplier returns to 1, removing the additional logit sharpening used during training. This training–inference mismatch may impair short-input retrieval and explain the observed collapse to 0% NIAH accuracy.
To address your question about the 0% NIAH accuracy at 4K, we additionally evaluated the same 64K-trained 1B checkpoint from the scaled setting with the inference multiplier fixed at 1.63139, matching its value during the 64K training stage. The 4K NIAH accuracy increased from 0% to 100%. Below, we provide the settings and full results of this additional evaluation, together with a second setting that tests scaling only beyond the 64K training window.
RoPE-ID (scaled training, fixed inference scaling). This is the setting used in the recovery experiment above. We reused the scaled-training checkpoints and fixed the inference multiplier at 1.63139 for all evaluation lengths to examine its effect on both short-input retrieval and length extrapolation.
RoPE-ID (no scale + scaling beyond 64K). We also tested a modified temperature-scaling rule by replacing the original 4096-token reference with the final training context length of 65,536 tokens. We reused the checkpoint trained without scaling and applied the following multiplier during inference:
Tables R.2 and R.3 below report these additional results alongside RoPE-ID (scaled), RoPE-ID (no scale), and MoPE for both the 410M and 1B models.
Fixing the inference multiplier restores 100% NIAH accuracy at 1K and 4K using the same scaled-training checkpoints, but lowers 128K accuracy to 21.23% for 410M and 77.87% for 1B.
RoPE-ID (no scale + scaling beyond 64K), listed as method 4 in the tables, provides a stronger baseline for comparison with MoPE by preserving within-window performance while improving 128K accuracy from 2.13% to 91.53% for 410M and from 79.70% to 97.27% for 1B. MoPE still outperforms all four RoPE-ID settings at every tested length beyond 64K for both model sizes, reaching 94.87% and 98.80% at 128K for 410M and 1B, respectively.
Table R.2. Single-needle NIAH accuracy (%) for 64K-trained 410M models, averaged over three runs.
| Method | 1K | 4K | 8K | 16K | 24K | 32K | 48K | 64K | 80K | 96K | 112K | 128K | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | RoPE-ID (scaled) | 20.30 | 7.57 | 24.23 | 46.97 | 61.50 | 78.50 | 93.93 | 96.37 | 95.77 | 92.40 | 84.83 | 85.17 |
| 2 | RoPE-ID (scaled training, fixed inference scaling) | 100.00 | 100.00 | 97.87 | 99.40 | 97.90 | 95.13 | 97.90 | 96.37 | 90.33 | 75.43 | 50.00 | 21.23 |
| 3 | RoPE-ID (no scale) | 100.00 | 100.00 | 99.10 | 99.40 | 98.50 | 97.57 | 98.20 | 95.43 | 91.20 | 75.77 | 29.07 | 2.13 |
| 4 | RoPE-ID (no scale + scaling beyond 64K) | 100.00 | 100.00 | 99.10 | 99.40 | 98.50 | 97.57 | 98.20 | 95.43 | 97.60 | 97.30 | 94.87 | 91.53 |
| 5 | MoPE | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 99.10 | 94.87 |
Table R.3. Single-needle NIAH accuracy (%) for 64K-trained 1B models, averaged over three runs.
| Method | 1K | 4K | 8K | 16K | 24K | 32K | 48K | 64K | 80K | 96K | 112K | 128K | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | RoPE-ID (scaled) | 0.30 | 0.00 | 3.63 | 49.10 | 75.17 | 88.17 | 96.97 | 93.97 | 97.87 | 94.87 | 94.53 | 96.97 |
| 2 | RoPE-ID (scaled training, fixed inference scaling) | 100.00 | 100.00 | 100.00 | 100.00 | 99.70 | 99.70 | 98.20 | 93.97 | 93.93 | 88.47 | 79.70 | 77.87 |
| 3 | RoPE-ID (no scale) | 100.00 | 100.00 | 100.00 | 99.10 | 98.80 | 99.10 | 98.20 | 98.50 | 96.07 | 92.13 | 83.63 | 79.70 |
| 4 | RoPE-ID (no scale + scaling beyond 64K) | 100.00 | 100.00 | 100.00 | 99.10 | 98.80 | 99.10 | 98.20 | 98.50 | 97.57 | 98.20 | 97.27 | 97.27 |
| 5 | MoPE | 100.00 | 100.00 | 100.00 | 99.70 | 100.00 | 100.00 | 100.00 | 99.70 | 100.00 | 99.70 | 100.00 | 98.80 |
Dear reviewer XVfv:
Thank you for your valuable feedback on multi-needle reasoning and the RoPE-ID comparison. We provide additional results and clarify the experimental settings below and will revise the manuscript accordingly.
1. Could you compare MoPE with FoPE or other strong baselines when several facts need to be retrieved from the same context? This would help check whether the improvement extends beyond single-needle retrieval.
We have added a multi-needle reasoning evaluation for the 64K-trained 1B models, adapted from NeedleBench (Li et al., 2025). The task requires models to retrieve supporting statements from different positions in a long context and combine them to answer a question. We evaluate nine nominal context lengths from 4K to 128K tokens, using the same questions across models and lengths. The first supporting statement is inserted at approximately 25% depth, with subsequent statements spaced by approximately 10% of the context length. We evaluate 30 samples at each context length and report keyword-based scores.
Below is an actual example from our evaluation set, using a Paul Graham essay as the haystack. Unrelated essay text is omitted for clarity.
MoPE achieves the highest score at seven of the nine evaluated context lengths, including 80K, 96K, and 112K. At 112K, MoPE scores 7.59, compared with 4.44 for the strongest baseline. These results suggest that MoPE's benefits extend beyond single-needle retrieval to tasks requiring the integration of multiple pieces of evidence. However, MoPE does not lead at 64K or 128K.
All models achieve low absolute scores on this task. This may partly be because they are base models pretrained on limited data without supervised fine-tuning (SFT), which could limit their ability to follow instructions and reason across multiple supporting statements.
Table R.1. Multi-needle reasoning results for 64K-trained 1B models, averaged over three runs.
Li, M., Zhang, S., Zhang, T., Duan, H., Liu, Y., & Chen, K. (2025). NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densities. Trans. Mach. Learn. Res., 2025.