acceptodds
Under review as a conference paper at ICLR 2027

Delayed Positional Adjustment for Context Extension

Abstract

Extending a pretrained language model to longer inputs changes two things at once: the training sequences grow longer, and the rotary position embedding (RoPE) frequencies are rescaled to a target such as position interpolation (PI) or YaRN, usually from the first update. We ask when to introduce the target frequencies. We introduce delayed positional adjustment (DEPA), which trains at the new length with the original frequencies, switches once to the target after 90% of the continuation updates, and adapts for the remaining 10%. We compare DEPA with static adjustment, which applies the target from the first update, using pairs matched on the pretrained model, final frequencies, token budget, and optimizer schedule. On 137M-parameter models extended from 1K to 4K tokens on PG19, DEPA lowers perplexity (PPL) beyond the trained window in every tested pair. Under our custom YaRN-family target (a YaRN variant with our own frequency mask), whose static runs already extrapolate well, five constant-learning-rate pairs show a reduction of about 1.9 in 16K-token PPL at a 0.03-PPL cost at 4K. Under PI and published YaRN, DEPA lowers 16K PPL from about 105–190 to below 32. The gain persists on a second pretrained base, on WikiText-103, for a 20-layer model, and at 19M parameters. When fixed target tokens are scored with progressively longer inputs, DEPA reduces added-context harm by 78–80% on PG19. Release timing is a consequential, easily controlled choice that context-extension recipes should set and report.

Primary area:
foundation or frontier models, including LLMs
Keywords:
rotary position embeddings, context extension

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_D3HxOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The work explores the idea that RoPE context length extension via position interpolation (PI) or YaRN fine-tuning should be done in two phases: first, train on longer context without change, then change the RoPE frequencies and temperature for the remainder of the extension training, typically 10%. This appears to be a novel idea with the potential to improve outcomes at no additional cost. The experiments provided show significantly improved validation-set perplexity within the region 2-4x beyond a 4x finetuned length, but mildly worse perplexity at 1x the finetuned length.

Critical strengths

The strongest part of the paper is the experimental evidence of reduced perplexity at 2-4x beyond a 4x finetuned length. This is the fundamental result, and supports the idea that a period of training at longer context with the model held unchanged is important prior to applying other extension methods such as PI or YaRN. The multiple seeds and ranges shown are helpful demonstrating confidence within the limitations of the experimental regime.

Critical weaknesses

The weakest parts of the paper are the quality of the writing, lack of comparative evidence about what happens within the original model context length through the finetuned length, the small scale and training dataset of the experiments, and the lack of retrieval and context specific benchmarks.

The paper is dense with numerical information, and is exceptionally difficult to read, idiomatic of common LLM output. The abstract uses terminology (e.g. "release") only somewhat defined or implicitly stated later in the paper itself; one should not be required to read the paper to understand the abstract. The 53 pages of appendices are quite challenging to approach. Such a large volume of information needs exceptional description, organization and summarization, which are not present.

While the evidence of ppl improvement at 8x and 16x context is intriguing, the ppl is mildly worse at the end of the 4k finetuned context length. This only emphasizes the need for evaluation at shorter lengths to see whether or not the method damages the model more at those lengths than YaRN or PI would.

As admitted in the limitations section and Appendix S, the scale of the experiments are extremely small, and are trained on very little data. Much smaller, indeed, than the experiments shown in YaRN paper upon which the method is based. Furthermore, the 137M model is pretrained at 1k context on PG19, which I view as an inappropriate corpus for this context length. If you examine this corpus, the beginning of many of the documents are wildly different than the expected contents later on, and we have found it is an inconsistent testing/training ground at best for important retrieval and context tests. I recommend, at a minimum, testing the method on the same models as used in the YaRN paper with similar context lengths. If you are looking for a different long context dataset, ProLong may be of use. There may also be some smaller modern well trained models available you could use for more extensive evaluations if compute is a major hurdle.

Finally, as admitted in Appendix S, the current lack of downstream benchmarks unfortunately dramatically limits the scope of utility for this research, making it indicative that there may be something of interest related to the method, but stopping short of indicating that it could be used effectively in practice for real world model inference.

Questions for authors

Further explanation of the described geometric motivation in the introduction and section 3 would be useful. I am unclear on exactly what is meant about the hold training "on the angle range that 16K inference will use, whereas the static arm must extrapolate fourfold in angle". There seems to be an implicit conflation of the 4x finetuned context extension to 4k, and the further 4x extrapolation to 16k. If these two expansions need to be done in equal ratio, I do not see that requirement listed in the paper or ablated. Is the goal to always have the finetuned context expansion ratio match the eventual ratio further extrapolated beyond the finetuned context? If so, this is an interesting point that should be made more clearly with ablations demonstrating its importance.

It would be helpful if you could explain the rationale and evidence for the custom YaRN-family target. Although the paper provides results for both classic YaRN and the custom YaRN target, the focus on the custom target muddies the conclusion significantly. Further, it is not clear if the custom target is specifically useful due to some special properties of the paper's pretraining regime, or is generally applicable to other models.

I would be open to increasing my score if the experimental evidence was dramatically enhanced in all three ways outlined in the Critical Weaknesses above. I strongly recommend significant writing improvements, but strong enough experiments would be a very dramatic improvement even in the face of only moderately revised writing.

Reviewer_EegNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies when RoPE frequencies should be adjusted during context extension. It proposes Delayed Positional Adjustment (DEPA), which trains on longer sequences using the original frequencies before switching to the target configuration late in continuation training. Matched experiments show consistent improvements in extrapolation perplexity across positional scaling recipes, model configurations, and datasets. Further analysis attributes the gains primarily to reduced degradation from additional context rather than improved long-range information retrieval.

Critical strengths
  • It's a simple and well-motivated idea. The paper identifies positional adjustment timing as an overlooked design choice and provides an intuitive angle-coverage explanation for delayed rescaling.
  • The paired experiments match pretrained checkpoints, token budgets, final RoPE configurations, and optimizer schedules, making the observed timing effect convincing.
  • The authors provide useful analysis, e.g., the same-target evaluations distinguish lower perplexity from effective context utilization.
Critical weaknesses
  • My main concerns lie in the gap between improved extrapolation perplexity and actual long-context capability. Although DEPA substantially reduces PPL beyond the training window, the same-target evaluations consistently show negative added-context utility, meaning that additional context still hurts prediction. More importantly, the counterfactual retrieval experiments remain near chance and show no consistent improvement in distant information retrieval. Therefore, the evidence primarily supports reduced long-context degradation, rather than improved utilization of long-range dependencies. This distinction limits the practical significance of the reported gains, even though the paper appropriately acknowledges the limitation.
  • The improvements partly reflect weak static extrapolation baselines. The largest gains occur when static PI and published YaRN suffer severe PPL degradation, whereas the improvement is much smaller under the stronger custom YaRN configuration. The matched comparisons establish a timing effect but do not demonstrate superiority over independently optimized context extension recipes.
  • The model scale and training regime limit generalizability. The largest model has only 137M parameters, and the main experiments extend context from 1K to 4K. Moreover, the 137M models receive only about 205M pretraining tokens before substantial continuation training, making it unclear whether the findings transfer to mature, well-pretrained LLMs.
Questions for authors
  • What exactly is learned during the source-frequency hold phase?
  • Can the contribution of angle coverage be isolated from general optimization effects?
Reviewer_WfuaOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when RoPE frequency rescaling should be introduced during context-extension continued pretraining. DEPA trains at the extended sequence length using the original frequencies, then switches to a final PI/YaRN target for the last 10% of updates. In matched comparisons with static adjustment from the first step, it consistently improves perplexity beyond the trained context window across several small-model settings.

Critical strengths
  1. The comparison is well controlled. Static and DEPA runs share the pretrained checkpoint, final positional target, continuation token budget, and optimizer schedule. The constant-learning-rate results help rule out learning-rate decay as the sole explanation.
  2. The paper reports useful checks and limitations. It evaluates several positional targets, seeds, corpora, and model variants, and reports in-window trade-offs and negative retrieval findings rather than presenting the extrapolation PPL gains as unqualified wins.
Critical weaknesses

1. The practical significance of the perplexity gains is not yet established.

DEPA reduces the damage from longer inputs, but the added-context utility in Table 4 remains negative. The counterfactual retrieval probes in Appendix L are near chance and show no consistent content-use advantage. The authors acknowledge this distinction; nonetheless, the evidence currently supports improved long-input language modeling more clearly than improved use of distant information. A capability evaluation on a task where the underlying model can meaningfully retrieve or integrate remote evidence would strengthen the contribution.

2. The demonstrated generality is limited by model scale and pretraining coverage.

The largest model has 137M parameters and was pretrained on roughly 205M tokens before extension. Although the additional corpora and architectures broaden the evidence, they do not establish whether the timing effect transfers to substantially larger or more thoroughly pretrained models. Since the method changes adaptation to a pretrained positional representation, this gap is particularly relevant to its intended application.

3. The comparative advantage over stronger extension recipes remains uncertain.

The most dramatic gains occur under PI and published YaRN when static extrapolation is already very poor, whereas gains under the stronger custom YaRN-family target are more modest. Moreover, the differences from exposure-matched gradual transitions are small, and the appendix does not establish a uniform advantage over smooth transitions with a final adaptation phase. The matched comparisons isolate a timing effect, but do not yet establish DEPA as a broadly preferred recipe over separately tuned static or gradual alternatives.

4. The theoretical analysis provides conditional intuition rather than an explanatory or predictive account of the method.

The rotation bounds and recovery derivations appear valid under their stated assumptions. However, the recovery analysis assumes a local contraction model without establishing its applicability to the relevant training dynamics. The resulting balance condition depends on unknown preparedness and recovery quantities and does not determine the proposed release fraction. Consequently, the analysis offers limited guidance on when delayed adjustment should help or how to choose its timing for a new model and target configuration.

Questions for authors
  1. Given the negative added-context utility and near-chance retrieval results, is there evidence that DEPA improves distant-information use on a task where the underlying model has meaningful retrieval capability? Without such evidence, would a narrower framing around extrapolation perplexity better reflect the demonstrated contribution?

  2. Does the timing effect persist for substantially larger or more thoroughly pretrained models, particularly those with longer original context windows? Such evidence would materially affect the scope of the conclusions.

  3. How much advantage remains when static and gradual alternatives are separately tuned under comparable compute budgets? In particular, does the abrupt switch offer a meaningful quality or efficiency benefit over a late smooth transition followed by target adaptation?

  4. Can the recovery analysis produce a release criterion using measurements available before the switch, and can that criterion predict useful timings across different models or positional targets? If not, which experimentally testable predictions does the theory provide beyond motivating a hold–adaptation trade-off?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.