acceptodds
Under review as a conference paper at ICLR 2027

Don't Stop Long-Context Continual Pre-Training Too Early: An Empirical Scaling Law

Abstract

Long-context continual pre-training (LCCP) is widely used to improve long-context capability, but how its benefits scale with the number of tokens used for LCCP remains unclear. We investigate this question through modeling, monitoring, and validating the benefits of scaling LCCP. First, we establish an empirical additive power law relating long-context performance to model size and the respective token budgets of short-context pre-training and LCCP. Experiments across five model scales and more than one hundred short- and long-context budget configurations show that the fitted law reproduces the observed trends and extrapolates well to held-out model scales. Second, we introduce Constant-Last4, a training and observation protocol that averages the four most recent consecutive checkpoints from a constant-learning-rate run to enable stable monitoring along a single extensible training trajectory. Under this protocol, the same scaling-law form fits well and accurately predicts the LCCP performance trajectory of a 1.22B-parameter model trained on approximately 100B LCCP tokens. Finally, downstream evaluations along this trajectory show continued improvements in long-context performance, with no clear sign of saturation within the evaluated range. These gains persist after both 128K context extension and supervised fine-tuning, with benefits extending to long-context understanding and agentic tasks. Together, these results argue for continuing to scale LCCP rather than stopping early: its gains are stably observable along an extensible trajectory, quantitatively predictable, and transferable to downstream tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
scaling law, long context, long-context continual pretraining

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_1RtLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how long-context language modeling performance scales with the amount of long-context continual pre-training (LCCP). The authors propose an empirical additive power law that models log(LongPPL) as a function of the number of non-embedding model parameters, short-context pre-training tokens, and long-context continual pre-training tokens. The scaling law is evaluated using 109 training configurations across five model scales and demonstrates good predictive accuracy when extrapolating to held-out scales.

The paper also introduces Constant-Last4, a training and observation protocol that averages the four most recent checkpoints from a constant-learning-rate training trajectory. This approach provides more stable evaluation measurements while allowing training to continue without a predefined endpoint. The proposed scaling law is further validated on a 1.22B-parameter model trained for approximately 100B LCCP tokens.

Finally, the authors investigate whether improvements in LongPPL translate into downstream capabilities. Results on RULER, long-context benchmarks, and agentic tasks indicate that additional LCCP can improve performance, with benefits generally persisting after 128K context extension and supervised fine-tuning

Critical strengths
  1. Systematic empirical investigation and predictive scaling-law formulation.

A major strength is the systematic study of how long-context performance depends on model size and the token budgets of two distinct training stages. Separating short-context pre-training tokens from LCCP tokens is a meaningful modeling choice because continual pre-training starts from an already trained model rather than from random initialization. The experiments cover 109 configurations across five model scales, and the scaling law achieves strong held-out-scale predictive performance, with reported R2>0.95R^2 > 0.95 and MAPE below 2% in log(LongPPL) space under the linear-decay setup (Section 3, Figure 3).

The paper also compares alternative scaling-law formulations in Appendix A.2, which strengthens the empirical justification for the proposed additive structure. This contribution is consequential because it provides a quantitative framework for estimating the marginal benefits of additional long-context training rather than relying entirely on individual training runs.

  1. Practical training protocol supported by downstream validation.

Constant-Last4 is a useful practical contribution. Constant-learning-rate training is attractive for extensible pre-training but produces noisier checkpoints and worse endpoint performance than matched linear-decay schedules. The authors show that averaging four consecutive checkpoints substantially improves stability and closely matches the linear-decay baseline across the evaluated configurations (Section 4, Figure 4).

Importantly, the analysis goes beyond fitting a perplexity metric. The paper examines whether additional LCCP produces downstream benefits during 32K training, after 128K context extension, and after supervised fine-tuning on long-context and agentic tasks (Section 5, Figures 7–8). These evaluations make the work more practically relevant and provide useful evidence that the measured improvements are not confined to the scaling-law fitting objective.

Critical weaknesses
  1. Limited evidence for the generality of the proposed scaling law beyond the evaluated training setup.

Although the experiments cover multiple model scales and training budgets, they primarily use one architecture family, a fixed long-context data mixture, and a common short-context pre-training recipe. Furthermore, the largest evaluated model contains approximately 1.22B total parameters, which is substantially smaller than the models for which large-scale LCCP budget allocation may be most consequential.

The primary scaling-law target is also LongPPL computed on GovReport using token positions selected by a fixed Llama-3.1-8B reference model. While the metric is well motivated and the downstream evaluations partially address this limitation, the fitted relationship has not been shown to transfer to other long-context data distributions or evaluation metrics. The negative fitted offset, which the authors appropriately acknowledge in Section 3.2, also means that the law should be interpreted as an empirical finite-range relationship rather than a general asymptotic scaling law.

These limitations matter because the paper's central motivation is to guide decisions about how far to scale LCCP. The current evidence strongly supports prediction within the studied training family, but it provides less certainty about whether the same exponents and scaling behavior apply to larger models, alternative architectures, or substantially different training mixtures.

  1. Insufficient analysis of the compute-efficiency and opportunity cost of additional LCCP.

The paper demonstrates that increasing LCCP tokens generally improves LongPPL and several downstream benchmarks. However, this does not by itself establish that continuing LCCP is the most effective use of additional training compute.

The additive scaling law explicitly models separate short-context and long-context token budgets, creating a natural opportunity to investigate their trade-offs. Nevertheless, the paper does not provide a sufficiently direct compute-normalized comparison between allocating additional tokens to LCCP and alternative training strategies, such as extending short-context pre-training or redistributing the training budget.

This distinction is particularly important because long-context sequences may incur substantially different computational costs, and the downstream improvements are not uniform across all tasks. A comparison of marginal performance gains per unit of training compute, ideally including matched-compute alternatives, would make the central practical conclusion considerably stronger.

Questions for authors
  1. Generalization of the scaling law. Have the authors tested whether the fitted scaling exponents remain stable when changing the LCCP data mixture, model architecture, or evaluation corpus? Even a limited additional validation with an alternative long-context dataset would help determine whether the observed relationship is specific to GovReport-based LongPPL or reflects a more general property of LCCP.

  2. Compute-optimal allocation between training stages. Given the separate DsD_s and DlD_l terms in the proposed scaling law, can the authors analyze the marginal benefit of allocating additional compute to LCCP versus short-context pre-training? A compute-normalized comparison or an analysis using the fitted exponents would materially strengthen the practical recommendation to continue LCCP.

  3. Robustness of the downstream conclusions. Figure 7 and the per-subtask analysis indicate that the RULER-32K gains are driven primarily by intermediate-difficulty retrieval tasks, while Figure 8 shows some fluctuations across later checkpoints. Can the authors provide a quantitative analysis of how consistently the improvements generalize across individual downstream tasks, rather than relying primarily on aggregate trends? This would help clarify whether increasing LCCP produces broad capability improvements or primarily benefits a subset of long-context tasks.

4 comments
AuthorsOpenReview ↗

W1 & Q1: Generality beyond the training setup and evaluation corpus

We thank the reviewer for the careful review and for recognizing our separate modeling of short-context and LCCP budgets, as well as the practical value of Constant-Last4 and the downstream validation. The three questions on generality, compute efficiency, and downstream robustness are all to the point; we respond below, and the new analyses are included in a new Section 6 (Discussion) of the revision.

W1 & Q1: Generality beyond the training setup and evaluation corpus

Limited compute prevents full validation on other model families, data mixtures, or larger scales; we now state the scope of the law explicitly in the Limitations paragraph of Section 6.

For the limitation you highlighted, that LongPPL is based only on GovReport with key tokens selected by a fixed Llama-3.1-8B, we built a new evaluation set: 929 financial documents from DocFinQA and DocMath, with key tokens selected by the newer and stronger Qwen3.8-27B. We re-evaluated the models of both scaling experiments in the paper (Linear-Decay, LD; Constant-Last4, CL4) and, as in the paper, fit on S1–S4 and extrapolate to S5. The fitted parameters are:

Law (S1–S4 fit)AAα\alphaBsB_sβs\beta_sBlB_lβl\beta_lCC
GovReport / CL4108.4460.2443228.8540.231352.1420.2253−0.614
GovReport / LD61.7380.2084186.7200.220669.7430.2346−0.818
DocFinQA+DocMath / CL4304.3430.3193303.9840.2542108.3800.26990.367
DocFinQA+DocMath / LD212.0970.2994184.1200.2291118.5950.26740.257
Protocol / Eval setS5 extrap. R2R^2S5 extrap. MAPEFit R2R^2Fit MAPE
CL4 / GovReport0.97180.66%0.9951.19%
LD / GovReport0.95161.71%0.9941.44%
CL4 / DocFinQA+DocMath0.95950.50%0.99341.12%
LD / DocFinQA+DocMath0.94990.87%0.99271.16%

Functional form. With a new reference model and a new domain, the same form still fits and extrapolates well, so it is not specific to GovReport or the Llama-3.1-8B reference model. Since key tokens are selected by a fixed rule, LongPPL on the new set still mainly measures long-context ability rather than domain expertise.

Stability of the exponents. The exponents themselves are not identical: under CL4, for example, α\alpha changes from 0.244 to 0.319 and βl\beta_l from 0.225 to 0.270. This is expected, since the difficulty distribution of key tokens differs between evaluation sets. What matters more in practice is the stage allocation derived from the exponents, which is nearly unchanged, as will be shown in subsequent replies.

Negative offset. On DocFinQA+DocMath, the fitted CC turns positive (0.367 for CL4, 0.257 for LD). The raw LongPPL data show that financial text has clearly higher token-level perplexity than GovReport, which explains the sign change and indicates that CC acts as a set-dependent overall offset. This agrees with your reading and with our discussion in Section 3.2: the law should be understood as an empirical relationship within a finite range rather than an asymptotic scaling law. These results are in Section 6 (Figure 9) and Appendix A.6.

AuthorsOpenReview ↗

W2 & Q2: Compute efficiency of additional LCCP and allocation between stages

W2 & Q2: Compute efficiency of additional LCCP and allocation between stages

As you note, the separate DsD_s and DlD_l terms make this trade-off analyzable, and the higher per-token cost of long sequences must be included. We added a closed-form analysis and compute-matched experiments (Section 6, Eq. 3, Table 2, Figure 10).

Cost of long sequences. We use measured training time rather than theoretical FLOPs. In our Megatron setup, 32K and 4K steps contain the same number of tokens (32K uses activation recomputation to avoid context/sequence parallelism), and a 32K token costs r=2.53r=2.53 times a 4K token.

Derivation and results. With total compute F=Ds+rDlF=D_s+rD_l, the ratio of log-LongPPL reductions per unit of compute on LCCP versus SCP is M=K Ds1+βs/Dl1+βlM=K\,D_s^{1+\beta_s}/D_l^{1+\beta_l} with K=βlBl/(rβsBs)K=\beta_lB_l/(r\beta_sB_s), and M=1M=1 gives the break-even ratio ρ∗=Dl∗/Ds=K1/(1+βl)Ds(βs−βl)/(1+βl)\rho^*=D_l^*/D_s=K^{1/(1+\beta_l)}D_s^{(\beta_s-\beta_l)/(1+\beta_l)}, which is independent of NN and CC and nearly independent of DsD_s. At fixed FF, the optimal allocation satisfies exactly M=1M=1.

For S5 (Ds=450ND_s=450N): at 10N, LCCP is 11–13 times as compute-efficient as SCP, and about twice as efficient at 40N; break-even is at 70–77N; the optimum gives LCCP 28–30% of the S5 total compute, and stopping at 10N costs 0.08–0.10 in log LongPPL versus at most 0.006 at 120N. Stopping too early is thus much more costly than stopping too late.

Compute-matched experiment. We trained two new S5 trajectories, Ds=360ND_s=360N to 160N and Ds=540ND_s=540N to 80N, and compared them with the original Ds=450ND_s=450N trajectory at matched FF (Figure 10). The 360N trajectory stays ahead of the 450N one until its LCCP reaches about 70N, and the 450N trajectory stays ahead of the 540N one until its LCCP reaches about 100N. Thus, with little LCCP, extra compute is better spent on LCCP, and SCP wins only after the break-even point; at both crossings, the predicted optimum lies between the two compared allocations.

Marginal return with a fixed base model. In practice, SCP is often trained once and reused, so LCCP has its own budget and its marginal return is what matters. We give a three-level criterion: (1) the break-even point is a lower bound, since before it LCCP outperforms even extending SCP, so stopping earlier is premature; (2) beyond it, with each 10N of LCCP at a fixed cost (about 66 H800 GPU-hours on S5), training continues while the predicted reduction for the next 10N exceeds a price λ\lambda; e.g., λ=0.005\lambda=0.005 stops at about 80N and λ=0.003\lambda=0.003 at about 124N; (3) directly evaluated RULER-32K serves as an auxiliary check.

Scope and implication. This allocation optimizes long-context performance as measured by LongPPL and ignores short-context performance, world knowledge, reasoning, and other abilities, so real pre-training must weigh more factors; ρ∗\rho^* is also obtained within our scale range.

AuthorsOpenReview ↗

Q3: Robustness of the downstream conclusions

Q3: Robustness of the downstream conclusions

Thank you for the careful observation. We address the points about Figures 7 and 8 in turn.

Per-task curves are in the appendix. Figure 14 shows the RULER-32K subtasks, Figure 15 the RULER-128K subtasks, Figures 16 and 17 the four long-context benchmarks (LongBench, GraphWalks, LongReason, BABILong) after 2.1B- and 4.2B-token long-context SFT, and Figure 18 GAIA and BFCL-WebSearch.

RULER-32K: gains concentrated on intermediate-difficulty subtasks are expected. RULER subtasks span a wide range of difficulty. Easy tasks such as single-key NIAH are already saturated early in training, while harder subtasks remain difficult even at about 100B LCCP tokens. Inspecting model outputs, we find that the 1.22B model does not yet have reliable in-context learning ability, so subtasks that depend on the in-prompt example, such as CWE and VT, stay near zero. The room for LCCP-driven improvement is therefore mainly in intermediate-difficulty tasks, consistent with Figure 14.

Downstream tasks after SFT: a per-task quantitative test. The main text reports aggregate scores to reduce single-task fluctuations caused by uneven SFT data sampling, which better reflects changes in overall capability. To answer your question, we trained the 2.1B-token long-context SFT on each LCCP checkpoint (Constant-Last4) with 3 different seeds and tested each task separately:

TaskMean change 10N → 120NSlope per LCCP doubling [95% CI]Spearman (p)Seed SD at same checkpointVariance explained by LCCP
LongBench+2.06+0.77 [0.45, 1.09]0.93 (0.003)0.850.68
GraphWalks+7.65+1.77 [0.70, 2.84]1.00 (< 0.001)3.130.45
LongReason+14.27+4.09 [2.79, 5.39]0.75 (0.033)2.800.85
BABILong+6.73+2.07 [1.36, 2.78]0.89 (0.006)1.930.74
4-task mean+7.68+2.17 [1.78, 2.56]0.96 (0.001)1.000.92

Slope: coefficient of a linear regression of score on log⁡2(LCCP)\log_2(\mathrm{LCCP}) (21 points). Spearman correlation is computed on the seed means of the 7 checkpoints, with one-sided permutation p-values. The last column is η2\eta^2 from one-way ANOVA.

  1. The gains are not confined to a few tasks. The slope CIs of all four tasks exclude 0 and all Spearman tests are significant, so each task improves with LCCP on its own.
  2. Part of the per-task fluctuation comes from SFT randomness. At the same checkpoint, the seed-to-seed SD reaches 2–3 points on single tasks, so per-task curves (3-seed means in Figure 16; a single seed in Figure 17) are expected to fluctuate. Averaging the four tasks reduces the seed SD to 1.0 and raises the variance explained by LCCP to 0.92.
  3. Tasks partly trade off against each other. At the same checkpoint, a seed that scores higher on GraphWalks tends to score lower on BABILong: their deviations from the checkpoint mean correlate at −0.63. This suggests some competition between tasks in a small-capacity SFT, which aggregation averages out.

Search-agent tasks. In Figure 18, GAIA and BFCL-WebSearch move in partly opposite directions: peaks in GAIA often coincide with dips in BFCL-WebSearch. This matches the task competition seen at the seed level and suggests that the two tasks also compete in a small-capacity SFT. Averaging them, as in Figure 8(c), partly cancels this effect and shows the trend of overall agentic performance.

Summary

Thank you for the recognition and constructive feedback. On generality, the law fits and extrapolates on a new LongPPL set with a different reference model and domain; the exponents change, but the stage-allocation conclusions do not. On compute efficiency, the measured cost ratio r=2.53r=2.53 yields a closed-form break-even ratio and compute-optimal allocation, validated by compute-matched trajectories, together with a stopping criterion for a fixed base model. On downstream robustness, a 3-seed per-task test shows that all four downstream tasks improve significantly with LCCP, with different magnitudes; single-task fluctuations stem mainly from SFT randomness and task competition, which is why the main text reports aggregates. The new evaluation set and the compute analysis are in Section 6 of the revision, and the per-task analysis will be added to the appendix.

Dear Reviewer 1RtL,

We sincerely thank you again for your careful review and for your positive assessment of our work. To address your concerns, we built a new LongPPL evaluation set from financial documents (DocFinQA and DocMath), with key tokens selected by Qwen3.8-27B instead of Llama-3.1-8B. On this set, the law still fits and extrapolates well, and the derived stage allocation is nearly unchanged. We also derived a closed-form compute-optimal allocation between short-context pre-training and LCCP that accounts for the measured cost of long sequences. It assigns roughly 30% of the total training compute to LCCP and is validated by compute-matched trajectories. Finally, a 3-seed per-task analysis shows that all four downstream tasks improve significantly with LCCP. The first two are in Section 6 of the revised paper, and the per-task analysis will be added to the appendix.

As the discussion period is coming to an end, we would be very grateful if you could let us know whether our responses have addressed your concerns. We would be happy to clarify any remaining questions.

Best regards, Authors

Reviewer_42UqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how the benefits of long-context continual pre-training (LCCP) scale with its training budget. It establishes an empirical additive power law relating log(LongPPL) to model size and the respective token budgets of short-context pre-training and LCCP. It also introduces CONSTANT-LAST4, which averages the four most recent checkpoints from a constant-learning-rate run to enable stable monitoring along an extensible training trajectory. Experiments on a 1.22B-parameter model trained with approximately 100B LCCP tokens show continued improvements in long-context performance, with gains remaining observable after 128K context extension and supervised fine-tuning for long-context and agentic tasks.

Critical strengths
  1. The paper studies 109 training configurations across five model scales and validates the proposed scaling law on held-out larger models, providing empirical support within the studied setting.

  2. The paper observes continued gains from longer LCCP and evaluates whether these gains persist after 128K context extension and supervised fine-tuning, including on long-context and agentic tasks.

Critical weaknesses
  1. The practical guidance implied by the title is not sufficiently developed. The paper shows continued improvements from longer LCCP, but does not clearly define what constitutes stopping “too early.” Continued improvement alone does not establish that further training is worthwhile. An analysis of marginal gains relative to computational cost would help translate the fitted scaling law into actionable guidance on when to continue or stop training.

  2. Despite covering 109 training configurations, the scaling-law evaluation is limited to a single model family, a fixed LCCP data mixture, and LongPPL measured on GovReport. Its generalizability to other model families, training mixtures, and evaluation domains remains unclear.

  3. The novelty of CONSTANT-LAST4 is limited. Constant-learning-rate training with checkpoint averaging has been explored in prior work, including WSM and early weight averaging. The contribution mainly lies in adapting and evaluating this approach for LCCP.

Questions for authors
  1. The title advises against stopping LCCP too early, but what constitutes “too early”? For example, training for another 10B tokens might improve the score by 0.1 points, while another 100B tokens might yield only 0.2 points. Although performance continues to improve, the computational cost may be substantial. In practice, how should one decide whether further LCCP is worthwhile? Can you provide a criterion for continuing or stopping training based on predicted gains and computational cost?

  2. The scaling law accurately predicts log(LongPPL), but how accurately does it predict the additional improvement from further training? Could you compare predicted and observed loss reductions over held-out training intervals and report uncertainty estimates?

  3. Under a fixed total compute budget, does allocating more compute to LCCP yield greater downstream improvements than allocating it to short-context pre-training or SFT? Could you provide a compute-matched comparison to support?

  4. Does the same scaling-law form remain predictive on other long-context evaluation domains?

  5. Figure 1 could be clearer and more concise. The numerous icons, boxes, and small labels make it visually crowded. The miniature plots are difficult to interpret and provide limited quantitative information. A simpler overview highlighting the experimental design and main findings would communicate the contribution more effectively.

4 comments
AuthorsOpenReview ↗

W1, Q1 & Q3 (Part I): What is "too early", compute allocation, and a stopping criterion

We thank the reviewer for the detailed and constructive comments. We agree that the lack of an operational definition of "too early" was the main gap of the paper. We added theory and new experiments on this point, now in a new Section 6 (Discussion), and address the other questions below.

W1, Q1 & Q3: What is "too early", compute allocation, and a stopping criterion

Short answer. We define stopping "too early" as stopping before the break-even ratio Dl∗≈0.16 DsD_l^*\approx0.16\, D_s (about 70N on S5). Before this point, the same compute lowers LongPPL more when spent on LCCP than on short-context pre-training (SCP). Details follow for two practical cases: LCCP competing with SCP for a fixed budget, and LCCP as a separate budget after SCP is finished.

1. LCCP and SCP share a compute budget

Cost ratio. We use measured wall-clock cost rather than theoretical FLOPs. In our Megatron setup, 32K and 4K steps contain the same number of tokens (the 32K stage uses activation recomputation); the average step times are 3.8 s and 1.5 s, so one long-context token costs r=2.53r=2.53 short-context tokens.

Derivation. Let ℓ=log⁡LongPPL=AN−α+BsDs−βs+BlDl−βl+C\ell=\log\mathrm{LongPPL}=AN^{-\alpha}+B_sD_s^{-\beta_s}+B_lD_l^{-\beta_l}+C and F=Ds+rDlF=D_s+rD_l. One more unit of compute buys one SCP token or 1/r1/r LCCP token, and the ratio of the resulting reductions in ℓ\ell is

M=∂ℓ/∂Dlr ∂ℓ/∂Ds=K Ds1+βsDl1+βl,K=βlBlr βsBs.M=\frac{\partial\ell/\partial D_l}{r\,\partial\ell/\partial D_s}=K\,\frac{D_s^{1+\beta_s}}{D_l^{1+\beta_l}},\qquad K=\frac{\beta_lB_l}{r\,\beta_sB_s}.

LCCP is the better use of compute when M>1M>1. Setting M=1M=1 gives the break-even ratio

ρ∗=Dl∗Ds=K11+βl Dsβs−βl1+βl.\rho^*=\frac{D_l^*}{D_s}=K^{\frac{1}{1+\beta_l}}\,D_s^{\frac{\beta_s-\beta_l}{1+\beta_l}}.

ρ∗\rho^* does not involve AA, NN, or CC. Since βs≈βl\beta_s\approx\beta_l, the exponent on DsD_s is about 0.03 or less in magnitude across the four fitted laws, so ρ∗\rho^* is nearly constant in DsD_s. At fixed FF, the first-order condition of the optimal allocation is exactly M=1M=1, so the compute-optimal allocation also lies on Dl=ρ∗DsD_l=\rho^*D_s.

Results (S5, Ds=450ND_s=450N, r=2.53r=2.53; point estimates are in Table 2 of the revision, and the bootstrap intervals in brackets are reported here):

MetricGovReport CL4GovReport LDDocFin CL4DocFin LD
MM at Dl=10ND_l=10N / 40N40N10.9 [8.9, 12.6] / 2.011.9 / 2.112.4 / 2.113.4 / 2.3
Break-even Dl∗D_l^* (ρ∗\rho^*)70N [61, 78] (0.156)74N [64, 84] (0.165)73N (0.161)77N (0.172)
Optimal LCCP share of FF28%29%29%30%
Gap to optimum: stop at 10N / 120N0.085 / 0.0060.098 / 0.0050.076 / 0.0040.089 / 0.004

At 10N, the same compute lowers log LongPPL about 11 times more on LCCP than on SCP, and still about twice as much at 40N. With the total compute fixed at that of the S5 run (450+120×2.53≈754N450+120\times2.53\approx754N), the optimum is about 540N of SCP plus 84N of LCCP, i.e., 28% of compute on LCCP. Stopping at 10N costs more than ten times as much as training to 120N, so stopping too early is far more costly than stopping too late.

Consistency across evaluation sets. The exponents fitted on the two evaluation sets differ (e.g., βl\beta_l is 0.225 vs. 0.270 under CL4), yet the three metrics are nearly identical. The allocation advice therefore *does not hinge on a particular evaluation set.

Compute-matched experiment (Q3). To test the analysis, we trained two new S5 trajectories with different splits at the same training cost: Ds=360ND_s=360N trained to Dl=160ND_l=160N, and Ds=540ND_s=540N trained to Dl=80ND_l=80N, compared with the original Ds=450ND_s=450N trajectory trained to 120N on a common axis F/N=(Ds+rDl)/NF/N=(D_s+rD_l)/N (new Figure 10). At a fixed training cost, neither more DsD_s nor more DlD_l is uniformly better:

  • 360N vs. 450N. At matched FF, the 360N trajectory is better until its LCCP reaches about 70N (the 450N trajectory is then at about 35N); only then does the 450N trajectory overtake it.
  • 450N vs. 540N. Likewise, the 450N trajectory is better until its LCCP reaches about 100N; only then does the 540N trajectory, at about 65N, overtake it.

Scope and implication. This allocation optimizes LongPPL alone, as a proxy for long-context performance, and ignores short-context performance, world knowledge, reasoning, and other capabilities; ρ∗\rho^* is also obtained within our model and budget range and needs verification at larger scales. Even so, it is informative that, for long-context ability alone, the optimum already allocates about 30% of compute to LCCP (token ratio about 0.16).

AuthorsOpenReview ↗

W1, Q1 & Q3 (Part II): What is "too early", compute allocation, and a stopping criterion

2. SCP is fixed: marginal gains and a stopping criterion

This case is also common in practice. SCP usually takes most of the compute, even when LCCP is merged into mid-training; it also involves architecture decisions, and as the first stage, changing it requires rerunning all later stages. SCP is therefore typically trained once at scale and reused, and LCCP acts as continual pre-training on a fixed base. What matters then is the marginal return of LCCP itself rather than its trade-off with SCP. We propose a three-level criterion.

(1) Lower bound: the break-even point. Even in the strongest competing case, where SCP could still be extended, continuing LCCP before Dl≈0.16 DsD_l\approx0.16\,D_s (about 70N on S5) lowers LongPPL more than spending the same compute on SCP. Stopping before this point is therefore premature whether or not SCP can be changed. This is our operational definition of "too early" in the title.

(2) Beyond break-even: a cost threshold. Each 10N of LCCP has a fixed cost. The practitioner can set a price λ\lambda, the minimum reduction in log LongPPL required per 10N. Because single LongPPL measurements are noisy, we use the fitted law to predict the reduction Δ^10N\hat\Delta_{10N} from the next 10N. For S5 under the GovReport CL4 law:

Price λ\lambda (min. reduction per 10N)0.0050.0040.0030.002
Stopping point≈80N≈97N≈124N≈174N

The scaling law makes these numbers computable in advance and turns "is further training worthwhile?" into an explicit cost comparison.

(3) Auxiliary downstream check: RULER-32K. Downstream scores give a more direct signal of marginal benefit. We recommend directly evaluating RULER-32K as the auxiliary signal: if it is still improving over recent checkpoints, the LongPPL reduction is still turning into long-context capability, which supports continuing.

Overall, when the budget allows, LCCP should be treated as a stage that needs sufficient training: longer LCCP yields stronger long-context performance, which carries over to downstream long-context and agentic tasks after SFT.

3. LCCP versus post-training (Q3)

Our long-context SFT experiments use two SFT budgets, 2.1B tokens (Figure 8b) and 4.2B tokens (Appendix Figure 17), which give downstream averages for combinations of SFT and LCCP budgets (Constant-Last4):

SFT \ LCCP10N20N40N60N80N100N120N
2.1B†19.0121.9824.7825.7125.7127.0426.69
4.2B26.4925.7426.8428.1928.3729.1229.97

† Mean over 3 SFT seeds; the 4.2B results use a single seed.

If LCCP and SFT shared one compute budget, spending more tokens on SFT would raise downstream scores more directly. At the same time, the two gains stack: longer LCCP helps under both SFT budgets (2.1B: 19.01 → 26.69; 4.2B: 26.49 → 29.97).

In practice, however, we do not view LCCP and post-training as competing for one budget, for three reasons.

  • First, they serve different goals: LCCP broadly strengthens base capabilities, whereas SFT turns them into task performance.
  • Second, SFT requires high-quality supervised data, often tied to downstream domains, which is costly to build and hard to scale, whereas LCCP needs only unsupervised corpora that are cheaper and easier to scale.
  • Third, their gains add up rather than trade off, so scaling LCCP can still improve the model whenever compute allows.
AuthorsOpenReview ↗

Responses to Other Concerns: Generalizability, Novelty, and More

Q2: Accuracy and uncertainty of the predicted additional improvement

For decision-making, what matters is not the absolute LongPPL but how much further training will reduce it, which is exactly the quantity the stopping criterion relies on. We test this with the fixed-model simplified law (Section 4.3, Figure 6) on the three S5 trajectories (Ds=360N/450N/540ND_s=360N/450N/540N): we fit only the first half of each trajectory's checkpoints and predict the second half. To measure the accuracy of the predicted reduction directly, we define

κ(x)=y^half(c)−y^half(x)y^full(c)−y^full(x),y=log⁡(LongPPL),\kappa(x)=\frac{\hat y_{\mathrm{half}}(c)-\hat y_{\mathrm{half}}(x)}{\hat y_{\mathrm{full}}(c)-\hat y_{\mathrm{full}}(x)},\qquad y=\log(\mathrm{LongPPL}),

where cc is the fitting cutoff, y^half\hat y_{\mathrm{half}} is the curve fitted on the first half only, and y^full\hat y_{\mathrm{full}} is the curve fitted on the full trajectory, used as a noise-smoothed reference reduction. κ=1\kappa=1 means the predicted reduction matches the reference; κ<1\kappa<1 (>1>1) means under- (over-) estimation. We use the full-trajectory fit rather than raw observations as the reference because single-checkpoint noise is a sizable fraction of the held-out reductions, which would make a ratio of raw observations itself noisy.

TrajectoryFit range (points)Held-out range (points)κ\kappa (midpoint) [95% CI]κ\kappa (endpoint) [95% CI]MAPER2R^2
Ds=360ND_s=360N≤80N (16)85–160N (16)0.990 at 120N [0.779, 1.222]0.990 at 160N [0.765, 1.242]0.40%0.57
Ds=450ND_s=450N≤60N (12)65–120N (12)1.060 at 90N [0.844, 1.340]1.065 at 120N [0.831, 1.374]0.36%0.64
Ds=540ND_s=540N≤40N (8)45–80N (8)0.921 at 60N [0.697, 1.250]0.913 at 80N [0.674, 1.277]0.42%0.85
  • The predicted reductions are close to the reference. The midpoint estimates are 0.990, 1.060, and 0.921, and the endpoint estimates are 0.990, 1.065, and 0.913, all within 9% of 1. The 360N trajectory, with the most and densest fitting points, is the most accurate; even the 540N trajectory, with only 8 fitting points, gives a reasonable estimate.
  • Uncertainty. We run 2,000 circular block-bootstrap resamples (block length 3) of the fitting-segment residuals and refit the first-half curve, keeping the full-trajectory curve fixed, so the intervals reflect how prefix noise affects the predicted reduction. All three 95% intervals cover 1. In the stopping criterion, the lower end of the interval can be used as a conservative estimate.

W2 & Q4: Generality across model families, data mixtures, and evaluation domains

Limited compute prevents full experiments on other model families, LCCP mixtures, or bigger models. For the evaluation domain (Q4), we built a new LongPPL set from 929 financial documents in DocFinQA and DocMath, with key tokens selected by Qwen3.8-27B, and refit the law on S1–S4 under both LD and CL4, extrapolating to S5:

Protocol / Eval setS5 extrap. R2R^2S5 extrap. MAPEFit R2R^2Fit MAPE
CL4 / GovReport0.97180.66%0.9951.19%
LD / GovReport0.95161.71%0.9941.44%
CL4 / DocFinQA+DocMath0.95950.50%0.99341.12%
LD / DocFinQA+DocMath0.94990.87%0.99271.16%

On a set with a new domain and a new reference model, the same form still fits and extrapolates well, and, more importantly, the compute-allocation conclusions are nearly unchanged (Table 2). The law and its practical guidance are therefore not specific to GovReport or the Llama-3.1-8B reference model. Since key tokens are selected by a fixed rule, LongPPL on the new set still mainly measures long-context ability rather than domain expertise. These results are in Section 6 (Figure 9) and Appendix A.6.

W3: Novelty of Constant-Last4

We agree. Constant learning rates with checkpoint averaging are not new, and WSM and early weight averaging, both cited in the paper, study them systematically. The paper sees Constant-Last4 as a training and observation protocol for LCCP and does not claim the averaging technique itself as new. Its contribution is the validation in the LCCP setting: it closely matches budget-matched Linear-Decay, and the same scaling law fits under it, thereby enabling extensible monitoring and prediction from early checkpoints. We will revise the wording to present Constant-Last4 explicitly as an adaptation of existing methods and to make the attribution clearer.

Q5: Readability of Figure 1

Thank you for the suggestion. We agree that the original Figure 1 is too crowded. We will redesign it by removing the redundant icons and replacing the miniature plots with real quantitative data and the main findings. The redesigned Figure 1 will replace the original in the revised paper.

Dear Reviewer 42Uq,

We sincerely thank you again for your thoughtful and detailed review, which has helped us improve the paper substantially. To address your concerns, we now give “too early” an operational definition derived from the fitted law. For long-context performance, the compute-optimal allocation assigns roughly 30% of the total training compute to LCCP. Stopping LCCP before it reaches this share is premature, because up to that point the same compute lowers LongPPL more on LCCP than on short-context pre-training. Beyond this point, we give a cost-based stopping criterion. Two new compute-matched trajectories support this analysis. We also added held-out tests of the predicted additional improvement with bootstrap confidence intervals, and a second evaluation set built from financial documents (DocFinQA and DocMath) with key tokens selected by Qwen3.8-27B instead of Llama-3.1-8B. Finally, we clarified that Constant-Last4 is an adaptation of existing methods, and the contribution lies in the validation in the scaling LCCP setting. We also redesigned Figure 1. The main additions are in the new Section 6 of the revised paper.

As the discussion period is coming to an end, we would be very grateful if you could let us know whether our responses have addressed your concerns. We would be happy to clarify any remaining questions.

Best regards, Authors

Reviewer_4SY8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper investigates the scaling behavior of long-context continual pre-training (LCCP). The authors propose an empirical scaling law and a Constant-Last4 protocol for stable evaluation and demonstrate predictable performance improvements with increased LCCP tokens.

19 Nov 2026: Those additions submitted by the author largely resolve my original concerns. I maintain my rating of 3 (weak acceptance), and adjust my rating of soundness from 2 to 3.

Critical strengths
  1. The paper addresses a practical question in LCCP and provides systematic experiments across multiple model scales and training budgets to validate the proposed scaling law.
Critical weaknesses
  1. Limited generalizability. The scaling law is mainly validated on GovReport LongPPL and models up to 1.22B parameters, leaving its generalization to larger models and other datasets unclear.
  2. Lack of compute-matched comparisons. The paper does not compare additional LCCP with alternative training strategies under the same compute budget.
Questions for authors
  1. How well does the proposed scaling law generalize to alternative long-context evaluation datasets or larger model architectures?
  2. How should an appropriate LCCP training budget or stopping point be selected?
3 comments
AuthorsOpenReview ↗

Generalization to other evaluation data

We thank the reviewer for the careful review and for recognizing our systematic experiments across model scales and training budgets. Both concerns, the generality of the scaling law and the lack of compute-matched comparisons, are addressed below with new experiments and analysis, which are now included in a new Section 6 (Discussion) of the revised paper.

W1 & Q1: Generalization to other evaluation data and larger models

Training other model families or larger models is beyond our compute budget during the rebuttal period. We therefore addressed the most direct limitation: LongPPL was computed only on GovReport, with key tokens selected by Llama-3.1-8B.

New evaluation set. We built a LongPPL set from 929 financial documents in DocFinQA and DocMath and selected key tokens with the newer and stronger Qwen3.8-27B under the same selection rule, so both the domain and the reference model differ from the original setup. We re-evaluated all Linear-Decay (LD) and Constant-Last4 (CL4) models, refit the law on S1–S4, and extrapolated to the held-out S5:

Protocol / Eval setS5 extrap. R2R^2S5 extrap. MAPEFit R2R^2Fit MAPE
CL4 / GovReport0.97180.66%0.9951.19%
LD / GovReport0.95161.71%0.9941.44%
CL4 / DocFinQA+DocMath0.95950.50%0.99341.12%
LD / DocFinQA+DocMath0.94990.87%0.99271.16%

Fit: 100 S1–S4 configurations; held-out: 24 (CL4) / 9 (LD) S5 points. MAPE is computed in log⁡(LongPPL)\log(\mathrm{LongPPL}) space.

The same functional form fits and extrapolates on the new set about as well as on GovReport, and the compute allocation derived from it is nearly unchanged (see W2 & Q2). The law is therefore not tied to a single corpus or reference model. Because key tokens are selected by a fixed rule based on long-range dependence, LongPPL on the new set still mainly measures long-context modeling rather than financial knowledge. These results are now in Section 6 (Figure 9), with details of the new set in Appendix A.6.

AuthorsOpenReview ↗

Compute-matched comparison

W2 & Q2: Compute-matched comparison and choosing the LCCP budget and stopping point

We agree that the paper lacked both a compute-matched comparison and actionable guidance on the LCCP budget. We address them for two practical cases: (i) LCCP competes with short-context pre-training (SCP) for the same total compute; (ii) SCP is already finished and LCCP has its own budget. Both cases are common in reality.

Case (i): LCCP and SCP share compute.

In our Megatron setup, 32K and 4K steps contain the same number of tokens, and a 32K step takes r=2.53r=2.53 times as long (3.8 s vs. 1.5 s). Measuring compute in SCP tokens, the total compute is F=Ds+rDlF=D_s+rD_l. From the fitted law we derive in closed form (Section 6, Eq. 3) the ratio MM of the log-LongPPL reduction per unit of compute spent on LCCP versus SCP, and the break-even ratio ρ∗=Dl∗/Ds\rho^*=D_l^* /D_s at which M=1M=1. Because M=1M=1 is also the first-order condition for minimizing the loss at fixed FF, the compute-optimal allocation lies on Dl=ρ∗DsD_l=\rho^*D_s. For S5 with Ds=450ND_s=450N (new Table 2):

MetricGovReport CL4GovReport LDDocFin CL4DocFin LD
MM at Dl=10ND_l=10N / 40N40N10.9 / 2.011.9 / 2.112.4 / 2.113.4 / 2.3
Break-even Dl∗D_l^* (ρ∗\rho^*)70N (0.156)74N (0.165)73N (0.161)77N (0.172)
Optimal LCCP share of FF28%29%29%30%
Gap to optimum: stop at 10N / 120N0.085 / 0.0060.098 / 0.0050.076 / 0.0040.089 / 0.004

At 10N, LCCP is 11–13 times as compute-efficient as SCP, and it remains the better use of compute until about 70–77N (ρ∗≈0.16\rho^*\approx0.16–0.170.17). At the S5 total compute,the optimum gives LCCP 28–30% of the compute; stopping at 10N leaves log LongPPL 0.08–0.10 above the optimum, versus at most 0.006 at 120N. All four laws agree.

Compute-matched experiment. We trained two new S5 trajectories, Ds=360ND_s=360N to Dl=160ND_l=160N and Ds=540ND_s=540N to Dl=80ND_l=80N, and compared them with the original Ds=450ND_s=450N trajectory at matched FF (see new Figure 10 in paper). Neither more SCP nor more LCCP is uniformly better: the 450N trajectory overtakes the 360N one only after the latter reaches about 70N of LCCP, and the 540N trajectory overtakes the 450N one only after the latter reaches about 100N. At both crossings, the predicted optimum Dl=ρ∗DsD_l=\rho^*D_s lies between the two compared allocations, consistent with the analysis.

Case (ii): SCP is fixed and LCCP has its own budget.

This is also a common case in practice, and we give a three-level stopping criterion. (1) Lower bound: stopping before the break-even point (Dl≈0.16 DsD_l\approx0.16\,D_s, about 70N on S5) is premature, because there LCCP outperforms even extending SCP. (2) Beyond it, stopping is a cost decision. Each 10N of LCCP has a fixed cost (about 66 H800 GPU-hours on S5); the practitioner sets a price λ\lambda, the minimum log-LongPPL reduction required per 10N, and stops once the reduction predicted by the law for the next 10N falls below λ\lambda. On S5, λ=0.005\lambda=0.005 and 0.0030.003 stop LCCP at about 80N and 124N. (3) As an auxiliary check, training continues while directly evaluated RULER-32K is still improving.

This analysis takes LongPPL as the sole objective and does not account for short-context ability, world knowledge, or other capabilities that real pre-training must also weigh; The derivation, Table 2, Figure 10, and criteria (1)–(2) are included in Section 6 of the revision.

We hope these additions, namely a second evaluation set with a different domain and reference model, a closed-form compute allocation validated by compute-matched runs, and a stopping criterion, address your concerns. Thank you again for the helpful comments.

Dear Reviewer 4SY8,

We sincerely thank you again for your time and constructive comments. To address your concerns, we built a second LongPPL evaluation set from financial documents (DocFinQA and DocMath), with key tokens selected by Qwen3.8-27B instead of Llama-3.1-8B. On this set, the scaling law fits and extrapolates to the held-out S5 as well as on GovReport. We also derived a closed-form compute allocation between short-context pre-training and LCCP and validated it with two new compute-matched S5 trajectories. It shows that LCCP should receive roughly 30% of the total training compute. Building on this, we give a stopping criterion for choosing the LCCP budget. These results are now in Section 6 of the revised paper.

As the discussion period is coming to an end, we would be very grateful if you could let us know whether our responses have addressed your concerns. We would be happy to clarify any remaining questions.

Best regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.