- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how long-context language modeling performance scales with the amount of long-context continual pre-training (LCCP). The authors propose an empirical additive power law that models log(LongPPL) as a function of the number of non-embedding model parameters, short-context pre-training tokens, and long-context continual pre-training tokens. The scaling law is evaluated using 109 training configurations across five model scales and demonstrates good predictive accuracy when extrapolating to held-out scales.
The paper also introduces Constant-Last4, a training and observation protocol that averages the four most recent checkpoints from a constant-learning-rate training trajectory. This approach provides more stable evaluation measurements while allowing training to continue without a predefined endpoint. The proposed scaling law is further validated on a 1.22B-parameter model trained for approximately 100B LCCP tokens.
Finally, the authors investigate whether improvements in LongPPL translate into downstream capabilities. Results on RULER, long-context benchmarks, and agentic tasks indicate that additional LCCP can improve performance, with benefits generally persisting after 128K context extension and supervised fine-tuning
Critical strengths
- Systematic empirical investigation and predictive scaling-law formulation.
A major strength is the systematic study of how long-context performance depends on model size and the token budgets of two distinct training stages. Separating short-context pre-training tokens from LCCP tokens is a meaningful modeling choice because continual pre-training starts from an already trained model rather than from random initialization. The experiments cover 109 configurations across five model scales, and the scaling law achieves strong held-out-scale predictive performance, with reported and MAPE below 2% in log(LongPPL) space under the linear-decay setup (Section 3, Figure 3).
The paper also compares alternative scaling-law formulations in Appendix A.2, which strengthens the empirical justification for the proposed additive structure. This contribution is consequential because it provides a quantitative framework for estimating the marginal benefits of additional long-context training rather than relying entirely on individual training runs.
- Practical training protocol supported by downstream validation.
Constant-Last4 is a useful practical contribution. Constant-learning-rate training is attractive for extensible pre-training but produces noisier checkpoints and worse endpoint performance than matched linear-decay schedules. The authors show that averaging four consecutive checkpoints substantially improves stability and closely matches the linear-decay baseline across the evaluated configurations (Section 4, Figure 4).
Importantly, the analysis goes beyond fitting a perplexity metric. The paper examines whether additional LCCP produces downstream benefits during 32K training, after 128K context extension, and after supervised fine-tuning on long-context and agentic tasks (Section 5, Figures 7–8). These evaluations make the work more practically relevant and provide useful evidence that the measured improvements are not confined to the scaling-law fitting objective.
Critical weaknesses
- Limited evidence for the generality of the proposed scaling law beyond the evaluated training setup.
Although the experiments cover multiple model scales and training budgets, they primarily use one architecture family, a fixed long-context data mixture, and a common short-context pre-training recipe. Furthermore, the largest evaluated model contains approximately 1.22B total parameters, which is substantially smaller than the models for which large-scale LCCP budget allocation may be most consequential.
The primary scaling-law target is also LongPPL computed on GovReport using token positions selected by a fixed Llama-3.1-8B reference model. While the metric is well motivated and the downstream evaluations partially address this limitation, the fitted relationship has not been shown to transfer to other long-context data distributions or evaluation metrics. The negative fitted offset, which the authors appropriately acknowledge in Section 3.2, also means that the law should be interpreted as an empirical finite-range relationship rather than a general asymptotic scaling law.
These limitations matter because the paper's central motivation is to guide decisions about how far to scale LCCP. The current evidence strongly supports prediction within the studied training family, but it provides less certainty about whether the same exponents and scaling behavior apply to larger models, alternative architectures, or substantially different training mixtures.
- Insufficient analysis of the compute-efficiency and opportunity cost of additional LCCP.
The paper demonstrates that increasing LCCP tokens generally improves LongPPL and several downstream benchmarks. However, this does not by itself establish that continuing LCCP is the most effective use of additional training compute.
The additive scaling law explicitly models separate short-context and long-context token budgets, creating a natural opportunity to investigate their trade-offs. Nevertheless, the paper does not provide a sufficiently direct compute-normalized comparison between allocating additional tokens to LCCP and alternative training strategies, such as extending short-context pre-training or redistributing the training budget.
This distinction is particularly important because long-context sequences may incur substantially different computational costs, and the downstream improvements are not uniform across all tasks. A comparison of marginal performance gains per unit of training compute, ideally including matched-compute alternatives, would make the central practical conclusion considerably stronger.
Questions for authors
-
Generalization of the scaling law. Have the authors tested whether the fitted scaling exponents remain stable when changing the LCCP data mixture, model architecture, or evaluation corpus? Even a limited additional validation with an alternative long-context dataset would help determine whether the observed relationship is specific to GovReport-based LongPPL or reflects a more general property of LCCP.
-
Compute-optimal allocation between training stages. Given the separate and terms in the proposed scaling law, can the authors analyze the marginal benefit of allocating additional compute to LCCP versus short-context pre-training? A compute-normalized comparison or an analysis using the fitted exponents would materially strengthen the practical recommendation to continue LCCP.
-
Robustness of the downstream conclusions. Figure 7 and the per-subtask analysis indicate that the RULER-32K gains are driven primarily by intermediate-difficulty retrieval tasks, while Figure 8 shows some fluctuations across later checkpoints. Can the authors provide a quantitative analysis of how consistently the improvements generalize across individual downstream tasks, rather than relying primarily on aggregate trends? This would help clarify whether increasing LCCP produces broad capability improvements or primarily benefits a subset of long-context tasks.
4 comments
W2 & Q2: Compute efficiency of additional LCCP and allocation between stages
W2 & Q2: Compute efficiency of additional LCCP and allocation between stages
As you note, the separate and terms make this trade-off analyzable, and the higher per-token cost of long sequences must be included. We added a closed-form analysis and compute-matched experiments (Section 6, Eq. 3, Table 2, Figure 10).
Cost of long sequences. We use measured training time rather than theoretical FLOPs. In our Megatron setup, 32K and 4K steps contain the same number of tokens (32K uses activation recomputation to avoid context/sequence parallelism), and a 32K token costs times a 4K token.
Derivation and results. With total compute , the ratio of log-LongPPL reductions per unit of compute on LCCP versus SCP is with , and gives the break-even ratio , which is independent of and and nearly independent of . At fixed , the optimal allocation satisfies exactly .
For S5 (): at 10N, LCCP is 11–13 times as compute-efficient as SCP, and about twice as efficient at 40N; break-even is at 70–77N; the optimum gives LCCP 28–30% of the S5 total compute, and stopping at 10N costs 0.08–0.10 in log LongPPL versus at most 0.006 at 120N. Stopping too early is thus much more costly than stopping too late.
Compute-matched experiment. We trained two new S5 trajectories, to 160N and to 80N, and compared them with the original trajectory at matched (Figure 10). The 360N trajectory stays ahead of the 450N one until its LCCP reaches about 70N, and the 450N trajectory stays ahead of the 540N one until its LCCP reaches about 100N. Thus, with little LCCP, extra compute is better spent on LCCP, and SCP wins only after the break-even point; at both crossings, the predicted optimum lies between the two compared allocations.
Marginal return with a fixed base model. In practice, SCP is often trained once and reused, so LCCP has its own budget and its marginal return is what matters. We give a three-level criterion: (1) the break-even point is a lower bound, since before it LCCP outperforms even extending SCP, so stopping earlier is premature; (2) beyond it, with each 10N of LCCP at a fixed cost (about 66 H800 GPU-hours on S5), training continues while the predicted reduction for the next 10N exceeds a price ; e.g., stops at about 80N and at about 124N; (3) directly evaluated RULER-32K serves as an auxiliary check.
Scope and implication. This allocation optimizes long-context performance as measured by LongPPL and ignores short-context performance, world knowledge, reasoning, and other abilities, so real pre-training must weigh more factors; is also obtained within our scale range.
Q3: Robustness of the downstream conclusions
Q3: Robustness of the downstream conclusions
Thank you for the careful observation. We address the points about Figures 7 and 8 in turn.
Per-task curves are in the appendix. Figure 14 shows the RULER-32K subtasks, Figure 15 the RULER-128K subtasks, Figures 16 and 17 the four long-context benchmarks (LongBench, GraphWalks, LongReason, BABILong) after 2.1B- and 4.2B-token long-context SFT, and Figure 18 GAIA and BFCL-WebSearch.
RULER-32K: gains concentrated on intermediate-difficulty subtasks are expected. RULER subtasks span a wide range of difficulty. Easy tasks such as single-key NIAH are already saturated early in training, while harder subtasks remain difficult even at about 100B LCCP tokens. Inspecting model outputs, we find that the 1.22B model does not yet have reliable in-context learning ability, so subtasks that depend on the in-prompt example, such as CWE and VT, stay near zero. The room for LCCP-driven improvement is therefore mainly in intermediate-difficulty tasks, consistent with Figure 14.
Downstream tasks after SFT: a per-task quantitative test. The main text reports aggregate scores to reduce single-task fluctuations caused by uneven SFT data sampling, which better reflects changes in overall capability. To answer your question, we trained the 2.1B-token long-context SFT on each LCCP checkpoint (Constant-Last4) with 3 different seeds and tested each task separately:
| Task | Mean change 10N → 120N | Slope per LCCP doubling [95% CI] | Spearman (p) | Seed SD at same checkpoint | Variance explained by LCCP |
|---|---|---|---|---|---|
| LongBench | +2.06 | +0.77 [0.45, 1.09] | 0.93 (0.003) | 0.85 | 0.68 |
| GraphWalks | +7.65 | +1.77 [0.70, 2.84] | 1.00 (< 0.001) | 3.13 | 0.45 |
| LongReason | +14.27 | +4.09 [2.79, 5.39] | 0.75 (0.033) | 2.80 | 0.85 |
| BABILong | +6.73 | +2.07 [1.36, 2.78] | 0.89 (0.006) | 1.93 | 0.74 |
| 4-task mean | +7.68 | +2.17 [1.78, 2.56] | 0.96 (0.001) | 1.00 | 0.92 |
Slope: coefficient of a linear regression of score on (21 points). Spearman correlation is computed on the seed means of the 7 checkpoints, with one-sided permutation p-values. The last column is from one-way ANOVA.
- The gains are not confined to a few tasks. The slope CIs of all four tasks exclude 0 and all Spearman tests are significant, so each task improves with LCCP on its own.
- Part of the per-task fluctuation comes from SFT randomness. At the same checkpoint, the seed-to-seed SD reaches 2–3 points on single tasks, so per-task curves (3-seed means in Figure 16; a single seed in Figure 17) are expected to fluctuate. Averaging the four tasks reduces the seed SD to 1.0 and raises the variance explained by LCCP to 0.92.
- Tasks partly trade off against each other. At the same checkpoint, a seed that scores higher on GraphWalks tends to score lower on BABILong: their deviations from the checkpoint mean correlate at −0.63. This suggests some competition between tasks in a small-capacity SFT, which aggregation averages out.
Search-agent tasks. In Figure 18, GAIA and BFCL-WebSearch move in partly opposite directions: peaks in GAIA often coincide with dips in BFCL-WebSearch. This matches the task competition seen at the seed level and suggests that the two tasks also compete in a small-capacity SFT. Averaging them, as in Figure 8(c), partly cancels this effect and shows the trend of overall agentic performance.
Summary
Thank you for the recognition and constructive feedback. On generality, the law fits and extrapolates on a new LongPPL set with a different reference model and domain; the exponents change, but the stage-allocation conclusions do not. On compute efficiency, the measured cost ratio yields a closed-form break-even ratio and compute-optimal allocation, validated by compute-matched trajectories, together with a stopping criterion for a fixed base model. On downstream robustness, a 3-seed per-task test shows that all four downstream tasks improve significantly with LCCP, with different magnitudes; single-task fluctuations stem mainly from SFT randomness and task competition, which is why the main text reports aggregates. The new evaluation set and the compute analysis are in Section 6 of the revision, and the per-task analysis will be added to the appendix.
Dear Reviewer 1RtL,
We sincerely thank you again for your careful review and for your positive assessment of our work. To address your concerns, we built a new LongPPL evaluation set from financial documents (DocFinQA and DocMath), with key tokens selected by Qwen3.8-27B instead of Llama-3.1-8B. On this set, the law still fits and extrapolates well, and the derived stage allocation is nearly unchanged. We also derived a closed-form compute-optimal allocation between short-context pre-training and LCCP that accounts for the measured cost of long sequences. It assigns roughly 30% of the total training compute to LCCP and is validated by compute-matched trajectories. Finally, a 3-seed per-task analysis shows that all four downstream tasks improve significantly with LCCP. The first two are in Section 6 of the revised paper, and the per-task analysis will be added to the appendix.
As the discussion period is coming to an end, we would be very grateful if you could let us know whether our responses have addressed your concerns. We would be happy to clarify any remaining questions.
Best regards, Authors
W1 & Q1: Generality beyond the training setup and evaluation corpus
We thank the reviewer for the careful review and for recognizing our separate modeling of short-context and LCCP budgets, as well as the practical value of Constant-Last4 and the downstream validation. The three questions on generality, compute efficiency, and downstream robustness are all to the point; we respond below, and the new analyses are included in a new Section 6 (Discussion) of the revision.
W1 & Q1: Generality beyond the training setup and evaluation corpus
Limited compute prevents full validation on other model families, data mixtures, or larger scales; we now state the scope of the law explicitly in the Limitations paragraph of Section 6.
For the limitation you highlighted, that LongPPL is based only on GovReport with key tokens selected by a fixed Llama-3.1-8B, we built a new evaluation set: 929 financial documents from DocFinQA and DocMath, with key tokens selected by the newer and stronger Qwen3.8-27B. We re-evaluated the models of both scaling experiments in the paper (Linear-Decay, LD; Constant-Last4, CL4) and, as in the paper, fit on S1–S4 and extrapolate to S5. The fitted parameters are:
Functional form. With a new reference model and a new domain, the same form still fits and extrapolates well, so it is not specific to GovReport or the Llama-3.1-8B reference model. Since key tokens are selected by a fixed rule, LongPPL on the new set still mainly measures long-context ability rather than domain expertise.
Stability of the exponents. The exponents themselves are not identical: under CL4, for example, α changes from 0.244 to 0.319 and βl from 0.225 to 0.270. This is expected, since the difficulty distribution of key tokens differs between evaluation sets. What matters more in practice is the stage allocation derived from the exponents, which is nearly unchanged, as will be shown in subsequent replies.
Negative offset. On DocFinQA+DocMath, the fitted C turns positive (0.367 for CL4, 0.257 for LD). The raw LongPPL data show that financial text has clearly higher token-level perplexity than GovReport, which explains the sign change and indicates that C acts as a set-dependent overall offset. This agrees with your reading and with our discussion in Section 3.2: the law should be understood as an empirical relationship within a finite range rather than an asymptotic scaling law. These results are in Section 6 (Figure 9) and Appendix A.6.