- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors point out that existing pre-training paradigms for time series foundation models, namely next-token prediction (NTP) and next-N-token prediction (NNTP), have important limitations. NTP focuses on local information and suffers from error accumulation in multi-step forecasting, while NNTP uses a fixed forecasting horizon and therefore adapts poorly to practical use. To address these problems, the authors propose a new pre-training paradigm, Next-x-Token Prediction (NxTP). It trains the model on forecasting horizons of varying lengths, enabling it to directly predict sufficiently long future sequences in a single decoding step. Experiments on two benchmarks show that NxTP consistently improves forecasting performance and accelerates inference in long-term scenarios.
Critical strengths
- The paper addresses an important limitation of time series foundation models (TSFMs): they must serve tasks with different forecast lengths. Explicitly varying the training horizon is an interesting way to meet this requirement while reducing reliance on repeated autoregressive rollouts.
- The proposed pre-training paradigm is shown to be robust across different decoder-only TSFMs.
- The paper is well written, and the main idea is simple and easy to apply.
Critical weaknesses
- The connection between the theoretical analysis and the proposed horizon distribution remains incomplete. Theorem 2 establishes an advantage only when the training and downstream horizon distributions are sufficiently close, but the paper does not show that the proposed log-uniform distribution satisfies this condition.
- The ablation compares log-uniform sampling only with uniform sampling and a deterministic maximum horizon. These controls do not directly test the distribution-matching explanation in Theorem 2. An additional distribution estimated from training (or validation) task horizons, or specified from deployment requirements, would help establish when log-uniform sampling is preferable.
- The authors only use decoder-only architectures as baselines, so the benefit of this paradigm on other architectures, such as encoder-decoder models, remains unclear.
- The details of the experiments used to generate Figure 2 are not provided. For example, which dataset was used, how many repetitions were run, and do other models show the same behavior?
Questions for authors
Could the authors please address the 4 points raised in the "Weaknesses" section?
2 comments
Thanks the authors for their response. My concerns have been addressed, and the newly added results further demonstrate the generalizability of the framework. I will raise my score to accept.
Response to Reviewer GZDs
We sincerely thank Reviewer GZDs for their insightful comments. The following addresses their concerns and provides answers to their questions. Changes in the revised version are marked in blue.
We thank the reviewer for these insightful comments, which we consider closely related, and we address them jointly below.
(1) Rationale for log-uniform distribution. Our primary goal is to develop a general-purpose autoregressive pretraining paradigm for TSFMs that generalizes across diverse downstream forecasting scenarios, rather than optimizing for a specific horizon distribution. In practice, downstream applications involve substantially different forecasting horizons, ranging from short-term to long-term prediction. Without prior knowledge of the downstream horizon distribution, assuming that future tasks will favor either short or long horizons would introduce an undesirable inductive bias. We therefore adopt log-uniform sampling as a general-purpose design choice. Specifically, its scale-uniform property ensures balanced coverage across different temporal scales, allowing the model to learn from diverse forecasting horizons without being overly biased toward any particular scale.
(2) Connection to Theorem 2 and empirical validation. We acknowledge that Theorem 2 does not establish that our chosen log-uniform distribution necessarily satisfies the condition in Eq. (9). Since the actual horizon distribution of downstream applications is generally unknown, directly verifying this condition in practice is challenging. However, our experiments demonstrate the effectiveness of log-uniform sampling on both the long-term forecasting benchmark, which is biased toward longer horizons, and GIFT-Eval, whose task distribution is more heavily weighted toward shorter horizons. These results demonstrate that log-uniform distribution remains effective across substantially different downstream horizon distributions, even when they deviate from a scale-uniform distribution.
Thank you for this suggestion. We have incorporated ChronosBolt, a representative encoder-decoder model, as an additional backbone. Due to time constraints and the substantial experimental workload, we have currently only evaluated its small variant. The corresponding results have been added to Tables 1 and 2 in the revised manuscript, and we report them below. As can be observed, NxTP consistently improves ChronosBolt's performance, demonstrating its effectiveness on encoder-decoder architectures as well. We plan to further include experiments with the base variant in the camera-ready version.
Long-term forecasting benchmark:
GIFT-Eval:
We thank the reviewer for pointing out this point. Figure 2 uses TimesFM-70M as the backbone and evaluates its predictions on the validation set sampled from the pretraining corpus, containing data that were never seen during pretraining. Since inference with a trained foundation model is deterministic, we perform only one run for each forecasting horizon, without additional repetitions. We have revised the caption of Figure 2 to make the description clearer.
Furthermore, in the revised manuscript, we have included additional examples from other models to examine whether the observed behavior generalizes beyond TimesFM-70M in Section 4.3. Figure 3 presents the corresponding results for Timer-30M and TimesFM-200M, where similar behaviors can be observed in both models: they exhibit sharp increases in MAE when multiple decoding steps are required, highlighting the issue of error accumulation. Meanwhile, their errors decrease at integer multiples of the pretraining prediction length, revealing a bias toward specific horizons. In contrast, NxTP achieves consistently lower MAE with substantially smoother error-growth curves. These results suggest that NxTP effectively mitigates both error accumulation and horizon-specific bias, thereby enhancing the model's ability to generalize across forecasting horizons.
Thank you for your time and efforts. We hope this has addressed your concerns and answered your questions. Please don’t hesitate to reach out if you have any further questions.