acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Autoregressive Time Series Pretraining with Next--Token Prediction

17 views

Abstract

Autoregressive pretraining is a central paradigm for building foundation models, yet its effectiveness for time series foundation models (TSFMs) remains controversial. Conventional next-token prediction (NTP) provides only local supervision and suffers from error accumulation. Existing Next--Token Prediction (NNTP) approaches that predict multiple future tokens per step alleviate these issues to some extent, but introduce a fixed-horizon bias that may hinder generalization across diverse forecasting horizons. To address these limitations, we reformulate autoregressive time series pretraining as Next--Token Prediction (NxTP), which dynamically samples the prediction horizon for each token based on the available context. By enabling models to directly predict sufficiently long future sequences within a single decoding step, NxTP substantially reduces the need for autoregressive rollouts while improving generalization across forecasting horizons. Extensive experiments on two well-established benchmarks demonstrate that NxTP consistently improves forecasting performance and horizon generalization while substantially accelerating inference in long-term forecasting scenarios.

Primary area:
learning on time series and dynamical systems
Keywords:
Autoregressive Time Series Pretraining, Time Series Foundation Models, Next-x-Token Prediction

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_GZDsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors point out that existing pre-training paradigms for time series foundation models, namely next-token prediction (NTP) and next-N-token prediction (NNTP), have important limitations. NTP focuses on local information and suffers from error accumulation in multi-step forecasting, while NNTP uses a fixed forecasting horizon and therefore adapts poorly to practical use. To address these problems, the authors propose a new pre-training paradigm, Next-x-Token Prediction (NxTP). It trains the model on forecasting horizons of varying lengths, enabling it to directly predict sufficiently long future sequences in a single decoding step. Experiments on two benchmarks show that NxTP consistently improves forecasting performance and accelerates inference in long-term scenarios.

Critical strengths
  1. The paper addresses an important limitation of time series foundation models (TSFMs): they must serve tasks with different forecast lengths. Explicitly varying the training horizon is an interesting way to meet this requirement while reducing reliance on repeated autoregressive rollouts.
  2. The proposed pre-training paradigm is shown to be robust across different decoder-only TSFMs.
  3. The paper is well written, and the main idea is simple and easy to apply.
Critical weaknesses
  1. The connection between the theoretical analysis and the proposed horizon distribution remains incomplete. Theorem 2 establishes an advantage only when the training and downstream horizon distributions are sufficiently close, but the paper does not show that the proposed log-uniform distribution satisfies this condition.
  2. The ablation compares log-uniform sampling only with uniform sampling and a deterministic maximum horizon. These controls do not directly test the distribution-matching explanation in Theorem 2. An additional distribution estimated from training (or validation) task horizons, or specified from deployment requirements, would help establish when log-uniform sampling is preferable.
  3. The authors only use decoder-only architectures as baselines, so the benefit of this paradigm on other architectures, such as encoder-decoder models, remains unclear.
  4. The details of the experiments used to generate Figure 2 are not provided. For example, which dataset was used, how many repetitions were run, and do other models show the same behavior?
Questions for authors

Could the authors please address the 4 points raised in the "Weaknesses" section?

2 comments
AuthorsOpenReview ↗

Response to Reviewer GZDs

We sincerely thank Reviewer GZDs for their insightful comments. The following addresses their concerns and provides answers to their questions. Changes in the revised version are marked in blue.


W1 & W2: Theoretical and empirical justification for log-uniform distribution

We thank the reviewer for these insightful comments, which we consider closely related, and we address them jointly below.

(1) Rationale for log-uniform distribution. Our primary goal is to develop a general-purpose autoregressive pretraining paradigm for TSFMs that generalizes across diverse downstream forecasting scenarios, rather than optimizing for a specific horizon distribution. In practice, downstream applications involve substantially different forecasting horizons, ranging from short-term to long-term prediction. Without prior knowledge of the downstream horizon distribution, assuming that future tasks will favor either short or long horizons would introduce an undesirable inductive bias. We therefore adopt log-uniform sampling as a general-purpose design choice. Specifically, its scale-uniform property ensures balanced coverage across different temporal scales, allowing the model to learn from diverse forecasting horizons without being overly biased toward any particular scale.

(2) Connection to Theorem 2 and empirical validation. We acknowledge that Theorem 2 does not establish that our chosen log-uniform distribution necessarily satisfies the condition in Eq. (9). Since the actual horizon distribution of downstream applications is generally unknown, directly verifying this condition in practice is challenging. However, our experiments demonstrate the effectiveness of log-uniform sampling on both the long-term forecasting benchmark, which is biased toward longer horizons, and GIFT-Eval, whose task distribution is more heavily weighted toward shorter horizons. These results demonstrate that log-uniform distribution remains effective across substantially different downstream horizon distributions, even when they deviate from a scale-uniform distribution.

W3: Effectiveness of NxTP on encoder-decoder models

Thank you for this suggestion. We have incorporated ChronosBolt, a representative encoder-decoder model, as an additional backbone. Due to time constraints and the substantial experimental workload, we have currently only evaluated its small variant. The corresponding results have been added to Tables 1 and 2 in the revised manuscript, and we report them below. As can be observed, NxTP consistently improves ChronosBolt's performance, demonstrating its effectiveness on encoder-decoder architectures as well. We plan to further include experiments with the base variant in the camera-ready version.

Long-term forecasting benchmark:

ETTh1ETTh2ETTm1ETTm2ECLWeather
MSEMAEMSEMAEMSEMAEMSEMAEMSEMAEMSEMAE
ChronosBolt-small0.4110.4090.3610.3840.3880.4040.2840.3170.1670.2540.2500.269
+NxTP0.3960.4040.3490.3810.3680.3770.2720.3130.1630.2500.2490.269
Δ\Delta3.65%1.22%3.22%0.78%5.15%6.68%4.23%1.26%2.40%1.57%0.004%0.0%

GIFT-Eval:

MASECRPS
ChronosBolt-small0.7880.565
+NxTP0.7800.558

W4: Details of Figure 2 and behaviors of other models

We thank the reviewer for pointing out this point. Figure 2 uses TimesFM-70M as the backbone and evaluates its predictions on the validation set sampled from the pretraining corpus, containing data that were never seen during pretraining. Since inference with a trained foundation model is deterministic, we perform only one run for each forecasting horizon, without additional repetitions. We have revised the caption of Figure 2 to make the description clearer.

Furthermore, in the revised manuscript, we have included additional examples from other models to examine whether the observed behavior generalizes beyond TimesFM-70M in Section 4.3. Figure 3 presents the corresponding results for Timer-30M and TimesFM-200M, where similar behaviors can be observed in both models: they exhibit sharp increases in MAE when multiple decoding steps are required, highlighting the issue of error accumulation. Meanwhile, their errors decrease at integer multiples of the pretraining prediction length, revealing a bias toward specific horizons. In contrast, NxTP achieves consistently lower MAE with substantially smoother error-growth curves. These results suggest that NxTP effectively mitigates both error accumulation and horizon-specific bias, thereby enhancing the model's ability to generalize across forecasting horizons.


Thank you for your time and efforts. We hope this has addressed your concerns and answered your questions. Please don’t hesitate to reach out if you have any further questions.

Thanks the authors for their response. My concerns have been addressed, and the newly added results further demonstrate the generalizability of the framework. I will raise my score to accept.

Reviewer_2Ww5OpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Next-x-Token Prediction (NxTP), a novel reformulation of autoregressive time series pretraining that dynamically samples prediction horizons instead of using a fixed prediction length, improving generalization across forecasting horizons. Moreover, it substantially reduces the need for autoregressive rollouts by predicting sufficiently long future sequences within a single decoding step. Experiments on long-term forecasting and GIFT-Eval benchmarks demonstrate its effectiveness.

Critical strengths
  1. Interesting and novel idea on sampling the prediction horizons during pretraining. This simple and broadly applicable pretraining reformulation addresses the limitations of fixed-horizon objectives without modifying the backbone architecture. Its effectiveness is demonstrated across six model configurations, with consistent improvements on both long-term forecasting and GIFT-Eval benchmarks.
  2. Meaningful improvement in long-horizon inference efficiency. By generating long future sequences in a single decoding step, NxTP achieves substantial speedups over conventional NTP while mitigating error accumulation.
Critical weaknesses
  1. The evidence for the generalization claims is insufficient. The empirical evaluation primarily uses a fixed set of prediction horizons (i.e., 96, 192, 336, 720). The claimed generalization to arbitrary horizons is therefore not fully established, particularly for horizons beyond the tested range and under substantial distribution shifts.
  2. Figure 2 presents results for only one model. Extending this analysis to additional models would strengthen the claim.
Questions for authors
  1. Could the authors provide error curves across horizons for additional backbones, including horizons beyond the standard evaluation settings?
  2. Since the theoretical analysis indicates that NxTP's advantage depends on the alignment between the training and downstream horizon distributions, how sensitive is performance to the choice of the log-uniform distribution?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 2Ww5

We sincerely thank Reviewer 2Ww5 for their insightful comments. The following addresses their concerns and provides answers to their questions. Changes in the revised version are marked in blue.


W1, W2 & Q1: More evidence on generalization to arbitrary forecasting horizons

We thank the reviewer for these insightful comments, which we consider closely related, and we address them jointly below.

The benchmark protocols only evaluate performance at specific forecasting horizons. We agree that additional examples are needed to provide stronger evidence of generalization to arbitrary forecasting horizons. In the revised manuscript, we have included additional examples from other models in Section 4.3. Figure 3 presents the corresponding results for Timer-30M and TimesFM-200M. The results demonstrate consistent improvements brought by NxTP across different model architectures. Notably, both models exhibit sharp increases in MAE when multi-step autoregressive rollouts are required, highlighting the issue of error accumulation. Meanwhile, their errors decrease at integer multiples of the pretraining prediction length, revealing a bias toward specific forecasting horizons. In contrast, NxTP achieves consistently lower MAE with substantially smoother error-growth curves. These results suggest that NxTP effectively mitigates both error accumulation and horizon-specific biases, thereby enhancing the model's ability to generalize across arbitrary forecasting horizons.

Q2: Sensitivity of the horizon distribution

We thank the reviewer for raising this important question. We adopt log-uniform sampling because it provides uniform coverage across different temporal scales, offering balanced performance across diverse forecasting horizons. Downstream forecasting applications involve a wide range of prediction horizons, from short-term to long-term. Since our goal is to develop a general-purpose foundation model, we aim to avoid biasing pretraining toward any particular horizon range. Log-uniform sampling therefore offers a natural choice, allowing the model to learn across different temporal scales and better support diverse downstream applications.

Our ablation studies further investigate the impact of different horizon sampling distributions and demonstrate that log-uniform sampling achieves better forecasting performance than alternative distributions without scale-uniform coverage. Furthermore, we conduct additional experiments to examine the sensitivity of log-uniform sampling to its maximum horizon, Hmax⁡H_{\max}, as shown in the table below. The results show that neither an excessively small nor an excessively large Hmax⁡H_{\max} yields optimal performance. This suggests that an appropriate choice of Hmax⁡H_{\max} is important for balancing prediction coverage and learning difficulty. Overall, these results support the effectiveness of our log-uniform sampling strategy and demonstrate that it provides a robust balance across diverse forecasting horizons.

ETTh1ETTh2ETTm1ETTm2ECLWeather
MSEMAEMSEMAEMSEMAEMSEMAEMSEMAEMSEMAE
Hmax⁡=480H_{\max}=4800.3940.4100.3490.3860.3620.3830.3000.3410.1730.2570.2670.281
Hmax⁡=1920H_{\max}=19200.3990.4100.3540.3870.3680.3910.2990.3430.1720.2570.2640.280
NxTP0.3910.4010.3440.3820.3420.3740.2680.3170.1630.2510.252

Thank you for your time and efforts. We hope this has addressed your concerns and answered your questions. Please don’t hesitate to reach out if you have any further questions.

Thank you for the rebuttal. My concerns have been addressed, and I will keep my score.

Reviewer_Fk3MOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper redefine autoregressive pretraining for time series as next-x-token prediction (NXTP). Instead of using a fixed prediction length in conventional NTP or NNTP, NXTP dynamically samples the prediction horizon for each token, mitigating horizon-specific bias and error accumulation during pretraining. Experiments on two widely recognized benchmarks demonstrate that NXTP can consistently improve forecasting performance and inference efficiency.

Critical strengths
  1. The main strength of NXTP is that it effectively addresses horizon-specific bias and error accumulation, while reducing the computational cost of autoregressive rollouts. The empirical results show that NXTP consistently improves forecasting performance across six backbones.
  2. A key contribution of this paper is its theoretical analysis, which clearly reflects the limitations of existing methods with fixed-horizon pretraining that will induce a horizon-specific bias.
  3. The paper is well-written with clear motivation. The idea and the proposed method are straightforward yet effective.
Critical weaknesses
  1. The theoretical analysis of NXTP's effectiveness appears to rely on strong assumptions that may limit the applicability of its guarantees in practice.
  2. The paper does not analyze the scaling behavior of NXTP, which I consider an important factor in determining the effectiveness of a pretraining paradigm.
Questions for authors
  1. Regarding Proposition 1, the conclusion holds only under the condition specified in Eq. (4). The authors should clarify whether this condition is readily satisfied in practical scenarios?
  2. To mitigate horizon-specific bias, the most straightforward approach would be to sample the prediction horizon directly from a distribution h∼q(h)h \sim q(h). Why, then, does the proposed formulation introduce a conditional distribution q(h∣t)q(h \mid t) that depends on the context position tt?
  3. The paper adopts a log-uniform distribution, and the ablation study shows that this choice outperforms uniform and deterministic alternatives. Beyond these comparisons, can the authors explain why log-uniform sampling is particularly suitable for time series pretraining?
  4. NXTP uses a prediction head that directly generates predictions up to Hmax⁡H_{\max}. How does the choice of Hmax⁡H_{\max} affect forecasting accuracy and model complexity?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Fk3M (1/2)

We sincerely thank Reviewer Fk3M for their insightful comments. The following addresses their concerns and provides answers to their questions. Changes in the revised version are marked in blue.


W1 & Q1: Validity of the Theoretical Assumption

We thank the reviewer for these insightful comments, which we consider closely related, and we address them jointly below.

We agree that Eq. (4) is a non-degeneracy condition rather than a universally satisfied assumption. It requires that the optimization objectives associated with different prediction horizons are not perfectly aligned. This condition is generally reasonable in practical multi-step forecasting models, where model parameters are shared across prediction steps and finite model capacity can introduce trade-offs among horizon-specific objectives.

For example, consider a standard linear regression model with squared loss, ℓi(w)=12(w⊤xi−yi)2\ell_i(\mathbf w)=\frac{1}{2}(\mathbf w^\top\mathbf x_i-y_i)^2. Although the optimal parameters minimize the aggregate loss over all NN prediction steps, they do not necessarily minimize the loss over its first HH steps. Consequently, the aggregate gradient contributed by either the prefix or tail objectives can remain nonzero at the full-horizon optimum. For example, with N=2N=2, H=1H=1, and two observations (x1,y1)=(1,1)(\mathbf x_1,y_1)=(1,1) and (x2,y2)=(1,2)(\mathbf x_2,y_2)=(1,2), the full-horizon optimum is w2⋆=1.5w_2^\star=1.5, whereas the prefix gradient is ∇wℓ1(w2⋆)=0.5≠0\nabla_w\ell_1(w_2^\star)=0.5\neq 0. This illustrates that Eq. (4) can naturally hold even in simple regression settings.

We emphasize that our proposition characterizes the horizon-specific optimization bias under this non-degeneracy condition, and we clarify this condition and its practical interpretation in Appendix A.1 of the revised manuscript.

W2: Scaling behavior of NxTP

We thank the reviewer for raising this important point. We agree that scaling behavior is a key factor in assessing the effectiveness of a pretraining paradigm. To our knowledge, however, a clear scaling law for TSFMs has not yet been established, though we share the reviewer's view that this is an important direction.

For this reason, when conducting our experiments, we deliberately included backbone variants of different model sizes. As shown in Table 1, however, increasing model size did not yield a significant improvement. Due to limited computational resources, we were unable to systematically study scaling beyond the original configurations already covered in our setup, and we leave a more thorough investigation of scaling behavior to future work (Appendix E).

Q2: Rationale for Conditioning qq on Context Position

We agree that sampling the prediction horizon from a position-agnostic distribution is indeed a straightforward baseline. However, conditioning the horizon distribution qq on the context position offers several important benefits. First, it provides a better match between task difficulty and context length: predicting a long horizon from a short context is overly difficult, whereas predicting a short horizon from a long context is overly easy. Conditioning on context position allows the horizon distribution to adapt accordingly, avoiding both extremes. Second, it better reflects real-world settings, where the prediction horizon is often chosen as a function of the available context length. This makes the proposed formulation more realistic and practically relevant.

AuthorsOpenReview ↗

Response to Reviewer Fk3M (2/2)

Q3: Rationale for log-uniform distribution

We thank the reviewer for raising this question. We adopt log-uniform sampling because it provides uniform coverage across different temporal scales, offering balanced performance across diverse forecasting horizons. Downstream forecasting applications involve a wide range of prediction horizons, from short-term to long-term. Since our goal is to develop a general-purpose foundation model, we aim to avoid biasing pretraining toward any particular horizon range. Log-uniform sampling therefore offers a natural choice, allowing the model to learn across different temporal scales and better support diverse downstream applications. Our ablation studies demonstrate that log-uniform sampling consistently outperforms non-scale-uniform alternatives. We have further clarified the rationale behind this design choice in the revised manuscript (Line 247-250).

Q4: Affect of Hmax⁡H_{\max}

The maximum prediction horizon Hmax⁡H_{\max} determines the output dimension of the prediction head. Consequently, the computational complexity and parameter count of the prediction head scale linearly with Hmax⁡H_{\max}, i.e., O(Hmax⁡)\mathcal{O}(H_{\max}).

To further investigate the impact of Hmax⁡H_{\max} on forecasting performance, we conduct additional experiments with different choices of Hmax⁡H_{\max}. As shown in the following table, we observe that neither an excessively small nor an excessively large Hmax⁡H_{\max} leads to optimal performance. This can be attributed to the trade-off between prediction coverage and learning difficulty. Specifically, a small Hmax⁡H_{\max} still requires multi-step rollouts for long-horizon forecasting, leading to error accumulation, whereas an excessively large Hmax⁡H_{\max} makes the prediction task more difficult for model to learn. These results suggest that an appropriately chosen Hmax⁡H_{\max} is important for balancing forecasting accuracy and model complexity. We included these results in the Section 4.5 of the manuscript.

ETTh1ETTh2ETTm1ETTm2ECLWeather
MSEMAEMSEMAEMSEMAEMSEMAEMSEMAEMSEMAE
Hmax⁡=480H_{\max}=4800.3940.4100.3490.3860.3620.3830.3000.3410.1730.2570.2670.281
Hmax⁡=1920H_{\max}=19200.3990.4100.3540.3870.3680.3910.2990.3430.1720.2570.2640.280
NxTP0.3910.4010.3440.3820.3420.3740.2680.3170.1630.2510.2520.272

Thank you for your time and efforts. We hope this has addressed your concerns and answered your questions. Please don’t hesitate to reach out if you have any further questions.

Thanks for the rebuttal. The additional clarifications and experiments have addressed my concerns. After reading the comments from the other reviewers, I support its acceptance and will keep my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.