acceptodds
Under review as a conference paper at ICLR 2027

A Forecasting Odyssey: Charting the Billion-Scale Landscape of Time-Series Forecasting

Abstract

Time-series forecasting—using historical patterns to predict future outcomes—has gained significant attention due to its broad practical impact nowadays. Despite decades of methodological advances and benchmark development, existing evaluation frameworks still exhibit critical limitations: (i) many studies rely on homogeneous or imbalanced datasets with limited cross-domain diversity and quality assurance; (ii) model coverage is often limited, overlooking simple yet effective baselines; and (iii) insufficient statistical and domain-specific analyses lead to superficial conclusions. To address these limitations, we introduce TimeFCST, a billion-scale benchmark with systematic quality assurance that evaluates 100 forecasting methods, roughly 5 more than prior studies, across statistical, machine learning, deep learning, and foundation models. Beyond aggregate leaderboards, we provide, to our knowledge for the first time, rigorous statistical validation of comparative findings over large-scale experiments and a failure-profiling framework spanning six error patterns. Together, these answer three questions: What works, when does it work, and how does it fail? Our evaluation reveals: (i) modern forecasting models lead aggregate rankings, yet statistically significant gains over well-tuned simple classical baselines emerge only among leading foundation models especially on UTS; (ii) model effectiveness varies across domains and settings, with simple classical methods matching or even outperforming modern models in domains such as nature, retail and economics; (iii) failure profiles reveal distinct error patterns hidden by aggregate metrics: deep learning and foundation models exhibit more phase and localized errors, with foundation models showing stronger underprediction bias; and (iv) additionally, we identify saturated datasets that obscure methodological progress and show that broad, diverse benchmark coverage reveals otherwise hidden, statistically significant differences among models. In summary, TimeFCST charts the evolving landscape of time-series forecasting, offering insights into recent advances and directions for future research.

Primary area:
datasets and benchmarks
Keywords:
Time-series Forecasting, Forecasting Benchmarks, Baseline Evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Aes6OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

TimeFCST evaluates 100 forecasting methods on 128 public datasets, evenly split between univariate and multivariate series, with on the order of one billion observations after curation. The study reports aggregate rankings, differences across domains, and a decomposition of forecast error into six failure modes.

Critical strengths
  • The comparison includes classical statistical and machine-learning methods that recent leaderboards often omit. On univariate series, Huber and ETS rank above approximately 70% of the 100 evaluated methods.
  • The Friedman–Nemenyi tests support a specific conclusion. On univariate data, FACT is not significantly separated from the tuned classical baselines, whereas TimesFM is. On multivariate data, deep-learning and foundation models are significantly separated from those baselines.
  • Figure 6 qualifies the aggregate ranking. Recent models lead on cloud, healthcare, and mobility, whereas economics, nature, and retail do not follow the same ordering.
  • The curation procedure is documented. Minor gaps are forward-filled, extreme artifacts are flagged by three anomaly detectors and then manually inspected, and the release includes code together with results over three random seeds.
Critical weaknesses
  • The introduction describes the hyperparameter optimization as systematic. Appendix F limits the search to four configurations per model. This budget is insufficient to support a claim of comparable tuning across model families.
  • Series identified as random walks are removed, with Exchange given as an example, because a naive forecast is already difficult to improve upon. This exclusion also removes a regime in which many recent models have limited advantage, so the retained collection is less demanding than the original archive.
  • The failure scores use oracle repairs with access to the ground truth, including a median offset, an additional slope, replacement of a local segment, and a small number of Fourier components of the label. A high recovery score for the level mode is the expected outcome of an optimal constant correction. The paper does not establish that the six operators correspond to distinct sources of error.
  • Leakage for TimesFM and Chronos is reported as approximately 10% or lower, without a definition of overlap. These models are evaluated zero-shot, while the task-specific models are trained on the target series. Both differences affect the univariate result in which foundation models are reported to lead.
Questions for authors
  • Are the series in TimeFCST-50M included in the reported test results, and were the four hyperparameter configurations selected on the test horizon?
  • Please define how the leakage rate is computed, and report the rank of TimesFM after all overlapping series are excluded.
  • If Exchange and the other excluded random-walk series are restored, do the significance groupings in Figure 5 remain the same?
6 comments
AuthorsOpenReview ↗

for reviewer Aes6 (part 1: Tuning budget and comparability)

Thank you for raising this concern about the hyperparameter search budget. We agree that evaluating more configurations gives each model a better chance of finding settings that improve its performance. However, we would like to emphasize the computational cost of substantially expanding the search across approximately 100 models, compared with prior benchmarks such as TFB, which evaluated 21 models. To make tuning feasible at this scale, we use TimeFCST-50M, a dedicated tuning subset balanced across data domains. To help understanding, we have revised and documented the details for TimeFCST-50M and hyperparameter tuning process in Appendix F.

To assess the practical effect of the search budget, we selected representative models and compared their default settings with the configurations selected under our original four-configuration search and a doubled budget of eight configurations. Table~8 in Appendix H shows that, the original four-configuration search improved performance over the defaults. Doubling the budget to eight configurations yielded further gains, but with diminishing returns. These results demonstrate the effectiveness of our original tuning protocol for the models tested: it captured most of the improvement through hyperparameter tuning while using half the search budget. We have included this comparison in the revised manuscript.

Furthermore, as mentioned in the response to Reviewer oShf, we also intend to maintain TimeFCST as a regularly updated benchmark and to expand the hyperparameter search over time. One way to allocate tuning resources fairly across models is to give each model the similar computing budget, so that more expensive models complete fewer trials within the allowed time. We believe the current protocol provides an effective starting point, and we will continue to improve the benchmark in future releases.

Table 8: Normalized MASE relative to Naive Seasonal (lower is better) on 64 univariate datasets under different hyperparameter tuning budgets. Configurations are selected on TimeFCST-50M. Parentheses show the difference from the default configuration (tuned minus default).

SettingETS (Stat.)ARIMA (Stat.)Huber (ML)RandomForest (ML)FACT (DL)TimeXer (DL)
Default0.94820.99000.87291.05240.83880.8669
4-config tuning0.9308 (−0.0174)0.9728 (−0.0172)0.8714 (−0.0015)1.0279 (−0.0245)0.8307 (−0.0081)0.8548 (−0.0121)
8-config tuning0.9254 (−0.0228)0.9596 (−0.0304)0.8686 (−0.0043)0.9951 (−0.0573)0.8256 (−0.0132)0.8420 (−0.0249)
AuthorsOpenReview ↗

for reviewer Aes6 (part 2: Data cleaning process)

Thank you for highlighting the value of evaluating models on random-walk-like series. We agree that settings where a naive forecast is difficult to beat are also informative, particularly for understanding when recent forecasting models offer limited gains over simpler methods. The exclusion narrows the scope of the main benchmark to settings where performance differences among forecasting methods are more pronounced.

As mentioned in our response to Reviewer Mq9A (W1/Q1), we evaluated the representative models on cleaned datasets and newly prepared datasets with random-walk-like series detected from our cleaning stage (Figure~23(a) in Appendix H). On the prepared random-walk-like datasets, ETS and ARIMA ranked higher, and their normalized MASE distributions overlapped with those of several foundation models.

We observed a similar trend in a separate experiment comparing the datasets with and without trimming long inactive regions (Figure~23(b) in Appendix H). When these regions were retained, simpler methods such as Huber and RMLP became more competitive. Both experiments show that the performance differences among models narrow and become harder to distinguish as the data become noisier. Together, the two analyses illustrate settings in which simpler methods can remain competitive with recent forecasting models on the noisy data. We have included both analyses in the revised manuscript (Appendix H) to show how model performance in these settings with "low-quality" datasets complements the main benchmark results.

AuthorsOpenReview ↗

for reviewer Aes6 (part 3: Failure profile and error mode)

Thank you for raising this question about how to interpret the six error modes. We designed these modes to capture recurring error patterns across models and data domains. Our goal is to identify patterns that recur across models and data domains and use them to guide further model improvements. As the resulting recovery scores are post-hoc diagnostics and the error modes are not necessarily independent, we acknowledge that a forecast error arising from a single source can yield recovery under more than one mode. The gap between the matching mode’s recovery score and those of the other modes indicates how clearly the profile distinguishes that error pattern.

To test how well the modes distinguish known error patterns, we injected forecasting error from different modes into a known MAE-optimal forecast and applied all six error modes to each modified forecast (Figures 24 and 25 in Appendix H). The matching mode achieved the highest mean recovery score for every injected error type, while the other modes generally yielded much lower scores. We also observed some overlap between modes. For example, the amplitude correction partially recovered an injected phase error, although the phase correction achieved substantially higher recovery. Because the mode targeting the injected error consistently gave the largest recovery, this overlap does not affect the overall separability of the modes, but it points to a limitation that we discuss further in Appendix H.

In the separate optimal-forecast controls discussed in our response to Reviewer Mq9A, we also assessed how much recovery each mode can obtain from unpredictable variation alone. We have included both experiments in the revised manuscript (Appendix H) and explain how users can interpret recurring patterns in the full recovery profile when investigating model behavior.

Reviewer_oShfOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a new benchmark (TimeFCST) for time series forecasting research, improving upon current benchmarks in several respects: more datasets and models, better preprocessing of uncurated datasets, and the addition of statistical testing. The paper first briefly traces common limitations in forecasting benchmarks, and then presents results by analyzing the different domains, systematic model failures, and forecasting scalability.

Critical strengths
  • The paper runs many experiments and presents a (yet) larger benchmark for more datasets and models in TSF. In particular, two relevant contributions are the increase in the number of experimental domains and the rigorous cleaning and quality assurance pipeline.

  • The paper is systematic in the experimental analysis and presents results in a clear manner.

Critical weaknesses

Although the paper is well-presented and systematic, as highlighted in the strengths section, its overall informativeness remains very poor. Most analyses revisit questions already investigated in prior work, even if on a larger scale, without providing substantially new insights or lessons that advance the TSF literature. My main recommendation to the authors is to carefully review the existing literature before embarking on an "Odyssey" of experiments. A more thorough assessment of prior work would have revealed that many of the reported findings have already been established, raising questions about the study's novelty and scientific contribution. Furthermore, I have critical concerns regarding the reproducibility of the HP search results.

Substantial overlap with prior work [1]. Several analyses presented in the paper closely overlap with those already published in [1], which is not cited: (1) Analysis of dataset coverage across domains, sampling frequencies, and dimensionalities (Figs. 1, 12 vs. Fig. 2 of [1]); (2) Showing how benchmark composition affects model rankings and conclusions about superiority (Sec. 5.4, Fig. 7 vs. Sec. 4.1, Fig. 4 of [1]); (3) Demonstrating that model rankings vary across datasets and that simpler architectures can outperform more sophisticated ones (Secs. 5.1–5.2 vs. Sec. 3, Fig. 1 and Appendix H of [1]); (4) Investigating relationships between model performance and temporal patterns, including trends, seasonality, and inter-variable dependencies (Sec. 5.2, Fig. 16 vs. Sec. 3 of [1]); (5) Examining whether exploiting inter-channel relationships improves multivariate forecasting (Fig. 7(a) vs. Fig. 3 of [1]); (6) Highlighting how inadequate tuning and omission of competitive baselines distort model comparisons (Secs. 4.2, 5.1 vs. Secs. 4.2–4.3 of [1]); (7) Applying nonparametric statistical tests, including the Friedman test, to assess whether apparent performance gains are meaningful (Sec. 5.1, Fig. 5 vs. Sec. 4.5 of [1]); (8) Examining whether forecasting improvements justify increased model complexity and computational costs (Sec. 5.5, Fig. 10 vs. Sec. 4.6 of [1]); (9) Adopting frequency-aware prediction horizons, an evaluation recommendation already advocated in [1] (Sec. 4.3 vs. Sec. 5 of [1]).

Although the paper expands the evaluation scale and introduces additional analyses, these overlaps should be acknowledged, and the novel contributions should be clearly distinguished.

Inconsistent superiority of FMs: The finding that FMs lead aggregate rankings but do not consistently outperform simpler supervised approaches (Sec. 5.1) is already well established. GIFT-Eval [4] highlighted the competitiveness of supervised methods, particularly PatchTST, while [2] demonstrated that carefully tuned baselines can match or outperform several FMs. Therefore, the reported competitiveness of classical baselines and the absence of consistent FM superiority do not constitute new insights.

Domain-dependent FM performance: The finding that "FM models generally perform strongly, but their dominance is highly domain-dependent" (Sec. 5.2) is not novel as well. The ICML 2026 position paper [5] already evaluated seven TSFMs across 72 datasets from six domains, showing substantial domain-dependent performance variations. Moreover, [6] demonstrated how performance depends on the relationship between pre-training and downstream distributions. Hence, the paper confirms well-known findings rather than adding additional knowledge regarding TSFMs.

Channel-dependent vs. channel-independent forecasting: The finding that the benefits of channel-dependent models depend on data characteristics (Sec. 5.2, Fig. 7) is not new, as [3] already studied when inter-variable dependencies are useful. While the present paper finds larger gains on highly correlated datasets, [3] showed that correlation does not necessarily imply useful dependencies. The authors point to broader dataset coverage but do not explain the differences in these findings, as [3] is again not cited.

Benchmarking methodology: The ICML 2026 paper [7], which is not cited, already demonstrated that differences in model configuration, preprocessing, and architectural components can strongly affect forecasting comparisons, sometimes more than the forecasting architecture itself. While the present paper evaluates more models and datasets, it does not systematically control for these factors, making it difficult to determine what drives the reported performance differences.

Benchmark reproducibility: The paper doesn't provide any details of the HPO search procedures, casting doubt on the effective reproducibility of the benchmark results and the extension of the benchmark with new models, since adding a new model to the benchmark would inevitably require the details of the HPO search to be available for fair comparisons. The released repository (hp_model_config[1-4].json files), like the paper (Tables 4-7), includes only the optimal HPs found, not the ranges of values considered during optimization, or the HPO search strategy or any early-stopping criterion employed.

[1] There are no Champions in Supervised Long-Term Time Series Forecasting, TMLR 2026

[2] Specialized Foundation Models Struggle to Beat Supervised Baselines, ICLR 2025

[3] A Closer Look at Transformers for Time Series Forecasting: Understanding Why They Work and Where They Struggle, ICML 2025

[4] GIFT-Eval: A Benchmark for General Time Series Forecasting Model Evaluation, NeurIPS 2024 Workshop

[5] Position: Time-Series Foundation Models Require Explicit Domain-Level Benchmarks, ICML 2026

[6] Does Cross-Domain Pre-Training Truly Help Time-Series Foundation Models?, ICLR 2025 Workshop

[7] Position: Current Benchmarking Hinders Real Progress in Deep Learning for Time Series Forecasting, ICML 2026

Questions for authors

See Weaknesses

7 comments
AuthorsOpenReview ↗

for reviewer oShf (part 1: contributions of proposed work)

Thank you for highlighting the connections between TimeFCST and recent forecasting studies. Several of the cited studies, including [1], [5], and [7], appeared in 2026 and represent parallel efforts to improve forecasting evaluation. We cite all mentioned works and acknowledge where their findings converge with ours in our revised version. We view this convergence as complementary evidence, particularly for observations such as how forecasting performance varies across datasets and domains.

We believe that TimeFCST complements these studies by evaluating approximately 100 models across 128 datasets, spanning statistical, machine learning, deep learning, and foundation models. Beyond scaling up the evaluation, this breadth enables fine-grained model comparisons across domains, temporal patterns, and forecasting horizons, yielding insights that generalize beyond one or two domains or model families (Sections~5.2--5.3). It also lets us track progress on each dataset, for example whether performance has saturated or is still improving, based on a large set of models developed over decades. Prior work [1] also used statistical significance tests to compare forecasting models. In TimeFCST, we apply these tests at a larger scale, across multiple model families and a broad collection of univariate and multivariate datasets. Furthermore, TimeFCST also introduces recovery-based error profiles to characterize recurring forecast errors and show how models with similar aggregate accuracy can differ in the kinds of errors they make. We clarify how these contributions build on the related work in the revised manuscript.

AuthorsOpenReview ↗

for reviewer oShf (part 2: Benchmark reproducibility W6)

Thank you for pointing out that more detail is needed to illustrate evaluation settings and reproduce the hyperparameter search. To keep tuning feasible across approximately 100 models, we evaluated four candidate configuration sets per model on TimeFCST-50M, a dedicated tuning subset balanced across data domains. After the hyperparameter tuning stage, we selected the best-performing configuration on the TimeFCST-50M and used it for final evaluation on the TimeFCST-1B. All deep learning models used a similar training framework with early stopping enabled. For reference, we have documented the details in training, hyperparameter tuning and evaluation in Appendix F.

Additionally, to assess the effect of the search budget in hyperparameter (HP) tuning, we compared the default settings, the current four-configuration search, and an expanded search with eight configurations for six representative models. As shown in Table 8, the current HP tuning plan reduced aggregate normalized MASE relative to the defaults across datasets. Doubling the budget yielded additional gains of varying magnitude, with additional gains of varying magnitude. These results show that the current HP tuning plan improved performance over the defaults and that the conclusion was stable under the budgets tested.

We intend to maintain TimeFCST as a regularly updated benchmark and to expand the hyperparameter search over time. One way to allocate tuning resources fairly across models is to give each model the same computing budget, so that more expensive models complete fewer trials within the allowed time. We believe the current protocol provides an effective starting point for representative hyperparameter tuning, and we will continue to improve it in future releases.

Table 8: Normalized MASE relative to Naive Seasonal (lower is better) on 64 univariate datasets under different hyperparameter tuning budgets. Configurations are selected on TimeFCST-50M. Parentheses show the difference from the default configuration (tuned minus default).

SettingETS (Stat.)ARIMA (Stat.)Huber (ML)RandomForest (ML)FACT (DL)TimeXer (DL)
Default0.94820.99000.87291.05240.83880.8669
4-config tuning0.9308 (−0.0174)0.9728 (−0.0172)0.8714 (−0.0015)1.0279 (−0.0245)0.8307 (−0.0081)0.8548 (−0.0121)
8-config tuning0.9254 (−0.0228)0.9596 (−0.0304)0.8686 (−0.0043)0.9951 (−0.0573)0.8256 (−0.0132)0.8420 (−0.0249)
AuthorsOpenReview ↗

for reviewer oShf (part 3: More details in comparison with prior work W1)

We thank reviewer oShf for pointing out some overlap with prior work [1]. We appreciate the opportunity to clarify the in-depth analysis from our work, which leads to detailed empirical conclusions and recommendations.

  • For (1), we argue that while prior work [1] also shows similar flaws of existing benchmarks in terms of data domain diversity and sample frequency, the purpose of our analysis is to also demonstrate that our benchmark, TimeFCST, achieves a more balanced data domain distribution and covers a variety of demand patterns.
  • For (2), prior work [1] lacks thorough analysis of benchmark composition: it only ablates on a single dataset and draws conclusions regarding dataset selection. However, in our work, we dive into much more fine-grained data patterns (trend, seasonality, etc.) at the time-series level. Additionally, we also examine how dataset composition shapes benchmark conclusions by tracking the benchmark's performance trend over time, where we conclude that over-reliance on saturated datasets limits discriminability and biases assessments of progress.
  • For (3), while prior work [1] does show dataset-specific performance for their evaluated methods, our analysis goes beyond dataset-level performance metrics. Specifically, in our Sec 5.1 and 5.2, we reveal how data patterns, forecasting horizons, and channel dependency can affect model's performance. Moreover, although both prior work [1] and our work both have found that ``simpler architectures can outperform more sophisticated ones'', our work expands evaluation upon the most recent state-of-the-art deep learning models (FACT, SimpleTM) and foundation models (Chronos, TimesFM), making the same claim more reliable and sync with the most recent TSF advances.
  • For (4), although prior work [1] does investigate the inter-variate property for different methods, prior work [1] only analyzes these patterns in a qualitative way: it lacks quantitative analyses regarding trend, seasonality, transition, or stationarity, all of which are presented in our work.
  • For (5), while we prior work [1] does make an attempt to analyze channel dependency in MTS forecasting, the scale of its experiment is limited and cannot be well generalized: only two methods are compared (PatchTST against iTransformer). Nevertheless, in our work, we conduct a significantly more comprehensive analysis across a total of 47 deep learning models to avoid cherry-picking conclusions from just 2-models comparison. The exact channel-dependent/channel-independent property for each deep learning model can be found in the Appendix Table 6.
  • For (6), while prior work [1] does indeed emphasize the importance of sufficient hyperparameter tuning and overlooked baselines, it does not provide specific solution to how hyperparameter tuning can be done effectively and efficiently, neither what specific models are exactly strong yet overlooked baselines. In our work, we introduce efficient hyperparameter optimization by curating TimeFCST-50M that maintains a similarly balanced data distribution in comparison to our TimeFCST-1B and provides dedicated validation sets for efficient tuning. We also explicitly point out exactly overlooked strong baselines: Huber and ETS for UTS forecasting and RMLP, Mamba for MTS forecasting.
  • For (7), while we acknowledge that prior work [1] discovers the flaw of not performing statistical test in previous benchmarks, it hasn't performed it at scale beyond their toy example (iPatch model). Our work, on the other hand, actually integrates statistical testing into all result analyses across all 100 models, making our results more reliable.
  • For (8), while we acknowledge that prior work [1] emphasizes the trade-off between model performance and efficiency, our work enriches this trade-off discussion by also integrating foundation models. This is especially important because these foundation models already shift much of the cost to large-scale offline pre-training; during inference, they can just do zero-shot forecasting and skip training. We argue that adding in these foundation models are crucial to understand the efficiency tradeoff landscape.
  • For (9), while prior work [1] stresses defining meaningful forecast horizons and our work adopts this benchmarking practice, our contribution is beyond curating horizon numbers for each dataset. In fact, we also integrate forecasting horizons into our analyses, observing that FMs and DL models generally maintain more stable performance from short- to long-term forecasting, shedding light on horizon-aware forecasting model selection.

In short, we thank again for pointing out prior work [1], and in our revised submission, we have included it in the related work.

Reviewer_Mq9AOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents TimeFCST, a benchmark with 128 datasets and 100 forecasting methods. It compares classical methods, deep learning models, and foundation models across different forecasting settings. It also introduces six error profiles to study differences that average accuracy may miss. The results show that modern models lead overall, while simple methods remain competitive in several settings.

Critical strengths
  1. The experiments cover a wide range of datasets and methods. The inclusion of strong classical baselines makes the comparison useful for assessing recent progress in forecasting.

  2. The error profiles offer a useful way to compare models with similar average accuracy. They could help identify differences that are hard to see from the overall rankings.

Critical weaknesses
  1. The data filtering choices would benefit from further discussion. Real-world time series can be noisy, incomplete, inactive for long periods, or difficult to predict, and limited predictability does not necessarily imply poor data quality. Section 4.1 describes trimming inactive regions "particularly in test regions," while Appendix E.2 excludes random-walk-like series after checking naive and SOTA model performance. It would be helpful to report the filtering thresholds and the number of series affected. A comparison before and after filtering would also help clarify the scope of the benchmark and whether the main conclusions depend on these choices.

  2. The error profiles are useful, though their interpretation as model weaknesses would benefit from further validation. The corrections in Appendix E.3 use the observed future and can also reduce errors caused by unpredictable variation, even when the original forecast is optimal given the available history. The permutation-based calibration for Localized Error is helpful, though it is less clear how the other modes account for this effect. Controlled examples with known optimal forecasts would help clarify when a high recovery score indicates a model limitation.

Questions for authors
  1. What thresholds were used at each filtering stage, and how many series were removed or trimmed? Do the main conclusions hold when the excluded series or regions are retained?

  2. What profiles do forecasts that are optimal under MAE receive on controlled data with unpredictable variation?

  3. How were seasonal periods and model orders selected for each dataset?

5 comments
AuthorsOpenReview ↗

for reviewer Mq9A (part 1: data filtering choices)

Thank you for raising this point about data filtering choices. We agree that noise, missing observations, prolonged inactivity, and limited predictability are all genuine characteristics of real-world time series. Limited predictability alone does not imply poor data quality. We excluded random-walk-like series to focus the benchmark on settings where forecasting methods have more opportunity to improve upon simple baselines. Prolonged constant stretches can reflect genuine inactivity or data collection issues such as sensor or logging failures. We have revised Section 4.1 and Appendix E.2 to document the filtering rules and thresholds, explain how trimming was applied in the test regions, and report how many series were affected by each rule.

We also added two analyses, each covering ten TimeFCST datasets, with results visualized in Figure 23, Appendix H (revised version). First, we compared the retained benchmark series with the excluded random-walk-like series from the same datasets. ETS and ARIMA ranked higher on the excluded subset, and their normalized MASE distributions overlapped with those of several foundation models.

Second, we evaluated the same series with and without trimming their long constant regions. When these regions were retained, TimesFM, Huber, and RMLP moved up in the rankings, while TiRex remained among the leading methods. Foundation and deep learning models occupied most of the leading positions in both versions, although several showed varying degrees of degradation on the uncleaned data.

Across both analyses, the performance differences among several leading models narrowed, and some simpler baselines became competitive. We have added these results to the revised manuscript (Appendix H) to clarify the benchmark’s scope and show how model comparisons can change on random-walk-like series and in the presence of prolonged inactivity.

AuthorsOpenReview ↗

for reviewer Mq9A (part 2: error profile discussion)

Thank you for raising these questions about the interpretation of the error profiles and for suggesting controlled experiments with known optimal forecasts. Your feedback helped us strengthen the empirical validation and explain more clearly how readers can use the recovery scores.

Our error profiles and recovery scores are post-hoc diagnostics that quantify how much each correction reduces forecast error against the observed future. The intuition is that, a correction that fits only a few parameters typically needs a consistent pattern across multiple time steps to recover substantial error, which unpredictable noise often lacks. For example, a global level shift can substantially reduce error when the forecast is systematically too high or too low. For the independent noise, a forecast horizon window makes a persistent offset due to chance less likely, so a single global shift tends to yield a smaller post-hoc recovery score. We conducted two controlled experiments to assess how well the profiles distinguish known error types and how much recovery can arise from unpredictable variation alone. Experiment details and results can be found in Appendix H (``Analysis of forecasting failure profiles'').

First, we independently injected each of the six error types into a known MAE-optimal forecast. In every case, the correction corresponding to the injected error achieved the highest mean recovery, while the other corrections generally yielded much lower scores.

Second, we applied the same corrections to known MAE-optimal forecasts without injecting any forecast errors. For periodic signals with independent or correlated noise, raw recovery scores were generally small. We also identify and acknowledge some weakness cases in our designs, e.g., for a constant signal with unpredictable spikes, the correction yielded appreciable recovery at the shorter horizon, even though the constant forecast was MAE-optimal.

Our goal is to provide a post-hoc framework for identifying recurring error patterns across forecasting models and data domains. The experiments show that the error profiles have the power to distinguish different error types, with robustness to unpredictable variation under optimal forecast settings. In the revised manuscript, we will include both experiments and discuss the strengths and limitations of the proposed error profile so that users can apply it to their own forecasts.

AuthorsOpenReview ↗

for reviewer Mq9A (part 3: Seasonal period and model order)

Thank you for asking us to clarify how seasonal periods and model orders were selected. When the dataset metadata indicated a clear seasonal cycle, we used the corresponding period based on the sampling frequency. For example, for the hourly ETTh datasets, we used a daily seasonal period of 24 observations. When the metadata did not provide a clear basis for choosing a seasonal period, we estimated it from the autocorrelation function (ACF).

We would first like to clarify our understanding of “model order” in this question. We take it to refer to the order parameters of statistical forecasting models, such as the autoregressive, differencing, and moving-average orders (p,d,q) in ARIMA. In our benchmark, these order parameters are treated as hyperparameters and selected through the hyperparameter search. We have revised the manuscript to provide the full details, including the search space and selection procedure for each model, in Appendix~F. If you had a different meaning in mind, we would appreciate clarification and would be happy to address it.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thanks to All Reviewers and Summary of Responses

We thank all reviewers for their careful reading and constructive feedback. We are encouraged that the reviewers found the evaluation broad and systematic, covering a wide range of datasets, domains, and methods (Mq9A, oShf). They also noted that the inclusion of strong classical baselines, which recent leaderboards often omit, makes the comparison useful for assessing recent progress (Mq9A, Aes6); that the cleaning and quality-assurance pipeline is rigorous and well documented (oShf, Aes6); that the Friedman–Nemenyi tests support specific conclusions about which model families are significantly separated from tuned baselines (Aes6); and that the failure profiles offer a useful way to compare models with similar aggregate accuracy (Mq9A).

Below, we summarize the main revisions. Detailed responses are provided in the individual replies. The revised manuscript has been uploaded to OpenReview, with all changes highlighted in blue for ease of review.

Summary of revisions

  • Data filtering (Mq9A, Aes6): We document the filtering thresholds and the number of affected series (Section 4.1, Appendix E.2), and add analyses of how model rankings change on the excluded and untrimmed series (Appendix H).
  • Failure profiles (Mq9A, Aes6): We add controlled experiments with known MAE-optimal forecasts to validate the six modes and discuss their limitations (Appendix H).
  • Hyperparameter tuning (oShf, Aes6): We describe the tuning protocol and the construction of TimeFCST-50M (Appendix F), and add a search-budget analysis (Appendix H, Table 8).
  • Pretraining overlap (Aes6): We define the overlap rate and recompute the rankings after excluding potentially overlapping datasets (Appendix H).
  • Related work (oShf): We cite and discuss the works raised by the reviewer and clarify how TimeFCST differs from them in Sections 2.2, 3, and 5.2.

We hope these revisions address the reviewers’ concerns, and we would be happy to provide further clarification during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.