- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
TimeFCST evaluates 100 forecasting methods on 128 public datasets, evenly split between univariate and multivariate series, with on the order of one billion observations after curation. The study reports aggregate rankings, differences across domains, and a decomposition of forecast error into six failure modes.
Critical strengths
- The comparison includes classical statistical and machine-learning methods that recent leaderboards often omit. On univariate series, Huber and ETS rank above approximately 70% of the 100 evaluated methods.
- The Friedman–Nemenyi tests support a specific conclusion. On univariate data, FACT is not significantly separated from the tuned classical baselines, whereas TimesFM is. On multivariate data, deep-learning and foundation models are significantly separated from those baselines.
- Figure 6 qualifies the aggregate ranking. Recent models lead on cloud, healthcare, and mobility, whereas economics, nature, and retail do not follow the same ordering.
- The curation procedure is documented. Minor gaps are forward-filled, extreme artifacts are flagged by three anomaly detectors and then manually inspected, and the release includes code together with results over three random seeds.
Critical weaknesses
- The introduction describes the hyperparameter optimization as systematic. Appendix F limits the search to four configurations per model. This budget is insufficient to support a claim of comparable tuning across model families.
- Series identified as random walks are removed, with Exchange given as an example, because a naive forecast is already difficult to improve upon. This exclusion also removes a regime in which many recent models have limited advantage, so the retained collection is less demanding than the original archive.
- The failure scores use oracle repairs with access to the ground truth, including a median offset, an additional slope, replacement of a local segment, and a small number of Fourier components of the label. A high recovery score for the level mode is the expected outcome of an optimal constant correction. The paper does not establish that the six operators correspond to distinct sources of error.
- Leakage for TimesFM and Chronos is reported as approximately 10% or lower, without a definition of overlap. These models are evaluated zero-shot, while the task-specific models are trained on the target series. Both differences affect the univariate result in which foundation models are reported to lead.
Questions for authors
- Are the series in TimeFCST-50M included in the reported test results, and were the four hyperparameter configurations selected on the test horizon?
- Please define how the leakage rate is computed, and report the rank of TimesFM after all overlapping series are excluded.
- If Exchange and the other excluded random-walk series are restored, do the significance groupings in Figure 5 remain the same?
6 comments
for reviewer Aes6 (part 2: Data cleaning process)
Thank you for highlighting the value of evaluating models on random-walk-like series. We agree that settings where a naive forecast is difficult to beat are also informative, particularly for understanding when recent forecasting models offer limited gains over simpler methods. The exclusion narrows the scope of the main benchmark to settings where performance differences among forecasting methods are more pronounced.
As mentioned in our response to Reviewer Mq9A (W1/Q1), we evaluated the representative models on cleaned datasets and newly prepared datasets with random-walk-like series detected from our cleaning stage (Figure~23(a) in Appendix H). On the prepared random-walk-like datasets, ETS and ARIMA ranked higher, and their normalized MASE distributions overlapped with those of several foundation models.
We observed a similar trend in a separate experiment comparing the datasets with and without trimming long inactive regions (Figure~23(b) in Appendix H). When these regions were retained, simpler methods such as Huber and RMLP became more competitive. Both experiments show that the performance differences among models narrow and become harder to distinguish as the data become noisier. Together, the two analyses illustrate settings in which simpler methods can remain competitive with recent forecasting models on the noisy data. We have included both analyses in the revised manuscript (Appendix H) to show how model performance in these settings with "low-quality" datasets complements the main benchmark results.
for reviewer Aes6 (part 3: Failure profile and error mode)
Thank you for raising this question about how to interpret the six error modes. We designed these modes to capture recurring error patterns across models and data domains. Our goal is to identify patterns that recur across models and data domains and use them to guide further model improvements. As the resulting recovery scores are post-hoc diagnostics and the error modes are not necessarily independent, we acknowledge that a forecast error arising from a single source can yield recovery under more than one mode. The gap between the matching mode’s recovery score and those of the other modes indicates how clearly the profile distinguishes that error pattern.
To test how well the modes distinguish known error patterns, we injected forecasting error from different modes into a known MAE-optimal forecast and applied all six error modes to each modified forecast (Figures 24 and 25 in Appendix H). The matching mode achieved the highest mean recovery score for every injected error type, while the other modes generally yielded much lower scores. We also observed some overlap between modes. For example, the amplitude correction partially recovered an injected phase error, although the phase correction achieved substantially higher recovery. Because the mode targeting the injected error consistently gave the largest recovery, this overlap does not affect the overall separability of the modes, but it points to a limitation that we discuss further in Appendix H.
In the separate optimal-forecast controls discussed in our response to Reviewer Mq9A, we also assessed how much recovery each mode can obtain from unpredictable variation alone. We have included both experiments in the revised manuscript (Appendix H) and explain how users can interpret recurring patterns in the full recovery profile when investigating model behavior.
for reviewer Aes6 (part 1: Tuning budget and comparability)
Thank you for raising this concern about the hyperparameter search budget. We agree that evaluating more configurations gives each model a better chance of finding settings that improve its performance. However, we would like to emphasize the computational cost of substantially expanding the search across approximately 100 models, compared with prior benchmarks such as TFB, which evaluated 21 models. To make tuning feasible at this scale, we use TimeFCST-50M, a dedicated tuning subset balanced across data domains. To help understanding, we have revised and documented the details for TimeFCST-50M and hyperparameter tuning process in Appendix F.
To assess the practical effect of the search budget, we selected representative models and compared their default settings with the configurations selected under our original four-configuration search and a doubled budget of eight configurations. Table~8 in Appendix H shows that, the original four-configuration search improved performance over the defaults. Doubling the budget to eight configurations yielded further gains, but with diminishing returns. These results demonstrate the effectiveness of our original tuning protocol for the models tested: it captured most of the improvement through hyperparameter tuning while using half the search budget. We have included this comparison in the revised manuscript.
Furthermore, as mentioned in the response to Reviewer oShf, we also intend to maintain TimeFCST as a regularly updated benchmark and to expand the hyperparameter search over time. One way to allocate tuning resources fairly across models is to give each model the similar computing budget, so that more expensive models complete fewer trials within the allowed time. We believe the current protocol provides an effective starting point, and we will continue to improve the benchmark in future releases.
Table 8: Normalized MASE relative to Naive Seasonal (lower is better) on 64 univariate datasets under different hyperparameter tuning budgets. Configurations are selected on TimeFCST-50M. Parentheses show the difference from the default configuration (tuned minus default).