- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that existing time series forecasting benchmarks typically consider a limited set of preprocessing techniques, potentially favoring certain models over others and compromising the fairness of the comparison, particularly in the presence of non-stationary environments. The authors evaluate forecasting models using different preprocessing techniques, showing that these choices can substantially affect performance and that models incorporating preprocessing mechanisms internally may be favored over those that do not. The authors collect the experimental results obtained across different preprocessing configurations in a metadataset.
Critical strengths
The paper addresses a relevant and well-known problem in the literature [1]: benchmarks play a central role in determining which models are considered state of the art, yet they have limitations in terms of evaluation fairness. The paper specifically focuses on differences in the preprocessing techniques used by models, which are usually overlooked in benchmarks. A strength of the paper is that it analyzes the impact of different preprocessing techniques on model performance and investigates whether benchmarks relying on simple scaling may favor models that incorporate preprocessing internally over those that do not.
[1] Brigato et al., There are no Champions in Supervised Long-Term Time Series Forecasting, TMLR 2026
Critical weaknesses
-
The impact of preprocessing and other implementation choices on benchmark fairness has already been identified and addressed in previous works, such as [2,3,4]. However, the authors do not acknowledge that these relevant works have already identified the same problem, nor do they adequately clarify the additional contributions of their analysis relative to the existing literature. As a result, the novelty and significance of the paper's contribution are unclear.
-
The experimental analysis does not seem sufficiently comprehensive to provide substantial new insights beyond those already established in the existing literature. In particular, the evaluation is restricted to a subset of the M4 dataset, uses fixed hyperparameters, and excludes the multivariate setting, limiting the conclusions that can be drawn regarding the performance comparison of different architectures. The lack of hyperparameter tuning also makes some claims about the performance gaps resulting from different preprocessing techniques insufficiently supported (for example, the claim made in the abstract). Additionally, the results appear to be reported from single runs rather than averaged across multiple random seeds, with means and standard deviations. Therefore, it is not possible to assess whether the observed performance differences and rankings reported in the tables are robust to random model initialization. Finally, the interpretation of some results is sometimes stronger than what the experimental evidence supports. For instance, referring to Table 3, the authors state that IP models achieve lower MASE than all non-IP models when no preprocessing is applied. However, in the first row of the table, XGBoost (MASE = 2.75), a non-IP model, outperforms PatchTST (MASE = 2.77), an IP model.
[2] Qiu et al., TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methods, 2024 VLDB Endowment
[3] Moretti et al., Position: Current Benchmarking Hinders Real Progress in Deep Learning for Time Series Forecasting, ICML 2026
[4] Shao et al., Exploring Progress in Multivariate Time Series Forecasting: Comprehensive Benchmarking and Heterogeneity Analysis, IEEE Transactions on Knowledge and Data Engineering 2025
Questions for authors
Questions:
- How is the Oracle preprocessing pipeline selected for each time series? Is it selected based on model performance at test time?
- How did you choose the subset of M4 series to use, apart from excluding the yearly series? This selection criterion should be specified.
- Do the datasets D_i in Section 3 consist of a single time series from M4 each in your experiments?
Comments:
- The paper sometimes states that the framework improves “interpretability“, but this term does not seem appropriate unless it is better contextualized.
- The distinction between preprocessing techniques referred to as “scaling” and those intended to address “nonstationarity” is not clearly specified in Table 1. In particular, it would be helpful to clarify which preprocessing transformations are included in each category.
- The Preliminaries section does not specify the dimensions of the variables. For example, the forecast is introduced without specifying the forecasting horizon.
- Table 3 is not well rendered, partially hiding model name
2 comments
Thank you for the clarifications. I acknowledge that you cite Shao et al. in the Introduction. However, the main body does not clearly position the work with respect to them and to the other works mentioned in my review, e.g. by recognizing that they identify similar issues with benchmarking practices. The main body also lacks a related work section. Since the contribution is mainly empirical, the use of fixed hyperparameters, a single seed, and only a univariate subset of M4 strongly limits the conclusions that can be drawn from the analysis. Therefore, I will keep my score.
Response to Reviewer 9UfS
Thank you for the detailed review and for recognizing the relevance of preprocessing to benchmark fairness.
1. Prior work and contribution. The original submission already positions our study relative to TFB and Shao et al. (BasicTS+). The Introduction identifies the gap: nonstationarity transformations are rarely treated as systematic benchmark dimensions. Table 1 compares benchmark coverage; Appendix B discusses their scope and contrasts TimeRecipe's module-level analysis with external preprocessing across model families. This positioning was present at submission.
Our contribution is a controlled comparison of 16 reversible pipelines across 11 models and approximately 29,000 series: about 5.1 million evaluations under fixed forecasting configurations. It measures within-model sensitivity, ranking changes, and pipeline preferences associated with series properties and internal preprocessing. PATM releases results, costs, and descriptors for further analysis and selection research.
We will make this positioning more prominent, incorporate Moretti and Brigato, and refine the TimeRecipe comparison to acknowledge its normalization and decomposition experiments. We defend this specific analysis and resource without claiming priority for recognizing the importance of preprocessing or implementation choices.
2. Coverage, hyperparameters, and repeated runs. The univariate M4 subset spans domains and frequencies but limits generalization to multivariate tasks and other corpora. The approximately 5.1 million evaluations prioritized pipeline coverage; repeating the grid across seeds and hyperparameters would multiply the computational workload. Fixed configurations enable within-model sensitivity comparisons; statistical methods retain automated component/order selection (Appendix F).
This design does not establish seed robustness or remove preprocessing-hyperparameter interactions. Standard deviations across series do not measure initialization variability. We will qualify architecture comparisons, including the abstract, as results for the evaluated configurations, without claiming a fully tuned leaderboard or causal separation of architecture and preprocessing.
3. Incorrect all-IP comparison in Table 3. Correct: without external preprocessing, XGBoost (2.75 MASE) outperforms PatchTST (2.77). We will state: "Without external preprocessing, five of the six IP models have lower mean MASE than the best-performing non-IP model." Table values are unchanged; we claim no significance for the 0.02 difference.
4. How is Oracle selected? Oracle selects the minimum recorded test MASE across 16 pipelines for each model and series. Transformation parameters are fitted on training data, but selection uses test outcomes. Gains are therefore retrospective within the tested grid, not a deployable selection procedure's performance. We will clarify this in the abstract and results. The 27%-87% figures average per-series percentage reductions, not percentage reductions in aggregate mean loss.
5. Why this M4 subset? Table 2 lists quarterly (24,000 series), weekly (359), daily (4,227), and hourly (414). Yearly data were excluded because seasonal differencing was inapplicable. Monthly evaluation was particularly computationally demanding and outside the completed results. This resource-limited selection does not cover all M4. We will replace "the full breadth of the M4 dataset" with "the evaluated M4 subset."
6. Does each D_i denote one series? Yes: each D_i is one M4 forecasting task with its training history, held-out horizon, and descriptors. A global model can train across multiple tasks; D_i does not imply a separate global fit per series.
7. Use of "interpretability." We mean transparency about performance dependence on pipeline choices, rather than explanations of individual forecasts. We will use "transparency of the evaluation protocol."
8. Scaling versus nonstationarity in Table 1. Q is z-score scaling, V is Box-Cox variance stabilization, U is first-order differencing, and S is seasonal differencing. Scaling does not generally remove trends or seasonal unit roots. The nonstationarity-pipeline column concerns benchmark-level comparisons of external transformations, not models' internal normalization or decomposition.
9. Dimensions and forecast horizon. Training series X_i is in R^{T_i}, input window s in R^{L_i}, forecast y-hat_i in R^{H_i}, and descriptors f_i in R^{40}. Table 2 gives H_i as 8, 13, 14, and 48 for quarterly, weekly, daily, and hourly series. We will add dimensions and the horizon definition to the preliminaries.
10. Table 3 rendering. We will fix clipped model names with compact, defined abbreviations, preserving all numerical results.