acceptodds
Under review as a conference paper at ICLR 2027

Are We Really Benchmarking Forecasting Models? The Impact of Preprocessing on Time Series Performance

Abstract

While established literature underscores the pivotal role of preprocessing in forecasting accuracy, this stage remains largely overlooked in current research. Modern benchmarks typically resort to simple scaling, failing to account for critical transformations required to address nonstationarity, such as differencing. This omission creates a significant structural preprocessing bias that favors models with built-in data treatments while obscuring the true potential of simpler architectures. We study this effect through a preprocessing-aware benchmark that evaluates 11 forecasting models across 16 reversible preprocessing pipelines on 29,000 M4 time series. Our results identify preprocessing as a key driver of forecasting performance. Optimizing preprocessing per series yields gains of approximately 27% to 87% across all evaluated models, with architectures lacking internalized preprocessing experiencing the most substantial improvements. This allows simpler architectures to become highly competitive with complex, state-of-the-art models in modern forecasting benchmarks. All resources and experimental results from this benchmark are stored in a comprehensive metadataset to support future metalearning tasks.

Primary area:
datasets and benchmarks
Keywords:
Machine Learning, Time Series Forecasting, Benchmark, Preprocessing, Nonstationarity

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_9UfSOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that existing time series forecasting benchmarks typically consider a limited set of preprocessing techniques, potentially favoring certain models over others and compromising the fairness of the comparison, particularly in the presence of non-stationary environments. The authors evaluate forecasting models using different preprocessing techniques, showing that these choices can substantially affect performance and that models incorporating preprocessing mechanisms internally may be favored over those that do not. The authors collect the experimental results obtained across different preprocessing configurations in a metadataset.

Critical strengths

The paper addresses a relevant and well-known problem in the literature [1]: benchmarks play a central role in determining which models are considered state of the art, yet they have limitations in terms of evaluation fairness. The paper specifically focuses on differences in the preprocessing techniques used by models, which are usually overlooked in benchmarks. A strength of the paper is that it analyzes the impact of different preprocessing techniques on model performance and investigates whether benchmarks relying on simple scaling may favor models that incorporate preprocessing internally over those that do not.

[1] Brigato et al., There are no Champions in Supervised Long-Term Time Series Forecasting, TMLR 2026

Critical weaknesses
  1. The impact of preprocessing and other implementation choices on benchmark fairness has already been identified and addressed in previous works, such as [2,3,4]. However, the authors do not acknowledge that these relevant works have already identified the same problem, nor do they adequately clarify the additional contributions of their analysis relative to the existing literature. As a result, the novelty and significance of the paper's contribution are unclear.

  2. The experimental analysis does not seem sufficiently comprehensive to provide substantial new insights beyond those already established in the existing literature. In particular, the evaluation is restricted to a subset of the M4 dataset, uses fixed hyperparameters, and excludes the multivariate setting, limiting the conclusions that can be drawn regarding the performance comparison of different architectures. The lack of hyperparameter tuning also makes some claims about the performance gaps resulting from different preprocessing techniques insufficiently supported (for example, the claim made in the abstract). Additionally, the results appear to be reported from single runs rather than averaged across multiple random seeds, with means and standard deviations. Therefore, it is not possible to assess whether the observed performance differences and rankings reported in the tables are robust to random model initialization. Finally, the interpretation of some results is sometimes stronger than what the experimental evidence supports. For instance, referring to Table 3, the authors state that IP models achieve lower MASE than all non-IP models when no preprocessing is applied. However, in the first row of the table, XGBoost (MASE = 2.75), a non-IP model, outperforms PatchTST (MASE = 2.77), an IP model.

[2] Qiu et al., TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methods, 2024 VLDB Endowment

[3] Moretti et al., Position: Current Benchmarking Hinders Real Progress in Deep Learning for Time Series Forecasting, ICML 2026

[4] Shao et al., Exploring Progress in Multivariate Time Series Forecasting: Comprehensive Benchmarking and Heterogeneity Analysis, IEEE Transactions on Knowledge and Data Engineering 2025

Questions for authors

Questions:

  1. How is the Oracle preprocessing pipeline selected for each time series? Is it selected based on model performance at test time?
  2. How did you choose the subset of M4 series to use, apart from excluding the yearly series? This selection criterion should be specified.
  3. Do the datasets D_i in Section 3 consist of a single time series from M4 each in your experiments?

Comments:

  1. The paper sometimes states that the framework improves “interpretability“, but this term does not seem appropriate unless it is better contextualized.
  2. The distinction between preprocessing techniques referred to as “scaling” and those intended to address “nonstationarity” is not clearly specified in Table 1. In particular, it would be helpful to clarify which preprocessing transformations are included in each category.
  3. The Preliminaries section does not specify the dimensions of the variables. For example, the forecast is introduced without specifying the forecasting horizon.
  4. Table 3 is not well rendered, partially hiding model name
2 comments

Response to Reviewer 9UfS

Thank you for the detailed review and for recognizing the relevance of preprocessing to benchmark fairness.

1. Prior work and contribution. The original submission already positions our study relative to TFB and Shao et al. (BasicTS+). The Introduction identifies the gap: nonstationarity transformations are rarely treated as systematic benchmark dimensions. Table 1 compares benchmark coverage; Appendix B discusses their scope and contrasts TimeRecipe's module-level analysis with external preprocessing across model families. This positioning was present at submission.

Our contribution is a controlled comparison of 16 reversible pipelines across 11 models and approximately 29,000 series: about 5.1 million evaluations under fixed forecasting configurations. It measures within-model sensitivity, ranking changes, and pipeline preferences associated with series properties and internal preprocessing. PATM releases results, costs, and descriptors for further analysis and selection research.

We will make this positioning more prominent, incorporate Moretti and Brigato, and refine the TimeRecipe comparison to acknowledge its normalization and decomposition experiments. We defend this specific analysis and resource without claiming priority for recognizing the importance of preprocessing or implementation choices.

2. Coverage, hyperparameters, and repeated runs. The univariate M4 subset spans domains and frequencies but limits generalization to multivariate tasks and other corpora. The approximately 5.1 million evaluations prioritized pipeline coverage; repeating the grid across seeds and hyperparameters would multiply the computational workload. Fixed configurations enable within-model sensitivity comparisons; statistical methods retain automated component/order selection (Appendix F).

This design does not establish seed robustness or remove preprocessing-hyperparameter interactions. Standard deviations across series do not measure initialization variability. We will qualify architecture comparisons, including the abstract, as results for the evaluated configurations, without claiming a fully tuned leaderboard or causal separation of architecture and preprocessing.

3. Incorrect all-IP comparison in Table 3. Correct: without external preprocessing, XGBoost (2.75 MASE) outperforms PatchTST (2.77). We will state: "Without external preprocessing, five of the six IP models have lower mean MASE than the best-performing non-IP model." Table values are unchanged; we claim no significance for the 0.02 difference.

4. How is Oracle selected? Oracle selects the minimum recorded test MASE across 16 pipelines for each model and series. Transformation parameters are fitted on training data, but selection uses test outcomes. Gains are therefore retrospective within the tested grid, not a deployable selection procedure's performance. We will clarify this in the abstract and results. The 27%-87% figures average per-series percentage reductions, not percentage reductions in aggregate mean loss.

5. Why this M4 subset? Table 2 lists quarterly (24,000 series), weekly (359), daily (4,227), and hourly (414). Yearly data were excluded because seasonal differencing was inapplicable. Monthly evaluation was particularly computationally demanding and outside the completed results. This resource-limited selection does not cover all M4. We will replace "the full breadth of the M4 dataset" with "the evaluated M4 subset."

6. Does each D_i denote one series? Yes: each D_i is one M4 forecasting task with its training history, held-out horizon, and descriptors. A global model can train across multiple tasks; D_i does not imply a separate global fit per series.

7. Use of "interpretability." We mean transparency about performance dependence on pipeline choices, rather than explanations of individual forecasts. We will use "transparency of the evaluation protocol."

8. Scaling versus nonstationarity in Table 1. Q is z-score scaling, V is Box-Cox variance stabilization, U is first-order differencing, and S is seasonal differencing. Scaling does not generally remove trends or seasonal unit roots. The nonstationarity-pipeline column concerns benchmark-level comparisons of external transformations, not models' internal normalization or decomposition.

9. Dimensions and forecast horizon. Training series X_i is in R^{T_i}, input window s in R^{L_i}, forecast y-hat_i in R^{H_i}, and descriptors f_i in R^{40}. Table 2 gives H_i as 8, 13, 14, and 48 for quarterly, weekly, daily, and hourly series. We will add dimensions and the horizon definition to the preliminaries.

10. Table 3 rendering. We will fix clipped model names with compact, defined abbreviations, preserving all numerical results.

Thank you for the clarifications. I acknowledge that you cite Shao et al. in the Introduction. However, the main body does not clearly position the work with respect to them and to the other works mentioned in my review, e.g. by recognizing that they identify similar issues with benchmarking practices. The main body also lacks a related work section. Since the contribution is mainly empirical, the use of fixed hyperparameters, a single seed, and only a univariate subset of M4 strongly limits the conclusions that can be drawn from the analysis. Therefore, I will keep my score.

Reviewer_C2KEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that current time series forecasting benchmarks fix a minimal global preprocessing (e.g. scaling only), and favors models with internalized preprocessing (IP-Models, e.g. ARIMA, ETS, Theta, TimesNet, PatchTST, Chronos2-Synth) and underestimates non-IP-Models (LR, XGBoost, MLP, N-BEATS, Informer). This work then builds a preprocessing-aware benchmark: 11 models x 16 reversible pipelines (V{none,boxcox} x S{none,sdiff} x U{none,diff} x Q{none,z-score}) on ~29k M4 series (Quarterly/Weekly/Daily/Hourly), ~5.1M runs in total. This paper reports MASE as main metric plus MSE/RMSE/MAE/sMAPE, analyze fixed-global vs per-series-oracle, intra- vs inter-model ranking, property-conditioned preferences, local vs global training, and training time / GPU cost.

The paper argues, per-series oracle preprocessing gives 27%-87% gains, larger for non-IP-Models (e.g. Informer 86.7%, MLP 59%), allowing simple models to close the gap with SOTA; rankings flip once preprocessing is tuned per series.

Critical strengths
  1. This study is at scale and systematicity, including 11 x 16 x 29k = 5.1M experiments, which is sufficient for benchmarks. The pipeline design is clean, reversible, leakage-controlled (fit on train only), order fixed V->S->U->Q with justification.

  2. The model coverage includes statistical, ML, deep, transformer, foundation. It also uses Chronos2-Synth (synthetic-only pretraining) to avoid M4 leakage.

Critical weaknesses
  1. TimeRecipe already benchmarks preprocessing as a part of their benchmark (Instance Norm x Series Decomposition, 4 combos). This work expands the grid to 16 classical reversible transforms (V/S/U/Q). It reads as a preprocessing-only extension with larger scale but less novelty.

  2. Table 3 contains 1.38e37, 5.19e35 etc. from applying seasonal differencing to non-seasonal series plus inverse Box-Cox amplification. Including such results into presentation and analysis becomes strange and less meaningful. Fig.3/6 plot them with truncated axes also looks very strange, which of all may decrease the credibility of the benchmark study.

  3. The insights of the work are not actionable at a glance. For instance, a reader with a new model still must brute-force all 16 pipelines on test to figure out what pre-processing is needed for his/her own model.

  4. Pre-processing helps is already well-known and it is not surprising that differencing or stabilization helps non-IP-Models on non-stationary data, particularly after RevIN comes out. The surprising signals in this work could be IP peaks at X/V/Q/VQ but collapses under VUS/USQ, U helps LR but hurts N-BEATS, Q hurts XGBoost, it would be more insightful if the study could provide in-depth analysis on such cases to understand when pre-processing fails.

Questions for authors

Please refer to the weakness

1 comment

Response to Reviewer C2KE

Thank you for recognizing the scale, systematic design, and model coverage of the study. We address your four concerns below.

1. Relationship to TimeRecipe and novelty. The original Appendix B already positions our study relative to TimeRecipe: it contrasts TimeRecipe's module-level benchmarking with our comparison of external, reversible pipelines across heterogeneous model families. The Introduction and Table 1 similarly position the study relative to existing benchmarks, including TFB and BasicTS+. The distinction was therefore present in the submission. We will refine the TimeRecipe discussion to explicitly acknowledge that its normalization and decomposition experiments are relevant preprocessing comparisons.

The contribution goes beyond expanding the number of pipeline combinations. We apply a common grid of 16 reversible external pipelines across statistical, machine-learning, neural, transformer, and foundation models, and examine how the model-pipeline interaction changes absolute errors, relative rankings, and preferences conditioned on series properties. This includes comparing the benefits and failures of external transformations in models with and without built-in preprocessing under the evaluated configurations. Approximately 5.1 million evaluations on 29,000 series support these comparisons, and PATM releases their results, costs, and series descriptors. The contribution is this comparative evidence and reusable resource; the general observation that preprocessing helps is the motivation already established by prior work.

2. Extreme errors and truncated axes. We retained the extreme values because removing the configurations that failed would conceal an important limitation of applying the same pipeline indiscriminately. The Results section already discusses how unnecessary differencing can amplify noise and how inverse Box-Cox reconstruction can magnify forecast errors on the original scale. This is an explanation for the failure mechanism, rather than a run-by-run causal diagnosis of every extreme value. Such values should be read as catastrophic failures, not fine-grained differences in ordinary forecasting accuracy. The axes were truncated for readability, while the full magnitudes remain reported in Table 3. Omitting them would make the benchmark less transparent.

3. Actionability and Oracle selection. Oracle chooses the lowest test MASE among the 16 pipelines separately for each model and series. It measures the best observed result within the tested grid with hindsight; it is not an operational recommendation to select pipelines on a new test set. Practical selection would use a validation window within the available history. The property-conditioned results provide candidate choices, and PATM supports developing selectors, but neither establishes a validated selection rule for unseen series. We will explicitly label the oracle gains as retrospective in the abstract and results. The reported 27%-87% gains are averages of per-series percentage reductions. We will also align the main-text gain descriptions with that definition and Appendix I.2.

4. What the failures teach us. The existing results already identify model-specific effects beyond the general statement that preprocessing helps. Table 3 shows LR improving from 2.87 to 2.27 MASE with first differencing, while N-BEATS worsens from 4.65 to 5.91; scaling changes XGBoost from 2.75 to 3.71. The poor results under stacked transformations also show that internal preprocessing does not confer protection against every external pipeline. These opposing responses explain why a common preprocessing choice can change the apparent comparison between models, which is a central question of the benchmark. Differencing can simplify changing levels but can also remove useful structure or compound existing transformations. These are possible mechanisms, the aggregate results do not establish the cause of each model-specific failure, including XGBoost's scaling result. We will qualify the IP/non-IP interpretation as an observed tendency under the tested configurations, rather than universal robustness or proof that internal preprocessing alone causes the differences.

Reviewer_sVw1OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This is a dataset and benchmark contribution paper, but the major contribution is conducting an in-depth analysis of forecasting approaches coupled with the preprocessing module selection. I cross-checked the dataset and code links, and both work.

Critical strengths

Preprocessing plays an important role, and this paper demonstrates that aspect. Overall, this is a very heavy benchmark-style paper, and I expected to spend significant time collecting the dataset. But overall, it's just a benchmark.

Critical weaknesses
  • The paper title may need a bit of attention. Preprocessing may have an impact on time series tasks (like forecasting, etc.).

  • Does Table 1 contain all the latest TSFM benchmarks? If there are more, please list them also. Also, we use a tick mark symbol for Yes; here, we use X to indicate Yes.

  • I recall an AutoML approach with several joint optimizers, whose objective was to jointly search the search space for preprocessing and a basic ML/SL model. I think the current paper should bring this concept; a few examples include:

    • AutoAI-TS: AutoAI for Time Series Forecasting
    • Catch22: Canonical Time-series CHaracteristics
    • FLOps: On Learning Important Time Series Features for Real-Valued Prediction
  • Section 2: Please revise to include more recent work or benchmarks on TSFM and related prior art, as discussed earlier.

  • Section 3 is easy to understand and can be compressed. I had a few suggestions, mainly coming from TabArea. Please discuss the selected dataset clearly. Please discuss the selected algorithm, its parameters, and the source of its implementation. Include a table listing the clear transformation features. Also, include the pre-processor parameters. These are crucial details and are currently missing from the main paper.

  • Incorporating the feature preprocessor has implications, such as reverse feature engineering, time complexity, space complexity, or features being in a non-temporal domain. These categories help clarify the work.

  • Please look at the TabArena benchmark, and can anything from this benchmark be transferred there to showcase the benefits.

Questions for authors

I have no questions as the paper needs work as highlighted in the weakness section.

2 comments

Response to Reviewer sVw1

Thank you for your review, for checking the artifacts, and for recognizing the importance of preprocessing. We address each point below.

1. Contribution and title. Our contribution is a benchmarking study and the PATM resource: 11 forecasting models, 16 reversible pipelines, and approximately 29,000 series, yielding about 5.1 million model-pipeline-series evaluations. The question is how preprocessing changes model comparisons and rankings across model families. The title's reference to forecasting models identifies the task studied. We do not generalize the experiments to other time-series tasks. The contribution rests on the systematic comparison and reusable results rather than on discovering that preprocessing matters.

2. Table 1 and recent benchmarks. Table 1 contains representative benchmarks, including GIFT-Eval and TSFM-Bench. It is not an exhaustive catalogue. Additional relevant examples include fev-bench and the observability benchmark BOOM. The literature review is in Appendix B, while Section 2 provides preliminaries. The caption explicitly defines X as Yes; a checkmark would also be a reasonable convention. This affirmative symbol is separate from pipeline X, which denotes no external preprocessing.

3. AutoML and feature-based approaches. Our focus is the gap in benchmarking research: measuring how model comparisons depend on the preprocessing protocol. AutoAI-TS addresses automated forecasting-pipeline selection, whereas catch22 provides time-series descriptors and FLOps selects relevant features. These are relevant connections, but an AutoML-system comparison would evaluate a different contribution. PATM provides outcomes and descriptors that can support selection methods without proposing a new optimizer here.

4. Dataset, algorithms, and documentation. The corpus covers M4 quarterly, weekly, daily, and hourly groups. Table 2 reports their sizes, seasonal lags, and horizons. The models span statistical, machine-learning, neural, transformer, and foundation approaches. Chronos2-Synth was chosen to avoid direct M4 overlap in real-data pretraining. Implementation libraries are identified in Appendix E, model settings in Appendix F, and local/global training in Appendix G, alongside the released code. Section 3 defines the reusable benchmark structure. Section 4 supplies the experimental choices. We acknowledge that collecting these details in the main text would make them easier to find.

5. Transformation features and parameters. The four components are summarized here. Each can also be omitted, giving 16 combinations. Parameters are fitted using training data.

ComponentTransformation and parameters
VBox-Cox; ten equally spaced lambda candidates in [0, 1], selected by a transformed-variance criterion on training data.
UFirst-order differencing, lag 1.
SSeasonal differencing; lags 4, 52, 7, and 24 for quarterly, weekly, daily, and hourly series.
QZ-score scaling using the training mean and standard deviation.

Appendix C separately lists the 40 series descriptors, and Appendix D describes the transformations.

6. Reconstruction, complexity, and temporal representation. These operations preserve the temporal sequence, with differencing shortening the available history. Forecasts are reconstructed by reversing the transformations using fitted parameters and retained training-boundary values. The descriptors are metadata for analysis, not a non-temporal replacement for forecasting inputs. For a fixed lambda grid and seasonal lag, preprocessing takes linear time in series length and a direct implementation uses linear storage. These costs are distinct from model fitting and pipeline search. Reported timings cover preprocessing plus training and exclude inference and search.

7. TabArena. TabArena is a useful reference for explicit configurations, validation protocols, and reproducible evaluation. These principles transfer to forecasting benchmarks. A direct experiment on its tabular tasks would require a different evaluation design. The present evidence concerns forecasting and the interaction between models and reversible temporal transformations.

Reviewer_sVw1OpenReview ↗

Rebuttal

I read the review, and as discussed in the main review, the paper needs significant alignment. I do not fully agree with bullet 3. TabArena is a good baseline.

Also, the author submitted a response toward the end of the review period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.