- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a metric called MAPS to measure how predictable a multivariate time series forecasting task is when given its covariates, lookback window and forecasting horizons. The basic idea is to check whether the similar historical samples lead to consistent future target vectors, by collecting the k nearest neighbours of each input window with PCA and measuring how well their future target vectors are aligned by cosine similarity and generating a score. This metric is verified on synthetic data and real-world tasks. The paper also includes ablations, sensitivity analysis, and a distribution shift experiment.
Critical strengths
The paper discusses a question about estimating the predictability of forecasting tasks, which could help interpret benchmark results.
Critical weaknesses
I appreciate the efforts to propose a metric responsible for task difficulty without a model. However, it sounds circular evaluation when the only real-data evidence is the performance of a trained model in section 4 (Crossformer, Sonnet). It is unclear what additional information MAPS can provide more than the performance of trained models.
As a metric, it usually helps us to understand how far a given model is from the achievable performance on a task. However, MAPS can’t tell me whether a bad model should be measured under this task or not since it will not reflect the best performance for the model potentially achieved. Besides, MAPS will change with the hyperparameters settings (Figure 6) in the same tasks, which raises concerns it captures intrinsic predictability for a task independent of the analysis settings.
Under distribution shift, MAPS computed on training data shows almost no relationship with downstream performance. This suggests that MAPS mainly describes data that has already been observed, rather than providing guidance before deployment.
MAPS requires PCA and an exhaustive k-NN search, with complexity that is quadratic in the number of windows N and grows with length and channels. This cost could be higher than fitting a lightweight baseline with computing its MAE or MSE.
On real data, MAPS is not distinguishable from ForeCA. ForeCA here uses only the target series and does not depend on L, H, or the covariates. Yet across the 12 configurations with H ≥ 12, the two differ by at most 0.006 against ρ and 0.009 against s, over only 20 locations, and no significance test against ForeCA is reported.
The paper motivates MAPS by the need to interpret results on forecasting benchmarks, citing ETT, Electricity, Traffic, Weather, Exchange-Rate, and ILI, as well as ProbTS, GIFT-Eval, and fev-bench. Yet none of these benchmarks is used.
Several inconsistencies and typos inside the papers. Like In Table S5, ForeCA values are identical to those in Table 1 but marked with ↑/↓ arrows. In section 3, “Performing PCA cost” should be “costs”, line 108 “C is number of covariates”.
Questions for authors
1 Beyond correlating with trained-model performance, can you show some concrete cases where MAPS changes a decision, for example selecting covariates, a lookback window, or a horizon, and we might reach a similar choice through training a model at higher cost?
2 How should we interpret an absolute MAPS value, given that it changes with stride, PCA setting, and k on the same task (Fig. 6A, Table 4, Table S2)?
3 Why does ForeCA use only the target series and near-identical correlations with MAPS in Table 1? Could you report a significance test against ForeCA?
4 Have you applied MAPS to any of the standard benchmarks cited in the introduction (e.g., ETT, Electricity, Traffic, or GIFT-Eval)? Does it explain the known differences in model performance across their tasks?
2 comments
Thank you for your detailed response and clarifications. However, my main concerns remain unresolved, particularly the lack of concrete evidence demonstrating the practical value of MAPS beyond existing metrics or lightweight forecasting baselines, the absence of statistical significance tests against ForeCA, and the limited evaluation on standard forecasting benchmarks. Besides, it is hard for me to find if you respond to my questions or weaknesses. Therefore, I prefer to maintain my current score.
Reply to reviewer rvWo
We thank the reviewer for providing a comprehensive review of our work.
Model Performance as Reference. Forecasting performance is used as an external validation signal on real-world data, where ground truth predictability is unavailable. We follow prior work (Wang et al. 2025) to compare the predictability score against forecasting performance. We further reduce model-specific bias by obtaining consistent results with both Sonnet and Crossformer, while our synthetic experiments provide model-independent validation against known predictability. Hence, we do not consider our evaluation to be circular.
Interpreting MAPS. As shown in Fig. 6 and Tables 4/S2, MAPS remains robust for strides of 5-100, for PCA thresholds that retain 70–90% of the variance, and a broad range of k values (2-1024). MAPS should be interpreted as relative predictability across tasks under fixed settings. It does not quantify a model's achievable performance (as it is model-independent). If a model performs poorly on a task with a high MAPS score, it indicates that the model has not been fully using the predictive information available in the data. Additionally, as MAPS is not computationally intensive, there are ways in which it can be used to inform modelling, i.e. running MAPS on 50% of the training data and comparing to the full data set will inform to what extent more data help in making the task more predictable.
Distribution Shift. In our experiment, we intentionally introduced a substantial distribution shift. Under this setting, MAPS and all baseline measures fail to predict downstream performance using the training data alone. In the real-world WEA and ILI scenarios, the distribution shifts are considerably smaller. We have clarified this point in Section 4.5 of the revised manuscript.
Computational Cost. Figure S6 shows that MAPS' runtime remains stable as the number of variables, lookback length, and horizon increase due to the fixed dimension of the PCA representation. We acknowledge that exhaustive k-NN remains quadratic in N and may be costly for very large datasets.
MAPS vs.ForeCA. ForeCA's strong performance on WEA is expected because the dataset has relatively strong seasonal patterns, and ForeCA operates on the frequency domain. However, this advantage does not generalise on the synthetic systems that do not have as strong periodicities. To this end, ForeCA aligns poorly with known predictability (Fig. 2), whereas MAPS remains accurate. Moreover, ForeCA cannot reflect changes in L, H, or covariates.
Task-Design Guidance. MAPS can be used to inform the design of forecasting solutions. Some examples are listed below:
Benchmark datasets. Weather and ILI provide many regions and support multiple test seasons, enabling more robust analysis when measuring predictability. Standard benchmarks such as ETT, Electricity, Traffic, and Exchange offer fewer independent test sets or comparable locations, making them less suitable for this specific question.
Stylistic corrections. We thank the reviewer for flagging these issues and have corrected them in the revised PDF.