acceptodds
Under review as a conference paper at ICLR 2027

Model-agnostic predictability quantification of multivariable time series

Abstract

Multivariable time series forecasting plays an important role in various domains, where predictions depend on complex interactions among covariates. Forecasting models are commonly evaluated across numerous benchmark datasets. However, forecasting accuracy reflects both model capability and task predictability, making it difficult to characterise the difficulty of a task independently of the model used. Existing predictability measures often fail to capture nonlinear dependencies or are limited to univariate series. In this work, we propose Multivariable Predictability Score (MAPS), a model-agnostic metric for measuring the intrinsic predictability of multivariable time series forecasting tasks. MAPS uses covariate information to assess predictability across different lookback windows and forecasting horizons, and provides a quantitative assessment of the predictive information available for forecasting. We evaluate MAPS across synthetic and real-world forecasting datasets against 6 baseline predictability measures. The former confirms that our method closely aligns with the expected reference predictability (r=0.963). Empirical results further corroborate that MAPS yields more reliable projections in the majority of experimental configurations.

Primary area:
learning on time series and dynamical systems
Keywords:
Time Series Forecasting, Predictability Quantification, Multivariable Forecasting

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rvWoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a metric called MAPS to measure how predictable a multivariate time series forecasting task is when given its covariates, lookback window and forecasting horizons. The basic idea is to check whether the similar historical samples lead to consistent future target vectors, by collecting the k nearest neighbours of each input window with PCA and measuring how well their future target vectors are aligned by cosine similarity and generating a score. This metric is verified on synthetic data and real-world tasks. The paper also includes ablations, sensitivity analysis, and a distribution shift experiment.

Critical strengths

The paper discusses a question about estimating the predictability of forecasting tasks, which could help interpret benchmark results.

Critical weaknesses

I appreciate the efforts to propose a metric responsible for task difficulty without a model. However, it sounds circular evaluation when the only real-data evidence is the performance of a trained model in section 4 (Crossformer, Sonnet). It is unclear what additional information MAPS can provide more than the performance of trained models.

As a metric, it usually helps us to understand how far a given model is from the achievable performance on a task. However, MAPS can’t tell me whether a bad model should be measured under this task or not since it will not reflect the best performance for the model potentially achieved. Besides, MAPS will change with the hyperparameters settings (Figure 6) in the same tasks, which raises concerns it captures intrinsic predictability for a task independent of the analysis settings.

Under distribution shift, MAPS computed on training data shows almost no relationship with downstream performance. This suggests that MAPS mainly describes data that has already been observed, rather than providing guidance before deployment.

MAPS requires PCA and an exhaustive k-NN search, with complexity that is quadratic in the number of windows N and grows with length and channels. This cost could be higher than fitting a lightweight baseline with computing its MAE or MSE.

On real data, MAPS is not distinguishable from ForeCA. ForeCA here uses only the target series and does not depend on L, H, or the covariates. Yet across the 12 configurations with H ≥ 12, the two differ by at most 0.006 against ρ and 0.009 against s, over only 20 locations, and no significance test against ForeCA is reported.

The paper motivates MAPS by the need to interpret results on forecasting benchmarks, citing ETT, Electricity, Traffic, Weather, Exchange-Rate, and ILI, as well as ProbTS, GIFT-Eval, and fev-bench. Yet none of these benchmarks is used.

Several inconsistencies and typos inside the papers. Like In Table S5, ForeCA values are identical to those in Table 1 but marked with ↑/↓ arrows. In section 3, “Performing PCA cost” should be “costs”, line 108 “C is number of covariates”.

Questions for authors

1 Beyond correlating with trained-model performance, can you show some concrete cases where MAPS changes a decision, for example selecting covariates, a lookback window, or a horizon, and we might reach a similar choice through training a model at higher cost?

2 How should we interpret an absolute MAPS value, given that it changes with stride, PCA setting, and k on the same task (Fig. 6A, Table 4, Table S2)?

3 Why does ForeCA use only the target series and near-identical correlations with MAPS in Table 1? Could you report a significance test against ForeCA?

4 Have you applied MAPS to any of the standard benchmarks cited in the introduction (e.g., ETT, Electricity, Traffic, or GIFT-Eval)? Does it explain the known differences in model performance across their tasks?

2 comments
AuthorsOpenReview ↗

Reply to reviewer rvWo

We thank the reviewer for providing a comprehensive review of our work.

Model Performance as Reference. Forecasting performance is used as an external validation signal on real-world data, where ground truth predictability is unavailable. We follow prior work (Wang et al. 2025) to compare the predictability score against forecasting performance. We further reduce model-specific bias by obtaining consistent results with both Sonnet and Crossformer, while our synthetic experiments provide model-independent validation against known predictability. Hence, we do not consider our evaluation to be circular.

Interpreting MAPS. As shown in Fig. 6 and Tables 4/S2, MAPS remains robust for strides of 5-100, for PCA thresholds that retain 70–90% of the variance, and a broad range of kk values (2-1024). MAPS should be interpreted as relative predictability across tasks under fixed settings. It does not quantify a model's achievable performance (as it is model-independent). If a model performs poorly on a task with a high MAPS score, it indicates that the model has not been fully using the predictive information available in the data. Additionally, as MAPS is not computationally intensive, there are ways in which it can be used to inform modelling, i.e. running MAPS on 50% of the training data and comparing to the full data set will inform to what extent more data help in making the task more predictable.

Distribution Shift. In our experiment, we intentionally introduced a substantial distribution shift. Under this setting, MAPS and all baseline measures fail to predict downstream performance using the training data alone. In the real-world WEA and ILI scenarios, the distribution shifts are considerably smaller. We have clarified this point in Section 4.5 of the revised manuscript.

Computational Cost. Figure S6 shows that MAPS' runtime remains stable as the number of variables, lookback length, and horizon increase due to the fixed dimension of the PCA representation. We acknowledge that exhaustive kk-NN remains quadratic in NN and may be costly for very large datasets.

MAPS vs.ForeCA. ForeCA's strong performance on WEA is expected because the dataset has relatively strong seasonal patterns, and ForeCA operates on the frequency domain. However, this advantage does not generalise on the synthetic systems that do not have as strong periodicities. To this end, ForeCA aligns poorly with known predictability (Fig. 2), whereas MAPS remains accurate. Moreover, ForeCA cannot reflect changes in LL, HH, or covariates.

Task-Design Guidance. MAPS can be used to inform the design of forecasting solutions. Some examples are listed below:

  1. Covariate set selection. MAPS can compare candidate covariate sets and identify whether additional variables provide useful predictive information before training a forecasting model.
  2. Task configuration. MAPS can compare lookback windows and identify horizons at which predictability becomes too low, helping select an informative history length and a feasible forecasting horizon.
  3. In the synthetic experiments, MAPS follows the same changes across forecasting horizons as the performance of a trained model, suggesting that it can guide task selection without repeatedly training forecasting models.

Benchmark datasets. Weather and ILI provide many regions and support multiple test seasons, enabling more robust analysis when measuring predictability. Standard benchmarks such as ETT, Electricity, Traffic, and Exchange offer fewer independent test sets or comparable locations, making them less suitable for this specific question.

Stylistic corrections. We thank the reviewer for flagging these issues and have corrected them in the revised PDF.

Thank you for your detailed response and clarifications. However, my main concerns remain unresolved, particularly the lack of concrete evidence demonstrating the practical value of MAPS beyond existing metrics or lightweight forecasting baselines, the absence of statistical significance tests against ForeCA, and the limited evaluation on standard forecasting benchmarks. Besides, it is hard for me to find if you respond to my questions or weaknesses. Therefore, I prefer to maintain my current score.

Reviewer_jNnsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MAPS, an evaluation metric or a predictability metric to understand the predictive power in a multivariate time series dataset. This is critical to assess the ceiling of improvement that can be obtained with respect to forecasting accuracy. The approach is quite simple: For a given test sample from the dataset, MAPS identifies the nearest neighbors (from the training samples) in the PCA-reduced input space and checks for the cosine similarity of the forecasts for all neighbors. High similarity indicates high predictability.

Critical strengths
  1. MAPS is very simple, intuitive, and has very few bells and whistles.

  2. The synthetic toy datasets help explain the improvement provided by MAPS over traditional baselines.

Critical weaknesses

Why Sonnet and Crossformer alone, and why not zero-shot multivariate forecasters like Chronos2, Toto2, TimesFM3, etc? The results are hinged on correlation with a single model’s (at max 2) predictive power, which does not marginalize the modeling choice. If the results hold over multiple multivariable forecasting architectures and zero-shot models, the correlation could have higher statistical significance.

Questions for authors

Check critical weaknesses. Additionally,

  1. Are the nearest neighbors chosen causally? That is important for assessing the predictability of time series datasets.

  2. The sweep over L is performed for real-world datasets, but not for synthetic datasets. I think showing results on synthetic datasets where there is a complex lagged relationship with varying lags (a dataset with a lag of 2, another with 8, another with 64, and so on) is necessary.

  3. Can the authors provide some intuitive toy examples, where a group of datasets obtained from increasing levels of complex relationships have the same predictability?

  4. Wouldn’t a ridge regression model with suitable feature engineering be a more competitive baseline than a simple ridge regression model?

  5. For the cases with a distribution shift between the train and test cases, the authors have an example from a real-world dataset. However, the exact ablation can be shown more intuitively using synthetic datasets where the functional relationship between the features (inputs) and the target (output) varies over time.

2 comments
AuthorsOpenReview ↗

Reply to reviewer jNns

We thank the reviewer for acknowledging clarity of our method and the value of our synthetic examples.

W1: We focus on Sonnet and Crossformer because they are specifically trained and tuned for multivariable time series forecasting for each target. These models provide SOTA performance on the explored data sets. Their performance therefore provides a closer approximation to the predictability attainable from the data compared to zero-shot or fine-tuned foundation models. Notably, SOTA foundation models perform worse than persistence models on our ILI forecasting task from Google search trends. Adding more models does not increase statistical significance or establish consistency, because the correlation with MAPS will depend on each model's forecasting capability.

Q1: Nearest neighbours are not chosen causally. They are selected from all input windows in the training data. Choosing the neighbours causally would introduce a severe computational cost, especially for data sets with larger number of covariates (the ILI data set being an example).

Q2: Effect of lag. We thank the reviewer and have conducted additional experiments using the synthetic functions from Figure S4 (exponential, damped sine, logarithm, logarithm + seasonality, power law, logistic, chirp, sawtooth, Gaussian). Each function is applied to a single input variable at a lag d∈{2,4,8,16,32,64,128}d \in \{2, 4, 8, 16, 32, 64, 128\}, and L∈{1,…,256}L \in \{1, \ldots, 256\}. Across all settings, MAPS correlates with forecasting performance at r = 0.96, compared with 0.73 for MI, and at most 0.38 for the other baselines.

Q3: Figure 2 and Figure S4 include synthetic datasets with varying complexity. These datasets have the same expected predictability of 1, and MAPS assigns consistently high scores of 0.9650.965–0.9990.999. We have revised the discussion in Section C.1 to highlight that these datasets share the same expected predictability despite their different relationship complexity.

Q4: A feature-engineered ridge model could perform better. However, its performance would depend on the feature engineering process. This would make the comparison depend on task-specific modelling choices, which does not provide a consistent evaluation setup across various datasets. We therefore use the same simple ridge specification across all datasets. We additionally verified that varying its regularisation strength does not change the conclusions. Specifically, the table below shows varying the ridge penalty α on the synthetic experiment of Figure 3. We report the correlation r between ridge performance (ρ) and the reference predictability, averaged over the six synthetic systems.

Ridge αr
10^-3.924
10^-2.924
10^-1.924
1.924
10.924
10^2.922
10^3.920
10^4.915
10^5.895

Q5: We added a controlled synthetic experiment where the input–target relationship changes over time, while keeping the input and target marginal distributions fixed. As the shift increases, the MAPS score computed on the training data remains unchanged, while the score computed on the test data declines along with the forecasting performance, as expected. This conclusion matches the distribution shift analysis on the real-world data. We have added this analysis with synthetic data in the revised version in Section 4.5.

Thank you to the authors for the rebuttal. However, a few questions still remain unclarified.

  1. Q1. I think my major issue with non-causal windows being used for evaluation is that it does not respect the temporal nature of the forecasting task. The forecasting tasks are always designed in benchmarks and in real-world use cases such that the test timestamps do not overlap with the training timestamps. Given this, the predictability metric should also follow the same principle.

  2. Q5. Distribution shifts cause forecasting performance to differ as expected, and this is typical of real-world datasets where there is non-stationarity. That is why I think that the windows should be chosen causally for all the experiments.

Reviewer_ASS7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MAPS (Multivariable Predictability Score), a model-agnostic metric for quantifying the intrinsic predictability of multivariate time series forecasting tasks. Motivated by the fact that forecasting accuracy conflates model capability with task difficulty while existing predictability measures either fail to capture nonlinear dependencies or are limited to univariate series MAPS leverages covariate information to assess predictability across different lookback windows and forecasting horizons, without requiring a forecasting model. Experiments on synthetic and real-world datasets against 6 baseline measures show that MAPS closely aligns with the expected reference predictability (r = 0.963) and yields more reliable assessments than existing measures in the majority of configurations.

Critical strengths
  1. MAPS addresses the shortcomings of existing measures that rely on forecasting models, ignore covariates, or are limited to univariate series. It is a model-agnostic multivariate predictability measure that explicitly accounts for covariates, lookback windows, and forecasting horizons, without requiring any forecasting model.

  2. On synthetic time series with diverse dynamics and known predictability, MAPS closely follows the reference predictability (average linear correlation r = 0.963), providing direct evidence of the metric’s validity.

  3. Across multivariate Weather and ILI forecasting tasks under multiple lookback and horizon configurations, MAPS exhibits the strongest relationship with empirical forecasting performance compared to 5 existing measures (r = 0.961 on Weather and r = 0.827 on ILI).

Critical weaknesses
  1. The real-world evaluation covers only two task families: Weather (20 locations) and ILI (England plus 3 US regions) both of which are constructed from work by the same group. There are no standard long-term forecasting benchmarks such as ETT, Electricity, Traffic, or Exchange, nor any datasets from other domains such as finance, energy, or retail.

  2. While the paper is motivated by "assessing task predictability prior to modelling," under distribution shift, MAPS computed on training data alone correlates with downstream performance at only r = 0.088 (not significant), rising to 0.684 only after the deployment data are observed effectively undermining its claimed primary use case.

  3. The direction-based measure discards amplitude information, making it not equivalent to the mainstream MAE/MSE evaluation; taking the absolute value after averaging over the neighborhood may misclassify actually predictable dynamics as unpredictable when positive and negative directions are mixed, and this lacks theoretical or empirical justification.

Questions for authors

Please answer the questions in the weakness section.

4 comments
AuthorsOpenReview ↗

Reply to reviewer ASS7

We thank the reviewer for recognizing the clarity of our work and the strong performance of MAPS across synthetic and real-world datasets.

W1: Weather and ILI provide many regions and support multiple test seasons, enabling more robust analysis when measuring predictability. Standard benchmarks such as ETT, Electricity, Traffic, and Exchange offer fewer independent test sets or comparable locations, making them less suitable for this specific question.

W2: In our experiment, we intentionally introduced a substantial distribution shift. Under this setting, MAPS and all baseline measures fail to predict downstream performance using the training data alone. In the real-world WEA and ILI scenarios, the distribution shifts are considerably smaller. We have clarified this point in Section 4.5 of the revised manuscript.

W3: We note that the directional scores are averaged over the selected nearest neighbours. Intuitively, both 11 and −1-1 indicate high predictability, and taking the absolute value before averaging would keep both scores close to 11. However, if the directional scores differ greatly among nearest neighbours, this means that similar inputs lead to different outputs, making the time series less predictable. Additionally, we compared the two choices by correlating the scores with Sonnet’s forecasting performance across 20 Weather locations, averaged across (L,H)(L,H) settings. Results are shown in the table below, where MAPS correlates better with the model performance. We therefore do average first and then take the absolute value. We have clarified this justification in our revised version in Section 3.1.

Weather
Abs → Avg0.943
Avg → Abs (MAPS)0.974

Thanks the authors for their detailed rebuttal and the additional experiments.

  1. Regarding W1, the authors' explanation is that Weather and ILI provide many regions and support multiple test seasons, making them more suitable for predictability assessment. We understand this rationale, but we still have the following concern: ETT, Electricity, Traffic, and Exchange are precisely the standard benchmarks cited in the paper's Introduction to motivate this work (ProbTS, GIFT-Eval, and fev-bench are also cited but not used), yet none of them appears in any experiment. Could the authors report MAPS scores and their relationship with forecasting performance on at least one of these standard datasets?

  2. The additional synthetic experiment and the clarification in Section 4.5 clearly demonstrate that MAPS measures the predictability of the data distribution on which it is computed, and the authors honestly acknowledge in the Limitations that "a score computed on training data may not characterise an unseen deployment distribution." We accept that the score is informative in the real-world WEA/ILI scenarios (where the distribution shift is small), and that no baseline performs better under shift. However, we would like to draw the authors' attention to the fact that the phrase "task predictability prior to modelling" is still retained in the abstract and introduction, which is inconsistent with the framing in Section 4.5 and the Limitations. We hope the authors will soften this claim accordingly in the revised version (e.g., by changing it to "task predictability can be assessed prior to modelling under the assumption that the training distribution approximates the deployment distribution"), so as to avoid readers misunderstanding the scope of applicability of the method.

  3. Regarding W3, the authors provide both an intuitive justification and an empirical ablation, which is corroborated by the "w/ signed directions" ablation results in Table 3 (0.902 vs. 0.961). The evidence is self-consistent.

AuthorsOpenReview ↗

Additional results

  1. We thank the reviewer for this suggestion. We have provided results on the benchmark datasets ETT (ETTh1, ETTh2, ETTm1, ETTm2), Electricity, and Traffic. We set L = 96 and H = 24. To compensate for the small number of tasks in these benchmarks, we use every ETT variable as the target in turn, and 20 randomly sampled series each from Electricity and Traffic. Across these tasks, MAPS shows the highest correlation with Sonnet's forecasting performance (r = 0.75), and on Electricity (r = 0.95), and is comparable to ForeCA on ETT (0.66 vs. 0.67).
MethodAll tasksETTElectricityTraffic
MAPS0.750.540.950.55
ForeCA0.470.530.890.60
Spectral0.440.060.71-0.17
PC0.57-0.080.770.50
MI-0.060.200.630.02
Persistence-0.410.28-0.71-0.06
  1. We thank the reviewer for their advice and will revise the claim in our paper.

  2. We thank the reviewer for acknowledging our justification.

Thank the authors for the thorough rebuttal and additional experiments. The supplementary results on ETT/Electricity/Traffic largely address my request (MAPS: r=0.75 overall, r=0.95 on Electricity). I would appreciate a brief explanation of why ForeCA slightly outperforms MAPS on ETT and Traffic; adding Exchange or a GIFT-Eval subset in the appendix would be even better, but if that is infeasible, a statement in the Limitations would suffice.

Comments on the paper

1 comment
Area_Chair_rYRAOpenReview ↗

Discussion encouraged in this period!

Hi all reviewers,

We’re about halfway through the discussion period. This is a friendly reminder to check the authors' replies. If the authors have answered questions or responded to critiques in the reviews, please leave a comment for the authors!

Thank you so much!

Best,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.