- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether the generative component in probabilistic multivariate time-series models contributes to learning the dependence between time points along the forecast horizon within a given time series (single channel). The authors offer the following empirical and theoretical findings:
- The current main evaluation metrics, namely the Continuous Ranked Probability Score (CRPS) and the Quantile Interval Coverage Error (QICE), are limited by definition to evaluating error per time point (marginals) rather than the error of the estimated joint law across the horizon. The consequence is that, as noted by the authors: “A forecaster that gets every marginal exactly right and the joint law entirely wrong scores perfectly.”
- A matched parametric floor, which keeps the deterministic backbone and replaces the generative component with a diagonal Gaussian over the horizon with independent noise per step, beats the probabilistic time series model in the majority of cases. Measuring the autocorrelation between residuals shows that while the model produces consistent forecasting trajectories, it remains far from the ground truth correlation in the majority of cases.
- The authors pinpoint the training objective of probabilistic time series forecasting models as the main limitation, as it does not provide a supervisory signal for learning temporal covariance. This is backed up by theoretical analysis of the properties of the metrics and the model objective, as well as by an empirical study of two proposed fixes that replace the KDE likelihood term with the variogram score or the energy score, respectively (both also introduced by the authors to evaluate temporal dependence learning).
The analysis includes seven standard forecasting datasets and four representative models: TMDM, NsDiff, D3U, and PPM, although not all analyses are run on all four models (e.g., the parametric floor is only evaluated on PPM and NsDiff).
In essence, the paper shows that generative components in probabilistic multivariate time-series models do not contribute much to learning temporal dependence and ties this to their objective, which lacks the respective supervision signal. Because standard metrics do not evaluate how well models capture temporal dependence either, it remains an unaddressed blind spot. While this is intuitive after the fact, the authors clearly reveal this blind spot and back it up with theoretical and empirical results. Finally, they propose fixes that add a covariance supervisory signal and demonstrate that they mitigate the issue to some extent.
Critical strengths
- The paper exposes a blind spot in both current training objectives and standard evaluation protocols (e.g., CRPS, QICE) of probabilistic time series models, which reward marginal pointwise accuracy while remaining blind to temporal dependence across the horizon. The main claims are backed up by theoretical and empirical analysis.
- The authors propose two computationally lightweight mitigations that introduce a supervisory signal for temporal dependence (replacing the KDE likelihood with the variogram or energy score). The energy score based fix outperforms the baseline in the majority of cases.
Critical weaknesses
- The paper reads like a technical report requiring further distillation. The core findings could be stated much more simply and concisely, leaving space to demonstrate the practical implications of this gap. In addition, some parts read like a list and can be refined to make the reading experience easier (e.g. Limitations section reads like a generated list).
- The paper would benefit from running on more benchmarks and models to strengthen the proposed mitigation and some of the analyses supporting the paper's claims.
Questions for authors
- Can the authors offer a distilled version of the key insights and findings in their paper?
- Have the authors attempted to link improved learning of the joint distribution to concrete downstream tasks or additional measures (e.g., autoregressive forecasting versus direct horizon forecasting)?