acceptodds
Under review as a conference paper at ICLR 2027

Marginal Scoring Hides What Generative Forecasters Learn

Abstract

Probabilistic multivariate time-series forecasting is dominated by a family of models that pair a deterministic backbone with a generative component, and that report CRPS and QICE as headline numbers. We show that both metrics, as implemented throughout this literature, are computed independently at every (time step, channel) element: they score the predictive marginals and are blind, by construction, to dependence across the forecast horizon. We make the consequence concrete. Taking a recent state-of-the-art model and holding its backbone, normalisation, training objective, sampling budget and evaluation code fixed, we replace only the generative sampler with a diagonal Gaussian head that cannot represent any temporal dependence. This deliberately dependence-free floor beats the published model on CRPS in of datasethorizon cells (by –, at seed standard deviations of at most ) and is generally better calibrated; the four exceptions are the two highest-dimensional benchmarks, where its deficit is –, and a lookback sweep shows that which side wins CRPS is a regime question, decided jointly by dimensionality and context length. Under a dependence-sensitive diagnostic there is no regime: the lag-1 autocorrelation of the sampler's residual paths — what it adds beyond the conditional mean, which is zero for the floor by construction — is positive for the generative model in every one of the cells we ran, across seven benchmarks, two horizons and three lookbacks. It is also small: the true forecast-error autocorrelation is – everywhere, and the generative model recovers – of it on the - and -series benchmarks, and all of it only on the two high-dimensional ones — exactly the cells where it also wins CRPS. The two metric families therefore rank the same pair of models in opposite directions, and the literature reports only one of them. We then isolate the cause. Across three datasets, giving the sampler an explicit horizon covariance buys – less autocorrelation than changing the objective does, so the objective, not the architecture, is the binding constraint. Replacing it with a joint proper scoring rule recovers – of the ground-truth autocorrelation (versus – for the published model), and an energy-score/KDE hybrid dominates the published model on CRPS, QICE, energy score and variogram score simultaneously on all three.

Primary area:
learning on time series and dynamical systems
Keywords:
probabilistic time-series forecasting, proper scoring rules, CRPS, evaluation methodology, temporal dependence, diffusion models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RQS2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether the generative component in probabilistic multivariate time-series models contributes to learning the dependence between time points along the forecast horizon within a given time series (single channel). The authors offer the following empirical and theoretical findings:

  • The current main evaluation metrics, namely the Continuous Ranked Probability Score (CRPS) and the Quantile Interval Coverage Error (QICE), are limited by definition to evaluating error per time point (marginals) rather than the error of the estimated joint law across the horizon. The consequence is that, as noted by the authors: “A forecaster that gets every marginal exactly right and the joint law entirely wrong scores perfectly.”
  • A matched parametric floor, which keeps the deterministic backbone and replaces the generative component with a diagonal Gaussian over the horizon with independent noise per step, beats the probabilistic time series model in the majority of cases. Measuring the autocorrelation between residuals shows that while the model produces consistent forecasting trajectories, it remains far from the ground truth correlation in the majority of cases.
  • The authors pinpoint the training objective of probabilistic time series forecasting models as the main limitation, as it does not provide a supervisory signal for learning temporal covariance. This is backed up by theoretical analysis of the properties of the metrics and the model objective, as well as by an empirical study of two proposed fixes that replace the KDE likelihood term with the variogram score or the energy score, respectively (both also introduced by the authors to evaluate temporal dependence learning).

The analysis includes seven standard forecasting datasets and four representative models: TMDM, NsDiff, D3U, and PPM, although not all analyses are run on all four models (e.g., the parametric floor is only evaluated on PPM and NsDiff).

In essence, the paper shows that generative components in probabilistic multivariate time-series models do not contribute much to learning temporal dependence and ties this to their objective, which lacks the respective supervision signal. Because standard metrics do not evaluate how well models capture temporal dependence either, it remains an unaddressed blind spot. While this is intuitive after the fact, the authors clearly reveal this blind spot and back it up with theoretical and empirical results. Finally, they propose fixes that add a covariance supervisory signal and demonstrate that they mitigate the issue to some extent.

Critical strengths
  1. The paper exposes a blind spot in both current training objectives and standard evaluation protocols (e.g., CRPS, QICE) of probabilistic time series models, which reward marginal pointwise accuracy while remaining blind to temporal dependence across the horizon. The main claims are backed up by theoretical and empirical analysis.
  2. The authors propose two computationally lightweight mitigations that introduce a supervisory signal for temporal dependence (replacing the KDE likelihood with the variogram or energy score). The energy score based fix outperforms the baseline in the majority of cases.
Critical weaknesses
  1. The paper reads like a technical report requiring further distillation. The core findings could be stated much more simply and concisely, leaving space to demonstrate the practical implications of this gap. In addition, some parts read like a list and can be refined to make the reading experience easier (e.g. Limitations section reads like a generated list).
  2. The paper would benefit from running on more benchmarks and models to strengthen the proposed mitigation and some of the analyses supporting the paper's claims.
Questions for authors
  1. Can the authors offer a distilled version of the key insights and findings in their paper?
  2. Have the authors attempted to link improved learning of the joint distribution to concrete downstream tasks or additional measures (e.g., autoregressive forecasting versus direct horizon forecasting)?
Reviewer_rCTROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether commonly reported metrics adequately evaluate temporal dependence in probabilistic multivariate time-series forecasting. Unlike prior critiques of CRPS-Sum, it identifies a distinct limitation of per-element CRPS and QICE through theoretical analysis and an audit of existing implementations. To demonstrate its practical significance, the authors replace PPM's generative sampler with a capacity-matched diagonal Gaussian baseline while preserving its training and evaluation setup. Across seven benchmarks, this simpler baseline achieves lower CRPS in 10 of 14 configurations despite lacking residual temporal dependence, whereas PPM performs better on dependence-sensitive evaluations. Further structure–objective experiments show that changing the training objective is more effective than increasing covariance capacity alone. The paper thus contributes both an evaluation diagnostic and evidence that dependence-aware objectives can improve temporal structure, although variogram-score training may compromise marginal calibration.

Critical strengths

1. The paper provides strong evidence of the limitations of commonly used evaluation metrics by combining theoretical analysis with empirical results. The authors mathematically demonstrate that per-element CRPS and QICE cannot distinguish joint predictive distributions with identical marginals but different temporal dependence structures. Furthermore, Table 1 shows that, across 14 experimental configurations, the Gaussian Floor, which cannot model residual temporal dependence, achieves lower CRPS in 10 configurations, whereas PPM achieves better Variogram Scores in 12 configurations and residual temporal autocorrelations closer to the ground truth in all 14. These findings connect the theoretical evaluation blind spot to differences in practical model rankings, demonstrating that marginal metrics alone may not adequately assess the quality of joint predictive distributions produced by generative forecasters. This strengthens the practical relevance of the research problem and the credibility of the paper's conclusions.

2. The paper further investigates the factors limiting temporal dependence learning in generative forecasting models through controlled experiments. In Section 7 (Table 2), the authors systematically vary the covariance structure and training objective to distinguish the effects of model expressiveness from those of the optimization objective. The results show that introducing low-rank covariance under the original pointwise KDE objective yields only marginal improvements in temporal autocorrelation, whereas dependence-aware training objectives lead to substantially greater improvements. This experimental design strengthens the paper by moving beyond identifying the limitations of existing evaluation metrics to investigating the potential reasons why generative forecasting models fail to fully capture temporal dependence, providing empirical evidence to guide the design of improved training objectives.

Critical weaknesses

1. The evaluation of the joint predictive distribution remains incomplete. While the paper highlights the limitations of per-element marginal metrics in capturing temporal dependence, its empirical assessment of temporal dependence modeling still relies primarily on the Variogram Score and temporal autocorrelation. In Section 8 (Table 2), the authors show that training with the Variogram Score can improve dependence-sensitive metrics while potentially overestimating correlations at certain lags. Furthermore, Section 9 acknowledges that these metrics only summarize certain aspects of the joint predictive distribution. Consequently, although the experiments demonstrate improvements in specific temporal dependence metrics, it remains unclear to what extent these improvements reflect greater accuracy of the full joint predictive distribution. This limits the strength of the paper's conclusions regarding improvements in temporal dependence modeling.

Questions for authors
  1. Could the authors provide additional diagnostics beyond the Variogram Score and temporal autocorrelation to evaluate the quality of the joint predictive distribution? For example, a comparison of multi-lag correlation structures or other dependence-sensitive diagnostics could help determine whether the reported improvements reflect broader improvements in temporal dependence modeling. Evidence that these improvements extend beyond the currently reported metrics would address my main concern and strengthen my assessment of the paper.

  2. In Section 7 (Table 2), is the limited improvement from adding low-rank covariance under the pointwise KDE objective primarily due to the objective's inability to reward temporal dependence? If so, how should we interpret the substantially larger improvements observed when low-rank covariance is combined with dependence-aware objectives? Could the authors clarify the interaction between model expressiveness and the training objective?

Reviewer_mUssOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper observes that a line of probabilistic forecasting papers is evaluated with two marginal distribution metrics: CRPS and QICE. These metrics alone cannot capture the dependency between the time points.

The authors formalized 3 propositions on what marginal scores cannot do and what joint objectives identify. Empirically:

  1. A simple Gaussian head (floor; derived from PPM) with diagonal variances beat the PPM model in terms of CRPS in 10/14 cases.
  2. However, the floor is worse at the lag-1 autocorrelation compared to the PPM.
  3. The authors studied whether the effect is because of the structure of the model or the loss function of the model and argued that the issue is with the loss function.

Training with the Energy score + KDE on low-rank + diagonal Gaussian outperformed PPM on all CRPS, QICE, and variogram scores.

Critical strengths
  1. Identified a real gap in the evaluation of probabilistic models of the lineage TMDM, NsDiff, DD3U, and PPM. These models are evaluated only using CRPS and QICE, which cannot measure temporal dependencies. This gap extends beyond the studied lineage, as many other probabilistic models are evaluated with univariate distribution metrics.

  2. A simple Gaussian model with diagonal covariance outperforming the PPM model often, in terms of CRPS is an insightful result. This shows what many probabilistic models lack: a simple yet strong baseline. This exposes what the literature lacks: predict the Gaussian distribution with the same encoder and see how it performs in terms of CRPS compared to complex distribution prediction models like diffusion.

  3. A practical and cheap fix for the line of models selected by the authors is useful. Train the model with dependency metrics such as variogram or energy score/KDE hybrid. With energy score/KDE hybrid loss function, a simple low-rank + diagonal covariance Gaussian beats PPM on both univariate and multivariate metrics.

Critical weaknesses
  1. The scope of the paper is limited to the PPM model, although the authors did bring in the discussion of NsDiff for one dataset. Hence, it is hard to generalize the conclusions to other models. While it is possible that the Gaussian baseline could be competitive, it was not shown for other encoders, and hence conclusions cannot be generalized.

  2. 1-lag autocorrelation is considered a key diagnostic to show how the independent Gaussian prediction models are worse than dependence-aware models. However, it is not a useful diagnostic because it does not truly evaluate the quality of the predicted distributions. The number is averaged over all windows, which reduces the forecast to a single summary. Hence, incorrect forecasts can score identically, and averaging over windows lets errors in opposite directions cancel out.

  3. Writing is dense in some places. The text is filled with a lot of numbers, making it hard to follow which ones are important.

Questions for authors

In addition to the weaknesses:

  1. Can the authors also provide point prediction accuracy for the floor and PPM? Is the CRPS gain due to a better distribution or just a better mean?
  2. For the PPM model, the authors stressed diagnosing with the 1-lag autocorrelation only on the residuals. However, they chose to use the full distribution when using NsDiff. Can the authors provide residual autocorrelation for NsDiff as well?
  3. In the introduction, the authors mentioned that "a deterministic forecast supplied a conditional mean". However, the value provided by the deterministic component is not the mean of the distribution but the starting point for the transformation. This needs to be clarified. (clarification)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.