acceptodds
Withdrawn from ICLR 2027

Beyond Average Error through Oracle-Informed Stress Tests for Time-Series Forecasting

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Average squared error cannot reveal whether forecasting performance degrades because the future becomes less predictable or because forecasts move farther from the conditional mean. We introduce paired, mechanism-controlled stress tests that decompose changes in expected squared error at each lead time into environmental risk and forecast–oracle distance, using an origin-conditioned predictive oracle unavailable to the evaluated models. Three end-to-end controls have known attribution. Specifically, the null, environmental-only, and information-gap controls verify that the pipeline assigns changes to the correct component. We then apply the benchmark to 24 deployable forecasters. Under frequent switching, 14 methods have higher realized MSE but lower oracle distance; under outlier-variance feedback, 19 have higher MSE but lower scale-standardized MSE. Short- and long-lead stress-response rankings have Spearman correlation 0.624, revealing substantial horizon-dependent reordering. We then study multivariate relation shifts. Across six models and three coupling severities, oracle distance accounts for only 0.7–3.9% of the decomposed expected-risk increase, and environmental-risk majority persists in an eight-channel system and a matched-difficulty audit of Ring, Block, and Hub relations. Finally, prespecified contrasts on independent data-generating process (DGP) realizations show that several visually compelling discovery profiles, including trend accumulation and the hypothesized switching reversal, do not replicate. The benchmark thus combines component-wise diagnosis with a held-out stability audit. It complements real-data out-of-distribution evaluation, which measures performance under realistic shifts when exact oracle attribution is unavailable.

Primary area:
datasets and benchmarks
Keywords:
Time-series forecasting, forecasting benchmarks, stress testing, distribution shift, out-of-distribution evaluation, forecast evaluation, uncertainty decomposition, predictive oracle, mechanism-controlled evaluation, robustness analysis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NNNeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a controlled benchmark designed to diagnose changes in time-series forecasting error. By employing paired baseline and shifted environments, the benchmark decomposes the change in expected squared error into changes in environmental risk and forecast–oracle distance. Results are reported by the type of underlying mechanism and individual forecast lead. The experiments encompass multiple mechanism families and deployable forecasting models. The findings reveal discrepancies among various diagnostic metrics and show that the models’ stress-response rankings can vary depending on the forecast lead. In the VAR systems evaluated, environmental risk accounts for most of the increase in expected squared error.

Critical strengths
  1. The paper studies an interesting and important question in time-series forecasting.

  2. The experimental evaluation is extensive, and the provided code supports reproducibility.

Critical weaknesses
  1. In the evaluated stable, zero-mean VARs, stronger coupling increases noise propagation, while long-horizon conditional means approach zero. A zero predictor might show similar environmental-risk dominance, so the model-specific insight from evaluating these forecasters is unclear without that comparison.

  2. Although Eq. (2) separates information restriction from model approximation and estimation error, the main experiments report only their combined oracle distance. This limits what the benchmark can say about how much of the gap better modeling could reduce under the same information set.

Questions for authors

None

Reviewer_2hYxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents paired, mechanism-controlled stress tests for time-series forecasting. Using an evaluator-only predictive oracle, it separates changes in expected squared error into environmental risk and forecast-oracle distance, with reporting by mechanism and forecast lead. The framework is evaluated on 24 deployable forecasters through known-attribution controls, multivariate coupling-shift experiments, and independent held-out checks.

Critical strengths
  1. A useful diagnostic perspective. Separating environmental risk from forecast-oracle distance clarifies why higher raw error need not mean that predictions have moved farther from the conditional mean. Mechanism- and lead-specific reporting also exposes differences that aggregate scores can conceal, making the evaluation more informative.

  2. End-to-end controls and independent confirmation. Known-attribution controls test the evaluation pipeline, while prespecified contrasts on independent seeds distinguish exploratory patterns from replicated findings. Reporting non-replications rather than only favorable results strengthens confidence in the stated scope of the conclusions.

Critical weaknesses
  1. The incremental diagnostic value over prior work remains insufficiently demonstrated. SynTSBench already evaluates predictions against both noisy observations and the underlying clean signal. Although this paper adds origin-conditioned oracles, strict pairing, and lead-wise analysis, a direct comparison or concrete examples are needed to establish which substantive diagnoses these additions enable beyond the existing approach.

  2. The strongest attribution finding may depend heavily on the generator and horizon aggregation. As acknowledged in Section 5.3, stronger coupling in the chosen linear-Gaussian VARs amplifies innovation uncertainty. The headline attribution fractions also aggregate 192 leads. Lead-wise decompositions and constant-prediction and fitted-VAR baselines in the same multivariate experiment would clarify what this finding reveals beyond the generator’s properties and the chosen aggregation.

Questions for authors

Please see the Critical Weaknesses section. I would consider raising my score if these concerns are satisfactorily addressed.

Reviewer_p1NNOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a method in time-series forecasting for distinguishing between higher "real" future variance and "model error" in predicting the conditional mean. This provides a mechanism for error attribution and can support comparisons between different time-series methods in heterogeneous environments.

Critical strengths

Bias-variance tradeoffs such as the one being studied here are important in machine learning in general.

The paper has a lot of empirical results which seem to be correct.

Critical weaknesses

The biggest weakness is that there is no empirical comparison against any prior work. Section 6 lists a bunch of prior works but there is no head-to-head comparison against any of them. And there really should be some baseline comparison against the literature, because evaluating model bias vs real-world inherent variance is a longstanding problem in machine learning.

Another major issue is the motivation. It is not made clear why we would want to do this. What downstream method is being improved by this technique? What scientific question is being answered?

Much of the paper is hard to follow or incoherently structured. For example, Figure 1 contains so much information that the text is impossible to read if the paper is printed out. The "Forecasting contract" paragraph consists of a bunch of text that repeatedly expresses that the future is not included in the information used to make a prediction (a totally standard assumption that should go without saying) expressed in multiple redundant ways: "The model never receives DGP parameters, latent or realized futures, or oracle paths"; "No future latent state, event, or observation enters either information set"; "contains only explicitly declared pre-origin generator state." At one point, the paper says "Appendix E gives assumptions and proof" but we shouldn't hide assumptions in an appendix: assumptions required for a result are important and belong in the main body of the paper if results are in the main body of the paper. Broadly, the paper just lacks motivation throughout.

Questions for authors

How does this work compare empirically against previous approaches that try to do the same thing? If no previous work has ever tried to do anything like this such that no comparison is possible, why should readers care about this problem?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.