- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that in time series forecasting, claims of transferability to conditions not observed during training should be backed by evidence that the model behaves appropriately under such conditions, thus, evidence of in-distribution predictive accuracy is not sufficient. The paper explains how a numerical mapping objective, i.e., minimizing the prediction error given past observations, may partially resemble "temporal evolution forecasting" without implying it. Subsequently, the work introduces six diagnostic protocols that allow comparing the behavior of a learned model under specific changing conditions in controlled settings. Finally, the paper demonstrates, on real-world data, how certain quantities (e.g., forecast revision, sensitivity) may behave differently from forecasting error.
Critical strengths
- The overall claim that in-distribution accuracy is not sufficient for claims of transferability across conditions not observed during training is generally agreeable. Studying diagnostics to recognize when a model is faithfully modeling the intended relationship rather than exploiting shortcuts is a relevant research question, in particular now that foundation models for time series forecasting are increasingly popular.
Critical weaknesses
- The paper central contributions are unclear and appear insufficient. Temporal Evolution Forecasting is only qualified as requiring forecast behavior to be compatible with the evolution of the target system. The authors do not provide a sufficiently precise and general criterion to assess whether a model satisfies this requirement beyond the selected diagnostic protocols. It is also unclear what the proposed framework adds to existing notions of robustness under distribution shift or system identification, and which problems can be addressed under this framework that could not be addressed under existing perspectives. Ultimately, it's unclear what the paper contribution is beyond arguing that in-distribution performance should not be used to support claims of out-of-distribution performance. The paper does not provide evidence of such unsubstantiated claims being prominent in the literature, or having systematically led to misleading conclusions.
- The proposed diagnostics seem to provide a collection of examples of how evaluating forecasting accuracy may fail to characterize model behavior under out-of-distribution conditions. However, it is unclear how the framework translates into practical tests for real world scenarios. Indeed, experiments on real-world data do not evaluate response correctness to changing conditions, as ground truth controls are not available. The experiments in Fig. 6 show how certain measurable quantities (e.g., sensitivity, forecast revision) can behave distinctly from forecasting error. However, this is not particularly surprising, and it is unclear how these quantities provide useful evidence to evaluating correct temporal evolution forecasting.
Questions for authors
- Could the authors provide an operational criterion to define if a learned model conforms to Temporal Evolution Forecasting, beyond the controlled tests presented in the paper?
- What is the fundamental difference between this framework and other notions like robustness under distribution shift or system identification?
- How can the proposed framework evaluate behavioral requirements on real-world data? Does the evaluation require a case-by-case ad-hoc design of diagnostics?
Minor suggestions:
- I suggest clarifying the relevant differences between evaluated ML models (e.g., Direct-last, One-step AR, Window Direct Ridge), contextually with the experiments. These details are relevant to understand the comparisons in the experimental sections, and one should not need to fetch them from the appendix.
- After Equation 2, the text refers to which does not appear in the equation. I suggest the author assess whether it is a typo or a missing element in the equation.