acceptodds
Under review as a conference paper at ICLR 2027

Does Accuracy Imply Correctness in Temporal Evolution Forecasting?

Abstract

Claims of transferable temporal evolution require response evidence matched to the declared condition changes. Low error on natural trajectories alone does not supply that evidence. Temporal Evolution Forecasting (TEF) expresses this behavioral requirement through declared information interfaces and conditional-response diagnostics. Controlled experiments show why familiar evidence cannot substitute for these tests: longer histories can improve natural accuracy while amplifying response error, an autoregressive model's local advantage can disappear during rollout, and mechanism-matched positive controls can recover the intended response. Across seven real-world benchmarks, forecast revisions and input sensitivity distinguish properties that natural error alone does not describe, without supplying intervention-response ground truth. Authors should therefore state which evolution capability they claim, evaluate natural forecasts alongside responses to the relevant changes, and establish that their diagnostics reward the intended behavior. The six protocols studied here illustrate this requirement; the appropriate evidence depends on the claim and on which paired outcomes the setting can identify.

Primary area:
learning on time series and dynamical systems
Keywords:
Time series Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Cc5nOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that in time series forecasting, claims of transferability to conditions not observed during training should be backed by evidence that the model behaves appropriately under such conditions, thus, evidence of in-distribution predictive accuracy is not sufficient. The paper explains how a numerical mapping objective, i.e., minimizing the prediction error given past observations, may partially resemble "temporal evolution forecasting" without implying it. Subsequently, the work introduces six diagnostic protocols that allow comparing the behavior of a learned model under specific changing conditions in controlled settings. Finally, the paper demonstrates, on real-world data, how certain quantities (e.g., forecast revision, sensitivity) may behave differently from forecasting error.

Critical strengths
  • The overall claim that in-distribution accuracy is not sufficient for claims of transferability across conditions not observed during training is generally agreeable. Studying diagnostics to recognize when a model is faithfully modeling the intended relationship rather than exploiting shortcuts is a relevant research question, in particular now that foundation models for time series forecasting are increasingly popular.
Critical weaknesses
  • The paper central contributions are unclear and appear insufficient. Temporal Evolution Forecasting is only qualified as requiring forecast behavior to be compatible with the evolution of the target system. The authors do not provide a sufficiently precise and general criterion to assess whether a model satisfies this requirement beyond the selected diagnostic protocols. It is also unclear what the proposed framework adds to existing notions of robustness under distribution shift or system identification, and which problems can be addressed under this framework that could not be addressed under existing perspectives. Ultimately, it's unclear what the paper contribution is beyond arguing that in-distribution performance should not be used to support claims of out-of-distribution performance. The paper does not provide evidence of such unsubstantiated claims being prominent in the literature, or having systematically led to misleading conclusions.
  • The proposed diagnostics seem to provide a collection of examples of how evaluating forecasting accuracy may fail to characterize model behavior under out-of-distribution conditions. However, it is unclear how the framework translates into practical tests for real world scenarios. Indeed, experiments on real-world data do not evaluate response correctness to changing conditions, as ground truth controls are not available. The experiments in Fig. 6 show how certain measurable quantities (e.g., sensitivity, forecast revision) can behave distinctly from forecasting error. However, this is not particularly surprising, and it is unclear how these quantities provide useful evidence to evaluating correct temporal evolution forecasting.
Questions for authors
  • Could the authors provide an operational criterion to define if a learned model conforms to Temporal Evolution Forecasting, beyond the controlled tests presented in the paper?
  • What is the fundamental difference between this framework and other notions like robustness under distribution shift or system identification?
  • How can the proposed framework evaluate behavioral requirements on real-world data? Does the evaluation require a case-by-case ad-hoc design of diagnostics?

Minor suggestions:

  • I suggest clarifying the relevant differences between evaluated ML models (e.g., Direct-last, One-step AR, Window Direct Ridge), contextually with the experiments. These details are relevant to understand the comparisons in the experimental sections, and one should not need to fetch them from the appendix.
  • After Equation 2, the text refers to PnatXP_{nat}^X which does not appear in the equation. I suggest the author assess whether it is a typo or a missing element in the equation.
Reviewer_SJH9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper questions whether low forecasting error on natural trajectories implies that a model has learned a transferable temporal evolution rule. It introduces Temporal Evolution Forecasting (TEF), which evaluates forecasting models not only by natural prediction error but also by their responses to declared condition changes through 6 complementary protocols. Controlled experiments on known dynamical systems show that natural accuracy and response correctness can diverge, while experiments on seven real-world datasets examine forecast revision and input sensitivity as complementary observational diagnostics.

Critical strengths
  1. The figures are well-depicted and the paper is easy to follow.
  2. The paper clearly distinguishes accurate prediction on the observed distribution from correct responses after a state, parameter, or regime change.
  3. The paper treats IID, State OOD, Parameter OOD, Restart, State Edit, and Regime Edit as tests of different claims, which makes the evaluation logic more rigorous and avoids conflating observational prediction with causal or mechanistic identification.
  4. The Duffing history-length experiment shows that natural nMSE can improve substantially while Effect Error becomes much worse, directly supporting the central argument.
Critical weaknesses
  1. Two types of related work are suggested to be discussed separately. Direct forecasting comparators, such as SOFTS[1], DUET[2], and TimeFilter[3], take numerical histories as input and predict numerical futures, closely matching the observational forecasting task considered in this paper; they could therefore be adapted to the controlled datasets without changing the prediction target. Conceptually related works, such as tcKAE[4] and It's TIME[5], study temporal consistency or richer task- and pattern-level evaluation, and would help better position the novelty and scope of TEF even though they do not instantiate the same evaluation setting.

  2. A change in the final observation may correspond to a true state reset, but it could also indicate sensor corruption or a change in an unobserved driver, which would require different responses. The current experiments stipulate the reset interpretation rather than testing whether a forecaster can distinguish among these cases; a reset-aware interface or a comparison between labeled resets and observation corruptions would make the evaluation more practically meaningful.

  3. Effect Error requires paired ground-truth futures before and after an intervention, which are generally unavailable in observational benchmarks. Consequently, the real-world experiments can demonstrate that accuracy, forecast revision, and sensitivity capture different properties, but cannot directly establish response correctness.

  4. Although seven datasets are considered, the fitted Ridge models use channel-independent mappings and therefore do not test cross-channel behavior. The paper could be strengthened by applying the proposed observational diagnostics to several modern joint-channel deep or foundation forecasting models.

  5. The text in Figures 4 and 5 is relatively small. Increasing the font size would improve readability.

[1] Han, Lu, et al. "Softs: Efficient multivariate time series forecasting with series-core fusion." in NeurIPS 2024.

[2] Qiu, Xiangfei, et al. "Duet: Dual clustering enhanced multivariate time series forecasting." in KDD 2025.

[3] Yifan Hu et al. "TimeFilter: Patch-Specific Spatial-Temporal Graph Filtration for Time Series Forecasting." in ICML 2025.

[4] Nayak, Indranil, et al. "Temporally consistent Koopman autoencoders for forecasting dynamical systems." Scientific Reports 15.1 (2025): 22127.

[5] Qiao, Zhongzheng, et al. "It's TIME: Towards the Next Generation of Time Series Forecasting Benchmarks." in ICML 2026.

Questions for authors
  1. How does TEF differ from and complement closely related work on temporal consistency and modern forecasting evaluation?
  2. How can the state-edit protocol distinguish a true state reset from sensor corruption or changes in unobserved drivers?
  3. How should TEF be applied when paired intervention ground truth is unavailable in real-world forecasting datasets?
  4. Can the authors evaluate the proposed diagnostics on modern joint-channel multivariate or foundation forecasting models?
Reviewer_zcBtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on the problem that forecasting accuracy does not guarantee correct responses to changes in system conditions. It proposes Temporal Evolution Forecasting and complementary diagnostics. It conducts experiments on three dynamical systems and demonstrates that the natural forecasting accuracy and response correctness can diverge. The experiment also show that longer histories, autoregressive architectures, and larger models do not guarantee the correct responses. The paper argues that one should match evaluation protocols to specific capabilities of the forecasting models.

Critical strengths
  1. The paper provides counterexamples and empirical demonstration to support its speculation that benchmark accuracy might hide failures.
  2. It conducts controlled experiments that differentiate natural forecasting accuracy from response correctness under changes in system condition. It also uses known dynamics to provide ground truth for evaluation of these responses.
Critical weaknesses
  1. The discussion in the related work section should explain more clearly how TEF posits in the existing literature and what it adds beyond these methods.
  2. The state-reset diagnostic creates histories outside the natural training distribution. But the real world experiments has no ground truth for response correctness. The practical meaning of this diagnostic is limited. It would be better if they can evaluate a more practical setting and show that the diagnostics helps identify models with more accurate forecasting under those changes.
  3. The experiments compare model families and scales, but these comparisons are confounded by unequal access to the system parameters, different training schedules, and changes in scaling factors. The comparisons are not fair enough to support the conclusion. It would be better if the authors could align information access, training conditions across models, and scaling factors to isolate their effects.
Questions for authors
  1. What does TEF contribute beyond existing work and how it posit itself in terms of novelty?
  2. Is it possible demonstrate a real-world setting where these diagnostics help select a model that performs better under actual changed conditions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.