acceptodds
Under review as a conference paper at ICLR 2027

When to Trust the Forecast: Post-hoc Failure Risk Estimation for Time Series Forecasting

Abstract

Long-term time series forecasting requires more than low average error: in deployment, users often need to assess whether a generated trajectory is likely to contain large forecast errors. Standard evaluation protocols average errors over samples, variables, and horizons, thereby obscuring severe localized errors that may occur within an otherwise accurate forecast. We formulate post-hoc forecast failure risk estimation for long-term time series forecasting, where a fixed forecasting backbone, its historical input window, and its generated trajectory are used to estimate the risk of large forecast errors before the ground truth is observed. We propose \modelname, a backbone-agnostic model that addresses this problem by fusing historical dynamics, forecast behavior, and diagnostic mismatches between the history and forecast trajectory. Concretely, \modelname decomposes the history into trend and residual tokens, encodes the predicted trajectory with temporal-difference features, constructs diagnostic tokens from mismatch signals such as slope, volatility, range, jump, and lead-time position, and uses a Failure Decoder with Horizon Queries to produce fine-grained risk estimates along the forecast trajectory. By turning post-hoc forecast assessment from sequence-level scoring into fine-grained failure risk estimation along the forecast trajectory, \modelname provides actionable risk signals for long-term forecasting systems.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Time series, Time series forecasting

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_zPCNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed a new problem of forecast risk classification. The goal is to generate a risk label for each forecast. They used validation set to construct groundtruth labels and built dedicated neural network architecture for these risk classification tasks. Experiments demonstrated the proposed method can have better risk classification accuracy than other general time series classification methods.

Critical strengths
  • It’s a new problem framing for time series forecasting, instead of predicting distribution or confidence interval, the paper proposes a classification setting for risks.
  • The experiments demonstrate good risk classification performance from different angles: classification accuracy, error reduction when removing risk classifications; robustness when defining error on different percentiles; and ablations on different components.
Critical weaknesses
  • The definition of risks is simplified. 80 percentile of very small errors is still very small errors, may not be a risk; 20 percentile of big errors is still big errors, should not be neglected.
  • I found the presentation of technical details lacks clarity. I have many detailed questions about Section 4. See more in the question section.

=== Post rebuttal updates ===

The authors improved clarity in the new revision and I increased my score to weak accept.

Questions for authors
  • What positional embedding in (4) is used?
  • How to get E in (5), it seems is a concatenation of e? What roles do the ctrc_{tr} and cresc_{res} play? LN is LayerNormalization?
  • What is forecast-prefix in Section 4.4? Are there definitions of all used features?
  • In (8), Y^h,d−XL,d\hat{Y}_{h,d} - X_{L,d}, why compare a future time step to a history step L? Which L should one use?
  • In (8), how to exactly compute shists^{hist}, average of slope of last k steps and over D variables? They need to be precise.
  • In (9), What are the motivations for that definition? Why are the terms defined in that way? One includes more and more forecasts as h increases, thus larger std; should it be a moving window of size k on the forecasts because you are comparing with the last k steps in the history.
  • For query embedding in (11), it seems to just map an integer h to a vector; is the embedding learnt or is it positional embedding for future timesteps?
  • How is fused memory M defined in (12)?
  • How are the validation datasets (that used to construct the labels) derived?
  • How do you compute Recall@Top20%? How is the Top selected? Based on risk probability? If so, it needs to be clear.
4 comments
AuthorsOpenReview ↗

Gentle Reminder: Follow-up on Our Rebuttal and Additional Experiments

Dear Reviewer,

We appreciate your time and insightful comments. Since the discussion period wraps up in about a day, we kindly invite you to take a look at our rebuttal and additional experiments when you have a moment. We remain available to address any further questions ahead of the deadline.

Thank you for your consideration,

The Authors

AuthorsOpenReview ↗

Follow-up on Our Response Before the Discussion Window Closes

Dear Reviewer,

Thank you for your constructive review and time. As there are fewer than 12 hours left before the discussion phase ends, we wanted to gently follow up in case you have a brief moment to check our response and additional experiments. We are happy to answer any remaining questions before the deadline.

Warm regards,

The Authors

AuthorsOpenReview ↗

Quick Reminder:  2 Hours Left in Discussion

Dear Reviewers,

Just a quick note that there are about 2 hours remaining in the discussion window. We are standing by and ready to respond right away if you have any brief follow-up questions.

Best regards,

The Authors

AuthorsOpenReview ↗

Final Reminder: <30 Minutes Left in Discussion

Dear Reviewers,

As the discussion window wraps up in under 30 minutes, we just wanted to check in one last time. We are actively monitoring this page and ready to respond immediately should anything else come to mind.

We truly appreciate your time and constructive insights throughout this process.

Best regards,

The Authors

Reviewer_tw4bOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors focus on an important but relatively underexplored issue in long-term time-series forecasting: a model may achieve a low average forecasting error while still producing large and potentially harmful errors at particular forecast steps. To address this issue, the paper proposes TSGuard, a post-hoc risk estimation framework that attempts to identify failure-prone forecast steps before the corresponding ground truth becomes available. The main characteristic of TSGuard is that it jointly examines the historical context, the behavior of the generated forecast, and several explicit mismatch signals between them, while using horizon-specific queries to produce a fine-grained risk score along the forecast trajectory. The empirical results across multiple datasets and forecasting backbones are generally promising, showing consistent improvements over the compared baselines in failure identification and risk-based forecast retention. Nevertheless, I still have several concerns regarding the overall formulation, methodological novelty, evaluation protocol, and the practical interpretation of the reported risk scores.

Critical strengths
  1. The paper addresses the important problem of identifying unreliable forecast steps before the ground truth becomes available. 2) TSGuard can be applied to different pretrained forecasting models without modifying or retraining the forecasting backbone. 3) The method produces horizon-wise risk scores instead of assigning a single reliability score to the entire forecast trajectory. 4) It jointly considers historical patterns, forecast behavior, and history–forecast mismatches. 5) The experiments cover seven datasets, five forecasting backbones, and both short- and long-horizon settings. 6) TSGuard generally achieves strong results across AUROC, Recall@Top20%, and AUPRC.
Critical weaknesses

The mathematical notation needs a careful editorial review. In particular, matrices and vectors are not consistently distinguished from scalars, making several expressions unnecessarily difficult to read. The authors should adopt a clear notation convention, apply it throughout the manuscript, and check the equations accordingly.

  1. If this were a newly identified problem, I would consider the current implementation satisfactory. Unfortunately, this topic has already been explored in the existing literature, while the proposed approach still relies heavily on established techniques. Therefore, the overall methodological novelty is limited.
  2. The evaluation mainly measures risk ranking, but does not demonstrate that the predicted risk scores are well calibrated. 3) Some competing methods are trained with their original objectives rather than the same failure-classification objective, raising concerns about comparison fairness. 4) The out-of-fold forecasts used to train TSGuard may differ substantially from the final backbone forecasts used during validation and testing. 5) The failure label averages errors across all variables, which may still conceal severe errors in individual channels.
Questions for authors

My main questions and concerns are detailed in the Weaknesses section above. I would appreciate the authors’ responses to these concerns.

Reviewer_G8YgOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces TSGuard, which is a supervised model that predicts finegrained failure risk for a given time-series forecaster. It combines past observations, future forecast, and a set of engineered mismatch features to identify errors exceeding validation-derived thresholds. Experiments report improved failure detection and lower error on predictions retained after risk-based filtering.

Critical strengths

The paper propose a practical question, which is that when a forecast is already given, can we look at it and the past observation, tells the finegrained reliability over the predicted horizon. The authors use rigorous out of fold split selection, which is a sensible precaution against optimistic training errors.

Critical weaknesses

The significance of the problem formulation is not sufficiently addressed. The problem that the paper porposed, predicting whether forecast error exceeds a threshold, is closely related to existing error estimation, and uncertainty quantification methods over time series. Although the paper acknowledges and compares with some related methods, it does not convincingly establish why this formulation constitutes a substantial research advance beyond a more temporally detailed prediction target.

Limited explanatory insight. TSGuard combines standard encoders, engineered mismatch features, and cross attention. The reported gains are encouraging, but the analysis provides limited understanding of which failure patterns these components capture, when they are necessary, and when the approach is likely to fail.

Questions for authors
  • With the architecture and inputs held fixed, how does binary failure prediction compare with regression on continuous error magnitude? Does threshold-based supervision provide benefits beyond alignment? This would help isolate the contribution of the proposed supervision from that of the architecture.

  • Does TSGuard identify which steps within a forecast are likely to fail, or mainly identify entire forecast windows that are difficult? How much improvement does step-level scoring provide over assigning one risk score to each window?

3 comments
AuthorsOpenReview ↗

Gentle Follow-up: Rebuttal & Additional Experiments (<12 Hours Remaining)

Dear Reviewer,

Thank you once again for your time and thoughtful feedback. With less than 12 hours remaining in the discussion period, we kindly invite you to review our rebuttal and additional experiments if your schedule permits. Please let us know if any points need further clarification before the window closes.

Best regards,

The Authors

AuthorsOpenReview ↗

Quick Reminder:  2 Hours Left in Discussion

Dear Reviewers,

Just a quick note that there are about two hours remaining in the discussion window. We are standing by and ready to respond right away if you have any brief follow-up questions.

Best regards,

The Authors

AuthorsOpenReview ↗

Final Note: Less Than 30 Minutes Remaining

Dear Reviewers,

As the discussion window wraps up in under 30 minutes, we just wanted to check in one last time. We are actively monitoring this page and ready to respond immediately should anything else come to mind.

We truly appreciate your time and constructive insights throughout this process.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Following Up on Our Rebuttal — Happy to Address Any Further Questions

Dear Reviewer,

We appreciate your time and insightful comments. Since the discussion period wraps up in about a day, we kindly invite you to take a look at our rebuttal and additional experiments when you have a moment. We remain available to address any further questions ahead of the deadline.

Thank you for your consideration,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.