acceptodds
Under review as a conference paper at ICLR 2027

When to Apply Forecast Corrections: Opportunities, Recoverability, and Risk under Delayed Feedback

Abstract

Lightweight forecast corrections can improve a frozen forecaster on average, but heterogeneous effects make the correct action forecast-dependent, and the decisive loss difference is unavailable when a multi-step forecast is issued. We therefore study correction selection separately from correction generation through a matched delayed-feedback evaluation that freezes each candidate pair and distinguishes average correction capacity, retrospective opportunity, issue-time recovery, exposure, and local downside. In the primary Prompt-Z/PatchTST study, retrospective opportunity appears throughout the evaluated matrix, whereas the tested issue-time policies recover only a limited and heterogeneous share. The learned router improves over simple non-anticipating rules and the minimal linear router, but larger routers, regression-risk-aligned objectives, and simple mixtures do not yield consistent further gains under matched Validation/CAL control. Operatingpoint changes jointly alter recovery and intervention exposure, and positive aggregate recovery can coexist with adverse cells and local periods. A separately audited iTransformer scope check supports the same qualitative separation, while the conclusions remain bounded to the evaluated candidate pairs and open-loop replay

Primary area:
learning on time series and dynamical systems
Keywords:
online time series forecasting, delayed feedback, forecast correction, correction routing, issue-time recoverability

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_cJrVOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper is situated in a forecasting context, where the goal is to make accurate predictions about outcomes up until time t+Ht+H based on some history up until time tt. Over time the ideal prediction function may change, meaning that the prediction model might need adjustments. In the context of this paper, these adjustments are not implemented by adjusting/refitting the model, but rather by producing two alternatives: the original prediction and a corrected variant. Then a router chooses whether to use the original or the corrected prediction. The central research question of this paper is then how well such routers work. To measure success, the paper introduces a range of metrics that, for example, compare the perfect routing (in hindsight) with the chosen routing (given only the past). The finding is that router performance really depends on the context, for example, it can be the case that always choosing the correction would be harmful but still sometimes the correction is more accurate than the original.

Critical strengths

In principle it makes sense to come up with evaluation metrics for the routing and to quantify the performnace of different routers on a representative set of datasets.

Critical weaknesses
  • The central issue of the paper is the clarity of the writing. Even with considerable effort I had trouble to even understand the introduction; I only understood it after having parsed the whole paper.
  • I am not yet convinced by the significance of the research question in the first place. Why this complicated setup with original model, correction, and router? Why not directly update the model? The router/correction have no signal that a refit would not have? I would love to see a comparison of the routing approach against a more direct approach.
  • I am not yet convinced that all the introduced metrics are really interesting. For example, why would we care about how well routing could have worked if we could have chosen in hindsight? A realistic 'cealing' would instead measure how well the routing could ideally be given the available signal? Also, in hindsight even random noise can be turned into a useful signal?
  • I am not convinced yet that the datasets and settings in the empirical study really justify more general conclusions, also taking into account the limited theoretical contribution.
Questions for authors

Can you try to rewrite the conclusion paragraph in Section 6 in simple words that help someone who is not directly from the field to understand what the central contribution and benefit of your methodology is? As a practicioner who considers using a router approach to correct forecasting, what can I take from the study?

Reviewer_F2TrOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper tries to analyze the flaws of current evaluation standard of post hoc correction of time series models. They claim that current evaluation just care about the average improvement of a corrector while this might hide the severe suboptimality of some correctors. They propose one measure that looks at the opportunity missed by the post hoc corrections

Critical strengths
  • The paper shows the limitations of some post hoc corrections approaches.
Critical weaknesses
  • The paper is exceptionally difficult to follow, due to understandable phrasing, dense terminology, and heavy jargon (e.g., "matched delayed-feedback evaluation," "delayed regression-deferral," "issue-enqueue-mature-update," and "CAL-TEST chronology") that obscures the core message.

  • The primary contribution boils down to introducing a new diagnostic measurement framework for post-hoc time-series corrections. For an ICLR-level venue, this is insufficient. A contribution of this nature requires a much deeper analytical depth, such as providing a rigorous theoretical analysis that explains why existing post-hoc approaches are fundamentally suboptimal along with concrete guidance on how to fix them, or introducing a framework that fundamentally transforms how models are trained or evaluated.

  • Given the low methodological innovation, one would expect a far more exhaustive and extensive empirical evaluation. The paper lacks a broad comparison across a wider array of time-series models, diverse correction modules, and modern time-series foundation models.

Questions for authors

1- What is concretely the take away message to the community of post hoc corrections in time series ? (Do not use usual evaluation protocol, design new methods) ? I have hard time this goal!

2- Can we have a recipe of how should post hoc corrections be trained ?

Reviewer_tz83OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This study shows when to apply forecast corrections under delayed feedback, distinguishing the benefit of always correcting, the opportunity available to a hindsight oracle, and the gain recovered by a learned router. Across seven datasets, learned routing generally improves on always correcting but captures only a limited share of the oracle gain, while positive average gains can coexist with local losses. The matched evaluation is useful, although the conclusions remain specific to the evaluated candidate pairs. Formatting issues and unclear figures make parts of the paper harder to follow.

Critical strengths
  1. The matched evaluation holds the Frozen/Fixed candidate pair fixed across policies and measures correction capacity, retrospective opportunity, and issue-time recovery on the same records. This separates correction selection from correction generation. I regard this as a substantive strength because it allows the contribution of routing to be assessed without confounding it with differences in candidate quality. 
  2. The primary 84-cell study provides evidence that substantial retrospective opportunity can coexist with limited and heterogeneous issue-time recovery. Larger routers, regression-risk-aligned objectives, and simple mixtures do not consistently improve recovery under matched Validation/CAL control. The analysis also retains adverse cells and local periods rather than reporting aggregate gains alone. These findings give the paper empirical value by showing the limits of the tested policies and the local downside that average recovery can obscure.
Critical weaknesses
  1. My main concern is that the paper measures the Oracle–Learned gap more clearly than it explains it. The theoretical results are mostly standard conditional-risk arguments adapted to this setting, and no new deferral objective is proposed. The experiments show that recovery is limited, but they cannot separate the effects of unavailable information, the chosen features, estimation, and operating control. This leaves the main empirical finding largely descriptive. 
  2. The evidence is also tied closely to Prompt-Z. The primary experiments use Prompt-Z with PatchTST, and the additional iTransformer study changes the backbone but keeps the same correction generator. Since the available opportunity and its recoverability depend on the candidate pair, it is unclear whether the same pattern would hold for other correction mechanisms. This limits how broadly the empirical findings can be interpreted.
Questions for authors

The questions below are ordered by their importance from high to low.

  1. Which finding most advances our understanding beyond documenting limited issue-time recovery? The sources of the Oracle–Learned gap remain unresolved. Can feature ablations or another diagnostic help explain what determines recovery?

  2. Both the primary study and the iTransformer scope check use Prompt-Z. How does this generator affect the amount and predictability of retrospective opportunity? Evidence from another correction mechanism, or analysis relating candidate-pair properties to recovery, would strengthen the empirical contribution.

  3. Could the authors add a hindsight reference with an override budget matched to each policy’s realized exposure? The unconstrained Oracle remains a valid reference, but this additional comparison would show how much opportunity remains at a comparable intervention level.

  4. Table 5 reports Frozen MSE values of 0.196, 0.744, and 0.304 for ETTh1 at H=1H=1 across seeds. What explains this variation? Were training settings and checkpoint-selection rules fixed, and how does candidate quality affect the recovery results?

  5. How stable are the objective comparisons across chronological CAL periods and nearby values of the 0.60 harmful-override precision requirement? The existing exposure sweeps do not directly address sensitivity to these choices.

  6. Why does the moving-block uncertainty analysis use 48 forecast windows? Are the intervals stable under other reasonable block lengths, given overlapping windows and the shared residual state?

  7. Learned improves over Fixed in 79/84 cells, but Table 6 reports mean/median relative MSE reductions of 0.4617%/0.1213%. How many cells remain improved under several stated relative-gain tolerances?

Presentation problems: In Figure 1, the arrow routing, particularly the curved purple arrows, makes misunderstand (p. 2, lines 54–70). Figure 2(b) has overlapping legend and summary annotations (p. 7, lines 324–337). In Figure 3, text overlaps the plotted content, and several labels are too small; the opposite W/T/L conventions in panels (b) and (c) are also easy to miss (p. 8, lines 378–394). Figure 4 has insufficient spacing between panels, with annotations extending into neighboring panel space (p. 9, lines 432–448). The unresolved “Appendices C–??” reference should also be corrected (p. 11, line 568).

These presentation problems make the framework and results difficult to follow. They are the main reason I lowered my recommendation from Weak Reject to Reject.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.