acceptodds
Under review as a conference paper at ICLR 2027

When Should a Forecast Be Corrected? Prediction Quality Is Not Decision Quality

Abstract

Post-hoc forecasting systems can leave two predictions for the same future target: a baseline forecast and a proposed revision. The system must decide whether to apply the revision before the target, and hence its true benefit, is known. We study this candidate-conditioned decision as forecast override. Under squared error, a revision's realized benefit depends on its alignment with the baseline residual and its magnitude; forecasts with similar errors can therefore favor opposite actions. We evaluate preserve-or-apply policies with Selection Regret, the excess loss relative to choosing the better action in hindsight. On Weather, a residual-based policy has regret 0.01164, compared with 0.03173 for preserving and 0.09048 for applying every revision. On ETTm2, its point-estimate regret is higher than that of the calibration-selected constant action, although the paired interval includes zero. Across forecasting backbones and candidate sources, predictive scores and average correction gains do not consistently rank execution policies. These results motivate evaluating a proposed revision as a decision whose value depends on the candidate and the information available when it is executed.

Primary area:
learning on time series and dynamical systems
Keywords:
time series forecasting, post-hoc forecast correction, forecast override, selection regret, decision-focused learning, forecast evaluation, time series foundation models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_W9MSOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a decision that arises in post-hoc forecasting systems. A baseline forecast y₀ and a candidate revision c (from a residual corrector, an output-space adapter, or a learned reviser) are both available. The system must decide, before the target is observed, whether to keep y₀ or output y₀ + c. The authors call this binary choice forecast override. They argue it should be evaluated separately from how well the candidate is generated or predicted.

The formal contributions are:

  1. When applying helps. Under squared error, the relative gain from applying the candidate is Γ = 2ρr − r². Here ρ is the cosine alignment between the candidate and the baseline residual, and r is the candidate's size relative to that residual. So applying helps only when 2ρ > r. The authors credit this alignment–magnitude trade-off to δ-Adapter and use it per instance.
  2. Selection Regret. This is the excess loss over the better of the two actions in hindsight. It splits into missed opportunity and harmful exposure. The authors prove that for a policy that thresholds predicted utility, the error in predicted utility bounds the regret.
  3. Two sources of lost value. At the population level, the total lost value splits into a Bayes evidence gap and a policy gap. The authors state explicitly that their experiments cannot separate the two.

Empirically, the paper evaluates preserve/apply policies on ETTm2, Weather and Electricity.

  • Primary grid. It uses Chronos-Bolt tiny with a generic residual candidate and four matched prediction targets: full residual, direct utility, alignment and projection.
  • Robustness checks. These swap the backbone (TimesFM 2.5 over 14 dataset–horizon settings, and Moirai 2.0 on three settings) and the candidate family (residual, polynomial, rank-1). They also use learned PIR revisions and official COSA outputs.

The main findings:

  • Base-forecast error alone does not determine the right action: windows with nearly equal error can favor opposite actions.
  • Both wrong-direction and oversized candidates occur in practice.
  • Strong predictive scores (cosine, AUROC, R²) do not consistently translate into lower Selection Regret than the best constant action. The prediction-based policies clearly win on Weather (residual policy regret 0.0116 vs. 0.0317 for always preserving and 0.0905 for always applying). They are not significantly better on ETTm2 and are unstable on Electricity.
  • PIR and COSA improve average MSE while 20–50% of their individual revisions are harmful. Learned gates do not reliably beat always applying PIR's revisions.

The authors conclude that a proposed revision should be evaluated as a decision tied to that specific candidate, rather than through prediction quality or average correction gain.

Critical strengths
  1. A real and under-examined decision point in post-hoc forecasting. The paper separates two questions that recent post-hoc work (PIR, COSA, δ-Adapter, CTRL) treats as one: how to generate a revision, and whether to execute it. Keeping y₀ as an explicit default action, and judging each concrete candidate against it, is a clean framing that practitioners can use. It applies to any source of corrections, whether learned correctors, reconciliation, or externally supplied adjustments.

  2. A decision-relevant evaluation metric. Selection Regret, split into missed opportunity and harmful exposure (Eq. 6, Table 10), explains why policies with similar correction rates differ in value. For example, Weather is dominated by harmful exposure while ETTm2 is dominated by missed benefit. Aggregate MSE, AUROC, or cosine similarity cannot show this, and the decomposition is more useful than a single "regret closed" number.

  3. The math is correct and connects to the decision. I checked the derivations; they are correct. The direction–scale form (Eq. 4) turns the known alignment–magnitude trade-off into a per-instance rule for when applying helps. Table 19 confirms that both failure routes, wrong-direction and overscaled candidates, occur in practice, so both terms in the rule matter empirically. Proposition S1 explains why utility MAE, unlike ranking metrics, bounds decision regret.

  4. Careful experimental protocol. Calibration and test windows are strictly separated in time, correction-utility labels are fit out-of-fold with purging, and test futures are never used for any selection step. Paired circular moving-block bootstrap intervals (block lengths up to 16) and five-seed reruns are reported. This is more rigorous than is typical in time-series forecasting papers and makes the positive Weather result credible.

  5. External validation on a published correction system. Evaluating unmodified official COSA outputs and exported PIR revisions shows that the effect is not an artifact of the authors' own candidate construction. Both systems improve average MSE while 20–50% of their individual revisions are harmful, which concretely supports the paper's central claim.

  6. Honest scoping of claims. The authors credit the alignment–magnitude identity to prior work. They state that the Bayes evidence gap is not estimated, so no information-theoretic limit is claimed, and they report null and unfavorable settings alongside the positive one. This candor is appreciated, though it also exposes how limited the positive evidence is (see Weaknesses).

Critical weaknesses
  1. The theory adds little that is new. The authors credit the main identity (Eq. 3–4) to δ-Adapter, and it is a few lines of algebra. Selection Regret is ordinary regret for a binary decision. Proposition S1 (regret ≤ |U − Û|) is the standard bound for a classifier that thresholds a predicted value. Proposition S2 is Jensen's inequality, i.e. the value of extra information. Each result is correct, but none goes beyond textbook results, and together they do not make a theoretical contribution of ICLR depth. The paper's main claim, that prediction quality does not imply decision quality, is already well established in decision-focused learning, which the paper itself cites.

  2. The closest prior work is missing. The preserve-or-apply decision has long been studied for judgmental forecast adjustments:

    • Blattberg & Hoch (1990) combine the model and the manager 50/50, which amounts to a fixed partial application of the correction.
    • Fildes et al. (2009, IJF) characterize which supply-chain overrides help or hurt by their sign and size, which is essentially an empirical preserve/apply rule.
    • Forecast Value Added (FVA) analysis evaluates exactly whether an override improves on the baseline.

Two further connections are also missing. The |U|-weighted utility-sign classifier is textbook example-dependent cost-sensitive learning (Elkan, 2001; Zadrozny et al., 2003). Preserve/apply is also a two-model forecast combination restricted to weights 0 or 1 (Bates & Granger, 1969). Without positioning against these, the paper's novelty is overstated.

  1. No method that works reliably, and no explanation of when gating helps. The empirical message is that results depend on the setting. Prediction-based policies clearly win only on Weather. On ETTm2 they do not beat always applying in any of 5 seeds, and Electricity is unstable. On PIR, learned gates do not reliably beat always applying the revision. The paper does not identify which properties of a dataset, backbone, or candidate predict a favorable regime. As a result, practitioners cannot act on the findings, and the work reads as a collection of case studies rather than a general result.

  2. Partial application is not considered. The paper's own algebra gives the best fraction of the candidate to apply per instance: α* = ⟨e, c⟩/‖c‖² = ρ/r. Applying y₀ + α·c with α between 0 and 1 is always at least as good as choosing between 0 and 1 only. The randomized version in Appendix A.4 is a different object. Continuous shrinkage is the obvious baseline that real systems would use. Its absence leaves open whether the reported failures of binary gating simply come from restricting the action to all-or-nothing.

  3. The evidence base is small.

    • The primary grid uses 64 test windows per setting, a single target channel per dataset, and calibration sets of 128–512 windows. The backbone is the smallest Chronos-Bolt model.
    • Many of the key intervals cross zero: ETTm2, Electricity, most TimesFM settings, and the PIR gates.
    • The abstract leads with an ETTm2 comparison whose paired interval includes zero.
    • Moirai covers only 3 settings and COSA only 2, yet the conclusion speaks of results across "three TSFM families."

Multivariate forecasting, longer horizons, and larger-scale benchmarks are absent.

  1. Some results look anomalous and are unexplained.
    • Several utility-sign AUROCs are far below 0.5: TimesFM Electricity 0.083–0.138 (Table 11), Moirai Electricity 0.128 (Table 12), and a residual AUROC of 0.056 at Electricity/h=96.
    • In these same settings the utility policy is catastrophically worse than the constant action (e.g. 0.284 vs. 0.0049 at Electricity/h=48).

An AUROC near 0.1 means the score ranks almost perfectly in the inverted direction. A sign-flipped score would reach AUROC ≈ 0.9. This points to a systematic shift between calibration and test, or a pipeline error. Either way it needs diagnosis, because these settings drive the "strongly negative" Electricity conclusions in Section 5.3 and Table 13.

  1. The analysis only applies to squared error. The direction–scale geometry, and therefore the mechanism analysis in Table 2 and Table 19, holds only for MSE. Many forecasting applications are evaluated with quantile/pinball loss, MAE, or WAPE. The paper does not discuss how the decision boundary changes under these losses. "Decision quality" is also still measured by forecast loss rather than by a downstream decision cost. The execution-cost term κ (Appendix A.4) is introduced but never used in the experiments.

  2. The presentation makes results hard to verify.

    • The "generic-residual candidate," which drives most experiments, is never defined in the main text.
    • Seven experimental grids (Table 4) produce similarly named numbers that cannot be compared across grids. For example, the Weather/Chronos residual-candidate advantage is 0.0333 in Table 13 but 0.0739 implied by Table 1. Readers must constantly track which grid a number comes from.
    • Table numbering skips (no Tables 5, 15, or 18), which suggests incomplete editing.
    • The prose is heavily hedged. Many sentences qualify their claims so carefully that it is hard to extract a clear takeaway.
Questions for authors

I am open to raising my score if the authors adequately address the questions below. Questions 1–4 matter most to my assessment.

  1. How does continuous shrinkage, y₀ + α̂(Z)·c with a learned α̂ ∈ [0, 1], compare with binary gating and with the two constant policies on the primary grid, PIR, and COSA? A simple calibration-fitted global α would also be useful as a baseline. If shrinkage consistently beats the constant actions, that would substantially strengthen the practical contribution. If it does not, that would strengthen the claim that the execution problem is genuinely hard.

  2. Please explain the utility-sign AUROCs of 0.056–0.138 on Electricity (Tables 11–12). Is this caused by a shift between calibration and test (e.g., in the sign of the residual), by a sign or label convention error, or by something else? Do the Electricity conclusions in Section 5.3 and Table 13 still hold once this is resolved?

  3. Can the authors identify properties, measurable before deployment, that predict whether a setting is favorable (Weather-like) or unfavorable (ETTm2-like)? Candidates include the base rate of beneficial revisions, the distribution of ρ and r on calibration data, or how stable the residual sign is over time. Even a simple diagnostic validated across the TimesFM grid would turn the findings into actionable guidance.

  4. How does forecast override differ, conceptually or technically, from:

    • (a) the judgmental-adjustment literature (Blattberg & Hoch, 1990; Fildes et al., 2009) and Forecast Value Added analysis;
    • (b) example-dependent cost-sensitive classification, which the |U|-weighted classifier instantiates;
    • (c) two-model forecast combination with weights restricted to {0, 1}?

    What is new beyond applying an existing decision lens to ML-based post-hoc correctors?

  5. Please define the generic-residual, polynomial, and rank-1 candidates precisely in the main text. What information is each fitted on, and how is c constructed at inference time?

  6. How does the decision boundary look under pinball/quantile loss or MAE? Is there an analogue of the 2ρ > r condition? Do the main empirical conclusions (Table 1, Table 3) still hold when utility is measured under these losses?

  7. Can the primary target comparison be extended beyond 64 test windows and a single channel? For example: all channels of each dataset, rolling-origin evaluation over several test periods, or the larger Chronos-Bolt models. As it stands, only Weather produces intervals that exclude zero. Would the ETTm2 and Electricity conclusions survive a larger test set?

  8. Calibration chooses always-preserve (regret 0.04316), yet always-apply has lower held-out regret (0.03546). Is this caused by the small calibration set (N_cal = 128) or by a shift over time? How sensitive are the constant-action reference choices to calibration size in general?

  9. Could the authors provide a single table that reproduces the Table 1 Weather/Chronos configuration inside the candidate-family grid (G5)? This would explain why the residual-candidate advantage is 0.0333 in Table 13 but about 0.074 in Table 1, and show which factor causes the difference.

  10. Have the authors evaluated whether policy rankings by Selection Regret agree with rankings by a downstream decision cost, such as a newsvendor or inventory cost, possibly using the execution-cost term κ from Appendix A.4? This would support the claim that the paper evaluates decision quality rather than a re-weighted forecast loss.

7 comments
AuthorsOpenReview ↗

Response to Reviewer W9MS — Q1: Continuous Shrinkage and Binary Actions

Thank you for raising this important point. We agree that partial application is an essential comparator: comparing only Preserve and Apply is not sufficient when a proposed correction can be applied at an intermediate strength. Your suggestion helped us identify a gap in our original evaluation.

We therefore added Preserve (alpha=0), Apply (alpha=1), Half (alpha=0.5), calibration-fitted Global shrinkage, and evidence-adaptive shrinkage. We also revisited the original G1 binary results and directly replayed the Full-Residual policy from frozen per-window actions, checking them against stored baseline losses, utilities, and policy losses. No model was refitted, and no test labels were used to select an execution rule.

The resulting held-out MSE comparisons are:

Source / test windowsPreserveApplyHalfGlobalAdaptiveSource binaryResidual binary
Chronos G1 ETTm2 / 64.038479.009884.017873.009884.009680.012214.011864
Chronos G1 Weather / 64.056111.114857.058285.091446.098671.041005.036020
Chronos G1 Electricity / 64.390753.383051.375427.375525.381158.388323.388382
PIR ETTm2 / 2,189.280573.246281.247941.243298.244371.245771—
PIR Weather / 1,993.245390.216175.220429.215359.214863.214451—
PIR Electricity / 937.755645.211025.339851.210716.209895.211025—

All values are held-out MSE, not binary Selection Regret. Source binary denotes the original Direct-Utility MLP in Chronos G1 and utility-weighted gate in PIR. Residual binary denotes the distinct Full-Residual MLP in primary G1; no corresponding PIR policy exists. G1 and PIR use different backbones, candidates, and windows, so scores should not be compared across rows.

The paired analysis clarifies where the evidence is stronger and where it remains uncertain. On G1 Weather, Full-Residual achieves .036020 MSE versus .058285 (Half), .091446 (Global), and .098671 (Adaptive). Full-Residual-minus-reference differences with 95% block-16 CIs are: Half, -.022265 [-.028086, -.017046]; Global, -.055426 [-.070197, -.039678]; and Adaptive, -.062651 [-.084184, -.040993]. All three marginal intervals lie below zero. However, Full-Residual versus the separate Direct-Utility binary policy is -.004985 [-.012034, .001125], which crosses zero. We therefore do not claim an established advantage between these two binary learners.

Results elsewhere qualify this finding. On G1 ETTm2, Full-Residual (.011864) has higher point MSE than Apply (.009884) and Adaptive (.009680); the paired differences are +.001980 [-.001253, .005259] and +.002184 [-.000489, .005067], both crossing zero. On G1 Electricity, Half (.375427) has lower point MSE than Full-Residual (.388382), but the difference +.012955 [-.004505, .032645] is also uncertain. On the PIR ETTm2 tail, Adaptive minus Apply is -.001910 [-.003788, -.000393].

We distinguish the original Weather Full-Residual-minus-Preserve result (-.02009; original block-16 CI [-.03235, -.00759]) from the newly archived bootstrap rerun, which uses seed 20271106 and yields a slightly different interval. We report them separately. The revised appendix provides all 18 additional G1 contrasts. These are marginal, unadjusted intervals, not familywise significance claims.

For official COSA/PatchTST, the frozen ETTm2 and Weather streams lack independent pre-test data for prospective shrinkage fitting; selecting a coefficient using test outcomes would be retrospective. Separately, a self-trained DLinear + native-COSA exchange-rate h96 batch-end sensitivity gives Apply .116154 and Adaptive .116412 over 704 future-supported windows, with an Adaptive-minus-Apply block-96 CI of [-.000413, .001133]. This is not an independently calibrated published-COSA checkpoint comparison.

We agree that continuous shrinkage is an important alternative and that binary execution should not be presented as universally superior without this comparison. The added experiments make the evaluation more complete, but also confirm that results depend on the setting. Our revised claim is deliberately limited: neither binary gating nor fractional application consistently dominates across the evaluated frozen systems.

Thank you again for identifying this missing comparison. It helped us make both the evidence and the scope of our conclusions more precise.

AuthorsOpenReview ↗

Response to Reviewer W9MS — Q2: Electricity AUROC and Utility Audit

Thank you for pointing out the unusually low AUROC values on Electricity. We agree that these results require careful diagnosis rather than being attributed to distribution shift without further evidence.

We re-examined the frozen Electricity settings by checking the same window and candidate assignments, stored labels, and source hashes against the exact squared-error identity U = 2A - C, where A = d^(-1) <y-y0, c> and C = d^(-1) ||c||^2. These checks did not reveal a sign error or a utility-label mismatch.

An important clarification is that the reported extremes involve different prediction targets. For TimesFM Electricity at h24/h48/h96, the Direct-Utility AUROCs are .1379/.08889/.08333, whereas the Full-Residual-induced AUROCs are .2966/.9333/.05556, respectively. For Moirai at h48, the corresponding values are .1278 (Direct Utility) and .9278 (Full Residual). Thus, the .05556 value refers to the h96 residual-derived score; it does not indicate that a single Direct-Utility score was silently sign-reversed.

We also examined the underlying candidate utility. On TimesFM h48, the beneficial-correction fraction changes from 8/12 in the available calibration suffix to 1/63 on the test windows. The test-set means are A = .014132, C = .409957, and U = -.381693. This provides evidence of a substantial calibration-to-test mismatch in candidate utility and is consistent with the poor transfer of a constant execution action.

However, we want to be careful not to overinterpret this observation. It does not establish the causal source of the extreme score-ranking AUROC. Calibration size, nonstationarity, correction magnitude, and target-dependent fitting remain potentially confounded. In particular, inverting scores using test labels would be an invalid diagnostic procedure.

We therefore retain Electricity as an anomalous and adverse setting with verified utility provenance. We have not claimed that all Electricity settings fail universally, nor that a single distribution-shift mechanism has been uniquely identified.

We appreciate this question because it prompted us to distinguish a verified label and sign audit from a causal explanation of the observed ranking behavior.

AuthorsOpenReview ↗

Response to Reviewer W9MS — Q3–Q4: Predeployment Identification and Research Contribution

Thank you for raising these two important questions. We agree that a useful execution policy should ideally identify favorable correction regimes before deployment, and that the distinction from established decision-making literature needs to be made explicit.

Q3: Can the favorable regime be identified before deployment?

We evaluated calibration-only summaries of benefit prevalence and correction scale, using leave-one-dataset-out regime prediction across 14 TimesFM settings.

The diagnostic identified favorable or unfavorable labels in only 6/14 settings and achieved worse mean regret than a constant reference.

We also evaluated a separately defined conservative action menu over 26 settings. It performed worse than Preserve in 6/26 settings, worse than Apply in 7/26 settings, and worse than the retrospectively best fixed action in 13/26 settings.

These results do not support a reliable predeployment go/no-go diagnostic.

Similarly, the concentration of PIR Weather gains in particular channels was identified using test outcomes. We therefore treat it as a retrospective sensitivity analysis rather than an inference-time decision rule.

We believe this negative finding is important: historical beneficial-correction prevalence cannot currently serve as a reliable deployment safety certificate.

Q4: What is new relative to established decision-making and forecast adjustment?

We have substantially revised the Introduction and Related Work to acknowledge the closest precedents, including Blattberg–Hoch and Fildes et al. on judgmental adjustment, Forecast Value Added, Bates–Granger forecast combination, TimeFuse, and example-dependent cost-sensitive classification (Elkan; Zadrozny et al.).

We also no longer present convex mixing, binary regret, the MSE alignment identity, or |U|-weighted classification as new mathematical contributions.

Our specific empirical focus is an already proposed correction from a frozen forecasting pipeline, together with the inference-time evidence available to decide whether it should be executed.

We evaluate three related but distinct aspects:

Identical forecast–candidate pairs under fixed, predicted, and fractional execution actions.

Prediction-target quality separately from the resulting action regret.

Harmful execution and missed benefit separately, including null and adverse regimes.

This distinction is important in the experiments. On Chronos Weather, Full-Residual performs better than both fixed actions, whereas on ETTm2 it does not improve upon calibration-selected Apply. On native PIR, improved mean MSE coexists with harmful individual revisions, without establishing a successful learned gate.

We have therefore narrowed our contribution to a candidate-conditioned evaluation and diagnosis, rather than claiming a universally effective policy, a new decision-theoretic bound, or forecasting state-of-the-art performance.

We appreciate this question because it encouraged us to state more precisely what the paper establishes and, equally importantly, what it does not establish.

Reviewer_4hNxOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper proposes a method for revising forecasts in a post-hoc forecasting system.

Critical strengths

The theoretical derivations in Section 3.2 at least seem to be correct.

Critical weaknesses

The paper does not really make a case for why we should care about the results or what those results mean for anything in the broader machine learning space. The introduction lists many "recent systems" that solve related problems, but none of these systems are compared against in the experiments. As far as I can tell there is no comparison against an existing baseline system for time-series prediction. Overall the paper is just generally incoherent, with little in the way of motivation that explains why the text is going in the direction it is going.

Figure 1 makes very little sense. The subfigures are labeled (a) (b) (c) and (d) but then these labels aren't mentioned anywhere in the text.

Questions for authors

How does this work compare empirically against other published works on the same datasets?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 4hNx — Research Motivation, Existing Systems, and Figure 1

Thank you for this helpful comment. After revisiting the original presentation, we agree that we did not explain the motivation of the problem as clearly as we should have. In particular, the transition from forecasting accuracy to the decision of whether a correction should actually be used was too brief. We have spent considerable effort revising this part of the manuscript.

The central question we want to study is relatively simple: suppose a forecasting model has already produced a forecast y0y_0, and an existing correction method proposes an alternative y0+cy_0+c. Should we always accept that correction, or are there instances where keeping the original forecast is the better choice?

This distinction is important to us. Our work does not propose another forecasting architecture or attempt to replace existing forecasting systems. Instead, we study the decision that follows an already generated correction. A corrected forecasting system may achieve better average MSE, but that alone does not tell us whether every individual correction is beneficial.

We have also made the connection to existing forecasting systems more explicit. Our primary controlled comparison uses frozen Chronos-Bolt forecasts and a specified residual candidate, with TimesFM and Moirai providing additional backbone settings. More importantly, we evaluate existing learned revision systems within their own forecasting pipelines.

For example, PIR/PatchTST reduces MSE from .2018 to .1758 on ETTm2 and from .1941 to .1679 on Weather. Yet 25.2% and 20.5% of the corresponding individual corrections are harmful. Similarly, official frozen COSA/PatchTST reduces Weather MSE from .172041 to .168322, while 39.1% of its individual corrections are harmful.

We believe these examples help illustrate the motivation more concretely: improvements in average forecasting accuracy do not necessarily mean that every proposed revision should be accepted. We also want to be precise about what these experiments establish. The PIR and COSA results are within-system Preserve-versus-Apply comparisons, not claims that our method achieves better forecasting accuracy than their published implementations under equal computational budgets.

We have also revised the explanation of Figure 1. The revised Introduction and caption now walk through the four panels: (a) the Forecast–Predict–Decide–Act–Evaluate process; (b) how similar forecast errors can lead to opposite execution decisions; (c) why accurate prediction does not necessarily imply successful execution; and (d) the Correctability Gap, representing the benefit that a policy fails to recover.

The figure artwork itself remains unchanged. We have therefore focused on making its intended message and the role of each panel clearer, rather than claiming to have redesigned the illustration.

Thank you again for directing our attention to these aspects. Your comment helped us recognize that the motivation should not be left for readers to infer from the experimental results. We hope the revised Introduction and Figure 1 now provide a more direct and coherent explanation of the problem we are trying to study.

AuthorsOpenReview ↗

Follow-up to Reviewer 4hNx

After reading your comments again, we realized that our earlier response still left an important question insufficiently answered: why should a machine learning researcher care about this problem?

We apologize for not making the motivation clear enough in the original manuscript.

Consider a forecasting system that already has an original prediction and a post-hoc correction. The correction may improve the average forecasting error, but that does not mean every individual revision is beneficial. In an operational system, the original forecast is still available, so accepting a proposed correction is itself a decision with consequences. This is the practical situation we want to understand.

We have tried to make this question concrete by evaluating existing correction systems rather than relying only on our own experimental pipeline.

For PIR/PatchTST, MSE decreases from 0.2018 to 0.1758 on ETTm2 and from 0.1941 to 0.1679 on Weather. However, 25.2% and 20.5% of the corresponding individual corrections are harmful.

The COSA results provide another example. On ETTm2, its MSE change is +0.000036, with a paired 95% confidence interval of [-0.000371, 0.000456]; 49.7% of individual corrections are harmful. On Weather, COSA reduces MSE by 0.003719, with a confidence interval of [-0.004675, -0.002738], while 39.1% of individual corrections remain harmful.

These results are central to our motivation. They show that improvements in average forecasting accuracy do not fully describe what happens when a correction is applied to individual predictions.

Our Chronos experiments further examine whether an execution policy can make this decision more effectively. On Weather, Full-Residual achieves a Selection Regret of 0.01164, compared with 0.03173 for Preserve and 0.09048 for Apply. This is a positive result that we consider meaningful. ETTm2, however, does not show the same advantage.

We also want to clarify the scope of these comparisons. PIR and COSA are evaluated within their respective forecasting pipelines, using their original correction outputs and matched evaluation windows. We are not claiming that our approach outperforms their published forecasting architectures under an equal computational budget. Our question concerns what happens after those systems have already proposed a correction.

We have revised the Introduction and Figure 1 explanation accordingly. Panel (a) describes the complete forecasting and execution workflow; panels (b) and (c) illustrate why similar forecast errors can lead to different correction decisions and why a good prediction signal does not guarantee a successful intervention. Panel (d) shows the Correctability Gap. We have added explicit references to all four panels in the manuscript. The original artwork remains unchanged.

Looking back, we should have introduced this operational question before presenting the mathematical formulation and experimental details. We appreciate your criticism because it helped us recognize that the motivation must be understandable on its own, rather than something readers are expected to reconstruct from the results.

Thank you for taking the time to point this out.

This author response does suggest some of my concerns might be addressed in a revision, but as no revisions seem to be present in the OpenReview revision history, I am unable to evaluate this. At least as described in the author response, the revision seems substantial enough that it warrants another round of peer review, which would give reviewers time to consider the new revised manuscript.

AuthorsOpenReview ↗

Reply to Reviewer 4hNx’s Follow-up on the Revised Manuscript

Thank you for telling us that you could not see the revision. You were right that our description alone was insufficient to assess the changes. The blue-marked PDF is now available in Revision History.

Your review helped us see that we had left readers to infer why the correction decision mattered. The revised Introduction and Figure 1 caption (pp. 1–2) now make that choice explicit before introducing the formalism, and the caption walks through each of the four panels.

Your question about published systems also has a more direct answer in Section 5.3 (pp. 7–8). We use the original PIR/PatchTST and COSA/PatchTST outputs and compare applying their corrections with keeping their own base forecasts on the same windows. In Table 4, COSA reduces average Weather MSE by 0.003719, while 39.1% of individual revisions increase error. We think this is a concrete reason for forecasting researchers to look at what happens when individual corrections are applied alongside the average accuracy of the system.

We appreciate that a substantive revision takes time to read. If the discussion window allows, we would be grateful if you could revisit those pages and reconsider your assessment of the paper’s clarity and empirical contribution.

Reviewer_tqe1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies when to accept a correction of an initial model forecast. The model first predicts a forecast and a separate correction system proposes a revised forecast. Before the actual outcome is known, a decision rule must choose between keeping the original forecast or accepting the revision. The paper compares several different decision rules. These rules use available information (e.g., past observations) to estimate whether the correction will reduce error. The authors evaluate those decisions afterward using selection regret: the extra error incurred compared with choosing the better forecast in hindsight. The experiments show that selective correction helps in some settings, but does not consistently outperform just accepting every correction or keeping all original forecasts.

Critical strengths
  1. Experiments cover several forecasting models and correction systems.
  2. Paper reports mixed results clearly. It shows where selective correction helps and also where simple rules perform well or better. Also acknowledges when differences are statistically uncertain.
Critical weaknesses
  1. The paper does not propose a consistently better decision method. The observation that good predictions do not always lead to good decisions is already recognized in prior work.
  2. As shown, selective correction works well in some settings, but it does not show statistically reliable improvement over the existing PIR correction system.
  3. More explanation is needed on why some decision rules fail. Poor results could be due to inaccurate benefit estimates, poorly chosen acceptance thresholds, etc. The paper investigates some of these possibilities but it does not clearly separate their effects or give a clear explanation of the observed behavior.
Questions for authors

See Weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer tqe1 — Main Contribution, PIR Results, and Failure Diagnosis

Thank you for your encouraging assessment of the multi-system experiments and, in particular, our effort to report both positive and negative results. We appreciate that you recognized this aspect of the work.

Your feedback also gave us an opportunity to reconsider how we describe the main contribution. We realized that the distinction we want to make could be stated more clearly: predicting whether a correction will be useful, generating a good correction candidate, and deciding whether to execute that candidate are three different problems. A model can perform well at the first two without necessarily solving the third.

This distinction is what motivates our use of Selection Regret. Rather than evaluating only the quality of a prediction target or the average performance of a corrected forecasting system, we evaluate the execution decision against Preserve and Apply on the same forecast–candidate pair.

The experimental results are deliberately mixed, and we believe both sides are important.

On Chronos Weather, the original Full-Residual rule achieves a regret of .01164, compared with .03173 for Preserve and .09048 for Apply. Its paired block-16 difference against Preserve is -.02009, with a 95% confidence interval of [-.03235, -.00759]. This provides a reproducible example in which a learned execution rule improves upon the fixed alternatives.

However, ETTm2 tells a different story. Full-Residual regret is .00404, compared with .00206 for calibration-selected Apply, and the paired interval crosses zero. We do not regard this as a result to be explained away. It is an important indication that the benefit of learning an execution rule depends on the underlying setting.

The native PIR experiments provide another useful perspective. Although the corrected forecasts improve mean MSE, harmful individual revisions remain. Moreover, the learned-gate improvements on held-out ETTm2 and Weather are only .00051 and .00172, respectively, with moving-block intervals that include zero.

For us, this leads to an important distinction: identifying harmful corrections after observing their outcomes is not the same as predicting, before execution, which corrections can safely be rejected. Our experiments suggest that the latter remains a difficult problem.

In the revised manuscript, we have therefore brought together the analysis of prediction targets, the MSE alignment and magnitude components, binary execution thresholds, continuous partial application, temporal sensitivity, and calibration-subset effects. These analyses help explain why a strong prediction metric alone does not determine the preferred execution action. At the same time, we do not claim that they causally explain every failure of the learned gate.

We have consequently narrowed the contribution to a candidate-conditioned evaluation of correction execution, supported by both positive and null regimes. We do not claim to have established a universally optimal policy or a deployment-safe selector.

Thank you again for your thoughtful assessment. We believe that taking the mixed results seriously, rather than presenting only the settings in which our approach works, has made the central message of the paper more precise and, ultimately, more honest.

AuthorsOpenReview ↗

Follow-up to Reviewer tqe1

After revisiting your review and our earlier response, we realized that we had focused too much on summarizing the additional experiments. We did not address your main concern as directly as we should have: what does this work actually establish, given that our decision rules do not consistently outperform simple alternatives?

We agree that this is an important limitation. We also think the mixed results deserve a closer explanation.

On Chronos Weather, the Full-Residual policy achieves a Selection Regret of 0.01164, compared with 0.03173 for Preserve and 0.09048 for Apply. The paired block-16 confidence interval against Preserve is [-0.03235, -0.00759]. We are encouraged by this result because it demonstrates that an execution policy can recover benefits that neither fixed action captures.

At the same time, ETTm2 tells a different story. Even with a cosine similarity of 0.821 and a utility-sign AUROC of 0.888, the induced policy has a regret of 0.00404, compared with 0.00206 for calibration-selected Apply. The difference is not statistically established. On PIR, the learned gate also fails to demonstrate a reliable improvement over native execution: the paired confidence intervals for ETTm2 and Weather both include zero.

We took your question about failure mechanisms seriously. In the revised analysis, we examined correction direction and magnitude separately. For harmful generic-residual revisions, 61.2% of Weather cases have the wrong direction, while the remaining 38.8% are positively aligned but excessively large. ETTm2 shows a different balance, with 46.2% wrong-direction and 53.8% over-scaled corrections. We also examined threshold calibration. On Weather, calibration substantially reduces the regret of the direct-utility score, while it does not improve the residual-based score. These findings help distinguish candidate errors from score and threshold errors, although we acknowledge that they do not establish a complete causal explanation of policy failure.

We have also reconsidered how we describe the contribution. The distinction between prediction quality and decision quality is not new by itself, and we do not claim otherwise. What we aim to contribute is a controlled evaluation of the execution decision for an already generated correction, using matched forecast-candidate pairs across several forecasting and correction systems. The positive, null, and adverse results are all part of that evaluation.

We understand why the lack of a consistently superior policy raises questions about the paper's contribution. We hope the revised manuscript makes clearer what the experiments establish, where the approach works, and where the evidence does not support a stronger claim.

Thank you for pushing us to examine these questions more carefully. Your comments helped us make the scope of the paper more precise.

Thank you for your response. Unfortunately, no learned rule reliably beats simple baselines and the narrowed contribution (candidate-conditioned evaluation using selection regret) seems a bit narrow to me, I retain my score.

AuthorsOpenReview ↗

Reply to Reviewer tqe1’s Follow-up on the Empirical Contribution

Thank you for being clear about your remaining concern. We agree that the experiments do not establish a learned rule that reliably beats simple baselines across settings. We still think the matched comparisons establish something useful about when learning the decision pays off.

On Chronos Weather, Full-Residual lowers MSE relative to Half by 0.022265, with a paired block-16 interval for the difference of [−0.028086, −0.017046]. The intervals against Global and Adaptive shrinkage also favor Full-Residual (Tables 2 and 21). On ETTm2, the comparison with Apply remains inconclusive, as do the learned-gate improvements over native PIR.

Your question about why the rules fail also led us to spell out the threshold check in Section 5.2 (p. 7). Holding the Weather Direct-Utility score fixed and calibrating its threshold lowers regret from 0.07376 to 0.01786. With the residual score, regret moves from 0.01687 to 0.01755, and the paired interval includes zero. This identifies a case where a better threshold makes a substantial difference. The remaining contribution of score estimation error is still open.

We know your concern is about the scope of the contribution as a whole. We hope the revised paper makes a stronger case that these comparisons are informative even when a fixed action is hard to beat. We would be grateful if you would reconsider your recommendation in light of the revised analysis.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Revisions

We sincerely thank all reviewers for their thoughtful and constructive feedback. Their comments helped us identify several places where the original manuscript needed clearer positioning, stronger empirical comparisons, and a more careful discussion of its limitations.

In response, we have revised the manuscript and conducted additional analyses. The main changes are summarized below.

  1. Clarifying the research question and contribution. We have revised the Abstract, Introduction, and Related Work to better distinguish candidate-conditioned correction execution from existing forecast adjustment, forecast combination, and cost-sensitive decision-making. Our focus is whether an already proposed correction should be executed for a particular instance, rather than developing another forecasting model.
  2. Strengthening the execution comparisons. We added matched comparisons among Preserve, Apply, Half, Global shrinkage, Adaptive shrinkage, and the original binary execution policies. We also restored and independently verified the original Full-Residual decisions using frozen per-window records. Additional paired uncertainty estimates are reported in the revised appendix. Importantly, the Weather results support improvements over several fractional alternatives, but do not establish an advantage over the separate Direct-Utility binary policy.
  3. Expanding sensitivity analyses. We added MAE and pinball-loss comparisons, calibration-window sensitivity analyses, and a synthetic execution-cost experiment. These experiments help clarify the dependence of execution decisions on the loss function, calibration data, and assumed action cost. We explicitly distinguish these sensitivities from retrained non-MSE policies or measured economic benefits.
  4. Improving presentation and discussing limitations. We revised the explanation of Figure 1(a)–(d), clarified the differences between experimental configurations, and expanded the discussion of adverse and inconclusive results. In particular, the additional experiments do not support a reliable predeployment go/no-go diagnostic. We have also made the scope of our claims more precise. We do not claim that binary gating or fractional correction is universally superior, nor do we propose a new decision-theoretic guarantee. Rather, we provide a controlled, candidate-conditioned evaluation of when an existing forecast correction is beneficial to execute under a specified loss.

We appreciate the reviewers' suggestions, which have helped us make the manuscript more rigorous, transparent, and appropriately scoped.

AuthorsOpenReview ↗

Revised Manuscript and Key Findings

Dear Reviewers and Area Chair,

Thank you for taking the time to discuss the paper with us. The revised manuscript is now available in Revision History, with the main changes marked in blue. We appreciated being asked to explain both why this decision matters and what our experiments actually establish.

The Introduction and Figure 1 (pp. 1–2) now begin with the choice a forecasting system faces when both its original forecast and a proposed correction are available. Section 5.3 (pp. 7–8) makes that choice concrete using the original outputs of PIR and COSA. On Weather, for example, COSA lowers average MSE while increasing error on 39.1% of individual windows. That is why we think correction execution deserves to be evaluated in its own right.

The matched comparisons in Tables 2 and 21 (pp. 5 and 25) show that selective execution can add value. On Chronos Weather, Full-Residual improves on Half, Global, and Adaptive shrinkage, with paired intervals favoring Full-Residual in each comparison. We also report where the case for a learned rule remains inconclusive, including the ETTm2 comparison with Apply and the learned gates on native PIR. The revised Section 5.2 (p. 7) shows how much threshold calibration helps one Weather score and how little it helps the other.

We believe these comparisons make the paper useful to researchers evaluating forecast corrections. They show when a learned rule earns its place against simple alternatives and where the evidence falls short. We would appreciate your considering the revised manuscript and these comparisons in your final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.