- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies a decision that arises in post-hoc forecasting systems. A baseline forecast y₀ and a candidate revision c (from a residual corrector, an output-space adapter, or a learned reviser) are both available. The system must decide, before the target is observed, whether to keep y₀ or output y₀ + c. The authors call this binary choice forecast override. They argue it should be evaluated separately from how well the candidate is generated or predicted.
The formal contributions are:
- When applying helps. Under squared error, the relative gain from applying the candidate is Γ = 2ρr − r². Here ρ is the cosine alignment between the candidate and the baseline residual, and r is the candidate's size relative to that residual. So applying helps only when 2ρ > r. The authors credit this alignment–magnitude trade-off to δ-Adapter and use it per instance.
- Selection Regret. This is the excess loss over the better of the two actions in hindsight. It splits into missed opportunity and harmful exposure. The authors prove that for a policy that thresholds predicted utility, the error in predicted utility bounds the regret.
- Two sources of lost value. At the population level, the total lost value splits into a Bayes evidence gap and a policy gap. The authors state explicitly that their experiments cannot separate the two.
Empirically, the paper evaluates preserve/apply policies on ETTm2, Weather and Electricity.
- Primary grid. It uses Chronos-Bolt tiny with a generic residual candidate and four matched prediction targets: full residual, direct utility, alignment and projection.
- Robustness checks. These swap the backbone (TimesFM 2.5 over 14 dataset–horizon settings, and Moirai 2.0 on three settings) and the candidate family (residual, polynomial, rank-1). They also use learned PIR revisions and official COSA outputs.
The main findings:
- Base-forecast error alone does not determine the right action: windows with nearly equal error can favor opposite actions.
- Both wrong-direction and oversized candidates occur in practice.
- Strong predictive scores (cosine, AUROC, R²) do not consistently translate into lower Selection Regret than the best constant action. The prediction-based policies clearly win on Weather (residual policy regret 0.0116 vs. 0.0317 for always preserving and 0.0905 for always applying). They are not significantly better on ETTm2 and are unstable on Electricity.
- PIR and COSA improve average MSE while 20–50% of their individual revisions are harmful. Learned gates do not reliably beat always applying PIR's revisions.
The authors conclude that a proposed revision should be evaluated as a decision tied to that specific candidate, rather than through prediction quality or average correction gain.
Critical strengths
-
A real and under-examined decision point in post-hoc forecasting. The paper separates two questions that recent post-hoc work (PIR, COSA, δ-Adapter, CTRL) treats as one: how to generate a revision, and whether to execute it. Keeping y₀ as an explicit default action, and judging each concrete candidate against it, is a clean framing that practitioners can use. It applies to any source of corrections, whether learned correctors, reconciliation, or externally supplied adjustments.
-
A decision-relevant evaluation metric. Selection Regret, split into missed opportunity and harmful exposure (Eq. 6, Table 10), explains why policies with similar correction rates differ in value. For example, Weather is dominated by harmful exposure while ETTm2 is dominated by missed benefit. Aggregate MSE, AUROC, or cosine similarity cannot show this, and the decomposition is more useful than a single "regret closed" number.
-
The math is correct and connects to the decision. I checked the derivations; they are correct. The direction–scale form (Eq. 4) turns the known alignment–magnitude trade-off into a per-instance rule for when applying helps. Table 19 confirms that both failure routes, wrong-direction and overscaled candidates, occur in practice, so both terms in the rule matter empirically. Proposition S1 explains why utility MAE, unlike ranking metrics, bounds decision regret.
-
Careful experimental protocol. Calibration and test windows are strictly separated in time, correction-utility labels are fit out-of-fold with purging, and test futures are never used for any selection step. Paired circular moving-block bootstrap intervals (block lengths up to 16) and five-seed reruns are reported. This is more rigorous than is typical in time-series forecasting papers and makes the positive Weather result credible.
-
External validation on a published correction system. Evaluating unmodified official COSA outputs and exported PIR revisions shows that the effect is not an artifact of the authors' own candidate construction. Both systems improve average MSE while 20–50% of their individual revisions are harmful, which concretely supports the paper's central claim.
-
Honest scoping of claims. The authors credit the alignment–magnitude identity to prior work. They state that the Bayes evidence gap is not estimated, so no information-theoretic limit is claimed, and they report null and unfavorable settings alongside the positive one. This candor is appreciated, though it also exposes how limited the positive evidence is (see Weaknesses).
Critical weaknesses
-
The theory adds little that is new. The authors credit the main identity (Eq. 3–4) to δ-Adapter, and it is a few lines of algebra. Selection Regret is ordinary regret for a binary decision. Proposition S1 (regret ≤ |U − Û|) is the standard bound for a classifier that thresholds a predicted value. Proposition S2 is Jensen's inequality, i.e. the value of extra information. Each result is correct, but none goes beyond textbook results, and together they do not make a theoretical contribution of ICLR depth. The paper's main claim, that prediction quality does not imply decision quality, is already well established in decision-focused learning, which the paper itself cites.
-
The closest prior work is missing. The preserve-or-apply decision has long been studied for judgmental forecast adjustments:
- Blattberg & Hoch (1990) combine the model and the manager 50/50, which amounts to a fixed partial application of the correction.
- Fildes et al. (2009, IJF) characterize which supply-chain overrides help or hurt by their sign and size, which is essentially an empirical preserve/apply rule.
- Forecast Value Added (FVA) analysis evaluates exactly whether an override improves on the baseline.
Two further connections are also missing. The |U|-weighted utility-sign classifier is textbook example-dependent cost-sensitive learning (Elkan, 2001; Zadrozny et al., 2003). Preserve/apply is also a two-model forecast combination restricted to weights 0 or 1 (Bates & Granger, 1969). Without positioning against these, the paper's novelty is overstated.
-
No method that works reliably, and no explanation of when gating helps. The empirical message is that results depend on the setting. Prediction-based policies clearly win only on Weather. On ETTm2 they do not beat always applying in any of 5 seeds, and Electricity is unstable. On PIR, learned gates do not reliably beat always applying the revision. The paper does not identify which properties of a dataset, backbone, or candidate predict a favorable regime. As a result, practitioners cannot act on the findings, and the work reads as a collection of case studies rather than a general result.
-
Partial application is not considered. The paper's own algebra gives the best fraction of the candidate to apply per instance: α* = ⟨e, c⟩/‖c‖² = ρ/r. Applying y₀ + α·c with α between 0 and 1 is always at least as good as choosing between 0 and 1 only. The randomized version in Appendix A.4 is a different object. Continuous shrinkage is the obvious baseline that real systems would use. Its absence leaves open whether the reported failures of binary gating simply come from restricting the action to all-or-nothing.
-
The evidence base is small.
- The primary grid uses 64 test windows per setting, a single target channel per dataset, and calibration sets of 128–512 windows. The backbone is the smallest Chronos-Bolt model.
- Many of the key intervals cross zero: ETTm2, Electricity, most TimesFM settings, and the PIR gates.
- The abstract leads with an ETTm2 comparison whose paired interval includes zero.
- Moirai covers only 3 settings and COSA only 2, yet the conclusion speaks of results across "three TSFM families."
Multivariate forecasting, longer horizons, and larger-scale benchmarks are absent.
- Some results look anomalous and are unexplained.
- Several utility-sign AUROCs are far below 0.5: TimesFM Electricity 0.083–0.138 (Table 11), Moirai Electricity 0.128 (Table 12), and a residual AUROC of 0.056 at Electricity/h=96.
- In these same settings the utility policy is catastrophically worse than the constant action (e.g. 0.284 vs. 0.0049 at Electricity/h=48).
An AUROC near 0.1 means the score ranks almost perfectly in the inverted direction. A sign-flipped score would reach AUROC ≈ 0.9. This points to a systematic shift between calibration and test, or a pipeline error. Either way it needs diagnosis, because these settings drive the "strongly negative" Electricity conclusions in Section 5.3 and Table 13.
-
The analysis only applies to squared error. The direction–scale geometry, and therefore the mechanism analysis in Table 2 and Table 19, holds only for MSE. Many forecasting applications are evaluated with quantile/pinball loss, MAE, or WAPE. The paper does not discuss how the decision boundary changes under these losses. "Decision quality" is also still measured by forecast loss rather than by a downstream decision cost. The execution-cost term κ (Appendix A.4) is introduced but never used in the experiments.
-
The presentation makes results hard to verify.
- The "generic-residual candidate," which drives most experiments, is never defined in the main text.
- Seven experimental grids (Table 4) produce similarly named numbers that cannot be compared across grids. For example, the Weather/Chronos residual-candidate advantage is 0.0333 in Table 13 but 0.0739 implied by Table 1. Readers must constantly track which grid a number comes from.
- Table numbering skips (no Tables 5, 15, or 18), which suggests incomplete editing.
- The prose is heavily hedged. Many sentences qualify their claims so carefully that it is hard to extract a clear takeaway.
Questions for authors
I am open to raising my score if the authors adequately address the questions below. Questions 1–4 matter most to my assessment.
-
How does continuous shrinkage, y₀ + α̂(Z)·c with a learned α̂ ∈ [0, 1], compare with binary gating and with the two constant policies on the primary grid, PIR, and COSA? A simple calibration-fitted global α would also be useful as a baseline. If shrinkage consistently beats the constant actions, that would substantially strengthen the practical contribution. If it does not, that would strengthen the claim that the execution problem is genuinely hard.
-
Please explain the utility-sign AUROCs of 0.056–0.138 on Electricity (Tables 11–12). Is this caused by a shift between calibration and test (e.g., in the sign of the residual), by a sign or label convention error, or by something else? Do the Electricity conclusions in Section 5.3 and Table 13 still hold once this is resolved?
-
Can the authors identify properties, measurable before deployment, that predict whether a setting is favorable (Weather-like) or unfavorable (ETTm2-like)? Candidates include the base rate of beneficial revisions, the distribution of ρ and r on calibration data, or how stable the residual sign is over time. Even a simple diagnostic validated across the TimesFM grid would turn the findings into actionable guidance.
-
How does forecast override differ, conceptually or technically, from:
- (a) the judgmental-adjustment literature (Blattberg & Hoch, 1990; Fildes et al., 2009) and Forecast Value Added analysis;
- (b) example-dependent cost-sensitive classification, which the |U|-weighted classifier instantiates;
- (c) two-model forecast combination with weights restricted to {0, 1}?
What is new beyond applying an existing decision lens to ML-based post-hoc correctors?
-
Please define the generic-residual, polynomial, and rank-1 candidates precisely in the main text. What information is each fitted on, and how is c constructed at inference time?
-
How does the decision boundary look under pinball/quantile loss or MAE? Is there an analogue of the 2ρ > r condition? Do the main empirical conclusions (Table 1, Table 3) still hold when utility is measured under these losses?
-
Can the primary target comparison be extended beyond 64 test windows and a single channel? For example: all channels of each dataset, rolling-origin evaluation over several test periods, or the larger Chronos-Bolt models. As it stands, only Weather produces intervals that exclude zero. Would the ETTm2 and Electricity conclusions survive a larger test set?
-
Calibration chooses always-preserve (regret 0.04316), yet always-apply has lower held-out regret (0.03546). Is this caused by the small calibration set (N_cal = 128) or by a shift over time? How sensitive are the constant-action reference choices to calibration size in general?
-
Could the authors provide a single table that reproduces the Table 1 Weather/Chronos configuration inside the candidate-family grid (G5)? This would explain why the residual-candidate advantage is 0.0333 in Table 13 but about 0.074 in Table 1, and show which factor causes the difference.
-
Have the authors evaluated whether policy rankings by Selection Regret agree with rankings by a downstream decision cost, such as a newsvendor or inventory cost, possibly using the execution-cost term κ from Appendix A.4? This would support the claim that the paper evaluates decision quality rather than a re-weighted forecast loss.
7 comments
Response to Reviewer W9MS — Q2: Electricity AUROC and Utility Audit
Thank you for pointing out the unusually low AUROC values on Electricity. We agree that these results require careful diagnosis rather than being attributed to distribution shift without further evidence.
We re-examined the frozen Electricity settings by checking the same window and candidate assignments, stored labels, and source hashes against the exact squared-error identity U = 2A - C, where A = d^(-1) <y-y0, c> and C = d^(-1) ||c||^2. These checks did not reveal a sign error or a utility-label mismatch.
An important clarification is that the reported extremes involve different prediction targets. For TimesFM Electricity at h24/h48/h96, the Direct-Utility AUROCs are .1379/.08889/.08333, whereas the Full-Residual-induced AUROCs are .2966/.9333/.05556, respectively. For Moirai at h48, the corresponding values are .1278 (Direct Utility) and .9278 (Full Residual). Thus, the .05556 value refers to the h96 residual-derived score; it does not indicate that a single Direct-Utility score was silently sign-reversed.
We also examined the underlying candidate utility. On TimesFM h48, the beneficial-correction fraction changes from 8/12 in the available calibration suffix to 1/63 on the test windows. The test-set means are A = .014132, C = .409957, and U = -.381693. This provides evidence of a substantial calibration-to-test mismatch in candidate utility and is consistent with the poor transfer of a constant execution action.
However, we want to be careful not to overinterpret this observation. It does not establish the causal source of the extreme score-ranking AUROC. Calibration size, nonstationarity, correction magnitude, and target-dependent fitting remain potentially confounded. In particular, inverting scores using test labels would be an invalid diagnostic procedure.
We therefore retain Electricity as an anomalous and adverse setting with verified utility provenance. We have not claimed that all Electricity settings fail universally, nor that a single distribution-shift mechanism has been uniquely identified.
We appreciate this question because it prompted us to distinguish a verified label and sign audit from a causal explanation of the observed ranking behavior.
Response to Reviewer W9MS — Q3–Q4: Predeployment Identification and Research Contribution
Thank you for raising these two important questions. We agree that a useful execution policy should ideally identify favorable correction regimes before deployment, and that the distinction from established decision-making literature needs to be made explicit.
Q3: Can the favorable regime be identified before deployment?
We evaluated calibration-only summaries of benefit prevalence and correction scale, using leave-one-dataset-out regime prediction across 14 TimesFM settings.
The diagnostic identified favorable or unfavorable labels in only 6/14 settings and achieved worse mean regret than a constant reference.
We also evaluated a separately defined conservative action menu over 26 settings. It performed worse than Preserve in 6/26 settings, worse than Apply in 7/26 settings, and worse than the retrospectively best fixed action in 13/26 settings.
These results do not support a reliable predeployment go/no-go diagnostic.
Similarly, the concentration of PIR Weather gains in particular channels was identified using test outcomes. We therefore treat it as a retrospective sensitivity analysis rather than an inference-time decision rule.
We believe this negative finding is important: historical beneficial-correction prevalence cannot currently serve as a reliable deployment safety certificate.
Q4: What is new relative to established decision-making and forecast adjustment?
We have substantially revised the Introduction and Related Work to acknowledge the closest precedents, including Blattberg–Hoch and Fildes et al. on judgmental adjustment, Forecast Value Added, Bates–Granger forecast combination, TimeFuse, and example-dependent cost-sensitive classification (Elkan; Zadrozny et al.).
We also no longer present convex mixing, binary regret, the MSE alignment identity, or |U|-weighted classification as new mathematical contributions.
Our specific empirical focus is an already proposed correction from a frozen forecasting pipeline, together with the inference-time evidence available to decide whether it should be executed.
We evaluate three related but distinct aspects:
Identical forecast–candidate pairs under fixed, predicted, and fractional execution actions.
Prediction-target quality separately from the resulting action regret.
Harmful execution and missed benefit separately, including null and adverse regimes.
This distinction is important in the experiments. On Chronos Weather, Full-Residual performs better than both fixed actions, whereas on ETTm2 it does not improve upon calibration-selected Apply. On native PIR, improved mean MSE coexists with harmful individual revisions, without establishing a successful learned gate.
We have therefore narrowed our contribution to a candidate-conditioned evaluation and diagnosis, rather than claiming a universally effective policy, a new decision-theoretic bound, or forecasting state-of-the-art performance.
We appreciate this question because it encouraged us to state more precisely what the paper establishes and, equally importantly, what it does not establish.
Response to Reviewer W9MS — Q1: Continuous Shrinkage and Binary Actions
Thank you for raising this important point. We agree that partial application is an essential comparator: comparing only Preserve and Apply is not sufficient when a proposed correction can be applied at an intermediate strength. Your suggestion helped us identify a gap in our original evaluation.
We therefore added Preserve (alpha=0), Apply (alpha=1), Half (alpha=0.5), calibration-fitted Global shrinkage, and evidence-adaptive shrinkage. We also revisited the original G1 binary results and directly replayed the Full-Residual policy from frozen per-window actions, checking them against stored baseline losses, utilities, and policy losses. No model was refitted, and no test labels were used to select an execution rule.
The resulting held-out MSE comparisons are:
All values are held-out MSE, not binary Selection Regret. Source binary denotes the original Direct-Utility MLP in Chronos G1 and utility-weighted gate in PIR. Residual binary denotes the distinct Full-Residual MLP in primary G1; no corresponding PIR policy exists. G1 and PIR use different backbones, candidates, and windows, so scores should not be compared across rows.
The paired analysis clarifies where the evidence is stronger and where it remains uncertain. On G1 Weather, Full-Residual achieves .036020 MSE versus .058285 (Half), .091446 (Global), and .098671 (Adaptive). Full-Residual-minus-reference differences with 95% block-16 CIs are: Half, -.022265 [-.028086, -.017046]; Global, -.055426 [-.070197, -.039678]; and Adaptive, -.062651 [-.084184, -.040993]. All three marginal intervals lie below zero. However, Full-Residual versus the separate Direct-Utility binary policy is -.004985 [-.012034, .001125], which crosses zero. We therefore do not claim an established advantage between these two binary learners.
Results elsewhere qualify this finding. On G1 ETTm2, Full-Residual (.011864) has higher point MSE than Apply (.009884) and Adaptive (.009680); the paired differences are +.001980 [-.001253, .005259] and +.002184 [-.000489, .005067], both crossing zero. On G1 Electricity, Half (.375427) has lower point MSE than Full-Residual (.388382), but the difference +.012955 [-.004505, .032645] is also uncertain. On the PIR ETTm2 tail, Adaptive minus Apply is -.001910 [-.003788, -.000393].
We distinguish the original Weather Full-Residual-minus-Preserve result (-.02009; original block-16 CI [-.03235, -.00759]) from the newly archived bootstrap rerun, which uses seed 20271106 and yields a slightly different interval. We report them separately. The revised appendix provides all 18 additional G1 contrasts. These are marginal, unadjusted intervals, not familywise significance claims.
For official COSA/PatchTST, the frozen ETTm2 and Weather streams lack independent pre-test data for prospective shrinkage fitting; selecting a coefficient using test outcomes would be retrospective. Separately, a self-trained DLinear + native-COSA exchange-rate h96 batch-end sensitivity gives Apply .116154 and Adaptive .116412 over 704 future-supported windows, with an Adaptive-minus-Apply block-96 CI of [-.000413, .001133]. This is not an independently calibrated published-COSA checkpoint comparison.
We agree that continuous shrinkage is an important alternative and that binary execution should not be presented as universally superior without this comparison. The added experiments make the evaluation more complete, but also confirm that results depend on the setting. Our revised claim is deliberately limited: neither binary gating nor fractional application consistently dominates across the evaluated frozen systems.
Thank you again for identifying this missing comparison. It helped us make both the evidence and the scope of our conclusions more precise.