acceptodds
Under review as a conference paper at ICLR 2027

Learning Predictive Uncertainty for Frozen Time-Series Forecasters

Abstract

Combining time-series point forecasts does not by itself provide reliable uncertainty estimates for the resulting ensemble. Error prediction has its own learning targets and may require different expert weights from point forecasting. We propose GETUP (Gated Ensemble for Time-Series Uncertainty Prediction), a simple approach that learns these weights separately. Its zero-shot implementation uses two independent gates based on the input spectrum: one combines point forecasts, while the other combines error estimates learned after freezing the point branch. Separating error magnitude from residual shape supports both Gaussian and non-Gaussian predictive distributions. We evaluate GETUP on the TFB and expanded ProbTS benchmark grids, each covering 36 dataset–horizon combinations, alongside in-domain experiments on six datasets and four forecast horizons. Benchmark comparisons show improvements over established forecasting baselines, while controlled experiments examine the contributions of predictor-specific error learning, input-dependent scales, and residual shape. The construction supports zero-shot forecasting, uncertainty estimation for frozen pretrained models, and training on target history. Across these settings, explicitly learning forecast errors provides a simple way to improve predictive uncertainty while preserving the point forecasts.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
probabilistic time series forecasting, uncertainty quantification, calibration, predictive distributions, ensemble methods, zero-shot forecasting, evaluation of probabilistic forecasts, proper scoring rules

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LJueOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GETUP, a method for estimating predictive uncertainty in time-series forecasting while keeping the point forecaster fixed. Its ensemble version uses two independent gates based on the input’s frequency content: one combines point forecasts, while the other combines predictions of forecast-error magnitude. These error estimates provide an input-dependent scale that can be combined with Gaussian or non-Gaussian residual distributions to produce probabilistic forecasts.

Critical strengths

The paper has a useful practical goal: adding predictive uncertainty to a frozen forecaster while preserving its point predictions. There is a broad evaluation and the controlled result showing a 3.74% pooled CRPS reduction from input-dependent scales at fixed point forecasts and residual shape.

Critical weaknesses

Although the paper evaluates calibration, substantial undercoverage remains in the zero-shot experiments. A comparison with a suitable post-hoc recalibration baseline would help establish whether the proposed uncertainty model improves calibration beyond simpler corrections. In Appendix C.2, the authors replace the uncertainty gate’s weights with those of the point-prediction gate while keeping the trained experts fixed, supporting the usefulness of different weights for the two tasks. However, this substitution disadvantages the shared-gate configuration: the point gate was trained for forecasting, and the error experts were trained with a different gate. A fairer comparison would train a shared gate for both tasks from scratch, using the same experts, data, and comparable tuning. The experiments show that the trained uncertainty branch benefits from its own gate; it does not show that a properly trained joint gate would be worse. The paper compares GETUP with several forecasting baselines, but those comparisons also change architecture, training, and uncertainty estimation. They therefore cannot establish that the gains come specifically from using a mixture of experts. Section 4.1 explicitly acknowledges missing single-expert controls for both the point and uncertainty ensembles. To establish this claim, the authors would need to compare GETUP with a single point expert and a single error predictor under matched training and evaluation conditions. Thus, the paper shows that its complete ensemble system is competitive, but does not demonstrate that multiple experts are necessary for its improvements on these datasets. My main concern is that predicting a frozen model’s errors, separating location from scale, and combining ensemble uncertainty are established ideas, as the paper itself explains. The clearest new element is separate gates for point forecasting and uncertainty estimation. This is a useful architectural extension rather than a sufficiently substantial ICLR contribution.

Questions for authors

see weakness comments

5 comments
AuthorsOpenReview ↗

Response to Reviewer LJue (1/5): summary of the revision and where each point is answered

Read together, the three reviews showed us that the submitted text did not convey clearly what the paper contributes or how its results fit together. We have therefore rewritten the main text around the two problems the paper addresses: obtaining calibrated predictive distributions from existing point forecasters, in-domain and zero-shot, and evaluating them by more than CRPS (Sec. 1–3); the construction, an evaluation protocol and the experiments follow (Sec. 4–5). App. I lists the changes from the submitted version, whose PDF remains in the OpenReview revision history. The submitted models carry the suffix -old and keep their numbers except for the corrections listed in App. I; plain names denote the current versions (Sec. 5). We use the same names below. The experiments requested in the reviews are added, mostly in Sec. 5.3.

Re-reading the paper is not necessary to see our response: below we answer each point directly, give the exact location where the paper addresses it, and report every experiment the review asks for. Section, table and figure numbers refer to the revised PDF.

We thank the reviewer for the careful reading. The review raises one main concern and asks for three experiments; we list each and where the revision answers it.

The review asks forRevised PDF
A contribution beyond separate gates (main concern)Sec. 1, 3–4, App. E
Complete systems kept apart from controlled comparisonsSec. 5, Tables 1–4
A post-hoc recalibration baseline for the zero-shot undercoverageSec. 4, 5.3, Table 2; App. B, H
A trained shared gate with the same experts and data, in place of the serving-time substitutionSec. 5.3, Table 2; App. H, Table 37
A single point expert and a single error predictor under matched conditionsSec. 5.3, Tables 2–3; App. H
AuthorsOpenReview ↗

Response to Reviewer LJue (2/5): main concern, contribution beyond separate gates

Main concern: what the paper contributes beyond separate gates (Sec. 1, 3–4, App. E)

We clarify the contribution as the revised introduction now states it (Sec. 1). It is organised around two problems (Sec. 3):

  • The engineering problem (Sec. 3.1). Point forecasters improve every few months, while each probabilistic forecaster is a separate generative design that every advance on the point side restarts, and every foundation model ships its own sampler. GETUP is the core contribution: one construction for both regimes (Sec. 4). Point forecasts are frozen, a second model learns the magnitude of their own errors, and a separately fitted residual law gives the shape. In-domain, GETUP-ID is a soup that absorbs a stronger point model by adding it as a member; zero-shot, GETUP-ZS is a cap, a gated bank of linear frequency experts trained once on a source corpus, that sits on any frozen point predictor, a foundation model included (GETUP-Point, GETUP-ZS+Aurora). This is the practical goal the review names as the paper's strength: uncertainty added to a frozen forecaster with its point predictions preserved. The separate spectral gate is a component of the zero-shot cap, needed for the bank to serve as one (see the gate controls in Part 5), not the contribution itself; the in-domain soup has no spectral gate at all, its error heads are combined by a ridge stacker.
  • The evaluation problem (Sec. 3.2). CRPS alone does not see whether a predictive law can be trusted, and parameters do not measure cost. The protocol (Sec. 4.1) reports CRPS, NLL, coverage and ENCE together and counts inference cost as operations per forecast window.
  • The evidence (Sec. 5): rank, CRPS and NMAE in both regimes with the cost of each system, an account of every design decision (Tables 2–3), and the finding that strong samplers lose to a trivial control built from their own point forecast: the control has lower NLL in 102 of 104 settings, while GETUP-ID-old and GETUP-ID beat their own controls on CRPS and NLL in all but 7 and 9 of 126 comparisons (Sec. 5.4, Table 4). We think this needs to be stated clearly to a community that ranks by CRPS alone.

The device is old; the forecaster is new. As the review notes, predicting a frozen model's errors, separating location from scale and combining ensemble uncertainty are established ideas, and the paper discusses them (Sec. 1, App. E): ARCH-type recursions make the magnitude of the next error predictable (Engle, 1982; Bollerslev, 1986), Glejser (1969) regressed the absolute residuals of a fixed least-squares fit on its inputs, and Lowe & Zapart (1999) trained a second network on the squared residuals of a fixed one; DEUP and PDHR are recent instances. The status of the device is not what the paper is about. GETUP replaces the fixed recursion by a learned map of the whole history (Sec. 2), and the paper claims a class of probabilistic forecasters built on it that is strong on all four scores at once, at a fraction of the cost of sampling, and that improves whenever its point forecasters do (Sec. 1); to our knowledge, none had been built that competes with the leading generative and foundation models on the standard long-horizon benchmarks. Zero-shot, GETUP variants hold the six best mean CRPS ranks on the ProbTS grid, the bank alone at 1.14 GFLOP per window against 823 for Aurora; in-domain, GETUP-ID has the lowest mean CRPS (0.149 against 0.150 for VolDy-VAE) and the best mean CRPS and NMAE ranks (2.32 and 2.34 against 2.52 and 2.84; Table 1).

AuthorsOpenReview ↗

Response to Reviewer LJue (3/5): main concern (cont.); complete systems and controlled comparisons

A non-obvious use of a frequency-expert bank. Super-Linear is a point forecaster. Turning its bank into the zero-shot cap, by giving every frequency a point expert and an error expert and routing the error bank by its own gate (Sec. 4; App. A.6), and then replacing the point side altogether by the output of another model while keeping the error machinery (GETUP-ZS+Aurora, GETUP-Point), is a construction of this paper. The same separation makes the point side replaceable in both regimes: zero-shot, the point branch is exchanged for a foundation model; in-domain, the soup absorbs a more accurate model as a member and the error branch is refitted by the unchanged procedure (with four OLinear members, CRPS falls by 1.38% per cell on average, in 57 of 72 cells; Table 3). The soup's error branch itself, value–error pairs stacked by ridge regression onto the error of the served forecast with a per-channel validation anchor, and the residual shape fitted separately from the scale (Sec. 4, App. A.5), are also introduced here; the submitted text presented them as implementation details.

On the size of the contribution. The review reads the work as a useful architectural extension. We see its strength precisely in its simplicity: an approach fundamentally different from the mainstream of heavy sampling-based models, applicable to any point forecaster in both regimes, ahead of the leading models zero-shot and, in-domain, on all four scores under each model's own predictive law, at a fraction of their operations. It is also at an early stage with a high capacity for adaptation: during this discussion alone, small changes improved it further (a scale factor fitted on the source corpus at least halves the median NLL and ENCE of the zero-shot bank, Table 2; OLinear members lower in-domain CRPS by 1.4% per cell, Table 3). We would therefore not measure its novelty by the sophistication of its parts. If the number of variants obscures the main line, we are glad to move secondary variants to the appendix.

Complete systems and controlled comparisons (Sec. 5)

As the review notes, the benchmark comparisons change architecture, training and uncertainty estimation at once. Sec. 5 now keeps the kinds of evidence apart. Sec. 5.1–5.2 compare complete systems, each with its cost. Zero-shot, on the 24 complete cells GETUP-ZS+Aurora, Aurora's deterministic forecast with the bank's error branch and the source-corpus factor, has the lowest mean CRPS, 0.155, and the best mean CRPS rank, 2.33, against 0.173 and 7.96 for Aurora; GETUP-ZS-old (0.156, rank 2.88) and GETUP-ZS (0.157, rank 2.96) follow. The bank's NMAE is level with Aurora's (0.193 against 0.192), and its MSE on the TFB grid is 2.07% higher than published Aurora's (Tables 1, 11). On the 28 complete in-domain cells of the ProbTS grid, GETUP-ID, measured on three seeds, has the lowest mean CRPS (0.149 against 0.150 for VolDy-VAE and 0.158 for GETUP-ID-old) and the best mean CRPS and NMAE ranks (2.32 and 2.34 against 2.52 and 2.84 for VolDy-VAE); VolDy-VAE keeps the lowest mean NMAE (0.188 against 0.189) and leads on Illness, which the harness scores on two test windows per horizon; without Illness GETUP-ID is further ahead in mean CRPS (0.1487 against 0.1568) and rank (1.58 against 2.77) on the remaining 24 cells (Table 1, Sec. 5.1). Against GETUP-ID-old, GETUP-ID lowers ProbTS CRPS from 0.153 to 0.149 on the four ETT sets, Exchange and Weather (lower in 20 of 24 cells) and from 0.171–0.202 to 0.131–0.166 on Illness (Table 20, App. G); on Electricity and Traffic its runs with OLinear's published settings are queued or in progress. Sec. 5.3 changes one component at a time and keeps everything else, the point forecast included, fixed (Tables 2–3); the three experiments the review asks for are there, and we report them below. Sec. 5.4 scores each model against its own trivial control on identical windows (Table 4).

Reviewer_C7eAOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that combining time-series point forecasts does not by itself provide reliable uncertainty estimates for the resulting ensemble, and that error prediction has its own learning targets and may require different expert weights from point forecasting. It therefore proposes GETUP, which learns the combination weights for error prediction separately. The point predictor is first trained or loaded and then frozen, after which an error model is trained to predict its absolute residuals: the predicted error magnitude supplies the scale, while a separately chosen residual distribution (Gaussian, an empirical distribution of historical errors, or a mixture centred on trajectories from a generative forecaster) determines the shape. The zero-shot implementation extends the Super-Linear spectrum-gated linear ensemble with an independently trained error branch, using two independent gates based on the input spectrum to combine point forecasts and error estimates, respectively. The same residual-prediction principle is also applied to uncertainty estimation for the frozen pretrained model AURORA and to an in-domain setting trained on the target dataset's history. The authors evaluate the method on TFB, an expanded ProbTS benchmark, and in-domain settings, reporting improvements in probabilistic metrics over existing baselines, while the zero-shot version's own point accuracy is slightly worse than AURORA's published results. Controlled experiments further examine the contributions of predictor-specific error learning, input-dependent scales, and residual shape.

Critical strengths
  1. The approach is simple and has low inference overhead: it directly outputs the distribution parameters without sampling as many times as in generative models.
  2. It is applicable to a wide range of scenarios: the same idea covers three Settings: zero-shot transfer, frozen pre-trained model uncertainty estimation, and in-domain training.
Critical weaknesses

Limited originality: There is a clear precedent for learning an error model separately on top of a frozen point predictor (e.g. DEUP [1]). The architecture of the zero-shot implementation is directly based on Super-Linear [2], and the difference mainly lies in adding an independent gating and combining the point prediction model for error prediction.

  1. Lack of comparison with similar methods: no comparison with β-NLL [3], faithful heteroscedastic regression [4] and other schemes, and no comparison with DEUP and other post-hoc error modeling methods on the same base.

  2. Insufficient demonstration of necessity of gating: ablation experiments replace weights in the inference stage, while the error expert itself is trained under the error-gated route, the effect decline is expected, and the retrained shared-gated or ungated control should be done.

  3. Generalization unverified: The error gating mechanism is only implemented on Super-Linear, and its effect is not verified on other base models.

  4. The baseline results on ProbTS are all copied from the original paper and mixed with the self-test results of the authors, which lacks a unified comparison, which weakens the credibility of the conclusion. Meanwhile, the point prediction accuracy of the zero-shot version is weaker than that of AURORA.

[1] Lahlou, Salem, et al. "DEUP: Direct epistemic uncertainty prediction." arXiv preprint arXiv:2102.08501 (2021).

[2] Nochumsohn, Liran, et al. "Super-Linear: A Lightweight Pretrained Mixture of Linear Experts for Time Series Forecasting." arXiv preprint arXiv:2509.15105 (2025).

[3] Seitzer, Maximilian, et al. "On the pitfalls of heteroscedastic uncertainty estimation with probabilistic neural networks." ICLR (2025).

[4] Stirn, A., Wessels, H., Schertzer, M., Pereira, L., Sanjana, N. & Knowles, D.. Faithful Heteroscedastic Regression with Neural Networks. AISTATS (2023)

Questions for authors
  1. The core claim of the paper is that error prediction requires different expert weights than point prediction, but existing gated ablation just replaces weights in the inference stage. Can the authors provide controls for retraining, such as shared gating or error branching without using gating, and give a deep enough analysis to justify independent error gating?

  2. The authors describe [5] PDHR as a "particularly close" method to this paper. Besides the addition of error gating, what is the essential difference between GETUP and PDHR and DEUP? Can these methods be compared on the same pedestal?

[5] Jordahn, Mikkel, et al. "Practical deep heteroskedastic regression." arXiv preprint arXiv:2603.01750 (2026).

7 comments
AuthorsOpenReview ↗

Response to Reviewer C7eA (1/6): summary of the revision and where each request is answered

Read together, the three reviews showed us that the submitted text did not convey clearly what the paper contributes or how its results fit together. We have therefore rewritten the main text around the two problems the paper addresses: obtaining calibrated predictive distributions from existing point forecasters, in-domain and zero-shot, and evaluating them by more than CRPS (Sec. 1–3); the construction, an evaluation protocol and the experiments follow (Sec. 4–5). App. I lists the changes from the submitted version; the submitted PDF remains in the OpenReview revision history. The submitted models carry the suffix -old; plain names denote the current versions (Sec. 5). We use the same names below. The experiments requested in the reviews are added, mostly in Sec. 5.3.

Re-reading the paper is not necessary to see our response: below we answer each point directly, give the exact location where the paper addresses it, and report every experiment the review asks for. Section, table and figure numbers refer to the revised PDF.

We thank the reviewer for the careful reading. Since the review's main concern is soundness, we first list every request and where the revision answers it.

The review asks forRevised PDF
Originality relative to DEUP and Super-Linear (W1)Sec. 1–4, App. E
β-NLL and faithful heteroscedastic regression on the same base (W2)Sec. 5.3, Table 2
DEUP and other post-hoc error models on the same base (W2, Q2)Sec. 5.3, Table 2
Retrained shared-gate control (W3, Q1)Table 2; App. H, Table 37
Retrained no-gate control (W3, Q1)Table 2; Table 37
The error model on a base other than Super-Linear (W4)Sec. 5.1, Table 1; Sec. 5.4, Tables 13–14; App. G, Table 36; App. H, Table 38
Published and self-measured results kept apart (W5a)Sec. 5, Fig. 3, Tables 1, 4, 13–14, 20
Point accuracy against Aurora (W5b)Sec. 5.1–5.2, Table 1, Fig. 3, Tables 11, 13–14, 19
Essential difference from DEUP and PDHR (Q2)Sec. 5.3, App. E
AuthorsOpenReview ↗

Response to Reviewer C7eA (2/6): W1 originality (DEUP, Super-Linear)

W1. Originality: DEUP and Super-Linear (Sec. 1–4, App. E)

We clarify the contribution as the revised introduction now states it (Sec. 1). It is organised around two problems (Sec. 3):

  • The engineering problem. Point forecasters improve every few months, while each probabilistic forecaster is a separate generative design, so every advance on the point side restarts the probabilistic one, and every foundation model ships its own sampler (Sec. 3.1). GETUP is one construction for both regimes (Sec. 4): point forecasts are frozen, a second model learns the magnitude of their own errors, and a separately fitted residual law gives the shape. In-domain, GETUP-ID is a soup that absorbs a stronger point model by adding it as a member; zero-shot, GETUP-ZS is a gated bank of linear frequency experts, trained once on a source corpus, that caps any frozen point predictor, a foundation model included (GETUP-Point, GETUP-ZS+Aurora).
  • The evaluation problem. CRPS alone does not see whether a predictive law can be trusted, and parameters do not measure cost (Sec. 3.2). The protocol (Sec. 4.1) reports CRPS, NLL, coverage and ENCE together and counts inference cost as operations per forecast window.
  • The evidence (Sec. 5): rank, CRPS and NMAE in both regimes with the cost of each system, an account of every design decision (Tables 2–3), and the finding that strong samplers lose to a trivial control built from their own point forecast: the control has lower NLL in 102 of 104 settings, while GETUP-ID-old and GETUP-ID beat their own controls on CRPS and NLL in all but 7 and 9 of 126 comparisons (Sec. 5.4, Table 4).

DEUP: the device is old; the forecaster is new. Predicting the size of a forecaster's error separately from the forecast is an old device: ARCH-type recursions make the magnitude of the next error predictable (Engle, 1982; Bollerslev, 1986), Glejser (1969) regressed the absolute residuals of a fixed least-squares fit on its inputs, and Lowe & Zapart (1999) trained a second network on the squared residuals of a fixed one; DEUP and PDHR are recent instances (Sec. 1, App. E). GETUP replaces the fixed recursion by a learned map of the whole history (Sec. 2). The paper claims a class of probabilistic forecasters built on the device that is strong on all four scores at once, at a fraction of the cost of sampling, and that improves whenever its point forecasters do (Sec. 1); to our knowledge, none had been built that competes with the leading generative and foundation models on the standard long-horizon benchmarks. Zero-shot, GETUP variants hold the six best mean CRPS ranks on the ProbTS grid: with Aurora's point forecast, GETUP-ZS+Aurora has the lowest mean CRPS (0.155 against 0.173 for Aurora), and the bank alone needs 1.14 GFLOP per window against 823 for Aurora; in-domain, GETUP-ID has the lowest mean CRPS (0.149 against 0.150 for VolDy-VAE) and the best mean CRPS and NMAE ranks (2.32 and 2.34 against 2.52 and 2.84; Table 1), and recomposing the soup as three PatchTrendFormer, four OLinear and three MLP members lowers CRPS by 1.38% per cell, in 57 of 72 cells (Table 3). DEUP- and PDHR-style error models are compared with ours on the same frozen forecast below (W2, Q2).

Super-Linear. Super-Linear is a point forecaster, and its spectral routing is its own (App. E). The zero-shot cap turns its bank into an uncertainty model: every frequency gets a point expert and an error expert, the error bank is routed by its own gate (Sec. 4; App. A.6), and the point side can be replaced altogether by another model's forecast while the error side is kept (GETUP-ZS+Aurora). The difference the review identifies, an independent error gate and an error bank serving a point forecast, is what makes the bank a probabilistic forecaster, and W3 shows that the separate gate is needed; it is a component of the zero-shot cap, needed for the bank to serve as one, not the contribution itself.

On the size of the contribution. The review lists simplicity and low inference cost as strengths, and the paper's claim rests on exactly these: no sampling, a cap on the point forecasters that exist, and a new point model absorbed at no design cost (Sec. 3.1). During this discussion alone small changes improved it further: the corpus-fitted scale factor at least halves the median NLL and ENCE of the zero-shot bank (Table 2). We would not measure novelty by the sophistication of the parts; if the number of variants obscures the main line, we are glad to move secondary variants to the appendix.

AuthorsOpenReview ↗

Response to Reviewer C7eA (3/6): W3/Q1 necessity of the error gate

W3 / Q1. Necessity of the error gate: trained controls (Sec. 5.3, Table 2; App. H, Table 37)

As the review notes, the submitted ablation substituted the weights at serving time. The revision retrains the controls. We first clarify what is trained. The point experts and the point gate are trained first, as in Super-Linear; with the point side frozen, each frequency error expert is trained on its own frequency stream without any gate, and only the error gate and the complementary error experts are trained through the error gate's routing (App. A.6). The one-gate and no-gate controls retrain exactly these, with the same recipe and three seeds of the error stage, on the windows of App. C.3.

The error gate is needed. Before calibration it is ahead of averaging the trained error experts on NLL and ENCE (median NLL 2.50 against 5.27 at H=96 and 1.78 against 5.51 at H=720, median ENCE 1.34 against 1.80 and 1.17 against 1.96), while on CRPS the average is lower at H=96 (0.3105 against 0.3130) and the gate at H=720 (0.3331 against 0.3394; Table 37). With the source-corpus factor it has lower CRPS, NLL and ENCE than the average at both horizons under all three training losses (Table 2). Sharing the point gate is the worst option: retrained under that routing, the error experts' intervals collapse to 32–36% coverage and CRPS rises by 10–12%.

The controls in detail:

  • One gate, taken from the point side. Routing the error bank by a frozen copy of the point gate, with the complementary experts retrained under that routing, raises CRPS in all 33 dataset–seed pairs, by 11.8% at H=96 and 10.4% at H=720 (Table 37); it has the worst NLL, coverage and ENCE in Table 2. The serving-time substitution measured 10.79% and 12.80% (Table 17): the train–test routing mismatch was not what produced the gap.
  • One gate trained on point and error: a single gate shared by both banks and trained in the error stage on both objectives—starting from the trained point gate, with the point experts frozen and the complementary error experts retrained—over the grid L_point + λ·L_error, λ ∈ {0.1, 1, 10}, and the Gaussian NLL, each at two gate steps (10⁻³ and the recipe's 0.1), one training seed per arm. At its own step of 10⁻³ it is worse than two gates on CRPS in every dataset at both horizons (on average over datasets by +3.5% to +12.8% for the squared-error objectives; the NLL objective diverges) and moves the point forecast (on average +0.3% to +13.1% MSE); at the recipe step of 0.1 the point MSE rises on average by up to 43% for the squared-error objectives. A gate shared by point and error costs either uncertainty or accuracy; in GETUP the point side is frozen before the error branch is fitted, so the served forecast never changes (Sec. 4).
  • No gate. Before calibration and under the submitted absolute-error objective, the gate is ahead of the average on CRPS in 9 of 18 dataset–seed pairs at H=96 and 9 of 15 at H=720, on NLL in 11/18 and 11/15; under the CRPS objective, on CRPS in 15/18 and 11/15 and on NLL in 14/18 and 12/15 (App. H). With the factor fitted on the source corpus (W2), for the absolute error: CRPS 0.3053 against 0.3134 and 0.3308 against 0.3425, median NLL 1.26 against 1.76 and 0.95 against 2.01 (Table 2).

Counts are over dataset–seed pairs because one run of the error stage varies between seeds (half-range of mean CRPS up to 0.0056; Table 37). App. C.3 keeps the routing analysis (the two gates select different experts, Jaccard overlap 0.154–0.380) and the serving-time substitution, labelled as such.

Reviewer_ytNYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GETUP, a framework for learning predictive uncertainty around time-series point forecasts while keeping the point-forecasting branch fixed. The central motivation is that weights optimized for point accuracy need not be suitable for estimating the error of the resulting ensemble. The zero-shot variant, GETUP-ZS, extends a spectrum-gated bank of linear forecasting experts with a separately trained error branch. One gate combines point predictions, while an independent gate combines estimates of residual magnitude. The paper considers two uncertainty constructions. Residual Uncertainty (RU) directly predicts the absolute residual of the combined forecast. Disagreement-Augmented Residual Uncertainty (DAU) predicts expert-specific residual magnitudes and adds a term based on disagreement between expert forecasts. The evaluation covers zero-shot comparisons on TFB and an expanded ProbTS grid, controlled uncertainty attachment experiments, in-domain forecasting, irregular sampling, calibration diagnostics, residual-shape studies, and computational measurements. The authors report that GETUP-ZS improves pooled CRPS over several source-trained forecasting systems, while predictor-specific error fitting improves likelihood and interval coverage for a frozen AURORA point forecast.

Critical strengths
  1. A point-forecast ensemble can achieve low MSE while still producing poorly calibrated uncertainty estimates. Learning the residuals of the actual served forecast, while leaving that forecast unchanged, is practically relevant for pretrained forecasting models whose parameters may be expensive or impossible to update.
  2. GETUP-POINT and the borrowed-error-bank control use identical AURORA point forecasts. The resulting reduction in NLL from 1.3490 to 1.0597 and improvement in 90% coverage from 73.75% to 81.65% provide evidence that an error model should be trained for the predictor whose errors it is intended to describe.
  3. The location-scale formulation supports Gaussian, empirical-residual, and mixture distributions without modifying the point predictor. The moment-matched Gaussian control is a sensible way to investigate whether non-Gaussian mixture shape contributes beyond the first two moments, although its current experimental coverage is limited.
Critical weaknesses
  1. The most specific new component appears to be the use of a separate spectral gate for residual experts in a time-series expert bank. This is useful, but narrower than the breadth suggested by the collection of GETUP-ZS, GETUP-POINT, GETUP-MIX, and GETUP-ID results. The paper would benefit from identifying one core algorithmic contribution and distinguishing it from applications of the general residual-modeling principle.
  2. Appendix C.2 replaces the error gate’s weights at inference while retaining error experts trained under the original two-gate system. This shows that the trained error gate and point gate behave differently, and that replacing the learned error weights generally hurts performance. It does not establish that two independently trained gates outperform a properly trained single-gate alternative. A shared-gate model could train its error experts under the routing distribution they will actually receive. Because the current serving-time substitutions create a train-test routing mismatch, the large CRPS differences cannot be attributed solely to the need for two gates. This is a central missing baseline for the claimed zero-shot architecture.
  3. The 7.1% pooled CRPS reduction over published AURORA is a complete-system comparison: GETUP-ZS and AURORA differ in both their point forecasts and their predictive distributions. GETUP-ZS is also 2.07% worse in relative MSE than published AURORA. The result therefore does not directly establish that GETUP provides a better uncertainty model for an equally accurate point predictor. In addition, sampled distributions, empirical residual laws, Gaussian approximations, and benchmark quantile approximations are used in different comparisons. The manuscript explicitly documents these differences, but the resulting numbers should be presented as separate experimental findings rather than as one consolidated superiority claim.
Questions for authors
  1. Can you train a matched single-gate baseline end to end, using the same gate for point and error experts throughout training and inference? This seems necessary to establish that independent gates provide value beyond the routing mismatch created by the current serving-time substitution.
  2. How do the main GETUP-ID results change when the uncertainty models are trained on cross-fitted or rolling-origin residuals? In particular, does the 3.74% CRPS improvement over the historical-scale control remain?
  3. Why is the conditional mean absolute residual used directly as a Gaussian scale in the default construction? Have you tested the analytical Gaussian conversion factor, a validation-fitted factor, or direct optimization of CRPS/NLL?
  4. Table 2 shows that refitting the AURORA-specific error head improves NLL and coverage but slightly worsens CRPS and AUC. Which notion of uncertainty quality is the primary objective, and how should these conflicting metrics affect the paper’s central claim?
5 comments
AuthorsOpenReview ↗

Response to Reviewer ytNY (1/5): summary of the revision; W1 contribution

Read together, the three reviews showed us that the submitted text did not convey clearly what the paper contributes or how its results fit together. We have therefore rewritten the main text around the two problems the paper addresses: obtaining calibrated predictive distributions from existing point forecasters, in-domain and zero-shot, and evaluating them by more than CRPS (Sec. 1–3); the construction, an evaluation protocol and the experiments follow (Sec. 4–5). App. I lists the changes from the submitted version, whose PDF remains in the OpenReview revision history. The submitted models carry the suffix -old and keep their numbers except for the corrections listed in App. I, while plain names denote the current versions (Sec. 5); we use the same names below. The experiments requested in the reviews are added, mostly in Sec. 5.3 and App. H.

Re-reading the paper is not necessary to see our response: below we answer each point directly, give the exact location where the paper addresses it, and report every experiment the review asks for. Section, table and figure numbers refer to the revised PDF.

Where each point is answered. W1 → Sec. 1, Sec. 3–4, App. E. W2/Q1 → Sec. 5.3, Table 2, App. A.6, App. C.3 (the submitted App. C.2), App. H. W3 → Sec. 5.1–5.2, Table 1, Fig. 3–4, App. C, App. G. Q2 → Sec. 5.3, Table 3, App. H. Q3 → Sec. 2, Sec. 4, Table 2, App. B.4, App. G. Q4 → Sec. 3.2, Sec. 5.4, App. H.

W1. One core contribution versus applications of a general principle

The revised introduction now names the core contribution (Sec. 1). It is organised around two problems (Sec. 3):

  • The engineering problem (Sec. 3.1). Point forecasters improve every few months, while each probabilistic forecaster is a separate generative design that every advance on the point side restarts, and every foundation model ships its own sampler. GETUP is the core contribution: one construction for both regimes (Sec. 4). Point forecasts are frozen, a second model learns the magnitude of their own errors, and a separately fitted residual law gives the shape. In-domain, GETUP-ID is a soup that absorbs a stronger point model by adding it as a member; zero-shot, GETUP-ZS is a cap, a gated bank of linear frequency experts trained once on a source corpus, that sits on any frozen point predictor, a foundation model included (GETUP-Point, GETUP-ZS+Aurora). This is the setting of the review's first strength: the served forecast stays unchanged, which is what pretrained models that cannot be updated need. The separate spectral gate is a component of the zero-shot cap, needed for the bank to serve as one (W2), not the contribution itself.
  • The evaluation problem (Sec. 3.2). CRPS alone does not see whether a predictive law can be trusted, and parameters do not measure cost. The protocol (Sec. 4.1) reports CRPS, NLL, coverage and ENCE together and counts inference cost as operations per forecast window.
  • The evidence (Sec. 5): rank, CRPS and NMAE in both regimes with the cost of each system, an account of every design decision (Tables 2–3), and the finding that strong samplers lose to a trivial control built from their own point forecast: the control has lower NLL in 102 of 104 settings, while GETUP-ID-old and GETUP-ID beat their own controls on CRPS and NLL in all but 7 and 9 of 126 comparisons (Sec. 5.4, Table 4). We think this needs to be stated clearly to a community that ranks by CRPS alone.

The device is old; the forecaster is new. Predicting the size of a forecaster's error separately from the forecast is an old and obvious device: ARCH-type recursions make the magnitude of the next error predictable (Engle, 1982; Bollerslev, 1986), Glejser (1969) regressed the absolute residuals of a fixed least-squares fit on its inputs, and Lowe & Zapart (1999) trained a second network on the squared residuals of a fixed one; DEUP and PDHR are recent instances (Sec. 1, App. E). The status of the device is not what the paper is about. GETUP replaces the fixed recursion by a learned map of the whole history (Sec. 2), and the paper claims a class of probabilistic forecasters built on it that is strong on all four scores at once, at a fraction of the cost of sampling, and that improves whenever its point forecasters do (Sec. 1); to our knowledge, none had been built that competes with the leading generative and foundation models on the standard long-horizon benchmarks. Zero-shot, GETUP variants hold the six best mean CRPS ranks on the ProbTS grid, the bank alone at 1.14 GFLOP per window against 823 for Aurora; in-domain, GETUP-ID has the lowest mean CRPS (0.149 against 0.150 for VolDy-VAE) and the best mean CRPS and NMAE ranks (2.32 and 2.34 against 2.52 and 2.84; Table 1).

AuthorsOpenReview ↗

Response to Reviewer ytNY (2/5): W1 (cont.); W2/Q1 trained shared-gate and no-gate controls

A non-obvious use of a frequency-expert bank. Super-Linear is a point forecaster. Turning its bank into the zero-shot cap, by giving every frequency a point expert and an error expert and routing the error bank by its own gate (Sec. 4; App. A.6), and then replacing the point side altogether by the output of another model while keeping the error machinery (GETUP-ZS+Aurora, GETUP-Point), is a construction of this paper. The same separation makes the point side replaceable in both regimes: zero-shot, the point branch is exchanged for a foundation model; in-domain, the soup absorbs a more accurate model as a member and the error branch is refitted by the unchanged procedure (with four OLinear members, CRPS falls by 1.38% per cell on average, in 57 of 72 cells; Table 3). The soup's error branch itself, value–error pairs stacked by ridge regression onto the error of the served forecast with a per-channel validation anchor, and the residual shape fitted separately from the scale (Sec. 4, App. A.5), are also introduced here; the submitted text presented them as implementation details.

On the size of the contribution. Some reviews consider the approach too simple for an ICLR contribution. We see its strength precisely there: an approach fundamentally different from the mainstream of heavy sampling-based models, simpler, applicable to any point forecaster in both regimes, ahead of the leading models zero-shot and, in-domain, on all four scores under each model's own predictive law, at a fraction of their operations. It is also at an early stage with a high capacity for adaptation: during this discussion alone, small changes improved it further (a scale factor fitted on the source corpus at least halves the median NLL and ENCE of the zero-shot bank, Table 2; OLinear members lower in-domain CRPS by 1.4% per cell, Table 3). We would therefore not measure its novelty by the sophistication of its parts. If the number of variants obscures the main line, we are glad to move secondary variants to the appendix.

What the ensembles add (Table 3, block B; App. H). Reducing GETUP-ID-old to one point model and one error head (K=N=1; MLP, PatchTrendFormer, RLinear; 72 settings) raises CRPS by 6.8%, 5.0% and 9.0%, higher than the full ensemble in 69, 72 and 72 of 72 settings; each single expert keeps 97%, 78% and 74% of the full ensemble's advantage over its trivial control.

W2/Q1. Trained shared-gate and no-gate controls (Sec. 5.3, Table 2; App. H, Table 37)

The error gate is needed. Before calibration it is ahead of averaging the trained error experts on NLL and ENCE (median NLL 2.50 against 5.27 at H=96 and 1.78 against 5.51 at H=720, median ENCE 1.34 against 1.80 and 1.17 against 1.96), while on CRPS the average is lower at H=96 (0.3105 against 0.3130) and the gate at H=720 (0.3331 against 0.3394; Table 37). With the source-corpus factor it has lower CRPS, NLL and ENCE than the average at both horizons under all three training losses (Table 2). Sharing the point gate is the worst option: retrained under that routing, the error experts' intervals collapse to 32–36% coverage and CRPS rises by 10–12%.

The routing mismatch. In the construction the frequency error experts are trained without any gate and the point gate on the point loss alone (App. A.6). Routing the error bank by a frozen copy of the point gate, with the complementary error experts retrained under that routing (three seeds of the error stage), therefore leaves the cost where the serving-time substitution put it: CRPS rises in all 33 dataset–seed pairs, by 11.8% at H=96 and 10.4% at H=720, against 10.79% and 12.80% for the substitution (Table 17). The train–test routing mismatch was not what produced the gap.

This suggested a stronger test: a single gate shared by both banks and trained in the error stage on both objectives—starting from the trained point gate, with the point experts frozen and the complementary error experts retrained—over the grid L_point + λ·L_error, λ ∈ {0.1, 1, 10}, and the Gaussian NLL, each at two gate steps (10⁻³ and the recipe's 0.1), one training seed per arm. At its own step of 10⁻³ it is worse than two gates on CRPS in every dataset at both horizons (on average over datasets by +3.5% to +12.8% for the squared-error objectives; the NLL objective diverges) and moves the point forecast (on average +0.3% to +13.1% MSE); at the recipe step of 0.1 the point MSE rises on average by up to 43% for the squared-error objectives (App. H). A gate shared by point and error either costs uncertainty or costs accuracy.

AuthorsOpenReview ↗

Response to Reviewer ytNY (3/5): W2/Q1 (cont.); W3 complete systems vs. equal point forecasts

No gate (W2/Q1, continued). Per dataset–seed pair at the raw scale, the gate is ahead of the average on CRPS in 9 of 18 at H=96 and 9 of 15 at H=720 (App. H). With the scale conversion the reviewer proposed in Q3, for the absolute-error objective, CRPS is 0.3053 against 0.3134 and 0.3308 against 0.3425, and median NLL 1.26 against 1.76 and 0.95 against 2.01 (Table 2). The revised experiments thus show the need for the second gate more broadly than the submitted ones.

W3. Complete systems versus equal point forecasts; estimators

Sec. 5 now reports these as separate findings, each naming what is scored, together with the cost of each system.

GETUP-ZS-old is a trade-off, not a point-accuracy claim. The 7.1% of the submitted text is its pooled CRPS reduction against published Aurora over all 36 cells, lower in 30 (Table 19); for the current GETUP-ZS it is 5.7%, lower in 28. On the 24 complete zero-shot cells of the ProbTS grid its mean CRPS is 0.156 against 0.173 for Aurora and its mean CRPS rank 2.88 against 7.96, while its NMAE is level with Aurora's (0.193 against 0.192; Table 1) and its MSE on the TFB grid is 2.07% higher (Table 11, Sec. 6). It needs 1.14 GFLOP per window against 823 for Aurora's 100 sampled trajectories (Sec. 5.2, Fig. 4). The current GETUP-ZS, a Gaussian law scaled by the source-corpus factor, is slightly behind it on this grid (0.157, rank 2.96; Sec. 5.1, App. G; see Q3).

An equally accurate point predictor: the attachments to Aurora. Both attachments keep Aurora's deterministic forecast and change only the scale, so they answer this question directly. GETUP-ZS+Aurora (the bank's error branch with the source-corpus factor) has the lowest zero-shot mean CRPS of Table 1, 0.155 (rank 2.33), against 0.173 for Aurora and 0.171 for Aurora^ours (our run of Aurora, a Gaussian with the mean and spread of its 100 samples); GETUP-Point-old (an error head fitted to Aurora's residuals on the source corpus) has 0.162, and GETUP-Point, the same head with its corpus factor, 0.164 (rank 5.12). On identical windows against a Gaussian with the mean and spread of Aurora's 100 samples (Tables 13–14), GETUP-Point-old lowers mean NLL from 39.2 to 1.074 ± 0.014 (two fits of the head), brings coverage closer to 0.90 and lowers ENCE in all 36 cells, and lowers CRPS in 28; with the source-corpus factor, GETUP-Point and GETUP-ZS+Aurora reach NLL 0.926 and 0.736 and CRPS 0.3192 and 0.3143, each with coverage closer to nominal and lower ENCE in all 36 cells and lower CRPS in 26 and 28.

The shared forecast is our run of Aurora's deterministic head: on the TFB grid its MSE is 1.59% below published Aurora's, 95% interval [−3.87, +0.50]% (Table 11), and both attachments have Aurora's NMAE, 0.192 (Table 1). Pooled over all 36 cells, CRPS is 7.8% below published Aurora's for GETUP-ZS+Aurora (lower in 32) and 2.5% for GETUP-Point-old (lower in 27), and 0.4% above it for GETUP-Point (lower in 26; Table 19). On both axes at once, the front of mean NMAE against mean CRPS consists of GETUP-ZS+Aurora, GETUP-ZS+Aurora-old and Aurora^ours (Fig. 3); cell by cell a GETUP variant is on the front in 33 of 36 cells and forms all of it in 25 (App. C.2).

In-domain, under each model's own predictive law on the 24 cells at L=96, GETUP-ID-old and GETUP-ID are both ahead of TARFVAE and VolDy-VAE on CRPS, NLL, coverage and ENCE; GETUP-ID has the lowest CRPS (0.2905 against 0.2944), GETUP-ID-old the lowest NLL and ENCE and the lowest cost, 0.546 GFLOP per window against 7.25 for TARFVAE, 1.14 for VolDy-VAE and 2.76 for GETUP-ID (Table 4, Fig. 5). On the 28 complete in-domain cells of the ProbTS grid, GETUP-ID, measured on three seeds, has the lowest mean CRPS (0.149 against 0.150 for VolDy-VAE and 0.158 for GETUP-ID-old) and the best mean CRPS and NMAE ranks (2.32 and 2.34 against 2.52 and 2.84 for VolDy-VAE); VolDy-VAE keeps the lowest mean NMAE (0.188 against 0.189) and leads on Illness, which the harness scores on two test windows per horizon; without Illness GETUP-ID is further ahead in mean CRPS (0.1487 against 0.1568) and rank (1.58 against 2.77) on the remaining 24 cells (Table 1, Sec. 5.1). GETUP-ID needs 2.41 GFLOP per window on these cells (with GETUP-ID-old's count on Illness, where the soup's operations are not yet counted), and GETUP-ID-old half VolDy-VAE's operations, 0.514 against 1.01 GFLOP (Table 1). The current GETUP-ID, with OLinear members, lowers CRPS against GETUP-ID-old by 1.4% per cell (Table 3). Against GETUP-ID-old, GETUP-ID lowers ProbTS CRPS from 0.153 to 0.149 on the four ETT sets, Exchange and Weather (lower in 20 of 24 cells) and from 0.171–0.202 to 0.131–0.166 on Illness (Table 20, App. G); on Electricity and Traffic its runs with OLinear's published settings are queued or in progress.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Updated revision (17 Nov): GETUP-ID on Electricity, one correction

We have uploaded a further revision of the PDF. Compared with the revision our responses refer to, it changes only the points below; table and appendix numbers and every other number quoted in our responses are unchanged.

1. The current GETUP-ID on Electricity. The soup with OLinear members, trained with OLinear's published Electricity settings, has finished on all four horizons with three seeds. ProbTS CRPS at H = 96 / 192 / 336 / 720:

  • GETUP-ID: 0.059 / 0.065 / 0.072 / 0.078
  • GETUP-ID-old: 0.066 / 0.072 / 0.078 / 0.090
  • VolDy-VAE: 0.069 / 0.075 / 0.078 / 0.080

It is the lowest in the grid at every Electricity horizon and forms the per-cell Pareto front there alone (Table 20, Table 16); pooled over its 32 cells, its CRPS is 1.0% below VolDy-VAE's, lower in 24 of 32 (Table 19). Table 1 does not change: Electricity is not among its complete cells, because some baselines have no published value at H = 336 and 720. Where our responses say that the Electricity and Traffic runs of GETUP-ID are in progress, Electricity is now complete; Traffic is still running, and there GETUP-ID remains GETUP-ID-old by definition (marked ≡ in the tables).

2. One correction. In Table 20 the GETUP-ID-old cell for Traffic at H = 336 averaged two seeds, one of which had been scored before its trained model was final. We removed that seed; the cell now shows the remaining one (0.206, marked as one seed of three). None of the numbers in our responses depends on this cell.

3. Two Traffic cells of GETUP-ID-old's per-cell law grid (L = 96, H = 336 and 720) gained a second seed; the counts in Table 4 are unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.