- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes GETUP, a method for estimating predictive uncertainty in time-series forecasting while keeping the point forecaster fixed. Its ensemble version uses two independent gates based on the input’s frequency content: one combines point forecasts, while the other combines predictions of forecast-error magnitude. These error estimates provide an input-dependent scale that can be combined with Gaussian or non-Gaussian residual distributions to produce probabilistic forecasts.
Critical strengths
The paper has a useful practical goal: adding predictive uncertainty to a frozen forecaster while preserving its point predictions. There is a broad evaluation and the controlled result showing a 3.74% pooled CRPS reduction from input-dependent scales at fixed point forecasts and residual shape.
Critical weaknesses
Although the paper evaluates calibration, substantial undercoverage remains in the zero-shot experiments. A comparison with a suitable post-hoc recalibration baseline would help establish whether the proposed uncertainty model improves calibration beyond simpler corrections. In Appendix C.2, the authors replace the uncertainty gate’s weights with those of the point-prediction gate while keeping the trained experts fixed, supporting the usefulness of different weights for the two tasks. However, this substitution disadvantages the shared-gate configuration: the point gate was trained for forecasting, and the error experts were trained with a different gate. A fairer comparison would train a shared gate for both tasks from scratch, using the same experts, data, and comparable tuning. The experiments show that the trained uncertainty branch benefits from its own gate; it does not show that a properly trained joint gate would be worse. The paper compares GETUP with several forecasting baselines, but those comparisons also change architecture, training, and uncertainty estimation. They therefore cannot establish that the gains come specifically from using a mixture of experts. Section 4.1 explicitly acknowledges missing single-expert controls for both the point and uncertainty ensembles. To establish this claim, the authors would need to compare GETUP with a single point expert and a single error predictor under matched training and evaluation conditions. Thus, the paper shows that its complete ensemble system is competitive, but does not demonstrate that multiple experts are necessary for its improvements on these datasets. My main concern is that predicting a frozen model’s errors, separating location from scale, and combining ensemble uncertainty are established ideas, as the paper itself explains. The clearest new element is separate gates for point forecasting and uncertainty estimation. This is a useful architectural extension rather than a sufficiently substantial ICLR contribution.
Questions for authors
see weakness comments
5 comments
Response to Reviewer LJue (2/5): main concern, contribution beyond separate gates
Main concern: what the paper contributes beyond separate gates (Sec. 1, 3–4, App. E)
We clarify the contribution as the revised introduction now states it (Sec. 1). It is organised around two problems (Sec. 3):
- The engineering problem (Sec. 3.1). Point forecasters improve every few months, while each probabilistic forecaster is a separate generative design that every advance on the point side restarts, and every foundation model ships its own sampler. GETUP is the core contribution: one construction for both regimes (Sec. 4). Point forecasts are frozen, a second model learns the magnitude of their own errors, and a separately fitted residual law gives the shape. In-domain, GETUP-ID is a soup that absorbs a stronger point model by adding it as a member; zero-shot, GETUP-ZS is a cap, a gated bank of linear frequency experts trained once on a source corpus, that sits on any frozen point predictor, a foundation model included (GETUP-Point, GETUP-ZS+Aurora). This is the practical goal the review names as the paper's strength: uncertainty added to a frozen forecaster with its point predictions preserved. The separate spectral gate is a component of the zero-shot cap, needed for the bank to serve as one (see the gate controls in Part 5), not the contribution itself; the in-domain soup has no spectral gate at all, its error heads are combined by a ridge stacker.
- The evaluation problem (Sec. 3.2). CRPS alone does not see whether a predictive law can be trusted, and parameters do not measure cost. The protocol (Sec. 4.1) reports CRPS, NLL, coverage and ENCE together and counts inference cost as operations per forecast window.
- The evidence (Sec. 5): rank, CRPS and NMAE in both regimes with the cost of each system, an account of every design decision (Tables 2–3), and the finding that strong samplers lose to a trivial control built from their own point forecast: the control has lower NLL in 102 of 104 settings, while GETUP-ID-old and GETUP-ID beat their own controls on CRPS and NLL in all but 7 and 9 of 126 comparisons (Sec. 5.4, Table 4). We think this needs to be stated clearly to a community that ranks by CRPS alone.
The device is old; the forecaster is new. As the review notes, predicting a frozen model's errors, separating location from scale and combining ensemble uncertainty are established ideas, and the paper discusses them (Sec. 1, App. E): ARCH-type recursions make the magnitude of the next error predictable (Engle, 1982; Bollerslev, 1986), Glejser (1969) regressed the absolute residuals of a fixed least-squares fit on its inputs, and Lowe & Zapart (1999) trained a second network on the squared residuals of a fixed one; DEUP and PDHR are recent instances. The status of the device is not what the paper is about. GETUP replaces the fixed recursion by a learned map of the whole history (Sec. 2), and the paper claims a class of probabilistic forecasters built on it that is strong on all four scores at once, at a fraction of the cost of sampling, and that improves whenever its point forecasters do (Sec. 1); to our knowledge, none had been built that competes with the leading generative and foundation models on the standard long-horizon benchmarks. Zero-shot, GETUP variants hold the six best mean CRPS ranks on the ProbTS grid, the bank alone at 1.14 GFLOP per window against 823 for Aurora; in-domain, GETUP-ID has the lowest mean CRPS (0.149 against 0.150 for VolDy-VAE) and the best mean CRPS and NMAE ranks (2.32 and 2.34 against 2.52 and 2.84; Table 1).
Response to Reviewer LJue (3/5): main concern (cont.); complete systems and controlled comparisons
A non-obvious use of a frequency-expert bank. Super-Linear is a point forecaster. Turning its bank into the zero-shot cap, by giving every frequency a point expert and an error expert and routing the error bank by its own gate (Sec. 4; App. A.6), and then replacing the point side altogether by the output of another model while keeping the error machinery (GETUP-ZS+Aurora, GETUP-Point), is a construction of this paper. The same separation makes the point side replaceable in both regimes: zero-shot, the point branch is exchanged for a foundation model; in-domain, the soup absorbs a more accurate model as a member and the error branch is refitted by the unchanged procedure (with four OLinear members, CRPS falls by 1.38% per cell on average, in 57 of 72 cells; Table 3). The soup's error branch itself, value–error pairs stacked by ridge regression onto the error of the served forecast with a per-channel validation anchor, and the residual shape fitted separately from the scale (Sec. 4, App. A.5), are also introduced here; the submitted text presented them as implementation details.
On the size of the contribution. The review reads the work as a useful architectural extension. We see its strength precisely in its simplicity: an approach fundamentally different from the mainstream of heavy sampling-based models, applicable to any point forecaster in both regimes, ahead of the leading models zero-shot and, in-domain, on all four scores under each model's own predictive law, at a fraction of their operations. It is also at an early stage with a high capacity for adaptation: during this discussion alone, small changes improved it further (a scale factor fitted on the source corpus at least halves the median NLL and ENCE of the zero-shot bank, Table 2; OLinear members lower in-domain CRPS by 1.4% per cell, Table 3). We would therefore not measure its novelty by the sophistication of its parts. If the number of variants obscures the main line, we are glad to move secondary variants to the appendix.
Complete systems and controlled comparisons (Sec. 5)
As the review notes, the benchmark comparisons change architecture, training and uncertainty estimation at once. Sec. 5 now keeps the kinds of evidence apart. Sec. 5.1–5.2 compare complete systems, each with its cost. Zero-shot, on the 24 complete cells GETUP-ZS+Aurora, Aurora's deterministic forecast with the bank's error branch and the source-corpus factor, has the lowest mean CRPS, 0.155, and the best mean CRPS rank, 2.33, against 0.173 and 7.96 for Aurora; GETUP-ZS-old (0.156, rank 2.88) and GETUP-ZS (0.157, rank 2.96) follow. The bank's NMAE is level with Aurora's (0.193 against 0.192), and its MSE on the TFB grid is 2.07% higher than published Aurora's (Tables 1, 11). On the 28 complete in-domain cells of the ProbTS grid, GETUP-ID, measured on three seeds, has the lowest mean CRPS (0.149 against 0.150 for VolDy-VAE and 0.158 for GETUP-ID-old) and the best mean CRPS and NMAE ranks (2.32 and 2.34 against 2.52 and 2.84 for VolDy-VAE); VolDy-VAE keeps the lowest mean NMAE (0.188 against 0.189) and leads on Illness, which the harness scores on two test windows per horizon; without Illness GETUP-ID is further ahead in mean CRPS (0.1487 against 0.1568) and rank (1.58 against 2.77) on the remaining 24 cells (Table 1, Sec. 5.1). Against GETUP-ID-old, GETUP-ID lowers ProbTS CRPS from 0.153 to 0.149 on the four ETT sets, Exchange and Weather (lower in 20 of 24 cells) and from 0.171–0.202 to 0.131–0.166 on Illness (Table 20, App. G); on Electricity and Traffic its runs with OLinear's published settings are queued or in progress. Sec. 5.3 changes one component at a time and keeps everything else, the point forecast included, fixed (Tables 2–3); the three experiments the review asks for are there, and we report them below. Sec. 5.4 scores each model against its own trivial control on identical windows (Table 4).
Response to Reviewer LJue (1/5): summary of the revision and where each point is answered
Read together, the three reviews showed us that the submitted text did not convey clearly what the paper contributes or how its results fit together. We have therefore rewritten the main text around the two problems the paper addresses: obtaining calibrated predictive distributions from existing point forecasters, in-domain and zero-shot, and evaluating them by more than CRPS (Sec. 1–3); the construction, an evaluation protocol and the experiments follow (Sec. 4–5). App. I lists the changes from the submitted version, whose PDF remains in the OpenReview revision history. The submitted models carry the suffix -old and keep their numbers except for the corrections listed in App. I; plain names denote the current versions (Sec. 5). We use the same names below. The experiments requested in the reviews are added, mostly in Sec. 5.3.
Re-reading the paper is not necessary to see our response: below we answer each point directly, give the exact location where the paper addresses it, and report every experiment the review asks for. Section, table and figure numbers refer to the revised PDF.
We thank the reviewer for the careful reading. The review raises one main concern and asks for three experiments; we list each and where the revision answers it.