- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
Suppose you have multiple forecasting models, and you want to decide which model to use. One possible procedure is to (1) calibrate all models on historical data (2) test their calibrated predictions on validation data (3) choose the best performing model (4) recalibrate this model on fresh data before deployment. However, the authors point out that the model selected after historical calibration in (1) may not be the model that performs best (in expectation) when calibrated on fresh data as in (4).
The authors therefore propose two technical contributions.
(1) A faster way to evaluate forecasts under repeated calibration (Section 3). One method of handling multiple sets of calibration data is to use cross validation and to average over all possible splits. The authors derive a faster way to compute this average using the fact that forecast errors are ranked and that the interval score has a piecewise linear structure.
(2) A statistically conservative way to choose between similar forecasts (Section 4). The authors derive a tighter bound on the difference in calibrated interval scores. Then, using concentration inequalities, they get a decision rule that effectively says to keep the current forecast unless there is sufficient evidence that an alternative should have lower expected score after recalibration. The main novelty here is in tightening the bound.
Experiments show that the proposed method of computing the average is faster than the standard method (Table 2). They also show that the tighter bound allows the conservative selection rule to be helpful in some settings (Table 3). However, it does not outperform quantile regression on real world time series data.
Critical strengths
The paper leverages the structure of calibration to derive two concrete technical contributions. These contributions seem sound.
Critical weaknesses
(1) The novelty of the first technical contribution is unclear. Methods for computing averages over cross validation splits are well-established, with existing work on efficient computation of U-statistics. The authors could more clearly establish what is novel beyond specializing the U-statistics framework to calibrated interval scores. The speed comparisons should also be made against existing U-statistics methods (rather than only against the authors’ own custom baselines).
(2) The practical utility of the second technical contribution seems limited. The improvement in model selection is mostly demonstrated in synthetic settings, and the conservative selection rules do not outperform quantile regression on real world data.
(3) The language is compressed and difficult for a human to read.
Questions for authors
-
How does the algorithm in Section 3 differ from established methods for computing averages over cross validation splits and U-statistics?
-
For what practical settings is the tighter bound in Section 4 expected to be useful?
Thank you for separating the questions of novelty, usefulness and presentation.
(1) What is new in the computation? Averaging cross-validation splits and computing complete U-statistics are established ideas. Even the probabilities that a subsample quantile takes each rank are classical (Kaigh and Lachenbruch, 1982). The additional step is scoring a held-out observation at that quantile. Its interval score is piecewise linear with two breakpoints. We sort the residuals once and reuse weighted prefix sums across every possible query. Deleting a row changes its role both as a query and as a calibration point. Accounting for those two changes gives all N deletions and the complete mean in O(N log N) per already-fitted forecast. The target and pool size stay fixed. Section 3 and Appendix A give the identity and algorithm.
(2) Which rule changed real forecast choices? We used the asymptotic jackknife threshold with Section 3’s delete-one variance. Algorithm 1 has a finite-sample guarantee and was tested on synthetic data. On the 153 fitted series, we used the first 27 origins to select, the next 13 to calibrate and the last 12 to score. The cautious rule switched on 40 series, helping 36 and hurting four. Unguarded averaging switched on 110, helping 73 and hurting 37. Compared with keeping the reference, the cautious rule’s mean loss fell by .09455 (whole-dataset resampling interval [−.1811, −.0260]). Its difference from unguarded averaging was +.01386 [−.03030, .06634], leaving the direction unresolved. The single chronological split had the lowest observed mean. When we instead reused 20 of the 40 selection origins for final calibration, the cautious rule helped 48 series and hurt seven. Against keeping the reference its mean score changed by −.10717, compared with −.10912 for always using local and −.12914 for unguarded averaging. The three policies hurt 7, 50 and 27 series, respectively. The threshold raised mean loss by .02197 versus unguarded averaging. These are retrospective choices on dependent series, without a nominal safety guarantee (Appendix P.3).
What did the software comparison measure? Chen et al.’s u-stats 0.7.6 and our sorted method computed the same two means in ten of twelve original calls. Two u-stats calls hit 20 seconds. In the two-machine repeat with a 45-second cap, all 320 calls at N=16 or 64 matched. At N=512 the 160 generic calls reached the 2 GiB memory limit. These are limits for the tested encoding. Table 2 compares a different task, the mean and all deletions, against our batched dense implementation (Tables 2, 52 and Appendix P).
When can the Section 4 bound change a decision? Consider an existing forecast whose owner proposes a bounded adjustment to its center. The fitted adjustment itself can bound the endpoint movement on the entire deployment domain. With independent selection observations, a negative upper bound supports the update. Holding the Hoeffding rule and labels fixed, changing only the bound moves zero favorable settings to 12 of 72 (Table 3). In a separate fitted synthetic comparison, the complete Bernstein rule kept the reference at both smaller budgets; at the largest it switched in 38 of 128 comparisons (Tables 54–55).
How does this relate to the fixed forecasts and the single-split comparison? In Appendix F the four step selectors minimize validation loss without a confidence guard. Fixed quantile regression beats them in both sources at both label delays; a native online procedure has extra feedback. None of its 34 eligible windows had enough selection labels (32) for the actual final pool (at least 59). In the separate nine-row-pool comparison, complete averaging beats one reused pool in three of four settings. For immediate traffic the score falls from 2963.9263 to 2511.6836 vehicles/hour, tying the fixed linear forecast (Table 35). The repeated-split average matches the complete average to displayed precision in three cells and scores slightly better in the fourth. A new comparison on all 153 public series fixes the candidate pair and uses the same 40 historical origins for both choices. At deployment pool size 20, averaging changed the choice on 23 series; 13 scored better and ten worse on 12 later origins. This is a retrospective choice comparison, with no confidence-based switching claim.
(3) What changed for readers? The introduction diagrams selection, calibration and scoring. Table 1 shows the ranking reversal in decimals, with exact values below. Section 3 distinguishes the score calculation from the inference assumptions. Table 3a gives better and worse series counts separately, with disjoint calibration first and both fitted choices scored on later observations. Appendix O holds the longer derivations.