acceptodds
Under review as a conference paper at ICLR 2027

The Price of Choosing: Forecast Selection Under Fresh Calibration

Abstract

When a forecast chosen with one calibration pool is calibrated again before deployment, the two calibrations need not rank candidates alike. We make the expected interval score after a fresh, finite calibration pool the comparison target. An exact construction shows that reusing one pool can select the worse forecast regardless of the number of queries. For A fixed predictors and N observations, rank weights and two hinges compute the complete-subset risk estimate and every fixed-pool-size delete-one value exactly in O(AN log N) arithmetic operations; across 12 settings, this is 1.44–24.69 times faster than a custom batched dense reference computing the same outputs. For globally nearby endpoints with a common bounded scale, calibration ranks give a paired score envelope without bounding the response, and its uniform constant 8 is sharp across ranks. Label-cost bounds then separate exact assessment from reliable selection. In a 72-setting unbounded-support study, the envelope enables beneficial controlled updates in up to 20 settings, while a query-level bound never updates. On nine public forecasting datasets, calibration removes most of a matched fitting gain, from 5.34 to 1.52 percent, and from 7.05 to 3.13 percent over ten seeds; in a prespecified external test, the seven hourly datasets, one of its two primary groups, retain 62 percent of the gain. Pre-calibration comparisons can therefore overstate the surviving difference. Candidate forecasts should be compared by their risk after the calibration they will actually receive, and its exact estimate is now cheap to compute.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
conformal prediction, uncertainty quantification, forecast selection, model selection, calibration, interval score, time series forecasting

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_65s3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

Suppose you have multiple forecasting models, and you want to decide which model to use. One possible procedure is to (1) calibrate all models on historical data (2) test their calibrated predictions on validation data (3) choose the best performing model (4) recalibrate this model on fresh data before deployment. However, the authors point out that the model selected after historical calibration in (1) may not be the model that performs best (in expectation) when calibrated on fresh data as in (4).

The authors therefore propose two technical contributions.

(1) A faster way to evaluate forecasts under repeated calibration (Section 3). One method of handling multiple sets of calibration data is to use cross validation and to average over all possible splits. The authors derive a faster way to compute this average using the fact that forecast errors are ranked and that the interval score has a piecewise linear structure.

(2) A statistically conservative way to choose between similar forecasts (Section 4). The authors derive a tighter bound on the difference in calibrated interval scores. Then, using concentration inequalities, they get a decision rule that effectively says to keep the current forecast unless there is sufficient evidence that an alternative should have lower expected score after recalibration. The main novelty here is in tightening the bound.

Experiments show that the proposed method of computing the average is faster than the standard method (Table 2). They also show that the tighter bound allows the conservative selection rule to be helpful in some settings (Table 3). However, it does not outperform quantile regression on real world time series data.

Critical strengths

The paper leverages the structure of calibration to derive two concrete technical contributions. These contributions seem sound.

Critical weaknesses

(1) The novelty of the first technical contribution is unclear. Methods for computing averages over cross validation splits are well-established, with existing work on efficient computation of U-statistics. The authors could more clearly establish what is novel beyond specializing the U-statistics framework to calibrated interval scores. The speed comparisons should also be made against existing U-statistics methods (rather than only against the authors’ own custom baselines).

(2) The practical utility of the second technical contribution seems limited. The improvement in model selection is mostly demonstrated in synthetic settings, and the conservative selection rules do not outperform quantile regression on real world data.

(3) The language is compressed and difficult for a human to read.

Questions for authors
  1. How does the algorithm in Section 3 differ from established methods for computing averages over cross validation splits and U-statistics?

  2. For what practical settings is the tighter bound in Section 4 expected to be useful?

1 comment

Thank you for separating the questions of novelty, usefulness and presentation.

(1) What is new in the computation? Averaging cross-validation splits and computing complete U-statistics are established ideas. Even the probabilities that a subsample quantile takes each rank are classical (Kaigh and Lachenbruch, 1982). The additional step is scoring a held-out observation at that quantile. Its interval score is piecewise linear with two breakpoints. We sort the residuals once and reuse weighted prefix sums across every possible query. Deleting a row changes its role both as a query and as a calibration point. Accounting for those two changes gives all N deletions and the complete mean in O(N log N) per already-fitted forecast. The target and pool size stay fixed. Section 3 and Appendix A give the identity and algorithm.

(2) Which rule changed real forecast choices? We used the asymptotic jackknife threshold with Section 3’s delete-one variance. Algorithm 1 has a finite-sample guarantee and was tested on synthetic data. On the 153 fitted series, we used the first 27 origins to select, the next 13 to calibrate and the last 12 to score. The cautious rule switched on 40 series, helping 36 and hurting four. Unguarded averaging switched on 110, helping 73 and hurting 37. Compared with keeping the reference, the cautious rule’s mean loss fell by .09455 (whole-dataset resampling interval [−.1811, −.0260]). Its difference from unguarded averaging was +.01386 [−.03030, .06634], leaving the direction unresolved. The single chronological split had the lowest observed mean. When we instead reused 20 of the 40 selection origins for final calibration, the cautious rule helped 48 series and hurt seven. Against keeping the reference its mean score changed by −.10717, compared with −.10912 for always using local and −.12914 for unguarded averaging. The three policies hurt 7, 50 and 27 series, respectively. The threshold raised mean loss by .02197 versus unguarded averaging. These are retrospective choices on dependent series, without a nominal safety guarantee (Appendix P.3).

What did the software comparison measure? Chen et al.’s u-stats 0.7.6 and our sorted method computed the same two means in ten of twelve original calls. Two u-stats calls hit 20 seconds. In the two-machine repeat with a 45-second cap, all 320 calls at N=16 or 64 matched. At N=512 the 160 generic calls reached the 2 GiB memory limit. These are limits for the tested encoding. Table 2 compares a different task, the mean and all deletions, against our batched dense implementation (Tables 2, 52 and Appendix P).

When can the Section 4 bound change a decision? Consider an existing forecast whose owner proposes a bounded adjustment to its center. The fitted adjustment itself can bound the endpoint movement on the entire deployment domain. With independent selection observations, a negative upper bound supports the update. Holding the Hoeffding rule and labels fixed, changing only the bound moves zero favorable settings to 12 of 72 (Table 3). In a separate fitted synthetic comparison, the complete Bernstein rule kept the reference at both smaller budgets; at the largest it switched in 38 of 128 comparisons (Tables 54–55).

How does this relate to the fixed forecasts and the single-split comparison? In Appendix F the four step selectors minimize validation loss without a confidence guard. Fixed quantile regression beats them in both sources at both label delays; a native online procedure has extra feedback. None of its 34 eligible windows had enough selection labels (32) for the actual final pool (at least 59). In the separate nine-row-pool comparison, complete averaging beats one reused pool in three of four settings. For immediate traffic the score falls from 2963.9263 to 2511.6836 vehicles/hour, tying the fixed linear forecast (Table 35). The repeated-split average matches the complete average to displayed precision in three cells and scores slightly better in the fourth. A new comparison on all 153 public series fixes the candidate pair and uses the same 40 historical origins for both choices. At deployment pool size 20, averaging changed the choice on 23 series; 13 scored better and ten worse on 12 later origins. This is a retrospective choice comparison, with no confidence-based switching claim.

(3) What changed for readers? The introduction diagrams selection, calibration and scoring. Table 1 shows the ranking reversal in decimals, with exact values below. Section 3 distinguishes the score calculation from the inference assumptions. Table 3a gives better and worse series counts separately, with disjoint calibration first and both fitted choices scored on later observations. Appendix O holds the longer derivations.

Reviewer_y665OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates forecast selection when candidate prediction intervals undergo fresh calibration before deployment. It demonstrates that evaluating forecasts using a reused calibration pool can lead to incorrect rankings, even with arbitrarily many evaluation queries. To address this issue, the authors develop an exact, computationally efficient estimator of expected post-calibration interval score, establish a sharp paired-score bound for nearby forecasts, and derive label-complexity results for reliable selection. Synthetic experiments and real-world forecasting benchmarks are presented to evaluate computational efficiency, controlled selection, and the impact of calibration on forecasting performance.

Critical strengths
  1. The paper identifies an important distinction between forecast evaluation under a fixed calibration pool and the expected performance after fresh calibration. The counterexample in Section 2 provides a clear demonstration that increasing the number of evaluation queries does not necessarily resolve the misranking induced by reusing a calibration pool. This motivates a meaningful assessment target that accounts for the randomness of future calibration.
  2. The label-complexity analysis clarifies that computing an exact complete-subset estimator does not eliminate the statistical cost of determining which candidate is better. The lower-bound constructions and concentration-based selection rules provide a complementary perspective on when reliable forecast selection is feasible.
Critical weaknesses
  1. The controlled-selection guarantee relies on restrictive assumptions and they are difficult to verify for independently trained forecasting models, particularly in time-series settings where temporal dependence and distribution shift are common.
  2. The effectiveness of the proposed selection framework is not convincingly demonstrated for fitted forecasting models. The main positive controlled-selection results in Section 5.2 are obtained using fixed analytic predictors rather than learned models. In the separate fitted-candidate experiments reported in Appendix B.5, all five confidence-based selection rules make zero updates across 256 fitted cases, even with relatively large selection budgets. Moreover, the chronological experiments in Appendix F show that the tested selection procedures have worse aggregate final interval scores than the fixed quantile-regression controls across all four source-delay settings.
Questions for authors
  1. What is the practical advantage of computing the exact complete-U statistic rather than using independent-block or incomplete-U estimators under the same label and computational budgets?
  2. What factors explain the different attenuation behaviors observed across the hourly, subhourly, and daily forecasting groups?
  3. Since the motivating applications largely concern time-series forecasting, can the proposed risk estimation or confidence-based selection framework be extended to temporally dependent observations under suitable mixing or stability assumptions?
1 comment

Thank you for asking how the uncertainty calculation connects to actual choices.

W2. What happens with fitted forecasts? On 153 series we selected on 27 origins, calibrated on 13 and scored on 12. The asymptotic jackknife threshold switched on 40 series (36 improved, four worsened); unguarded averaging switched on 110 (73 improved, 37 worsened). Against keeping the reference, cautious mean loss changed by −.09455 (whole-dataset resampling interval [−.1811, −.0260]). Against unguarded averaging the difference was +.01386 [−.03030, .06634], leaving its direction unresolved. A single split had the lowest mean. With the size-20 pool reused for selection, the threshold helped 48 and hurt seven. Loss changes versus keeping were −.10717 for the threshold, −.10912 for always local and −.12914 for unguarded averaging. The threshold gave .02197 higher mean loss than unguarded averaging. Its improvement over keeping survived every dataset omission, although ETTh2 alone worsened. These series are dependent (Table 3a; Appendix P.3).

Q1. What does exact averaging buy at the same budget? Hold the N labelled observations fixed. A randomly drawn block estimate averages to the complete statistic over its possible splits. Complete averaging therefore removes split noise. Population estimation error still remains. The same decomposition holds for uniformly sampled incomplete U-statistics. Smaller estimation error alone does not order the risk of the forecast chosen by minimization (Theorem 3).

In each of 72 analytic settings both estimates used the same 32 independent selection draws and fixed functions. The median disjoint-block/complete variance ratio was 2.30 across 36 Gaussian settings and 1.69 across 36 Student-t settings. One empirical ratio fell below one. Appendix P reports the full ranges and separates the synthetic fitted study, where variability across fits cannot be identified as within-fit variance.

Time changes the choice. On 48 mean-only cells, disjoint blocks took .43 ms and complete prefix averaging 1.28 ms (medians). Incomplete averaging with N sampled blocks took 11.098 ms. Time-matched incomplete averaging took 1.333 ms and had median approximation RMSE .8651, measured on the vector of risks. For scale, the median within-cell mean absolute risk was 8.617; the corresponding absolute alternative-minus-first score contrast was .139. The complete pass also supplies the exact deletion values for paired inference (Tables 2, 51–52).

Q2. Why do the frequency groups differ? Calibration changes width, miss frequency and miss distance. The hourly raw score contrast separates into −.01685 from width and −.09759 from miss penalties. After calibration these become −.05484 and −.01617. Width explains most of the remaining hourly advantage: .6205 of the raw score contrast survives. On subhourly data the local intervals are narrower, but miss penalties outweigh this and the local average score is worse before and after calibration. In the corrected secondary daily analysis, retention is .9978 and Crypto supplies the favorable contrast. Removing Finland Traffic reverses both subhourly contrast signs. Frequency, scale, dataset mix and fit quality change together, so the decompositions do not establish a causal frequency effect (Section 5.3, Appendix M).

W1 and Q3. What survives time dependence or shift? With a specified ordered deployment episode, its final observation is the query. Earlier observations may be dependent. Independent episodes permit an ordinary block-level bound. Separating stationary episodes by a gap g with a known absolute-regularity coefficient adds at most (J−1)β(g) to the failure budget for J episodes. The single last query pays the 1/α range. Alternatively, if the deployment episode is within total variation τ of the independent reference, a contrast bounded by M changes by at most 2Mτ. Shift of the selection law also needs its own allowance. The observed series do not supply those certificates (Appendix P.1).

Section 4 also needs IID selection rows, a common bounded scale and bounded endpoint displacement. A bounded update can supply the last condition by design. In the original sequential synthetic fit five finite rules kept the reference in all 256 cases. Joint least squares yielded 38 complete Bernstein switches in 128 comparisons at the largest budget and none at smaller budgets. The same fits recur (Tables 53–55). Algorithm 1 was tested on synthetic data. Fixed quantile regression beat all four unguarded Appendix F selectors. All 34 eligible windows had 32 selection rows for final pools of at least 59. In a separate size-nine study, complete averaging beat one reused pool in three of four settings and the fixed quantile forecast in one of four (Table 35).

Reviewer_a7hMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies forecast selection when the chosen predictor undergoes fresh calibration before deployment. The authors show that reusing one calibration pool can select a worse forecast, regardless of the number of evaluation queries. The authors propose an exact algorithm to compute the complete-subset risk estimate and all delete-one estimates in O(ANlog⁡N)O(AN\log N) time. They also establish a sharp paired score envelope and derive statistical guarantees and label-cost bounds for reliable selection. Their synthetic experiments demonstrate speedups and beneficial controlled updates under unbounded response distributions. Their real-world experiments show that fresh calibration substantially reduces a matched fitting gain, from 5.34% to 1.52% on nine public forecasting datasets.

Critical strengths
  1. I think the paper studies an interesting and practically relevant problem of forecast selection under fresh calibration. The authors clearly motivate the importance of evaluating forecasts after the calibration they will actually receive. They also provide a theoretical construction showing that reusing one calibration pool can lead to persistent selection errors.
  2. The authors develop an exact algorithm for computing the complete-subset risk estimate and all delete-one estimates in O(ANlog⁡N)O(AN\log N) time. They also establish a sharp paired score envelope and derive statistical guarantees and label-cost bounds for reliable selection. These results provide useful tools for calibration-aware forecast comparison.
  3. The authors conduct computational benchmarks, synthetic experiments, and real-world forecasting studies. The experiments demonstrate substantial computational speedups and the usefulness of the paired score envelope for controlled updates. The real-world results also provide interesting evidence that fresh calibration can substantially attenuate improvements in fitted forecasting models.
Critical weaknesses

Perhaps the first convern is that the statistical guarantees for reliable selection require globally nearby endpoints, a common bounded scale, and IID selection observations independent of fitting. These assumptions appear rather restrictive for practical forecasting applications, especially when comparing substantially different predictors or working with dependent time-series data. It is also unclear how the required global endpoint bounds can be certified for complex forecasting models. The authors should clarify the practical applicability of their guarantees and discuss whether these assumptions can be relaxed.

Another concern is that although the synthetic experiments demonstrate beneficial controlled updates, they use fixed analytic predictors rather than trained forecasting models. The real-world experiments mainly show that fresh calibration attenuates fitting gains, but do not directly demonstrate that the proposed selection algorithm improves forecast selection. Moreover, the paper reports that the confidence-based rules make no updates in a separate fitted-predictor experiment. It would be helpful to see stronger empirical evidence that the proposed approach leads to meaningful improvements over existing selection methods in realistic forecasting settings.

Questions for authors

N/A

1 comment

Thank you for asking where the assumptions and empirical benefits extend.

W1. Which assumptions can a practitioner relax? An exact average over the given data can be computed for any fixed pair of interval predictors. Their endpoints can be far apart and the rows can be dependent. Interpreting its expectation as risk after a fresh independent calibration sample requires selection rows drawn independently of each other and of fitting. Pointwise asymptotic delete-one inference avoids the endpoint-distance condition if the scale is bounded, second moments exist and the first-order variance is positive. Growing pools require further local regularity and a nondegenerate limit (Section 3 and Appendix O).

The finite-sample rule in Section 4 needs IID selection rows independent of fitting as well as a common bounded scale and a global bound on endpoint changes. One can certify that last condition by design, for example by fitting a bounded correction to a deployed forecaster. A maximum on a finite validation set cannot certify it. Under signed correction a constant shift of standardized radius cancels (Appendix B). Appendix P.1 also allows dependence within an ordered episode when episodes are independent or a between-episode mixing allowance is known. Designating the final observation as the query costs a range of order 1/α. The public series here supply neither episode certificate.

W2. Does selection work with actual fitted forecasts? We added a retrospective comparison on 153 series with three separate blocks: 27 earlier origins to choose, 13 later origins to calibrate, and the final 12 to score. The cautious delete-one rule switched to the local predictor on 40 series. It helped 36 and hurt four. Unguarded averaging helped 73 and hurt 37 in 110 switches. Against keeping the reference, mean loss changed by −.09455 for the cautious rule, −.10583 for always using local, −.10841 for unguarded averaging and −.11617 for a single selection split. The whole-dataset resampling interval for the cautious rule versus keeping was [−.1811, −.0260]. Its mean loss difference from unguarded averaging was +.01386 [−.03030, .06634], leaving the direction unresolved (Appendix P.3).

A second comparison retains a 20-origin calibration pool drawn from the 40 older selection origins. Here the cautious rule helped 48 and hurt seven, giving a mean change of −.10717 from keeping the reference (−1.56%). Always switching had a mean change of −.10912 but hurt 50 series, and unguarded averaging had −.12914 and hurt 27. The cautious rule had .02197 higher mean loss than unguarded averaging. Removing any single dataset preserved its improvement over keeping the reference; ETTh2 alone had a higher mean score. Both designs use previously studied, time-ordered series and a pointwise asymptotic threshold. Neither estimates a guaranteed future error rate.

The finite-sample rule was also tested on newly fitted synthetic predictors. Sequential fitting had produced small adjustments and five rules kept the reference in all 256 cases. Joint least squares gives larger adjustments with a certifiable bound. At the two smaller budgets neither Bernstein rule switched. With 40,960 or 81,920 labels, complete and block Bernstein switched in 38 and 30 of 128 comparisons; the same fits recur across comparisons. Always using the fitted adjustment gave both favorable and adverse condition means, while complete Bernstein’s means ranged from −.01568 to zero (Tables 53–55). This separates the stronger guarantee, which used many labels, from the empirical risk trade-off seen in the actual forecasts.

Comments on the paper

1 comment

Thank you very much to all the reviewers. Whether comparing forecasts after recalibration changes a real choice, and whether a cautious choice pays off. We have added a test on 153 public series in which the three roles use separate observations. The first 27 origins select a forecast, the next 13 calibrate it and the final 12 score it. The asymptotic jackknife threshold, using the delete-one variance in Section 3, switched away from the reference on 40 series. Later interval loss fell on 36 and rose on four. Unguarded complete averaging switched on 110 series, helping 73 and hurting 37. The cautious rule avoided 33 harmful switches and passed up 37 helpful ones. Its mean loss fell by .09455 against keeping the reference (whole-dataset resampling interval [−.1811, −.0260]). Against unguarded averaging the mean difference was +.01386 [−.03030, .06634], leaving its direction unresolved. Of five rules, a single chronological selection split had the lowest average loss in this comparison (Section 5.3, Table 3a; Appendix P.3).

A second comparison uses a larger calibration pool. Its final 20 calibration origins also enter the 40-origin selection set, while all 12 scoring origins are later. The cautious rule helped 48 series and hurt seven. Against keeping the reference, mean loss changed by −.10717 (−1.56%) for the threshold, −.10912 for always using local and −.12914 for unguarded averaging. Unguarded averaging helped 90 and hurt 27. The threshold’s mean was .02197 higher. It improved on keeping the reference after each one-dataset omission, although ETTh2 itself had higher loss. These two retrospective comparisons distinguish avoiding harmful switches from obtaining the lowest average loss. Temporal dependence precludes a nominal error-rate guarantee here.

Table 1 gives the ranking reversal in decimals; Table 3a shows all five choices and their later scores. Sections 3–4 identify the calculation and its inference assumptions. Complete cross-validation and the subsample-quantile weights are classical. Two breakpoints in the interval score let us compute the mean and every fixed-size deletion after one sort. The finite-sample switching rule additionally needs independent selection rows, a common bounded scale and a bound on endpoint movement. Holding the labels, predictors and confidence rule fixed, the tighter range alone changes zero favorable settings to 12 of 72 in the controlled test (Table 3).

The two-machine software comparison is in Appendix P. The nine-dataset fitting comparison in Section 5.3 still shows that a 5.34% raw gain shrinks to 1.52% after calibration. Together the results show why the comparison should use the recalibrated intervals that will be issued, and what a forecasting team gives up when it asks for a more cautious change.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.