acceptodds
Under review as a conference paper at ICLR 2027

Bellman conformal inference: Calibrating prediction intervals for time series

Abstract

We introduce Bellman Conformal Inference (BCI), a framework that wraps around any time series forecasting model and provides calibrated prediction intervals. Unlike existing methods, BCI is able to leverage multi-step ahead forecasts and explicitly optimize the average interval lengths by solving a one-dimensional stochastic control problem (SCP) at each time step. In particular, we use the dynamic programming algorithm to find the optimal policy for the SCP. We prove that BCI achieves long-term coverage under arbitrary distribution shifts and temporal dependence, even with poor multi-step ahead forecasts. In addition, we prove that BCI is near optimal for its surrogate model predictive control problem when the raw forecast intervals are approximately calibrated. We find empirically that BCI avoids uninformative intervals with infinite lengths and generates substantially shorter prediction intervals when the raw forecast intervals are poorly calibrated.

Primary area:
learning on time series and dynamical systems
Keywords:
conformal prediction, time series, uncertainty quantification, online learning, distribution-free inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1YgBOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Bellman Conformal Inference (BCI), a framework that uses multi-step forecasts to improve prediction interval efficiency while controlling long-run average miscoverage in time series. BCI wraps an existing forecasting model, estimates the behavior of forecast errors through an empirical distribution of historical uncalibrated PIT values, and combines this distribution with future interval-length curves to formulate a finite-horizon stochastic control problem. The objective trades off interval length against a penalty for excessive cumulative miscoverage. A Bellman recursion uses the number of errors accumulated within the planning horizon as its state to select the current nominal miscoverage level. The method executes only the current action, updates the miscoverage penalty weight after observing the outcome, and replans. The paper establishes long-run average coverage and, under additional assumptions, studies marginal coverage, approximate optimality for a surrogate planning objective, and interval-length comparisons with ACI in a particular setting. Experiments assess calibration and interval efficiency on stock returns, volatility, and search popularity forecasting tasks.

Critical strengths
  1. The paper addresses a well-motivated question about the trade-off between coverage and efficiency in online interval calibration. Incorporating future interval-length curves into the decision allows the method to account for differences in the cost of obtaining coverage across time. This provides a useful modeling perspective on allocating a miscoverage budget, extending approaches that primarily adjust the nominal level in response to past errors.

  2. The framework separates coverage control from the accuracy of the forecasting model. Under the stated structural assumptions on the interval family, the long-run average coverage guarantee relies on online feedback and boundary safeguards, without requiring a correctly specified forecaster or accurate planning distributions. This makes the approach compatible with different forecasting models while retaining an explicit validity guarantee under model misspecification and distribution shift.

  3. The paper examines long-run average coverage, marginal coverage under additional conditions, and interval efficiency, and demonstrates compatibility with both neural forecasting models and GARCH. The empirical discussion also considers the calibration of the original intervals, the planning horizon, and the occurrence of infinite intervals. These analyses help identify conditions under which the method is useful and explain why its gains vary across forecasting problems.

Critical weaknesses
  1. The decision spaces in the optimization problem, Bellman recursion, and efficiency theorem are not consistently specified. Equation (4) and Theorem 3.3 optimize an action vector fixed at the planning time, and Appendix E explicitly compares objectives over this vector space. In contrast, the recursion in Equations (20)–(22) optimizes a feedback policy whose future actions depend on the accumulated error state. These problems generally have different optimal values and can produce different current actions. The discrepancy also affects the proof: Appendix E cancels the interval-length term between the two objectives because the actions are fixed. Under a feedback policy, future actions and length costs depend on the simulated history, so changing the disturbance law can also change expected length costs. As written, the analysis therefore does not establish that DP exactly solves Equation (4), or that Theorem 3.3 applies to the implemented algorithm. A consistent policy formulation and corresponding proof are needed.

  2. The surrogate planning model is not sufficiently connected to the interval efficiency achieved during actual operation. BCI uses current multi-step interval lengths as proxies for the lengths obtained after future reforecasting, and models future errors as independent draws from an empirical distribution of recent one-step PIT values. Both approximations can distort the allocation of miscoverage under substantial distribution shift or miscalibration. Theorem 3.3 holds the future length functions fixed and controls the surrogate objective error caused by replacing the PIT distribution; a small bound also requires approximately conditionally uniform PITs. It does not control errors in the anticipated length curves or establish a cumulative length guarantee for the receding-horizon procedure. Since the largest empirical gains occur when the original intervals are poorly calibrated, the theory provides limited explanation of this central benefit. Direct analysis or controlled experiments are needed to assess sensitivity to inaccurate lookahead information and temporal dependence in forecast errors.

  3. The independent value of Bellman planning over simpler online cost optimization remains insufficiently established. The long-run coverage guarantee follows from the penalty update and boundary safeguards and does not depend on the particular internal optimizer. The strict length advantage in Appendix F concerns a KL-regularized variant and requires sufficiently strong regularization and specific relations between the methods' step sizes, whereas the main experiments use no regularization. This comparison does not isolate the benefit of lookahead from regularization and feedback strength. Although the paper discusses horizon sensitivity, including a single-step configuration, it does not provide sufficiently detailed matched comparisons holding the historical PIT estimator, feedback rule, and tuning budget fixed. In particular, it remains unclear how much is gained from genuine future length information relative to single-step optimization or planning with the current length curve repeated across the horizon. The attribution of the reported efficiency gains to the central new mechanism is therefore incomplete.

Questions for authors
  1. Appendix B.2 states that the empirical-CDF optimization can be solved exactly by searching historical PIT values. However, the miscoverage convention uses an action greater than or equal to the PIT, so the empirical CDF jumps upward at these values. With a continuous decreasing length function, the infimum may be approached immediately to the left of a jump without being attained at the jump itself. How is this handled in the implementation? Does the solver use left-limit evaluations or approximately optimal actions, and how is its optimization tolerance determined?

  2. The formal conditions for Theorem 3.2 include a transition density bounded below uniformly over past states on compact outcome sets, bounded predictive location and scale, and interior actions separated from the boundaries. Which experimental forecasters and data-generating settings satisfy these conditions? When forecasting parameters are continually refitted, is the joint state in Appendix D, consisting of a finite observation window, PIT history, and penalty weight, still a time-homogeneous Markov process, or must the fitted parameters and training state also be included?

  3. Appendix A.3 selects step sizes using the full evaluation sequence to match ACI's local coverage variability. If hyperparameters are instead chosen on an earlier temporal validation segment and then frozen, or updated through a strictly prospective rule, does the coverage–length trade-off persist on subsequent observations? Could the authors report comparisons with ACI and PID under such a protocol to clarify the gains attainable in online deployment?

4 comments

We thank the reviewer for the insightful comments. Please see our detailed responses to weaknesses W1-W3 and questions Q1-Q3

**W1: The decision spaces in the optimization problem, Bellman recursion, and efficiency theorem..."

Our response: We use (4) to motivate the objective function. The notation min⁡αt∣t,…,αt+T−1∣t\min_{\alpha_{t|t}, \ldots, \alpha_{t+T-1|t}} in (4) is for illustration and it does not mean BCI literally optimizes αt∣t,…,αt+T−1∣t\alpha_{t|t}, \ldots, \alpha_{t+T-1|t} as TT scalars. We will remove the minimization in (4) and emphasize that this is the objective function we will be using for the stochastic control problem. We agree that Theorem 3.3 in our submitted version did not apply to the implemented algorithm. In the revision, we have updated it for the actual DP algorithm we used. The conditions remain the same and the bound only differs by a constant. We thank the reviewer for pointing this out. Below is a sketch. Setting. Fix a planning horizon T≥1T\ge 1, a window size B≥4B\ge 4, a time t≥B+1t\ge B+1, and a target miscoverage rate αˉ∈(0,1)\bar{\alpha}\in(0,1). Let Pt⋆P_t^\star be the true joint law of the next TT uncalibrated PITs (βt,…,βt+T−1)(\beta_t,\ldots,\beta_{t+T-1}) conditional on the history Ft−1\mathcal{F}_{t-1}. BCI plans under Pt(B)=(μ^t(B))⊗TP_t^{(B)}=(\widehat{\mu}_t^{(B)})^{\otimes T}, which draws each future PIT independently from the empirical distribution of the past BB PITs.

Assumptions. Let Gs∘(u)=Pr⁡(βs≤u∣Fs−1)G_s^\circ(u)=\Pr(\beta_s\le u\mid\mathcal{F}_{s-1}). Assume that, for every s≥1s\ge 1 and u,v∈[0,1]u,v\in[0,1],

sup⁡u∈[0,1]∣Gs∘(u)−u∣≤κs,∣Gs∘(u)−Gs∘(v)∣≤M∣u−v∣,\sup_{u\in[0,1]}|G_s^\circ(u)-u|\le\kappa_s,\qquad |G_s^\circ(u)-G_s^\circ(v)|\le M|u-v|,

where M>0M > 0 and (κs)(\kappa_s) is deterministic, nonnegative, and nonincreasing. Assume also that each planned interval-length curve satisfies Ls∣t≥0L_{s\mid t}\ge 0 and Ls∣t(1)=0L_{s\mid t}(1)=0, that ∣λt∣<∞|\lambda_t| < \infty, and that τ≥0\tau\ge 0 with K(⋅,αˉ)K(\cdot,\bar{\alpha}) nonnegative and finite.

Theorem 3.3 (efficiency of empirical PIT planning). Let π^t\widehat{\pi}_t minimize the empirical Bellman objective at every stage and error count, as in (26). Let JtP(π)J_t^{P}(\pi) denote the expected planning cost in (4) under law PP, and let Πt\Pi_t be the class of policies whose actions depend on the stage and the cumulative error count. For every δ∈(0,1)\delta\in(0,1), with probability at least 1−δ1-\delta,

0≤JtPt⋆(π^t)−inf⁡π∈ΠtJtPt⋆(π)≤4∣λt∣(log⁡(B/δ)+MB+κt−B).0\le J_t^{P_t^\star}(\widehat{\pi}_t)-\inf_{\pi\in\Pi_t}J_t^{P_t^\star}(\pi)\le 4|\lambda_t|\left(\frac{\sqrt{\log(B/\delta)}+M}{\sqrt{B}}+\kappa_{t-B}\right).

W2: The surrogate planning model is not sufficiently connected to the interval efficiency achieved during actual operation...

**Our response: ** We would like to emphasize that the validity of BCI (i.e., average coverage control) does not require independence of βt\beta_t, empirical distribution of recent PIT values to be correctly specified, or the length function of multi-step ahead forecasting intervals to be good approximations. They are simply a working model inside the optimizer. Theorem 3.1 allows arbitrary temporal dependence, distribution shift, or poor multi-step ahead forecasting intervals, as long as Assumption 2.1 holds for the nominal prediction intervals, which is almost always the case by design. This is similar in spirit to the broader conformal inference literature, where a working model is used without needing to assume it is correct.

Theorem 3.3 investigates the efficiency when the working model described above is accurate. It does not invalidate the validity claim in Theorem 3.1. This is also a standard exercise in the conformal inference literature which often shows the method is efficient with a good working model.

AuthorsOpenReview ↗

(Continued)

We thank the reviewer for their comments and questions. Q1. The empirical CDF and the DP. For the empirical CDF jumps, we use the left-limit evaluations. For other cases where the minimum is not attained, we select a solution whose objective value is within ϵ\epsilon of the infimum in (4), for a sufficiently small ϵ>0\epsilon>0.

Q2. Conditions of Theorem 3.2 under continual refitting. We will walk through each of the assumptions, Assumptions D.1-D.4 in Appendix D.2, for Theorem 3.2 and talk about the requirements for the forecaster and data-generating distributions. Assumption D.1 requires the true transition density to be bounded from below. This is a common regularity assumption. Assumption D.2 covers most practical prediction intervals, which are based on estimated conditional mean and variance. For the forecaster, we have added a sentence above Theorem 3.2 and a more detailed remark below Assumption D.2 showing that it allows for rolling-window fit of the underlying model (namely the finite observation window you mentioned), which is a common approach in practice. Assumption D.3 is automatically satisfied when a finite grid of α\alpha is used (e.g., {0.01, 0.02, …, 0.99}). Assumption D.4 is a regularity condition that avoids extreme distributional shifts.

See the updated paper for the details.

Q3: tuning on an earlier segment and freezing. We re-ran the comparison with BCI's c and PID's (η, K_I) tuned on the first 60% of each series (matched to ACI's 50-day rolling standard deviation of miscoverage for the first 60% of the data, with ACI's γ fixed), then frozen and evaluated on the last 40%. BCI uses T = 5 for returns and Google Trends and T = 3 for volatility. Reduction = 100(1 − BCI/method).

DatasetγAvg. length (BCI / ACI / PID)Reduction vs ACI / vs PID (%)Miscoverage % (BCI / ACI / PID)Frac. ∞ % (BCI / ACI / PID)
Return-AMD0.10.1437 / 0.1487 / 0.1541+3.3 / +6.89.96 / 9.96 / 10.396.42 / 12.31 / 0.00
Return-AMD0.0080.1388 / 0.1605 / 0.1380+13.5 / −0.610.39 / 9.74 / 10.60*0.54 / 0.00 / 0.00
Return-Amazon0.10.0997 / 0.1291 / 0.1295+22.8 / +23.010.03 / 9.65 / 10.181.99 / 17.76 / 0.00
Return-Amazon0.0080.0948 / 0.1388 / 0.0871+31.7 / −8.810.11 / 9.65 / 10.110.23 / 0.00 / 0.00
Return-Nvidia0.10.1261 / 0.1437 / 0.1473+12.2 / +14.410.16 / 10.06 / 10.1613.09 / 11.91 / 0.00
Return-Nvidia0.0080.1270 / 0.1536 / 0.1214+17.3 / −4.610.06 / 9.38* / 10.55*12.60 / 0.00 / 0.00
Volatility-AMD0.19.3077 / 10.1724 / 9.8340+8.5 / +5.410.01 / 10.01 / 10.013.68 / 14.96 / 12.16
Volatility-AMD0.0087.6612 / 7.4855 / 8.2249−2.3 / +6.99.91 / 10.21 / 10.060.00 / 0.00 / 3.12
Volatility-Amazon0.14.8189 / 8.1797 / 7.3490+41.1 / +34.49.95 / 9.91 / 10.000.00 / 13.92 / 10.67
Volatility-Amazon0.0084.2562 / 4.1651 / 4.3483−2.2 / +2.110.00 / 9.95 / 9.910.00 / 0.00 / 0.25
Volatility-Nvidia0.16.6720 / 9.4444 / 7.9661+29.4 / +16.210.01 / 10.01 / 10.060.00 / 13.90 / 7.24
Volatility-Nvidia0.0085.9586 / 6.1048 / 6.5275+2.4 / +8.710.16 / 10.11 / 10.110.00 / 0.00 / 1.36
Google Trends0.14.0574 / 5.0930 / 4.6730+20.3 / +13.29.76 / 9.92 / 9.924.48 / 14.88 / 0.00
Google Trends0.0083.9281 / 5.0074 / 3.7725+21.6 / −4.19.92 / 10.40 / 10.56*0.00 / 0.00 / 0.00

*Mean miscoverage outside 10 ± 0.5%. PID follows the main tables' rule: the selected (η, K_I) is kept unless it produces exploding intervals on the tuning rows, in which case K_I is capped at the next grid value below and the selection is repeated; this applied to 4 of the 14 settings. Unbounded intervals are charged the observed outcome range (square-root scale for volatility).

BCI is shorter than ACI in 12 of 14 settings and than PID in 10 of 14. The two losses to ACI are small and occur under loose control (about −2%). The four losses to PID also all occur under loose control. BCI's miscoverage stays within 10 ± 0.5% in all 14 settings, while PID is outside the band in three of its four wins, so those wins are partly bought with lower coverage. BCI's share of infinite intervals is highest on Return-Nvidia (about 13% at both γ), which we regard as a limitation.

W3: value of Bellman planning. Appendix F.2 now shows that BCI with τ = 0 has shorter intervals than ACI under miscalibration, without stationarity or KL regularization. For the value of looking ahead, we compared the paper's horizon with T = 1 (the same empirical-PIT estimator, feedback rule and tuning protocol), re-tuning c for each T and matching to ACI. Reduction in interval length vs ACI (γ = 0.1):

DatasetT = 1T = 3T = 5
Return-AMD−1.9%−1.3%+2.7%
Return-Amazon+19.2%+18.4%+18.7%
Return-Nvidia−15.5%−2.2%+9.5%
Volatility-AMD+10.0%+11.1%+12.2%
Volatility-Amazon+33.5%+34.5%+32.0%
Volatility-Nvidia+30.8%+31.7%+27.5%
Google Trends−2.1%+8.5%+14.6%

Longer horizons also produce more full-space intervals at γ = 0.1.

Reviewer_8RTDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a conformal approach to time-series forecasting based on model predictive control, entitled Bellman Conformal Inference (BCI), that adaptively calibrates prediction intervals. For this, BCI formulates the problem as a stochastic control system that simulates the process over the next T steps and then moves backward along the horizon, trading off interval efficiency and coverage via a dynamic penalty. Then it solves this optimization problem using dynamic programming and updates the penalty term like ACI. BCI provides formal guarantees for long-run coverage. Experimental results on various univariate time series data show that BCI yields shorter intervals than ACI and conformal PID, particularly when the base intervals are poorly calibrated.

Critical strengths
  • A new formulation of conformal calibration as a model predictive control problem over multi-step interval forecasts.
  • The theoretical guarantees include long-run coverage, marginal coverage results, and an efficiency bound.
  • BCI keeps the model-agnostic property of the conformal approach by showing that it can be wrapped around classical and modern forecasters such as GARCH and transformers/LSTM.
Critical weaknesses
  • BCI relies on a strong assumption of full-space intervals when the penalty term exceeds the maximum, which can yield uninformatively large and potentially infinite intervals in the worst cases.
  • The theoretical justification for the assumption of independence over a finite-horizon objective requires approximate calibration and Lipschitz conditions, which may not hold when nominal intervals are very poor.
  • The paper does not consider more recent approaches to online conformal prediction such as OLCP-Hedge [1], RCCP [2], AcMCP [3], and AgACI [4].

[1] Lai, Yuheng, Raskutti, Garvesh (2026). Online Localized Conformal Prediction. arXiv:2605.05497

[2] Jin, S., Kim, K., Lee, J., & Lee, Y. (2026). Retrieval-corrected conformal prediction for time series. arXiv:2608.10553.

[3] Wang, X., & Hyndman, R. J. (2024). Online conformal inference for multi-step time series forecasting. arXiv:2410.13115.

[4] Zaffran, M., Feron, O., Goude, Y., Josse, J.; Dieuleveut, A.. (2022). Adaptive Conformal Predictions for Time Series. 39th International Conference on Machine Learning.

Questions for authors
  1. What would be the results of BCI compared to AgACI and OLCP-Hedge using the electricity time series data?
  2. What would be the formal guarantee of the Markov-based marginal coverage result, or at least the underlying assumptions and implications of the current informal statement?
4 comments

We thank the reviewer for the insightful comments. Please see our detailed responses to weaknesses W1-W3 and questions Q1-Q2

Weakness 1: BCI relies on a strong assumption of full-space intervals when the penalty term exceeds the maximum,...

Our response: Any method that guarantees long-term coverage against arbitrary adversarial outcomes, including ACI and its variants (Gibbs & Cand` es, 2021; Gibbs & Candes, 2022; Angelopoulos et al., 2023), must allow the possibility of full-space intervals. Otherwise, after observing each generated interval, an adversary could choose YtY_t outside it, giving errt=1err_t = 1, for every tt and violating (1). To formalize the statement, we have added Proposition C.1 in Appendix C.2 in the revision. Below we give a sketch.

Setting. Let Y⊆R\mathcal{Y}\subseteq\mathbb{R} be the outcome space and let αˉ∈(0,1)\bar{\alpha}\in(0,1) be the target miscoverage rate. At each time tt, a generates an interval Ct(1−β)C_t(1-\beta). An adversary observes the realized interval before choosing Yt∈YY_t\in\mathcal{Y}.

Proposition C.1 (necessity of full-space intervals). Suppose the procedure guarantees, against every such adversary,

lim sup⁡K→∞1K∑t=1K1Yt∉It≤αˉalmost surely.\limsup_{K\to\infty}\frac{1}{K}\sum_{t=1}^{K}\mathbf{1}\\{Y_t\notin I_t\\}\le\bar{\alpha}\quad\text{almost surely}.

Then there exists an adversary under which

lim inf⁡K→∞1K∑t=1K1Y⊆It≥1−αˉalmost surely.\liminf_{K\to\infty}\frac{1}{K}\sum_{t=1}^{K}\mathbf{1}\\{\mathcal{Y}\subseteq I_t\\}\ge 1-\bar{\alpha}\quad\text{almost surely}.

Weakness 2: The theoretical justification for the assumption of independence over a finite-horizon objective ...

Our response: We would like to emphasize that independence is just a working model inside the optimizer. Theorem 3.1 allows arbitrary temporal dependence, insofar as Assumption 2.1 holds for the nominal prediction intervals, which is almost always the case by design. Put another way, even if βt\beta_t is not conditionally independent, BCI still guarantees average coverage control. This is similar in spirit to the broader conformal inference literature, where a working model is used without needing to assume it is correct.

The choice of i.i.d. βt\beta_t has two benefits. First, Theorem 3.2 shows that if the raw intervals are indeed well-calibrated, this choice is approximately optimal. Second, it makes the stochastic control problem easy to solve, as described in Appendix B.1.

​​I appreciate the authors' clarifications. However, they don't fully address my concerns or answer my questions. Therefore, I maintain my score.

Our apologies for the incomplete response. Regarding W3 and Q1, the revised PDF adds AgACI, DtACI, OLCP-Hedge and AcMCP on all eight series, including electricity (ELEC2). The matched comparisons are in Section 4.3 and Tables 3–6 (Appendix A.2). We did not compare with RCCP because it does not have a coverage guarantee. A comparison with ACI and PID using hyperparameters frozen from the first 60% of each series, with no matching on the test segment, is in our response to 1YgB.

AuthorsOpenReview ↗

Q2 Markov-based marginal coverage

Q2. The formal guarantee and assumptions are in Appendix D (the main text gives an informal version because of the length limit). The assumptions are Assumptions D.1–D.4 in Appendix D.2: D.1 is a common condition on the density; D.2 covers most practical prediction intervals, and Remark D.1 shows that it allows rolling-window fitting of the underlying model; D.3 is automatically satisfied when a finite grid of α is used (e.g., {0.01, 0.02, …, 0.99}); D.4 is a regularity condition that avoids extreme distributional shifts. The formal statement is Theorem D.1 in Appendix D.3.

Reviewer_5vPPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Bellman Conformal Inference (BCI) is a framework designed to calibrate time series forecasts while optimizing prediction interval lengths through a stochastic control problem solved at each time step using dynamic programming. Compared to adaptive conformal inference (ACI) and Conformal PID, the authors empirically show that BCI can produce shorter prediction intervals, particularly when the underlying forecasts are poorly calibrated. The paper also provides theoretical guarantees on long-term coverage, marginal coverage under Markov assumptions, and near-optimality of the stochastic control formulation.

Critical strengths

S1. The intuition behind considering multiple steps in the calibration process is compelling, and its potential efficiency benefits are well argued. The authors' formulation of the multi-step problem as a predictive control problem is sound, and developing an exact solution algorithm via dynamic programming is practical for short prediction horizons (e.g., T=3).

S2. The presentation of results through metrics and plots is informative, highlighting the advantages of BCI. Empirically, BCI can produce shorter prediction intervals than ACI and Conformal PID. The sensitivity analysis on the prediction horizon strengthens the experimental evaluation.

S3. The paper provides additional theoretical results characterizing the behavior of BCI, including marginal coverage under Markov assumptions (Theorem 3.2) and near-optimality of the stochastic control formulation (Theorem 3.3). These results strengthen the theoretical understanding of the proposed framework.

Critical weaknesses

W1. While the theoretical results beyond long-term coverage guarantees are valuable, a gap remains between these guarantees and the main empirical advantages of BCI. In particular, Theorem 3.3 establishes the near-optimality of the surrogate stochastic control problem under the assumption of approximately calibrated conditional PIT distributions. However, the empirical advantages of BCI are most pronounced when the underlying forecasts are poorly calibrated. It is therefore unclear to what extent this result explains the efficiency improvements observed in practice. Moreover, the theorem does not directly establish improvements in the actual lengths of prediction intervals compared to alternative conformal methods.

W2. The practical significance of some theoretical results could be better explained. In particular, Theorem 3.2 establishes an asymptotic marginal coverage guarantee averaged over (q) consecutive steps. The additional assumptions required for this result are also restrictive. It would be helpful to clarify what this guarantee provides beyond the existing long-term coverage guarantee and under what practical conditions it is meaningful.

W3. The paper is sometimes challenging to follow. An illustration of the full pipeline introduced at the beginning of Section 3 and a symbol table in the appendix would significantly help comprehension. Algorithm 1 could also provide more details on the optimization procedure and its implementation.

Questions for authors

I would appreciate further clarification on the theoretical concerns raised in my review.

3 comments

We thank the reviewer for the insightful comments. Please see our detailed responses to your questions W1-W3.

W1. While the theoretical results beyond long-term coverage guarantees are valuable, a gap remains between these guarantees and the main empirical advantages of BCI ...

Our response: In the submitted version, we had an Appendix F (now Appendix F.1) showing that BCI with τ>0\tau > 0 has provably shorter intervals than ACI for well-calibrated nominal prediction intervals when both of them have stationary trajectories, as Gibbs and Candes (2023) and our Theorem 3.2 suggest.

In the revision, we have added Appendix F.2 that provides a class of examples where the unregularized BCI (τ=0\tau=0) has provably shorter intervals than ACI when the nominal prediction intervals are miscalibrated, without assuming their trajectories are stationary. We provide a sketch below.

Setting. Let αˉ\bar{\alpha} be the target miscoverage rate. Before predicting YtY_t, we observe VtV_t. The covariates are i.i.d., with VtV_t independent of the history Ht−1\mathcal{H}_{t-1}. At nominal miscoverage level aa, the interval is Ct(1−a)=C(1−a;Vt)C_t(1-a)=C(1-a;V_t). For each method m∈A,Bm\in\\{A,B\\}, let errtm=1Yt∉Ct(1−αtm)\mathrm{err}_t^m=\mathbf{1}\\{Y_t\notin C_t(1-\alpha_t^m)\\}.

Assumption F.4 (miscalibration). For every a∈[0,1]a\in[0,1],

\Pr\bigl\\{Y_t\notin C_t(1-a)\mid\mathcal{H}_{t-1},V_t\bigr\\}=G(a),

where GG is continuous and strictly increasing, with G(0)=0G(0)=0 and G(1)=1G(1)=1. Thus u=G(a)u=G(a) is the actual miscoverage probability. Define the interval length at actual miscoverage uu by

ℓv(u)=∣C(1−G−1(u);v)∣.\ell_v(u)=\bigl|C(1-G^{-1}(u);v)\bigr|.

ACI. With γA>0\gamma_A > 0 and fixed z1∈[0,1]z_1\in[0,1], set

αtA=min⁡1,max⁡0,zt,zt+1=zt+γA(αˉ−errtA).\alpha_t^A=\min\\{1,\max\\{0,z_t\\}\\},\qquad z_{t+1}=z_t+\gamma_A(\bar{\alpha}-\mathrm{err}_t^A).

BCI. BCI uses the true GG, a planning horizon TT, and τ=0\tau=0. When 0<λt<λmax⁡0 < \lambda_t < \lambda_{\max}, it chooses a policy to minimize

E[ℓVt(u0)+∑j=1T−1ℓt,jfuture(uj)+λt(1T∑j=0T−1Ej−αˉ)+].\mathbb{E}\left[\ell_{V_t}(u_0)+\sum_{j=1}^{T-1}\ell_{t,j}^{\mathrm{future}}(u_j)+\lambda_t\left(\frac{1}{T}\sum_{j=0}^{T-1}E_j-\bar{\alpha}\right)_{+}\right].

Here uju_j may depend on the number of earlier simulated errors, and EjE_j is a simulated error with conditional probability uju_j. BCI uses the first action utB=u0⋆u_t^B=u_0^\star. It sets utB=1u_t^B=1 when λt≤0\lambda_t\le 0 and utB=0u_t^B=0 when λt≥λmax⁡\lambda_t\ge\lambda_{\max}. With fixed λ1∈[0,λmax⁡]\lambda_1\in[0,\lambda_{\max}], set

αtB=G−1(utB),λt+1=λt+γB(errtB−αˉ).\alpha_t^B=G^{-1}(u_t^B),\qquad \lambda_{t+1}=\lambda_t+\gamma_B(\mathrm{err}_t^B-\bar{\alpha}).

Assumption F.5 (length curves). The map (v,u)↦ℓv(u)(v,u)\mapsto\ell_v(u) is jointly measurable. All current and future length curves are continuously differentiable, nonincreasing, and convex, with common endpoints ℓ(0)=W<∞\ell(0)=W < \infty and ℓ(1)=0\ell(1)=0. The future curves are deterministic and strictly convex.

Assumption F.6 (tuning parameters). Let MM be the supremum of −ℓ′(0)-\ell'(0) over all current and future length curves. Assume T≥2T\ge 2, 0<αˉ<1/T0 < \bar{\alpha} < 1/T, M<∞M < \infty, and

λmax⁡≥M1/T−αˉ,0<γB<λmax⁡.\lambda_{\max}\ge\frac{M}{1/T-\bar{\alpha}},\qquad 0 < \gamma_B < \lambda_{\max}.

Assumption F.7 (variation over time). The future length curves satisfy

∑j=1T−1sup⁡u∈[0,1]∣ℓt,jfuture(u)−ℓt−1,jfuture(u)∣⟶0.\sum_{j=1}^{T-1}\sup_{u\in[0,1]}\left|\ell_{t,j}^{\mathrm{future}}(u)-\ell_{t-1,j}^{\mathrm{future}}(u)\right|\longrightarrow 0.

Theorem F.2. Let VV have the common distribution of the VtV_t. Define L⋆L^\star as the shortest expected interval length attainable by choosing miscoverage as a function of VV, subject to average miscoverage αˉ\bar{\alpha}. Let Δ\Delta be the gain over using the same miscoverage αˉ\bar{\alpha} at every covariate value:

L⋆=inf⁡u:V→[0,1],;E[u(V)]=αˉE[ℓV(u(V))],Δ=E[ℓV(αˉ)]−L⋆.L^\star=\inf_{u:\mathcal{V}\to[0,1],\\;\mathbb{E}[u(V)]=\bar{\alpha}}\mathbb{E}[\ell_V(u(V))],\qquad \Delta=\mathbb{E}[\ell_V(\bar{\alpha})]-L^\star.

Under Assumptions F.4–F.7, both methods attain the target miscoverage, with no stationarity assumption on their trajectories:

1K∑t=1Kerrtm⟶αˉalmost surely,m∈A,B.\frac{1}{K}\sum_{t=1}^{K}\mathrm{err}_t^m\longrightarrow\bar{\alpha}\quad\text{almost surely},\qquad m\in\\{A,B\\}.

Let LKm=K−1∑t=1KE∣Ct(1−αtm)∣L_K^m=K^{-1}\sum_{t=1}^K\mathbb{E}|C_t(1-\alpha_t^m)| be the expected average interval length. Then

lim inf⁡K→∞LKA≥E[ℓV(αˉ)],lim sup⁡K→∞LKB≤L⋆+γB2Tαˉ(1−αˉ).\liminf_{K\to\infty}L_K^A\ge\mathbb{E}[\ell_V(\bar{\alpha})],\qquad \limsup_{K\to\infty}L_K^B\le L^\star+\frac{\gamma_B}{2T}\bar{\alpha}(1-\bar{\alpha}).

Consequently, BCI has strictly shorter asymptotic expected average length whenever

Δ>γB2Tαˉ(1−αˉ).\Delta > \frac{\gamma_B}{2T}\bar{\alpha}(1-\bar{\alpha}).

In particular, we develop low-level assumptions in Proposition F.1 to ensure Δ>0\Delta > 0. We also show in Corollary F.4 that the conclusion remains when GG is estimated by the historical βt\beta_t's.

AuthorsOpenReview ↗

(Continued)

W2. The practical significance of some theoretical results could be better explained...

Our response: We show in the end of Theorem 3.2 that when is drawn from the conditional stationary distribution given the past γ\gamma’s. It can be seen from the proof that this condition implies a stationary stochastic process, which is also assumed in Section 4.2 of the ACI paper (Gibbs and Candes, 2021).

Without this initial condition that γ1\gamma_1 is drawn from the stationary distribution, we are unaware of any analysis of expected coverage when YtY_t is stationary. The main challenge is that, when γ1\gamma_1 is fixed, γt\gamma_t is a deterministic sequence of the past observations and γ1\gamma_1, hence stationarity of YtY_t does not translate into the joint stationarity of (Yt,γt)(Y_t, \gamma_t). Instead, we prove the qq-step average coverage under the assumptions in Appendix D.2. We argue that the assumptions are not restrictive: D.1 is a common condition on the density; D.2 covers most practical prediction intervals (we add a remark below it showing that it allows for rolling-window fit of the underlying model); D.3 is automatically satisfied when a finite grid of α\alpha is used (e.g., 0.01,0.02,…,0.99\\{0.01, 0.02, \ldots, 0.99\\}); D.4 is a regularity condition that avoids extreme distributional shifts.

Moreover, we show that the qq-step average control is not an artifact of our proof argument. Instead, even for ACI, is cyclic when for some integer. For example, consider and lim⁡t→∞E[errt]\lim_{t\to\infty}\mathbb{E}[\mathrm{err}_t] is cyclic when αˉ=1/q\bar{\alpha}=1/q for some integer q>0q > 0.

For example, consider αˉ=1/2\bar{\alpha}=1/2. The ACI update rule is

αt+1=αt+12(12−errt).\alpha_{t+1}=\alpha_t+\frac{1}{2}\left(\frac{1}{2}-\mathrm{err}_t\right).

Let Yt∼iidUnif(1/3,5/12)Y_t\stackrel{\mathrm{iid}}{\sim}\mathrm{Unif}(1/3,5/12) and C(1−α)=[α,1]C(1-\alpha)=[\alpha,1]. Starting from α1=1/4\alpha_1=1/4, we have almost surely α2n−1=14,err2n−1=0\alpha_{2n-1}=\frac{1}{4},\quad \mathrm{err}_{2n-1}=0 and α2n=12,err2n=1\alpha_{2n}=\frac{1}{2},\quad \mathrm{err}_{2n}=1. Thus, although (Yt)(Y_t) is iid, the expected coverage alternates between 11 and 00. It does not converge pointwise, while every two-period average equals the target coverage 1/21/2.

In sum, we believe this is a very challenging yet important question. Our Theorem 3.2 reveals an interesting cyclic structure for stationary YtY_t under reasonable conditions. The structure is fundamental to ACI-style methods.

AuthorsOpenReview ↗

(Continued)

W3 (clarity). Thank you for your comment. In the revision, we added a pipeline figure at the start of Section 3 (Figure 2) showing how the forecasts and PIT history feed the Bellman DP, how the DP output α_t gives the interval, and how λ_{t+1} is updated after observing Y_t. We also added a notation table (Table 7, Appendix B). Lastly, we expanded Algorithm 1 to list its hyperparameters and inputs and the DP steps (backward recursion over the horizon using the cumulative error count as the state), and to state the output rule α_t = 0 when λ_t ≥ λ_max.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revisions to paper

We would like to thank all the reviewers. The revision (i) updates all tables and figures and adds ELEC2; (ii) states the forecasting models and horizons (Transformer with moving-average variance estimation, T = 5, for returns, Google Trends and ELEC2; GARCH(2,2), T = 3, for volatility; all fit on rolling windows); (iii) adds comparisons with AgACI, DtACI, OLCP-Hedge and AcMCP (Section 4.3, Appendix A.2); and (iv) adds a pipeline figure, a table of notation and a more detailed Algorithm 1.

Theory: we added Proposition C.1 (any method with long-run coverage guarantees must allow full-space intervals), Appendix F.2 (unregularized BCI has provably shorter intervals than ACI under miscalibrated nominal intervals, without stationarity), an updated Theorem 3.3 that applies to the implemented DP policy, and Remark D.1 with a sentence above Theorem 3.2 showing that the assumptions allow rolling-window fitting. BCI can produce full-space intervals, as any method with long-run coverage guarantees must (Proposition C.1), but they are rare in our experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.