- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Bellman Conformal Inference (BCI), a framework that uses multi-step forecasts to improve prediction interval efficiency while controlling long-run average miscoverage in time series. BCI wraps an existing forecasting model, estimates the behavior of forecast errors through an empirical distribution of historical uncalibrated PIT values, and combines this distribution with future interval-length curves to formulate a finite-horizon stochastic control problem. The objective trades off interval length against a penalty for excessive cumulative miscoverage. A Bellman recursion uses the number of errors accumulated within the planning horizon as its state to select the current nominal miscoverage level. The method executes only the current action, updates the miscoverage penalty weight after observing the outcome, and replans. The paper establishes long-run average coverage and, under additional assumptions, studies marginal coverage, approximate optimality for a surrogate planning objective, and interval-length comparisons with ACI in a particular setting. Experiments assess calibration and interval efficiency on stock returns, volatility, and search popularity forecasting tasks.
Critical strengths
-
The paper addresses a well-motivated question about the trade-off between coverage and efficiency in online interval calibration. Incorporating future interval-length curves into the decision allows the method to account for differences in the cost of obtaining coverage across time. This provides a useful modeling perspective on allocating a miscoverage budget, extending approaches that primarily adjust the nominal level in response to past errors.
-
The framework separates coverage control from the accuracy of the forecasting model. Under the stated structural assumptions on the interval family, the long-run average coverage guarantee relies on online feedback and boundary safeguards, without requiring a correctly specified forecaster or accurate planning distributions. This makes the approach compatible with different forecasting models while retaining an explicit validity guarantee under model misspecification and distribution shift.
-
The paper examines long-run average coverage, marginal coverage under additional conditions, and interval efficiency, and demonstrates compatibility with both neural forecasting models and GARCH. The empirical discussion also considers the calibration of the original intervals, the planning horizon, and the occurrence of infinite intervals. These analyses help identify conditions under which the method is useful and explain why its gains vary across forecasting problems.
Critical weaknesses
-
The decision spaces in the optimization problem, Bellman recursion, and efficiency theorem are not consistently specified. Equation (4) and Theorem 3.3 optimize an action vector fixed at the planning time, and Appendix E explicitly compares objectives over this vector space. In contrast, the recursion in Equations (20)–(22) optimizes a feedback policy whose future actions depend on the accumulated error state. These problems generally have different optimal values and can produce different current actions. The discrepancy also affects the proof: Appendix E cancels the interval-length term between the two objectives because the actions are fixed. Under a feedback policy, future actions and length costs depend on the simulated history, so changing the disturbance law can also change expected length costs. As written, the analysis therefore does not establish that DP exactly solves Equation (4), or that Theorem 3.3 applies to the implemented algorithm. A consistent policy formulation and corresponding proof are needed.
-
The surrogate planning model is not sufficiently connected to the interval efficiency achieved during actual operation. BCI uses current multi-step interval lengths as proxies for the lengths obtained after future reforecasting, and models future errors as independent draws from an empirical distribution of recent one-step PIT values. Both approximations can distort the allocation of miscoverage under substantial distribution shift or miscalibration. Theorem 3.3 holds the future length functions fixed and controls the surrogate objective error caused by replacing the PIT distribution; a small bound also requires approximately conditionally uniform PITs. It does not control errors in the anticipated length curves or establish a cumulative length guarantee for the receding-horizon procedure. Since the largest empirical gains occur when the original intervals are poorly calibrated, the theory provides limited explanation of this central benefit. Direct analysis or controlled experiments are needed to assess sensitivity to inaccurate lookahead information and temporal dependence in forecast errors.
-
The independent value of Bellman planning over simpler online cost optimization remains insufficiently established. The long-run coverage guarantee follows from the penalty update and boundary safeguards and does not depend on the particular internal optimizer. The strict length advantage in Appendix F concerns a KL-regularized variant and requires sufficiently strong regularization and specific relations between the methods' step sizes, whereas the main experiments use no regularization. This comparison does not isolate the benefit of lookahead from regularization and feedback strength. Although the paper discusses horizon sensitivity, including a single-step configuration, it does not provide sufficiently detailed matched comparisons holding the historical PIT estimator, feedback rule, and tuning budget fixed. In particular, it remains unclear how much is gained from genuine future length information relative to single-step optimization or planning with the current length curve repeated across the horizon. The attribution of the reported efficiency gains to the central new mechanism is therefore incomplete.
Questions for authors
-
Appendix B.2 states that the empirical-CDF optimization can be solved exactly by searching historical PIT values. However, the miscoverage convention uses an action greater than or equal to the PIT, so the empirical CDF jumps upward at these values. With a continuous decreasing length function, the infimum may be approached immediately to the left of a jump without being attained at the jump itself. How is this handled in the implementation? Does the solver use left-limit evaluations or approximately optimal actions, and how is its optimization tolerance determined?
-
The formal conditions for Theorem 3.2 include a transition density bounded below uniformly over past states on compact outcome sets, bounded predictive location and scale, and interior actions separated from the boundaries. Which experimental forecasters and data-generating settings satisfy these conditions? When forecasting parameters are continually refitted, is the joint state in Appendix D, consisting of a finite observation window, PIT history, and penalty weight, still a time-homogeneous Markov process, or must the fitted parameters and training state also be included?
-
Appendix A.3 selects step sizes using the full evaluation sequence to match ACI's local coverage variability. If hyperparameters are instead chosen on an earlier temporal validation segment and then frozen, or updated through a strictly prospective rule, does the coverage–length trade-off persist on subsequent observations? Could the authors report comparisons with ACI and PID under such a protocol to clarify the gains attainable in online deployment?
4 comments
(Continued)
We thank the reviewer for their comments and questions. Q1. The empirical CDF and the DP. For the empirical CDF jumps, we use the left-limit evaluations. For other cases where the minimum is not attained, we select a solution whose objective value is within of the infimum in (4), for a sufficiently small .
Q2. Conditions of Theorem 3.2 under continual refitting. We will walk through each of the assumptions, Assumptions D.1-D.4 in Appendix D.2, for Theorem 3.2 and talk about the requirements for the forecaster and data-generating distributions. Assumption D.1 requires the true transition density to be bounded from below. This is a common regularity assumption. Assumption D.2 covers most practical prediction intervals, which are based on estimated conditional mean and variance. For the forecaster, we have added a sentence above Theorem 3.2 and a more detailed remark below Assumption D.2 showing that it allows for rolling-window fit of the underlying model (namely the finite observation window you mentioned), which is a common approach in practice. Assumption D.3 is automatically satisfied when a finite grid of is used (e.g., {0.01, 0.02, …, 0.99}). Assumption D.4 is a regularity condition that avoids extreme distributional shifts.
See the updated paper for the details.
Q3
Q3: tuning on an earlier segment and freezing. We re-ran the comparison with BCI's c and PID's (η, K_I) tuned on the first 60% of each series (matched to ACI's 50-day rolling standard deviation of miscoverage for the first 60% of the data, with ACI's γ fixed), then frozen and evaluated on the last 40%. BCI uses T = 5 for returns and Google Trends and T = 3 for volatility. Reduction = 100(1 − BCI/method).
| Dataset | γ | Avg. length (BCI / ACI / PID) | Reduction vs ACI / vs PID (%) | Miscoverage % (BCI / ACI / PID) | Frac. ∞ % (BCI / ACI / PID) |
|---|---|---|---|---|---|
| Return-AMD | 0.1 | 0.1437 / 0.1487 / 0.1541 | +3.3 / +6.8 | 9.96 / 9.96 / 10.39 | 6.42 / 12.31 / 0.00 |
| Return-AMD | 0.008 | 0.1388 / 0.1605 / 0.1380 | +13.5 / −0.6 | 10.39 / 9.74 / 10.60* | 0.54 / 0.00 / 0.00 |
| Return-Amazon | 0.1 | 0.0997 / 0.1291 / 0.1295 | +22.8 / +23.0 | 10.03 / 9.65 / 10.18 | 1.99 / 17.76 / 0.00 |
| Return-Amazon | 0.008 | 0.0948 / 0.1388 / 0.0871 | +31.7 / −8.8 | 10.11 / 9.65 / 10.11 | 0.23 / 0.00 / 0.00 |
| Return-Nvidia | 0.1 | 0.1261 / 0.1437 / 0.1473 | +12.2 / +14.4 | 10.16 / 10.06 / 10.16 | 13.09 / 11.91 / 0.00 |
| Return-Nvidia | 0.008 | 0.1270 / 0.1536 / 0.1214 | +17.3 / −4.6 | 10.06 / 9.38* / 10.55* | 12.60 / 0.00 / 0.00 |
| Volatility-AMD | 0.1 | 9.3077 / 10.1724 / 9.8340 | +8.5 / +5.4 | 10.01 / 10.01 / 10.01 | 3.68 / 14.96 / 12.16 |
| Volatility-AMD | 0.008 | 7.6612 / 7.4855 / 8.2249 | −2.3 / +6.9 | 9.91 / 10.21 / 10.06 | 0.00 / 0.00 / 3.12 |
| Volatility-Amazon | 0.1 | 4.8189 / 8.1797 / 7.3490 | +41.1 / +34.4 | 9.95 / 9.91 / 10.00 | 0.00 / 13.92 / 10.67 |
| Volatility-Amazon | 0.008 | 4.2562 / 4.1651 / 4.3483 | −2.2 / +2.1 | 10.00 / 9.95 / 9.91 | 0.00 / 0.00 / 0.25 |
| Volatility-Nvidia | 0.1 | 6.6720 / 9.4444 / 7.9661 | +29.4 / +16.2 | 10.01 / 10.01 / 10.06 | 0.00 / 13.90 / 7.24 |
| Volatility-Nvidia | 0.008 | 5.9586 / 6.1048 / 6.5275 | +2.4 / +8.7 | 10.16 / 10.11 / 10.11 | 0.00 / 0.00 / 1.36 |
| Google Trends | 0.1 | 4.0574 / 5.0930 / 4.6730 | +20.3 / +13.2 | 9.76 / 9.92 / 9.92 | 4.48 / 14.88 / 0.00 |
| Google Trends | 0.008 | 3.9281 / 5.0074 / 3.7725 | +21.6 / −4.1 | 9.92 / 10.40 / 10.56* | 0.00 / 0.00 / 0.00 |
*Mean miscoverage outside 10 ± 0.5%. PID follows the main tables' rule: the selected (η, K_I) is kept unless it produces exploding intervals on the tuning rows, in which case K_I is capped at the next grid value below and the selection is repeated; this applied to 4 of the 14 settings. Unbounded intervals are charged the observed outcome range (square-root scale for volatility).
BCI is shorter than ACI in 12 of 14 settings and than PID in 10 of 14. The two losses to ACI are small and occur under loose control (about −2%). The four losses to PID also all occur under loose control. BCI's miscoverage stays within 10 ± 0.5% in all 14 settings, while PID is outside the band in three of its four wins, so those wins are partly bought with lower coverage. BCI's share of infinite intervals is highest on Return-Nvidia (about 13% at both γ), which we regard as a limitation.
W3
W3: value of Bellman planning. Appendix F.2 now shows that BCI with τ = 0 has shorter intervals than ACI under miscalibration, without stationarity or KL regularization. For the value of looking ahead, we compared the paper's horizon with T = 1 (the same empirical-PIT estimator, feedback rule and tuning protocol), re-tuning c for each T and matching to ACI. Reduction in interval length vs ACI (γ = 0.1):
| Dataset | T = 1 | T = 3 | T = 5 |
|---|---|---|---|
| Return-AMD | −1.9% | −1.3% | +2.7% |
| Return-Amazon | +19.2% | +18.4% | +18.7% |
| Return-Nvidia | −15.5% | −2.2% | +9.5% |
| Volatility-AMD | +10.0% | +11.1% | +12.2% |
| Volatility-Amazon | +33.5% | +34.5% | +32.0% |
| Volatility-Nvidia | +30.8% | +31.7% | +27.5% |
| Google Trends | −2.1% | +8.5% | +14.6% |
Longer horizons also produce more full-space intervals at γ = 0.1.
We thank the reviewer for the insightful comments. Please see our detailed responses to weaknesses W1-W3 and questions Q1-Q3
**W1: The decision spaces in the optimization problem, Bellman recursion, and efficiency theorem..."
Our response: We use (4) to motivate the objective function. The notation minαt∣t,…,αt+T−1∣t in (4) is for illustration and it does not mean BCI literally optimizes αt∣t,…,αt+T−1∣t as T scalars. We will remove the minimization in (4) and emphasize that this is the objective function we will be using for the stochastic control problem. We agree that Theorem 3.3 in our submitted version did not apply to the implemented algorithm. In the revision, we have updated it for the actual DP algorithm we used. The conditions remain the same and the bound only differs by a constant. We thank the reviewer for pointing this out. Below is a sketch. Setting. Fix a planning horizon T≥1, a window size B≥4, a time t≥B+1, and a target miscoverage rate αˉ∈(0,1). Let Pt⋆ be the true joint law of the next T uncalibrated PITs (βt,…,βt+T−1) conditional on the history Ft−1. BCI plans under Pt(B)=(μt(B))⊗T, which draws each future PIT independently from the empirical distribution of the past B PITs.
Assumptions. Let Gs∘(u)=Pr(βs≤u∣Fs−1). Assume that, for every s≥1 and u,v∈[0,1],
supu∈[0,1]∣Gs∘(u)−u∣≤κs,∣Gs∘(u)−Gs∘(v)∣≤M∣u−v∣,
where M>0 and (κs) is deterministic, nonnegative, and nonincreasing. Assume also that each planned interval-length curve satisfies Ls∣t≥0 and Ls∣t(1)=0, that ∣λt∣<∞, and that τ≥0 with K(⋅,αˉ) nonnegative and finite.
Theorem 3.3 (efficiency of empirical PIT planning). Let πt minimize the empirical Bellman objective at every stage and error count, as in (26). Let JtP(π) denote the expected planning cost in (4) under law P, and let Πt be the class of policies whose actions depend on the stage and the cumulative error count. For every δ∈(0,1), with probability at least 1−δ,
0≤JtPt⋆(πt)−infπ∈ΠtJtPt⋆(π)≤4∣λt∣(Blog(B/δ)+M+κt−B).
W2: The surrogate planning model is not sufficiently connected to the interval efficiency achieved during actual operation...
**Our response: ** We would like to emphasize that the validity of BCI (i.e., average coverage control) does not require independence of βt, empirical distribution of recent PIT values to be correctly specified, or the length function of multi-step ahead forecasting intervals to be good approximations. They are simply a working model inside the optimizer. Theorem 3.1 allows arbitrary temporal dependence, distribution shift, or poor multi-step ahead forecasting intervals, as long as Assumption 2.1 holds for the nominal prediction intervals, which is almost always the case by design. This is similar in spirit to the broader conformal inference literature, where a working model is used without needing to assume it is correct.
Theorem 3.3 investigates the efficiency when the working model described above is accurate. It does not invalidate the validity claim in Theorem 3.1. This is also a standard exercise in the conformal inference literature which often shows the method is efficient with a good working model.