acceptodds
Under review as a conference paper at ICLR 2027

OBCP: Online Backward Conformal Prediction under Distribution Shift

Abstract

Backward conformal prediction (BCP) controls prediction-set size through a data-dependent level. In online prediction, true labels are revealed after predictions are made. As the data distribution changes, the true-label BCP ratio may no longer satisfy the e-value expectation condition, and past data may no longer reflect current risk. We propose Online Backward Conformal Prediction (OBCP), extending BCP to dynamic prediction and risk reporting under a fixed set-size budget. We introduce a conditional inflation factor and construct a clipped product surrogate to bound miscoverage risk without relying on the e-value condition. We derive a three-term risk bound combining a recency-weighted average of past clipped products, a target-drift term, and a finite-sample concentration term. When a valid drift allowance is available, this bound yields fixed-time high-probability certificates for current conditional miscoverage. We also establish exact quantitative relations between the clipped-product statistic and empirical error counting. Simulations compare oracle-drift certificates, while a chronological real-data audit evaluates the observable product statistic.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
conformal prediction, uncertainty quantification, online prediction, distribution shift, risk estimation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_npL1OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Online Backward Conformal Prediction (OBCP), extending backward conformal prediction to sequential classification under distribution shift while enforcing a fixed prediction-set size budget. The authors introduce a clipped-product surrogate that upper-bounds the miscoverage indicator without relying on the e-value expectation condition. They establish a fixed-time high-probability risk bound consisting of a recency-weighted historical statistic, a target-drift term, and a concentration term. The paper further investigates the relationship between the proposed surrogate and empirical error counting, theoretically characterizing when the former yields tighter risk certificates. Experiments on synthetic distribution shifts and the CLEAR10 dataset demonstrate the behavior of the proposed risk-reporting framework.

Critical strengths
  1. Unlike conventional conformal prediction approaches that primarily control coverage while allowing prediction-set sizes to vary, the proposed framework emphasizes risk reporting under a fixed set-size budget. This complementary perspective is well motivated.

  2. The introduction of effective sample size and weighted mean lag provides an intuitive interpretation of the tradeoff between emphasizing recent observations and retaining sufficient statistical information.

Critical weaknesses
  1. Theorem 1 provides a risk bound involving the target-drift term Γβ,t(w)\Gamma_{\beta,t}(w), which depends on unknown conditional expectations. Corollary 1 replaces this term with a prediction-time allowance, but the framework does not provide a generally applicable procedure for obtaining such an allowance from observed data.
  2. The paper defines the conditional surrogate target as βt∘=E[Qt(1)∣Gt]\beta_t^\circ=\mathbb E[Q_t^{(1)}\mid\mathcal G_t], where Gt\mathcal G_t includes the current input XtX_t. Consequently, variations in βt∘\beta_t^\circ may arise not only from changes in the underlying data-generating distribution but also from differences between consecutive inputs. Even under a stationary i.i.d. data distribution, the conditional risk and conditional surrogate expectation can fluctuate substantially across heterogeneous inputs. The paper would benefit from a clearer discussion of the assumptions under which conditional target variation remains sufficiently small, particularly when the covariate distribution is heterogeneous.
Questions for authors
  1. How can a valid drift allowance be constructed in a realistic deployment setting? Corollary 1 requires a prediction-time upper bound on Γβ,t(w)\Gamma_{\beta,t}(w). Can the authors provide an implementable estimator or conservative bound based only on observable online data, without access to the true conditional label distributions? In particular, could such a procedure generate nonvacuous certificates on CLEAR10 or another real-world dataset?
  2. Since the candidate ratios preserve the ordering of the nonconformity scores, it appears that feasible prediction sets reduce to selecting the BB smallest-score labels when scores are distinct. Can the authors confirm this characterization? If so, what is the main advantage of the online backward construction over applying a fixed top-BB predictor with a separate sequential risk-monitoring procedure?
4 comments

We thank the reviewer for their careful assessment and valuable feedback. We have strengthened the manuscript to address the key concerns. We respond to each point below.

Responses to Critical Weaknesses

Weakness 1. We have addressed this concern by constructing a prediction-time observable drift allowance under a simple normalized-mean stability condition, without requiring unknown conditional label distributions or future outcomes. On CLEAR10, our construction yields nonvacuous full risk certificates at 82.6882.68\\% of evaluation times. We have added the construction and results in blue to Section 6 and Appendix C.8, with further details in our response to Question 1 below.

Weakness 2. We agree that heterogeneous covariates can cause conditional-target variation even under stationarity. Our theory accounts for such variation without restricting the underlying shift mechanism. To address this concern, we have added a four-dimensional Gaussian experiment (Section 5.3; response to Reviewer zbzx, Question 1) and observable CLEAR10 certificates under an explicit stability condition (Section 6; Question 1 below), both highlighted in blue in the revised manuscript.

Responses to Questions

Question 1. Yes. We now provide a concrete, prediction-time implementable drift allowance that yields nonvacuous full risk certificates on CLEAR10, using only observable online information, without requiring the true conditional label distributions or future outcomes.

Before presenting our construction, we first emphasize that we strictly exclude any future information from prediction-time computation. Without additional conditions, there is no general procedure that can guarantee nontrivial conditional risk certificates in arbitrary online settings. This is a fundamental information limitation of online conditional risk certification, independent of the dataset, the prediction method, or the specific form of the drift correction. It is not a limitation specific to OBCP. We have added a rigorous proof in Appendix A.4, highlighted in blue.

Nevertheless, a simple normalized-mean stability condition already suffices to construct a nontrivial observable allowance. At each feasible time, let ℓt=min⁡y∈Yut(y)\ell_t=\min_{y\in\mathcal{Y}}u_t(y) and Zt=(Qt(1)−ℓt)/(1−ℓt)∈[0,1]Z_t=(Q_t^{(1)}-\ell_t)/(1-\ell_t)\in[0,1]. Writing θt=E[Zt∣Gt]\theta_t = \mathbb{E}[Z_t \mid \mathcal{G}_t] gives the exact identity βt∘=ℓt+(1−ℓt)θt\beta_t^{\circ} = \ell_t + (1-\ell_t)\theta_t.

Under the stability condition θt=θ∈[0,1]\theta_t=\theta\in[0,1], we use previously revealed normalized products to construct a confidence lower bound θ‾t−1\underline{\theta}_{t-1} and obtain

Γ‾t,w=(1−θ‾t−1)∑q=1mAq(w)∣ℓt−q+1−ℓt−q∣.\overline{\Gamma} _{t,w} = (1 - \underline{\theta} _{t-1}) \sum _{q=1}^{m} A_q(w) \left| \ell _{t-q+1} - \ell _{t-q} \right|.

This upper-bounds our original accumulated Γβ,t(w)\Gamma_{\beta,t}(w) with high probability, so Corollary 1 applies directly without modification. We have added this construction in blue to Section 6, with the detailed confidence procedure and proof in Appendix C.8. We also tested the temporal stability of the normalized-product mean on CLEAR10, obtaining encouraging results (HAC(20) Wald test, p=0.534p=0.534). The test is detailed in Appendix C.8 (highlighted in blue).

We evaluate the resulting prediction-time observable drift allowance and the complete three-term risk certificate on the chronological CLEAR10 stream:

SettingΓ‾\overline{\Gamma} (mean/median)Full bound (mean/median)Nonvacuous Rate
Γ‾t,w\overline{\Gamma}_{t,w}0.2275/0.16640.2275/0.16640.7854/0.71800.7854/0.718082.6882.68\\%

The complete numerical results have been added in blue to Appendix C.8 (Table 10). The core implementation code for this CLEAR10 experiment is provided in the supplementary material under the CLEAR10_Rebuttal folder. These findings demonstrate that our original Gamma-based certificate is practically implementable using only observable online information and need not be overly conservative.

Question 2. We thank the reviewer for this insightful observation. Importantly, the connection with Top-BB prediction concerns label selection, rather than the conditional risk-certification framework developed in our work. In fact, the relationship between fixed-size prediction and Top-TT selection is already recognized in classical BCP (Gauthier et al., 2025, Remark 3.4), where empirical Top-TT error counting is explicitly discussed as a natural alternative. Similarly, when our construction is feasible, the reference-score sum is positive, and candidate scores are distinct, OBCP selects the BB smallest-score labels. This shared selection property does not, however, determine how prediction risk should be monitored and certified over time, which is the central focus of our work.

The central novelty of OBCP lies in its risk-certification framework for fixed-budget online predictions, which goes substantially beyond Top-BB label selection. Specifically, we construct an inflation-corrected clipped-product surrogate that pointwise upper-bounds miscoverage under distribution shift. Equation (3) establishes its exact decomposition into empirical error counting and an additional soft component that retains continuous boundary information. Theorem 1 provides a fixed-time conditional risk bound combining historical products, target drift, and sampling uncertainty, while Corollary 1 yields a prediction-time certificate given a valid drift allowance. Furthermore, Corollary 3 characterizes the tradeoff between the additional conservatism of the Product statistic and the potential improvement in target stability, which can lead to tighter risk certificates. These theoretical properties and certification guarantees are not provided by Top-BB selection alone.

Beyond our existing comparison with empirical counting (Sections 3.4 and 4.3), we further consider an independent and straightforward score-aware monitor constructed directly from the Top-BB selection threshold, without relying on BCP or OBCP:

Q_t^{\mathrm{Margin}} = \min\left\\{\frac{S(X_t,Y_t)}{S_{t,(B+1)}},1\right\\},

where S(Xt,Yt)S(X_t,Y_t) is the nonconformity score of the realized label, and St,(B+1)S_{t,(B+1)} is the (B+1)(B+1)-th smallest score among all candidate labels, corresponding to the Top-BB selection threshold. The corresponding rolling monitoring statistic is β^t,wMargin=∑ℓ=1mwℓQt−ℓMargin\widehat{\beta} _{t,w}^{\mathrm{Margin}} = \sum _{\ell=1}^{m} w _{\ell} Q _{t-\ell}^{\mathrm{Margin}}. For a fair comparison, both methods use the same three-component risk certification framework: a historical surrogate average, a drift allowance, and a concentration radius.

Using identical Top-BB predictions in an exploratory controlled simulation with AR(1) covariates, ordered-probit labels, and fixed softmax NLL scores (K=5K=5, B=3B=3, L=50L=50, m=200m=200), we evaluate both monitors over 200200 independent trajectories. As in Section 5, we use the known data-generating distribution to compute the oracle accumulated drift.

MetricOBCPPlain Margin
Historical surrogate average0.0860.073
Oracle Gamma0.2780.315
Full certificate0.4510.475

The full certificate is the sum of the historical surrogate average, Oracle Gamma, and a common concentration radius, which is omitted as a separate row for brevity. These results illustrate that backward normalization can reduce accumulated target drift sufficiently to offset its additional conservatism, yielding a tighter oracle risk certificate. We will further clarify the Top-BB equivalence and the novelty of our monitoring construction in the revised manuscript.


Closing Remark. We sincerely thank the reviewer for the careful evaluation and constructive feedback. We greatly appreciate the opportunity to strengthen the manuscript through this discussion. If any questions or concerns remain, we would be very grateful to hear them and would be happy to provide further clarification.

References

Gauthier, E., Bach, F., and Jordan, M. I. (2025). Backward Conformal Prediction. The Thirty-ninth Conference on Neural Information Processing Systems (NeurIPS 2025).

Thank you for the detailed rebuttal and revisions. Most of my concerns have been addressed and I have raised my score to 3.

We sincerely thank the reviewer for taking the time to carefully consider our rebuttal and revised manuscript. We truly appreciate your constructive feedback and reconsideration of our work.

Reviewer_zbzxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper extends backward conformal prediction to an online setting where each prediction set must obey a fixed label budget and the true label is revealed after prediction. Since the backward ratio need not remain an e-value under distribution shift, the selected backward level is not treated as a conditional-risk certificate. Instead, the method constructs a clipped product Qt(1)=min⁡{α~tRt,Yt,1}Q_t^{(1)}=\min\{\widetilde\alpha_t R_{t,Y_t},1\} that pointwise upper-bounds miscoverage. With deterministic recency weights and a fully feasible calendar window, a martingale concentration argument yields

+\Gamma_{\beta,t}(w) +\sqrt{\frac{\log(1/\eta)}{2m_{\mathrm{eff}}(w)}}.$$ Here the terms are historical clipped products, variation of their conditional means, and sampling uncertainty. The guarantee is fixed-time and becomes observable only when a valid prediction-time drift allowance is supplied. The paper also establishes the exact decomposition of Product into Count plus a nonnegative covered-event soft term, and characterizes when a smaller product-target drift compensates for that extra conservatism. Simulations use known conditional laws and oracle drift to compare weighting schemes and counting certificates. A chronological CLEAR10 experiment instead audits the observable historical summaries against future-block empirical errors; it does not validate a deployable non-oracle conditional-risk certificate.
Critical strengths
  1. The guarantee is carefully scoped and the main argument is transparent. The paper avoids applying the e-value expectation condition after exchangeability has been lost. The pointwise inequality between error and clipped product is immediate from the threshold construction, and Appendix A.1 correctly separates the deterministic drift comparison from conditional Hoeffding concentration. The zero-extension argument handles infeasible observations without silently converting calendar windows into selected feasible samples. The authors explicitly distinguish a fixed-time implication on feasible windows from a simultaneous deployment guarantee, and distinguish an oracle drift term from an observable allowance (§§3–4). I did not find a mathematical counterexample to the stated bound under these assumptions.

  2. The Product-versus-Count comparison exposes a real bias–stability tradeoff rather than claiming unconditional superiority. Equation (3) makes clear that Product is necessarily at least as conservative as Count on identical history. Corollary 3 then identifies the only way the complete product certificate can improve: its drift savings must exceed its historical soft surplus. The near-boundary simulations deliberately demonstrate both favorable and unfavorable regimes, including a reversal in favor of Count when switching slows (Table 5). The CLEAR10 experiment preserves chronological prediction/feedback ordering and explicitly labels its future-100 target as a retrospective audit rather than the unobserved current conditional risk. These qualifications are important strengths.

Critical weaknesses
  1. The essential input needed for operational certification is not instantiated, and conditional-target drift is a demanding assumption even without temporal distribution shift. Corollary 1 requires an externally valid allowance for Γβ,t\Gamma_{\beta,t}, while Appendix A.3 gives sufficient constructions in terms of quantities such as conditional-label-law variation. These are mathematically useful reductions, but no non-oracle procedure or application with a justified numerical allowance is evaluated. Every complete simulated certificate uses oracle drift; the real-data study evaluates only historical summaries (§§5–6).

    The distinction between temporal distribution shift and the variation actually required by the theorem is especially important. Since βt∘=E[Qt(1)∣Gt]\beta_t^\circ=\mathbb E[Q_t^{(1)}\mid\mathcal G_t] conditions on the current covariate XtX_t, heterogeneous i.i.d. inputs can produce substantial adjacent conditional-target changes even when the population distribution is stationary. Likewise, the total-variation term in Appendix A.3 compares conditional label laws at different observed inputs, not simply consecutive population distributions. The constant-covariate simulations avoid this difficulty. Thus, a small externally known rate of population drift does not automatically supply the needed allowance in the image-stream setting. The limitation is acknowledged in part by treating the allowance as an input; nevertheless, it is the main obstacle to translating the paper's conditional theorem into the advertised use case.

  2. The comparative evidence does not yet separate useful boundary information from added conservatism and the choice of drift bound. On CLEAR10, Table 7 gives a mean clipped product of 0.08836 at B=3B=3, while the realized error frequency is 0.02343. Product's zero shortfall against the future-100 audit therefore needs to be read alongside its substantial extra margin. The 4,651 overlapping audit windows are useful descriptive evaluations, but zero shortfall alone does not establish superiority over a suitably conservative counting summary. A comparison of overestimation, alert responsiveness, or tightness at matched shortfall would be more informative.

    There is also a sharper oracle counting comparator available from the same proof. Define

    =\left[r_t^*-\sum_{\ell=1}^m w_\ell r_{t-\ell}^*\right]_+.$$ Replacing the telescoping total-variation step with this direct endpoint-to-history difference gives, with the same fixed-time concentration radius and feasibility convention, $$r_t^*\leq\widehat r^{\mathrm{count}}_{t,w} +D^r_{t,w} +\sqrt{\frac{\log(1/\eta)}{2m_{\mathrm{eff}}(w)}}.$$ This is a reviewer-derived *oracle* comparator, not a proposed observable certificate. Nevertheless, $D^r_{t,w}\leq\Gamma_{r,t}(w)$, and it does not repeatedly charge oscillations that cancel in the weighted history. For the periodic 0.08/0.16 risk paths with a 200-step uniform window and switching blocks of 25 or 50 steps, a window entirely inside the periodic regime averages risk to 0.12, so $D^r_{t,w}$ is only 0 or 0.04. In contrast, Table 6's mean accumulated counting drifts are 0.3216 and 0.1608. Consequently, the reported oracle advantage is partly a comparison against an accumulated-variation correction that is loose for the constructed oscillatory paths. This does not invalidate Corollary 3 or its stated comparison, but the paper should establish whether the intended practical advantage survives a sharper matched counting analysis or follows from a realistic class of *observable* allowances.
Questions for authors
  1. A non-oracle use case. Can the authors provide at least one covariate-rich setting in which the drift allowance is specified independently of the evaluation labels, justified under explicit assumptions, and leads to nonvacuous certificates? Please distinguish changes caused by different current covariates from temporal changes in the data law. If this remains outside scope, please frame the contribution more explicitly as a drift-conditional analysis of a surrogate rather than an end-to-end current-risk certification method.

  2. A stronger matched counting comparison. How do the near-boundary results change when Count uses the direct Dt,wrD^r_{t,w} correction above, or when both methods use the sharpest allowance available under the same stated drift information? On CLEAR10, can the authors report overestimation and a matched-margin counting baseline, with its margin selected on separate earlier data rather than the retrospective audit? These comparisons would clarify whether the soft boundary term supplies actionable information beyond purchasing a larger reporting margin.

  3. Relevant missing work and the distinct guarantee. I wonder whether the following work is directly relevant:

    • Zhiyu Zhang, David Bombara, and Heng Yang, Discounted Adaptive Online Learning: Towards Better Regularization, ICML 2024. This studies adaptive forgetting in nonstationary online learning and its application to online conformal prediction.
4 comments

Responses to Weaknesses

We thank the reviewer for the constructive feedback.

Weakness 1. We address these concerns in our response to Question 1 below.

Weakness 2. We investigate these tradeoffs in Question 2 (Sharp Drift Part) below and Question 2 (CLEAR10 Matched-Margin Part) in a separate response.

Responses to Questions

Question 1. We thank the reviewer for this suggestion. We have added a covariate-rich experiment with four-dimensional inputs in Section 5.3 and Appendix B.3 of the revised manuscript. Specifically, we consider Xt=vHt+ξt⊥\mathbf X_t=vH_t+\xi_t^\perp, where v=(1,1,1,1)⊤/2v=(1,1,1,1)^\top/2, Ht=0.5+0.7St+UtH_t=0.5+0.7S_t+U_t, StS_t is a stationary two-state Markov chain, UtU_t is a stationary Gaussian AR(1) process, and ξt⊥\xi_t^\perp is orthogonal Gaussian noise. Labels follow an ordered probit model based on v⊤Xtv^\top\mathbf X_t. Assuming that vv is known and the Gaussian label-noise standard deviation satisfies σY≥σ0>0\sigma_Y\geq\sigma_0>0, we obtain

\operatorname{TV}(\pi_j,\pi_{j-1})\leq\delta_j:=\min\left\\{1,\frac{|v^\top(\mathbf X_j-\mathbf X_{j-1})|}{\sigma_0\sqrt{2\pi}}\right\\}.

Together with the observable product-vector differences, this bound gives a prediction-time drift allowance through the construction in Appendix A.3. The joint data distribution remains stationary, and P(Y∣X=x)P(Y\mid\mathbf X=x) is fixed over time. Changes in πj\pi_j arise from different realized covariates Xj\mathbf X_j, rather than temporal changes in the data law. Thus, the experiment explicitly distinguishes covariate-induced conditional variation from temporal distribution shift.

Across 200200 simulated trajectories, the mean complete certificate is 0.69610.6961, and 72.2972.29\\% of the certificates are nontrivial (<1<1). The results show that observable certificates can remain informative under substantial covariate heterogeneity, although they become more conservative near abrupt state changes.

Question 2 - Sharp Drift Part. In the original near-boundary experiment, replacing Count's accumulated drift with the proposed direct correction Dr,t+(w)D_{r,t}^{+}(w) reduces its mean certificates to 0.22660.2266, 0.22740.2274, and 0.22720.2272 for H=25,50,200H=25,50,200, respectively, below the original Product certificates (0.26910.2691, 0.26840.2684, and 0.26780.2678). We agree that accumulated variation can be conservative when successive changes cancel. We use Γ\Gamma because it admits natural upper bounds based on local target variation without requiring the true conditional targets or the directions of their changes. This formulation accommodates different application-specific drift assumptions without restricting the underlying shift mechanism. Moreover, its weighted structure makes the recency–uncertainty tradeoff explicit through the mean lag and effective sample size (Corollary 2). Our choice is motivated by analytical flexibility and interpretability, rather than a claim that Γ\Gamma is the sharpest correction.

We further investigate the reviewer's second part of the question under shared, incomplete model information. We consider a five-class ordered-probit model, in which labels are generated by thresholding a latent Gaussian response, with stationary two-dimensional Gaussian covariates. Both methods know the same model structure and parameter uncertainty set Θ=[−2.5,0.5]\Theta=[-2.5,0.5], but not the true parameter θ⋆=−2\theta_\star=-2. For each target f∈u,vf\in\\{u,v\\}, we compute its smallest uniformly valid direct drift allowance within this model class,

G_{f,t}^{\mathrm{sharp}}=\left[\sup_{\theta\in\Theta}\left\\{a_t^f(\theta)-\sum_{\ell=1}^{m}w_\ell a_{t-\ell}^f(\theta)\right\\}\right]_+,

where ajf(θ)=∑yπj(y;θ)fj(y)a_j^f(\theta)=\sum_y\pi_j(y;\theta)f_j(y), uj(y)=min⁡α~jRj,y,1u_j(y)=\min\\{\widetilde\alpha_jR_{j,y},1\\}, and vj(y)=1α~jRj,y≥1v_j(y)=\mathbf{1}\\{\widetilde\alpha_jR_{j,y}\geq1\\}. Both methods use identical predictions, histories, and concentration radii, with conservative numerical optimization.

We compare a switching setting, where candidate labels exchange positions across the Top-BB prediction boundary as covariates vary, with a no-switch control that eliminates these crossings by modifying the score geometry while preserving the same data-generating process. Results over 200200 independent confirmation trajectories are:

SettingTrue riskProductCount
Switching0.01150.01150.11410.11410.14380.1438
No-switch0.00620.00620.11340.11340.09610.0961

True risks are computed only for post-hoc evaluation. Following exploratory comparisons, we report results for both switching and no-switch settings under matched sharp model-constrained allowances and shared incomplete model information. Product yields a tighter certificate in the switching setting, whereas Count performs better in the no-switch setting. Consistent with Corollary 3, these findings illustrate a setting-dependent tradeoff rather than uniform superiority.

Question 2 - CLEAR10 Matched-Margin Part. We conducted additional experiments on CLEAR10 to compare Product with an earlier-calibrated Counting baseline, including the requested overestimation analysis. We retain the original setting with B=3B=3, L=m=200L=m=200, uniform weighting (U200), and the future-100100 retrospective audit. Specifically, we add a fixed margin δ\delta to r^t,wcount\widehat r_{t,w}^{\mathrm{count}}. We consider two calibration methods: Mean-matched, using the average historical soft surplus, and Shortfall-matched, using the maximum observed Counting shortfall:

\delta_{\mathrm{SF}} = \max_{t\in\mathcal T_{\mathrm{cal}}}\left[\widehat r_{t,100}^{\mathrm{future}}-\widehat r_{t,w}^{\mathrm{count}}\right]_+.$$ Here, $\mathcal T_{\mathrm{cal}}$ denotes the calibration period. Both margins are computed from earlier data and remain fixed during evaluation. All calibration future-$100$ blocks are completed before evaluation begins, so no evaluation-period labels are used to calculate the margins. We consider two chronological calibration settings in an exploratory sensitivity analysis, with reference to the two shaded shortfall episodes in Figure 2 of Section 6 (zero-based indices). The Early setting uses calibration anchors $200–300$ and evaluation anchors $400–4850$, leaving both episodes outside calibration. The Extended setting uses calibration anchors $200–900$ and evaluation anchors $1000–4850$, including the first episode in calibration while leaving the second for evaluation. | Setting | Margin $\delta$ | Under (P) | Under (C+$\delta$) | Mean Over (P) | Mean Over (C+$\delta$) | |:---|:---|:---|:---|:---|:---| | Early / Mean | $.06663$ | $0/4451$ | $0/4451$ | $\mathbf{.06418}$ | $.06650$ | | Early / Shortfall | $.02000$ | $\mathbf{0/4451}$ | $574/4451$ | $.06418$ | $.02152$ | | Extended / Mean | $.05897$ | $\mathbf{0/3851}$ | $16/3851$ | $.06534$ | $.05922$ | | Extended / Shortfall | $.05000$ | $\mathbf{0/3851}$ | $17/3851$ | $.06534$ | $.05029$ | *P = Product; C+$\delta$ = Count+$\delta$; Under = underestimation count; Mean Over = mean overestimation.* Product achieves zero underestimation in all four settings without an additional calibrated margin. In Early / Mean, it also achieves lower mean overestimation than calibrated Counting ($.06418$ versus $.06650$), while Counting still underestimates risk in the other three settings. These results suggest that Product's time-varying soft surplus can offer a better balance between risk underestimation and conservatism, addressing the concern raised in Critical Weakness 2. **Question 3.** We thank the reviewer for highlighting this relevant work. We have added a comparison with Zhang et al. (2024) in the Introduction, highlighted in blue. Both approaches address nonstationary online prediction without assuming exchangeability, but pursue complementary objectives. **Temporal weighting.** With a constant discount factor, Zhang et al.'s forgetting weights have the same geometric form as our exponential recency kernels after normalization. Their weights apply to losses and gradients, whereas ours aggregate clipped-product risk surrogates. Both methods balance the use of recent observations against the need for sufficient historical data. Zhang et al. analyze discounted regret, gradient variance, and effective horizon, while OBCP characterizes drift and concentration through weighted mean lag and effective sample size. **Objectives and guarantees.** Zhang et al. fix a target miscoverage level $\alpha$ and adapt prediction-set radii using FTRL, establishing finite-time discounted coverage and regret guarantees. OBCP instead fixes a set-size budget $B$ and uses backward-ratio-based clipped products, retaining boundary information beyond binary error counting. While Zhang et al. control discounted empirical miscoverage in the pinball-loss setting, OBCP bounds current conditional miscoverage with fixed-time high probability, given a valid prediction-time drift allowance and window feasibility. Neither guarantee directly implies the other. --- **Closing Remark.** We sincerely thank the reviewer for the constructive feedback and thoughtful suggestions. We hope that our clarifications and additional analyses have addressed the concerns raised. If any questions remain or further clarification would be helpful, please let us know. We would be happy to provide additional details. ### References Zhang, Z., Bombara, D., and Yang, H. (2024). Discounted Adaptive Online Learning: Towards Better Regularization. *Proceedings of the 41st International Conference on Machine Learning (ICML 2024).*

I thank the authors for the response. I keep my original rating.

Thank you for reviewing our responses. We really appreciate your time and the valuable comments you provided throughout the review process.

Reviewer_qovYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper concentrates on backward conformal prediction (BCP). It argues that the true-label BCP ratio may no longer satisfy the e-value expectation condition, and past data may no longer reflect current risk, as the data distribution changes. To address this issue, this paper proposes Online Backward Conformal Prediction (OBCP), extending BCP to dynamic prediction and risk reporting under a fixed set-size budget. The main contributions can be summarized as:

  1. Characterize the inflation of the backward ratio and construct a clipped product that pointwise upper-bounds prediction errors, without assuming the e-value expectation condition.

  2. Derive a fixed-time bound separating historical products, target drift, and finite-sample uncertainty.

  3. Show that the clipped-product statistic equals weighted error counting plus a nonnegative soft component from covered predictions.

Critical strengths

1.This paper is well written and easy to follow.

2.Interesting ideas. This paper proposes Online Backward Conformal Prediction (OBCP), extending BCP to dynamic prediction and risk reporting under a fixed set-size budget.

Critical weaknesses

Main concern. The correctness of Corollary 1 is based on the exist of prediction-time quantity Γ‾t,w\overline{\Gamma} _{t,w} satisfies PΓβ,t(w)≤Γ‾t,w≥1−ξ\mathbb{P}\\{ \Gamma _{\beta,t}(w) \leq \overline{\Gamma} _{t,w}\\} \geq 1-\xi. However, in the real-world scenario, it is impossible to calculate Γ‾t,w\overline{\Gamma} _{t,w} without prior assumptions. This is supported in Section 6 which states that "This examines the historical surrogate, not the full high-probability certificate". The author should discuss in depth in the main text how to provide a feasible upper bound construction scheme for Γ‾t,w\overline{\Gamma} _{t,w} without prior assumptions.

Questions for authors

Please address the concerns in Critical Weaknesses.

1 comment

Responses to Questions

Question 1. We thank the reviewer for raising this important issue. We agree that a valid and nontrivial drift allowance generally requires assumptions on the conditional label mechanism. Without such assumptions, no prediction-time method can guarantee a uniformly nontrivial upper bound on the current conditional risk. We now make this limitation explicit in Appendix A.4 and clarify that the drift allowance in Corollary 1 is an assumption-dependent input rather than an estimator automatically supplied by OBCP.

To demonstrate how such an allowance can be constructed, we have added a four-dimensional covariate experiment in Section 5.3 and Appendix B.3. Under an explicit ordered probit model and a known lower bound on the Gaussian label-noise standard deviation, we derive a prediction-time drift allowance using observed covariates and previously revealed feedback, without oracle conditional probabilities or unrevealed labels.

We have also added a real-data construction on CLEAR10 in Section 6 and Appendix C.8. Under an explicitly stated normalized-mean stability condition, we derive an observable allowance for the original accumulated target drift. The resulting complete three-term expression is nontrivial at 82.6882.68\\% of the evaluated reporting times. It uses only prediction-time information.

These additions clarify the scope of our contribution: OBCP provides a risk-certification framework when a valid drift allowance is available, and the new experiments demonstrate concrete ways to construct such allowances under explicit assumptions.

Comments on the paper

1 comment

Dear Reviewers,

Authors’ kindly responded to address your concerns. Please check rebuttals and share your opinions with authors. Thanks for your contributions!

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.