acceptodds
Under review as a conference paper at ICLR 2027

Sequential Confidence Sets for Coverage-Constrained Conformal Model Selection

Abstract

Modern conformal forecasting systems often maintain several adaptive pipelines that differ in base forecasters, conformity scores, calibration windows, and update rules. Comparing them is difficult because coverage is a hard constraint, whereas efficiency should be optimized only among feasible pipelines. We formulate this problem as sequential inference for a stochastic constrained argmin. At each time step, the target is the set of minimum-cost pipelines satisfying multiple prefix-average conditional miscoverage constraints. We introduce Coverage-Constrained Sequential Model Confidence Sets (CC-SMCS), which separate certifiably feasible, possibly feasible, and possibly constrained-optimal pipelines. Using simultaneous martingale confidence sequences, CC-SMCS projects a rectangular confidence region onto the constrained argmin and admits an exact closed-form rule. With probability at least , it contains every constrained-optimal pipeline simultaneously over all times. This finite-sample guarantee requires no stationarity or mixing assumptions and remains valid under data-dependent stopping. We also establish an impossibility result for safe certification at the coverage boundary and extend the construction to delayed multi-horizon feedback and outcome-dependent efficiency objectives.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Conformal prediction, Sequential model selection, Anytime-valid inference, Time-series forecasting

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_m5LCOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper considers a problem of certifying conformal systems for a coverage-constrained problem optimizer and introduces CC-SMCS to construct a sequentially valid model set with theoretical guarantees. Importantly, the proposed CC-SMCS is shown to achieve a time-uniform coverage guarantee (Theorem 5.2). Simulation studies demonstrate the empirical performance of the proposed approach. Finally, real data experiments with two datasets in the main text show how it performs in such situations.

Critical strengths
  1. The paper considers an important issue: addressing uncertainty regarding different conformal pipelines satisfying coverage constraints. It adopts the martingale confidence sequencing theory.

  2. Theorem 5.2 and Proposition 5.3 are important results providing targeted guarantees for the proposed CC-SMCS. These time-uniform results are useful in the problem context. Simulations provide supporting empirical evidence. More importantly, a closed-form projection-based solution (Eq. (16)) is practically appealing.

Critical weaknesses
  1. The presentation of the paper made it really difficult to understand the paper's contributions. I had a tough time reading through the paper since the ideas are not well explained; the paper uses diverse terminology throughout the paper without proper context or definitions. Lines 40-42 (A scalar score...an under-covering method): Which score does this refer to? Also, what are the feasible and infeasible methods in lines 44-45? How does the current work differ from some recent works (Arnold et al. 2026)?

  2. I found only one baseline comparison (Arnold et al. 2026) in the simulation study without error bars (or are there error bars?). Real data experiments lack such comparisons.

Questions for authors
  1. What do lines 61-62 mean?

  2. How does the current work differ from some recent works (Arnold et al., 2026)? Could you please clarify what the technical novelties of the proposed approach are? It appears that the specific application to the coverage-constrained problem adopts established sequential confidence sequence analysis.

  3. In the simulations, it is stated that the baseline Weak-SMCS uses a different score (lines 430-431). Does it mean it solves a different problem? How is it a fair baseline comparison? In the introduction (lines 40-45), the paper talks about some alternatives. Why are these not considered in the simulation comparisons?

3 comments
AuthorsOpenReview ↗

Response to Reviewer m5LC

We thank the reviewer for the comments. We have revised the paper based on your helpful comments.

W1 & Q1: Clarity and Presentation

A1: The scalar score in the original Lines 40 to 42 is an evaluation loss such as the Winkler score. The conformity score is used to construct prediction intervals. Lines 44 to 45 refer to pipelines that satisfy all prescribed coverage constraints or violate at least one. The Introduction explains these roles, and Section 3 defines feasibility.

Lines 61 to 62 concern the scope of the set guarantee. CC-SMCS retains every constrained optimizer on the joint confidence event. A retained pipeline can still have unresolved feasibility. Membership alone gives neither a feasibility certificate nor a guarantee of predictive coverage after selection. Sections 3 and 4 distinguish certified feasibility, possible feasibility, and membership in the projected set.

W1 & Q2: Technical Contributions Beyond Arnold et al. (2026)

A2: Arnold et al. (2026) construct time-uniform confidence sets based on average conditional expected loss differences, whereas we target minimum width subject to coverage constraints. To clarify this distinction, we have added an example in Section 4: for Yt∼Unif(0,1)Y_t\sim\mathrm{Unif}(0,1), intervals C1=[0.045,0.997]C_1=[0.045,0.997], C2=[0,0.970]C_2=[0,0.970], and C3=[0.025,0.945]C_3=[0.025,0.945] have coverage probabilities (0.952,0.970,0.920)(0.952,0.970,0.920) and expected Winkler scores (0.99268,0.98800,0.99300)(0.99268,0.98800,0.99300). Under a 95% coverage requirement, C1C_1 is the narrowest feasible interval, whereas C2C_2 minimizes the expected Winkler score. This illustrates that a Winkler-score optimizer can differ from the minimum-width pipeline meeting the coverage requirement. We have clarified this distinction in the Introduction, Related Work, and Section 4.

We use established martingale confidence sequences to bound coverage risks. The technical contribution is the exact rectangular projection of these simultaneous bounds onto the constrained argmin (Proposition 4.1). Among possibly feasible candidates, the projection removes a candidate when a certified feasible candidate has strictly lower observed cost. Given the risk bounds, projection takes O(MR)O(MR) operations for MM candidates and RR constraints. Theorem 5.2 combines these steps to give time-uniform retention of every constrained optimizer.

W2 & Q3: Baseline Comparisons

A3: Possible keeps the possibly feasible set and omits cost projection. Hoeffding applies the constrained projection with Hoeffding risk bounds. Empirical implements a two-stage alternative by minimizing width among candidates that pass the previous complete day's empirical coverage check.

Weak-SMCS targets weak superiority based on predictably normalized pairwise Winkler-score differences (Appendix C). Table 2 evaluates all nine procedures on the same simulated paths and reports how often each reported set retains the constrained oracle. For score-based methods this measures alignment with the constrained target. Their original guarantees concern their own targets.

Table 2 reports 95% Monte Carlo intervals, and Figure 1 shows pointwise 95% Monte Carlo bands. The real-data study includes Weak-SMCS and other baselines on the same candidate libraries and evaluation window, with a common deployment rule for all set procedures. Figures 3 and 4 show the empirical coverage and interval width of deployed predictions with bootstrap error bars. Sections 6.1 and 6.2 and Appendices H and I give the comparison protocols and results.

I thank authors for their response and revision of the paper. Some of my concerns regarding clarity and presentation are now resolved. Authors response also confirms my concern about the technical novelty: use established martingale confidence sequences to bound coverage risks. The technical contribution is the exact rectangular projection of these simultaneous bounds onto the constrained argmin (Proposition 4.1). The primary contribution lies in the rectangular projection. I believe this projection appears to follow naturally from simultaneous confidence bounds. Even though this construction appears useful and practically useful, I am not convinced that this suggests substantial technical contribution. Therefore, I am maintaining my assessment of the paper.

AuthorsOpenReview ↗

Response to Reviewer m5LC

Thank you for your follow-up comments and for acknowledging the improvements in clarity. We agree that the risk confidence sequences use established martingale results. Our previous response focused too narrowly on Proposition 4.1. The paper also studies when the reported set can recover the constrained oracle and when the coverage boundary prevents singleton identification. We would like to explain how these results contribute to the theoretical analysis.

1. A boundary limitation that applies to any rule satisfying the same retention requirement.

Proposition 5.3 is separate from the projection result. It considers two candidates with fully known costs. At the coverage boundary, the cheaper candidate is the unique constrained optimizer. Nevertheless, for any rule satisfying simultaneous oracle retention for every parameter value in the Bernoulli model, the probability of ever reporting the correct singleton at the boundary is at most δ\delta:

Pτ(∃t≥1:S^t={1})≤δ.P_{\tau}\left(\exists t\geq1:\widehat{\mathcal S}_t=\{1\}\right)\leq\delta.

This result does not depend on the confidence bounds or projection used by CC-SMCS. It shows that even a unique optimizer with a known, strict cost advantage need not be identified as a singleton with high probability. Uncertainty about feasibility can remain an obstacle after uncertainty about costs has been removed.

2. Recovery conditions and a lower bound near the boundary.

Appendix B.4 gives sufficient exposure and margin conditions under which the reported set eventually equals the current constrained oracle on the joint confidence event.

Appendix B.5 also gives a finite-sample lower bound in the two-candidate example. When p=τ−Δp=\tau-\Delta, any rule satisfying the same retention requirement that excludes the more expensive candidate by time nn with probability at least 1−β1-\beta must satisfy

n≥kl⁡(1−β,δ)kl⁡(τ−Δ,τ).n\geq\frac{\operatorname{kl}(1-\beta,\delta)}{\operatorname{kl}(\tau-\Delta,\tau)}.

For fixed τ\tau, δ\delta, and β\beta, this lower bound grows at rate Δ−2\Delta^{-2} as the feasibility margin approaches zero.

We therefore ask that the theoretical contribution be assessed through these results together. Proposition 4.1 gives the exact set supported by the rectangular risk region. The recovery conditions explain when this set can contract to the oracle. The boundary result and lower bound show limits that apply beyond our particular construction. The established confidence-sequence results provide the risk guarantees, while these results describe what can be concluded about the constrained optimizer. We hope these results clarify the broader theoretical contribution of the paper.

Reviewer_Ytn5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies sequential selection of conformal pipelines under multiple coverage constraints. It formulates the target as the set of minimum-cost pipelines satisfying prefix-average conditional miscoverage constraints and proposes Coverage-Constrained Sequential Model Confidence Sets (CC-SMCS). The method constructs simultaneous martingale confidence sequences for the constraint quantities, classifies candidates as certifiably feasible or possibly feasible, and projects the resulting confidence region onto the constrained argmin. Its main theoretical result is a time-uniform non-exclusion guarantee: with probability at least (1-δ\delta), the constructed set contains every constrained-optimal pipeline at every time. The paper also discusses the difficulty of safe certification at the coverage boundary and evaluates the method in synthetic scenarios and time-series forecasting experiments.

Critical strengths

The paper explicitly separates coverage feasibility from efficiency optimization, which is a useful perspective for conformal forecasting systems where coverage is a constraint rather than merely one component of a scalar performance score. Moreover, the use of simultaneous martingale confidence sequences supports inference across repeated evaluations and data-dependent stopping without requiring stationarity or mixing assumptions.

Critical weaknesses
  1. Limited practical guarantee. Although the paper considers hard coverage constraints, the returned set may still contain methods whose coverage feasibility has not been established. Thus, the method guarantees that truly optimal feasible methods are not missed, but it does not guarantee that every retained method is feasible. This limits its usefulness when coverage violations must be avoided.

  2. In addition, the experiments do not provide a sufficiently convincing comparison with baselines targeting the same constrained selection objective.

  3. Section 4.1 is poorly written and full of equations. It reads like a draft instead of a formal manuscript.

Questions for authors
  1. If coverage is essential, particularly in safety-critical applications, why does CC-SMCS retain pipelines that are “possibly feasible but more efficient”? Such pipelines have not been certified to satisfy the coverage constraint, so their inclusion appears inconsistent with treating coverage as a hard requirement.

  2. If slight coverage violations are acceptable, what does CC-SMCS gain over optimizing a scalar loss that combines coverage and efficiency? What practical decision does the constrained-oracle confidence set support beyond the separately certified-feasible set, particularly when coverage violations are unacceptable?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Ytn5

We thank the reviewer for the comments. We have revised the paper based on your helpful comments.

W1 & Q1: Coverage Constraints and Practical Guarantees

A1: Coverage requirements are hard constraints on the oracle target. A narrower candidate whose risk bounds still permit feasibility can be the constrained optimum. Excluding it before the evidence rules it out can remove the true optimizer. CC-SMCS retains candidates compatible with constrained optimality and reports the certified feasible set separately. The retained set can include an infeasible candidate whose risk bounds still permit feasibility.

Theorem 5.2 gives a joint event with probability at least 1−δ1-\delta on which, at every evaluation time, all certified candidates satisfy the prescribed prefix-average conditional miscoverage constraints and every constrained optimizer is retained. The guarantee also permits zero coverage tolerance. The certified feasible set identifies candidates with established feasibility, while CC-SMCS identifies candidates that remain compatible with constrained optimality. These outputs support evaluating a new pipeline against certified alternatives while accounting for unresolved feasibility. Section 5 also bounds the extra observed cost of requiring certification, as discussed in A4. Coverage for subsequent predictions or adaptive switching requires a separate guarantee.

W2: Baseline Comparisons

A2: Section 6.1 compares nine procedures. Possible uses the same coverage-risk bounds as CC-SMCS and omits cost projection. Hoeffding uses the same constrained projection with Hoeffding risk bounds. In the group-coverage experiment, CC-SMCS reduces the mean set size relative to Possible from 3.000 to 2.026 under the overall constraint and from 2.428 to 1.688 under overall and group constraints. Both methods retain the respective oracle throughout evaluation on all 1,000 paths in both designs (Appendix H.1, Table 14).

W3: Presentation of the Method

A3: Section 4 explains how the risk bounds define the possibly feasible and certified feasible sets, then how observed costs determine the projected set. Appendix B gives the technical arguments.

Q2: Scalar Loss and the Value of the Constrained-Oracle Confidence Set

A4: Our framework treats coverage as an explicit constraint rather than a penalty in a scalar loss. The coverage threshold is prespecified, and zero additional tolerance (εr=0\varepsilon_r=0) is also permitted. Our objective is to identify the minimum-width pipeline satisfying this constraint, which need not coincide with the optimizer of a scalar loss such as the Winkler score. To illustrate this distinction, we have added an example in Section 4. For Yt∼Unif(0,1)Y_t\sim\mathrm{Unif}(0,1), intervals C1=[0.045,0.997]C_1=[0.045,0.997], C2=[0,0.970]C_2=[0,0.970], and C3=[0.025,0.945]C_3=[0.025,0.945] have coverage probabilities (0.952,0.970,0.920)(0.952,0.970,0.920) and expected Winkler scores (0.99268,0.98800,0.99300)(0.99268,0.98800,0.99300). Under a 95% coverage requirement, C1C_1 is the narrowest feasible interval, whereas C2C_2 minimizes the expected Winkler score.

The confidence set keeps candidates whose feasibility remains unresolved available for further comparison. Proposition 5.3 explains the difficulty of certification at the coverage boundary. In Simulation 1 at δ=0.10\delta=0.10, CC-SMCS retains the boundary oracle M4M_4 throughout evaluation on all 1,000 paths. It certifies M4M_4 at least once during evaluation on one path (Table 13).

Section 5 bounds the extra observed cost of requiring certification. When the certified feasible set is nonempty, subtracting the lowest possibly feasible cost from the lowest certified feasible cost gives an upper bound on the gap between the cheapest certified pipeline and the constrained optimum on the joint confidence event. A small bound shows that choosing the cheapest certified pipeline incurs little extra observed cost. A larger bound leaves room for an unresolved candidate to improve on the certified choice.

I thank the authors for their response, which addresses my questions about the empirical results. However, I think the contribution of this paper is not significant for ICLR. In addition, the writing still needs to be improved. Therefore, I will maintain my initial assessment.

Reviewer_cqEFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CC-SMCS, a method for sequentially screening conformal prediction pipelines for time-series tasks. It maintains a set of potentially optimal pipelines based on coverage constraints and interval efficiency, with a time-uniform guarantee that the true constrained optimizers are retained. The experiments generally support the claimed screening behavior.

Critical strengths
  1. The method explicitly separates coverage requirements from efficiency optimization. As new observations arrive, it can exclude pipelines with sufficient evidence of undercoverage or those dominated by a certified feasible alternative.
  2. The simulations and real-data experiments support the method’s ability to retain optimal candidates and adjust the candidate set as coverage and interval widths change.
Critical weaknesses
  1. The practical benefit over existing adaptive conformal methods needs clearer justification. ACI already adjusts its effective miscoverage level online to maintain long-run coverage. Although this differs from filtering multiple pipelines, the paper should explain when the additional screening layer provides meaningful benefits over a well-tuned adaptive method. Efficiency also depends on the nonconformity score and calibration strategy; stronger configurations using learned scores or group-aware calibration would provide useful comparisons.
  2. Although ACI and other conformal methods are included as candidates, the paper does not directly compare an end-to-end CC-SMCS deployment strategy against using ACI alone. Such a comparison would require specifying how to select a single pipeline from the retained set and how to handle an empty set, then evaluating coverage, interval width, and computational cost.
Questions for authors

See weakness.

2 comments
AuthorsOpenReview ↗

Response to Reviewer cqEF

We thank the reviewer for the comments. We have revised the paper based on your helpful comments.

W1: Practical Benefits Beyond Adaptive Conformal Methods

A1: Our primary objective is to evaluate conformal pipelines under prescribed coverage constraints, using mean interval width as the efficiency objective. A new method can enter a prespecified library and be assessed alongside established pipelines as outcomes accumulate. Theorem 5.2 guarantees simultaneous retention of every constrained optimizer over time, allowing researchers to inspect the set repeatedly and report it at a data-dependent time.

ACI adjusts the miscoverage level within a pipeline. CC-SMCS compares pipelines with different scores and calibration strategies. In a library containing tuned ACI, the screening step can rule it out if the risk bounds establish undercoverage or if another pipeline with smaller observed mean width is certified feasible.

We select ACI's learning rate and residual window and AgACI's residual window before testing by mean 95% Winkler loss. AgACI retains all 30 experts. The library includes CQR with learned conditional quantiles and MVP$^{\ast}$ with overall, daytime, and nighttime calibration groups. Appendices F.3 and F.4 describe the finite-width MVP$^{\ast}$ construction and the tuning protocol.

W2: End-to-End Deployment Comparison

A2: Section 6.2 specifies a common deployment rule. Before each prediction, the rule keeps the current pipeline if it belongs to the set reported after the last outcome. Otherwise it selects a retained pipeline with the smallest past cumulative mean width, breaking ties in a fixed order. An empty set leads to tuned ACI. All candidate pipelines continue updating.

Figures 3 and 4 compare empirical coverage and mean interval width on twelve tasks from four datasets and three forecasting backbones. On French electricity prices with DLinear, CC-SMCS attains 96.52% empirical coverage compared with 95.05% for tuned ACI, using 88.3% of ACI's mean width (Table 18). Possible uses the same coverage bounds and omits cost projection. Under the common deployment rule, Possible matches ACI in both metrics on all twelve tasks, which isolates the effect of cost projection in the comparison with CC-SMCS. CC-SMCS itself continues to use ACI on several tasks.

On French prices with DLinear, the median online time for the full seven-candidate system is 2.096 ms per update, compared with 0.079 ms for tuned ACI alone (Table 3). Most of the full system time is spent running and updating the candidates. The median paired increment over maintaining the shared candidate library is 0.297 ms per update. Appendix I.1 gives the timing protocol.

Thank you for adding the experiments and clarifying the deployment rule. I acknowledge the novelty of your sequential screening framework. However, the experiments remain incomplete, particularly against classical methods that explicitly train nonconformity scores to improve interval efficiency (e.g., Uncertainty Quantification over Graph with Conformalized Graph Neural Networks). I therefore maintain my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.