- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper considers a problem of certifying conformal systems for a coverage-constrained problem optimizer and introduces CC-SMCS to construct a sequentially valid model set with theoretical guarantees. Importantly, the proposed CC-SMCS is shown to achieve a time-uniform coverage guarantee (Theorem 5.2). Simulation studies demonstrate the empirical performance of the proposed approach. Finally, real data experiments with two datasets in the main text show how it performs in such situations.
Critical strengths
-
The paper considers an important issue: addressing uncertainty regarding different conformal pipelines satisfying coverage constraints. It adopts the martingale confidence sequencing theory.
-
Theorem 5.2 and Proposition 5.3 are important results providing targeted guarantees for the proposed CC-SMCS. These time-uniform results are useful in the problem context. Simulations provide supporting empirical evidence. More importantly, a closed-form projection-based solution (Eq. (16)) is practically appealing.
Critical weaknesses
-
The presentation of the paper made it really difficult to understand the paper's contributions. I had a tough time reading through the paper since the ideas are not well explained; the paper uses diverse terminology throughout the paper without proper context or definitions. Lines 40-42 (A scalar score...an under-covering method): Which score does this refer to? Also, what are the feasible and infeasible methods in lines 44-45? How does the current work differ from some recent works (Arnold et al. 2026)?
-
I found only one baseline comparison (Arnold et al. 2026) in the simulation study without error bars (or are there error bars?). Real data experiments lack such comparisons.
Questions for authors
-
What do lines 61-62 mean?
-
How does the current work differ from some recent works (Arnold et al., 2026)? Could you please clarify what the technical novelties of the proposed approach are? It appears that the specific application to the coverage-constrained problem adopts established sequential confidence sequence analysis.
-
In the simulations, it is stated that the baseline Weak-SMCS uses a different score (lines 430-431). Does it mean it solves a different problem? How is it a fair baseline comparison? In the introduction (lines 40-45), the paper talks about some alternatives. Why are these not considered in the simulation comparisons?
3 comments
I thank authors for their response and revision of the paper. Some of my concerns regarding clarity and presentation are now resolved. Authors response also confirms my concern about the technical novelty: use established martingale confidence sequences to bound coverage risks. The technical contribution is the exact rectangular projection of these simultaneous bounds onto the constrained argmin (Proposition 4.1). The primary contribution lies in the rectangular projection. I believe this projection appears to follow naturally from simultaneous confidence bounds. Even though this construction appears useful and practically useful, I am not convinced that this suggests substantial technical contribution. Therefore, I am maintaining my assessment of the paper.
Response to Reviewer m5LC
Thank you for your follow-up comments and for acknowledging the improvements in clarity. We agree that the risk confidence sequences use established martingale results. Our previous response focused too narrowly on Proposition 4.1. The paper also studies when the reported set can recover the constrained oracle and when the coverage boundary prevents singleton identification. We would like to explain how these results contribute to the theoretical analysis.
1. A boundary limitation that applies to any rule satisfying the same retention requirement.
Proposition 5.3 is separate from the projection result. It considers two candidates with fully known costs. At the coverage boundary, the cheaper candidate is the unique constrained optimizer. Nevertheless, for any rule satisfying simultaneous oracle retention for every parameter value in the Bernoulli model, the probability of ever reporting the correct singleton at the boundary is at most :
This result does not depend on the confidence bounds or projection used by CC-SMCS. It shows that even a unique optimizer with a known, strict cost advantage need not be identified as a singleton with high probability. Uncertainty about feasibility can remain an obstacle after uncertainty about costs has been removed.
2. Recovery conditions and a lower bound near the boundary.
Appendix B.4 gives sufficient exposure and margin conditions under which the reported set eventually equals the current constrained oracle on the joint confidence event.
Appendix B.5 also gives a finite-sample lower bound in the two-candidate example. When , any rule satisfying the same retention requirement that excludes the more expensive candidate by time with probability at least must satisfy
For fixed , , and , this lower bound grows at rate as the feasibility margin approaches zero.
We therefore ask that the theoretical contribution be assessed through these results together. Proposition 4.1 gives the exact set supported by the rectangular risk region. The recovery conditions explain when this set can contract to the oracle. The boundary result and lower bound show limits that apply beyond our particular construction. The established confidence-sequence results provide the risk guarantees, while these results describe what can be concluded about the constrained optimizer. We hope these results clarify the broader theoretical contribution of the paper.
Response to Reviewer m5LC
We thank the reviewer for the comments. We have revised the paper based on your helpful comments.
W1 & Q1: Clarity and Presentation
A1: The scalar score in the original Lines 40 to 42 is an evaluation loss such as the Winkler score. The conformity score is used to construct prediction intervals. Lines 44 to 45 refer to pipelines that satisfy all prescribed coverage constraints or violate at least one. The Introduction explains these roles, and Section 3 defines feasibility.
Lines 61 to 62 concern the scope of the set guarantee. CC-SMCS retains every constrained optimizer on the joint confidence event. A retained pipeline can still have unresolved feasibility. Membership alone gives neither a feasibility certificate nor a guarantee of predictive coverage after selection. Sections 3 and 4 distinguish certified feasibility, possible feasibility, and membership in the projected set.
W1 & Q2: Technical Contributions Beyond Arnold et al. (2026)
A2: Arnold et al. (2026) construct time-uniform confidence sets based on average conditional expected loss differences, whereas we target minimum width subject to coverage constraints. To clarify this distinction, we have added an example in Section 4: for Yt∼Unif(0,1), intervals C1=[0.045,0.997], C2=[0,0.970], and C3=[0.025,0.945] have coverage probabilities (0.952,0.970,0.920) and expected Winkler scores (0.99268,0.98800,0.99300). Under a 95% coverage requirement, C1 is the narrowest feasible interval, whereas C2 minimizes the expected Winkler score. This illustrates that a Winkler-score optimizer can differ from the minimum-width pipeline meeting the coverage requirement. We have clarified this distinction in the Introduction, Related Work, and Section 4.
We use established martingale confidence sequences to bound coverage risks. The technical contribution is the exact rectangular projection of these simultaneous bounds onto the constrained argmin (Proposition 4.1). Among possibly feasible candidates, the projection removes a candidate when a certified feasible candidate has strictly lower observed cost. Given the risk bounds, projection takes O(MR) operations for M candidates and R constraints. Theorem 5.2 combines these steps to give time-uniform retention of every constrained optimizer.
W2 & Q3: Baseline Comparisons
A3: Possible keeps the possibly feasible set and omits cost projection. Hoeffding applies the constrained projection with Hoeffding risk bounds. Empirical implements a two-stage alternative by minimizing width among candidates that pass the previous complete day's empirical coverage check.
Weak-SMCS targets weak superiority based on predictably normalized pairwise Winkler-score differences (Appendix C). Table 2 evaluates all nine procedures on the same simulated paths and reports how often each reported set retains the constrained oracle. For score-based methods this measures alignment with the constrained target. Their original guarantees concern their own targets.
Table 2 reports 95% Monte Carlo intervals, and Figure 1 shows pointwise 95% Monte Carlo bands. The real-data study includes Weak-SMCS and other baselines on the same candidate libraries and evaluation window, with a common deployment rule for all set procedures. Figures 3 and 4 show the empirical coverage and interval width of deployed predictions with bootstrap error bars. Sections 6.1 and 6.2 and Appendices H and I give the comparison protocols and results.