- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the computational problem behind honest conformal prediction under label shift when the importance weights are unknown and only known to lie in a finite-sample confidence region. The authors propose QuiCK, with a coordinate-wise exact solver for hyperbox confidence regions and an LP-based solver for joint polytope regions, together with warm starts and sensitivity-based pruning to compute worst-case weighted quantile thresholds for all classes. The statistical coverage guarantee is inherited from worst-case calibration over a valid weight confidence region, while the main contribution of this paper is making that optimization substantially faster.
Critical strengths
S1. The paper identifies a concrete computational bottleneck in honest conformal prediction.
S2. The separation between calibration-sample counts and class-level weights is useful.
S3. The experimental evaluation covers both statistical validity and computational behavior.
Critical weaknesses
W1. The statistical framework itself is largely inherited from prior work, and the novelty is primarily computational.
The honest conformal construction based on a confidence region for the unknown importance weights is already present in Li et al. (2026), and the same prior work also reports exact maximization over marginal hyperboxes. The paper should be much more precise about how much of the statistical methodology is actually new.
W2. The computational experiments omit the most relevant exact baselines.
The paper's main claim is computational efficiency, yet the primary nontrivial computational baseline is BBHR, a heuristic boundary-biased MCMC procedure introduced by the authors. There is no runtime comparison against straightforward exact alternatives such as Charnes-Cooper, Dinkelbach iterations, a naive threshold scan with LPs, or a per-class LP/binary-search implementation without the proposed warm start and sensitivity pruning. Without these comparisons, it is difficult to quantify how much speedup comes from the genuinely new algorithmic ideas.
W3. The proposed honest prediction sets can be extremely conservative.
For a nominal 90% target, the reported coverage is often around 97% in small-sample regimes,. Thus, the method obtains honesty partly by paying a very large statistical-efficiency cost. Computational efficiency alone does not resolve this limitation.
W4. The claimed high-dimensional scalability is partly masked by increasing the amount of estimation data with .
When studying the number of classes, the paper sets . Thus larger label spaces are automatically given proportionally more data. This isolates computational scaling, but it does not demonstrate statistical scalability at a fixed data budget. Moreover, constructing the weight confidence region requires estimating a confusion matrix, which itself becomes statistically and computationally difficult as grows.
W5. The empirical language sometimes overstates what the experiments establish.
Statements such as the experiments "strictly guarantee" or "perfectly provide" coverage conflate empirical coverage observed over 200 trials with the theorem-level guarantee. Likewise, the manuscript repeatedly uses highly promotional phrases such as "fundamentally fuses," "absolute statistical validity," and "remarkable" scalability. A more precise and less promotional presentation would improve the paper.
Questions for authors
See weaknesses
Thank you for the thorough review, especially for recognizing the value of separating calibration-sample counts from class-level weights. We will add more exact baselines and tone down the overstated language.