- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates the Split Conformal Prediction method (SCP) under the condition of small calibration sample size and high target coverage, which makes SCP return the entire label space quite frequently. The authors construct Beta-quantile envelopes around each rank of the calibration set, which allows them to classify the ranks as forced-in, ambiguous, or forced-out. Based on that, the authors define two sets: the inner set C⁻ and the outer set C⁺, and any prediction set falling in between these two would have a coverage level between a lower bound LB and an upper bound UB. Using the above, the authors suggest three algorithms: a random mixture of the two sets achieving the target coverage exactly, a projected rule using a reference distribution of the scores to assign ambiguous ranks and a stratified version for subgroups. The authors run experiments on synthetic scores and on three datasets: California Housing, Diabetes and CIFAR-10.
Critical strengths
-
The paper deals with a real problem. For small m and large τ, SCP outputs the entire label space and this commonly occurs in class conditional settings.
-
The concept of forced-in/ambiguous/forced-out is well-clarified so one can easily identify what rank choices are yet to be made at a particular m.
-
The reference rule (Proposition 3.10) ensures that the set remains in [C⁻, C⁺]. That is why the selection of an “unfavorable” reference distribution can not contradict the provided coverage guarantees (LB and UB). This is a reasonable approach to utilizing external data.
Critical weaknesses
-
Missing baselines. The main results compare only against plain SCP. For exact coverage, randomized SCP is the natural baseline but it is not used on the California Housing and Diabetes datasets. Comparisons with the simple max-score rule and with SPI would help evaluate the proposed methods.
-
The description of SPI also needs some correcting. The authors say their method “avoids model misspecification and additional exchangeability assumptions required for synthetic data”, but SPI claims finite-sample coverage without assumptions on the real or synthetic data distributions.
-
Formatting and structure.
- Table 3 goes outside the column margin.
- There is no single experiments section describing the baselines, setup and choice of β, which makes the paper’s structure hard to follow.
- The classification results, the stratified results, related work and limitations are not in the main paper.
- The Figure 14 sub-captions do not match the plot titles.
-
Comparisons are not at matched coverage.
- In cases where calibration is small but coverage is high, SCP tends to return the whole label space. The improvement is measured against SCP’s capped size, so it depends on the cap the authors chose: 100 in the heteroskedastic setting and Range(Y) = 321 on the Diabetes dataset.
- Tables 2 and 3 report gains but not coverage.
- For CIFAR-10, the text reports Ĉλ coverage for only two settings (0.944 at τ = 0.95 and 0.989 at τ = 0.99, both with m = 10).
Questions for authors
-
Compare your methods with randomized SCP, SPI and the max-score threshold at the same coverage, including on the real datasets? Also, what are the exact CIFAR-10 coverages for every setting, and did you break tied scores at random?
-
When k = m + 1, does C⁻ under your definitions include rank m + 1, making it the full label space? If not, which threshold is used instead?
-
Which β values and cutoff convention produced each table? How was C⁻ computed at m = 10, τ = 0.97, where β_min appears to make it the full label space?