- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Conformal prediction provides marginal coverage guarantees under the exchangeability assumption between calibration and test data, but does not guarantee conditional coverage, meaning that coverage across subgroups may deviate from the user-specified level. A common approach to addressing this limitation is to apply normalized conformal prediction, which adapts prediction interval lengths to the estimated difficulty of each prediction by normalizing the residuals. In this paper, the authors introduce a framework for systematically studying how the strength of this adaptation affects the coverage and interval length across subgroups.
Critical strengths
-
The paper addresses a central limitation of conformal prediction: marginal coverage guarantees do not necessarily ensure valid coverage across subgroups. The authors emphasize how this limitation becomes particularly dangerous in clinical applications, where under-coverage of certain patient groups may have harmful consequences. This motivates the need to study how the error adaptation strength affects prediction interval lengths and coverage across groups.
-
The paper is written concisely and is well organized. The motivation, proposed methods, and theoretical and empirical results are well structured.
-
The paper provides a comprehensive empirical evaluation across several datasets, including two clinical cohorts and synthetic data with known heteroscedasticity. The experiments include several target coverage levels, adaptation strengths, and difficulty estimators. They also include comparisons with various baselines such as split conformal prediction, mondrian conformal prediction, CQR, and CASCADE. Additionally, the authors include several evaluation metrics, such as marginal coverage, group coverage, coverage gaps, worst-group coverage, and interval length.
Critical weaknesses
-
The methodological novelty of the paper is quite limited, as the proposed method is closely related to existing normalized conformal prediction methods and seems to be a straightforward extension. In particular, the adaptation strength parameter essentially interpolates between constant-width and fully normalized prediction intervals. Thus, the main contribution is the empirical investigation of adaptation strength and its effect on interval length and coverage across subgroups rather than a development of a novel method.
-
Although the paper demonstrates that the adaptation strength substantially affects conditional coverage and interval length, it remains unclear how a user should choose this parameter in practice. While the authors investigate data-driven selection strategies in Appendix C, their effectiveness remains unclear. Empirically it seems that the preferred strength varies across datasets, coverage levels, and evaluation groups. It remains unclear whether the reported improvements can be reliably achieved through data-driven selections.
-
While the presented method guarantees marginal coverage for each fixed adaptation strength, the authors provide limited theoretical study of how the choice of w affects subgroup coverage and interval length. In particular, it would be interesting to characterize the conditions under which is preferable to full adaptation (), and how the optimal strength depends on the underlying data distribution and the accuracy of the difficulty estimator. Such an analysis could help explain the empirical findings and strengthen the paper overall.
Questions for authors
The font size in Figures 1 and 2 is relatively small, please make it larger.
3 comments
I thank the authors for their efforts in addressing my concerns and for providing additional theoretical analysis and a principled approach to selecting the adaptation strength parameter. However, I remain concerned about the novelty of the overall contribution, as the proposed method appears to be closely related to existing normalized conformal prediction approaches. Furthermore, while the new theoretical analysis provides useful insights, the main result appears to follow relatively directly from the well-established principle that scaling residuals by their conditional quantiles can achieve balanced conditional coverage under appropriate assumptions. Therefore, I will maintain my original ratings.
Response
We thank the reviewer for taking the time to go through our manuscript and revisions! We make it explicit from the initial version that our approach borrows from normalized CP and use it to study coverage allocation in a two-stage model. Our theoretical contribution is the characterization of the balancing strength within the adaptation family and, under additional assumptions, how coverage changes. What motivates our approach is the distribution of such coverage, since in clinical settings some patient subpopulations may remain under or over-covered, and stronger adaptation can improve or worsen this. The added analysis helps explain these patterns. Your feedback helped improve the manuscript, and we are thankful for that.
Response
Thank you so much for the time spent in our manuscript! Your questions motivated an interesting set of experiments that we have included in the manuscript.
W1 (Novelty). We agree that CASCADE-CRC builds on normalized conformal prediction, and Section 2.3 makes the link explicit (w=1/(1+γ)). Our response to Reviewer aHXd, W1 covers novelty and the clinical motivation in full. In short, the revision adds three things: Proposition 1, which gives the balancing strength and shows when it falls below one (W3), the training-only estimate R^α (W2), and a stronger member of the family that pairs R^α with a score predicting the (1−α) error quantile (Section 4.3). Section 4.1 and Appendix Table 6 now show which patients gain and lose coverage as w changes.
Where does it outperform existing methods? The comparison table is in our response to Reviewer gzGc, W3, and in Appendix Table 11. CASCADE-CRC is better than constant length in 5 of 12 settings and worse in none. It wins more often than it loses against Mondrian, conditional conformal prediction (F3), localized CP, rectified scores and batch multivalid CP, and has the lowest CovGap of all methods on PPMI at 80%. Against CQR, the strongest baseline, it ties in 10 of 12 settings. There, the differences are that CASCADE-CRC's intervals are centered on the point prediction and shorter at high targets.
W2 (Choosing the strength in practice).
W3 (When is partial scaling preferable?). Proposition 1 (Section 2.4, proof in Appendix C) answers this. If the conditional error quantile is affine in the score, coverage given the score is flat at wα†. If the scaled error is also continuous and independent of the score (with a unique calibration multiplier at each w), "easy" patients can be over-covered and "hard" patients under-covered below wα†, and the reverse above it. Relative interval length follows Eq. 8. So, under these assumptions and for balance conditional on the score, the balancing strength is below one exactly when part of the error is not tracked by the score (θ0>0). The proposition says nothing about optimal length or balance in other groups. Appendix F shows how this plays out empirically:
Q1 (Figure fonts). Thanks for pointing this out! Both figures now have bigger labels.