acceptodds
Under review as a conference paper at ICLR 2027

Coverage Allocation in Two-Stage Conformal Prediction

Abstract

Conformal prediction (CP) provides marginal coverage guarantees under exchangeability and is generally evaluated through mean coverage and interval length. In clinical settings, mean coverage says nothing about whether coverage is balanced across patient groups. One way to address this is by adapting interval length to prediction difficulty, but stronger adaptation does not necessarily lead to better intervals, and can leave patient subpopulations under-covered. We introduce CASCADE-CRC, a framework for studying how adaptation strength affects coverage and interval length reallocation in two-stage conformal prediction. We evaluate on inpatient and outpatient Parkinson's disease cohorts, an external zero-inflated insurance dataset, and synthetic data with known heteroscedasticity. Our experiments reveal that adaptation strength reallocates interval length and, with it, coverage across groups, reducing the coverage gap (CovGap) by on the outpatient cohort while keeping marginal coverage, with benefits that vary by setting. Our findings emphasize the importance of adaptation strength as a modeling choice, motivating joint evaluation of group coverage and interval length.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Uncertainty Quantification, Conformal Prediction, Parkinson's Disease, Neuroscience, Neurology

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GZBwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Conformal prediction provides marginal coverage guarantees under the exchangeability assumption between calibration and test data, but does not guarantee conditional coverage, meaning that coverage across subgroups may deviate from the user-specified level. A common approach to addressing this limitation is to apply normalized conformal prediction, which adapts prediction interval lengths to the estimated difficulty of each prediction by normalizing the residuals. In this paper, the authors introduce a framework for systematically studying how the strength of this adaptation affects the coverage and interval length across subgroups.

Critical strengths
  1. The paper addresses a central limitation of conformal prediction: marginal coverage guarantees do not necessarily ensure valid coverage across subgroups. The authors emphasize how this limitation becomes particularly dangerous in clinical applications, where under-coverage of certain patient groups may have harmful consequences. This motivates the need to study how the error adaptation strength affects prediction interval lengths and coverage across groups.

  2. The paper is written concisely and is well organized. The motivation, proposed methods, and theoretical and empirical results are well structured.

  3. The paper provides a comprehensive empirical evaluation across several datasets, including two clinical cohorts and synthetic data with known heteroscedasticity. The experiments include several target coverage levels, adaptation strengths, and difficulty estimators. They also include comparisons with various baselines such as split conformal prediction, mondrian conformal prediction, CQR, and CASCADE. Additionally, the authors include several evaluation metrics, such as marginal coverage, group coverage, coverage gaps, worst-group coverage, and interval length.

Critical weaknesses
  1. The methodological novelty of the paper is quite limited, as the proposed method is closely related to existing normalized conformal prediction methods and seems to be a straightforward extension. In particular, the adaptation strength parameter essentially interpolates between constant-width and fully normalized prediction intervals. Thus, the main contribution is the empirical investigation of adaptation strength and its effect on interval length and coverage across subgroups rather than a development of a novel method.

  2. Although the paper demonstrates that the adaptation strength substantially affects conditional coverage and interval length, it remains unclear how a user should choose this parameter in practice. While the authors investigate data-driven selection strategies in Appendix C, their effectiveness remains unclear. Empirically it seems that the preferred strength varies across datasets, coverage levels, and evaluation groups. It remains unclear whether the reported improvements can be reliably achieved through data-driven selections.

  3. While the presented method guarantees marginal coverage for each fixed adaptation strength, the authors provide limited theoretical study of how the choice of w affects subgroup coverage and interval length. In particular, it would be interesting to characterize the conditions under which w<1w<1 is preferable to full adaptation (w=1w=1), and how the optimal strength depends on the underlying data distribution and the accuracy of the difficulty estimator. Such an analysis could help explain the empirical findings and strengthen the paper overall.

Questions for authors

The font size in Figures 1 and 2 is relatively small, please make it larger.

3 comments

Thank you so much for the time spent in our manuscript! Your questions motivated an interesting set of experiments that we have included in the manuscript.

W1 (Novelty). We agree that CASCADE-CRC builds on normalized conformal prediction, and Section 2.3 makes the link explicit (w=1/(1+γ)w=1/(1+\gamma)). Our response to Reviewer aHXd, W1 covers novelty and the clinical motivation in full. In short, the revision adds three things: Proposition 1, which gives the balancing strength and shows when it falls below one (W3), the training-only estimate R^α\hat R_\alpha (W2), and a stronger member of the family that pairs R^α\hat R_\alpha with a score predicting the (1−α)(1-\alpha) error quantile (Section 4.3). Section 4.1 and Appendix Table 6 now show which patients gain and lose coverage as ww changes.

Where does it outperform existing methods? The comparison table is in our response to Reviewer gzGc, W3, and in Appendix Table 11. CASCADE-CRC is better than constant length in 5 of 12 settings and worse in none. It wins more often than it loses against Mondrian, conditional conformal prediction (F3), localized CP, rectified scores and batch multivalid CP, and has the lowest CovGap of all methods on PPMI at 80%. Against CQR, the strongest baseline, it ties in 10 of 12 settings. There, the differences are that CASCADE-CRC's intervals are centered on the point prediction and shorter at high targets.

W2 (Choosing the strength in practice).

  • Rule (Section 2.4). Set w=R^αw=\hat R_\alpha, the training-only estimate defined in the general response, then calibrate as usual.
  • Guarantee. Coverage of at least 1−α1-\alpha holds exactly, because no calibration or evaluation outcome is used. Exact balance cannot be guaranteed in finite samples for an estimated strength. Under the conditions of Proposition 1, and if the error law given the score does not depend on how many encounters a patient contributes, R^α\hat R_\alpha is consistent for the balancing strength (Appendix C). Weighting encounters to match one-per-patient sampling gives R^α=0.22\hat R_\alpha=0.22 instead of 0.26 on PPMI at 80% and raises CovGap by 0.004 [0.001, 0.007]. At the other targets and on XYZ Health the change is at most 0.003 (Appendix E).
  • Evidence (Section 4.3, Table 3). With the quantile score, w=R^αw=\hat R_\alpha improves balance over constant length in 5 of 12 settings and is worse in none. Among the methods compared, it also falls least behind the best fixed strength chosen in hindsight. On XYZ Health, where no strength helps, R^α≈0\hat R_\alpha\approx0 recovers constant length.
  • Guide (Section 5, Appendix K). If R^α\hat R_\alpha is near 0, use constant length. If it is between 0 and 1, use it, and if it is near 1, use full scaling. Recompute it for each target. With only hundreds of magnitude-stage training units, do not default to w=1w=1. With fewer than about 25 units per evaluation group, report the noise floor before ranking methods. Use CQR when errors are asymmetric and the range need not contain the point forecast, and Mondrian or conditional conformal prediction when coverage must hold within large predefined groups.

W3 (When is partial scaling preferable?). Proposition 1 (Section 2.4, proof in Appendix C) answers this. If the conditional error quantile is affine in the score, coverage given the score is flat at wα†w^\dagger_\alpha. If the scaled error is also continuous and independent of the score (with a unique calibration multiplier at each ww), "easy" patients can be over-covered and "hard" patients under-covered below wα†w^\dagger_\alpha, and the reverse above it. Relative interval length follows Eq. 8. So, under these assumptions and for balance conditional on the score, the balancing strength is below one exactly when part of the error is not tracked by the score (θ0>0\theta_0>0). The proposition says nothing about optimal length or balance in other groups. Appendix F shows how this plays out empirically:

  • estimator accuracy: the balancing strength rises with the score's training size and with better-tuned scores. In simulations it stays below one even for the true noise scale, consistent with the residual also containing the point predictor's error, which the noise scale does not track.
  • score target: scores that target the error quantile give higher strengths than mean-error scores.
  • data distribution: stronger heteroscedasticity raises the strength, and large errors that do not grow with the score lower it, mostly at high targets (freMTPL2).

Q1 (Figure fonts). Thanks for pointing this out! Both figures now have bigger labels.

I thank the authors for their efforts in addressing my concerns and for providing additional theoretical analysis and a principled approach to selecting the adaptation strength parameter. However, I remain concerned about the novelty of the overall contribution, as the proposed method appears to be closely related to existing normalized conformal prediction approaches. Furthermore, while the new theoretical analysis provides useful insights, the main result appears to follow relatively directly from the well-established principle that scaling residuals by their conditional quantiles can achieve balanced conditional coverage under appropriate assumptions. Therefore, I will maintain my original ratings.

We thank the reviewer for taking the time to go through our manuscript and revisions! We make it explicit from the initial version that our approach borrows from normalized CP and use it to study coverage allocation in a two-stage model. Our theoretical contribution is the characterization of the balancing strength within the adaptation family and, under additional assumptions, how coverage changes. What motivates our approach is the distribution of such coverage, since in clinical settings some patient subpopulations may remain under or over-covered, and stronger adaptation can improve or worsen this. The added analysis helps explain these patterns. Your feedback helped improve the manuscript, and we are thankful for that.

Reviewer_aHXdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how the strength of residual-scale adaptation redistributes prediction-interval length and coverage across groups. CASCADE-CRC interpolates between a constant scale and a normalized difficulty estimate through sigma_w(x) = (1−w) + w·sigma_tilde(x), and separately calibrates each fixed strength. With binary miscoverage loss, this is normalized split conformal prediction. Experiments use two Parkinson's disease cohorts, a positive-claim insurance subset, and heteroscedastic synthetic data. The main finding is that intermediate adaptation can improve group coverage balance, while full adaptation can overcorrect; the preferred strength depends on the score, target coverage, and evaluation partition.

Critical strengths

S1. The paper isolates a useful modeling choice and reports its failures rather than claiming uniform superiority. Holding the predictor, calibration observations, and evaluation partition fixed makes the strength sweep interpretable. The study shows that more adaptation can worsen coverage balance, that benefits fade or reverse at higher targets, and that the preferred strength changes when groups are defined by previous dose. These are worthwhile findings for practitioners who might otherwise assume full residual scaling is always preferable.

S2. The scope of validity and clinical interpretation is carefully stated. The manuscript distinguishes fixed-strength marginal validity from group validity, data-dependent strength selection, and end-to-end routing. Patient-level separation and one eligible calibration/evaluation encounter per patient address an important dependence issue. The dose-scale and zero-exclusion analyses further explain why improved CovGap does not establish safe or useful medication recommendations. These qualifications increase the credibility of the empirical study.

Critical weaknesses

W1. The methodological novelty and specifically two-stage contribution are limited. Section 2.3 explicitly identifies the interpolation as a reparameterization of the sensitivity constant in normalized conformal prediction. Under the implemented binary loss, CRC reduces to the usual normalized split-conformal quantile. Equation (10) also shows that the CASCADE comparator already uses the same affine scale form with a different difficulty score. Thus, the contribution is primarily the experimental characterization of adaptation strength, not a new conformal calibration method or risk-control result.

The two-stage setting contributes less to the actual evaluation than the framing suggests. Section 2.1 routes records using the observed fact that a change occurred, not a first-stage prediction. First-stage outputs enter only as candidate difficulty features, and Section 4.5 finds little benefit from them. The insurance experiment likewise evaluates positive outcomes. These choices are legitimate for a conditional regression study, but do not test uncertainty propagation or decision errors in a deployed two-stage pipeline. I appreciate that the paper expressly disclaims an end-to-end guarantee; the remaining issue is whether its name and motivation convey the actual contribution accurately.

An empirical contribution can still be sufficient. Here, however, stronger explanatory evidence is needed about why partial scaling helps: for example, whether it compensates for estimation noise, regression to the mean, a mean-versus-tail residual target, or genuine mismatch in the difficulty ranking. The oracle-noise-scale and independent synthetic design are good foundations, but the current results mostly identify favorable strengths retrospectively rather than establishing a transferable principle for choosing them.

W2. The headline improvement is retrospective, while finite-group noise and valid strength selection remain unresolved. The 31.7% improvement on PPMI is a reduction from approximately 0.060 to 0.041 at a favorable evaluated strength, an absolute change of about 1.9 percentage points. At the same target, CQR already achieves CovGap of 0.043 with mean length 1.55, compared with 0.041 and 1.63 for w=0.50 (Table 1). This is evidence of an allocation effect, but does not yet demonstrate a meaningful advantage over CQR. The most favorable strength also changes by dataset, target, score, and grouping.

Appendix C appropriately distinguishes an empirical selector from a certifying selector and states that its certificate does not extend the fixed-strength marginal guarantee to the selected rule. However, it does not provide a numerical table of selected strengths, certification frequencies, out-of-sample coverage, lengths, and gaps. The main text's statement that selection improves three of four datasets cannot substitute for those results. A deployable selection protocol should reserve final calibration data after strength selection, or otherwise justify validity of the selected procedure.

There is also a substantial estimation issue in interpreting CovGap and FSC as population imbalance. CovGap averages absolute empirical deviations, and FSC takes the minimum empirical coverage; both are affected by group sample size. The XYZ Health groups average only about 17–24 evaluation patients (Table 5). At 90% nominal coverage, an exactly calibrated group of roughly 20 patients can exhibit an absolute empirical coverage error of several percentage points from binomial variation alone. Indeed, Figure 4 contrasts a split-averaged CovGap of 0.059 with a deviation of only 0.009 computed from the averaged group coverages. This does not invalidate the paired comparisons, but it does mean that the level of the observed gap cannot be interpreted directly as systematic misallocation. The manuscript acknowledges split sensitivity and reuses the cohorts transparently; a matched sample-size null experiment or coverage uncertainty analysis is still needed to show which apparent allocation patterns exceed measurement noise.

Questions for authors

Q1 — W1, contribution clarification. What is the main advance beyond the existing normalized-conformal sensitivity parameter and the affine scale already used by CASCADE? Can you state the contribution explicitly as an empirical study, and explain which conclusions specifically require a two-stage setting rather than ordinary conditional regression on nonzero outcomes?

Q2 — W1, explanatory evidence. Using the existing synthetic benchmark, can you compare oracle difficulty, estimated mean-residual difficulty, and estimated tail-residual difficulty across calibration and score-training sample sizes? Does the favorable intermediate strength persist with a well-estimated oracle-aligned scale, or is it primarily compensating for score-estimation error? This distinction would change the generality of the central conclusion.

Q3 — W2, essential selection results. Please provide the complete results of Appendix C: selected-strength distributions, certification rates, test CovGap/FSC, marginal coverage, and length for both selection rules. Can a selection-then-independent-final-calibration protocol retain validity and improve on constant scaling and CQR without using evaluation outcomes?

Q4 — W2, uncertainty and practical significance. How much of the XYZ Health gap and worst-group shortfall would be expected from its small group counts under equal population coverage? A matched null simulation or groupwise uncertainty analysis, together with paired comparisons against CQR at independently chosen strengths, would help distinguish systematic reallocation from finite-test noise and retrospective selection.

4 comments

Thank you so much for your review of our manuscript. We have made sure to address these concerns.

W1 (Novelty). We agree with your three points, and Section 2.3 now states them. Normalized and localized conformal prediction have been around for years (Papadopoulos et al., 2011, Lei et al., 2018, Guan, 2023), and our ww reparameterizes the normalized sensitivity constant. As far as we know, though, these tools have not been used to ask how the degree of adaptation allocates coverage across patient groups, or what that means clinically. That is the gap our paper addresses. CASCADE uses one fixed parameter, whereas CASCADE-CRC studies the strength itself and which strength balances coverage. The improvement has three parts:

  • The method. CASCADE-CRC calibrates each strength for the magnitude stage of a two-stage clinical model, and now picks it with R^α\hat R_\alpha (general response) without giving up the coverage guarantee (Section 2.4).
  • The study of ww. Moderate adaptation balances coverage across patient groups, and full adaptation over-corrects (Sections 4.1–4.3). Proposition 1 (new, Section 2.4) explains when. With r=∣Y−f^(X)∣r=|Y-\hat f(X)|, S=σ~(X)S=\tilde\sigma(X) and qα(s)q_\alpha(s) the (1−α)(1-\alpha) quantile of rr given S=sS=s, if qα(s)=θ0+θ1sq_\alpha(s)=\theta_0+\theta_1 s, then at wα†=θ1/(θ0+θ1)w^\dagger_\alpha=\theta_1/(\theta_0+\theta_1) the scale is proportional to qαq_\alpha and coverage given SS converges to 1−α1-\alpha at every score value. If r/qα(S)r/q_\alpha(S) is also continuous and independent of SS (with a unique calibration multiplier at each ww), coverage moves monotonically in SS at any other ww, and at wα†w^\dagger_\alpha it equals ⌈(m+1)(1−α)⌉/(m+1)\lceil(m+1)(1-\alpha)\rceil/(m+1) at every score value in finite samples, averaging over the calibration data. So wα†<1w^\dagger_\alpha<1 exactly when θ0>0\theta_0>0, that is, when part of the error is not tracked by the score. Our training-size experiments are consistent with score-estimation error pushing the balancing strength below one on our data (Appendix F).
  • Who gains and loses coverage. In Parkinson's disease dosing, constant length over-covers the lowest-difficulty stratum, mostly patients with a recorded prior dose, and under-covers the highest, which has more patients without one (46% against 24%) and larger dose changes. CASCADE-CRC allocates coverage toward the latter (Section 4.1, Appendix Table 6, Appendix E).

W1 (Two-stage setting). Section 2.1 now separates two settings. In the main analysis, the model predicts the amount of the change, and we use the encounters with a recorded dose change in our data. In the new automated mode (Appendix E), the Stage-1 classifier decides whether the dose changes. Encounters predicted to change get a CASCADE-CRC interval calibrated on those encounters, and the rest get an interval calibrated on encounters predicted not to change. We report coverage conditional on the Stage-1 decision and overall, and how classifier errors affect the intervals. Q1 lists which conclusions depend on the two-stage architecture.

W1 (Why partial scaling helps). Under Proposition 1 partial scaling balances coverage along the score when part of the error is not tracked by the score (θ0>0\theta_0>0), and R^α\hat R_\alpha turns that into a rule. We tested each candidate listed (Appendix F):

  • estimation noise: the score's training size, in simulations and on PPMI
  • regression to the mean: an oracle scale with added noise
  • mean versus tail target: mean-residual against tail-residual and quantile scores
  • ranking mismatch: different scores at a fixed strength (Section 4.6, Appendix Table 4)

W2 (Headline result and CQR). The headline now uses w=R^αw=\hat R_\alpha (Section 4.3). On PPMI at 80%, CASCADE-CRC with the quantile score reaches CovGap 0.041 at mean length 1.59 (paper's score: 0.046 at 1.60, constant length: 0.060 at 1.60), against 0.043 at 1.55 for CQR. Unlike the retrospective 0.041 at w=0.50w=0.50, this strength is chosen on training data alone. The constant-length gap it starts from (0.060) is well above the noise floor (0.036), but after the Nadeau–Bengio correction for patients shared across splits its interval includes zero (Q4). It ties CQR in 10 of 12 settings (Table 3), so we do not claim better balance than CQR. What differs is the form of the interval. CASCADE-CRC is symmetric around f^(x)\hat f(x), so every interval contains the model's point forecast of the dose change, whereas CQR's interval excludes it for 5.2% of PPMI patients. CASCADE-CRC's intervals are also shorter at high targets (PPMI 95%: 2.80 with the paper's score and 2.85 with the quantile score, against 3.19, Table 1, Appendix Table 11). And the favorable strength does change by dataset, target and score, which is why R^α\hat R_\alpha is re-estimated for each.

W2 (Selection protocol). We addressed both points. First, Appendix G now gives the numerical table for both selectors of the original Appendix C at every target (selected strengths, certification frequencies, out-of-sample coverage, lengths and gaps). The empirical selector picks the strength with the lowest CovGap on half of the calibration set. The certifying selector moves away from constant length only when confidence bounds show a lower CovGap. Second, R^α\hat R_\alpha is a deployable protocol: the strength is chosen on Dhold\mathcal D_{\mathrm{hold}}, which is training data and shares no patients with the calibration or evaluation sets, and the full calibration set is then used only to compute λ^\hat\lambda for that fixed strength. No calibration data is spent on selection, so validity holds exactly (Section 2.4).

W2 (Finite-group noise). We added a noise floor and per-group Clopper–Pearson intervals (Section 3, Appendix E). Please see Q4.

Q1 (Main advance and two-stage conclusions). The main advance is in W1 (Novelty). Most of our conclusions concern the magnitude stage and would also hold for conditional regression on nonzero outcomes. The two-stage setting enters in four places:

  • (a) Calibration. Patients contribute several encounters, including encounters without a change. Sampling one change per patient avoids treating repeated encounters as independent units, and exchangeability then rests on how patients and their eligible encounters are sampled (Section 2.1).
  • (b) Deployment. When Stage 1 decides, its false positives add a point mass at zero to the calibration data, and mean interval length rose for every method. Coverage still holds conditional on the decision, but balance changes (Appendix E).
  • (c) Stage-1 features. Adding Stage-1 outputs to the difficulty score changes balance little compared with the choice of strength (Section 4.6).
  • (d) Why the balancing strength is low. Zero inflation leaves the magnitude stage only the encounters with a change to learn the score from (about 720 on PPMI, 140 on XYZ Health). Our training-size experiments are consistent with the resulting estimation error pushing the balancing strength well below one (Appendix F).

Q2 (Oracle versus estimated difficulty). Yes (Appendix F). We ran simulations with exact conditional coverage, comparing oracle, oracle with noise, estimated mean-residual, estimated tail-residual and log-linear scores. The results are consistent with the intermediate strength compensating for score-estimation error:

scorebest ww as the score's training set growslarge-sample slope ratio (20,000 draws)
oracle (true σ)0.850.85
oracle × noise (log-sd 0.3 / 0.6)0.55 / 0.330.53 / 0.28
estimated mean-residual (paper)0.31 → 0.730.32 → 0.72
estimated tail-residual0.50 → 0.900.50 → 0.90

The best strength tracks this slope ratio, rises as the score improves and barely moves with calibration size. Even the oracle balances best at 0.85 rather than one. The residual ∣Y−f^(X)∣|Y-\hat f(X)| also contains the error of f^\hat f, which the true noise scale does not track, and the oracle result is consistent with that error favoring partial adaptation. On real data, R^α\hat R_\alpha rises with the size of Dhold\mathcal D_{\mathrm{hold}} on PPMI.

Q3 (Complete selection results). Appendix G reports both selectors (W2) for every dataset and target. The certifying selector never certified a strength, so it always returned constant length. The empirical selector helps where the effect is large (PPMI, synthetic) and is neutral elsewhere. On your second question: yes, R^α\hat R_\alpha is selection followed by independent calibration (W2). It improves on constant length (Table 3). Appendix E provides a comparison with CQR.

Q4 (Noise and comparison with CQR). Most of the XYZ Health gap and worst-group shortfall is explained by group size. At 90% and 95% constant length sits at the noise floor, and the worst-group coverage expected under exact calibration (0.72 at 80%) lies within the observed FSC range (0.64–0.75). On PPMI the constant-length gap (0.060) is well above its floor (0.036), and a noise-adjusted root-mean-square gap, which subtracts the expected binomial variance from each group's squared deviation, falls from 0.062 with constant length to 0.033 with R^α\hat R_\alpha. Against CQR at R^α\hat R_\alpha, the paired CovGap differences are +0.002 [−0.006, +0.011] at 80% and +0.003 [−0.004, +0.010] at 90%. Because the 30 splits reuse the same patients, the split-to-split intervals can be too narrow. After the Nadeau–Bengio correction for this overlap, every clinical CovGap interval includes zero.

The response clarifies the mechanism and selection protocol, but novelty remains a concern, and the dependence-adjusted clinical results do not establish a clear improvement in coverage balance over the baselines. I will maintain my score.

Thank you for replying to our rebuttal. We appreciate the time spent reviewing our manuscript. We will make sure the remaining concerns and limitations are reflected in the paper.

Reviewer_gzGcOpenReview ↗
Rating
2
Confidence
2(+1)
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper addresses drug dose prediction in a two-stage framework: a classifier first predicts whether a patient’s dose will be modified, and then, if so, a regressor predicts the magnitude of this change. The authors propose to build conformal prediction intervals for this second stage by interpolating, via a parameter ww, between constant-size sets (absolute residuals) and sets normalized by an estimate of the dispersion σ~(x)\tilde\sigma(x). The stated goal is to obtain more balanced coverage across patient subgroups of varying difficulty. The validation is empirical and relies on a few small clinical datasets, evaluated with the CovGap and FSC metrics (minimum coverage over groups).

Critical strengths

The applied motivation is real and relevant: quantifying uncertainty on dose adjustments is an important clinical problem, and the question of coverage fairness across patient subpopulations is legitimate in this context. The proposed method is also simple to implement. In my view, however, these points are not enough to offset the weaknesses below, as the methodological contribution and the validation are not at the expected level.

Critical weaknesses
  • The problem and the contribution are poorly defined, and the “two-stage” aspect is not addressed. The title and introduction emphasize a two-stage formulation (classification then regression), but the paper in practice only studies the regression stage. The classifier is not analyzed (Section 2.4 refers to another work), and the difficulties specific to coupling the two stages (propagation of classifier errors, conditioning of the stage-2 calibration on the stage-1 decision, overall coverage guarantee) are ignored. Section 4.5 even seems to indicate that stage 1 contributes little. The clinical setting thus serves more as a motivation than as an actual object of study, and the proposed method does not specifically address the stated problem.
  • Limited novelty and a very insufficient positioning in the literature, resulting in an unconvincing experimental validation.

The method is not new. Normalizing scores by an estimate of the dispersion is standard (Papadopoulos et al., 2011; Lei et al., 2018, cited line 178). The parameter ww in Equation (4) largely corresponds to Equation (29) of Papadopoulos et al. (2011). If σ~(x)\tilde\sigma(x) is well estimated and the noise is symmetric, w=1w = 1 allows approximating conditional coverage. w<1w < 1 then acts as a post hoc regularization of an overly dispersed or overfitted \tilde\sigma(x), rather than as a new strategy.

Above all, the literature review misses many approaches designed precisely for the problem at hand, namely conditional or group coverage:

  • The method of Gibbs et al. (2025) guarantees exact conditional coverage over a finite class of groups. It is cited but not compared against, even though the paper’s evaluation groups (tertiles of a score computed on X) fall exactly within this framework. It is the most natural baseline, and its absence is hard to justify.
  • Multivalid conformal prediction (Bastani et al., 2022; Jung et al., 2023) also addresses simultaneous coverage over groups.
  • Localized approaches or those based on the conditional density are also omitted: CD-split (Izbicki et al., 2020), localized CP (Guan, 2023), as well as the rectification of conformity scores (Plassier et al., 2025). More generally, Dheur et al. (2025) provide a broader review covering more methods.

None of these methods is used as a baseline. It is therefore impossible to know whether the proposed approach brings anything compared to the state of the art, in addition to being of limited novelty.

The experimental evaluation does not settle the question either. The CovGap and FSC metrics depend heavily on the choice of groups and require enough samples per group to be interpretable. However, the test sets in Table 1 contain only 239 and 61 points in total: the estimated per-group coverages are therefore very noisy, and the conclusions unreliable. Conditional coverage diagnostics better suited to small samples exist, for example L1-ERT (Braun et al., 2026). Moreover, more balanced coverage across difficulty groups guarantees nothing for other subgroups that may be more clinically important (sex, age, treatment duration), which makes this criterion rather arbitrary. The experimental scope (number of datasets, baselines, dispersion models) is very narrow.

Finally, the approach does not rely on any theoretical analysis. For a ww fixed in advance, the only guarantee is marginal coverage, which follows directly from split conformal prediction and holds for any score. Nothing is established about what the method is supposed to improve: no group or conditional coverage result, even approximate or asymptotic, no characterization of the situations in which w<1w < 1 would be preferable to w=1w = 1 (for example as a function of the estimation error of \tilde\sigma), and no analysis of the size of the resulting intervals. The approaches omitted from the literature, on the contrary, offer explicit guarantees: exact coverage over a finite class of groups, asymptotic conditional validity for localized or conditional-density-based methods. In the absence of such guarantees, the claimed benefits rest entirely on experiments whose scope and statistical reliability are, as noted above, limited.

References

  • Bastani, O., et al. (2022). Practical adversarial multivalid conformal prediction. Advances in Neural Information Processing Systems (NeurIPS)
  • Braun, et al. (2026). Conditional coverage diagnostics for conformal prediction. International Conference on Machine Learning (ICML)
  • Dheur, V., et al. (2025). A Unified Comparative Study with Generalized Conformity Scores for Multi-Output Conformal Regression. International Conference on Machine Learning (ICML)
  • Gibbs, I., et al. (2025). Conformal prediction with conditional guarantees. Journal of the Royal Statistical Society Series B: Statistical Methodology, 87(4), 1100-1126
  • Guan, L. (2023). Localized conformal prediction: A generalized inference framework for conformal prediction. Biometrika, 110(1), 33–50
  • Izbicki, R., et al. (2020). Flexible distribution-free conditional predictive bands using density estimators. International Conference on Artificial Intelligence and Statistics (AISTATS)
  • Jung, C., et al. (2023). Batch multivalid conformal prediction. International Conference on Learning Representations (ICLR)
  • Lei, J., et al. (2018). Distribution-free predictive inference for regression. Journal of the American Statistical Association, 113(523), 1094–1111
  • Papadopoulos, H., et al. (2011). Regression conformal prediction with nearest neighbours. Journal of Artificial Intelligence Research, 40, 815–840
  • Plassier, V., et al. (2025). Rectifying conformity scores for better conditional coverage. International Conference on Machine Learning (ICML)

ADDED POST REBUTTAL

Thank you for the rebuttal and the responses.
I still believe that the paper is not very clear, especially now that many new metrics and experiments have been added. This makes the contribution difficult to identify, and the paper overclaims the two-stage process, while the second stage is the main focus of the work.
I carefully read the new comparisons, but I now believe that the method does not significantly improve the empirical results over existing strategies such as CQR or rectifying the non-conformity scores, which reduces the contribution.
I therefore maintain my rating.

Questions for authors

Requests for clarification that could change my assessment:

  • Can you precisely formalize the two-stage conformal prediction problem? What coverage guarantee are you targeting (overall marginal, or conditional on the classifier’s decision)? How are stage-1 errors accounted for in the stage-2 calibration? Your evaluation groups are tertiles of a function of XX, i.e., a finite class of groups. Why not compare to the method of Gibbs et al. (2025), which guarantees exact coverage over such groups? What advantage does your approach offer over it, as well as over multivalid and localized methods?

  • The best strengths in Tables 1–2 and the 31.7% CovGap reduction in the abstract are selected retrospectively on the evaluation data. The selection procedure of Appendix C is not used for the main results, and its certifying rule never certifies an improvement. Could you report results for a ww selected without access to evaluation outcomes, and state what validity guarantee holds for the selected ww?

  • Do the gains observed for w<1w < 1 not simply reflect a poorly learned σ~(⋅)\tilde\sigma(\cdot)? Have you tried more expressive or better-tuned dispersion models? In that case, would w=1w = 1 remain suboptimal?

  • How many points does each group used for CovGap and FSC contain, and what are the confidence intervals associated with the estimated coverages, particularly on the 61-point dataset?

  • Could you evaluate coverage on clinically relevant subgroups (sex, age, etc.), and complement CovGap and FSC with diagnostics suited to small samples, such as L1-ERT (Braun et al., 2026).

Suggestions (without direct impact on my recommendation):

  • Restructure the paper to better identify the contributions. The problem addressed should be clearly stated, it should be specified whether the goal is to introduce a method or to highlight a phenomenon, and what is new should be distinguished from what builds on existing work. In particular, the “two-stage” framework should not be emphasized (title, introduction) when only the regression stage is studied. Either the paper genuinely addresses the coupling of the two stages, or it is refocused on conformal regression, and the title and introduction are adjusted accordingly.
  • Can you provide a theoretical result, even asymptotic or under assumptions on the quality of \tilde\sigma, justifying the improvement in group coverage for w<1w < 1?
3 comments

Thank you so much for the thorough review! Please find our responses below.

W1 (Problem definition and two-stage scope). Section 1 now frames the problem as: how much should a learned difficulty score change interval length in the magnitude stage, and how does that choice move coverage between patient groups? We characterize the strength that balances coverage (Proposition 1, W5) and estimate it from training data with R^α\hat R_\alpha (general response). Our response to Reviewer aHXd, W1 provides the full statement of the contribution.

On the two stages, Section 2.1 now states the scope. In the main analysis, Stage 2 is calibrated and evaluated on the encounters with a recorded dose change in our data. To study the coupling the reviewer describes, we added an automated mode (Appendix E) in which the Stage-1 classifier decides whether the dose changes and each predicted class is calibrated separately. Coverage then holds conditional on the classifier's decision. The classifier's false positives add a point mass at zero, and the mean interval length of every method rose to cover it (PPMI at 80%: 5.71 for CASCADE-CRC, 5.73 for CQR, 6.34 for Mondrian, against 1.60 in the main analysis). In this mode Mondrian and CQR balance better than CASCADE-CRC on PPMI, and we report that.

We agree that Stage-1 outputs add little as difficulty features (Section 4.6). Our response to Reviewer aHXd, Q1 lists where the two-stage structure does matter. Section 4.1 and Appendix Table 6 now show which patients gain and lose coverage as ww changes.

W2 (Novelty and positioning). We agree that CASCADE-CRC borrows from normalized conformal prediction. Section 2.3 shows that the scale σ~(x)+γ\tilde\sigma(x)+\gamma of Papadopoulos et al. (2011, Eq. 29) is proportional to σw(x)\sigma_w(x) with w=1/(1+γ)w=1/(1+\gamma). We also agree that w<1w<1 regularizes an over-dispersed σ~\tilde\sigma: since σ~\tilde\sigma has mean one on Dhold\mathcal D_{\mathrm{hold}}, σw=1+w(σ~−1)\sigma_w=1+w(\tilde\sigma-1) shrinks its deviations by the factor ww. What the revision adds is how much to shrink. Under Proposition 1 the relative variation of the conditional error quantile is qα(s)/qα(1)=1+wα†(s−1)q_\alpha(s)/q_\alpha(1)=1+w^\dagger_\alpha(s-1), so in relative terms it varies wα†w^\dagger_\alpha times as much as σ~\tilde\sigma (W5), and R^α\hat R_\alpha estimates this before calibration, so the regularization is not post hoc. Our response to Reviewer aHXd (W1) expands on novelty, and W3 below has the comparisons with additional baselines.

W3 (Baselines). Added in Section 2.5 with their description in Appendix D:

  • conditional conformal prediction (Gibbs et al., 2025) with three function classes: F1 (Mondrian on the difficulty tertiles), F2 = span{1, σ~\tilde\sigma}, and F3 (overlapping difficulty, age and sex groups)
  • batch multivalid CP (BatchGCP, Jung et al., 2023)
  • localized CP (Guan, 2023, Hore and Barber, 2025)
  • rectified scores (Plassier et al., 2025)

Bastani et al. is an online method and does not fit the batch split-conformal setting. CD-split can return unions of intervals, which do not give a single dose range, so we discuss it in Section 2.5 alongside Dheur et al.

CovGap at the 80% target, with mean interval length in parentheses (all targets in Appendix Table 11):

PPMIXYZ HealthfreMTPL2Synthetic*
constant length0.060 (1.60)0.099 (2.54)0.025 (2.52)0.118 (2.76)
Mondrian0.045 (1.61)0.106 (2.65)0.021 (2.52)0.069 (2.89)
CQR0.043 (1.55)0.100 (2.40)0.024 (2.62)0.044 (2.54)
conditional conformal F20.043 (1.62)0.100 (2.63)0.019 (2.50)0.062 (2.88)
conditional conformal F30.044 (1.63)0.100 (2.91)0.021 (2.53)n/a
localized CP0.052 (∞)0.098 (∞)0.021 (∞)0.094 (∞)
rectified scores (ratio)0.044 (1.65)0.102 (2.72)0.033 (2.63)0.035 (2.93)
BatchGCP0.042 (1.59)0.099 (2.52)0.021 (2.50)0.070 (2.79)
CASCADE-CRC, primary score0.046 (1.60)0.102 (2.59)0.021 (2.51)0.067 (2.81)
CASCADE-CRC, quantile score0.041 (1.59)0.098 (2.55)0.021 (2.51)0.052 (2.81)

*Synthetic groups use the true noise scale. F3 needs age and sex, which the synthetic data lack. Both CASCADE-CRC rows use w=R^αw=\hat R_\alpha.

Table 3 compares CASCADE-CRC's CovGap with each comparator's in 12 settings: 4 datasets × 3 targets (9 for F3). Groups are the predicted-difficulty tertiles, on the synthetic data too. In each setting, CASCADE-CRC counts as better if the paired 95% interval for the CovGap difference is below zero, tied if it contains zero, and worse if it is above zero.

CASCADE-CRC vsconstantMondrianCQRF2F3localized CPrectifiedBatchGCP
quantile score5/7/03/9/01/10/11/10/11/8/04/8/05/6/15/7/0
primary score5/7/04/8/01/10/13/8/12/7/04/8/05/7/05/7/0

No setting flags CASCADE-CRC as worse than constant length. It wins more often than it loses against Mondrian, F3, localized CP, rectified scores and BatchGCP, and ties CQR and F2 in most settings. A tie does not establish equivalence. Table 3 also reports each method's length relative to constant length, its share of infinite intervals and whether it is centered on the point prediction. CASCADE-CRC's intervals are centered and were finite in every setting. Section 4.3 discusses these results.

W4 (Evaluation, sample size and subgroups).

  • Small groups. We added a noise floor (the CovGap expected from group size alone) and per-group Clopper–Pearson intervals (Section 3, Appendix E). With 17–24 patients per group, XYZ Health cannot rank methods at 90% and 95% (Q4).
  • Small-sample diagnostics. L1-ERT (Appendix E, Q5).
  • Clinical subgroups. Age, sex, time since diagnosis (proxy for treatment duration) and previous dose (Section 4.4, Q5).
  • Scope. Four new baselines (W3), more dispersion models (Section 4.6, Appendix F), and simulations over noise distributions, score-training sizes and calibration sizes (Appendix F).

W5 (Theoretical analysis). We added Proposition 1 (Section 2.4, proof in Appendix C). Suppose the conditional (1−α)(1-\alpha) quantile of the error is affine in S=σ~(X)S=\tilde\sigma(X), qα(s)=θ0+θ1sq_\alpha(s)=\theta_0+\theta_1 s, and the error has a positive density there, so the quantile is unique. Then at wα†=θ1/(θ0+θ1)w^\dagger_\alpha=\theta_1/(\theta_0+\theta_1), coverage conditional on SS converges to 1−α1-\alpha. If in addition, the scaled error has a continuous distribution and is independent of SS, and the calibration multiplier is unique at each ww, coverage is monotone in SS at any other ww, and at wα†w^\dagger_\alpha coverage given SS equals ⌈(m+1)(1−α)⌉/(m+1)\lceil(m+1)(1-\alpha)\rceil/(m+1) at every score value for every calibration size mm, averaging over the calibration data. That finite-sample statement holds at the oracle strength. Under these assumptions, the method improves coverage along the score. The balancing strength is also below one exactly when θ0>0\theta_0>0. It takes the affine form as given, but our simulations show that a noisier score yields a lower balancing strength (Appendix F). For interval size, Eq. 8 gives relative lengths across patients at any ww, with Table 3 and Appendix Table 11 reporting the lengths.

Q1 (Formalization, stage-1 errors and Gibbs et al.).

  • Target guarantee (Sections 2.1, 2.3). In the main analysis the target population is P1=L(X,Y∣Z=1)P_1=\mathcal L(X,Y\mid Z=1), encounters with a recorded dose change, one per patient. For any ww fixed before calibration, including R^α\hat R_\alpha, we target marginal coverage, P(Y∗∈C^w(X∗))≥1−α\mathbb P(Y_*\in\widehat C_w(X_*))\ge1-\alpha under P1P_1.
  • Overall or conditional on the classifier's decision. In the automated mode each predicted class is calibrated separately, so coverage holds conditional on the classifier's decision and therefore also overall (W1).
  • Stage-1 errors. They enter the calibration only in the automated mode (W1).
  • Gibbs et al., multivalid and localized methods. Now compared (W3). CASCADE-CRC offers comparable or better balance with closed-form intervals that are centered on the point prediction and were finite in every setting, and it needs no group definitions at deployment.

Q2 (Strength chosen without evaluation outcomes). Yes. We now report w=R^αw=\hat R_\alpha, which uses only Dhold\mathcal D_{\mathrm{hold}}, so the marginal guarantee holds exactly (Section 2.4). On PPMI at 80%, CovGap falls from 0.060 to 0.046 with the paper's score and 0.041 with the quantile score, at unchanged mean length. On XYZ Health, R^α≈0\hat R_\alpha\approx0 recovers constant length. Section 4.3, Table 3 and Appendix Table 11 contain the full results, and Appendix G the original selection rules.

Q3 (Better-tuned dispersion models). Largely yes, which is expected in the magnitude stage, where the score is learned from few encounters. In simulations a noisier σ~\tilde\sigma gives a larger intercept θ0\theta_0 and a lower balancing strength. We tried more expressive and better-tuned models (regularized and bagged boosting, a log-linear ridge model and the quantile scores of Section 2.6). The balancing strength rises but stays below one on our data, so w=1w=1 remains slightly suboptimal. In simulations it approaches one only when the dispersion model is correctly specified (Appendix F). R^α\hat R_\alpha adapts to whatever score is used.

Q4 (Group sizes and confidence intervals). Each group holds about 65–88 units on PPMI, 17–24 on XYZ Health and 379 on freMTPL2 per split, and every result averages 30 splits (Appendix Table 6). On XYZ Health a group's 95% Clopper–Pearson interval is about ±0.15 wide and contains the target most of the time. PPMI is more informative: its intervals are about ±0.10 wide, and the constant-length CovGap (0.060) is well above its floor (0.036). A noise-adjusted root-mean-square gap falls from 0.062 with constant length to 0.033 with R^α\hat R_\alpha. Because the 30 splits reuse the same patients, the split-to-split intervals are too narrow. After the Nadeau–Bengio correction for this overlap, every clinical CovGap interval includes zero (Appendix E).

Q5 (Clinical subgroups and L1-ERT). We added age (both cohorts), sex and time since diagnosis (XYZ Health), previous dose (PPMI) and L1-ERT (Section 4.4, Appendix E).

Su1 (Structure and contributions). See W1. The introduction now says that the paper both characterizes a phenomenon and gives a way to act on it, and it separates prior work (Section 2.3) from what is new (Sections 2.4, 4.3).

Su2 (Theoretical result). Please see W5.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their assessment of our manuscript. Please find below how we have addressed the reviews and updated the manuscript.

Our approach. CASCADE-CRC builds prediction intervals for the magnitude stage of a two-stage model (Stage 1 predicts whether the dose changes, Stage 2 by how much). The interval is C^w(x)=[f^(x)−λ^wσw(x), f^(x)+λ^wσw(x)]\widehat C_w(x)=[\hat f(x)-\hat\lambda_w\sigma_w(x),\ \hat f(x)+\hat\lambda_w\sigma_w(x)] with σw(x)=(1−w)+wσ~(x)\sigma_w(x)=(1-w)+w\tilde\sigma(x), where σ~\tilde\sigma is the normalized difficulty score and ww is the adaptation strength (w=0w=0: constant length, w=1w=1: full scaling by σ~\tilde\sigma).

Main changes

  1. Choosing the strength (Sections 2.4, 4.3). We estimate the balancing strength from training data alone. On Dhold\mathcal D_{\mathrm{hold}} we fit a (1−α)(1-\alpha) linear quantile regression of ∣Y−f^(X)∣|Y-\hat f(X)| on out-of-fold scores, θ^0+θ^1σ~(X)\hat\theta_0+\hat\theta_1\tilde\sigma(X), and set R^α=θ^1/(θ^0+θ^1)\hat R_\alpha=\hat\theta_1/(\hat\theta_0+\hat\theta_1). The coverage guarantee is unchanged. On PPMI at 80%, CovGap drops from 0.060 with constant length to 0.046, and to 0.041 with a score that predicts the (1−α)(1-\alpha) error quantile (the quantile score), at the same mean length.
  2. Theory and mechanism (Section 2.4, Appendices C and F). Proposition 1 characterizes the strength that balances coverage across the difficulty score, wα†w^\dagger_\alpha, and when it falls below one. Appendix C proves it and the consistency of R^α\hat R_\alpha, and Appendix F tests the mechanism in simulations and on real data.
  3. Comparisons and uncertainty (Sections 2.5, 3, 4.4, Table 3, Appendix E). We added conditional conformal prediction (Gibbs et al., 2025), localized CP (Guan, 2023, Hore and Barber, 2025), rectified scores (Plassier et al., 2025) and batch multivalid CP (Jung et al., 2023), compared in paired tests. We also report noise floors, per-group confidence intervals, dependence-corrected comparisons and clinical subgroups.

We also added an automated mode in which Stage 1 decides whether the dose changes (Appendix E), a practical guide (Section 5) for practitioners, the full results of the original selection rules (now Appendix G) and larger fonts for figures 1 and 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.