- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper examines how Huber contamination in both training and calibration sets impacts the interval length of split conformal prediction (SCP). The authors derive finite-sample upper bounds on the interval-length deviation relative to a clean population oracle, and establish a local minimax lower bound on a restricted location submodel. They also exactly characterize the worst-case effects of calibration contamination. The finite-sample upper-bound analysis is further extended to CQR and adaptive residual scores, while experiments illustrate the effects of contamination.
Critical strengths
- Solid theoretical foundation. The decomposition of interval-length deviation into training and calibration errors is clean and intuitive. The finite-sample upper bounds, paired with the local minimax lower bound (and matching attainable rates under certain conditions), form a rigorous analytical framework.
- Good analysis of calibration contamination. I liked Proposition 4.1. It doesn't just bound the error; it also characterizes the worst-case population-level interval inflation and contraction, identifies the breakdown threshold under an unbounded-score-support assumption, and constructs contaminating distributions that attain the extrema below this threshold.
- Comprehensive empirical validation. The experiments are well-designed, covering various contamination mechanisms and three different nonconformity scores across synthetic and real datasets. The ablation comparing Bernoulli and fixed-fraction replacement is a nice touch that clarifies finite-sample effects.
Critical weaknesses
- Mismatch between theoretical assumptions and empirical models. Theorem 4.1 relies heavily on uniform replacement stability. While the authors prove this for regularized linear quantile regression (Example 4.1), the real-world experiments use neural networks. Since no corresponding stability guarantees are established for the neural networks used in the experiments, it is hard to tell how well the theoretical bounds actually translate to these settings.
- Limited exploration of joint contamination in experiments. The theory nicely accommodates arbitrary and distinct contamination distributions ( and ), but the joint-contamination results in Figures 2, 4b, and 6 are based on the same fixed point-mass distribution for both stages. Furthermore, the real-data experiments only look at calibration contamination. Testing with distinct or more structured contaminations across both stages would better demonstrate the generality of the results.
- Lack of direct quantitative evaluation of the bounds. The experiments illustrate the qualitative trends predicted by the theory, but do not directly assess the tightness of the finite-sample bounds. For example, the slope in Figure 3a captures empirical sensitivity rather than the actual uniform stability constant , and the reference curve in Figure 4a drops the sampling terms from Theorem 4.1. The paper would be more convincing if it included a direct numerical assessment of how tight these bounds are in practice.
- Narrow scope of the minimax lower bound. Theorem 4.2 is proved on a very specific intercept-only location submodel with asymmetric noise and constrained estimators. In particular, the local linear identifiability condition excludes smooth noise distributions symmetric about zero, for which the first-order quantile sensitivity vanishes. I am left wondering if the claimed rate sharpness holds up in more general regression settings.
Questions for authors
- While the experiments include several real-world datasets, the contamination is still artificially introduced through settings such as mean shifts and variance inflation. Could the authors discuss how their theoretical findings might be useful in practice, where the contamination levels and the underlying clean distribution are typically unknown? Also, would it be possible to evaluate the findings on datasets with naturally occurring label noise or outliers to better understand their practical relevance?
See Weaknesses for other concerns.
4 comments
Response (Part 2)
Weakness 4: Narrow scope of the minimax lower bound
We thank the reviewer for this keen insight. We agree that, when restricted to symmetric noise distributions, the training-side lower bound may be smaller, as the first-order sensitivity of the residual quantile vanishes.
However, we would like to clarify that our theoretical framework accommodates a **broader** class of distributions, including both symmetric and asymmetric noise. Minimax lower bounds are typically established by constructing sufficiently difficult instances. Our asymmetric location submodel provides such an instance, yielding a lower bound that locally matches an attainable upper bound up to constant factors under the stated conditions.
If we restrict attention to the narrower class of symmetric noise distributions, the training-side lower bound may improve, and the upper bound could potentially also be sharpened by exploiting symmetry. This does not contradict the minimax lower bound established using the asymmetric submodel. We have clarified this distinction in Remark 4.2.
Q1: "While the experiments include ... their practical relevance?"
We thank the reviewer for this important question.
Practical relevance. The length-sensitivity bounds apply uniformly over contaminating distributions under the stated assumptions. When the calibration contamination level is unknown, a known upper bound can be used for conservative coverage correction (Section 4.3.2). Obtaining reliable upper bounds on contamination levels in practice remains future work.
Naturally occurring contamination. We added a DESI redshift experiment (Appendix F.6). Among 2,308 galaxies with expert-reviewed redshifts from visual inspection, 71 (3.08%) exhibit catastrophic errors in the automated Redrock estimates (Figure 35A). Across 40 splits, calibrating with Redrock labels increases the mean interval length from 0.266 to 0.285 while holding the predictor fixed (Figure 35B). Controlled Huber mixtures built from the observed catastrophic errors show increasing interval-length deviations as the contamination level rises (Figure 35C).
These results strengthen the practical relevance of our theoretical findings beyond artificially introduced contamination.
We thank the reviewer again for the thoughtful and constructive feedback. We hope that these theoretical clarifications and additional experiments have adequately addressed the reviewer's concerns.
Thank you for the detailed response and substantial additional experiments, which have addressed most of my concerns. I appreciate the authors' efforts in strengthening the paper and I'm happy to maintain my positive rating.
Thank you for taking the time to review our response. We appreciate your feedback and continued support.
We sincerely thank the reviewer for the positive assessment of our work and for recognizing the contributions of our theoretical analysis and empirical evaluation. We also appreciate the constructive suggestions, which have helped us further strengthen the connection between theory and experiments. We address each weakness below.
Weakness 1: "Mismatch between theoretical assumptions and empirical models"
We thank the reviewer for this important comment. First, we would like to clarify that our original real-world experiments considered calibration contamination only, and therefore did not directly involve the uniform replacement-stability assumption. To strengthen the connection between theory and practice, we have made two improvements.
First, we have extended Example 4.1 to establish uniform replacement stability for a class of nonlinear neural networks. Under bounded parameter derivatives, convex Lipschitz losses, and sufficiently strong regularization, Definition 4.1 holds with an explicit stability constant L=λ−cH2cB2. Appendix F.1.1 describes the tanh network and gives its replacement-stability constants for exact regularized fits.
Second, we have added experiments on four real datasets with mean-shift contamination in training and heavy-tailed contamination in calibration (Figure 19). The observed interval-length deviations are consistent with the distinct effects of training and calibration contamination predicted by our analysis.
These additions strengthen the empirical connection to our stability-based theoretical framework.
Weakness 2: Limited exploration of joint contamination in experiments
We thank the reviewer for this suggestion. In the initial submission, we prioritized the challenging point-mass contamination setting for the joint-contamination experiments in Figures 2, 4b, and 6 due to space limitations. Nevertheless, we also considered four contamination families—mean shift, variance inflation, heavy-tailed noise, and covariate shift with heteroscedastic noise—in Figures 8, 9, 12, 13, 14, and 16, although these experiments focused on calibration contamination.
We expanded the joint-contamination experiments to consider distinct contamination distributions at the two stages. Figure 18 examines different training and calibration point masses in a Gaussian simulation, and Figure 19 combines mean-shift training contamination with heavy-tailed calibration contamination on four real datasets. Appendix F.3.1 compares the same contamination types (Figures 20 and 21), and Appendix F.3.2 examines different contamination types across the stages (Figures 22 through 24).
These additional experiments demonstrate the effects of different contamination types at the two stages and provide broader empirical support for our theoretical framework, which allows arbitrary and distinct Q1 and Q2.
Weakness 3: Lack of direct quantitative evaluation of the bounds
We thank the reviewer for this valuable suggestion. We agree that a direct quantitative evaluation of the theoretical bounds would strengthen our empirical analysis.
First, we clarify that the reference curve in Figure 4a intentionally includes only the population calibration-contamination term to isolate its dependence on δ2. It is not intended to represent the complete finite-sample upper bound.
To address the reviewer's concern, we have made the following additions.
1. Explicit stability constants. We extended Example 4.1 to cover regularized smooth neural networks and derived replacement-stability constants for the exact regularized fits of the tanh network used in our additional experiments (Appendix F.1.1).
2. Direct numerical evaluation of the complete upper bound. Appendix F.2 (Figure 17) compares the complete four-term upper bound in Theorem 4.1 with empirical interval-length deviations in a Gaussian simulation with the smooth tanh network. At (δ1,δ2)=(0.05,0.04), the empirical 95th percentile over 20 runs is 0.676 relative to the numerical clean population reference. The theoretical bound for exact fits and the exact clean population oracle is 8.216, illustrating its conservativeness in this setting.