acceptodds
Under review as a conference paper at ICLR 2027

Split Conformal Interval Length under Huber Contamination

Abstract

Existing analyses of interval length in split conformal prediction largely focus on clean data. However, how contamination at the training and calibration stages alters interval length relative to the corresponding clean population oracle remains understudied. In this paper, we consider Huber contamination and derive a finite-sample upper bound that holds uniformly over contaminating distributions, with leading terms scaling as , in addition to a clean-sample estimation term. We further establish a local minimax lower bound that matches an attainable fixed-confidence upper rate, up to constant factors, on a regular asymmetric location submodel. For calibration contamination, we obtain an exact worst-case characterization of the population interval-length deviation. We also extend the upper-bound results to conformalized quantile regression and adaptive residual scores. Experiments on synthetic and real-world datasets illustrate the distinct effects of training and calibration contamination and are consistent with our theoretical analysis.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Conformal Prediction, Robustness, Uncertainty Quantification

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_cZarOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper examines how Huber contamination in both training and calibration sets impacts the interval length of split conformal prediction (SCP). The authors derive finite-sample upper bounds on the interval-length deviation relative to a clean population oracle, and establish a local minimax lower bound on a restricted location submodel. They also exactly characterize the worst-case effects of calibration contamination. The finite-sample upper-bound analysis is further extended to CQR and adaptive residual scores, while experiments illustrate the effects of contamination.

Critical strengths
  1. Solid theoretical foundation. The decomposition of interval-length deviation into training and calibration errors is clean and intuitive. The finite-sample upper bounds, paired with the local minimax lower bound (and matching attainable rates under certain conditions), form a rigorous analytical framework.
  2. Good analysis of calibration contamination. I liked Proposition 4.1. It doesn't just bound the error; it also characterizes the worst-case population-level interval inflation and contraction, identifies the breakdown threshold under an unbounded-score-support assumption, and constructs contaminating distributions that attain the extrema below this threshold.
  3. Comprehensive empirical validation. The experiments are well-designed, covering various contamination mechanisms and three different nonconformity scores across synthetic and real datasets. The ablation comparing Bernoulli and fixed-fraction replacement is a nice touch that clarifies finite-sample effects.
Critical weaknesses
  1. Mismatch between theoretical assumptions and empirical models. Theorem 4.1 relies heavily on uniform replacement stability. While the authors prove this for regularized linear quantile regression (Example 4.1), the real-world experiments use neural networks. Since no corresponding stability guarantees are established for the neural networks used in the experiments, it is hard to tell how well the theoretical bounds actually translate to these settings.
  2. Limited exploration of joint contamination in experiments. The theory nicely accommodates arbitrary and distinct contamination distributions (Q1Q_1 and Q2Q_2), but the joint-contamination results in Figures 2, 4b, and 6 are based on the same fixed point-mass distribution for both stages. Furthermore, the real-data experiments only look at calibration contamination. Testing with distinct or more structured contaminations across both stages would better demonstrate the generality of the results.
  3. Lack of direct quantitative evaluation of the bounds. The experiments illustrate the qualitative trends predicted by the theory, but do not directly assess the tightness of the finite-sample bounds. For example, the slope in Figure 3a captures empirical sensitivity rather than the actual uniform stability constant LL, and the reference curve in Figure 4a drops the sampling terms from Theorem 4.1. The paper would be more convincing if it included a direct numerical assessment of how tight these bounds are in practice.
  4. Narrow scope of the minimax lower bound. Theorem 4.2 is proved on a very specific intercept-only location submodel with asymmetric noise and constrained estimators. In particular, the local linear identifiability condition excludes smooth noise distributions symmetric about zero, for which the first-order quantile sensitivity vanishes. I am left wondering if the claimed rate sharpness holds up in more general regression settings.
Questions for authors
  1. While the experiments include several real-world datasets, the contamination is still artificially introduced through settings such as mean shifts and variance inflation. Could the authors discuss how their theoretical findings might be useful in practice, where the contamination levels and the underlying clean distribution are typically unknown? Also, would it be possible to evaluate the findings on datasets with naturally occurring label noise or outliers to better understand their practical relevance?

See Weaknesses for other concerns.

4 comments

We sincerely thank the reviewer for the positive assessment of our work and for recognizing the contributions of our theoretical analysis and empirical evaluation. We also appreciate the constructive suggestions, which have helped us further strengthen the connection between theory and experiments. We address each weakness below.

Weakness 1: "Mismatch between theoretical assumptions and empirical models"

We thank the reviewer for this important comment. First, we would like to clarify that our original real-world experiments considered calibration contamination only, and therefore did not directly involve the uniform replacement-stability assumption. To strengthen the connection between theory and practice, we have made two improvements.

First, we have extended Example 4.1 to establish uniform replacement stability for a class of nonlinear neural networks. Under bounded parameter derivatives, convex Lipschitz losses, and sufficiently strong regularization, Definition 4.1 holds with an explicit stability constant L=2cB2λ−cHL=\frac{2cB^2}{\lambda-cH}. Appendix F.1.1 describes the tanh network and gives its replacement-stability constants for exact regularized fits.

Second, we have added experiments on four real datasets with mean-shift contamination in training and heavy-tailed contamination in calibration (Figure 19). The observed interval-length deviations are consistent with the distinct effects of training and calibration contamination predicted by our analysis.

These additions strengthen the empirical connection to our stability-based theoretical framework.

Weakness 2: Limited exploration of joint contamination in experiments

We thank the reviewer for this suggestion. In the initial submission, we prioritized the challenging point-mass contamination setting for the joint-contamination experiments in Figures 2, 4b, and 6 due to space limitations. Nevertheless, we also considered four contamination families—mean shift, variance inflation, heavy-tailed noise, and covariate shift with heteroscedastic noise—in Figures 8, 9, 12, 13, 14, and 16, although these experiments focused on calibration contamination.

We expanded the joint-contamination experiments to consider distinct contamination distributions at the two stages. Figure 18 examines different training and calibration point masses in a Gaussian simulation, and Figure 19 combines mean-shift training contamination with heavy-tailed calibration contamination on four real datasets. Appendix F.3.1 compares the same contamination types (Figures 20 and 21), and Appendix F.3.2 examines different contamination types across the stages (Figures 22 through 24).

These additional experiments demonstrate the effects of different contamination types at the two stages and provide broader empirical support for our theoretical framework, which allows arbitrary and distinct Q1Q_1 and Q2Q_2.

Weakness 3: Lack of direct quantitative evaluation of the bounds

We thank the reviewer for this valuable suggestion. We agree that a direct quantitative evaluation of the theoretical bounds would strengthen our empirical analysis.

First, we clarify that the reference curve in Figure 4a intentionally includes only the population calibration-contamination term to isolate its dependence on δ2\delta_2. It is not intended to represent the complete finite-sample upper bound.

To address the reviewer's concern, we have made the following additions.

1. Explicit stability constants. We extended Example 4.1 to cover regularized smooth neural networks and derived replacement-stability constants for the exact regularized fits of the tanh network used in our additional experiments (Appendix F.1.1).

2. Direct numerical evaluation of the complete upper bound. Appendix F.2 (Figure 17) compares the complete four-term upper bound in Theorem 4.1 with empirical interval-length deviations in a Gaussian simulation with the smooth tanh network. At (δ1,δ2)=(0.05,0.04)(\delta_1,\delta_2)=(0.05,0.04), the empirical 95th percentile over 20 runs is 0.6760.676 relative to the numerical clean population reference. The theoretical bound for exact fits and the exact clean population oracle is 8.2168.216, illustrating its conservativeness in this setting.

AuthorsOpenReview ↗

Response (Part 2)

Weakness 4: Narrow scope of the minimax lower bound

We thank the reviewer for this keen insight. We agree that, when restricted to symmetric noise distributions, the training-side lower bound may be smaller, as the first-order sensitivity of the residual quantile vanishes.

However, we would like to clarify that our theoretical framework accommodates a **broader** class of distributions, including both symmetric and asymmetric noise. Minimax lower bounds are typically established by constructing sufficiently difficult instances. Our asymmetric location submodel provides such an instance, yielding a lower bound that locally matches an attainable upper bound up to constant factors under the stated conditions.

If we restrict attention to the narrower class of symmetric noise distributions, the training-side lower bound may improve, and the upper bound could potentially also be sharpened by exploiting symmetry. This does not contradict the minimax lower bound established using the asymmetric submodel. We have clarified this distinction in Remark 4.2.

Q1: "While the experiments include ... their practical relevance?"

We thank the reviewer for this important question.

Practical relevance. The length-sensitivity bounds apply uniformly over contaminating distributions under the stated assumptions. When the calibration contamination level is unknown, a known upper bound δˉ2∈[δ2,1)\bar{\delta}_2\in[\delta_2,1) can be used for conservative coverage correction (Section 4.3.2). Obtaining reliable upper bounds on contamination levels in practice remains future work.

Naturally occurring contamination. We added a DESI redshift experiment (Appendix F.6). Among 2,308 galaxies with expert-reviewed redshifts from visual inspection, 71 (3.08%) exhibit catastrophic errors in the automated Redrock estimates (Figure 35A). Across 40 splits, calibrating with Redrock labels increases the mean interval length from 0.266 to 0.285 while holding the predictor fixed (Figure 35B). Controlled Huber mixtures built from the observed catastrophic errors show increasing interval-length deviations as the contamination level rises (Figure 35C).

These results strengthen the practical relevance of our theoretical findings beyond artificially introduced contamination.


We thank the reviewer again for the thoughtful and constructive feedback. We hope that these theoretical clarifications and additional experiments have adequately addressed the reviewer's concerns.

Thank you for the detailed response and substantial additional experiments, which have addressed most of my concerns. I appreciate the authors' efforts in strengthening the paper and I'm happy to maintain my positive rating.

Thank you for taking the time to review our response. We appreciate your feedback and continued support.

Reviewer_xQVXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies split conformal prediction interval-length robustness under Huber contamination in training and calibration data. It establishes finite-sample upper bounds, a matching local minimax lower bound on an asymmetric location submodel, and an exact worst-case characterization of calibration contamination, including its breakdown threshold. Next, extensions to CQR and adaptive residual scores are also provided. Finally, synthetic and real-data experiments illustrate the theoretical findings.

Critical strengths
  1. The paper provides a systematic theoretical analysis of interval-length robustness under joint contamination. The decomposition clearly distinguishes the effects of training and calibration contamination.
  2. The theoretical results are rigorous and relatively comprehensive. In particular, the exact worst-case calibration characterization and local minimax lower bound strengthen the analysis.
Critical weaknesses
  1. The minimax optimality result has limited generality. The matching lower bound relies on a specific asymmetric location submodel and does not establish optimality for broader regression settings. The authors have clarified the validity and scope of their hard-instance construction. However, its first-order training term relies on a reference predictor that is not locally length-optimal, limiting its interpretation as an intrinsic interval-efficiency barrier. (Partially solved)

  2. The experiments provide limited quantitative validation of the theoretical guarantees. The original concerns regarding bound tightness and the applicability of stability assumptions to practical learning algorithms have been substantially addressed through additional theoretical analysis and experiments. (Solved)

  3. The Huber-specific theoretical novelty remains insufficiently established. The main upper-bound rates appear extendable to broader i.i.d. TV contamination using maximal coupling, DKW, and quantile perturbation. Even the finite-sample breakdown threshold appears to extend beyond Huber mixtures. Although the exact asymmetric quantile envelopes are valuable, it remains unclear which central statistical conclusions fundamentally depend on the Huber structure, rather than representing sharp applications of broadly applicable statistical techniques. (Unresolved)

Questions for authors

To what extent can the minimax optimality result be generalized beyond the asymmetric location submodel? In particular, would the training-contamination rate remain tight under symmetric noise distributions, where interval length may exhibit only second-order sensitivity to prediction errors?

7 comments

We sincerely thank the reviewer for the thoughtful assessment of our work and for recognizing the contributions of our theoretical analysis. We appreciate the constructive suggestions regarding the generality of our minimax result and the empirical validation of our theoretical guarantees. We have revised the manuscript and added further experiments to address these concerns.

Weakness 1: "The minimax optimality result has limited generality."

We thank the reviewer for raising this point. We would like to clarify the role of the asymmetric location submodel in our minimax analysis.

Minimax lower bounds are typically established by constructing sufficiently difficult instances that incur large estimation errors under the considered criterion. In our setting, the relevant error is the interval-length deviation from the clean population oracle. Our asymmetric location submodel serves as such a hard instance, for which the lower bound locally matches an attainable upper bound up to constant factors.

Importantly, our general theoretical framework accommodates both symmetric and asymmetric noise distributions. The asymmetric location submodel is used to establish the lower bound, rather than being a restriction imposed on the general upper-bound analysis. The presence of easier instances, such as symmetric noise distributions, does not contradict the lower-bound construction.

Our upper-bound analysis applies to general regression learners satisfying the stated stability and regularity conditions. For the minimax analysis, we construct a regular asymmetric location submodel that serves as a hard instance, yielding a lower bound that locally matches an attainable upper bound up to constant factors under the stated conditions. We have clarified the scope of this matching result in Section 4.2 and Appendices B.8–B.9, and added Remark 4.2 to explain the distinction between symmetric and asymmetric noise.

When additional structural information, such as noise symmetry, is available, it may be possible to exploit this property to obtain sharper upper bounds. Our analysis instead considers the broader setting without imposing such restrictions. Extending the matching minimax analysis to more general regression models remains an interesting direction for future work.

Weakness 2: "The experiments provide limited quantitative validation of the theoretical guarantees."

We thank the reviewer for this suggestion. We have strengthened both the theoretical justification of the stability condition and its empirical validation.

Applicability of uniform replacement stability.

We first clarify that uniform replacement stability, defined in Definition 4.1, is a sufficient condition for our upper-bound guarantee, rather than a property automatically satisfied by arbitrary learning algorithms. For example, ordinary least-squares regression with unbounded responses need not satisfy this condition, since replacing a single training label with an arbitrarily large value can cause an arbitrarily large change in the fitted predictor.

To demonstrate the applicability of our theory, we have extended Example 4.1 beyond regularized linear quantile regression to a broader class of regularized predictors, including smooth neural networks under explicit conditions. In particular, we establish the uniform replacement-stability constant L=2cB2λ−cHL=\frac{2cB^2}{\lambda-cH} where BB bounds the predictor's parameter gradient, HH bounds its parameter Hessian, cc is the Lipschitz constant of the loss, and λ>cH\lambda>cH is the regularization parameter. Appendix F.1.1 describes the tanh network and gives its replacement-stability constants for exact regularized fits.

Figure 18 uses this network to examine distinct training and calibration point masses in a Gaussian simulation. Figure 19 uses mean-shift contamination in training and heavy-tailed contamination in calibration on four real datasets. These results illustrate the distinct contributions of training and calibration contamination to interval-length deviations.

Quantitative evaluation of the upper bounds.

Appendix F.2 (Figure 17) evaluates the complete four-term finite-sample upper bound in Theorem 4.1 for the absolute-residual score in a Gaussian simulation with the smooth tanh network. At (δ1,δ2)=(0.05,0.04)(\delta_1,\delta_2)=(0.05,0.04), the empirical 95th percentile of the interval-length deviation from the numerical clean population reference is 0.6760.676 over 20 runs. The theoretical upper bound for exact fits and the exact clean population oracle is 8.2168.216, about 12 times the empirical percentile.

AuthorsOpenReview ↗

Response (Part 2)

Regarding the Question: "To what extent can the minimax optimality result ...?"

We thank the reviewer for this question. As discussed in our response to Weakness 1, our asymmetric location submodel provides a hard instance for establishing the minimax lower bound within the stated setting. Under symmetric noise, the first-order sensitivity vanishes, and the training-side lower bound can be smaller. We have clarified this distinction in Remark 4.2.


We thank the reviewer again for the constructive comments. We hope these clarifications and additional results adequately address the reviewer's concerns.

I apologize for my delayed follow-up, partly because I have been taking additional time to understand the novelty of several theoretical components, particularly whether the reliance on the Huber mixture structure is essential to deriving many of the upper bounds.

I sincerely thank the authors for their detailed responses and substantial additional experiments. I appreciate the improved empirical validation, particularly the explicit stability analysis and direct numerical evaluation of the theoretical bounds.

However, since this is primarily a theoretical paper, my central concern remains the significance of its theoretical contributions relative to existing literature.

First, the main upper bound appears to follow from combining standard algorithmic stability (Bousquet & Elisseeff, 2002), DKW concentration, and quantile perturbation arguments. Similar rates seem potentially obtainable under more general TV/CDF perturbations or bounded adversarial replacements. Moreover, calibration contamination and conformal efficiency have already been studied by Clarkson et al. (2024), while finite-sample oracle-length deviations under clean data have been analyzed by Yao et al. (2026). Likewise, the breakdown threshold at δ2=α\delta_2=\alpha largely reflects classical upper-quantile breakdown behavior. Which theoretical insights fundamentally require the Huber mixture structure, and what is genuinely new beyond these existing results?

Second, while the asymmetric hard instance establishes a valid lower bound, the underlying Huber-neighborhood indistinguishability argument is closely related to classical robust minimax theory (Chen et al., 2016). The matching result remains restricted to a specific location submodel and constrained estimator class. Furthermore, first-order interval-length sensitivity can vanish even under asymmetric noise when the population predictor is locally length-optimal. The joint lower bound is also obtained by combining separate training- and calibration-side hard instances, rather than identifying a genuinely joint contamination mechanism. These observations raise questions about the generality and distinctiveness of the claimed minimax rate.

I encourage the authors to clarify these distinctions and explicitly discuss the limitations of the theoretical novelty and generality in the main manuscript. The additional experiments are valuable, but the central question for a theory-focused paper is what fundamentally new statistical understanding the results provide.

References:

[1] Bousquet, O., & Elisseeff, A. (2002). Stability and Generalization. JMLR.

[2] Clarkson, J., Xu, W., Cucuringu, M., Swan, Y., & Reinert, G. (2024). Split Conformal Prediction under Data Contamination. PMLR 230.

[3] Yao, Y., He, L., & Gastpar, M. (2026). Non-Asymptotic Analysis of Efficiency in Conformalized Regression. ICLR 2026.

[4] Chen, M., Gao, C., & Ren, Z. (2016). A General Decision Theory for Huber's ε-Contamination Model. Electronic Journal of Statistics.

Reviewer_WRR7OpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies the split conformal interval length under Huber contamination at the training and calibration stages. It gives an upper bound under replacement stability and local score-density conditions, a local minimax lower bound and a matching attainable upper rate, up to constant factors, on an asymmetric location submodel, and an exact population calibration breakdown result. It extends the upper analysis to CQR and adaptive scores and studies the two contamination stages on synthetic and real regression data.

Critical strengths
  • A clear separation of the two contamination stages: The four-part decomposition separates training sensitivity, clean training error, calibration sampling error, and calibration bias. The resulting high-probability upper bound is uniform over contaminating distributions under the stated stability and local density conditions.
  • An exact population calibration analysis: This work characterizes worst-case length inflation, contraction, and breakdown under its stated support condition. These results give a clear account of how calibration contamination affects interval length and complement the finite-sample upper bound.
Critical weaknesses

The following points concern clarity, practical scope, and further validation. They do not overturn the stated theoretical results or my recommendation to accept.

  1. The theoretical scope and finite-sample conditions could be easier to interpret. The first-order training lower term relies on asymmetry. For a smooth noise density, qg′(0)=[g(−q⋆)−g(q⋆)]/[g(q⋆)+g(−q⋆)]q_g'(0)=[g(-q_\star)-g(q_\star)]/[g(q_\star)+g(-q_\star)], which is zero for symmetric noise. A short explanation in Section 4.2 and a symmetric-location comparison would clarify why the local lower bound need not describe every noise model. Lemma 4.3 also uses a local density neighborhood, but “sufficiently large” leaves the required calibration sample size unclear. Please give a sufficient condition using the neighborhood margin and explain the deterioration as δ2\delta_2 approaches α\alpha; uniformity over contaminating distributions at a fixed contamination level does not remove this issue. A compact table of assumptions and a numerical upper-bound diagnostic would help. The general Huber decision-theory context [1] could also help place the local lower bound.
  2. The experiments give limited evidence for the full theoretical setting. The explicit stability example uses regularized linear quantile regression, while the real-data experiments use neural networks and keep training clean. A real-data training-contamination experiment with a learner whose assumptions can be checked would strengthen the link to theory. Synthetic length errors use a population oracle, whereas real-data errors use a paired clean split-conformal reference; the captions should make this difference clear. Reusing a fixed test set across the 20 repetitions also limits the uncertainty assessment, especially when interpreting mean coverage below 0.9. Independently redrawn test splits, or a separation of the sources of variation, would help. Empirical reference curves should remain distinct from uniform bounds, and any correction using an estimated contamination level should account for its uncertainty. A brief statement on keeping score selection separate from final calibration would also clarify the role of tuning bias [2].
Questions for authors
  1. Can you add a symmetric-noise comparison and state an explicit calibration sample condition for the local density argument, including how it changes near δ2=α\delta_2=\alpha?
  2. Can you strengthen the empirical link with a stable learner under real-data training contamination, clarify the reference targets in the captions, and assess uncertainty with independently redrawn test splits?

References

[1] Chen, M., Gao, C., & Ren, Z. (2016). A general decision theory for Huber's epsilon-contamination model. Electronic Journal of Statistics, 10(2), 3752–3774. https://doi.org/10.1214/16-EJS1216

[2] Zeng, H., Liu, K., Jing, B., & Wei, H. (2025). Parametric Scaling Law of Tuning Bias in Conformal Prediction. Proceedings of the 42nd International Conference on Machine Learning, PMLR 267, 74133–74156. https://proceedings.mlr.press/v267/zeng25e.html

4 comments

We sincerely thank the reviewer for the positive assessment of our work and for recognizing the contributions of our theoretical analysis. We also appreciate the constructive suggestions, which have helped us further strengthen the theoretical presentation and empirical validation. We address each point below.

Weakness 1: "The theoretical scope and finite-sample conditions could be easier to interpret."

The first-order training lower term relies ... clarify why the local lower bound need not describe every noise model.

We thank the reviewer for the suggestion. We have added Remark 4.2 to the revised manuscript to clarify the distinction between symmetric and asymmetric noise. Our theoretical framework accommodates both cases. We emphasize that minimax lower bounds are typically established by constructing sufficiently difficult instances. We therefore consider an asymmetric noise distribution, which yields a lower bound that locally matches an attainable upper bound up to constant factors. Under symmetric noise, the first-order sensitivity of the residual quantile vanishes, so the asymmetric lower-bound construction does not directly apply. A smaller training-side minimax rate may be attainable under additional assumptions. The upper bound could also potentially be improved by exploiting symmetry. Exploring algorithms that explicitly exploit noise symmetry, together with potentially sharper upper bounds, would be an interesting direction for future work.

Lemma 4.3 also uses a local density neighborhood, but ... does not remove this issue.

We thank the reviewer for this suggestion. Appendix B.1 gives the local-neighborhood condition for Lemma 4.3. Let dI>0d_{\mathcal I}>0 be such that [q−(δ2)−dI,q+(δ2)+dI]⊂I[q_-(\delta_2)-d_{\mathcal I},q_+(\delta_2)+d_{\mathcal I}]\subset\mathcal I. The following sample size suffices.

\frac{2\log(2/\gamma)}{[(1-\delta_2)r_0d_{\mathcal I}]^2}, \frac{4}{(1-\delta_2)r_0d_{\mathcal I}}, \frac{1-\alpha}{\alpha} \right).$$ The first two thresholds make each term in $\sqrt{\log(2/\gamma)/(2n_2)}+2/n_2$ smaller than $(1-\delta_2)r_0d_{\mathcal I}/2$. The third ensures that the conformal rank $k_{n_2}$ is at most $n_2$. As $\delta_2$ approaches $\alpha$, the upper quantile $q_+(\delta_2)$ moves further into the tail, potentially reducing the local density lower bound $r_0$ and requiring a larger calibration sample size\. We have included the discussion in Section 4\.1\.2\. > A compact table of assumptions and a numerical upper\-bound diagnostic would help\. > > We thank the reviewer for these helpful suggestions. Table 2 in Appendix A.2 summarizes the key assumptions of our main theoretical results. Appendix F.2 (Figure 17) evaluates the complete four-term finite-sample upper bound in Theorem 4.1 in a Gaussian simulation with a smooth tanh neural network and an explicit replacement-stability constant. At $(\delta_1,\delta_2)=(0.05,0.04)$, the theoretical upper bound is $8.216$ and the empirical 95th percentile of the interval-length deviation over 20 runs is $0.676$. The gap illustrates the conservativeness of the bound in this setting. > The general Huber decision\-theory context \[1\] could also help place the local lower bound\. > > We thank the reviewer for pointing out this connection\. our training\-side lower\-bound construction follows a similar principle of indistinguishability under overlapping Huber contamination neighborhoods\. Our analysis further extends this reasoning to conformal interval\-length deviation and incorporates calibration\-specific lower bounds\. We have added this connection to Section 4\.2\.
AuthorsOpenReview ↗

Response (Part 2)

Weakness 2: "The experiments give limited evidence for the full theoretical setting."

The explicit stability example uses ... would strengthen the link to theory.

We thank the reviewer for this suggestion. We have strengthened the connection between our theoretical assumptions and real-data experiments in two ways. First, we extended Example 4.1 to cover regularized smooth neural networks under explicit conditions. Appendix F.1.1 describes the tanh network and gives its replacement-stability constants for exact regularized fits. Second, we added experiments on four real datasets with mean-shift contamination in training and heavy-tailed contamination in calibration (Figure 19). The observed interval-length deviations are consistent with the distinct effects of training and calibration contamination predicted by our analysis. These new experiments complement our original calibration-only experiments and strengthen the empirical connection to the stability-based theory.

Synthetic length errors use a population oracle, whereas real-data errors use a paired clean split-conformal reference; the captions should make this difference clear.

We thank the reviewer for pointing this out. We have clarified the reference used to compute interval-length deviations in the relevant figure captions: synthetic experiments use the clean population oracle, whereas real-data experiments use paired clean split-conformal intervals constructed within each repetition.

Reusing a fixed test set across the 20 repetitions ... would help.

For the additional smooth tanh experiments, we independently redraw the training, calibration, and test splits across 20 repetitions (Appendix F.1.2). This includes the coverage-correction experiments in Figures 28–29 (Appendix F.4), allowing the reported variability to reflect test-split variation. The original experiments in Appendix D retain fixed test sets.

Empirical reference curves should remain distinct from uniform bounds ... clarify the role of tuning bias [2].

We thank the reviewer for these suggestions. We have clarified the reference curves in the figure captions. Appendix F.2 (Figure 17) compares empirical deviations from a numerical clean population reference with the complete finite-sample upper bound, which applies to exact fits and the exact clean population oracle.

Unknown contamination level. Section 4.3.2 establishes conservative coverage correction when a valid upper bound δˉ2≥δ2\bar{\delta}_2\geq\delta_2 is known. This also suggests a confidence-adjusted extension: if an independent validation sample with verified labels provides an upper confidence bound on δ2\delta_2 with failure probability at most η<α\eta<\alpha, then using α′=(α−η)(1−δˉ2)\alpha'=(\alpha-\eta)(1-\bar{\delta}_2) accounts for the uncertainty in estimating the contamination level. Estimating contamination levels without auxiliary information remains future work.

Tuning bias. We agree that data-dependent score selection may introduce tuning bias. Section 5 clarifies that we compare three prespecified scores without selecting one using the final calibration set. We thank reviewer for the related reference [2] and have cited it in the main text.

Regarding Q1: "Can you add a symmetric-noise comparison ..."

We thank the reviewer for these suggestions. Remark 4.2 explains the role of asymmetry in the minimax lower bound. As shown in Appendix B.8, the derivative qg′(0)q_g'(0) vanishes under symmetric noise but is nonzero for our asymmetric hard instance. Above, we give an explicit sufficient calibration sample size derived from the condition in Appendix B.1. Section 4.1.2 discusses why the required sample size can increase as δ2\delta_2 approaches α\alpha.

Regarding Q2: "Can you strengthen .... independently redrawn test splits?"

We extended Example 4.1 to cover regularized smooth neural networks and give the stability constants for the tanh network in Appendix F.1.1. Figure 19 reports the real-data joint-contamination results against paired clean split-conformal intervals. These experiments use independently redrawn training, calibration, and test splits (Appendix F.1.2).


We thank the reviewer again for the thoughtful and constructive feedback. We hope our responses have adequately addressed the reviewer's concerns.

Thank you for the detailed responses. I have read the revised manuscript and appreciate the clarifications on the lower-bound scope and calibration sample conditions. The added joint-contamination experiments, clearer reference targets, and independently redrawn test splits also address my concerns about empirical validation.

These revisions strengthen the paper and my positive assessment. I maintain my rating of 4 (clear acceptance) and support acceptance. Thank you again for the response.

BTW: One minor editorial point: please capitalize “additional” in the Appendix F entry in the table of contents (page 14, line 746).

Thank you for reviewing the revised manuscript and for your continued support. We’ve capitalized “Additional” in the Appendix F entry in the table of contents of the revised PDF.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.