acceptodds
Withdrawn from ICLR 2027

Provably Robust Conformal Prediction against Arbitrary Data Poisoning Attacks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Conformal Prediction (CP) provides a principled approach to uncertainty quantification in machine learning by constructing prediction sets that provably contain the true label with high coverage. However, recent studies have shown that CP is highly vulnerable to data poisoning attacks: manipulating even a small fraction of training or calibration samples can significantly inflate prediction set sizes or violate coverage guarantees, undermining CP’s reliability. We propose JointCP, a robust CP framework to defend against arbitrary data poisoning attacks with formal coverage guarantees, i.e., the coverage is still valid when a fraction of training and calibration samples used in CP are arbitrarily perturbed (e.g., deleted, added, modified, or combined in any manner). Results on multiple image classification datasets demonstrate the effectiveness and robustness of JointCP from both empirical and theoretical perspectives.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
conformal prediction, data poisoning, certified defenses, machine learning security, adversarial robustness

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_4MHkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper’s core contribution is to propose a robust conformal prediction framework built upon two-stage voting aggregation, delivering distribution-free and data-dependent coverage guarantees.

Critical strengths

Theoretically, it derives coverage lower bounds under both clean and poisoned settings. Experimentally, it achieves high coverage with acceptable set sizes across multiple image classification datasets, outperforming baselines such as RPS.

Critical weaknesses

I have several concerns about this work. First, does poisoning during training or calibration include canonical adversarial examples such as FGSM and PGD? Furthermore, m plays an important role in the theory; a small value of m can severely violate exchangeability. In addition, would failures arise when evaluating on adversarial test samples?

Questions for authors

Beyond the aforementioned weaknesses, the paper also lacks discussion of recent relevant works, including:

  • Enhancing adversarial robustness with conformal prediction: A framework for guaranteed model reliability
  • Ensuring Calibration Robustness in Split Conformal Prediction Under Adversarial Attacks
Reviewer_NDujOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper provides a conformal prediction methodology that can be provably robust to adversarial data insertion, deletion and modification attacks to the training and calibration data (i.e. against poisoning attacks). The paper also derives robustness certificates for the proposed approach.

Critical strengths

The problem of provable robustness against data poisoning attacks in conformal prediction is important, and the authors correctly and sufficiently motivate this problem. The paper is also generally written in a clear way and easy to follow (except as discussed in weaknesses). Threat model and approach is clearly described, background appears methodologically correct with the given related work as context.

Critical weaknesses

Incremental approach. In the current state, this work reads too incremental to the approach proposed in [1]. While the authors discuss the differences in section 4.4, this attribution is too late and not honest. Large parts of the abstract, introduction, background & threat model read in essence identical to [1]. In particular the main approach in section 3 is identical to [1] except for step (4) where this paper proposes a two-level prediction-set aggregation. This paper requires substantial efforts to improve clarity how it contributes beyond existing works, which does not appear easy to address given the current state.

Missing justification for the core contribution. We can only count step (4) of section 3.1 and the corresponding robustness certificates as contributions that go beyond [1]. The problem here is that the paper currently does not motivate or justify the two-level aggregation approach, why is it needed exactly? In which use-cases should we adopt your approach, and why? Specifically, what is the benefit of your two-level aggregation over the majority prediction sets used in [1]? Is it required to obtain the guaranteed lower bound on coverage, and why can't we get this bound for the approach proposed in [1]? This justification and discussion is missing and would be critical to correctly assess the impact of this work.

Missing motivation and justification for parameter α\alpha. The parameter α\alpha appears central to the proposed approach, yet the authors only introduce it as "an aggregation parameter" (L209) that "controls the trade-off between coverage and prediction set size" (L230). The motivation for this parameter is rather difficult to follow because you never clearly explain the purpose of introducing this additional parameter despite referring to it already in the introduction. Could you please elaborate why you need to introduce it? Why do you require α>1\alpha>1, what happens for α→1\alpha \rightarrow 1, and why do you default to α=2\alpha=2?

Unclear comparison to related work. The authors provide a direct comparison to [1] in Table 2 and show that their approach is empirically better than [1]. There are two problems with this comparison:

  • First, the comparison is not fair as it only directly compares the certificates, not the coverage guarantee and sets sizes on clean data. For example, for a fixed significance level ϵ\epsilon, [1] would guarantee 1−ϵ1-\epsilon coverage on clean data, whereas you can only guarantee 1−α2ϵ(α−1)2<1−ϵ1-\frac{\alpha^2\epsilon}{(\alpha-1)^2} < 1-\epsilon (e.g. 1−4ϵ1-4\epsilon at the default α=2\alpha=2), i.e. this comparison is neither careful nor useful.
  • Second, the comparison is only empirical, but what is the mathematical argument for this improvement? Is the scenario (m1,m2,ϵm_1, m_2, \epsilon) a picked setting in which your approach dominates, or are you always dominating their approach? Why did you select SVHN for this comparison, instead of e.g. CIFAR10? How does this direct comparison change for example for m1=0m_1=0?

A fair empirical comparison would be required, but a direct mathematical comparison of the two approaches would make the improvement argument more sound.

Missing limitation discussion. The main paper misses a careful discussion of the limitations specific to this approach, the current discussion provided in Appendix E is too broad and not sufficient. For example, one limitation is that the current guarantee quickly deteriorates for larger radii. In addition, the proposed approach significantly reduces the coverage guarantee even for clean data: e.g. for a significance level of 0.05, the guaranteed lower bound on coverage is only 80%. This questions the practical applicability of the proposed methodology: Since empirical coverage seems to remain high, we lose the tight coverage guarantee on clean data. Critically, this should also be discussed in the comparison to the closely related works [1] and [2].

Missing comparison to related work. You only cite [2] in the context of robustness guarantees against inference-time perturbations, however, in fact they also provide provable robustness guarantees against data poisoning. How does your approach relate to theirs?

Minor: The caption of Figure 1 is too dense and hard to follow. Legends in the figures in the main text are very difficult to read, they are too small. Grammar mistake in Line 105.

My current recommendation is clear reject based on the arguments provided in weaknesses 2, 3, 4 and 5.


References

[1] Scholten et al., Provably reliable conformal prediction sets in the presence of data poisoning, ICLR 2025.

[2] Zargarbashi et al., Robust Yet Efficient Conformal Prediction Sets, ICML 2024.

Questions for authors

Questions see weaknesses.

Reviewer_UPqgOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper proposes JointCP, a defense method against data poisoning attacks utilizing conformal predictions. The method proceeds in four stages after partitioning the training and calibration set: training a classifier on each training partition, calibrating each classifier on every calibration partition, constructing the prediction sets, and aggregating these sets through a two-level aggregation mechanism. The aggregation combines prediction sets across calibration partitions for each classifier, then combines the resulting sets across classifiers. The authors evaluates the method on SVHN, CIFAR-10, and CIFAR-100.

Critical strengths

None

Critical weaknesses

I have concerns about the extent of the contribution relative to prior works. The methodology of JointCP, including partitioning the training and calibration set, as well as constructing the prediction sets, have significant overlap with [1]. Overall, the methodological contribution, in the way it is currently presented, appears incremental, and its significance is not clearly established. In particular, the motivation for the two-level aggregation remains unclear and should be more thoroughly explained.

There are significant presentation / formatting issues that are undermining the readability and credibility of this work. The transitions between the problem setup, method, and theoretical analysis need improvement, particularly in explaining how the design choices follow from the stated objectives. There are also various citation formatting, awkward phrasing, and grammatical issues throughout the paper. The manuscript would benefit from substantive reorganization and thorough language editing. I have included some, but not all, examples below:

Pg.1:

  • Please fix the in-text citation formats
  • Line 39: “probability 90%”
  • The word “and” is bolded in Line 48.
  • Line 49: “Figure 1 an overview”

Pg.2:

  • Please avoid introducing notations in the caption of a figure and explain the method in plain text. ...
Questions for authors
  • Could the author please clarify the motivation for using the two-level aggregation mechanism? I.e., is using the two-level aggregation a necessary condition for obtaining the distribution-free coverage guarantee? The authors could also show ablation results with m2=1m_2 = 1 to show what the calibration-level vote contributes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.