Incremental approach. In the current state, this work reads too incremental to the approach proposed in [1]. While the authors discuss the differences in section 4.4, this attribution is too late and not honest. Large parts of the abstract, introduction, background & threat model read in essence identical to [1]. In particular the main approach in section 3 is identical to [1] except for step (4) where this paper proposes a two-level prediction-set aggregation. This paper requires substantial efforts to improve clarity how it contributes beyond existing works, which does not appear easy to address given the current state.
Missing justification for the core contribution. We can only count step (4) of section 3.1 and the corresponding robustness certificates as contributions that go beyond [1]. The problem here is that the paper currently does not motivate or justify the two-level aggregation approach, why is it needed exactly? In which use-cases should we adopt your approach, and why? Specifically, what is the benefit of your two-level aggregation over the majority prediction sets used in [1]? Is it required to obtain the guaranteed lower bound on coverage, and why can't we get this bound for the approach proposed in [1]? This justification and discussion is missing and would be critical to correctly assess the impact of this work.
Missing motivation and justification for parameter α. The parameter α appears central to the proposed approach, yet the authors only introduce it as "an aggregation parameter" (L209) that "controls the trade-off between coverage and prediction set size" (L230). The motivation for this parameter is rather difficult to follow because you never clearly explain the purpose of introducing this additional parameter despite referring to it already in the introduction. Could you please elaborate why you need to introduce it? Why do you require α>1, what happens for α→1, and why do you default to α=2?
Unclear comparison to related work. The authors provide a direct comparison to [1] in Table 2 and show that their approach is empirically better than [1]. There are two problems with this comparison:
- First, the comparison is not fair as it only directly compares the certificates, not the coverage guarantee and sets sizes on clean data. For example, for a fixed significance level ϵ, [1] would guarantee 1−ϵ coverage on clean data, whereas you can only guarantee 1−(α−1)2α2ϵ<1−ϵ (e.g. 1−4ϵ at the default α=2), i.e. this comparison is neither careful nor useful.
- Second, the comparison is only empirical, but what is the mathematical argument for this improvement? Is the scenario (m1,m2,ϵ) a picked setting in which your approach dominates, or are you always dominating their approach? Why did you select SVHN for this comparison, instead of e.g. CIFAR10? How does this direct comparison change for example for m1=0?
A fair empirical comparison would be required, but a direct mathematical comparison of the two approaches would make the improvement argument more sound.
Missing limitation discussion. The main paper misses a careful discussion of the limitations specific to this approach, the current discussion provided in Appendix E is too broad and not sufficient. For example, one limitation is that the current guarantee quickly deteriorates for larger radii. In addition, the proposed approach significantly reduces the coverage guarantee even for clean data: e.g. for a significance level of 0.05, the guaranteed lower bound on coverage is only 80%. This questions the practical applicability of the proposed methodology: Since empirical coverage seems to remain high, we lose the tight coverage guarantee on clean data. Critically, this should also be discussed in the comparison to the closely related works [1] and [2].
Missing comparison to related work. You only cite [2] in the context of robustness guarantees against inference-time perturbations, however, in fact they also provide provable robustness guarantees against data poisoning. How does your approach relate to theirs?
Minor: The caption of Figure 1 is too dense and hard to follow. Legends in the figures in the main text are very difficult to read, they are too small. Grammar mistake in Line 105.
My current recommendation is clear reject based on the arguments provided in weaknesses 2, 3, 4 and 5.
References
[1] Scholten et al., Provably reliable conformal prediction sets in the presence of data poisoning, ICLR 2025.
[2] Zargarbashi et al., Robust Yet Efficient Conformal Prediction Sets, ICML 2024.