- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies calibration poisoning in conformal anomaly detection, showing that high-scoring contaminated samples can inflate the detection threshold and significantly reduce detection power while maintaining high coverage. To address this issue, the authors propose RCAD, which combines symmetric trimming with a quantile-level correction, and RCAD-U, which uses a clean reference set to estimate contamination without requiring a known contamination rate. Experiments across multiple datasets and anomaly detectors show that the proposed methods can substantially improve power retention compared to standard conformal calibration.
Critical strengths
-
The paper is generally well organized and clearly written. The motivation and proposed calibration methods are easy to understand.
-
The paper investigates a practically relevant problem in conformal anomaly detection, highlighting the potential mismatch between coverage and detection power under calibration contamination.
Critical weaknesses
-
The proposed methods are closely related to existing robust calibration and trimming-based approaches. In particular, Bashari et al. [1] have already investigated power degradation under contaminated reference data and considered trimming-based defenses. Although the paper lists Naive-Trim as a baseline, it does not provide a clear quantitative comparison to establish the benefit of the proposed level correction over simpler trimming strategies. Moreover, several relevant studies on conformal prediction under poisoning attacks, including [2–4], are missing from the related work. While these studies consider different prediction tasks and attack settings, they are important for positioning the contribution. The authors should better distinguish their methodological novelty from existing robust conformal calibration approaches and provide more direct comparisons.
-
RCAD mainly addresses upper-tail contamination that increases the detection threshold, but it can perform poorly under lower-tail contamination. Section 5.4 reports worst-case coverage dropping to approximately 0.50 against a nominal target of 0.90. Although RCAD-U attempts to address different contamination placements, its CDF-based estimates may not accurately capture the contamination mass, especially when contaminated scores fall in the middle of the distribution. Furthermore, RCAD-U relies on a relatively clean reference set, and its finite-sample coverage guarantee is not established.
-
The theoretical analysis assumes i.i.d. calibration samples drawn from the mixture distribution , while the experiments replace a fixed number of calibration samples with high-scoring candidates selected by an attacker. These two settings are not necessarily equivalent, and it is unclear whether the theoretical guarantees extend to the implemented adversarial attacks. The attacker’s objective, whether reducing detection power or violating coverage, should also be more explicitly defined.
-
Several theoretical arguments appear incomplete or incorrect. Proposition 8 uses the empirical clean-sample fraction to derive population coverage, but fixing the contamination count does not eliminate the sampling fluctuations. Proposition 12 claims a breakdown point of , which does not generally match the replacement breakdown point of the -th order statistic. Corollary 7 also relies on a CDF-to-quantile argument that may fail when is continuous but not strictly increasing. These results require further justification or corrected proofs.
References
[1] Robust Conformal Outlier Detection under Contaminated Reference Data, ICML 2025.
[2] Robust Yet Efficient Conformal Prediction Sets, ICML 2024.
[3] Provably Reliable Conformal Prediction Sets in the Presence of Data Poisoning, ICLR 2025.
[4] Data Poisoning Attacks against Conformal Prediction, ICML 2024.
Questions for authors
-
What is the main advantage of the proposed quantile correction over simpler approaches such as Naive-Trim or one-sided trimming? A direct comparison would help clarify the methodological contribution.
-
Does RCAD-U provide a theoretical coverage guarantee under general contamination placements, particularly when contaminated scores fall in the middle of the distribution?
-
The theoretical analysis assumes i.i.d. samples from a mixture distribution, while the experiments use fixed-budget adversarial replacement. Could the authors clarify whether the theoretical guarantees still hold under the implemented attack setting, and what assumptions are needed?
-
Could the authors further justify the theoretical results in Corollary 7, Proposition 8, and Proposition 12? In particular, Corollary 7 appears to require additional assumptions on , while Proposition 8 does not clearly justify the transition from empirical to population coverage. The claimed breakdown point of in Proposition 12 also seems inconsistent with the breakdown point of the corresponding order statistic. These issues raise concerns about the validity of the theoretical claims.