- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The author proposes CAVET-AD, a detector-agnostic post-processing threshold calibration framework: it maps detector scores to a reference tail space via a frozen reference ECDF, refines the reference upper tail using extreme value theory, and controls a bounded adaptive runtime budget with a batch tail excess response ; when rtail approaches zero, it reverts to a conservative fixed rule.
Critical strengths
- The value of this work lies first and foremost in its problem framework: instead of continuing to ask "how to tune thresholds," it explicitly shifts to "when one should tune thresholds." This point holds practical significance in anomaly detection deployment.
- It is worth affirming that the author proactively reported multiple negative results and boundary conditions.
Critical weaknesses
- As currently written, this paper could easily be perceived as just "another pipeline combining EVT and adaptive budgeting," with its main selling point being the increase in average F1 score from 0.475 to 0.579.
- Leave-one-out dataset strategy is a cross-fitted evaluation, not a dynamic deployment on a real-time stream.
- The work compares SPOT, Conformal–BH, and various fixed thresholds, but does not perform a same-condition comparison with methods that directly estimate the deployment mixing ratio or contamination factor.
- Section 5.7 and Appendix G.5's controlled covariate shift tests show that on CreditCard2013, when shift=1.00, the average rtail rises to 0.3390, but the adaptive F1 drops sharply instead.
Some typoes:
- The dataset naming is inconsistent. The text alternates between CreditCard2013, CREDITCARD2013, Credit Card 2013, and CICIDS2018, CSE-CIC-IDS2018, CICIDS2018, etc. It is recommended to standardize the abbreviations and strictly adhere to them after their first appearance.
Questions for authors
- Can you supplement a method that directly estimates deployment prevalence or contamination proportion as a baseline, and explain under which assumptions CAVET-AD should outperform the direct estimate, and under which conditions it should underperform?
- The biconditional KS threshold in Appendix G.9 is a study-specific post-hoc threshold. Can it be transformed into a transferable, fixed a priori rule, or at least one validated on a second dataset? If not, should the main conclusions be further restricted to a pure prevalence-shift protocol?
Response to the Rebuttal
I appreciate the authors’ time and effort for the rebuttal. Yet I’m unconvinced that the main concerns have been addressed.
For example, for Q1, the authors acknowledge that a prevalence estimate does not answer the "when to adapt" question. Besides, the results in Tables 12 and 13 seem to suggest a more nuanced relationship between tail informativeness and adaptation benefit, which I feel can raise more concerns. While the reported held-out results are encouraging, Table 14 shows that the proposed gate does not outperform adaptive. Besides, I also noticed that related concerns about EVT and prevalence baselines were also raised by other reviewers.
Overall, I appreciate the authors' clarifications, but the some concerns remain unresolved. I therefore maintain my original score.