- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper argues that a detector can have perfect false-alarm validity while being completely blind to a distribution shift that leaves its score distribution unchanged. It proposes an approach that recursively splits existing score groups rather than replacing them, with the objective of exposing specified within-group discrepancies.
Critical strengths
- The work points out that a valid conformal score need not be a sufficient representation for every relevant alternative, i.e., the distribution of the scores can remain unchanged under a target distribution.
- Refinement is a natural approach to tackle the problem. This approach also yields interpretable theoretical results.
Critical weaknesses
-
QuantTree constructs a recursive binary partition of the observation space specifically for multivariate change detection. The distinction between the proposed approach and QuantTree from an algorithmic and mathematical perspective has not been clarified.
-
The Algorithm 1 details and the descriptions in the main text are not sufficient to provide a good understanding for reproducibility. Section 4 is too sparse for a good understanding.
-
The approach "searches supplied target changes; it does not generate new attacks during refinement". In realistic anomaly detection, the changes are typically unknown in advance.
Questions for authors
-
How is the "logistic-halfspace dictionary with axis-aligned fallbacks" done? What is the computational complexity?
-
What is the precise novelty relative to multicalibrated partition refinement and recursive partition in QuantTree?
-
The method explicitly searches supplied target changes. What happens when the true future alternative Q is outside the declared portfolio?
-
In contrast to the theory, the Pearson vs Gini improvement does not meet the target criterion and the results do not establish that the proposed Pearson split criterion is better than a standard supervised partition criterion.
-
The paper concludes that positive design-family certificates did not establish competitive unseen-family ranking. Does this mean that certifying that a partition captures information about declared/training alternatives does not imply that the partition is a good anomaly representation for unseen alternatives? How can the method repair a detector in a practical application?
-
The difference between CoFi-E (0.58 AUROC) and Isolation Forest (0.75) is large. Why?
-
There should be comparisons to other baseline approaches like sequential predictive two-sample testing, sequential kernelized testing and nonparametric e-detectors.
-
The current audit guarantee appears to depend critically on the target/search specification being fixed independently of the audit sample. The precise adaptive-data-selection boundary should be stated more explicitly.
-
Conformal systems commonly operate with a fixed calibration set rather than continuously obtaining fresh samples. How to ensure the runtime guarantee?
3 comments
Response (2/3) – unknown changes, Pearson, and AUROC
Q3 (future change outside the declared target portfolio). Nested refinement still preserves the inherited grouping's population rank information for an unlisted alternative (Corollary 1). If CoFi-E+ includes the inherited evidence process with weight w, its evidence is at least w times that component's evidence at every time (Sec. 4.5). These safeguards do not ensure equal finite-reference delay or competitive unseen-family rankings. Theorem 3 (App. B.11) gives conditional transfer results requiring target-law coverage. No fixed finite grouping can generally expose every unknown change; our guarantee is retention, with additional certified gain for the supplied targets.
Q4 (Pearson versus Gini). Pearson is the default because selection and certification use the same information gain (Theorem 1), and Theorem 2 proves the stated Pearson-greedy rate. This does not establish a detection advantage: the corrected exact-K contrast is 0.25% [−1.66, 2.07], below the prespecified 10% criterion (Sec. 6.3). Gini is an interpretable alternative criterion and receives the same independent certificates (Remark 2; App. D.5). The contribution remains applicable with either criterion.
Q5 (do target certificates predict unseen-anomaly rankings? how is repair used?). Certificates establish gain for specified changes, not generic unseen-family ranking. The practical use is to repair a missed behavior identified through incident review, red-teaming, or recurring false negatives. Representative examples are split into proposal, selection, and audit roles; accepted refinements are certified and frozen before monitoring. In the clean panel, Pearson maps improve target AUROC to 0.821 versus 0.733 for the original Isolation Forest score, while other-family AUROC is 0.643 (App. D.10). This supports repairing an identified blind spot alongside the existing detector.
Q6 (historical 0.582 versus 0.754 AUROC). The historical evaluation excluded each test family from repair; the gap across its 605 folds remains a negative result (App. D.12, Table 18). Two diagnostics identify relevant factors:
- Ranking: our new intervention holds repaired groups fixed and refits only their ranking using additional future-family selection labels. Pearson other-family AUROC rises 0.643 → 0.742; with the same extra supervision, repaired groups add 0.034 [0.010, 0.063] over the original four groups (App. D.10).
- Coverage: the audit already reported in the submission finds that including the held-out family among the targets raises AUROC 0.5722 → 0.6137 across 13 datasets (App. D.12).
The ranking intervention uses extra labels on a separate panel, so it is not an unsupervised improvement or a full explanation of the historical gap. Together, the diagnostics show why target coverage and the fitted ranking matter when assessing transfer from repaired groups.
Response (3/3) – baselines, audit adaptivity, and calibration
Q7 (sequential baselines). Section 6.5 and Appendix D.9 (Table 16) add all requested families on 145 clean fits with paired evaluation, the same nominal 5% anytime false-alarm criterion, and a common restricted-delay endpoint:
| Procedure | Delay ↓ | Detection | FA | Target examples | Refs/step |
|---|---|---|---|---|---|
| QuantTree partition + common KT | 103.81 | 0.479 | 0.034 | no | 63 |
| Kernel betting | 63.77 | 0.804 | 0.003 | no | 1 |
| Certified CoFi-E | 53.09 | 0.812 | 0.034 | yes | 63 |
| Predictive test, target-initialized | 42.90 | 0.884 | 0.025 | yes | 1 |
| Predictive test, no target initialization | 150.05 | 0.057 | 0.000 | no | 1 |
| E-detector, adapted to anytime PFA | 32.47 | 0.924 | 0.006 | yes | 1 |
CoFi-E reduces restricted delay versus QuantTree partitions by 48.86% [36.54, 62.36] and kernel betting by 16.75% [8.62, 26.10] (approximate Bonferroni-adjusted 98.75% four-dataset-cluster intervals). The e-detector is faster; the difference from the target-initialized predictive test is inconclusive. QuantTree uses common KT evidence rather than its native test. The faster direct methods use target examples and full features; this panel does not measure earlier-family retention.
Retention motivates repair: the separate historical panel reports earlier-family detection of 97.1% for CoFi-E versus 25.3%/36.0% for logistic/HGB replacement, under the qualifications disclosed for Table 1. This does not establish retention superiority over the new sequential baselines. Together with population-information retention (Corollary 1), it supports repair as a useful option when preserving an existing detector's distinctions matters.
Q8 (audit adaptive-selection boundary). Before audit, the method fixes preprocessing, targets, directions, thresholds, complete paths, and the entire family of parent/child comparisons for which simultaneous bounds are allocated (Sec. 4.3; Apps. B.7–B.8). Audit outcomes may select a certified prefix or a parent-consistent branch subset within that fixed family. They cannot create new cuts, refit features, change targets, or move thresholds without fresh independent audit data or another valid adaptive argument. This is the boundary required for the claimed simultaneous error control.
Q9 (fixed calibration rather than fresh references). Appendix B.12 (new Proposition 4) supplies a separate online conformal construction requiring only one initial calibration set. Freeze the representation first, then rank each new score among the calibration scores, previous monitored scores, and itself, with randomized ties. Under joint exchangeability conditional on the frozen design, these sequential p-values are iid uniform. Nonnegative betting functions with integral at most one, chosen using only the design and past p-values, give anytime false-alarm control. The guarantee is marginal over the calibration sample, not conditional on its realized values. The evaluated score-group KT update still requires fresh normal references; the online construction's detection power is not evaluated here.
Response (1/3) – presentation, implementation, and novelty
Thank you for the detailed criticism of presentation, reproducibility, and novelty. We revised the presentation without replacing the submitted primary method.
Presentation and Q1 (candidate construction/complexity). Figure 1 now separates (a) the inherited score's blind spot, (b) joint refinement versus marginal merging, and (c) propose–select–audit–freeze. The Introduction adds a concrete ZIP-code/street-level example. Section 4.1 and the expanded 13-step Algorithm 1 identify data roles explicitly; Sec. 4.1 defines the round cap T, minimum child support, and FitAndFreeze, while Appendices C.1–C.3 specify implementation and measured cost.
In the matched study: balanced logistic regression on proposal normal/target examples supplies one direction per target; we add min(15,d) seeded coordinate directions. For each direction and current group, candidate thresholds are the seven normal-selection quantiles 1/8,...,7/8. A greedy selection-data search starts from four inherited groups, adds at most 12 splits (K ∈ {8,16}), checks at least six normal rows per child, and caps evaluations at 16,384. The complete path is fixed before auditing. The stated complexity is O(sDn(d + log n)) for search, O(n_w d) per logistic-fit iteration, and four exact count intervals per proposed split (App. C.3). Median proposal/search/audit times are 6.15/57.21/3.42 ms (App. C.3).
Q2 (QuantTree and multicalibrated partition refinement). Our contribution is an audited repair of an inherited detector, with an explicit retention constraint and per-split gain certificates. Recursive splitting, nesting, and data processing are shared ingredients. The analysis (1) characterizes score-law collisions and why fixed-weight marginal merging remains blind when every component score law is unchanged (Sec. 3); (2) quantifies information recovered by nested splits and proves retention and conditional greedy progress (Theorems 1–2; Corollary 1); and (3) connects independent certification to valid rank monitoring after freezing (Secs. 4–5).
Appendix E.1 (Table 19) now compares the original formulations: QuantTree builds normal-data histograms for change detection; multicalibrated partitions target importance-weight accuracy. We do not claim these approaches cannot be extended with inherited groups, auditing, or conformal monitoring. In a partition-only comparison using common KT evidence, CoFi-E's delay is 53.09 versus 103.81 for QuantTree partitions (App. D.9); this is not a comparison with QuantTree's native test and CoFi-E uses target examples. The novelty claim is the integrated repair objective plus independently audited guarantees, not binary partitioning itself.