- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes to test for a distributional change within an ordered sequence by comparing a sequence-level detection score against the same score computed on historical reference sequences using a randomized conformal p-value. Finite-sample validity is established. Further results include multiscale calibration, location-scale invariant scores, and FDR control.
Critical strengths
Finite-sample exact validity is established. The proposed approach is natural and simple.
Critical weaknesses
-
The main novelty is application of the standard conformal p-value to ordered sequences, which is incremental. More detailed comparison to the related literature on finite-sample distribution-free change-point and including them as baselines should be done.
-
The approach is valid for a test sequence that is exchangeable with the references, but not specifically for testing "no change-point". This is because any difference between the test and reference like a distribution shift can cause the hypothesis rejection even if there is no change-point.
Questions for authors
-
Section 2.5 handles distribution shift only for coordinatewise location-scale shifts. The pooled-standardized CUSUM scores in Example 2 are not invariant to coordinatewise rescaling. How about the case in Example 1? It seems that Theorem 1 is essentially only applicable for the random-forest CLR score?
-
To use Theorem 2, do you need to know and if so, how to find this?
-
In the DNN and ReCap-DNN comparison, it is unclear why the DNN uses a fixed 0.5 threshold. Can this be tuned to achieve optimal performance? ReCaP-DNN additionally refits the final layer per test sequence, treating the test sequence as label 0. That biases its score downward, and it is also computationally costly.
-
HASC sequences come from six participants and are cut from longer recordings. If references and test windows overlap in recordings or come from the same sessions, they are not independent. Please clarify how this was avoided in the experiments.
-
The screw-driving study is not a change-point problem but outlier detection. This aligns with the concern in W2. The paper should be positioned correctly to align with the theory.
-
The FDR control in Section 2.6 is not tested.
-
What happens when references are contaminated with change-points? A sensitivity analysis is needed.
2 comments
Thanks for the rebuttal. The application of the theoretical results remains unclear in practice and the novelty remains limited.
We thank the reviewer for the detailed questions. We clarify the scope of the theory and summarize the additional evidence.
Additional comparison (W1).
The supplementary ART comparison holds the sparse-PCA/rank-CUSUM detection score fixed and compares within-sequence permutation calibration with calibration using no-change reference sequences (Table 6 in the Supplementary Material).
Score-level exchangeability and the scope of Theorem 1 (W2; Q1).
ReCaP calibrates one score per sequence using no-change reference sequences. It allows dependence within each sequence and requires exchangeability only of the resulting null scores. The no-change hypothesis alone does not guarantee this condition; the score must also remove nuisance differences that do not carry change-point information.
Theorem 1 is not specific to random forests. The required invariance also holds for several standard classifiers when fitted within the sequence, including tree-based methods (CART, random forests, and GBDT), unpenalized logistic regression with an intercept, and LDA/QDA. SVMs and neural networks can also be used after within-sequence standardization. For the CUSUM example, the raw q=2 score is not generally invariant to coordinatewise rescaling; applying coordinatewise standardization within each sequence gives an invariant score.
Theorem 2 and the industrial application (Q2; Q5).
Theorem 2 gives a guarantee when a valid bound on the deviation of the null p-value distribution from uniformity is available. The bound, denoted by ρ, is generally unknown; it must be justified or bounded under additional assumptions, and the block-based estimate used here is heuristic. We will make this distinction clear. We will also describe the screw-driving study as operation-level anomaly screening. It is an additional application of reference calibration, while the HASC and simulation studies address changes within sequences.
Classifier thresholds (Q3).
For a fixed classifier, no-change validation data used to select a probability cutoff can instead be used as ReCaP calibration references. This avoids a separate cutoff choice and gives decisions at a prespecified significance level. In the supplementary recording-separated HASC comparison, both fixed-head ReCaP and pooled ReCaP-DNN have higher TPR than the tuned-threshold DNN across all five settings, while empirical FPR remains below 0.05.
These comparisons have different empirical FPRs; Table 3 in the Supplementary Material gives the retrospective matched-FPR results. Table 4 in the Supplementary Material reports refitting costs. In pooled fitting, the target and references receive label zero and are ranked within the same fit, without using the true target label.
HASC recording separation (Q4).
The revised evaluation assigns 18 recordings from seven participants to disjoint training, validation, no-change-pool, and test roles (Table 1 in the Supplementary Material). RF and ReCaP-RF use the same test windows as the DNN methods over ten recording-level reassignments (Table 5 in the Supplementary Material).
FDR and reference contamination (Q6–Q7).
In the supplementary fixed-score Gaussian multiple-testing experiment, reference-calibrated p-values followed by BH yielded empirical FDR below 0.05 across all considered mixtures, with power increasing as the non-null proportion increased. Across ten HASC reassignments, fixed-score BH and scalar-score AdaDetect–BH made no discoveries at q=0.05; at q=0.10, discoveries occurred for π1=0.3 and 0.5 (Table 11 in the Supplementary Material). In the Gaussian reference-contamination experiment, the specified mean-change contamination made the procedure increasingly conservative and reduced power. These additional experiments support the claims in the settings considered; they were not part of the original main-paper experiments and do not imply validity under arbitrary reference contamination. Tables 10 and 9 in the Supplementary Material give the Gaussian FDR and contamination settings and results.