acceptodds
Under review as a conference paper at ICLR 2027

ReCaP: Calibrating Change-Point Tests with No-Change Reference Sequences

Abstract

Testing an ordered sequence for a distributional change is difficult when a useful statistic has an unknown null distribution. We address this problem by using historical sequences known to contain no change as a reference sample. We compute the same statistic for the test sequence and each reference sequence, then use the resulting statistics to form a randomized conformal -value. If these statistics are exchangeable under the null, the -value controls Type I error in finite samples. Each statistic may be computed from its own sequence, from the pooled sample of the reference sequences and the test sequence, or from that pooled sample together with optional sequences labelled as containing a change. We also establish finite-sample Type I error control under specified forms of heterogeneity across sequences. A multiscale extension detects multiple changes while retaining the same validity guarantee. Numerical studies demonstrate that our method yields empirical Type I error close to the nominal level and comparable power. We apply the method to detect activity transitions in wearable-sensor data and screen industrial screw-driving operations for anomalies.

Primary area:
learning on time series and dynamical systems
Keywords:
Change-point detection, Conformal inference, Novelty detection, Finite-sample inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_LBaBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes to test for a distributional change within an ordered sequence by comparing a sequence-level detection score against the same score computed on historical reference sequences using a randomized conformal p-value. Finite-sample validity is established. Further results include multiscale calibration, location-scale invariant scores, and FDR control.

Critical strengths

Finite-sample exact validity is established. The proposed approach is natural and simple.

Critical weaknesses
  1. The main novelty is application of the standard conformal p-value to ordered sequences, which is incremental. More detailed comparison to the related literature on finite-sample distribution-free change-point and including them as baselines should be done.

  2. The approach is valid for a test sequence that is exchangeable with the references, but not specifically for testing "no change-point". This is because any difference between the test and reference like a distribution shift can cause the hypothesis rejection even if there is no change-point.

Questions for authors
  1. Section 2.5 handles distribution shift only for coordinatewise location-scale shifts. The pooled-standardized CUSUM scores in Example 2 are not invariant to coordinatewise rescaling. How about the case q=2q=2 in Example 1? It seems that Theorem 1 is essentially only applicable for the random-forest CLR score?

  2. To use Theorem 2, do you need to know ρ\rho and if so, how to find this?

  3. In the DNN and ReCap-DNN comparison, it is unclear why the DNN uses a fixed 0.5 threshold. Can this be tuned to achieve optimal performance? ReCaP-DNN additionally refits the final layer per test sequence, treating the test sequence as label 0. That biases its score downward, and it is also computationally costly.

  4. HASC sequences come from six participants and are cut from longer recordings. If references and test windows overlap in recordings or come from the same sessions, they are not independent. Please clarify how this was avoided in the experiments.

  5. The screw-driving study is not a change-point problem but outlier detection. This aligns with the concern in W2. The paper should be positioned correctly to align with the theory.

  6. The FDR control in Section 2.6 is not tested.

  7. What happens when references are contaminated with change-points? A sensitivity analysis is needed.

2 comments

We thank the reviewer for the detailed questions. We clarify the scope of the theory and summarize the additional evidence.

Additional comparison (W1).

The supplementary ART comparison holds the sparse-PCA/rank-CUSUM detection score fixed and compares within-sequence permutation calibration with calibration using no-change reference sequences (Table 6 in the Supplementary Material).

Score-level exchangeability and the scope of Theorem 1 (W2; Q1).

ReCaP calibrates one score per sequence using no-change reference sequences. It allows dependence within each sequence and requires exchangeability only of the resulting null scores. The no-change hypothesis alone does not guarantee this condition; the score must also remove nuisance differences that do not carry change-point information.

Theorem 1 is not specific to random forests. The required invariance also holds for several standard classifiers when fitted within the sequence, including tree-based methods (CART, random forests, and GBDT), unpenalized logistic regression with an intercept, and LDA/QDA. SVMs and neural networks can also be used after within-sequence standardization. For the CUSUM example, the raw q=2q=2 score is not generally invariant to coordinatewise rescaling; applying coordinatewise standardization within each sequence gives an invariant score.

Theorem 2 and the industrial application (Q2; Q5).

Theorem 2 gives a guarantee when a valid bound on the deviation of the null pp-value distribution from uniformity is available. The bound, denoted by ρ\rho, is generally unknown; it must be justified or bounded under additional assumptions, and the block-based estimate used here is heuristic. We will make this distinction clear. We will also describe the screw-driving study as operation-level anomaly screening. It is an additional application of reference calibration, while the HASC and simulation studies address changes within sequences.

Classifier thresholds (Q3).

For a fixed classifier, no-change validation data used to select a probability cutoff can instead be used as ReCaP calibration references. This avoids a separate cutoff choice and gives decisions at a prespecified significance level. In the supplementary recording-separated HASC comparison, both fixed-head ReCaP and pooled ReCaP-DNN have higher TPR than the tuned-threshold DNN across all five settings, while empirical FPR remains below 0.05.

These comparisons have different empirical FPRs; Table 3 in the Supplementary Material gives the retrospective matched-FPR results. Table 4 in the Supplementary Material reports refitting costs. In pooled fitting, the target and references receive label zero and are ranked within the same fit, without using the true target label.

HASC recording separation (Q4).

The revised evaluation assigns 18 recordings from seven participants to disjoint training, validation, no-change-pool, and test roles (Table 1 in the Supplementary Material). RF and ReCaP-RF use the same test windows as the DNN methods over ten recording-level reassignments (Table 5 in the Supplementary Material).

FDR and reference contamination (Q6–Q7).

In the supplementary fixed-score Gaussian multiple-testing experiment, reference-calibrated pp-values followed by BH yielded empirical FDR below 0.05 across all considered mixtures, with power increasing as the non-null proportion increased. Across ten HASC reassignments, fixed-score BH and scalar-score AdaDetect–BH made no discoveries at q=0.05q=0.05; at q=0.10q=0.10, discoveries occurred for π1=0.3\pi_1=0.3 and 0.50.5 (Table 11 in the Supplementary Material). In the Gaussian reference-contamination experiment, the specified mean-change contamination made the procedure increasingly conservative and reduced power. These additional experiments support the claims in the settings considered; they were not part of the original main-paper experiments and do not imply validity under arbitrary reference contamination. Tables 10 and 9 in the Supplementary Material give the Gaussian FDR and contamination settings and results.

Thanks for the rebuttal. The application of the theoretical results remains unclear in practice and the novelty remains limited.

Reviewer_vTuqOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies the problem of testing an ordered sequence for a distributional change when historical sequences without a changepoint are available. ReCaP computes a detection score for each reference sequence and the test sequence, then constructs a randomized conformal p-value by comparing these scores. Under conditional exchangeability of the scores, the test has exact finite-sample Type I error control, while allowing for temporal dependence within sequences. Their framework accommodates classical CUSUM statistics, random-forest classifier scores, and classifiers trained using additional sequences containing changes. The authors also consider scores that aggregate evidence across multiple intervals, establish validity under certain forms of heterogeneity across sequences, and apply AdaDetect to control the FDR when testing multiple sequences. Their experiments cover Gaussian mean changes, activity transitions, and industrial anomaly screening.

Critical strengths
  1. Using historical no-change sequences is a useful way to calibrate a test when the test statistic’s null distribution is difficult to derive. The method allows dependence within each sequence, provided the reference and test scores are exchangeable under the null. This makes it applicable to a broad range of classical and learned statistics.
  2. The experiments show that calibration using reference sequences can control false positives while retaining useful power. In the activity-recognition example, the calibrated neural-network classifier keeps the false-positive rate close to 5% across different training class proportions, with similar detection rates to the original classifier in most settings.
Critical weaknesses
  1. The methodological contribution beyond existing conformal inference techniques is limited. The main validity result is a standard conformal rank argument, the results for location–scale transformations and multiple intervals follow immediately from invariance and exchangeability, and the FDR guarantee uses an existing procedure. Theorem 2 assumes a bound on the unknown null distribution of the p-value, but does not derive it from assumptions on how the reference and test sequences differ. The proposed method for estimating this bound has no validity guarantee and can substantially reduce power. Consequently, the paper offers limited new theory for situations in which the reference and test sequences are not directly comparable.
  2. The experiments do not establish an advantage over simpler calibration methods. The neural-network baselines classify at a fixed probability threshold of 0.5, while the proposed method explicitly targets a false-positive rate of 0.05. A more informative comparison would calibrate the same classifier using held-out no-change sequences and compare power at the same false-positive rate. This would help determine whether refitting the classifier for each test sequence provides a meaningful benefit. Similarly, the experiment with temporal dependence shows that the particular baseline calibration fails, but does not compare against methods designed to handle such dependence. The industrial application concerns anomalous operations rather than changes within a sequence, so it provides limited evidence for the contribution to change-point testing specifically.
Questions for authors
  1. How does the proposed method compare with training a classifier once and calibrating its scores using held-out no-change sequences? At the same false-positive rate and with comparable amounts of data, does refitting the classifier for each test sequence improve power, and how much additional computation does it require?
  2. Can the bound in Theorem 2 be derived from assumptions on how the reference and test sequences differ? Are there conditions under which the proposed block-based estimate gives a valid bound for the full-sequence p-value?
  3. In the HASC experiments, are the pretraining, reference, and test samples separated by their original recordings? Can windows overlap across these samples, and how does the sampling procedure support the exchangeability assumption after conditioning on the pretrained model?
2 comments

We thank the reviewer for the careful assessment, especially the questions about the methodological contribution, classifier comparisons, temporal dependence, and the HASC data split.

Contribution and positioning (W1).

The contribution of ReCaP is a calibration framework for change-point scores. It uses historical no-change sequences to calibrate one score per sequence, so it does not require exchangeability of the observations within a sequence. This allows the framework to be used with existing classical and learned detection statistics, while the validity condition is stated directly at the score level. The supplementary ART comparison keeps the detection score fixed and examines the change in calibration (Table 6 in the Supplementary Material).

Classifier comparison and refitting (W2; Q1).

We agree that comparison with a DNN using a fixed 0.5 cutoff alone does not isolate the benefit of calibration. In an additional recording-separated HASC experiment conducted for this response, we therefore compared a tuned-threshold DNN, a fixed DNN calibrated by ReCaP, and the test-specific pooled ReCaP-DNN. All DNN variants use the same frozen encoder, labelled-change examples, normal-data budget, and test sequences within each reassignment and each rr. The tuned-threshold and fixed-head ReCaP procedures use the same fitted classifier and held-out no-change pool; they differ in how that pool is used.

Table R3. HASC detection rates and amortized head-stage costs over ten recording-level splits.

rrMethodFPRTPRMERHead ms/test
0.1DNN-0.50.00000.43500.28250.227
DNN-tuned0.01500.48500.26500.190
Fixed-DNN + ReCaP0.02150.50350.25900.240
Pooled ReCaP-DNN0.02600.53130.24749.668
0.5DNN-0.50.05500.62000.21750.237
DNN-tuned0.00500.52500.24000.188
Fixed-DNN + ReCaP0.00830.55280.22780.244
Pooled ReCaP-DNN0.03080.56700.23199.680
0.9DNN-0.50.11000.72500.19250.219
DNN-tuned0.01000.54000.23500.175
Fixed-DNN + ReCaP0.01250.56180.22540.230
Pooled ReCaP-DNN0.04050.56750.236510.402
—RF (package-default)0.91501.00000.4575—
—ReCaP-RF0.05550.63630.2096—

Both ReCaP variants achieve higher TPRs than DNN-tuned with empirical FPRs below 0.050.05. The retrospective matched-FPR comparison shows higher TPR for pooled refitting at r=0.1r=0.1 and for fixed-head calibration at r=0.5,0.9r=0.5,0.9 (Table 3 in the Supplementary Material). Their amortized head-stage costs are approximately 1010 and 0.240.24 ms per test, respectively.

Temporal dependence and Theorem 2 (W1–W2; Q2).

The original paper already includes a Gaussian AR(1) experiment. The supplementary ART comparison adds a recent method that uses within-sequence permutation calibration, while keeping the detection score fixed. We also implemented the block Gaussian multiplier CUSUM test for time series in Yu and Chen (2021, Remark 2). On the same AR(1) design and using the same max-coordinate CUSUM score, ReCaP achieved empirical size 0.048 and power 0.214. The best-calibrated block choice among those examined, M=5M=5, had size 0.034 and power 0.156; smaller and larger blocks showed substantial size distortion. This provides a direct comparison with a dependence-aware calibration method and also illustrates the importance of block-length selection. Regarding Theorem 2, the theorem treats the deviation bound (denoted by ρ\rho) as an assumption. It is generally unknown and must be justified or conservatively bounded under additional assumptions on the reference and test distributions. The current block-based estimate is heuristic and is not itself a finite-sample certificate.

HASC data separation (Q3).

The original window-level split allowed overlapping raw samples between reference and test windows. We therefore reran the HASC comparison using a recording-separated protocol. Each recording is assigned exclusively to encoder training, validation, the no-change pool, or testing, eliminating shared raw samples across these roles. All methods, including RF and ReCaP-RF, are evaluated on the same test windows over ten recording-level reassignments (Tables 1 and 5 in the Supplementary Material).

Industrial application (W2).

The HASC and Gaussian experiments directly evaluate within-sequence change detection. The industrial study demonstrates a complementary application of the same calibration framework to anomaly screening of complete operations.

I appreciate the new HASC experiments, which address my concern about overlapping recordings, but the gains from refitting are mixed at matched false-positive rates and the correction in Theorem 2 still relies on a bound whose estimate is heuristic, so I’m keeping my original score.

Reviewer_oKtvOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper propose a new hypothesis test for time series data based on a randomized conformal p-value. Through experiments, it demonstrates that the proposed method can successfully control FDR while it can detect different time series accurately.

Critical strengths
  • The proposed method is theoretically sounds.
  • The proposed method works in real data.
Critical weaknesses
  • The proposed method was compared with classical methods
  • It is validated only for the proposed method and it is difficult assess how good the proposed method compared with existing methods.
  • For null hypothesis, it assumes i.i.d, while time series data is non-i.i.d.
Questions for authors
  • How good the proposed method with more recent methods?
  • The proposd method assume i.i.d. If we assume i.i.d., we can use most of hypothesis test approaches. How we can extend it to non-i.i.d. setup?
1 comment

We thank the reviewer for raising the questions of temporal dependence and comparison with recent methods.

Temporal dependence (W3; Q2).

ReCaP does not require observations within a sequence to be i.i.d. or exchangeable. Its finite-sample validity relies on exchangeability of the sequence-level scores under the null. The Gaussian AR(1) experiment in the original paper illustrates this setting. We also conducted a supplementary comparison with ART, using exactly the same sparse-PCA/rank-CUSUM score and changing only the calibration procedure. Table R2 uses 500 null and 500 alternative sequences per setting. The methods perform similarly under i.i.d. observations. Under AR(1), ReCaP-ART has size near 0.050.05, whereas ART has substantial size inflation, precluding a size-matched power comparison. Table 6 in the Supplementary Material gives the experimental details.

Table R2. ART and ReCaP-ART under independent and AR(1) observations, at nominal level 0.050.05.

& \text{i.i.d. Gaussian} & \text{AR(1), }\rho=0.5 \\\\ \hline \begin{array}{l} \text{Method} \\\\ \hline \text{ART} \\\\ \text{ReCaP-ART} \end{array} & \begin{array}{cc} \text{Size} & \text{Power} \\\\ \hline 0.066 & 1.000 \\\\ 0.068 & 1.000 \end{array} & \begin{array}{cc} \text{Size} & \text{Power} \\\\ \hline 0.930 & 1.000 \\\\ 0.046 & 0.948 \end{array} \\\\ \hline \end{array}$$ **Comparison with recent methods (W1–W2; Q1).** The comparison is with the recent method of Cui et al. (2026), published in the *Journal of the Royal Statistical Society: Series B (Statistical Methodology)*. Our contribution is the calibration framework: it can use the same score while replacing within-sequence permutation calibration with calibration across no-change reference sequences. The supplementary experiment holds the score fixed, so it isolates this difference in calibration and shows its relevance when observations within a sequence are dependent.
Reviewer_3jfrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ReCaP, which is a framework for CPMs that compares the score on the target sequence with the scores got on historical sequences known to be "nominal", i.e., without changes. This allow the framework not to derive or approximate a null distribution. The framework relies on the exact control of type I error and covers CUSUM, RF and NNs (on two real applications).

Critical strengths
  • the idea to separated the choice of the score from the calibration is simple and useful
  • The finite-sample guarantee is clear and well described
  • intersting technical solutions have been considered (i.e., symmetric pooling, architecture description, disjoint split)
  • Results are promising
Critical weaknesses
  • The idea seems to be quite incremental and the novelty limited
  • what proposed seems to strictly depend on the assumption (exchange test and history) which is quite strong is real-world settings. What happens when this assumption does not hold?
  • what happens when n0 is not equal to 200?
  • the real applications rely on threshold-based classifiers. Why? How to tune the threshold?
Questions for authors

See previous box

1 comment

We thank the reviewer for the constructive assessment and for recognizing the value of separating score construction from calibration. We clarify the contribution and summarize the additional evidence below.

Contribution and exchangeability (W1–W2).

ReCaP provides a general calibration framework for existing change-point detection statistics. Its key difference from conventional within-sequence distribution-free methods is that it calibrates one score per sequence using historical no-change reference sequences. The observations within a sequence need not be exchangeable, so temporal dependence within each sequence is allowed. The required condition is exchangeability of the scores from the reference sequences and a null test sequence. This can be more realistic than requiring the raw sequences themselves to be exchangeable: an appropriate detection score may remove baseline or scale differences that are unrelated to the change of interest. Theorem 1 gives a formal example under sequence-specific location–scale transformations.

The no-change hypothesis alone does not guarantee comparable reference and test scores. When score exchangeability is not exact, Theorem 2 gives a guarantee if a valid bound on the deviation of the null pp-value distribution from uniformity is available. The block-based estimate in the paper is a practical heuristic for this quantity; we will clarify the distinction between the theorem's condition and its estimation.

Reference sample size (W3).

The choice n0=200n_0=200 is an experimental setting, not a theoretical requirement. Proposition 1 applies to any n0≥1n_0\geq1 with randomized pp-values. In an additional Gaussian sensitivity experiment conducted for this response, increasing n0n_0 from 20 to 200 raised ReCaP-Combined power from 0.600 to 0.720 in the sparse setting, with smaller gains beyond 200. Empirical size remained broadly near the nominal level. This supports n0=200n_0=200 as a practical choice for the settings considered.

Classifier thresholds (W4).

A classifier used at a target false-positive rate usually needs no-change validation data to choose its probability cutoff. ReCaP uses these data directly as calibration references: classifier outputs remain continuous scores, and their ranks produce pp-values tested at a prespecified level.

We added DNN-tuned and Fixed-DNN + ReCaP to a recording-separated HASC comparison. These two methods share the same fitted classifier and held-out no-change pool. In all three settings shown in Table R1, both ReCaP variants have higher TPRs than DNN-tuned and empirical FPRs below 0.050.05, with Pooled ReCaP-DNN achieving the highest TPR among the calibrated methods. Tables 1 and 2 in the Supplementary Material provide the data allocation and full results.

Table R1. HASC classifier thresholds and reference calibration.

rrMethodFPRTPR
0.1DNN-0.50.00000.4350
DNN-tuned0.01500.4850
Fixed-DNN + ReCaP0.02150.5035
Pooled ReCaP-DNN0.02600.5313
0.5DNN-0.50.05500.6200
DNN-tuned0.00500.5250
Fixed-DNN + ReCaP0.00830.5528
Pooled ReCaP-DNN0.03080.5670
0.9DNN-0.50.11000.7250
DNN-tuned0.01000.5400
Fixed-DNN + ReCaP0.01250.5618
Pooled ReCaP-DNN0.04050.5675

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.