- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes WONDER, an unsupervised online change-point detector that compares consecutive sliding windows using a neural approximation to the Wasserstein distance. The critic is updated incrementally with a gradient penalty. The authors derive concentration bounds and guarantees for false alarms and detection delay for the idealized statistic. Experiments on synthetic data and MNIST report lower detection delays than five baselines.
Critical strengths
1.The method combines an adaptive discrepancy with incremental neural training, providing a practical approach to detecting distributional changes without labeled change events.
2.The analysis explicitly handles partially contaminated windows and dependence between overlapping tests. These considerations improve its relevance to online detection.
Critical weaknesses
-
The guarantees do not adequately cover the implemented detector. The theory assumes an exact supremum over strictly Lipschitz, norm-bounded networks, whereas the implementation uses a soft gradient penalty and eight updates per step. At target ARL 1,000, the theoretical thresholds exceed the statistic’s maximum possible value under both experimental window configurations, making them uninformative about detection performance. Even if the trained critic satisfies the assumed constraints, false-alarm bounds transfer, but detection guarantees require control of the optimization gap.
-
The experimental comparisons do not establish matched actual ARL. Thresholds are calibrated using marginal statistic quantiles rather than measured first-alarm times. With temporally dependent statistics, this does not establish the claimed ARL. Calibration at ARL 100,000 also relies on approximately 100,000 observations, providing very limited information about the relevant tail.
Questions for authors
Which guarantees apply to the implemented statistic, and how are the Lipschitz and layer-norm assumptions ensured?
Are independently measured ARLs available? Does WONDER retain its advantage when methods are compared at matched achieved ARL?
Can the authors demonstrate a configuration where the theoretical threshold provides both meaningful false-alarm control and nontrivial detection power?
3 comments
Thank you for the clarification and the additional experiments. I have two follow-up questions.
The guarantees can be directly transferred to the detector that is run by normalizing its statistic.
Does the table in part (B) use this normalized statistic, or the original statistic with a fixed threshold?
The thresholds at measured ARL reported here rest on 60 to 340 simulated false alarms per method.
Could you report each method’s measured ARL and uncertainty intervals for both ARL and EDD? Could the remaining ARL mismatch explain the smaller delay differences?
Further Response to Reviewer gmXQ
We thank the Reviewer for the follow-up questions.
Replies to questions:
Does the table in part (B) use the normalized statistic, or the original statistic with a fixed threshold?
The original statistic of Algorithm 1 with a constant threshold, set by Monte Carlo as for every baseline. We thank the Reviewer for this insightful question, which led us to also run under the protocol of part (B), with a constant Monte Carlo threshold. EDD at measured ARL :
| Statistic | Mixed normal | Laplace | Uniform | MNIST mean |
|---|---|---|---|---|
| 2.06 | 1.11 | 1.09 | 4.70 | |
| 2.02 | 1.14 | 1.17 | 5.12 |
Here the gradient steps, step size and penalty of are re-tuned on separate streams, which changes the step size only. The two statistics behave similarly: they differ by at most 0.08 sample on the synthetic data, and is 0.4 sample slower on MNIST. We therefore keep in the experiments.
Measured ARLs, uncertainty intervals, and the remaining ARL mismatch.
Each threshold is matched to a measured ARL of on its method's null streams, so the six measured ARLs are equal by construction and their uncertainty is the sampling error of the calibration. With 500 null streams and 1,000 delay runs per method on the synthetic data (MNIST: 200 and 100 per digit), each threshold now rests on 310 to 336 false alarms (MNIST: 77 to 80). The table gives 95% bootstrap intervals. The threshold is re-matched on each resample, so the EDD intervals include the calibration error.
| WONDER | RFF-MMD | OKCUSUM | NEWMA | ScanB | NN-CUSUM | |
|---|---|---|---|---|---|---|
| Synthetic: ARL () | 1.00 [0.90, 1.12] | 1.00 [0.89, 1.13] | 1.00 [0.89, 1.12] | 1.00 [0.90, 1.12] | 1.00 [0.90, 1.13] | 1.00 [0.91, 1.10] |
| EDD, mixed normal | 2.06 [1.98, 2.15] | 3.29 [3.12, 3.49] | 8.41 [8.18, 8.68] | 8.99 [8.78, 9.21] | 17.4 [17.2, 17.7] | 42.7 [41.3, 43.8] |
| EDD, Laplace | 1.11 [1.09, 1.14] | 1.15 [1.13, 1.18] | 3.93 [3.90, 3.97] | 4.16 [4.09, 4.22] | 10.3 [10.1, 10.4] | 16.6 [16.0, 17.0] |
| EDD, Uniform | 1.09 [1.08, 1.11] | 1.17 [1.14, 1.21] | 3.96 [3.94, 3.98] | 4.73 [4.66, 4.80] | 10.6 [10.5, 10.8] | 21.4 [20.6, 22.1] |
| MNIST: ARL () | 1.01 [0.83, 1.27] | 1.01 [0.81, 1.26] | 1.01 [0.82, 1.29] | 1.00 [0.81, 1.26] | 1.01 [0.81, 1.30] | 1.02 [0.82, 1.30] |
| EDD, MNIST mean | 4.70 [4.56, 4.82] | 4.98 [4.85, 5.13] | 6.53 [6.31, 6.65] | 8.24 [8.08, 8.37] | 14.3 [14.0, 14.5] | 14.0 [13.5, 15.0] |
The remaining mismatch does not explain the differences with RFF-MMD, the closest baseline. The 95% intervals of the EDD difference (RFF-MMD minus WONDER), which also include the calibration error, exclude zero.
Response to Reviewer gmXQ
We thank the Reviewer for reading our work and for the constructive comments. Shared concerns are answered in parts (A) and (B) of the general response.
Replies to weaknesses:
The guarantees do not cover the implemented detector, and the theoretical thresholds are uninformative about detection.
As the Reviewer notes, false-alarm bounds transfer once the trained critic satisfies the Lipschitz and layer norm constraints, and detection requires control of the optimization gap. Part (A) makes both statements exact: a normalization places the trained critic in FNN by construction, and Theorem 4.11 then holds with WNN(P,Q) replaced by the separation st that the critic achieves.
The Reviewer's computation is correct. At γ=103 and with ∏ℓMℓ=1, the threshold of Theorem 4.9 equals 17.0,B for (n,m)=(32,2) and 9.5,B for (256,5) (16.0,B for the re-tuned (64,2)), whereas a 1-Lipschitz statistic cannot exceed 2B. With such small test windows the theorems say nothing about detection, and the experiments rely on Monte Carlo thresholds only: the analysis provides the scaling of the threshold in n, m, the depth and γ, not a value that leads to an immediate empirical threshold.
The comparisons are not made at matched actual ARL.
The thresholds follow the standard protocol (part (B)), with the rule and code of [R1], which guarantees E∞[T]≥γ/2 for every method when the quantile is exact but, as the Reviewer notes, not equal achieved ARLs; we thus also measured it from first-alarm times. The concern about the tail is justified: with that code's default (100 null streams of 1,023 samples), the quantile at nominal 105 rests on about one exceedance, and the measured ARL of that threshold for WONDER is about 3.6×104 on the synthetic streams and 1.0×104 on MNIST (where the statistic also drifts upward). The thresholds at measured ARL 104 reported here rest on 60 to 340 simulated false alarms per method.
Replies to questions:
Which guarantees apply to the implemented statistic, and how are its assumptions ensured?
Theorems 4.9–4.10, for the normalized statistic g~t of part (A) or, equivalently, for the statistic g^t of Algorithm 1 with its threshold multiplied by the current weight norms. The Lipschitz and layer-norm assumptions are then ensured by the normalization itself: the rescaled weights have unit Frobenius norm, hence spectral norm at most 1. The delay bound holds only with WNN(P,Q) replaced by the achieved separation st, which cannot be verified a priori.
Does WONDER keep its advantage at matched measured ARL?
At measured ARL 104, reported as an example (table in part (B)), it does on the synthetic streams and, on average over the digits, on MNIST. On the synthetic streams, the re-tuned WONDER has a smaller detection delay than every baseline (2.07 / 1.12 / 1.10 against 3.29 / 1.15 / 1.17 for RFF-MMD, the best of them).
On MNIST, at matched measured ARL 104, the EDD per digit is:
WONDER is faster than RFF-MMD on digits 2, 3, 4, 5, 6, 8 and 9, and slower on digits 1 and 7. The four other baselines are slower than WONDER on every digit.
Is there a configuration where the theoretical threshold controls false alarms and still detects changes?
Yes, but only for large windows and large changes. For the normalized function class, Theorems 4.9 and 4.11 give E∞[T]≥γ together with detection within m post-change samples with probability at least 1−β as soon as
BWNN(P,Q) ≥ (n1+m1)(2Cd+32ln(6γ)+2ln(1/β)).
For d=2, γ=103 and β=0.05, the last factor equals 25.6. For this class WNN(P,Q)≤B, with equality for two point masses at antipodal points of norm B. Even this largest change requires n=m≥2,626, and a change with WNN(P,Q)=B/2 requires n=m≥10,504. These large windows reflect that the results are distribution-free: they hold for every pair of distributions on the ball of radius B with the given WNN(P,Q).
The constants can be reduced: Proposition 4.4 bounds the population Rademacher complexity directly, so Corollary 4.6 follows from Lemma A.2 alone. The threshold of Theorem 4.9 then becomes Cn,mCd+B2(1/n+1/m)ln(2γ), and the requirements above drop to n=m≥918 and 3,672.
[R1] Kalinke and Gavioli-Akilagun, NeurIPS, 2025.