- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces an unsupervised time-series anomaly detector built entirely on a score function learned with denoising score matching. Each subsequence is scored with a window-averaged Hyvärinen statistic J, which needs neither a partition function nor an energy. The paper also gives a theoretical analysis of when this statistic detects anomalies and a rule for choosing the smoothing length. The authors claim that this score-only approach performs on par with the strongest baseline, PaAno, on the univariate benchmark (0.588 vs 0.584 VUS-PR; no significant difference), at under a third of its runtime, while it ranks in the middle on the multivariate benchmark.
Critical strengths
-
The paper gives an exact formula for how much higher the statistic is on anomalies than on normal data. It shows that detection works when anomalies differ strongly from normal data, and fails when anomalies are more concentrated than normal data. It also shows a real example consistent with this, a low-variance flat-line anomaly on which the statistic drops instead of rising.
-
The error bound in Proposition 1 does not assume that the learned score is the gradient of an energy. Hence, the method works with any trained score network, including diffusion-style models where no likelihood is available. This makes it an easy way to reuse an existing score model as an anomaly detector, at the cost of one score evaluation and a divergence estimate per window.
-
The method works well even though the network is a simple MLP trained with standard denoising score matching. On the univariate benchmark, even the energy-parameterized variant (0.561 VUS-PR) comes close to PaAno (0.584), which is considerably more elaborate. It is also about 0.07 or more ahead of all the other neural baselines the authors ran, and it stays ahead when the paper's audit applies the method's moving-average smoothing to the scores of five of them.
Critical weaknesses
-
The paper says all hyperparameters are chosen on the tuning split and the architecture is fixed to the energy-parameterized MLP, but the main claims (abstract, contributions, and the comparison with PaAno) use the direct score network instead. In the tuning split, the energy network outperforms the direct network (0.514 vs 0.504 VUS-P; Tables 4 and 18). The direct network is better only on the evaluation set (0.588 vs 0.561), which should be used to report results, not to choose between models. This choice matters because, against PaAno, the direct network is ahead by 0.004, while the model selected on the tuning split is behind by 0.023.
-
The method performs among the best on the univariate benchmark, but the paper does not show that the Hyvärinen statistic J itself is the reason. Most of J's own gain comes from the moving-average smoothing, which takes the direct network from 0.506 without smoothing to 0.588 with it, and the paper's audit shows the same smoothing also helps the baselines. In addition, when the network provides energy, that energy performs at least as well as J (0.577 vs 0.561 for the energy network). The contribution therefore rests on the case where no energy is available.
-
PaAno is reported as averaged over three seeds, but it was effectively run with only one. Its configuration fixes the seed to 2021 (HP_list.py, line 143), PaAno applies it when training starts (PaAno_PAI.py, line 616), and the experiment runner never passes the run seed, so the three runs are identical. This can be seen in the paper itself, where PaAno is reported as 0.584 ± 0.000 (Table 1) and some of its multivariate scores have a standard deviation of exactly zero (Table 8). When I ran PaAno on one series with four different seeds, its VUS-PR ranged from 0.574 to 0.620. The statistical tests are still valid, but the claimed lead over PaAno (0.588 vs 0.584) may depend on this single run.
Questions for authors
Q1: Was the direct score network chosen as the main model before you saw the evaluation results, and if so, on what basis?
Q1.1: Following the tuning split as described would select the energy network, yielding a main result of 0.561, which is below PaAno (0.584) and the same network's energy readout. How would this change the claims of the paper?
Q2: Is there a setting where J clearly beats an energy readout, or where the statistic itself, and not the smoothing, explains the gain over the baselines?
Q2.1: For example, how do pointwise J and pointwise energy compare with each other and with the smoothed baselines?
Q3: In Section 6, the paper reports the method as "TOST-equivalent" to PaAno within ±0.05. This test treats the 350 series as independent, but they come from only 23 datasets, and your own source-clustered interval ([−0.109, +0.092]) is wider than the margin. Will you remove the equivalence claim or redo the test to account for the source datasets?
Q4: Can you confirm that PaAno's fixed seed overrides the experiment seed in all three runs?
Suggestion: Please rerun PaAno with different seeds, report its mean and standard deviation, and check whether the direct network is still ahead.
Q5 premise: The paper states that the runtimes are directly comparable because each job used a single GPU worker. However, the launch scripts run 16 to 20 jobs in parallel on 8 GPUs, so several models seem to have shared the same GPU at the same time, and not always in the same way (16 parallel jobs for the direct network, 20 for PaAno). Timing both methods one at a time on a single GPU for 40 time series from the benchmark, I found a 2.0x speed-up over PaAno, instead of the reported 3.4x, although my hardware is different, which may also explain part of the gap.
Q5.1: Were the timing runs done with one job per GPU?
Q5.2: If not, can you measure the runtimes again under controlled conditions?
2 comments
Thank you for the detailed response. I checked the revised PDF. The PaAno re-seeding (Table 31), the controlled timing (Appendix I), the withdrawn TOST, the protocol-ordered Table 1, and the ROC disclosure in the abstract address the protocol issues I raised, and I appreciate the transparency. However, these corrections remove the paper’s main empirical claim without replacing it, so my overall assessment is unchanged.
The headline result is now a null result. The original submission claimed parity with PaAno and a small lead. The revision correctly claims “neither a lead nor a tie,” and PaAno is significantly ahead under VUS-ROC and AUC-ROC. The rebuttal’s statement that a score-only model is “enough to reach PaAno-level VUS-PR” contradicts this. With a source-clustered CI of [−0.110, +0.092], the benchmark cannot distinguish methods $\sim$0.1 apart, so “no detected difference” is weak evidence of anything. In addition, the network, the readout, and the decision to sweep were all influenced by evaluation results. The protocol-selected model is the energy network, for which PaAno is nominally ahead, with a runtime advantage of about , not (right?).
The proposed statistic is not what drives performance. The divergence term, which is the object of Corollary 1 and what distinguishes J from the score norm, adds +0.023 on TSB-AD-U. This is not significant under clustering, and 84% of it comes from one source (Exathlon). Without it, the method is the MSMA score norm at a single scale plus a moving average, as the revision itself acknowledges. Before averaging, J is not distinguishable from the smoothed CNN, DTE, and LSTM-AD. After equal averaging, its lead survives clustering only over TranAD. When an energy is available, the energy readout is at least as good. The theoretical contribution is a one-step corollary of a classical identity, and its predicted failure mode is illustrated on a single series.
Reply to Reviewer M3mv
Thank you. Your review identified real protocol problems, including a seed bug in the benchmark's own PaAno configuration. We fixed all of them, and the main result holds under the stricter protocol you asked for.
W1 / Q1. Model choice. The direct network was chosen after evaluation results were available, because a model with no energy is the case the paper is about. The revision says so and reports results in protocol order (Table 1):
Q1.1. How the claims change. The claim survives the strict protocol: whichever row one takes, no difference from PaAno is detected under VUS-PR, and that includes the tuning-selected model. We now also report all four metrics (Table 34). PaAno is significantly ahead under VUS-ROC and AUC-ROC, and the abstract says so.
W2 / Q2. J vs smoothing. You were right that averaging carries most of the gain, and the paper now says so. We tested every comparison you asked about:
This is why the paper's claim is about score-only models, not about J beating an energy. When a model has an energy, use it. When it has only a score, as diffusion models do, this paper shows it is enough to reach PaAno-level VUS-PR.
Q2.1. Smoothed baselines. Equal averaging lifts all five audited baselines by 0.017–0.233. Every gain is significant per series and under source clustering, and the ranks change (Table 4). That is a benchmark finding of its own.
W3 / Q4. PaAno seeds. Confirmed. TSB-AD's
HP_list.pyfixesseed = 2021andcalib_seed = bank_seed = 42, overriding the run seed. We re-ran with all three tied to the run seed:Q3. TOST. Withdrawn. It passes per series (p = 0.007) but not clustered by source (p = 0.19).
Q5. Timing. The earlier numbers came from shared-GPU sweeps, so "directly comparable" is removed. We re-measured one job at a time on one NVIDIA T4 over 40 random series: PaAno takes 3.3× as long on average, 1.9×–5.0× per series (Appendix I). Your 2.0× is inside that range.
We also checked tuning/evaluation overlap. Thirteen tuning series repeat an evaluation channel, and σ selection is unchanged without them.
Every protocol issue you raised is now fixed and documented, and the central result holds under the stricter analysis. We would be grateful if you would reconsider your score on the revised paper.