- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes OneShot, a weighted ensemble approach for time-series anomaly detection. Given one confirmed anomaly event, OneShot evaluates each detector on that event together with sampled, presumed-normal points, then converts these performance estimates into softmax ensemble weights.
Critical strengths
The paper studies a practically motivated setting in which a deployed anomaly detection system receives occasional operator feedback.
Critical weaknesses
- Generalization across anomaly types remains unclear. The method assumes that detector performance on one confirmed event is informative about other events in the time series. A more fine-grained analysis should distinguish hidden anomalies that resemble the confirmed event from those with different characteristics. Transfer across related time series would also help establish the usefulness of a confirmed event beyond the series on which it was observed.
- The concern regarding practicability of the proposed approach. For example, OneShot is not applicable on almost half of the benchmark datasets which contains only one labeled anomaly.
Questions for authors
- Does baseline ensemble share the same type of normalization? Does score normalization use the entire time series, including future/hidden points?
- Could you reconcile the exact-label results in Tables 2 and 9? “Best on labeled window” obtains 0.672 VUS-PR in Table 2 but 0.568 in Table 9, while OneShot obtains 0.710 and 0.699, respectively. Table 9 should also include ensemble baselines so that its robustness gains can be attributed specifically to the proposed weighting strategy.
- Could you explain why confirming the earliest events sometimes performs worse than selecting events uniformly at random, given that the former better reflects practical deployment?
4 comments
I thank the authors for the detailed response! The proposed approach relies on the transferability of a confirmed anomaly event to hidden events. Therefore, further analysis of transferability is needed for Table R2. Also, why do the numbers in Table R5 differ from one another?
We sincerely thank the reviewer for engaging in the discussion and for the thoughtful follow-up questions!
Both questions helped us strengthen the paper, and we have posted a second updated PDF (changes in brown).
Changes from the first updated PDF:
- [Appendix P.5] New analysis of when a confirmation transfers, with Table 41 (Q1)
- [Table 9] New column "Correct" (share of the confirmed window that is truly anomalous) and grouped rows (Q2).
- Thanks to the reviewer's careful reading, the "Shifted by half" row now matches Table R5 (Top-3 average 0.665, OneShot 0.662)
- [Section 5.4] The discussion of Table 9 follows the new column
Q1: When does a confirmation transfer to a related series?
We thank the reviewer for this insightful suggestion, and we agree that the transfer relies on the confirmed event being informative for the target. Since OneShot transfers which detectors to trust, the natural measure is whether the two series rank the detectors alike. For the trials of Table R2, we computed the Spearman correlation between the per-detector accuracies of the source and the target series, and split the trials into three equal groups by :
Table R8 (Appendix P.5, Table 41). Transfer by detector agreement
- Trials of Table R2, univariate, VUS-PR on the whole target series
- Agreement: Spearman between source and target detector accuracies
| Agreement | Mean | Score average | Best on window | OneShot |
|---|---|---|---|---|
| High | 0.82 | 0.666 | 0.717 | 0.788 |
| Mid | 0.53 | 0.574 | 0.579 | 0.677 |
| Low | -0.01 | 0.529 | 0.345 | 0.468 |
- The gain grows with the agreement: +18% over the score average in both the high and mid groups (), and the per-trial gain correlates with (Spearman 0.40). This confirms that the transfer works through the mechanism of OneShot, i.e., the confirmation identifies the detectors that also suit the target.
- Series of the same dataset mostly agree: in 84% of the trials (median 0.54), which is why the transfer helps on average in Table R2.
- When the two series rank the detectors independently (), the confirmation carries little information about the target, and OneShot stays well above the window winner (0.468 vs. 0.345, ), since its soft weight does not bet on a single detector.
- The multivariate collection shows the same trend (high group: 0.409 vs. 0.282 for the score average).
In deployment, this suggests confirming the event on a series of the same source (e.g., the same machine or sensor type), which is how related series arise in practice.
Q2: Why do the numbers in Table R5 differ across rows?
We thank the reviewer for the careful check.
The rows differ in which series can be scored, not in the hidden part:
- The score average uses no label, so it changes only with the set of series. On the 146 series of the Exact, Shifted, and Normal rows, it is identical (0.603). The shorter windows of the Half and Quarter rows cannot be scored on some series, which leaves subsets of 133 and 114 series.
- Within each row, every method uses the same series, confirmed window, and hidden part, so each row is a paired comparison.
- The Exact row (0.699) differs from Table 2 (0.710) because the corruption study draws its own trials (confirmed events and normal points).
To make the rows easier to read, we added the column "Correct" to Table 9:
Table R5 (revised Table 9). Imperfect labels
- Series and pool of Table 2, VUS-PR
| Confirmed event | Correct | Series | Score average | Top-3 average | Best on window | OneShot |
|---|---|---|---|---|---|---|
| Window within the event | ||||||
| Exact | 100% | 146 | 0.603 | 0.684 | 0.621 | 0.699 |
| Half of it | 100% | 133 | 0.614 | 0.684 | 0.609 | 0.696 |
| Quarter of it | 100% | 114 | 0.649 | 0.679 | 0.611 | 0.698 |
| Window (partly) off the event | ||||||
| Shifted by half | 53% | 146 | 0.603 | 0.665 | 0.587 | 0.662 |
| Shifted by its length | 7% | 146 | 0.603 | 0.499 | 0.420 | 0.503 |
| Normal window | 0% | 146 | 0.603 | 0.337 | 0.237 | 0.339 |
- When the confirmation is CORRECT, OneShot is the best of all methods, including the Top-3 average (), even when only a quarter of the event is confirmed.
- When the window is (partly) OFF the event, the drop of OneShot is itself evidence that it truly uses the confirmation: its accuracy follows the share of correct points (0.699, 0.662, 0.503, 0.339 at 100%, 53%, 7%, 0%), which shows that its gain comes from the confirmed event rather than from averaging the pool. Even so, it stays tied with the best ensemble at 53% (), leads it on both larger pools (Table R6), and stays above the window winner in every row.
We hope that these results address the remaining questions and can be reflected in the final assessment!
If any concern remains unresolved, please let us know and we will gladly address it.
Dear Reviewer 9Pwc,
As the discussion period is coming to a close in a few hours, we would like to thank you again for your time and consideration. We would be happy to address any remaining questions or comments before the discussion closes.
Best regards,
The Authors
Please let me know if I can provide any further clarification!
If my responses have sufficiently addressed your concerns, I would be grateful if you would consider revisiting your score.