- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates why replacing the graph in a time-series anomaly detector sometimes leaves detection performance unchanged. It proposes a four-layer diagnostic procedure covering graph execution, predictive information, model utilization, and anomaly scoring. The authors provide theoretical analysis for directed cycles, test the procedure on planted failure cases, and apply it to several existing detectors and real datasets.
Critical strengths
-
The paper addresses an important methodological problem. An unchanged anomaly score does not necessarily mean that graph information is useless. Separating graph execution, information availability, model utilization, and scoring offers a more informative alternative to conventional ablation analysis.
-
The study goes beyond conceptual discussion. The diagnostic rules recover 101 of 120 planted labels, identify graph interfaces that ignore supplied replacements, and lead to concrete changes in GDN. The authors also report unsuccessful interventions rather than presenting every diagnosis as a performance improvement.
Critical weaknesses
-
The diagnostic rules are less reliable when multiple mechanisms influence the same outcome. All nineteen errors in the planted benchmark occur in partial-utilization cases. Since real detectors can contain several interacting bottlenecks, the accuracy of the diagnosis outside single-absorber settings remains uncertain.
-
The evidence for practical intervention benefits is mixed. The controlled-cycle experiment shows a substantial improvement, but the SMD repair mainly improves prediction without a corresponding significant improvement in anomaly detection. This limits the evidence that the proposed diagnosis consistently leads to better detectors.
Questions for authors
-
How should the diagnostic procedure interpret cases in which two or more layers simultaneously absorb a graph replacement?
-
When the identified repair improves prediction but not anomaly ranking, what criterion determines whether the diagnosis has been practically useful?
5 comments
Response to Reviewer VmGf (2/2)
W2 / Q2. When a repair improves prediction but not ranking, what criterion determines whether the diagnosis was useful?
A diagnosis is useful when it predicts, before the run, which change will move the outcome and which will not (§7). We fixed 11 such predictions in advance and count all of them; 9 held.
The case you describe: the root weight on SMD. It closes at least 91% of GDN's normal-risk gap, yet SMD's AP does not rise (0.380 against 0.394). Rerun after the repair, the diagnosis says why and where to act next: the graph is now used on only 5 of 25 objects (rule R0), 4 objects have no available cross-channel information and on 6 GDN's original sources miss it, and on injected cross-channel anomalies the official max-over-channels score absorbs most of the signal (+0.108 on the injected channels, +0.018 under the official score). This explains why improving prediction alone did not move the ranking, and it predicted, before the run, that changing the score would. Changing the score then moved the outcome on the anomalies it targets, as predicted (+0.094 [+0.022, +0.177]). SMD's own labels are mostly single-channel deviations, where a cross-channel score helps only on objects that self-history detects poorly (Spearman −0.60 with the self-history AP), so we recommend a secondary score that keeps both streams (App. I.3).
All predictions tested (11 in three studies; also listed in App. I.8).
- Cycle: a selector gradient restores detection (held: +0.417 [+0.399, +0.434], 20/20 seeds); removing only the gradient from the trainable selector loses it (held: −0.278); the attention change alone does not restore it (held: +0.001 [−0.011, +0.013]).
- SMD, injected cross-channel anomalies: a label-free contrast score lets informative sources reach the ranking (held: +0.094), widens the learned-minus-random contrast of GDN's sources (held: +0.011 to +0.048), and does so beyond the official score (held: +0.037); for repaired GDN, whose learned graph has no advantage even when only the injected channels are read (+0.002), changing the score does not create one (held: +0.001 [−0.009, +0.012]).
- SMD, GDN utilization repair: an attention-free linear source path trained jointly with GDN would lift u above 0.5 (not held: median u from −8.13 to 0.13) and let GDN's learned graph beat a random one (not held: +0.006 [−0.007, +0.025]); under the official score the contrast stays within ±0.01 (held), and without the path the learned graph does not beat the random one on the injected channels (held at the edge of its tolerance: +0.010).
What the two misses show. The repair moved the measured gap without crossing the threshold, and the outcome did not move; with the path, GDN's selector also picked weaker sources (App. I.4). A located layer tells where to act; it does not guarantee that the first repair closes it. To place the remaining absorption, we kept the sources fixed and changed only the path that reads them (App. I.8): read by a ridge path fitted on normal data, informative (Granger-selected) sources are used (median u 0.73) and beat random ones under a full-channel contrast score on fresh injections (+0.105 AP, p = 0.002), and GDN's own learned sources beat random ones on the injected channels (+0.070, secondary endpoint). Under the same score, three changes to GDN's own message path do not, so the absorption sits in GDN's message path, not in its sources. These later repairs met 2 of their 6 primary endpoints; App. I.8 lists all of them.
In short: by this criterion the diagnosis was useful on the cycle and on SMD's injected anomalies, and where a first repair fell short it still located the remaining absorption. On SMD's own labels the learned graph does not beat a random one after any repair; the abstract, §6.2, and the conclusion say so.
We thank the reviewer again for questions that sharpened the paper, and we hope the revision strengthens your confidence in it.
Gentle Follow-up Before the Discussion Closes
Dear Reviewer VmGf,
Thank you again for your constructive review and positive assessment of our work. As the author–reviewer discussion period closes soon (November 18 AoE), we gently ask whether our response and the revised PDF address your two questions—on interpreting simultaneous multi-layer absorption via per-layer verdicts, and on the pre-registered intervention criterion when a repair improves normal prediction without moving the ranking. We would be happy to clarify any remaining point before the window closes.
Best regards, The Authors
Response to Reviewer VmGf (1/2)
Thank you for the positive assessment and for two pointed questions. They led to the two additions we consider most important: per-layer verdicts when several layers absorb at once, and a criterion for when a diagnosis has been useful.
W1 / Q1. How should the procedure interpret cases where two or more layers absorb the same replacement?
Every layer now returns its own verdict, and the report keeps the full verdict vector; the first absorbing layer gives only a summary label. With two absorbers planted at once, both are flagged in 29 of 30 runs.
How the procedure reads it. The summary label is the first absorbing layer in the order execution → information → utilization → scoring; the report keeps every layer's verdict, including partial retention at the information layer, together with u and ΔAP (§3.1, Algorithm 1). The submitted rules let any AP change override the earlier verdicts (a "scoring-route precedence"). We removed this override, because it caused one of the two partial-utilization misses.
Two absorbers at once (new, App. I.1). Three cases plant two absorbers: diluted static attention plus noise bursts (utilization and scoring), history retention plus bursts (information and scoring), and a symmetrizing detector on a retaining graph (execution and information). The per-layer verdicts flag both planted layers in 29/30 runs (pre-registered criterion: at least 24). The summary label names the first planted layer in every run of the first and third cases. In the retention case the replacement keeps 26% of the gain while AP still falls; the frozen rule R2 summarizes this as "retained" and names information (30/30 overall), whereas the rule of §3.1 reports partial retention and names a later layer (20/30 overall; Table I.3). Both readings are in the verdict vector, which is why we report the vector, not a single label.
Accuracy with single absorbers. All 19 planted misses came from how the rules read the utilization measurement: an absolute threshold, and the precedence. Reading utilization relative to the available information, u = e/g < 0.5, was frozen after the first run and confirmed on fresh seeds (20/20 partial-utilization runs; R2 120/120 overall, against 101/120 for the pre-registered rules, which we keep as the reported benchmark result) and on partial-utilization mechanisms outside the benchmark whose ground truth does not use u (120/120, against 80–100/120 for the absolute threshold; App. I.1). The rule of §3.1 (fixed after these runs) applies the same relative criterion to the gain the replacement removes.
Utilization as a degree. Over a sweep of distractor strength, u falls with the replacement's effect on AP (Spearman 0.97). At seven distractors the graph is utilization-limited while the replacement still lowers AP by 0.47, so every label is reported with u and ΔAP rather than alone (Table I.4).
A real detector with several absorbers. SMD's GDN is such a case, and the revised §6.2 reports it layer by layer rather than with one label: 4 of 25 objects have no available cross-channel information, on 6 GDN's selector misses it, on the other 15 GDN does not use it (its graph path degrades normal prediction), and the score hides cross-channel deviations. Each verdict names the component to examine next.