- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies the benchmarking setting on OpenSet GAD. It reveals that existing works use an oracle rule instead of a validation rule that practitioners can apply. Moreover, it proposes OUTPOST, a heavily engineered method whose performance is tied to baselines.
Critical strengths
-
The paper investigates an interesting research problem: the reliability and robustness of benchmarking in open-set graph anomaly detection. From a real-world deployment perspective, this is a useful yet critical problem to research.
-
This paper identifies the gap between research and deployed detectors - the oracle rule.
Critical weaknesses
-
The scope of this work is overly narrow, limiting its potential impact and relevance. There are only two representative studies in this field for now. Conducting an empirical analysis of such a limited body of research is unlikely to generate broad interest within the research community.
-
The author's opinion on the second convention is misleading and irresponsible. The authors claim in the Introduction that "Baseline numbers are copied from earlier papers ..." However, this claim does not appear to be supported by the existing open-set GAD literature. In particular, a comparison of the baseline results reported in NSReg and DEMO reveals differences in the reported values, as shown below.
| Paper | Method | Photo | Computers | CS |
|---|---|---|---|---|
| NSReg | TAM | 0.621±0.004 | 0.435±0.002 | 0.638±0.009 |
| NSReg | 0.836±0.031 | 0.752±0.019 | 0.953±0.008 | |
| DEMO | TAM | 0.6045±0.015 | 0.4432±0.014 | 0.6398±0.011 |
| NSReg | 0.8360±0.012 | 0.7403±0.021 | 0.9032±0.035 |
These discrepancies indicate that DEMO did not simply copy the published baseline results from NSReg. Despite being an easy-to-spot issue, this incorrect statement still appears in the paper and lies at the center of the motivation, making me doubt the quality and technical solidity of the paper.
-
OUTPOST is badly designed. a) It is not well-motivated and introduced properly. The authors rigidly combine this work with other experimental analyses. b) It is just a stacking of engineered components, rather than a well-designed innovation. There is no insight or novelty behind these modules. c) Despite claiming to be a competitive detector, as shown in Table 1, it performs worse than NSReg on 5 of 7 datasets when evaluated under the validation rule favored by the authors. d) It cannot even address the problem highlighted by the authors. As the results in Table 1 show, removing the oracle rule causes OUTPOST's performance to drop significantly.
-
This paper is poorly written and organized. From an organizational perspective, despite being submitted to datasets and benchmarks, the method OUTPOST is still a major part of the contribution, which confuses people about what it aims to address.
Questions for authors
- Why do you include the method OUTPOST in your paper, despite it performing worse than NSReg on the majority of datasets?
3 comments
Response to Reviewer ppLt (2/2)
W1. Scope
"The scope of this work is overly narrow ... There are only two representative studies in this field for now."
The field is small, which is one reason to examine its conventions before later work adopts them; DEMO alone compares against 15 methods. Several findings also apply beyond the two papers. Validation saturation can arise in any benchmark built from relabeled classes with a small seen-class validation set, and the tie band below seed noise and the ablation gains collected only by the oracle rule concern semi-supervised GAD in general. The reporting checklist in Appendix L applies to any detector, not only the three studied here.
W3(a, b). Motivation and design of OUTPOST
"It is not well-motivated ... just a stacking of engineered components, rather than a well-designed innovation."
We built OUTPOST for the audit, and the revision states this role (contribution (iii) and Section 4). Measuring the contribution of a component requires a method in which every component can be removed, and a comparison on all eight graphs requires a method that runs on ogbn-mag, where neither baseline does. OUTPOST is built on established techniques, and we claim the findings it enables, not the components themselves. For example, pseudo-labeling carries the gains on the semi-synthetic graphs and gives none on real fraud, and the atlas gate has no measurable effect, which we report.
W3(c). Worse than NSReg under the validation rule
"... it performs worse than NSReg on 5 of 7 datasets when evaluated under the validation rule favored by the authors."
This is correct and is stated in the abstract. OUTPOST still has to be competitive for its ablations to be informative. Under the validation rule, OUTPOST is significantly ahead of NSReg on CS and Yelp, behind on Amazon, T-Finance, and ogbn-arxiv, and not significantly different on Photo and Computers (Table 10); in AUC-PR it leads on Computers, CS, and Yelp (Table 8). Which detector is better on most graphs therefore depends on the rule and the metric. We do not claim that OUTPOST is the best detector.
W3(d). Drop without the oracle rule
"... removing the oracle rule causes OUTPOST's performance to drop significantly."
The baselines show the same drop. On Photo, the oracle bonus is 0.114 for DEMO, 0.083 for NSReg, and 0.079 for OUTPOST; on Computers, OUTPOST's is the largest (0.080, against 0.051 and 0.052). Across methods it ranges from 0.041 to 0.114 on the semi-synthetic graphs and from 0.002 to 0.014 on the real ones, so it depends on the benchmark more than on the method (general response, points 3 and 4).
W4. Organization
"... the method OUTPOST is still a major part of the contribution, which confuses people about what it aims to address."
We agree. In the revision, contribution (iii) states the finding first and describes OUTPOST by its role in the ablations, Section 4 opens with this role and is shorter (details moved to Appendix B), and the SimSample analysis is in Appendix H.
Q1. Why include OUTPOST?
"Why do you include the method OUTPOST in your paper, despite it performing worse than NSReg on the majority of datasets?"
We use OUTPOST for the ablations, and it is the only one of the three methods that runs on all eight graphs. It is also competitive: it leads on CS and Yelp under all three rules, and under the validation rule it is significantly ahead of NSReg on both (p = 0.027 each).
Thank you to the authors for their responses. The authors' responses have addressed my concerns about ethical review. Despite that, my concerns about the quality of the work, the narrowed scope, and marginal innovation remain. I therefore keep my score unchanged.
Response to Reviewer ppLt (1/2): W2 and ethics concern
We thank the reviewer for checking our description of prior work against the sources. This comment addresses W2 and the ethics concern; the remaining points follow in a second comment.
W2 and the ethics concern
The reviewer is correct that "copied" does not describe the two open-set papers. NSReg runs its own baselines, and DEMO trains its baselines on its own split (its Appendix F.1), which accounts for the differences in the reviewer's table. We have corrected the abstract, the introduction, and Table 5:
The point we intended is in the second half of the original sentence: values produced under different splits, budgets, and seed counts cannot be paired. The reviewer's table illustrates it. NSReg on CS is 0.953 in the NSReg paper and 0.9032 in DEMO, a gap of 0.050 that exceeds the 0.042 by which DEMO reports leading NSReg on that graph.
The first convention. We verified it in the same way. In DEMO's released trainer.py, the function eval keeps the maximum test value of each metric over all epochs and returns it as the result. NSReg's released runner (runners/train_runner.py) evaluates the test set during training and defines no separate validation set. Section 3 now states this.
Effect on the results. The correction changes no result. Every paired comparison and test in the paper uses our own re-runs on identical splits and seeds. Transcribed values appear only in a gray reference column, in Appendix D, and in one comparison on ogbn-mag, where neither DEMO nor NSReg can run; each is marked as unpaired.
We hope the correction resolves the concern, and we would be grateful if the reviewer would reconsider the ethics flag in light of it.