- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes a protocol for measuring how much a linearized GCN surrogate attack biases robustness comparisons between graph transformers and GCNs. It introduces a paired, reference-relative estimand, the surrogate’s shortfall against a strong reference attack on the victim minus the same shortfall on a GCN baseline. It also introduces a shadow-model diagnostic with at most 50 score queries per target. Experimental results show that the linearized surrogate can overstate transformer robustness and even invert model rankings, and that a Jaccard edge filter mainly reduces transferability rather than vulnerability.
Critical strengths
- The paper raises an important point: robustness comparisons depend on the attacker used. It shows that weak surrogates can reverse which model looks more robust.
- The statistical evaluation is careful, using paired seeds on common targets, confidence intervals, and multiplicity correction, and the null and negative results are reported honestly.
- The defence analysis is insightful, showing that a filter-aware attacker without victim weights or queries recovers most of the Jaccard filter’s apparent protection.
Critical weaknesses
Weaknesses and Questions
- What is the key novelty of this work compared with existing adaptive attacks, and what practical benefit does it provides?
- Why frame the study around graph transformers when similar findings appear for GAT and GIN?
- Why use a weak white-box reference when stronger adaptive attacks are available?
- How realistic is the threat model when the attacker uses ground-truth labels and victim training labels?
- Why not evaluate recent defenses such as GNNGuard and RUNG, and stronger attacks such as EvA?
- Has the protocol been evaluated on larger datasets such as OGB-Products?
Questions for authors
See Weaknesses and Questions.