acceptodds
Under review as a conference paper at ICLR 2027

Robust to the Wrong Attacker: Separating Model from Attacker in Graph Transformer Robustness

Abstract

Graph-transformer robustness comparisons can use edge edits chosen by a linearized GCN surrogate. Poor surrogate fit can understate vulnerability and reverse model rankings. We introduce a paired, reference-relative estimand: the surrogate's shortfall against a stated strong attack on the victim minus the corresponding shortfall on a named GCN baseline, evaluated on common targets and paired over ten seeds. Our contribution is a controlled evaluation protocol and paired estimand for measuring surrogate-induced bias; the attack procedure is deliberately conventional. On Cora and Citeseer, linearized-surrogate GCN-to-transformer gaps are 16 to 24 points versus 4 to 13 under full-access greedy search. The excess is positive in all eight victim-graph cells against a plain GCN and separated from zero in six; against an accuracy-matched GCN it is positive in six and separated above zero in five. In three of these sixteen comparisons the surrogate inverts the ranking. On Pubmed, at nearly equal clean accuracy, the excess is 9.9 and 9.4 points against the two GCNs, both separated from zero. An exhaustive per-step candidate reference yields positive estimates separated from zero for the GPS-style transformer against both baselines on both graphs. No excess is detected for the official GPSConv layer on Cora or for an SGFormer-style victim, so the claim concerns surrogate fit, not attention. With an attacker-owned shadow and at most 50 score queries per target, our diagnostic falls no more than 5 points below greedy white-box search across 16 complete-record conditions and sometimes exceeds it; in the official pipeline of Foth et al. (2025) it flips 61 percent of targets against 63 for their adaptive white-box attack. The protocol also separates defences: adversarial training lowers attack success, whereas a Jaccard edge filter mainly reduces transferability. Of its apparent 32.3-point protection against blind transfer on Cora, filter-aware transfer recovers 24.7 without victim weights or victim queries.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
graph transformers, adversarial robustness, structure attacks, black-box attacks, surrogate fidelity, graph neural networks, robustness evaluation, node classification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xXw3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a protocol for measuring how much a linearized GCN surrogate attack biases robustness comparisons between graph transformers and GCNs. It introduces a paired, reference-relative estimand, the surrogate’s shortfall against a strong reference attack on the victim minus the same shortfall on a GCN baseline. It also introduces a shadow-model diagnostic with at most 50 score queries per target. Experimental results show that the linearized surrogate can overstate transformer robustness and even invert model rankings, and that a Jaccard edge filter mainly reduces transferability rather than vulnerability.

Critical strengths
  1. The paper raises an important point: robustness comparisons depend on the attacker used. It shows that weak surrogates can reverse which model looks more robust.
  2. The statistical evaluation is careful, using paired seeds on common targets, confidence intervals, and multiplicity correction, and the null and negative results are reported honestly.
  3. The defence analysis is insightful, showing that a filter-aware attacker without victim weights or queries recovers most of the Jaccard filter’s apparent protection.
Critical weaknesses

Weaknesses and Questions

  1. What is the key novelty of this work compared with existing adaptive attacks, and what practical benefit does it provides?
  2. Why frame the study around graph transformers when similar findings appear for GAT and GIN?
  3. Why use a weak white-box reference when stronger adaptive attacks are available?
  4. How realistic is the threat model when the attacker uses ground-truth labels and victim training labels?
  5. Why not evaluate recent defenses such as GNNGuard and RUNG, and stronger attacks such as EvA?
  6. Has the protocol been evaluated on larger datasets such as OGB-Products?
Questions for authors

See Weaknesses and Questions.

Reviewer_c6ovOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on the robustness of the graph attack problem. It finds that the attack methods really matter when comparing graph transformers and GCNs. Previous linearized GCN surrogates are not effective for graph transformers. The evaluation is also affected by the quality of the attacker. Based on this finding, the paper proposes a new controlled evaluation protocol to measure surrogate-induced bias. The experiments are conducted mainly on Cora and CiteSeer with some additional larger datasets.

Critical strengths

The paper defines a relative measure to reflect the robustness of graph models compared with GCNs. It conducts many experiments to analyze how different attack methods affect performance.

Critical weaknesses
  • The graph transformers implemented by the authors get much lower performance than GCNs (7-12 points), which limits the reliability of the paper.
  • The novelty of the paper is limited. The paper mainly uses existing methods to analyze the robustness of some graph methods. The findings are also not promising.
Questions for authors

The paper focuses on fair robustness evaluation of graph attacks. How do these robustness findings help further graph research, such as new architectures of GNNs or graph transformers?

Reviewer_D24rOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks how much of a graph transformer's apparent robustness under attacks generated by a linearized GCN comes from poor attack transfer. On shared targets that the compared models classify correctly, it measures each model's attack-success shortfall relative to a stronger reference, then subtracts the GCN baseline's shortfall to obtain a paired statistic, Δ. For some model–dataset combinations, linearized attacks exaggerate robustness differences and can reverse model rankings. The paper also evaluates trained-shadow and limited-query diagnostics. Its Jaccard-filter experiments show that blocking transferred edits can give a substantially different picture from evaluation against an attacker that accounts for the defence.

Critical strengths

1.A valuable part of this paper is that it does not simply compare attack-success rates measured on different target sets. It pairs the models on shared targets and includes accuracy matching, retuning, and an official GPS-layer control. Target selection itself changes the reported gap, so these controls matter. The positive excess of roughly 9–10 percentage points on Pubmed, where clean accuracies are close, also shows that the observation is not confined to the initially weaker transformer models. The supplied target-level flags allow independent reaggregation of the main comparisons, and an up-to-five-edit reanalysis retains all three ranking reversals. 2.The strongest Jaccard result is that knowledge of the filter and its threshold alone raises attack success on Cora from 26.0% to 50.7%, without victim weights or victim queries. This directly shows that much of the apparent protection comes from the attacker failing to account for preprocessing. Adversarial training behaves differently on the shared targets, so this experiment does distinguish two forms of defensive effect.

Critical weaknesses
  1. flip() changes both A[t,j] and A[j,t], but grad_scores() differentiates an unconstrained dense adjacency and ranks edits using only (1−2A[t,j])·g[t,j]. For this implementation, the first-order term along the actual undirected edit is generally (1−2A[t,j])·(g[t,j]+g[j,t])`. The forward computation does not tie these two input variables, so a symmetric input does not make the scores equivalent. The function feeds both greedy white-box search and shadow proposals, potentially affecting the reference values in Δ and the query-to-white-box comparisons. I would like a symmetric-gradient or finite-difference validation and a rerun on the same targets. 2.The supplementary records resolve the concern that the ranking reversals might disappear simply by retaining success at earlier budgets: all three survive an up-to-five-edit evaluation. Changing attack aggregation does materially change the numbers, however. In an independent reanalysis that unions the three linearized attacks' successes per target within five edits, positive Δ intervals excluding zero fall from 11 to 8 out of 16 comparisons, with 7 surviving BH correction; ranking reversals with both gap intervals excluding zero fall from 3 to 2. For example, the original Cora/GCN excess falls from 14.2 to 7.7 percentage points, and the accuracy-matched contrast from 7.7 to 2.7. The mismatch remains, but the three-reversal count and headline magnitudes should be explicitly tied to the mean-attack convention.
Questions for authors

1.Why does the greedy score use only g[t] when each undirected edit changes two adjacency entries? Please validate the score with symmetric parameterization or finite differences and report paired Δ and the hybrid-to-reference gap using the symmetric gradient. Both the victim and GCN arms need updating. 2.Section 3 and Algorithm 1 specify five shadow and five random proposals per step. The code instead takes M//2 + len(done) + 1 shadow proposals before appending random candidates and truncating to M. With a sufficiently large valid pool, this retains six shadow proposals at the first step, and the mixture can shift further later. Please identify the rule used in the reported runs and check whether fixed 5+5 sampling materially changes the results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.