- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose GRL-Reliability, a multi-axis benchmark for evaluating Graph Representation Learning Reliability. They evaluate twelve representative methods, spanning topology-only embeddings, supervised GNNs, self-supervised graph models, and GFMs, on twenty-five graph datasets under standardized evaluation conditions.
Critical strengths
It's a benchmark paper and it correctly cover many graph learning methods with separate evaluations of relevant reliability risks. It correctly details protocols, statistical analysis, severity sweeps and make the results easy to access The paper also add an analysis on pretraining unseen graphs which is a useful check.
Critical weaknesses
-
Performance drops can give misleading robustness rankings. For instance on BACE, GraphSAGE has the smallest scaffold gap, yet OFA achieves substantially higher scaffold-test AUC (66.6 vs. 56.9). Even if absolute scores are reported, the cross-axis analysis ranks robustness using drops. This weakens the conclusion on model selection since smaller degradation does not necessarily mean better performance under deployment shifts.
-
The overlap in pertaining limits OOD evidence: The temporal and inductive-entity experiments use graphs included in pretraining, so downstream-held-out nodes or entities are not necessarily unseen during pretraining. The paper acknowledges this, but the unseen-graph analysis does not resolve it for these two scenarios. Consequently, these results provide limited evidence of generalization to genuinely unseen future data or entities.
Questions for authors
-
Do the cross-axis conclusions hold when robustness is ranked by absolute corrupted/OOD performance rather than performance drop?
-
Can the temporal and inductive-entity experiments exclude future nodes and held-out entities from pretraining?
-
How stable are the attribution rankings under another explainer, particularly one that scores edges directly?
-
Have you considered feature missingness beyond Gaussian noise on embeddings? Ferrini et al., “Rethinking GNNs and Missing Features: Challenges, Evaluation and a Robust Solution,” (ICML 2026) studies how missingness mechanisms affect robustness evaluation. Discussing this connection would help clarify which deployment failures the current corruption tests capture and which they leave out.
5 comments
Response to RNAQ (2/2): GNNExplainer and Feature Missingness
Q3: a second explainer that scores edges directly. We added GNNExplainer, which learns an edge mask directly, under the same deletion protocol for all eight methods. Its method ranking moderately agrees with the gradient-saliency ranking (mean per-dataset Kendall τ = 0.59, from 0.29 on Elec-Computers to 0.93 on WikiCS), and the same three methods (GFT, GraphSAGE, GIT) form the top three by average rank under both explainers (Appendix G.5, referenced in §3.2.5).
Q4: feature missingness (Ferrini et al., ICML 2026). Following Ferrini et al., we added three test-time missingness mechanisms: element-wise MCAR, node-wise MCAR and value-dependent MNAR, at rates 0.1–0.9 with mean imputation, for the eight methods evaluated on every axis, with models trained on complete features (Appendix G.1.3, Table 11c).
- Dense text embeddings do not mask missingness. At 90% missingness, accuracy drops on average over four datasets by 28–40 points under both MCAR mechanisms and by 8–18 points under MNAR.
- Gaussian noise is not a proxy for missingness at high severity. Method rankings at 90% missingness agree only weakly to moderately with rankings under the strongest feature noise (mean τ = 0.16–0.50 across mechanisms), whereas rankings at 50% missingness agree more with those at the middle noise level (τ = 0.63–0.69). This extends Finding 1 (robustness is signal-specific).
- §3.2.1 points to these results. With these additions, the corruption tests cover deployment-time feature noise, edge loss and feature missingness, and leave out training-time missingness and missingness-aware models (§4).
Accuracy (%) at 90% test-time missingness, averaged over CiteSeer, DBLP, PubMed and WikiCS (Table 11c):
| Method | Element MCAR | Node MCAR | MNAR |
|---|---|---|---|
| GCN | 39.3 | 37.3 | 64.5 |
| GAT | 42.8 | 39.5 | 64.7 |
| GraphSAGE | 45.2 | 37.2 | 66.0 |
| BGRL | 33.9 | 33.4 | 54.8 |
| GraphMAE | 38.6 | 38.8 | 59.8 |
| UniGraph2 | 34.5 | 42.3 | 56.5 |
| GFT | 37.5 | 36.1 | 54.0 |
| GIT | 39.3 | 34.7 | 60.1 |
Thank you for the detailed response and the substantial additional experiments. The absolute-performance and signed-gap analyses (Table 11b), the re-pretraining study (Table 10b), the GNNExplainer comparison, and the missingness experiments address my main concerns, and I have verified that the corresponding changes are in the revised manuscript. I am raising my score accordingly.
A few minor points remain:
- OFA, which also pretrains on the shared corpus and is the leader on the arXiv temporal split and on FB15K237, is not included in the re-pretraining study. Please include it or explain why it was excluded.
- Finding 1 is still supported only by drops in the main text. A brief absolute-performance check (e.g., rank agreement between the two corruption channels in absolute terms) would make it consistent with the rest of the revision.
- The GNNExplainer results are reported only in the text; a per-method table in Appendix G.5 would be helpful.
Response to RNAQ (1/2): Absolute Rankings and Re-pretraining
We thank the reviewer for the constructive comments and the pointer to Ferrini et al. All four questions are addressed with new results in the revised manuscript (changes highlighted in blue).
W1/Q1: drop-based vs. absolute ranking. The main cross-axis patterns remain supported: Insights 1–2 under absolute performance, and Insight 3 under signed gaps (§3.3). Appendix F (Table 11b) re-ranks the eight methods evaluated on every axis by absolute performance and by the signed gap.
Average rank (1 = best) of the eight methods evaluated on every axis (Table 11b; attribution from Table 11):
W2/Q2: pretraining overlap in temporal and inductive-entity experiments. Yes. We re-pretrained BGRL, GraphMAE, GIT, UniGraph2, GFT and LLaGA after removing all arXiv papers from 2017 onward (the temporal OOD test period; 59% of arXiv nodes) and the union of the held-out test entities across the inductive splits of WN18RR and FB15K237 (about 58% of entities) together with their edges (about 18% of KG edges remain), and re-ran the original evaluation protocol (Appendix E, Table 10b).
Before → after removing the OOD test data from pretraining (Table 10b):