Benchmarking Reliability in Graph Representation Learning

Graph representation learning (GRL) has evolved from topology-only graph embeddings to task-specific supervised GNNs and, more recently, to reusable representations and graph foundation models (GFMs). However, existing GRL benchmarks primarily evaluate clean-setting performance, transfer, adaptation, and task coverage, while reliability-relevant evaluations are typically conducted one axis at a time under different method pools and protocols. As a result, it remains unclear how modern GRL methods compare under matched conditions when deployment conditions affect graph signals, graph contexts, label support, structural groups, or predictive evidence. We introduce GRL-Reliability, a multi-axis benchmark for evaluating GRL reliability. GRL-Reliability evaluates twelve representative methods, spanning topology-only embeddings, supervised GNNs, self-supervised graph models, and GFMs, on twenty-five graph datasets under standardized evaluation conditions while preserving method-native adaptation where available. The benchmark covers five evaluation axes: corruption robustness, OOD generalization, class imbalance, fairness, and interpretation, with per-axis and subcondition reporting rather than a single aggregate score. Our analysis reveals three cross-axis insights. First, even tests that intervene on the same graph factor can produce distinct method profiles, showing that closely related tests capture different aspects of reliability. Second, foundation-era methods show axis-specific strengths rather than broad reliability advantages. Third, degree shift and rare-class coverage remain challenging across the evaluated method set, revealing deployment regimes where current GRL methods still have substantial room for improvement. The benchmark, evaluation protocols, and code are available at: https://anonymous.4open.science/r/GRL-Reliability-1889/.