A Modern Approach to GNN Model Selection for Node Property Prediction

In the graph machine learning field, graph neural networks (GNNs) have become a standard class of models. However, most studies that use such models for their experiments overlook several important aspects of model selection. First, they often do not pay much attention to how GNN backbone is composed and instead use some fixed, arbitrarily chosen sequence of operations. Second, they normally do not consider message-passing mechanism as an option for architecture tuning and choose not to include it in the search space, while treating GNNs with different aggregation functions as completely independent methods. Lastly, they may not invest enough resources in GNN hyperparameter tuning, while using quite simple optimization techniques such as random search over a coarse grid of values. We believe these issues can notably limit the potential of GNNs as a class of models, reduce their predictive performance, and ultimately diminish their practical utility. In this work, we propose to make a step beyond the existing practices in GNN model selection and show how to construct a modern GNN baseline that better accounts for the given graph data and prediction task. Specifically, we investigate how the joint optimization of architecture backbone, its message-passing mechanism, and other hyperparameters over a complex configuration space can extend and outperform conventional GNN tuning approaches. By incorporating each mentioned aspect of model selection into a single tuning procedure and using a standard Bayesian optimization technique, we can achieve notably higher predictive performance in node property prediction than by using many other methods proposed in prior works on GNN model design and implemented in graph deep learning packages. The introduction of a flexible generalized aggregation function further improves the performance and provides more evidence that various prediction tasks may require finer tuning of the message-passing behavior. Our study provides a new perspective on what model selection in modern GNNs should account for and how their architecture can be optimized to obtain stronger results in practice.