- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces mGNN, which jointly tunes GNN backbone layout, message-passing mechanism, and training hyperparameters using Bayesian optimization, and extends it with a generalized aggregation function. Its main contribution is a unfied model-selection framework evaluated on 25 node prediction datasets against 21 baselines.
Critical strengths
- A practical, modular framework for jointly tuning GNN architecture and training hyperparameters
- broad evalution across 25 datasets covering node classification and regression tasks.
- Controlled component ablations, with the remaining hyperparameters independently re-tuned, provide useful evidence for task-dependent architectural choices; the descriptive importance analysis complements these results.
Critical weaknesses
- The ablations support tuning architectural choices rather than fixing them, but do not directly isolate the benefit of joint optimization over staged tuning of the same options. Section 4 fixes components rather than optimizing them sequentially, while the reported first-order importance analysis does not characterize interactions. A budget-matched staged-tuning baseline would clarify the additional value of joint search.
- The attention-head search ranges are inconsistent. Tables 8–9 allow mGNN and mGNN† to search over {1, 2, 4, 8}, whereas the attention-based baselines in Tables 3,4 and 6,7 are restricted to {2, 4, 8}. The effect of this additional option is not isolated, weakening attribution of the performance gains to the proposed design.
- Search efficiency is not adequately evaluated. All methods receive 100 trials, but equal trial counts do not ensure equal computational cost. No performance-versus-budget curves or comparisons under matched wall-clock or GPU-hour budgets are reported. It remains unclear whether the expanded search space offers a favorable performance–cost trade-off
Questions for authors
- How does joint search compare with an architecture-first, training-hyperparameter-second procedure over the same options and with the same total trial budget? Could you specify the budget allocation between stages?
- Why is the single-head option excluded from the attention-based baselines? Report results with aligned attention-head ranges and how often H = 1 is selected by mGNN and mGNN†.
- Could you report validation-selected performance versus trial count and cumulative GPU time, including smaller budgets and the startup-sampling policy? Under matched hardware and compute budgets, how long do mGNN, mGNN†, and the strongest baselines take to reach comparable performance?
3 comments
W3 and Q3. Performance and computational cost under matched hardware
To investigate how predictive performance depends on the computational budget, we compare the proposed and variants and the strongest baseline Classic GCN over the first 100 trials on matched hardware, including 25 random startup trials. The reported test performance comes from configurations selected by the validation score, using one training seed during tuning. For each dataset, the common time budget is the shortest time taken by the three methods to complete these 100 trials. The following results use that common budget , with Classic GCN as the reference for and .
| Method | ||||
|---|---|---|---|---|
| \mathbf{+1.20\\%} | ||||
| Classic GCN | \underline{0.00\\%} |
The advantage of over the strongest baseline Classic GCN remains when the methods receive equal GPU time on each dataset. Please note that slightly falls behind the Classic GCN baseline in these particular experiments due to the selection of datasets, especially the lack of other Graphland datasets that pose a notable challenge for the considered baselines (see more results in Table 10 of Appendix D).
Next, we also compare the computational costs of the full tuning runs across the selected datasets. The following table reports total tuning time, training time of the selected models, and parameter count. The columns report signed percentage changes relative to Classic GCN. Totals for the selected models cover one configuration selected by the validation score per dataset, using training times recorded during tuning.
| Method | Tuning time, GPU-hours | Tuning | Training time, GPU-hours | Training | Parameters, | Parameters |
|---|---|---|---|---|---|---|
| Classic GCN | — | — | — |
As can be seen, the introduced mGNN variants require notably less time for tuning and training, but typically have more parameters in the best configuration.
Finally, we compare the time needed to reach comparable performance using the recorded tuning runs. We use the test performance of Classic GCN's configuration selected by the validation score from the first 100 trials as the reference on each dataset. The table reports the time each variant's best configuration first reaches or exceeds this reference score, as a percentage of the baseline's first time to reach it. Here, means equal time, smaller values mean faster, and a dash indicates that the reference score was not reached within 100 trials.
| Dataset | ||
|---|---|---|
lastfm-asia | — | — |
facebook | ||
flickr | — | — |
roman-empire | ||
amazon-ratings | — | |
questions | — | |
tolokers-2 | ||
artnet-exp | ||
city-roads-M | ||
artnet-views |
reaches this target on datasets and takes less time than Classic GCN to first reach the target on of those . reaches it on datasets and uses less time on of those . Please refer to Appendix E.7, which provides curves showing test performance against trial count and cumulative GPU-hours.
Official Comment by Reviewer aDxa
Thanks to the authors for the rebuttal. The additional experiments address the points I raised. I understand that limited computational resources meant running most of them on a subset of 10 graph datasets, and I appreciate that the authors clearly reported where the baselines outperformed mGNN.
we still plan to conduct these experiments on all 25 graph datasets used in our submission and include the results in the paper.
Please be sure to include the full results in the revised version of paper.
I am inclined toward a weak accept now, but would like to see more discussion from other reviewers before increasing my score.
We thank the reviewer for their detailed feedback and concrete suggestions for strengthening the experimental comparisons. We address the raised concerns below.
Because of the number of requested experiments and limited computational resources, the experiments below use a subset of 10 graph datasets described in Appendix E.1, which enables us to conduct them relatively quickly but also reliably distinguish the compared models. However, we still plan to conduct these experiments on all 25 graph datasets used in our submission and include the results in the paper.
W1 and Q1. Joint versus staged hyperparameter tuning
To assess the benefit of joint tuning, we compare it with staged tuning under the same trial budget. The staged alternative allocates 50 trials to architecture selection at fixed training settings, then 50 trials to tuning those settings for the selected architecture. Its stages use 13 and 12 random startup trials, matching the total of 25 for joint tuning. The results are provided below. Elo and RI use joint tuning as the reference.
Joint tuning gives stronger aggregate performance under the same budget. Since both procedures lead in the pure mean test performance on half of the datasets, they have the same NS, but staged tuning has large relative losses on
questionsandartnet-views. These results support joint tuning as the default option under the considered budget. Please refer to Appendix E.4 for the exact stage settings and results for each dataset.W2 and Q2. We overlooked the difference in the search spaces for the number of attention heads, and we thank the reviewer for noticing it and bringing it to our attention. To fix the comparison, we extend the Classic and Critical GAT and LGT search spaces from H∈2,4,8 to H∈1,2,4,8, matching the search space used in our mGNN models. We retune these four baselines with the same budget, original training functions, and remaining search settings. The aggregate metrics below are computed over these four baselines and both mGNN variants on the selected 10 datasets.
Original baseline head range 2,4,8:
Expanded baseline head range 1,2,4,8:
mGNN† remains first in all three aggregate measures, but Classic GAT overtakes mGNN for second place in this reduced set of experiments. Please note that the selected graph datasets omit other challenging GraphLand datasets on which mGNN performs strongly (see Table 10 in Appendix D). In the original benchmark including all 25 datasets, mGNN† selects H=1 on 2/25 datasets, while mGNN selects attention on 16/25 datasets and uses H=1 in 2/16 of those configurations. Please refer to Appendix E.8 for the comparison figure and other details.