acceptodds
Under review as a conference paper at ICLR 2027

A Modern Approach to GNN Model Selection for Node Property Prediction

Abstract

In the graph machine learning field, graph neural networks (GNNs) have become a standard class of models. However, most studies that use such models for their experiments overlook several important aspects of model selection. First, they often do not pay much attention to how GNN backbone is composed and instead use some fixed, arbitrarily chosen sequence of operations. Second, they normally do not consider message-passing mechanism as an option for architecture tuning and choose not to include it in the search space, while treating GNNs with different aggregation functions as completely independent methods. Lastly, they may not invest enough resources in GNN hyperparameter tuning, while using quite simple optimization techniques such as random search over a coarse grid of values. We believe these issues can notably limit the potential of GNNs as a class of models, reduce their predictive performance, and ultimately diminish their practical utility. In this work, we propose to make a step beyond the existing practices in GNN model selection and show how to construct a modern GNN baseline that better accounts for the given graph data and prediction task. Specifically, we investigate how the joint optimization of architecture backbone, its message-passing mechanism, and other hyperparameters over a complex configuration space can extend and outperform conventional GNN tuning approaches. By incorporating each mentioned aspect of model selection into a single tuning procedure and using a standard Bayesian optimization technique, we can achieve notably higher predictive performance in node property prediction than by using many other methods proposed in prior works on GNN model design and implemented in graph deep learning packages. The introduction of a flexible generalized aggregation function further improves the performance and provides more evidence that various prediction tasks may require finer tuning of the message-passing behavior. Our study provides a new perspective on what model selection in modern GNNs should account for and how their architecture can be optimized to obtain stronger results in practice.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
graph neural network, model selection, architecture design, hyperparameter optimization, strong baseline, node property prediction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_aDxaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces mGNN, which jointly tunes GNN backbone layout, message-passing mechanism, and training hyperparameters using Bayesian optimization, and extends it with a generalized aggregation function. Its main contribution is a unfied model-selection framework evaluated on 25 node prediction datasets against 21 baselines.

Critical strengths
  1. A practical, modular framework for jointly tuning GNN architecture and training hyperparameters
  2. broad evalution across 25 datasets covering node classification and regression tasks.
  3. Controlled component ablations, with the remaining hyperparameters independently re-tuned, provide useful evidence for task-dependent architectural choices; the descriptive importance analysis complements these results.
Critical weaknesses
  1. The ablations support tuning architectural choices rather than fixing them, but do not directly isolate the benefit of joint optimization over staged tuning of the same options. Section 4 fixes components rather than optimizing them sequentially, while the reported first-order importance analysis does not characterize interactions. A budget-matched staged-tuning baseline would clarify the additional value of joint search.
  2. The attention-head search ranges are inconsistent. Tables 8–9 allow mGNN and mGNN† to search over {1, 2, 4, 8}, whereas the attention-based baselines in Tables 3,4 and 6,7 are restricted to {2, 4, 8}. The effect of this additional option is not isolated, weakening attribution of the performance gains to the proposed design.
  3. Search efficiency is not adequately evaluated. All methods receive 100 trials, but equal trial counts do not ensure equal computational cost. No performance-versus-budget curves or comparisons under matched wall-clock or GPU-hour budgets are reported. It remains unclear whether the expanded search space offers a favorable performance–cost trade-off
Questions for authors
  1. How does joint search compare with an architecture-first, training-hyperparameter-second procedure over the same options and with the same total trial budget? Could you specify the budget allocation between stages?
  2. Why is the single-head option excluded from the attention-based baselines? Report results with aligned attention-head ranges and how often H = 1 is selected by mGNN and mGNN†.
  3. Could you report validation-selected performance versus trial count and cumulative GPU time, including smaller budgets and the startup-sampling policy? Under matched hardware and compute budgets, how long do mGNN, mGNN†, and the strongest baselines take to reach comparable performance?
3 comments

We thank the reviewer for their detailed feedback and concrete suggestions for strengthening the experimental comparisons. We address the raised concerns below.

Because of the number of requested experiments and limited computational resources, the experiments below use a subset of 10 graph datasets described in Appendix E.1, which enables us to conduct them relatively quickly but also reliably distinguish the compared models. However, we still plan to conduct these experiments on all 25 graph datasets used in our submission and include the results in the paper.

W1 and Q1. Joint versus staged hyperparameter tuning

To assess the benefit of joint tuning, we compare it with staged tuning under the same trial budget. The staged alternative allocates 50 trials to architecture selection at fixed training settings, then 50 trials to tuning those settings for the selected architecture. Its stages use 13 and 12 random startup trials, matching the total of 25 for joint tuning. The results are provided below. Elo\mathrm{Elo} and RI\mathrm{RI} use joint tuning as the reference.

Tuning procedureFR↓\mathrm{FR}\downarrowNS↑\mathrm{NS}\uparrowElo↑\mathrm{Elo}\uparrowRI↑\mathrm{RI}\uparrow
Joint1.2\mathbf{1.2}0.5\mathbf{0.5}1000\mathbf{1000}\mathbf{0.00\\%}
Staged1.4‾\underline{1.4}0.5\mathbf{0.5}930‾\underline{930}\underline{-2.56\\%}

Joint tuning gives stronger aggregate performance under the same budget. Since both procedures lead in the pure mean test performance on half of the datasets, they have the same NS\mathrm{NS}, but staged tuning has large relative losses on questions and artnet-views. These results support joint tuning as the default option under the considered budget. Please refer to Appendix E.4 for the exact stage settings and results for each dataset.

W2 and Q2. We overlooked the difference in the search spaces for the number of attention heads, and we thank the reviewer for noticing it and bringing it to our attention. To fix the comparison, we extend the Classic and Critical GAT and LGT search spaces from H∈2,4,8H \in \\{2,4,8\\} to H∈1,2,4,8H \in \\{1,2,4,8\\}, matching the search space used in our mGNN\mathrm{mGNN} models. We retune these four baselines with the same budget, original training functions, and remaining search settings. The aggregate metrics below are computed over these four baselines and both mGNN\mathrm{mGNN} variants on the selected 10 datasets.

Original baseline head range 2,4,8\\{2,4,8\\}:

MethodFR↓\mathrm{FR}\downarrowNS↑\mathrm{NS}\uparrowElo↑\mathrm{Elo}\uparrow
mGNN†\mathrm{mGNN}^{\dagger}1.7\mathbf{1.7}0.78\mathbf{0.78}1135\mathbf{1135}
mGNN\mathrm{mGNN}2.2‾\underline{2.2}0.57‾\underline{0.57}1000‾\underline{1000}
Classic GAT2.42.40.500.50991991
Classic LGT3.03.00.310.31858858
Critical GAT2.42.40.430.43939939
Critical LGT2.82.80.310.31902902

Expanded baseline head range 1,2,4,8\\{1,2,4,8\\}:

MethodFR↓\mathrm{FR}\downarrowNS↑\mathrm{NS}\uparrowElo↑\mathrm{Elo}\uparrow
mGNN†\mathrm{mGNN}^{\dagger}1.7\mathbf{1.7}0.79\mathbf{0.79}1137\mathbf{1137}
mGNN\mathrm{mGNN}2.32.30.520.5210001000
Classic GAT2.2‾\underline{2.2}0.56‾\underline{0.56}1046‾\underline{1046}
Classic LGT3.13.10.300.30857857
Critical GAT2.42.40.480.48997997
Critical LGT3.33.30.200.20823823

mGNN†\mathrm{mGNN}^{\dagger} remains first in all three aggregate measures, but Classic GAT overtakes mGNN\mathrm{mGNN} for second place in this reduced set of experiments. Please note that the selected graph datasets omit other challenging GraphLand datasets on which mGNN\mathrm{mGNN} performs strongly (see Table 10 in Appendix D). In the original benchmark including all 25 datasets, mGNN†\mathrm{mGNN}^{\dagger} selects H=1H = 1 on 2/252/25 datasets, while mGNN\mathrm{mGNN} selects attention on 16/2516/25 datasets and uses H=1H = 1 in 2/162/16 of those configurations. Please refer to Appendix E.8 for the comparison figure and other details.

W3 and Q3. Performance and computational cost under matched hardware

To investigate how predictive performance depends on the computational budget, we compare the proposed mGNN†\mathrm{mGNN}^{\dagger} and mGNN\mathrm{mGNN} variants and the strongest baseline Classic GCN over the first 100 trials on matched hardware, including 25 random startup trials. The reported test performance comes from configurations selected by the validation score, using one training seed during tuning. For each dataset, the common time budget TT is the shortest time taken by the three methods to complete these 100 trials. The following results use that common budget TT, with Classic GCN as the reference for Elo\mathrm{Elo} and RI\mathrm{RI}.

MethodFR↓\mathrm{FR}\downarrowNS↑\mathrm{NS}\uparrowElo↑\mathrm{Elo}\uparrowRI↑\mathrm{RI}\uparrow
mGNN†\mathrm{mGNN}^{\dagger}1.6\mathbf{1.6}0.73\mathbf{0.73}1123\mathbf{1123}\mathbf{+1.20\\%}
mGNN\mathrm{mGNN}2.32.30.410.41951951−1.63-1.63\\%
Classic GCN2.1‾\underline{2.1}0.43‾\underline{0.43}1000‾\underline{1000}\underline{0.00\\%}

The advantage of mGNN†\mathrm{mGNN}^{\dagger} over the strongest baseline Classic GCN remains when the methods receive equal GPU time on each dataset. Please note that mGNN\mathrm{mGNN} slightly falls behind the Classic GCN baseline in these particular experiments due to the selection of datasets, especially the lack of other Graphland datasets that pose a notable challenge for the considered baselines (see more results in Table 10 of Appendix D).

Next, we also compare the computational costs of the full tuning runs across the selected datasets. The following table reports total tuning time, training time of the selected models, and parameter count. The Δ\Delta columns report signed percentage changes relative to Classic GCN. Totals for the selected models cover one configuration selected by the validation score per dataset, using training times recorded during tuning.

MethodTuning time, GPU-hoursTuning Δ\DeltaTraining time, GPU-hoursTraining Δ\DeltaParameters, ×106\times 10^6Parameters Δ\Delta
mGNN†\mathrm{mGNN}^{\dagger}33.9933.99−13.6-13.6\\%0.370.37−3.0-3.0\\%33.5433.54+80.2+80.2\\%
mGNN\mathrm{mGNN}24.8524.85−36.8-36.8\\%0.280.28−25.2-25.2\\%19.0019.00+2.1+2.1\\%
Classic GCN39.3539.35—0.380.38—18.6118.61—

As can be seen, the introduced mGNN variants require notably less time for tuning and training, but typically have more parameters in the best configuration.

Finally, we compare the time needed to reach comparable performance using the recorded tuning runs. We use the test performance of Classic GCN's configuration selected by the validation score from the first 100 trials as the reference on each dataset. The table reports the time each variant's best configuration first reaches or exceeds this reference score, as a percentage of the baseline's first time to reach it. Here, 100%100\% means equal time, smaller values mean faster, and a dash indicates that the reference score was not reached within 100 trials.

DatasetmGNN†\mathrm{mGNN}^{\dagger}mGNN\mathrm{mGNN}
lastfm-asia——
facebook32.732.7\\%21.421.4\\%
flickr——
roman-empire27.027.0\\%19.319.3\\%
amazon-ratings24.524.5\\%—
questions126.9126.9\\%—
tolokers-2150.5150.5\\%10.710.7\\%
artnet-exp54.954.9\\%39.739.7\\%
city-roads-M123.1123.1\\%44.544.5\\%
artnet-views25.825.8\\%222.4222.4\\%

mGNN†\mathrm{mGNN}^{\dagger} reaches this target on 8/108/10 datasets and takes less time than Classic GCN to first reach the target on 55 of those 88. mGNN\mathrm{mGNN} reaches it on 6/106/10 datasets and uses less time on 55 of those 66. Please refer to Appendix E.7, which provides curves showing test performance against trial count and cumulative GPU-hours.

Reviewer_aDxaOpenReview ↗

Official Comment by Reviewer aDxa

Thanks to the authors for the rebuttal. The additional experiments address the points I raised. I understand that limited computational resources meant running most of them on a subset of 10 graph datasets, and I appreciate that the authors clearly reported where the baselines outperformed mGNN.

we still plan to conduct these experiments on all 25 graph datasets used in our submission and include the results in the paper.

Please be sure to include the full results in the revised version of paper.

I am inclined toward a weak accept now, but would like to see more discussion from other reviewers before increasing my score.

Reviewer_EBqfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes mGNN, a framework for graph neural network model selection that jointly optimizes backbone architecture, message-passing mechanisms, and training hyperparameters using Tree-structured Parzen Estimator Bayesian optimization. The search space includes multiple backbone layouts, normalization configurations, aggregation functions, and standard architectural and optimization hyperparameters. The authors further introduce mGNN†\text{mGNN}^\dagger, which uses a generalized attention-based aggregation function with tunable temperature, attention sink, and degree adjustment.

Critical strengths
  • The paper addresses an important issue in GNN evaluation: predictive performance can depend substantially on backbone design, aggregation mechanisms, and hyperparameter tuning. The emphasis on stronger, task-adaptive baselines is valuable for the graph machine learning community.
  • The evaluation covers 25 datasets spanning multiclass classification, binary classification, and regression.
  • The proposed attention-based aggregation function provides a flexible way to approximate several conventional aggregation behaviors.
Critical weaknesses
  • Although the authors discuss GraphGym and several GNN NAS methods, the distinction between the proposed framework and existing graph AutoML approaches could be made more explicit. In particular, GraphGym and AutoGL already provide frameworks for GNN architecture exploration and automated model selection.
  • It remains unclear whether mGNN†\text{mGNN}^\dagger benefits specifically from black-box tuning of (τ,s)(\tau, s) or if making (τ,s)(\tau, s) learnable per-head/per-layer parameters within mGNN would achieve the same (or better) expressivity while shrinking the HPO search space by two dimensions
Questions for authors
  • How do the rankings of TPE, random search, and genetic search change with different budgets, such as 25, 50, 100, and 200 trials?
  • Can the authors report training time, parameter counts, and total tuning cost for mGNN, mGNN†\text{mGNN}^\dagger, and the strongest competing baselines?
  • What are the individual contributions of the attention temperature, sink, and degree adjustment? How frequently does optimization select intermediate aggregation regimes rather than configurations close to conventional aggregation functions?
2 comments

We thank the reviewer for their insightful questions and helpful suggestions for clarifying our contribution. We address the raised concerns below.

Because of the number of requested experiments and limited computational resources, the experiments below use a subset of 10 graph datasets described in Appendix E.1, which enables us to conduct them relatively quickly but also reliably distinguish the compared models. However, we still plan to conduct these experiments on all 25 graph datasets used in our submission and include the results in the paper.

W1. Relation to GraphGym and AutoGL

Our central question is how to construct a model selection procedure that can produce a strong and reliable GNN baseline for node property prediction. In our study, we show that backbone layout, message passing, and training settings should be selected together for each prediction task, and it can be done through a simple, joint tuning procedure based on Bayesian optimization over a complex search space.

In contrast, GraphGym [1] studies which GNN design choices work well across tasks and when successful architectures transfer between tasks. Its initial design space includes architecture and training settings, but its transfer experiments use a reduced grid of hyperparameter values with fixed training settings rather than tuning across the full search space independently for each task. Our procedure instead uses Bayesian optimization to select backbone layout, message passing, and training settings specifically for the given task.

Further, AutoGL [2] automates feature processing, model tuning, and optional ensembling. Its published implementation jointly tunes model and training hyperparameters separately for each supplied candidate model type, but does not select among those types within a single hyperparameter search. Our approach instead treats backbone layout and aggregation function as jointly tuned components on par with other standard hyperparameters in the same search space.

[1] You et al., "Design Space for Graph Neural Networks," 2020.

[2] Guan et al., "AutoGL: A Library for Automated Graph Learning," 2021.

W2. Tuned versus learned temperature and sink

To investigate whether the introduced temperature τ\tau and sink ss knobs can be learned during training, we remove these two hyperparameters from the search space and learn them separately for each attention head and aggregation layer, initializing them at τ=1\tau = 1 and s=0s = 0 and keeping them within the original ranges. The results are provided below. Elo\mathrm{Elo} and RI\mathrm{RI} use the tuned variant as the reference.

Aggregation parametersFR↓\mathrm{FR}\downarrowNS↑\mathrm{NS}\uparrowElo↑\mathrm{Elo}\uparrowRI↑\mathrm{RI}\uparrow
Tuned1.1\mathbf{1.1}0.6\mathbf{0.6}1000\mathbf{1000}\mathbf{0.00\\%}
Learned1.6‾\underline{1.6}0.4‾\underline{0.4}915‾\underline{915}\underline{-3.38\\%}

Tuned temperature τ\tau and sink ss give stronger results in all four aggregate measures. Although learning these parameters improves the performance on some datasets, the aggregate results favor preserving their global tuning in the proposed model selection procedure. Please refer to Appendix E.5 for the detailed results on each dataset.

Q1. Comparison of configuration samplers across trial budgets

To compare different samplers, we extend their runs to 200 trials, use the same 25 random startup configurations as in our initial submission, and plot the curves showing how the test performance of the best configuration selected by the validation score changes with increasing trial budget. To summarize the behavior observed in these curves, we average each aggregate measure over the considered budgets after the shared startup, using either uniform weights or weights proportional to 1/b1/b for bb denoting the number of trials to give more importance to earlier budgets. The integrated scores are provided below. RI\mathrm{RI} is measured relative to the result obtained after the shared 25 startup trials, which also serves as the Elo\mathrm{Elo} reference rated 10001000.

Budget weightingSamplerFR↓\mathrm{FR}\downarrowNS↑\mathrm{NS}\uparrowElo↑\mathrm{Elo}\uparrowRI↑\mathrm{RI}\uparrow
UniformBayesian1.76\mathbf{1.76}0.60\mathbf{0.60}1110.2\mathbf{1110.2}\mathbf{+0.45\\%}
Genetic1.84‾\underline{1.84}0.58‾\underline{0.58}1093.7‾\underline{1093.7}−0.37-0.37\\%
Random2.072.070.440.441028.91028.9\underline{-0.21\\%}
Inverse-budgetBayesian1.70\mathbf{1.70}0.61\mathbf{0.61}1072.4\mathbf{1072.4}\mathbf{+0.26\\%}
Genetic1.79‾\underline{1.79}0.58‾\underline{0.58}1053.1‾\underline{1053.1}−0.68-0.68\\%
Random1.881.880.510.511020.91020.9\underline{-0.20\\%}

The exact ordering may depend on the trial budget: the Random sampler leads most aggregate measures at 50 trials, the Bayesian leads at 100 trials, and the Genetic leads at 200 trials. However, the Bayesian sampler leads all four integrated scores under both weightings and is the only sampler with positive average RI\mathrm{RI} over startup in these searches. These results support the usage of Bayesian optimization as a strong default option for the proposed model selection procedure. Please refer to Appendix E.3 for the complete aggregate curves and curves for each dataset.

Q2. Our response to Reviewer aDxa under W3 and Q3 reports tuning and training time cost together with parameter counts. Please refer to Appendix E.7 for the detailed results under matched hardware.

Q3. Selected attention distributions and sink mass

To investigate which aggregation behaviors are selected for different graph datasets, we normalize neighbor entropy by its value for uniform attention and measure the attention mass assigned to the sink. We also classify node-level attention distributions as ∼Uniform\sim\mathrm{Uniform} for normalized entropy ⩾0.9\geqslant 0.9, ∼Argmax\sim\mathrm{Argmax} for normalized entropy ⩽0.1\leqslant 0.1, and Intermediate between them. After computing such statistics, we average them over attention heads, aggregation layers, and training seeds. Some representative and interesting results are provided below.

DatasetNormalized entropySink mass~UniformIntermediate~Argmax
lastfm-asia98.5998.59\\%4.934.93\\%98.3498.34\\%1.661.66\\%0.000.00\\%
facebook59.6059.60\\%11.4411.44\\%19.8019.80\\%70.6470.64\\%9.579.57\\%
questions99.7599.75\\%78.3078.30\\%99.9699.96\\%0.040.04\\%0.000.00\\%

The tuned configurations of mGNN†\mathrm{mGNN}^{\dagger} vary in both neighbor attention and sink mass across datasets. For example, lastfm-asia and questions favor nearly uniform neighbor attention, with mean sink masses of 4.934.93\\% and 78.3078.30\\%, respectively. On facebook, 70.6470.64\\% of distributions have intermediate entropy, reflecting more selective weighting of neighbors.

Together with general response R1 about the contribution of hyperparameters in the generalized aggregation function, our observations support that it is important to tune the aggregation behavior for each prediction task. Please refer to Appendix E.6 for the detailed results.

Reviewer_a6REOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies GNN model selection for node prediction. The authors jointly tune backbone layout, aggregation, and training hyperparameters using TPE. An extended version introduces a generalized aggregator with adjustable temperature, an attention sink, and degree adjustment. Experiments cover 25 datasets and include ablations that fix individual components while retuning the remaining parameters.

Critical strengths

I agree with the paper’s view of model selection. An aggregator’s performance depends on its surrounding architecture and training setup, so evaluating how well a complete selection procedure performs is useful. The experiments test this directly: when fixing a backbone or aggregation choice, the authors retune the remaining parameters. This makes the component ablations convincing. The coverage of 25 datasets and the individual results also make it clear where the gains occur.

The mGNN design is simple and effective. It achieves particularly clear gains on GraphLand. The leading methods are closer on classic datasets, and the proposed method does not win everywhere. These differences are reported clearly, making mGNN a potentially useful GNN baseline.

Critical weaknesses

The paper clearly explains the generalized aggregator’s limiting regimes and demonstrates its overall benefit. However, it does not separately ablate temperature, the attention sink, and degree adjustment. Their individual contributions, and whether all three are needed for the reported gains, remain unclear.

Repetitions of the complete search procedure are not reported. Section 3 describes evaluating the selected configuration with five seeds, which captures retraining variability but not search-stage variability. Clarifying whether independent searches were repeated would help assess the reliability of the search-strategy comparisons.

Questions for authors
  1. Did you repeat the complete hyperparameter search with different sampler seeds? If so, how consistent were the results, particularly the comparison between TPE and random search?

  2. Do you have ablation results showing the individual contributions of temperature, the attention sink, and degree adjustment in extended mGNN? These would help clarify which components account for its gains over mGNN.

2 comments

We thank the reviewer for their careful reading and constructive feedback. We address the raised concerns below.

Because of the number of requested experiments and limited computational resources, the experiments below use a subset of 10 graph datasets described in Appendix E.1, which enables us to conduct them relatively quickly but also reliably distinguish the compared models. However, we still plan to conduct these experiments on all 25 graph datasets used in our submission and include the results in the paper.

W2 and Q1. Robustness to the random seed of configuration sampler

To assess robustness to the random seed of configuration sampler, we repeat the full tuning runs for both mGNN\mathrm{mGNN} variants, keeping the training and preprocessing seeds used at the tuning stage fixed. Given the limited computational resources and time available for the response, we have so far repeated these searches only for TPE. In the table below, the best Classic and best Critical baseline report the best values among GCN, GAT, SAGE, and LGT within each family, selected separately for each metric and sampler seed. Elo\mathrm{Elo} is tied to mGNN\mathrm{mGNN} at 10001000 for each seed.

Sampler seedMethodFR↓\mathrm{FR}\downarrowNS↑\mathrm{NS}\uparrowElo↑\mathrm{Elo}\uparrow
00mGNN†\mathrm{mGNN}^{\dagger}1.9\mathbf{1.9}0.82\mathbf{0.82}1119\mathbf{1119}
mGNN\mathrm{mGNN}2.5‾\underline{2.5}0.630.631000‾\underline{1000}
Best Classic2.72.70.64‾\underline{0.64}980980
Best Critical3.03.00.550.55927927
11mGNN†\mathrm{mGNN}^{\dagger}1.9\mathbf{1.9}0.84\mathbf{0.84}1074\mathbf{1074}
mGNN\mathrm{mGNN}2.1‾\underline{2.1}0.76‾\underline{0.76}1000‾\underline{1000}
Best Classic2.82.80.660.66897897
Best Critical3.03.00.570.57847847
22mGNN†\mathrm{mGNN}^{\dagger}1.9\mathbf{1.9}0.81\mathbf{0.81}1072\mathbf{1072}
mGNN\mathrm{mGNN}2.6‾\underline{2.6}0.73‾\underline{0.73}1000‾\underline{1000}
Best Classic2.6‾\underline{2.6}0.710.71965965
Best Critical3.13.10.590.59927927

The advantage of the mGNN\mathrm{mGNN} model variants over the Classic and Critical baselines persists across all three sampler seeds. Separate paired permutation tests for each model variant give Holm-adjusted pp-values of 0.7100.710 for mGNN†\mathrm{mGNN}^{\dagger} and 0.6320.632 for mGNN\mathrm{mGNN}, providing insufficient evidence of a systematic effect of the sampler seed in these experiments. Please refer to Appendix E.9 for more details.

W1 and Q2. Please refer to general response R1 that answers your question about the contribution of hyperparameters in the generalized aggregation function.

Thank you for the additional experiments. The comparison with staged tuning is limited to one setup, and the relative performance of TPE and random search across seeds remains unclear. I will keep my score.

Comments on the paper

2 comments

We thank the reviewers for their thoughtful and constructive comments. In this general response, we address the question raised by several reviewers about the contribution of hyperparameters in the generalized aggregation function and provide the results for the requested component ablations on all 25 datasets.

R1. Contributions of the generalized aggregation hyperparameters

To isolate the contribution of hyperparameters of the proposed generalized aggregation, we fix them one at a time and independently retune the remaining hyperparameters on all datasets. The results are provided below. RI\mathrm{RI} is measured relative to the default variant where each hyperparameter is tuned as usual, which also serves as the Elo\mathrm{Elo} reference rated 10001000.

VariantFR↓\mathrm{FR}\downarrowNS↑\mathrm{NS}\uparrowElo↑\mathrm{Elo}\uparrowRI↑\mathrm{RI}\uparrow
Everything tuned, mGNN†\mathrm{mGNN}^{\dagger}1.56\mathbf{1.56}0.62\mathbf{0.62}1000\mathbf{1000}\mathbf{0.00\\%}
Temperature fixed, τ=1\tau=11.921.920.410.41891891−0.92-0.92\\%
Sink disabled, s=−∞s=-\infty1.801.800.520.52921921−0.61-0.61\\%
Degree adjustment disabled, γuv=1\gamma_{uv}=11.761.760.570.57986986−0.65-0.65\\%

Each ablation reduces the aggregate test performance, supporting that the introduced aggregation knobs should be included in the model selection procedure to achieve the strongest result on average. Please refer to Appendix E.2 for more details.

Dear Reviewers,

As the discussion period is ending, we would like to ensure we have addressed all your concerns. Our rebuttal includes the requested experiments, such as component ablations, comparisons of configuration samplers, joint versus staged tuning, learned versus tuned aggregation, matched-hardware cost and performance, matched attention-head ranges, and sampler-seed robustness.

If there are any additional points or feedback you would like us to consider, please let us know. Thank you for your time and effort in reviewing our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.