- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work proposes GraphPack, a node-level prediction version of the recent method TabPack. It precomputes a shared bank of views (propagated features and training labels), samples view subsets and hyperparameters for 64 MLPs, and trains them efficiently as a packed module. Online greedy selection then combines MLPs and LightGBM candidates to improve the validation score and produce the final ensemble model. The goal of this method is to achieve performance similar to or better than tuned graph learning baselines without per-dataset hyperparameter tuning (search), which often takes substantial time. GraphPack achieved the best or tied performance on seven of ten GraphLand datasets under the RH split (random splits with 50% training labels), compared with the evaluated GNN, LightGBM, and SIGN pipelines. It also had a substantially lower measured workflow cost than tuning and ensembling the SIGN baseline.
Critical strengths
- Ensembling is an effective, efficient, and widely used approach in tabular learning, but I believe its practical value is often overlooked in graph learning. Showing its effectiveness and efficiency in graph learning is a practical contribution of this work.
- Ablations show that sampling view subsets brings benefits over giving every member the entire bank, supporting structural diversity as a contributor to the method’s performance. This highlights the importance of view diversity in building effective graph ensembles.
Critical weaknesses
- Questionable transferability: This work uses 16 datasets for development, while only one web graph with three different tasks is used for held-out evaluation, if I understood correctly. While achieving strong performance across diverse datasets with a single method is difficult, I believe this experimental setup provides limited support for its generalizability and transferability to unseen graphs.
- I believe the purpose of this method is to eliminate per-dataset hyperparameter tuning by ensembling inexpensive and diverse decoupled predictors. While the results are promising, graph foundation models, as discussed by the authors in the related work section, share a same target problem. Although the paper includes a comparison in Appendix D.4, I believe the cost and performance comparison should be discussed more prominently in the main experimental section.
Questions for authors
- GraphPack performs worse than GNNs on the city-road datasets. Does this observation reflect fundamental limitations of decoupled GNNs? If GraphPack inherits these limitations, is its design also fundamentally limited on graphs where decoupled methods underperform?
- GraphPack underperforms fine-tuned GraphPFN in the low-label regime. Do the authors have comparisons with GNNs on these same datasets and splits? If GraphPack also underperforms GNNs in the low-label regime, I believe this limitation should be adequately discussed in the main paper.
- How did the authors ensure a fair comparison on the datasets used for development, which make up most of the evaluation in the paper? Doesn’t using these datasets for development give GraphPack an unfair advantage over the other baselines?
- GNN baselines use the validation set for hyperparameter search and checkpoint selection, while GraphPack uses it for early stopping and online greedy ensemble selection. While it is hard to say which method benefits more from the validation set, I believe the difference in how it is used could affect the comparison. Can GraphPack still outperform the baselines when ensemble selection uses an inner split of the training set under a matched training-label budget? If so, this would provide stronger evidence.
5 comments
Response to Reviewer 9dwR (2/4)
W1 — Transfer to new graph domains (continued)
GraphPack achieves 79.32 AP without an additional per-dataset search, exceeding both local controls and the seven published AP references shown here. Its mean ROC-AUC also exceeds both local controls. Elliptic's released features include neighborhood aggregates.
Why the fixed procedure can adapt to new graphs. The reusable rules specify which predictors to sample; training and selection determine how those predictors are combined for each graph. Figure 4 shows this adaptation under shared sampling rules: the selected view mix varies by graph, and selected label-view use tracks its measured benefit. The new social and financial results test this mechanism's practical value beyond the development tasks.
Manuscript revision: Sections 4.1 and 4.4 introduce the expanded transfer evaluation. Appendix H.1 (Tables 25–27) gives both new-graph protocols and results, including every BlueSky target and reported metric. Q3 below explains how this complements the development-benchmark comparisons.
W2 — Graph foundation models and cost in the main paper
The main paper now compares GraphPack with graph foundation models in accuracy and computational cost. Section 4.4 highlights the accuracy/cost comparison; Table 28 in Appendix H.2 details GraphPFN's in-context-learning (ICL), full fine-tuning (FT), and adapter fine-tuning (AFT) modes alongside GraphPack's fixed training-and-selection workflow. ICL also avoids downstream hyperparameter search, making it a directly relevant comparison. GraphPack completes the measured workflows in 10.0 s and 29.3 s without a pretrained predictor checkpoint, and Q2 below details its gains over the conventional GNN controls.
| Method | tolokers-2 (AP) | city-roads-M () | tolokers-2 runtime (s) | city-roads-M runtime (s) |
|---|---|---|---|---|
| GraphPack | 58.88 ± 0.65 | 64.11 ± 0.19 | 10.0† | 29.3† |
| Plain GAT (incl. tuning) | 55.21 ± 0.26 | 58.90 ± 0.26 | 765.4 | 703.7 |
| GAT+label (incl. tuning) | 55.10 ± 0.18 | 60.58 ± 0.16 | 760.7 | 703.1 |
| GCN | 56.27 ± 0.31 | 58.82 ± 0.25 | 27.24 ± 1.44 | 27.29 ± 3.55 |
| GraphPFN ICL | 61.29 ± 0.12 | 64.85 ± 0.13 | 7.03 ± 0.04 | 16.22 ± 0.02 |
| GraphPFN full FT | 62.80 ± 0.39 | 67.30 ± 0.21 | 195.44 ± 35.03 | 1023.22 ± 232.98 |
| GraphPFN adapter FT | 61.98 ± 0.58 | 67.53 ± 0.18 | — | — |
GraphPack and both GAT variants are measured locally on an A800 80 GB. GCN and GraphPFN values are from GraphPFN v4, Tables 1, 7, and 9; their runtimes are the authors' end-to-end measurements at optimal hyperparameters on an A100 80 GB; GraphPFN runtimes include the default ten-member ensemble. “—” means no corresponding runtime is included here. † GraphPack times are complete-run measurements from the original local profiling batch; accuracy and runtime come from separate evaluation and profiling batches.
This gives a direct view of the available workflows: GraphPack's reported times are lower than full FT and higher than ICL, while Q2 details its gains over the conventional GNNs. GraphPack trains without a pretrained predictor checkpoint; supplied node features may themselves be pretrained embeddings.
Peak GPU memory. Separate A800 80 GB profiles cover preprocessing, bank/graph construction, final training, selection, and scoring, excluding preceding search. The table gives the maximum of three process GPU peaks, sampled by NVML every 100 ms.
| Method | tolokers-2 GPU peak (GiB) | city-roads-M GPU peak (GiB) |
|---|---|---|
| GraphPack | 1.725 | 2.477 |
| SIGN+lab tune-then-ensemble | 2.857 | 3.689 |
| GAT+label | 2.072 | 2.594 |
GraphPack has lower observed GPU peaks than both baselines. These are the separate training profiles from Appendix B.6; the table above reports RL accuracy and workflow costs.
Manuscript revision: Section 4.4 presents the accuracy/cost comparison. Appendix H.2 (Table 28) lists GNN and GraphPFN results with their sources, and Appendix B.6 gives memory profiles. Appendices C.10/H.5 separately identify architecture-only inference timings.
Response to Reviewer 9dwR (3/4)
Q1 — Task adaptation on road graphs
GraphPack adapts to a road-prediction task through both learned predictors and ensemble selection. It improves over SIGN+lab and its tune-then-ensemble extension on both city-roads datasets; the new city-roads-M low-label comparison also gives a higher mean than GAT+label.
Evidence that the proposed design benefits road graphs. In Table 3, GraphPack without GBDT achieves mean scores of 77.80 and 76.78 on city-roads-M and city-roads-L, compared with 77.53 and 76.06 for SIGN+lab tune-then-ensemble; the full GraphPack achieves 78.00 and 76.91. Thus, its improvement over this decoupled pipeline is present even without tree candidates. Separately, the controlled neural-only ablation in Table 4 supports the value of sampling views on city-roads-M: sampling both views and training hyperparameters achieves 77.77 ± 0.12, compared with 77.37 ± 0.04 when every member receives all views.
The new city-roads-M comparisons extend this evidence: GraphPack reaches 64.11 ± 0.19 versus GAT+label's 60.58 ± 0.16 on the official RL split. Under strict inner selection, their means are 75.85 ± 0.08 and 75.83 ± 0.26, respectively (Q2/Q4; Appendix H.2/H.4).
Why precomputed views still permit task adaptation. For illustration, consider a bank of multi-hop features , where denotes node features and a fixed graph propagation operator. Even a linear readout can learn how to combine these views:
The weight blocks are fitted from training labels, so even this simple readout learns which propagation scales to emphasize. GraphPack learns nonlinear MLPs over different view subsets and then selects complementary predictions. The architectural distinction is where task-specific learning occurs: GraphPack learns over a bank of propagation scales and view families, while message-passing GNNs learn transformations between propagation steps. The road-graph comparisons and view ablation show the practical benefit of GraphPack's task adaptation within the decoupled approach.
Manuscript revision: Section 4.2 and Appendix H.3 connect task-adaptive prediction with the road-graph comparisons and structural ablation. Appendix G describes adaptation over the precomputed bank; Appendix H.4 reports the inner-selection experiment.
Q2 — GNN comparisons on the same low-label splits
GraphPack exceeds both locally tuned GAT variants and four published conventional GNN baselines on the same official low-label splits. We use identical RL masks (10%/10%/80% train/validation/test). Each local GAT variant receives its own 15-point validation-selected grid; GraphPack keeps its defaults. These added runs complement the submitted RL results retained in Appendix D.4. W2 reports the local results; the table below gives the published GNN comparisons.
| Method | tolokers-2 (AP) | city-roads-M () |
|---|---|---|
| GraphPack | 58.88 ± 0.65 | 64.11 ± 0.19 |
| GCN | 56.27 ± 0.31 | 58.82 ± 0.25 |
| GraphSAGE | 54.43 ± 0.34 | 59.44 ± 0.27 |
| Plain GAT | 57.41 ± 0.85 | 59.86 ± 0.20 |
| GT | 56.98 ± 0.55 | 59.55 ± 0.28 |
GraphPack reports three runs in this work. The tuned GNN results are from GraphPFN v4, Table 1, with ten runs on the official RL split. These GNNs include residual connections, layer normalization, and MLP blocks; GT uses neighborhood attention.
Relative to the locally tuned plain GAT, GraphPack gains 3.67 AP and 5.21 R² points; relative to GAT+label, it gains 3.78 and 3.53 points. The positive GNN comparison therefore extends across both label-input choices and the four published architectures.
Manuscript revision: Section 4.4 highlights the low-label GNN result. Appendix H.2 (Table 28) reports local and published comparisons, with GraphPFN variants and costs presented separately in W2 above.
Response to Reviewer 9dwR (1/4)
Thank you for recognizing the practical value of ensembling in graph learning and the role of structural-view diversity. We have addressed each weakness and question with new experiments and focused manuscript revisions.
Three additions directly strengthen the contribution:
For easy navigation, W1, W2, and Q1–Q4 are answered separately below. New performance results are mean ± sample SD over three seeds. AP, ROC-AUC, R², and Spearman are multiplied by 100; BlueSky MAE uses log1p engagement units. Published results retain their source protocols.
W1 — Transfer to new graph domains
The fixed GraphPack procedure now covers three held-out graphs from different domains: web, social, and financial. The added BlueSky-quotes and Elliptic experiments test the same model defaults on previously unused graphs and improve on both newly trained controls.
For both added graphs, the model defaults and seeds were fixed before observing their effectiveness results, and all declared targets and seeds were retained. What transfers is the training procedure and its configuration distribution: predictors are fitted on the new graph, without transferring fitted predictor weights or conducting a new GraphPack hyperparameter search.
BlueSky-quotes: a new temporal social graph. We use the released GraphBench 0.2.0 features and chronological snapshots to predict log1p likes, replies, and reposts. The chronological train/validation/test partitions contain 289,136/286,425/311,272 node-snapshot observations. Propagation stays within each snapshot; label views use training labels only. All three targets and seeds are included.
The table summarizes aggregate MAE for our three locally evaluated methods and six published reference methods from GraphBench v5, Table 2, with the source identified for each row.
Our aggregate MAE averages the three targets within each seed, then reports the mean and sample SD across seeds. GraphPack improves on both local controls in every target-wise R², MAE, and Spearman score. Its aggregate MAE of 0.7387 is also below the published means of DeepSets, MLP, GNN, and GCN+, supporting the competitiveness of the fixed defaults on this previously unused temporal graph.
For likes, replies, and reposts, respectively, the 100R2 results are:
Elliptic: a new financial-transaction graph. We use chronological fit/validation/test periods 1–29/30–34/35–49 on Elliptic, with the same fixed model defaults and seeds 0/1/2.
The table also includes seven independently published references from GADBench (NeurIPS 2023), Tables 4 and 13. Its tuned-method values are point estimates under the source protocol; AUPRC is computed as average precision.