acceptodds
Under review as a conference paper at ICLR 2027

GNNM: Parameter-Efficient BatchEnsembles for Graph Neural Networks

Abstract

Ensembles are an effective way to improve the predictive performance of neural networks, but their computational and memory costs limit their use in many settings. Modern parameter-efficient ensembles address this issue by sharing the main model weights across ensemble members and introducing only a small number of additional parameters, yet their effectiveness in message passing architectures remains unclear. We introduce GNNM, a lightweight ensemble modification for residual GNNs that replaces only the input and output linear projectors with BatchEnsemble parameterisations while leaving the graph backbone unchanged. This design allows the same modification to be applied to different GNN models with minimal parameter overhead and without altering their message passing layers. We evaluate GNNM on five standard node classification benchmarks and compare it with both individual model baselines and explicit ensemble baselines under the same protocol for hyperparameter selection. The results show that a GNNM variant achieves the best score on every dataset. On average, GNNM improves over the individual model for every backbone and performs on par with or better than an explicit ensemble of the same size while using about four times fewer parameters, while its training cost is comparable to that of the explicit ensemble. The gains are strongest for SAGE, GAT-sep, GT, and GT-sep, while for GAT the explicit ensemble remains stronger. These results show that projector level ensembling can serve as a simple and parameter-efficient mechanism for improving residual GNNs while preserving the structure of the underlying graph model.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Graph neural networks, ancestry inference, IBD graphs, node classification, semi-inductive learning, BatchEnsemble, parameter-efficient ensembling, residual connections, graph representation learning.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KueGOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper adapts BatchEnsemble/TabM-style parameter sharing to residual GNNs. GNNM introduces member-specific multiplicative factors and biases only in the input and output linear projectors, while sharing message-passing parameters. Each member maintains separate hidden representations and passes sequentially through the shared backbone. Training averages member losses and accumulates gradients before updating parameters; inference averages raw logits. GNNM used nearly as few parameters as a single model and approximately one-quarter as many as four-member ENS.

Critical strengths
  1. The method integrates readily with different backbones and clearly reduces parameter storage. GNNM changes only the input and output projectors, preserving the message-passing rules of existing residual GNNs. Its parameter count remains close to a single model rather than four independent copies. With SAGE, GT, and GT-sep, GNNM outperformed both BASE and ENS on all five tasks. These results support the practical value of parameter sharing for some graph backbones.
Critical weaknesses
  1. The two-projector design lacks the closest controls. Section 2.3 discusses shallow ensembles, but the main comparisons include only BASE and independent ENS. Output-only heads, input-only modulation, and BatchEnsemble within message-passing layers are missing. Without these controls, the results do not establish the additional value of modulating both projectors over simpler shared ensembles or justify their placement.

  2. The graph-specific motivation lacks evidence about member diversity. The paper argues that message passing may reduce diversity but does not measure member correlation, disagreement, representation similarity, or gains from ensembling individual members. Nor does it connect these measures to graph structure or depth. The unexplained differences between SAGE/GT and weaker GAT configurations limit understanding of when GNNM should work.

  3. Parameter savings do not establish practical cost advantages. Tables 6 and 8 show approximately fourfold parameter savings, but slightly slower training steps and similar peak memory. Costs average eight backbones and five depths rather than Table 1’s selected configurations. Matched accuracy–training/inference cost comparisons, including cheaper shared-representation heads, are needed to identify the settings where GNNM offers a practical advantage.

  4. The ablations do not isolate the proposed initialization mechanism. Ensemble-size and initialization experiments cover only Roman-empire, three backbones, five layers, and five splits. Xavier-all changes both member diversity and activation scale. Its weaker results do not isolate the roles of first-layer diversity and identity-like output initialization. Excluding weaker GCN/GAT configurations also limits explanations of the main results.

Questions for authors
  1. Under the same backbone and training/selection protocol, how do output-only shared-representation heads, input-only modulation, full input+output GNNM, and broader BatchEnsemble placements compare? Which results establish the additional value of modulating both projectors over simpler shared ensembles?

  2. Could you report member correlation/disagreement and ensemble gains over individual members, relating these measures to backbone, depth, or graph structure? Can these statistics explain the more consistent gains with SAGE/GT and weaker results with GAT, supporting the motivation about message passing and diversity?

  3. When initial activation scales are kept comparable, how do changes to first-layer random signs and identity-like output initialization affect results? Do experiments including weaker GAT/GCN configuration show that initialization or ensemble size affects the main findings beyond the three backbones tested on Roman-empire?

1 comment
AuthorsOpenReview ↗

Response to Reviewer KueG

Response to Reviewer KueG

We thank the reviewer for the detailed review and the precise questions, which led to most of the new experiments.

W1 / Q1. Closest architectural comparisons and broader placements

All three reviewers asked what each projector contributes. Our response to Gv82 presents the ablations in Table R3, including the variant with BatchEnsemble in the backbone’s linear layers.


W2 / Q2. Member diversity

We now measure member diversity (Section 5.3) on validation nodes from the first split, with each method at its validation-selected depth. For each pair of members, disagreement is the percentage of nodes assigned different labels. Error correlation is the Pearson correlation between their error indicators (1 for a wrong prediction, 0 for a correct one). We average both metrics over the six member pairs.

MetricENSGNNM
Pairwise disagreement9.0%9.2%
Error correlation0.790.78
Ensemble gain over mean member (points)+0.63+1.06

Table R4. Means across five datasets and seven backbones. Ensemble gain is the validation score from averaging the members’ logits minus the mean of their individual scores. GNNM has a larger gain in 27 of 35 backbone–dataset combinations.

Across datasets, the ensemble gain is largest on Amazon-ratings and smallest on Minesweeper. Appendix G reports that the mean GNNM advantage over ENS rises from +0.12 points at one layer to +0.37 at four layers.


W3. Practical cost

Reviewer Gv82 summarizes the trade-off as GNNM “remains competitive with explicit ensembles using substantially fewer parameters.” We agree, and we see the main benefit of GNNM in its accuracy rather than in its cost.

We trained all methods to convergence using the same rule for early stopping (Table 1 of the revised paper). Under this protocol, GNNM is on average more accurate than the explicit ensemble (+0.41 points over ENS and +1.07 over BASE in Table 1) while using one quarter of its parameters. Its training step takes about as long as that of the ensemble with four members (median ratio 1.00 when both have the same depth). Training speed is not the claim of the paper. The benefit of GNNM is a more accurate model with one quarter of the parameters.

The model with four readout heads costs less because the heads share one pass through the backbone. For a five-layer GAT on Roman-empire, one training step takes 31.9 ms for BASE and 33.0 ms for the model with four readout heads. It takes 127.6 ms for ENS, which updates all four members, and 126.6 ms for GNNM. It scores 81.46, compared with 81.94 for the single model and 83.99 for GNNM (Table R3 in our response to Gv82). A cheaper shared model is therefore available, but it does not provide the accuracy gain of ensembling.


W4 / Q3. Isolating the initialization

On Roman-empire, the new ablation covers all seven backbones, including GCN and GAT, all 10 splits, and depths from one to five layers. The earlier ablation used three backbones, a depth of five layers, and five splits. Table 5 of the revised paper reports the scores below. The BASE, full GNNM, and output projector results also appear in Table R3 of our response to Gv82.

  • Comparable activation scales. We rescaled the Xavier-initialized factors to unit root mean square, matching the scale of our ±1 factors. The rescaled variant reaches 82.43, compared with 82.56 for Xavier-all and 83.99 for our initialization. The gap persists after matching the scales.
  • First-layer random signs. Initializing all factors to one makes the members identical at the start and drops accuracy to 81.55, the level of BASE. This points to the importance of member-specific variation at initialization.
  • Identity-like output initialization. With modulation in the output projector only, identity-like (r = 1, 81.49) and random-sign (r = ±1, 81.29) initialization perform similarly and both stay at the level of BASE. The results point to first-layer diversity as the important component.
  • Ensemble size on all backbones. Accuracy grows monotonically across ensemble sizes. It reaches 81.47, 82.59, 83.99, and 84.64 for M = 1, 2, 4, and 8. M = 8 improves over M = 4 on six of seven backbones, with GAT as the exception. A single member performs at the level of BASE, suggesting that the gain comes from combining members.
Reviewer_Gv82OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether GNN ensembles can improve prediction without duplicating the full model’s parameters. It proposes GNNM, which applies BatchEnsemble to the input and output projectors while sharing all message-passing parameters. Members are trained jointly with member-specific modulation vectors, and their logits are averaged at inference.

The evidence is primarily empirical, rather than based on theoretical guarantees. Experiments compare GNNM with single models and four-member explicit ensembles across seven backbone families and five heterophilous node-classification datasets, using ten official splits. Additional studies examine ensemble size, initialization, computational costs, and one graph-classification task. GNNM improves average performance over single models and remains competitive with explicit ensembles while using approximately one quarter of their parameters. Gains vary across backbones, and superiority over explicit ensembles is not robustly established after accounting for dependencies across results. Training time remains comparable to explicit ensembling, so the main benefit is parameter efficiency rather than computational acceleration.

Critical strengths
  1. Clear and practical motivation. The paper addresses a well-defined question: whether GNNs can retain the predictive benefits of ensembling without duplicating the full model’s parameters.
  2. Simple and effective approach. The projector-level BatchEnsemble modification is lightweight and reusable across different backbones. Despite its simplicity, it improves over single models in many settings and remains competitive with explicit ensembles using substantially fewer parameters.
  3. Clear presentation. The paper is well organized and easy to follow. The architecture, training procedure, and distinction between parameter sharing and computational savings are clearly explained.
Critical weaknesses

The central architectural choice is not sufficiently isolated. The ablations vary ensemble size and initialization but do not compare input-only, output-only, and input-plus-output designs. In particular, a shared backbone with multiple output heads could provide a simpler and computationally cheaper alternative. These controls would clarify whether member-specific transformations at both ends are necessary and what additional benefit the proposed two-projector design provides.

Questions for authors

My concerns are in weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer Gv82

Response to Reviewer Gv82

We thank the reviewer for the positive assessment of the motivation, simplicity, and presentation.

Isolating the central architectural choice

All three reviewers (t16d, Gv82, and KueG) asked what each projector contributes. Table R3 reports mean test accuracy across seven backbones on Roman-empire (Section 5.2 and Table 5 of the revised paper).

VariantMean test accuracy (%)
BASE81.94
ENS (M = 4)82.89
GNNM (input + output)83.99
Input projector only84.00
Output projector only (r = 1 / r = ±1)81.49 / 81.29
GNNM + BatchEnsemble on backbone linear layers83.94
Shared backbone + four linear input–output pairs82.44
Shared backbone + four linear readout heads (one pass)81.46

Table R3. Roman-empire architecture ablations. For each variant, backbone, and split, we selected the best depth (1–5 layers) by validation accuracy. We averaged the corresponding test accuracies over 10 splits and seven backbones. Bold marks the highest mean. Underlining marks the submitted GNNM setup.

In the BatchEnsemble input projector, each of the four members uses its own learned scales on the node features before and after the shared linear map. This gives each member a different representation at the input to the shared backbone. Input-only modulation matches full GNNM, while output-only modulation stays at the level of BASE. Adding BatchEnsemble factors to every linear layer in the backbone while keeping GNNM's input and output projectors reaches similar accuracy to full GNNM. The independent-projector variant uses one pair of ordinary linear input and output projectors per member and scores 1.55 points below GNNM. The single-pass variant uses one shared linear input projector and one backbone pass to feed four linear readout heads, with accuracy at the level of a single model.

Some of the ablated variants are indeed cheaper. The model with four readout heads runs the backbone once. For a five-layer GAT on Roman-empire, its training step takes 33.0 ms, close to 31.9 ms for BASE, against 127.6 ms for ENS, which updates all four members, and 126.6 ms for GNNM. GNNM with two instances takes 63.9 ms, and a single instance runs one pass like BASE. These faster variants do not pay off in accuracy. Averaged over the seven backbones, the readout heads reach 81.46, a single instance 81.47, and two instances 82.59, all below ENS (82.89) and GNNM (83.99) (Table R3 and Table 5 of the revised paper).

We also added an analysis of member diversity (Section 5.3). Although the backbone is shared, GNNM members are about as diverse as independently trained models (pairwise disagreement 9.2% vs. 9.0%, error correlation 0.78 vs. 0.79). Averaging them yields a larger gain (+1.06 vs. +0.63 points over the mean member).

Reviewer_t16dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GNNM, which applies BatchEnsemble-style modulation to the input and output projectors while sharing the message-passing backbone across ensemble members. The authors compare GNNM with single models and four-member explicit ensembles across seven backbones and five node-classification datasets. The results suggest that GNNM can achieve performance close to explicit ensembles with roughly one quarter of their parameters, while requiring similar training time.

Critical strengths
  1. The parameter savings are clear. GNNM keeps its parameter count close to that of a single model while achieving performance comparable to four-member explicit ensembles in the tested settings.

  2. The statistical analysis is careful about dependencies between datasets and backbones.

Critical weaknesses
  1. The contribution of the two-projector design is not sufficiently established. The ablations examine ensemble size and initialization, but do not compare input-only modulation, output-only modulation, or a shared backbone with multiple ordinary readout heads. Given existing BatchEnsemble and shallow GNN ensemble methods, these comparisons would help establish whether both projectors are needed and what they add over simpler sharing schemes.

  2. The conclusions may depend on the training settings. Appendix A fixes learning rate, width, dropout, and weight decay, while mainly selecting depth. This is a reasonable controlled comparison, but different sharing schemes may favor different optimization and regularization settings. A modest, budget-matched search would help establish whether the reported performance relationships are robust.

  3. The evaluation covers a relatively narrow range of tasks. Testing several backbones is useful, but does not replace broader dataset coverage. More diverse graph structures and prediction tasks would strengthen the case that the method is broadly applicable.

Questions for authors

See Weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer t16d

Response to Reviewer t16d

Thank you for the review.

W1. Contribution of the two-projector design

All three reviewers asked whether both projectors are needed. Our response to Gv82 reports the ablations in Table R3, including the comparison with ordinary readout heads.


W2. Dependence on training settings

We agree that a fixed configuration may favour one sharing scheme. We ran the largest search on GAT because, as an attention backbone, it has more hyperparameters to tune: attention dropout and the number of heads in addition to learning rate, weight decay, and dropout. This gives the largest grid, with 24 configurations. We then took the two pairs of learning rate and weight decay that ranked best on GAT, (0.0001, 0) and (0.0003, 0.01), and transferred them to the other six backbones together with the default (0.00003, 0). On each dataset and backbone, each method selected its settings and a depth from one to five layers using validation splits 0–2.

DatasetGap in Table 1Tuned gap (backbones where GNNM leads)
Roman Empire+1.10+0.59 (5/7)
Amazon Ratings+0.08+0.08 (5/7)
Minesweeper−0.38−0.44 (1/7)
Tolokers+0.16+0.25 (5/7)
Questions+1.08+0.85 (7/7)

Table R1. Each gap is GNNM minus ENS in percentage points, averaged over the seven backbones.


W3. Range of tasks

We evaluated two tasks beyond node classification, link prediction and graph classification. The graph classification evaluation includes molecular property prediction (Section 5.4 and Appendix F).

We used the published Open Graph Benchmark data and splits for ogbl-collab and ogbg-molbace [1]. For Collab, we used GCN and GraphSAGE from the OGB example and BUDDY from the authors' implementation. All three are benchmarked together on Collab in [2]. For MolBACE, we used the GCN from the OGB molecular example.

For PROTEINS from TUDataset [3], we report the GraphSAGE comparison with global mean pooling from our submission, using the same ten outer folds.

We applied GNNM to these existing backbones and compared it with the single model and an ordinary four-member ensemble on the same data and splits.

DatasetTask typeBackboneMetricBASEENSGNNM
ogbl-collabLink predictionGCNHits@2047.7751.9653.59
ogbl-collabLink predictionGCNHits@5058.0760.4161.45
ogbl-collabLink predictionGCNHits@10063.8366.3667.24
ogbl-collabLink predictionGraphSAGEHits@2047.8251.4452.59
ogbl-collabLink predictionGraphSAGEHits@5057.9060.5560.55
ogbl-collabLink predictionGraphSAGEHits@10064.1265.9366.18
ogbl-collabLink predictionBUDDYHits@2058.7559.0858.30
ogbl-collabLink predictionBUDDYHits@5065.6866.5266.50
ogbl-collabLink predictionBUDDYHits@10070.0470.5470.75
ogbg-molbaceGraph classificationGCNROC-AUC80.6082.3782.71
PROTEINSGraph classificationGraphSAGEAccuracy73.3572.9073.62

Table R2. Mean test scores over five seeds per Collab backbone, ten seeds for MolBACE, and ten outer folds for PROTEINS. GNNM leads on both graph classification datasets and most link prediction comparisons.

On MolBACE, shared BatchNorm mixed running statistics across members. We gave each member its own running mean and variance while keeping the affine parameters shared. See Appendix F.

References

[1] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open Graph Benchmark: Datasets for Machine Learning on Graphs. NeurIPS, 2020.

[2] Benjamin Paul Chamberlain, Sergey Shirobokov, Emanuele Rossi, Fabrizio Frasca, Thomas Markovich, Nils Hammerla, Michael M. Bronstein, and Max Hansmire. Graph Neural Networks for Link Prediction with Subgraph Sketching. ICLR, 2023.

[3] Christopher Morris, Nils M. Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann. TUDataset: A collection of benchmark datasets for learning with graphs. ICML Workshop on Graph Representation Learning and Beyond, 2020.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.