- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper adapts BatchEnsemble/TabM-style parameter sharing to residual GNNs. GNNM introduces member-specific multiplicative factors and biases only in the input and output linear projectors, while sharing message-passing parameters. Each member maintains separate hidden representations and passes sequentially through the shared backbone. Training averages member losses and accumulates gradients before updating parameters; inference averages raw logits. GNNM used nearly as few parameters as a single model and approximately one-quarter as many as four-member ENS.
Critical strengths
- The method integrates readily with different backbones and clearly reduces parameter storage. GNNM changes only the input and output projectors, preserving the message-passing rules of existing residual GNNs. Its parameter count remains close to a single model rather than four independent copies. With SAGE, GT, and GT-sep, GNNM outperformed both BASE and ENS on all five tasks. These results support the practical value of parameter sharing for some graph backbones.
Critical weaknesses
-
The two-projector design lacks the closest controls. Section 2.3 discusses shallow ensembles, but the main comparisons include only BASE and independent ENS. Output-only heads, input-only modulation, and BatchEnsemble within message-passing layers are missing. Without these controls, the results do not establish the additional value of modulating both projectors over simpler shared ensembles or justify their placement.
-
The graph-specific motivation lacks evidence about member diversity. The paper argues that message passing may reduce diversity but does not measure member correlation, disagreement, representation similarity, or gains from ensembling individual members. Nor does it connect these measures to graph structure or depth. The unexplained differences between SAGE/GT and weaker GAT configurations limit understanding of when GNNM should work.
-
Parameter savings do not establish practical cost advantages. Tables 6 and 8 show approximately fourfold parameter savings, but slightly slower training steps and similar peak memory. Costs average eight backbones and five depths rather than Table 1’s selected configurations. Matched accuracy–training/inference cost comparisons, including cheaper shared-representation heads, are needed to identify the settings where GNNM offers a practical advantage.
-
The ablations do not isolate the proposed initialization mechanism. Ensemble-size and initialization experiments cover only Roman-empire, three backbones, five layers, and five splits. Xavier-all changes both member diversity and activation scale. Its weaker results do not isolate the roles of first-layer diversity and identity-like output initialization. Excluding weaker GCN/GAT configurations also limits explanations of the main results.
Questions for authors
-
Under the same backbone and training/selection protocol, how do output-only shared-representation heads, input-only modulation, full input+output GNNM, and broader BatchEnsemble placements compare? Which results establish the additional value of modulating both projectors over simpler shared ensembles?
-
Could you report member correlation/disagreement and ensemble gains over individual members, relating these measures to backbone, depth, or graph structure? Can these statistics explain the more consistent gains with SAGE/GT and weaker results with GAT, supporting the motivation about message passing and diversity?
-
When initial activation scales are kept comparable, how do changes to first-layer random signs and identity-like output initialization affect results? Do experiments including weaker GAT/GCN configuration show that initialization or ensemble size affects the main findings beyond the three backbones tested on Roman-empire?
Response to Reviewer KueG
Response to Reviewer KueG
We thank the reviewer for the detailed review and the precise questions, which led to most of the new experiments.
W1 / Q1. Closest architectural comparisons and broader placements
All three reviewers asked what each projector contributes. Our response to Gv82 presents the ablations in Table R3, including the variant with BatchEnsemble in the backbone’s linear layers.
W2 / Q2. Member diversity
We now measure member diversity (Section 5.3) on validation nodes from the first split, with each method at its validation-selected depth. For each pair of members, disagreement is the percentage of nodes assigned different labels. Error correlation is the Pearson correlation between their error indicators (1 for a wrong prediction, 0 for a correct one). We average both metrics over the six member pairs.
Table R4. Means across five datasets and seven backbones. Ensemble gain is the validation score from averaging the members’ logits minus the mean of their individual scores. GNNM has a larger gain in 27 of 35 backbone–dataset combinations.
Across datasets, the ensemble gain is largest on Amazon-ratings and smallest on Minesweeper. Appendix G reports that the mean GNNM advantage over ENS rises from +0.12 points at one layer to +0.37 at four layers.
W3. Practical cost
Reviewer Gv82 summarizes the trade-off as GNNM “remains competitive with explicit ensembles using substantially fewer parameters.” We agree, and we see the main benefit of GNNM in its accuracy rather than in its cost.
We trained all methods to convergence using the same rule for early stopping (Table 1 of the revised paper). Under this protocol, GNNM is on average more accurate than the explicit ensemble (+0.41 points over ENS and +1.07 over BASE in Table 1) while using one quarter of its parameters. Its training step takes about as long as that of the ensemble with four members (median ratio 1.00 when both have the same depth). Training speed is not the claim of the paper. The benefit of GNNM is a more accurate model with one quarter of the parameters.
The model with four readout heads costs less because the heads share one pass through the backbone. For a five-layer GAT on Roman-empire, one training step takes 31.9 ms for BASE and 33.0 ms for the model with four readout heads. It takes 127.6 ms for ENS, which updates all four members, and 126.6 ms for GNNM. It scores 81.46, compared with 81.94 for the single model and 83.99 for GNNM (Table R3 in our response to Gv82). A cheaper shared model is therefore available, but it does not provide the accuracy gain of ensembling.
W4 / Q3. Isolating the initialization
On Roman-empire, the new ablation covers all seven backbones, including GCN and GAT, all 10 splits, and depths from one to five layers. The earlier ablation used three backbones, a depth of five layers, and five splits. Table 5 of the revised paper reports the scores below. The BASE, full GNNM, and output projector results also appear in Table R3 of our response to Gv82.