- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper demonstrates the trainability issues of deeper GNNs and the vanishing gradient problem in GNNs through diagnostic experiments, then proposes an optimization technique to improve the trainability of deep GNNs through a topology-based backward pass as opposed to vanilla backpropagation.
Critical strengths
The probe experiments are interesting and contribute to identifying the vanishing gradient problem. Optimization of GNNs, on which this paper focuses, is an open area for research that should be explored more.
Critical weaknesses
-
Previous works that introduce initialization schemes for the vanishing gradient problem in GNNs [1-3] already show near-zero gradient norms in the backward pass at larger depths. Therefore, novelty claims regarding problem identification are not strong.
-
No long-range benchmarks, such as LRGB, are used for evaluation, although the paper repeatedly mentions them as a direct application of deeper GNNs. The main result in the paper is shown for L=6, although some experiments are also conducted at 10 or 20 layers. L=6 is hardly deep, and performance of KAFT deteriorates significantly at larger depths.
-
Settings where KAFT is evaluated or discussed as applicable are limited, and even under those, the improvements are only marginal. In fact, [3] shows better performance at deeper layers than reported in this paper in a similar setup without any residual connections, regularization, or normalization. Also, residual connections in GNNs not only enable identity mappings to preserve, at higher depths, the performance of smaller depths but also help in the backward pass, as corroborated by this paper, where gains shrink on the GIN architecture. There is no discussion of whether KAFT should be an alternative to residual links and why or why not. Furthermore, it is also incompatible with approaches that alter graph structure for regularization, such as DropEdge, and several rewiring approaches that may be necessary to mitigate over-squashing.
-
The authors mention "gradient oversmoothing" in the related work but do not discuss how it relates to their solution. A follow-up work on "backward oversmoothing" [4] also distinguishes it from vanishing gradients. Such a discussion is missing from this paper, and it is unclear whether the aim is to address vanishing gradients or backward oversmoothing.
-
The backward hop cap K hyperparameter is critical, but its sensitivity is only briefly analyzed in one dataset setting.
-
Missing information in the appendix (??) in several places on pages 15–19 indicates an incomplete submission.
-
Minor: The lack of spacing between table captions, subsection headings, and paragraphs hampers readability, particularly on page 7.
[1] On the Initialization of Graph Neural Networks (ICML 2023).
[2] Are GATs Out of Balance? (NeurIPS 2023).
[3] Orthogonal Graph Neural Networks (AAAI 2022).
[4] Backward Oversmoothing: Why is it hard to train deep Graph Neural Networks? (ICML 2026).
Questions for authors
-
Are the experiments full-batch? Can KAFT handle batched optimization?
-
The broader perspective shared in the concluding discussion mentions interesting applications where the gains may be meaningful. Could the authors elaborate on how these could be illustrated?
-
What is the procedure for integrating KAFT into standard GNN training in PyTorch Geometric?
12 comments
Response to Reviewer K7qV (cont. 2/6)
Critical 3. "[3] shows better performance at deeper layers than reported in this paper in a similar setup without any residual connections, regularization, or normalization."
Response: The setup of [3] is not free of regularization, and its deep results do not come from a plain stack. Three points from [3]'s own text and from the reviewer's own reference [2]:
- [3]'s abstract lists "hybrid weight initialization, orthogonal transformation, and orthogonal regularization", and its Eq. 7 adds that orthogonality penalty to the loss, with weight λ.
- Plain Ortho-GCN is reported only up to 8 layers (Table 6: 2, 4 and 8 layers). The 16–64-layer entries (Table 8) are Ortho-GCNII, which is built on GCNII, a residual backbone.
- The reviewer's reference [2] describes [3] as "a combination of orthogonal initialization and regularization."
This machinery is costly and fragile. It brings three hyper-parameters: the orthogonalization iterations T, the hybrid-initialization weight β and the penalty weight λ. [3] selects them by a random search on validation accuracy, and with its official code the result depends on that search: on CiteSeer at 8 layers, configurations with β ≥ 0.7 or T ≤ 2 reach 18–47% (general response, item 2(a)). These settings were found for BP. Comparing BP on Ortho-GCN with KAFT on a plain GCN compares two forward models; the paper compares BP and KAFT on the same forward model because that isolates the backward rule.
KAFT needs none of it. With plain Glorot initialization, no penalty and no added hyper-parameters, KAFT never collapses at 10–20 layers, where BP does in 235 of 240 runs, and gains +3.1 to +28.8 points on DBLP at 8–20 layers under L2.
The machinery stacks on KAFT. Trained on the Ortho-GCN forward model with [3]'s settings, which were tuned for BP, KAFT is within 0.6 points of BP in 11 of 12 cells (Cora and DBLP, 8–20 layers, two recipes, equal tuning budget). With β, λ, gains and feedback initialization searched for KAFT, against BP with [3]'s published settings and its own learning-rate search, KAFT is on par with BP on average (+0.33 points over 12 cells), with gains up to +5.71 points (Cora, 20 layers). On [3]'s own protocol (CiteSeer, 8 layers, up to 1500 epochs with early stopping on validation loss, each rule with its own 40-configuration search over T, β, λ and learning rate), KAFT matches BP: 66.0% against 65.2% (n.s., 10 paired seeds). BP's value there matches our run of [3]'s official code (65.17%). Our values are test accuracy at the best-validation-loss epoch; the 68.59% reported in [3] is a best-test figure. The full equal-budget tables and [3]'s official code and protocol are in the revised appendix.
[3]'s depth comes from forward machinery that has to be tuned for the training rule; KAFT removes the failure without it and, on [3]'s own protocol, matches BP.
Response to Reviewer K7qV (cont. 3/6)
Critical 4. "No long-range benchmarks, such as LRGB, are used for evaluation, although the paper repeatedly mentions them as a direct application of deeper GNNs. The main result in the paper is shown for L=6 ... L=6 is hardly deep, and performance of KAFT deteriorates significantly at larger depths."
Response: We cite LRGB (Dwivedi et al., 2022) as motivation for why depth matters, not as an evaluated domain; the revision says so. L = 5–6 is chosen as the onset of BP's failure, where accuracy is still high enough for a clean comparison over 96 settings. Depth is tested separately at 8–20 layers on plain GCNs. Under the L2 recipe KAFT's advantage grows with depth: with L2 for both rules, the DBLP gap is +3.1, +14.9, +28.7 and +28.8 points at 8, 12, 16 and 20 layers: KAFT goes from 82.1% to 74.5% and BP from 79.0% to 45.7%. BP ends at uniform predictions in 235 of 240 runs at Cora 10/20 and DBLP 16/20 layers, KAFT in none. The cause is measurable at step 0: at 10 layers BP's first-layer task signal is 0.084× the L2 term and KAFT's is 78×, so BP's step follows the regularizer. With no weight decay for either rule, KAFT is ahead by +0.9, +2.1 and +2.1 points at DBLP 8, 12 and 16 layers and by +3.2 at Cora 10 layers, all significant. Across three decay recipes (L2, decoupled, none), KAFT moves by at most 3.1 points in each of these six settings, BP by up to 35.5. On plain GCNs with Glorot initialization at 8–16 layers, BP is significantly ahead at no depth and under no recipe, and under the L2 recipe KAFT is far ahead. Without decay the margin also grows with depth on CiteSeer: with learning rates re-selected per rule on seeds 0–2 and tested on held-out seeds 3–19, KAFT is ahead by +3.8, +6.6 and +10.8 points at 6, 10 and 20 layers (BH q ≤ 0.0011); over all 18 settings of this depth sweep (CiteSeer, PubMed, DBLP; 6–20 layers; both recipes), KAFT is significantly ahead in 15 and BP in none. At 20 layers, forward-side initialization lifts KAFT further: the balanced initialization of [2] takes KAFT on Cora from 48.8% to 75.5%.
Response to Reviewer K7qV (1/6)
We thank the reviewer for a detailed review, for finding the probe experiments "interesting," and for agreeing that GNN optimization "is an open area for research that should be explored more." The review's reasons for rejection are the missing appendix values, novelty relative to [1–3], evaluation depth, and the size of the gains. We answer each below with new results.
Protocol for the numbers below. Unless stated otherwise, the BP–KAFT comparisons in this reply come from reruns in which both rules use the same decay recipe and the same saved initializations. The main recipe is the BP recipe of the submission, PyTorch Adam with L2 weight decay 5e-4; we also report no decay for both (details in the general response).
Critical 1. "Missing information in the appendix (??) in several places on pages 15–19 indicates an incomplete submission."
Response: All 41 placeholders are filled from the original run logs. They sit in Appendices I–N: the precision-audit text and Tables 14–15 (Appendix I), the DBLP split proportions (Appendix J), and learning-rate grids, depths and p-values in the tables of Appendices K–N. The full values for Tables 14–15 are in our reply to FpAz; for example, the Table 14 maximum gradient entries are 1.05e-26 to 2.09e-26. We also re-measured the precision audit on 120 runs.
Critical 2. "Previous works that introduce initialization schemes for the vanishing gradient problem in GNNs [1-3] already show near-zero gradient norms in the backward pass at larger depths. Therefore, novelty claims regarding problem identification are not strong."
Response: [1–3] and KAFT start from the same symptom and answer it in different places. [1–3] add machinery to the forward side: a new initialization, or constraints and a penalty on the weights, each with its own settings, chosen for BP. KAFT changes the backward rule and adds nothing to the forward model. Our contribution is on that backward side:
The two are complementary, not competing. The balanced initialization of [2] stacks on KAFT: it lifts KAFT from 70.0% to 78.3% on Cora at 10 layers and from 48.8% to 75.5% at 20 layers (L2 recipe), far above BP without it (32.8% and 25.6%).
All three references are now cited, and the revision states this distinction; we also removed "first" from the diagnosis claim.
[1–3] add forward machinery tuned for BP; KAFT fixes the backward side without it, and the two combine.