acceptodds
Under review as a conference paper at ICLR 2027

Deep Message-Passing GNNs Have a Backward Problem: Kronecker-Aligned Feedback Training

Abstract

Efforts to train deep message-passing graph neural networks have largely focused on modifying the forward computation to mitigate oversmoothing. However, preserving informative hidden representations alone does not ensure effective learning: supervision can still attenuate through the backward chain of graph propagation, feature transformations, and activation gates. In plain ten-layer GCN and GraphSAGE models on three citation graphs, backpropagation (BP) produces hidden-layer gradients too small to change any FP32 parameter, while frozen probes on the same hidden states remain well above chance. This motivates treating backward credit assignment as an explicit design choice. We introduce Kronecker-Aligned Feedback Training (KAFT), which exploits the Kronecker structure of graph-convolution Jacobians to deliver graph-aware feedback directly to hidden layers, leaving the forward and inference computation unchanged. KAFT combines a capped operator on the forward graph with learned feature-feedback matrices, aligned to directional targets obtained by propagating and normalizing probes through downstream weights. We show that this target satisfies a positive-semidefinite alignment property and give sufficient conditions under which the KAFT update descends and converges to stationarity. Across four node-classification datasets, four backbones, two depths, and three learning rates, KAFT improves accuracy over BP by 3.1 percentage points on average and is significantly better in 76 of 96 matched settings and significantly worse in 4; gains shrink on backbones with built-in identity paths such as GIN. The benefit grows once depth exceeds the point where BP stops updating: on DBLP it rises from 4.3 points at 8 layers to 30.4 points at 20 layers. With BatchNorm or LayerNorm in the same forward model, KAFT is significantly better in four of six comparisons and never significantly worse. In preliminary three-seed experiments, KAFT raises Cora link-prediction ROC-AUC from 64.5% to 81.4% with a six-layer encoder and matches BP on mini-batch graph classification. These results suggest that the backward rule, not only the forward operator, is a useful design axis for deep message-passing GNNs.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
graph neural networks, feedback alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_K7qVOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper demonstrates the trainability issues of deeper GNNs and the vanishing gradient problem in GNNs through diagnostic experiments, then proposes an optimization technique to improve the trainability of deep GNNs through a topology-based backward pass as opposed to vanilla backpropagation.

Critical strengths

The probe experiments are interesting and contribute to identifying the vanishing gradient problem. Optimization of GNNs, on which this paper focuses, is an open area for research that should be explored more.

Critical weaknesses
  1. Previous works that introduce initialization schemes for the vanishing gradient problem in GNNs [1-3] already show near-zero gradient norms in the backward pass at larger depths. Therefore, novelty claims regarding problem identification are not strong.

  2. No long-range benchmarks, such as LRGB, are used for evaluation, although the paper repeatedly mentions them as a direct application of deeper GNNs. The main result in the paper is shown for L=6, although some experiments are also conducted at 10 or 20 layers. L=6 is hardly deep, and performance of KAFT deteriorates significantly at larger depths.

  3. Settings where KAFT is evaluated or discussed as applicable are limited, and even under those, the improvements are only marginal. In fact, [3] shows better performance at deeper layers than reported in this paper in a similar setup without any residual connections, regularization, or normalization. Also, residual connections in GNNs not only enable identity mappings to preserve, at higher depths, the performance of smaller depths but also help in the backward pass, as corroborated by this paper, where gains shrink on the GIN architecture. There is no discussion of whether KAFT should be an alternative to residual links and why or why not. Furthermore, it is also incompatible with approaches that alter graph structure for regularization, such as DropEdge, and several rewiring approaches that may be necessary to mitigate over-squashing.

  4. The authors mention "gradient oversmoothing" in the related work but do not discuss how it relates to their solution. A follow-up work on "backward oversmoothing" [4] also distinguishes it from vanishing gradients. Such a discussion is missing from this paper, and it is unclear whether the aim is to address vanishing gradients or backward oversmoothing.

  5. The backward hop cap K hyperparameter is critical, but its sensitivity is only briefly analyzed in one dataset setting.

  6. Missing information in the appendix (??) in several places on pages 15–19 indicates an incomplete submission.

  7. Minor: The lack of spacing between table captions, subsection headings, and paragraphs hampers readability, particularly on page 7.

[1] On the Initialization of Graph Neural Networks (ICML 2023).

[2] Are GATs Out of Balance? (NeurIPS 2023).

[3] Orthogonal Graph Neural Networks (AAAI 2022).

[4] Backward Oversmoothing: Why is it hard to train deep Graph Neural Networks? (ICML 2026).

Questions for authors
  1. Are the experiments full-batch? Can KAFT handle batched optimization?

  2. The broader perspective shared in the concluding discussion mentions interesting applications where the gains may be meaningful. Could the authors elaborate on how these could be illustrated?

  3. What is the procedure for integrating KAFT into standard GNN training in PyTorch Geometric?

12 comments
AuthorsOpenReview ↗

Response to Reviewer K7qV (1/6)

We thank the reviewer for a detailed review, for finding the probe experiments "interesting," and for agreeing that GNN optimization "is an open area for research that should be explored more." The review's reasons for rejection are the missing appendix values, novelty relative to [1–3], evaluation depth, and the size of the gains. We answer each below with new results.

Protocol for the numbers below. Unless stated otherwise, the BP–KAFT comparisons in this reply come from reruns in which both rules use the same decay recipe and the same saved initializations. The main recipe is the BP recipe of the submission, PyTorch Adam with L2 weight decay 5e-4; we also report no decay for both (details in the general response).

Critical 1. "Missing information in the appendix (??) in several places on pages 15–19 indicates an incomplete submission."

Response: All 41 placeholders are filled from the original run logs. They sit in Appendices I–N: the precision-audit text and Tables 14–15 (Appendix I), the DBLP split proportions (Appendix J), and learning-rate grids, depths and p-values in the tables of Appendices K–N. The full values for Tables 14–15 are in our reply to FpAz; for example, the Table 14 maximum gradient entries are 1.05e-26 to 2.09e-26. We also re-measured the precision audit on 120 runs.

Critical 2. "Previous works that introduce initialization schemes for the vanishing gradient problem in GNNs [1-3] already show near-zero gradient norms in the backward pass at larger depths. Therefore, novelty claims regarding problem identification are not strong."

Response: [1–3] and KAFT start from the same symptom and answer it in different places. [1–3] add machinery to the forward side: a new initialization, or constraints and a penalty on the weights, each with its own settings, chosen for BP. KAFT changes the backward rule and adds nothing to the forward model. Our contribution is on that backward side:

  • A diagnosis on the same checkpoints. Frozen probes on BP's penultimate hidden state stay far above chance (36.9–63.6% at 10 layers against chance levels of 14–33%, Table 14), while the current task gradient changes no element of BP's next optimizer step (120 of 120 runs). The class information is in the forward pass; the optimizer step carries zero task signal. Near-zero gradient norms do not show this; the probe-versus-step test on the same checkpoint does.
  • The collapse, quantified. Weight over-decaying under L2 is known (Lu et al., 2024; Cong et al., 2021); we measure it on the same model for both rules. Under Adam with L2, BP's first-layer task signal is 0.084× the regularizer at step 0, and KAFT's is 78×; BP then ends at uniform predictions in 235 of 240 deep runs, KAFT in none.
  • A fix that needs no forward machinery. With plain Glorot initialization and no penalty, KAFT gains +3.1 to +28.8 points over BP on DBLP at 8–20 layers under L2, and +3.19 points on average over the 96 settings of Table 2 (72 settings significant for KAFT, 4 for BP).

The two are complementary, not competing. The balanced initialization of [2] stacks on KAFT: it lifts KAFT from 70.0% to 78.3% on Cora at 10 layers and from 48.8% to 75.5% at 20 layers (L2 recipe), far above BP without it (32.8% and 25.6%).

All three references are now cited, and the revision states this distinction; we also removed "first" from the diagnosis claim.

[1–3] add forward machinery tuned for BP; KAFT fixes the backward side without it, and the two combine.

AuthorsOpenReview ↗

Response to Reviewer K7qV (cont. 2/6)

Critical 3. "[3] shows better performance at deeper layers than reported in this paper in a similar setup without any residual connections, regularization, or normalization."

Response: The setup of [3] is not free of regularization, and its deep results do not come from a plain stack. Three points from [3]'s own text and from the reviewer's own reference [2]:

  • [3]'s abstract lists "hybrid weight initialization, orthogonal transformation, and orthogonal regularization", and its Eq. 7 adds that orthogonality penalty to the loss, with weight λ.
  • Plain Ortho-GCN is reported only up to 8 layers (Table 6: 2, 4 and 8 layers). The 16–64-layer entries (Table 8) are Ortho-GCNII, which is built on GCNII, a residual backbone.
  • The reviewer's reference [2] describes [3] as "a combination of orthogonal initialization and regularization."

This machinery is costly and fragile. It brings three hyper-parameters: the orthogonalization iterations T, the hybrid-initialization weight β and the penalty weight λ. [3] selects them by a random search on validation accuracy, and with its official code the result depends on that search: on CiteSeer at 8 layers, configurations with β ≥ 0.7 or T ≤ 2 reach 18–47% (general response, item 2(a)). These settings were found for BP. Comparing BP on Ortho-GCN with KAFT on a plain GCN compares two forward models; the paper compares BP and KAFT on the same forward model because that isolates the backward rule.

KAFT needs none of it. With plain Glorot initialization, no penalty and no added hyper-parameters, KAFT never collapses at 10–20 layers, where BP does in 235 of 240 runs, and gains +3.1 to +28.8 points on DBLP at 8–20 layers under L2.

The machinery stacks on KAFT. Trained on the Ortho-GCN forward model with [3]'s settings, which were tuned for BP, KAFT is within 0.6 points of BP in 11 of 12 cells (Cora and DBLP, 8–20 layers, two recipes, equal tuning budget). With β, λ, gains and feedback initialization searched for KAFT, against BP with [3]'s published settings and its own learning-rate search, KAFT is on par with BP on average (+0.33 points over 12 cells), with gains up to +5.71 points (Cora, 20 layers). On [3]'s own protocol (CiteSeer, 8 layers, up to 1500 epochs with early stopping on validation loss, each rule with its own 40-configuration search over T, β, λ and learning rate), KAFT matches BP: 66.0% against 65.2% (n.s., 10 paired seeds). BP's value there matches our run of [3]'s official code (65.17%). Our values are test accuracy at the best-validation-loss epoch; the 68.59% reported in [3] is a best-test figure. The full equal-budget tables and [3]'s official code and protocol are in the revised appendix.

[3]'s depth comes from forward machinery that has to be tuned for the training rule; KAFT removes the failure without it and, on [3]'s own protocol, matches BP.

AuthorsOpenReview ↗

Response to Reviewer K7qV (cont. 3/6)

Critical 4. "No long-range benchmarks, such as LRGB, are used for evaluation, although the paper repeatedly mentions them as a direct application of deeper GNNs. The main result in the paper is shown for L=6 ... L=6 is hardly deep, and performance of KAFT deteriorates significantly at larger depths."

Response: We cite LRGB (Dwivedi et al., 2022) as motivation for why depth matters, not as an evaluated domain; the revision says so. L = 5–6 is chosen as the onset of BP's failure, where accuracy is still high enough for a clean comparison over 96 settings. Depth is tested separately at 8–20 layers on plain GCNs. Under the L2 recipe KAFT's advantage grows with depth: with L2 for both rules, the DBLP gap is +3.1, +14.9, +28.7 and +28.8 points at 8, 12, 16 and 20 layers: KAFT goes from 82.1% to 74.5% and BP from 79.0% to 45.7%. BP ends at uniform predictions in 235 of 240 runs at Cora 10/20 and DBLP 16/20 layers, KAFT in none. The cause is measurable at step 0: at 10 layers BP's first-layer task signal is 0.084× the L2 term and KAFT's is 78×, so BP's step follows the regularizer. With no weight decay for either rule, KAFT is ahead by +0.9, +2.1 and +2.1 points at DBLP 8, 12 and 16 layers and by +3.2 at Cora 10 layers, all significant. Across three decay recipes (L2, decoupled, none), KAFT moves by at most 3.1 points in each of these six settings, BP by up to 35.5. On plain GCNs with Glorot initialization at 8–16 layers, BP is significantly ahead at no depth and under no recipe, and under the L2 recipe KAFT is far ahead. Without decay the margin also grows with depth on CiteSeer: with learning rates re-selected per rule on seeds 0–2 and tested on held-out seeds 3–19, KAFT is ahead by +3.8, +6.6 and +10.8 points at 6, 10 and 20 layers (BH q ≤ 0.0011); over all 18 settings of this depth sweep (CiteSeer, PubMed, DBLP; 6–20 layers; both recipes), KAFT is significantly ahead in 15 and BP in none. At 20 layers, forward-side initialization lifts KAFT further: the balanced initialization of [2] takes KAFT on Cora from 48.8% to 75.5%.

Reviewer_FpAzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies backward-signal attenuation in deep message-passing GNNs. It reports that hidden representations can retain probe-recoverable class information even when backpropagation produces numerically ineffective parameter updates. Motivated by the Kronecker factorization of graph-convolution Jacobians, it proposes Kronecker-Aligned Feedback Training (KAFT). KAFT preserves the forward architecture while replacing hidden-layer backpropagation with graph-structured feedback and learned feature-feedback matrices aligned to chain-normalized probe targets. Experiments evaluate node classification across several backbones and depths, compatibility with forward stabilizers, topology controls, and preliminary applications beyond node classification. The appendix establishes a PSD alignment property and conditional descent guarantees.

Critical strengths
  1. A motivated intervention in backward credit assignment. The paper connects the graph-convolution Jacobian’s structure to a concrete alternative feedback rule. The chain-normalized probe construction provides a principled way to learn feature-feedback directions, and its PSD alignment property gives useful mathematical support. The topology controls provide evidence that correspondence between the forward and backward graphs matters.

  2. Substantial empirical investigation under matched forward architectures. The evaluation includes multiple backbones, depths, learning rates, and predominantly 20-seed comparisons, together with normalization, identity-path, and graph-operator ablations. Improvements persist in some normalized models: for example, KAFT improves the ten-layer LayerNorm GCN on Cora by 2.35 percentage points. These comparisons strengthen the evidence that backward-rule design can contribute beyond training an unstabilized backbone.

Critical weaknesses
  1. The central diagnostic remains incompletely substantiated and does not fully isolate backward transport from numerical scaling. Appendix I appropriately distinguishes a zero FP32 norm from elementwise-zero gradients and reports actual parameter-update checks. However, Table 14 leaves the FP64 maximum gradients and output-adjacent errors as “??”, while Table 15 omits the depths defining the claimed transport boundaries. These are quantities needed to assess the principal diagnosis. Furthermore, above-chance probes on standardized representations establish recoverable class structure, but do not isolate the cause of training failure: Appendix I acknowledges substantial forward-activation attenuation as well. A comparison with simpler BP gradient-rescaling or layerwise-normalization controls would help determine whether KAFT’s structural feedback direction is essential beyond restoring update magnitude. This uncertainty directly affects the paper’s explanatory contribution.

  2. The evidence for the specific alignment mechanism is weaker than the evidence that the overall method improves selected depth-stressed models. Comparisons with DFA and DFA-GNN are useful, but the paper does not sufficiently separate the benefit of learned directional alignment from changes in feedback scale and graph-side transport. Matched-scale random or fixed feedback controls would make this distinction clearer. The theoretical results only partly resolve this issue: the PSD cross-kernel result applies to a matched linear setting, while the practical method includes nonlinear gates, capped propagation, diffusion, approximate feedback tracking, and Adam. Appendix Q explicitly makes descent conditional on alignment and perturbation-margin assumptions, without establishing that these conditions hold in the principal deep-network experiments. Consequently, the proposed mechanism is plausible, but its necessity and theoretical coverage remain insufficiently demonstrated.

Questions for authors
  1. Can you provide the missing numerical values in Tables 14–15 and specify the initialization and exact optimizer-update procedure used in the precision audit, including the treatment of Adam moments and weight decay?

  2. How does KAFT compare with BP using layerwise gradient normalization or rescaling, and with random or fixed feedback matched to KAFT’s feedback magnitude? These controls would help distinguish structural alignment from scale restoration.

  3. Which alignment or perturbation-margin conditions from Appendix Q have been measured for the practical nonlinear algorithm, especially beyond the BP transport boundary? If they have not been verified, how should the theoretical guarantee be interpreted relative to the empirical claims?

4 comments
AuthorsOpenReview ↗

Response to Reviewer FpAz (1/4)

We thank the reviewer for a precise and fair review, and for naming the exact controls that test our claims. We are glad the reviewer finds KAFT "a motivated intervention in backward credit assignment," the chain-normalized probe construction "a principled way to learn feature-feedback directions," the topology controls "evidence that correspondence between the forward and backward graphs matters," and the evaluation a "substantial empirical investigation under matched forward architectures." We ran the controls the reviewer asked for. The reviewer's two main concerns are that the diagnosis does not separate backward transport from numerical scale, and that the role of the feedback structure is not separated from feedback scale. The new results answer both.

Critical 1. "Can you provide the missing numerical values in Tables 14–15 and specify the initialization and exact optimizer-update procedure used in the precision audit, including the treatment of Adam moments and weight decay?"

Response: Yes.

Table 14 (L = 10, 20 seeds per row):

dataset / backbonemax |grad| (FP64 accumulation)output-adjacent error
Cora / GCN1.82e-261.451e-4
Cora / GraphSAGE1.43e-261.610e-4
CiteSeer / GCN1.11e-261.772e-4
CiteSeer / GraphSAGE1.05e-261.857e-4
PubMed / GCN1.78e-262.035e-4
PubMed / GraphSAGE2.09e-262.359e-4

Table 15: DBLP norm-nonzero at L = 10, norm-zero in 4/5 seeds at L = 15 and 5/5 at L = 20; Flickr norm-nonzero at L = 20 (shallow/deep norm ratio ≈ 2.2e-11), norm-zero in 3/3 at L = 30; ogbn-arxiv norm-nonzero at L = 10, norm-zero in 3/3 at L = 20; Coauthor-Physics stable through about L = 26, increasingly unstable from L = 28 to 32.

Initialization. Glorot-uniform, bias-free layers, width 64, one saved draw per seed, used by both rules.

Optimizer in the audit (BP). torch.optim.Adam(lr=1e-2, weight_decay=5e-4) with default β = (0.9, 0.999) and ε = 1e-8, one parameter group covering every layer, full batch, 100 epochs; the audit is taken at the 100-epoch checkpoint. In this optimizer the decay is an L2 term: g ← ∇L + λθ is formed first and enters both moment estimates.

The four measurements, and how moments and decay enter each (re-run on 120 runs: 3 datasets × GCN/GraphSAGE × 20 seeds):

  1. The native FP32 gradient norm is 0 in every hidden layer in every run, while individual entries are nonzero: layer-0 medians are 1.4e-29 to 1.4e-28, and the largest entry is 2.1e-26.
  2. The FP64-accumulated norm matches a full FP64 recompute to a relative 1e-7 to 3e-4. The manual output-layer gradient and autograd differ by at most 1.0e-32 (the original note gave 1.6e-33 for seed 0; the re-run gives 2.3e-33).
  3. Moments and decay excluded: a fresh Adam step (zero moments, no decay) on this gradient changes 0 parameters.
  4. Moments and decay included (new): from the cloned weights and optimizer state, the next step with and without the current task gradient is identical in every element, in 120 of 120 runs.

So at the audit point, the current task gradient makes no representable difference; the step is set by the L2 term and the stored moments.

KAFT's update, and the shared-recipe reruns. KAFT forms Adam moments from its assigned direction and applies the same coefficient in decoupled (AdamW) form (Loshchilov & Hutter, 2019): θ ← θ − lr·(m̂/(√v̂ + ε) + λθ). The revised Appendix A gives both updates. With BP's form for both rules, the 96-setting result reproduces: mean gain +3.19 points, with 72 settings significant for KAFT and 4 for BP after BH correction (submitted: +3.09, 72 and 4). KAFT's own mean moves by +0.07. With no decay for either rule the gain is +1.48 (65 and 6), and KAFT is ahead in 82 of 96 settings. In link prediction with no decay for both, KAFT is ahead by +1.9 and +3.6 ROC-AUC over 20 seeds.

AuthorsOpenReview ↗

Response to Reviewer FpAz (cont. 2/4)

Critical 2. The reviewer notes that the diagnostic "does not fully isolate backward transport from numerical scaling": above-chance probes "establish recoverable class structure, but do not isolate the cause of training failure," and Appendix I "acknowledges substantial forward-activation attenuation as well."

Response: The probes establish that the information is present; the cause is isolated by interventions that hold the forward model fixed and change only the update. First, the depth study under three shared decay recipes (plain GCN, Glorot init; BP / KAFT test accuracy; * p < 0.05, paired; learning rate chosen per rule on seed 0, seeds 1–19 reported):

L2 in Adam for bothdecoupled for bothno decay for both
Cora, 10 layers32.8 / 70.0*67.2 / 70.9*68.3 / 71.5*
Cora, 2025.6 / 48.8*50.8 / 51.950.6 / 50.4
DBLP, 879.0 / 82.1*79.6 / 81.7*81.1 / 82.0*
DBLP, 1266.4 / 81.3*80.1 / 81.8*79.6 / 81.7*
DBLP, 1649.9 / 78.6*78.1 / 80.2*78.1 / 80.3*
DBLP, 2045.7 / 74.5*76.4 / 76.374.7 / 75.6

Along each row KAFT's accuracy moves by at most 3.1 points and BP's by up to 35.5. With L2 inside Adam, BP ends at uniform predictions in 235 of 240 runs at Cora 10/20 and DBLP 16/20 layers (three learning rates, 20 seeds). With the same coefficient in decoupled form, or with no decay, it never does, and KAFT never does under any of the three. Against BP without the L2 term, KAFT is ahead up to 16 layers (+0.9 to +3.2 points, all four significant).

The mechanism is measured per step (Cora, 10 layers, lr 1e-3, 20 seeds). At step 0, with identical weights and activations, BP's first-layer task signal is 0.084× the L2 term λ‖W‖ and KAFT's is 78×. BP's step follows the decay: its first-layer norm falls from 11.1 to 6.4e-4 and all 20 runs stay at uniform predictions. KAFT's weights grow (11.1 → 24.7) and it reaches 72.4%. At this learning rate BP collapses while its FP32 hidden gradients are still nonzero (0 of 20 runs reach an FP32-null update). So the cause is the competition inside the update, and FP32 underflow is its endpoint; Section 2.3 now says so. A fork from BP's own state at step 25 (weights and Adam moments, replayed bit-exactly) gives 23.5% if BP continues, 63.8% with BP's gradient rescaled per layer to KAFT's update norm, and 67.8% with KAFT.

At the same forward state, the update decides whether training escapes. The next point separates the two things the update carries: its scale and its direction.

The question is not whether the forward pass attenuates, but whether the update still follows the task.

AuthorsOpenReview ↗

Response to Reviewer FpAz (cont. 3/4)

Critical 3. "How does KAFT compare with BP using layerwise gradient normalization or rescaling, and with random or fixed feedback matched to KAFT's feedback magnitude? These controls would help distinguish structural alignment from scale restoration."

Response: New six-arm control on three settings, two recipes, 20 paired seeds, each arm at its own learning rate selected on validation. Test accuracy; in brackets, Δ against full KAFT; * p < 0.05.

armCora GCN, 6 layers (L2 / none)Cora GCN+LN, 10 layers (L2 / none)DBLP GCN, 16 layers (L2 / none)
BP68.5 (−8.35*) / 74.9 (−2.25*)75.9 (−2.41*) / 75.3 (−2.85*)49.9 (−28.58*) / 78.1 (−1.54, p = 0.12)
BP rescaled per layer to KAFT's update norm74.0 (−2.85*) / 74.5 (−2.68*)75.7 (−2.62*) / 75.5 (−2.57*)76.4 (−2.12, p = 0.055) / 78.2 (−1.41, p = 0.18)
BP with LARS-style per-layer normalization74.2 (−2.66*) / 74.2 (−2.91*)75.1 (−3.20*) / 75.7 (−2.38*)77.6 (−0.88, p = 0.43) / 77.4 (−2.29*)
KAFT, fixed random R (same column normalization)75.6 (−1.33*) / 74.9 (−2.22*)76.5 (−1.80*) / 76.5 (−1.62*)63.6 (−14.91*) / 66.2 (−13.51*)
KAFT, R aligned once then frozen77.9 (+1.06*) / 78.3 (+1.09*)78.3 (+0.03) / 78.2 (+0.12)79.8 (+1.29, p = 0.069) / 80.1 (+0.48, p = 0.19)
full KAFT76.9 / 77.278.3 / 78.178.5 / 79.7

What the control shows:

  • The learned feature feedback matters. With topology, gates, column normalization and recipe held fixed, replacing the learned R with a fixed random R costs 1.3–2.2 points at 6–10 layers and 13.5–14.9 points at 16 layers. This arm shares KAFT's column normalization. A second random arm, rescaled every step to KAFT's per-layer update norm (CiteSeer and PubMed, 10 paired seeds), does no better: −0.24 points against the first on average, significant in 0 of 12 cells, and both random arms are 17–33 points below KAFT at 16 layers. The gap comes from the direction of the feedback, not its scale.
  • The rule adds accuracy where scale is not the problem. Where BP already trains (Cora at 6 layers without decay, Cora with LayerNorm under both recipes, DBLP at 16 layers without decay), rescaling moves BP by at most 0.4 points, and LARS-style normalization by at most 0.8 (not significant). KAFT stays 2.57–2.68 points ahead of the rescaled BP on the three Cora settings (p ≤ 3.6e-4), and 2.3–3.2 points ahead of LARS-style BP in all four of these settings (p ≤ 0.026), including DBLP at 16 layers without decay.
  • Where BP collapses, KAFT stays ahead of both normalized arms. Under L2, per-layer normalization lifts BP out of collapse: LARS-style BP gains +5.7 points on Cora at 6 layers and +27.7 on DBLP at 16 layers, with no uniform runs (BP: 18 of 20). KAFT is still ahead of the rescaled and the LARS-style arm in all three L2 settings, by 2.1–2.9 and 0.9–3.2 points, significantly on both Cora settings.
  • Alignment can be a one-time cost. R aligned once and then frozen matches full KAFT, and is 1.1 points better on Cora at 6 layers.

On layerwise gradient normalization by name, the control has two such arms. The rescaled-BP arm normalizes each hidden layer's gradient to KAFT's own per-layer update norm, which tests directly whether KAFT helps by restoring magnitude. The LARS-style arm (You et al., 2017) uses the standard fixed target instead: each weight gradient is rescaled to the norm of its weight matrix, g_l ← g_l·‖W_l‖/‖g_l‖ (trust coefficient 1), and then enters the same Adam step as BP, so that the base optimizer stays the same for every arm. KAFT is ahead of both arms in all 12 comparisons, significantly in 9. On CiteSeer the six arms repeat the Cora pattern: KAFT is significantly ahead of LARS-style BP in 4 of 6 settings (+4.0 to +9.5 points) and of rescaled BP in 3 (+3.9 to +8.1). We attribute this lead to the rule as a whole, graph-side spreading of the error plus aligned feature feedback, and not to the feature direction alone; the evidence is firm on Cora and CiteSeer; on DBLP at 16 layers and on PubMed the margins are smaller and mostly not significant (full six-arm tables in the appendix).

Matching the update scale is not enough: KAFT's rule, graph-side spreading plus aligned feature feedback, carries accuracy that normalized BP does not reach.

AuthorsOpenReview ↗

Response to Reviewer FpAz (cont. 4/4)

Strongly recommended. "Which alignment or perturbation-margin conditions from Appendix Q have been measured for the practical nonlinear algorithm, especially beyond the BP transport boundary? If they have not been verified, how should the theoretical guarantee be interpreted relative to the empirical claims?"

Response: We now measure Theorem 1's two quantities at every step: μ = ⟨∇L, u⟩/‖∇L‖² and M = ‖u‖/‖∇L‖, with ∇L in FP64 so that they stay defined past the FP32 boundary (Cora GCN, 6, 10 and 20 layers, 20 seeds, both recipes, two learning rates; new: DBLP and LayerNorm stacks). Appendix Q already says the condition "must be imposed on the actual preconditioned moment direction", so we measure both KAFT's raw direction and its actual Adam step.

  • KAFT's raw direction is a descent direction (μ > 0) in 72–100% of steps (median over seeds; worst seed 52%), so alignment is negative in some steps at every depth.
  • The same measurement on DBLP (6, 10 and 20 layers) and on LayerNorm stacks (Cora and DBLP, 10 layers), 10 seeds each, both recipes and two learning rates: the raw direction is a descent direction in 57–97% of steps on DBLP and 48–100% on the LayerNorm stacks (median over seeds).
  • KAFT's actual Adam step descends the data loss in 81–99% of steps (median; worst seed 72%).
  • For comparison, BP's actual Adam step under L2 at 10 layers (lr 1e-3) descends the data loss in only 10.7% of steps; against the L2-regularized objective it descends in 100%. The step follows the decay term.
  • M grows with depth, from about 170 at 6 layers to 700 at 10 and 2×10⁴ at 20, because the true gradient shrinks while KAFT's direction does not. The step-size bound η ≤ μ/(βM²) was not part of this measurement.

How to read the theory, in three tiers. (i) Lemma 2 (the PSD target) is exact, as is the alignment identity of Lemma 1. (ii) Corollary 1 is exact for the matched linear model. (iii) Theorem 1 is conditional: it gives sufficient conditions for descent. For the practical nonlinear algorithm we measured the alignment condition, which is positive in 72–100% of steps on Cora, 57–97% on DBLP and 48–100% on LayerNorm stacks, including beyond the BP boundary; the step-size condition was not measured. The measurements therefore support the premise of Theorem 1 in most steps; they do not certify it. The revised Appendix Q reports this table and states the three tiers.

Leftover weakness. "the PSD cross-kernel result applies to a matched linear setting, while the practical method includes nonlinear gates, capped propagation, diffusion, approximate feedback tracking, and Adam."

Response: Appendix Q treats these parts as an explicit perturbation Δ of the matched kernel and bounds it with Weyl's inequality. The measured μ above is the empirical counterpart of that margin. The method's design follows from the theory: the target is PSD by construction (exact), and the measured alignment is positive in most steps across the datasets and stacks we tested (empirical).

We hope the shared-recipe reproduction, the direction control and the measured conditions resolve the soundness concerns. If they do, we would be grateful if the reviewer would consider raising the score.

References

  • Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019. arXiv:1711.05101.
  • Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv:1708.03888, 2017.
Reviewer_E4a8OpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

the paper investigates optimization difficulties in deep message-passing graph neural networks and present an alternative credit-assignment mechanism that exploits the Kronecker structure of message-passing Jacobians

Critical strengths

The most defensible novelty is replacing sequential backpropagation with graph-aware, learned feedback. Demonstrating improved training at substantial depth without changing the forward architecture.

Critical weaknesses

Vanishing gradients in deep GNNs are not a new discovery, and the connection between oversmoothing, spectral contraction, and gradient propagation has already been studied. The authors are trying to make is that forward representations can remain informative even when the backward learning signal becomes numerically ineffective. But presenting backward gradient failure as a fundamentally new explanation for deep-GNN failure is questionable. This concept is also studied from the point of view of injectivity in 'Scalable Message Passing Neural Networks: No Need for Attention in Large Graph Representation Learning' (ICLR Gram) and from the point of view of asymptotic behavior in 'Understanding convolution on graphs via energies' (TMLR) (missing references).

Questions for authors

I have questions regarding the paper's central diagnostic claim:

Is KAFT solving a distinct backward credit-assignment problem, or compensating for forward-induced gradient attenuation? The paper shows that hidden representations retain linearly recoverable class information while FP32 parameter updates become ineffective. But in tthe appendix also acknowledges severe forward activation attenuation.

Could the authors clarify what evidence establishes backward transport as an independently limiting mechanism, rather than another manifestation of the same forward spectral contraction responsible for oversmoothing? --> this is important to understand

Does KAFT offer meaningful advantages over well-tuned deep GNN architectures? --> we know from GCNII, SMPNNs etc that residual connections are very important (conventional forward stabilizers already recover much of the performance as shown in Table 4) using simple approaches so why overcomplicate things? Can KAFT improve the performance of already stabilized GNNs further?

8 comments
AuthorsOpenReview ↗

Response to Reviewer E4a8 (1/4)

We thank the reviewer for the positive assessment and for questions that go to the center of the paper. We are glad the reviewer sees "the most defensible novelty" in "replacing sequential backpropagation with graph-aware, learned feedback," "demonstrating improved training at substantial depth without changing the forward architecture." We also thank the reviewer for the two references; both are now cited.

Protocol for the numbers below. Unless stated otherwise, the BP–KAFT comparisons in this reply come from reruns in which both rules use the same decay recipe and the same saved initializations. The main recipe is the BP recipe of the submission, PyTorch Adam with L2 weight decay 5e-4; we also report no decay for both (details in the general response). With the shared recipe the 96-setting result reproduces: +3.19 points on average over 20 paired seeds, with 72 settings significant for KAFT and 4 for BP (submitted: +3.09, 72 and 4).

Critical 1. "Vanishing gradients in deep GNNs are not a new discovery, and the connection between oversmoothing, spectral contraction, and gradient propagation has already been studied ... presenting backward gradient failure as a fundamentally new explanation for deep-GNN failure is questionable." The reviewer also names two missing references.

Response: The paper builds on this line of work (Arroyo et al., 2025; Park & Kim, 2024; Park et al., 2025), and its contribution is the backward rule and the evidence for it, not a new account of why gradients vanish. The revision makes this explicit: "first" is removed from the diagnosis claim, and one sentence is corrected, where we had written that spectral contraction causes vanishing gradients "independently of forward collapse"; the text now follows Arroyo et al., who state that over-smoothing "is directly related to the mechanism causing vanishing gradients." We added Sáez de Ocáriz Borde et al. (2024), who show that residual connections are needed to keep universal approximation in graph convolutions, and Di Giovanni et al. (2023), who show through an energy view that, given a residual connection, graph convolutions can smooth or sharpen features depending on the weights. Both results concern the limit of the forward model; our claim concerns training at finite depth.

What the paper contributes:

  1. A test on the same checkpoints. After 100 epochs of BP at 10 layers, frozen linear probes on the penultimate hidden state reach 36.9–63.6% (Table 14; chance is 14–33%), while the current task gradient changes no element of BP's next optimizer step in 120 of 120 runs. The information is still in the forward pass; the update no longer uses it.
  2. A measurement of the collapse. Weight over-decaying under L2 is known (Lu et al., 2024; Cong et al., 2021). We quantify it on the same model for both rules: BP's first-layer task signal is 0.084× the L2 term at step 0; KAFT's is 78×. This is why BP collapses under Adam with L2 and KAFT does not (Part 2 below).
  3. A backward rule derived from the Jacobian. The message-passing Jacobian factors into a graph term and a feature term (a Kronecker product). KAFT keeps the graph factor of the forward model and learns the feature factor, aligned to a target with a PSD alignment property (Lemma 2), with sufficient conditions for descent (Theorem 1).
  4. Constant feedback depth. Hidden-layer feedback has dependency depth Θ(K+1) instead of Θ(L) (Table 1), which gives 1.59–1.73× faster training steps on Cora and parity on DBLP (Table 10).

The novelty we claim is the backward rule, not the observation that gradients vanish.

AuthorsOpenReview ↗

Response to Reviewer E4a8 (cont. 2/4)

Critical 2. "Is KAFT solving a distinct backward credit-assignment problem, or compensating for forward-induced gradient attenuation? ... Could the authors clarify what evidence establishes backward transport as an independently limiting mechanism, rather than another manifestation of the same forward spectral contraction responsible for oversmoothing? --> this is important to understand"

Response: Not an independent cause; an independent point of repair. Forward contraction and backward attenuation share the same weights and gates (Arroyo et al., 2025). What we add is that training can be restored by changing only the update. The test is whether training can be rescued from the backward side alone, with the forward model held fixed. Four interventions do exactly that (Cora, plain 10-layer GCN, 20 seeds, lr 1e-3, same initializations unless stated):

  • Same state, different backward rule. At step 0, with identical weights and activations, BP's first-layer task signal is 0.084× the L2 term and KAFT's is 78×. BP's Adam step follows the L2 term: its first-layer weight norm falls from 11.1 to 6.4e-4, and all 20 runs end at uniform predictions (24.0%). KAFT's first-layer norm grows to 24.7 and it reaches 72.4%.
  • Fork from BP's own trajectory. From BP's state at step 25 (weights and Adam moments, replayed bit-exactly), continuing with BP gives 23.5% (20 of 20 runs uniform). Rescaling BP's own gradient per layer to KAFT's update norm gives 63.8%, and switching the hidden backward rule to KAFT gives 67.8% (0 of 20 uniform). At lr 1e-2 the early fork gives 31.6%, 63.1% and 48.3%: both backward changes escape, BP does not.
  • Remove the competing term. With no weight decay and nothing else changed, BP trains the same model to 71.1% (KAFT 72.9%).
  • Beyond scale. Where BP already trains (6-layer GCN without decay; 10-layer GCN with LayerNorm under both recipes), rescaling BP to KAFT's per-layer norm changes its accuracy by at most 0.4 points, and KAFT is 2.57–2.68 points above the rescaled BP (p ≤ 3.6e-4). KAFT with a fixed random feedback matrix under the same column normalization is 1.3–2.2 points below KAFT at 6–10 layers and 13.5–14.9 below on DBLP at 16 layers. Rescaling the random feedback every step to KAFT's per-layer update norm leaves it where it was (−0.24 points against the unscaled random arm on average, 0 of 12 cells significant; CiteSeer and PubMed), 18.9–33.4 points below KAFT at 16 layers: the gap is direction, not scale.

In the first three, the forward computation is unchanged and training is restored by changing the update alone. That makes the backward side an independent point of repair: at the same forward state, BP's attenuated signal loses to the regularizer and KAFT's does not. The fourth shows that KAFT's rule (graph-side spreading of the error plus aligned feature feedback) adds accuracy beyond restoring the update scale. On CiteSeer the same control repeats the pattern: KAFT is significantly ahead of LARS-style BP in 4 of 6 settings and of rescaled BP in 3.

The forward state can be the same; the backward rule decides whether training escapes.

Depth under shared recipes. With L2 for both rules, the DBLP gap grows from +3.1 points at 8 layers to +28.8 at 20, and BP ends at uniform predictions in 235 of 240 runs at Cora 10/20 and DBLP 16/20 layers, while KAFT never does. With no decay for both, KAFT is ahead by +0.9 to +3.2 points up to 16 layers, significant in all four comparisons. With learning rates re-selected per rule on seeds 0–2 and tested on held-out seeds 3–19, KAFT's no-decay margin on CiteSeer is +3.8 / +6.6 / +10.8 points at 6 / 10 / 20 layers (BH q ≤ 0.0011). Across three shared recipes (L2, decoupled, none), KAFT's accuracy moves by at most 3.1 points in each of these six settings, and BP's by up to 35.5. Section 2 now cites over-decaying as prior art and reports this measurement.

AuthorsOpenReview ↗

Response to Reviewer E4a8 (cont. 3/4)

Strongly recommended. "Does KAFT offer meaningful advantages over well-tuned deep GNN architectures? --> we know from GCNII, SMPNNs etc that residual connections are very important ... so why overcomplicate things? Can KAFT improve the performance of already stabilized GNNs further?"

Response: KAFT is the simpler of the two routes. Stabilized architectures fix deep training by adding machinery to the forward model, and that machinery brings its own hyper-parameters, tuned for BP. KAFT fixes the failure without it, and stabilizing initializations and regularizers can still be added on top.

The machinery is costly and tuned for BP. Ortho-GConv (Guo et al., 2022), for example, adds three hyper-parameters (T, β, λ) and an orthogonality penalty. Its settings are chosen by a random search on validation accuracy, and with its official code on CiteSeer at 8 layers the result depends on that search: the five best configurations all have β ≤ 0.2 and T ≥ 3, and configurations with β ≥ 0.7 or T ≤ 2 reach 18–47% (general response, item 2(a)).

KAFT needs none of it. On plain GCNs with Glorot initialization, KAFT never collapses where BP does in 235 of 240 runs, and gains +3.1 to +28.8 points on DBLP at 8–20 layers under L2. With normalization at 10 layers, it adds up to +10.3 points with the forward model unchanged (Δ = KAFT − BP, 19 held-out paired seeds, each rule at the learning rate selected in the submission; * p < 0.05):

L = 10BN, L2 for bothLN, L2 for bothBN, no decayLN, no decay
Cora+7.92*+2.02*+7.89*+2.72*
CiteSeer+0.77+2.74*+6.24*+3.43*
PubMed+10.28*−1.01+9.21*−0.10

KAFT is significant in 4 of 6 cells with L2 and 5 of 6 without; BP is significant in none. GIN (Xu et al., 2019), with sum aggregation and a weighted self term, gains +0.42 points over its 24 settings of Table 2 with L2 for both and +1.32 with no decay.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response to All (1/3)

We thank the three reviewers for careful and constructive reviews. They agree on what KAFT contributes: E4a8 sees "the most defensible novelty" in "replacing sequential backpropagation with graph-aware, learned feedback"; FpAz calls KAFT "a motivated intervention in backward credit assignment" whose PSD alignment property "gives useful mathematical support"; K7qV finds the probe experiments "interesting" and agrees that GNN optimization "is an open area for research that should be explored more." Each reviewer reply below is self-contained for its main points. This note collects what they share.

1. What KAFT is, and what it does. KAFT is a backward rule for the forward model one wants to keep. It replaces layer-by-layer backpropagation through hidden layers with learned, graph-aware feedback: the graph factor of the message-passing Jacobian is kept, and the feature factor is learned and aligned to a PSD target (Lemma 2), with sufficient conditions for descent (Theorem 1). Architecture, parameters and inference cost are unchanged. In reruns where both rules share the same decay recipe and the same saved initializations (20 paired seeds):

  • Over the 96 settings of Table 2, KAFT gains +3.19 points on average, significant in 72 settings for KAFT and 4 for BP (Benjamini–Hochberg, q = 0.05).
  • On deep plain GCNs, BP ends at uniform predictions in 235 of 240 runs (Cora 10/20, DBLP 16/20 layers); KAFT in 0 of 240. On DBLP the gap grows from +3.1 points at 8 layers to +28.8 at 20.
  • At step 0 of a 10-layer GCN on Cora, with identical weights, BP's first-layer task signal is 0.084× the L2 term and KAFT's is 78×. This quantifies the weight over-decaying described by Lu et al. (2024, SkipNode) and Cong et al. (2021): BP's attenuated task signal loses to the regularizer, and KAFT's does not.

2. Stabilizing machinery versus a backward rule. Several comments propose that BP with a stabilizing initialization or regularizer (the initialization schemes [1–3] named by K7qV, Ortho-GConv in particular) is the better answer to deep-GNN training failure. Our answer has three parts.

(a) The machinery is costly and fragile, and it is tuned for BP. Ortho-GConv adds three hyper-parameters (orthogonalization iterations T, hybrid-initialization weight β, penalty weight λ) and an orthogonality penalty in the loss. With its official code on CiteSeer at 8 layers, the script defaults give 44.31 ± 4.73. The random search described in its appendix (20 configurations × 10 runs, selection on validation) is needed to reach 65.17 ± 4.19; the paper reports 68.59 ± 1.74. The five best configurations all have β ≤ 0.2 and T ≥ 3; configurations with β ≥ 0.7 or T ≤ 2 reach 18–47%, with standard deviations up to 16.5.

(b) KAFT needs none of it to fix the failure. The results in item 1 use the submitted forward models with standard Glorot initialization, no penalty and no added hyper-parameters on the forward side.

(c) The machinery is orthogonal to KAFT and stacks on it. Balanced orthogonal initialization ([2], Mustafa et al., 2023) lifts KAFT itself from 70.0% to 78.3% on Cora at 10 layers and from 48.8% to 75.5% at 20 layers (L2 recipe), far above BP without that machinery (32.8% and 25.6% with L2; 68.3% and 50.6% with no decay). On the Ortho-GCN forward model, with [3]'s settings, which were tuned for BP, KAFT is within 0.6 points of BP in 11 of 12 cells. With each rule given settings chosen for it, KAFT beats BP on ResGCN and matches it on Ortho-GCN. On a residual GCN (ResGCN, Li et al., 2019; Cora, 8–20 layers), both rules searched the same grid of stabilizing settings (residual scale, LayerNorm placement, initialization gain; KAFT also its feedback initialization), then their own optimizer grid, with selection on seeds 0–2. KAFT's choice was the same in all six cells (LayerNorm after the residual add, gain 0.5, orthogonal feedback initialization). It reaches 80.1–81.5% at 16 and 20 layers and beats BP's best on the same grid in 5 of 6 cells, by +1.75 to +3.67 points (BH q ≤ 3.2e-4, 17 held-out seeds). On Ortho-GCN with initialization and regularization settings searched for KAFT (β, λ, gains and feedback initialization; 216 configurations per cell) against BP with [3]'s published settings and its own learning-rate search, KAFT is on par on average (+0.33 points over 12 cells), with gains up to +5.71 points (Cora, 20 layers). Residual connections change the forward model itself; KAFT changes how a given forward model is trained, and the two combine. The full equal-budget tables for every stabilized stack are in the revised appendix.

A stabilizing initialization or regularizer is an add-on that must be tuned for the training rule; KAFT fixes the failure without one and can use one on top.

AuthorsOpenReview ↗

Response to All (cont. 2/3)

3. New experiments (all finished; key tables in the revised appendix):

  1. Shared-recipe reruns of Tables 2, 3, 6, 7, 18 and 24 and of the depth study (L2 in Adam for both, no decay for both; the depth study also with decoupled decay for both).
  2. Six-arm scale-versus-direction control: BP, BP rescaled per layer to KAFT's update norm, BP with LARS-style per-layer gradient normalization, KAFT with a fixed random feedback matrix, KAFT with feedback aligned once, full KAFT.
  3. Per-step logs of the collapse mechanism, and forks from BP's own optimizer state.
  4. Theorem 1's two quantities measured at every step in FP64 at 6, 10 and 20 layers.
  5. Stabilized forward models at 8, 16 and 20 layers (balanced orthogonal initialization, Ortho-GCN, GCNII-style and ResGCN stacks), with the same tuning budget of 12 configurations for each rule, selected on validation over seeds 0–2 and reported on held-out seeds 3–19; and the official Ortho-GConv code and protocol.
  6. Hop cap K swept over three datasets, three depths and two recipes, with the Dirichlet energy of the hidden error.
  7. DropEdge with a backward graph synchronized to the forward graph.
  8. A 120-run re-measurement of the precision audit, including a new check on the actual optimizer step.
  9. Each rule with stabilizing settings chosen for it: initialization and regularization searched for KAFT on balanced-initialization, Ortho-GCN and GCNII-style stacks (30 cells), against BP with the published settings; and ResGCN with the same grid of residual scale, LayerNorm placement and initialization gain searched for each rule (6 cells). Ortho-GConv's own protocol on Cora and CiteSeer at 8 layers, with a 40-configuration search for each rule.
  10. Learning rates and decay recipe re-selected per rule on seeds 0–2 and tested on held-out seeds 3–19 (Table 6 layout and depth study); shared-recipe reruns of Tables 4, 19 and 22; the six-arm control on CiteSeer and PubMed with a random-feedback arm rescaled to KAFT's per-layer update norm; the hop-cap sweep with label spreading removed; Theorem 1's quantities on DBLP and on LayerNorm stacks.
AuthorsOpenReview ↗

Response to All (cont. 3/3)

4. Revisions in the PDF (uploaded by 18 November):

  • All 41 "??" placeholders are filled from the run logs.
  • The abstract uses the shared-recipe values: +3.2 points with 72 and 4 significant settings under L2 for both, +1.5 (65 and 6) without decay. The depth sentence states both recipes: on DBLP, +3.1 points at 8 layers to +28.8 at 20 with L2, where BP collapses; +0.9 to +2.1 up to 16 layers without decay. The FP32 sentence names the recipe (Adam with L2 weight decay).
  • Section 2 cites weight over-decaying as prior art (Lu et al., 2024; Cong et al., 2021) and adds our measurement of it (0.084× against 78×). Section 4.5 adds the stacking results of item 2(c) and states how KAFT relates to residual connections: they change the forward model, and KAFT changes how a given forward model is trained.
  • Link prediction (Table 24) is reported over 20 seeds with no decay for both rules: +1.9 and +3.6 ROC-AUC. Under L2 neither rule trains this encoder, and the table says so.
  • Table 7 (20 layers, four larger graphs) is reported under both shared recipes with the same bias-free forward model for all rules; the caption states that accuracies are taken at the validation peak and reports the final training loss.
  • The sentence on combining KAFT with forward-side stabilizers names the depth at which it was tested (L = 6).
  • Section 2.3 and Appendix I name the recipe and learning rate of the FP32 measurement and add the collapse mechanism.
  • "First" is removed from the diagnosis claim; the sentence on Arroyo et al. (2025) is corrected; the references named by the reviewers are added.
  • Appendix Q reports the measured Theorem 1 quantities on Cora, DBLP and LayerNorm stacks. Appendix F adds a PyTorch Geometric listing.
  • New appendix sections give the full per-cell tables of items 3.5, 3.9 and 3.10, with each rule's selected configuration, held-out-seed results and BH-corrected tests.
  • Figure 1 is re-rendered as one vector figure, and the spacing on page 7 is fixed.

5. Decay specification (clarification of Appendix A). Appendix A gives one optimizer line for both rules (Adam, weight decay 5 × 10⁻⁴). The two rules apply that coefficient in different forms: BP uses PyTorch Adam with weight_decay=5e-4, so the L2 term λθ enters both moment estimates; KAFT applies it in decoupled (AdamW) form (Loshchilov & Hutter, 2019), θ ← θ − lr·(m̂/(√v̂ + ε) + λθ). The revised Appendix A gives both updates and the recipe of every table. All numbers above come from reruns with one shared recipe; under the submitted recipes the 96-setting result is +3.09 (72 and 4), under L2 for both +3.19 (72 and 4), and KAFT's mean is 75.24, 75.31 and 75.24 under the three recipes.

References

  • [1] Jiahang Li, Yakun Song, Xiang Song, and David Paul Wipf. On the initialization of graph neural networks. In International Conference on Machine Learning (ICML), 2023. arXiv:2312.02622.
  • [2] Nimrah Mustafa, Aleksandar Bojchevski, and Rebekka Burkholz. Are GATs out of balance? In Advances in Neural Information Processing Systems (NeurIPS), 2023. arXiv:2310.07235.
  • [3] Kai Guo, Kaixiong Zhou, Xia Hu, Yu Li, Yi Chang, and Xin Wang. Orthogonal graph neural networks. In AAAI Conference on Artificial Intelligence, 2022. arXiv:2109.11338.
  • Guohao Li, Matthias Müller, Ali Thabet, and Bernard Ghanem. DeepGCNs: Can GCNs go as deep as CNNs? In IEEE International Conference on Computer Vision (ICCV), 2019. doi:10.1109/ICCV.2019.00936. arXiv:1904.03751.
  • Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019. arXiv:1711.05101.
  • Weigang Lu, Yibing Zhan, Binbin Lin, Ziyu Guan, Liu Liu, Baosheng Yu, Wei Zhao, Yaming Yang, and Dacheng Tao. SkipNode: On alleviating performance degradation for deep graph convolutional networks. IEEE Transactions on Knowledge and Data Engineering, 36:7030–7043, 2024. doi:10.1109/TKDE.2024.3374701. arXiv:2112.11628.
  • Weilin Cong, Morteza Ramezani, and Mehrdad Mahdavi. On provable benefits of depth in training graph convolutional networks. In Advances in Neural Information Processing Systems (NeurIPS), 2021. arXiv:2110.15174.
  • Álvaro Arroyo, Alessio Gravina, Benjamin Gutteridge, Federico Barbero, Claudio Gallicchio, Xiaowen Dong, Michael Bronstein, and Pierre Vandergheynst. On vanishing gradients, over-smoothing, and over-squashing in GNNs: Bridging recurrent and graph learning. arXiv:2502.10818, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.