acceptodds
Under review as a conference paper at ICLR 2027

d-STABLE: Learning Residual Allocations Under a Controlled Depth Budget

Abstract

The sensitivity of a residual network—how much it amplifies a small input change across all layers—is a product over layers, so it is governed by one quantity: the total *residual budget*, the summed strength of the per-layer residual branches. We show that this budget and its allocation across layers are two separable levers. The budget's growth rate in depth sets the *stability class*: a linear budget makes worst-case sensitivity exponential, a square-root budget (Fixup, Stable ResNet) sub-exponential, and a *logarithmic* budget polynomial, so is the critical rate. The allocation leaves the class unchanged and sets accuracy within it. Prior stabilizers couple the two: fixed schedules freeze both, and learned gates (ReZero, LayerScale) let the budget drift. **d-STABLE** is a reparameterization that holds the total budget at exactly at every optimization step while the optimizer learns the allocation. When every branch is norm-bounded this gives an a-priori polynomial bound on sensitivity; otherwise the pinned budget is an inductive bias whose amplification we measure. Our experiments separate the two levers. On `ogbn-arxiv` at 64 layers, unscaled residual GCN/SAGE collapse to 16–22% accuracy through gradient expansion, not oversmoothing. Keeping the residual mass small prevents the collapse: Fixup, ReZero, and a fixed logarithmic schedule all reach the same 69.5–69.8% plateau. The allocation supplies the margin above that plateau, reaching % (+1.7 over a fixed profile at an identical budget), ahead of GCNII, LayerScale, and Gradient Gating. On this benchmark the margin comes from the allocation's shape, not its training; enforcing the a-priori bound costs 1.8 points. On `ImageNet-1k`, where residual networks already train stably, the budget costs 0.5 top-1 points on ResNet-50 and 1.8 on DeiT-Tiny, where vanilla, LayerScale, and a DeepNorm-style rule are more accurate.

Primary area:
learning theory
Keywords:
residual networks, training stability, depth scaling, signal propagation, residual scaling schedules, deep graph neural networks, oversmoothing, Lipschitz sensitivity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ySwYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper separates the total magnitude of residual coefficients from their allocation across depth. It proposes normalized signed or nonnegative coefficients whose absolute sum is bounded by beta log(d) throughout training. Under uniform bounds on the complete residual-branch Jacobians, a standard product inequality then gives a polynomial upper bound on input sensitivity. Experiments study synthetic chains, several graph-network architectures and ImageNet models. The graph experiments report improved deep-network accuracy and include controls for matched budgets, frozen allocations and branch normalization. The paper also reports cases where learning the allocation adds little and where the budget reduces ImageNet accuracy.

Critical strengths
  1. The parameterization is simple and makes a relevant quantity explicit. Theorem 1 cleanly states the full-branch assumptions, and the paper generally distinguishes a worst-case sensitivity guarantee from optimizer convergence and adversarial robustness.
  2. The empirical investigation includes unusually informative controls: matched-budget and frozen-profile comparisons, norm-on/off experiments, multiple graph operators and repeated seeds for the main result. I appreciate the explicit reporting that the ogbn-arxiv margin largely comes from profile shape rather than learning, and that ImageNet accuracy can decrease. These findings help delimit the method's usefulness.
Critical weaknesses
  1. The implemented “a-priori certified” configuration does not yet establish the claimed certificate. Appendix E.1 (p.20) states that normalization uses one power iteration and that realized branch norms reach 1.15. This does not enforce a uniform bound of 1, while Table 8 labels d^beta as an a-priori reference. Observing sampled J_p95 below that reference cannot certify a supremum over inputs. I appreciate the explicit disclosure of the approximation, but the distinction matters because the claimed cost of obtaining a certificate is a central result. Please either enforce a defensible full-branch norm bound and use its actual exponent, or describe these runs as empirical approximate control. It should also be clear whether the measured Jacobian is for the residual stack or includes the encoder and classifier excluded from the stated bound.

  2. The evidence does not yet isolate the central mechanism and its advantage over simpler alternatives. The claim that residual-1 fails through gradient expansion rather than oversmoothing needs reconciliation with Table 10 (p.23), which reports feature-MAD=0.00 at depths 32 and 64 and defines zero as directional collapse. Increasing normalized Dirichlet energy and directional feature collapse need not be mutually exclusive. Moreover, the frozen-profile result is valuable evidence that allocation learning is unnecessary for the headline graph gain; the random-sign control identified in Appendix E.2 would clarify whether the improvement is primarily a signed-initialization effect. The comparator terminology also matters: the fixed s_k=1/sqrt(d) control called “Fixup” is not the full initialization-and-learned-multiplier method of Zhang et al. (2019, Section 3). Renaming that control, or explaining the implemented correspondence, would make the scope of the comparative claims more reliable.

Questions for authors
  1. What full-branch operator bound is actually enforced in the configuration called certified? Can Table 8 report the corresponding valid upper bound, and specify the input/output map used for J_p95? Please also reconcile the normalization description of the heterophilic rows in Appendix E.5 with the main-text scope statement.
  2. Can the energy and feature-MAD measurements be reconciled on the same checkpoints? Do they support coexisting amplification and directional collapse, rather than excluding one mechanism? The residual-1+SN case with J_p95=7.0 but 11.7% accuracy in Table 7 is particularly informative here.
  3. At the same budget, how does a frozen random-sign, uniform-magnitude allocation compare with the learned and frozen-converged profiles? Please clarify whether the “Fixup” implementation contains the original initialization, biases and learned multipliers, or is only a constant-scale control.
3 comments

Dear Reviewer ySwY,

Thank you for the close reading. Each of your three questions changed the paper.

Q1 / W1: the certificate

What full-branch operator bound is actually enforced in the configuration called certified? … Table 8 labels dβd^\beta as an a-priori reference.

You are right: one power-iteration step under-reads σ(W)\sigma(W), and the realized branch norms reach 1.151.15, so the submitted runs did not enforce ρ≤1\rho \le 1. We made three changes.

  • Exact normalization. We replaced the one-step estimate with an exact SVD. A unit test checks ∥Wk∥2=1±10−5\lVert W_k\rVert_2 = 1 \pm 10^{-5} at every layer after training; a second shows the one-step version exceeding 1 after a few updates.
  • Retrained. β=1/2/4\beta = 1 / 2 / 4 reach 69.81±0.1269.81 \pm 0.12 / 71.13±0.2471.13 \pm 0.24 / 71.51±0.0671.51 \pm 0.06 (seeds 3–7; uncertified headline 71.48±0.2071.48 \pm 0.20).
  • Old runs relabeled. They are now an "approximate control", reported with the valid bound dρmax⁡βd^{\rho_{\max}\beta} from the measured ρmax⁡=1.15\rho_{\max} = 1.15.

Which map. The paper's Jp95J_{p95} is ∥∂ logits[v]/∂X∥\lVert \partial\,\text{logits}[v] / \partial X\rVert, encoder and classifier included, whereas Theorem 1 bounds the residual stack ∂H(d)/∂H(0)\partial H^{(d)} / \partial H^{(0)}. The revision reports both. On the exact-SN runs (d=64d=64, seeds 3–7):

β=1\beta=1β=2\beta=2β=4\beta=4
stack norm log⁡∥Jeff∥\log\lVert J_{\text{eff}}\rVert3.033.034.284.284.904.90
bound βlog⁡d\beta\log d4.164.168.328.3216.6416.64

The bound holds at every β\beta and is tightest at β=1\beta=1.

Heterophilic rows. The d-STABLE rows used SN and the baselines did not; the revision states this confound in the table and in the scope sentence.

Q2 / W2: energy and feature-MAD on the same checkpoints

Can the energy and feature-MAD measurements be reconciled on the same checkpoints?

Yes, and they coexist. On one set of checkpoints (d=32,64d = 32, 64; seeds 3–5) we measured Dirichlet energy, cosine feature-MAD, Jp95J_{p95}, the stack norm and per-layer gradient norms. At d=64d=64 (best checkpoint, mean of 3 seeds):

test acc. (%)log⁡∥Jeff∥\log\lVert J_{\text{eff}}\rVertfeature-MADper-layer grad. norm
residual-112.512.518.218.20.0000.00010610^{6}–10710^{7}
residual-1 + exact SN11.511.53.63.60.010.01
1/d1/\sqrt d scale69.469.45.95.90.450.45≤0.2\le 0.2
ReZero69.869.84.14.10.510.51≤0.2\le 0.2
d-STABLE71.671.65.05.00.830.83≤0.2\le 0.2

At d=32d=32, Jp95J_{p95} is 252252 for residual-1 against 3131–3434 for the 1/d1/\sqrt d scale, ReZero and d-STABLE (seeds 3–5; the submission's 157157 is seeds 0–2). At d=64d=64 the input-to-logit Jp95J_{p95} of these checkpoints does not fit in GPU memory, so we report the stack norm there; the submission's 7.8×1067.8\times10^{6} at d=64d=64 stands.

We therefore withdraw "not oversmoothing" and write: the residual-1 collapse is accompanied by gradient expansion; directional collapse (MAD →0\to 0) co-occurs.

Residual-1 + SN (Jp95=7.0J_{p95} = 7.0 at 11.711.7%) is the case that separates the two. Bounding the branch norm does not rescue a Θ(d)\Theta(d) budget, because the budget itself sets the bound: with ρk≤1\rho_k \le 1 and ∣sk∣=1|s_k| = 1, Theorem 1 allows ∥Jeff∥\lVert J_{\text{eff}}\rVert up to 2d2^d. At d=64d=64 the residual-1 + exact-SN stack starts at log⁡∥Jeff∥=12.9\log\lVert J_{\text{eff}}\rVert = 12.9 (d-STABLE: 0.60.6), with first-layer gradient norms near 2×1042\times10^{4} (d-STABLE: 0.010.01). Training brings the stack down to 3.63.6, inside the working range, yet the runs end directionally collapsed. At d=32d=32, two of three seeds train (63.863.8, 62.662.6%) and one collapses (22.822.8%).

Q3 / W2: random-sign control, and "Fixup"

At the same budget and a uniform magnitude ∣sk∣|s_k|:

signsone-step SN, β=1\beta=1one-step SN, β=4\beta=4no SN
all positive69.06±0.0669.06 \pm 0.0670.22±0.3470.22 \pm 0.3469.57±0.3769.57 \pm 0.37
random69.54±0.0869.54 \pm 0.0871.37±0.2571.37 \pm 0.2571.39±0.2071.39 \pm 0.20
alternating69.58±0.0869.58 \pm 0.0871.48±0.2471.48 \pm 0.2471.62±0.1171.62 \pm 0.11
learned69.58±0.0969.58 \pm 0.0971.63±0.2171.63 \pm 0.2171.48±0.2071.48 \pm 0.20

A frozen converged allocation gives 71.44±0.1971.44 \pm 0.19. Random and alternating signs tie the learned allocation, so the improvement is a sign-balance effect, not a learning or initialization-magnitude effect.

Linear-ended check. With the branch ending linearly (sign absorbable into WW) and exact SN at β=4\beta=4, all-positive and alternating tie (71.03±0.2871.03 \pm 0.28 vs 71.05±0.1371.05 \pm 0.13), so the sign effect is specific to the ReLU-ended branch, as Sec. 3.4 predicts.

"Fixup". Correct: our control is the constant scale sk=1/ds_k = 1/\sqrt d, not Zhang et al.'s full method (zero-initialized last layer, scalar biases and multipliers). The text now calls it "the 1/d1/\sqrt d scale" (rows and legends that still read "Fixup" denote this scale, stated in Sec. 4.2) and says that the full method is not run; its released code targets ResNets and Transformers, not GNNs.

Thanks for running these experiments. The exact-SVD normalization and sign controls resolve my main technical concerns, and I have raised soundness to 3. I am keeping the overall rating at 2 because the added value over fixed signed allocations is still limited. Please also clean up the old certificate and oversmoothing claims in Appendix E.1.

Thank you for reading the new results and for raising soundness. Your comment led us to one more control, and it changes a claim we made in the rebuttal. Changes of this round are in red in the uploaded PDF; earlier ones stay in blue.

1. Fixed signed allocations: you are right, and the depth result was confounded

We had answered that the logarithmic budget is needed at depth, because the 1/d1/\sqrt d scale degrades at d=128d = 128 and 256256 while d-STABLE holds. That comparison mixed two things: the 1/d1/\sqrt d scale was same-signed and d-STABLE has mixed signs. We reran it with alternating signs at a fixed uniform magnitude (GCN on ogbn-arxiv, same recipe, seeds 3–5):

dd1/d1/\sqrt d, same sign1/d1/\sqrt d, alternatinglog⁡d/d\log d/d, alternatingd-STABLE (learned)
6469.45±0.2969.45 \pm 0.2971.68±0.4571.68 \pm 0.4571.57±0.0571.57 \pm 0.0571.61±0.1071.61 \pm 0.10
12864.15±1.8564.15 \pm 1.8571.25±0.7571.25 \pm 0.7571.37±0.2371.37 \pm 0.2371.63±0.1871.63 \pm 0.18
25644.1±19.944.1 \pm 19.971.01±0.4871.01 \pm 0.4871.65±0.2071.65 \pm 0.2071.78±0.1771.78 \pm 0.17
  • Most of the depth separation is a sign effect. With alternating signs the 1/d1/\sqrt d scale trains on every seed at d=256d = 256 and stays within 0.60.6 points of the logarithmic budget. That gap is of the size of its seed spread, so up to 256256 layers we have no evidence that accuracy needs the logarithmic rate. The polynomial bound does need it.
  • A fixed alternating allocation matches the learned one at every depth, as you said.
  • Training does not find the sign balance. Started from all-positive signs, the learned allocation stays at 69.85±0.0869.85 \pm 0.08 (seeds 3–7) and at most one of its 64 signs changes.
  • Sign balance does not rescue a linear budget: residual-1 with sk=±1s_k = \pm 1 reaches 31.8±9.131.8 \pm 9.1.

We corrected the abstract, the introduction, Sec. 4.2 (the larger-depth item and Table 3, which now has the two alternating columns), the conclusion, and Appendices A and F. The statement in our general response and in our reply to Reviewer Vz6e (W3) that the logarithmic budget "holds where the d\sqrt d one degrades" should be read with this correction; we will post a note there.

2. What the paper contributes after this

  • The sign of the residual scale is an architectural degree of freedom when the branch ends in a ReLU. Same-signed scales plateau at d=64d = 64 and collapse at d=256d = 256; balanced signs remove both. With a linear-ended branch, which can absorb the sign, the effect vanishes (71.0371.03 vs 71.0571.05), as the symmetry result predicts.
  • The exact certificate: an a-priori bound dβd^{\beta} at no accuracy cost at β=4\beta = 4 on ogbn-arxiv, and at a measured cost of 1111 AP on Peptides-func.

We agree that learning the allocation adds no accuracy, and that the method contribution is narrower than the first version claimed. The revision now says so.

3. Appendix E.1

Fixed as you asked, and the same two claims are corrected where they also appeared in Sec. 2 and Appendix A.

  • Certificate. The one-step runs are no longer called certified: the paragraph is titled "One-step SN as an approximate control", and Table 7 lists dβd^{\beta} as a reference, not as a bound for those runs.
  • Oversmoothing. "The collapse is not oversmoothing" is removed from the text and captions. E.1 now says that the Dirichlet energy of every residual arm grows, and that the cosine feature-MAD of residual-1 is zero, so directional collapse co-occurs with the expansion.
  • Attribution. E.1 no longer says that "only the budget delivers" the accuracy; it attributes the margin among budgeted arms to sign balance.
Reviewer_Vz6eOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a normalized residual-gating parameterization method. Novelty lies in the logarithmic absolute-budget constraint, not per-layer gating. Ablation shows that allocation initialization or shape accounts for the improvement. However, benefit of learned gating appears limited.

Critical strengths

This paper provides thorough analysis on the budget-allocation view as a transparent theoretical justification. Experiments shown are meaningfully controlled and clear interpretable formulation of residual scaling.

Critical weaknesses
  1. The quantitative results doesn't demonstrate the necessity of learning the gates. As shown in Table 12 and 11, learned allocation only provides marginally improvement over frozen converged allocation which doesn't justify the added cost of this method.

  2. Empirical gains are regime-dependent as shown in Table 5 where there are accuracy degradation on every test ImageNet architecture. Table 4 also shows that d-STABLE doesn't consistently outperform existing stabilization methods across GNN operators.

  3. Shown quantitative results in Figure 7 and table 2 do not cleanly establish that log budget growth is necessary for observed stabilization. I.e. existing alternatives already avoid collapse at the tested depth. My understanding might be wrong, please confirm.

Questions for authors

Listed in weaknesses.

2 comments

Dear Reviewer Vz6e,

We appreciate your comments, and address your comments as follows.

W1: learning the allocation

learned allocation only provides marginally improvement over frozen converged allocation.

We agree, and the revision says so up front. On ogbn-arxiv the allocation barely moves from its initialization, and training it at 100×100\times the step changes nothing (71.44±0.2771.44 \pm 0.27 vs 71.48±0.2071.48 \pm 0.20). What carries the +1.7+1.7 over a fixed 1/k1/k profile at the same budget is the allocation's sign balance: a fixed uniform ∣sk∣|s_k| with random or alternating signs ties the learned profile (general comment, item 2).

The contribution we now claim is:

  1. the pinned log⁡d\log d budget (stability class);
  2. a sign-balanced allocation;
  3. an exact certificate at no accuracy cost at β=4\beta=4 (71.51±0.0671.51 \pm 0.06 vs 71.48±0.2071.48 \pm 0.20).

Cost. The parameterization adds dd scalar parameters and one normalization of dd scalars per step: +0.8+0.8% time per epoch at d=64d=64 and +1.0+1.0% at d=256d=256 against the fixed 1/d1/\sqrt d scale (same GPU model, same queue). The exact-SVD certificate adds a further ~30%.

W2: ImageNet, and consistency across operators

accuracy degradation on every tested ImageNet architecture; Table 4 not consistent across operators.

Correct, and we now frame ImageNet as the cost of the budget where networks are already stable. Two findings explain the size of that cost.

  • The ResNet-50 tanh cost (~2 points) is the parameterization collapsing onto one block, not the budget. Tanh magnitudes with positive signs give 75.275.2 and softplus magnitudes with mixed signs give 76.876.8, against 74.774.7 (tanh) and 76.976.9 (softplus); single seed.
  • Pre-normalized blocks already realize a d\sqrt d-class budget (the branch is 0-homogeneous in the stream), so a pinned log⁡d\log d budget can only constrain them.

For the cross-architecture table (Table 4 in the submission, now in the appendix for space) we state per operator where d-STABLE leads and where it does not: residual-1 collapses on GCN, GraphSAGE and GCNII at d=64d=64, trails by 4.64.6 points on GAT at d=16d=16, and ties d-STABLE on TransformerConv at d=8d=8 (71.571.5 vs 71.671.6).

W3: is the logarithmic rate needed?

existing alternatives already avoid collapse at the tested depth.

Correct at d=64d=64: d=8\sqrt d = 8 and log⁡d≈4.2\log d \approx 4.2 are both small, and at that depth the 1/d1/\sqrt d control trails a fixed log⁡d\log d profile by only 0.40.4 points. The accuracy separation between classes appeared only in the synthetic chains up to d=1024d=1024. We ran d=128d=128 and d=256d=256 on ogbn-arxiv (seeds 3–5, same recipe):

depthresidual-11/d1/\sqrt d scaleReZerod-STABLE (log⁡d\log d)d-STABLE + exact SN, β=4\beta=4
6412.5±8.512.5 \pm 8.569.45±0.2969.45 \pm 0.2969.78±0.1569.78 \pm 0.1571.61±0.1071.61 \pm 0.1071.51±0.0671.51 \pm 0.06
12816.9±9.616.9 \pm 9.664.15±1.8564.15 \pm 1.8570.38±0.2970.38 \pm 0.2971.63±0.1871.63 \pm 0.1871.63±0.1671.63 \pm 0.16
256—44.1±19.944.1 \pm 19.970.90±0.1670.90 \pm 0.1671.78±0.1771.78 \pm 0.1771.62±0.1971.62 \pm 0.19

The 1/d1/\sqrt d scale degrades with depth (−5.3-5.3 points from d=64d=64 to 128128; at 256256 one of three seeds collapses to a constant prediction, 21.621.6%), while the log⁡d\log d budget holds at 71.671.6–71.871.8 at every depth. ReZero also trains at d=256d=256: its learned scales stay small (∑k∣αk∣=1.05\sum_k|\alpha_k| = 1.05 at d=256d=256, below log⁡d=5.5\log d = 5.5), which places it in the same polynomial class; it trails d-STABLE by 0.90.9–1.81.8 points. So the d\sqrt d-vs-log⁡d\log d separation that the synthetic chains showed at d=1024d=1024 appears in accuracy on a real task by d=128d=128–256256.

Thanks for the detailed response. Though, it doesn't clearly answer my question and response seems convoluted. I'll stick with my original scores.

Reviewer_N3kVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper analyzes the sensitivity of residual networks and shows that it is governed by the total residual budget, the summed strength of the per-layer residual branches. The authors argue that the budget and its allocation across layers are separable: the budget's growth rate with depth d determines the stability class, with log d identified as the critical rate for polynomial sensitivity, while the allocation determines accuracy within that class. Based on this, they propose d-STABLE, a reparameterization that fixes the total budget at beta-log d while the optimizer learns the allocation. Experiments are conducted on GNNs (ogbn-arxiv), ResNet-50, and DeiT on ImageNet-1k. On deep GNNs, the method prevents accuracy collapse and improves over existing stabilizers, while on ImageNet it incurs a small accuracy cost.

Critical strengths

none

Critical weaknesses
  1. The unified view is not well justified. The paper frames oversmoothing, rank collapse, and exploding gradients as two ends of a single phenomenon governed by the residual budget. However, these failure modes are typically attributed to specific components, such as aggregation in GNNs and self-attention in Transformers, and are also affected by normalization and activations. This unification is the paper's central conceptual contribution. If the budget alone cannot account for component-specific effects, the theory’s explanatory scope is much narrower than claimed.

  2. The method underperforms the vanilla baseline on standard large-scale benchmarks. On ImageNet-1k, d-STABLE reduces top-1 accuracy relative to the vanilla models. Its clear benefit appears mainly on deep GNNs on a single benchmark (ogbn-arxiv). ImageNet with ResNet and Transformers is the most widely used setting for residual networks. A method that degrades accuracy there, while helping only in a narrow deep-GNN setting, has limited practical value, and this weakens the claim that the residual-budget view applies generally across architectures.

Questions for authors

Q1. The Introduction opens with the claim that “depth is the most direct way to give a model more computation” and frames oversmoothing, rank collapse, and exploding gradients as "the same object at opposite ends." However, these phenomena are typically attributed to specific architectural components, such as repeated neighborhood aggregation in GNNs and self-attention in Transformers, and are also affected by normalization and activations. How does the residual-budget view account for these components? Is the budget alone sufficient to explain the different behaviors observed across GNNs, ResNet, and DeiT?

Q2. The equation in the Introduction uses notation that has not been defined, which makes it difficult to follow. More broadly, many results are stated without derivations or explanation. Please define all notation before use and provide the supporting derivations.

Q3. The findings vary across datasets and network structures, yet the experiments do not cover a broad range of existing architectures. How general are the conclusions, and how should practitioners apply them? Additional architectures would strengthen the empirical support.

2 comments

Dear Reviewer N3kV,

We appreciate your feedback and your time. We address your concerns as follows.

W1 / Q1: is the budget alone sufficient?

Is the budget alone sufficient to explain the different behaviors observed across GNNs, ResNet, and DeiT?

No, and the revision no longer claims it. The bound is on the weighted budget Sd=∑kρk∣sk∣S_d = \sum_k \rho_k |s_k|, where ρk\rho_k is the full-branch Jacobian norm. Component effects (aggregation in a GNN, attention in a Transformer, normalization, activation) enter through ρk\rho_k; the budget ∑k∣sk∣\sum_k|s_k| is the factor a residual design controls directly. Measured on 40 checkpoints across branch types, normalizations and schedules, SdS_d orders the end-to-end amplification (Spearman 0.830.83), whereas ∑k∣sk∣\sum_k|s_k| alone does not (0.160.16). We removed the "same object at opposite ends" sentence and rewrote the introduction around this two-factor statement.

Q2: undefined notation

Every symbol in the introduction's display is now defined before it appears, and each stated result points to where it is derived (Sec. 3, with proofs in the appendix).

W2 / Q3: ImageNet cost, breadth, and guidance

degrades accuracy on ImageNet … helps only in a narrow deep-GNN setting.

We agree that ImageNet shows a cost (0.50.5 points on ResNet-50 softplus) and now say where the method applies: deep residual stacks whose branches are not already normalized in the stream. Beyond ogbn-arxiv the paper covers pubmed, heterophilic graphs, GraphSAGE/GAT/GCNII operators, and the long-range Peptides-func benchmark (a controlled depth test on one minimal backbone; rerun to convergence, β≥4\beta \ge 4 exceeds the learned gates by 0.020.02–0.030.03 AP). New in the revision:

  • DeeperGCN ResGEN-28 (official code and recipe; our five native seeds give 71.80±0.2771.80 \pm 0.27 against the reported 71.92±0.1671.92 \pm 0.16): d-STABLE reaches 71.96±0.2171.96 \pm 0.21 (β=1\beta=1) and 71.79±0.1371.79 \pm 0.13 (β=4\beta=4), n=5n=5 each. This is parity, as expected for a pre-normalized block.
  • Depth 128 and 256 on GCN: the 1/d1/\sqrt d scale falls to 64.264.2 and 44.144.1 while d-STABLE holds at 71.671.6 and 71.871.8 (reply to Vz6e, W3).
  • Deep pre-LN ViT on CIFAR-100 (width 192, patch 4, 300 epochs, one seed; mean over the last 5 epochs):
depth 12depth 48
vanilla71.2371.2369.9369.93
LayerScale72.2672.2672.6072.60
d-STABLE (softplus)73.0873.0872.1572.15

d-STABLE leads at depth 12; at depth 48 LayerScale leads by 0.450.45, and both stay ahead of vanilla.

A "when to use it" paragraph gives practitioners the rule: the budget binds when branches are unnormalized or post-activation and the stack is deep; with pre-normalization it is a cost.

Thank you for the replies. However, the proposed method does not clearly outperform the compared methods. In addition, the manuscript still needs further improvement. Therefore, I maintain my previous score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.