- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper separates the total magnitude of residual coefficients from their allocation across depth. It proposes normalized signed or nonnegative coefficients whose absolute sum is bounded by beta log(d) throughout training. Under uniform bounds on the complete residual-branch Jacobians, a standard product inequality then gives a polynomial upper bound on input sensitivity. Experiments study synthetic chains, several graph-network architectures and ImageNet models. The graph experiments report improved deep-network accuracy and include controls for matched budgets, frozen allocations and branch normalization. The paper also reports cases where learning the allocation adds little and where the budget reduces ImageNet accuracy.
Critical strengths
- The parameterization is simple and makes a relevant quantity explicit. Theorem 1 cleanly states the full-branch assumptions, and the paper generally distinguishes a worst-case sensitivity guarantee from optimizer convergence and adversarial robustness.
- The empirical investigation includes unusually informative controls: matched-budget and frozen-profile comparisons, norm-on/off experiments, multiple graph operators and repeated seeds for the main result. I appreciate the explicit reporting that the ogbn-arxiv margin largely comes from profile shape rather than learning, and that ImageNet accuracy can decrease. These findings help delimit the method's usefulness.
Critical weaknesses
-
The implemented “a-priori certified” configuration does not yet establish the claimed certificate. Appendix E.1 (p.20) states that normalization uses one power iteration and that realized branch norms reach 1.15. This does not enforce a uniform bound of 1, while Table 8 labels d^beta as an a-priori reference. Observing sampled J_p95 below that reference cannot certify a supremum over inputs. I appreciate the explicit disclosure of the approximation, but the distinction matters because the claimed cost of obtaining a certificate is a central result. Please either enforce a defensible full-branch norm bound and use its actual exponent, or describe these runs as empirical approximate control. It should also be clear whether the measured Jacobian is for the residual stack or includes the encoder and classifier excluded from the stated bound.
-
The evidence does not yet isolate the central mechanism and its advantage over simpler alternatives. The claim that residual-1 fails through gradient expansion rather than oversmoothing needs reconciliation with Table 10 (p.23), which reports feature-MAD=0.00 at depths 32 and 64 and defines zero as directional collapse. Increasing normalized Dirichlet energy and directional feature collapse need not be mutually exclusive. Moreover, the frozen-profile result is valuable evidence that allocation learning is unnecessary for the headline graph gain; the random-sign control identified in Appendix E.2 would clarify whether the improvement is primarily a signed-initialization effect. The comparator terminology also matters: the fixed s_k=1/sqrt(d) control called “Fixup” is not the full initialization-and-learned-multiplier method of Zhang et al. (2019, Section 3). Renaming that control, or explaining the implemented correspondence, would make the scope of the comparative claims more reliable.
Questions for authors
- What full-branch operator bound is actually enforced in the configuration called certified? Can Table 8 report the corresponding valid upper bound, and specify the input/output map used for J_p95? Please also reconcile the normalization description of the heterophilic rows in Appendix E.5 with the main-text scope statement.
- Can the energy and feature-MAD measurements be reconciled on the same checkpoints? Do they support coexisting amplification and directional collapse, rather than excluding one mechanism? The residual-1+SN case with J_p95=7.0 but 11.7% accuracy in Table 7 is particularly informative here.
- At the same budget, how does a frozen random-sign, uniform-magnitude allocation compare with the learned and frozen-converged profiles? Please clarify whether the “Fixup” implementation contains the original initialization, biases and learned multipliers, or is only a constant-scale control.
3 comments
Thanks for running these experiments. The exact-SVD normalization and sign controls resolve my main technical concerns, and I have raised soundness to 3. I am keeping the overall rating at 2 because the added value over fixed signed allocations is still limited. Please also clean up the old certificate and oversmoothing claims in Appendix E.1.
Thank you for reading the new results and for raising soundness. Your comment led us to one more control, and it changes a claim we made in the rebuttal. Changes of this round are in red in the uploaded PDF; earlier ones stay in blue.
1. Fixed signed allocations: you are right, and the depth result was confounded
We had answered that the logarithmic budget is needed at depth, because the scale degrades at and while d-STABLE holds. That comparison mixed two things: the scale was same-signed and d-STABLE has mixed signs. We reran it with alternating signs at a fixed uniform magnitude (GCN on ogbn-arxiv, same recipe, seeds 3–5):
| , same sign | , alternating | , alternating | d-STABLE (learned) | |
|---|---|---|---|---|
| 64 | ||||
| 128 | ||||
| 256 |
- Most of the depth separation is a sign effect. With alternating signs the scale trains on every seed at and stays within points of the logarithmic budget. That gap is of the size of its seed spread, so up to layers we have no evidence that accuracy needs the logarithmic rate. The polynomial bound does need it.
- A fixed alternating allocation matches the learned one at every depth, as you said.
- Training does not find the sign balance. Started from all-positive signs, the learned allocation stays at (seeds 3–7) and at most one of its 64 signs changes.
- Sign balance does not rescue a linear budget: residual-1 with reaches .
We corrected the abstract, the introduction, Sec. 4.2 (the larger-depth item and Table 3, which now has the two alternating columns), the conclusion, and Appendices A and F. The statement in our general response and in our reply to Reviewer Vz6e (W3) that the logarithmic budget "holds where the one degrades" should be read with this correction; we will post a note there.
2. What the paper contributes after this
- The sign of the residual scale is an architectural degree of freedom when the branch ends in a ReLU. Same-signed scales plateau at and collapse at ; balanced signs remove both. With a linear-ended branch, which can absorb the sign, the effect vanishes ( vs ), as the symmetry result predicts.
- The exact certificate: an a-priori bound at no accuracy cost at on
ogbn-arxiv, and at a measured cost of AP on Peptides-func.
We agree that learning the allocation adds no accuracy, and that the method contribution is narrower than the first version claimed. The revision now says so.
3. Appendix E.1
Fixed as you asked, and the same two claims are corrected where they also appeared in Sec. 2 and Appendix A.
- Certificate. The one-step runs are no longer called certified: the paragraph is titled "One-step SN as an approximate control", and Table 7 lists as a reference, not as a bound for those runs.
- Oversmoothing. "The collapse is not oversmoothing" is removed from the text and captions. E.1 now says that the Dirichlet energy of every residual arm grows, and that the cosine feature-MAD of residual-1 is zero, so directional collapse co-occurs with the expansion.
- Attribution. E.1 no longer says that "only the budget delivers" the accuracy; it attributes the margin among budgeted arms to sign balance.
Dear Reviewer ySwY,
Thank you for the close reading. Each of your three questions changed the paper.
Q1 / W1: the certificate
You are right: one power-iteration step under-reads σ(W), and the realized branch norms reach 1.15, so the submitted runs did not enforce ρ≤1. We made three changes.
Which map. The paper's Jp95 is ∥∂logits[v]/∂X∥, encoder and classifier included, whereas Theorem 1 bounds the residual stack ∂H(d)/∂H(0). The revision reports both. On the exact-SN runs (d=64, seeds 3–7):
The bound holds at every β and is tightest at β=1.
Heterophilic rows. The d-STABLE rows used SN and the baselines did not; the revision states this confound in the table and in the scope sentence.
Q2 / W2: energy and feature-MAD on the same checkpoints
Yes, and they coexist. On one set of checkpoints (d=32,64; seeds 3–5) we measured Dirichlet energy, cosine feature-MAD, Jp95, the stack norm and per-layer gradient norms. At d=64 (best checkpoint, mean of 3 seeds):
At d=32, Jp95 is 252 for residual-1 against 31–34 for the 1/d scale, ReZero and d-STABLE (seeds 3–5; the submission's 157 is seeds 0–2). At d=64 the input-to-logit Jp95 of these checkpoints does not fit in GPU memory, so we report the stack norm there; the submission's 7.8×106 at d=64 stands.
We therefore withdraw "not oversmoothing" and write: the residual-1 collapse is accompanied by gradient expansion; directional collapse (MAD →0) co-occurs.
Residual-1 + SN (Jp95=7.0 at 11.7%) is the case that separates the two. Bounding the branch norm does not rescue a Θ(d) budget, because the budget itself sets the bound: with ρk≤1 and ∣sk∣=1, Theorem 1 allows ∥Jeff∥ up to 2d. At d=64 the residual-1 + exact-SN stack starts at log∥Jeff∥=12.9 (d-STABLE: 0.6), with first-layer gradient norms near 2×104 (d-STABLE: 0.01). Training brings the stack down to 3.6, inside the working range, yet the runs end directionally collapsed. At d=32, two of three seeds train (63.8, 62.6%) and one collapses (22.8%).
Q3 / W2: random-sign control, and "Fixup"
At the same budget and a uniform magnitude ∣sk∣:
A frozen converged allocation gives 71.44±0.19. Random and alternating signs tie the learned allocation, so the improvement is a sign-balance effect, not a learning or initialization-magnitude effect.
Linear-ended check. With the branch ending linearly (sign absorbable into W) and exact SN at β=4, all-positive and alternating tie (71.03±0.28 vs 71.05±0.13), so the sign effect is specific to the ReLU-ended branch, as Sec. 3.4 predicts.
"Fixup". Correct: our control is the constant scale sk=1/d, not Zhang et al.'s full method (zero-initialized last layer, scalar biases and multipliers). The text now calls it "the 1/d scale" (rows and legends that still read "Fixup" denote this scale, stated in Sec. 4.2) and says that the full method is not run; its released code targets ResNets and Transformers, not GNNs.