acceptodds
Under review as a conference paper at ICLR 2027

A Qualitative Test-Risk Mechanism for Scaling Behavior in Normalized Residual Networks

Abstract

The scaling behavior, in which test performance often improves as model size and data increase, is a central empirical phenomenon in modern deep learning, yet its theoretical basis remains incomplete. In this paper, we study depth expansion in normalized residual networks: starting from a trained model in an old hypothesis class, we insert a new residual block at an intermediate layer and ask when such an expansion can yield a provable improvement in test risk. We develop a unified framework that decomposes this question into representational gain, optimization gain, and generalization transfer. First, under a first-order descent condition near zero initialization, we prove that the expanded hypothesis class contains an auxiliary jumpboard model with strictly smaller population risk than the original model. Second, under norm control tailored to post-normalized residual architectures, we establish a norm-based Rademacher complexity bound for the expanded model class. These ingredients lead to two complementary test-risk guarantees: one route passes through population risk and is tighter when a positive population margin is available, while the other works directly at the train/test level, avoids Hoeffding transfer, and is more robust in degenerate regimes. Together, these results provide a theorem-driven mechanism under which residual depth expansion can improve test performance in normalized residual networks. More broadly, they suggest that scaling is inherently joint: depth creates new improving directions, width enhances the finite-sample observability of weak signals, and data determines whether the statistical cost of expansion can be controlled.

Primary area:
foundation or frontier models, including LLMs
Keywords:
scaling law

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6HnwOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether inserting a residual block into a trained normalized residual network can improve test performance. Its framework combines a locally improving “jumpboard” model, an empirical selection rule, and uniform generalization bounds. It additionally analyzes train–test activation-gradient alignment to motivate a beneficial role for width, and presents experiments on residual networks and LLaMA-style models.

Critical strengths
  1. The paper addresses a meaningful distinction between enlarging a hypothesis class and improving the test performance of a selected model.

  2. The basic risk-comparison inequalities are valid given the stated selection rule and an appropriate fixed-class uniform generalization bound. The fallback construction usefully avoids assuming global optimization.

  3. The discussion of zero-output initialization distinguishes zeroing the final projection from zeroing the entire residual branch. The treatment of stabilized normalization also clarifies how normalization singularities can be avoided.

Critical weaknesses
  1. The paper gives limited insight into why adding depth improves test risk. Theorems 1–2 combine uniform generalization bounds with an empirical comparison rule that applies to general model classes. Condition 1 already assumes that the inserted block has a first-order descent direction. Once this is assumed, the existence of a better jumpboard model follows from differentiability. What remains unexplained is why trained residual networks should have such directions and when the resulting gains are large enough to exceed the generalization penalty. These are central questions for the claimed depth-scaling mechanism.

  2. The link between width and test-risk improvement remains unclear for general residual blocks. Condition 1 concerns a descent direction in the inserted block’s parameters. Theorem 4 instead studies μ⊤g\mu^\top g, the alignment of average activation gradients. Even when these gradients align, the selected parameter direction need not reduce test loss. Corollary 3 connects the two under an additional realizability assumption, but this assumption is not established for the general residual blocks discussed. Without that connection, better gradient alignment does not show that width makes residual updates more likely to improve test performance.

  3. The sufficient depth–width–data condition in Appendix M.5 is not established. The argument uses Δk=O(k−1/β)\Delta_k=O(k^{-1/\beta}) and a signal proportional to N(P)Δk1+βN(P)\Delta_k^{1+\beta} to upper-bound the misalignment probability. However, the probability bound contains the reciprocal of the signal. Controlling this reciprocal from above requires a lower bound on the signal, which the upper bound on Δk\Delta_k does not provide. The mixed covariance term also requires control in the weak-signal regime; controlling the leading terms alone is insufficient. These gaps leave the stated sufficient reliability condition unsupported by the given assumptions.

Questions for authors
  1. What architecture-specific insight establishes useful depth expansion beyond the local descent assumption in Condition 1 and the generalization argument in Theorems 1–2? Is there a concrete setting analyzed in the paper where the jumpboard improvement can be shown to exceed the generalization penalty?

  2. For which residual-block parameterizations does the additional realizability assumption in Corollary 3 hold? Can you establish this connection for the blocks covered by your main claims, so that μ⊤g>0\mu^\top g>0 implies test-loss descent along the selected residual update?

  3. In Appendix M.5, what lower bound on ∥μˉk,P∥2\|\bar\mu_{k,P}\|^2 justifies the claimed upper bound on misalignment probability? Since Δk=O(k−1/β)\Delta_k=O(k^{-1/\beta}) provides only an upper bound on the signal, how should the sufficient depth–width–data condition be derived, including control of the mixed covariance term?

8 comments
AuthorsOpenReview ↗

Response to Reviewer 6Hnw (1/5): W1/Q1 (1)

We thank the reviewer for the careful reading, and in particular for the precise analysis of Appendix M.5. Our response is split into five comments. Page numbers refer to the revised PDF.

W1 / Q1 (part 1 of 2)

1. Scope. The main text proves conditional, qualitative statements. Page 3 states that Condition 1 is not meant to hold for every trained model or insertion layer, and that we make no strict-improvement claim when it fails. We agree that the comparison inequalities of Theorems 1 and 2 hold for general model classes; this is by design. Once the architecture-independent comparison step is isolated, the architecture-specific content sits in three places: the exact embedding and tangent directions of zero-output residual blocks (Appendices C and P); the norm-based generalization bound for normalized residual networks (Lemma 1); and train–test alignment on the residual stream (Theorem 4, Corollary 3).

2. Status of Condition 1. For a fixed insertion layer, Condition 1 is necessary and sufficient for the inserted block to admit a strict first-order descent direction, and it is equivalent to a nonzero risk gradient with respect to the new block's parameters (Proposition 1, Appendix P.4, pp. 66–67). For zero-output projection blocks it is equivalent to Cpop=E[qψ⊤]≠0C_{\rm pop}=\mathbb E[q\psi^\top]\neq 0 (Corollary 6, p. 70, where this matrix is denoted MpopM_{\rm pop}). A recent preprint [X] gives a corresponding characterization under a fixed function-preserving first-order growth protocol: at each candidate location, Condition 1 is equivalent to a nonvanishing parameter gradient of the inserted block, and to the activation-gradient signal not being orthogonal to the block's tangent space. The equivalence itself is elementary; its role is to fix the status of the condition. Any strict improvement obtained through a first-order direction of the new block requires Condition 1, so it delimits the scope of the first-order mechanism, and the hypothesis of our conditional statements cannot be weakened within that mechanism. "First-order" is an essential qualifier: higher-order or non-local improvements lie outside the equivalence (Appendix P.10).

3. What the residual structure contributes. Take hU,V(z)=VψU(z)h_{U,V}(z)=V\psi_U(z) and zero only the output projection (V=0V=0).

  • The old model is embedded exactly, fold∗∈Hnewf^\ast_{\rm old}\in\mathcal H_{\rm new}, so a fallback comparison is always available. In the worst case the comparator is the old model itself, and Appendix H (p. 38) gives Ltest(fnew)≤Ltest(fold∗)−ΔERM+2(ϵM+ϵK)L_{\rm test}(f_{\rm new})\le L_{\rm test}(f^\ast_{\rm old})-\Delta_{\rm ERM}+2(\epsilon_M+\epsilon_K), with no convergence assumption on the optimizer.
  • The tangent directions of the new parameters do not vanish: along V(η)=ηΔVV(\eta)=\eta\Delta V the first-order perturbation is ΔVψU(z)\Delta V\psi_U(z). Zeroing the entire branch can make the parameter Jacobian vanish, and the input Jacobian is identically zero at V=0V=0 (Appendices C.5 and C.6, pp. 19–23).
  • In this parameterization Condition 1 becomes a computable criterion, Cpop≠0C_{\rm pop}\neq 0; with ΔV=−Cpop\Delta V=-C_{\rm pop} the directional derivative equals −∥Cpop∥F2<0-\Vert C_{\rm pop}\Vert_F^2\lt 0. Its training-set version CS≠0C_S\neq 0 yields a constructible training-side jumpboard (Corollary 1, pp. 21–22).
AuthorsOpenReview ↗

Response to Reviewer 6Hnw (1/5): W1/Q1 (2)

W1 / Q1 (part 2 of 2)

4. Why a trained network can still satisfy Condition 1.

  • Stationarity of the old model in its own parameters removes only the directions those parameters can generate. It does not make the activation-gradient signal qq orthogonal to the tangent directions of the new block (Appendices C.7 and P.8, pp. 23 and 71). In the decomposition of Appendix C, Condition 1 requires (i) a descent direction in hidden space and (ii) that the block realizes one such direction to first order. For (i) to fail one needs E[q⊤ϕ(z)]=0\mathbb E[q^\top\phi(z)]=0 for every perturbation ϕ\phi of the representation, which is far stronger than parameter stationarity.
  • For a given zero-output projection block, Condition 1 fails if and only if Cpop=0C_{\rm pop}=0, i.e., E[qiψj]=0\mathbb E[q_i\psi_j]=0 for every coordinate of the signal and every residual feature. If this holds at every admissible layer, the model is in the deepest-model regime of Section 5, where we claim no first-order strict improvement.
  • We do not claim that the residual architecture alone guarantees non-orthogonality. What the paper provides is the exact characterization of the condition, its compatibility with stationarity of the old model, and measurable evidence. In Figure 2 and Table 3 the norm of the mean activation gradient is clearly nonzero (it decays with depth in Figure 2). In Table 1, the jumpboard built from the activation gradient and realized by an actual residual block has lower training loss and lower test loss than the corresponding old model at all three growth steps.

5. When the gain exceeds the generalization penalty.

Criterion. Theorems 1 and 2 give ΔRS+ΔERM>2ϵM\Delta_R^S+\Delta_{\rm ERM}\gt 2\epsilon_M and ΔRtest+ΔERM>2(ϵM+ϵK)\Delta_R^{\rm test}+\Delta_{\rm ERM}\gt 2(\epsilon_M+\epsilon_K).

A concrete setting (population route, Theorem 7, p. 40). Fix the reference model and an insertion layer satisfying the population form of Condition 1 (Cpop≠0C_{\rm pop}\neq 0 for zero-output projection blocks). The margin ΔR>0\Delta_R\gt 0 of the population jumpboard is determined by the data distribution and the reference model only; it does not depend on the training sample size MM. Meanwhile ϵM=O~(BℓdΓarch/M)\epsilon_M=\tilde O\big(B_\ell\sqrt{d\Gamma_{\rm arch}/M}\big) (Lemma 1, Corollary 2). Theorem 7 gives, with probability at least 1−2δ1-2\delta,

R(fnew)≤R(fold∗)−ΔR−ΔERM+2ϵM.R(f_{\rm new})\le R(f^\ast_{\rm old})-\Delta_R-\Delta_{\rm ERM}+2\epsilon_M.

By Appendix F.1 (pp. 33–34), M≳Bℓ2(dΓarch+log⁡(1/δ))/ΔR2M\gtrsim B_\ell^2\big(d\Gamma_{\rm arch}+\log(1/\delta)\big)/\Delta_R^2 ensures 2ϵM<ΔR2\epsilon_M\lt \Delta_R, hence R(fnew)<R(fold∗)R(f_{\rm new})\lt R(f^\ast_{\rm old}) even when ΔERM=0\Delta_{\rm ERM}=0. The hypothesis is Condition 2: the final model is no worse on the training set than the comparison jumpboard.

Limits of this statement. Under the C1C^1 assumptions of the main text, ΔR\Delta_R is only known to be positive; its size is not quantified. Appendix J (p. 41) already describes it as a directional certificate and notes that in practice the dominant improvement is often carried by ΔERM\Delta_{\rm ERM}. The main text therefore answers which inequality decides and what order of sample size is needed; it does not claim a closed-form lower bound on ΔR\Delta_R.

Role of the experiments. Table 1 is an empirical observation of the mechanism, not a numerical verification that the gain exceeds the theoretical bound. Norm-based bounds are numerically loose, and we make no such claim.

6. A further quantitative supplement (new Appendix M; not needed for the main results). Assume in addition a curvature bound along the update ray (M2), boundedness at the current prediction (M3), and fresh direction samples (M5). Consider the update V=−(λ/βV)C^V=-(\lambda/\beta_V)\widehat C built from the empirical direction C^\widehat C (the CSC_S of Appendix C). With the step-size factor λ=min⁡{1,A/Γ}\lambda=\min\lbrace 1,A/\Gamma\rbrace, Theorem M.5 (p. 51) gives, with probability at least 1−δ1-\delta, whenever s≥2a\sqrt{s}\ge 2a,

R(fk)−R(fk+1)≥Dk:=116βVmin⁡{s,s2e2},s=∥Cpop∥F2,R(f_k)-R(f_{k+1})\ge D_k:=\frac{1}{16\beta_V}\min\Big\lbrace s,\frac{s^2}{e^2}\Big\rbrace,\qquad s=\Vert C_{\rm pop}\Vert_F^2,

where aa and ee are explicit O(M−1/2)O(M^{-1/2}) noise terms; Lemma M.6 gives a guarantee of the same type for a fully data-driven step size. For softmax cross-entropy with a fixed full-row-rank linear head, Corollary M.11 (pp. 57–58) verifies (M1)–(M3) and provides all constants. This is a setting in which the gain has an explicit lower bound that already accounts for the direction-estimation error. When the hypotheses of both results hold, inserting it into Theorem 1 gives the sufficient condition Dk+ΔERM>2ϵMD_k+\Delta_{\rm ERM}\gt 2\epsilon_M for strict improvement of the final model.

AuthorsOpenReview ↗

Response to Reviewer 6Hnw (3/5): W2/Q2

W2 / Q2

1. Which parameterizations satisfy the condition of Corollary 3. It holds for residual blocks with a trainable output bias, hU,V,a(z)=VψU(z)+ah_{U,V,a}(z)=V\psi_U(z)+a, initialized at V=0V=0, a=0a=0. Along the path V(η)=0V(\eta)=0, a(η)=−ημa(\eta)=-\eta\mu, one has hη(z)≡−ημh_\eta(z)\equiv-\eta\mu for every input, hence

ddηLtrain(fη)∣η=0=−∥μ∥2,ddηLtest(fη)∣η=0=−μ⊤g.\frac{d}{d\eta}L_{\rm train}(f_\eta)\Big\vert_{\eta=0}=-\Vert \mu\Vert^2,\qquad\frac{d}{d\eta}L_{\rm test}(f_\eta)\Big\vert_{\eta=0}=-\mu^\top g.

For the bias parameter, the parameter gradient is the mean activation gradient: ∇aLtrain=μ\nabla_a L_{\rm train}=\mu and ∇aLtest=g\nabla_a L_{\rm test}=g. In this direction the training-set form of Condition 1 reads μ≠0\mu\neq 0, the selected parameter direction is −μ-\mu, and the quantity μ⊤g\mu^\top g controlled by Theorem 4 is exactly the first-order change of the test loss along it. For such blocks, μ⊤g>0\mu^\top g\gt 0 implies strict test-loss descent for sufficiently small steps, without any additional realizability assumption (Corollary 3, p. 39).

2. Correspondence with the experiments. Appendix N.5 (pp. 59–61) defines the sequence-level signal as the derivative of the loss with respect to an additive residual shared by all token positions,

qs=∂∂uLs(hs,1+u,…,hs,T+u)∣u=0=∑t=1T∇hs,tLs.q_s=\frac{\partial}{\partial u}L_s(h_{s,1}+u,\dots,h_{s,T}+u)\Big\vert_{u=0}=\sum_{t=1}^{T}\nabla_{h_{s,t}}L_s.

This is the gradient of the loss with respect to the output bias of a zero-initialized residual block inserted at the measurement position. Table 2 therefore estimates the probability that updating this bias along −μM-\mu_M decreases the test loss to first order. Two caveats: Table 2 measures the sign of the first-order alignment, not the finite-step loss change after fully training the new block; and the correspondence concerns the inserted block, so we do not claim that existing LLaMA sublayers carry output biases.

3. General output-projection blocks. For h(z)=Vψ(z)h(z)=V\psi(z) with features ψ=ψU0\psi=\psi_{U_0} fixed at initialization (as in Corollary 1), the training-selected direction is ΔV=−CS\Delta V=-C_S with CS=1M∑iqiψ(zi)⊤C_S=\frac1M\sum_i q_i\psi(z_i)^\top, and

ddηLtest(fη)∣η=0=−⟨CS,Ctest⟩F,Ctest:=1K∑jq~jψ(z~j)⊤.\frac{d}{d\eta}L_{\rm test}(f_\eta)\Big\vert_{\eta=0}=-\langle C_S,C_{\rm test}\rangle_F,\qquad C_{\rm test}:=\frac1K\sum_j\tilde q_j\psi(\tilde z_j)^\top.

So the reviewer is right: for general blocks, μ⊤g>0\mu^\top g\gt 0 does not by itself give test descent along this direction; the relevant quantity is ⟨CS,Ctest⟩F\langle C_S,C_{\rm test}\rangle_F. The proof of Theorem 4, however, uses only that μ\mu and gg are two independent sample means of one random vector with nonzero mean and bounded largest covariance eigenvalue. Under the same sampling convention (with the feature initialization independent of the samples), applying it verbatim to ξ:=vec(qψ(z)⊤)\xi:=\mathrm{vec}(q\psi(z)^\top), with mean vec(Cpop)\mathrm{vec}(C_{\rm pop}) and covariance Σξ\Sigma_\xi, gives

Pr⁡(⟨CS,Ctest⟩F≤0)≤4λmax⁡(Σξ)M∥Cpop∥F2+4λmax⁡(Σξ)K∥Cpop∥F2+4λmax⁡(Σξ)tr(Σξ)KM∥Cpop∥F4.\Pr\big(\langle C_S,C_{\rm test}\rangle_F\le 0\big)\le\frac{4\lambda_{\max}(\Sigma_\xi)}{M\Vert C_{\rm pop}\Vert_F^2}+\frac{4\lambda_{\max}(\Sigma_\xi)}{K\Vert C_{\rm pop}\Vert_F^2}+\frac{4\lambda_{\max}(\Sigma_\xi)\mathrm{tr}(\Sigma_\xi)}{KM\Vert C_{\rm pop}\Vert_F^4}.

Here Cpop≠0C_{\rm pop}\neq 0 is exactly Condition 1. Thus, for the zero-output projection blocks covered by our main claims, the link between the parameter direction selected under Condition 1 and first-order test descent is given by the same theorem, with (μ,g,μˉ,Σ)(\mu,g,\bar\mu,\Sigma) replaced by (CS,Ctest,Cpop,Σξ)(C_S,C_{\rm test},C_{\rm pop},\Sigma_\xi). The bias case is ψ≡1\psi\equiv 1, where the bound reduces to Theorem 4. We will add this as a remark in Appendices E and H, and state in the main text that Corollary 3 as originally written concerns the bias-type direction.

4. Scope of the width statement. The form O(1MN+1KN)O\big(\frac{1}{MN}+\frac{1}{KN}\big) is derived for the bias-type direction, under Assumption 3 and the uniformly active regime (∥μˉ∥2=Θ(N)\Vert \bar\mu\Vert^2=\Theta(N), tr(Σ)≤Nτ2\mathrm{tr}(\Sigma)\le N\tau^2); Appendix J (p. 42) already calls it the bias-type special case. For general projection blocks the same role is played by the ratio of ∥Cpop∥F2\Vert C_{\rm pop}\Vert_F^2 to λmax⁡(Σξ)\lambda_{\max}(\Sigma_\xi). Whether this ratio grows with width depends on the features, and we make no such claim for general blocks. Empirically, the alignment probability in Table 2 increases with width and sample size, and the signal ∥μ^∥2\Vert \hat\mu\Vert^2 in Table 3 increases monotonically with width (from 0.15 to 0.26), though more slowly than linearly. We present these as evidence consistent with the mechanism, not as a verification of Θ(N)\Theta(N); when the uniformly active assumption fails, the general bound with the measured ∥μˉ∥2\Vert \bar\mu\Vert^2 should be used. We will qualify the width statements of Section 5 accordingly.

Reviewer_pmPFOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper attempts to address three questions: Is there truly a better function within a larger model space? Can training algorithms find such a function? Can the gains in training be transferred to unseen data? The paper assumes a specific scenario: after training a residual network, inserting a new residual block—under what conditions can it be proven that the test loss decreases? The theory consists of three parts: 1. If there exists a descent direction, there exists a locally better model. 2. Through backtracking selection, the final model's training loss is guaranteed to be no worse than the baseline model. 3. Only when the gains exceed the generalization error can test improvement be ensured.

Critical strengths
  1. The issue is important, and the analysis object is clear. The conclusion demonstrates stronger capabilities in deep network expression than mere proof.
  2. The feasibility of parameterizing the descent direction was discussed.
  3. The experimental setup is practical, attempting to directly examine the theoretical components.
Critical weaknesses

The details for experimental reproducibility and some appendix arguments remain insufficient. The main text introduces the C4 experiment with a 32-layer LLaMA-style model, but Appendix N primarily provides CIFAR training settings.

Questions for authors

Does this paper truly derive the scaling law? It appears that the main result provides conditional qualitative improvement guarantees but does not directly derive a universal power law.

2 comments
AuthorsOpenReview ↗

Response to Reviewer pmPF (1/2): Full C4/LLaMA setup added (Appendix N.5) and Appendix M rewritten

We thank the reviewer for the positive assessment and the accurate summary. The three-step structure the reviewer identifies is exactly the logic of the paper: a descent direction yields a locally better jumpboard model; the fallback (backtracking) selection guarantees that the final model is no worse than the jumpboard on the training set; and test improvement is guaranteed only when the gains exceed the generalization error. We address the two points below. Changes in the revised PDF are marked in red.

W1. Reproducibility of the C4 experiment and appendix arguments.

Thank you for pointing this out. We added Appendix N.5, which gives the complete setup of the LLaMA-style experiments behind Tables 2 and 3:

  • Models. Six decoder-only LLaMA-style configurations with 32 blocks and widths N∈{512,1024,1536,2048,2560,3072}N\in\{512,1024,1536,2048,2560,3072\}: pre-RMSNorm, grouped-query attention with RoPE (head dimension 128, query:KV heads = 4:1), SwiGLU feed-forward networks with intermediate dimension 3.5N3.5N, and a final RMSNorm followed by a linear LM head. Non-embedding parameter counts range from 0.1091B to 3.9261B (Table 5).
  • Pretraining. From scratch with the Llama 3 tokenizer, context length 2048, global batch of 480 sequences and 10B tokens per run, on a fixed mixture (C4 35%, FineWeb-Edu 30%, SlimPajama 20%, OpenWebText2 10%, Wikipedia/StackExchange/arXiv 5%). Each width is trained with 8 seeds (48 models). AdamW with (β1,β2)=(0.9,0.95)(\beta_1,\beta_2)=(0.9,0.95) and weight decay 0.1; 2,000 warmup steps to 3×10−43\times10^{-4}, cosine decay to 1.5×10−71.5\times10^{-7}; gradient clipping 1.0; BF16; no dropout; no post-training.
  • Clarification. C4 is the evaluation corpus on which activation gradients are measured, not the pretraining corpus. We revised the wording in Section 6 accordingly.
  • Measurement protocol. Gradients are taken at the output of the last Transformer block, before the final RMSNorm and the LM head. The sequence-level signal is the derivative of the token-normalized sequence loss with respect to an additive shift shared by all positions (Eq. (13)). Table 2 draws from two fixed, disjoint pools with independent sampling with replacement, M=K∈{1,2,4,8}M=K\in\{1,2,4,8\}, and 500,000 trials per (width, seed, MM), so the conditional Monte Carlo standard error is at most 7.1×10−47.1\times10^{-4}. Table 3 reports the squared norm of the averaged gradient, computed within each model before aggregation across seeds.

Regarding the appendix arguments, we rewrote Appendix M with complete proofs under explicit assumptions. All statements are high-probability bounds with explicit constants, replacing the earlier heuristic derivation. If the reviewer has other arguments in mind, we would be glad to expand them.

AuthorsOpenReview ↗

Response to Reviewer pmPF (2/2): On the scaling law: qualitative mechanism and a conditional power-law envelope

Q1. Does the paper truly derive the scaling law?

The reviewer's reading is correct: the paper does not derive a universal power law, and this is not its aim. As the title indicates, we provide a qualitative mechanism behind scaling behavior. Residual depth expansion pays a statistical cost of at most 2(ϵM+ϵK)2(\epsilon_M+\epsilon_K); it does not worsen the test risk when the representation and optimization gains cover this cost, and it strictly improves it when they exceed it. This explains why depth, width and data must grow jointly, but it does not by itself determine the functional form of scaling.

A power law needs additional assumptions on how the remaining first-order signal decays with the excess risk, and such assumptions depend on the data distribution and the learned representation. The revised Appendix M gives one such route. A zero-output block is inserted and its output projection is updated by one gradient step on fresh samples. Assume in addition that the fraction of the remaining gradient captured by the new block's features decays with the excess risk at a rate governed by an exponent bb, with constants κ0\kappa_0 and ϱ\varrho. Then, with probability at least 1−δ−δψ1-\delta-\delta_\psi, the excess risk after kk inserted blocks satisfies (Theorem M.9) Δk≤(Δ0−b+b κ0216 ϱ2 k)−1/b\Delta_k\le\Big(\Delta_0^{-b}+\frac{b\,\kappa_0^2}{16\,\varrho^2}\,k\Big)^{-1/b} until it reaches a noise floor set by the per-step sample size MM: Δsat1+b≲ϱ2κ02(ιM+ι2M2)\Delta_{\rm sat}^{1+b}\lesssim\frac{\varrho^2}{\kappa_0^2}\big(\frac{\iota}{M}+\frac{\iota^2}{M^2}\big) with ι=log⁡(2n/δ)\iota=\log(2n/\delta) (Remark M.4).

Three caveats apply:

  1. The result is conditional on this additional assumption.
  2. It is an upper envelope, not a matching asymptotic law.
  3. The exponent 1/b1/b is determined by the data and the learned representation, not by the residual architecture.

The architecture supplies the mechanism:

  • a zero-output block keeps a nonzero first-order direction (Appendix C);
  • the new block's gradient depends on the functional gradient only through its projection onto the subspace capturable at the insertion layer (Lemma M.1);
  • near-identity residual maps are injective, so growth does not destroy information in the representation (Corollary M.11).

The exponent, by contrast, is data-dependent. This is consistent with the empirical observation that scaling exponents differ across tasks. Our experiments (near-diagonal gradient covariance, decay of the first-order signal with depth, growth with width) are consistent with such assumptions, but they do not constitute a test of the exponent.

We revised the text accordingly: Section 4 now states this scope explicitly, and the parameter-count exponent table of the previous Appendix M has been removed.

Reviewer_gvUpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the risk and test behaviors of normalized residual networks when the depth is increased by inserting a new residual block. Two main results are proved. The first one ensures that, under some assumptions, adding a new residual block improves the population risk strictly compared to the original model when the combined representation and optimization gains are more than the generalization gap. The second result focuses on the empirical test performance and it is shown that the finite-test-set margin and the optimization gain are more than the generalization gap. Several other insights on depth, width, and data are discussed in the paper. For example, depth provides new improving directions, width enhances the finite-sample observability, and adding more data can offset the statistical price of having a larger model. Experimental results show that some assumptions are valid including the activation-gradient covariance is near-diagonal. Also, it is shown that the growth-trained networks outperform the models trained from scratch and increasing the width can be a stabilizer.

Critical strengths

The paper is well-written in the sense that the assumptions are clearly explained and discussed (or even verified). The breakdown of representation, optimization, and generalization makes the work easier to follow. The findings on the depth, width, and data, and their interplay seems interesting.

Critical weaknesses
  1. The main theorems do not mention any specific training algorithms. I feel this is probably too much of abstraction. The results would be more convincing if some algorithms can be discussed and perhaps some comments about how to make a good algorithm for achieving the desired "representation/optimization > generalization" can be shared. The theorems are currently optimization-agnostic due to the dependence on the Condition 2.
  2. There seems to be a mismatch between the theorems and the models that were trained end-to-end. The theoretical setting relies on a frozen model with a newly inserted block and the training of the inserted block. However, in some parts of the experiments and also in reality, we train the models end-to-end. In this sense, the theorems seem to be not practical for real-world end-to-end training of a deeper residual network.
  3. The work focuses on the behavior of inserting a new residual block (qualitative). It would be more interesting to see some quantitative results or formulas to illustrate how much gains a model can have if a number of new blocks are added.
Questions for authors
  1. Are there specific (constructive) training algorithms to guarantee the optimization gains? If yes, what is the complexity compared to SGD?
  2. Is it possible to extend/generalize the single-block jumpboard analysis to multi-block or a number of residual blocks (where multiple blocks are trained while the original layers are still frozen)?
  3. Based on the above question, if possible, would that shed light on getting quantitative results? Is it possible to get a quantitative scaling law for adding a number of layers?
  4. A residual network is almost always trained in the end-to-end fashion for the best performance. The jumpboard model only trains the single-blockbeing added. Is there a better modeling approach for end-to-end training?
8 comments
AuthorsOpenReview ↗

Clarifications on Training Algorithms, Multi-Block Extensions, Scaling Laws, and End-to-End Training(1)

Thank you for recognizing our decomposition into representational gain, optimization gain, and generalization transfer, as well as our analysis of the relationship among depth, width, and data. Your questions help us clarify the respective roles of the comparison theorems in the main text, the jumpboard construction, and the quantitative extension in the appendix. We respond to each point below.

1. Training algorithms and complexity relative to SGD (W1, Q1)

Theorems 1 and 2 deliberately remain agnostic to the training algorithm: they study the conditions under which candidate models returned by different training procedures can be guaranteed to improve risk. Condition 2 on pp. 3–4 of the original manuscript explicitly states that the candidate model falgf_{\mathrm{alg}} may come from either growth training or scratch training. The final model is determined by the following empirical selection rule:

$f_{\mathrm{new}} \in\arg\min_{f\in{f_{\mathrm{alg}},\widetilde f}} L_{\mathrm{train}}(f).$

Accordingly, the main theorems do not require a particular optimizer, such as SGD, and do not rely on a global convergence guarantee for any optimizer. The effect of the training algorithm enters the comparison through the empirical optimization gain (ΔERM)(\Delta_{\mathrm{ERM}}) of the final model relative to the jumpboard. The central question of this paper is when representational and optimization gains are sufficient to offset the statistical cost. This question can be studied separately from the convergence rate of a particular optimizer.

The question of whether a constructive algorithm exists can be addressed at three levels:

  1. The fallback guarantee applies to any algorithm. The selection rule in Condition 2 ensures (ΔERM≥0)(\Delta_{\mathrm{ERM}}\ge0). The comparison model can be the old model itself: both the statement of Theorem 1 and the proof of Theorem 2 in Appendix H explicitly allow this degenerate case and yield Ltest(fnew)≤Ltest(fold∗)ΔERM+2(ϵM+ϵK).L_{\mathrm{test}}(f_{\mathrm{new}}) \le L_{\mathrm{test}}(f_{\mathrm{old}}^\ast) \Delta_{\mathrm{ERM}}+2(\epsilon_M+\epsilon_K). Constructing this comparator via a zero-output block requires no additional optimization. The selection step only requires comparing the empirical losses of the candidate and old models on the same training set, and existing evaluation results can be reused.

  2. A jumpboard that improves the training loss can be constructed. For the zero-output residual parameterization in Corollary 1, if the empirical gradient with respect to the new block’s output projection is nonzero at the origin, a sufficiently small step in its negative direction, for example with a step size selected by backtracking, yields a jumpboard with strictly lower training loss. Improvement in test loss or population risk is then guaranteed under the corresponding risk-comparison conditions. Section 6.4 also provides another experimental construction.

  3. A strictly positive (ΔERM)(\Delta_{\mathrm{ERM}}) depends on the particular optimizer; we do not assert that it must occur.

As a constructive supplement, the rewritten Appendix M provides an algorithm for growing a zero-output residual block: it fixes the residual features, estimates the actual parameter gradient of the new block’s output projection, and updates the projection in the negative gradient direction. Lemma M.6 and Theorem M.10 provide a conservative step size that can be constructed from samples when the curvature and gradient envelope bounds are known. Under the stated conditions, the algorithm guarantees with high probability that population risk does not increase, and guarantees a strict decrease above the specified threshold. This guarantee holds even when the additional optimization gain is ΔERM=0\Delta_{\mathrm{ERM}}=0.

Each growth step requires a batch gradient estimate and the accumulation of step-size statistics. The second-order U-statistic used for signal estimation can be computed from the squared norm of the sum of gradients and the sum of their individual squared norms, without enumerating all sample pairs. If the output projection has size (N×m)(N\times m) and signal estimation uses (M′)(M') samples, the matrix accumulation costs (O(M′Nm))(O(M'Nm)) once the per-sample gradients and features are available. This describes the cost of a single growth step; we do not claim that its overall convergence complexity is uniformly better or worse than that of full SGD training.

For algorithm design, the theorem provides three principles: use the selection rule as a fallback; start growth training from the jumpboard; and control the generalization term through norm constraints (Lemma 1). Table 1 is consistent with these principles: at all three growth steps, growth training achieves lower test loss than training the same architecture from scratch (0.2626 versus 0.2874, 0.2580 versus 0.2822, and 0.2510 versus 0.2948).

AuthorsOpenReview ↗

Clarifications on Training Algorithms, Multi-Block Extensions, Scaling Laws, and End-to-End Training(2)

  1. Extension to multiple residual blocks and joint training (Q2)

The main text uses the insertion of a single block to present the basic improvement mechanism. The same comparison framework extends directly to multiple residual blocks.

Adding multiple blocks simultaneously. Consider a fixed model class (Hnew(r))(\mathcal H_{\mathrm{new}}^{(r)}) obtained by adding (r) blocks and satisfying the corresponding norm and regularity conditions. Since the added blocks permit zero output, setting the other (r-1) blocks to zero preserves any single-block insertion path within this class. If at least one of the selected insertion locations satisfies Condition 1, the original improving jumpboard belongs to this class.

The candidate model can then be obtained by jointly training all newly added blocks or by updating all old and new layers end to end. As long as Condition 2 holds, the comparison proofs of Theorems 1 and 2 remain unchanged, with the generalization terms replaced by the uniform bounds for the full class containing all added blocks. For example, let (ϵM(r))(\epsilon_M^{(r)}) denote the corresponding training-side generalization bound. Then, with probability at least (1−δ)(1-\delta),

$R(f_{\mathrm{new}}) \le R(f_{\mathrm{old}}^\ast) -\Delta_R^S-\Delta_{\mathrm{ERM}} +2\epsilon_M^{(r)}.$

Here, the representational gain (ΔRS)(\Delta_R^S) is supplied by a constructible jumpboard in the class containing all added blocks, while the additional empirical gain from joint training enters through (ΔERM)(\Delta_{\mathrm{ERM}}).

Sequential growth. Consider (f0→f1→⋯→fn)(f_0\to f_1\to\cdots\to f_n); Table 1 reports three such sequential growth steps. If the insertion locations and number of steps are fixed in advance, all intermediate models belong to the final class (H(n))(\mathcal H^{(n)}), with later blocks set to zero. On the uniform generalization event for (H(n))(\mathcal H^{(n)}), which has probability at least (1−2δ)(1-2\delta), summing the training-loss relations implied by the selection rule across steps gives

$L_{\mathrm{test}}(f_n) \le L_{\mathrm{test}}(f_0) -\sum_{k<n}\bigl( \Delta_{\mathrm{train},k}+\Delta_{\mathrm{ERM},k} \bigr) +2(\epsilon_M+\epsilon_K).$

Here, (Δtrain,k≥0)(\Delta_{\mathrm{train},k}\ge0) is the training-loss gain of the jumpboard relative to (fk)(f_k) at step (k), (ΔERM,k≥0)(\Delta_{\mathrm{ERM},k}\ge0) is the additional empirical gain at that step, and (ϵM,ϵK)(\epsilon_M,\epsilon_K) are evaluated for the final class. Thus, the gains accumulate across steps, while the generalization cost is incurred only once. This cost increases with the number of added blocks (Lemma 1), which is precisely what we mean by “data must keep pace with depth.” This conclusion places no restriction on the training method used at each step.

Both joint training of multiple blocks and sequential growth can therefore be accommodated within the same comparison framework. Appendix M further analyzes the cumulative decrease and rate for a specific sequential growth algorithm. We will add this explanation of the extension to multiple blocks to the revised manuscript and distinguish it clearly from the results for the specific algorithm in the appendix.

AuthorsOpenReview ↗

Clarifications on Training Algorithms, Multi-Block Extensions, Scaling Laws, and End-to-End Training(3)

  1. Qualitative contributions and quantitative scaling laws (W3, Q3)

The primary goal of this paper is to establish a conditional qualitative mechanism: when residual expansion can be guaranteed not to increase risk, and when it can be guaranteed to reduce risk strictly. Specifically, expansion incurs a statistical cost of at most (2(ϵM+ϵK))(2(\epsilon_M+\epsilon_K)); it is non-degrading when the gains cover this cost and strictly beneficial when they exceed it. Pages 1 and 5 of the original manuscript explicitly state this scope and explain that Appendix M discusses a power-law extension under additional model-dependent conditions. The main text does not claim that residual architecture alone implies a universal power law relating model size to loss, and Theorems 1, 2, and 4 do not depend on Appendix M.

We agree that quantitative results for repeated expansion can deepen our understanding of this mechanism. We have therefore rewritten Appendix M, replacing the previous quantitative derivation with a finite-sample descent analysis of a concrete growth algorithm and an upper envelope for risk over multiple steps. For example, Theorem M.10, which uses an implementable step size, gives the following inequality under its assumptions and on the corresponding high-probability event whenever the risk gap is above the specified threshold:

$\Delta_{k+1} \le \Delta_k-\frac{c_{\mathrm{lo}}}{64}\Delta_k^{1+b}.$

Consequently, up to and including the first threshold crossing specified in the theorem,

$\Delta_k \le \left( \Delta_0^{-b}+\frac{b,c_{\mathrm{lo}}}{64}k \right)^{-1/b}.$

Here, (Δk)(\Delta_k) is the risk gap relative to the reference predictor specified in the appendix, and the exponent and constants are determined by conditions such as quantitative capture. This is a power-law upper envelope under additional conditions. The residual structure provides an identity embedding of the original model and new perturbation directions. How much remaining signal these directions capture, and how that signal decays with the risk gap, also depend on the data distribution, the current representation, and the mathematical properties of the loss. These quantitative conditions cannot be derived from residual structure alone.

Our experiments are compatible with such assumptions: Figure 2 shows that the first-order signal decreases with depth, while Table 3 shows that it increases with width. These results provide empirical support for the quantitative extension, but we do not claim that they directly test the exponent. The qualitative mechanism in the main text and the conditional rate in the appendix should therefore be distinguished.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.