- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether inserting a residual block into a trained normalized residual network can improve test performance. Its framework combines a locally improving “jumpboard” model, an empirical selection rule, and uniform generalization bounds. It additionally analyzes train–test activation-gradient alignment to motivate a beneficial role for width, and presents experiments on residual networks and LLaMA-style models.
Critical strengths
-
The paper addresses a meaningful distinction between enlarging a hypothesis class and improving the test performance of a selected model.
-
The basic risk-comparison inequalities are valid given the stated selection rule and an appropriate fixed-class uniform generalization bound. The fallback construction usefully avoids assuming global optimization.
-
The discussion of zero-output initialization distinguishes zeroing the final projection from zeroing the entire residual branch. The treatment of stabilized normalization also clarifies how normalization singularities can be avoided.
Critical weaknesses
-
The paper gives limited insight into why adding depth improves test risk. Theorems 1–2 combine uniform generalization bounds with an empirical comparison rule that applies to general model classes. Condition 1 already assumes that the inserted block has a first-order descent direction. Once this is assumed, the existence of a better jumpboard model follows from differentiability. What remains unexplained is why trained residual networks should have such directions and when the resulting gains are large enough to exceed the generalization penalty. These are central questions for the claimed depth-scaling mechanism.
-
The link between width and test-risk improvement remains unclear for general residual blocks. Condition 1 concerns a descent direction in the inserted block’s parameters. Theorem 4 instead studies , the alignment of average activation gradients. Even when these gradients align, the selected parameter direction need not reduce test loss. Corollary 3 connects the two under an additional realizability assumption, but this assumption is not established for the general residual blocks discussed. Without that connection, better gradient alignment does not show that width makes residual updates more likely to improve test performance.
-
The sufficient depth–width–data condition in Appendix M.5 is not established. The argument uses and a signal proportional to to upper-bound the misalignment probability. However, the probability bound contains the reciprocal of the signal. Controlling this reciprocal from above requires a lower bound on the signal, which the upper bound on does not provide. The mixed covariance term also requires control in the weak-signal regime; controlling the leading terms alone is insufficient. These gaps leave the stated sufficient reliability condition unsupported by the given assumptions.
Questions for authors
-
What architecture-specific insight establishes useful depth expansion beyond the local descent assumption in Condition 1 and the generalization argument in Theorems 1–2? Is there a concrete setting analyzed in the paper where the jumpboard improvement can be shown to exceed the generalization penalty?
-
For which residual-block parameterizations does the additional realizability assumption in Corollary 3 hold? Can you establish this connection for the blocks covered by your main claims, so that implies test-loss descent along the selected residual update?
-
In Appendix M.5, what lower bound on justifies the claimed upper bound on misalignment probability? Since provides only an upper bound on the signal, how should the sufficient depth–width–data condition be derived, including control of the mixed covariance term?
8 comments
Response to Reviewer 6Hnw (1/5): W1/Q1 (2)
W1 / Q1 (part 2 of 2)
4. Why a trained network can still satisfy Condition 1.
- Stationarity of the old model in its own parameters removes only the directions those parameters can generate. It does not make the activation-gradient signal orthogonal to the tangent directions of the new block (Appendices C.7 and P.8, pp. 23 and 71). In the decomposition of Appendix C, Condition 1 requires (i) a descent direction in hidden space and (ii) that the block realizes one such direction to first order. For (i) to fail one needs for every perturbation of the representation, which is far stronger than parameter stationarity.
- For a given zero-output projection block, Condition 1 fails if and only if , i.e., for every coordinate of the signal and every residual feature. If this holds at every admissible layer, the model is in the deepest-model regime of Section 5, where we claim no first-order strict improvement.
- We do not claim that the residual architecture alone guarantees non-orthogonality. What the paper provides is the exact characterization of the condition, its compatibility with stationarity of the old model, and measurable evidence. In Figure 2 and Table 3 the norm of the mean activation gradient is clearly nonzero (it decays with depth in Figure 2). In Table 1, the jumpboard built from the activation gradient and realized by an actual residual block has lower training loss and lower test loss than the corresponding old model at all three growth steps.
5. When the gain exceeds the generalization penalty.
Criterion. Theorems 1 and 2 give and .
A concrete setting (population route, Theorem 7, p. 40). Fix the reference model and an insertion layer satisfying the population form of Condition 1 ( for zero-output projection blocks). The margin of the population jumpboard is determined by the data distribution and the reference model only; it does not depend on the training sample size . Meanwhile (Lemma 1, Corollary 2). Theorem 7 gives, with probability at least ,
By Appendix F.1 (pp. 33–34), ensures , hence even when . The hypothesis is Condition 2: the final model is no worse on the training set than the comparison jumpboard.
Limits of this statement. Under the assumptions of the main text, is only known to be positive; its size is not quantified. Appendix J (p. 41) already describes it as a directional certificate and notes that in practice the dominant improvement is often carried by . The main text therefore answers which inequality decides and what order of sample size is needed; it does not claim a closed-form lower bound on .
Role of the experiments. Table 1 is an empirical observation of the mechanism, not a numerical verification that the gain exceeds the theoretical bound. Norm-based bounds are numerically loose, and we make no such claim.
6. A further quantitative supplement (new Appendix M; not needed for the main results). Assume in addition a curvature bound along the update ray (M2), boundedness at the current prediction (M3), and fresh direction samples (M5). Consider the update built from the empirical direction (the of Appendix C). With the step-size factor , Theorem M.5 (p. 51) gives, with probability at least , whenever ,
where and are explicit noise terms; Lemma M.6 gives a guarantee of the same type for a fully data-driven step size. For softmax cross-entropy with a fixed full-row-rank linear head, Corollary M.11 (pp. 57–58) verifies (M1)–(M3) and provides all constants. This is a setting in which the gain has an explicit lower bound that already accounts for the direction-estimation error. When the hypotheses of both results hold, inserting it into Theorem 1 gives the sufficient condition for strict improvement of the final model.
Response to Reviewer 6Hnw (3/5): W2/Q2
W2 / Q2
1. Which parameterizations satisfy the condition of Corollary 3. It holds for residual blocks with a trainable output bias, , initialized at , . Along the path , , one has for every input, hence
For the bias parameter, the parameter gradient is the mean activation gradient: and . In this direction the training-set form of Condition 1 reads , the selected parameter direction is , and the quantity controlled by Theorem 4 is exactly the first-order change of the test loss along it. For such blocks, implies strict test-loss descent for sufficiently small steps, without any additional realizability assumption (Corollary 3, p. 39).
2. Correspondence with the experiments. Appendix N.5 (pp. 59–61) defines the sequence-level signal as the derivative of the loss with respect to an additive residual shared by all token positions,
This is the gradient of the loss with respect to the output bias of a zero-initialized residual block inserted at the measurement position. Table 2 therefore estimates the probability that updating this bias along decreases the test loss to first order. Two caveats: Table 2 measures the sign of the first-order alignment, not the finite-step loss change after fully training the new block; and the correspondence concerns the inserted block, so we do not claim that existing LLaMA sublayers carry output biases.
3. General output-projection blocks. For with features fixed at initialization (as in Corollary 1), the training-selected direction is with , and
So the reviewer is right: for general blocks, does not by itself give test descent along this direction; the relevant quantity is . The proof of Theorem 4, however, uses only that and are two independent sample means of one random vector with nonzero mean and bounded largest covariance eigenvalue. Under the same sampling convention (with the feature initialization independent of the samples), applying it verbatim to , with mean and covariance , gives
Here is exactly Condition 1. Thus, for the zero-output projection blocks covered by our main claims, the link between the parameter direction selected under Condition 1 and first-order test descent is given by the same theorem, with replaced by . The bias case is , where the bound reduces to Theorem 4. We will add this as a remark in Appendices E and H, and state in the main text that Corollary 3 as originally written concerns the bias-type direction.
4. Scope of the width statement. The form is derived for the bias-type direction, under Assumption 3 and the uniformly active regime (, ); Appendix J (p. 42) already calls it the bias-type special case. For general projection blocks the same role is played by the ratio of to . Whether this ratio grows with width depends on the features, and we make no such claim for general blocks. Empirically, the alignment probability in Table 2 increases with width and sample size, and the signal in Table 3 increases monotonically with width (from 0.15 to 0.26), though more slowly than linearly. We present these as evidence consistent with the mechanism, not as a verification of ; when the uniformly active assumption fails, the general bound with the measured should be used. We will qualify the width statements of Section 5 accordingly.
Response to Reviewer 6Hnw (1/5): W1/Q1 (1)
We thank the reviewer for the careful reading, and in particular for the precise analysis of Appendix M.5. Our response is split into five comments. Page numbers refer to the revised PDF.
W1 / Q1 (part 1 of 2)
1. Scope. The main text proves conditional, qualitative statements. Page 3 states that Condition 1 is not meant to hold for every trained model or insertion layer, and that we make no strict-improvement claim when it fails. We agree that the comparison inequalities of Theorems 1 and 2 hold for general model classes; this is by design. Once the architecture-independent comparison step is isolated, the architecture-specific content sits in three places: the exact embedding and tangent directions of zero-output residual blocks (Appendices C and P); the norm-based generalization bound for normalized residual networks (Lemma 1); and train–test alignment on the residual stream (Theorem 4, Corollary 3).
2. Status of Condition 1. For a fixed insertion layer, Condition 1 is necessary and sufficient for the inserted block to admit a strict first-order descent direction, and it is equivalent to a nonzero risk gradient with respect to the new block's parameters (Proposition 1, Appendix P.4, pp. 66–67). For zero-output projection blocks it is equivalent to Cpop=E[qψ⊤]=0 (Corollary 6, p. 70, where this matrix is denoted Mpop). A recent preprint [X] gives a corresponding characterization under a fixed function-preserving first-order growth protocol: at each candidate location, Condition 1 is equivalent to a nonvanishing parameter gradient of the inserted block, and to the activation-gradient signal not being orthogonal to the block's tangent space. The equivalence itself is elementary; its role is to fix the status of the condition. Any strict improvement obtained through a first-order direction of the new block requires Condition 1, so it delimits the scope of the first-order mechanism, and the hypothesis of our conditional statements cannot be weakened within that mechanism. "First-order" is an essential qualifier: higher-order or non-local improvements lie outside the equivalence (Appendix P.10).
3. What the residual structure contributes. Take hU,V(z)=VψU(z) and zero only the output projection (V=0).