- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
For an un-normalized Gaussian residual network with a terminal ReLU, the paper develops a theory that inverts the residual scale from a prescribed fraction of terminal features that are zero at initialization (the inactive fraction). For a single input, the state energy follows the ODE (Prop 3.1), and as the state width and depth grow at arbitrary relative rates with the branch width finite (even ), the inactive fraction converges in to (Thm 3.2). For positive inputs there is a unique for each (Thm 3.3), and the dependence on input scale is invariant for ReLU / leaky ReLU, decreasing for GELU, and increasing for SELU (Thm 3.4). For several inputs, the paper studies features that are zero on every input (zero columns of the readout): the joint empirical law of coordinate rows converges to , where the covariance of is given by a Gaussian-kernel flow (Thm 4.1); for ReLU / leaky ReLU a unique inverse exists for positive or centered Gaussian inputs (Thm 4.2); there are positive input laws with identical first- and second-order marginals but different zero-column fractions and expected ranks (Prop 4.3, Cor E.5); and a transfer rule holds for leaky ReLU (Cor 4.4). Experiments confirm calibration errors below 0.2 points, the opposite corrections for GELU / SELU, the joint predictions, and the fitting difference in the parity example, and report that short classification runs show no consistent accuracy difference.
Critical strengths
-
The inverse problem of choosing the initialization scale from a target inactivity rather than a target variance is solved rigorously in a joint limit with arbitrary relative rates of width and depth and finite branch width. The paper uses existing covariance-propagation tools while making clear why a non-Gaussian input coordinate law must be retained. The theory is solid.
-
The construction of Prop 4.3 / Cor E.5. Input laws that pairwise statistics (the mean kernel) cannot distinguish change the expected rank and fitting error of a readout with a finite feature budget, shown both by proof and by experiment. Beyond initialization, this is a general caution for describing finite feature collections through kernels.
Critical weaknesses
-
The practical significance is limited. The object is a special residual structure, un-normalized with a single terminal ReLU, and the implications for modern residual networks (with normalization and no terminal ReLU) are not direct. The short classification experiments show no consistent accuracy difference (App. I), and the authors acknowledge that the results concern properties at initialization.
-
Most of the main text calibrates the single-input marginal inactivity, whereas the quantity relevant to readout capacity is the joint silent fraction; the gap between the two (Figure 1) is the very motivation of the paper. The joint inverse problem (Thm 4.2) is solved only for ReLU / leaky ReLU with positive or Gaussian inputs, not for GELU / SELU or general signed inputs.
Questions for authors
-
In Thm 4.1 with , the increment covariance between inputs depends at each layer on a single scalar product and converges to the Gaussian kernel only through averaging over depth. How does the convergence rate depend on (is there a term analogous to of Prop 3.1 on the joint side)? I would like to confirm that in the experiments is justified by this rate.
-
Is it correct that for centered inputs the marginal inverse problem degenerates (the fraction is for every )? If so, does the difference between He initialization and the target of this paper reduce to the fact that marginal inactivity becomes controllable only in a residual structure whose identity path carries positive inputs, or do you claim an essentially new prescription on the joint side even for centered inputs?
6 comments
Response to Reviewer GA9u (2/3)
W2. Most of the main text calibrates the marginal inactivity for a single input, whereas the quantity relevant to readout capacity is the joint silent fraction. The joint inverse (Theorem 4.2) is solved only for ReLU and leaky ReLU with positive or Gaussian inputs.
A. Thank you for these remarks. We agree that the joint fraction is the quantity that matters for the readout. The revised Introduction and the new Section 6.3 now treat it directly, a new experiment in Appendix K.5 separates the two fractions, and Appendices E.6 and E.7 extend the joint inverse on the input side and delimit it on the activation side.
On the balance between marginal and joint calibration, the main text now gives the joint fraction more room. The Introduction defines both fractions and the joint empirical law in words, and Section 6.3 deals only with joint inactivity on real input sets. The offset experiment of Appendix K.5 moves the joint fraction while the marginal fraction stays fixed (Mackey-Glass, , , errors relative to ).
| Offset | 0 | 1 | 2 | 4 | 6 |
|---|---|---|---|---|---|
| Inactive fraction for one input (%) | 49.6 | 49.6 | 49.9 | 50.1 | 50.1 |
| Joint silent fraction (%) | 0.4 | 8.6 | 20.6 | 33.0 | 38.4 |
| Test error, original columns | 1.00 | 0.81 | 0.82 | 0.99 | 1.14 |
| Test error, silent columns reflected | 1.00 | 0.72 | 0.60 | 0.55 | 0.56 |
The marginal fraction stays at one half while the joint fraction grows from 0.4% to 38%, and the gap between the last two rows grows with it. With a silent fraction , the original columns perform like active columns, a relation that holds within 10% in each of the 120 settings.
On the activations for which the joint inverse is solved, Proposition E.4 now shows which activations keep correlations from decreasing. With we prove , with equality exactly for functions that are linear on each half line, and the normalized kernel satisfies on exactly in the equality case. This is the property on which the monotone inverse rests. ReLU and leaky ReLU have it and move the correlations strictly, whereas a linear branch leaves them fixed and gives no strict inverse. GELU and SELU have , and we give correlated inputs for which their joint inactivity decreases near , which rules out an inverse that increases for all inputs.
On the inputs, the joint inverse now goes beyond positive and Gaussian laws. In Appendix E.6 we extend it to signed laws that are not Gaussian, the scale mixtures with Gaussian and an independent positive gain shared by a row. Theorem 4.2 also no longer assumes a positive definite . Nonzero inputs that are pairwise nonproportional suffice, so training sets with are covered. For general signed inputs some assumption is necessary. In Appendix E.12 we give a signed law with a positive definite second moment matrix for which first decreases.
Q1. How does the convergence rate depend on ? Is there a term analogous to on the joint side? Is in the experiments justified by this rate?
A. Thank you for this question. Yes. The new Proposition E.1 gives the same term on the joint side, and at this term is smaller than the width term in our experiments. It bounds the state Gram matrix against the covariance ODE started from the same finite input Gram matrix.
The constant depends on , the Lipschitz constant of the activation, bounds on the input energies and a bound on the coefficient. The input Gram matrix may be singular.
The dependence on comes from the fluctuation of the branch. Conditionally on the past, the branch contributes with , whose mean is . At this is a random matrix of rank one. It enters the Gram increment with weight , so its fluctuation has conditional variance of order per layer. The fluctuations of different layers are martingale differences, and their sum over layers has variance of order .
This rate justifies . For and the branch term is smaller than the width term in the forecasting experiments () and in Figure 13 (), up to constants. Figure 13 confirms the calibration directly and includes . Over its 36 conditions with and , the largest deviation of a mean silent fraction from its target is 0.019, at and , which is the size of the standard error of that mean.
Response to Reviewer GA9u (3/3)
Q2. Is it correct that for centered inputs the marginal inverse degenerates (the fraction is 1/2 for every )? If so, does the difference from He initialization reduce to the fact that marginal inactivity is controllable only when the identity path carries positive inputs, or do you claim an essentially new prescription on the joint side even for centered inputs?
A. Thank you for this precise question. Yes, the marginal inverse degenerates for centered inputs, and we do claim a new prescription on the joint side. Appendix E.12 now states both points.
On the first part, for centered Gaussian input rows, and more generally for symmetric input laws, the marginal fraction equals for every , since for symmetric . Symmetry is the right condition. A law with mass at and at has mean zero, and its inactive fraction tends to as .
The difference from He initialization is more than the positivity of the identity path. The marginal inverse works because the identity path carries positive inputs, which supply nonzero thresholds, but the resulting coefficient uses more than the input variance. For the ReLU target it is about 1.52 for constant positive inputs and 1.18 for half normal inputs with the same second moment, whereas a rule based on the input variance alone assigns both the same scale.
On the joint side the prescription is new also for centered inputs. By Theorem 4.2(ii), increases strictly from to while every marginal stays at , so each intermediate target fixes a unique coefficient. For two inputs with terminal correlation ,
In this Gaussian case, prescribing joint inactivity is equivalent to prescribing a terminal correlation (Corollary E.3), and the inverse expresses that choice as an expected number of unused features. For laws that are not Gaussian the equivalence fails, which is the content of the parity construction. There, equal pairwise statistics come with different joint inactivity and different expected rank.
The comparison with leaky ReLU makes the difference from variance matching explicit. Equal accumulated variance requires , whereas an equal normalized joint feature law requires . The offset experiment in our answer to W2 shows the same point empirically. The marginal fraction is fixed at one half while the joint fraction, and with it the readout error, changes.
Thank you again for your time and thoughtful review. We hope that our responses and the revised manuscript address your concerns, and we would be glad to discuss any remaining questions.
Response to Reviewer GA9u (1/3)
Thank you for taking the time to review our paper and for your detailed and precise feedback. We have carefully addressed each of your points, and all changes are marked in red in the revised manuscript.
A. Thank you for raising this point. We added Section 6.3 and Appendix K, with experiments on real inputs which show that the joint law is useful in practice, for fixed features and for trained weights.
On the practical significance, the joint law now predicts measured quantities on real input sets. We use two standard chaotic forecasting tasks, the Mackey-Glass series and the Lorenz system, in which a ridge readout on M terminal features of a frozen ReLU residual network predicts a future value from eight lagged values. The joint law gives the silent fraction from the Gram matrix of the training inputs alone.
These silent columns cost accuracy. With raw inputs, which are not standardized, 33% and 21% of the columns are silent, and selecting 256 active columns lowers the test error by 47% and 9% (Table 24). The prediction can also be inverted for a design choice (Tables 25 and 26). For the coefficient, the inverse gives the largest value at which M columns still contain, in expectation, the N−1 active columns needed to fit N points. With 256 points and four budgets between 272 and 320 columns, the predicted values on Mackey-Glass run from 2.38 to 4.99 and the measured ones from 2.34 to 5.02. For an input offset, a constant coordinate appended to the standardized inputs, the inverse gives its size for a target silent fraction without any search. Each entry lists Mackey-Glass first and Lorenz second, and reflection negates the preactivation of a silent column so that it becomes active.
The results are no longer confined to initialization. In Appendix K.7 we prove that a silent column takes no part in training until it reopens, and in Appendix K.9 we measure what this costs. In a network with n=256 whose residual weights are all trained, the readout reads sixteen columns. Arm B reads sixteen active columns and arm A reads eight active and eight silent ones, over 24 networks. All entries are test accuracies in %, and the data set is MNIST unless stated (Table 31).
The cost of the silent columns survives training under gradient descent, SGD and Adam with the smaller step, and it disappears only under Adam with the larger step. A network that starts with all features silent cannot learn at all. On raw Mackey-Glass inputs at a=1 this happens in 12% of the networks of width 2. The product Dn for a silent fraction D and width n, which is exact at a=0 and an approximation at a=1, gives 11.8% (Appendix K.8).
The absence of an accuracy difference in Appendix I is consistent with these results. Its runs prescribe the marginal fraction of single inputs and use Adam at steps selected by validation, so they do not control the number of silent columns. The joint quantity is visible there as well. In each of its runs that ends at chance accuracy, no terminal feature is active on the calibration inputs.
On the architecture, the statement on training is not tied to our residual structure. Its proof uses only that a linear readout follows a ReLU layer, so it holds for any earlier layers. The limit theorems remain specific to this structure. Appendix E.12 now states which steps use the terminal ReLU and the absence of normalization, and Appendix J treats a branch with RMS normalization.