acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Inactivity in Residual Networks

Abstract

Controlling activation variance does not determine how many features are zero on every input. Such features contribute no signal to a linear readout on that input set. We develop an inverse theory that chooses a residual initialization scale from a prescribed fraction of inactive features. For Gaussian branches followed by a terminal ReLU, we prove convergence of the joint empirical feature law on any fixed input set as state width and depth grow at arbitrary relative rates, allowing branch width to remain finite. The law yields unique inverses for marginal inactivity on positive inputs and, for ReLU or leaky ReLU branches, joint inactivity on positive or centered Gaussian input laws. The marginal inverse explains why input rescaling requires opposite corrections for GELU and SELU, while ReLU and leaky ReLU are invariant. We also construct positive non-Gaussian input laws with identical pairwise terminal feature distributions but different unused-feature fractions and expected readout ranks at every fixed budget of sampled limiting features. Experiments verify calibration and the predicted fitting differences. The results identify the input and activation information needed to control initial feature usage beyond variance.

Primary area:
learning theory
Keywords:
Weight Initialization, Signal Propagation, Deep Learning Theory, Residual Networks

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GA9uOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

For an un-normalized Gaussian residual network xk+1=xk+aLBkϕ(Akxk)x_{k+1} = x_k + \frac{a}{\sqrt{L}} B_k \phi(A_k x_k) with a terminal ReLU, the paper develops a theory that inverts the residual scale aa from a prescribed fraction pp of terminal features that are zero at initialization (the inactive fraction). For a single input, the state energy follows the ODE qˉ′(t)=a2vϕ(qˉ(t))\bar q'(t) = a^2 v_\phi(\bar q(t)) (Prop 3.1), and as the state width nn and depth LL grow at arbitrary relative rates with the branch width mm finite (even m=1m=1), the inactive fraction converges in L2L^2 to ∫Φ(−u/w) dν(u)\int \Phi(-u/\sqrt{w})\,d\nu(u) (Thm 3.2). For positive inputs there is a unique apa_p for each p∈(0,1/2)p \in (0, 1/2) (Thm 3.3), and the dependence on input scale is invariant for ReLU / leaky ReLU, decreasing for GELU, and increasing for SELU (Thm 3.4). For several inputs, the paper studies features that are zero on every input (zero columns of the readout): the joint empirical law of coordinate rows converges to U+GaU + G_a, where the covariance of GaG_a is given by a Gaussian-kernel flow WaW_a (Thm 4.1); for ReLU / leaky ReLU a unique inverse exists for positive or centered Gaussian inputs (Thm 4.2); there are positive input laws with identical first- and second-order marginals but different zero-column fractions and expected ranks (Prop 4.3, Cor E.5); and a transfer rule aβ=a0/(1−β)a_\beta = a_0/(1-\beta) holds for leaky ReLU (Cor 4.4). Experiments confirm calibration errors below 0.2 points, the opposite corrections for GELU / SELU, the joint predictions, and the fitting difference in the parity example, and report that short classification runs show no consistent accuracy difference.

Critical strengths
  1. The inverse problem of choosing the initialization scale from a target inactivity rather than a target variance is solved rigorously in a joint limit with arbitrary relative rates of width and depth and finite branch width. The paper uses existing covariance-propagation tools while making clear why a non-Gaussian input coordinate law must be retained. The theory is solid.

  2. The construction of Prop 4.3 / Cor E.5. Input laws that pairwise statistics (the mean kernel) cannot distinguish change the expected rank and fitting error of a readout with a finite feature budget, shown both by proof and by experiment. Beyond initialization, this is a general caution for describing finite feature collections through kernels.

Critical weaknesses
  1. The practical significance is limited. The object is a special residual structure, un-normalized with a single terminal ReLU, and the implications for modern residual networks (with normalization and no terminal ReLU) are not direct. The short classification experiments show no consistent accuracy difference (App. I), and the authors acknowledge that the results concern properties at initialization.

  2. Most of the main text calibrates the single-input marginal inactivity, whereas the quantity relevant to readout capacity is the joint silent fraction; the gap between the two (Figure 1) is the very motivation of the paper. The joint inverse problem (Thm 4.2) is solved only for ReLU / leaky ReLU with positive or Gaussian inputs, not for GELU / SELU or general signed inputs.

Questions for authors
  1. In Thm 4.1 with m=1m=1, the increment covariance between inputs depends at each layer on a single scalar product ϕ(Akx(s)) ϕ(Akx(t))\phi(A_k x^{(s)})\,\phi(A_k x^{(t)}) and converges to the Gaussian kernel only through averaging over depth. How does the convergence rate depend on mm (is there a term analogous to 1/(mL)1/(mL) of Prop 3.1 on the joint side)? I would like to confirm that m=32m=32 in the experiments is justified by this rate.

  2. Is it correct that for centered inputs the marginal inverse problem degenerates (the fraction is 1/21/2 for every aa)? If so, does the difference between He initialization and the target of this paper reduce to the fact that marginal inactivity becomes controllable only in a residual structure whose identity path carries positive inputs, or do you claim an essentially new prescription on the joint side even for centered inputs?

6 comments
AuthorsOpenReview ↗

Response to Reviewer GA9u (1/3)

Thank you for taking the time to review our paper and for your detailed and precise feedback. We have carefully addressed each of your points, and all changes are marked in red in the revised manuscript.

W1. The practical significance is limited. The object is a special residual structure without normalization and with a single terminal ReLU, and the implications for modern residual networks are not direct. The short classification experiments show no consistent accuracy difference (Appendix I), and the results concern properties at initialization.

A. Thank you for raising this point. We added Section 6.3 and Appendix K, with experiments on real inputs which show that the joint law is useful in practice, for fixed features and for trained weights.

On the practical significance, the joint law now predicts measured quantities on real input sets. We use two standard chaotic forecasting tasks, the Mackey-Glass series and the Lorenz system, in which a ridge readout on MM terminal features of a frozen ReLU residual network predicts a future value from eight lagged values. The joint law gives the silent fraction from the Gram matrix of the training inputs alone.

Input setNumber of inputsSettingsMean error of the predicted silent fraction
Subset of the training inputs64400.63 points
All training inputs2,000800.46 points

These silent columns cost accuracy. With raw inputs, which are not standardized, 33% and 21% of the columns are silent, and selecting 256 active columns lowers the test error by 47% and 9% (Table 24). The prediction can also be inverted for a design choice (Tables 25 and 26). For the coefficient, the inverse gives the largest value at which MM columns still contain, in expectation, the N−1N-1 active columns needed to fit NN points. With 256 points and four budgets between 272 and 320 columns, the predicted values on Mackey-Glass run from 2.38 to 4.99 and the measured ones from 2.34 to 5.02. For an input offset, a constant coordinate appended to the standardized inputs, the inverse gives its size for a target silent fraction without any search. Each entry lists Mackey-Glass first and Lorenz second, and reflection negates the preactivation of a silent column so that it becomes active.

Target silent fractionRealizedError relative to zero offsetError relative to a validation search over aa
10%, original columns10.0% / 8.7%0.79 / 0.960.86 / 0.98
25%, silent columns reflected24.3% / 24.7%0.55 / 0.920.60 / 0.94

The results are no longer confined to initialization. In Appendix K.7 we prove that a silent column takes no part in training until it reopens, and in Appendix K.9 we measure what this costs. In a network with n=256n=256 whose residual weights are all trained, the readout reads sixteen columns. Arm B reads sixteen active columns and arm A reads eight active and eight silent ones, over 24 networks. All entries are test accuracies in %, and the data set is MNIST unless stated (Table 31).

TrainingUpdatesBACost, B minus A [95% interval]
Fixed features32.025.07.0 [6.0, 7.9]
Gradient descent10061.352.29.1 [6.3, 11.9]
Gradient descent1,50071.669.42.2 [1.4, 2.9]
SGD with momentum1,50069.165.73.4 [2.3, 4.4]
SGD with momentum, Fashion-MNIST1,50067.866.61.2 [0.7, 1.7]
Adam, step 10−410^{-4}60065.762.53.2 [0.8, 5.6]
Adam, step 10−310^{-3}60074.975.2−0.2 [−1.3, 0.8]

The cost of the silent columns survives training under gradient descent, SGD and Adam with the smaller step, and it disappears only under Adam with the larger step. A network that starts with all features silent cannot learn at all. On raw Mackey-Glass inputs at a=1a=1 this happens in 12% of the networks of width 2. The product DnD^n for a silent fraction DD and width nn, which is exact at a=0a=0 and an approximation at a=1a=1, gives 11.8% (Appendix K.8).

The absence of an accuracy difference in Appendix I is consistent with these results. Its runs prescribe the marginal fraction of single inputs and use Adam at steps selected by validation, so they do not control the number of silent columns. The joint quantity is visible there as well. In each of its runs that ends at chance accuracy, no terminal feature is active on the calibration inputs.

On the architecture, the statement on training is not tied to our residual structure. Its proof uses only that a linear readout follows a ReLU layer, so it holds for any earlier layers. The limit theorems remain specific to this structure. Appendix E.12 now states which steps use the terminal ReLU and the absence of normalization, and Appendix J treats a branch with RMS normalization.

AuthorsOpenReview ↗

Response to Reviewer GA9u (2/3)

W2. Most of the main text calibrates the marginal inactivity for a single input, whereas the quantity relevant to readout capacity is the joint silent fraction. The joint inverse (Theorem 4.2) is solved only for ReLU and leaky ReLU with positive or Gaussian inputs.

A. Thank you for these remarks. We agree that the joint fraction is the quantity that matters for the readout. The revised Introduction and the new Section 6.3 now treat it directly, a new experiment in Appendix K.5 separates the two fractions, and Appendices E.6 and E.7 extend the joint inverse on the input side and delimit it on the activation side.

On the balance between marginal and joint calibration, the main text now gives the joint fraction more room. The Introduction defines both fractions and the joint empirical law in words, and Section 6.3 deals only with joint inactivity on real input sets. The offset experiment of Appendix K.5 moves the joint fraction while the marginal fraction stays fixed (Mackey-Glass, a=1a=1, M=128M=128, errors relative to c=0c=0).

Offset cc01246
Inactive fraction for one input (%)49.649.649.950.150.1
Joint silent fraction (%)0.48.620.633.038.4
Test error, original columns1.000.810.820.991.14
Test error, silent columns reflected1.000.720.600.550.56

The marginal fraction stays at one half while the joint fraction grows from 0.4% to 38%, and the gap between the last two rows grows with it. With a silent fraction DD, the MM original columns perform like (1−D)M(1-D)M active columns, a relation that holds within 10% in each of the 120 settings.

On the activations for which the joint inverse is solved, Proposition E.4 now shows which activations keep correlations from decreasing. With χϕ(q)=qE[ϕ′(qG)2]/vϕ(q)\chi_\phi(q)=q \mathbb E[\phi'(\sqrt qG)^2]/v_\phi(q) we prove χϕ(q)≥1\chi_\phi(q)\ge1, with equality exactly for functions that are linear on each half line, and the normalized kernel satisfies ρq(c)≥c\rho_q(c)\ge c on [0,1][0,1] exactly in the equality case. This is the property on which the monotone inverse rests. ReLU and leaky ReLU have it and move the correlations strictly, whereas a linear branch leaves them fixed and gives no strict inverse. GELU and SELU have χϕ>1\chi_\phi > 1, and we give correlated inputs for which their joint inactivity decreases near a=0a=0, which rules out an inverse that increases for all inputs.

On the inputs, the joint inverse now goes beyond positive and Gaussian laws. In Appendix E.6 we extend it to signed laws that are not Gaussian, the scale mixtures U=RZU=RZ with Gaussian ZZ and an independent positive gain RR shared by a row. Theorem 4.2 also no longer assumes a positive definite C0C_0. Nonzero inputs that are pairwise nonproportional suffice, so training sets with N>dN > d are covered. For general signed inputs some assumption is necessary. In Appendix E.12 we give a signed law with a positive definite second moment matrix for which D2(a)D_2(a) first decreases.

Q1. How does the convergence rate depend on mm? Is there a term analogous to 1/(mL)1/(mL) on the joint side? Is m=32m=32 in the experiments justified by this rate?

A. Thank you for this question. Yes. The new Proposition E.1 gives the same 1/(mL)1/(mL) term on the joint side, and at m=32m=32 this term is smaller than the width term in our experiments. It bounds the state Gram matrix Ck=XkTXk/nC_k=X_k^{\mathsf T}X_k/n against the covariance ODE started from the same finite input Gram matrix.

Emax⁡k≤L∥Ck−Cˉ(k/L)∥F2≤K(1n+1mL+1L2)\mathbb E\max_{k\le L}\Vert C_k-\bar C(k/L)\Vert_{\rm F}^2\le K\Big(\frac1n+\frac1{mL}+\frac1{L^2}\Big)

The constant KK depends on NN, the Lipschitz constant of the activation, bounds on the input energies and a bound on the coefficient. The input Gram matrix may be singular.

The dependence on mm comes from the fluctuation of the branch. Conditionally on the past, the branch contributes Hk=m−1∑j≤mϕ(zkj)ϕ(zkj)TH_k=m^{-1}\sum_{j\le m}\phi(z_{kj})\phi(z_{kj})^{\mathsf T} with zkj∼N(0,Ck)z_{kj}\sim N(0,C_k), whose mean is Kϕ(Ck)\mathsf K_\phi(C_k). At m=1m=1 this is a random matrix of rank one. It enters the Gram increment with weight a2/La^2/L, so its fluctuation has conditional variance of order 1/(mL2)1/(mL^2) per layer. The fluctuations of different layers are martingale differences, and their sum over LL layers has variance of order 1/(mL)1/(mL).

This rate justifies m=32m=32. For m=32m=32 and L=64L=64 the branch term 1/(mL)=1/20481/(mL)=1/2048 is smaller than the width term 1/n1/n in the forecasting experiments (n=512n=512) and in Figure 13 (n=1000n=1000), up to constants. Figure 13 confirms the calibration directly and includes m=1m=1. Over its 36 conditions with m∈{1,32}m\in\lbrace 1,32\rbrace and L∈{64,512}L\in\lbrace 64,512\rbrace, the largest deviation of a mean silent fraction from its target is 0.019, at m=1m=1 and L=64L=64, which is the size of the standard error of that mean.

AuthorsOpenReview ↗

Response to Reviewer GA9u (3/3)

Q2. Is it correct that for centered inputs the marginal inverse degenerates (the fraction is 1/2 for every aa)? If so, does the difference from He initialization reduce to the fact that marginal inactivity is controllable only when the identity path carries positive inputs, or do you claim an essentially new prescription on the joint side even for centered inputs?

A. Thank you for this precise question. Yes, the marginal inverse degenerates for centered inputs, and we do claim a new prescription on the joint side. Appendix E.12 now states both points.

On the first part, for centered Gaussian input rows, and more generally for symmetric input laws, the marginal fraction equals 1/21/2 for every aa, since EΦ(−U/w)=1/2\mathbb E\Phi(-U/\sqrt w)=1/2 for symmetric UU. Symmetry is the right condition. A law with mass 2/32/3 at −1-1 and 1/31/3 at 22 has mean zero, and its inactive fraction tends to 2/32/3 as a↓0a\downarrow0.

The difference from He initialization is more than the positivity of the identity path. The marginal inverse works because the identity path carries positive inputs, which supply nonzero thresholds, but the resulting coefficient uses more than the input variance. For the ReLU target p=0.25p=0.25 it is about 1.52 for constant positive inputs and 1.18 for half normal inputs with the same second moment, whereas a rule based on the input variance alone assigns both the same scale.

On the joint side the prescription is new also for centered inputs. By Theorem 4.2(ii), DN(a)D_N(a) increases strictly from Pr⁡(U≤0)\Pr(U\le0) to 1/21/2 while every marginal stays at 1/21/2, so each intermediate target fixes a unique coefficient. For two inputs with terminal correlation c(a)c(a),

D2(a)=14+arcsin⁡c(a)2π.D_2(a)=\frac14+\frac{\arcsin c(a)}{2\pi}.

In this Gaussian case, prescribing joint inactivity is equivalent to prescribing a terminal correlation (Corollary E.3), and the inverse expresses that choice as an expected number of unused features. For laws that are not Gaussian the equivalence fails, which is the content of the parity construction. There, equal pairwise statistics come with different joint inactivity and different expected rank.

The comparison with leaky ReLU makes the difference from variance matching explicit. Equal accumulated variance requires aβ=a0/1+β2a_\beta=a_0/\sqrt{1+\beta^2}, whereas an equal normalized joint feature law requires aβ=a0/(1−β)a_\beta=a_0/(1-\beta). The offset experiment in our answer to W2 shows the same point empirically. The marginal fraction is fixed at one half while the joint fraction, and with it the readout error, changes.

Thank you again for your time and thoughtful review. We hope that our responses and the revised manuscript address your concerns, and we would be glad to discuss any remaining questions.

Reviewer_SQN8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies inactivity at initialization as state width and depth tend to infinity, while branch width may remain finite. It considers residual networks with Gaussian weights and a terminal ReLU activation function. For a fixed number of inputs, it derives a limiting joint empirical feature law and uses it to choose a residual coefficient for a target inactivity level. It also constructs input laws with identical pairwise terminal feature laws but different joint inactivity and expected readout ranks.

Critical strengths
  1. The analysis explains why matching single-input inactivity need not match feature usage across inputs. It connects joint inactivity to shared-weight dependence and the input coordinate-row law, including when branch width remains finite.

  2. The parity construction shows that identical pairwise terminal feature laws can still give different expected readout ranks at a fixed finite budget of sampled limiting features. This gives a concrete fitting difference that the mean kernel does not capture.

Critical weaknesses

1. The importance of the inverse problem needs stronger justification. The paper shows that unused columns can limit readout fitting, but it does not sufficiently explain why setting a tolerance on unused columns is an important objective for choosing the residual coefficient. Even if this objective is useful for fixed features, it remains unclear what controlling inactivity at initialization tells us about the network after training. As the paper acknowledges, changing the residual weights can reopen silent features. The readout rank results alone therefore do not establish this connection.

2. The scope of the joint results and the presentation need clearer explanation. The joint results assume that C0=E[UU⊤]C_0=\mathbb{E}[UU^\top] is positive definite. In the Gaussian linear-embedding example, it is singular when N>dN>d. The main text should explain this restriction means for applying the joint inverse to a training set. Definitions and notation also make the paper hard to follow. For instance, the introduction should explicitly connect marginal/joint inactivity to per-input zeros/features zero on every input, and explain the joint empirical feature law. Examples of notation issues include the use of 1\mathbf{1} for both indicators and the all-ones vector, and changes between lowercase and uppercase vector symbols whose meanings and dimensions need clearer explanation.

Questions for authors

How should the tolerance on unused columns be chosen? Please explain why the largest residual coefficient meeting this constraint is useful, and whether the intended benefit concerns readout fitting with fixed features or training all weights.

8 comments
AuthorsOpenReview ↗

Response to Reviewer SQN8 (1/3)

We sincerely thank you for your time and thoughtful review. We have carefully addressed each of your concerns, and all changes are marked in red in the revised manuscript.

W1-1. The importance of the inverse problem needs stronger justification. It is not sufficiently explained why setting a tolerance on unused columns is an important objective for choosing the residual coefficient.

Q1-1. How should the tolerance on unused columns be chosen, and why is the largest residual coefficient meeting this constraint useful?

A. Thank you for these questions, which we answer together. We revised Sections 4 and 6.3 and added Appendix K.4 with Table 25. The tolerance is the admissible number of unused readout columns. The use of the readout determines it, and the largest coefficient that meets it is the end of the range in which the readout retains enough active columns. Section 4 now states how the use of the readout fixes the tolerance, Section 6.3 reports predicted and measured coefficients, and Appendix K.4 contains the rules and the experiment described below.

A tolerance on unused columns is an important objective because an unused column is lost to the readout. A column that is zero on every training input receives zero weight in a ridge readout or in a least squares readout of minimum norm. A readout on MM columns, ZZ of them unused, is therefore the readout on the other M−ZM-Z columns, on training and on test inputs. A tolerance of bb columns thus guarantees a budget of M−bM-b columns, which the coefficient controls.

The tolerance should be chosen from the use of the readout, which fixes bb. For fitting, NN points need N−1N-1 active columns, so b=M−N+1b=M-N+1, with no free parameter. For prediction, a readout with unused fraction DD has the test error of (1−D)M(1-D)M active columns. If this error falls like M−sM^{-s}, the unused columns multiply it by (1−D)−s(1-D)^{-s}, and an accepted factor 1+ε1+\varepsilon gives D≤1−(1+ε)−1/sD\le1-(1+\varepsilon)^{-1/s}.

The largest coefficient meeting the tolerance is useful because it is the right end of the admissible range. The unused fraction grows with the coefficient (Theorem 4.2), so the coefficients that meet a tolerance form an interval 0≤a≤ab/M0\le a\le a_{b/M}. Its right end summarizes this interval in one number, which the inverse returns from the Gram matrix of the training inputs before any weight is drawn. For fitting, the expected number of active columns falls below the required N−1N-1 as soon as the coefficient exceeds it. Inside the interval the choice is left to validation, because the coefficient also changes the features themselves.

We tested both rules on ten new repetitions per task, each with its own trajectory and network, with standardized inputs so that every unused column is due to the coefficient. The table compares the predicted with the measured largest admissible coefficient. For fitting, the measured value is the first coefficient with fewer than N−1N-1 active columns. For prediction, it is the coefficient at which the cost of the unused columns reaches the accepted increase, and its prediction uses the error curve of active columns.

RequirementToleranceMackey-Glass, predicted / measuredLorenz, predicted / measured
Fit 256 points with 272 columns17 columns2.38 / 2.342.58 / 2.61
Fit 256 points with 288 columns33 columns3.27 / 3.283.49 / 3.57
Fit 256 points with 304 columns49 columns4.11 / 3.934.33 / 4.57
Fit 256 points with 320 columns65 columns4.99 / 5.025.22 / 4.57
Test error at most 5% above that of active columnsfrom the error curve2.10 / 2.114.27 / 4.29
Test error at most 10% above that of active columnsfrom the error curve2.97 / 3.107.88 / 7.88

Over the eight fitting cases, predicted and measured boundaries differ by 0.16 on average. Half of the fifty fits per budget stop interpolating at 2.33, 3.21, 3.92 and 4.83 on Mackey-Glass and at 2.63, 3.31, 3.98 and 4.66 on Lorenz, close to these values. With raw inputs the predicted unused fraction is already 34% and 22% at a=0a=0, before the residual branches act, and it exceeds the fitting tolerance in 79 of the 80 cases. No coefficient is then admissible, and indeed no readout interpolates at a=0a=0.

The interval is also useful in practice. A search restricted to the coefficients whose predicted unused fraction rises by at most five percentage points over a=0a=0 returns the model of the full search in 98 of 100 fits and saves 21% of the selection time (Table 2). The inverse can also be solved for a constant input coordinate. The offsets that it gives for 10% and 25% unused columns lower the test error on Mackey-Glass to 0.79 and, with the unused columns reflected, to 0.55 of its value without offset (Appendix K.5).

AuthorsOpenReview ↗

Response to Reviewer SQN8 (2/3)

W1-2. Even if this objective is useful for fixed features, it remains unclear what controlling inactivity at initialization tells us about the network after training, since changing the residual weights can reopen silent features.

Q1-2. Does the intended benefit concern readout fitting with fixed features or training all weights?

A. Thank you for raising this point. We added a proof to Appendix K.7, new experiments in Appendices K.8 and K.9, and their results to Section 6.3. The benefit concerns both settings. For fixed features it is exact, as described above. For trained networks an unused column is absent from training until it reopens, which slows learning and lowers the accuracy after training.

Controlling inactivity at initialization tells us how many columns take part in training from the start. A column that is unused on every training input receives no gradient and passes none on. Gradient descent, SGD and Adam therefore follow exactly the trajectory of the network without that column, until the column reopens. A network with ZZ unused columns thus starts training as a network that is narrower by ZZ columns (Appendix K.7).

Changing the residual weights does reopen unused columns, but only a part of them. We followed all 706 initially unused features of 56 MNIST runs. Under gradient descent 45% of them are still unused after 1,500 updates, and 37% are never active (Table 28).

The benefit therefore remains when all weights are trained. To isolate the unused columns at a fixed coefficient, the readout reads sixteen columns of a network with n=256n=256 whose residual weights are all trained (Appendix K.9, Table 31). We compare sixteen active columns (arm B) with eight active and eight unused columns (arm A), from the same initial weights, over 24 networks. Entries are test accuracy in %, on MNIST unless stated.

TrainingUpdatesBACost, B minus A [95% interval]
Fixed features32.025.07.0 [6.0, 7.9]
Gradient descent10061.352.29.1 [6.3, 11.9]
Gradient descent1,50071.669.42.2 [1.4, 2.9]
SGD with momentum10062.650.612.0 [10.3, 13.8]
SGD with momentum1,50069.165.73.4 [2.3, 4.4]
SGD with momentum, Fashion-MNIST10065.356.68.7 [6.3, 11.1]
SGD with momentum, Fashion-MNIST1,50067.866.61.2 [0.7, 1.7]
Adam, step 10−410^{-4}10063.454.19.3 [7.4, 11.3]
Adam, step 10−410^{-4}60065.762.53.2 [0.8, 5.6]

Unused columns at initialization slow learning and leave a lower accuracy at the end of training, under all three optimizers and on both data sets. Under gradient descent the training set is fitted after a median of 340 updates against 165. Only Adam with the larger step 10−310^{-3} reopens the columns within a few updates and removes the cost. Across coefficients the picture is the same. When the inverse sets the initial unused fraction to 0.5%, 2%, 5%, 10% and 20%, the test accuracy after gradient descent is 78.1%, 76.0%, 73.5%, 68.7% and 56.5% (Table 29).

The extreme case is a network whose terminal features are all unused (Appendix K.8). No gradient then reaches the residual weights, and the network cannot learn. At a=0a=0 the features are independent, and such a start has probability exactly DnD^n, where DD is the unused fraction and nn the width. At a=1a=1 the features are coupled through the shared weights, and DnD^n is an approximation that treats them as independent. It stays close to what we observe. Among 5,000 initializations at a=1a=1 on raw Mackey-Glass inputs, 12.1%, 4.0% and 1.6% of the networks of width 2, 3 and 4 start this way, against 11.8%, 4.0% and 1.4%. A tolerance δ\delta on such starts then corresponds to D≤δ1/nD\le\delta^{1/n}, exactly at a=0a=0 and approximately at a=1a=1.

AuthorsOpenReview ↗

Response to Reviewer SQN8 (3/3)

W2. The joint results assume that C0=E[UU⊤]C_0=\mathbb E[UU^\top] is positive definite. In the Gaussian linear embedding example it is singular when N>dN>d, and the main text should explain what this restriction means for applying the joint inverse to a training set. Definitions and notation also make the paper hard to follow, for instance marginal and joint inactivity, the joint empirical feature law, the symbol 1 for indicators and for the all ones vector, and lowercase and uppercase vector symbols.

A. Thank you for this valuable suggestion. We agree that requiring a positive definite C0C_0 excluded every training set with N>dN>d. We removed this requirement from Theorems 4.1 and 4.2, explained in Section 4 what the new assumptions mean for a training set, and rewrote the definitions and the notation.

The joint results no longer need a positive definite C0C_0. Theorem 4.1 now requires positive diagonal entries of C0C_0, which means nonzero inputs. Theorem 4.2 requires in addition ∣(C0)st∣<(C0)ss(C0)tt|(C_0)_{st}|<\sqrt{(C_0)_{ss}(C_0)_{tt}} for every pair. A Gram matrix may therefore be singular, as it is whenever N>dN>d. The proofs still hold because the ReLU or leaky ReLU kernel of such a Gram matrix is positive definite, which makes the covariance added by the residual stack positive definite for every a>0a>0 (Lemma E.2). Proposition E.1 adds the rate 1/n+1/(mL)+1/L21/n+1/(mL)+1/L^2 for the Gram matrix, again without a positive smallest eigenvalue. Appendix E.12 gives the details.

For a training set embedded by a Gaussian matrix, the new assumptions mean that the joint inverse applies as soon as the raw inputs are nonzero and no two of them are proportional. A subset of the training inputs also suffices. A feature that is unused on the full set is unused on every subset, so the calibrated fraction of a subset bounds the fraction of the full set. Our experiments are of this kind. The inputs have eight or nine coordinates, so every Gram matrix below is singular, and the prediction remains accurate.

Input setNumber of inputsSettingsMean error of the predicted unused fraction
Subset of the training inputs (Section 6.3)64400.63 points
All training inputs, offsets (Appendix K.5)2,000800.46 points
All training inputs, coefficients up to 6 (Appendix K.4)2,000480.41 points

We also rewrote the definitions and the notation that you list.

  • The Introduction now defines marginal inactivity as the fraction of features that are zero for one input and joint inactivity as the fraction that are zero on every input.
  • The joint empirical feature law is the law of the values that a uniformly drawn coordinate takes across all inputs.
  • The indicator is written I{⋅}\mathbb I\{\cdot\} and the all ones vector 1N\mathbf 1_N.
  • Lowercase symbols are vectors in Rn\mathbb R^n for one input, and uppercase symbols are indexed by the NN inputs (Table 3 in Appendix A).
  • Section 4 states that, after the terminal ReLU, a row of Xk∈Rn×NX_k\in\mathbb R^{n\times N} becomes a column of the readout matrix, so a jointly silent row is an unused readout column.

Thank you again for your time and thoughtful review. We hope that our responses and the revised manuscript address your concerns, and if so, we kindly ask you to consider revisiting your rating. We would be glad to discuss any remaining questions.

Reviewer_qtNnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper investigates inactive features in residual neural networks and how to control their fraction at initialization. Previous work has studied the variance of feature activation; however, controlling variance alone does not determine the fraction of features that are zero on every input.

The paper develops a method for choosing the residual initialization scale to achieve a prescribed inactive-feature fraction.

The authors consider a simple, mathematically tractable model: an unnormalized residual network of the form

xk+1=xk+aLBkϕ(Akxk)x_{k+1}=x_k+\frac{a}{\sqrt L}B_k\phi(A_kx_k),

with independent Gaussian matrices AkA_k and BkB_k, branch activations chosen from ReLU, leaky ReLU, GELU, and SELU, and a single ReLU after the final block.

For strictly positive inputs satisfying the stated assumptions, the authors establish the existence of a unique scale aa that achieves a prescribed marginal inactive characteristic fraction p∈(0,1/2)p\in(0,1/2) in the width–depth limit.

The analysis is then extended to a fixed set of inputs. The authors define a feature as silent on this set if its terminal output is zero on every input. By tracking the joint input-coordinate law and the evolution of the second-moment matrix, they prove convergence of the silent-feature fraction to DN(a)D_N(a). For ReLU and leaky ReLU branches, they establish a unique scale achieving any target d∈(0,1/2)d\in(0,1/2) for positive input laws and any target d∈(Pr⁡(U≤0),1/2)d\in(\Pr(U\le0),1/2) for centered Gaussian input laws, under the stated assumptions.

The authors show that pairwise statistics alone cannot determine joint inactivity or finite-feature fitting capacity for a linear readout.

Finally, they compared the scales required for ReLU and leaky ReLU, distinguishing variance matching from matching the normalized joint feature law for centered Gaussian inputs.

Critical strengths

-) The paper provides a mathematical way to select the residual initialization scale to achieve a desired fraction of inactive features (in the case of the specified model and under a Gaussian distribution of matrix AkA_k and BkB_k). Rather than only predicting inactivity (at initialization) for a given scale, the authors establish conditions under which a unique scale achieves the prescribed target in the joint width–depth limit. The results are reported for the ReLU, Leaky ReLU, GELU and SELU activations.

-) The paper constructs two joint input-coordinate laws with identical pairwise limiting feature distributions, and therefore the same mean kernel, but different fractions of features that are zero on every input. With a fixed finite number of sampled features, these distributions also yield different expected feature-matrix ranks. Using this construction, the authors show that the mean kernel alone can miss differences in finite-feature fitting capacity.

Critical weaknesses

The paper provides a clear characterization of inactivity at initialization under independent Gaussian weights, with experiments supporting the predictions at finite widths and depths. I would appreciate a broader discussion of how these findings relate to subsequent full-network training, where initially silent features may reactivate, and which conclusions might extend beyond Gaussian initialization. In particular, the authors report in Appendix I that ``correcting initial activity does not produce an accuracy advantage here.'' Discussing whether and when initial feature usage is expected to influence subsequent learning would help readers assess the practical scope of the results.

Questions for authors

-) Regarding the experimental results, do the available measurements suggest an association between the initial joint silent-feature fraction and subsequent optimization or final performance? If feature-level trajectories are available, it would also be informative to know whether initially silent features tend to reactivate. Are there theoretical arguments or relevant references that help interpret this connection?

-) The forward limit law and marginal calibration results already cover a general class of activations. Could the authors clarify which additional properties are needed to extend the joint-inactivity inverse beyond the considered activation functions? Can the results be generalized to broad conditions on the activation, rather than established for specific choices?

The following questions concern possible extensions; a qualitative discussion would be sufficient.

-) I am curious about how generalizable the results are, beyond the assumptions studied? In particular, which parts of the analysis require Gaussian weights, and what difficulties would arise with other independent weight distributions? Similarly, beyond positive and centered Gaussian input laws, are there other input distributions for which the inactive-feature fraction increases strictly with the residual scale, ensuring a unique scale for each attainable target?

-) Could the framework accommodate deeper residual branches in place of Bϕ(Ax)B\phi(Ax)? What can be the main obstacle there?

4 comments
AuthorsOpenReview ↗

Response to Reviewer qtNn (1/3)

We sincerely appreciate the time you took to read and review our paper. We have carefully addressed each of your concerns, and all changes are marked in red in the revised manuscript.

W1. A broader discussion of how these findings relate to subsequent training of the full network, where initially silent features may reactivate, and of which conclusions extend beyond Gaussian initialization would help. Appendix I reports that correcting initial activity does not produce an accuracy advantage. Whether and when is initial feature usage expected to influence subsequent learning?

Q1-1. Do the available measurements suggest an association between the initial joint silent fraction and subsequent optimization or final performance?

A. Thank you for these questions. Yes. The final accuracy decreases with the initial silent fraction, and a controlled experiment shows that the silent features themselves slow optimization and lower the final accuracy.

Association (Appendix K.8, Table 29). We used the inverse to set five initial joint silent fractions and trained all residual weights on MNIST (n=256n=256, L=64L=64, eight initializations).

Target silent fraction0.5%2%5%10%20%
Silent features at initialization (%)0.31.73.98.819.8
Test accuracy, gradient descent (%)78.176.073.568.756.5
Test accuracy, Adam, step 10−410^{-4} (%)83.481.778.873.262.0
Test accuracy, Adam, step 10−310^{-3} (%)84.984.182.980.269.4

The accuracy falls with the initial silent fraction under all three optimizers, although every run fits the training set. Since the coefficient changes together with the silent fraction here, we separated the two in a second experiment.

Controlled experiment (Appendix K.9, Table 31). The coefficient is fixed and only the number of silent features differs. The readout of a network with n=256n=256 reads sixteen terminal features, and all residual weights are trained. In arm B all sixteen are active at initialization, and in arm A eight of them are silent. Both arms start from the same weights, and we repeat this for 24 networks. The table reports test accuracy in % (MNIST unless Fashion-MNIST is named).

TrainingUpdatesBACost, B minus A [95% interval]
Fixed features32.025.07.0 [6.0, 7.9]
Gradient descent10061.352.29.1 [6.3, 11.9]
Gradient descent1,50071.669.42.2 [1.4, 2.9]
SGD with momentum10062.650.612.0 [10.3, 13.8]
SGD with momentum1,50069.165.73.4 [2.3, 4.4]
SGD with momentum, Fashion-MNIST10065.356.68.7 [6.3, 11.1]
SGD with momentum, Fashion-MNIST1,50067.866.61.2 [0.7, 1.7]
Adam, step 10−410^{-4}10063.454.19.3 [7.4, 11.3]
Adam, step 10−410^{-4}60065.762.53.2 [0.8, 5.6]
Adam, step 10−310^{-3}60074.975.2−0.2 [−1.3, 0.8]

Silent features slow optimization and lower the final accuracy under gradient descent, SGD and Adam with the smaller step. Arm B fits the training set after a median of 165 updates of gradient descent and arm A after 340. With fixed features the cost is permanent, and it disappears only under Adam with the larger step.

Initial feature usage matters in three cases. These are fixed features, weights that move slowly, as under gradient descent, SGD and Adam with a small step, and networks whose terminal features are all silent (Q1-3 below). Appendix I prescribes the marginal fraction of single inputs and trains with Adam at steps selected by validation, so it does not control the number of silent features. The last case still appears there. Every run of Appendix I that ends at chance accuracy has all terminal features silent on the calibration inputs. The statement on training (Q1-3) does not use the weight distribution, so it extends beyond Gaussian initialization. Section 6.3 now states these findings in the main text.

AuthorsOpenReview ↗

Response to Reviewer qtNn (2/3)

Q1-2. If trajectories of individual features are available, do initially silent features tend to reactivate?

Q1-3. Are there theoretical arguments or relevant references that help interpret this connection?

A. Thank you for these questions. Initially silent features do reactivate, at a speed set by the optimizer, and we prove that until then a silent feature takes no part in training.

Reactivation (Table 28). We tracked the 706 initially silent features of the 56 runs of each optimizer.

OptimizerStill silent at the endNever activeMedian update of first activation
Gradient descent, 1,500 updates316 (45%)263 (37%)135
Adam, step 10−410^{-4}, 600 updates175 (25%)138 (20%)22
Adam, step 10−310^{-3}, 600 updates87 (12%)8 (1%)2

Under gradient descent and Adam at the smaller step, 37% and 20% of the silent features are never active, and the cost remains. At the larger step all but 1% reopen, and the cost disappears.

Theory (Appendix K.7). A feature that is silent on all training inputs has zero gradient for its readout weight and its preactivation. Gradient descent, SGD and Adam therefore follow the trajectory of the network without this feature until it reopens. It can reopen because the shared residual weights keep moving through the active features, and it stays silent while its preactivations move by less than its initial margin. In the controlled experiment the outputs with and without the silent columns coincide in all 24 networks until the first reopening, under gradient descent and Adam.

When all terminal features are silent, the residual weights receive no gradient and the network cannot learn. For a=0a=0 the terminal features come from independent rows of the embedding, so such a start has probability exactly DnD^n (silent fraction DD, width nn). For a=1a=1 they share the residual weights, and DnD^n is an approximation. On raw Mackey-Glass inputs it gives 11.8%, 4.0% and 1.4% for widths 2, 3 and 4, and we observe 12.1%, 4.0% and 1.6% among 5,000 initializations (Appendix K.8). The seven such starts in Table 30 do not learn, and all seven learn after reflection.

References. The connection to earlier work, now in Appendix K.7, is as follows. Tentative and permanent death of ReLU units at initialization [1] is close to our distinction between features that reopen and those that never do. A deep ReLU network with symmetric initialization becomes constant with probability tending to one as depth grows [2], the analogue of our completely silent case. Dormant units accumulate during deep reinforcement learning, and recycling them helps [3]. In sparse autoencoders, features that oppose the mean of the input activations are dead at initialization and can stay dead throughout training [4].

Q2. Which additional properties are needed to extend the inverse for joint inactivity beyond the considered activation functions? Can the results be generalized to broad conditions on the activation?

A. Thank you for this question. Beyond the forward law, the joint inverse needs control of how the correlations between inputs evolve, and Appendix E.7 now states this for general activations.

  • General conditions. For positive inputs it suffices that the added covariance WaW_a is positive definite for a>0a>0, with variances that increase strictly in aa and correlations that do not decrease. For centered Gaussian inputs it suffices that C0+WaC_0+W_a is positive definite for a>0a>0, with correlations that do not decrease and at least one that increases strictly on every interval. Then DN(a)D_N(a) increases strictly, also for unequal input energies.
  • Which activations keep correlations from decreasing. For a Lipschitz activation with ϕ(0)=0\phi(0)=0 and input energy qq, let χϕ(q)=q E[ϕ′(qG)2]/vϕ(q)\chi_\phi(q)=q\,\mathbb E[\phi'(\sqrt qG)^2]/v_\phi(q). Proposition E.4 shows that χϕ(q)≥1\chi_\phi(q)\ge1, and that the normalized kernel leaves no correlation in [0,1][0,1] smaller than before exactly when χϕ(q)=1\chi_\phi(q)=1, that is, when the activation is linear on each half line. ReLU and leaky ReLU belong to this case and increase the correlations strictly, which gives the unique inverse of Theorem 4.2, whereas a linear branch leaves them fixed. GELU and SELU have χϕ>1\chi_\phi>1, and we exhibit correlated inputs for which their joint inactivity decreases for small aa. This excludes an inverse that increases for all inputs.

[1] Shin & Karniadakis, Trainability of ReLU networks and data-dependent initialization, J. Mach. Learn. Model. Comput., 2020

[2] Lu et al., Dying ReLU and initialization: theory and numerical examples, Commun. Comput. Phys., 2020

[3] Sokar et al., The dormant neuron phenomenon in deep reinforcement learning, ICML, 2023

[4] Simon et al., On the relationship between activation outliers and feature death in sparse autoencoders, arXiv:2605.31518, 2026

AuthorsOpenReview ↗

# Response to Reviewer qtNn (3/3)

Q3. Which parts of the analysis require Gaussian weights, and what difficulties would arise with other independent weight distributions? Beyond positive and centered Gaussian input laws, are there other input distributions for which the inactive fraction increases strictly with the residual scale?

A. Thank you for raising these two points. Gaussian weights enter the proofs in exactly two places, and the joint inverse now also covers a family of signed input laws that are not Gaussian.

Weights. We added this discussion to Appendix E.12. Given the current states, each row of AkXkA_kX_k is exactly N(0,Ck)N(0,C_k). Given the branch outputs, each row of Bkϕ(AkXk)B_k\phi(A_kX_k) is exactly Gaussian with the empirical branch covariance. Both identities hold at any width, which is what allows mm to stay fixed, even at one. For other independent centered entries of BkB_k with the same variance, the conditional covariance is unchanged, and a martingale central limit theorem over depth is the natural tool. It needs moment bounds, a Lindeberg condition and concentration of the Gram matrix. Replacing AkA_k is harder, because the preactivations are then only approximately Gaussian. One has to ensure that no single coordinate dominates, for instance max⁡i∥Xk,i∥/n→0\max_i\|X_{k,i}\|/\sqrt n\to0, and to propagate this through depth. The statement on training in our answer to Q1-3 needs no assumption on the weights.

Inputs. We added this extension to Appendix E.6. The new family consists of scale mixtures U=RZU=RZ with Z∼N(0,C0)Z\sim N(0,C_0) and an independent positive scalar RR with ER2=1\mathbb ER^2=1, shared by the NN inputs of a row. For ReLU or leaky ReLU and a Gram matrix with positive diagonal and pairwise correlations below one in absolute value, DN(a)D_N(a) increases continuously and strictly from Pr⁡(Z≤0)\Pr(Z\le0) to 1/21/2, so each target in between has a unique coefficient. The proof conditions on R=rR=r. The covariance is then r2C0+War^2C_0+W_a, whose correlations mix the initial ones with those of the added noise, with a weight that increases in aa. The Gaussian argument applies for each rr, and averaging over RR keeps strictness. A Gaussian embedding followed by random positive row gains is a concrete example. The same theorem now also covers singular Gram matrices, which admits more inputs than input dimensions.

Some assumption on signed laws is needed. Two independent coordinates that take the value −1-1 with probability 1/31/3 and 22 with probability 2/32/3 have a positive definite second moment matrix and D2(0)=1/9D_2(0)=1/9, yet D2(a)D_2(a) first decreases (Appendix E.12).

Q4. Could the framework accommodate deeper residual branches in place of Bϕ(Ax)B\phi(Ax)? What is the main obstacle?

A. Thank you for this question. We added a discussion of deeper branches to Appendix E.12. The strategy carries over, and the main obstacle is the finite internal width of the branch.

If the branch ends with an independent centered linear map, the residual increments remain centered, fresh weights in each block keep the averaging over depth, and the covariance drift is given by the expected second moment map of the whole branch. With finite internal width the intermediate covariance matrices inside the branch are random, and in general E[Kϕ(H)]≠Kϕ(EH)\mathbb E[\mathsf K_\phi(H)]\ne\mathsf K_\phi(\mathbb EH), so the drift differs from a composition of kernels at infinite width. A proof would identify this effective map and establish moment bounds and enough regularity for the comparison with the Gram flow. A joint inverse would need, in addition, monotone correlations and thresholds for the effective map. Growing internal widths remove the difficulty through concentration.

Finally, we revised the passages that were hard to read. The Introduction now says in words what marginal and joint inactivity are, and Section 4 spells out what each matrix and vector contains.

Thank you again for your time and thoughtful review. We hope that our responses and the revised manuscript address your concerns, and if so, we kindly ask you to consider revisiting your rating. We would be glad to discuss any remaining questions.

Dear Reviewer qtNn,

We believe we have carefully and comprehensively addressed all your concerns and questions. As the discussion period is set to close in less than 30 hours, we would be grateful for any additional suggestions or specific points to further enhance our manuscript.

We sincerely hope you might reconsider your evaluation or share further insights to help us strengthen our work.

Thank you once again for your support and thoughtful guidance throughout this process. We deeply value your time and effort in reviewing our work.

Best regards,
Authors of Submission 59255

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

Dear Reviewers and Area Chair,

Thank you for your careful reading and helpful suggestions. We appreciate your comments on the inverse theory as width and depth grow with branch width held fixed (qtNn, GA9u), the distinction between inactivity on one input and unused columns across inputs (SQN8), and the parity construction (qtNn, SQN8, GA9u). We have uploaded the revised manuscript. Changes in the PDF are highlighted in red.

Main concerns and revisions

1. How can we choose an inactivity target and use it to set the residual coefficient before training? (SQN8, GA9u)

The number of columns the output layer needs gives a practical target. In a regression task requiring at least two columns to represent the outputs, we allowed a 5% risk of starting with fewer than two. Using forward passes alone, we chose the largest coefficient meeting this limit. Both this rule and validation over 13 trained candidates selected a=3.0a=3.0 at 20 layers and a=3.5a=3.5 at 80 layers (Section 6.3 and Appendix K.11). We also tested the inverse formula directly on Mackey-Glass, a nonlinear time series forecasting task. At 128 layers, a target of 5% unused columns gave a=1.97a=1.97 on average from the training inputs alone. After training, test error was close to that at a=2a=2 at both state widths tested (Appendix K.11, Table 34).

2. Does choosing the initialization this way improve prediction after all weights are trained? (qtNn, SQN8, GA9u)

On three regression tasks, choosing the coefficient from the new training inputs gave lower test error than reusing a=2a=2, the value that worked well on Mackey-Glass. In optical depth recovery, error fell from 0.4328 to 0.2673, a 38.2% reduction, with lower error on all 24 test datasets (Section 6.3 and Appendix K.11, Table 35). Table 36 also shows that a single coefficient chosen by validation on Mackey-Glass and optical depth recovery gave a lower mean test error on those two tasks than the inverse. We also held the coefficient fixed and used the inverse to set the initial bias. On Mackey-Glass, targeting 10% unused columns gave the lowest mean test error of the five targets tested. Error was 36% lower than with a constant input of one and 61% lower than with a coefficient chosen by validation from three candidates (Appendix K.10, Table 32).

3. How broadly does the theory apply, and how does its accuracy depend on network size? (qtNn, SQN8, GA9u)

The joint results now cover settings with more inputs than input dimensions (Theorems 4.1 and 4.2), and the inverse extends to Gaussian scale mixtures (Appendix E.6). Proposition E.1 bounds the mean squared error of the state Gram matrix by a term of order 1/n+1/(mL)+1/L21/n+1/(mL)+1/L^2, where nn is state width, mm is branch width, and LL is depth. Proposition E.4 shows that for GELU and SELU, joint inactivity can decrease as the coefficient increases.


Further revisions

Point raisedRevision
Unused columns before and during training (qtNn, SQN8, GA9u)We predict when too few fixed columns remain to fit the data (Section 4 and Appendix K.4, Table 25) and show when initially unused columns delay learning (Appendices K.7–K.9).
Definitions and scope (qtNn, SQN8, GA9u)We clarified the notation in the Introduction, Section 4, and Table 3. The offset experiment varies joint inactivity at fixed marginal inactivity (Appendix K.5). Appendix E.12 discusses centered inputs, other weight distributions, and deeper branches.
Motivation and prior work (GA9u)Section 1 distinguishes our contribution from Price et al. (2024). Section 2 explains why we use a terminal ReLU.

Discussion to date

  • GA9u raised their rating from 3 to 4, noting that the revised theory and experiments addressed their questions about branch width, variance-based initialization, and practical significance.
  • SQN8 recognized the theoretical contribution and maintained a rating of 2, asking for more direct evidence that inactivity can guide the choice of initialization. In our reply, we showed that choosing the initial bias from an inactivity target lowered test error on Mackey-Glass (Appendix K.10). On three regression tasks, choosing the residual coefficient from the new training inputs also gave lower test error than reusing a=2a=2 (Appendix K.11). We have not yet received further comments and would welcome the reviewer’s thoughts on these results.
  • qtNn gave a rating of 2 and asked how inactivity at initialization affects subsequent training. We addressed this question with a proof and training experiments in Appendices K.7–K.9 and have not yet received a follow-up.

Thank you again for the time and care you have given our paper.

Best regards,
Authors of Submission 59255

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.