acceptodds
Under review as a conference paper at ICLR 2027

Drift Is Not Enough: A Diffusive Floor for Gradient Propagation in Recurrent Networks

Abstract

Whether a recurrent model learns long-range dependencies is determined by whether its gradients survive backpropagation over long horizons without exploding or vanishing. This behavior is governed by the top Lyapunov exponent , which characterizes the asymptotic rate of gradient growth or decay. While recent regularization methods seek to ensure marginal stability by driving this exponent to zero, the dynamics governing the non-autonomous gradient propagation remain insufficiently characterized. In this work, we model gradient dynamics as a linear cocycle over an ergodic Markov chain. Within this framework, we prove that the Lyapunov exponent spectrum of Recurrent Neural Network (RNN) gradient dynamics is well-defined and independent of the input realization, acting as a deterministic Law of Large Numbers drift. Crucially, while this drift rate governs the asymptotic scaling, gradient transport over any finite horizon exhibits second-order fluctuations governed by a companion Central Limit Theorem with asymptotic variance . Consequently, eliminating drift () does not prevent degradation, but leaves a superpolynomial, two-sided dispersion scaling as , rendering vanishing and explosion equiprobable. We analytically prove that the asymptotic variance is strictly non-degenerate () for any choice of recurrent weights in vanilla RNNs with saturating activations, showing that gradient dispersion is an inescapable feature of these models. In our experiments, penalizing both the drift (gradient flossing) and the variance reduces empirical diffusion by 63–74% and improves psMNIST accuracy from (unregularized) and (flossing alone) to . Because this formulation requires only a recurrent state, it extends beyond vanilla RNNs, and we observe the same growth in gated, orthogonal and selective state-space recurrent models.

Primary area:
learning on time series and dynamical systems
Keywords:
recurrent neural networks, exploding and vanishing gradients, Lyapunov exponents, gradient propagation, marginal stability, edge of chaos, random dynamical systems, long-range dependencies, regularization

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_QgacOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that zeroing the leading Lyapunov exponent λ1\lambda_1 only removes average log growth, not the accumulated fluctuations in log norms. Under stated assumptions the authors prove a CLT with positive asymptotic variance σ2\sigma^2, derive a propagation horizon combining drift and fluctuations, and add a βσ2\beta\sigma^2 penalty to Gradient Flossing, estimating σ2\sigma^2 from batch means of the per-step QR stretches.

Critical strengths
  • The framework is more complete than looking at the exponent alone: λ1\lambda_1 gives the average trend, σ2\sigma^2 the accumulated fluctuations, and both enter the propagation length. Establishing nondegenerate fluctuations under explicit assumptions is a real contribution.

  • The experiments go beyond a single baseline. Table 1 sweeps α\alpha for flossing; at α=30\alpha=30 flossing reaches a final λ1\lambda_1 similar to the proposed method, but the latter gets lower σ\sigma and higher psMNIST accuracy (0.860±0.0050.860\pm0.005 vs 0.826±0.0090.826\pm0.009). Remeasuring on independent long trajectories helps. This is decent evidence that variance control matters in practice.

Critical weaknesses
  1. The link to finite-window flossing needs more work, especially how window length affects the comparison. Using Appendix D.8's notation, the single-trajectory window estimate is fˉ=m−1∑t=1mft\bar f=m^{-1}\sum_{t=1}^{m}f_t. Under stationarity, Proposition D.10 gives Eπ[ft]=λ1\mathbb E_\pi[f_t]=\lambda_1. With Rn=Cov⁡π(f1,f1+n)R _ n=\operatorname{Cov} _ \pi(f_1,f_{1+n}) and absolutely summable autocovariances, Equation (33) gives σ2=R0+2∑n≥1Rn\sigma^2=R_0+2\sum _ {n\ge1}R_n, hence
=\lambda_1^2+\frac{\sigma^2}{m}+o(m^{-1}).$$ So finite-window flossing already penalizes fluctuations implicitly; the added term mainly makes the weight independently adjustable. For the leading-exponent part, when the bias of $\widehat\sigma^2$ is negligible, the expected penalties are roughly $$\alpha\lambda_1^2+\frac{\alpha}{m}\sigma^2, \qquad \alpha\lambda_1^2+\left(\frac{\alpha}{m}+\beta\right)\sigma^2.$$ This is not an equivalence between the training methods, and it assumes stationarity and long windows. What's missing is a joint $m\times\alpha$ comparison with task performance, which would show how robust the benefit of independent weighting is to window choice. I'm asking to strengthen existing evidence, not claiming the decomposition refutes the main claim about variance control. 2. **The acknowledged theoretical limitations should show up more consistently in the main text.** The CLT and strict positivity need i.i.d. inputs, a full-row-rank input map ($d_u\ge d_h$), and dense nonsingular recurrent weights. Section 6 flags the low-dimensional, correlated-input results as empirical, and Remark D.2 says $d_u\ge d_h$ is a requirement of the current proof. So this is mostly a presentation issue. Appendix E.2 already ties the roughly $m^{0.66}$ growth on psMNIST to correlated inputs violating Assumption 1, and says so explicitly. The main text and captions should be equally clear, and "diffusive scaling" shouldn't be used so broadly that it blurs the distinction from the $\sqrt m$ prediction. The step from a fixed-vector forward Jacobian product to actual backward gradients, whose directions depend on data and loss, also needs more explanation.
Questions for authors
  1. Implementation and contribution. Is the Lyapunov penalty averaged after squaring each trajectory's exponent, or is the exponent averaged across trajectories before squaring? Do you agree with the single-trajectory second-moment decomposition above, and with reading the added term as independently weighting long-run fluctuations? The aggregation rule changes the implicit variance penalty.

  2. What would change my assessment. A small joint comparison of window length mm and α\alpha under comparable training and tuning budgets, reporting task performance for k=1k=1 or an independently weighted leading exponent. Also, how sensitive is the training-time σ^2\widehat\sigma^2 estimate to block length ℓ\ell? That would separate long-run fluctuation control from finite-window estimation effects.

  3. Connection to actual gradients. Given the qualifications in Limitations, Remark D.2, and Appendix E.2: how does the fixed-vector forward-product result apply to backward gradients that depend on data and loss? Please state the required conditions, and align the main text and captions on correlated inputs, other recurrent architectures, and the 0.660.66 scaling with the existing scope statements.

2 comments

Q1. Aggregation. The exponent is averaged before squaring: λ^i\hat\lambda_i is the mean over N=32N=32 trajectories and m=128m=128 steps, and the penalty is α∑iλ^i2\alpha\sum_i\hat\lambda_i^2.

W1. Implicit fluctuation penalty. We thank the reviewer for this derivation. We agree with it and with the reading of the added term, and will add both to the paper: flossing does penalize σ2\sigma^2, and βσ2\beta\sigma^2 makes this weight independently adjustable. In our runs the estimate is averaged before squaring, so E[λ^12]≈λ12+σ2/(mN)\mathbb E[\hat\lambda_1^2]\approx\lambda_1^2+\sigma^2/(mN), an implicit weight of α/4096\alpha/4096; with per-trajectory squaring it is α/m\alpha/m. The implicit weight, however, cannot be made large. It grows only with α\alpha, which also pushes the drift harder (training fails at α=100\alpha=100, Table 1), or with a shorter window, which no longer measures the asymptotic σ2\sigma^2. It also decreases as 1/(mN)1/(mN), so the more accurately flossing estimates λ1\lambda_1, the less it penalizes σ\sigma. Within the range that trains, the weight is at most about 11, where the β\beta sweep shows only a modest effect; the main gains need β≈3\beta\approx3–3030, which the implicit term would reach only at α\alpha in the thousands or above (R5). The α\alpha sweep agrees: raising α\alpha from 11 to 3030 lowers σ\sigma from 0.1160.116 to 0.0780.078, and no further (0.0900.090 at α=100\alpha=100), against 0.0380.038 with βσ2\beta\sigma^2 (Table 1).

The reason to penalize σ\sigma directly comes from the CLT. σ\sigma is determined by the weights, cannot be zero (Prop. 4.2), and together with λ1\lambda_1 determines the usable horizon m∗m^\ast; even at λ1=0\lambda_1=0, the transported gradient spreads as e±σme^{\pm\sigma\sqrt m}. We will add the decomposition to Section 5.2, crediting the reviewer, with an appendix collecting the α\alpha and β\beta sweeps.

Q2. Window, α\alpha and block length. Per-trajectory flossing at m∈{32,128}×α∈{1,30}m\in\{32,128\}\times\alpha\in\{1,30\} has an implicit weight between 0.0080.008 and 0.940.94, within the range β≤1\beta\le1 of the sweep in R5. At α=30\alpha=30 the drift weight also changes; flossing at α=30\alpha=30 alone reaches 0.8260.826. We believe this answers the question the grid would test, so we have not trained it. If the reviewer still considers it worthwhile, we will run it and report the results.

For the block length, see R3: the estimate of σ\sigma for the variance-penalized network is flat in ℓ\ell, so we do not expect the training block length to matter, although we have not trained with other values of ℓ\ell. These measurements also correct App. D.8, which states that the short-block bias is negative: for flossing at α=30\alpha=30, short blocks overestimate σ\sigma.

W2. Limitations in the main text. We agree that they should be stated next to the theorem and not only in Section 6. After Theorem 4.1 we will add a sentence stating that the proof requires i.i.d. inputs, du≥dhd_u\ge d_h and dense nonsingular WhW_h, and that the trained networks are covered only empirically. The captions of Figs. 2–3 will say the same, the larger exponent under dependent inputs (R1) will be stated as outside the theorem, and "diffusive" will be used only for m\sqrt m growth.

Q3. Fixed vectors and loss gradients. Theorem 4.1 holds for any vector vv that does not depend on the inputs inside the window [t,t+m][t,t+m]; through its proof, vv may depend on everything before tt. The loss gradient is not of this kind. It is transported backward, J⊤gTJ^\top g_T with gT=∂L/∂hTg_T=\partial\mathcal L/\partial h_T, and gTg_T is computed from hTh_T, hence from the same inputs that make up JJ. Two conditions are therefore needed. First, backward transport must obey the same CLT. It has the same λ1\lambda_1, since JJ and J⊤J^\top have the same singular values, and we expect the same CLT through the time-reversed chain, but have not proved its minorization. Second, the direction of the transported gradient must forget its initial condition. Then log⁡∥J⊤gT∥−log⁡∥J⊤w∥\log\|J^\top g_T\|-\log\|J^\top w\| stays bounded for a fixed ww, and the data-dependent start shifts the mean by a constant without changing the spread.

The check in R2 confirms this on the real task: the loss gradient spreads like a fixed vector to within about 10%, shifted by a constant. We will add this check as an appendix and state the two conditions in the Limitations.

Many thanks for the detailed response. The authors have addressed most of my concerns. Therefore, I am happy to recommend acceptance.

Reviewer_xs1AOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper is a reasonable paper that shows that propagation of log gradient norms through a recurrent network exhibits diffusive growth even when the mean is controlled to avoid vanishing or exploding gradients on average. The proof of this is challenging because RNNs are like feedforward networks with the same weight matrix in each layer, and so correlations across layers need to be taken into account to prove a Gaussian CLT for the log gradient norm. This CLT is empirically validated. They attempt to improve performance with a variance penalty but the results, while positive, are weak - not much performance improvement, though training is faster - but the asymptotic performance gain is not much higher.

Critical strengths

The proofs of the CLT of log gradient norms are nice.

Critical weaknesses

The empirical improvements stemming from the variance penalization while positive are weak. The paper also does not do justice to the literature on gradient propagation and jacobian singular values in feedforward networks. Discussing this literature, where a lot of work has been done, would be helpful in setting a contrast to the recurrent case. For example: https://arxiv.org/abs/1802.09979 computes an analytic formula for the entire Lyapunov (singular value) spectrum of the Jacobian in the feedforward case. Probably worth citing this and related papers in the feedforward case.

Questions for authors

Why is the absolute level of performance on MNIST and CIFAR10 so low compared to state of the art? is it because you are using a vanilla RNN?

2 comments

Size of the gains. The absolute numbers are far from the state of the art, and the paper does not claim otherwise. The relevant comparison is within a fixed model: the penalty changes only the training objective of a fixed vanilla tanh RNN with 18k parameters. On psMNIST, test accuracy rises from 0.8110.811 to 0.8600.860, which removes a quarter of the errors. With the same training protocol, a GRU of the same width reaches 0.8430.843 with 2.9×2.9\times the parameters, and an LSTM 0.8750.875 with 3.8×3.8\times. Against flossing at matched drift, the penalty adds 3.43.4 points (Welch t=3.4t=3.4). On the adding problem at T=300T=300, it halves the MSE and solves 8/88/8 seeds against 5/85/8. These experiments test the prediction that, at fixed drift, a lower σ\sigma lengthens the usable horizon, and the prediction holds in every setting where the drift is already centered. We will add the GRU and LSTM accuracies next to Table 1.

Feedforward literature. Thank you for the pointer. We looked at the paper the reviewer mentions (Pennington, Schoenholz & Ganguli, 2018) and at the related feedforward work, and at how they relate to the recurrent case, and we will extend the discussion of the related work accordingly. The result we find closest to ours is Hanin & Nica (2020), who show that in deep ReLU networks log⁡∥Jx∥\log\|Jx\| is approximately Gaussian with variance proportional to depth/width. As the reviewer notes in the summary, the recurrent case differs in that the same weight matrix appears in every factor and the factors are coupled through the hidden state; our CLT treats this dependence through a Markov chain.

In the recurrent case, the analogous scaling would make σ\sigma vanish at large width. We therefore measured σ\sigma as a function of the width in untrained tanh RNNs (R4). With independent input to every unit, σ2dh\sigma^2d_h is constant, so that Varlog⁡∥J0→mv∥≈c m/dh\mathrm{Var}\log\|J_{0\to m}v\|\approx c\,m/d_h, and horizon/width plays the role of depth/width. With a scalar input, σ\sigma barely decreases with width, which we attribute to all units being driven by the same input value. We will add this measurement as an appendix.

Q. Absolute accuracy. Yes. The model is a vanilla tanh RNN with dh=128d_h=128 (18k parameters), chosen to isolate gradient transport. State-of-the-art results (above 98%) are obtained by structured state-space models with many more parameters.

Thank you. I maintain my score of clear acceptance.

Reviewer_pkvxOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper revisits the problem of vanishing and exploding gradients in recurrent neural networks, focusing on the input-driven setting. Existing approaches such as gradient flossing penalize the squared magnitudes of the leading Lyapunov exponents to control the average rate of gradient growth or decay. The main argument of this paper is that controlling this drift alone does not ensure reliable gradient propagation over arbitrarily long time horizons.

The authors first establish that, under their assumptions, the Lyapunov spectrum is deterministic and depends on the network parameters and input distribution rather than the particular input realization. They then prove a central limit theorem for the log-magnitude of directional Jacobian transport over time. This identifies a diffusion term in addition to the drift: even when the top Lyapunov exponent is zero, fluctuations in the log-magnitude grow on the order of m\sqrt{m}. For vanilla RNNs satisfying the theorem’s assumptions, the authors further prove that the asymptotic variance is strictly positive.

The authors use these results to derive a leading-order estimate of the usable gradient horizon, m⋆m^\star, and propose adding a variance penalty to gradient flossing. The experiments on psMNIST, the adding problem, and noise-padded CIFAR-10 show that this additional penalty can improve training when diffusion limits gradient transport. They also examine several recurrent architectures outside the scope of the theorem, providing empirical evidence that the drift–diffusion description is useful more broadly.

Critical strengths
  • I found the distinction between the Lyapunov drift and the fluctuations around it particularly useful. The result that the asymptotic variance remains strictly positive under the stated assumptions explains why setting the top Lyapunov exponent to zero is not enough. The usable-horizon estimate also helps connect this result to the finite time horizons relevant for learning.

  • The authors identify a quantity that is missing from drift-based regularization and give a practical way to estimate and penalize it using the QR computation already used by gradient flossing. I also appreciated the comparison at nearly matched drift, which helps show that the improvement is not explained only by moving the top Lyapunov exponent closer to zero.

Critical weaknesses
  • The experiments on psMNIST, the adding problem, and noise-padded CIFAR-10 are useful for isolating issues with gradient transport. However, I would have liked to see the method tested on a broader range of sequence-learning applications. This would help establish how useful the additional regularization is beyond these controlled benchmarks. I still think the theoretical contribution is substantial enough to make the paper an important contribution.

  • In particular, the CLT assumes i.i.d. inputs. These conditions are restrictive for many practical sequence tasks. The authors acknowledge this and provide experiments outside the assumptions, which I appreciate. However, the change in the observed scaling exponent under correlated psMNIST inputs leaves an open question about how the theoretical description should be modified in that setting.

Questions for authors

Ouestions

  • Can the analysis say more about inputs with temporal correlations? Figure 2 suggests that dispersion persists under psMNIST inputs, but the observed exponent changes from approximately 1/21/2 to 0.660.66. Appendix E.2 also acknowledges this difference. Under what assumptions on temporal dependence would the CLT and m\sqrt{m} scaling continue to hold, and when should we expect a different scaling law? Even a discussion of a restricted case, such as inputs satisfying suitable assumptions, would be useful.

  • In Section 5.3, the authors attribute the drop in accuracy at large variance penalties to a restriction of network expressivity. Could the authors explain this connection more precisely? The penalty measures fluctuations in stretching along the leading transported direction, so I would like to understand which task-relevant dynamics are restricted when these fluctuations are suppressed. Could the observed drop also reflect an optimization effect or a change in the drift? The experiments clearly show that making the penalty too strong can hurt performance, but the reason for attributing this specifically to expressivity could be explained more clearly.

Presentation suggestions

  • Please define “non-autonomous” in plain language when it first appears, for example by explaining that the recurrent dynamics are driven by a time-varying external input. This distinction is central to the paper and would help readers who are less familiar with the terminology.

  • Around line 171, it would be helpful to state explicitly that the matrix spectral 2-norm equals the largest singular value, ∣J∣2=σ1(J)|J|_2=\sigma_1(J).

  • Although the distinction is already stated, I would suggest using a different symbol for the diffusion scale in Theorem 4.1, such as σdiff\sigma_{\mathrm{diff}}, to avoid confusion with the singular values σi\sigma_i. This would also make it clearer that the diffusion scale is a statistic of the trajectory-dependent stretching process, rather than a singular value of the recurrent matrix.

2 comments

Thank you for the precise questions.

Broader applications. We agree that the experiments do not establish how useful the penalty is on general sequence tasks, and we will say so in the Limitations. We chose psMNIST, the adding problem and noise-padded CIFAR-10 because the horizon each task requires is known or controllable, so that a change in accuracy can be attributed to (λ1,σ)(\lambda_1,\sigma). The measurement of (λ1,σ,m∗)(\lambda_1,\sigma,m^\ast) applies more broadly, since it needs only a recurrent state and the QR recursion; we report it for seven cells.

Q1. Correlated inputs. The proof uses i.i.d. inputs only to make (ht,[vt])(h_t,[v_t]) a Markov chain with a minorization. If the inputs are a geometrically ergodic Markov chain ξt\xi_t, then (ξt,ht,[vt])(\xi_t,h_t,[v_t]) is again a Markov chain, and Meyn & Tweedie (Thm. 17.0.1) gives σm\sigma\sqrt m growth provided the enlarged chain satisfies a minorization, which we have not proved. This is the natural extension of the theory in this work, and we will state it as such in the paper. When the dependence does not decay fast enough (long-range dependence, or a latent variable fixed for the whole sequence), we expect a larger exponent.

The measurements are consistent with this (R1). With shuffled text the spread follows σm\sigma\sqrt m, and with natural text the exponent is about 0.70.7, close to the 0.660.66 of psMNIST. The drift is unchanged, and at long horizons the fluctuations exceed those under i.i.d. inputs (2.52.5 times at m=16,000m=16{,}000), so the two-term description remains with a different exponent. We will add this discussion to the Limitations, where we will also state that the exponent 1/21/2 in m∗m^\ast relies on the assumptions of the theorem.

Q2. Accuracy at large β\beta. We agree: the runs do not show that the drop comes from a loss of expressivity, and we will remove that attribution from the paper. The drop is not overfitting: from β=30\beta=30 to 10001000, training accuracy falls from 0.880.88 to 0.810.81 and the train–validation gap shrinks (0.019→0.0080.019\to0.008). It is also not caused by the drift: ∣λ1∣|\lambda_1| at β=1000\beta=1000 (≈0.005\approx0.005) is no larger than at β=30\beta=30 (0.00670.0067), and m∗m^\ast is larger (372372 against 189189). The loss of accuracy is therefore a direct effect of the strong penalty on the fit of the training data. We hypothesize that σ\sigma measures how much the input modulates the stretching of the transported gradient, and that forcing it down pushes the network towards input-independent, LTI-like transport (the LTI SSM has σ=0\sigma=0). The current runs cannot distinguish between a restriction of what the network can represent and slower optimization, so the paper will report the drop without attributing it to either.

Presentation. We will make the three changes: (i) define non-autonomous at first use ("the one-step Jacobians change over time because the input drives them"); (ii) state ∥J∥2=s1(J)\|J\|_2=s_1(J) at line 171; (iii) rename the singular values to sis_i and keep σ\sigma for the diffusion, since σ\sigma is standard for a central limit theorem (CLT) and appears in every figure. If the reviewer prefers σdiff\sigma_{\text{diff}}, we will use it.

Thank you for the response and additional experiments. The discussion of correlated inputs is helpful, and removing the attribution to reduced expressivity addresses my concern.

I also agree with Reviewer Qgac that the finite-window (α\alpha) regularization already contains an implicit variance penalty. The additional (β\beta) term should therefore be described as allowing variance to be controlled independently of the drift penalty.

Broader applications remain a limitation, but I think that the contribution is valuable. I maintain my recommendation of accept.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Official comment to all reviewers (2/2)

(R3) Block length of the batch-means estimator (Qgac). The penalty estimates σ\sigma by batch means, with blocks of ℓ=11\ell=11 steps during training. Blocks that are too short bias the estimate, so the comparison at matched drift could depend on this choice. We therefore re-measured the trained checkpoints with ℓ\ell from 11 to 3136, on the concatenated psMNIST stream of R1:

σ\sigmaℓ=11\ell=11321287843136
flossing, α=30\alpha=300.0980.0850.0780.0770.075
flossing +βσ2+\beta\sigma^20.0380.0370.0380.0420.043

The estimate for the variance-penalized network barely changes with ℓ\ell, and flossing at matched drift retains 1.71.7–2.6×2.6\times more diffusion at every ℓ\ell (2.12.1–2.7×2.7\times under i.i.d. inputs). The comparison does not depend on the block length.

(R4) Dependence of σ\sigma on width (xs1A). Hanin & Nica (2020) show that, in deep feedforward ReLU networks, the variance of the log-norm of the gradient scales as depth/width, so that it vanishes at large width. We measured σ\sigma against width in untrained tanh RNNs with dh=32d_h=32 to 512512. With independent input to every unit (du=dhd_u=d_h), σ2dh\sigma^2 d_h stays at 1.21.2–1.41.4, so that σ∝1/dh\sigma\propto1/\sqrt{d_h}, the recurrent counterpart of that law. With a scalar input (du=1d_u=1, as in our sequence tasks), σ\sigma decreases only from 0.300.30 to 0.210.21 over the same range. A likely reason is that a scalar input drives all units with the same random value at each step, so that their fluctuations do not average out across units. We have not tested intermediate input dimensions, which would be of interest and which we leave to future work.

(R5) Implicit variance penalty in flossing (Qgac). As Reviewer Qgac shows, flossing already penalizes σ2\sigma^2 through the variance of its finite-window estimate of λ1\lambda_1. The implicit weight is α/(mN)\alpha/(mN) in our implementation (λ^1\hat\lambda_1 averaged over N=32N=32 trajectories of m=128m=128 steps before squaring), or α/m\alpha/m with per-trajectory squaring. For the settings that train (m≥32m\ge32, α≤30\alpha\le30) it is at most about 11. The β\beta sweep of the paper (flossing at α=1\alpha=1 plus βσ2\beta\sigma^2, psMNIST, 3–5 seeds) shows what a weight of each size does:

β\beta00.10.3131030
accuracy0.8300.8370.8360.8410.8520.8540.860
σ\sigma0.120.110.110.090.080.060.04

Weights up to β≈1\beta\approx1 lower σ\sigma modestly (0.12→0.090.12\to0.09) and change accuracy by about one point; the main gains come from β≈3\beta\approx3 to 3030. At m=128m=128, reaching β=3\beta=3 through the implicit term would require α≈12,000\alpha\approx12{,}000 in our implementation (≈380\approx380 with per-trajectory squaring), and β=30\beta=30 would require α≈120,000\alpha\approx120{,}000 (≈3,800\approx3{,}800). Flossing already fails to train at α=100\alpha=100 (Table 1).

Changes to the paper. We post these replies now so that the reviewers can see our answers and the new results, and will upload a revised PDF with the following changes in the next few days. (i) After Theorem 4.1, the conditions of the proof (i.i.d. inputs, du≥dhd_u\ge d_h, dense nonsingular WhW_h), with the captions of Figs. 2–3 aligned. (ii) In Fig. 2b, we will replace the psMNIST curve with the text task and its shuffled control (R1), and use "diffusive" only for m\sqrt m growth. We will also describe the concatenated stream on which (λ1,σ,m∗)(\lambda_1,\sigma,m^\ast) are measured for the networks trained on psMNIST. (iii) R2 and R4 as short appendices. (iv) The decomposition of Qgac in Section 5.2, with the α\alpha and β\beta sweeps in an appendix (R5). (v) A paragraph on the feedforward literature, and GRU and LSTM accuracies next to Table 1. (vi) In the Limitations: dependent inputs and the exponent 1/21/2 of m∗m^\ast, the conditions for loss gradients, and the restriction to controlled benchmarks. (vii) The sentence on expressivity at large β\beta and the sentence on the bias sign in App. D.8 corrected. (viii) The notation changes of pkvx.

AuthorsOpenReview ↗

Official comment to all reviewers (1/2)

We thank the three reviewers for the time they devoted to the paper and for their constructive reviews. We list here five results, four of them new, to which the individual replies refer (R1–R2 in this comment, R3–R5 and the changes to the paper in the next).

(R1) Inputs with temporal dependence (pkvx, Qgac). The psMNIST curve of Fig. 2b was measured on a stream of concatenated, independently drawn images. Such a stream is dependent only within each image, so it is not a clean test of dependent inputs, and the paper did not explain this clearly. We therefore repeated the test on a naturally long stream, driving the same untrained network with character-level text (text8) and, as a control, with the same characters shuffled. With shuffled characters the spread follows σm\sigma\sqrt m with no fitted parameter (observed/predicted 1.021.02–1.061.06, local slopes 0.430.43–0.560.56). With the text in its natural order the spread grows approximately as m0.7m^{0.7} over m∈[250,16000]m\in[250,16000] (local slopes 0.620.62–0.800.80 across three symbol encodings), while λ1\lambda_1 is unchanged (−0.40-0.40 in both cases). This agrees with the exponent 0.660.66 reported for psMNIST: under naturally dependent inputs the fluctuations grow faster than m\sqrt m, and Theorem 4.1 does not cover this case.

(R2) Loss gradients (Qgac). Theorem 4.1 is stated for the transport of a vector that does not depend on the inputs in the transport window. The gradient of the loss differs in two respects: it is transported backward, as J⊤gTJ^\top g_T with gT=∂L/∂hTg_T=\partial\mathcal L/\partial h_T, and gTg_T depends on the same inputs that generate JJ. We therefore compared it with fixed vectors on the task itself, using four trained psMNIST checkpoints and 512 single test images. The table reports the standard deviation across images of the change in log-norm over mm steps, for the loss gradient, a fixed covector ww and a fixed vector vv:

m=100m=100200400700
unregularized1.35/1.31/1.371.96/1.95/1.973.10/3.04/3.154.70/4.68/4.89
flossing, α=30\alpha=300.99/1.04/0.921.32/1.41/1.321.79/1.91/1.822.49/2.55/2.47
flossing +βσ2+\beta\sigma^20.60/0.56/0.530.74/0.70/0.690.94/0.93/0.871.14/1.16/1.09

The three agree to within about 10% at every horizon, and the mean change for the loss gradient differs from that for a fixed ww by a constant of 1.0–1.4 nats, independent of mm. The loss gradient thus fluctuates as the vector of Theorem 4.1 does, and the effect of the penalty carries over to it: at matched drift, its spread over 700 steps is 2.5 nats with flossing and 1.1 nats with βσ2\beta\sigma^2.

AuthorsOpenReview ↗

Revised paper.

We have uploaded a revised version of the paper. It contains the changes described in our previous comment, together with minor corrections in the text, figures, and appendices.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.