- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that zeroing the leading Lyapunov exponent only removes average log growth, not the accumulated fluctuations in log norms. Under stated assumptions the authors prove a CLT with positive asymptotic variance , derive a propagation horizon combining drift and fluctuations, and add a penalty to Gradient Flossing, estimating from batch means of the per-step QR stretches.
Critical strengths
-
The framework is more complete than looking at the exponent alone: gives the average trend, the accumulated fluctuations, and both enter the propagation length. Establishing nondegenerate fluctuations under explicit assumptions is a real contribution.
-
The experiments go beyond a single baseline. Table 1 sweeps for flossing; at flossing reaches a final similar to the proposed method, but the latter gets lower and higher psMNIST accuracy ( vs ). Remeasuring on independent long trajectories helps. This is decent evidence that variance control matters in practice.
Critical weaknesses
- The link to finite-window flossing needs more work, especially how window length affects the comparison. Using Appendix D.8's notation, the single-trajectory window estimate is . Under stationarity, Proposition D.10 gives . With and absolutely summable autocovariances, Equation (33) gives , hence
Questions for authors
-
Implementation and contribution. Is the Lyapunov penalty averaged after squaring each trajectory's exponent, or is the exponent averaged across trajectories before squaring? Do you agree with the single-trajectory second-moment decomposition above, and with reading the added term as independently weighting long-run fluctuations? The aggregation rule changes the implicit variance penalty.
-
What would change my assessment. A small joint comparison of window length and under comparable training and tuning budgets, reporting task performance for or an independently weighted leading exponent. Also, how sensitive is the training-time estimate to block length ? That would separate long-run fluctuation control from finite-window estimation effects.
-
Connection to actual gradients. Given the qualifications in Limitations, Remark D.2, and Appendix E.2: how does the fixed-vector forward-product result apply to backward gradients that depend on data and loss? Please state the required conditions, and align the main text and captions on correlated inputs, other recurrent architectures, and the scaling with the existing scope statements.
2 comments
Many thanks for the detailed response. The authors have addressed most of my concerns. Therefore, I am happy to recommend acceptance.
Q1. Aggregation. The exponent is averaged before squaring: λ^i is the mean over N=32 trajectories and m=128 steps, and the penalty is α∑iλ^i2.
W1. Implicit fluctuation penalty. We thank the reviewer for this derivation. We agree with it and with the reading of the added term, and will add both to the paper: flossing does penalize σ2, and βσ2 makes this weight independently adjustable. In our runs the estimate is averaged before squaring, so E[λ^12]≈λ12+σ2/(mN), an implicit weight of α/4096; with per-trajectory squaring it is α/m. The implicit weight, however, cannot be made large. It grows only with α, which also pushes the drift harder (training fails at α=100, Table 1), or with a shorter window, which no longer measures the asymptotic σ2. It also decreases as 1/(mN), so the more accurately flossing estimates λ1, the less it penalizes σ. Within the range that trains, the weight is at most about 1, where the β sweep shows only a modest effect; the main gains need β≈3–30, which the implicit term would reach only at α in the thousands or above (R5). The α sweep agrees: raising α from 1 to 30 lowers σ from 0.116 to 0.078, and no further (0.090 at α=100), against 0.038 with βσ2 (Table 1).
The reason to penalize σ directly comes from the CLT. σ is determined by the weights, cannot be zero (Prop. 4.2), and together with λ1 determines the usable horizon m∗; even at λ1=0, the transported gradient spreads as e±σm. We will add the decomposition to Section 5.2, crediting the reviewer, with an appendix collecting the α and β sweeps.
Q2. Window, α and block length. Per-trajectory flossing at m∈{32,128}×α∈{1,30} has an implicit weight between 0.008 and 0.94, within the range β≤1 of the sweep in R5. At α=30 the drift weight also changes; flossing at α=30 alone reaches 0.826. We believe this answers the question the grid would test, so we have not trained it. If the reviewer still considers it worthwhile, we will run it and report the results.
For the block length, see R3: the estimate of σ for the variance-penalized network is flat in ℓ, so we do not expect the training block length to matter, although we have not trained with other values of ℓ. These measurements also correct App. D.8, which states that the short-block bias is negative: for flossing at α=30, short blocks overestimate σ.
W2. Limitations in the main text. We agree that they should be stated next to the theorem and not only in Section 6. After Theorem 4.1 we will add a sentence stating that the proof requires i.i.d. inputs, du≥dh and dense nonsingular Wh, and that the trained networks are covered only empirically. The captions of Figs. 2–3 will say the same, the larger exponent under dependent inputs (R1) will be stated as outside the theorem, and "diffusive" will be used only for m growth.
Q3. Fixed vectors and loss gradients. Theorem 4.1 holds for any vector v that does not depend on the inputs inside the window [t,t+m]; through its proof, v may depend on everything before t. The loss gradient is not of this kind. It is transported backward, J⊤gT with gT=∂L/∂hT, and gT is computed from hT, hence from the same inputs that make up J. Two conditions are therefore needed. First, backward transport must obey the same CLT. It has the same λ1, since J and J⊤ have the same singular values, and we expect the same CLT through the time-reversed chain, but have not proved its minorization. Second, the direction of the transported gradient must forget its initial condition. Then log∥J⊤gT∥−log∥J⊤w∥ stays bounded for a fixed w, and the data-dependent start shifts the mean by a constant without changing the spread.
The check in R2 confirms this on the real task: the loss gradient spreads like a fixed vector to within about 10%, shifted by a constant. We will add this check as an appendix and state the two conditions in the Limitations.