acceptodds
Under review as a conference paper at ICLR 2027

Direct State Feedback: Fast RNN Training with Stationary Backward Dynamics

Abstract

Backpropagation through time (BPTT) trains recurrent neural networks by transporting errors through a sequence of time-varying state Jacobians, making temporal gradient propagation inherently sequential. Inspired by feedback alignment, we introduce Direct State Feedback (DSF), an alternative, model-agnostic training method for recurrent networks. DSF relies on a temporal feedback rule that replaces the recurrent Jacobians used during the backward pass with a fixed diagonal approximation, while preserving the original nonlinear forward dynamics. This turns temporal error transport into a stationary linear recurrence that can be efficiently evaluated with parallel scans. We further combine fixed feedback with exact gradients over short horizons, yielding T-DSF, which continuously interpolates between truncated BPTT and full temporal credit assignment. Despite its aggressive backward approximation, DSF retains much of the performance of BPTT across language modeling and memory tasks, while substantially outperforming purely local or truncated updates and reducing training cost at scales up to one billion parameters. Gradient diagnostics further show that DSF updates remain positively aligned with BPTT despite increasing approximation error at long horizons. These results suggest that recurrent networks can be trained with simple, stationary feedback dynamics in place of exact Jacobian transport, extending the feedback-alignment principle from depth to time.

Primary area:
optimization
Keywords:
RNNs, alternative to BPTT, efficient training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Lt7xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Direct State Feedback (DSF), a training method for recurrent networks that replaces the time-varying recurrent Jacobians in the backward pass with a fixed diagonal feedback operator. This makes the backward recurrence stationary and allows it to be computed using a parallel prefix scan. The paper also introduces T-DSF, which combines exact short-horizon BPTT with DSF for longer-range credit assignment.

The experiments cover several recurrent architectures, language modeling, and memory tasks. DSF consistently improves over fully truncated BPTT, while T-DSF offers a useful trade-off between training cost and performance. My main concerns are the extent of the technical novelty, the relevance of the theoretical bound to the observed behavior, and the justification for the fixed feedback operator.

Critical strengths
  1. The method is simple, and the experiments show that it is useful. Replacing the backward recurrence with a stationary diagonal operator gives a clear route to parallel computation. DSF consistently improves over fully truncated BPTT, and T-DSF offers a practical compromise between exact local gradients and cheaper long-range credit assignment.

  2. The evaluation goes beyond a single architecture or task. The paper tests multiple recurrent architectures, sequence lengths, memory tasks, and model scales. I also appreciate the gradient diagnostics in Section 4.7. They help identify what DSF is actually doing and show that its effectiveness cannot simply be attributed to closely reproducing the BPTT gradient.

Critical weaknesses
  1. The technical advance over existing ideas appears limited. Fixed feedback operators have a substantial history in Feedback Alignment, and stationary diagonal recurrences with scan-based computation are established tools in state-space models. Applying these ideas to the temporal backward dynamics of nonlinear RNNs is interesting, but the paper needs to make clearer what new algorithmic insight this combination provides. The empirical usefulness is more convincing to me than the claimed methodological advance.

  2. The theory and the choice of feedback operator do not yet explain the observed results. Proposition 1 bounds the gradient difference under contractivity and bounded Jacobian-deviation assumptions, but the paper does not establish whether these assumptions hold in the trained models. Section 4.7 reports a cosine similarity of only 0.114 between the DSF and BPTT gradients for recurrent parameters at sequence length 4096. This does not contradict the proposition, but it leaves unclear why the bound helps explain successful training at long horizons.

The choice of the feedback operator is also central here. The default spectrum, linearly spaced from 0.1 to 0.9, is selected in one six-layer GRU/OpenWebText setting and then reused across substantially different settings. Since this spectrum controls how quickly the backward signal decays, broader sensitivity evidence is needed to justify the choice. Together, these issues leave the success of the surrogate gradient insufficiently explained.

Questions for authors
  1. Can you report measurements of the recurrent Jacobian norms and their deviations from the fixed feedback operator during training? These measurements would help determine whether the assumptions in Proposition 1 describe the models used in the experiments.

  2. How should Proposition 1 be interpreted in light of the low gradient alignment at long sequence lengths? In particular, what does the bound tell us about training effectiveness when the DSF gradient differs substantially from the BPTT gradient?

  3. How sensitive are the results to the feedback spectrum across architectures and tasks? Does the spectrum selected in the GRU/OpenWebText experiment remain competitive with other choices in the remaining settings, and is there a criterion for choosing it without task-specific tuning?

4 comments
AuthorsOpenReview ↗

Rebuttal [1/3]

We thank the reviewer for the thoughtful review and for recognizing the practical value of the approach. We agree that our contribution should be more clearly separated from existing feedback-alignment methods and parallel recurrence algorithms.

W.1 — Novelty of combining fixed feedback and parallel scans

W.1 "The technical advance over existing ideas appears limited. Fixed feedback operators have a substantial history in Feedback Alignment, and stationary diagonal recurrences with scan-based computation are established tools in state-space models. [...] The paper needs to make clearer what new algorithmic insight this combination provides."

Response. We agree that fixed feedback matrices, diagonal linear recurrences, and parallel scans are well-established tools, and we do not claim novelty for these individual ingredients. Our contribution is the way they are combined to train nonlinear RNNs: DSF keeps the nonlinear forward pass unchanged, and replaces only the true Jacobians in the backward recurrence of the hidden-state gradients with a fixed diagonal matrix,

\quad\longrightarrow\quad \widetilde g_t=e_t+\widetilde g_{t+1}A.$$ This matters computationally: exact BPTT multiplies time-dependent, generally dense Jacobians, whereas DSF only needs powers of a single diagonal matrix, i.e., element-wise products. The backward pass then becomes a linear recurrence that a parallel scan computes with $O(\log T)$ sequential depth. Our implementation uses $O(Td\log T)$ work (Table 1), and work-efficient scans reduce this to $O(Td)$. Unlike state-space models, DSF does not change the architecture: it is a training method for existing nonlinear RNNs, here GRUs, LSTMs, sLSTMs, and vanilla RNNs, and up to 1B parameters for GRUs and sLSTMs. We also acknowledge important predecessors: Han et al. (2020) already use temporal feedback with matrix powers, ModProp (Liu et al., 2022) uses fixed temporal filters with eligibility traces, and RFLO (Murray, 2019) uses fixed leaky eligibility traces. DSF can indeed be written exactly in a forward, eligibility-trace form, $$R_t=AR_{t-1}+P_t, \qquad \sum_tP_t^\top\widetilde g_t = \sum_tR_t^\top e_t,$$ with $P_t$ the local parameter Jacobian, so we do not claim a new class of temporal credit filters. The practical difference is that DSF propagates $d$-dimensional hidden-state gradients backward, instead of storing the potentially large $d\times p$ eligibility traces. T-DSF additionally combines exact BPTT over short horizons with DSF beyond them; this is a practical hybrid, not a new principle. **New finding: alignment in time.** The revision also shows that DSF behaves like Feedback Alignment beyond the backward computation itself: as in Feedback Alignment, the network adapts to the fixed feedback. When trained with DSF, the true Jacobians become aligned with $A$ (new Section 4.7, Appendix D). The correlation between their diagonals and $A$ reaches 0.89, 0.79, and 0.60 at 75M, 166M, and 1B parameters, against roughly 0 for BPTT and FT-BPTT, and it appears early in training. We will revise the presentation to state this contribution clearly, rather than suggesting novelty for the individual ingredients.

W.2, Q.1, Q.2 — Proposition 1, Jacobian assumptions, and low gradient alignment

W.2 "The theory and the choice of feedback operator do not yet explain the observed results. Proposition 1 bounds the gradient difference under contractivity and bounded Jacobian-deviation assumptions, but the paper does not establish whether these assumptions hold in the trained models. Section 4.7 reports a cosine similarity of only 0.114 between the DSF and BPTT gradients for recurrent parameters at sequence length 4096."

Q.1 "Can you report measurements of the recurrent Jacobian norms and their deviations from the fixed feedback operator during training?"

Q.2 "How should Proposition 1 be interpreted in light of the low gradient alignment at long sequence lengths? [...] What does the bound tell us about training effectiveness when the DSF gradient differs substantially from the BPTT gradient?"

Response. We agree that Proposition 1 alone does not explain the observed training behavior, and that its assumptions needed to be checked. We revised the proposition and measured all its assumptions on the trained models.

Revised Proposition 1. The original assumption of contractive Jacobians does not hold in practice (see below). The revised Proposition 1 (Section 3.3) only requires the products of the true Jacobians, Pt,t′=Jt′⋯Jt+1P_{t,t'}=J_{t'}\cdots J_{t+1}, to be summable: ∑t′≥t∥Pt,t′∥≤κ\sum_{t'\ge t}\|P_{t,t'}\|\le\kappa, which allows ∥Jt∥>1\|J_t\|>1. With ∥Jt−A∥≤ε\|J_t-A\|\le\varepsilon, ∥A∥<1\|A\|<1, and ∥et∥≤E\|e_t\|\le E, it gives, for any sequence length TT,

\leq \frac{\varepsilon E\kappa}{1-\|A\|}.$$ **Measurements (Q.1).** The new Appendix C reports the Jacobian norms, their deviation from $A$, and the summability constant $\kappa$ on the final 75M, 166M, and 1B models, and along training for the 75M model (Tables 13–14, Figures 8–9): | Rule | $\max_t\|J_t\|$ (75M / 166M / 1B) | Contractive steps | $\kappa$ | Deviation $\|J_t-A\|$ | Deviation along the DSF signal | |---|---|---|---|---|---| | BPTT | 4.82 / 3.04 / 1.68 | 0% / 0% / 12% | 269 / 279 / 263 | 2.44 / 1.67 / 1.12 | 0.55 / 0.49 / 0.44 | | DSF | 3.46 / 3.91 / 1.69 | 0% / 0% / 10% | 304 / 399 / 215 | 2.06 / 1.71 / 0.90 | **0.32 / 0.26 / 0.25** | | FT-BPTT | 3.38 / 2.41 / 1.28 | 0% / 0% / 41% | 71 / 98 / 129 | 2.16 / 1.26 / 1.02 | 0.57 / 0.45 / 0.44 | All steps are contractive at initialization, but almost none are after a few percent of training, which is why we replaced this assumption. The constant $\kappa$, in contrast, stays finite, and on 8192-token streams the partial sums saturate before the training length (they increase by at most 2.1% from 1028 to 7295 steps). These are measurements on finite samples, not a proof that the assumption holds for all trajectories. The deviation $\|J_t-A\|$ is not small, mostly because of the off-diagonal part of the Jacobians. However, the proof only uses the deviation along the DSF signal, $\|\widetilde g_t(J_t-A)\|/\|\widetilde g_t\|$, which is 40–47% lower for DSF than for BPTT and FT-BPTT. This is explained by a new finding: training with DSF makes the Jacobians align with $A$ (Section 4.7). **Interpretation of the low gradient cosine (Q.2).** The cosine of 0.114 (now Section 4.8) was measured on a single recurrent layer at initialization, with synthetic inputs and five seeds, not on trained models; the revised Section 4.8 and Appendix A.10 now state this protocol explicitly. At initialization, the Jacobians are not yet aligned with $A$, so this setting is closer to a worst case. Moreover, the cosine stays positive and 85–89% of the coordinates have the same sign as the BPTT gradient at all lengths, so the DSF update remains a descent direction to first order, even when its angle to the BPTT gradient is large. We agree that the proposition bounds the error on the hidden-state gradients, not on the parameter gradients, and does not by itself guarantee descent or convergence; we renamed it accordingly ("Uniform State-Adjoint Approximation Error").

W.2, Q.3 — Spectrum selection and generalization across architectures

W.2 "The default spectrum, linearly spaced from 0.1 to 0.9, is selected in one six-layer GRU/OpenWebText setting and then reused across substantially different settings. Since this spectrum controls how quickly the backward signal decays, broader sensitivity evidence is needed to justify the choice."

Q.3 "How sensitive are the results to the feedback spectrum across architectures and tasks? Does the spectrum selected in the GRU/OpenWebText experiment remain competitive with other choices in the remaining settings, and is there a criterion for choosing it without task-specific tuning?"

Response. We agree that the choice of the spectrum is important, as it sets how far back in time errors are propagated: an error kk steps later is weighted by aika_i^k. The default A=diag⁡(linspace⁡(0.1,0.9,d))A=\operatorname{diag}(\operatorname{linspace}(0.1,0.9,d)) was selected on the six-layer GRU on OpenWebText (Appendix E, Table 16), where it was the best of the tested spectra. We then kept it unchanged for all architectures (GRU, LSTM, sLSTM, vanilla RNN) and scales (up to the 1B multi-head GRU, where each head uses diag⁡(linspace⁡(0.1,0.9,128))\operatorname{diag}(\operatorname{linspace}(0.1,0.9,128))), and it gave consistent results without any tuning. We did not run a full spectrum sweep for every architecture, however.

Sensitivity across tasks (Q.3). To study sensitivity on a very different task, we added a spectrum ablation on the copy task with delays up to 8192 (new Appendix E.1, Table 17, five seeds), next to the language-modeling ablation of Table 16:

Copy task: choice of AAExact match
BPTT (reference)99.78 ± 0.05
linspace⁡(0.1,0.9)\operatorname{linspace}(0.1,0.9) (default)43.5 ± 6.3
linspace⁡(0.1,0.99)\operatorname{linspace}(0.1,0.99)79.6 ± 6.2
linspace⁡(0.1,0.999)\operatorname{linspace}(0.1,0.999)81.2 ± 3.3
Log-spaced time scales, ai∈[0.1,0.999]a_i\in[0.1,0.999]99.18 ± 0.20
Language modeling (6-layer GRU): choice of AAValidation PPL
BPTT (reference)40.78
0.5I0.5I49.14
linspace⁡(0.2,0.8)\operatorname{linspace}(0.2,0.8)46.22
linspace⁡(0.1,0.9)\operatorname{linspace}(0.1,0.9) (default)44.60
linspace⁡(0,ρ)\operatorname{linspace}(0,\rho)45.59
logspace⁡(0.1,ρ)\operatorname{logspace}(0.1,\rho)58.59
linspace⁡(0.9,ρ)\operatorname{linspace}(0.9,\rho)201.58

Here ρ=1−10−9\rho=1-10^{-9}.

The two tasks prefer opposite spectra: long time scales help a lot on the copy task, where log-spaced time scales match BPTT, but entries close to one hurt language modeling, and even caused training to diverge in the original setup of our 1B experiment.

A criterion for choosing AA. Proposition 1 explains these opposite trends: the DSF signal satisfies ∥g~t∥≤E/(1−∥A∥)\|\widetilde g_t\|\le E/(1-\|A\|), which becomes large when ∥A∥\|A\| approaches one and errors arrive at every time step, as in language modeling. In the copy task, errors are sparse, so the DSF signal stays bounded and long time scales can carry credit across long delays. This suggests a simple rule without task-specific tuning: a contractive spectrum spread over short and medium time scales, such as our default, for densely supervised tasks, and longer, log-spaced time scales for sparsely supervised tasks with long delays. We do not claim this rule is optimal for every architecture.

Thank you for the clarifications and additional experiments. These responses largely address my concerns regarding the theoretical assumptions and the selection of the feedback operator AA. I also appreciate the clearer positioning of T-DSF as a practical hybrid and the added Jacobian alignment experiments supporting its connection to Feedback Alignment. However, my main concern about methodological novelty remains: BPTT and FT-BPTT do not use the fixed feedback operator AA, and their lack of alignment with AA is therefore largely expected, so this comparison provides only limited evidence about the role of the alignment mechanism. Overall, the rebuttal substantially strengthens the paper, although I still have some reservations about the methodological contribution. I therefore appreciate the progress but will keep my current score.

Reviewer_9xWvOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Direct State Feedback (DSF): an alternative, model-agnostic training method for recurrent networks. Inspired by the "feedback alignment" paradigm, it relies on a temporal feedback rule that replaces the recurrent Jacobians used during the backward pass with a fixed diagonal approximation. This enables parallel temporal error propagation. In addition, the paper proposes a combination of fixed feedback with exact gradients over short horizons (T-DSF), thereby using exact Jacobians within short segments, initialized with approximate gradients computed using DSF. Experiments show better predictive performance than fully truncated training, often approaching BPTT at lower computational cost; theory bounds the backward-recursion approximation error under contractivity and bounded-deviation assumptions.

Critical strengths

The method is clearly formulated and easy to understand. It preserves the nonlinear forward dynamics while enabling efficient parallel temporal error propagation. The experiments spans several recurrent architectures, tasks, and model scales exceeding one billion parameters. DSF improves substantially over fully truncated training at similar computational cost, while T-DSF improves over truncated BPTT at matched segment lengths with little additional overhead.

Critical weaknesses
  1. The novelty of the contribution is hard to assess, as the paper has not clearly isolated what is new beyond existing recurrent feedback-alignment methods and stationary approximations to temporal credit assignment. Relevant works include, e.g.:

  2. Proposition 1 provides limited explanatory value under restrictive assumptions. More specifically, it requires both the exact Jacobians and the feedback matrix AA to be uniformly contractive, together with bounded local errors and Jacobian deviation. Choosing a contractive AA establishes only one part; however, the paper does not establish uniform contractivity of the exact Jacobians or uniform boundedness of local errors. Moreover, the bound concerns hidden-state adjoints, not gradient alignment or optimization convergence.

  3. The choice of AA lacks a principled selection method (besides contractiveness). The authors do not provide sufficient insight into why their specific choice should be effective across tasks beyond those considered in the experiments. Moreover, choosing ∣A∣<1\vert A \vert < 1 may introduce a strong bias toward recent timesteps. The implications for learning long-range dependencies deserve further discussion.

Questions for authors
  • How restrictive is uniform contractiveness of the exact state Jacobians (condition (ii) of Proposition 1), and does it holds in the evaluated networks?

  • Could choosing ∣A∣<1|A|<1 adversely affect training when long-range dependencies are crucial (similar to the effect of vanishing gradients)? How should the feedback spectrum be selected for tasks requiring longer temporal credit assignment?

3 comments
AuthorsOpenReview ↗

Rebuttal [1/3]

We thank the reviewer for the constructive review, and for pointing out that our contribution should be more clearly separated from previous feedback-alignment methods for RNNs and from other fixed approximations of temporal credit assignment.

W.1 — Novelty relative to Folchini, Han, and ModProp

W.1 "The novelty of the contribution is hard to assess, as the paper has not clearly isolated what is new beyond existing recurrent feedback-alignment methods and stationary approximations to temporal credit assignment."

Relevant works: Folchini et al. (2025/2026), Han et al. (2020), and Liu et al. (2022, ModProp).

Response. We agree that fixed feedback and fixed temporal credit assignment existed before DSF, and that our related-work section does not discuss these works enough. However, the cited methods do not use the same learning rule as DSF.

Folchini et al. (2025/2026). Their method (Eqs. 4–5) sends the output error directly to each time step through a fixed feedback matrix BB. The recurrent parameters are then updated with local derivatives, but errors are never propagated from one hidden state to the previous one. DSF, in contrast, keeps a backward recurrence between hidden states, with AA in place of the true Jacobians:

\qquad\text{i.e.,}\qquad \widetilde g_t=\sum_{s=t}^T e_sA^{s-t}.$$ With a loss at the last time step only, the error reaching time step $t$ is multiplied by $A^{T-t}$, which is different at each time step. A single fixed projection shared by all time steps cannot reproduce this in general. A different projection at each time step could, so the difference lies in the methods as specified, not in what such a family of methods can represent in principle. **Han et al. (2020).** Han et al. already use temporal feedback with powers of the backward weights, through an upper-triangular block matrix (Eq. 9), so we do not claim novelty for using matrix powers in temporal feedback. With well-chosen blocks, this matrix could even reproduce the DSF kernel $A^{s-t}$. Their final method, however, uses sparse, sign-binarized blocks (Eqs. 11–12), and does not use the same update of the recurrent parameters. DSF instead obtains the whole kernel from a single diagonal matrix, and computes it with a parallel scan, without building a dense temporal feedback matrix. **Liu et al. (2022, ModProp).** ModProp is an important precedent for fixed long-range temporal credit assignment. It uses local eligibility traces and neuromodulatory filters with coefficients $\mu^{k-1}W^k$ (Eqs. 4–6), which differ in general from the DSF kernel $A^k$: matching the first lag requires $A=W$, and matching the second requires $(\mu-1)W^2=0$, i.e., $\mu=1$ or $W^2=0$. Even then, the full parameter updates would only coincide with matching local derivatives and learning signals. ModProp also carries credit through eligibility and neuromodulatory signals, whereas DSF directly replaces the Jacobians in the backward recurrence. **Link with eligibility traces.** DSF can also be written exactly in a forward, eligibility-trace form. With $P_t$ the local parameter Jacobian and: $$R_t=AR_{t-1}+P_t, \qquad \sum_tP_t^\top\widetilde g_t = \sum_tR_t^\top e_t.$$ DSF is therefore mathematically related to eligibility-based learning, and with $A=\alpha I$ it matches a restricted form of RFLO. We do not claim to introduce a new class of temporal credit filters. **Our contribution.** DSF replaces the true Jacobians with a fixed diagonal matrix in the backward pass only: the forward pass and the local parameter derivatives are unchanged. Because the matrix is diagonal, the backward pass becomes a simple linear recurrence that can be computed with a parallel scan, and we show that this works across RNN architectures and up to 1B parameters. We also find that, as in Feedback Alignment, the network adapts to the fixed feedback: when trained with DSF, the true Jacobians become aligned with $A$ (new Section 4.7). The correlation between their diagonals and $A$ reaches 0.89, 0.79, and 0.60 at 75M, 166M, and 1B parameters, against roughly 0 for BPTT and FT-BPTT, and it appears early in training. To our knowledge, this has not been reported for the methods cited above. We will revise the related-work discussion to explicitly acknowledge these antecedents and narrow the novelty claims accordingly.

W.2, Q.1 — Restrictive assumptions of Proposition 1

W.2 "Proposition 1 provides limited explanatory value under restrictive assumptions. [...] Choosing a contractive A establishes only one part; however, the paper does not establish uniform contractivity of the exact Jacobians or uniform boundedness of local errors. Moreover, the bound concerns hidden-state adjoints, not gradient alignment or optimization convergence."

Q.1 "How restrictive is uniform contractiveness of the exact state Jacobians (condition (ii) of Proposition 1), and does it holds in the evaluated networks?"

Response. We agree with the reviewer: the original assumption of uniformly contractive Jacobians was too restrictive, and it does not hold in our trained models. We have therefore revised Proposition 1 (Section 3.3) and added an empirical check of all its assumptions (new Appendix C).

Revised Proposition 1. Let Pt,t′=Jt′Jt′−1⋯Jt+1P_{t,t'}=J_{t'}J_{t'-1}\cdots J_{t+1} be the product of the true Jacobians between tt and t′t'. We now assume (i) ∥Jt−A∥≤ε\|J_t-A\|\le\varepsilon, (ii) ∥A∥<1\|A\|<1 and ∑t′=tT−1∥Pt,t′∥≤κ\sum_{t'=t}^{T-1}\|P_{t,t'}\|\le\kappa, and (iii) ∥et∥≤E\|e_t\|\le E. Then, for any tt and any sequence length TT,

\leq \frac{\varepsilon E\kappa}{1-\|A\|}.$$ Assumption (ii) no longer requires every Jacobian to be contractive: it only requires the products of the true Jacobians to be summable, and it allows $\|J_t\|>1$ at some or all steps. It is implied by one-step contractivity ($\|J_t\|\le M<1$ gives $\kappa\le 1/(1-M)$), so the new result includes the original one as a special case. The proof is also shorter (Appendix B.1). **Does it hold in the evaluated networks? (Q.1)** Appendix C measures these quantities on the final 75M, 166M, and 1B models, and along training for the 75M model (Tables 13–14, Figures 8–9): | Rule | $\max_t\|J_t\|$ (75M / 166M / 1B) | Contractive steps (75M / 166M / 1B) | $\kappa$ (75M / 166M / 1B) | |---|---|---|---| | BPTT | 4.82 / 3.04 / 1.68 | 0% / 0% / 12% | 269 / 279 / 263 | | DSF | 3.46 / 3.91 / 1.69 | 0% / 0% / 10% | 304 / 399 / 215 | | FT-BPTT | 3.38 / 2.41 / 1.28 | 0% / 0% / 41% | 71 / 98 / 129 | One-step contractivity thus fails: all steps are contractive at initialization, but almost none are after a few percent of training. The summability constant $\kappa$, in contrast, remains finite in all models. On validation streams of 8192 tokens, eight times the training length of the 75M models, the partial sums $\kappa(n)$ saturate before the training length: | $n$ | 125 | 1028 | 7295 | |---|---|---|---| | BPTT | 110.5 | 236.6 | 241.7 | | DSF | 146.4 | 266.1 | 267.4 | | FT-BPTT | 70.2 | 74.2 | 74.2 | These are finite-sample measurements on the evaluated trajectories and horizons: they support assumption (ii) in practice but cannot prove uniform summability over arbitrary trajectories and sequence lengths. **Deviation from $A$.** The operator-norm deviation $\varepsilon$ is not small (median 2.06, 1.71, and 0.90 for DSF at 75M, 166M, and 1B), mostly because of the off-diagonal part of the Jacobians. However, the proof only uses the deviation along the DSF signal, $\|\widetilde g_t(J_t-A)\|/\|\widetilde g_t\|$, which can replace $\varepsilon$ in the bound. This quantity is 40–47% lower for DSF (0.32, 0.26, 0.25) than for BPTT (0.55, 0.49, 0.44) and FT-BPTT, which we relate to a new finding: training with DSF makes the true Jacobians align with $A$ (Section 4.7, diagonal correlation 0.89, 0.79, and 0.60, versus ≈0 for BPTT and FT-BPTT). **Scope.** We agree that the bound is on the gradients with respect to the hidden states, not on the parameter gradients, and we renamed the proposition accordingly ("Uniform State-Adjoint Approximation Error"). It is a worst-case bound that holds under its assumptions, and it does not by itself guarantee aligned parameter gradients or convergence.

W.3, Q.2 — Feedback spectrum and long-range credit assignment

W.3 "The choice of A lacks a principled selection method (besides contractiveness). [...] Choosing ∥A∥<1\|A\|<1 may introduce a strong bias toward recent timesteps."

Q.2 "Could choosing ∥A∥<1\|A\|<1 adversely affect training when long-range dependencies are crucial (similar to the effect of vanishing gradients)? How should the feedback spectrum be selected for tasks requiring longer temporal credit assignment?"

Response. We agree that the spectrum of AA is a central design choice, as it sets how far back in time errors are propagated. For diagonal A=diag⁡(a1,…,ad)A=\operatorname{diag}(a_1,\ldots,a_d), the contribution of an error occurring kk time steps later is weighted, coordinate-wise, by aika_i^k, i.e., it decays with time scale τi=−1/log⁡ai\tau_i=-1/\log a_i. Our default schedule A=diag⁡(linspace⁡(0.1,0.9,d))A=\operatorname{diag}(\operatorname{linspace}(0.1,0.9,d)) therefore covers a range of decay time scales, but an isolated error is strongly attenuated after many steps: the contribution of an error kk steps later is bounded by 0.9k0.9^k. We agree that this induces a recency bias that can limit direct long-range credit assignment.

Does ∥A∥<1\|A\|<1 hurt long-range tasks? (Q.2) Yes, it can, and we now quantify it on the copy task with delays up to 8192 (new Appendix E.1, Table 17, five seeds):

Choice of AAToken accuracyExact match
BPTT (reference)99.97 ± 0.0199.78 ± 0.05
linspace⁡(0.1,0.9)\operatorname{linspace}(0.1,0.9) (default)88.5 ± 1.843.5 ± 6.3
linspace⁡(0.1,0.99)\operatorname{linspace}(0.1,0.99)96.8 ± 1.179.6 ± 6.2
linspace⁡(0.1,0.999)\operatorname{linspace}(0.1,0.999)97.0 ± 0.681.2 ± 3.3
linspace⁡(0,1)\operatorname{linspace}(0,1)96.7 ± 1.079.5 ± 5.6
Log-spaced time scales, ai∈[0.1,0.999]a_i\in[0.1,0.999]99.88 ± 0.0399.18 ± 0.20

Longer time scales help considerably on this task, and log-spaced time scales (ai=1−1/τia_i=1-1/\tau_i with τi\tau_i log-uniform in [1.11,1000][1.11,1000]) match BPTT. On language modeling, the opposite holds (Appendix E, Table 16, six-layer GRU on OpenWebText): linspace⁡(0.1,0.9)\operatorname{linspace}(0.1,0.9) reaches 44.60 PPL, versus 46.22 for linspace⁡(0.2,0.8)\operatorname{linspace}(0.2,0.8), 49.14 for 0.5I0.5I, and 201.58 for linspace⁡(0.9,ρ)\operatorname{linspace}(0.9,\rho) with ρ\rho close to one. In the original setup of our 1B experiment, a feedback operator with entries too close to one even caused training to diverge.

How to select the spectrum. Proposition 1 explains these opposite trends: the DSF signal satisfies ∥g~t∥≤E/(1−∥A∥)\|\widetilde g_t\|\le E/(1-\|A\|), which becomes large when ∥A∥\|A\| approaches one and errors are injected at every time step, as in language modeling. In the copy task, the errors are sparse, so the DSF signal remains bounded even with ∥A∥\|A\| close to one, and long time scales can carry credit across long delays. This gives a practical guideline: a contractive spectrum spread over short and medium time scales, such as our default, for densely supervised tasks, and longer, log-spaced time scales for sparsely supervised tasks with long delays. We do not claim a universal selection rule, and we keep the default AA in the main text for consistency across tasks.

Reviewer_LdphOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Nonlinear RNNs (such as GRUs, LSTMs, xLSTM, sLSTM) offer efficient inference but are hard to optimize due to back-propagation through time (BPTT) being sequential in T resulting in exploding or vanishing gradients and incorrect credit assignment without careful architectural tuneup. Literature has attacked this problem with many different routes: truncated BPTT or auxiliary / synthetic rewards, online RTRL approximation methods, parallel approximations like ParaRNN, DEER, or linear recurrences like Mamba, State-space models, LRU, etc.

This work attacks this BPTT problem from model agnostic route, where nonlinear RNN forward pass is kept intact, while the backward in-time transport is replaced by making it stationary using feedback alignment. It probes whether a deliberately wrong but parallelizable temporal backward pass can still train well.

This work proposes Direct State Feedback (DSF) which enables fast RNN training with stationary backward dynamics. Typical backward pass for RNN can be denoted as

gt=et+gt+1At+1g_t = e_t + g_{t+1} A_{t+1} where ete_t denotes the local error at timestep t, At+1A_{t+1} denotes the Jacobians over the hidden state (these vary with time since the forward process is recursive). This equation provides a way to calculate gradients from the final timestep towards the initial timestep and this computation is sequential in nature.

Feedback Alignment (FA) has explored an exact mechanism of backward propagation in deep neural networks through the layers, instead of computing the Jacobians over the hidden states (which would be sequential in a deep neural network), it replaces the exact computation with an approximation by using a fixed feedback matrix instead of the Jacobians and directly propagating the output reward through this matrix. Thus, speeding up the backward pass substantially since there’s no sequential dependency anymore.

DSF replaces the exact recurrence g_t = e_t + g_{t+1}A_t with g̃_t = e_t + g̃_{t+1}A, where A = diag(linspace(0.1, 0.9, d)) is fixed. The backward pass becomes a linear time-invariant diagonal recurrence, computed with a parallel prefix scan in O(log T) sequential depth. Theoretical analysis shows a uniform bound on ∥gt−g~t∥\| g_t - \tilde{g}_t \| assuming contractivity and bounded Jacobian deviation.

This paper also introduces a variant called T-DSF, which combines truncated BPTT and DSF. T-DSF runs exact BPTT within segments of length K and uses DSF to seed each segment boundary, so it generalizes T-BPTT (A = 0 recovers it).

Empirical evaluation consists of gradient-alignment diagnostics and various RNN cells on WikiText-103 and OpenWebText, with parameters up to 183M in size and one large model at 1B.

DSF consistently sits between FT-BPTT and BPTT, usually much closer to BPTT. For example, on WikiText-103 with context 256 the perplexities are 31.5 (DSF), 28.3 (BPTT) and 46.6 (FT-BPTT). Layer-level runtime roughly matches FT-BPTT and is about 3× faster than BPTT at long sequences.

Critical strengths
  • DSF is model-agnostic, keeps forward pass exact and seems easy to implement with existing scan kernels on backward pass with feedback alignment.
  • T-DSF blends both T-BPTT (exact gradient computation within a truncated timeframe) and approximate feedback alignment through DSF beyond the window. It gives consistent gains at matched K, with large gains at small K (e.g., in Table 5, for WikiText-103, 31.5 vs 46.6 at K = 1, and 30.8 vs 38.9 at K = 2), for only about 2% computational overhead (Table 10).
  • Extensive evaluation on various benchmarks (WikiText-103, OpenWebText, Selective Recall on MQAR and Long-Range memory test on copy task) show its efficacy over T-BPTT.
Critical weaknesses

  • Feedback alignment framing is a bit incorrect in this context. FA tries to align the forward weights with random feedback, this work does not test whether trained Jacobian’s diagonals align with A over training, which would imply connection to FA. This method is conceptually closer to methods like
  • Table~3 result with 1B parameter model trained on OpenWebText points to some issues in BPTT or DSF training.
    • FT-BPTT, at 29.28, is within 0.6 PPL of BPTT at 28.68. In this setup, temporal credit assignment barely matters, so the benchmark doesn’t discriminate between methods.
    • DSF at 22.54 beats both by about 6 PPL, from a single one-epoch run with no seeds. That points to an optimization artifact, such as BPTT gradient pathologies in a 36-layer GRU or tuning favoring one method, rather than a property of DSF.
    • The end-to-end speedup is only about 18% (35.7 h vs 29.3 h), versus the “3×” layer-level claim.
  • Missing some important baselines in comparison
    • Exact parallel training of nonlinear RNNs: ParaRNN (https://arxiv.org/abs/2510.21450) solves the nonlinear recurrences in parallel with Newton iterations and custom parallel reductions, and trains 7B-parameter LSTM/GRU adaptations to perplexity comparable to Transformers and Mamba2.
    • Linear-recurrence models trained exactly: Mamba, GLA, LRU, etc.
    • A Transformer reference point.
  • There are some inconsistencies in the reported results
    • FT-BPTT is 46.52 for both GRU and sLSTM in Table 2, and the same value appears in Table 4 at context length 2048.
    • The 6-layer OpenWebText BPTT baseline is 40.78 in Tables 8, 9 and 12 but 41.42 in Table 5.
    • WikiText BPTT is 28.32 in Table 4 but 28.55 in Table 5.
  • Missing literature connection
Questions for authors
  • In most experiments, DSF always looses to BPTT and sometimes looses badly. Many comparisons show the improvement in runtime compared to BPTT, what happens when DSF is given same training budget as BPTT? For ex. in Sec. 4.2, if DSF had same budget as BPTT (i.e., 140 hours), would you expect it to reach the same performance as BPTT?
  • I’m unable to understand why one would prefer DSF over BPTT, when its leaving performance on the table since BPTT gives you better performance than DSF.
  • For Table.3 (1B experiment), do you have any results for sLSTM? Since even in Table.2, the gap between BPTT and DSF for GRU was somewhat nearby.
  • Can you reconcile the BPTT baseline discrepancies (40.78 vs 41.42; 28.32 vs 28.55) and the repeated 46.52?
  • Which A was used for the copy task and MQAR? With max(a_i) = 0.9, how does DSF propagate useful credit over more than 2000 steps?
  • Similar to FA, do trained Jacobians align with the fixed feedback matrix during training?
  • How does DSF compare against methods like ParaRNN and DEER or quasi-DEER?
  • Do you have any comparison with state-space models? Since they also enable parallel training and have often been compared to transformers on many sequential modeling tasks.
  • Can you report end-to-end wall-clock time and memory footprint for main experiments? It would help in substiating claims on the efficiency.
  • Can you explain on 1B experiment, why is FT-BPTT and BPTT are nearly similar in performance? Did you probe why DSF suddenly outperforms both significantly? Do you have other seeds or learning rate sweep per method?
5 comments
AuthorsOpenReview ↗

Rebuttal [1/5]

We thank the reviewer for the detailed and constructive assessment. We address each concern below, grouping related weaknesses and questions to avoid repetition.

W.1, Q.6 — Feedback Alignment, RFLO, e-prop, and Jacobian alignment

W.1 "Feedback alignment framing is a bit incorrect in this context. FA tries to align the forward weights with random feedback [...] This method is conceptually closer to methods like RFLO [and] Eligibility Propagation."

Q.6 "Similar to FA, do trained Jacobians align with the fixed feedback matrix during training?"

Response. We agree that RFLO (Murray, 2019) and e-prop (Bellec et al., 2020), as well as the more recent Generalising E-prop to Deep Networks (Millidge, 2025), are particularly relevant predecessors and deserve a more prominent discussion.

The relationship is deeper than a conceptual analogy. Using column-vector notation, let Pt=∂ht/∂θP_t=\partial h_t/\partial\theta denote the exact local parameter Jacobian. DSF computes

\qquad \widetilde G=\sum_tP_t^\top\widetilde g_t.$$ Defining stationary eligibility traces by $$R_t=AR_{t-1}+P_t,\qquad R_0=0,$$ a rearrangement of the temporal sums gives the exact identity $$\boxed{ \widetilde G=\sum_tR_t^\top e_t. }$$ Thus, DSF admits an exactly equivalent eligibility-trace formulation. In particular, for $A=\alpha I$ and matched local derivatives and learning signals, its parameter updates coincide with a restricted RFLO-style leaky eligibility rule. The standard RFLO algorithm additionally uses random output-error feedback and is therefore not generally identical to DSF. The practical distinction is representational: DSF transports state adjoints through the backward pass, avoiding explicit storage of the potentially large forward sensitivity tensors. Unlike fully local online rules, DSF also retains conventional spatial differentiation across network layers. **New finding: the Jacobians align with the feedback operator (Q.6).** Our original gradient diagnostics measured agreement with BPTT parameter gradients, not the alignment of the trained recurrent Jacobians with the fixed feedback operator. Following the reviewer's question, we measured the latter directly (new Section 4.7, Table 6 and Figure 6, and Appendix D), and found that **the answer is yes: training with DSF makes the true Jacobians align with $A$**, a temporal analogue of Feedback Alignment. The main metric is the *diagonal correlation*, the Pearson correlation across hidden units between the time-averaged diagonal of the true Jacobians, $\overline{(J_t)_{ii}}$, and the feedback values $a_i$ (chance level 0). Appendix D adds the Frobenius cosine used in the Feedback Alignment literature, compared with its chance level obtained by permuting $A$. | Model | Rule | Diagonal correlation | Frobenius cosine (chance) | Diagonal deviation ↓ | |---|---|---|---|---| | 75M GRU | BPTT | −0.02 | 0.50 (0.50) | 0.47 | | | **DSF** | **0.89** | **0.79** (0.69) | **0.30** | | | FT-BPTT | 0.00 | 0.68 (0.69) | 0.41 | | 166M multi-head GRU | BPTT | 0.01 | 0.76 (0.76) | 0.46 | | | **DSF** | **0.79** | **0.85** (0.77) | **0.35** | | | FT-BPTT | 0.02 | 0.85 (0.85) | 0.48 | | 1B multi-head GRU | BPTT | −0.01 | 0.82 (0.82) | 0.45 | | | **DSF** | **0.60** | **0.89** (0.84) | **0.37** | | | FT-BPTT | −0.01 | 0.88 (0.88) | 0.49 | The alignment is specific to DSF at all scales, while BPTT and FT-BPTT stay at chance level. It emerges early in training: in the 75M model, the diagonal correlation goes from −0.01 at initialization to 0.35 after 300 updates, 0.65 after 1,000 updates, and 0.89 at the end of training. It also connects to our theory: the error bound only depends on the Jacobian mismatch along the DSF signal, $\|\widetilde g_t(J_t-A)\|/\|\widetilde g_t\|$, which is 40–47% lower for DSF than for BPTT and FT-BPTT at all scales (Appendix C). These observations establish emergent alignment, although they do not by themselves prove that it causes successful optimization.

W.2, Q.3, Q.10 — Interpretation of the 1B-parameter experiment

W.2 "FT-BPTT, at 29.28, is within 0.6 PPL of BPTT at 28.68 [...] DSF at 22.54 beats both by about 6 PPL, from a single one-epoch run with no seeds. That points to an optimization artifact [...] rather than a property of DSF."

Q.3 "For Table.3 (1B experiment), do you have any results for sLSTM?"

Q.10 "Can you explain on 1B experiment, why is FT-BPTT and BPTT are nearly similar in performance? [...] Do you have other seeds or learning rate sweep per method?"

Response. We thank the reviewer for this careful analysis, with which we agree: the original 1B result was an artifact, not a property of DSF. The original large models were obtained by naively scaling a dense GRU, and in this setting, BPTT itself trained poorly (28.68 PPL, barely better than FT-BPTT at 29.28).

We have fixed this in the revision. Following the way recurrent networks are successfully scaled in recent architectures such as the sLSTM of xLSTM (Beck et al., 2024), we now use a multi-head GRU, whose recurrent weights are block-diagonal with heads of size 128 (Appendix A.2). With this architecture, the results are coherent at all scales: BPTT is best, DSF stays within a stable gap of BPTT, and FT-BPTT falls far behind (Table 2):

ModelParamsBPTTDSFFT-BPTT
GRU, 12 layers × 6 heads166M29.7734.11 (+15%)49.88 (+68%)
sLSTM, 12 layers × 2 heads183M28.0835.12 (+25%)46.52 (+66%)
GRU, 30 layers × 12 heads1.02B18.7021.95 (+17%)40.18 (+115%)

The relative gap between DSF and BPTT does not widen with scale (+15% at 166M, +17% at 1B), whereas the gap of FT-BPTT grows. We removed the original dense-GRU 1B result and its discussion from the paper.

Learning-rate sweep (Q.10). For the 166M multi-head GRU, we swept the learning rate of each rule (Appendix A.4, Figure 7, Table 8):

Learning rateBPTTDSFFT-BPTT
3×10−43\times10^{-4}29.7734.1149.88
6×10−46\times10^{-4}28.0932.6947.01
10−310^{-3}27.4332.3545.40
2×10−32\times10^{-3}27.1332.3541.90
4×10−34\times10^{-3}27.3433.4343.76
6×10−36\times10^{-3}28.5134.7045.61

The ordering BPTT < DSF < FT-BPTT holds at every learning rate. All three rules share the same best learning rate, 2×10−32\times10^{-3}, and degrade at higher values, so the sweep brackets the optimum. The gap of FT-BPTT is therefore not a tuning artifact: it remains 54% above BPTT at its best learning rate.

Billion-parameter sLSTM (Q.3). We also trained a billion-parameter sLSTM at context length 1024 (Section 4.4, Figure 5). Both runs were trained for about 94 GPU-hours; the values below are those of their last validation checkpoints:

1B sLSTMBPTTDSF
Validation PPL at the last checkpoint22.05 (after 94.14 h)26.59 (after 93.82 h)
Time to reach PPL ≤ 3375.32 h56.30 h

These are single, unfinished runs, not converged results. They show that DSF remains trainable at this scale and reaches intermediate perplexities earlier, while BPTT obtains a better perplexity at the same late-stage budget.

W.2, Q.9 — Kernel-level versus end-to-end efficiency

W.2 "The end-to-end speedup is only about 18% (35.7 h vs 29.3 h), versus the '3×' layer-level claim."

Q.9 "Can you report end-to-end wall-clock time and memory footprint for main experiments?"

Response. We agree that kernel-level and end-to-end speedups must be clearly distinguished. The approximately threefold acceleration in Figure 4 concerns the recurrent-layer computation only, not complete training.

The original billion-parameter experiments did not use our optimized CUDA backward implementation. With the optimized implementation, in the setup of the submission (dense 1B GRU), the measured end-to-end training durations are 35.7 hours for BPTT, 21.6 hours for DSF, and 21.5 hours for FT-BPTT: DSF achieves a 1.65× end-to-end speedup over BPTT, at nearly the same cost as FT-BPTT. On a six-layer OpenWebText GRU, a complete epoch takes 6,920 seconds for BPTT versus 4,221 seconds for optimized DSF, also a 1.64× end-to-end speedup, with final perplexities of 42.10 and 45.48 (Section 4.4, Figure 5).

Note that the new multi-head GRU runs of Table 2 do not use our optimized DSF kernel yet, which was written for the dense GRU. Their training times (about 35 hours on 4 H100 GPUs for all three rules) are therefore not representative of the cost of DSF. We will port the optimized kernel to the multi-head GRU and report the corresponding timings in the final version.

W.3, Q.7, Q.8 — Missing baselines: ParaRNN, DEER, state-space models, and Transformers

W.3 "Missing some important baselines [...] Exact parallel training of nonlinear RNNs: ParaRNN [...] Linear-recurrence models trained exactly: Mamba, GLA, LRU, etc. [...] A Transformer reference point."

Q.7 "How does DSF compare against methods like ParaRNN and DEER or quasi-DEER?"

Q.8 "Do you have any comparison with state-space models? Since they also enable parallel training and have often been compared to transformers on many sequential modeling tasks."

Response. We agree that these methods are important for positioning DSF among efficient sequence-model training approaches. They nevertheless address different computational objects.

DSF retains the original nonlinear recurrent forward dynamics and approximates only the temporal state-adjoint transport during training. In contrast, ParaRNN (Danieli et al., 2026), DEER (Lim et al., 2024), and quasi-DEER methods primarily seek to parallelize the evaluation of nonlinear recurrent dynamics through numerical solution techniques. Similarly, Mamba, GLA, and LRU achieve efficient sequence processing by introducing structured forward recurrences: they change the underlying sequence-model architecture rather than approximating the backward dynamics of an unchanged nonlinear RNN.

These methods are therefore important system-level comparisons, but they are not interchangeable baselines for evaluating the effect of replacing temporal Jacobians with fixed feedback. Our main comparisons with BPTT and truncated BPTT on the same recurrent architectures isolate the consequences of the learning rule while controlling the forward model. We will clarify this distinction in the related-work discussion, and we agree that broader system-level comparisons are necessary before claiming superiority over alternative efficient sequence models.

To provide reference points, we additionally trained a billion-parameter Mamba and a billion-parameter Transformer on OpenWebText, on the same 4 H100 GPUs as our GRU models:

1B modelValidation PPLTraining time
Transformer16.1228.57 h
Mamba21.6824.17 h
Dense GRU + DSF (submission setup, optimized DSF kernel)22.5421.6 h
Multi-head GRU + DSF (revised setup, without the optimized kernel)21.95~36 h

The Transformer remains the strongest model, and Mamba reaches a perplexity comparable to the GRU trained with DSF. In the submission setup, DSF required about 10% less training time than Mamba. The revised multi-head GRU improves the perplexity of DSF, but its timing does not yet benefit from the optimized kernel (see above). Since these models use different architectures, this comparison reflects their practical efficiency–quality trade-off rather than an isolated comparison of training algorithms. We will include these results in the revised manuscript.

W.4, Q.4 — Inconsistencies in reported results

W.4 "FT-BPTT is 46.52 for both GRU and sLSTM in Table 2 [...] The 6-layer OpenWebText BPTT baseline is 40.78 in Tables 8, 9 and 12 but 41.42 in Table 5. WikiText BPTT is 28.32 in Table 4 but 28.55 in Table 5."

Q.4 "Can you reconcile the BPTT baseline discrepancies (40.78 vs 41.42; 28.32 vs 28.55) and the repeated 46.52?"

Response. We thank the reviewer for pointing out these inconsistencies. These were reporting errors in the manuscript, and we apologize for the confusion.

The revised large-scale comparison separates the two architectures: the 166M multi-head GRU obtains 49.88 FT-BPTT PPL, while the 183M sLSTM obtains 46.52 FT-BPTT PPL (Table 2). These results correspond to different experiments; the repeated 46.52 entry in the original GRU/sLSTM comparison was a reporting error.

For the BPTT reference baselines, the correct values are 40.78 for the six-layer GRU on OpenWebText and 28.32 for the three-layer GRU on WikiText-103. We have corrected the affected entries in the revised manuscript: Table 4 (previously Table 5) now reports 28.32 and 40.78, consistent with Tables 3 and 16.

These corrections concern only the reporting of the results and do not change the conclusions of our experiments.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: summary of the revision

We thank the reviewers for their careful and constructive reviews. They found the method simple and the paper clear, and their questions focused on scaling, the theory, the choice of AA, and efficiency. The revised PDF answers these questions and completes the empirical analysis. Revised text is in blue, and NEW / FIX margin notes mark the main additions.


1. Scaling (Table 2, Appendix A.4)

The large-scale models now use a multi-head GRU, following how sLSTM scales its recurrence. The results are consistent from 166M to 1B parameters (validation perplexity on OpenWebText):

ModelBPTTDSFFT-BPTT
GRU, 166M29.7734.1149.88
sLSTM, 183M28.0835.1246.52
GRU, 1.02B18.7021.9540.18

The ranking is the same at every scale, and DSF recovers 62–85% of the gap between FT-BPTT and BPTT.

A learning-rate sweep on the 166M model (Figure 7, Table 8) confirms that the gaps do not come from tuning: at the best learning rate of each method, 27.13 / 32.35 / 41.90.


2. DSF training aligns the Jacobians with AA (Section 4.7)

We observe a striking and nontrivial behavior: although the feedback operator AA is fixed and ignores the true recurrent Jacobians JtJ_t, DSF training makes the JtJ_t align with AA. This is the temporal counterpart of feedback alignment in feedforward networks: the network adapts to its feedback, which brings the DSF signal closer to the true gradient. The effect appears at every scale, emerges early in training, and is absent with BPTT and FT-BPTT:

Diagonal correlation between JtJ_t and AA75M166M1B
BPTT−0.020.01−0.01
DSF0.890.790.60
FT-BPTT0.000.02−0.01

This alignment likely explains why a fixed feedback operator trains deep recurrent networks well (see also point 3).


3. Theory: a weaker, empirically verified assumption (Section 3.3, Appendices B.1 and C)

Proposition 1 now relies on a weaker and more realistic assumption, which we verify on trained models. It no longer requires each recurrent Jacobian JtJ_t to be contractive: it only requires the products of Jacobians over time to be summable, i.e., the influence of a future error on the current state decays on average, with a finite total gain κ\kappa. The distance between the DSF signal and the true gradient is then at most εEκ/(1−∥A∥)\varepsilon E\kappa/(1-\|A\|) for any sequence length, where ε\varepsilon bounds the deviation ∥Jt−A∥\|J_t-A\| and EE bounds the per-step error. Appendix C measures these quantities on the trained 75M, 166M, and 1B models: κ\kappa stays finite, even on sequences eight times longer than in training. Moreover, the proof only uses the deviation Jt−AJ_t-A along the DSF signal, which is 40–47% lower for DSF than for BPTT and FT-BPTT, reflecting the alignment above.


4. Choice of AA (Appendix E)

The default A=diag⁡(linspace⁡(0.1,0.9))A=\operatorname{diag}(\operatorname{linspace}(0.1,0.9)) is used for all main results. A new ablation on the copy task (Appendix E.1, five seeds) shows how to adapt it to long delays:

Copy task, choice of AAExact match (%)
BPTT (reference)99.78 ± 0.05
Default43.5 ± 6.3
linspace⁡(0.1,0.99)\operatorname{linspace}(0.1,0.99)79.6 ± 6.2
Log-spaced time scales, ai∈[0.1,0.999]a_i\in[0.1,0.999]99.18 ± 0.20

On language modeling, entries close to one do not help (Table 16). Proposition 1 explains both trends: the DSF signal scales as E/(1−∥A∥)E/(1-\|A\|) when errors are dense, but stays bounded when they are sparse. This gives a simple guideline: the default for densely supervised tasks, and longer, log-spaced time scales for sparse long-range tasks.


5. Efficiency (Section 4.4, Figure 5)

Perplexity against wall-clock time: with the optimized backward kernel, DSF gives a 1.64× end-to-end speedup over BPTT on the six-layer GRU.


6. Other updates

All copy-task results (Table 5) are now averaged over five seeds, with all methods trained in the same setup. The gradient-diagnostics protocol is detailed in Appendix A.10, and reported baseline values are harmonized across tables.

For the final version, we will expand the related work with the references suggested by the reviewers, and add the Transformer and Mamba reference points given in our response to Reviewer Ldph.

The method and its main takeaways are unchanged; the revision completes the evidence behind them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.