Direct State Feedback: Fast RNN Training with Stationary Backward Dynamics

Backpropagation through time (BPTT) trains recurrent neural networks by transporting errors through a sequence of time-varying state Jacobians, making temporal gradient propagation inherently sequential. Inspired by feedback alignment, we introduce Direct State Feedback (DSF), an alternative, model-agnostic training method for recurrent networks. DSF relies on a temporal feedback rule that replaces the recurrent Jacobians used during the backward pass with a fixed diagonal approximation, while preserving the original nonlinear forward dynamics. This turns temporal error transport into a stationary linear recurrence that can be efficiently evaluated with parallel scans. We further combine fixed feedback with exact gradients over short horizons, yielding T-DSF, which continuously interpolates between truncated BPTT and full temporal credit assignment. Despite its aggressive backward approximation, DSF retains much of the performance of BPTT across language modeling and memory tasks, while substantially outperforming purely local or truncated updates and reducing training cost at scales up to one billion parameters. Gradient diagnostics further show that DSF updates remain positively aligned with BPTT despite increasing approximation error at long horizons. These results suggest that recurrent networks can be trained with simple, stationary feedback dynamics in place of exact Jacobian transport, extending the feedback-alignment principle from depth to time.