Neural Real-Time Recurrent Learning

Recurrent neural networks (RNNs) are commonly trained using backpropagation through time. However, its memory cost grows with sequence length, and it prevents online updates of the RNN parameters. Real-Time Recurrent Learning (RTRL) enables online parameter updates at a fixed memory cost. It maintains an influence matrix describing how parameter changes affect the state of the network. Despite these benefits, both its memory and per-step computational costs scale poorly with network width. Prior work has addressed these limitations either by restricting the recurrent architecture to make RTRL tractable or by approximating the influence matrix, notably through low-rank tensor representations. In this work, we introduce Neural Real-Time Recurrent Learning (NRTRL), which learns a compact representation of the influence matrix. NRTRL uses neural networks to update this representation over time and reconstruct the corresponding influence estimate, allowing the approximation mechanism itself to be learned. We instantiate this approach by reusing the RNN cell along a short virtual trajectory, with learned low-rank perturbations to its parameters and learned inputs. We show that this construction learns influence estimates that are accurate enough to enable learning. We evaluate NRTRL on pixel-by-pixel MNIST and character-level Penn Treebank.