Recurrent network implementations of sequential Gaussian Process regression

In noisy environments, biological and artificial agents often approximate optimal Bayesian inference. However, whether and how neural circuits support such probabilistic computation remains unknown. Here, we investigate neural representations of uncertainty in recurrent neural networks trained to perform sequential Gaussian process (GP) regression, a rich setting where the target Bayesian computation is exactly specified and an expanding dataset must be compressed into a finite-capacity hidden state. We show that RNNs learn to implement reduced-rank recursive Bayesian regression to approximate GP inference. These networks learn an approximately linear belief representation over the leading eigenfunctions of the GP kernel, and this representation emerges through successive stage-like transitions during training. Causal perturbations to the hidden state confirm the identified representations are linearly steerable. Lastly, we develop a novel framework for examining how belief covariance is encoded in neural activity. Our results demonstrate how finite-state recurrent networks can implement sequential probabilistic inference, and offer a methodology for identifying probabilistic representations of uncertainty in artificial and biological dynamical systems.