acceptodds
Under review as a conference paper at ICLR 2027

Recurrent network implementations of sequential Gaussian Process regression

Abstract

In noisy environments, biological and artificial agents often approximate optimal Bayesian inference. However, whether and how neural circuits support such probabilistic computation remains unknown. Here, we investigate neural representations of uncertainty in recurrent neural networks trained to perform sequential Gaussian process (GP) regression, a rich setting where the target Bayesian computation is exactly specified and an expanding dataset must be compressed into a finite-capacity hidden state. We show that RNNs learn to implement reduced-rank recursive Bayesian regression to approximate GP inference. These networks learn an approximately linear belief representation over the leading eigenfunctions of the GP kernel, and this representation emerges through successive stage-like transitions during training. Causal perturbations to the hidden state confirm the identified representations are linearly steerable. Lastly, we develop a novel framework for examining how belief covariance is encoded in neural activity. Our results demonstrate how finite-state recurrent networks can implement sequential probabilistic inference, and offer a methodology for identifying probabilistic representations of uncertainty in artificial and biological dynamical systems.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Recurrent neural networks, Gaussian processes, belief representations, nonautonomous dynamical systems

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZWCYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how recurrent neural networks represent and perform sequential probabilistic inference. The authors train GRUs to carry out sequential Gaussian process (GP) regression, where observations are presented one at a time and the network predicts both the mean and uncertainty at each query location. By comparing network behavior and hidden-state representations with an analytically specified Bayesian filter, the paper argues that the trained networks approximate GP inference through reduced-rank recursive Bayesian regression over the leading eigenfunctions of the GP kernel. The paper further reports that the posterior mean is approximately linearly represented in the hidden state, that the corresponding eigenmodes emerge sequentially during training, and that uncertainty is better explained by covariance-like rather than precision-like parameterizations. Finally, hidden-state interventions are used to test whether the identified representations causally influence the network’s predictions.

Critical strengths
  1. Sequential GP regression provides a controlled setting in which the target Bayesian computation is known exactly, making it particularly suitable for studying whether recurrent networks learn probabilistic representations.

  2. The paper does not rely on a single diagnostic. It estimates the network’s effective rank using predictive behavior, the spectrum of its effective kernel, and the dimensionality of hidden representations. The agreement among these measurements in the main GRU setting is encouraging.

  3. The hidden-state perturbations go beyond standard linear probing. Editing the inferred posterior-mean representation produces the expected change in predictive means, while the uncertainty interventions affect predictive variance after controlling for changes in the mean. These experiments provide meaningful evidence that the identified representations are functionally relevant.

  4. Interpolating between covariance- and precision-based representations using the power-Euclidean family is an interesting methodological contribution for studying uncertainty representations in artificial and potentially biological neural systems.

Critical weaknesses
  1. The experimental setting is too narrow to establish the generality of the conclusions.

The main experiments use a one-dimensional input on a fixed interval, a stationary RBF kernel with mostly fixed hyperparameters, Gaussian observation noise, and a relatively small set of recurrent architectures. It is therefore unclear whether the conclusions extend to multidimensional inputs, other kernels, nonuniform input distributions, non-Gaussian or heteroscedastic noise, or more complex sequential-inference problems. The neuroscience implications should consequently be stated more cautiously.

  1. The interpretation that the GRU “implements” reduced-rank recursive Bayesian regression is stronger than the current evidence supports.

The experiments establish that the network’s behavior and hidden representations are consistent with a reduced-rank Bayesian solution. However, they do not directly show that the GRU performs the specific recursive updates derived in the paper. In particular, the work does not identify neural variables corresponding to the innovation or Kalman gain, nor does it demonstrate that one-step hidden-state transitions match the analytical Bayesian update. Alternative learned algorithms could produce similar outputs and linearly decodable belief states.

  1. The effective rank of approximately seven is insufficiently characterized.

The agreement among rank estimates in the main setting is interesting, but there is no reason for seven to be a universal or structurally special value. The learned rank should depend on the GP eigenspectrum, kernel lengthscale, noise level, context length, hidden-state capacity, and training budget. Although the appendix varies some of these factors, it primarily reports predictive loss rather than how the learned rank changes. Controlled experiments with known finite-rank kernels or systematically varied spectra are needed to show that the network adaptively selects the relevant rank rather than reaching an architecture- or optimization-dependent limit.

  1. The precise form of the uncertainty representation is not convincingly identified.

The encoding scores for Σ\Sigma, Σ1/2\Sigma^{1/2}, Σ1/4\Sigma^{1/4}, and log⁡Σ\log\Sigma are nearly identical. Thus, the shallow optimum at α=1/4\alpha=1/4 should not be interpreted as evidence that the network specifically encodes Σ1/4\Sigma^{1/4}. The stronger conclusion supported by the results is only that covariance/expectation-side parameterizations are more linearly aligned with the hidden state than precision/natural-side parameterizations. Moreover, direct covariance interventions also alter the predictive mean and require an additional mean-clamping operation, indicating that the two representations are not cleanly separated.

Questions for authors
  1. Can the authors directly test whether the GRU’s one-step state transition implements the analytical recursive Bayesian update—for example, by identifying representations of the prediction error/innovation and Kalman gain, or by testing whether an edited belief state evolves according to the correct update on subsequent observations?

  2. How does the learned effective rank vary with kernel lengthscale, observation noise, context length, hidden size, and training duration? In particular, would the network recover different ranks on synthetic GP tasks whose true kernel ranks are explicitly set to, for example, 3, 7, and 12?

  3. Do the principal conclusions hold for multidimensional inputs, nonuniform or out-of-distribution query distributions, different kernel families, and heteroscedastic or non-Gaussian observation noise?

2 comments

We greatly appreciate the reviewer's thoughtful feedback on our manuscript and took great care to incorporate many of their criticisms. Our response is separated into two comments.

(1) The experimental setting is too narrow to establish the generality of the conclusions.

(3) The effective rank of approximately seven is insufficiently characterized.

We thank the reviewer for this feedback and have accordingly added new analyses to the manuscript. Specifically, Figure 5 (new) now shows key analyses on a broader range of recurrent architectures. This analysis strengthens our results because it suggests a natural dichotomy in the belief state parameterisation: We find that nonlinear (gated) RNNs learn expectation parameter representations (since expectation parameter update dynamics are inherently nonlinear), while modern structure state space models (SSMs, which have linear dynamics) learn natural parameter representations. As suggested by the reviewer, we also include Figure 8 (new) in the appendix, which varies RBF kernel hyperparameters, Gaussian observation noise, hidden state capacity, and the true (truncated) rank of the prior kernel used during training and demonstrates approximate robustness in our three rank estimation procedures. Crucially, when training samples are generated from rank-MM truncated GPs (rather than the full GP), the three rank estimates seem to recover this rank decently well. We believe the architectural generalisation in Figure 5 and the hyperparameter generalisation in Figure 8 address the reviewer's primary concerns and help to strengthen our underlying arguments. The reviewer also recommended multidimensional inputs, other kernels, and non-Gaussian observation models, which we believe are valuable extensions of this work that we intend to investigate in future work; however, we were not able to include those extensions in this work.

(4) The precise form of the uncertainty representation is not convincingly identified.

We direct the reviewer to Section 4.3 and Figure 5 for our extended results across architectures. While we concede that the exact value α⋆\alpha^\star should not be overinterpreted given the flatness of the maximum (as now explicitly articulated in footnote 3), the fact that gated nonlinear networks and SSMs reveal optima at different ends of the power α\alpha spectrum strengthens our claim that the particular choice of parameterisation is important in identifying the format of the network's internal representation. Crucially, we now not only report encoding R2R^2 as a function of power α\alpha (and a new power β\beta which controls how much the mean component of the belief state is transformed by the covariance, i.e. Σβμ\mathbf \Sigma^\beta \boldsymbol \mu) but also report the steerability R2R^2 as a function of those parameters (showing how effective linear steering is at each α,β\alpha,\beta point). Thus, our covariance analysis is not simply correlational---it directly impacts how effectively one can causally manipulate the network's internal representation.

(2) The interpretation that the GRU “implements” reduced-rank recursive Bayesian regression is stronger than the current evidence supports.

We agree with the reviewer's assessment that our work does not explicitly identify e.g. where the Kalman gain is implemented in trained networks, and have accordingly softened our language somewhat in light of this concern (e.g., we have refrained from using the word "implements"). However, we do believe our analysis demonstrates that the networks learn a reduced-rank Bayesian regression solution insofar as the Bayesian sufficient statistics can be recovered from linear encoding and can be used to linearly steer the network's internal representation. By definition, if the Bayesian sufficient statistics can be linearly read-out at each timestep (which they can be), the decoded trajectory itself satisfies the Kalman recursion approximately. While we do not direct causal manipulations at these identified Kalman gain representations (and concede that as an alternate methodology), we believe our correlational and causal results on the rank-MM Bayesian weight-space sufficient statistics adequately establish that the network approximate reduced-rank Bayesian regression.

We would greatly appreciate the reviewer's thoughts on these arguments and look forward to their feedback.

Reviewer_9EhiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates how RNN (specifically a GRU) trained on synthetic sequential Gaussian Process (GP) regression data learns to represent uncertainty. Through linear encoding models and causal perturbations, the authors demonstrate that the network empirically converges to a reduced-rank recursive Bayesian approximation.

Critical strengths
  1. The motivation is clear and the narrative is self-contained.
  2. Figure 2 looks neat and the manuscript is easy to follow.
Critical weaknesses
  1. Truncating a GP into finite basis is a well-studied technique [1], and since sequential GP regression can be solved by Kalman filtering, it's not surprising that the optimal solution is a recursive linear update. Training a finite-state model (here a GRU) to minimize loss on this exact task and discovering that it learns the optimal recursive Bayesian update is not surprising.
  2. Only toy experiments based on synthetic 1D inputs and an RBF kernel are considered.
  3. The paper only considers legacy GRU, however the current SOTA recurrent sequence models (e.g., HiPPO, S4, Mamba) is built on State-Space Models (SSMs). In particular, [2] seems to be a relevant work where they explicitly design a recurrent GP using the HiPPO framework (a renovated RNN). I think a submission to ICLR 2027 would need to contrast its empirical analysis on GRU with modern recurrent sequence architectures to stay relevant.

Overall, while the empirical analysis of the GRU state is self-contained, showing that a legacy RNN architecture can learn a decade-old textbook update in a 1D toy setup doesn't seem to meet the novelty and impact standard for ICLR 2027.

[1] Rasmussen & Williams. Gaussian Processes for Machine Learning. MIT Press 2006

[2] Chen et al. Recurrent Memory for Online Interdomain Gaussian Processes. NeurIPS 2025

Questions for authors

See Weakness.

2 comments

We greatly appreciate the reviewers thoughtful comments on our manuscript. We have written a few responses below to the points raised by the reviewer.

Truncating a GP into finite basis is a well-studied technique [1], and since sequential GP regression can be solved by Kalman filtering, it's not surprising that the optimal solution is a recursive linear update. Training a finite-state model (here a GRU) to minimize loss on this exact task and discovering that it learns the optimal recursive Bayesian update is not surprising.

While we agree that finite-basis approximations to GP regression have been studied extensively, Kalman filtering does not generically solve GP regression for kernels like the RBF kernel. Kalman filtering's recursive updating is only the optimal solution after a truncated basis is chosen (for nondegenerate kernels; for degenerate and half-integer Matérn kernels, we agree that Kalman filtering is always an optimal solution). How this basis is chosen by the network and the rank of this approximation are nontrivial. Additionally, the networks studied in the main text were allowed to have large capacity (for the GRU, 128 hidden states with gating variables), leaving open the possibility that the network could have learned the dual function space solution requiring no recursive/reduced-rank approximation; that it doesn't (and seems to generalise to much longer context sizes) is a useful result for future investigation. Beyond the question of whether the network learns true probabilistic inference and how it does this algorithmically, our work also focuses on the exact format of the network's representation assuming it has learnt probabilistic inference, which has not been done systematically in previous work. Our work investigates what kind of representation the network learns, and our finding that it learns a flat/linear and steerable code was surprising and novel. Importantly, prior work on neural coding of uncertainty has focused primarily on low-dimensional and/or categorical inference problems rather than high-dimensional, continuous inference tasks (as is the case with GP regression). Finally, while the debate between natural or expectation parameter representations in neural networks has received several theoretical accounts (Ma et al., 2006; Vértes & Sahani, 2018), previous work has not provided falsifiable evidence in favor of one parameterisation over the other. Our work does.

We have added a discussion of some of these points in Section 5 (Discussion) and would greatly appreciate the reviewer's thoughts on these arguments.

Only toy experiments based on synthetic 1D inputs and an RBF kernel are considered.

We greatly appreciate this feedback; while we were unable to incorporate a systematic analysis in higher dimensions and for other kernels (both of which are insightful extensions that we intend to pursue in future work), we direct the reviewer to supplementary Figure 8 (new), which shows rank estimates for a broader range of task hyperparameters (GRU hidden size, GP lengthscale, observation noise, and true prior kernel rank).

The paper only considers legacy GRU, however the current SOTA recurrent sequence models (e.g., HiPPO, S4, Mamba) is built on State-Space Models (SSMs). In particular, [2] seems to be a relevant work where they explicitly design a recurrent GP using the HiPPO framework (a renovated RNN). I think a submission to ICLR 2027 would need to contrast its empirical analysis on GRU with modern recurrent sequence architectures to stay relevant.

We greatly appreciate this feedback and have accordingly added analysis of several additional architectures to the manuscript. Figure 5 (new) now compares key results from the paper on networks with an update-gate RNN (UGRNN) unit (Collins et al., 2017), LSTM unit (Hochreiter & Schmidhuber, 1997), generic diagonal linear unit with nonlinear readout (a simple structure state space model), linear recurrent unit (Orvieto et al., 2023), and S5 unit (Smith et al., 2023). The latter three models (SSMs) have two layers to allow for enough expressivity to solve the task comparably well to the GRUs. See section 4.3 for a discussion of these results. In brief, analysis of SSMs and two alternative gated architectures reveals an approximate universality of the linear belief manifold and linear steerability results with a predictable dichotomy in the covariance representation results: Gated networks, which have nonlinear dynamics, learn approximately expectation parameter codes, whereas SSMs, which have linear dynamics, learn natural parameter codes. We believe this strengthens our analysis, given that the nonlinear/linear dichotomy is exactly what theory predicts: expectation parameter dynamics are nonlinear with a linear readout, whereas natural parameter dynamics are linear with a nonlinear readout.

I thank the authors for the rebuttal. However I don't think my concerns are adequately addressed, especially on the practical value of this paper. Hence I keep my score unchanged.

Reviewer_vwixOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper observes that Gauss Process regression requires a kernel matrix that grows with the data, which a fixed-size RNN cannot store. The authors suggest that the network instead does Bayesian regression on a small number of the kernel’s leading basis functions. Their empirical study then show that a trained RNN behaves this way.

Critical strengths

The strength is in seeing that and empirically showing the only a small number of basis functions is necessary, which indicate theat the original infinite dimensional problem can be reduced to finite dimensional

Critical weaknesses

The biggest weakness is leaving the mathematical objects not defined or used before a definition is given. This makes reading difficult and what is worst the main points mights may not be conveyed to the reader.

Questions for authors

none

1 comment

We thank the reviewer for taking the time to read our paper and provide a review. We are writing to kindly request a few clarifications from this reviewer. We would greatly appreciate if the reviewer could provide examples of where we might have left "the mathematical objects not defined or used before a definition is given," and if there are areas of improvement we can focus on to address the reviewer's concerns about the informativeness and soundness of the manuscript. Given that the other two reviewers, despite their criticisms, found the paper understandable (and even "clear and easy to follow" in one case), we would greatly appreciate these clarifications so as to strengthen the paper, especially since the "Questions for Authors" section was left blank. Thank you very much for your time, and we look forward to hearing your feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of rebuttal-period manuscript changes

We sincerely appreciate the thoughtful feedback provided by the reviewers and have taken considerable care in addressing many of their concerns. Some of this has been addressed in the respective replies to the reviewers, but here, we would like to provide an overview of how changes have been integrated into the manuscript.

Figure 5 (main) and Section 4.3 (new): Reviewer 9Ehi raised the concern that our analysis focused primarily on a legacy nonlinear recurrent architecture (the GRU) and not on more state-of-the-art state space models (SSMs). We now include a comparative analysis of five additional architectures, two gated nonlinear recurrent architectures (LSTMs (Hochreiter & Schmidhuber, 1997) and UGRNNs (Collins et al., 2017)) and three variants of SSMs (a simple diagonal SSM, the LRU (Orvieto et al., 2023), and S5 (Smith et al., 2023)). In pursuing this broader comparative analysis, we sought to test the hypothesis that gated network (with nonlinear dynamics) were better suited for expectation parameter representations, while SSMs (linear dynamics) were better suited for natural parameters. Our analysis (Figure 5) suggests exactly this, and we provide the relevant discussion in Section 4.3.

We believe this new analysis also addresses both 9Ehi's and ZWCY's concerns about the limited scope of our results (one architecture and a fixed set of hyperparameters).

Discussion: We now include a paragraph discussing more explicitly the motivation and novelty of our work, based on our response to reviewer 9Ehi.

Figure 8 (appendix, new): 9Ehi and ZWCY raised the concern that the original results may not be general because only one architecture and set of hyperparameters were analysed. Additionally, ZWCY expressed concern that the rank estimation were not adequately validated. As a result, we have included a systematic comparison of the three rank estimation procedures (two behavioural, one representational) across a range of hyperparameters, namely GP lengthscale ℓ\ell, recurrent layer size HH, observation noise σn\sigma_n, and the rank of the prior covariance (using finite-rank truncations of the KL-expansion as suggested by ZWCY). In the lattermost case, we show that the rank estimates approximately recover the true rank, and in general, Figure 8 serves to demonstrate that our results are robust and not unique to the particular choice of hyperparameters used in the original study.

Figure 9 (appendix, new): To further illustrate that the networks surprisingly learn something akin to reduced-rank recursive regression, we reproduce key results in a network trained with largely the same hyperparameters and architecture (GRU) except that trials have context size 40 rather than 100. In this regime, the number of total tokens (80) is less than the number of hidden neurons, leaving open the possibility that the network learns an input-memorising (and/or potentially function-space GP) approximation. It instead reproduces the results from the main text, suggesting that even in the input-memorising regime, the inductive biases of the network favor a reduced-rank recursive approximation.


Additional changes

Figure 4 (main): Several changes have been made:

  • The former Figure 5 (linear steering of covariance) has now been subsumed into Figure 4.
  • Along with Figure 3, the summary panel now includes random matched-norm edits (hatched bars) as another baseline.
  • Belief state encoding profiles now vary along two parameters, α\alpha (as before) and a new parameter β\beta. β\beta controls how much the mean μt\mu_t is transformed by the covariance. In this more accurate parameterisation, expectation parameters sit at α=1,β=0\alpha=1, \beta=0 and natural parameters sit at α=β=−1\alpha=\beta=-1 (up to bias terms and/or multiplicative constants).
  • Panel B now shows α,β\alpha, \beta profiles for steerability R2R^2 (i.e., how effective linear steering is). The optimum of this profile is selected for subsequent steering experiments. This is also used for visualisation and steering examples in Figure 5.

We would like to thank the reviewers and area chairs once again for their feedback and guidance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.