acceptodds
Under review as a conference paper at ICLR 2027

When Does Probabilistic Inference Improve Task Learning for Stochastic RNNs?

Abstract

Recurrent neural networks (RNNs) are used both to solve temporal tasks and to model the dynamics of physical and biological systems. In both settings, recurrent dynamics are often subject to noise, but how this stochasticity is handled in learning algorithms differs. Task optimization typically minimizes output error along noisy rollouts of network dynamics using backpropagation through time (BPTT). System identification instead infers latent trajectories conditioned on observed outputs and learns from this posterior, for example, via expectation–maximization (EM). Can such inference-based learning of recurrent weights also benefit task optimization? We compare EM-based posterior-inference-based EM with prior-rollout BPTT training and show that the two coincide in the low-noise limit, in both objective and optimization dynamics, but diverge when noise is non-negligible. In this regime, we identify theoretically and empirically three task families where posterior inference improves task learning: long-delay tasks, where it avoids the exponential, delay-dependent decay of BPTT's learning signal; tasks with distribution shifts, where conditioning latent activity on targets reduces catastrophic interference; and tasks with ambiguous targets, where EM captures temporal structure that output-error minimization ignores. Finally, we show that several alternatives to BPTT can be interpreted as different approximate E-steps within a unified EM framework, suggesting that advances in state-space-model inference may offer principled routes to new BPTT alternatives. Together, these results illuminate when inference-based training is advantageous over prior-rollout training in stochastic learning systems.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Recurrent neural network, Probabilistic methods, Gaussian process, Task-optimized models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xFXbOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes to train stochastic RNNs using the EM algorithm instead of BPTT. EM involves inference over the latent states (i.e., over the system noise), which alternates with weight updates. This alleviates the weakness of BPTT implementations that must sample from the noise prior, and it avoids vanishing gradients because information is propagated through time (over multiple EM iterations) via the inferred states. Experiments identify regimes in which EM outperforms BPTT, and further theoretical results derive existing BPTT alternatives within the EM framework.

Critical strengths

Elegant theoretical unification Promising new direction(s) for RNN optimization

Critical weaknesses

Toy problems. Unclear how the EM methods will scale.

Questions for authors

Why is (10) claimed to be equivalent to (5)? The predictive distribution is not symmetric nor is the variance independent of WW.

The claim in Appendix F was confusing until I realized training is done without teacher forcing. (Otherwise, the model could learn from the dependence of hT2h_{T_2} on xT1+1=yT1x_{T_1+1}=y_{T_1}; dropping the expectation over zz in (97) would be incorrect because f2f_2 would depend on zz via hT1+1h_{T_1+1}.) IIUC, this is an unusual task because it cannot (even in principle) be solved above chance: the expected square errors at TiT_i (i=1,2i=1,2) are both equal to ∣∣y^Ti∣∣2+dY||\hat{y}_{T_i}||^2+d_Y (see (92)). So my question is how the EM-trained model behaves. In context 1, does it randomly generate either two positive outputs or two negative outputs, even though this yields greater error than outputting (0,0)?

235: there is no fig 1C. Lower part of Fig 2B?

4 comments
AuthorsOpenReview ↗

Question 1 & 2

Thank you for the positive assessment of our work, particularly for recognizing the value of the theoretical unification that EM algorithm provides. We also appreciate the questions and suggestions, which will help us clarify the manuscript and strengthen the experimental comparisons.

1. Relationship between Eq. 5 and Eq. 10 (question 1)

Thank you for pointing out the ambiguity in our discussion of the relationship between the marginal-likelihood objective (Eq. 5) and the Bayesian loss (Eq. 10). We agree that these objectives are not generally equivalent, particularly when the predictive covariance depends on the network parameters.

More precisely, minimizing the negative marginal log-likelihood in Eq. 5 will recover the conditional distribution of the outputs given the data inputs. Its predictive mean coincides with the Bayes-optimal predictor under squared error and therefore minimizes Eq. 10. However, this does not imply that the two objectives are equivalent for arbitrary model parameters, or that approximate EM necessarily finds the minimum of Eq. 10. We will correct this statement in Section 3.1 and Appendix D.

2. Structural sequence generation and teacher forcing (question 2)

We agree that the structural sequence generation task requires a clearer explanation.

As you correctly noted, prior-rollout MSE training without teacher forcing cannot learn this task. Because the target outputs have zero conditional means, the expected BPTT gradient contains no information about their cross-time correlations, as shown in Appendix G.

However, we agree that this limitation can be overcome by introducing teacher forcing. For example, when the first observed output is used as an additional input to the network, BPTT can learn to predict whether the second output should be identical to or opposite to the first. We implemented this teacher-forced BPTT baseline and found that although it can learn the conditional prediction task, its performance deteriorates at high state-noise levels, where EM achieves lower prediction error:

State noiseBPTT Bayesian lossEM Bayesian lossTF Bayesian loss
0.10.98360.03770.0137
0.20.98350.03610.0175
0.30.98310.04630.0233
0.40.98240.03740.0309
0.50.98200.05060.0549
0.60.98180.03980.0525
0.70.98180.04440.0616
0.80.98210.04400.0779
0.90.98260.03650.1500
1.00.98340.03630.2009
1.50.98740.04800.9157
2.00.99230.04340.9834
2.50.99760.04570.9838

The EM’s superior performance may arise from its explicit treatment of uncertainty in stochastic network dynamics through posterior inference.

We would also like to highlight an important difference between teacher forcing and posterior-based inference. An EM-trained generative network provides a model of the joint output distribution, which can be used for different prediction and inference tasks without retraining its weights. For example, the same EM-trained model can, in principle:

  1. Generate a pair of outputs without conditioning on either observation.
  2. Predict the second output conditioned on the first observation.
  3. Infer the first output conditioned on the second observation.

These operations can be performed by conditioning the same generative model on different subsets of observed outputs. In contrast, the teacher-forced BPTT baseline considered here is explicitly trained for one particular conditional-prediction task, where the first observed output is provided as an input to predict the second. Although such a network also induces a joint output distribution, its training objective does not explicitly encourage accurate modeling of that distribution or prediction under other conditioning protocols. Posterior-based inference therefore provides a more flexible framework for using the same learned network across different prediction tasks.

Regarding the behavior of the EM-trained network, the target distribution in our structural sequence generation task satisfies yT1=zy_{T1} = z and yT2=czy_{T2} = cz, where zz is a zero-mean Gaussian vector and c=±1c = \pm 1 specifies whether the two outputs should be identical (c=1c = 1) or opposite (c=−1c = -1). EM-trained model will correctly learn this joint Gaussian distribution in each context. In context 1 (c=1c=1), a correctly learned model would generate pairs of identical output vectors, but these vectors need not be entirely positive or negative, since each coordinate of zz is a Gaussian variable and can have either sign.

AuthorsOpenReview ↗

Toy problems and computational scaling

We agree that our experiments focus on relatively small networks and controlled tasks. These settings were chosen to isolate the mechanisms underlying the differences between posterior-based learning and prior-rollout BPTT and to connect the numerical results to our theoretical analyses.

Regarding computational scalability, we acknowledge that the Gaussian-smoothing EM algorithm used in our experiments is more expensive than BPTT. As analyzed in Appendix B, for a network with NN neurons and sequence length TT, this EM implementation has time complexity O(TN3)O(TN^3) and memory complexity O(TN2)O(TN^2), compared with O(TN2)O(TN^2) time and O(TN)O(TN) memory for a single-rollout BPTT calculation.

We have also examined the effect of network size on task performance in Fig. C.2, where EM maintains an advantage over the tested BPTT procedure across different network sizes. Our current goal is therefore not to claim that the proposed implementation is already competitive with BPTT in computational efficiency and on practical benchmarks, but to establish conditions under which posterior-based learning provides advantages and to identify the mechanisms responsible for these advantages.

Finally, we will correct the reference to Fig. 1C, which should refer to the lower panels of Fig. 1B.

Thank you again for these constructive comments, which have helped us clarify the framing and strengthen the experimental evidence. We believe that these clarifications and additional analyses provide a clearer understanding of the distinct roles of learning joint output distributions, performing conditional inference, and optimizing network parameters under different objectives.

Thanks for the replies, which address all my questions. Regarding the sequence generation task, notice that BPTT would also fail on a simpler task with a single timepoint, y_1 ~ N(0,1). The difference lies in the KL vs square loss: EM will learn to output the distribution while BPTT learns only the mean. (Of course if squared error is the objective, then predicting the mean is better.) EM's success on this task can still be traced to posterior inference: Roughly, on observing y_1 it computes a posterior for xi_1 and then updates W based on that posterior estimate, thus bootstrapping its way into learning a correlation between xi and y. The two-step task you tested works for essentially the same reason: EM learns the joint distribution (bivariate Gaussian with +1 or -1 correlation) while BPTT learns only the mean which is (0,0) in both conditions. This seems like a separate mechanism from the one emphasized in the paper, that BPTT must sample from the noise prior while EM can focus on the posterior where the bulk of the marginal likelihood lies. Just some further thoughts that you may or may not find useful.

Thank you for your further point on the separate mechanism about BPTT with prior-rollout learns the conditional mean (point prediction) while EM learns the joint output distribution. We fully agree that this is conceptually distinct from the mechanism highlighted in the manuscript, namely that BPTT samples from prior while EM uses posterior.

We also hope to add that both mechanisms could be viewed as a single mechanism under the EM framework: in the expression of ELBO (Eq. 32), if we use the prior distribution as the variational distribution q, it naturally yields the prior-rollout MSE. In contrast, if we use posterior as the variational distribution q, it gives the marginal likelihood loss. So in this sense, the fact BPTT learns conditional mean is a mathematical consequence of sampling from the prior.

We believe this unified perspective is a useful contribution of our theoretical framework, as it helps explain how different choices of the variational distribution lead to different learning objectives. Thank you again for this thoughtful comment, which has helped us clarify the conceptual distinctions and strengthen the unified perspective presented in our manuscript.

Reviewer_TR7nOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Authors present an interesting story, comparing two different approaches to training stochastic RNNs: backpropagation-through-time (BPTT), using rollouts as Monte-Carlo approximations of state noise, and expectation-maximization (EM), which infers a posterior over likely latent trajectories given both the inputs and the outputs and uses those to update the network parameters.

They make the following claims:

  1. EM and BPTT coincide in the limit of low state noise
  2. EM is superior on long-delay tasks, tasks with distribution shifts, and tasks with ambiguous targets
  3. Several different BPTT-alternatives can be cast as different approximate E-steps in the EM framework

They support these claims with both simulation and theoretical analysis.

Critical strengths

I found the comparison of BPTT-alternatives to various E-step approximation approaches to be interesting and creative. This seems to be a novel insight and sells the authors perspective that the systems identification framework is quite helpful in reasoning about these network objects.

Critical weaknesses
  1. Overall, the framing of the paper as comparing backpropagation-through-time (BPTT) and expectation maximization (EM) doesn’t seem to accurately describe the actual scientific questions being tackled in the paper. It seems more accurate to say that the paper tackles BPTT used to optimize MSE loss to EM. This is an important distinction, because, while MSE is standard for many/most uses, one could certainly use BPTT to optimize other losses, including distributional losses. For example, the RNN could be trained to emit the mean/covariance of a Gaussian distribution over the output conditional on the input; this could be trained with BPTT on the log-likelihood, as far as I can tell. Indeed, the authors’ own results include a stochastic characterization of BPTT, that itself suggests ways that BPTT could be used with the log-likelihood loss directly (e.g., sampling multiple rollouts and computing the conditional likelihood of the output given the input and the generated sequence, averaging them, and taking the log of that as the overall log-likelihood). As such, it strikes me that the framing is overly ambitious given the scope of the actual analysis, which concerns really only point prediction optimized with MSE loss + BPTT algorithm.

  2. Some of the experimental results seem quite trivial given the involved factors. Firstly, the weakness of BPTT on long-delay tasks due to vanishing gradients is well-known, and the cause being that backpropagating the gradients involves a product at each timestep is well-understood. It’s precisely this weakness that led to the development of gated RNNs, which I was surprised to see were not included in the experiments (and even more surprised that they were not mentioned at all to my recollection). I understand the point is to show that the EM variant does not have this problem (or rather has it more weakly) but the presentation was not convincing in light of the above. Secondly, the sequence generation result is not at all surprising: the BPTT optimum, due to the MSE loss + scalar prediction setup, will be the conditional mean; necessarily, the network should learn to just output the same point each time since it has no mechanism for conditioning on the forced first output. In general, the outcome of this experiment seems more of a reflection of the fact that the EM network naturally can condition on the forced first output whereas the typical BPTT setup doesn’t have a natural avenue for this. (Though, note that again, given the authors’ stochastic characterization of BPTT, there probably is some sort of unnatural way of doing this that would test the learned representations separately from the generation procedure.)

  3. Some of the experiments seem to have technical issues. The fact that in the distribution-shift experiment, the EM approach is allowed to maintain running statistics across the distribution shift, seems like a serious confound and not just “an opportunity for future work to disentangle”, particularly when resetting the running statistics at the boundary is straightforward. It separately seems confounded to me that the distribution shift itself encompasses really three different distribution shifts: shift in the context distribution (uniform to only context 1), shift in the output mean (from u to u/2), and shift in the input noise (s to 3s). These should really be presented separately in addition to the joint result.

Questions for authors

None

8 comments
AuthorsOpenReview ↗

1. Clarifying the comparison between BPTT and EM (weakness 1, part 1/2)

Thank you for the thoughtful comments and for recognizing the value of interpreting alternative RNN learning algorithms through the framework of system identification. We sincerely apologize for the lack of clarity in our original framing and experimental comparisons. We address the three main concerns below and will revise the manuscript to clarify these distinctions and incorporate all additional experimental results.

1. Clarifying the comparison between BPTT and EM (weakness 1)

We fully agree that BPTT is a general method for computing gradients and is not restricted to minimizing MSE. To clarify, our study specifically compares minimizing the prior-rollout MSE objective (based on point prediction) with maximizing the marginal-likelihood objective across different task settings. The former is optimized via BPTT, and the latter via EM. Thus, the two approaches differ in both their training objectives and parameter updates.

Prior-rollout MSE has been widely used in task-optimized RNNs in neuroscience [1–6], where networks are trained to produce accurate responses despite intrinsic noise. Our central question is whether EM-based learning can improve task performance even when prior-rollout MSE, rather than likelihood, is the final performance metric.

We believe one of our most interesting findings is that optimizing the marginal likelihood with EM can yield a lower test prior-rollout MSE than directly optimizing the MSE with BPTT, as observed in our long-delay, distribution-shift, and structural sequence experiments. If BPTT reliably found the global minimum of the MSE objective, EM could not outperform it on this metric. Thus, the observed advantage suggests that EM yields more effective optimization dynamics, even when measured by MSE. Intuitively, this is because EM can better account for uncertainty in stochastic settings and has a natural mechanism for conditioning on the output.

As you suggested, BPTT can also be used to optimize the marginal likelihood by sampling multiple rollouts, averaging the conditional likelihood of the output given the input, and taking the log of that as the overall log-likelihood. In the limit of infinitely many rollout samples, BPTT on this prior-rollout likelihood objective is equivalent to generalized EM (GEM, Eq. 8) when using an exact posterior, as shown by Fisher’s identity in Eq. 37. However, this equivalence generally does not hold for a finite number of rollout samples. Furthermore, the prior-rollout likelihood BPTT update is generally not equivalent to the exact EM M-step used for the RNN model throughout our manuscript.

We compare four training algorithms on the delayed XOR task used in our manuscript: (1) prior-rollout MSE optimized using BPTT, (2) multi-rollout negative log-likelihood (NLL) optimized using BPTT, (3) generalized EM (GEM), and (4) EM with an exact M-step. Below are the test MSE (delay = 5, number of training data P = 200, performance measured by MSE):

State-noiseMSE BPTTNLL BPTTGEMEM
0.0200.0503 ± 0.02130.0502 ± 0.02050.2602 ± 0.00020.0309 ± 0.0051
0.0370.0433 ± 0.01040.0558 ± 0.01210.2799 ± 0.00310.0537 ± 0.0183
0.0690.0395 ± 0.01300.0622 ± 0.01460.2693 ± 0.02240.0395 ± 0.0130
0.1260.0375 ± 0.00910.0956 ± 0.03020.2429 ± 0.02850.0630 ± 0.0401
0.2330.0405 ± 0.00900.0980 ± 0.03010.1265 ± 0.04660.0783 ± 0.0459
0.4290.0659 ± 0.07880.0991 ± 0.03310.1750 ± 0.03980.0589 ± 0.0094
0.7930.2319 ± 0.08040.1396 ± 0.04870.2064 ± 0.03560.0724 ± 0.0496
1.4660.2607 ± 0.00030.2331 ± 0.06380.2145 ± 0.01940.0647 ± 0.0466
2.7070.2615 ± 0.00110.2996 ± 0.02050.2122 ± 0.01990.0559 ± 0.0211
5.0000.2630 ± 0.00290.3175 ± 0.02750.2154 ± 0.01110.0499 ± 0.0154

We find that EM with an exact M-step generally outperforms the other algorithms in this experiment and exhibits less severe vanishing-gradient effects. Thank you for suggesting these additional comparisons, which have helped us strengthen our results and better characterize the performance of alternative BPTT-based training methods.

[1] Yang, et al. Nature Neuroscience 22.2 (2019). [2] Driscoll, et al. Nature Neuroscience 27.7 (2024). [3] Song, et al. Elife 6 (2017). [4] Ehrlich, et al. Eneuro 8.1 (2021). [5] Dubreuil, et al. Nature Neuroscience 25.6 (2022). [6] Mante, et al. Nature 503.7474 (2013).

AuthorsOpenReview ↗

1. Clarifying the comparison between BPTT and EM (weakness 1, part 2/2)

These comparisons also motivate a hybrid training algorithm that uses EM during the initial stages of training to obtain more effective parameter updates, followed by BPTT to directly minimize the desired task objective, namely prior-rollout MSE. We find that this hybrid approach achieves lower test MSE than either EM or BPTT alone on the delayed XOR task (delay = 5, P=200 training examples):

State noisePrior-rollout MSE BPTTEMHybrid
0.0200.0441 ± 0.00290.0349 ± 0.00090.0391 ± 0.0018
0.0370.0497 ± 0.00640.0543 ± 0.00660.0417 ± 0.0016
0.0690.0417 ± 0.00330.0428 ± 0.00610.0405 ± 0.0017
0.1260.0423 ± 0.00210.0523 ± 0.00590.0404 ± 0.0014
0.2330.0404 ± 0.00220.0752 ± 0.01670.0403 ± 0.0022
0.4290.0390 ± 0.00090.0586 ± 0.00410.0415 ± 0.0023
0.7930.1779 ± 0.04020.0727 ± 0.02070.0403 ± 0.0018
1.4660.2605 ± 0.00010.0658 ± 0.01720.0402 ± 0.0017
2.7070.2610 ± 0.00020.0570 ± 0.00690.0405 ± 0.0018
5.0000.2622 ± 0.00060.0499 ± 0.00410.0394 ± 0.0014
AuthorsOpenReview ↗

2. Long-delay tasks and structural sequence generation (weakness 2, part 1/2)

Long-delay tasks. We agree that vanishing gradients in vanilla RNNs are well understood and that gated architectures were developed partly to address this difficulty. Our contribution is not to revisit this limitation, but to characterize how the learning signals generated by EM differ from those produced by prior-rollout BPTT in stochastic RNNs.

In Appendix E, we show that, under stable dynamics, the task-dependent component of the BPTT gradient decays exponentially with the delay, whereas the exact-EM update scales inversely with the delay. We believe that obtaining this explicit analytical distinction is nontrivial and helps explain why EM performs better in long-delay tasks.

We also agree that comparisons with gated recurrent architectures are important. Although gating can mitigate vanishing gradients, BPTT gradients in GRUs still involve products of temporal Jacobians [1]. We perform additional simulations on the delayed-XOR task for a moderate noise level (state noise level = 2), comparing GRUs trained with BPTT and vanilla RNNs trained with either EM and BPTT (number of training data P = 200, performance measured by prior rollout MSE):

DelayVanilla RNN BPTTVanilla RNN EMParameter-matched GRU BPTT
00.0494 ± 0.01120.0602 ± 0.00860.0998 ± 0.0007
20.0453 ± 0.00550.0542 ± 0.01290.1052 ± 0.0013
50.2394 ± 0.04270.0938 ± 0.03010.1145 ± 0.0025
100.2085 ± 0.10430.1754 ± 0.04550.1448 ± 0.0076
150.2608 ± 0.00040.1304 ± 0.02490.2401 ± 0.0451
200.2609 ± 0.00020.1581 ± 0.01320.2468 ± 0.0310
250.2613 ± 0.00050.1567 ± 0.01650.2624 ± 0.0004

We find that for long delays, vanilla RNNs trained with EM achieve lower MSE than both vanilla RNNs and GRUs trained with BPTT. This advantage diminishes at low state-noise levels, suggesting that EM is particularly effective in learning long-delay tasks when intrinsic noise is non-negligible. These findings are consistent with our theoretical analysis of the different delay dependencies of EM and BPTT parameter updates.

[1] Gilboa, Dar, et al. arXiv:1901.08987 (2019).

Reviewer_ZryGOpenReview ↗
Rating
1
Confidence
4(+1)
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The manuscript compares learning algorithms for fitting recurrent models to sequential data generated by latent dynamics, in particular the EM algorithm vs. BPTT applied to noisy rollouts of the observations. (Importantly, there is process as well as observation noise.) The authors show that samples from the model achieve lower MSE when trained with the BPTT scheme (indeed, lower than the teacher's variance, which the EM-trained model matches). On other experiments---a task with a delay period, a task in which the training data shift part way through training, and a task with temporal correlation between outputs---EM is superior (albeit more computationally expensive). Finally, the manuscript re-interprets alternatives to BPTT as different choices for the variational posterior.

Critical strengths

The manuscript is fairly easy to read and thoroughly covers cases of interest. Derivations are presented in great detail in a thorough appendix (not all of which this reviewer read). The overall goal of determining under which contexts BPTT on noisy rollouts is to be used in place of EM is a useful one, even if this paper doesn't perhaps get us there.

Critical weaknesses

The results are framed as identifying some cases in which one method is superior, some in which the other is. But BPTT on stochastic rollouts (although described as "standard") is really inferior in all cases (although it is cheaper), and this follows from the loss function itself. As the authors realize, the BPTT loss amounts to using the generative prior p(h|x) as the variational posterior, q(h). This will essentially always be worse than any other sensible variational posterior. Indeed, it amounts to ignoring the observations, so the longer the trajectory, the more inappropriate this prior becomes. Put a different way, the variational loss can be written as the joint KL divergence, i.e. (suppressing inputs for brevity) KL[pdata(Y)q(H)∣p(H,Y)],KL[p_\text{data}(Y)q(H) | p(H, Y)], which for the choice q(h) = p(h) reduces to Eq[KL[pdata(Y)∣p(Y∣H)]].E_q[KL[p_\text{data}(Y) | p(Y|H)]]. So the optimal model p under this loss ignores the latent dynamics.

This is (perhaps) obscured by the fact that the authors report MSE as in Eq. 9, which is the MSE of samples from the model. This MSE does not evaluate how well the model has learned the true underlying system. (Intuitively, we should never add noise to an estimator, which can only make it worse; and if the model is asked to minimize this MSE, it will simply learn to make less variable outputs.) The authors themselves note that the BPTT student achieves lower MSE than the teacher's variance---but this is a mark against (not in favor of) the student.

When only the "Bayesian loss" is considered (i.e., the standard MSE between noiseless model output and true data), EM is superior on all tasks---as one would expect, since it makes a better posterior approximation.

MINOR

  • The authors write that minimizing Eq. 5, the marginal cross entropy, is equivalent to maximizing the evidence lower bound. I think they should say the ELBO (as the name says) is a lower bound on (the negative of) Eq. 5. (This also affects the claims about the Bayes predictor at the end of Appendix C.)
  • The bottom panels of Fig. 1B are referred to in the text as Fig. 1C.
  • Fig. 2A for the delayed XOR task: the figure doesn't seem to match the task description. The means of the inputs look to be matched for all time. Also, the two outputs are equal for long stretches, but they are described as reporting a proposition and its negation. Perhaps I am misunderstanding it.
Questions for authors

Can the authors provide a setting in which using the prior p(h|x) as the variational posterior would be preferred to any standard variational posterior? Can they specify circumstances under which one which their sample-based MSE would be the right quantity to minimize?

7 comments
AuthorsOpenReview ↗

Clarifying the motivation and scope of our study

Thank you for your time and effort in reviewing our work and for finding it fairly easy to read and thoroughly covering the cases of interest. We apologize for not clearly motivating our study and our choice of objective in the previous version of the manuscript.

1. Motivation and scope

We would first like to clarify that the main focus of our study is to use sample-based MSE as the primary performance metric, and to characterize when and why the EM algorithm can outperform BPTT under this metric.

In computational neuroscience, there is a long history of training RNNs with intrinsic noise to perform cognitive tasks and subsequently analyzing their learned dynamics [1-7]. In these settings, the goal is to train the network to produce accurate responses despite fixed intrinsic noise. Therefore, minimizing the expected squared error of the network responses is a natural objective. We therefore believe that understanding how different learning algorithms affect task performance under this MSE metric is an important question for computational neuroscience and cognitive science.

Our central finding is that even when minimizing MSE is the ultimate goal, directly optimizing this objective using BPTT does not necessarily yield the best task performance. In particular, we find that EM, despite optimizing a different objective, the marginal likelihood, achieves lower test MSE. In particular, in tasks with long delay periods, distribution shifts and output correlations.

We believe this is a highly nontrivial result. It is true that, if BPTT could reliably find the global minimum of the MSE, EM could not outperform it under the MSE metric. The observed advantage of EM therefore highlights differences in optimization dynamics rather than an inherent advantage of the likelihood objective in minimizing MSE. Our theoretical analyses and numerical experiments suggest that a key mechanism underlying this advantage is EM's ability to account for uncertainty through posterior inference, thereby producing more effective parameter updates in the presence of stochasticity.

Together, these results suggest that a learning algorithm optimizing an objective different from the final task metric can nevertheless achieve better performance on that metric than an algorithm directly optimizing it. We believe that identifying when and why posterior-based EM training achieves lower MSE than prior-rollout BPTT is an important contribution of our work.

[1] Yang, et al. Nature Neuroscience 22.2 (2019). [2] Driscoll, et al. Nature Neuroscience 27.7 (2024). [3] Song, et al. Elife 6 (2017). [4] Ehrlich, et al. Eneuro 8.1 (2021). [5] Dubreuil, et al. Nature Neuroscience 25.6 (2022). [6] Mante, et al. Nature 503.7474 (2013).

AuthorsOpenReview ↗

2. When is sample-based MSE the appropriate objective? (weakness & question 1)

In our introduction, we contrast two ways of training RNNs with intrinsic noise: task optimization and system identification. In system identification, as you suggested, the goal is to recover the underlying latent dynamics that generate the observed sequential data. Maximizing the marginal likelihood (Eq. 5) is the correct objective, and the EM algorithm (using posterior inference) provides one approach to achieving it. But as we mentioned in the last response, there are many scenarios in which the goal is to produce the correct response in the presence of intrinsic noise, for which sample-based MSE is the appropriate objective. We explain this in more detail below:

a) Task optimization in noisy recurrent networks. For example, consider a recurrent network that must retain an input and produce an accurate response after a delay, while intrinsic noise remains present during execution. The question there is: how should the weights be learned when the network must produce an accurate response despite fixed intrinsic noise?

The relevant objective here is the expected squared error of each response, i.e., MSE. In our model (Eq. 1), the noise amplitudes are fixed, and the network must learn to perform the task despite this intrinsic variability. As you pointed out, minimizing MSE involves both matching the conditional mean of the network output to the target and reducing its output variance.

b) Noise as implicit regularization. Injecting noise into RNNs during MSE training can also provide implicit regularization [1,2]. This can help reduce overfitting when training data are limited and encourage the emergence of more robust and stable recurrent dynamics. Thus, noise can also play a constructive role during learning, even when it increases the variability of individual network responses.

We believe that examining sample-based MSE is an important research direction, and our work sheds light on when EM will outperform BPTT across various task settings.

[1] Lim, et al. NeurlPS 34 (2021). [2] Rungratsameetaweemana, et al. PNAS 122.3 (2025).

AuthorsOpenReview ↗

3. Is the prior preferable as a variational posterior? (question 2)

Thank you for suggesting an alternative formulation of the ELBO that more clearly illustrates the consequences of using the prior as the variational distribution. We agree that if the goal is to recover the underlying latent dynamics, the prior is not preferable to the exact posterior. As discussed above, our use of prior trajectories is instead motivated by the task-performance criterion (prior-rollout MSE). For the model considered here, minimizing MSE is equivalent, up to scaling and parameter-independent constants, to minimizing the negative ELBO evaluated at the prior. This equivalence does not imply that the prior provides a better posterior approximation. Rather, it highlights the distinction between two training objectives.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.