- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes to train stochastic RNNs using the EM algorithm instead of BPTT. EM involves inference over the latent states (i.e., over the system noise), which alternates with weight updates. This alleviates the weakness of BPTT implementations that must sample from the noise prior, and it avoids vanishing gradients because information is propagated through time (over multiple EM iterations) via the inferred states. Experiments identify regimes in which EM outperforms BPTT, and further theoretical results derive existing BPTT alternatives within the EM framework.
Critical strengths
Elegant theoretical unification Promising new direction(s) for RNN optimization
Critical weaknesses
Toy problems. Unclear how the EM methods will scale.
Questions for authors
Why is (10) claimed to be equivalent to (5)? The predictive distribution is not symmetric nor is the variance independent of .
The claim in Appendix F was confusing until I realized training is done without teacher forcing. (Otherwise, the model could learn from the dependence of on ; dropping the expectation over in (97) would be incorrect because would depend on via .) IIUC, this is an unusual task because it cannot (even in principle) be solved above chance: the expected square errors at () are both equal to (see (92)). So my question is how the EM-trained model behaves. In context 1, does it randomly generate either two positive outputs or two negative outputs, even though this yields greater error than outputting (0,0)?
235: there is no fig 1C. Lower part of Fig 2B?
4 comments
Toy problems and computational scaling
We agree that our experiments focus on relatively small networks and controlled tasks. These settings were chosen to isolate the mechanisms underlying the differences between posterior-based learning and prior-rollout BPTT and to connect the numerical results to our theoretical analyses.
Regarding computational scalability, we acknowledge that the Gaussian-smoothing EM algorithm used in our experiments is more expensive than BPTT. As analyzed in Appendix B, for a network with neurons and sequence length , this EM implementation has time complexity and memory complexity , compared with time and memory for a single-rollout BPTT calculation.
We have also examined the effect of network size on task performance in Fig. C.2, where EM maintains an advantage over the tested BPTT procedure across different network sizes. Our current goal is therefore not to claim that the proposed implementation is already competitive with BPTT in computational efficiency and on practical benchmarks, but to establish conditions under which posterior-based learning provides advantages and to identify the mechanisms responsible for these advantages.
Finally, we will correct the reference to Fig. 1C, which should refer to the lower panels of Fig. 1B.
Thank you again for these constructive comments, which have helped us clarify the framing and strengthen the experimental evidence. We believe that these clarifications and additional analyses provide a clearer understanding of the distinct roles of learning joint output distributions, performing conditional inference, and optimizing network parameters under different objectives.
Thanks for the replies, which address all my questions. Regarding the sequence generation task, notice that BPTT would also fail on a simpler task with a single timepoint, y_1 ~ N(0,1). The difference lies in the KL vs square loss: EM will learn to output the distribution while BPTT learns only the mean. (Of course if squared error is the objective, then predicting the mean is better.) EM's success on this task can still be traced to posterior inference: Roughly, on observing y_1 it computes a posterior for xi_1 and then updates W based on that posterior estimate, thus bootstrapping its way into learning a correlation between xi and y. The two-step task you tested works for essentially the same reason: EM learns the joint distribution (bivariate Gaussian with +1 or -1 correlation) while BPTT learns only the mean which is (0,0) in both conditions. This seems like a separate mechanism from the one emphasized in the paper, that BPTT must sample from the noise prior while EM can focus on the posterior where the bulk of the marginal likelihood lies. Just some further thoughts that you may or may not find useful.
Thank you for your further point on the separate mechanism about BPTT with prior-rollout learns the conditional mean (point prediction) while EM learns the joint output distribution. We fully agree that this is conceptually distinct from the mechanism highlighted in the manuscript, namely that BPTT samples from prior while EM uses posterior.
We also hope to add that both mechanisms could be viewed as a single mechanism under the EM framework: in the expression of ELBO (Eq. 32), if we use the prior distribution as the variational distribution q, it naturally yields the prior-rollout MSE. In contrast, if we use posterior as the variational distribution q, it gives the marginal likelihood loss. So in this sense, the fact BPTT learns conditional mean is a mathematical consequence of sampling from the prior.
We believe this unified perspective is a useful contribution of our theoretical framework, as it helps explain how different choices of the variational distribution lead to different learning objectives. Thank you again for this thoughtful comment, which has helped us clarify the conceptual distinctions and strengthen the unified perspective presented in our manuscript.
Question 1 & 2
Thank you for the positive assessment of our work, particularly for recognizing the value of the theoretical unification that EM algorithm provides. We also appreciate the questions and suggestions, which will help us clarify the manuscript and strengthen the experimental comparisons.
1. Relationship between Eq. 5 and Eq. 10 (question 1)
Thank you for pointing out the ambiguity in our discussion of the relationship between the marginal-likelihood objective (Eq. 5) and the Bayesian loss (Eq. 10). We agree that these objectives are not generally equivalent, particularly when the predictive covariance depends on the network parameters.
More precisely, minimizing the negative marginal log-likelihood in Eq. 5 will recover the conditional distribution of the outputs given the data inputs. Its predictive mean coincides with the Bayes-optimal predictor under squared error and therefore minimizes Eq. 10. However, this does not imply that the two objectives are equivalent for arbitrary model parameters, or that approximate EM necessarily finds the minimum of Eq. 10. We will correct this statement in Section 3.1 and Appendix D.
2. Structural sequence generation and teacher forcing (question 2)
We agree that the structural sequence generation task requires a clearer explanation.
As you correctly noted, prior-rollout MSE training without teacher forcing cannot learn this task. Because the target outputs have zero conditional means, the expected BPTT gradient contains no information about their cross-time correlations, as shown in Appendix G.
However, we agree that this limitation can be overcome by introducing teacher forcing. For example, when the first observed output is used as an additional input to the network, BPTT can learn to predict whether the second output should be identical to or opposite to the first. We implemented this teacher-forced BPTT baseline and found that although it can learn the conditional prediction task, its performance deteriorates at high state-noise levels, where EM achieves lower prediction error:
The EM’s superior performance may arise from its explicit treatment of uncertainty in stochastic network dynamics through posterior inference.
We would also like to highlight an important difference between teacher forcing and posterior-based inference. An EM-trained generative network provides a model of the joint output distribution, which can be used for different prediction and inference tasks without retraining its weights. For example, the same EM-trained model can, in principle:
These operations can be performed by conditioning the same generative model on different subsets of observed outputs. In contrast, the teacher-forced BPTT baseline considered here is explicitly trained for one particular conditional-prediction task, where the first observed output is provided as an input to predict the second. Although such a network also induces a joint output distribution, its training objective does not explicitly encourage accurate modeling of that distribution or prediction under other conditioning protocols. Posterior-based inference therefore provides a more flexible framework for using the same learned network across different prediction tasks.
Regarding the behavior of the EM-trained network, the target distribution in our structural sequence generation task satisfies yT1=z and yT2=cz, where z is a zero-mean Gaussian vector and c=±1 specifies whether the two outputs should be identical (c=1) or opposite (c=−1). EM-trained model will correctly learn this joint Gaussian distribution in each context. In context 1 (c=1), a correctly learned model would generate pairs of identical output vectors, but these vectors need not be entirely positive or negative, since each coordinate of z is a Gaussian variable and can have either sign.