- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The authors propose an estimator for the one-step-ahead conditional distribution of a multivariate stochastic process. Under the presented model assumptions, the authors prove that the predictive states can be injectively embedded via the conditional expectations of random Fourier features of future observations, where p is the Hausdorff dimension of the predictive state space. Finally, the authors provide convergence rate guarantees for their estimator using recurrent neural networks and mixture density networks.
Critical strengths
- Estimating the one-step ahead conditional distribution of a multivariate stochastic process is a relevant problem with many practical applications as the authors highliht in section A.1.
- Contribution 2 is elegant and provides valuable insights on the connection between the geometry of the state space and the number of features required to represent it.
Critical weaknesses
-
The paper suffer from serious presentation issues. Several pages worth of content are devoted to presenting assumptions; yet, most of them remain undiscussed. Similarly, the results in section 3.4 are not discussed and give no insight on the consequences of the main theoretical resuts. The authors also never explain why the rates they propose are fast. Finally, the authors do not include a related work section in the main text but relegate it to the appendix, making it difficult for the reader to contextualize the contribution. The authors cite a relevant work [1] that similarly employs random Fourier features, but they never explicitly discuss this connection, nor do they compare against it experimentally.
-
The estimator of , despite being the central object of the paper, is never stated explicitly. Likewise, the procedure in Algorithm 1 is generic and omits important mechanisms in line 3 and 4 of the algorithm, I suggest that the authors expand those steps or at least point to the equations in the paper responsible for those steps. Moreover, the experimental section does not verify the theoretical results of theorem 4 and 5. Additionally, the data generating process is fairly simple and low-dimensional, which does not allow to assess the robustness of the estimator in more realistic scenarios.
[1] Downey et al. ; Predictive state recurrent neural networks
Questions for authors
- Can the authors elaborate on why their rates are to be considered fast?
- Can the authors state the final estimator explicitly, and clarify which theorem applies to the estimator actually used in practice?
4 comments
Thanks again for your review! We'd appreciate your feedback, and we'd be very happy to address any remaining concerns.
I thank the authors for their rebuttal. The changes are welcome and improve the clarity of the manuscript. I still have several concerns:
- I agree with the authors that "formalizing ideas and asking fundamental questions is valuable because it can help clarify why certain ideas work", but it remains unclear to me how the proposed estimator and its properties clarify why RNNs, MDNs or other architrectures work. Section 2.4.1 now reads as a description of how a RNN works rather than clarifying why it works.
- The authors write "our theoretical investigations can also inform the development of architectures for new sequence models based on predictive state representations". Can the authors elaborate on this point? What principles and insights in the work can drive the development of new architectures?
- I do not agree with listing contributions 3 and 4 while putting the technical content in the appendix. I understand that other reviewers and I raised concerns about clarity due to the dense presentation but if the authors believe they form a core contribution they should rethink their presentation rather than simply moving it to the appendix.
- The new experiments are welcome, but they are not adequately discussed. For instance, in the C. elegans experiment the authors find it interesting that the k=3 instantiation of their method works best. Why is that the case? Why does performance inversely correlate with k in that experiment? Do the authors find the empirical convergence observed in their synthetic experiments consistent with the theoretical rates in the appendix?
Thank you for your thoughtful response to our rebuttal. We really do appreciate it.
I agree with the authors that "formalizing ideas and asking fundamental questions is valuable because it can help clarify why certain ideas work", but it remains unclear to me how the proposed estimator and its properties clarify why RNNs, MDNs or other architrectures work. Section 2.4.1 now reads as a description of how a RNN works rather than clarifying why it works.
We have updated that sentence to make it clearer that the "certain ideas" we were referring to are the ideas from the literature on predictive state representations (discussed in Sections 1.1.1 and 1.1.2), not RNNs and MDNs. The goal of our work is to explain why learning representations of predictive states works at all (this is not obvious), rather than explaining why RNNs or MDNs work (these are just particular architectures that our ideas can be implemented with). Sections 2.4.1 and 2.4.2 explain how our algorithm is implemented with an RNN and MDN, respectively, because those are the architectures used in our experiments (and supported by the convergence rate results).
The authors write "our theoretical investigations can also inform the development of architectures for new sequence models based on predictive state representations". Can the authors elaborate on this point? What principles and insights in the work can drive the development of new architectures?
We have updated this sentence as well. This is the subject of our ongoing work, so unfortunately we can't give more details about the new architecture, loss functions, etc., or even hints, without giving the ideas away. We really do dislike having to give you this answer here, but we hope you understand that we do have a good reason (and that we're not just being unnecessarily coy or dismissive of your good questions). As a compromise, we have changed that sentence to be more of a general statement.
I do not agree with listing contributions 3 and 4 while putting the technical content in the appendix. I understand that other reviewers and I raised concerns about clarity due to the dense presentation but if the authors believe they form a core contribution they should rethink their presentation rather than simply moving it to the appendix.
We have removed these contributions from the "Our Contributions". We agree that we either had to (1) remove the emphasis on contributions that are now in the appendix, or (2) somehow rework that technical material to fit it in the main text. We decided that it would be better to prioritize the improved exposition of our main ideas, so we went with option (1).
Everything moved to the appendix is now presented as an "additional result" or "bonus result", and everything in the main text is presented as a "main result" (we have added small comments throughout the main text to emphasize this). Our hope is that the convergence rate results are now understood by you and the other reviewers as nice "bonus material" that interested readers can go on and read about in the appendix.
We appreciate your help working with us to improve the exposition of our results. We think it's much more sensible now (main ideas presented clearly in main text, and bonus technical results in appendix for interested readers).
The new experiments are welcome, but they are not adequately discussed. For instance, in the C. elegans experiment the authors find it interesting that the k=3 instantiation of their method works best. Why is that the case? Why does performance inversely correlate with k in that experiment? Do the authors find the empirical convergence observed in their synthetic experiments consistent with the theoretical rates in the appendix?
We have added more discussion about the C. elegans experiment (see "Our interpretation"). For example, we discuss the interpretation that our estimator takes advantage of intrinsic structure in the C. elegans data which allows the past information to be compressed into a lower-dimensional representation (OTOH, the baseline vanilla MDN estimator is not able to take advantage of this structure).
Yes, the empirical convergence rates are consistent with our theoretical convergence rates. Of course, there are many unknown constants, so it's tough to say anything more rigorous than this (certainly, if the estimation error was constant or increasing with the sample size, then no, that would not be consistent with our theoretical convergence rate results).
We believe the paper is much better now after incorporating your second round of feedback.
Please let us know if you have any additional concerns.
Thank you for taking the time to carefully review our initial submission and provide constructive feedback. We have made major revisions to improve our paper based on your comments. We hope that these revisions address your concerns.
We agree with all of your points. We made the following changes: (1) we have restructured the paper to improve its readability and flow, (2) we moved the assumptions and theoretical convergence rate results to the appendix, while keeping the parameter estimation procedures for the recurrent neural network and mixture density network in the main text (so this is much clearer now), (3) we have clarified that we meant relatively faster rates are achievable when the predictive state space is lower-dimensional, and (4) we have moved the related work section to the main text which contains sections "Comparisons" and "Purpose of Our Work" to address your concerns about comparisons with \textit{Downey et al.; Predictive state recurrent neural networks} and better contextualizes our contributions. Regarding the insights from our results, we list some here: we clarify the probabilistic objects that are being represented, we explain when these representations based on random Fourier features actually identify the predictive state, we provide estimators and explain what probabilistic objects they are actually converging to, and we give sufficient conditions under which this convergence actually occurs.
We agree that the estimator Q^Z^t of the conditional distribution QZt should have been stated explicitly (previously, it was left implicit based on the definition from the estimated conditional density); please see the end of Section ``Implementation with Deep Neural Networks'' for the explicit estimator. We agree that the algorithm was too generic, so for the sake of concreteness we have added a section directly below Algorithm 1 that presents the details for estimating the parameters of the recurrent neural network and mixture density network. In the experiments section, we have added two higher-dimensional data generating processes and a real data example. Please let us know if you'd like to see additional experiments.
By faster we mean that the convergence rate can be improved when the dimension of the predictive state is lower. We have now rewritten all sentences about the speed of the convergence rate to clarify that we meant relatively faster rates can be achieved when the dimension of the predictive state space is lower (i.e., not that the convergence rates are always fast).
Please see the end of Section "Implementation with Deep Neural Networks" for the final estimator. Theorem 5 is about the out-of-sample L2 error based on continuing the fitted recurrence, and the plots in the experiments section show that this L2 error (averaged over 500 out-of-sample time steps) decreases as the training sample size n increases.
We are grateful for your thoughtful feedback, and we think the updated manuscript is significantly better. We would appreciate hearing your thoughts, so that we can address any remaining concerns.