- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies learning a state matrix , decoder parameters , and transition parameters from known matrices and , where each letter in an alphabet has it's own and . The state loss trains to approximate (with weight decay on the ), the decoder loss trains to approximate , and each transition loss trains to approximate the corresponding .
Theorem 9 proves that when is updated only using the state loss, is udpated using only the decoder loss, and each is updated only by its corresponding transition loss, then the gradient flow converges globally from any initialisation of that is full rank and arbitrary finite initialisations of and the . In contrast, Theorem 12 proves that when every parameter is updated by every loss that depends on it, the gradient flow can converge to non-optimal minima. Theorem 13 proves that after a finite time, it is possible to switch from the decoupled training to fully coupled updates while still converging.
When is a complete finite block of the target function’s finite-rank Hankel matrix and are its one-symbol shifts, the limiting parameters induce an exact linear recurrent realisation through the decoder and transitions .
Critical strengths
- The mathematical formulation is precise, and the assumptions and main theoretical results are clearly stated, although some of the mathematical arguments can be hard to follow. I did not identify any errors in the proofs of the results.
- The controlled synthetic experiments nicely demonstrate the key theoretical results.
Critical weaknesses
It is not clear how much relevance the results of this paper have outside the relatively narrow case considered, where one has access to a complete finite Hankel block and its one-symbol shifts, and follows the particular training procedure proposed by the paper. In particular, it is unclear if the difference between triangular and fully coupled training has any implications on training recurrent models in practice. In my opinion, this limits the paper’s apparent appeal for the general ICLR audience. However, my background is outside this particular theoretical area, and I welcome the other reviewers’ perspectives on the significance of these results.
Questions for authors
I have no questions for the authors.
Reminder to reviwers to read author's responses
Dear Reviewers,
Thank you for your efforts in providing thorough reviews.
As the discussion period is underway, I would like to kindly remind you to read the authors' responses to your reviews and provide follow-up comments where appropriate or leave a comment. In particular, please consider whether the responses address your questions and concerns, and update your assessment if the clarifications or additional information affect your initial evaluation.
Your engagement in this discussion is important for ensuring a fair and thorough review process.
Thank you again for your time and contributions!
Best regards, Area Chair