- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors propose Supervised Memory Training (SMT) for pretraining nonlinear RNNs, decoupling the learning of memory representations from memory dynamics. This avoids recurrent credit propagation by reducing RNN training to supervised learning on one-step memory transition labels. These labels are generated by an additional Transformer encoder-decoder model pair to learn predictive memory representations. A subsequent fine-tuning stage (DMT) mitigates accumulated errors during recurrent rollouts. Experiments demonstrate that SMT pretraining followed by DMT fine-tuning outperforms conventional BPTT across various RNN architectures and tasks, particularly in capturing long-range dependencies.
Critical strengths
Decoupling the learning of memory representations from the learning of memory dynamics is an interesting alternative for training RNNs beyond conventional BPTT. The proposed method is relatively simple and applicable to different nonlinear RNN architectures. The authors provide empirical evidence across several tasks that their approach can improve learning of long-range dependencies. The experimental investigation is relatively comprehensive, including different RNN architectures, synthetic and natural tasks, scaling experiments, several ablation studies, and analyses of gradient stability and memory drift.
Critical weaknesses
-
The actual computational advantage is not convincingly established. While the authors demonstrate improvements in sequential computation, these do not necessarily translate into reduced overall training costs. In particular, SMT requires jointly training a Transformer encoder-decoder model pair and an RNN, followed by sequential DMT fine-tuning. It remains unclear whether the full method (SMT→DMT) outperforms BPTT under comparable total computational budgets, e.g., in terms of total training FLOPs or GPU-hours. Without such a comparison, the practical significance of the claimed computational advantages remains uncertain.
-
The novelty relative to existing approaches should be stated more prominently. In particular, the connections to teacher-student distillation, predictive-state representations, and related approaches such as NextLat are substantial. Although connections are discussed in Appendix A, a more explicit discussion in the main text would help clarify the specific methodological contributions beyond existing approaches.
-
While SMT avoids recurrent unrolling during pretraining, the full method (SMT→DMT) additionally requires DMT, which again reintroduces sequential computation. Figure 13 shows that DMT substantially improves the final performance, indicating its importance for the overall approach. However, most experimental comparisons consider the full SMT→DMT pipeline, making it difficult to disentangle the contributions of the two training stages. In particular, it remains unclear to what extent the reported improvements can be attributed to SMT pretraining rather than subsequent rollout-based fine-tuning.
Questions for authors
-
Does SMT→DMT retain its advantages over BPTT and alternative parallel RNN training methods when the complete training pipelines are compared under equal computational budgets?
-
Although the authors explicitly exclude linear RNNs: Could SMT also benefit linear RNNs and modern state-space architectures such as Mamba? Investigating this would 1) help clarify whether SMT's advantages primarily stem from one-step credit assignment or from avoiding sequential computation during training; and 2) would increase practical relevance.
Additional feedback (not part of decision assessment)
- The statements about "chaotic dynamics" and "edge of chaos" are mathematically imprecise. Exploding gradients, dynamical instability, and chaos are distinct concepts. The authors should distinguish these properties more carefully, particularly given the interpretation of Lyapunov exponents and chaotic dynamics in Appendix Figure 11.
- I would strongly recommend moving Figure 1 below the abstract or to page 2. Its current placement before the abstract is unconventional and somewhat disrupts the reading flow.
2 comments
Thank you for your clarifications. I will keep my score.
k7c1 Rebuttal
Thank you for your useful review!
W1/Q1: See common response on FLOPs and wallclock
W2: We agree that connecting SMT to related work more would be beneficial. We will add more discussion/comparison of SMT to prior methods and move the related work to the main paper for camera ready.
W3: See common response for SMT vs DMT
Q2: As you said, we explicitly exclude linear RNNs since we are interested in nonlinear RNNs in their own right (see https://arxiv.org/abs/2603.03612). We don't expect SMT to help linear attention RNNs (e.g. Mamba) because they are already parallel and have one-step credit assignment across the sequence (the two problems that SMT solves). However, they solve this problem in a naive way: by having the architecture just add up the states: st=∑i=0tkiviT, which trivially creates one-step gradient path, without nonlinear expressivity.
Thanks for the additional feedback! We will update the language around the chaotic dynamics and edge of chaos.