acceptodds
Under review as a conference paper at ICLR 2027

Pretraining Recurrent Networks without Recurrence

Abstract

Training recurrent neural networks (RNNs) requires assigning credit across long sequences of computations. Standard backpropagation through time (BPTT) addresses this problem poorly: it is sequential in time, limiting parallelism, and suffers from vanishing or exploding gradients, making long-range associations difficult to learn. We propose Supervised Memory Training (SMT), a method for training nonlinear RNNs that sidesteps recurrent credit propagation entirely by reducing RNN training to supervised learning on one-step memory transition labels . SMT acquires these memory labels by training a Transformer-based encoder on a predictive state objective—retaining only information from the past necessary to predict the future. By decoupling what to remember from how to update memory, SMT enables time-parallel RNN training with a stable length gradient path between any two tokens—without ever unrolling the RNN. We find that SMT outperforms BPTT when pretraining various RNN architectures on tasks like language modeling and pixel sequence modeling. SMT enables nonlinear RNNs to better capture long-range dependencies and train in parallel, potentially unlocking the scaling of models that build temporal abstractions of past experience.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
RNNs, parallelism, vanishing gradients, credit assignment, transformers

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_k7c1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose Supervised Memory Training (SMT) for pretraining nonlinear RNNs, decoupling the learning of memory representations from memory dynamics. This avoids recurrent credit propagation by reducing RNN training to supervised learning on one-step memory transition labels. These labels are generated by an additional Transformer encoder-decoder model pair to learn predictive memory representations. A subsequent fine-tuning stage (DMT) mitigates accumulated errors during recurrent rollouts. Experiments demonstrate that SMT pretraining followed by DMT fine-tuning outperforms conventional BPTT across various RNN architectures and tasks, particularly in capturing long-range dependencies.

Critical strengths

Decoupling the learning of memory representations from the learning of memory dynamics is an interesting alternative for training RNNs beyond conventional BPTT. The proposed method is relatively simple and applicable to different nonlinear RNN architectures. The authors provide empirical evidence across several tasks that their approach can improve learning of long-range dependencies. The experimental investigation is relatively comprehensive, including different RNN architectures, synthetic and natural tasks, scaling experiments, several ablation studies, and analyses of gradient stability and memory drift.

Critical weaknesses
  1. The actual computational advantage is not convincingly established. While the authors demonstrate improvements in sequential computation, these do not necessarily translate into reduced overall training costs. In particular, SMT requires jointly training a Transformer encoder-decoder model pair and an RNN, followed by sequential DMT fine-tuning. It remains unclear whether the full method (SMT→DMT) outperforms BPTT under comparable total computational budgets, e.g., in terms of total training FLOPs or GPU-hours. Without such a comparison, the practical significance of the claimed computational advantages remains uncertain.

  2. The novelty relative to existing approaches should be stated more prominently. In particular, the connections to teacher-student distillation, predictive-state representations, and related approaches such as NextLat are substantial. Although connections are discussed in Appendix A, a more explicit discussion in the main text would help clarify the specific methodological contributions beyond existing approaches.

  3. While SMT avoids recurrent unrolling during pretraining, the full method (SMT→DMT) additionally requires DMT, which again reintroduces sequential computation. Figure 13 shows that DMT substantially improves the final performance, indicating its importance for the overall approach. However, most experimental comparisons consider the full SMT→DMT pipeline, making it difficult to disentangle the contributions of the two training stages. In particular, it remains unclear to what extent the reported improvements can be attributed to SMT pretraining rather than subsequent rollout-based fine-tuning.

Questions for authors
  • Does SMT→DMT retain its advantages over BPTT and alternative parallel RNN training methods when the complete training pipelines are compared under equal computational budgets?

  • Although the authors explicitly exclude linear RNNs: Could SMT also benefit linear RNNs and modern state-space architectures such as Mamba? Investigating this would 1) help clarify whether SMT's advantages primarily stem from one-step credit assignment or from avoiding sequential computation during training; and 2) would increase practical relevance.

Additional feedback (not part of decision assessment)

  • The statements about "chaotic dynamics" and "edge of chaos" are mathematically imprecise. Exploding gradients, dynamical instability, and chaos are distinct concepts. The authors should distinguish these properties more carefully, particularly given the interpretation of Lyapunov exponents and chaotic dynamics in Appendix Figure 11.
  • I would strongly recommend moving Figure 1 below the abstract or to page 2. Its current placement before the abstract is unconventional and somewhat disrupts the reading flow.
2 comments
AuthorsOpenReview ↗

k7c1 Rebuttal

Thank you for your useful review!

W1/Q1: See common response on FLOPs and wallclock

W2: We agree that connecting SMT to related work more would be beneficial. We will add more discussion/comparison of SMT to prior methods and move the related work to the main paper for camera ready.

W3: See common response for SMT vs DMT

Q2: As you said, we explicitly exclude linear RNNs since we are interested in nonlinear RNNs in their own right (see https://arxiv.org/abs/2603.03612). We don't expect SMT to help linear attention RNNs (e.g. Mamba) because they are already parallel and have one-step credit assignment across the sequence (the two problems that SMT solves). However, they solve this problem in a naive way: by having the architecture just add up the states: st=∑i=0tkiviTs_t=\sum_{i=0}^{t}k_iv_i^T, which trivially creates one-step gradient path, without nonlinear expressivity.

Thanks for the additional feedback! We will update the language around the chaotic dynamics and edge of chaos.

Thank you for your clarifications. I will keep my score.

Reviewer_DLEPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper underscores the importance of non-linear RNNs : constant memory (only keeps track of hidden state), cheap inference per token, and expressive state transitions than linear RNNs and transformers. It aims to solve their training bottleneck: i.e., sequential training through BPTT.

Paper studies the problem of training an RNN, which requires credit assignment to correct past input but the reward signal is typically delayed. In the worst case, an RNN updates its memory from timestep 1,2,..,T but the reward is only available at step T and thus for training through BPTT, the graph gets unrolled and the backpropagated signal either vanishes or explodes by the time it reaches early steps.

This work proposes SMT (supervised memory training) method to provide easier signals for training RNNs. It trains a bidirectional transformer encoder with learned register tokens which compresses the past context (x1,x2,..,xt) into a memory mt. A causal decoder predicts the future output from this memory mt and future input x_{t+1}. Thus, this encoder/decoder training yields compressed memories {m1, m2,.., mt) which can be used as intermediate signals for RNN training which predicts future hidden state given the past hidden state and future input.

This transformer encoder signal enables time-parallel learning of RNN, then RNNs just learns to update memory over time and use this signal to predict the output.

These two systems (non-linear RNN + encoder/decoder) are jointly trained to ensure that encoder learns Markovian memories which serves as the reward signals for RNN. Training includes a MSE loss between RNN hidden state prediction and the encoder learned memory. Alongside, there’s a cross-entropy loss between the output of deocder (using encoder predictions) and ground truth signal. Further, to stabilize the training there’s a small uniformity loss which prevents collapse. During training, for each sample, instead of providing reward for all timesteps, a single timestep is randomly sampled for efficiency.

Only deploying SMT has the issue that the non-linear RNN does not uses its own predicted memory as hidden state input, resulting in a train-test mismatch. Thus, this work proposes a fine-tuning phase, Dagger Memory Training (DMT), which fixes this by unrolling the RNN and uses RNN unrolled hidden states for state transition. This is a small fine-tuning stage after SMT and is inherently sequential like BPTT.

Empirical experiments have been done on small scale tasks including synthetic tasks (retrieval, copy, etc.), character level language modeling on TinyStories, Pixel-sequence MNIST and binarized sketchy. SMT→DMT path beats BPTT in many tasks. SMT→DMT is able to train RNNs while it struggles with vanilla stacked GRUs and LSTMs.

Critical strengths
  • Simple and easy reframing of training RNN into a supervised regression on transformer intermediate signals as reward.
  • O(1) credit assignment path without unrolling RNN avoids instability often observed in ParaRNN/DEER. Since the supervision is a direct reward from a larger capacity encoder/decoder teacher.
  • Good diagnostic analysis of per-timestep gradient magnitudes, drift analysis before and after DMT stage, etc.
  • Lists failures modes like failures on LSTM/GRUs and sequential nature of DMT.
Critical weaknesses
  • Fig~13 shows that DMT stage (which is sequential) is crucial and without this SMT (on its own) has far worse behavior (loss 4 before and 0.6 after DMT).
  • In order to train with SMT, one needs to train this additional encoder/decoder structure and since its a transformer, why would one not use this for the main task? Being a teacher in this setup, the transformer setups a ceiling on the performance on this task. The paper does not show where does SMT→DMT goes beyond the teacher performance.
  • Long-range credit assignment has not been thoroughly tested. Memory labels only see 256 tokens
  • Baselines are weak or missing
    • BPTT is run on a transformer backbone RNN, there are no standard RNN cells or variants like orthogonal RNNs / ODE-RNNs, etc.
    • Truncated BPTT is missing as a baseline
    • SMT→DMT does not work well on LSTM/GRUs which are still widely used recurrent cells in sequential modeling (as far as RNNs are concerned)
    • SMT fails on exactly the classical nonlinear RNNs. The explanation offered (“obsolete, lack RMSNorm”) is a hypothesis, not an ablation. It raises the question of whether SMT needs a Transformer-like transition with a lot of per-step depth, which weakens the efficiency story at inference.
    • ParaRNN is missing from the main comparison
    • Linear RNNs or state-space models like Mamba/DeltaNet etc. at matched scale and compute
  • No compute/inference efficiency results are presented. Training RNNs alongside Transformer network is not cheap and there needs to be some thoughts on what is the computational footprint and empirical latency measurement on training time.
  • The novelty margin over NextLat (Teoh et al., 2025) is narrow. The authors acknowledge NextLat resembles DMT and, with rollout length 1, SMT. The distinction (“our focus is the RNN”) is one of emphasis, not mechanism. A direct empirical comparison is needed.
  • Missing literature relevant papers:
Questions for authors
  • In the section 2.2 (practice), what happens when you sample all timesteps instead of sampling a single timestep t randomly? Its clear the compute cost shoots up, but do you see any gains in performance?
  • DMT seems to be unrolling the RNNs which implies a sequential dependency on BPTT. If so, how much of the performance gain comes from DMT vs only SMT? What is the breakdown of the training cost between the two phases since even though DMT is short fine-tuning, its sequential in nature.
  • What are the class of problems which require this structure, where standard transformers fail? There are hybrid variants which now store even less memory and various techniques to lower the computational and storage footprint of these models.
  • Do you have any comparisons against truncated BPTT, FPTT, and other auxiiliary loss based RNN training?
  • Why do stakced GRU/LSTM fail with proposed scheme? Why not add the same normalization schemes like RMSNorm and other bells and whistles to these cells like in your main recurrent cell? This seem like an issue in the proposed training scheme.
  • Do you have any empirical comparison against NextLat method?
  • Also, one important assumption in SMT is that whatever encoder outputs for the input context (x1,x2,..,xt) — why is this encoder output supposed to be the memory that RNN should learn? Since transformers would take up the encoder hidden states from (time 1,2,…,t) to answer the value at next step. This assumption needs some evidence.
  • Why do you need the uniformity loss? Why this exponential shape? What scenario leads to collapse? What are the hyper-parameters for the weights for different loss terms in Eq.~5?
  • Why are there no baselines corresponding to state space models or orthogonal RNNs? Also, wheres the performance of transformers on these task (that would give a ceiling on the performance)?
1 comment
AuthorsOpenReview ↗

DLEP Rebuttal

Thank you for your constructive criticism, which will improve our paper!

W1: See the common response for SMT vs DMT.

W2: RNNs are fundamentally different than Transformers. Transformer's memory grows with context and Transformers are limited in expressivity. Figure 15 shows the RNN also outperforms the teacher Transformer in context length generalization.

W3: We show context lengths up to 512 (Figure 6), which is well beyond what BPTT can reliably do credit assignment over in our settings. Moreover, our credit assignment is done by the teacher Transformer, an architecture that is already known to have great credit assignment capabilities over more than 10-100K context lengths, but is hard to show with academic resources. We will update our paper to mention this point.

W4: See the common response for the TBPTT baselines and GRU/LSTM failure explanation. We will move the ParaRNN baseline into the main paper for camera ready. Orthogonal RNNs are an appropriate baseline that we are currently running. Thank you for the suggestion!

W5: See common response on FLOPs and wallclock.

W6: We agree NextLat is the closest prior work and will discuss it prominently in the main text. The closest NextLat analog within our framework is SMT with T_f = 1. Our T_f ablations in the common response (updated Figure 25) show this T_f setting considerably underperforms (0.75 loss vs 0.55 loss on TinyStories), suggesting that the long-horizon predictive objective is a key empirical ingredient beyond NextLat. Beyond this, the two methods differ in core motivation: SMT is a practical algorithm designed specifically for training RNNs. NextLat was not designed primarily for RNNs, but rather for regularizing the Transformer, as is evident from the original paper. Its brief RNN experiment was added in a later revision, concurrent with our work. Moreover, training arbitrary RNN architectures (especially those with large memory states) with NextLat is highly non trivial, and would require substantial changes to the algorithm to do so. If time permits, we will add a direct NextLat comparison.

W7: Thank you for making us aware of this related work! We will update the paper to add these citations. We will explore adding FPTT as a baseline.

Q1: We find that performance increases by only a little, but makes training Tx more FLOP expensive, which is impractical to run on our academic budget.

Q2: See the common response for SMT vs DMT.

Q3: https://arxiv.org/abs/2603.03612 shows that Transformers and linear RNNs are theoretically limited in the problems they can solve, while nonlinear RNNs are not. These problems include complex state tracking (e.g. Figure 15 shows state tracking performance of Transformer vs our RNN).

Q4: See the common response for TBPTT baselines.

Q5: See the common response for SMT's failure on Vanilla Stacked GRU/LSTM

Q6: See W6.

Q7: Our formulation creates an explicit information bottleneck, similar to an auto-encoder, to construct the compressed memory state (Figure 10), so the decoder cannot attend to tokens outside of this memory state. Appendix F provides a proof for why the predictive state objective will induce a Markovian memory state sequence. Joint training will also explicitly incentivize Markovianness. All together, the temporal compression, Markovianness, and future prediction ability make this formulation almost exactly what RNNs need for sequence modeling.

Q8: Figure 20 shows the uniformity loss is not critical. However, we find that without it, the memory space is highly collapsed and not diverse, which may be a problem in certain setups, which is why we add it. We choose the exponential shape from https://arxiv.org/abs/2005.10242 since it is theoretically grounded in contrastive learning. For hyperparameters we choose lambda_dec=1, lambda_dyn=0.1, lambda_unif=0.001 by default.

Q9: In most figures in the paper, the SMT Encoder (blue) is a fixed-memory Transformer upperbound (fixed memory size for fairness). With SMT->DMT, our RNNs are able to match this upperbound and even go beyond it in length generalization (Fig 15).

Reviewer_NcUgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Supervised Memory Training, a method intended to avoid the temporal sequentiality and long-range gradient instability of backpropagation through time. The idea is to represent the past as a set of timestamped events and train a Transformer encoder to produce a compressed memory state that is predictive of the future. Consecutive memory states then serve as supervision for an RNN, which learns the one-step transition from (m_t, x_{t+1}) to m_{t+1} without unrolling the complete recurrent computation graph. This reduces the training gradient path between temporally distant information from O(T) to O(1) and permits parallel training across time. The paper compares SMT, SMT followed by DMT, and BPTT and the results suggest advantages in long-range dependency learning, gradient stability, and sequential computation, together with relatively smooth scaling with context length, memory capacity, and model size.

Critical strengths

The paper reformulates RNN training from the perspective of credit assignment and addresses an important problem with a conceptually novel approach. Separating learning what should be remembered from learning how memory should be updated provides a clear framework for time-parallel pretraining of nonlinear RNNs. The synthetic tasks cover gradient stability, memory capacity, state tracking, associative recall, and in-context learning, allowing targeted evaluation of the central claims. Pixel-sequence modeling on MNIST and Sketchy and language modeling on TinyStories offer useful, if limited, complements to the controlled tasks. The analyses of gradient path length, sequential FLOPs, memory bottlenecks, future-prediction horizon, and joint versus detached training are extensive. The paper also openly discusses limitations caused by teacher expressivity and memory-state drift.

Critical weaknesses

The evidence for fairness and practical efficiency remains incomplete. SMT requires training an additional Transformer encoder and decoder, followed by RNN training and potentially non-parallel DMT. The paper emphasizes sequential FLOPs but does not comprehensively compare total FLOPs, GPU hours, peak memory, or end-to-end wall-clock time, potentially understating the cost of teacher training, label generation, and the multistage procedure. The real-world tasks are relatively small, and TinyStories and pixel sequences are still far from modern large-scale language modeling. Standard language-modeling datasets, perplexity comparisons, and downstream evaluations are missing. Most comparisons focus on BPTT, while strong alternatives such as ParaRNN are largely placed in the appendix, and comparisons with truncated BPTT, state-space models, and other parallel RNN approaches are limited. Variance across repeated runs and statistical significance are also unclear.

Questions for authors
  1. When teacher training, decoder training, memory-label generation, and DMT are all included, how do SMT and BPTT compare in total FLOPs, GPU hours, peak memory, and actual training throughput?
  2. Were all baselines matched in parameter count, training data, and hyperparameter-search budget?
  3. Could the authors provide main-text comparisons with truncated BPTT, Mamba-like models, and ParaRNN on standard language-modeling datasets and substantially longer contexts?
  4. How much state drift accumulates when the SMT-trained RNN is rolled out freely, and how does this error scale with sequence length?
  5. How do the teacher's capacity limit, the future-prediction horizon T_f, and the potential non-uniqueness of the memory labels affect the final RNN?
2 comments
AuthorsOpenReview ↗

NcUg Rebuttal

Thank you for your insightful review!

Q1: See common response on FLOPs and wallclock.

Q2: Yes, the baselines are matched in parameter count, state size, exact architecture when possible, and hyperparameter search budget. We sweep some important hyperparameters and plot against tokens seen in Figure 5 and 22.

Q3: We added truncated BPTT baselines in Figure 24. ParaRNN baselines are in Figure 17. We will move them to the main paper for camera ready! We will also consider running larger scale experiments.

Q4: In Figure 13, we plot the rollout drift of the SMT-only RNNs. Raw drift (MSE distance between RNN generated memory state and encoder memory state) grows as a sigmoid curve (slow then fast then slow). DMT is critical to fixing this drift. We plot R^2 instead of raw MSE since it provides comparison with a average-predicting baseline. See the common response for importance of SMT vs DMT.

Q5: The teacher's capacity limits the RNN by directly controlling the quality of the memory labels. We expect smaller teachers to produce worse labels, and thus worse RNNs. Figure 8 and Figure 25 (new) show the impact of Tf on the synthetic tasks and TinyStories/MNIST: Empirically, Tf>>1 helps since it makes credit assignment explicit over long horizons. Potential non-uniqueness of memory labels is actually a central advantage of SMT: one critical job of the encoder is learning what to forget. Finding the right memory representation is about finding which contexts should be collapsed into the same memory state, and which should be separated.

The author's feedback solves most of my concerns. I will keep my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Common Response

FLOPs and Wallclock

We have updated the paper and added Figure 22 that plots test loss vs 1) SeqFLOPs, 2) FLOPs, 3) wallclock time, and 4) data. Note that wallclock=GPU hours since each run is only on one GPU. Our results show the following (> means beats in performance/efficiency):

  • SeqFLOPs: SMT+DMT > BPTT on Tinystories, and SMT+DMT >> BPTT on MNIST
  • FLOPs: SMT+DMT ≈ BPTT on Tinystories, and SMT+DMT > BPTT on MNIST
  • Wallclock: SMT+DMT > BPTT on Tinystories, and SMT+DMT >> BPTT on MNIST
  • Data: SMT+DMT ≈ BPTT on Tinystories, and SMT+DMT >> BPTT on MNIST

SMT+DMT measures the resources used by our entire pipeline (SMT, DMT, the teacher encoder, label generation, etc.). Note that the advantage of parallelization is that it allows our method to access more FLOPs in the same wallclock time (not necessarily be FLOP efficient). In fact, Transformers replaced RNNs primarily because they could unlock the use of more FLOPs, enabling scaling. The fact that SMT+DMT is also more FLOP efficient on long-range memory tasks (e.g. MNIST) is a separate advantage.

SMT vs DMT

We have updated the paper with Figure 23, that shows the importance of SMT vs DMT for the final performance. We want to highlight that DMT is a light sequential finetuning stage to stabilize rollouts after SMT's heavy parallel pretraining. This is directly analogous to Transformers: heavy parallel pretraining is followed up by light sequential RL to incite a specific desired behavior, with both being critical for final performance. Here are the results of an RNN trained on TinyStories with

  1. only SMT: loss=2.5 (drifts during rollout)
  2. only DMT: loss=2.0 (not enough RNN training)
  3. SMT->DMT: loss=0.5 (good performance)

Here is the amount of resources taken up:

  • FLOPs: SMT uses 7.5e17 FLOPs, DMT uses 5.0e15 FLOPs (0.67% of SMT)
  • Data: SMT uses 6.1e9 Tokens, DMT uses 5.6e6 tokens (0.09% of SMT)
  • SeqFLOPs: SMT uses 62.4e6 SeqFLOPs, DMT uses 8.1e6 SeqFLOPs (13.0% of SMT)
  • Wallclock: SMT takes 12 hours, and DMT takes 2.4 hours (20% of SMT)
  • (These numbers are for 150000 SMT iterations and 300 DMT iterations, which is enough for full performance)

Note also that DMT's sequential rollout is completely unoptimized, and can be improved further with engineering. SMT moves the weights by a distance of 311.7 from the init weights. In contrast, DMT only moves the weights by a distance of 12.9 from the SMT weights (4% of the distance). This all suggests that DMT is a lightweight finetuning stage, like RL is. That is why we focus our paper on SMT, which enables parallel pretraining of nonlinear RNNs.

SMT on Vanilla Stacked GRU/LSTM

We have updated the paper with Figure 21, that shows key ablations to understand why SMT fails on the vanilla stacked GRU/LSTM architectures. We update our original hypothesis for why. Our evidence suggests that two factors that contribute to failure:

  1. The hierarchical structure of the memory state itself, which is not fit for the SMT Encoder to output.
  2. The lack of expressivity of a shallow one-step mapping.

(By “hierarchical,” we mean a layered, depth-causal memory state: higher layers condition on lower layers’ states, but not vice versa. This is distinct from hierarchy in the multi-timescale sense, e.g., HM-RNNs or Clockwork RNNs.)

See the figure for more details on the ablation. We believe the current SMT formulation is well suited for a densely connected memory state and rich RNN transition functions, which is a valid, if not more practical regime that nonlinear RNNs should be operating in. Modifying SMT to work with hierarchical memory states is an interesting future direction.

Truncated BPTT Baselines

We have updated the paper with Figure 24, that shows Truncated BPTT baselines. We find that TBPTT does not significantly improve performance over BPTT, even when bigger batch sizes are used. This may be because truncation reduces the credit horizon length, further hindering BPTT's already poor credit assignment. Truncated BPTT is not able to match SMT.

Ablate Tf

We have updated the paper with Figure 25, that ablates Tf on TinyStories and MNIST. We find that larger Tf is empirically better for our natural datasets.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.