acceptodds
Under review as a conference paper at ICLR 2027

Temporal Recurrence Favors Fewer Layers

Abstract

In streaming tasks, recurrent models can carry latent computation across time, allowing each update to build on representations produced earlier. This raises a basic question: once temporal recurrence provides sequential computation across steps, how much depth is still needed within each step? Prior work has shown that recurrence can make shallow models competitive.We instead study this question as a compute-allocation problem, varying within-step depth, expert width, and the number of parallel experts per layer across several compute budgets. For each budget, we compare the best observed recurrent and non-recurrent allocations and the performance they achieve under approximately matched per-step computation. Across Sokoban and autoregressive FineWeb language modeling, we find that temporal recurrence shifts the best observed compute allocation toward substantially fewer layers, with comparable or better performance.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
temporal recurrence, mixture of experts, compute allocations, recurrent neural networks, shallow and deep architectures

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HRbTOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates how temporal recurrence changes the preferred allocation of computation between within-step depth, expert width, and parallel experts. They compare recurrent and non-recurrent models on Sokoban and FineWeb language modeling under fixed-compute. Recurrent models generally reach their best observed performance with two to four layers, while non-recurrent models benefit from substantially greater depth. Additional experiments examine aggregation, expert communication, architectural controls, planning probes, and hardware performance. The main contribution is a systematic allocation study showing that temporal recurrence can shift useful computation toward shallower, more parallel updates within the tested model families.

Critical strengths

S1: The compute-allocation framework provides a useful extension of earlier shallow-recurrence observations.

S2: The qualitative pattern appears across multiple budgets and two different domains.

S3: ConvGLU controls, aggregation ablations, and hardware measurements strengthen the empirical study.

Critical weaknesses

W1: FineWeb comparisons change streaming history, normalization, input injection, and cache handling alongside recurrence, preventing clean attribution.

W2: Exact FLOPs and compute mismatches are insufficiently documented. Language-model optima lack seed variability and receive uneven search coverage.

W3: Planning probes are correlational and use selected seeds and jointly solved levels, limiting mechanistic conclusions.

W4: Recurrent training remains substantially slower than Transformer training, despite reduced inference depth.

Questions for authors

Q1: Does the depth-allocation shift persist when streaming, normalization, input injection, and cache handling are matched?

Q2: What are the exact resource costs, and how stable are the preferred depths across seeds and longer training?

Q3: Do future-action probes and interventions on the carried state support the proposed planning mechanism?

1 comment

Thank you for the detailed feedback, particularly for requesting the architectural ablation in the language modeling. These additional ablations led us to revise our interpretation of the results, as discussed in Point 1 of our general response. Below, we address your questions and concerns.

FineWeb comparisons change streaming history, normalization, input injection, and cache handling alongside recurrence, preventing clean attribution

Points 1 and 2 of the general response address this issue. Apart from the recurrent feedback connection, the revised Recurrent Transformer differs from the default Transformer in sharing KV across experts and using streaming training with KV context carried across consecutive training windows. We ablate both choices in non-recurrent Transformers in Appendix B, Figure 10. These changes modestly affect performance, without changing the performance advantage of shallow Recurrent Transformers.

We also added normalization ablations in Appendix B, Figures 11–12, showing that postnorm can improve performance at shallow allocations while leading to earlier depth saturation in both model families. With learnable postnorm in both models, the Recurrent Transformer still reaches comparable performance with fewer layers and achieves a lower best observed validation loss than the non-recurrent Streaming Transformer.

Exact FLOPs and compute mismatches are insufficiently documented.

We added analytical forward FLOPs per token and parameter counts to Appendix P.

Language-model optima lack seed variability

In Table 7 (Appendix L) we added 2 additional seeds to a few runs of our small budget sweep for Recurrent and non recurrent Transformers and observed low variance.

Planning probes are correlational

To complement the probes, we added a periodic state-reset intervention. Please see Point 5 of our general response and Appendix I, Figure 21, for the details.

Recurrent training remains substantially slower

We discuss this at point 6 of the general response. We also note that our experiments demonstrate that substantial training runs are feasible: we trained a shallow recurrent model for 120B tokens in 3 days on 4 H100 (Appendix C). If a similar overhead persists at larger scales, we believe larger pretraining runs would also be feasible, although we did not measure it.

and longer training

We have added experiments extending training to 4.8B tokens for several depth allocations of recurrent and non-recurrent Transformers, and to 120B tokens for a two-layer Recurrent Transformer and a 16-layer Transformer. Shallow recurrent models remain competitive with deeper Transformers over these extended training budgets. Please see Point 6 of our general response and Appendix C for details.

Reviewer_Zw6dOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how temporal recurrence changes the allocation of computation across depth, expert width, and parallel experts. Experiments on Sokoban and FineWeb language modeling find that recurrent models favor fewer layers under approximately matched per-step compute budgets.

Critical strengths

None

Critical weaknesses
  1. The experimental settings are too limited to support the broader practical implications. Although the paper includes language modeling, it remains unclear whether the observed preference for fewer layers persists at larger model scales or translates into benefits on realistic downstream LLM tasks.

  2. The study does not derive sufficiently general or actionable conclusions from its observations. It is difficult to determine how a practitioner should transfer the findings to other architectures, domains, or tasks. An empirical contribution can be valuable without introducing a new method, but here the limited experimental scope and lack of transferable guidance leave the overall contribution insufficient for acceptance.

Overall, I recommend rejection. The paper offers a focused empirical report, but its practical relevance and the reach of its conclusions remain insufficiently established.

Questions for authors

What evidence supports extending the observed depth preference to larger models and downstream tasks? What concrete guidance can practitioners take from the results when choosing depth and recurrence for a different model or task?

1 comment

Thank you for your review. Below, we address your questions and concerns.

The experimental settings are too limited to support the broader practical implications… … it remains unclear whether the observed preference for fewer layers persists at larger model scales.

We have added scaling experiments that increase model size up to approximately 1.2B parameters and, separately, extend training to 120B tokens. Shallow recurrent models remain competitive with deeper Transformers across the tested scales. Please see Point 6 of our general response and Appendix C for details.

The study does not derive sufficiently general or actionable conclusions from its observations… …What concrete guidance can practitioners take from the results when choosing depth and recurrence for a different model or task?

Concrete guidance for practitioners. If a non-recurrent model benefits substantially from depth on a streaming task and inference latency matters, our results suggest testing a shallower recurrent model and reallocating the saved per-step computation to width or parallel experts. This may preserve performance while reducing latency. Compare these alternatives under approximately matched per-step compute, and assess whether the recurrent model meets the required performance with lower measured latency on the intended hardware.

Reviewer_LBvUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how temporal recurrence changes the allocation of approximately fixed per-step compute. A parallel-expert framework varies within-step depth, expert count, and width; the recurrent version carries latent state to the next environment step or token. On Sokoban and FineWeb, the best tested recurrent allocations generally use fewer layers than the best non-recurrent ones. In one FineWeb setting, a two-layer recurrent allocation reaches roughly the loss of the best sixteen-layer non-recurrent allocation. Probes and timing measurements provide additional evidence about planning-related representations and execution costs.

Critical strengths
  1. The allocation sweeps ask a better question than whether one shallow recurrent model can beat one deep baseline. Reallocating computation into width and parallel experts makes the observed depth preference interesting. The trend appears in both Sokoban and language modeling, rather than only one task.

  2. The ConvGLU control improves the Sokoban comparison by reducing the architectural differences between recurrent and non-recurrent agents. The paper also reports training throughput rather than inferring efficiency solely from fewer layers. These are useful checks on the interpretation.

Critical weaknesses
  1. The language-model comparison changes several factors alongside recurrence. The recurrent models carry latent state and KV context across training windows; the non-recurrent models train on independent sequences. The recurrent architecture additionally uses token-embedding inputs at each depth level, output normalization, and shared KV across experts. Matching average attention length does not match the available history or training procedure. Consequently, the two-versus-sixteen-layer result does not isolate the effect of the recurrent feedback connection. This is central because the contribution is supposed to explain how recurrence changes compute allocation, not merely present two model families with different preferred depths.

  2. The paper does not adequately distinguish reduced need for depth from difficulty training deep recurrent updates. Truncated BPTT can affect deeper recurrence differently, even if the non-recurrent baseline benefits from more layers. The planning probes do not resolve this: they use the best seed per allocation and levels solved by every compared policy, and the models receive new observations over the steps where probe recall improves. That is evidence consistent with continued planning, not an intervention showing it. The approximate compute proxy also needs configuration-level validation before drawing a strong allocation conclusion. Taken together, the results establish a promising empirical trend more clearly than its explanation.

Questions for authors
  1. Can the new arch be trained in parallel. If not, how can we make the training cost acceptable.

  2. In the Figure 5 setting with loss near 3.51, what is the shallow/deep frontier when both model families use the same continuous training streams, retained KV context, embedding inputs, output normalization, and per-layer KV sharing, with only the cross-token latent feedback enabled or disabled? This is the control needed to attribute the depth shift to recurrence rather than to the other design changes.

  3. Table 4 uses a 64-token recurrent training window. For one shallow and one deep recurrent allocation within the same budget, does increasing the BPTT window change the gap when total training tokens and tokens per update are held fixed? Please also show training and validation curves. If the deep model improves disproportionately, the preferred depth may depend on the optimization setup rather than a reduced need for within-step computation.

  4. For the 2-layer recurrent and s16-layer non-recurrent allocations defining the Figure 5 comparison, what are the actual architecture settings, parameter counts, forward FLOPs, and training FLOPs per token? Please include the embedding/readout and the different KV paths in the accounting.

1 comment

We thank the reviewer for the feedback and for requesting controlled ablations that led us to revise our interpretation of the results. Please see Point 1 of our general response. Below, we address your questions and comments.

The language-model comparison changes several factors alongside recurrence.

Points 1 and 2 of the general response address this issue. Apart from the recurrent feedback connection, the revised Recurrent Transformer differs from the default Transformer in sharing KV across experts and using streaming training with KV context carried across consecutive training windows. We ablate both choices in non-recurrent Transformers in Appendix B, Figure 10. These changes modestly affect performance, without changing the performance advantage of shallow Recurrent Transformers.

We also added normalization ablations in Appendix B, Figures 11–12, showing that postnorm can improve performance at shallow allocations while leading to earlier depth saturation in both model families. With learnable postnorm in both models, the Recurrent Transformer still reaches comparable performance with fewer layers and achieves a lower best observed validation loss than the non-recurrent Streaming Transformer.

Truncated BPTT can affect deeper recurrence differently

We agree with this. But our revised Recurrent Transformer continues to improve with depth. Plus, in Appendix D, we increase the training sequence length from 64 to 128, doubling the BPTT window, and the validation loss improves. This provides evidence that our training procedure with BPTT can successfully optimize longer recurrent computations.

Can the new arch be trained in parallel. If not, how can we make the training cost acceptable

We discuss this at point 6 of the general response. We also note that our experiments demonstrate that substantial training runs are feasible: we trained a shallow recurrent model for 120B tokens in 3 days on 4 H100 (Appendix C). If a similar overhead persists at larger scales, we believe larger pretraining runs would also be feasible, although we did not measure it.

For RL post-training, shallow recurrent models may benefit from lower rollout-generation latency, since both recurrent models and Transformers generate tokens autoregressively. But we have not evaluated the potential post-training speedup.

Further efficiency improvements may be possible through implementation changes and alternative training approaches, such as distilling deep Transformer representations into recurrent models or introducing chunk level recurrence.

The planning probes do not resolve this… …That is evidence consistent with continued planning, not an intervention showing it.

To complement the probes, we added a periodic state-reset intervention. Please see Point 5 of our general response and Appendix I, Figure 21, for the details.

forward FLOPs

We added analytical forward FLOPs per token and parameter counts to Appendix P. The FLOP estimates include the readout projection, and the parameter counts include token embeddings and the readout. Sharing KV across experts adds an aggregation of their K/V proposals, which adds little to the forward FLOP count.

Table 4 uses a 64-token recurrent training window. For one shallow and one deep recurrent allocation within the same budget, does increasing the BPTT window change the gap when total training tokens and tokens per update are held fixed?

We report this experiment in Appendix D, Figure 16. We compare BPTT windows of 32, 64 (default), and 128 tokens across depth allocations, keeping the total training-token budget and the number of supervised tokens per update (131,072) fixed. Increasing the BPTT window from 64 to 128 improves validation loss at all tested depths. Reducing the window from 64 to 32 worsens performance at all tested depths.

Comments on the paper

3 comments
AuthorsOpenReview ↗

general response to reviewers

We thank reviewers LBvU and HRbT for requesting more controlled comparisons between the recurrent and non-recurrent Transformers. These experiments helped us separate early depth saturation from the ability of shallow recurrent models to match deeper non-recurrent models, leading us to revise our interpretation.

In language modeling, we find that the early depth saturation observed in our original experiments depends on architectural choices, particularly normalization. The revised Recurrent Transformer continues to improve with greater depth. We therefore no longer present earlier saturation as a general consequence of recurrence. Appendix B reports the normalization ablation (Figure 11) underlying this revision.

The finding that recurrent models can reach the performance of substantially deeper non-recurrent models with fewer within-step layers, under approximately matched per-step compute, remains well-supported in the tested settings. We have revised the main text to focus on this distinction.

1. Main-text revisions.

The main text now emphasizes that recurrent models can reach a given performance with substantially fewer within-step layers than non-recurrent models under approximately matched per-step compute. The central research question remains how temporal recurrence changes the allocation of computation between within-step depth and parallel capacity. All newly added or edited text is red. We also updated Figure 3 to show the minimum observed depth among tested allocations needed to reach a given performance for recurrent and non-recurrent models.

We updated the Recurrent Transformer architecture used in the main text. It uses prenorm within each Transformer block, as in the non-recurrent baseline, without the additional normalization previously applied to each expert’s output. To introduce temporal recurrence, we RMS-normalize the previous token’s final residual stream, concatenate it with the current token embedding, and project the result back to the hidden dimension at the first layer. Subsequent layers receive only the preceding layer’s residual stream; unlike the previous recurrent variant, they do not receive additional token-embedding inputs.

2. Shared-KV and streaming ablations (Reviewers LBvU and HRbT).

Beyond the recurrent input connection described above, we examine two further differences from the default Transformer: KV sharing across experts within each layer and streaming training with KV context carried across consecutive training windows. The default Transformer uses separate keys and values for each expert and trains on independent sequences.

Appendix B, Figure 10, compares the default Transformer with a Streaming Transformer and then additionally introduces KV sharing. KV sharing produces similar or modestly worse validation loss, particularly at shallow allocations, while streaming modestly improves performance at greater depths. The recurrent models still retain their performance advantage under these ablations.

3. Normalization ablations (Reviewers LBvU and HRbT).

Appendix B, Figures 11 and 12, examines normalization choices for both the Recurrent Transformer and the non-recurrent Streaming Transformer. Adding RMSNorm to each expert’s output leads to earlier depth saturation in both model families: the postnorm variants reach their best observed performance at shallower depths and perform worse at greater depths. With learnable postnorm in both models, the Recurrent Transformer still reaches comparable performance with fewer layers and achieves a lower best observed validation loss. The revised main-text Recurrent Transformer, without this additional output normalization, continues to improve with depth. These results distinguish the architecture-dependent saturation pattern from the shallow-performance advantage of recurrence.

The blue curve in Figure 11, labeled "RT w/ parameter-free postnorm", uses the output-normalization scheme from the previous Recurrent Transformer version.

4. Scalability (Reviewers Zw6d and HRbT).

To test whether the observed patterns persist as the training-token budget and model size increase, we report three additional experiments in Appendix C.

a) Longer training across depth allocations. For the small FineWeb compute budget, we evaluate several depth allocations around the best-performing configurations of the Recurrent Transformer (RT) and Transformer, extending training to 4.8B tokens. The relative performance of the tested allocations remain broadly consistent across extended training (Figure 13).

b) Extended training of a shallow recurrent model. To test whether a shallow Recurrent Transformer remains competitive with a Transformer over substantially longer training, we train a two-layer RT and a 16-layer Transformer for 120B tokens (roughly 100x the original token budget). The RT performs better early in training, while the Transformer overtakes it later. The final cross-entropy gap is approximately 0.019 (Figure 14).

c) Larger models. To test whether shallow recurrent models remain competitive at larger model sizes, we scale the two-layer Recurrent Transformer by increasing the number of experts, reaching approximately 1.2B parameters at most. We scale the Transformer baseline by increasing its depth and width inspired by Cerebras-GPT [1]. The shallow recurrent models remain competitive across the tested scales (Figure 15).

[1] Dey, N., Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster. 2023

5. Interpretability and causal intervention in Sokoban (Reviewer HRbT, also relevant to Reviewer LBvU).

We added a reset intervention at the end of Appendix I, Figure 21, in which we reset the full recurrent state every five environment steps. The solve rate drops by 16 percentage points for shallow L=2, compared with 7.5 percentage points for deeper L=4. The shallower model also no longer catches up with the deeper model in planning performance, as measured by recall, which is consistent with our interpretation that it relies more heavily on planning across time steps.

We also highlight prior evidence that the planning representations learned by a ConvLSTM-based agent causally influence its behavior. Bush et al. [2] demonstrated that targeted interventions on these representations can steer the agent’s behavior in interpretable ways. Given the similarities in architecture and training procedure, we believe these findings support our interpretation.

[2] Bush, T, et al. Interpreting Emergent Planning in Model-Free Reinforcement Learning. ICLR2025.

6. Lower training throughput (Reviewers LBvU and HRbT).

We consider the roughly 2–4x training slowdown measured for our shallow recurrent configurations manageable for experimentation, given their potential for lower inference latency and greater architectural flexibility. Since submission, we have also substantially improved the throughput reported in Table 6 for (L > 1). Further throughput gains may be possible through implementation improvements or adaptations of training approaches such as chunkwise recurrence and supervised memory training [3].

[3] Kumar, A., and Isola, P. Pretraining Recurrent Networks without Recurrence. 2026

7. Missing details (Reviewers LBvU and HRbT).

Missing details with respect to compute and parameter count are added to Appendix P.

We also want to highlight that in the small- and medium-budget FineWeb comparisons, recurrent models with L=2L=2 achieve lower observed validation losses than the best tested default Transformer allocations, which use L=16L=16 and L=13L=13, respectively. In the long-context setting, a recurrent model with L=4L=4 also achieves a lower loss than the best default Transformer allocation, which uses L=16L=16. Recurrent Transformers continue to benefit from additional depth in all three settings. In Sokoban, the best L=2L=2 recurrent allocation outperforms all tested non-recurrent allocations at each compute budget, with the best non-recurrent allocations using L=8L=8 for the small and medium budgets and L=16L=16 for the large budget.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.