- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper investigates how temporal recurrence changes the preferred allocation of computation between within-step depth, expert width, and parallel experts. They compare recurrent and non-recurrent models on Sokoban and FineWeb language modeling under fixed-compute. Recurrent models generally reach their best observed performance with two to four layers, while non-recurrent models benefit from substantially greater depth. Additional experiments examine aggregation, expert communication, architectural controls, planning probes, and hardware performance. The main contribution is a systematic allocation study showing that temporal recurrence can shift useful computation toward shallower, more parallel updates within the tested model families.
Critical strengths
S1: The compute-allocation framework provides a useful extension of earlier shallow-recurrence observations.
S2: The qualitative pattern appears across multiple budgets and two different domains.
S3: ConvGLU controls, aggregation ablations, and hardware measurements strengthen the empirical study.
Critical weaknesses
W1: FineWeb comparisons change streaming history, normalization, input injection, and cache handling alongside recurrence, preventing clean attribution.
W2: Exact FLOPs and compute mismatches are insufficiently documented. Language-model optima lack seed variability and receive uneven search coverage.
W3: Planning probes are correlational and use selected seeds and jointly solved levels, limiting mechanistic conclusions.
W4: Recurrent training remains substantially slower than Transformer training, despite reduced inference depth.
Questions for authors
Q1: Does the depth-allocation shift persist when streaming, normalization, input injection, and cache handling are matched?
Q2: What are the exact resource costs, and how stable are the preferred depths across seeds and longer training?
Q3: Do future-action probes and interventions on the carried state support the proposed planning mechanism?
Thank you for the detailed feedback, particularly for requesting the architectural ablation in the language modeling. These additional ablations led us to revise our interpretation of the results, as discussed in Point 1 of our general response. Below, we address your questions and concerns.
Points 1 and 2 of the general response address this issue. Apart from the recurrent feedback connection, the revised Recurrent Transformer differs from the default Transformer in sharing KV across experts and using streaming training with KV context carried across consecutive training windows. We ablate both choices in non-recurrent Transformers in Appendix B, Figure 10. These changes modestly affect performance, without changing the performance advantage of shallow Recurrent Transformers.
We also added normalization ablations in Appendix B, Figures 11–12, showing that postnorm can improve performance at shallow allocations while leading to earlier depth saturation in both model families. With learnable postnorm in both models, the Recurrent Transformer still reaches comparable performance with fewer layers and achieves a lower best observed validation loss than the non-recurrent Streaming Transformer.
We added analytical forward FLOPs per token and parameter counts to Appendix P.
In Table 7 (Appendix L) we added 2 additional seeds to a few runs of our small budget sweep for Recurrent and non recurrent Transformers and observed low variance.
To complement the probes, we added a periodic state-reset intervention. Please see Point 5 of our general response and Appendix I, Figure 21, for the details.
We discuss this at point 6 of the general response. We also note that our experiments demonstrate that substantial training runs are feasible: we trained a shallow recurrent model for 120B tokens in 3 days on 4 H100 (Appendix C). If a similar overhead persists at larger scales, we believe larger pretraining runs would also be feasible, although we did not measure it.
We have added experiments extending training to 4.8B tokens for several depth allocations of recurrent and non-recurrent Transformers, and to 120B tokens for a two-layer Recurrent Transformer and a 16-layer Transformer. Shallow recurrent models remain competitive with deeper Transformers over these extended training budgets. Please see Point 6 of our general response and Appendix C for details.