- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces Gated Recurrent Transformers (GRT), a weight-shared, depth-recurrent Transformer architecture built upon the prelude–recurrent core–coda paradigm. To improve the expressiveness of repeated applications of the same Transformer block, the authors incorporate an elementwise (channel-wise) gating mechanism conditioned on both the current hidden state and the prelude representation, alongside Gaussian state/gate noise and uniform depth sampling during training. They also explore inference-time extensions enabled by this setup, including self-speculative decoding across recurrence depths and a post-training channel-wise KV cache compression codec, evaluating the approach on GPT-2 Small, Medium, and Large scales.
Critical strengths
1.Practical Engineering Integration: The paper combines depth-wise weight sharing, prelude re-injection, elementwise gating, and random depth training into a unified pipeline that supports elastic depth selection and self-speculative decoding at inference time.
2.Extensive Empirical Logging: The authors conduct a broad set of experiments and ablation runs across three GPT-2 model scales, documenting training dynamics, gate activation statistics, and KV cache compression strategies.
Critical weaknesses
1.Incremental Architectural Modification Without Deep Mechanistic Justification: At its core, the main architectural delta of GRT over prior depth-recurrent Transformers (e.g., Geiping et al., 2025) is the addition of an elementwise gate on the depth-wise residual update. However, the paper fails to provide a rigorous theoretical or mechanistic explanation for why elementwise gating is fundamentally necessary or effective in recurrent depth. By contrast, recent works that refine coarse updates into fine-grained channel-wise/elementwise gating (such as KDA's channel-wise decay refinement over GDN in recurrent sequence modeling) or works that explicitly parameterize and modulate cross-layer residual streams (such as AttnRes or mHC / Hyper-Connections) clearly articulate why fine-grained/multi-stream modulation is needed—analyzing how it resolves subspace bottlenecks, controls spectrum/gradient flow, or decouples memory retention across channels. Here, the elementwise gate is simply bolted onto the base recurrent architecture heuristic-style, accompanied only by descriptive post-hoc summary statistics rather than a deep analysis of why elementwise modulation works where simpler residual updates fail.
2.Burying Unfavorable Trade-offs and Limitations in the Appendix Hurts Readability and Transparency: A major issue with the manuscript is its presentation strategy. Many critical factors and practical trade-offs that are unfavorable to the proposed method—ranging from actual wall-clock latency overheads and true FLOPs/memory scaling under realistic batch sizes to the relative contribution of the gate versus existing prelude re-injection and the fidelity loss of KV compression—are tucked away across a sprawling appendix rather than being transparently confronted in the main text. This selective framing in the main body makes the paper difficult to read critically and obscures the true cost-benefit profile of the architecture.
3.Unconvincing Scaling Trends Against Standard Dense Baselines: Even with the proposed gating mechanism, the model still lags behind standard dense Transformers at Medium and Large scales under comparable compute budgets, raising questions about whether depth-wise gating truly closes the expressivity gap as model capacity grows.
Questions for authors
1.Why Elementwise Gating Specifically? Beyond empirical loss deltas on small models, can the authors provide a deeper theoretical or mechanistic analysis—similar in spirit to how KDA justifies channel-wise gating over GDN, or how residual-stream modulation works like AttnRes/mHC analyze cross-layer feature routing—explaining what specific representational or optimization bottleneck in recurrent depth requires per-element gating as opposed to scalar, per-head, or structured residual mixing?
2.Transparent Reporting of Trade-offs in the Main Text: Can the authors revise the main text to upfront and transparently integrate the unfavorable system and scaling trade-offs currently relegated to the appendix, so that readers get an accurate picture of the method's practical limitations without having to excavate the supplementary material?
3 comments
2/2
On scaling against dense (W3).
We appreciate the reviewer's interest in the scaling trend. GRT's gap to dense closes as data grows, and at GPT-2 XL depth GRT matches dense with 42% of its parameters and 0.71x its training compute.
Table R9. GRT minus dense validation loss (nats) for Small and Large. The 0.5x to 2x rows are measured (Figure 6), and the 2.5x row is extrapolated (Figure R2). Negative means GRT is better.
| Data budget | Small | Large |
|---|---|---|
| 0.5x | −0.10 | +0.10 |
| 1x | −0.01 | +0.06 |
| 2x | −0.01 | +0.02 |
| 2.5x (extrapolated) | −0.02 | +0.01 |
Traced back to our standard budget, GRT at XL depth would trail dense by about as much as at Large (+0.05 vs. +0.06 nats), and with 2.5x data it overtakes it.
Table R10. Validation loss at GPT-2 XL depth. The 2.5x row is measured, and the 0.5x to 2x rows trace the measured points back with the Large curve shapes (Figure R2).
| Data budget | Dense | GRT | GRT − dense |
|---|---|---|---|
| 0.5x (extrapolated) | 2.739 | 2.825 | +0.09 |
| 1x (extrapolated) | 2.639 | 2.685 | +0.05 |
| 2x (extrapolated) | 2.609 | 2.615 | +0.01 |
| 2.5x (measured) | 2.605 | 2.601 | −0.004 |
Table R4. Validation loss at GPT-2 XL depth, 24.6B tokens (2.5x the budget of Table 1).
| Model | Params | Train compute | Val loss |
|---|---|---|---|
| Dense GPT-2 XL | 1.56B | 1.00x | 2.605 |
| GRT 2+10x4+6 | 0.65B | 0.71x | 2.601 |
https://anonymous.4open.science/r/GRT-ICLR-2027-Rebuttal-Figures-5454/fig_extrapolation-1.png
Figure R2. Data scaling of Small and Large (measured, solid and dashed), extrapolated to 2.5x data (dotted, hollow markers), with the measured GPT-2 XL runs at 2.5x (stars) traced back with the Large curve shapes.
This trend matches the scaling laws for looped language models in Parcae [4], which find that at a fixed FLOP budget, looping and data should be increased together. We test this directly with the XL where we noticed that with 2.5x more data than our standard budget, the gap seen at Large disappears. Overall, with our proposed architecture we are able to beat the baseline with fewer training FLOPs.
References.
[1] Kimi Team. Attention Residuals. arXiv 2603.15031, 2026.
[2] Xie et al. mHC: Manifold-Constrained Hyper-Connections. arXiv 2512.24880, 2025.
[3] Kimi Team. Kimi Linear. arXiv 2510.26692, 2025.
[4] Prairie et al. Parcae: Scaling Laws For Stable Looped Language Models. arXiv 2604.12946, 2026.
Dear Reviewer SRjX,
Thank you again for your detailed review and thoughtful questions. We posted a detailed rebuttal to your review, split across two comments. As the discussion period closes shortly, we would be very grateful if you could take a look and let us know whether our response resolves your concerns, or raise any remaining points while there is still time for us to respond.
We would also highly appreciate it if you could re-consider your score in light of the additional experiments and revisions, which include a mechanistic analysis of the elementwise gate, the gate-granularity comparison, moving the latency, memory and FLOPs trade-offs into the main text with a new Limitations paragraph, and GPT-2 XL-depth training where GRT matches dense with 42% of the parameters.
Thank you again for your time and effort.
1/2
We thank the reviewer for recognizing the "practical engineering integration" that supports "elastic depth selection and self-speculative decoding", and the "broad set of experiments and ablation runs across three GPT-2 model scales".
On why an elementwise gate (W1, Q1).
We thank the reviewer for asking for a mechanistic justification, which led us to connect the gate to recent work. Section 3 motivates the gate by the need for one block to act as different layers at different recurrences. The gate also has three properties that connect it to the works the reviewer cites, now explained in Section 3 and Appendix H.
We tested point 3 directly by comparing gate granularities (Table 3 in the revised PDF).
Table R3. Validation loss by gate type, 20k steps.
Gating channel groups independently is what matters: a per-head gate matches the elementwise one, while a single scalar gate per token is 0.02 nats worse at both Small and Medium.
On trade-offs in the appendix (W2, Q2).
Thank you for this suggestion. We have moved the main trade-offs into the main text of the revised PDF. Table 1 now reports training and inference FLOPs for every method. Section 4 has a new Latency and Memory paragraph covering the compiled overhead, early exit against measured latency (Figure 3c), self-speculative latency and memory up to a 7B-shaped model, and Section 5 adds the factorial ablation (Table 3). The conclusion now has a Limitations paragraph. The full latency, memory and KV-compression tables remain in Appendix C, including the self-speculative latency below (Table 9).
Table R6. Decode latency (ms/token, compiled, batch 4, 1024-token prompt, 128 new tokens), relative to dense in parentheses. The last row is estimated as full-depth latency divided by the measured task-level speedups (range over ten tasks, Table 6).