acceptodds
Under review as a conference paper at ICLR 2027

Gated Recurrent Transformers: Expressive Depth Through Recurrent Modulation

Abstract

Scaling transformer language models creates an inherent tension between expressivity and memory efficiency. While unique weights across layers preserve functional specialization—from input-grounding to abstract refinement—they incur a substantial memory footprint. Conversely, standard depth-sharing enforces uniform transformations that collapse representational diversity and degrade modeling quality. We introduce the GATED RECURRENT TRANSFORMER (GRT), a recurrent depth transformer inspired by gated recurrent neural networks, where a lightweight projection and an elementwise update gate—conditioned on the hidden state, a fixed prelude output, and noise resampled at every step—modulate the recurrent update. This allows the model to specialize the input to the same few layers across recurrences, rather than requiring many unique layers to achieve functional diversity. Under an isoFLOPS constraint, a 3-layer GRT matches the accuracy of a 12-layer GPT-2 Small baseline with similar training and inference FLOPs, and leads a wide range of existing recurrent depth approaches in all nine scale-by-budget cells; at medium and large scale it approaches dense quality at the standard token budget and overtakes it at medium scale once that budget is doubled. Under an isoPARAMS constraint, deeper recurrence achieves a 2.76 validation loss versus 2.84 for a non-recurrent counterpart at matched parameter and data budget. Our results demonstrate that adaptive depth reuse is a principled strategy for trading parameters for quality. Intermediate recurrences additionally support lossless greedy self-speculative decoding without a separate draft model. Moreover, we propose an optional lightweight post-training optimization, Channel-wise KV Codec, to compress recurrent KV caches to the storage budget of a single K/V pair per token.

Primary area:
foundation or frontier models, including LLMs
Keywords:
recurrent depth, looped transformers, parameter sharing, adaptive computation, gating, language modeling, efficient inference, test-time compute

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_SRjXOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces Gated Recurrent Transformers (GRT), a weight-shared, depth-recurrent Transformer architecture built upon the prelude–recurrent core–coda paradigm. To improve the expressiveness of repeated applications of the same Transformer block, the authors incorporate an elementwise (channel-wise) gating mechanism conditioned on both the current hidden state and the prelude representation, alongside Gaussian state/gate noise and uniform depth sampling during training. They also explore inference-time extensions enabled by this setup, including self-speculative decoding across recurrence depths and a post-training channel-wise KV cache compression codec, evaluating the approach on GPT-2 Small, Medium, and Large scales.

Critical strengths

1.Practical Engineering Integration: The paper combines depth-wise weight sharing, prelude re-injection, elementwise gating, and random depth training into a unified pipeline that supports elastic depth selection and self-speculative decoding at inference time.

2.Extensive Empirical Logging: The authors conduct a broad set of experiments and ablation runs across three GPT-2 model scales, documenting training dynamics, gate activation statistics, and KV cache compression strategies.

Critical weaknesses

1.Incremental Architectural Modification Without Deep Mechanistic Justification: At its core, the main architectural delta of GRT over prior depth-recurrent Transformers (e.g., Geiping et al., 2025) is the addition of an elementwise gate on the depth-wise residual update. However, the paper fails to provide a rigorous theoretical or mechanistic explanation for why elementwise gating is fundamentally necessary or effective in recurrent depth. By contrast, recent works that refine coarse updates into fine-grained channel-wise/elementwise gating (such as KDA's channel-wise decay refinement over GDN in recurrent sequence modeling) or works that explicitly parameterize and modulate cross-layer residual streams (such as AttnRes or mHC / Hyper-Connections) clearly articulate why fine-grained/multi-stream modulation is needed—analyzing how it resolves subspace bottlenecks, controls spectrum/gradient flow, or decouples memory retention across channels. Here, the elementwise gate is simply bolted onto the base recurrent architecture heuristic-style, accompanied only by descriptive post-hoc summary statistics rather than a deep analysis of why elementwise modulation works where simpler residual updates fail.

2.Burying Unfavorable Trade-offs and Limitations in the Appendix Hurts Readability and Transparency: A major issue with the manuscript is its presentation strategy. Many critical factors and practical trade-offs that are unfavorable to the proposed method—ranging from actual wall-clock latency overheads and true FLOPs/memory scaling under realistic batch sizes to the relative contribution of the gate versus existing prelude re-injection and the fidelity loss of KV compression—are tucked away across a sprawling appendix rather than being transparently confronted in the main text. This selective framing in the main body makes the paper difficult to read critically and obscures the true cost-benefit profile of the architecture.

3.Unconvincing Scaling Trends Against Standard Dense Baselines: Even with the proposed gating mechanism, the model still lags behind standard dense Transformers at Medium and Large scales under comparable compute budgets, raising questions about whether depth-wise gating truly closes the expressivity gap as model capacity grows.

Questions for authors

1.Why Elementwise Gating Specifically? Beyond empirical loss deltas on small models, can the authors provide a deeper theoretical or mechanistic analysis—similar in spirit to how KDA justifies channel-wise gating over GDN, or how residual-stream modulation works like AttnRes/mHC analyze cross-layer feature routing—explaining what specific representational or optimization bottleneck in recurrent depth requires per-element gating as opposed to scalar, per-head, or structured residual mixing?

2.Transparent Reporting of Trade-offs in the Main Text: Can the authors revise the main text to upfront and transparently integrate the unfavorable system and scaling trade-offs currently relegated to the appendix, so that readers get an accurate picture of the method's practical limitations without having to excavate the supplementary material?

3 comments

We thank the reviewer for recognizing the "practical engineering integration" that supports "elastic depth selection and self-speculative decoding", and the "broad set of experiments and ablation runs across three GPT-2 model scales".

On why an elementwise gate (W1, Q1).

We thank the reviewer for asking for a mechanistic justification, which led us to connect the gate to recent work. Section 3 motivates the gate by the need for one block to act as different layers at different recurrences. The gate also has three properties that connect it to the works the reviewer cites, now explained in Section 3 and Appendix H.

  1. It is a learned, per-channel mixture over recurrences, as in AttnRes [1]. Each step keeps a share g of the old state and writes a share 1 − g of the block output, so after R steps the state is a weighted average of the prelude output and every block output, with weights that are non-negative and sum to 1 for every token and channel (Appendix H). Each channel therefore decides how much it reads from each recurrence. A plain residual stream sums all block outputs with fixed unit weights, which AttnRes identifies as the cause of hidden-state growth that dilutes each layer's contribution.
  2. It keeps an identity path with bounded gain, as in mHC [2]. An average can never exceed the largest value it averages, so no channel grows beyond its inputs however many recurrences run. In g ⊙ h the old state is simply scaled by the gate, so a channel with g ≈ 1 passes its value and gradient straight to the next step without going through the shared block, as in LSTM and GRU gates (Appendix H). This matters more with shared weights, since the same block is applied R times, and the +4 bias starts training near this identity path (g ≈ 0.98).
  3. It gives each channel its own retention, as KDA does over GDN [3]. KDA lets each channel decay at its own rate over time, and GRT does the same over depth. In GRT Large, the share of copy-saturated channels (g > 0.95) grows from 19.7% to 28.8% across the six steps while the other channels keep being rewritten (Figure 14b). The applied update norm falls from 17.6 to 6.5 while the block's proposal norm stays roughly constant, so the decay comes from the gate closing (Figure 15). A scalar gate would force one retention level on all channels.

We tested point 3 directly by comparing gate granularities (Table 3 in the revised PDF).

Table R3. Validation loss by gate type, 20k steps.

GateSmallMedium
Per-token (scalar)3.162.91
Per-head3.14In Progress
Elementwise (ours)3.142.89

Gating channel groups independently is what matters: a per-head gate matches the elementwise one, while a single scalar gate per token is 0.02 nats worse at both Small and Medium.

On trade-offs in the appendix (W2, Q2).

Thank you for this suggestion. We have moved the main trade-offs into the main text of the revised PDF. Table 1 now reports training and inference FLOPs for every method. Section 4 has a new Latency and Memory paragraph covering the compiled overhead, early exit against measured latency (Figure 3c), self-speculative latency and memory up to a 7B-shaped model, and Section 5 adds the factorial ablation (Table 3). The conclusion now has a Limitations paragraph. The full latency, memory and KV-compression tables remain in Appendix C, including the self-speculative latency below (Table 9).

Table R6. Decode latency (ms/token, compiled, batch 4, 1024-token prompt, 128 new tokens), relative to dense in parentheses. The last row is estimated as full-depth latency divided by the measured task-level speedups (range over ten tasks, Table 6).

DecodingMediumLarge
Dense GPT-22.54 (1.00x)3.33 (1.00x)
GRT, full depth2.81 (1.11x)3.67 (1.10x)
GRT + self-speculative2.02 to 2.38 (0.80 to 0.94x)2.51 to 3.08 (0.75 to 0.93x)

On scaling against dense (W3).

We appreciate the reviewer's interest in the scaling trend. GRT's gap to dense closes as data grows, and at GPT-2 XL depth GRT matches dense with 42% of its parameters and 0.71x its training compute.

Table R9. GRT minus dense validation loss (nats) for Small and Large. The 0.5x to 2x rows are measured (Figure 6), and the 2.5x row is extrapolated (Figure R2). Negative means GRT is better.

Data budgetSmallLarge
0.5x−0.10+0.10
1x−0.01+0.06
2x−0.01+0.02
2.5x (extrapolated)−0.02+0.01

Traced back to our standard budget, GRT at XL depth would trail dense by about as much as at Large (+0.05 vs. +0.06 nats), and with 2.5x data it overtakes it.

Table R10. Validation loss at GPT-2 XL depth. The 2.5x row is measured, and the 0.5x to 2x rows trace the measured points back with the Large curve shapes (Figure R2).

Data budgetDenseGRTGRT − dense
0.5x (extrapolated)2.7392.825+0.09
1x (extrapolated)2.6392.685+0.05
2x (extrapolated)2.6092.615+0.01
2.5x (measured)2.6052.601−0.004

Table R4. Validation loss at GPT-2 XL depth, 24.6B tokens (2.5x the budget of Table 1).

ModelParamsTrain computeVal loss
Dense GPT-2 XL1.56B1.00x2.605
GRT 2+10x4+60.65B0.71x2.601

https://anonymous.4open.science/r/GRT-ICLR-2027-Rebuttal-Figures-5454/fig_extrapolation-1.png

Figure R2. Data scaling of Small and Large (measured, solid and dashed), extrapolated to 2.5x data (dotted, hollow markers), with the measured GPT-2 XL runs at 2.5x (stars) traced back with the Large curve shapes.

This trend matches the scaling laws for looped language models in Parcae [4], which find that at a fixed FLOP budget, looping and data should be increased together. We test this directly with the XL where we noticed that with 2.5x more data than our standard budget, the gap seen at Large disappears. Overall, with our proposed architecture we are able to beat the baseline with fewer training FLOPs.

References.
[1] Kimi Team. Attention Residuals. arXiv 2603.15031, 2026.
[2] Xie et al. mHC: Manifold-Constrained Hyper-Connections. arXiv 2512.24880, 2025.
[3] Kimi Team. Kimi Linear. arXiv 2510.26692, 2025.
[4] Prairie et al. Parcae: Scaling Laws For Stable Looped Language Models. arXiv 2604.12946, 2026.

Dear Reviewer SRjX,

Thank you again for your detailed review and thoughtful questions. We posted a detailed rebuttal to your review, split across two comments. As the discussion period closes shortly, we would be very grateful if you could take a look and let us know whether our response resolves your concerns, or raise any remaining points while there is still time for us to respond.

We would also highly appreciate it if you could re-consider your score in light of the additional experiments and revisions, which include a mechanistic analysis of the elementwise gate, the gate-granularity comparison, moving the latency, memory and FLOPs trade-offs into the main text with a new Limitations paragraph, and GPT-2 XL-depth training where GRT matches dense with 42% of the parameters.

Thank you again for your time and effort.

Reviewer_1AiEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Gated Recurrent Transformers, which reuse the same transformer blocks across depth to reduce the number of parameters. A learned gate controls how much the hidden state changes at each recurrence, while conditioning on the original input and adding noise during training. The method achieves comparable performance with less parameters at small scale. The paper also proposes using intermediate recurrences for self-speculative decoding and compressing the KV cache across recurrences to further reduce inference memory.

Critical strengths
  1. The results look promising: the method achieves comparable quality with substantially fewer parameters and less peak memory during inference. I also appreciate the comparisons with other recurrent models and the ablations showing the contribution of each component.
  2. Self-speculative decoding is a very useful property of the approach especially for faster on device inference (with batch size 1)
Critical weaknesses
  1. The proposed approach is slower in wall-clock time during full-depth inference.
  2. The “isoFLOPs” results use ~28% more forward FLOPs at small scale and 6% more at medium/large scale compared to the dense baseline, so the comparison is not really at the same compute budget.
  3. The scale of the experiments is limited: the pretraining context is only 1,024 tokens (for comparison, OLMo 3 7B was pretrained with an 8K context), and all models have fewer than 1B parameters.
  4. Inference timing benchmark is limited: a 1,024-token prompt + 128 generated tokens. It is particularly a limitation for proposed codec approach to compress KV cache across shared block.
  5. Codec should be trained separately from the main model

Therefore, it is hard to understand how well the method scales and whether the extra overhead needed to make recurrent weight sharing work would become a problem for larger models and longer contexts.

Questions for authors
  1. Given that the authors have access to at least two H200 GPUs, would it be possible to benchmark inference with larger models using randomly initialized weights? This would help us understand how inference time and memory usage scale without requiring additional training. I think this comparison with a dense baseline would strengthen the paper significantly.
  2. This is not a question rather than a general comment: I am wondering if authors tried to decompose the KV into shared component + low rank approximation for each recurrent step. This would look like: K(r)=Kshared+AK(r)BK(r)K^{(r)} = K_{\text{shared}} + A_K^{(r)}B_K^{(r)}, same for values. I guess this might be interesting because then Codec would not be needed and it will simplify the approach.
3 comments

We thank the reviewer for finding the results "promising", with "substantially fewer parameters and less peak memory during inference", and for pointing out that self-speculative decoding is "a very useful property ... for faster on device inference".

On wall-clock latency (W1).

We agree, and thank the reviewer for highlighting the wall-clock cost. Under torch.compile, full-depth GRT decodes 10% slower than dense at Large (3.67 vs. 3.33 ms/token) and 11% slower at Medium (2.81 vs. 2.54), while at Large it uses 62% fewer parameters and 59% less peak memory (Table 7). Self-speculative decoding, which the reviewer highlighted, more than recovers this overhead.

Table R6. Decode latency (ms/token, compiled, batch 4, 1024-token prompt, 128 new tokens), relative to dense in parentheses. The last row is estimated as full-depth latency divided by the measured task-level speedups (range over ten tasks, Table 6).

DecodingMediumLarge
Dense GPT-22.54 (1.00x)3.33 (1.00x)
GRT, full depth2.81 (1.11x)3.67 (1.10x)
GRT + self-speculative2.02 to 2.38 (0.80 to 0.94x)2.51 to 3.08 (0.75 to 0.93x)

Early exit gives a second way to beat dense latency from the same checkpoint. We also timed every exit depth in the setting of Table 7 (batch 4, 1024-token prompt, 128 new tokens) and report it relative to the dense model of each scale (Table 8).

Table R7. Decode latency relative to dense (dense = 1.00).

DecodingMedium (R=4)Large (R=6)XL (R=4)
Dense GPT-21.00x1.00x1.00x
GRT, full depth1.08x1.07x1.04x
GRT, one step early0.86x0.92x0.83x

Exiting one step early makes GRT faster than dense at every scale. Relative to its own full depth, it saves 20% at Medium and 14% at Large, which is more than the 10 to 11% compiled overhead in Table 7. At Large that costs only 0.006 nats. At Medium it costs 0.06 nats. We plotted Figure 3b again with latency in place of FLOPs (Figure 3c in the revised PDF): https://anonymous.4open.science/r/GRT-ICLR-2027-Rebuttal-Figures-5454/fig_earlyexit_latency-1.png

Figure R3. Figure 3b with decode latency of each scale (stars: dense GPT-2).

On the "isoFLOPs" label (W2).

Thank you for pointing this out. We agree that the label is misleading, and we will rename the regime isoDepth, since it matches the number of block applications. At inference, GRT uses 28% more FLOPs than dense at Small and 6% more at Medium and Large. Training compute is a different matter. Because the depth is sampled uniformly during training, GRT trains with 0.78x, 0.73x and 0.69x the compute of the dense model at Small, Medium and Large. Despite using less training compute, GRT's loss is competitive with baseline. Table 1 in the revised PDF reports both numbers for every method.

On benchmarking larger models and longer contexts (W4, Q1).

We appreciate the suggestion to benchmark with random weights. Following it, we benchmarked decoding on one A100-80GB, from GPT-2 Medium up to a 7B-shaped model with d=4096 and 32 block applications (dense 6.6B parameters, GRT 2+7x4+2 with 2.5B), comparing the full, averaged and codec caches (Figure 10 in the revised PDF).

Table R8. Peak GPU memory (GiB), batch 1, 8K-token prompt plus 512 generated tokens. Parameters are dense / GRT.

CacheMedium (354M / 127M)Large (774M / 293M)XL (1.56B / 0.65B)7B-shape (6.6B / 2.5B)
Dense1.763.325.9423.8
GRT, full Rx cache1.472.484.2816.1
GRT, averaged1.191.713.429.0
GRT, codec1.442.034.1810.1

The relative saving grows with model size. At the 7B shape, GRT needs 0.68x the memory of dense with the full cache and 0.38x with the averaged cache. Our proposed codec saves memory close to that of averaged cache. Its column sits higher only because our unfused implementation materializes the reconstructed K and V. We plotted the same measurements relative to dense: https://anonymous.4open.science/r/GRT-ICLR-2027-Rebuttal-Figures-5454/fig_memory-1.png

Figure R4. Peak decoding memory relative to the dense model (labels in GiB).

On model size (W3).

Thank you for pushing on scale. We scaled to GPT-2 XL depth (48 block applications, d=1600) and trained both models for 50k steps (24.6B tokens). At GPT-2 XL depth, GRT matches the dense model with 42% of its parameters and 0.71x its training compute.

Table R4. Validation loss at GPT-2 XL depth, 24.6B tokens.

ModelParamsTrain computeVal loss
Dense GPT-2 XL1.56B1.00x2.605
GRT 2+10x4+60.65B0.71x2.601

https://anonymous.4open.science/r/GRT-ICLR-2027-Rebuttal-Figures-5454/fig_extrapolation-1.png

Figure R2. Data scaling of Small and Large (measured, solid and dashed), extrapolated to 2.5x data (dotted, hollow markers), with the measured GPT-2 XL runs at 2.5x (stars) traced back with the Large curve shapes.

On pretraining context length (W3).

The reviewer's point about context length is well taken. We pretrained GRT Small and dense GPT-2 Small at an 8K context for the full 20k steps, lowering the batch size by the same factor of 8 so tokens per step and the 9.8B-token budget are unchanged (Appendix D.2).

Table R5. Validation loss, Small, 9.8B tokens.

Model1K context8K context
Dense GPT-2 Small3.1543.152
GRT (ours)3.1413.140

GRT's lead over dense holds at an 8K context, so it does not depend on the short pretraining context.

On the codec and a shared plus low-rank cache (W5, Q2).

We thank the reviewer for the suggestion. The codec is already trained separately, on the frozen GRT model, and it is close in spirit to the proposed decomposition: a per-channel linear encoder and decoder reconstruct the K and V of every recurrence from one shared pair of latents (Appendix B). Learning K^(r) = K_shared + A^(r)B^(r) jointly during pretraining would remove the separate codec finetuning step, and we have added it to future work.

Dear Reviewer 1AiE,

Thank you again for your thoughtful review and constructive suggestions. We posted a detailed rebuttal to your review, split across two comments. As the discussion period closes shortly, we would be very grateful if you could take a look and let us know whether our response resolves your concerns, or raise any remaining points while there is still time for us to respond.

We would also highly appreciate it if you could re-consider your score in light of the additional experiments and revisions, which include training-FLOPs accounting for every method, decoding memory benchmarks up to a 7B-shaped model, GPT-2 XL-depth training, 8K-context pretraining, and early-exit latency measurements.

Thank you again for your time and effort.

Reviewer_xLw9OpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Gated Recurrent Transformers (GRT), an architecture that reuses transformer blocks across depth while applying learned elementwise gates to control recurrent state updates. The gates condition on the current hidden state and a fixed prelude representation, with additional stochastic perturbations during training. The authors evaluate GRT under both isoFLOPs and isoPARAMS settings against dense GPT-2 models and several recurrent transformer baselines. Results show improved parameter efficiency and competitive language modeling performance, supported by component ablations, downstream evaluations, and analysis of intermediate recurrences. The paper also explores self-speculative decoding and a post-training KV-cache compression method.

<<<< UPDATES >>>>

I overall think the recurrent weight sharing design (+ learned gating, which is very standard) a natural and relatively simple approach. The motivation on param efficiency makes sense to me. what i particularly liked about this work is that it ACTUALLY looked at inference latency & memory usage more than a validation loss. My original concerns were whether that isoFLOPs comparison actually controlled training compute at all.

After the rebuttal I think most of my concerns have been handled by the authors. It does look like the full depth inference is still 10% slower. somehow this means the speculative latency is calculated from speedup measurements, rather than direct;ly benchmarking under the same conditions. overall i maintain a positive rating and updated my confidence.

Critical strengths
  1. A simple and effective architectural modification. The proposed gating mechanism is a natural extension of recurrent neural networks to recurrence over transformer depth. The method introduces relatively little architectural complexity, and the component ablations suggest that gating provides meaningful improvements beyond standard weight sharing and prelude conditioning. I find the simplicity of the approach appealing.

  2. Broad empirical evaluation across relevant trade-offs. The authors compare GRT against several recent recurrent architectures at three model scales, considering both parameter and compute budgets. I particularly appreciate the inclusion of real inference latency, memory consumption, early-exit behavior, and multiple training data budgets. The results suggest that the architecture can offer a useful parameter-quality trade-off rather than improving one metric in isolation.

Critical weaknesses
  1. The isoFLOPs comparison does not appear to control actual training compute. The paper defines isoFLOPs primarily through the maximum number of transformer block applications, but recurrent models sample execution depth uniformly during training. Consequently, their average training FLOPs can differ substantially from those of fixed-depth dense baselines, despite using the same number of optimization steps and training tokens. This makes the comparison difficult to interpret as a controlled compute-quality experiment. I would like to see the actual cumulative training FLOPs for each model and comparisons at matched measured compute budgets.

  2. The empirical evidence for the architectural advantage is somewhat limited. Most results use a single training seed, with three-seed validation restricted to the small configuration. The component ablation is sequential, so it does not fully isolate the contributions or interactions of gating, prelude re-injection, and stochastic noise. Furthermore, the models are relatively small and trained on approximately 9.8B tokens. While this is sufficient to demonstrate a promising architectural direction, stronger claims about scaling and practical efficiency would benefit from additional validation, particularly under matched compute budgets.

Questions for authors
  1. Because recurrence depth is sampled uniformly during training, I'm not sure the maximum forward-pass FLOPs reflect the compute each method actually used. How do the cumulative training FLOPs compare across GRT, the dense baselines, and the other recurrent methods? Compute-matched learning curves, or final results reported against measured training FLOPs, would make the comparison much more convincing.

  2. I'd like a clearer picture of where GRT's gains come from. How much of the improvement is due to the adaptive elementwise gating itself, and how much to the extra learned projection, the prelude conditioning, or differences in optimization? A factorial ablation, or a comparison against a simpler scalar-gating variant, would help show whether the elementwise design is actually needed.

  3. Do the improvements over the recurrent baselines hold up across multiple seeds at the medium and large scales? I'm also unsure how to read the efficiency claims, given the reported 10–11% increase in compiled generation latency relative to dense models before self-speculative decoding is applied. Some discussion of this trade-off would be helpful.

3 comments

We thank the reviewer for describing the gate as "a simple and effective architectural modification" and for appreciating that we report "real inference latency, memory consumption, early-exit behavior, and multiple training data budgets".

On training compute (W1, Q1).

We appreciate the reviewer pressing on this point. The reviewer is right that our label matched block applications per forward pass rather than the compute actually spent, and we have renamed the regime isoDepth. Once training compute is counted, the comparison favours GRT. GRT samples r uniformly from 1 to R, so it runs (R+1)/2 recurrences on average. Our heavy-tail Poisson baseline uses a log-normal sampler with its mode at R and averages about 0.93R. MoR's expert-choice routing keeps a fraction (R−r+1)/R of the tokens at step r [1], which also gives (R+1)/2 on average. Ouro supervises every exit [2] and RRT trains at a fixed depth, so both run all R steps. FLOPs follow Eq. 6 (Appendix A.2) at each method's expected depth (Appendix A.3), excluding embeddings and the LM head as in Table 1.

Table R1. Training compute relative to dense (9.8B tokens) / validation loss. Bold is best. Underline is second best.

MethodSmallMediumLarge
Dense GPT-21.00x / 3.15‾\underline{3.15}1.00x / 2.841.00x / 2.71
MoR0.66x / 3.300.69x / 3.020.66x / 2.91
Heavy-tail Poisson1.04x / 3.230.96x / 2.970.96x / 2.93
Ouro1.11x / 3.191.02x / 2.931.02x / 2.82
RRT1.14x / 3.141.05x / 2.951.04x / 2.85
GRT (ours)0.78x‾\underline{0.78\text{x}} / 3.140.73x‾\underline{0.73\text{x}} / 2.89‾\underline{2.89}0.69x‾\underline{0.69\text{x}} / 2.77‾\underline{2.77}

GRT trains with 22 to 31% less compute than dense, and less than every recurrent baseline except MoR, which it beats by 0.13 to 0.16 nats. GRT matches dense at Small with 78% of its training compute and stays within 0.05 to 0.06 nats at Medium and Large with 69 to 73% of training compute. Meanwhile, Heavy-tail Poisson, Ouro and RRT all spend more compute than GRT, and only RRT at Small reaches the same loss. We plotted loss against training compute for every method (Figure 8 in the revised PDF) here: https://anonymous.4open.science/r/GRT-ICLR-2027-Rebuttal-Figures-5454/fig_train_compute-1.png

Figure R1. Validation loss against training compute (relative to dense).

On where the gains come from (W2, Q2).

Thank you for suggesting factorial ablations. We ran them on GRT Small (20k steps), changing one factor at a time from the full model (Table 3 in the revised PDF).

Table R2. Factorial ablations, GRT Small, 20k steps.

VariantGatePrelude re-injectionState noiseVal loss
GRT (ours)✓✓✓3.14
No gate✗✓✓3.19
No prelude re-injection✓✗✓3.18
No state noise✓✓✗3.15

The gate is the largest single contributor. Removing the gate costs the most (+0.05), followed by prelude re-injection (+0.04), while state noise adds 0.01.

On types of gates (Q2).

We thank the reviewer for asking whether the elementwise design is actually needed. We compared gate granularities at the same budget.

Table R3. Validation loss by gate type, 20k steps.

GateSmallMedium
Per-token (scalar)3.162.91
Per-head3.14In Progress
Elementwise (ours)3.142.89

The gain comes from letting channels update independently, not from the gate's parameter count. A per-token scalar gate uses one value for all channels, so every channel keeps the same share of its previous state, and it recovers only 0.03 of the gate's 0.05 nats. Per-head and elementwise gates let different channels keep different shares of their previous state, and both reach 3.14. The elementwise gate stays ahead at Medium (2.89 vs. 2.91), so its advantage is not specific to Small.

On seeds (W2, Q3).

We share the reviewer's concern about variance. Multi-seed runs at Medium and Large are beyond our budget. At Small, three seeds give 3.145 ± 0.004 for GRT and 3.188 ± 0.056 for dense, and GRT's worst seed (3.148) is below dense's best (3.154) (Appendix D.1). At Medium and Large, GRT's margin over the strongest recurrent baseline (Ouro) is 0.04 and 0.05 nats, at least ten times the seed standard deviation we measured for GRT.

On model and data scale (W2).

Thank you for raising the question of scale. Ideally we would compare at frontier-lab budgets, which is beyond our compute. We instead follow the scaling-law methodology of Kaplan et al. [3] and measure how the gap to dense changes across three model sizes and three data budgets (Figure 6). The gap shrinks as data grows with the Large variant delta between baseline and GRT falls from 0.10 to 0.06 to 0.02 nats between 0.5x and 2x data, and at Small GRT leads at every budget. To test where this trend leads, we scaled to GPT-2 XL depth and trained GRT 2+10x4+6 and dense GPT-2 XL for 50k steps (24.6B tokens, 2.5x the budget of Table 1). GRT reaches 2.601 against 2.605 for dense, with 0.65B instead of 1.56B parameters and 0.71x the training compute. At GPT-2 XL depth, GRT matches dense with 42% of the parameters and 0.71x the training compute.

Table R4. Validation loss at GPT-2 XL depth, 24.6B tokens.

ModelParamsTrain computeVal loss
Dense GPT-2 XL1.56B1.00x2.605
GRT 2+10x4+60.65B0.71x2.601

https://anonymous.4open.science/r/GRT-ICLR-2027-Rebuttal-Figures-5454/fig_extrapolation-1.png

Figure R2. Data scaling of Small and Large (measured, solid and dashed), extrapolated to 2.5x data (dotted, hollow markers), with the measured GPT-2 XL runs at 2.5x (stars) traced back with the Large curve shapes.

On pretraining context length (W2).

Thanks as well for noting the short pretraining context. We pretrained GRT Small and dense GPT-2 Small at an 8K context for the full 20k steps, lowering the batch size by the same factor of 8 so tokens per step and the 9.8B-token budget are unchanged (Appendix D.2).

Table R5. Validation loss, Small, 9.8B tokens.

Model1K context8K context
Dense GPT-2 Small3.1543.152
GRT (ours)3.1413.140

GRT's lead over dense holds at an 8K context, so it does not depend on the short pretraining context.

On the latency trade-off (Q3).

We are grateful the reviewer looked closely at the latency numbers. Under torch.compile, full-depth GRT decodes 10% slower than dense at Large and 11% slower at Medium, while at Large it uses 62% fewer parameters and 59% less peak memory (Table 7). However, Self-speculative decoding gives 1.19x to 1.46x over GRT's own full depth at Large (Table 6), more than the overhead. With self-speculative decoding, GRT is estimated to decode faster than dense at both scales.

Table R6. Decode latency (ms/token, compiled, batch 4, 1024-token prompt, 128 new tokens), relative to dense in parentheses. The last row is estimated as full-depth latency divided by the measured task-level speedups (range over ten tasks, Table 6).

DecodingMediumLarge
Dense GPT-22.54 (1.00x)3.33 (1.00x)
GRT, full depth2.81 (1.11x)3.67 (1.10x)
GRT + self-speculative2.02 to 2.38 (0.80 to 0.94x)2.51 to 3.08 (0.75 to 0.93x)

Moreover, we would like to indicate that GRT can beat or approach dense baseline in terms of latency and validation loss (while maintaining the memory savings advantage) by exiting early. We timed every exit depth in the Table 7 setting, normalized to the dense model of each scale. Exiting one recurrence before full depth runs at 0.86x the dense latency at Medium and 0.92x at Large, at a cost of 0.06 and 0.006 nats over full depth (Figure 3b). We plotted Figure 3b again with latency in place of FLOPs (Figure 3c in the revised PDF): https://anonymous.4open.science/r/GRT-ICLR-2027-Rebuttal-Figures-5454/fig_earlyexit_latency-1.png

Figure R3. Figure 3b with decode latency of each scale (stars: dense GPT-2).

References.
[1] Bae et al. Mixture-of-Recursions. NeurIPS 2025.
[2] Zhu et al. Scaling Latent Reasoning via Looped Language Models. arXiv 2510.25741, 2025.
[3] Kaplan et al. Scaling Laws for Neural Language Models. arXiv 2001.08361, 2020.

Dear Reviewer xLw9,

Thank you again for your careful review and constructive suggestions. We posted a detailed rebuttal to your review, split across two comments. As the discussion period closes shortly, we would be very grateful if you could take a look and let us know whether our response resolves your concerns, or raise any remaining points while there is still time for us to respond.

We would also highly appreciate it if you could re-consider your score in light of the additional experiments and revisions, which include training-FLOPs accounting for every method, factorial and gate-granularity ablations, GPT-2 XL-depth training at 2.5x the data budget, 8K-context pretraining, and early-exit latency measurements.

Thank you again for your time and effort.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Executive Summary

We thank all reviewers for their time. They found the proposed solution "a simple and effective architectural modification" (xLw9), the results "promising ... with substantially fewer parameters and less peak memory" (1AiE), and the pipeline a "practical engineering integration" (SRjX).

The main shared concern was compute accounting (xLw9, 1AiE). Our "isoFLOPs" label matched the number of block applications per forward pass. At inference, GRT costs 6% more FLOPs than the dense model at Medium and Large (28% at Small). However at training, since GRT samples its depth uniformly during training, it trains with 22 to 31% less compute than the dense model. Among the recurrent baselines only MoR trains with less, and GRT matches or beats every recurrent baseline at every scale (Table R1). We have renamed the regime isoDepth and now report training and inference FLOPs for every method.

Our results indicate that when training on larger budgets, GRT narrows the gap with the baseline and trends show it can eventually overtake it. This makes GRT even more reliable for frontier models that train on larger budgets. While in our paper we do not have enough compute to train for such budgets, we were able to perform scaling analysis that enable us to extrapolate to such conclusions. To verify that extrapolation, we trained GPT-2 XL depth at a large enough data budget, showing that GRT matches the dense model (2.601 vs. 2.605) with only 42% of its parameters and 0.71x its training compute.

This response adds training FLOPs for all methods (Table R1), factorial gate ablations including scalar and per-head gates (Table R2), early-exit latency (Table R3), decoding memory up to a 7B-shaped model (Table R4), 8K-context pretraining, larger scale training, and a mechanistic account of the gate.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.