- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces a recurrent transformer variant in which a latent recurrent state aggregating the content of the hidden activations at a deeper layer is fused into the hidden activations of a shallower layer at the following token via a dedicated fusion module. The recurrence is therefore over a block, and the paper offers evidence that selecting middle layers for such a block is most beneficial for reasoning tasks. Experiments on synthetic tasks highlight the advantage of combined recurrence and attention mechanisms. Experiments on standard LLM benchmarks show improvements for models up to 1B parameters trained from scratch, and for a single 1.7B parameter retro-fitted model (initialized from a pretrained regular transformer).
Critical strengths
- The paper is clearly written.
- The observations of how the global gating mechanism evolves over training are interesting.
Critical weaknesses
-
The novel content of this work is limited. [Koishekenov et al. 2025] already identified the benefits for reasoning of looping only over middle layers, although in the context of looping architectures rather than recurrent transformers, and did so retro-fitting a pretrained transformer.
-
Most other ingredients were already introduced by the works mentioned in the Related Work section.
-
The paper does not compare against any other hybrid recurrent/attention architecture (e.g. Griffin).
-
Experiments are limited to models of up to 1.7B. Unfortunately it is frequent to see gains measured at this scale vanish at larger scales (see the same paper cited above for an example).
-
A single experiment is not sufficient to support the sweeping claim that ‘existing transformers can be upgraded’ (L393).
Koishekenov, Y., Lipani, A. and Cancedda, N., 2025. Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughts. arXiv preprint arXiv:2510.07358.
Questions for authors
-
L043: Why would injecting the hidden representation of the last layer into the embeddings of the following token bypass the middle layers?
-
How do the local gating functions in the fusion module evolve during training?
3 comments
Response to Reviewer 9Gkg (part 2/3)
Response to Reviewer 9Gkg (part 2/3)
Weakness 2: most other ingredients already exist.
The individual components were introduced by works already in the related work section.
We agree that the individual ingredients have precedents, and the contribution is their combination together with the evidence about which part matters.
(a) Among architectures that recur across tokens, the middle-block design performs best in our pipeline. The Latent Recurrent and Full-bandwidth Transformers both send a deep state through the full depth of the network. We reimplemented both and trained them with the same data, optimizer and token budget as T²MLR at 135M. We find that the middle-block design is ahead of both full-depth designs:
| Model | Eval loss |
|---|---|
| Latent Recurrent Transformer | 2.957 |
| Full-bandwidth Transformer | 2.954 |
| T²MLR | 2.939 |
This is consistent with the finding we arrived at independently in Table 2, where middle-layer T²MLR variants perform better than full-layer T²MLR. The two comparisons point the same way, whether the full-depth alternative is a different architecture or our own full-layer variant. We would like to point out that these are recent concurrent works, which we cite here for reference rather than as direct precedents of this paper.
The parallel training procedure is validated against exact recurrent training at three scales in Table 6, and the pathway can be added to a model that has already been pretrained. In the revision, we will state the contribution in these terms in Section 1.
Weakness 3: no comparison against hybrid recurrent and attention architectures.
The paper does not compare against a hybrid such as Griffin.
(a) Griffin and T²MLR change different parts of the model, and only T²MLR can be added to a pretrained Transformer. Following your suggestion, we trained a parameter-matched Griffin-style model in our pipeline under the same recipe. The Griffin-style model reaches a lower eval loss at this scale:
| Model | Eval loss |
|---|---|
| Griffin-style | 2.912 |
| T²MLR, three seeds | 2.939 |
| Transformer, three seeds | 2.951 |
We see the two as complementary rather than as alternatives. Griffin replaces most attention layers with a recurrent block and carries state at a fixed depth, which changes the backbone. T²MLR adds a deep-to-shallow connection across tokens to an unchanged Transformer, which is what allows the pathway to be retrofitted into an already pretrained model. The table therefore compares two different design changes. Griffin is a new backbone that is trained from scratch, and the lower loss of the Griffin-style model at 135M does not bear on whether an existing pretrained Transformer can be improved by adding the T²MLR pathway.
In the revision, we will add this comparison to Section 4 and the discussion of scope to Section 5.
Weakness 4: experiments are limited in scale.
Gains measured below 2B parameters often vanish at larger scales.
(a) The gain holds across the scales we train from scratch. Our from-scratch experiments cover 135M, 362M and 1B parameters, and 10B and 50B tokens, with T²MLR ahead of the parameter-matched Transformer in every setting in Table 1. We agree that this does not establish behaviour beyond the scales we tested, and we state the result as covering the range we ran.
In the revision, we will scope the scaling statement in Section 4 to the parameter and token range we test.
Weakness 5: a single retrofit does not support the claim that existing transformers can be upgraded.
One retrofitting experiment is not enough for a general claim.
We agree the claim at L393 was too broad.
(a) We narrow the claim to the setting we demonstrate. What we show is that one pretrained Transformer, SmolLM2-1.7B-Instruct, can be given a middle-layer recurrent pathway and a brief finetune, after which it improves on GSM8K from 35.8 to 39.9 and on MATH500 from 12.8 to 18.0. Most of the gain remains at equal training compute. Stopped once it had used the compute of the full Transformer finetune, the T²MLR retrofit reaches 38.7 on GSM8K and 15.8 on MATH500. We also correct the description of that experiment. The base model has 24 layers, the window is layers 5 to 20, and training used the full OpenMathReasoning chain-of-thought split.
In the revision, we will rewrite L393 to state the narrower claim and correct the description of the retrofit setup in Section 4.
Response to Reviewer 9Gkg (part 3/3)
Response to Reviewer 9Gkg (part 3/3)
Question 1: why would injecting the last layer bypass the middle layers?
The sentence at L043 is unclear.
(a) The sentence was imprecise and we will rewrite it. When the last layer's representation is fed to the input of the next token, it passes back through every layer, so nothing is bypassed in the literal sense. What we meant is that the representation carried is the wrong one. A last-layer state is specialized for predicting the next token rather than for holding intermediate computation, whereas middle-layer states hold the intermediate computation that the next token can continue. The measurement above supports this reading. We find that the model depends on the carried state about thirteen times more for a middle window than for a last-block window, 1.9 nats against 0.15.
In the revision, we will rewrite L043 to say this directly.
Question 2: how do the local gating functions evolve during training?
The paper reports the global gates but not the local ones.
(a) The local input gate opens steadily while the recurrent gate stays graded. We tracked the local gates across training at 135M, alongside the global scales already in the appendix. We find that the two local gates behave differently:
| Gate | Start of training | End of training |
|---|---|---|
| Local input gate, mean | 0.50 | 0.94 |
| Local recurrent gate, mean | 0.55 | 0.55 |
The two global scales start at zero, so training begins from the plain Transformer. Within the first two thousand steps the recurrent scale rises to about 0.7 and the input scale falls to about -0.7, after which both change slowly. The local recurrent gate stays graded around 0.55 rather than saturating.
In the revision, we will add the local gates to the Learned Gating Behavior figure in the appendix.
Thank you again for your constructive review! We hope these new results address your concerns, and we are happy to answer any follow-up questions.
Response to Reviewer 9Gkg (part 1/3)
Response to Reviewer 9Gkg (part 1/3)
We thank the reviewer for the close reading and for finding the analysis of how the gating evolves over training interesting. Below, we first summarize our responses and then address each point in detail.
Summary of responses
Weakness 1: novelty relative to ETD.
(a) We do not claim the middle-layer observation, and the paper credits its sources. Middle-layer depth recurrence was used by [1] and [2], both of which the submission cites, and ETD [3] develops the idea further. What we establish is that the same preference holds for temporal recurrence, which is a nontrivial generalization from depth recurrence rather than a corollary of it. Depth recurrence repeats computation within one token. Temporal recurrence carries a representation to the next token, and the natural choice there is the final layer, since it summarizes the whole computation. We replaced the carried state with the state of an unrelated sequence and measured the cost over the following 256 tokens. We find the opposite of that expectation:
The model relies on a middle state about five times more than a full-depth one and about thirteen times more than a last-block one.
(b) The two mechanisms also differ in cost. ETD re-runs the middle block several times for every token, so decoding cost grows with the number of loops. T²MLR iterates only during training, and each generated token takes one forward pass. Following your suggestion to compare them directly, we trained an ETD-style model in our pipeline at 135M, looping the same block three times. We find that the two reach the same loss while the ETD-style model applies 1.4 times the layers at every generated token:
In the revision, we will state the attribution to [1], [2] and [3] in Section 5 and add both measurements to Section 4.
References
[1] N. Saunshi, N. Dikkala, Z. Li, S. Kumar, and S. J. Reddi. Reasoning with Latent Thoughts: On the Power of Looped Transformers. ICLR 2025.
[2] J. Geiping, S. McLeish, N. Jain, J. Kirchenbauer, S. Singh, B. R. Bartoldson, B. Kailkhura, A. Bhatele, and T. Goldstein. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach. arXiv:2502.05171, 2025.
[3] Y. Koishekenov, A. Lipani, and N. Cancedda. Encode, Think, Decode: Scaling Test-Time Reasoning with Recursive Latent Thoughts. ICLR 2026.