acceptodds
Under review as a conference paper at ICLR 2027

T²MLR: Transformer with Temporal Middle-Layer Recurrence

Abstract

Transformer reasoning is limited by autoregressive decoding, which repeatedly compresses rich hidden computation through token space and makes it difficult for intermediate reasoning states to persist across time. We introduce Transformers with Temporal Middle-Layer Recurrence (T²MLR), a Transformer-based latent reasoning architecture that fuses a cached middle-layer representation from the previous token directly into an earlier layer of the current token position, enabling abstract intermediate computation to persist across decoding steps with little inference overhead. Across natural-language pretraining and multi-hop reasoning finetuning, middle-layer T²MLR variants outperform data- and parameter-matched Transformer baselines in every setting we test, with gains that persist when scaling to 1B parameters or 50B pretraining tokens. Moreover, applying recurrence to only a localized middle-layer block (as little as 20% of the network) often outperforms full-layer recurrence. Importantly, T²MLR does not require pretraining from scratch: retrofitting the recurrent pathway into an existing pretrained 1.7B Transformer and briefly finetuning substantially improves math reasoning, lowering the barrier to practical adoption. These results suggest that effective latent reasoning in Transformers does not require recurrence through all layers, as in prior work, but can instead emerge more strongly from targeted middle-layer recurrence.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Transformers, Recurrence, State-tracking, Latent reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_9GkgOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces a recurrent transformer variant in which a latent recurrent state aggregating the content of the hidden activations at a deeper layer is fused into the hidden activations of a shallower layer at the following token via a dedicated fusion module. The recurrence is therefore over a block, and the paper offers evidence that selecting middle layers for such a block is most beneficial for reasoning tasks. Experiments on synthetic tasks highlight the advantage of combined recurrence and attention mechanisms. Experiments on standard LLM benchmarks show improvements for models up to 1B parameters trained from scratch, and for a single 1.7B parameter retro-fitted model (initialized from a pretrained regular transformer).

Critical strengths
  • The paper is clearly written.
  • The observations of how the global gating mechanism evolves over training are interesting.
Critical weaknesses
  • The novel content of this work is limited. [Koishekenov et al. 2025] already identified the benefits for reasoning of looping only over middle layers, although in the context of looping architectures rather than recurrent transformers, and did so retro-fitting a pretrained transformer.

  • Most other ingredients were already introduced by the works mentioned in the Related Work section.

  • The paper does not compare against any other hybrid recurrent/attention architecture (e.g. Griffin).

  • Experiments are limited to models of up to 1.7B. Unfortunately it is frequent to see gains measured at this scale vanish at larger scales (see the same paper cited above for an example).

  • A single experiment is not sufficient to support the sweeping claim that ‘existing transformers can be upgraded’ (L393).

Koishekenov, Y., Lipani, A. and Cancedda, N., 2025. Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughts. arXiv preprint arXiv:2510.07358.

Questions for authors
  • L043: Why would injecting the hidden representation of the last layer into the embeddings of the following token bypass the middle layers?

  • How do the local gating functions in the fusion module evolve during training?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 9Gkg (part 1/3)

Response to Reviewer 9Gkg (part 1/3)

We thank the reviewer for the close reading and for finding the analysis of how the gating evolves over training interesting. Below, we first summarize our responses and then address each point in detail.

Summary of responses

  • Weakness 1 (novelty relative to ETD). Middle-layer depth recurrence was used by earlier work that the submission cites. What we establish is that the preference holds for temporal recurrence, which is a nontrivial generalization from depth recurrence, and we give a direct measurement supporting it.
  • Weakness 2 (existing ingredients). The ingredients have precedents and the contribution is their combination. A middle block beats full-depth recurrence across tokens, which no earlier cross-token method tests. Our reimplementations of the Latent Recurrent and Full-bandwidth Transformers, which recur through the full depth, end level with the Transformer.
  • Weakness 3 (hybrid recurrent and attention models). We trained a parameter-matched Griffin-style model. Griffin replaces the backbone and is trained from scratch, while T²MLR is added to an unchanged Transformer and can be retrofitted. We report the comparison and explain why the two designs serve different use cases.
  • Weakness 4 and 5 (scale and the retrofit claim). Our from-scratch results hold to 1B parameters and 50B tokens. We agree the upgrade claim was too broad and we narrow it to the setting we demonstrate, where the gain also holds at matched training compute.
  • Questions. We rewrite the sentence at L043 and give the evolution of the local gates.

Weakness 1: novelty relative to ETD.

Prior work already identified the benefit of looping over middle layers, including in a retrofitting setting.

(a) We do not claim the middle-layer observation, and the paper credits its sources. Middle-layer depth recurrence was used by [1] and [2], both of which the submission cites, and ETD [3] develops the idea further. What we establish is that the same preference holds for temporal recurrence, which is a nontrivial generalization from depth recurrence rather than a corollary of it. Depth recurrence repeats computation within one token. Temporal recurrence carries a representation to the next token, and the natural choice there is the final layer, since it summarizes the whole computation. We replaced the carried state with the state of an unrelated sequence and measured the cost over the following 256 tokens. We find the opposite of that expectation:

Recurrence windowCost of replacing the carried state
Middle1.9 nats
Full-layer0.4 nats
Last block0.15 nats

The model relies on a middle state about five times more than a full-depth one and about thirteen times more than a last-block one.

(b) The two mechanisms also differ in cost. ETD re-runs the middle block several times for every token, so decoding cost grows with the number of loops. T²MLR iterates only during training, and each generated token takes one forward pass. Following your suggestion to compare them directly, we trained an ETD-style model in our pipeline at 135M, looping the same block three times. We find that the two reach the same loss while the ETD-style model applies 1.4 times the layers at every generated token:

ModelEval lossLayers per generated token
ETD-style, three loops2.9381.4x
T²MLR, three seeds2.9391.0x

In the revision, we will state the attribution to [1], [2] and [3] in Section 5 and add both measurements to Section 4.

References

[1] N. Saunshi, N. Dikkala, Z. Li, S. Kumar, and S. J. Reddi. Reasoning with Latent Thoughts: On the Power of Looped Transformers. ICLR 2025.

[2] J. Geiping, S. McLeish, N. Jain, J. Kirchenbauer, S. Singh, B. R. Bartoldson, B. Kailkhura, A. Bhatele, and T. Goldstein. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach. arXiv:2502.05171, 2025.

[3] Y. Koishekenov, A. Lipani, and N. Cancedda. Encode, Think, Decode: Scaling Test-Time Reasoning with Recursive Latent Thoughts. ICLR 2026.

AuthorsOpenReview ↗

Response to Reviewer 9Gkg (part 2/3)

Response to Reviewer 9Gkg (part 2/3)

Weakness 2: most other ingredients already exist.

The individual components were introduced by works already in the related work section.

We agree that the individual ingredients have precedents, and the contribution is their combination together with the evidence about which part matters.

(a) Among architectures that recur across tokens, the middle-block design performs best in our pipeline. The Latent Recurrent and Full-bandwidth Transformers both send a deep state through the full depth of the network. We reimplemented both and trained them with the same data, optimizer and token budget as T²MLR at 135M. We find that the middle-block design is ahead of both full-depth designs:

ModelEval loss
Latent Recurrent Transformer2.957
Full-bandwidth Transformer2.954
T²MLR2.939

This is consistent with the finding we arrived at independently in Table 2, where middle-layer T²MLR variants perform better than full-layer T²MLR. The two comparisons point the same way, whether the full-depth alternative is a different architecture or our own full-layer variant. We would like to point out that these are recent concurrent works, which we cite here for reference rather than as direct precedents of this paper.

The parallel training procedure is validated against exact recurrent training at three scales in Table 6, and the pathway can be added to a model that has already been pretrained. In the revision, we will state the contribution in these terms in Section 1.

Weakness 3: no comparison against hybrid recurrent and attention architectures.

The paper does not compare against a hybrid such as Griffin.

(a) Griffin and T²MLR change different parts of the model, and only T²MLR can be added to a pretrained Transformer. Following your suggestion, we trained a parameter-matched Griffin-style model in our pipeline under the same recipe. The Griffin-style model reaches a lower eval loss at this scale:

ModelEval loss
Griffin-style2.912
T²MLR, three seeds2.939
Transformer, three seeds2.951

We see the two as complementary rather than as alternatives. Griffin replaces most attention layers with a recurrent block and carries state at a fixed depth, which changes the backbone. T²MLR adds a deep-to-shallow connection across tokens to an unchanged Transformer, which is what allows the pathway to be retrofitted into an already pretrained model. The table therefore compares two different design changes. Griffin is a new backbone that is trained from scratch, and the lower loss of the Griffin-style model at 135M does not bear on whether an existing pretrained Transformer can be improved by adding the T²MLR pathway.

In the revision, we will add this comparison to Section 4 and the discussion of scope to Section 5.

Weakness 4: experiments are limited in scale.

Gains measured below 2B parameters often vanish at larger scales.

(a) The gain holds across the scales we train from scratch. Our from-scratch experiments cover 135M, 362M and 1B parameters, and 10B and 50B tokens, with T²MLR ahead of the parameter-matched Transformer in every setting in Table 1. We agree that this does not establish behaviour beyond the scales we tested, and we state the result as covering the range we ran.

In the revision, we will scope the scaling statement in Section 4 to the parameter and token range we test.

Weakness 5: a single retrofit does not support the claim that existing transformers can be upgraded.

One retrofitting experiment is not enough for a general claim.

We agree the claim at L393 was too broad.

(a) We narrow the claim to the setting we demonstrate. What we show is that one pretrained Transformer, SmolLM2-1.7B-Instruct, can be given a middle-layer recurrent pathway and a brief finetune, after which it improves on GSM8K from 35.8 to 39.9 and on MATH500 from 12.8 to 18.0. Most of the gain remains at equal training compute. Stopped once it had used the compute of the full Transformer finetune, the T²MLR retrofit reaches 38.7 on GSM8K and 15.8 on MATH500. We also correct the description of that experiment. The base model has 24 layers, the window is layers 5 to 20, and training used the full OpenMathReasoning chain-of-thought split.

In the revision, we will rewrite L393 to state the narrower claim and correct the description of the retrofit setup in Section 4.

AuthorsOpenReview ↗

Response to Reviewer 9Gkg (part 3/3)

Response to Reviewer 9Gkg (part 3/3)

Question 1: why would injecting the last layer bypass the middle layers?

The sentence at L043 is unclear.

(a) The sentence was imprecise and we will rewrite it. When the last layer's representation is fed to the input of the next token, it passes back through every layer, so nothing is bypassed in the literal sense. What we meant is that the representation carried is the wrong one. A last-layer state is specialized for predicting the next token rather than for holding intermediate computation, whereas middle-layer states hold the intermediate computation that the next token can continue. The measurement above supports this reading. We find that the model depends on the carried state about thirteen times more for a middle window than for a last-block window, 1.9 nats against 0.15.

In the revision, we will rewrite L043 to say this directly.

Question 2: how do the local gating functions evolve during training?

The paper reports the global gates but not the local ones.

(a) The local input gate opens steadily while the recurrent gate stays graded. We tracked the local gates across training at 135M, alongside the global scales already in the appendix. We find that the two local gates behave differently:

GateStart of trainingEnd of training
Local input gate, mean0.500.94
Local recurrent gate, mean0.550.55

The two global scales start at zero, so training begins from the plain Transformer. Within the first two thousand steps the recurrent scale rises to about 0.7 and the input scale falls to about -0.7, after which both change slowly. The local recurrent gate stays graded around 0.55 rather than saturating.

In the revision, we will add the local gates to the Learned Gating Behavior figure in the appendix.

Thank you again for your constructive review! We hope these new results address your concerns, and we are happy to answer any follow-up questions.

Reviewer_7icWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces T2MLR, an architecture with temporal recurrence through its middle layers. A gated fusion module injects a deeper representation from the previous token into an earlier layer of the current token, enabling deep-to-shallow information flow across decoding steps while retaining a single network pass per generated token. Training uses Jacobi refinement and truncated backpropagation to approximate recurrent computation in parallel. The paper evaluates S5-Retrieval, FineWeb-Edu pretraining, reasoning finetuning, and retrofitting a pretrained 1.7B model. It also studies recurrence location and width, compares looped/pause-token baselines, measures decoding and prefill overhead. Results show that selected middle-layer recurrence outperforms both parameter-matched Transformer baselines and full-layer recurrence variants.

Critical strengths
  1. Well-motivated and well-validated architectural design. T2MLR addresses the limited access of shallow layers to deeper representations from previous tokens. T2MLR injects the previous token's middle-layer hidden state into the earlier layer of current token and demonstrates the efficitiveness by extensive experiments: retrieval task, pretraining, retrofitting etc.
  2. Consistent gains over multiple baseliens. T2MLR improves accuracy over parameter- and data-matched Transformers. At 135M, the selected middle-layer variants also outperform the looped and pause-token baselines, despite these baselines requiring roughly twice the decoding computation.
Critical weaknesses
  1. Lower efficiency: Training is roughly 2–4x slower. At 135M, a Transformer trained for the same wall-clock budget achieves higher average zero-shot accuracy, 45.30% versus 44.14% (Table 11). Besides, exact prefill is sequential, and Jacobi prefill costs 2.7× for 6-layer's 8-step prefill(Table 8). Thus, the small decoding overhead does not imply comparable end-to-end inference cost, and practical benefits depend on the training budget and prompt/generation workload.
  2. Layer selection relies on sweeps: The choice of recurrent layers rests on a 135M sweep, with no clear rule for other models. The boundaries used at larger scales ((9,24)) and for retrofitting ((5,20)) are not explained, so applying T2MLR to a new model would likely require another costly sweep.
Questions for authors
  1. Training recipe: Full-bandwidth Transformer [1] trains top-to-bottom recurrence with only a few feedback passes added late in pretraining. Have you tried a similar schedule, and does middle-layer recurrence still outperform full-layer recurrence? If not, do you think a similar strategy would be needed to make T2MLR more efficient? Existing observations or discussion would be enough; no new experiments are required.
  2. Layer selection: Prior work selects recurrent layers with explicit metrics instead of sweeps; For example, ETD [2] uses the angular distance between consecutive layers to locate the reasoning block. Have you checked whether such metrics predict your best boundaries, e.g., (13,18) at 135M? A simple selection rule would make T2MLR easier to apply to new models.

[1] X. Wang et al. Full-bandwidth Transformer. arXiv:2608.08888, 2026.

[2] Y. Koishekenov, A. Lipani, and N. Cancedda. Encode, Think, Decode: Scaling Test-Time Reasoning with Recursive Latent Thoughts. ICLR 2026.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 7icW (part 1/3)

Response to Reviewer 7icW (part 1/3)

We thank the reviewer for the careful summary and for recognizing that T²MLR improves on parameter-matched and data-matched Transformers. Below, we first summarize our responses and then address each point in detail.

Summary of responses

  • Weakness 1 (efficiency). T²MLR spends more compute in training and generates better at close to the decoding cost of a Transformer. We agree that a Transformer given the same training compute is slightly ahead on zero-shot NLP, and we explain below why we think the extra training compute is well spent; on reasoning, the 1.7B retrofit stays ahead at matched training compute. We measured the full inference cost and found end-to-end latency within 7 to 11 percent of a Transformer for reasoning-length generations, and we retrained the models with a cheaper schedule that reduces the recurrent training overhead by about 2.5 times.
  • Weakness 2 (layer selection). The windows in our larger models were not swept. They follow one rule, which is that the recurrence window is centred on the network, and this holds exactly for all three models we train.
  • Question 1 (staged schedule). Our retrofit is that schedule taken to its limit and it works. A schedule with few Jacobi passes does not, because the parallel gradient agrees with the exact recurrent gradient only after about eight Jacobi passes; the local-sequential schedule is how we reduce the training cost.
  • Question 2 (selection by metric). The angular distance criterion of [2] selects a block starting at layer 12 to 13 against our 13 at 135M and lands within a few layers of our windows at the larger scales, so it gives a practitioner a cheap way to place the window without a sweep.

Weakness 1: lower efficiency.

Training is 2 to 4 times slower, and sequential or iterative prefill means the small decoding overhead does not imply comparable end-to-end inference cost.

T²MLR spends more compute in training and, in return, generates better at close to the decoding cost of a Transformer. The comparisons in the paper hold parameters, training data and decoding cost fixed, and T²MLR is more accurate under those comparisons. The price is training compute. As you point out, a Transformer given the same training compute, spent on additional training tokens, is slightly ahead on zero-shot NLP (45.30 against 44.14 in Table 11), and we should have said so in the main text. We think paying for the gain in training is worthwhile for three reasons. First, training compute is paid once, while decoding cost is paid at every generated token. Looped and pause-token models buy their gains with extra passes at every generated token, and T²MLR needs one pass. Second, a Transformer can turn extra training compute into accuracy only when additional data is available. In finetuning on a reasoning corpus, where the data is fixed, T²MLR is ahead even at equal training compute: the 1.7B T²MLR retrofit scores 38.7 on GSM8K and 15.8 on MATH500, and the Transformer finetune scores 35.8 and 12.8. An independent study of top-layer latent feedback (B. Huang et al., "Latent Feedback Is a Compute and Data Amplifier", alphaXiv, November 2026) makes the same comparison with the Transformer spending the matched compute on repeated epochs: the feedback model reaches a given loss with up to 48% less training compute when unique data per parameter is plentiful, and needs about 8% more when unique data is scarce. The feedback design in the study differs from ours, so we cite the study only for the role of the data regime. Third, the training overhead can be reduced: measurement (b) below cuts the overhead by about 2.5 times at the same accuracy. Two new measurements make the cost concrete.

(a) End-to-end inference is within 7 to 11 percent of a Transformer for reasoning-length generations. Following your suggestion, we measured prompt prefill and generation together on one H100 at 135M with a 2048 token prompt. Prefill with eight iterations takes 49 ms for the D=6 window against 20 ms for the Transformer, and generation takes 12.6 ms per token against 11.8 ms. We find that end-to-end latency is governed by decoding rather than by prefill:

Output tokensTransformerT²MLR D=6T²MLR D=14
1281.54 s1.67 s1.70 s
5126.08 s6.52 s6.55 s
204824.28 s25.95 s25.98 s

At 2048 output tokens the overhead is 7 percent for both windows. Prefill dominates only for outputs of a few tokens. Batching reduces its share further, from 2.45 times at batch 1 to 1.46 times at batch 32.

References

[2] Y. Koishekenov, A. Lipani, and N. Cancedda. Encode, Think, Decode: Scaling Test-Time Reasoning with Recursive Latent Thoughts. ICLR 2026.

AuthorsOpenReview ↗

Response to Reviewer 7icW (part 2/3)

Response to Reviewer 7icW (part 2/3)

Weakness 1 (continued).

(b) A cheaper training schedule reduces the recurrent overhead by about 2.5 times. To reduce the training cost, we retrained the models with a local-sequential, global-parallel schedule, adapting the cyclic Gauss-Seidel fixed-point iteration of [1, Section 3.3] to our recurrent cache. The paper resolves the token chain with Jacobi iteration, where every position is refreshed at once from the previous iteration, so information advances one position per pass. In the new schedule, positions are split into g strided groups by t mod g and the groups are updated in order within a single pass. Updates are therefore sequential inside any window of g consecutive tokens and parallel across the whole sequence. Each pass costs the same as a Jacobi pass because every position is still updated once, and one pass advances the recurrent chain g positions instead of one, so far fewer passes are needed. This is a training-time change only. The model is architecturally identical and we evaluate it through the same inference path. We find that the schedule cuts the recurrent training overhead by about 2.5 times at both scales while matching the model it replaces:

135M, recurrence window 8 to 21, 10B tokens.

FLOPsLossARC-CARC-EHSPIQAOBQAWGSciQAverage
Transformer1.00x3.11724.4945.5029.6460.5531.2048.6261.6043.08
T²MLR, Jacobi4.11x3.07923.8945.0330.3161.4329.4051.4666.0043.93
T²MLR, local-sequential1.65x3.06724.4045.2430.3360.6630.8052.7263.2043.91

362M, recurrence window 9 to 24, 10B tokens.

FLOPsLossARC-CARC-EHSPIQAOBQAWGSciQAverage
Transformer1.00x2.73227.1352.6537.1865.5134.0051.8571.0048.48
T²MLR, Jacobi4.56x2.71028.8452.4838.6666.0033.4052.0973.1049.22
T²MLR, local-sequential1.74x2.71228.5853.6238.9965.6733.6051.0773.6049.30

FLOPs overhead is forward plus backward relative to the Transformer backbone, including the language modelling head. The two schedules land within 0.08 points of each other on the seven-task average at both scales, and both improve on the Transformer.

In the revision, we will state next to Table 11 that T²MLR does not reach a given loss with less training compute, add the end-to-end latency measurement to Appendix B.5, and report the local-sequential schedule and its cost.

Weakness 2: layer selection relies on sweeps.

The recurrent layers rest on a 135M sweep, with no clear rule for other models.

The windows in our larger models were not swept, and we should have stated the rule we used.

(a) The recurrence window is centred on the network, which fixes its position without any search. Writing each band with zero-based layer indices (the paper writes the same windows one-based, as (13,18), (9,24) and (5,20)), we find that all three of our models place the window exactly at the centre:

ModelLayersWindowLayers beforeLayers after
135M3012 to 171212
1B328 to 2388
1.7B retrofit244 to 1944

The position therefore follows from the architecture, and the only quantity we choose is the width of the window. Table 2 shows that a width of about 20 percent of the network already captures the gain at 135M; the larger models use wider windows, and we did not sweep the width at those scales. Placing the window in a new model therefore needs no sweep.

In the revision, we will state the centring rule in Section 3 and give this table in the appendix.

Question 1: a staged schedule with few feedback passes.

Have you tried adding recurrence late in pretraining, as the Full-bandwidth Transformer does, and does middle-layer recurrence still win under such a schedule?

(a) Our retrofit is that schedule taken to its limit, and it works. The recurrent pathway is added only after the Transformer is fully pretrained and is then trained briefly. We find that a staged schedule works for middle-layer recurrence, improving GSM8K from 35.8 to 39.9 and MATH500 from 12.8 to 18.0. We have not run full-layer recurrence under the same schedule, so we cannot yet say whether the middle-layer advantage carries over to it.

References

[1] Zhang and Ren. WhiteMatter. arXiv:2608.18486, 2026.

AuthorsOpenReview ↗

Response to Reviewer 7icW (part 3/3)

Response to Reviewer 7icW (part 3/3)

Question 1 (continued).

(b) Dropping Jacobi passes does not reach the exact recurrent gradient; ordering the updates within a pass is what reduces the cost. We measured the cosine similarity between the parallel gradient and the exact recurrent gradient at 135M as a function of the number of Jacobi passes. We find that two passes are nearly uncorrelated with the exact gradient and about eight are needed for agreement:

PassesCosine with the exact recurrent gradient
20.13
80.985

Reducing the number of Jacobi passes is therefore not an option. The local-sequential schedule above advances the recurrent chain several positions per pass, and that schedule is our answer to making training cheaper.

In the revision, we will add this analysis to the appendix and connect it to the discussion of training cost.

Question 2: do metrics such as angular distance predict the boundaries?

Prior work selects recurrent layers with explicit metrics rather than sweeps.

(a) The angular distance criterion predicts our windows closely enough to replace a sweep. Following your suggestion, we computed the consecutive-layer angular distance criterion of [2] on our models without any training. We find that the criterion selects a block starting where our best 135M window starts, at layer 12 to 13 against our 13, and ending a few layers later. On the 1B and 1.7B models it lands within a few layers of our centred windows. The criterion is sensitive to how the curve is smoothed, so we present it as a cheap starting point to confirm placement rather than as an exact selector.

In the revision, we will report this comparison alongside the centring rule in Section 3.

Thank you again for your constructive review! We hope these new results address your concerns, and we are happy to answer any follow-up questions.

References

[2] Y. Koishekenov, A. Lipani, and N. Cancedda. Encode, Think, Decode: Scaling Test-Time Reasoning with Recursive Latent Thoughts. ICLR 2026.

Reviewer_2b5HOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes T²MLR, a Transformer variant that carries a cached middle-layer hidden state from the previous token to an earlier layer at the current token via a gated recurrent pathway updated by residual accumulation and RMSNorm. Training uses Jacobi iterations and truncated BPTT to retain parallel teacher forcing. T²MLR achieves near-perfect S5-Retrieval within training lengths, outperforming matched Transformer/LSTM baselines; improves zero-shot accuracy in 135M–1B pretraining up to 50B tokens; favors localized middle-layer recurrence; and, when retrofitted to SmolLM2-1.7B-Instruct, raises GSM8K from 35.78% to 39.88% and MATH500 from 12.80% to 18.00%. Results support middle-layer temporal recurrence, though added training and prompt-prefill costs complicate the efficiency tradeoff.

Critical strengths
  1. A simple, effective architectural modification with evidence of improved reasoning. The proposed deep-to-shallow recurrent connection is conceptually simple, preserves the standard Transformer backbone, and adds little computation during autoregressive token generation. The S5-Retrieval experiment (Figure 4) provides compelling evidence of the intended inductive bias: T²MLR successfully combines sequential state tracking and in-context retrieval, a setting where the evaluated Transformer and LSTM baselines struggle. The fixed-width recurrence-location ablations (Table 2) further show that middle-layer placement consistently outperforms early- and late-layer alternatives. These results provide meaningful evidence that the recurrent pathway contributes capabilities beyond increased parameter count.

  2. Broad empirical validation, including scalable training and retrofitting. The paper goes beyond a small-scale architectural demonstration, evaluating pretraining at multiple parameter and data scales (Table 1), downstream reasoning finetuning (Figure 5 and Table 9), and retrofitting an existing 1.7B pretrained model (Table 3). Importantly, the authors validate their approximate temporal-parallel training procedure against exact recurrent training at three model scales (Table 6), finding closely matched validation losses. The ability to improve an existing pretrained model without restarting pretraining is especially valuable for practical adoption.

Critical weaknesses
  1. Substantial training and prefill overhead weaken the practical efficiency argument. The main comparisons match parameters, training data, and decoding cost, but T²MLR requires substantially more training computation due to repeated Jacobi refinement. In Appendix C.3 (Table 11), a Transformer trained under the same wall-clock budget as T²MLR achieves a higher zero-shot average (45.30 vs. 44.14). The paper also reports training slowdowns of approximately 2–4×. Moreover, the low inference overhead in Table 7 primarily measures autoregressive generation. Appendix B.5 states that the main experiments use exact sequential prompt prefill; approximate parallel prefill reduces this bottleneck but still incurs considerable overhead: with eight Jacobi iterations, prefill takes 2.7× and 4.6× the Transformer baseline time for the D=6 and D=14 variants, respectively (Table 8). Although the authors acknowledge these tradeoffs, they materially affect the practical value proposition. The paper convincingly demonstrates improvements at fixed training data and decoding budgets, but not a consistent advantage under matched total computation or end-to-end inference latency.

  2. Empirical comparisons do not sufficiently establish advantages over the closest temporal-recurrence architectures. The paper discusses several closely related approaches in Section 5 and Appendix A, including the Feedback Transformer, Latent Recurrent Transformer, and Full-bandwidth Transformer. These methods also allow deeper representations to influence subsequent token computations. However, the primary experimental comparisons focus on conventional Transformers, LSTMs, looped Transformers, and pause-token models rather than comparable deep-to-shallow temporal-recurrence architectures. Although the recurrence-location ablations are informative, they are primarily conducted at 135M parameters. Consequently, it remains unclear how much of the observed improvement comes specifically from the proposed middle-layer placement and fusion design, as opposed to the broader benefit of adding a deep-to-shallow recurrent pathway. This limits the evidence supporting the paper’s distinct contribution relative to the closest architectural alternatives.

Questions for authors
  1. How does T²MLR compare with standard Transformers under matched end-to-end inference and training budgets on reasoning tasks? Table 11 indicates that additional Transformer training can outperform T²MLR on zero-shot NLP benchmarks, while Table 8 shows substantial prefill overhead. Could the authors provide end-to-end latency comparisons that include both prompt prefill and generation for representative prompt/output lengths? Additionally, does the advantage on downstream reasoning tasks persist when the baseline receives a comparable training-compute budget? These results would clarify whether T²MLR offers a practically favorable tradeoff for reasoning-focused deployments.

  2. Can the authors better isolate the benefit of middle-layer recurrence relative to existing temporal-feedback architectures? Could the authors provide a controlled comparison or architectural ablation involving a closely related deep-to-shallow recurrence mechanism under comparable training and inference budgets? For example, comparing different injection locations and fusion mechanisms under an otherwise identical training procedure would help establish which aspects of T²MLR are responsible for the improvements. This could materially strengthen the novelty and architectural conclusions.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 2b5H (part 1/3)

Response to Reviewer 2b5H (part 1/3)

We thank the reviewer for the positive assessment and for noting that the S5-Retrieval experiment gives clear evidence of the intended inductive bias. Below, we first summarize our responses and then address each point in detail.

Summary of responses

  • Weakness 1 (training and prefill overhead). T²MLR spends more compute in training and generates better at close to the decoding cost of a Transformer. We agree with the reading of Table 11 that a Transformer given the same training compute is slightly ahead on zero-shot NLP, and we explain below why we think the extra training compute is well spent. We measured the full inference cost. End-to-end latency including prompt prefill is within 7 to 11 percent of a Transformer for generation lengths typical of reasoning. We also retrained the models with a cheaper schedule that reduces the recurrent training overhead from 4.11 times to 1.65 times at 135M and from 4.56 times to 1.74 times at 362M.
  • Weakness 2 (closest temporal-recurrence architectures). We trained the Full-bandwidth and Latent Recurrent Transformers in our own pipeline at 135M. Both end level with the Transformer while T²MLR improves on it, and a new measurement shows the model relies on the carried state about five times more for a middle window than for full-layer recurrence.
  • Question 1 (matched budgets on reasoning). At a training budget matched to the complete Transformer finetune, the 1.7B retrofit still improves GSM8K from 35.8 to 38.7 and MATH500 from 12.8 to 15.8.
  • Question 2 (isolating the design). We trained eight variants of the fusion module with the window fixed. All of them improve on the Transformer and they span 0.016 in eval loss, so the gain does not come from the gating.

Weakness 1: training and prefill overhead.

The training slowdown and the prefill cost weaken the practical efficiency argument.

T²MLR spends more compute in training and, in return, generates better at close to the decoding cost of a Transformer. The comparisons in the paper hold parameters, training data and decoding cost fixed, and T²MLR is more accurate under those comparisons. The price is training compute. You are right that a Transformer given the same training compute, spent on additional training tokens, is slightly ahead on zero-shot NLP (45.30 against 44.14 in Table 11), and we should have said so in the main text. We think paying for the gain in training is worthwhile for three reasons. First, training compute is paid once, while decoding cost is paid at every generated token. Looped and pause-token models buy their gains with extra passes at every generated token, and T²MLR needs one pass. Second, a Transformer can turn extra training compute into accuracy only when additional data is available. When the training data is fixed, as in finetuning on a reasoning corpus, T²MLR stays ahead at matched training compute: the 1.7B retrofit improves GSM8K from 35.8 to 38.7 (Question 1 below). An independent study of top-layer latent feedback (B. Huang et al., "Latent Feedback Is a Compute and Data Amplifier", alphaXiv, November 2026) makes the same comparison with the Transformer spending the matched compute on repeated epochs: the feedback model reaches a given loss with up to 48% less training compute when unique data per parameter is plentiful, and needs about 8% more when unique data is scarce. The feedback design in the study differs from ours, so we cite the study only for the role of the data regime. Third, the training overhead can be reduced: measurement (b) below cuts the overhead by about 2.5 times at the same accuracy. Two new measurements make the cost concrete.

(a) End-to-end inference is within 7 to 11 percent of a Transformer for reasoning-length generations. Following your suggestion, we measured prompt prefill and generation together on one H100 at 135M with a 2048 token prompt. Prefill with eight iterations takes 49 ms for the D=6 window against 20 ms for the Transformer, and generation takes 12.6 ms per token against 11.8 ms. We find that end-to-end latency is governed by decoding rather than by prefill:

Output tokensTransformerT²MLR D=6T²MLR D=14
1281.54 s1.67 s1.70 s
5126.08 s6.52 s6.55 s
204824.28 s25.95 s25.98 s

Prefill is a one-time cost of tens of milliseconds while generation runs for seconds, so its share falls as the output grows. At 2048 output tokens the overhead is 7 percent for both windows. Prefill dominates only for very short outputs, reaching 1.94 times at a single generated token. Batching reduces the prefill share further, from 2.45 times at batch 1 to 1.46 times at batch 32.

AuthorsOpenReview ↗

Response to Reviewer 2b5H (part 2/3)

Response to Reviewer 2b5H (part 2/3)

Weakness 1 (continued).

(b) A cheaper training schedule reduces the recurrent overhead by about 2.5 times. To reduce the training cost, we retrained the models with a local-sequential, global-parallel schedule, adapting the cyclic Gauss-Seidel fixed-point iteration of [1, Section 3.3] to our recurrent cache. Positions are split into g strided groups by t mod g and the groups are updated in order within one pass, so updates are sequential inside any window of g consecutive tokens and parallel across the sequence. Each pass costs the same as a Jacobi pass, and one pass advances the recurrent chain g positions instead of one. This is a training-time change only. The model is architecturally identical and we evaluate it through the same inference path. We find that the schedule cuts the recurrent training overhead by about 2.5 times at both scales while matching the model it replaces:

135M, recurrence window 8 to 21, 10B tokens.

Backbone FLOPsLossARC-CARC-EHSPIQAOBQAWGSciQAverage
Transformer1.00x3.11724.4945.5029.6460.5531.2048.6261.6043.08
T²MLR, Jacobi4.11x3.07923.8945.0330.3161.4329.4051.4666.0043.93
T²MLR, local-sequential1.65x3.06724.4045.2430.3360.6630.8052.7263.2043.91

362M, recurrence window 9 to 24, 10B tokens.

Backbone FLOPsLossARC-CARC-EHSPIQAOBQAWGSciQAverage
Transformer1.00x2.73227.1352.6537.1865.5134.0051.8571.0048.48
T²MLR, Jacobi4.56x2.71028.8452.4838.6666.0033.4052.0973.1049.22
T²MLR, local-sequential1.74x2.71228.5853.6238.9965.6733.6051.0773.6049.30

FLOPs overhead is forward plus backward relative to the Transformer backbone, including the language modelling head. The two schedules land within 0.08 points of each other on the seven-task average at both scales, and both improve on the Transformer.

In the revision, we will state next to Table 11 that T²MLR does not reach a given loss with less training compute, add the end-to-end latency measurement to Appendix B.5, and report the local-sequential schedule and its cost.

Weakness 2: comparison against the closest temporal-recurrence architectures.

The primary comparisons use Transformers, LSTMs, looped and pause-token models rather than deep-to-shallow temporal recurrence.

Following your suggestion, we trained the closest architectures in our own pipeline at 135M, with the same data, optimizer and token budget for every model.

(a) A deep-to-shallow pathway on its own does not produce the gain. We reimplemented the Full-bandwidth and the Latent Recurrent Transformer and trained them under our recipe. We find that both end level with the Transformer while T²MLR improves on it:

ModelEval loss
Transformer, three seeds2.951
Full-bandwidth Transformer2.954
Latent Recurrent Transformer2.957
T²MLR, three seeds2.939

The Transformer seeds range from 2.945 to 2.955, and T²MLR is ahead of the Transformer in each paired seed. These two baselines are our own reimplementations at one seed each, tuned as far as the rebuttal period allowed, so we claim only that we could not obtain a gain from them in this setting.

(b) The Feedback Transformer cannot be trained at this budget. Its training is sequential in the token dimension. We measured it at 258 times the time per token of our pipeline, which puts a matched run out of reach for this submission.

(c) The model relies on the carried state about five times more for a middle window than for full-layer recurrence. We replaced the carried state with the state of an unrelated sequence and measured the cost over the following 256 tokens. We find that the middle window depends on the carried state far more than the alternatives:

Recurrence windowCost of replacing the carried state
Middle1.9 nats
Full-layer0.4 nats
Last block0.15 nats

This gives a direct mechanism for the placement result in Table 2. A middle window carries intermediate computation that the next token actually uses, while a last-block window carries a representation already specialized for predicting the next token.

In the revision, we will add these baselines and the state-replacement measurement to Section 4 and report the reimplementation details in the appendix.

References

[1] Zhang and Ren. WhiteMatter. arXiv:2608.18486, 2026.

AuthorsOpenReview ↗

Response to Reviewer 2b5H (part 3/3)

Response to Reviewer 2b5H (part 3/3)

Question 1: matched end-to-end inference and training budgets on reasoning tasks.

Does the reasoning advantage persist when the baseline receives a comparable training-compute budget?

(a) At matched training compute the retrofit still improves reasoning. Following your suggestion, we evaluated the 1.7B retrofit checkpoint whose training cost equals that of the complete Transformer finetune. We find that the reasoning gain holds at a matched budget:

ModelGSM8KMATH500
Transformer finetune35.812.8
T²MLR retrofit, matched compute38.715.8

The matched-compute checkpoint reaches most of the gain reported in Table 3, which uses the full retrofit budget.

In the revision, we will add the matched-compute row to Table 3.

Question 2: isolating middle-layer recurrence from the fusion design.

Which aspects of the design are responsible for the improvement?

(a) The fusion design is not what produces the gain. Following your suggestion, we fixed the window and trained eight variants of the fusion module, from the one in the paper down to a plain additive connection with no gates. We find that all eight improve on the Transformer and they differ from one another by at most 0.016 in eval loss:

VariantEval loss
Fusion module in the paper, three seeds2.939
Plain additive connection, three seeds2.934
Remaining six variants, one seed each2.932 to 2.948

The plain additive version is at least as good as the module in the paper. Together with the placement ablation in Table 2 and the state-replacement measurement above, this locates the gain in the recurrent pathway and its placement rather than in the gating.

In the revision, we will define the additive variant in the appendix, give the full table of eight variants, and revise the sentences in Section 3 that attribute the gain to the gating.

Thank you again for your constructive review! We hope these new results address your concerns, and we are happy to answer any follow-up questions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of revisions and new results (part 1/2)

We thank the reviewers for their careful reading. In response we trained ETD-style and Griffin-style baselines and our reimplementations of the Full-bandwidth and Latent Recurrent Transformers in our pipeline, repeated the main comparison over three seeds, ablated the fusion module, evaluated the 1.7B retrofit at matched training compute, measured end-to-end latency including prefill, and retrained T²MLR with a cheaper training schedule. Each reply to a reviewer is self-contained. This comment summarises, for the AC and all reviewers, the concerns raised across the reviews and what we did about each concern.

The gain of T²MLR comes from recurrence over the middle layers. Reviewers 2b5H (Weakness 2) and 9Gkg (Weaknesses 1 to 3) asked whether the gain is specific to T²MLR, or would also come from any deep-to-shallow connection or from earlier middle-layer methods. The Full-bandwidth and Latent Recurrent Transformers send a deep state through the full depth of the network. Our reimplementations of the two architectures, trained in our pipeline, end level with the Transformer, while T²MLR improves on the Transformer in each of three seeds. The reimplementations were tuned as far as the rebuttal period allowed. A fusion ablation over eight variants shows that the gain remains when the gates are removed, and the placement ablation in the paper (Table 2) shows that a middle-layer window beats full-layer, early and late windows. A new measurement shows that the model relies on the carried state most when the window is in the middle layers: replacing the carried state with the state of an unrelated sequence costs 1.9 nats for a middle window and 0.4 nats for full-layer recurrence. ETD loops a middle block within each token, and an ETD-style model in our pipeline reaches the same eval loss as T²MLR while applying 1.4 times the layers at every generated token. A Griffin-style model reaches a lower eval loss at 135M (2.912 against 2.939 for T²MLR); Griffin replaces the backbone and is trained from scratch, while T²MLR is added to an unchanged Transformer and can be retrofitted into a pretrained model.

T²MLR spends more compute in training and, in return, generates better at close to the decoding cost of a Transformer. Each generated token takes one forward pass, and measured end-to-end latency, including prompt prefill, is 7 to 11 percent above a Transformer at generation lengths typical of reasoning. Reviewers 2b5H and 7icW (Weakness 1 in both reviews) note that a Transformer given the same training compute, spent on additional training tokens, is slightly ahead on zero-shot NLP (Table 11), and the revised paper says so in the main text. We think the extra training compute is well spent for three reasons. Training compute is paid once, while decoding cost is paid at every generated token. On reasoning, T²MLR is ahead even at equal training compute: the 1.7B T²MLR retrofit scores 38.7 on GSM8K and the Transformer finetune scores 35.8. The training overhead can also be reduced, as the next paragraph shows.

A new training schedule reduces the training FLOPs of T²MLR by about 2.5 times with the same zero-shot accuracy. We retrained T²MLR with a local-sequential, global-parallel schedule, adapting the cyclic Gauss-Seidel fixed-point iteration of [1] to our recurrent cache. Each pass costs the same as a Jacobi pass and advances the recurrent chain several positions, so fewer passes are needed. The replies to Reviewers 2b5H and 7icW describe the schedule in full. The new schedule changes training only, and the resulting model is evaluated through the same inference path as before. In the table, FLOPs are relative to the Transformer and the average is over the seven zero-shot tasks.

ScaleWindowJacobi FLOPsNew schedule FLOPsJacobi averageNew schedule average
135M8 to 214.11x1.65x43.9343.91
362M9 to 244.56x1.74x49.2249.30

In our larger models the recurrence window is centred on the network, so its position was chosen without a sweep. Reviewer 7icW (Weakness 2) asked how the windows at larger scales were chosen. The sweep at 135M found the best window at the centre of the network, and the 1B model and the 1.7B retrofit place the window at the centre without a further sweep. The angular-distance criterion of ETD, computed without any training, selects a block within a few layers of each window, and the revised paper offers the criterion as a cheap check on the placement.

References

[1] Zhang and Ren. WhiteMatter. arXiv:2608.18486, 2026.

AuthorsOpenReview ↗

Summary of revisions and new results (part 2/2)

The from-scratch gain is shown up to 1B parameters and 50B tokens, and the retrofit gain is shown for one model, SmolLM2-1.7B-Instruct. Reviewer 9Gkg (Weaknesses 4 and 5) pointed out that one retrofit does not support the claim that existing Transformers can be upgraded in general. We agree, and the revised paper claims the retrofit result for SmolLM2-1.7B-Instruct only. The gain of that retrofit holds at matched training compute.

Additions to the paper. The revised paper adds the ETD-style and Griffin-style baselines, the comparison with our reimplementations of the Full-bandwidth and Latent Recurrent Transformers, the end-to-end latency table, the matched-compute retrofit result, the state-replacement measurement, a rule for choosing the layer window, an appendix section on a simplified additive fusion module, the local-sequential training schedule, and a retrofit claim stated for SmolLM2-1.7B-Instruct, the model we retrofit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.