acceptodds
Under review as a conference paper at ICLR 2027

Simply Stabilizing the Loop via Fully Looped Transformer

Abstract

Scaling model performance typically requires increasing model size. Looped Transformer (LT) offers a parameter-efficient alternative by repeatedly reusing the same Transformer blocks, converting additional computation into performance gains without increasing the parameter count or context length. However, increasing the number of loop iterations makes LT increasingly difficult to optimize, often leading to gradient oscillation, residual explosion, and training collapse. We propose the Fully Looped Transformer (FLT), a simple, parameter-free redesign that addresses these problems without adding learnable parameters or imposing specially designed optimization constraints. FLT combines a Fully Looped Architecture, which exposes the recurrent state to every layer, with Attention Injection, which controls recurrent information flow through the existing attention module. These modifications enable FLT to train stably through 12 loops in our tested configurations, while baseline looped models enter high-loss regimes. In the Base-size, 6-loop setting, FLT improves downstream-task accuracy by 14.9% relative to the original LT. Fixed-loop checkpoint evaluations further show that performance improves as inference depth approaches the trained depth, even without randomized loop-count training, revealing a depth-aware test-time compute interface. Extending this interface beyond the trained range is a promising research direction.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Looped Transformer, Pre-training Instability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_pxehOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores Looped Transformers and proposes Fully Looped Transformer (FLT). Instead of passing the previous loop output only into the next loop's input, FLT uses it as the attention query at every layer while the input to the subsequent loop is the input embedding. The paper attributes high-loop instability to gradient oscillation and residual explosion, and reports stable training through 12 loops, comparisons against LT/input-injection variants, several attention mechanisms, and fixed-checkpoint inference-depth sweeps.

Critical strengths
  1. Very simple stabilisation idea

I like how little FLT changes. Very simple and elegant idea. Reusing the existing attention projection to expose the recurrent state to every layer is simple, parameter free, and the reported high-loop stability is interesting.

  1. Breadth of ablations

I especially appreciate the authors testing GQA, MLA, SWA, and full attention rather than showing the method with only one attention implementation. Though I do not think it establishes the benefit of FLT itself without matched LT controls for each variant.

Critical weaknesses
  1. The proposed mechanism is not cleanly isolated.

I think the cleanest missing control is a rescaled residual on LT such as

hl+1=ll+1hl+1l+1fl(xl)h_{l+1} = \frac{l}{l+1} h_l + \frac{1}{l+1} f_l(x_l)

or

The next hidden state = (layers seen / layers seen + 1) * hidden state + (1 / layers seen + 1) * current sublayer output

This directly controls residual scale which could address the residual explosion identified in the paper.

  1. The experimental case for FLT over simpler stable alternatives is still limited.

Input injection is already quite stable and competitive. At Small scale it beats FLT on Avg5 at all three loop counts, and at Base-3 it scores 43.22 versus 41.53 for FLT. FLT only clearly pulls ahead at Base-6/9. Due to how FLOPs scale with looping depth higher loops are only favourable under parameter contained regime so common uses would only be in the 3 loop regime.

The scale is also smaller than the headline parameter counts suggest. Using the paper's own parameter formula,

N=2Vdmodel+12Ddmodel2,N = 2Vd_{\mathrm{model}} + 12Dd_{\mathrm{model}}^2,

the 318M Base model contains only roughly 85M non-embedding Transformer parameters, with most parameters in the untied embedding and LM head. I would be considerably more convinced by one experiment with a substantially larger actual backbone.

Questions for authors
  1. Diagnostics and mechanism.

Could you add a rescaled-residual control

hl+1=ll+1hl+1l+1fl(xl)h_{l+1} = \frac{l}{l+1} h_l + \frac{1}{l+1} f_l(x_l)

that keeps hidden-state scale roughly controlled? This would isolate the degradation caused by hidden-state norm growth.

  1. Why specifically inject through QQ, and what is the wall-clock overhead?

The paper motivates QQ injection partly through KV-cache compatibility. Why not compare QQ, KK, and VV injection directly?

Could you first report measured training throughput, prefill/decode latency, and KV-cache memory for FLT versus LT at matched model size and loop count? Since QQ comes from the previous-loop state while K,VK,V come from the current layer state, FLT cannot directly use the conventional single-input fused QKV projection. What implementation do you use, and what wall-clock overhead does this introduce?

Conversely, because every layer in a loop uses the same previous-loop hidden state as its QQ source, the layer-specific query projections could in principle be concatenated into one larger projection and then split across layers. I would be interested in whether this is used.

For KK- or VV-side injection, could the previous-loop hidden-state history instead be cached once and shared across layers while only the remaining projected stream stays layer-specific? This would greatly reduce cache storage. If you evaluated these alternatives, please report their cache-memory and latency tradeoffs.

  1. Attention variants.

I appreciate Table 2, but showing that FLT itself trains with four attention variants does not show that FLT is what provides the stability advantage under each variant. Could you include matched LT or input-injected LT controls?

Also, why does full attention have the worst WikiText PPL of all four variants, 41.12 versus 38.76--39.68, while BPB/Core do not show the same ordering?

  1. Scale and stable baselines.

Could you test one substantially larger non-embedding backbone? The current Base model is nominally 318M but only about 85M of this is the Transformer backbone, and the remaining advantage over input injection is much smaller than the headline comparison against vanilla LT.

7 comments
AuthorsOpenReview ↗

Response to Reviewer pxeh (1/2)

Dear Reviewer pxeh,

We greatly appreciate your positive assessment of our work, and in particular your attention to the ablation studies. Below are our responses to your critical weaknesses and questions.


Response to Critical Weaknesses 1 & Question 1

Thank you for sharing such an interesting insight. Following the formula you provided, we added a control, which we denote LT_rs. The experimental setting is the same as that of Figure 2: 6 layers, 12 loops. The conclusion is that a rescaled residual does not stabilize LT. LT_rs collapses just as vanilla LT does, with its validation BPB remaining at 2.37, while FLT has already dropped to 1.16 at the same step. By step 860 the collapse was unambiguous, so we terminated this run early.

The detailed results are shown below:

BPB↓Step 200Step 400Step 600Step 800Step 2000
LT_rs2.2192.3322.3882.374-
LT2.3352.3762.3752.3742.370
FLT1.3421.2271.1851.1641.078
MetricLT_rsLTFLT
loop_00 hidden-state norm4,10318,836486
Grad norm std96.81.540.08
Max grad norm2,44032.10.87
Steps with grad norm > 584201

The rescaled residual does lower the residual scale. The hidden-state norm is about 1/5 of that of LT, but it is still growing, and by step 800 it is already 8 times that of FLT. More importantly, gradient oscillation is far more severe than in vanilla LT: the standard deviation of the gradient norm rises from 1.54 to 96.8, and the maximum spike from 32 to 2,440. Adjusting the residual scaling coefficient alone neither constrains the residual stream persistently nor resolves gradient oscillation. This is exactly why FLT addresses these two symptoms with FLA and AI respectively.


Response to Critical Weaknesses 2 & Question 4

We agree that LT_i is better at Small and at Base with 3 loops, and we also agree that more loops only pay off under a parameter-constrained regime. But the parameter-constrained regime is exactly the setting this paper targets. In this setting, the point of looping is to trade the same parameters for better performance: under the same parameters and the same data, going from non-looped to FLT with 9 loops at Base raises Avg5 from 39.93 to 43.43. This requires performance to keep improving with the loop count, and FLT is the only variant at Base whose performance improves monotonically with the loop count.

To better demonstrate the advantage of our method, we trained a model with a larger backbone: 18 layers, containing 287M non-embedding parameters, 2.4× that of Base. Both FLT and LT_i loop 6 times and are trained for 10,000 steps with identical hyperparameters.

ModelStep 500 BPB↓Step 2000 BPB↓Step 10000 BPB↓CORE↑Avg5↑
FLT1.08400.95390.84350.2040.325
LT_i1.09230.96200.85110.1980.304

Throughout training, the validation BPB of FLT is lower than that of LT_i, with a gap comparable to that at the Base scale, and FLT also leads on both CORE and Avg5. Under the same setting, FLT_res, which removes AI, collapsed during training, showing that AI remains necessary on a larger backbone.

AuthorsOpenReview ↗

Response to Reviewer pxeh (2/2)

Response to Question 2

The implementation of FLT does not require splitting a fused QKV projection, and introduces no additional computational overhead. In our implementation, Q, K and V are already three separate linear projections; no fused QKV projection is used. FLT merely replaces the input of the Q projection with the output of the previous loop, so the parameter count and the FLOPs are identical to those of LT. We measured the overhead on a single A100 at the Base scale with 12 loops, and the three models are nearly identical. At 3, 6 and 9 loops, all gaps are likewise within 5%.

ModelTraining step (s)Peak training memory (GB)Prefill (ms)Decode (ms/token)
LT0.42434.599.396.5
LT_i0.44234.599.196.8
FLT0.42634.599.596.2

Thank you for the suggestion of concatenating the Q projections. In FLT, the input to the Q projection is identical across all layers within one loop, so concatenating the per-layer WQW_Q into a single matrix, performing one matrix multiplication and then splitting by layer is indeed feasible, and it adds no parameters. Our current implementation does not use this scheme; we will add this optimization to the public code.

Injecting through K/V would violate a design principle of FLT. FLT does not alter the K/V stream of standard attention, which is how it avoids redundant computation. We inject through Q so that the key-value stream retains the same form as in standard attention, making it directly compatible with existing KV cache mechanisms and with related KV-cache compression methods. If we injected through K or V, the cached content would become a projection of the previous-loop state. Your proposal of a cross-layer shared cache in fact shows that K/V injection requires a new cache design: caching a single copy of the previous-loop hidden state means that, for every generated token, every layer must recompute the projection over all past tokens, trading redundant computation for memory. This is exactly the extra design that FLT seeks to avoid.

Second, V injection contradicts the rationale of Attention Injection (AI). The output of attention is a weighted sum of value vectors: Q and K determine the weights, while V determines what is written into the residual stream. AI is designed so that the previous-loop state participates only in computing the weights, while the content written into the residual stream still comes from the current layer. Lines 253–256 of the paper read:

This is analogous to the role of gates in LSTM and GRU: the recurrent state only controls which information is selected, and is not written in directly. Under V injection, the previous-loop state becomes the content written into the residual stream, which amounts to adding the previous-loop state back into the residual stream after a linear projection, returning to the direct-addition scheme of FLT_res. The ablation in the main text shows that FLT_res cannot train stably at the Base scale with 12 loops. Q injection is therefore the only choice that satisfies both conditions.


Response to Question 3

Under the setting of Table 2 in our paper, both LT and input-injected LT already collapse during training. Figure 4 uses exactly the setting of Table 2, namely the Base scale with 12 loops: LT and FLT_res remain stuck at a high loss, LT_i and LT_ai collapse during training, and only FLT trains stably. It is therefore not possible to train the corresponding LT and input-injected LT as controls.

The ordering of PPL differs from that of BPB and Core mainly because the three are evaluated on different data. PPL is evaluated on WikiText-2, BPB on the validation set of FineWeb-Edu, and Core is an aggregate score over downstream tasks. All three may fluctuate under the influence of different attention mechanisms during training. This does not affect the conclusion we draw here: FLT is compatible with different attention variants.

We thank you for your insightful comments, and we hope our response resolves your questions about our work.

Thank you for testing the residual scaling control and adding the slightly larger-backbone comparison. Interesting and insightful results. The 287M non-embedding backbone experiment partially addresses my scale concern, and the improvement over input injection is encouraging. However, the larger-backbone evidence remains limited to a single run at one six-loop configuration. The backbone is also still relatively modest. These results strengthen the empirical case.

Reviewer_z8neOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Looping the same layers several times gives a model more compute without more parameters, but looped models get unstable as you add loops. This paper blames growing hidden states and early gradient oscillations, and proposes the Fully Looped Transformer: each loop restarts from the input embeddings, and every layer uses the previous loop's output as its attention query, so no parameters are added. They train small GPT-style models on FineWeb-Edu and compare against standard looped baselines at different loop counts. The takeaway is that a simple change lets looped models train stably and keep improving with more loops.

NOTE to AC: the PDF title doesn't match the OpenReview title and this triggered a desk reject on another paper in my pool but IMO you should not desk reject this paper bc both titles have the same concepts mentioned, it's just reworded

Critical strengths
  • Simple, parameter-free change that's easy to drop into existing looped models, and it works with several attention types (GQA, MLA, sliding window)
  • Fair comparison among looped variants: same hyperparameters and token budget, and code for everything is released
  • At the larger size, FLT is the only variant that keeps getting better from 3 to 6 to 9 loops, which is exactly what the paper sets out to fix
  • Upfront about limitations and keeps collapsed runs in the tables which is GREAT
Critical weaknesses

= There's no non-looped baseline, so it's unclear whether looping is worth it at all. Compute grows with loops, and FLT at 9 loops roughly matches the input-injection baseline at 3. Saunshi et al. and Mixture-of-Recursions both compare against same-parameter and same-compute models.

  • The gains are small. The 14.9% headline is against a baseline that had already gotten worse with more loops. Against the stronger baseline it's under a point from single runs, and at the smaller size the baselines win.
  • The stability motivation doesn't seem totally sound to me. The baselines already normalize the state between loops, and FLT's smaller hidden states are partly by construction since it restarts from the embeddings. Everything also uses one untuned learning rate, but Geiping et al. needed a much lower LR to stabilize looped models.
  • Close prior work isn't discussed. Parcae targets the same problem with the same diagnosis, and in the Feedback Transformer every layer also attends to top-layer states.
Questions for authors
  • How does this compare to Parcae, which identifies the same cause (hidden states blowing up across loops) and fixes it by constraining the update between loops? If the problem is just state growth, a principled constraint like that seems like a more direct fix, so it's unclear why restarting from the embeddings and changing the attention query is the right approach.
  • Does FLT actually let you adjust compute at test time? Performance falls apart when the loop count differs from training, so it seems like looping is just equivalent to a fixed-depth model with shared weights. What does looping buy you over a regular deeper model?
4 comments
AuthorsOpenReview ↗

Response to Reviewer z8ne (1/3)

Dear Reviewer z8ne,

Thank you for pointing out the mismatch between the title in the PDF and the title on OpenReview. This was a typo, and we have fixed it in the newly uploaded PDF.

We also appreciate your recognition of our current work. Below are our responses to your critical weaknesses and questions.


Response to Critical Weaknesses 1

The main contribution of this paper is a simple method for resolving the training instability of looped models. The value of looping itself lies in parameter and data efficiency, and FLT allows that value to be realized stably. The comparisons in the main text therefore focus on looped variants with the same parameter count and the same loop count. The compute of these baselines is identical.

To address whether looping itself is worthwhile, we added experiments on non-looped models at the Small and Base scales. The performance comparison between looped and non-looped models is shown below:

MetricSmall L1FLT Small L3Base L1FLT Base L3
PPL↓71.9362.7745.1640.48
BPB↓1.0611.0450.9170.898
Core↑6.205.7213.7215.38
Avg5↑33.0133.4039.9341.53

As we can see, under the same parameters and the same data, FLT outperforms the non-looped model on PPL, BPB and Avg5 at both scales. This indicates that looping is indeed a more data-efficient paradigm.

We agree that FLT at 9 loops is not more compute-efficient than LT_i at 3 loops, and this paper does not claim an advantage in compute efficiency. We did not train a deeper compute-matched non-looped model, because this paper focuses on efficiency under parameter and data constraints.


Response to Critical Weaknesses 2

We believe the comparison against LT is reasonable. LT is the most common baseline form of looped models, and Saunshi et al. (arXiv:2502.17416), cited by the reviewer, studies exactly this directly looped model. More importantly, the fact that LT degrades as the loop count increases is not a matter of a deliberately chosen "weak baseline". It is precisely the problem this paper sets out to solve. At the Base scale, the Avg5 of LT drops from 41.71 at 3 loops to 37.32 at 6 loops, which is even below the non-looped model at 39.93. The 14.9% measures exactly the gain obtained once FLT removes this degradation. At the same time, we also fully report the stronger baseline LT_i in Table 1 of the main text.

What we want to emphasize is that FLT is more stable and more scalable as the loop count and the scale increase: at the Small scale LT_i is better; at the Base scale, FLT surpasses LT_i at 6 and 9 loops, and is the only variant that improves monotonically with the loop count.

In Figure 4 of the main text, our experiments show that at the Base scale with 12 loops, only FLT completes training in a fully stable manner.

We further trained an 18-layer model with 636M parameters, twice the size of Base. Both models loop 6 times and are trained for 10,000 steps.

ModelStep 500 BPB↓Step 2000 BPB↓Step 10000 BPB↓CORE↑Avg5↑
FLT1.08400.95390.84350.2040.325
LT_i1.09230.96200.85110.1980.304

Throughout training, the validation BPB of FLT is lower than that of LT_i, and FLT also leads on both CORE and Avg5. All results come from single runs, but this advantage is present throughout training and does not appear only at a particular checkpoint.

AuthorsOpenReview ↗

Response to Reviewer z8ne (2/3)

Response to Critical Weaknesses 3

We thank the reviewer for this comment. We believe there is a misunderstanding here: the stability of FLT does not come from smaller hidden states.

The hidden states of FLT are not small because every loop restarts from the embedding. As the reviewer notes, all models normalize the state between loops, so the input magnitude of every loop is the same, and restarting does not give FLT a smaller starting point. Figure 2 in the main text shows the output of the first loop, in which the computation of LT and FLT is identical. The difference between the two comes from the different weights learned during training: the architecture of FLT lets the model avoid hidden-state growth during training. Moreover, FLT_res also restarts from the embedding at every loop, yet it collapses at Base with 12 loops and on the 18-layer model, showing that restarting alone does not bring stability.

Lowering the learning rate does not resolve the instability of looped models. In all our experiments we use the default learning rate that nanochat had tuned for non-looped models; we did not select one specifically for FLT. The setting of Geiping et al. (arXiv:2502.05171) cannot be compared directly with our method. They use AdamW while we primarily use Muon, and these two classes of optimizers accommodate different learning-rate scales. Moreover, while lowering the learning rate they also modified the normalization structure, so their stability is not necessarily the result of the low learning rate.

We also ran additional experiments under lowered learning rates, with the results shown below:

SettingFinal val BPB↓loop_00 hidden-state normMean grad normSteps triggering grad clipping
LT ×12.370~50,000——
LT ×0.51.2601,323, still growing——
LT ×0.11.197475.191900 / 1900
FLT ×11.077~1,4000.3113 / 1900

Lowering the learning rate does not resolve all the problems that cause LT to collapse. Although the hidden-state norm of LT is then smaller than that of FLT, gradient clipping is triggered at all 1900 steps, gradient oscillation is more pronounced, and learning is markedly slower. A small hidden state is therefore not equivalent to stable training. FLT, by contrast, trains stably without any learning-rate tuning, and this is precisely the advantage we wish to emphasize.

AuthorsOpenReview ↗

Response to Reviewer z8ne (3/3)

Response to Critical Weaknesses 4 & Question 1

Thank you for these valuable references. On examination, however, we find that although Parcae and FLT study a closely related problem, the two model settings differ fundamentally, which makes a fair comparison difficult. Parcae loops only a number of middle layers, with a non-looped segment before and after; its initial recurrent hidden state is sampled from a random normal distribution, and the input is injected at every loop through a newly introduced learnable matrix. The looped models used in this paper loop the entire model, and the loop starts from the input embeddings. In addition, the stability analysis of Parcae is derived for a linear system after removing the nonlinear parts of the Transformer block, and addresses only state growth. Because the architectural settings and the extra parameters required differ between the two, a direct comparison could not attribute the difference in stability to the stabilization method itself, so we did not make one.

Regarding Question 1, we would like to point out that the design of FLT does not merely suppress hidden-state growth. In Section 3 we diagnose two symptoms of the instability of looped models, and FLT is designed precisely against these two symptoms. The beginning of Section 4, lines 206–208, reads:

"In Section 3, we identified two patterns associated with Looped Transformer's training instability: gradient oscillation and residual explosion. We design Fully Looped Transformer to mitigate these two problems."

Among them, FLA suppresses residual explosion and AI suppresses gradient oscillation. Section 4.2, lines 248–251, explains the rationale behind AI:

"A key role of these gates is to modulate the information in residual flow within a bounded range, which helps control the magnitude of propagated signals and gradients. Inspired by this principle, we propose Attention Injection (AI) …"

Constraining hidden-state growth alone does not in itself resolve the instability of looped models, which is the conclusion we already drew in Critical Weaknesses 3. The contribution of FLT is that a single architectural change addresses both problems without adding any parameters. We will add a discussion of Parcae and of the above differences to the related work of the revised version.

Feedback Transformer is architecturally entirely different from our FLT. Feedback Transformer combines the outputs of all layers at each time step into a single memory through a weighted sum. The attention of each layer then draws its K/V from the memory of past tokens. It is also not a looped model: its layers do not share weights.


Response to Question 2

We acknowledge that performance does degrade when the inference depth differs from the training depth. What we want to emphasize here, however, is a trend: for FLT, the closer the inference depth is to the training depth, the better the performance. This holds across all three metrics, Core, Avg5 and BPB. This suggests that looped models have the potential to flexibly exploit test-time compute, but how to unlock this capability remains to be explored.

We also qualify this in our final conclusion in Section 5.4. Lines 476–477 of the paper read:

"Thus, FLT provides a useful depth-aware test-time compute interface within the trained regime, while reliable extrapolation beyond that regime remains a promising direction for future work."

In lines 26–27 of the abstract, we also state directly that the current model has limitations and that extending its test-time compute capability requires further research.

"Extending this interface beyond the trained range is a promising research direction."

Compared with deeper models, weight sharing brings data-efficiency gains and certain distinctive properties. At the same computational depth, a deeper model has more parameters and therefore requires more data to train. Looped models also hold particular advantages over non-looped ones; for example, they make it easier for a model to perform in-context learning [1] or to learn algorithms [2] from data. These topics are unrelated to the subject discussed in this paper, so we did not address them directly in the main text.

We thank you again for your insightful comments, and we hope our response addresses your concerns about our work.

[1] Chen, B., Li, X., Liang, Y., Shi, Z., & Song, Z. (2024). Bypassing the exponential dependency: Looped transformers efficiently learn in-context by multi-step gradient descent. arXiv preprint arXiv:2410.11268.

[2] Yang, Liu, et al. "Looped transformers are better at learning learning algorithms." International conference on learning representations. Vol. 2024. 2024.

Thanks for the detailed response and the new runs.

  • The stability argument is more convincing now, and my point about smaller hidden states being by construction is mostly addressed.
  • I still don't think it's clear that looping is worth it. The new non-looped model has the same parameters but a third of the compute, so FLT beating it is expected. Without a same-compute deeper model, the paper shows FLT fixes a problem with looping, but not the actual benefit of looping in the first place. I know other papers have established this but it matters if your experiments are in that setting or not because otherwise you might be fixing an instability in a sort of unrelated setting to the useful one.
  • The gains are still small and come from single runs. The bigger 18-layer model helps, but it's still just one run at one loop count. I know it's unfair to ask authors for more and more experiments, but I don't see the gains as significant enough to provide a clear signal.
  • On Parcae, I get that the setups differ, but both papers say they fix the same instability. Some direct comparison (or at least trying its constrained injection in your setup) would help show why FLT's fix is the better one.
  • Thanks for being upfront about test-time compute. I'd soften the "test-time compute interface" framing, since it only works well at the trained loop count.

I raised my score to a 3.

Reviewer_U6svOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper tackles the optimization bottlenecks in Looped Transformers at higher loop counts: 1) gradient oscillation and 2) cross-loop residual norm explosion. To circumvent these training instabilities without introducing additional parameter overhead, the authors propose the Fully Looped Transformer. The core contribution lies in two architectural modifications: 1) a Fully Looped Architecture that re-injects raw token embeddings while broadcasting the previous loop’s final state to all layers, and 2) Attention Injection, which repurposes existing projections for cross-attention, bounding injected signal magnitudes via Softmax normalization. Empirical validation demonstrates that the proposed successfully stabilizes training up to 12 loops across various attention variants.

Critical strengths

Parameter-Free Design: The proposed modifications are conceptually straightforward and introduce no additional parameter overhead. Repurposing existing attention projections for cross-attention is cost-effective.

Clear Experiment Design: The pretraining experiments on FineWeb-Edu, though restricted to relatively small model scales, demonstrate that FLT can recover the performance degradation observed in standard multi-loop LTs.

Critical weaknesses

Limited performance gains: the abstract highlights a substantial 14.9% improvement, but this is measured against a degraded 6-loop LT rather than a robust baseline. When compared to the standard Input Injection (Table 1), the Small FLT variant consistently underperforms across all loop counts. At the Base scale, the inefficiency becomes apparent: a 9-loop FLT barely beat 3-loop LT. This leaves the inherent value of the looping paradigm unproven.

Overstated test-time capabilities: the claimed "usable test-time compute interface" is contradicted by the findings in Figure 5, which illustrates collapse when inference depth diverges from training depth.

Questions for authors
  1. Can authors provide more empirical studies to show their proposed method's advantages?

  2. What are the PPL, Core, and Avg5 scores of 1-loop Small/Base models and compute-matched non-looped Transformers under the same setup, and why does Small FLT underperform LTiLT_i?

  3. Since all runs use no warm-up, does adding standard LR warm-up resolve the early 2000-step gradient oscillation in 9/12-loop LT or LT_i?

2 comments
AuthorsOpenReview ↗

Response to Reviewer U6sv (1/2)

Dear Reviewer U6sv,

We thank you for recognizing the cost-effectiveness of our design and the empirical results of our work. Below are our responses to your critical weaknesses and questions.


Response to Critical Weaknesses 1 & Question 1

While LT_i is stronger at the Small scale, FLT overtakes LT_i at 6 and 9 loops at the Base scale, which is the scale we want to emphasize. Moreover, among all the models, only FLT improves monotonically with the loop count.

To provide more convincing evidence, we trained FLT and LT_i under an 18-layer, 6-loop setting. Models in this setting contain 636M parameters, twice the size of Base. Limited by time and compute, we trained for 10,000 steps, with identical hyperparameters on both sides.

The comparison between the two models is shown below. Throughout training, the validation BPB of FLT is lower than that of LT_i, and FLT also achieves higher final task scores. This indicates that FLT has more to offer as scale increases.

ModelStep 500 BPB↓Step 2000 BPB↓Step 10000 BPB↓CORE↑Avg5↑
FLT1.08400.95390.84350.2040.325
LT_i1.09230.96200.85110.1980.304

In addition, we would like to clarify that the goal of this paper is to provide a simple method for stabilizing looped models. We also added experiments on non-looped models at the Small and Base scales. Their comparison against the looped models is shown below:

MetricSmall L1FLT Small L3Base L1FLT Base L3
PPL↓71.9362.7745.1640.48
BPB↓1.0611.0450.9170.898
Core↑6.205.7213.7215.38
Avg5↑33.0133.4039.9341.53

These experiments show that looped models are more data-efficient than non-looped models: at 3 loops, FLT outperforms the non-looped model on PPL, BPB and Avg5 at both scales. However, as reported in Table 1 of the main text, vanilla LT at 9 loops falls to an Avg5 of 24.93 at the Base scale, far below the 39.93 of the non-looped model. This shows that looping pays off only when training is stable, which is exactly the problem FLT solves.


Response to Critical Weaknesses 2

We acknowledge that performance does degrade when the inference depth differs from the training depth. However, we would like to clarify that we do not overstate the test-time capability of our model. What we emphasize is a trend: for FLT, the closer the inference depth is to the training depth, the better the performance. This holds across all three metrics, Core, Avg5 and BPB. This suggests that looped models have the potential to flexibly exploit test-time compute, but how to unlock this capability remains to be explored.

Our conclusion is stated with a corresponding qualification. Lines 476–477 of the paper read:

"Thus, FLT provides a useful depth-aware test-time compute interface within the trained regime, while reliable extrapolation beyond that regime remains a promising direction for future work."

In lines 26–27 of the abstract, we also state directly that the current model has limitations and that extending its test-time compute capability requires further research.

"Extending this interface beyond the trained range is a promising research direction."

AuthorsOpenReview ↗

Response to Reviewer U6sv (2/2)

Response to Question 2

Under the same parameters and the same data, FLT outperforms the 1-loop model on PPL, BPB and Avg5 at both the Small and Base scales, and on Core at the Base scale. The detailed results are given in the table under Critical Weaknesses 1 & Question 1 above.

We did not include a compute-matched comparison. The main contribution of this paper is a simple method for the training instability that is pervasive in looped models, rather than a method with higher data efficiency. Our comparisons therefore focus on looped variants with the same parameter count and the same computational depth.

Why does Small FLT underperform LT_i? We offer one possible explanation: the computational depth of a small model is still shallow, so it is already easier to train stably, and the advantage of FLT is therefore less visible at the Small scale. Our evidence is that as the parameter count grows, both the Base and the 18-layer FLT outperform LT_i. Moreover, Figure 4 in the main text shows that FLT is the only model that completes training stably at the Base scale with 12 loops, indicating that the computational depth of a model has a strong effect on its stability.


Response to Question 3

Warm-up does not resolve gradient oscillation. We ran additional experiments with warm-up. The results show that warm-up suppresses the hidden-state norm but amplifies gradient oscillation. The instability that appears during the training of looped models therefore cannot be fully eliminated by adjusting the learning rate in the early phase of training.

SettingFinal val BPB↓loop_00 hidden-state normSteps with grad norm > 5Max gradient spike
LT L91.1697,853836.2
LT L9 + warm-up 5001.1244,9267116.3
LT L122.37053,185432.1
LT L12 + warm-up 5001.1557,63295585
FLT L121.0775~1,400113.3

We thank you again for your insightful comments, and we hope our response addresses your concerns about our work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to all reviewers

Dear Area Chairs and Reviewers,

Thank you for your time and valuable feedback.

We thank the three reviewers for their careful and constructive reviews. We are encouraged that all three arrived at a largely consistent judgment about the method itself: the change FLT introduces is simple and elegant (pxeh), it adds no new parameters (pxeh, U6sv, z8ne), and it can be dropped directly into existing looped models (z8ne); it does not depend on a particular attention implementation, holding for GQA, MLA, sliding window, and full attention (pxeh, z8ne); it recovers the performance degradation of standard multi loop LT and trains stably up to 12 loops (U6sv), and at the larger scale it is the only variant that keeps improving as the loop count increases, which is exactly the problem this paper sets out to fix (z8ne). The reviewers also acknowledged that our comparisons use the same hyperparameters and token budget, that all code is released, and that collapsed runs are kept in the tables as they are (z8ne).

We would also like to restate the contribution of this paper here. The paper addresses one specific problem: looped Transformers collapse during training as the loop count grows, which makes the path of trading more loops for stronger capability unworkable at high loop counts. We first diagnose two observable symptoms that accompany these failures, then introduce two parameter free architectural modifications that each address one of them, and verify the feasibility of our proposed method FLT in a controlled setting.

In this discussion period we have run additional experiments in response to the reviewers' comments, which are described in the individual replies. We note that Reviewer z8ne and Reviewer pxeh have raised their scores to 3 after the discussion with us, and we are very grateful that both were willing to reassess our work. For the two points raised by Reviewer U6sv, the performance gains and the test time compute claims, we have offered clarifications and added the corresponding experimental evidence in our reply, and we hope this material helps in reassessing the paper.

Finally, we sincerely thank the three reviewers and the AC for the time and effort invested in this round.

Best regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.