- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper explores Looped Transformers and proposes Fully Looped Transformer (FLT). Instead of passing the previous loop output only into the next loop's input, FLT uses it as the attention query at every layer while the input to the subsequent loop is the input embedding. The paper attributes high-loop instability to gradient oscillation and residual explosion, and reports stable training through 12 loops, comparisons against LT/input-injection variants, several attention mechanisms, and fixed-checkpoint inference-depth sweeps.
Critical strengths
- Very simple stabilisation idea
I like how little FLT changes. Very simple and elegant idea. Reusing the existing attention projection to expose the recurrent state to every layer is simple, parameter free, and the reported high-loop stability is interesting.
- Breadth of ablations
I especially appreciate the authors testing GQA, MLA, SWA, and full attention rather than showing the method with only one attention implementation. Though I do not think it establishes the benefit of FLT itself without matched LT controls for each variant.
Critical weaknesses
- The proposed mechanism is not cleanly isolated.
I think the cleanest missing control is a rescaled residual on LT such as
or
The next hidden state = (layers seen / layers seen + 1) * hidden state + (1 / layers seen + 1) * current sublayer output
This directly controls residual scale which could address the residual explosion identified in the paper.
- The experimental case for FLT over simpler stable alternatives is still limited.
Input injection is already quite stable and competitive. At Small scale it beats FLT on Avg5 at all three loop counts, and at Base-3 it scores 43.22 versus 41.53 for FLT. FLT only clearly pulls ahead at Base-6/9. Due to how FLOPs scale with looping depth higher loops are only favourable under parameter contained regime so common uses would only be in the 3 loop regime.
The scale is also smaller than the headline parameter counts suggest. Using the paper's own parameter formula,
the 318M Base model contains only roughly 85M non-embedding Transformer parameters, with most parameters in the untied embedding and LM head. I would be considerably more convinced by one experiment with a substantially larger actual backbone.
Questions for authors
- Diagnostics and mechanism.
Could you add a rescaled-residual control
that keeps hidden-state scale roughly controlled? This would isolate the degradation caused by hidden-state norm growth.
- Why specifically inject through , and what is the wall-clock overhead?
The paper motivates injection partly through KV-cache compatibility. Why not compare , , and injection directly?
Could you first report measured training throughput, prefill/decode latency, and KV-cache memory for FLT versus LT at matched model size and loop count? Since comes from the previous-loop state while come from the current layer state, FLT cannot directly use the conventional single-input fused QKV projection. What implementation do you use, and what wall-clock overhead does this introduce?
Conversely, because every layer in a loop uses the same previous-loop hidden state as its source, the layer-specific query projections could in principle be concatenated into one larger projection and then split across layers. I would be interested in whether this is used.
For - or -side injection, could the previous-loop hidden-state history instead be cached once and shared across layers while only the remaining projected stream stays layer-specific? This would greatly reduce cache storage. If you evaluated these alternatives, please report their cache-memory and latency tradeoffs.
- Attention variants.
I appreciate Table 2, but showing that FLT itself trains with four attention variants does not show that FLT is what provides the stability advantage under each variant. Could you include matched LT or input-injected LT controls?
Also, why does full attention have the worst WikiText PPL of all four variants, 41.12 versus 38.76--39.68, while BPB/Core do not show the same ordering?
- Scale and stable baselines.
Could you test one substantially larger non-embedding backbone? The current Base model is nominally 318M but only about 85M of this is the Transformer backbone, and the remaining advantage over input injection is much smaller than the headline comparison against vanilla LT.
7 comments
Response to Reviewer pxeh (2/2)
Response to Question 2
The implementation of FLT does not require splitting a fused QKV projection, and introduces no additional computational overhead. In our implementation, Q, K and V are already three separate linear projections; no fused QKV projection is used. FLT merely replaces the input of the Q projection with the output of the previous loop, so the parameter count and the FLOPs are identical to those of LT. We measured the overhead on a single A100 at the Base scale with 12 loops, and the three models are nearly identical. At 3, 6 and 9 loops, all gaps are likewise within 5%.
| Model | Training step (s) | Peak training memory (GB) | Prefill (ms) | Decode (ms/token) |
|---|---|---|---|---|
| LT | 0.424 | 34.5 | 99.3 | 96.5 |
| LT_i | 0.442 | 34.5 | 99.1 | 96.8 |
| FLT | 0.426 | 34.5 | 99.5 | 96.2 |
Thank you for the suggestion of concatenating the Q projections. In FLT, the input to the Q projection is identical across all layers within one loop, so concatenating the per-layer into a single matrix, performing one matrix multiplication and then splitting by layer is indeed feasible, and it adds no parameters. Our current implementation does not use this scheme; we will add this optimization to the public code.
Injecting through K/V would violate a design principle of FLT. FLT does not alter the K/V stream of standard attention, which is how it avoids redundant computation. We inject through Q so that the key-value stream retains the same form as in standard attention, making it directly compatible with existing KV cache mechanisms and with related KV-cache compression methods. If we injected through K or V, the cached content would become a projection of the previous-loop state. Your proposal of a cross-layer shared cache in fact shows that K/V injection requires a new cache design: caching a single copy of the previous-loop hidden state means that, for every generated token, every layer must recompute the projection over all past tokens, trading redundant computation for memory. This is exactly the extra design that FLT seeks to avoid.
Second, V injection contradicts the rationale of Attention Injection (AI). The output of attention is a weighted sum of value vectors: Q and K determine the weights, while V determines what is written into the residual stream. AI is designed so that the previous-loop state participates only in computing the weights, while the content written into the residual stream still comes from the current layer. Lines 253–256 of the paper read:
This is analogous to the role of gates in LSTM and GRU: the recurrent state only controls which information is selected, and is not written in directly. Under V injection, the previous-loop state becomes the content written into the residual stream, which amounts to adding the previous-loop state back into the residual stream after a linear projection, returning to the direct-addition scheme of FLT_res. The ablation in the main text shows that FLT_res cannot train stably at the Base scale with 12 loops. Q injection is therefore the only choice that satisfies both conditions.
Response to Question 3
Under the setting of Table 2 in our paper, both LT and input-injected LT already collapse during training. Figure 4 uses exactly the setting of Table 2, namely the Base scale with 12 loops: LT and FLT_res remain stuck at a high loss, LT_i and LT_ai collapse during training, and only FLT trains stably. It is therefore not possible to train the corresponding LT and input-injected LT as controls.
The ordering of PPL differs from that of BPB and Core mainly because the three are evaluated on different data. PPL is evaluated on WikiText-2, BPB on the validation set of FineWeb-Edu, and Core is an aggregate score over downstream tasks. All three may fluctuate under the influence of different attention mechanisms during training. This does not affect the conclusion we draw here: FLT is compatible with different attention variants.
We thank you for your insightful comments, and we hope our response resolves your questions about our work.
Thank you for testing the residual scaling control and adding the slightly larger-backbone comparison. Interesting and insightful results. The 287M non-embedding backbone experiment partially addresses my scale concern, and the improvement over input injection is encouraging. However, the larger-backbone evidence remains limited to a single run at one six-loop configuration. The backbone is also still relatively modest. These results strengthen the empirical case.
Response to Reviewer pxeh (1/2)
Dear Reviewer pxeh,
We greatly appreciate your positive assessment of our work, and in particular your attention to the ablation studies. Below are our responses to your critical weaknesses and questions.
Response to Critical Weaknesses 1 & Question 1
Thank you for sharing such an interesting insight. Following the formula you provided, we added a control, which we denote LT_rs. The experimental setting is the same as that of Figure 2: 6 layers, 12 loops. The conclusion is that a rescaled residual does not stabilize LT. LT_rs collapses just as vanilla LT does, with its validation BPB remaining at 2.37, while FLT has already dropped to 1.16 at the same step. By step 860 the collapse was unambiguous, so we terminated this run early.
The detailed results are shown below:
The rescaled residual does lower the residual scale. The hidden-state norm is about 1/5 of that of LT, but it is still growing, and by step 800 it is already 8 times that of FLT. More importantly, gradient oscillation is far more severe than in vanilla LT: the standard deviation of the gradient norm rises from 1.54 to 96.8, and the maximum spike from 32 to 2,440. Adjusting the residual scaling coefficient alone neither constrains the residual stream persistently nor resolves gradient oscillation. This is exactly why FLT addresses these two symptoms with FLA and AI respectively.
Response to Critical Weaknesses 2 & Question 4
We agree that LT_i is better at Small and at Base with 3 loops, and we also agree that more loops only pay off under a parameter-constrained regime. But the parameter-constrained regime is exactly the setting this paper targets. In this setting, the point of looping is to trade the same parameters for better performance: under the same parameters and the same data, going from non-looped to FLT with 9 loops at Base raises Avg5 from 39.93 to 43.43. This requires performance to keep improving with the loop count, and FLT is the only variant at Base whose performance improves monotonically with the loop count.
To better demonstrate the advantage of our method, we trained a model with a larger backbone: 18 layers, containing 287M non-embedding parameters, 2.4× that of Base. Both FLT and LT_i loop 6 times and are trained for 10,000 steps with identical hyperparameters.
Throughout training, the validation BPB of FLT is lower than that of LT_i, with a gap comparable to that at the Base scale, and FLT also leads on both CORE and Avg5. Under the same setting, FLT_res, which removes AI, collapsed during training, showing that AI remains necessary on a larger backbone.