acceptodds
Withdrawn from ICLR 2027

T-LoopFormer: Token-Level Elastic-Depth Looped Transformers with Dynamic Routing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Looped Transformers have recently demonstrated strong performance in both reasoning and language tasks by reusing a shared set of parameters across multiple iterations, achieving parameter efficiency without sacrificing representational power. Besides, looped Transformers perform inference directly in the latent space (latent reasoning) to reduce the number of tokens consumed during inference, thereby achieving improved sample efficiency. However, these models typically apply a fixed recursion depth uniformly to every token, leading to suboptimal compute allocation and leaving significant efficiency gains on the table. In this work, we propose token-level elastic-depth looped transformers: T-LoopFormer. T-LoopFormer uses dynamic token-choice routing, which enables each token to adaptively determine its own number of loop iterations based on its hidden state, which could improve the token generation accuracy. Moreover, we further introduce recursion-wise KV cache, which maintains an independent key-value cache for each recursion loop of active tokens; this design ensures that tokens at different depths only attend to their corresponding cached states, effectively enabling faster autoregressive decoding. Extensive experiments show that T-LoopFormer achieves robust performance on a wide range of language modeling and zero-shot reasoning tasks, and our model could reach the lowest decoding latency, which validates the effectiveness of token-choice router and recursion-wise KV cache.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Looped Transformers, Latent Reasoning, Dynamic Routing, KV Cache

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wViyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces T-LoopFormer, a looped transformer architecture that allows for token-level elastic depth through dynamic routing using a depth prediction module. The authors manage to efficiently train the model with gather-scatter operations and apply recursion-wise KV cache to save computation and memory. Experiments on various tasks show that T-LoopFormer outperforms baseline models in terms of performance and efficiency.

Critical strengths
  1. The topic of this paper follows the trend of improving transformer efficiency and performance with looped architectures and dynamic compute allocation, which is currently a hot topic in the field.
  2. The experiments look incredibly nice: the authors show that T-LoopFormer outperforms baseline models with only 1/8 of the parameters under the same compute budget.
Critical weaknesses
  1. The presentation looks a bit rushed. The figures are stretched (Figure 2, 3, 4(a), 4(b)). Section 4.2 Ablation Study is an important part of the work while there are only few lines of description without any analysis.
  2. Missing comparison with strong baselines. Mixture-of-Recursions [1] is a famous work that also uses dynamic routing on looped transformers while the authors do not mention in the manuscript. I believe that there are plenty of other recent works on looped transformers that the authors should compare with.
  3. The authors does not mention details of the early exit baseline. It is known that how early exit is implemented can significantly affect the performance [2,3]. Simply saying "naive early exiting" makes the reader confused.
  4. Lack of novelty. The idea of dynamic routing on looped transformers is not something new. Simply changing a router does not seem to be a significant contribution (and not an intuitively reasonable choice as stated in appendix D), unless it significantly improves the performance compared to existing works. Unfortunately, the authors do not provide any comparison with named works like MoR or other recent works on looped transformers.

This work follows the recent trending topic of looped transformers but I believe 1) it requires a significant revision before it can be accepted and 2) the result is not strong enough to be accepted as a paper in ICLR.

[1] Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation. Sangmin Bae, Yujin Kim, Reza Bayat, Sungnyun Kim, Jiyoun Ha, Tal Schuster, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji, Aaron Courville, Se-Young Yun.
[2] BERT Loses Patience: Fast and Robust Inference with Early Exit. Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, Furu Wei.
[3] LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding. Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed A Aly, Beidi Chen, Carole-Jean Wu.

Questions for authors
  1. About TPOT. Theoretically, in T-LoopFormer, if we remove the router s.t. all tokens go through all layers with full depth, the model should have the same computational cost as a standard transformer. However, in Figure 3(b) and Table 2, this seems not be the case. Why is that?
  2. In Figure 1(b), the authors show that the average depth of the model is 3.69. Based on the pie chart, this number seems to be calculated as 1*36.85% + 2*23.73% + 3*8.14% + 4*13.87% + 5*2.16% + 6*11.26% + 7*3.77% + 8*0.23% = 2.708.
Reviewer_JiDPOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper adds token-specific recursion depths to a parameter-shared Transformer. A router predicts depth from the initial embedding; gather–scatter computation and recursion-wise KV caches reduce overhead. Experiments on a roughly 1B model report improved perplexity, benchmark accuracy, and decoding efficiency at the highest evaluated compute budget.

Critical strengths
  1. The architecture connects adaptive depth with practical cache and execution changes, reporting both accuracy and latency improvements.
  2. Router and cache ablations help separate their contributions to quality and efficiency.
Critical weaknesses
  1. The contextual basis of routing is unclear. The embedding-only input cannot distinguish the same token at the same position across different prefixes. Consequently, heterogeneous depth allocation alone does not establish adaptive reasoning, and the benefit over existing adaptive-depth approaches remains insufficiently isolated.

  2. Router training is insufficiently specified. Appendix C invokes an STE but does not clearly show its differentiable connection to task loss through hard token selection. Gradients through gathered values alone do not establish that the router learns useful depth decisions.

Questions for authors
  1. What evidence shows that embedding-only routing captures contextual difficulty rather than token and position preferences?
  2. What does one-shot routing add relative to ACT/MoD-style adaptive depth, and why is it preferable to routing from a processed hidden state?
  3. Where is the STE-valued gate connected to the task loss, and how does the router receive gradients despite discrete gather indices?
  4. Are the comparisons matched by nominal maximum depth or realized token-level FLOPs, and how does the latency advantage vary with batch size and context length?
  5. What explains the lower-budget degradation, and do the selected depths track reasoning difficulty or merely how often tokens reach the loop cap?
Reviewer_yCPxOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper extends LoopFormer with a token-level router that decides how many loops each token goes through. It also adds gather-scatter, so only active tokens are computed, and a per-loop KV cache that stores only the tokens that reach each loop. In the experiment, it matches the perplexity of a 24-layer baseline at 24x nominal FLOPs. It also reports the better decoding latency and training memory among the compared models.

Critical strengths
  • The efficiency result is promising: a 3-layer shared block, looped about 3.7 times per token on average, reaches the same perplexity as a 24-layer standard Transformer, with about 1/8 of the layer parameters and less than half of the per-token compute.
Critical weaknesses
  • The core ideas are already in Mixture-of-Recursions (Bae et al., 2025), which is not cited. MoR already uses routers to give each token its own recursion depth in a shared block, and it proposes "recursion-wise KV caching" with the same name and mechanism. The claim that prior looped models give every token the same depth is also not very accurate: the Universal Transformer (Dehghani et al., 2019) already has per-position halting.

  • Some key claims are not well supported.

    • The router only sees the token and position embeddings, so depth cannot depend on context. This does not match the "based on its hidden state" framing.
    • The 12x model, with at most 4 loops, is 3.5 PPL worse than the 24x model, even though the 24x model averages only 3.69 loops.
    • For the speed claims, the latency setup is not given.
    • "24x FLOPs" is the nominal maximum; actual compute is about half of that, so comparisons should be made at matched actual compute.
  1. Bae, S., Kim, Y., Bayat, R., Kim, S., Ha, J., Schuster, T., ... & Yun, S. Y. (2026). Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation. Advances in Neural Information Processing Systems, 38, 96572-96617.

  2. Dehghani, M., Gouws, S., Vinyals, O., Uszkoreit, J., & Kaiser, Ł. (2018). Universal transformers. arXiv preprint arXiv:1807.03819.

Questions for authors
  1. How does this differ from MoR, and how do the two compare empirically?
  2. Why does the router use only the initial embedding?
  3. Can you explain the gap between the 12x and 24x results?
  4. How was latency measured, and what do the results look like at matched actual compute?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.