- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work introduces T-LoopFormer, a looped transformer architecture that allows for token-level elastic depth through dynamic routing using a depth prediction module. The authors manage to efficiently train the model with gather-scatter operations and apply recursion-wise KV cache to save computation and memory. Experiments on various tasks show that T-LoopFormer outperforms baseline models in terms of performance and efficiency.
Critical strengths
- The topic of this paper follows the trend of improving transformer efficiency and performance with looped architectures and dynamic compute allocation, which is currently a hot topic in the field.
- The experiments look incredibly nice: the authors show that T-LoopFormer outperforms baseline models with only 1/8 of the parameters under the same compute budget.
Critical weaknesses
- The presentation looks a bit rushed. The figures are stretched (Figure 2, 3, 4(a), 4(b)). Section 4.2 Ablation Study is an important part of the work while there are only few lines of description without any analysis.
- Missing comparison with strong baselines. Mixture-of-Recursions [1] is a famous work that also uses dynamic routing on looped transformers while the authors do not mention in the manuscript. I believe that there are plenty of other recent works on looped transformers that the authors should compare with.
- The authors does not mention details of the early exit baseline. It is known that how early exit is implemented can significantly affect the performance [2,3]. Simply saying "naive early exiting" makes the reader confused.
- Lack of novelty. The idea of dynamic routing on looped transformers is not something new. Simply changing a router does not seem to be a significant contribution (and not an intuitively reasonable choice as stated in appendix D), unless it significantly improves the performance compared to existing works. Unfortunately, the authors do not provide any comparison with named works like MoR or other recent works on looped transformers.
This work follows the recent trending topic of looped transformers but I believe 1) it requires a significant revision before it can be accepted and 2) the result is not strong enough to be accepted as a paper in ICLR.
[1] Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation. Sangmin Bae, Yujin Kim, Reza Bayat, Sungnyun Kim, Jiyoun Ha, Tal Schuster, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji, Aaron Courville, Se-Young Yun.
[2] BERT Loses Patience: Fast and Robust Inference with Early Exit. Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, Furu Wei.
[3] LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding. Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed A Aly, Beidi Chen, Carole-Jean Wu.
Questions for authors
- About TPOT. Theoretically, in T-LoopFormer, if we remove the router s.t. all tokens go through all layers with full depth, the model should have the same computational cost as a standard transformer. However, in Figure 3(b) and Table 2, this seems not be the case. Why is that?
- In Figure 1(b), the authors show that the average depth of the model is 3.69. Based on the pie chart, this number seems to be calculated as
1*36.85% + 2*23.73% + 3*8.14% + 4*13.87% + 5*2.16% + 6*11.26% + 7*3.77% + 8*0.23% = 2.708.