- Rating
- 4(+2)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The authors study whether looping itself provides an advantage in Transformers under fair budget matching. Looped Transformers iterate a shared block of layers to obtain greater effective depth without adding parameters, and prior work has reported that this extra depth improves language modeling, multi-hop reasoning, and math. However, most prior evaluations compare models of the same size, in which the looped model also uses more FLOPs per token, so the reported gains confound architectural benefits with extra compute. Through three ablation studies, the authors arrive at an effective design recipe called SMELT (Sparse MoE Transformer, middle layers Loop Twice), which loops the middle half of the layers twice and compensates for the extra layer executions by narrowing the model. They scale both SMELT and the baseline up to 54B total non-embedding parameters and fit a separate Chinchilla-style scaling law to each architecture. The paper reports that SMELT’s loss decreases faster with compute, yielding a 14.7–18.0% training compute saving on the compute-optimal frontier at FLOPs.
Critical strengths
-
The comparison against non-loop baselines is relatively fair and therefore convincing. I especially appreciate that the paper tries to match not only per-token FLOPs, but also total parameters and KV cache.
-
The study is conducted on MoE models, which makes it more valuable for current large-scale model design.
-
The NLP downstream evaluation is fairly comprehensive. The paper evaluates on a broad set of downstream tasks and also includes analyses on sample length and in-context learning.
Critical weaknesses
-
The downstream evaluation does not include any math- or code-focused benchmarks. This is quite puzzling. Although the validation domains do include Math/STEM, the downstream tasks do not test these capabilities directly. Even basic and widely used benchmarks such as GSM8K or MATH500 are missing. It is therefore unclear whether the claimed advantages transfer to math and code tasks.
-
The explanation for why loop=3 underperforms loop=2 is not convincing. The authors mainly attribute this to the fact that, under matched budgets, higher loop counts require narrower models. However, this does not seem sufficient. In the loop=2 analysis, the second visit already shows larger write magnitudes, and the cosine difference of the writes may also become significant (as discussed in works such as LoopCoder-v2: Only Loop Once for Efficient Test-Time Computation Scaling). It is therefore possible that increasing the number of loops leads to unstable writes and thus performance degradation. More analysis on loop=3 should be provided. Otherwise, it is difficult to rule out the possibility that the chosen loop design is itself suboptimal.
-
The choice of looping the middle 50% of layers appears heuristic. This result may depend on the specific data or architecture used in this paper. As a result, I have concerns about the generality of the proposed recipe and whether the same choice would remain optimal for other models or training settings.
-
The paper does not include any extrapolation experiments on loop count. This further increases my concern about the proposed loop architecture design. Without such experiments, it is hard to know whether the reported recipe is robust.
Questions for authors
NA
3 comments
Rebuttal 2/n
W3: Generality of the 50% Span
We agree that the exact optimal span may vary across different datasets or architectures, and our goal is not to claim that 50% is universally optimal in every setting. Rather, we view 50% as a simple, effective baseline recipe:
- Selected empirically rather than heuristically, with strong performance nearby. We arrived at the 50% span via controlled sweeps at both and (Section 2.3, Figure 2(a), Table 9). Notably, all six nonzero spans (17% to 100%) beat the unlooped Baseline on validation loss, and neighboring spans perform nearly as well (at , 33%, 50%, and 67% differ by less than ), showing that looping works well across a wide middle region rather than only at 50%.
- Transfers reliably across scales as a fixed recipe. Once selected at
200M(), this simple 50% recipe transfers without any retuning across all 16 cells of our scaling grid (from 100M to 1.6B active parameters and up to 54B total parameters), outperforming the Baseline on validation loss in every cell. This is also consistent with prior studies that independently favor middle-layer looping across diverse setups [6–10].
W4: Test-Time Loop Extrapolation
SMELT is trained with a fixed loop count () and is not designed for test-time loop extrapolation. On the 1.6B, model, validation loss is for SMELT at vs. for the Baseline. Running the same checkpoint with 3 and 4 loops at test time, with the residual scaling changed to and accordingly, raises validation loss to and .
Enabling test-time extrapolation requires additional training techniques, such as randomized loop counts with input injection [8] and per-pass losses [11], and even with these, performance can still degrade beyond the trained depth [12]. LoopCoder-v2 [3] likewise keeps training and inference loop counts matched throughout. Our claim concerns this setting, where training and inference use the same loop count and SMELT consistently outperforms the Baseline.
Rebuttal 3/n
Revision. We will (1) add the post-SFT math and code results with the SFT protocol, (2) add the fixed-width control and the per-pass write and cosine measurements for and to Section 2.5 and Appendix B.2, (3) state explicitly that SMELT is one effective loop design rather than the optimal one, and that the measured savings are a lower bound for looping, and (4) add the test-time extrapolation result to the limitations.
[1] Yang, J., Lieret, K., Jimenez, C. E., Wettig, A., Khandpur, K., Zhang, Y., Hui, B., Press, O., Schmidt, L., & Yang, D. (2025). SWE-smith: Scaling data for software engineering agents. Advances in Neural Information Processing Systems, 38. https://arxiv.org/abs/2504.21798
[2] Deng, X., Da, J., Pan, E., He, Y. Y., Ide, C., Garg, K., Lauffer, N., Park, A., Rane, C., Sampath, K., Krishnan, M., Kundurthy, S. R., Hendryx, S. M., Wang, Z., Zhang, C. B. C., Jacobson, N., Liu, B., & Kenstler, B. (2026). SWE-Bench Pro: Can AI agents solve long-horizon software engineering tasks? Proceedings of Machine Learning Research, 306, 23917–23934. https://proceedings.mlr.press/v306/deng26f.html
[3] Yang, J., Guo, S., Zhang, W., Zheng, T., Du, Y., Li, H.-S., Wu, J., Song, Y., Xing, Y., Cai, Q., Huang, Z., Hao, C., Tao, R., Liu, X., Zhao, W. X., Tang, M., Lv, W., Zhou, M., & Dai, B. (2026). LoopCoder-v2: Only loop once for efficient test-time computation scaling. arXiv preprint arXiv:2606.18023.
[4] Gao, Z., Chen, Y., Xiao, Y., Yang, X., Tao, R., Zhou, J., & Dai, B. (2026). Loop the Loopies! arXiv preprint arXiv:2607.16051.
[5] Wu, B., Chen, M., Luo, X., Yan, S., Yu, Q., Xia, F., Zhang, T., Zhan, H., Zhong, Z., Zhou, X., Qiao, S., & Bin, X. (2025). Parallel loop transformer for efficient test-time computation scaling. arXiv preprint arXiv:2510.24824.
[6] Saunshi, N., Dikkala, N., Li, Z., Kumar, S., & Reddi, S. J. (2025). Reasoning with latent thoughts: On the power of looped transformers. International Conference on Learning Representations (ICLR).
[7] Kapl, F., Angelis, E., Maile, K., von Oswald, J., & Bauer, S. (2026). From growing to looping: A unified view of iterative computation in LLMs. arXiv preprint arXiv:2602.16490.
[8] Geiping, J., McLeish, S., Jain, N., Kirchenbauer, J., Singh, S., Bartoldson, B. R., Kailkhura, B., Bhatele, A., & Goldstein, T. (2025). Scaling up test-time compute with latent reasoning: A recurrent depth approach. Advances in Neural Information Processing Systems (NeurIPS), 38.
[9] Prairie, H., Novack, Z., Berg-Kirkpatrick, T., & Fu, D. Y. (2026). Parcae: Scaling laws for stable looped language models. arXiv preprint arXiv:2604.12946.
[10] Chen, W., Li, T., Huang, W., Yin, Y., Shang, L., & Qin, C. (2026). LoopMoE: Unifying iterative computation with mixture-of-experts for language modeling. arXiv preprint arXiv:2606.04438.
[11] Zhu, R.-J., Wang, Z., Hua, K., Zhang, T., Li, Z., Que, H., Wei, B., Wen, Z., Yin, F., Xing, H., Li, L., Shi, J., Ma, K., Li, S., Kergan, T., Smith, A. K., Qu, X., Hui, M., Wu, B., ... Eshraghian, J. K. (2025). Scaling latent reasoning via looped language models. arXiv preprint arXiv:2510.25741.
[12] Yang, X.-W., Han, Z., Zhang, X.-H., Wei, W.-D., Shao, J.-J., Guo, L.-Z., & Li, Y.-F. (2026). Stabilizing recurrent dynamics for test-time scalable latent reasoning in looped language models. Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306.
Rebuttal 1/n
We thank Reviewer 1FWf for finding our budget-matched comparisons fair and convincing, for recognizing the practical value of studying MoE models, and for appreciating the breadth of our downstream evaluations.
Before addressing each point in detail, we would like to gently clarify the core scope of our claim regarding W2–W4. Our primary goal in this paper is to answer a foundational question: does looping itself provide a net advantage when per-token FLOPs, total parameters, and KV cache are all strictly matched? We present SMELT as a simple, effective recipe to test this hypothesis across scales, rather than as the globally optimal loop architecture.
At the same time, we fully agree that validating math/code transfer and probing the stability and generality of the recipe are essential, and we provide new SFT experiments and analyses for W1–W4 below.
W1: Math and Code Benchmarks
The models in the paper are pretrained base models, so we used domain-specific CE Gain on held-out validation sets as a pretraining proxy (Figure 16(a), Appendix C.1), where Code (20.4%) and Math/STEM (16.6%) show larger compute savings than general web data (14.8%).
To test downstream math and code directly, we fine-tuned our largest matched pair (1.6B active, S≈97) with the same internal SFT data (≈ 120B tokens) and recipe, and evaluated both with the same harness. SMELT outperforms the Baseline on all four benchmarks.
W2: Why Does r=3 Underperform r=2?
To separate the third pass from the narrowing that FLOP matching requires, we trained a control: r=3 at SMELT's width and total parameters (1.33× FLOPs and KV cache).
200M, S≈85)