- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper contributes training recipes for looped language models through training from scratch and converting pretrained dense models.The from-scratch recipe combines 210B tokens of pretraining with 100B tokens of high-quality mid-training, using learning-rate warmup and stronger exit-gate regularization to stabilize training. The resulting 1.4B model outperforms a parameter-matched dense baseline across 12 benchmarks and approaches a larger dense model on mathematics and reading comprehension at comparable inference compute. For pretrained-model conversion, scalar-controlled embedding injection and a smoothed exit loss yield consistent, generally modest improvements over dense continued-training baselines.
Critical strengths
S1. The paper is clearly written and easy to follow.
S2. The from-scratch LoopLM substantially outperforms its parameter-matched dense baseline, particularly on mathematical reasoning and reading comprehension, and achieves competitive performance in these areas against a larger dense model at matched inference compute.
S3. The training stability analyses provide useful empirical insights into exit-gate collapse and the effects of learning-rate warmup, regularization, and loss smoothing.
S4. The appendices provide extensive training details and diagnostics, including model configurations, optimization settings, data mixtures, loss-spike counts, and gradient norms. These details support reproducibility and offer practical guidance for future research on training looped models.
Critical weaknesses
W1. The methodological novelty is limited. The from-scratch approach largely retains Ouro’s architecture and training objective, with modifications primarily to the training schedule, data mixture, and stabilization settings. The dense-to-loop conversion combines relatively simple input-reinjection and loss-smoothing mechanisms. While these refinements have practical value, the paper could more clearly articulate the new insights that distinguish its contribution from existing looped-model training recipes.
W2. In Table 2, the three-iteration LoopLM does not outperform dense-ouro-3.9B on any evaluated benchmark at matched inference compute, despite using approximately 1.4× more training compute. It matches the larger dense model on MATH500 and approaches it on other mathematics and reading-comprehension benchmarks, but trails on knowledge and commonsense tasks. These results support a parameter-efficiency advantage, while the benefits in terms of compute efficiency remain less clear.
W3. The mid-training stage accounts for approximately one-third of the total training tokens and, under the paper’s compute accounting, a similar fraction of training FLOPs. This token allocation is substantially higher than in recipes such as OLMo 2 and SmolLM3. Moreover, this stage uses instruction–response data with response-only supervision, making supervised mid-training a substantial component of the overall recipe. While Section 3.4 motivates this stage and Section 3.5 reports substantial gains, the paper does not explain how the specific 210B/100B allocation was selected or evaluate its sensitivity to alternative allocations. It therefore remains unclear how strongly the reported advantages depend on this particular training recipe.
Questions for authors
Q1. Regarding W2, Appendix I provides theoretical FLOPs estimates, but the practical benefits of LoopLM’s lower parameter count remain unclear. Could the authors report measured inference memory usage, throughput, and latency under comparable hardware and workload settings? These results would help identify deployment scenarios in which LoopLM offers a favorable trade-off despite the performance gap at matched inference compute.
Q2. Regarding W3, could the authors complement the existing evaluations with a cloze-completion task such as LAMBADA? Could they also compare LoopLM’s language-modeling performance with Ouro and other published models on a shared held-out corpus, such as wikitext or the Pile validation set?
Q3. Also regarding W3, Section 3.5 reports substantial improvements from mid-training. However, it remains unclear how sensitive the results are to the allocation of 210B pretraining tokens and 100B mid-training tokens. Could the authors vary this allocation under a fixed total training compute budget for both LoopLM and the dense baselines? If full-scale experiments are too costly, smaller-scale ablations would still provide useful evidence.
Q4. Figure 9(a)(b) shows that scalar input injection improves later iterations’ training losses relative to the first iteration. However, it remains unclear whether naive recurrence fails partly because the exit gate concentrates on the first iteration before later iterations have sufficiently adapted. Could the authors test a simple alternative that initially uses uniform per-iteration loss weights and subsequently enables the learned gate? Comparing this baseline with the proposed recipe under matched training budgets, using absolute held-out losses and downstream results, would help clarify the contribution of the proposed modifications.