acceptodds
Under review as a conference paper at ICLR 2027

Closing the Loop: Practical Training Recipes for Looped Language Models

Abstract

Looped language models increase effective depth by repeatedly applying a shared block of layers, but existing large-scale recipes require multi-stage training over trillions of tokens, and it remains difficult to separate the benefits of recurrence from differences in data and training. In this work, we establish practical recipes for looped language models, with three main results. (1) We develop a compute-efficient from-scratch training pipeline that reduces the training budget from 7.7T tokens in Ouro to 310B tokens while retaining strong reasoning performance. A two-stage recipe—pretraining followed by high-quality midtraining—together with learning-rate warmup and stronger exit-gate regularization enables stable recurrent training without the multi-stage schedule used in prior work. (2) We establish an end-to-end recipe spanning pretraining and reasoning-oriented midtraining. Under controlled comparisons, our 1.4B LoopLM outperforms a parameter-matched dense model trained on exactly the same data and token budget on all 12 evaluated benchmarks, including +14 points on GSM8K, +10 on MATH, and +22 on DROP. At matched inference compute, it approaches a 3.9B dense model on mathematical reasoning and reading comprehension while using only 36% as many parameters. (3) We introduce a minimal recipe for converting pretrained dense models into looped ones: a single learned input-mixing scalar and a smoothed exit loss are sufficient, with no step-specific parameters. Applied to Qwen3-1.7B-Base, the resulting Looped Qwen improves over an identically continued dense baseline on every evaluated benchmark across two independent data regimes, with statistically clear gains on GSM8K, MATH, and MMLU-Pro on the curated mixture. Together, these results make looped language models substantially cheaper to train from scratch and practical to introduce into existing pretrained checkpoints, while isolating the gains due to recurrence itself.

Primary area:
foundation or frontier models, including LLMs
Keywords:
looped language models, recurrent Transformers, latent reasoning, adaptive computation, parameter sharing, language model pretraining, model upcycling, efficient scaling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gGgiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper contributes training recipes for looped language models through training from scratch and converting pretrained dense models.The from-scratch recipe combines 210B tokens of pretraining with 100B tokens of high-quality mid-training, using learning-rate warmup and stronger exit-gate regularization to stabilize training. The resulting 1.4B model outperforms a parameter-matched dense baseline across 12 benchmarks and approaches a larger dense model on mathematics and reading comprehension at comparable inference compute. For pretrained-model conversion, scalar-controlled embedding injection and a smoothed exit loss yield consistent, generally modest improvements over dense continued-training baselines.

Critical strengths

S1. The paper is clearly written and easy to follow.

S2. The from-scratch LoopLM substantially outperforms its parameter-matched dense baseline, particularly on mathematical reasoning and reading comprehension, and achieves competitive performance in these areas against a larger dense model at matched inference compute.

S3. The training stability analyses provide useful empirical insights into exit-gate collapse and the effects of learning-rate warmup, regularization, and loss smoothing.

S4. The appendices provide extensive training details and diagnostics, including model configurations, optimization settings, data mixtures, loss-spike counts, and gradient norms. These details support reproducibility and offer practical guidance for future research on training looped models.

Critical weaknesses

W1. The methodological novelty is limited. The from-scratch approach largely retains Ouro’s architecture and training objective, with modifications primarily to the training schedule, data mixture, and stabilization settings. The dense-to-loop conversion combines relatively simple input-reinjection and loss-smoothing mechanisms. While these refinements have practical value, the paper could more clearly articulate the new insights that distinguish its contribution from existing looped-model training recipes.

W2. In Table 2, the three-iteration LoopLM does not outperform dense-ouro-3.9B on any evaluated benchmark at matched inference compute, despite using approximately 1.4× more training compute. It matches the larger dense model on MATH500 and approaches it on other mathematics and reading-comprehension benchmarks, but trails on knowledge and commonsense tasks. These results support a parameter-efficiency advantage, while the benefits in terms of compute efficiency remain less clear.

W3. The mid-training stage accounts for approximately one-third of the total training tokens and, under the paper’s compute accounting, a similar fraction of training FLOPs. This token allocation is substantially higher than in recipes such as OLMo 2 and SmolLM3. Moreover, this stage uses instruction–response data with response-only supervision, making supervised mid-training a substantial component of the overall recipe. While Section 3.4 motivates this stage and Section 3.5 reports substantial gains, the paper does not explain how the specific 210B/100B allocation was selected or evaluate its sensitivity to alternative allocations. It therefore remains unclear how strongly the reported advantages depend on this particular training recipe.

Questions for authors

Q1. Regarding W2, Appendix I provides theoretical FLOPs estimates, but the practical benefits of LoopLM’s lower parameter count remain unclear. Could the authors report measured inference memory usage, throughput, and latency under comparable hardware and workload settings? These results would help identify deployment scenarios in which LoopLM offers a favorable trade-off despite the performance gap at matched inference compute.

Q2. Regarding W3, could the authors complement the existing evaluations with a cloze-completion task such as LAMBADA? Could they also compare LoopLM’s language-modeling performance with Ouro and other published models on a shared held-out corpus, such as wikitext or the Pile validation set?

Q3. Also regarding W3, Section 3.5 reports substantial improvements from mid-training. However, it remains unclear how sensitive the results are to the allocation of 210B pretraining tokens and 100B mid-training tokens. Could the authors vary this allocation under a fixed total training compute budget for both LoopLM and the dense baselines? If full-scale experiments are too costly, smaller-scale ablations would still provide useful evidence.

Q4. Figure 9(a)(b) shows that scalar input injection improves later iterations’ training losses relative to the first iteration. However, it remains unclear whether naive recurrence fails partly because the exit gate concentrates on the first iteration before later iterations have sufficiently adapted. Could the authors test a simple alternative that initially uses uniform per-iteration loss weights and subsequently enables the learned gate? Comparing this baseline with the proposed recipe under matched training budgets, using absolute held-out losses and downstream results, would help clarify the contribution of the proposed modifications.

Reviewer_f3teOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper provides two training routes for looped language models: a simplified from-scratch Ouro recipe, and conversion of Qwen3-1.7B-Base into a looped model using a shared input-mixing scalar and exit-loss smoothing. It reports final task performance, diagnoses gate collapse, normalisation, and per-loop training behaviour, and provides anonymous supplementary code.

Critical strengths
  1. The technical-report and engineering value is substantial. The supplementary code includes training, conversion, checkpointing, and export paths, together with single-GPU smoke tests. The gate-collapse and per-loop loss diagnostics also offer useful practical training experience.

  2. The final results show parameter efficiency worth studying. With the same data and token budget, looped models outperform parameter-matched dense models on multiple tasks. Design choices such as the shared scalar also have support from local controlled experiments.

Critical weaknesses
  1. The research increment is not sufficiently established. The shared scalar and loss smoothing mainly adapt existing techniques; the appendix provides analysis results rather than a new analysis method. Relative to Ouro, AlwaysThink, and TaH, the evidence does not yet clearly establish what previously unresolved problem is solved or what reusable knowledge is gained beyond a successful configuration. Local experiments span different datasets and settings, without a controlled path connecting prior recipes to the final recipe. Improvements at the same token budget also do not directly establish superiority at the same training compute.

  2. Formal training and main-table evaluation are not connected by a complete reproduction path. The available code is a substantive strength, but main-experiment configurations, real-data preprocessing and sampling, transitions between stages, and the corresponding evaluation code and commands remain incomplete. Head tying and the actual number of unique parameters in the dense controls also need confirmation from formal configurations. The current materials do not yet let readers reliably reconstruct the main results; smoke tests alone do not verify the full experiments.

Overall assessment: The paper has substantial practical value as a technical report, but its new knowledge and progress over prior work remain unclear, and important gaps persist in the full reproduction path. I lean toward weak rejection. New empirical knowledge can itself constitute a contribution, provided that its significance is clear and supported by appropriate evidence.

Questions for authors
  1. What is the most important new question answered by this paper? Please focus on one of low-budget training, minimal conversion, or failure modes; compare with direct prior work under common conditions; and clarify the limitations of existing methods, what changes here, and the scope of the result. New theory or complex modules are not required.

  2. With a fixed reference implementation, data, and budget, could you provide a small number of key intermediate versions from Ouro or identifiable AlwaysThink/TaH variants to the final recipe, distinguishing matched comparisons from diagnostics across different settings?

  3. Could you provide formal configurations for a representative shared Qwen model and its dense control, data manifests and processing scripts, the stage-transition procedure, initial/final checkpoint identifiers, and the evaluation adapter and full commands used for the main table?

  4. Does the dense control use the same head-tying setup and parameter-counting convention? Please specify the conversion seed, initial checkpoint, fixed-depth evaluation, and export settings, and provide representative logs or repetitions commensurate with the core stability claim.

Reviewer_kqvcOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents a token-efficient recipe for training looped language models from scratch and converting existing dense models to looped language models. By reducing Ouro’s pipeline from 5 training stages to only a pretraining and midtraining stage, and adjusting learning rate warmup and a stronger KL penalty for the exit-gate distribution to prevent gate collapse, LoopLM can be trained on only 310B tokens compared to Ouro’s 7.7T, while outperforming a dense variant of Ouro at parameter-matched scales and approaching its performance at inference compute-matched scales. Furthermore, converting Qwen3-1.7B-Base to a looped LM outperforms the same base checkpoint continued on the same data.

Critical strengths

The authors present a significantly cheaper training recipe for looped LMs compared to Ouro, at only 4% of the token budget and only 3% of the training compute while matching dense Ouro at equal inference compute on math and reading comprehension tasks, and trailing it on knowledge tasks. They also show that exit-gate collapse can be prevented or recovered from by simple adjustments to warmup and KL penalty without the 5-stage schedule of Ouro, allowing for more stable training of looped LMs.

Critical weaknesses

The abstract asserts that the authors' LoopLM "retains strong reasoning performance" versus the Ouro model, but Table 3 shows large reductions in performance on MATH (-7) and MATH500 (−19).

It's unclear whether reducing the training token count relative to Ouro is really a contribution, since it appears that the Ouro model is massively over-trained relative to its capacity. The Chinchilla rule of thumb suggests that we use something between 20 x 1.4B = 28B to 20 x 3.9B = 78B training tokens, and Ouro used 7.7T tokens in total, which is 100x more than what you might expect. Being able to reduce the number of training tokens is not surprising.

The authors' LoopLM underperforms the dense 3.9B model in Table 2 for most tasks, and the dense model actually required fewer training FLOPs (7.1 x 10^21) than the looped model (9.9 x 10^21). It isn't clear that the memory savings from the authors' looping are worth it, if the price is that you need to spend more compute and perform worse on most tasks. Additionally, there may not be substantial memory savings at all for 4,096-token contexts, since the cost of storing the KV cache is comparable to the cost of storing the 1.4B parameters.

The benchmarks that the authors chose and the training data are closely related, and the size of the pretraining dataset was quite small (in the sense that there were only 16B unique tokens), which means that generalization to a broader range of tasks might be impaired. For example, the midtraining includes training splits of several benchmarks that were evaluated on (e.g., FLAN/DROP, GSM8K/MATH train), and large performance gaps relative to Ouro are on the more challenging benchmarks (e.g., MMLU-Pro: −24, MATH500: −19). I’d like to see harder, held-out evaluations (maybe BBH, GPQA, code generation) to judge whether the claimed improvements go beyond the benchmarks that were relatively easy or were close to the training distribution.

Questions for authors

Can the authors also report the total memory usage during inference? Is the KV cache shared across layers?

Table 2 evaluates on 12 benchmarks but Table 4 only evaluates on 8 of them. Can the authors provide the remaining 4 for the latter table for completeness?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.