acceptodds
Under review as a conference paper at ICLR 2027

Learning from Recurrence: Internalizing Depth-Induced Preferences in Looped Language Models

Abstract

Recurrent language models can improve predictions by repeatedly applying the same parameters, making depth a property of computation rather than model size. We ask whether behavior exposed by additional recurrence can be written back into the shared computation itself, a transfer we call preference write-back. Trajectories generated by deeper execution then self-distill only the shared recurrent block, without an additional correctness signal. On a 1B model retrofitted with recurrent depth, this update substantially improves both one-pass and full-depth execution on mathematical reasoning, across models trained for mean depths from 4 to 32: GSM8K accuracy at a model's own depth rises from 56.2% to 62.7% at mean depth 4, with comparable gains at 8 and 32. Moreover, updating the block through one recurrent pass consistently outperforms training on the same targets through the full recurrent depth, while requiring less compute and memory; the resulting update also remains useful when the block is reused at later passes. A controlled comparison of looping recipes shows that this advantage of shallow training is a property of how the recurrent model was trained, not of weight tying alone. These results show that recurrent depth is not only an inference-time resource: it can generate self-supervision that improves the shared operator in both single-pass and multi-pass execution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
looped language models, recurrent depth, self-distillation, test-time compute, latent reasoning, weight tying

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zBC3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates whether the benefits of additional recurrent computation in Looped Language Models can be internalized into their shared parameters. The authors propose a preference write-back approach that uses trajectories generated at deeper recurrent depths to self-distill the shared recurrent block through shallow execution, without additional correctness supervision. Experiments on an OLMo-2 1B model demonstrate improvements in mathematical reasoning across different recurrent depths. The paper further analyzes training depth, gradient behavior, input injection, and generalization to other tasks.

Critical strengths
  1. Interesting research motivation and empirical findings. The paper explores an interesting question of whether improvements obtained through additional recurrent computation can be incorporated into model parameters. The reported gains in mathematical reasoning suggest that deeper recurrent execution can provide useful self-supervision.

  2. Comprehensive experimental analysis. The experiments systematically investigate teacher depth, training depth, and inference depth. The ablation studies and gradient analysis provide useful insights into why shallow self-distillation can outperform full-depth training. The evaluation across multiple mathematical benchmarks and non-mathematical tasks further strengthens the empirical analysis.

Critical weaknesses
  1. Limited methodological novelty. The proposed preference write-back mechanism is essentially a form of self-distillation, where deeper model-generated trajectories supervise a shallower execution of the same model. While the application to Looped Language Models is interesting, the underlying training objective and optimization strategy are relatively straightforward. The methodological distinction from existing self-distillation approaches is not sufficiently substantial.

  2. Limited generalization across model architectures and scales. The experiments are conducted primarily on a single OLMo-2 1B backbone, with different recurrent-depth configurations. Although the paper evaluates several mathematical reasoning benchmarks, the results do not establish whether the proposed mechanism generalizes to other language model families or larger parameter scales. Moreover, the evaluations on 17 non-mathematical tasks mainly demonstrate performance preservation rather than meaningful improvements, leaving its broader applicability uncertain.

Questions for authors
  1. What is the fundamental methodological distinction between preference write-back and conventional deep-to-shallow self-distillation? Could the authors further clarify what makes the proposed mechanism specifically novel beyond its application to weight-shared recurrent models?

  2. Can the authors provide additional evidence that preference write-back generalizes to other backbone architectures or larger language models? Furthermore, why are the improvements less consistent across some mathematical benchmarks, such as SVAMP, despite the substantial gains observed on GSM8K?

8 comments

We sincerely thank the reviewer for the encouraging assessment and constructive feedback! We address the concerns below.

W1: Methodological distinction from conventional self-distillation.

We thank the reviewer for this question. Our framework develops recurrent reasoning during mid-training, then uses the model’s own deeper executions to improve the shared core. The objective is standard likelihood training on self-generated trajectories. The methodological distinction lies in the roles of teacher, learning and deployment depth. In a common deep-to-shallow distillation setup, the shallow predictor is trained for deployment. In preference write-back, shallow execution provides the learning path, while the updated core is reused across the deployed model’s deeper passes. The depths used to generate supervision, learn the update and deploy the model can therefore be chosen separately. We study which learning path improves recurrent inference and how this choice depends on the recurrent recipe.

First, learning depth is a separate choice from deployment depth, and the update need not act mainly at the pass used for learning.

  • One-pass learning improves four-pass inference more than four-pass learning does. We fix the teacher trajectories, objective, trainable parameters, and optimization settings, and change only the depth through which the core learns. One-pass learning reaches 62.70% GSM8K accuracy at four-pass inference. Four-pass learning reaches 56.86%, a 5.84-point difference. The corresponding difference on GSM1K is 3.32 points. Four forward passes with backpropagation through only the last pass yield 56.63%. Restricting backpropagation to the last pass does not reproduce the benefit of one-pass learning. The paper’s gradient measurements also show that the two learning depths provide different learning signals: with the loss placed after four passes, the gradient reaching the first-pass state has 7.3% of the norm of the one-pass gradient and a cosine similarity of 0.009 with it (Figure 3a).
  • Most of the gain comes from applying the updated core in later passes. Using the updated core only on the first pass changes four-pass accuracy by 0.08 points. Using it only on passes 2 to 4 recovers 6.29 of the 6.52-point gain. The update learned through one pass remains useful when the core is applied at later recurrent positions.

Second, recurrent execution supplies useful supervision, and the learning-depth advantage depends on the recurrent recipe.

  • Additional recurrence supplies supervision beyond one-pass self-training. With one-pass learning and four-pass inference, Table 5 reports 62.70% accuracy with four-pass teacher trajectories, compared with 59.29% with one-pass teacher trajectories, a 3.41-point difference.
  • The advantage of shallow learning changes with the recurrent recipe. The matched model without input injection, trained on exactly the same four-pass trajectories, no longer shows the advantage of shallow learning. Four-pass learning reaches 54.81% on GSM8K, compared with 54.74% for one-pass learning, and is better on GSM1K and SVAMP. We identify a one-pass learning route that strengthens later recurrent computation, and show that its advantage depends on the recipe used to develop that computation during mid-training.
  • Under the matched budget, write-back improves deep inference more than the tested KD alternatives. With 1,024 GSM8K training questions, the same four-pass teacher, and a fixed generation-and-learning budget, write-back improves four-pass GSM8K accuracy by 4.32 points relative to the original looped checkpoint. This compares with 2.12 for DistiLLM-2, the strongest KD alternative tested on GSM8K.

Comparison methods. Gold-answer SFT uses human solutions. SSD-based self-training learns from the model’s own unfiltered sampled responses [1]. Sequence KD trains the conventional student on exactly the same R4 trajectories used by write-back [2]. DistilLM-2 contrasts teacher- and student-generated responses [3], GKD learns from teacher feedback on on-policy student rollouts [4], and EOPD incorporates teacher entropy into on-policy distillation [5].

Evaluation depths. For the looped model, R4 is the main inference setting, where the shared core is repeatedly applied. R1 provides an additional reduced-depth reference using the same weights.

References

[1] Zhang et al. Embarrassingly Simple Self-Distillation Improves Code Generation. 2026.

[2] Kim and Rush. Sequence-Level Knowledge Distillation. EMNLP 2016.

[3] Ko et al. DistilLM-2: A Contrastive Approach Boosts the Distillation of LLMs. ICML 2025.

[4] Agarwal et al. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. ICLR 2024.

[5] Jin et al. Entropy-Aware On-Policy Distillation of Language Models. ICML 2026.

Table: Comparison of preference write-back with SFT, self-training, and KD using a shared random subset of 1,024 GSM8K training questions. Scores are exact-match accuracy (%) at inference depths R1 and R4. Parenthetical percentages indicate relative changes from the non-recurrent control (B50).

ModelTraining targetsTrainableLearning depthGSM8K (1)GSM8K (4)GSM1K (1)GSM1K (4)
Non-looped control (B50)———44.05—39.83—
+ SSD-style self-trainingown samplesfullR142.23 (-4.1%)—37.43 (-6.0%)—
+ Gold-answer SFThuman solutionsfullR144.73 (+1.5%)—40.17 (+0.8%)—
+ Sequence KDL0-R4 pathsfullR147.46 (+7.7%)—40.83 (+2.5%)—
+ EOPDL0-R4 teacherfullR145.72 (+3.8%)—39.50 (-0.8%)—
Looped control (L0)———40.8656.1836.9351.12
+ Gold-answer SFThuman solutionscoreR138.8955.04 (+25.0%)33.0350.79 (+27.5%)
+ Gold-answer SFThuman solutionscoreR438.5156.10 (+27.4%)34.7751.20 (+28.5%)
+ Gold-answer SFThuman solutionsfullR138.3654.28 (+23.2%)35.1046.89 (+17.7%)
+ Gold-answer SFThuman solutionsfullR440.0355.50 (+26.0%)33.0347.72 (+19.8%)
+ DistiLLM-2L0-R4 teachercoreR142.4658.30 (+32.4%)36.9350.71 (+27.3%)
+ GKD (on-policy)L0-R4 teachercoreR141.9356.71 (+28.7%)36.7649.05 (+23.1%)
+ EOPDL0-R4 teachercoreR141.7056.86 (+29.1%)36.5150.46 (+26.7%)
+ EOPDL0-R4 teacherfullR140.7155.80 (+26.7%)34.1146.56 (+16.9%)
+ Preference write-back (ours)L0-R4 pathscoreR142.7660.50 (+37.3%)39.5051.54 (+29.4%)

W2: Generalization across architectures and model scales.

We thank the reviewer for suggesting this extension. Preference write-back benefits the tested OLMo-, Llama-, and Qwen-based looped models, with gains appearing at different inference depths. Our controlled 50B-token comparisons of looping recipes under OLMo-2’s public mid-training setup identify a recurrent recipe that strengthens mathematical reasoning. Preference write-back then uses the model’s own deeper executions to further improve the shared recurrent core. The additional experiments test whether this write-back component remains useful under other looping recipes, using five public checkpoints spanning different backbones and recurrent designs, including the larger 3.5B Huginn. For this controlled comparison with limited data, all models share a fixed random subset of 1,024 GSM8K training questions. Each model learns from its own multi-pass generations by updating the shared core through one pass. R1 and Rm denote one-pass inference and the reported deeper inference setting, respectively.

The gains carry over to the deeper inference settings of both Retro-OLMo and Retro-LLaMA. On Retro-OLMo at R16, GSM8K and GSM1K accuracy increase by 6.0% and 5.0% relative, respectively. On Retro-LLaMA at R8, the corresponding improvements are 5.3% and 0.8%. At R1, the gains on these two benchmarks are 15.2% and 21.9% for Retro-OLMo, and 25.7% and 29.6% for Retro-LLaMA. These results extend the benefit of one-pass updates to deeper inference under another looping recipe and on the Llama backbone [1].

After write-back, Qwen-based LoopUS exceeds its original eight-pass accuracy with a single pass. GSM8K accuracy increases from 48.82% at the original R8 setting to 55.65% at the updated R1 setting, while GSM1K increases from 48.22% to 53.53%. These are relative improvements of 14.0% and 11.0%, respectively. Learning from deeper executions thus creates a stronger shallow deployment option in a model with gated recurrent updates, using the same small question pool [2].

Deep-inference accuracy is preserved or improved in all four tested models with fixed input injection. This includes our model, Retro-OLMo, Retro-LLaMA, and Huginn, which extends the observed benefit to the larger 3.5B model, with R32 accuracy improving by 0.9% on GSM8K and 1.2% on GSM1K under the same 1,024-question protocol [3]. Ouro, which does not repeatedly inject the fixed input representation, improves at R1 by 47.8% and 44.0% on the two benchmarks but declines at R4 [4]. Within OLMo-2, the paper’s matched ablation tests the effect of input injection on the learning-depth advantage. One-pass learning outperforms four-pass learning at R4 with injection, while this advantage does not persist without it. The looping recipe therefore matters for how shallow learning benefits deeper inference.

Table. Preference write-back across looped backbones (accuracy, %). 1 and m denote evaluation depths. Parentheses show relative change over the original model at the same depth, and N/A marks a zero baseline.

ModelGSM8K (1)GSM8K (m)GSM1K (1)GSM1K (m)
Own looped model (L0), m = 440.8656.1836.9351.12
+ Preference write-back42.76 (+4.6%)60.50 (+7.7%)39.50 (+7.0%)51.54 (+0.8%)
Retro-OLMo, m = 1629.4945.8724.6543.07
+ Preference write-back33.97 (+15.2%)48.60 (+6.0%)30.04 (+21.9%)45.23 (+5.0%)
Retro-LLaMA, m = 825.6346.1724.1543.98
+ Preference write-back32.22 (+25.7%)48.60 (+5.3%)31.29 (+29.6%)44.32 (+0.8%)
Huginn, m = 320.0032.370.0828.46
+ Preference write-back1.59 (N/A)32.68 (+0.9%)1.58 (+1800.0%)28.80 (+1.2%)
LoopUS, m = 80.0848.820.0848.22
+ Preference write-back55.65 (+73300.0%)48.67 (-0.3%)53.53 (+64400.0%)48.46 (+0.5%)
Ouro, m = 426.1679.6825.0677.43
+ Preference write-back38.67 (+47.8%)77.18 (-3.1%)36.10 (+44.0%)74.69 (-3.5%)

[1] McLeish et al. Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence. 2025.

[2] Park et al. LoopUS: Recasting Pretrained LLMs into Looped Latent Refinement Models. 2026.

[3] Geiping et al. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach. 2025.

[4] Zhu et al. Scaling Latent Reasoning via Looped Language Models. 2025.

Reviewer_xxHXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigated in how loop tf's deeper behavior can be helpful to the recurrent blocks. It distilled the trajectories produced by deeper looping of the block (than training) into the block by one-pass training. The experiments are done based on continue pretraining of OLMo2-1B base model, and answers several questions (that I understood clearly):

  1. the behavior of deeper looping of the same block can be learned by the block (one-pass), and the gain extends to deeper loops.
  2. the learned blocks affect more in the later recurrent passes.
  3. useful distillation signals arises across teacher depths and trajectory construction choices.
Critical strengths

Interesting work! I think this method is novel, and actually the result is somehow, to my surprise. Before looking into this paper, I thought this could be that the memorization of one pass should be much more than looped training. However, it seems that this mainly affect the later loops, which makes me thinking of more how this could happen.

Critical weaknesses

Look at the questions below.

Questions for authors

I have some questions

  1. Why this method is important/meaningful. and what's a fair/meaningful comparison (for example, for the same token budget, your method is better than looped tf training. whether the phenomenon is because you continue pretrained a normal model?)
  2. Why deeper behavior is meaningful? Why (any evidence?) figure 1 happens.
  3. the conclusion in table 5 doesn't appear robust to me? (if looking at teacher depth 32 versus 8 for the row group of mean depth 32)
7 comments

We sincerely thank the reviewer for the thoughtful and encouraging assessment of our work, as well as the constructive questions! We address each point below.

W1: Explain why the method is meaningful and provide fair comparisons.

We thank the reviewer for the question. Our framework makes the model’s own deeper reasoning a source of supervision for improving subsequent recurrent reasoning. Mid-training first develops useful recurrent computation. Preference write-back then learns from its deeper executions through a shallow update to the shared core, without requiring reference answers for the write-back questions or adding model parameters. Even in the paper’s largest write-back run, the supervised completion tokens amount to less than 0.03% of the 50B mid-training token count.

Our comparisons control both the data used during mid-training and the additional budget allocated to learning. The conventional and looped models share the pretrained initialization, mid-training data, data order, and 50B-token budget. For the additional learning comparisons, all methods use a fixed random subset of 1,024 GSM8K training questions drawn from material already included in mid-training. Gold-answer SFT revisits these questions using human solutions, directly testing the value of further supervised training on familiar mathematical material. On the looped checkpoint, these controls cover both core-only and full-parameter updates, with learning through either one or four passes. The primary learning comparisons use a common compute cap of approximately 1.575 × 10¹⁶ estimated matrix-multiplication FLOPs, covering trajectory generation, teacher scoring, and student training, with deployed looped-model performance evaluated at four passes.

Write-back uses this additional budget more effectively than the tested supervised and self-supervised alternatives.

Gold-answer SFT remains below write-back on both GSM8K and GSM1K across the tested update scopes and learning depths.

On GSM1K the observed gains are small at this budget: write-back changes four-pass accuracy by +0.4 points and the best SFT variant by +0.1.

With the same original four-pass teacher, write-back increases deployed GSM8K accuracy by 4.32 percentage points relative to the original looped checkpoint. This is more than twice the 2.12-point gain of DistilLM-2, the strongest tested KD alternative on GSM8K. We also apply self-training and sequence KD to the non-recurrent control. Sequence KD receives exactly the same deeper trajectories used by write-back, yet remains below the original looped checkpoint’s deployed performance. These comparisons test both the value of recurrent computation and how effectively its deeper behavior can be used for further learning.

On the same teacher-generated trajectories, one-pass learning improves deployed reasoning more than full-depth learning. This comparison holds the teacher-generated trajectories fixed and varies the learning depth. With identical four-pass targets, questions and settings, learning through one pass reaches 62.70% on GSM8K at four-pass inference, and learning through all four passes reaches 56.86% (Table 3). One-pass learning is also cheaper: 0.72 against 1.14 GPU-hours, with 14% less peak memory (App. D.2). The same ordering holds on GSM8K and GSM1K for the models trained at mean depths 2, 8 and 32. The 60.50% result comes from the 1,024-question controlled comparison, whereas 62.70% is the paper's main 50,000-question result.

Q1: Is this because we continued pretraining a normal model?

Pretrained initialization alone does not determine the advantage of shallow learning. Three results bear on it.

  • A working one-pass model is not required. In our model trained at mean depth 32, one pass scores only 4.09% on GSM8K. Learning through that pass still raises 32-pass accuracy from 57.01% to 62.09%.
  • A pretrained start is not enough. Our two mean-depth-4 models start from the same OLMo-2 1B checkpoint and use matched recurrent recipes with and without input injection. Without injection, one-pass learning loses its advantage over four-pass learning (Table 4).
  • The benefit varies across recurrent recipes and inference depths. We applied write-back to five public looped models with the same 1,024 questions. The larger observed full-depth gains on GSM8K are in the models built from a pretrained LM with input injection: our mean-depth-4 model (+4.3 points), Retro-OLMo (+2.7) and Retro-LLaMA (+2.4). The two models trained from scratch are mixed: Huginn gains 0.3 points and Ouro loses 2.5. LoopUS changes little at full depth, while its updated one-pass execution exceeds its original full-depth accuracy on both benchmarks.

The matched comparison shows that the recurrent recipe matters even with the pretrained initialization held fixed, while the public-model results extend the analysis to different recurrent designs.

Comparison methods. Gold-answer SFT uses human solutions. SSD-based self-training learns from the model’s own unfiltered sampled responses [1]. Sequence KD trains the conventional student on exactly the same R4 trajectories used by write-back [2]. DistilLM-2 contrasts teacher- and student-generated responses [3], GKD learns from teacher feedback on on-policy student rollouts [4], and EOPD incorporates teacher entropy into on-policy distillation [5].

Evaluation depths. For the looped model, R4 is the main inference setting, where the shared core is repeatedly applied. R1 provides an additional reduced-depth reference using the same weights.

References

[1] Zhang et al. Embarrassingly Simple Self-Distillation Improves Code Generation. 2026.

[2] Kim and Rush. Sequence-Level Knowledge Distillation. EMNLP 2016.

[3] Ko et al. DistilLM-2: A Contrastive Approach Boosts the Distillation of LLMs. ICML 2025.

[4] Agarwal et al. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. ICLR 2024.

[5] Jin et al. Entropy-Aware On-Policy Distillation of Language Models. ICML 2026.

Table 1: Comparison of preference write-back with SFT, self-training, and KD using a shared random subset of 1,024 GSM8K training questions. Scores are exact-match accuracy (%) at inference depths R1 and R4. Parenthetical percentages indicate relative changes from the non-recurrent control (B50).

ModelTraining targetsTrainableLearning depthGSM8K (1)GSM8K (4)GSM1K (1)GSM1K (4)
Non-looped control (B50)———44.05—39.83—
+ SSD-style self-trainingown samplesfullR142.23 (-4.1%)—37.43 (-6.0%)—
+ Gold-answer SFThuman solutionsfullR144.73 (+1.5%)—40.17 (+0.8%)—
+ Sequence KDL0-R4 pathsfullR147.46 (+7.7%)—40.83 (+2.5%)—
+ EOPDL0-R4 teacherfullR145.72 (+3.8%)—39.50 (-0.8%)—
Looped control (L0)———40.8656.1836.9351.12
+ Gold-answer SFThuman solutionscoreR138.8955.04 (+25.0%)33.0350.79 (+27.5%)
+ Gold-answer SFThuman solutionscoreR438.5156.10 (+27.4%)34.7751.20 (+28.5%)
+ Gold-answer SFThuman solutionsfullR138.3654.28 (+23.2%)35.1046.89 (+17.7%)
+ Gold-answer SFThuman solutionsfullR440.0355.50 (+26.0%)33.0347.72 (+19.8%)
+ DistiLLM-2L0-R4 teachercoreR142.4658.30 (+32.4%)36.9350.71 (+27.3%)
+ GKD (on-policy)L0-R4 teachercoreR141.9356.71 (+28.7%)36.7649.05 (+23.1%)
+ EOPDL0-R4 teachercoreR141.7056.86 (+29.1%)36.5150.46 (+26.7%)
+ EOPDL0-R4 teacherfullR140.7155.80 (+26.7%)34.1146.56 (+16.9%)
+ Preference write-back (ours)L0-R4 pathscoreR142.7660.50 (+37.3%)39.5051.54 (+29.4%)

Q2: Explain why deeper behavior is meaningful.

Good point! Deeper execution improves task performance and provides useful supervision. For the same mean-depth-4 checkpoint, GSM8K accuracy increases from 40.86% at one pass to 56.18% at four passes, a 15.3-point gain. Table 5 reports 62.70% accuracy with four-pass teacher trajectories, compared with 59.29% with one-pass teacher trajectories, a difference of 3.41 percentage points.

Figure 1 illustrates a pattern measured across a large collection of recurrent readouts. Its bars are schematic. As drawn, they show only sharpening: the candidates keep the same order and the leading one gains probability. Appendix B additionally quantifies candidate reordering, which is not shown in the schematic. The empirical analysis, including plotted distributions and ordering changes, is provided in the appendix section “Recurrence primarily reweights the top of the distribution.” We analyze 715,596 generated-token positions from all 7,473 GSM8K training questions, reading intermediate states through the same output stack along fixed four-pass generations. The top-ranked token remains unchanged across all four passes at 95.6% of positions. When the final choice differs from the first-pass choice, 99.4% of those final choices were already among the first-pass top five. Where the top token remains unchanged, its probability generally increases while total top-10 probability mass changes little.

The measured changes include candidate reordering beyond temperature sharpening. At positions whose top-ranked token remains unchanged, an average of 9.8% of pairwise orderings among the initial rank-2–10 candidates are reversed by the fourth pass. Positive scalar temperature rescaling preserves these orderings. Our write-back targets are greedy trajectories. They retain selected tokens and the contexts those choices create, rather than the teacher’s probability magnitudes. These ordering changes show that recurrence does more than sharpen probabilities. Rank-preserving sharpening alone would leave these targets unchanged. Changes in the preferred continuation can therefore become a learning signal for the shared core. The readout measurements and write-back results thus provide complementary evidence for useful changes in predictive preference.

Q3: Why does the refinement pattern occur?

There are two interpretations of what extra passes do in a looped model, as the introduction notes. Each pass may compute an intermediate result that the next one needs. Looped transformers trained from scratch on multi-hop reasoning behave this way [6]. Or every pass may revise the same prediction until the state stops changing, as in equilibrium models [7]. Two choices in our recipe are consistent with the second interpretation. Input injection re-supplies the input at every pass while the recurrence starts from noise (App. C.1), keeping the input representation available throughout recurrence. Depth is sampled during mid-training with one shared output stack, encouraging useful predictions at different depths. An independent study observes rapid fixed-point convergence in Huginn and retrofitted Llama, while Ouro, without input injection, does not reach a fixed point in the examined executions [8]. In our readouts the revisions concentrate among already plausible continuations and are larger where the first-pass readout is uncertain.

The pass-replacement results are consistent with later refinement reducing the influence of the initial update. One possible explanation is that later passes attenuate changes in the incoming state. In that case, using the updated core only at the first pass could have a diminishing effect, whereas reusing it at later passes repeatedly applies the learned change. If recurrence contracts differences between relevant states and the update changes each application only slightly, its repeated use could remain stable at greater depth. Table 2 directly tests where the updated core improves accuracy. Using the updated core at the first pass only leaves four-pass accuracy essentially unchanged (56.25% against 56.18%), while using it at passes 2 to 4 only reaches 62.47%, against 62.70% with the update at every pass. The shallow-learned update therefore remains useful on states reached later in recurrence, with passes 2 to 4 recovering almost all of the full-update gain.

Ouro shows that shallow gains need not extend to full-depth inference. Ouro is trained from scratch without input injection, and in the same independent study it does not reach a fixed point in the examined executions [8]. After write-back, its one-pass GSM8K accuracy rises from 26.16% to 38.67%, but its four-pass accuracy falls from 79.68% to 77.18% (GSM1K: 77.43% to 74.69%). In our model, Retro-OLMo, Retro-LLaMA and Huginn, which use input injection, full-depth accuracy is preserved or improved.

Reviewer_KG85OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether the benefits of additional recurrent computation in looped language models can be internalized into their shared parameters. The authors propose Preference Write-back, which self-distills trajectories generated at deeper recurrent depths by updating the shared recurrent block through shallow training. Experiments on OLMo-2 1B models with different recurrent depths evaluate mathematical reasoning performance and general capability retention. The results show improvements on GSM8K and demonstrate that single-pass training generally outperforms full-depth training.

Critical strengths
  1. Detailed analysis of recurrent depth. The experiments investigate the effects of teacher depth, training depth, and input injection, providing useful insights into recurrent model optimization.
Critical weaknesses
  1. Insufficient comparisons with conventional approaches. The paper lacks comparisons with standard SFT, self-training, and knowledge distillation under matched computational budgets. Consequently, it remains unclear whether the improvements arise from the proposed mechanism or conventional self-training.

  2. Limited and inconsistent empirical improvements. The gains are mainly observed on mathematical reasoning tasks and are not consistent across benchmarks. For example, performance on SVAMP decreases after self-distillation at larger recurrent depths. Moreover, the experiments do not establish a clear computational advantage over standard non-recurrent models.

Questions for authors
  1. Could the authors compare the proposed approach with conventional self-training or distillation on non-recurrent models using the same training data and comparable computational budgets?

  2. What explains the performance degradation on SVAMP at larger recurrent depths? Can the authors provide additional evidence of generalizability and computational efficiency?

7 comments

We sincerely thank the reviewer for the thoughtful assessment of our work and the constructive feedback! We address the concerns below.

W1: Comparisons with conventional approaches.

We thank the reviewer for suggesting these comparisons. Our recurrent framework outperforms the tested non-recurrent baselines, and write-back further improves the looped checkpoint beyond the tested SFT/KD updates. The full framework reaches 60.50% on GSM8K and 51.54% on GSM1K. It develops reasoning through recurrent mid-training, then learns from the model’s own deeper trajectories to improve the shared core. Building on OLMo-2’s public mid-training recipe, we keep the pretrained initialization, training data, data order, and 50B-token budget fixed across the conventional and looped models. For a controlled comparison with limited data, the non-recurrent and looped learning methods use the same fixed random subset of 1,024 questions from the GSM8K training split. The primary learning comparisons use a common compute cap of approximately 1.575 × 10¹⁶ estimated matrix-multiplication FLOPs, covering trajectory generation, teacher scoring, and student training.

The looped checkpoint already outperforms the tested non-recurrent baselines before write-back, including distillation using the same deeper trajectories. Looping repeatedly applies a shared core. R1 and R4 denote one and four recurrent passes at inference. R4 is the looped model’s deployed setting, while R1 provides a reduced-depth reference using the same weights. On the non-recurrent checkpoint, we compare gold-answer SFT, SSD-style self-training, sequence KD, and EOPD with full-parameter updates. Sequence KD receives exactly the same R4-generated trajectories used by write-back and achieves the strongest non-recurrent results, 47.46% on GSM8K and 40.83% on GSM1K. The original looped checkpoint already reaches 56.18% and 51.12% at R4, before write-back. Sequence KD benefits from the deeper trajectories, but remains below the original looped model at its deployed depth under this learning budget.

Starting from the same looped checkpoint, write-back achieves higher R4 accuracy than the tested SFT and KD updates. DistilLM-2, GKD, and EOPD all use the same original R4 teacher. Under the fixed budget, write-back increases deployed R4 accuracy on GSM8K by 4.32 percentage points relative to the original looped checkpoint. This is more than twice the 2.12-point gain of DistilLM-2, the strongest KD alternative on GSM8K. Additional gold-answer SFT controls cover both core-only and full-parameter updates and likewise remain below write-back on both benchmarks. With the recurrent teacher and evaluation depth held fixed, write-back makes more effective use of the model’s deeper executions than the tested KD alternatives. Among the tested updates to the same looped checkpoint, only our write-back improves both GSM8K and GSM1K at R4. It therefore strengthens the recurrent computation developed during mid-training. The paper’s separate learning-depth ablation further shows that, with the same four-pass teacher trajectories, one-pass learning reaches 62.70% at four-pass inference, compared with 56.86% for four-pass learning. The learning path therefore matters even when the deeper supervision is held fixed.

Comparison methods. Gold-answer SFT uses human solutions. SSD-based self-training learns from the model’s own unfiltered sampled responses [1]. Sequence KD trains the conventional student on exactly the same R4 trajectories used by write-back [2]. DistilLM-2 contrasts teacher- and student-generated responses [3], GKD learns from teacher feedback on on-policy student rollouts [4], and EOPD incorporates teacher entropy into on-policy distillation [5].

References

[1] Zhang et al. Embarrassingly Simple Self-Distillation Improves Code Generation. 2026.

[2] Kim and Rush. Sequence-Level Knowledge Distillation. EMNLP 2016.

[3] Ko et al. DistilLM-2: A Contrastive Approach Boosts the Distillation of LLMs. ICML 2025.

[4] Agarwal et al. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. ICLR 2024.

[5] Jin et al. Entropy-Aware On-Policy Distillation of Language Models. ICML 2026.

Table: Comparison of preference write-back with SFT, self-training, and KD using a shared random subset of 1,024 GSM8K training questions. Scores are exact-match accuracy (%) at inference depths R1 and R4. Parenthetical percentages indicate relative changes from the non-recurrent control (B50).

ModelTraining targetsTrainableLearning depthGSM8K (1)GSM8K (4)GSM1K (1)GSM1K (4)
Non-looped control (B50)———44.05—39.83—
+ SSD-style self-trainingown samplesfullR142.23 (-4.1%)—37.43 (-6.0%)—
+ Gold-answer SFThuman solutionsfullR144.73 (+1.5%)—40.17 (+0.8%)—
+ Sequence KDL0-R4 pathsfullR147.46 (+7.7%)—40.83 (+2.5%)—
+ EOPDL0-R4 teacherfullR145.72 (+3.8%)—39.50 (-0.8%)—
Looped control (L0)———40.8656.1836.9351.12
+ Gold-answer SFThuman solutionscoreR138.8955.04 (+25.0%)33.0350.79 (+27.5%)
+ Gold-answer SFThuman solutionscoreR438.5156.10 (+27.4%)34.7751.20 (+28.5%)
+ Gold-answer SFThuman solutionsfullR138.3654.28 (+23.2%)35.1046.89 (+17.7%)
+ Gold-answer SFThuman solutionsfullR440.0355.50 (+26.0%)33.0347.72 (+19.8%)
+ DistiLLM-2L0-R4 teachercoreR142.4658.30 (+32.4%)36.9350.71 (+27.3%)
+ GKD (on-policy)L0-R4 teachercoreR141.9356.71 (+28.7%)36.7649.05 (+23.1%)
+ EOPDL0-R4 teachercoreR141.7056.86 (+29.1%)36.5150.46 (+26.7%)
+ EOPDL0-R4 teacherfullR140.7155.80 (+26.7%)34.1146.56 (+16.9%)
+ Preference write-back (ours)L0-R4 pathscoreR142.7660.50 (+37.3%)39.5051.54 (+29.4%)

W2: Gains across tasks and depths.

We thank the reviewer for the comment. Our framework improves reasoning across mathematical benchmarks and also benefits non-mathematical tasks. Sharing a core across recurrent passes introduces an inductive bias toward reusable transformations, whose usefulness can vary by task. Controlled multi-hop studies show that this reuse can outperform shallow non-recurrent models with the same parameter count [1]. Previous language-model studies also find strong mathematical reasoning gains and task-dependent benefits from additional recurrence [1, 2]. Our framework builds on this bias to develop mathematical reasoning during mid-training and then improve it using the model’s own deeper executions.

The mathematical gains extend across benchmarks and recurrent depths. Across six mathematical benchmarks, the final models achieve 34.9–46.2% mean per-task relative improvements over the non-recurrent control at their deployed mean depths of 4, 8, and 32. Write-back improves 15 of the 18 comparisons relative to the corresponding looped checkpoints, including all six benchmarks at mean depth 4. GSM8K, GSM1K, GSM-Symbolic, and ASDiv improve across all three models.

The framework’s gains extend beyond mathematics, while write-back largely preserves these gains. Across 17 non-mathematical OLMES tasks covering knowledge, commonsense, science question answering, and reading comprehension, the final models achieve a mean per-task relative improvement of 5.6–7.1% over the non-recurrent control, with each model scoring higher on at least three quarters of the tasks. After write-back, the 17-task average changes by less than 0.2 percentage points from the corresponding looped checkpoint at each model’s deployed mean depth.

SVAMP retains gains from the recurrent framework, with a further improvement from write-back at mean depth 4. In the main results, which use uncurated pools, the mean-depth-4 model improves on SVAMP after write-back, while the deeper models show small decreases relative to their looped checkpoints. All three final models retain 14.5–16.3% relative improvements over the non-recurrent control. Write-back learns from the model’s own deeper generations on the augmented GSM8K portion of OpenMathInstruct-2, without SVAMP training examples.

Broader question coverage improves SVAMP transfer at a fixed training-set size. At R4, we compare random and curated pools of 10,000 questions. Curation uses question text alone to match the random pool’s length distribution, favor less common problem types, and retain at most one question from each near-duplicate cluster. SVAMP accuracy rises from 61.20% to 63.90%, a 4.4% relative improvement over random selection, and exceeds the looped checkpoint’s 63.30%. Both pools improve GSM8K, while the broader pool also improves SVAMP over the looped checkpoint, showing that question selection can strengthen transfer beyond the source benchmark.

[1] Saunshi et al. Reasoning with Latent Thoughts: On the Power of Looped Transformers. ICLR 2025.

[2] Geiping et al. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach. NeurIPS 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.