- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates whether the benefits of additional recurrent computation in Looped Language Models can be internalized into their shared parameters. The authors propose a preference write-back approach that uses trajectories generated at deeper recurrent depths to self-distill the shared recurrent block through shallow execution, without additional correctness supervision. Experiments on an OLMo-2 1B model demonstrate improvements in mathematical reasoning across different recurrent depths. The paper further analyzes training depth, gradient behavior, input injection, and generalization to other tasks.
Critical strengths
-
Interesting research motivation and empirical findings. The paper explores an interesting question of whether improvements obtained through additional recurrent computation can be incorporated into model parameters. The reported gains in mathematical reasoning suggest that deeper recurrent execution can provide useful self-supervision.
-
Comprehensive experimental analysis. The experiments systematically investigate teacher depth, training depth, and inference depth. The ablation studies and gradient analysis provide useful insights into why shallow self-distillation can outperform full-depth training. The evaluation across multiple mathematical benchmarks and non-mathematical tasks further strengthens the empirical analysis.
Critical weaknesses
-
Limited methodological novelty. The proposed preference write-back mechanism is essentially a form of self-distillation, where deeper model-generated trajectories supervise a shallower execution of the same model. While the application to Looped Language Models is interesting, the underlying training objective and optimization strategy are relatively straightforward. The methodological distinction from existing self-distillation approaches is not sufficiently substantial.
-
Limited generalization across model architectures and scales. The experiments are conducted primarily on a single OLMo-2 1B backbone, with different recurrent-depth configurations. Although the paper evaluates several mathematical reasoning benchmarks, the results do not establish whether the proposed mechanism generalizes to other language model families or larger parameter scales. Moreover, the evaluations on 17 non-mathematical tasks mainly demonstrate performance preservation rather than meaningful improvements, leaving its broader applicability uncertain.
Questions for authors
-
What is the fundamental methodological distinction between preference write-back and conventional deep-to-shallow self-distillation? Could the authors further clarify what makes the proposed mechanism specifically novel beyond its application to weight-shared recurrent models?
-
Can the authors provide additional evidence that preference write-back generalizes to other backbone architectures or larger language models? Furthermore, why are the improvements less consistent across some mathematical benchmarks, such as SVAMP, despite the substantial gains observed on GSM8K?
8 comments
Table: Comparison of preference write-back with SFT, self-training, and KD using a shared random subset of 1,024 GSM8K training questions. Scores are exact-match accuracy (%) at inference depths R1 and R4. Parenthetical percentages indicate relative changes from the non-recurrent control (B50).
| Model | Training targets | Trainable | Learning depth | GSM8K (1) | GSM8K (4) | GSM1K (1) | GSM1K (4) |
|---|---|---|---|---|---|---|---|
| Non-looped control (B50) | — | — | — | 44.05 | — | 39.83 | — |
| + SSD-style self-training | own samples | full | R1 | 42.23 (-4.1%) | — | 37.43 (-6.0%) | — |
| + Gold-answer SFT | human solutions | full | R1 | 44.73 (+1.5%) | — | 40.17 (+0.8%) | — |
| + Sequence KD | L0-R4 paths | full | R1 | 47.46 (+7.7%) | — | 40.83 (+2.5%) | — |
| + EOPD | L0-R4 teacher | full | R1 | 45.72 (+3.8%) | — | 39.50 (-0.8%) | — |
| Looped control (L0) | — | — | — | 40.86 | 56.18 | 36.93 | 51.12 |
| + Gold-answer SFT | human solutions | core | R1 | 38.89 | 55.04 (+25.0%) | 33.03 | 50.79 (+27.5%) |
| + Gold-answer SFT | human solutions | core | R4 | 38.51 | 56.10 (+27.4%) | 34.77 | 51.20 (+28.5%) |
| + Gold-answer SFT | human solutions | full | R1 | 38.36 | 54.28 (+23.2%) | 35.10 | 46.89 (+17.7%) |
| + Gold-answer SFT | human solutions | full | R4 | 40.03 | 55.50 (+26.0%) | 33.03 | 47.72 (+19.8%) |
| + DistiLLM-2 | L0-R4 teacher | core | R1 | 42.46 | 58.30 (+32.4%) | 36.93 | 50.71 (+27.3%) |
| + GKD (on-policy) | L0-R4 teacher | core | R1 | 41.93 | 56.71 (+28.7%) | 36.76 | 49.05 (+23.1%) |
| + EOPD | L0-R4 teacher | core | R1 | 41.70 | 56.86 (+29.1%) | 36.51 | 50.46 (+26.7%) |
| + EOPD | L0-R4 teacher | full | R1 | 40.71 | 55.80 (+26.7%) | 34.11 | 46.56 (+16.9%) |
| + Preference write-back (ours) | L0-R4 paths | core | R1 | 42.76 | 60.50 (+37.3%) | 39.50 | 51.54 (+29.4%) |
W2: Generalization across architectures and model scales.
We thank the reviewer for suggesting this extension. Preference write-back benefits the tested OLMo-, Llama-, and Qwen-based looped models, with gains appearing at different inference depths. Our controlled 50B-token comparisons of looping recipes under OLMo-2’s public mid-training setup identify a recurrent recipe that strengthens mathematical reasoning. Preference write-back then uses the model’s own deeper executions to further improve the shared recurrent core. The additional experiments test whether this write-back component remains useful under other looping recipes, using five public checkpoints spanning different backbones and recurrent designs, including the larger 3.5B Huginn. For this controlled comparison with limited data, all models share a fixed random subset of 1,024 GSM8K training questions. Each model learns from its own multi-pass generations by updating the shared core through one pass. R1 and Rm denote one-pass inference and the reported deeper inference setting, respectively.
The gains carry over to the deeper inference settings of both Retro-OLMo and Retro-LLaMA. On Retro-OLMo at R16, GSM8K and GSM1K accuracy increase by 6.0% and 5.0% relative, respectively. On Retro-LLaMA at R8, the corresponding improvements are 5.3% and 0.8%. At R1, the gains on these two benchmarks are 15.2% and 21.9% for Retro-OLMo, and 25.7% and 29.6% for Retro-LLaMA. These results extend the benefit of one-pass updates to deeper inference under another looping recipe and on the Llama backbone [1].
After write-back, Qwen-based LoopUS exceeds its original eight-pass accuracy with a single pass. GSM8K accuracy increases from 48.82% at the original R8 setting to 55.65% at the updated R1 setting, while GSM1K increases from 48.22% to 53.53%. These are relative improvements of 14.0% and 11.0%, respectively. Learning from deeper executions thus creates a stronger shallow deployment option in a model with gated recurrent updates, using the same small question pool [2].
Deep-inference accuracy is preserved or improved in all four tested models with fixed input injection. This includes our model, Retro-OLMo, Retro-LLaMA, and Huginn, which extends the observed benefit to the larger 3.5B model, with R32 accuracy improving by 0.9% on GSM8K and 1.2% on GSM1K under the same 1,024-question protocol [3]. Ouro, which does not repeatedly inject the fixed input representation, improves at R1 by 47.8% and 44.0% on the two benchmarks but declines at R4 [4]. Within OLMo-2, the paper’s matched ablation tests the effect of input injection on the learning-depth advantage. One-pass learning outperforms four-pass learning at R4 with injection, while this advantage does not persist without it. The looping recipe therefore matters for how shallow learning benefits deeper inference.
Table. Preference write-back across looped backbones (accuracy, %). 1 and m denote evaluation depths. Parentheses show relative change over the original model at the same depth, and N/A marks a zero baseline.
| Model | GSM8K (1) | GSM8K (m) | GSM1K (1) | GSM1K (m) |
|---|---|---|---|---|
| Own looped model (L0), m = 4 | 40.86 | 56.18 | 36.93 | 51.12 |
| + Preference write-back | 42.76 (+4.6%) | 60.50 (+7.7%) | 39.50 (+7.0%) | 51.54 (+0.8%) |
| Retro-OLMo, m = 16 | 29.49 | 45.87 | 24.65 | 43.07 |
| + Preference write-back | 33.97 (+15.2%) | 48.60 (+6.0%) | 30.04 (+21.9%) | 45.23 (+5.0%) |
| Retro-LLaMA, m = 8 | 25.63 | 46.17 | 24.15 | 43.98 |
| + Preference write-back | 32.22 (+25.7%) | 48.60 (+5.3%) | 31.29 (+29.6%) | 44.32 (+0.8%) |
| Huginn, m = 32 | 0.00 | 32.37 | 0.08 | 28.46 |
| + Preference write-back | 1.59 (N/A) | 32.68 (+0.9%) | 1.58 (+1800.0%) | 28.80 (+1.2%) |
| LoopUS, m = 8 | 0.08 | 48.82 | 0.08 | 48.22 |
| + Preference write-back | 55.65 (+73300.0%) | 48.67 (-0.3%) | 53.53 (+64400.0%) | 48.46 (+0.5%) |
| Ouro, m = 4 | 26.16 | 79.68 | 25.06 | 77.43 |
| + Preference write-back | 38.67 (+47.8%) | 77.18 (-3.1%) | 36.10 (+44.0%) | 74.69 (-3.5%) |
[1] McLeish et al. Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence. 2025.
[2] Park et al. LoopUS: Recasting Pretrained LLMs into Looped Latent Refinement Models. 2026.
[3] Geiping et al. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach. 2025.
[4] Zhu et al. Scaling Latent Reasoning via Looped Language Models. 2025.
We sincerely thank the reviewer for the encouraging assessment and constructive feedback! We address the concerns below.
We thank the reviewer for this question. Our framework develops recurrent reasoning during mid-training, then uses the model’s own deeper executions to improve the shared core. The objective is standard likelihood training on self-generated trajectories. The methodological distinction lies in the roles of teacher, learning and deployment depth. In a common deep-to-shallow distillation setup, the shallow predictor is trained for deployment. In preference write-back, shallow execution provides the learning path, while the updated core is reused across the deployed model’s deeper passes. The depths used to generate supervision, learn the update and deploy the model can therefore be chosen separately. We study which learning path improves recurrent inference and how this choice depends on the recurrent recipe.
First, learning depth is a separate choice from deployment depth, and the update need not act mainly at the pass used for learning.
Second, recurrent execution supplies useful supervision, and the learning-depth advantage depends on the recurrent recipe.
Comparison methods. Gold-answer SFT uses human solutions. SSD-based self-training learns from the model’s own unfiltered sampled responses [1]. Sequence KD trains the conventional student on exactly the same R4 trajectories used by write-back [2]. DistilLM-2 contrasts teacher- and student-generated responses [3], GKD learns from teacher feedback on on-policy student rollouts [4], and EOPD incorporates teacher entropy into on-policy distillation [5].
Evaluation depths. For the looped model, R4 is the main inference setting, where the shared core is repeatedly applied. R1 provides an additional reduced-depth reference using the same weights.
References
[1] Zhang et al. Embarrassingly Simple Self-Distillation Improves Code Generation. 2026.
[2] Kim and Rush. Sequence-Level Knowledge Distillation. EMNLP 2016.
[3] Ko et al. DistilLM-2: A Contrastive Approach Boosts the Distillation of LLMs. ICML 2025.
[4] Agarwal et al. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. ICLR 2024.
[5] Jin et al. Entropy-Aware On-Policy Distillation of Language Models. ICML 2026.