- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper focuses on the study of effects of temporal order of recurrent-depth assignments on the training of recurrent-depth or looped language models. When training looped LMs, each shared block can be applied a variable number of times. The authors carefully designed experiments (they called it matched-exposure design) that the same 20,500-update depth multiset is permuted to ensure only temporal order differs across different training runs. In particular, they employed training schedules that contain exactly the same number of training steps at each depth but in different orders: shuffled, ascending (shallow to deep), and descending (deep to shallow). The authors found that descending order consistently produces worse models than shuffled order, especially at the deepest trained depth. They also found that what matters most is training at deep depths late in the run, i.e., moving the deepest updates to the end of the descending schedule removes most of the penalty.
Critical strengths
-
The paper carefully designed their experiments to remove all the possible confounding factors in order to ensure fair comparison across training schedules. All the experimental runs have the same weight initialization, token order, and the learning rate. The only difference is the order of depths.
-
The authors also conducted interventional and probing studies to understand why certain results are observed. They found that simply moving the deepest training to the end removes most of the penalty. This finding could benefit future developers/practitioners for training looped transformers.
Critical weaknesses
-
The paper shows that order matters compared to a shuffled baseline, but it did not compare with simpler alternatives like fixed-depth training or other existing curricula, such as starting with a shallow depth and gradually increasing the mean towards a target depth (McLeish, et al., 2025). We still don’t know how big this effect is compared with the typical choices adopted by developed when training looped LMs.
-
The models evaluated are small (about 100 million parameters at most), and success is judged only by prediction loss on text. No evaluations were conducted on downstream tasks such as question answering or reasoning. It is therefore unclear whether the findings would hold to larger models or are important for the task applications of such models.
McLeish, et al., 2025. Teaching pretrained language models to think deeper with retrofitted recurrence. arXiv preprint arXiv:2511.07384.
Questions for authors
-
With the same compute budget, how do the experimented schedules compare with simply training at one fixed depth, or with a typical shallow-to-deep curriculum, such as the approach proposed in (McLeish, et al., 2025)?
-
Why were depths 5–13 selected for the primary experiment? Were these chosen before examining schedule effects? Did you observe similar effects for other bands of equal width and at different absolute depths?
-
If you vary how many deep updates are placed at the end, how does the benefit change? And does the change improve real task performance, or only the loss measure?
-
The experiments show that the order of depths matters, but not why. Changing the order also changes several other things, e.g., later training overwrites models learned from earlier learning, where each depth falls in the learning rate schedule, each depth would see different batches of training text, etc. The experiments do not isolate these factors from each other. How do you explain the cause of the experimental results?