acceptodds
Under review as a conference paper at ICLR 2027

Equal Depth Exposure, Different Models: Order Effects in Recurrent-Depth Training

Abstract

Recurrent-depth language models apply one shared block a variable number of times, and their training schedules are usually described only by how often each depth is used. We ask whether the order of depths changes the trained model when these counts are held exactly fixed. We train 67M-parameter models on permutations of one fixed multiset of 20,500 per-update depths, with identical counts for each depth across runs. Paired runs share initialization, data order, and learning rates; we evaluate each model at depths 1 to 16. Sorting depths in descending rather than shuffled order raises held-out loss at the deepest trained depth in all five seed pairs, by 0.25 nats on average (95% interval 0.20 to 0.31), and by about 0.6 nats three iterations beyond the trained band. The penalty retains its sign on larger and disjoint evaluation windows, a second multiset, a 102M-parameter model, a C4 corpus slice, and training without either auxiliary loss (three seed pairs). Ascending order, which resembles a depth curriculum, yields losses close to those of shuffled order on depths 5 to 13 but, after training on depths 2 to 10, raises loss by 0.29 nats six iterations beyond the trained band. The penalty depends on when the deepest updates arrive: moving the 5,000 deepest updates from the start to the end of the descending schedule leaves 2 to 6% of the penalty, and 300 to 2,050 further updates at depth 13 bring a checkpoint with 2 nats of excess loss to within 0.03 to 0.11 nats of the shuffled model. Depth order is therefore a training variable in its own right: schedules should be reported by when each depth is used, not only how often.

Primary area:
foundation or frontier models, including LLMs
Keywords:
depth scheduling, weight sharing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_p1yuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on the study of effects of temporal order of recurrent-depth assignments on the training of recurrent-depth or looped language models. When training looped LMs, each shared block can be applied a variable number of times. The authors carefully designed experiments (they called it matched-exposure design) that the same 20,500-update depth multiset is permuted to ensure only temporal order differs across different training runs. In particular, they employed training schedules that contain exactly the same number of training steps at each depth but in different orders: shuffled, ascending (shallow to deep), and descending (deep to shallow). The authors found that descending order consistently produces worse models than shuffled order, especially at the deepest trained depth. They also found that what matters most is training at deep depths late in the run, i.e., moving the deepest updates to the end of the descending schedule removes most of the penalty.

Critical strengths
  • The paper carefully designed their experiments to remove all the possible confounding factors in order to ensure fair comparison across training schedules. All the experimental runs have the same weight initialization, token order, and the learning rate. The only difference is the order of depths.

  • The authors also conducted interventional and probing studies to understand why certain results are observed. They found that simply moving the deepest training to the end removes most of the penalty. This finding could benefit future developers/practitioners for training looped transformers.

Critical weaknesses
  • The paper shows that order matters compared to a shuffled baseline, but it did not compare with simpler alternatives like fixed-depth training or other existing curricula, such as starting with a shallow depth and gradually increasing the mean towards a target depth (McLeish, et al., 2025). We still don’t know how big this effect is compared with the typical choices adopted by developed when training looped LMs.

  • The models evaluated are small (about 100 million parameters at most), and success is judged only by prediction loss on text. No evaluations were conducted on downstream tasks such as question answering or reasoning. It is therefore unclear whether the findings would hold to larger models or are important for the task applications of such models.

McLeish, et al., 2025. Teaching pretrained language models to think deeper with retrofitted recurrence. arXiv preprint arXiv:2511.07384.

Questions for authors
  1. With the same compute budget, how do the experimented schedules compare with simply training at one fixed depth, or with a typical shallow-to-deep curriculum, such as the approach proposed in (McLeish, et al., 2025)?

  2. Why were depths 5–13 selected for the primary experiment? Were these chosen before examining schedule effects? Did you observe similar effects for other bands of equal width and at different absolute depths?

  3. If you vary how many deep updates are placed at the end, how does the benefit change? And does the change improve real task performance, or only the loss measure?

  4. The experiments show that the order of depths matters, but not why. Changing the order also changes several other things, e.g., later training overwrites models learned from earlier learning, where each depth falls in the learning rate schedule, each depth would see different batches of training text, etc. The experiments do not isolate these factors from each other. How do you explain the cause of the experimental results?

Reviewer_HGZHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether the temporal order of training depths changes a recurrent-depth language model when each depth is used exactly the same number of times. It compares permutations of a fixed multiset of 20,500 depth assignments, pairing runs by initialization, token order, and per-step learning rate. Descending order increases held-out cross-entropy relative to shuffled order by 0.252 nats at the deepest trained depth and 0.596 nats three iterations beyond the training range, averaged over five seed pairs. Additional experiments vary the depth multiset, model width, corpus, and objective, while placement and continuation probes examine the role of training history. The main conclusion is that depth schedules should be described by their temporal arrangement as well as their exposure counts.

Critical strengths
  1. The design separates depth order from total exposure. Sections 2.1–2.2 hold the depth multiset and total core applications fixed, while matching initialization, token order, and the learning rate at each update within pairs. This directly addresses whether an exposure histogram is sufficient to characterize training. The paper explicitly treats depth's interactions with the data phase, learning rate, and optimizer state as part of the permutation intervention.

  2. The main empirical effect has several substantive checks. The descending–shuffled difference is positive in all five primary seed pairs at depth 13, with a mean of 0.2519 nats and a 95% interval of [0.1960, 0.3078]. The effect persists on expanded validation and disjoint test windows, and the three-seed plain-cross-entropy ablation shows that it does not require either auxiliary loss. The four-seed terminal-placement results in Table 5 further demonstrate that rearranging the same depth counts can substantially change the endpoint loss.

Critical weaknesses
  1. The training guidance derived from the order effect remains unclear. The paired experiments establish that order affects performance even when the usage count of each depth is held fixed across schedules. However, the main findings concern degradation under globally descending rather than shuffled schedules, while moving or adding deep updates mainly restores performance to near the shuffled baseline. Although these results show that the timing of deep updates matters, they do not yet clarify when deliberate depth-schedule design is necessary or which principles should guide the choice of a schedule.

  2. Generality across recurrent architectures and loop counts remains unclear. The experiments use a prelude–single shared recurrent block–coda design. Other models recur over the entire Transformer layer stack, as in LoopCoder-v2 [1], or repeat individual layers locally, as in Loopie [2]. Whether the observed order effect extends to these different recurrence structures remains unexamined. The main training schedules use 5–13 recurrent applications, and the additional 2–10 setting still covers a broad range. In comparison, Loopie [2] and SMELT [3] use two executions of their respective recurrent units. The behavior of the order effect in low-loop-count variable-depth training has not been established.

References [1] Jian Yang et al. LoopCoder-v2: Only Loop Once for Efficient Test-Time Computation Scaling. 2026. arXiv:2606.18023v1. [2] Zitian Gao et al. Loop the Loopies! 2026. arXiv:2607.16051v3. [3] Shaowen Wang et al. SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers. 2026. arXiv:2609.01343v3.

Questions for authors
  1. Based on the analyses, what empirically supported scheduling principles can the authors identify beyond avoiding a globally descending schedule? Under which training conditions does each principle apply, and which results support it?

  2. How does the observed order effect change under full-stack or layer-wise recurrence? Does it remain substantial when variable-depth training is restricted to low loop counts?

Reviewer_BqYfOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines how the order of training depths affects recurrent-depth language models when exposure at each depth is held fixed. The authors permute a fixed set of depth assignments while matching initialization, token order, and the learning rate at each update. They find that concentrating deep updates early can hurt performance at deep readouts, whereas moving them to the end largely removes the penalty. The study shows that a recurrent model's final behavior depends on the history of depth assignments as well as their total exposure.

Critical strengths
  • Holding the full depth multiset fixed separates the effect of ordering from simply using more deep updates. Matching initialization, token order, and the per-step learning-rate schedule makes this a clear test of the overall depth-permutation effect under the chosen training recipe.

  • The authors report unsuccessful attempts to predict the loss dynamics and discuss the limits of their explanatory probes. This transparency makes the evidence easier to assess and identifies useful questions for follow-up work.

Critical weaknesses
  • It is still unclear how these observations should change a practitioner's training recipe. A comparison with fixed-depth training or an established recurrence curriculum under the same compute budget would help put the practical importance of the order effect in context.

  • The claim that most of the penalty is already present before the final updates uses an intermediate descending checkpoint compared with the final shuffled checkpoint. This comparison also includes differences in training progress; could the authors show the gap between the two schedules at matching checkpoints to support the interpretation of when the penalty develops?

  • The ascending-order comparison across depth bands and the continuation probes each rely on a single training seed per setting. Repeating these specific comparisons would strengthen the secondary conclusions, whose evidence is currently less secure than the replicated descending–shuffled result.

Questions for authors

Please refer to the Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.