acceptodds
Under review as a conference paper at ICLR 2027

Looping Beyond Twice : A Scalable Recipe for Looped Mixture-of-Experts

Abstract

Looped Transformers introduce recurrent depth as a new scaling axis for large language models (LLMs): by repeatedly applying shared Transformer blocks, they increase effective depth without increasing parameter count. However, the benefits of looping remain unclear for large MoE LLMs under FLOPs-matched comparisons. The main reason is that the gains from additional iterations diminish quickly and can even turn into degradation, so the extra FLOPs spent on looping yield little substantial improvement. Consequently, prior work typically settles on two loops. In this paper, we take a closer look at why looped MoE fails to scale and identify two main obstacles. First, looping inherits and amplifies the curse of depth: hidden-state variance grows with each iteration as residual updates accumulate, which destabilizes deep recurrence and causes representations to drift. Second, looped MoE suffers from expert selection collapse: routers repeatedly select the same experts across loops, so extra iterations add computation without adding computational diversity. Guided by this diagnosis, we propose LOOM, built on a single principle: each loop should contribute new computation while keeping the recurrent state stable. LOOM stabilizes recurrence by scaling residual updates to bound variance growth and re-injecting the input embedding at every loop, and diversifies it through per-loop routers that engage different experts and a Looping Residual that carries earlier outputs forward. Experiments across 100M–1.7B models demonstrate stable recurrence up to 12 loops. Under matched FLOPs, the 700M model performs best at around 5 loops, reducing perplexity from 18.36 to 16.54 and improving average zero-shot accuracy from 38.84% to 39.53% over the non-looped baseline. Without FLOP matching, the 1.7B model trained on 60B tokens peaks at 9 loops, reducing perplexity from 9.62 to 7.77 and improving average zero-shot accuracy from 42.4% to 47.7%. The code is submitted.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Looped Transformers, Mixture-of-Experts, Recurrent Depth

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3ZVCOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper takes a closer look at why looped MoE fails to scale and identify two main obstacles. First, looping inherits and amplifies the curse of depth: hidden-state variance grows with each iteration as residual updates accumulate, which destabilizes deep recurrence and causes representations to drift. Second, looped MoE suffers from expert selection collapse: routers repeatedly select the same experts across loops, so extra iterations add computation without adding computational diversity. Guided by this diagnosis, this paper proposes LOOM, built on a single principle: each loop should contribute new computation while keeping the recurrent state stable. LOOM stabilizes recurrence by scaling residual updates to bound variance growth and re-injecting the input embedding at every loop, and diversifies it through per-loop routers that engage different experts and a Looping Residual that carries earlier outputs forward.

Critical strengths
  1. The paper pushes looped LLMs beyond the commonly used two-loop regime and shows that deeper looping can be made to work in practice. The overall method is well motivated and appears technically reasonable.

  2. The paper includes a practical optimization of the training procedure through segmented backpropagation.

Critical weaknesses
  1. The MoE setup is not well chosen and does not appear to follow current mainstream MoE-LLM designs such as DeepSeek or Qwen.
    In particular, the sparsity is relatively low, and the model uses too many shared experts. This makes the experimental setting look less representative of modern sparse MoE LLMs, and raises concerns about how well the conclusions transfer to more realistic large-scale MoE configurations.

  2. The downstream evaluation is relatively limited.
    Although the paper reports results on several commonsense reasoning benchmarks, it does not include widely used evaluations such as MMLU, GSM8K, or BoolQ. This makes the empirical picture less complete and weakens the evidence for the claimed generality of the method.

  3. The FLOP-matched comparison is not entirely convincing.
    In my view, this is not a very natural trade-off: if anything, a larger number of experts should become more beneficial at greater depth, and several prior works suggest similar trends. The paper chooses to keep hidden size and other backbone dimensions fixed while trading loop depth mostly against expert activation, which feels like a somewhat unusual and restrictive way to match compute.

  4. The paper does not directly analyze the cross-loop update geometry.
    While the paper studies activation variance and routing diversity, it does not provide a direct analysis of the cosine similarity between updates across loops or the magnitude of the update written at each loop (Ref: LoopCoder-v2). Without such diagnostics, it is still unclear what later loops are performing, which leaves some doubt about how the proposed looping mechanism is really working.

Questions for authors

NA

5 comments
AuthorsOpenReview ↗

Response to Reviewer 3ZVC (1/2)

We sincerely thank you for your thoughtful comments, and we are grateful that you found our attempt to extend looping beyond two iterations well motivated, and segmented backpropagation a practical training optimization. Please find our responses below.

W1: The MoE setup does not follow mainstream designs (DeepSeek, Qwen).

A1: Thank you for this suggestion. As requested, we added two additional setups that follow Qwen3 (arXiv:2505.09388; no shared expert) and DeepSeek-V3 (arXiv:2412.19437; one always-on shared expert). Both share one 10-layer MoE recipe and differ only in the number of loops. Hidden size is 512 and expert width is 192 throughout. When looping these backbones, all LOOM components are kept exactly as in the paper. Both families have ~300M parameters; the shared expert raises the activation rate from 10.9% to 12.7%. All runs use 6B tokens (1× Chinchilla), global batch 1024, lr 2e-4 with WSD, and segment length K = 3.

ModelLoopsEffective depthExpertsTop-kSharedActivation rateVal. ppl ↓Avg. acc ↑
Qwen110728010.9%26.1136.81
Qwen330728010.9%23.8137.46
Qwen660728010.9%19.3138.85
Qwen990728010.9%17.5639.56
Qwen12120728010.9%17.0539.57
DeepSeek110728112.7%25.2137.12
DeepSeek330728112.7%23.8237.09
DeepSeek660728112.7%19.7539.04
DeepSeek990728112.7%17.9239.85
DeepSeek12120728112.7%17.3339.24

LOOM transfers well to both sparse designs: perplexity falls steadily with more loops, and average accuracy improves by 2.76 (Qwen) and 2.73 (DeepSeek) points.

W2: Missing MMLU, GSM8K, and BoolQ.

A2: We appreciate this helpful suggestion. We evaluated the ~1.7B LOOM models of Table 4. As these tasks are challenging for models without task-specific training, we also report the NLL (lower is better) following the NLL-based pre-training evaluation protocol in [1, 2]:

LoopsMMLU ACC ↑MMLU NLL ↓BoolQ ACC ↑BoolQ NLL ↓GSM8K ACC ↑GSM8K NLL ↓
126.34.1555.34.521.41.92
327.43.8656.84.201.51.84
629.13.4259.63.721.71.72
929.93.3160.63.601.61.66
1229.83.3560.43.651.41.65

Deeper looping improves MMLU , BOOLQ and GSM8K, consistent with the gains on the commonsense benchmarks in the paper.

[1] Microsoft AI. MAI-Thinking-1: Building a hill-climbing machine. Technical report, 2026.

[2] Chen et al. Nexus: Same Pretraining Loss, Better Downstream Generalization via Common Minima. arXiv 2026.

AuthorsOpenReview ↗

Response to Reviewer 3ZVC (2/2)

W3: The FLOP-matched comparison trades loops against expert activation.

A3: Thank you for this suggestion. To further examine it, we trained two deeper non-looped families that stack distinct layers and match LOOM in total parameters (~700M), per-token FLOPs, and effective depth. Narrow width keeps 80 experts per layer and shrinks d=Id=I; fixed width keeps d=I=512d=I=512, shrinks the expert pool, and uses the same top-k as LOOM.

  • Model configurations:
MethodEffective depth102030405060
Narrow widthd=Id=I512368304264238216
Experts / top-k80/2680/2580/2580/2480/2480/24
Params (total / act.)707M/213M710M/212M717M/217M716M/210M721M/211M712M/211M
Fixed widthExperts / top-k80/2639/1226/819/515/412/3
Params (total / act.)707M/213M702M/205M712M/213M720M/209M719M/211M718M/210M
LOOMLoops / top-k1× / 262× / 123× / 84× / 55× / 46× / 3
  • Results:
MethodEffective depth102030405060
Narrow widthVal. ppl ↓18.3619.0119.5220.0720.5821.19
Avg. acc ↑38.8438.6738.4738.2137.7437.02
Fixed widthVal. ppl ↓18.3617.8417.4517.3817.4617.58
Avg. acc ↑38.8438.9139.0239.1339.0938.98
LOOMVal. ppl ↓18.3617.3716.9116.5816.5416.57
Avg. acc ↑38.8439.0039.3439.5339.5339.50

When depth comes from stacking distinct layers, it must be paid for in width [1] or in experts per layer [2], so the gains are limited. In contrast, LOOM achieves the lowest perplexity at every depth, showing that recurrent depth is an effective way to spend the same parameter and FLOP budget.

[1] Levine et al. Limits to Depth Efficiencies of Self-Attention. NeurIPS 2020.

[2] Krajewski et al. Scaling Laws for Fine-Grained Mixture of Experts. ICML 2024.

W4: No analysis of cross-loop update geometry.

A4: Thank you for this valuable suggestion. Following LoopCoder-v2 (arXiv:2606.18023, §3.2.1), we analyze the official ~350M, 9-loop LOOM and three ablations on 320 FineWeb documents (327,680 tokens). Let h(0)\mathbf{h}^{(0)} be the token embedding and h(r)\mathbf{h}^{(r)} the hidden state after loop rr; the net update of loop rr (including embedding re-injection) is u(r)=h(r)−h(r−1)\mathbf{u}^{(r)}=\mathbf{h}^{(r)}-\mathbf{h}^{(r-1)}. We report the following metrics, as in LoopCoder-v2 Table 3:

  • Step size δ(r)\delta^{(r)}: mean per-token ∥u(r)∥2\lVert\mathbf{u}^{(r)}\rVert_2; how far loop rr moves the hidden state (larger = bigger rewrite).
  • Effective rank: exponential of the entropy of the normalized singular values of the RMSNorm'd hidden states; the number of directions the tokens occupy (higher = richer representation).
MetricModel123456789
Step size δ\deltaLOOM24.4214.3313.146.884.336.607.544.556.90
Native no tech31.040.200.200.190.190.190.190.190.20
Only Res Scale20.8019.448.4711.8513.988.6613.3913.519.49
Only Embed Inject32.442.660.380.240.220.210.200.200.21
Effective rankLOOM249.0337.2359.7367.4368.7358.6368.1370.2359.0
Native no tech2.92.92.92.92.92.92.92.92.9
Only Res Scale180.394.197.7105.097.698.190.182.484.6
Only Embed Inject34.133.233.033.033.033.033.033.033.0
  • LOOM takes large steps in the early loops and progressively smaller ones later, while its effective rank stays high throughout. This indicates that later loops build on the work of earlier ones, refining the representation with smaller but still diverse updates.
  • Native and Only Embed Inject have small step sizes and ranks after the first loop, so the later loops barely change the hidden state and contribute little.
  • Only Res Scale keeps taking large steps, but its rank stays low, which means the loops keep rewriting the same few directions instead of working together.

We sincerely thank the reviewer again for the constructive feedback. We hope these clarifications are helpful, and we would warmly welcome any further comments or suggestions. Your feedback and recognition mean a great deal to us.

Dear Reviewer 3ZVC,

Thank you again for your thoughtful review. We have provided detailed responses and additional experimental evidence addressing your concerns, including Qwen- and DeepSeek-style MoE configurations, expanded downstream evaluations, additional compute-matched baselines, and analyses of cross-loop update geometry.

We would appreciate any further feedback you may have and would be happy to clarify any remaining questions or concerns.

Thank you for your time and consideration.

Reviewer_VEsbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a scalable recipe to scale to deeper layers for looped MoE Transformer, it identifies two issues: Re-using the same router across layers make different iterations engage the same subset of experts, so they equip each loop with a new erouter. They also introduce residual scaling for making signal propagation stable.

Critical strengths

The paper is easy to understand and the result supports the main claim that it is more scalable than baseline.

Critical weaknesses

Overall it is promising, but could you explain the key difference between the contribution II (stabilizing recurrence) and the prior work on looped Transformer (Sun et al., Noci et al, Wang et al)? Seems both are in scale of lambda / hsqrt(M) but you replace h with a t.

Questions for authors
  • Why it plateaus for Native and Only Embedding Inject in Figure 3, is it the diminishing return problem mentioned at the beginning of text?

  • Figure 4 it seems the similarity is till high (minimum 0.88)

  • In Algorithm 1, can you explain why stop gradient is used?

  • In Table 3, why residual scale has unrecoverable loss spike?

5 comments
AuthorsOpenReview ↗

Response to Reviewer VEsb

We thank the reviewer for the careful reading and insightful questions, and for finding the recipe promising. Please find our responses below.

W1: Difference between Contribution II and prior residual scaling (Sun et al., Noci et al., Wang et al.)?

A1: Thank you for pointing this out. The key point of Contribution II is the combination of residual scaling γ=λ/(HM)\gamma=\lambda/(H\sqrt{M}) and loop-dependent re-injection gt=λ/(tM)g_t=\lambda/(t\sqrt{M}) in h0t=(1−gt)hMt−1+gtxh^t_0=(1-g_t)h^{t-1}_M+g_t x, designed jointly to keep the activation variance stable across all loops. Constant HH gives every loop the same residual budget; decaying tt balances the input against the growing recurrent state. Swapping either index breaks this:

Residual γ\gammaInjection gtg_tVal. ppl ↓Avg. acc ↑Var. rangeVar. std ↓
λ/(HM)\lambda/(H\sqrt{M})λ/(tM)\lambda/(t\sqrt{M}) (LOOM)14.8041.150.98–1.080.027
λ/(tM)\lambda/(t\sqrt{M})λ/(tM)\lambda/(t\sqrt{M})15.2840.730.90–1.420.163
λ/(HM)\lambda/(H\sqrt{M})λ/(HM)\lambda/(H\sqrt{M})14.9441.060.93–1.250.087

Only LOOM keeps the variance within 0.98–1.08, giving the best results. We have made this more explicit in Contribution II of our revision.

Q1: Why do Native and Embed-inject plateau in Figure 3?

A2: This differs from the diminishing returns (gain per extra loop) in the introduction. The plateau arises because the gradients explode (up to 2.8×10152.8\times10^{15}) and clipping (norm 1.0) makes each step so small that updates are effectively frozen. Active params is a ratio: 100×Na/N100 \times N_a / N (%), where NaN_a is the number of parameters with ∣gi∣≥10−4max⁡j∣gj∣|g_i| \ge 10^{-4} \max_j |g_j| and NN the number of trainable parameters.

RunAct. variance ↓Loss ↓Active params ↑
Native4467.320.98%
Embed inject only11416.863.53%
LOOM1.042.7358.11%

Only 0.98% / 3.53% of parameters stay active in Native / Embed-inject (vs. 58.11% in LOOM), confirming frozen updates.

Q2: Similarity in Figure 4 is still high (min 0.88).

A3: We appreciate this careful observation. A cosine of ~0.88 already reflects a large change in expert selection (data from Figure 4):

CosineLoopsShared top-10Top-5, earlier loopTop-5, later loop
0.8841 vs 70/10e10, e26, e8, e0, e29e12, e15, e4, e17, e3
0.9106 vs 71/10e29, e8, e0, e27, e23e12, e15, e4, e17, e3
0.9521 vs 24/10e10, e26, e8, e0, e29e22, e12, e18, e10, e25
0.9914 vs 56/10e15, e12, e22, e17, e14e15, e24, e17, e12, e20

Forcing a very low cosine can instead hurt performance. We fix each loop to a nearly disjoint set of 6 experts (step 5,000):

VariantCross-loop cos.Val. ppl ↓Avg. acc ↑
LOOM0.95818.6239.8
Shared router0.99419.8339.2
Fixed 6 experts per loop0.11119.5639.0

Thus per-loop routers already diversify experts substantially, and flexible reuse gives the best results.

Q3: Why stop-gradient in Algorithm 1?

A4: Thank you for raising this point. The stop-gradient implements segmented backpropagation: every K=3K=3 loops, LOOM computes the loss, takes an optimizer step, and detaches (hMt,NH,DH)(h^t_M,N_H,D_H). This is a common technique for looped models, also used by HRM (arXiv:2506.21734), TRM (arXiv:2510.04871). It keeps memory constant in HH and gives H/KH/K updates per batch:

LoopsTime (seg./full)Memory (seg./full)Val. ppl ↓ (seg./full)Avg. acc ↑ (seg./full)
33.1/3.1 h12.5/12.5 GiB18.27/18.2439.38/39.40
66.3/11.2 h12.6/14.8 GiB15.35/17.8840.88/38.98
99.9/15.8 h12.6/17.0 GiB14.80/17.7841.15/39.34
1212.6/38.3 h12.6/19.3 GiB14.86/17.8241.42/38.86

Segmented backpropagation thus trains 1.6–3.0× faster with constant memory and consistently improves performance.

Q4: Why does residual scaling alone spike unrecoverably?

A5: Residual scaling stabilizes the activation variance but not training itself: the gradient norm explodes during training, while LOOM keeps it at 2.5–4:

StepVal. ppl ↓ (Res./LOOM)Grad. norm (Res./LOOM)Act. var. (Res./LOOM)
7,00026.51/16.777.1/2.91.04/1.03
end33.89/14.802.0×10142.0\times10^{14}/2.51.50/1.04

As in Q1, the exploded gradient combined with clipping (norm 1.0) leaves the weights unable to recover, causing the unrecoverable spike.

We sincerely thank the reviewer again for the constructive feedback. We hope these clarifications are helpful, and we would warmly welcome any further comments or suggestions. Your feedback and recognition mean a great deal to us.

In A3, can you explain why you choose "1 vs 7", "6 vs 7", ... etc? I am curious whether other pairs with similarity > 0.88 would have these expert selection behavior too

Q: Why were "1 vs 7", "6 vs 7", ... chosen, and do other pairs with similarity > 0.88 behave the same?

A: Thank you for the follow-up. The pairs in A3 are illustrative examples rather than selected by overlap, and the behavior is general. Below we compare loop 1 against all later loops, using the same data as Figure 4:

CosineLoopsShared top-10Top-5, loop 1Top-5, later loop
0.8821 vs 40/10e10, e26, e8, e0, e29e15, e12, e22, e17, e14
0.8841 vs 70/10e10, e26, e8, e0, e29e12, e15, e4, e17, e3
0.9061 vs 81/10e10, e26, e8, e0, e29e15, e24, e16, e17, e3
0.9111 vs 52/10e10, e26, e8, e0, e29e15, e24, e17, e12, e20
0.9521 vs 24/10e10, e26, e8, e0, e29e22, e12, e18, e10, e25
0.9581 vs 35/10e10, e26, e8, e0, e29e23, e29, e8, e27, e11
0.9661 vs 67/10e10, e26, e8, e0, e29e29, e8, e0, e27, e23
0.9681 vs 97/10e10, e26, e8, e0, e29e29, e8, e27, e0, e10

Thank you again; we are happy to address any further questions. Your feedback and recognition mean a great deal to us.

Reviewer_CUxCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work presents a recipe for training looped MoE LLMs past two looped iterations. This recipe consists of a combination of prior proposed techniques. The adopted techniques include residual scaling and input embedding connections. The authors also propose to untie routers, and replace the residual update for the attention mechanism output. Instead of the normal additive update, the authors introduce local and global vectors. The local vector is re-initialized for each loop while the global vector carries across loops with EMA updates. The attention residual is replaced with the sum of those two EMAs The authors also train with “segmented backpropagation” which is essentially a form of truncated BPTT. They inject language modeling supervision every k=3 loops. While early-exit style losses have been explored before, this work actually does an optimization step after each k loops then detaches various states and continues. So the model actually does multiple gradient updates per batch. This combination of techniques stabilizes/improves training at large depths.

The authors follow a pretty standard pre-training setup and validate their method with a combination of validation perplexity and standard downstream benchmarks. The baselines are various controlled LLM comparisons without looping or only including various aspects of their recipe. Pretty extensive ablations are presented along with analysis of activation variance to show the impact of their interventions. The authors present both FLOP-controlled experiments and FLOP-unconstrained experiments. Positive results are reported under both settings. Without the recipe, naively increasing looping is shown to actually hurt performance.

Critical strengths

Looped models are of increasing interest in the community so this is timely work. The recipe is a nice combination of relevant existing techniques (residual scaling, embedding re-injection) with novel components (attention residual formulation, segmented backprop).

The ablations generally show that each component of their recipe contributes to stabilizing/improving training and model performance. The activation variance analysis is a nice diagnostic of what is happening.

Critical weaknesses

The main weakness of the paper is the construction of a pretty non-standard unlooped baseline model. The compute-matched claim rests on that comparison. I think the non-standard baseline results from the attempt to control for both parameters and flops. The resulting non-looped baseline in Table 2 has 10 layers, d = 512, and uses 26 of 80 experts per token. Basically to match flops with a recurrent baseline, it was necessary to use a short model with unusually densely activated experts compared to normal best practice. However, deeper non-looped MoE baselines with fewer experts could also be constructed to match parameters and flops and have a bit more standard shape.

Similarly, FLOPs-controlled dense and MoE models with the same effective depth would be an informative comparison even if parameters are not strictly controlled. Looped models are basically always worse on FLOPs controlled settings when the non-looped model just has more params. Not saying this is necessarily bad, you can always trade off memory, FLOPS, etc. Without comparisons with more standard LLM configurations, it’s quite hard to contextualize the results here.

The specific design of the attention residual isn’t really ablated in depth (although its overall use is). I find the current design to be slightly arbitrary. How important are the separate local and global memories? How sensitive are things to the EMA weight? How were these things decided?

The aspect of segmented backprop actually updating the model multiple times per batch seems to be a bit buried in the algorithm description. I feel that more should be discussed of that in the main paper (if I’m understanding it correctly).

Questions for authors

How does the model compare with a deeper non-looped MoE baselines with fewer experts (or even dense) that is of a more typical shape? Having a more standard recipe for the non-looped baseline is critical for assessing the performance here.

Similarly, how do FLOPs-controlled dense and MoE models with the same effective depth perform? These would have more params, so looping wouldn't be expected to win. But it’s an informative upper bound.

How were various aspects of the attention residual decided? What components are critical? How important is the EMA param?

How do the gains change when the number of optimizer steps is controlled? So accumulating the losses into a single update per batch. In Table 8, shouldn’t H=3 with segmentation K=3 just be equivalent? Or am I missing something?

9 comments
AuthorsOpenReview ↗

Response to Reviewer CUxC (1/3)

We greatly appreciate the reviewer's constructive suggestions and encouraging comments on the timeliness, recipe design, and analyses of this work. Your valuable comments indeed help improve our paper! Please find our responses below.

W1 (sparsity): The Table 2 baseline activates experts unusually densely.

A1: We trained sparser non-looped baselines with 80 experts and the same top-k as LOOM at each depth (activation 15% → 5%), narrowing d=Id=I to keep ~700M total parameters.

MethodEffective depth2030405060
Sparse non-loopedd=Id=I368304264238216
Experts / top-k80/1280/880/580/480/3
Val. ppl ↓19.1219.9920.5322.1822.55
Avg. acc ↑38.6238.3238.1437.6037.48
LOOMLoops / top-k2× / 123× / 84× / 55× / 46× / 3
Val. ppl ↓17.3716.9116.5816.5416.57
Avg. acc ↑39.0039.3439.5339.5339.50

At the same sparsity, deeper non-looped models degrade, whereas LOOM improves with depth. In A2, we further report deeper, more standard-shaped non-looped baselines matched in both parameters and FLOPs.

W1 & Q1: Deeper non-looped MoE baselines with a more standard shape.

A2: We trained two deeper non-looped families that stack distinct layers to reach the same effective depth as LOOM, and match it in total parameters (~700M) and per-token FLOPs. Narrow width keeps 80 experts per layer and shrinks d=Id=I; fixed width keeps d=I=512d=I=512, shrinks the expert pool, and uses the same top-k as LOOM. All runs use the Table 2 setup.

  • Model configurations:
MethodEffective depth102030405060
Narrow widthd=Id=I512368304264238216
Experts / top-k80/2680/2580/2580/2480/2480/24
Params (total / act.)707M/213M710M/212M717M/217M716M/210M721M/211M712M/211M
Fixed widthExperts / top-k80/2639/1226/819/515/412/3
Params (total / act.)707M/213M702M/205M712M/213M720M/209M719M/211M718M/210M
LOOMLoops / top-k1× / 262× / 123× / 84× / 55× / 46× / 3
  • Results:
MethodEffective depth102030405060
Narrow widthVal. ppl ↓18.3619.0119.5220.0720.5821.19
Avg. acc ↑38.8438.6738.4738.2137.7437.02
Fixed widthVal. ppl ↓18.3617.8417.4517.3817.4617.58
Avg. acc ↑38.8438.9139.0239.1339.0938.98
LOOMVal. ppl ↓18.3617.3716.9116.5816.5416.57
Avg. acc ↑38.8439.0039.3439.5339.5339.50

Deeper stacking helps only with fixed width: adding layers improves the fixed-width family, whereas the narrow-width family degrades, since the extra depth must be paid for in width [1, 2] or in experts per layer [3]. In contrast, LOOM is consistently better, achieving the lowest perplexity at every depth.

[1] Levine et al. Limits to Depth Efficiencies of Self-Attention. NeurIPS 2020.

[2] Petty et al. The Impact of Depth on Compositional Generalization in Transformer Language Models. NAACL 2024.

[3] Krajewski et al. Scaling Laws for Fine-Grained Mixture of Experts. ICML 2024.

AuthorsOpenReview ↗

Response to Reviewer CUxC (2/3)

W2 & Q2: FLOP-unconstrained non-looped models as an upper bound.

A3: We keep the total parameters at ~700M (as LOOM) and activate all experts (top-k = number of experts, i.e., a dense FFN), at about 3× the FLOPs of LOOM. Depth is added in the same two ways as in A2. FLOPs are per token, relative to the Table 2 baseline.

MethodEffective depth102030405060
Narrow width (80/80)d=Id=I512368304264238216
FLOPs2.93×3.03×3.10×3.11×3.16×3.13×
Val. ppl ↓17.8818.5319.2119.5420.0320.62
Avg. acc ↑39.3839.0538.4138.7738.3037.58
Fixed widthExperts / top-k80/8039/3926/2619/1915/1512/12
FLOPs2.93×2.93×3.00×3.00×3.04×3.00×
Val. ppl ↓17.8816.7516.3416.1916.2019.27†
Avg. acc ↑39.3839.3539.5639.9239.8838.71†
LOOMLoops / top-k1× / 262× / 123× / 84× / 55× / 46× / 3
FLOPs1.00×1.00×1.07×1.00×1.07×1.07×
Val. ppl ↓18.3617.3716.9116.5816.5416.57
Avg. acc ↑38.8439.0039.3439.5339.5339.50

† Training of the fixed-width depth-60 model was unstable. The fixed-width, fully activated model is the strongest reference (16.19 ppl, 39.92 Avg. at depth 40). With the same parameters and roughly one third of its FLOPs, LOOM at 5 loops attains comparable performance (16.54 ppl, 39.53 Avg.) and outperforms all narrow-width models, approaching this upper bound at a much lower compute cost.

W3: Design of the Looping Residual.

A4: Both memories store the attention output oℓto^t_\ell with the same EMA (Eq. 3, β=0.5\beta=0.5) and differ only in lifetime: the local memory LL is reset at each loop (10 writes), while the global memory HH persists across loops (90 writes at 9 loops). The residual adds both readouts (Eq. 4). We have clarified this in Eq. 3 of our revision.

SettingLOOM (L+H)L onlyH onlyw/o Looping Residual
Val. ppl ↓14.8015.0515.1115.43
Avg. acc ↑41.1541.0340.9240.52

Both memories contribute, and combining them performs best. LOOM is also robust to the decay rate β\beta:

β\beta0.10.30.50.70.91.0
Val. ppl ↓15.2614.9814.8014.9015.2015.46
Avg. acc ↑40.4741.0341.1540.8640.7440.68

Over β∈[0.3,0.7]\beta\in[0.3,0.7], perplexity varies by at most 0.18 and average accuracy by at most 0.29, and every β∈[0.3,0.9]\beta\in[0.3,0.9] outperforms the model without Looping Residual. We therefore use β=0.5\beta=0.5 at all scales without further tuning.

AuthorsOpenReview ↗

Response to Reviewer CUxC (3/3)

Response to Reviewer CUxC (3/3)

W4: Segmented backpropagation performs multiple updates per batch.

A5: The detailed procedure of segmented backpropagation is as follows. Within one training step (input xx, targets yy), the HH loops are split into S=H/KS = H/K segments, and the step runs SS forward–backward–update cycles:

1,…,K⏟s=1∣K+1,…,2K⏟s=2∣⋯∣(S−1)K+1,…,H⏟s=S (last)\underset{s=1}{\underbrace{1,\dots,K}} \quad \Big\vert \quad \underset{s=2}{\underbrace{K+1,\dots,2K}} \quad \Big\vert \quad \cdots \quad \Big\vert \quad \underset{s=S \text{ (last)}}{\underbrace{(S-1)K+1,\dots,H}}

hMts=Fθs(sg(hMts−1),x),ts=min⁡(sK,H)h_M^{t_s} = F_{\theta_s}(\mathrm{sg}(h_M^{t_{s-1}}), x), \qquad t_s = \min(sK, H)

Ls=CE(LMHead(RMS(hMts)),y),θs+1←θs−η∇θsLs\mathcal L_s = \mathrm{CE}(\mathrm{LMHead}(\mathrm{RMS}(h_M^{t_s})), y), \quad \theta_{s+1} \leftarrow \theta_s - \eta \nabla_{\theta_s} \mathcal L_s

Here hMth_M^{t} is the hidden state after the last layer (MM) of loop tt, so hMts−1h_M^{t_{s-1}} is the state entering segment ss (hMt0=xh_M^{t_0} = x); FθsF_{\theta_s} runs the loops of segment ss (Eqs. 1–4) with the current weights θs\theta_s; sg(⋅)\mathrm{sg}(\cdot) stops the gradient; and η\eta is the learning rate. Segments s<Ss<S supervise intermediate outputs (deep supervision, as in HRM); the last segment supervises the final output hMHh_M^H used at inference, and θS+1\theta_{S+1} starts the next step. Hence

updates per step=S(3 at H=9, 4 at H=12, K=3),memory=O(K) vs. O(H).\text{updates per step} = S \quad (3 \text{ at } H=9, \text{ } 4 \text{ at } H=12, \text{ } K=3), \qquad \text{memory} = O(K) \text{ vs. } O(H).

We have made this more explicit in Section 3.3 of our revision. For more details on segmented backpropagation, please refer to our responses to Q3 and Q4.

Q3: Gains when the number of optimizer steps is controlled.

A6: We fix a 12-loop ~350M model (10B tokens, global batch 1024, 10k steps) and vary only KK. Every batch still runs all 12 loops and receives 12/K12/K updates; K=12K=12 is a single loss and a single update per batch. Longer segments use smaller micro-batches to fit in memory.

KKUpdates / batchMicro-batchs / stepVal. ppl ↓Avg. acc ↑
112165.7119.0539.18
26165.1715.2740.06
34164.5414.8641.42
4387.4114.8241.16
6286.9915.3140.52
121413.7517.8238.86

Compared with no segmentation (K=12K=12), a suitable segment length greatly speeds up training and improves performance. K=3K=3, the setting used in the paper, is both the fastest and the most accurate.

Q4: Should H=3H=3 with K=3K=3 be equivalent to full backpropagation?

A7: With H=K=3H=K=3, the two settings are indeed equivalent. The discrepancy in Table 8 was our oversight: due to time constraints, the two settings were run on different clusters. We have rerun all of them on the same cluster (memory measured at micro-batch 1):

LoopsBackpropTime (h)s / stepMemory (GiB)Val. ppl ↓Avg. acc ↑
3Segmented3.11.1312.518.2739.38
3Full3.11.1312.518.2439.40
6Segmented6.32.2612.615.3540.88
6Full11.23.9914.817.8838.98
9Segmented9.93.5712.614.8041.15
9Full15.85.6617.017.7839.34
12Segmented12.64.5412.614.8641.42
12Full38.313.7519.317.8238.86

At 3 loops the two settings coincide, as expected. From 6 loops on, full backpropagation uses more memory and time and stalls near 17.8 ppl, while segmented backpropagation keeps memory at 12.6 GiB and reaches 14.8–15.4 ppl. We have replaced Table 8 and Figure 5 with these results in our revision.

We thank the reviewer again for the constructive feedback and would be glad to address any further questions. Your feedback and recognition mean a great deal to us.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.