- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper takes a closer look at why looped MoE fails to scale and identify two main obstacles. First, looping inherits and amplifies the curse of depth: hidden-state variance grows with each iteration as residual updates accumulate, which destabilizes deep recurrence and causes representations to drift. Second, looped MoE suffers from expert selection collapse: routers repeatedly select the same experts across loops, so extra iterations add computation without adding computational diversity. Guided by this diagnosis, this paper proposes LOOM, built on a single principle: each loop should contribute new computation while keeping the recurrent state stable. LOOM stabilizes recurrence by scaling residual updates to bound variance growth and re-injecting the input embedding at every loop, and diversifies it through per-loop routers that engage different experts and a Looping Residual that carries earlier outputs forward.
Critical strengths
-
The paper pushes looped LLMs beyond the commonly used two-loop regime and shows that deeper looping can be made to work in practice. The overall method is well motivated and appears technically reasonable.
-
The paper includes a practical optimization of the training procedure through segmented backpropagation.
Critical weaknesses
-
The MoE setup is not well chosen and does not appear to follow current mainstream MoE-LLM designs such as DeepSeek or Qwen.
In particular, the sparsity is relatively low, and the model uses too many shared experts. This makes the experimental setting look less representative of modern sparse MoE LLMs, and raises concerns about how well the conclusions transfer to more realistic large-scale MoE configurations. -
The downstream evaluation is relatively limited.
Although the paper reports results on several commonsense reasoning benchmarks, it does not include widely used evaluations such as MMLU, GSM8K, or BoolQ. This makes the empirical picture less complete and weakens the evidence for the claimed generality of the method. -
The FLOP-matched comparison is not entirely convincing.
In my view, this is not a very natural trade-off: if anything, a larger number of experts should become more beneficial at greater depth, and several prior works suggest similar trends. The paper chooses to keep hidden size and other backbone dimensions fixed while trading loop depth mostly against expert activation, which feels like a somewhat unusual and restrictive way to match compute. -
The paper does not directly analyze the cross-loop update geometry.
While the paper studies activation variance and routing diversity, it does not provide a direct analysis of the cosine similarity between updates across loops or the magnitude of the update written at each loop (Ref: LoopCoder-v2). Without such diagnostics, it is still unclear what later loops are performing, which leaves some doubt about how the proposed looping mechanism is really working.
Questions for authors
NA
5 comments
Response to Reviewer 3ZVC (2/2)
W3: The FLOP-matched comparison trades loops against expert activation.
A3: Thank you for this suggestion. To further examine it, we trained two deeper non-looped families that stack distinct layers and match LOOM in total parameters (~700M), per-token FLOPs, and effective depth. Narrow width keeps 80 experts per layer and shrinks ; fixed width keeps , shrinks the expert pool, and uses the same top-k as LOOM.
- Model configurations:
| Method | Effective depth | 10 | 20 | 30 | 40 | 50 | 60 |
|---|---|---|---|---|---|---|---|
Narrow width | 512 | 368 | 304 | 264 | 238 | 216 | |
| Experts / top-k | 80/26 | 80/25 | 80/25 | 80/24 | 80/24 | 80/24 | |
| Params (total / act.) | 707M/213M | 710M/212M | 717M/217M | 716M/210M | 721M/211M | 712M/211M | |
Fixed width | Experts / top-k | 80/26 | 39/12 | 26/8 | 19/5 | 15/4 | 12/3 |
| Params (total / act.) | 707M/213M | 702M/205M | 712M/213M | 720M/209M | 719M/211M | 718M/210M | |
LOOM | Loops / top-k | 1× / 26 | 2× / 12 | 3× / 8 | 4× / 5 | 5× / 4 | 6× / 3 |
- Results:
| Method | Effective depth | 10 | 20 | 30 | 40 | 50 | 60 |
|---|---|---|---|---|---|---|---|
Narrow width | Val. ppl ↓ | 18.36 | 19.01 | 19.52 | 20.07 | 20.58 | 21.19 |
| Avg. acc ↑ | 38.84 | 38.67 | 38.47 | 38.21 | 37.74 | 37.02 | |
Fixed width | Val. ppl ↓ | 18.36 | 17.84 | 17.45 | 17.38 | 17.46 | 17.58 |
| Avg. acc ↑ | 38.84 | 38.91 | 39.02 | 39.13 | 39.09 | 38.98 | |
LOOM | Val. ppl ↓ | 18.36 | 17.37 | 16.91 | 16.58 | 16.54 | 16.57 |
| Avg. acc ↑ | 38.84 | 39.00 | 39.34 | 39.53 | 39.53 | 39.50 |
When depth comes from stacking distinct layers, it must be paid for in width [1] or in experts per layer [2], so the gains are limited. In contrast, LOOM achieves the lowest perplexity at every depth, showing that recurrent depth is an effective way to spend the same parameter and FLOP budget.
[1] Levine et al. Limits to Depth Efficiencies of Self-Attention. NeurIPS 2020.
[2] Krajewski et al. Scaling Laws for Fine-Grained Mixture of Experts. ICML 2024.
W4: No analysis of cross-loop update geometry.
A4: Thank you for this valuable suggestion. Following LoopCoder-v2 (arXiv:2606.18023, §3.2.1), we analyze the official ~350M, 9-loop LOOM and three ablations on 320 FineWeb documents (327,680 tokens). Let be the token embedding and the hidden state after loop ; the net update of loop (including embedding re-injection) is . We report the following metrics, as in LoopCoder-v2 Table 3:
- Step size : mean per-token ; how far loop moves the hidden state (larger = bigger rewrite).
- Effective rank: exponential of the entropy of the normalized singular values of the RMSNorm'd hidden states; the number of directions the tokens occupy (higher = richer representation).
| Metric | Model | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |
|---|---|---|---|---|---|---|---|---|---|---|
| Step size | LOOM | 24.42 | 14.33 | 13.14 | 6.88 | 4.33 | 6.60 | 7.54 | 4.55 | 6.90 |
Native no tech | 31.04 | 0.20 | 0.20 | 0.19 | 0.19 | 0.19 | 0.19 | 0.19 | 0.20 | |
Only Res Scale | 20.80 | 19.44 | 8.47 | 11.85 | 13.98 | 8.66 | 13.39 | 13.51 | 9.49 | |
Only Embed Inject | 32.44 | 2.66 | 0.38 | 0.24 | 0.22 | 0.21 | 0.20 | 0.20 | 0.21 | |
| Effective rank | LOOM | 249.0 | 337.2 | 359.7 | 367.4 | 368.7 | 358.6 | 368.1 | 370.2 | 359.0 |
Native no tech | 2.9 | 2.9 | 2.9 | 2.9 | 2.9 | 2.9 | 2.9 | 2.9 | 2.9 | |
Only Res Scale | 180.3 | 94.1 | 97.7 | 105.0 | 97.6 | 98.1 | 90.1 | 82.4 | 84.6 | |
Only Embed Inject | 34.1 | 33.2 | 33.0 | 33.0 | 33.0 | 33.0 | 33.0 | 33.0 | 33.0 |
LOOMtakes large steps in the early loops and progressively smaller ones later, while its effective rank stays high throughout. This indicates that later loops build on the work of earlier ones, refining the representation with smaller but still diverse updates.NativeandOnly Embed Injecthave small step sizes and ranks after the first loop, so the later loops barely change the hidden state and contribute little.Only Res Scalekeeps taking large steps, but its rank stays low, which means the loops keep rewriting the same few directions instead of working together.
We sincerely thank the reviewer again for the constructive feedback. We hope these clarifications are helpful, and we would warmly welcome any further comments or suggestions. Your feedback and recognition mean a great deal to us.
Dear Reviewer 3ZVC,
Thank you again for your thoughtful review. We have provided detailed responses and additional experimental evidence addressing your concerns, including Qwen- and DeepSeek-style MoE configurations, expanded downstream evaluations, additional compute-matched baselines, and analyses of cross-loop update geometry.
We would appreciate any further feedback you may have and would be happy to clarify any remaining questions or concerns.
Thank you for your time and consideration.
Response to Reviewer 3ZVC (1/2)
We sincerely thank you for your thoughtful comments, and we are grateful that you found our attempt to extend looping beyond two iterations well motivated, and segmented backpropagation a practical training optimization. Please find our responses below.
W1: The MoE setup does not follow mainstream designs (
DeepSeek,Qwen).A1: Thank you for this suggestion. As requested, we added two additional setups that follow Qwen3 (arXiv:2505.09388; no shared expert) and DeepSeek-V3 (arXiv:2412.19437; one always-on shared expert). Both share one
10-layerMoE recipe and differ only in the number of loops. Hidden size is512and expert width is192throughout. When looping these backbones, allLOOMcomponents are kept exactly as in the paper. Both families have~300Mparameters; the shared expert raises the activation rate from 10.9% to 12.7%. All runs use 6B tokens (1× Chinchilla), global batch 1024, lr 2e-4 with WSD, and segment length K = 3.QwenQwenQwenQwenQwenDeepSeekDeepSeekDeepSeekDeepSeekDeepSeekLOOMtransfers well to both sparse designs: perplexity falls steadily with more loops, and average accuracy improves by 2.76 (Qwen) and 2.73 (DeepSeek) points.W2: Missing MMLU, GSM8K, and BoolQ.
A2: We appreciate this helpful suggestion. We evaluated the ~1.7B
LOOMmodels of Table 4. As these tasks are challenging for models without task-specific training, we also report the NLL (lower is better) following the NLL-based pre-training evaluation protocol in [1, 2]:Deeper looping improves MMLU , BOOLQ and GSM8K, consistent with the gains on the commonsense benchmarks in the paper.
[1] Microsoft AI. MAI-Thinking-1: Building a hill-climbing machine. Technical report, 2026.
[2] Chen et al. Nexus: Same Pretraining Loss, Better Downstream Generalization via Common Minima. arXiv 2026.