- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how recurrence and sparsity jointly affect the scaling of looped MoE models. It proposes a bounded effective-parameter mapping that captures diminishing returns from looping and how greater sparsity raises and sustains these gains. The fitted law improves held-out loss prediction and guides model design under compute and memory budgets. Downstream experiments show that looping can reduce the parameters needed for comparable reasoning performance, at the cost of higher inference compute.
Critical strengths
-
The paper is clearly written and easy to follow. It helped me quickly understand the architecture of looped MoE models and their trade-offs.
-
I appreciate work like this that provides useful empirical insights and guidance for future research.
Critical weaknesses
-
The paper mainly contributes an empirical study of scaling laws.
-
The predicted optimal configurations receive limited direct validation. The trillion-token experiment trains only one law-selected recurrence setting. Varying inference depth on that checkpoint does not establish that the selected recurrence is optimal compared with separately trained alternatives under the same training budget.
-
The efficiency analysis relies mainly on approximate FLOPs and weight memory. Without actual latency and total memory measurements, including KV caches and runtime state, the practical benefits of looping remain unclear. The INT4 memory estimates also lack corresponding evaluations of quantized models.
-
The following two points are mainly for discussion. Models such as LongCat-Flash dynamically allocate computation by varying the number of active FFN experts. Could looped MoE adopt a similar design during training, giving difficult tokens more recurrent passes and easy tokens fewer? How might this affect the proposed scaling law under a fixed average compute budget?
-
Inspired by cross-layer DSA index reuse in models such as GLM-5.2, could looped MoE benefit from a regular hybrid structure—for example, interleaving recurrent MoE blocks with ordinary MoE layers that have separate weights? I would be interested in the authors’ thoughts on how such designs might change the trade-off between performance, compute, and memory.
Questions for authors
Any plan to train a bigger looped MoE? (3B, 8B or 30B) which can expand your scaling laws for looped MoE effectively.
6 comments
Response to Reviewer 9JrU [Part 2/4]
Response to Reviewer 9JrU [Part 2/4]
Q2. Direct validation of the predicted optimal configurations
As suggested, we directly validate the law-predicted recurrence against separately trained models at matched training compute, using the available observed losses from our scaling sweep. Specifically, we compare the law-predicted losses of LoopMoE () with separately trained models at varying recurrence , each using its compute-matched token budget , for A0.3B-1.3B LoopMoE at FLOPs and A1.0B-5.4B LoopMoE at FLOPs. As shown in Table I, the recurrence with the minimal law-predicted loss aligns with the recurrence with the minimal observed loss in both settings: for A0.3B-1.3B LoopMoE and for A1.0B-5.4B LoopMoE. Thus, the law identifies when looping pays off and when it does not. Moreover, Table II shows that on reasoning (mean of BBH and GSM8K), A0.3B-1.3B LoopMoE (with the recurrence of minimal predicted loss) nearly matches the larger A0.6B-2.9B MoE ( vs. ), similar to the findings in Table 2.
Table I. Predicted vs. observed loss of LoopMoE () at matched training compute. All looped models are trained separately with the same supervision. The minimal predicted and observed losses are shown in bold.
(a) A0.3B-1.3B LoopMoE at FLOPs (minimal predicted loss at )
| │ | R=1 | R=2 | R=3 | R=4 | R=6 | R=8 | |
|---|---|---|---|---|---|---|---|
| Loss (predicted) | │ | 2.2279 | 2.2239 | 2.2273 | 2.2346 | 2.2542 | 2.2761 |
| Loss (observed) | │ | 2.2292 | 2.2248 | 2.2272 | 2.2353 | 2.2625 | 2.2750 |
(b) A1.0B-5.4B LoopMoE at FLOPs (minimal predicted loss at )
| │ | R=1 | R=2 | R=3 | R=4 | R=6 | R=8 | |
|---|---|---|---|---|---|---|---|
| Loss (predicted) | │ | 2.0080 | 2.0150 | 2.0261 | 2.0390 | 2.0668 | 2.0946 |
| Loss (observed) | │ | 2.0056 | 2.0158 | 2.0282 | 2.0382 | 2.0670 | 2.0968 |
Table II. A0.6B-2.9B MoE vs. A0.3B-1.3B LoopMoE () on downstream benchmarks at matched training compute. Both models are trained with the same supervision. Overall is the mean of 14 benchmarks.
| Model | │ | R | │ | BBH³ | GSM8K⁸ | ARC‑C²⁵ | ARC‑E | OBQA | HS | PIQA | SIQA | Wino | BoolQ | DROP³ | MMLU⁵ | NQ⁵ | TQA⁵ | Overall |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| A0.6B‑2.9B MoE | │ | 1 | │ | 23.8 | 23.4 | 43.2 | 66.2 | 34.6 | 61.8 | 73.9 | 47.9 | 59.2 | 63.3 | 35.4 | 30.2 | 9.6 | 27.5 | 42.9 |
| A0.3B‑1.3B LoopMoE | │ | 2 | │ | 24.8 | 21.8 | 39.6 | 61.6 | 36.0 | 57.0 | 72.1 | 47.8 | 58.2 | 59.4 | 32.8 | 27.6 | 7.9 | 22.3 | 40.6 |
Response to Reviewer 9JrU [Part 3/4]
Response to Reviewer 9JrU [Part 3/4]
Q3. Practical efficiency: latency, total memory, and INT4 memory estimates
We thank the reviewer for this suggestion. We measured the practical runtime of the models in Table 2 on an H100 GPU: (1) latency: time per output token (TPOT, ms); and (2) runtime memory (GB): the peak memory measured by torch.cuda.max_memory_allocated, covering weights, KV cache, and runtime buffers.
Table III presents the practical runtime comparisons, where we compare the models in Table 2: non-looped MoE, LoopMoE, and LoopMoE with built-in speculative decoding [a] using the exit as a built-in draft model. Compared with the larger A0.6B-2.9B MoE, A0.3B-1.3B LoopMoE at matches its reasoning accuracy ( vs. ) with the peak memory ( vs. GB), at the cost of TPOT in latency ( vs. ms). However, as LoopMoE gives valid outputs at every recurrent pass, its first pass () can naturally serve as a built-in lossless draft model for speculative decoding [a] (where we set drafted tokens per verification step), which reduces this latency gap to ( vs. ms). As looping reuses the same weights and only adds a modest KV cache, these efficiency gains hold across test-time recurrences: from to , LoopMoE uses only – the peak memory of the larger MoE, while speculative decoding keeps its latency within –. At larger batch sizes or longer contexts, the KV cache takes a larger share of memory, which Section 4.2 notes as outside the scope of our weight-memory analysis.
Overall, these runtime results confirm the practical gains of looped MoE over a larger non-looped MoE: it matches the reasoning performance and keeps latency at about via speculative decoding with its built-in draft model, while using only about half of the runtime memory.
Table III. Runtime benchmarks of A0.6B-2.9B MoE and A0.3B-1.3B LoopMoE across test-time recurrence , measured on 1 H100 GPU with batch size 1 and the same inference stack (CUDA graphs and MoE kernels), with average over 3 runs on the same prompts at the trained context length. Inference FLOPs, TPOT, and peak memory ratios are normalized to the A0.6B-2.9B MoE baseline. TPOT: time per output token. Peak mem: peak GPU memory, including weights, KV cache, and runtime buffers.
| Model | │ | R | │ | Setting | │ | FLOPs ratio ↓ | TPOT ratio ↓ | Peak mem ratio ↓ | TPOT (ms) | Peak mem (GB) | │ | Reasoning (acc) | Overall (acc) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| A0.6B‑2.9B MoE | │ | 1 | │ | Non‑looped | │ | 1.0× | 1.0× | 1.0× | 5.52 | 6.05 | │ | 36.4 | 50.2 |
| A0.3B‑1.3B LoopMoE | │ | 1 | │ | Non‑looped | │ | 0.4× | 0.7× | 0.5× | 3.61 | 2.88 | │ | 14.6 | 36.6 |
| A0.3B‑1.3B LoopMoE | │ | 2 | │ | Looped | │ | 0.8× | 1.1× | 0.5× | 6.24 | 2.94 | │ | 28.4 | 43.0 |
| A0.3B‑1.3B LoopMoE | │ | 3 | │ | Looped | │ | 1.1× | 1.6× | 0.5× | 8.87 | 3.01 | │ | 35.6 | 46.0 |
| A0.3B‑1.3B LoopMoE | │ | 4 | │ | Looped | │ | 1.5× | 2.1× | 0.5× | 11.51 | 3.08 | │ | 36.5 | 46.7 |
| A0.3B‑1.3B LoopMoE | │ | 5 | │ | Looped | │ | 1.8× | 2.5× | 0.5× | 14.05 | 3.15 | │ | 36.5 | 46.9 |
| A0.3B‑1.3B LoopMoE | │ | 2 | │ | Looped + spec. decoding | │ | 0.8× | 0.9× | 0.5× | 5.00 | 3.17 | │ | 28.4 | 43.0 |
| A0.3B‑1.3B LoopMoE | │ | 3 | │ | Looped + spec. decoding | │ | 1.1× | 1.2× | 0.5× | 6.72 | 3.17 | │ | 35.6 | 46.0 |
| A0.3B‑1.3B LoopMoE | │ | 4 | │ | Looped + spec. decoding | │ | 1.5× | 1.2× | 0.5× | 6.68 | 3.30 | │ | 36.5 | 46.7 |
| A0.3B‑1.3B LoopMoE | │ | 5 | │ | Looped + spec. decoding | │ | 1.8× | 1.3× | 0.6× | 6.99 | 3.37 | │ | 36.5 | 46.9 |
To address the INT4 concern, we now compute weight memory in bfloat16 precision, matching the training precision, and have recomputed the memory-constrained optima. We leave quantized-model optimization (which is beyond the scope of this work) to future work.
References
[a] Leviathan et al. Fast Inference from Transformers via Speculative Decoding. ICML 2023.
Response to Reviewer 9JrU [Part 1/4]
Response to Reviewer 9JrU [Part 1/4]
We thank the reviewer for the thoughtful review and the interesting discussion points. We appreciate the recognition of our presentation (clearly written and easy to follow, helping readers quickly understand the architecture of looped MoE models and their trade-offs) and the value of our work (useful empirical insights and guidance for future research). Below, we address the questions and concerns, and we have revised our paper accordingly, with changes highlighted in blue.
Q1. Contribution beyond an empirical study of scaling laws
We thank the reviewer for this comment. Indeed, our work is grounded in empirically observed trends. In fact, neural scaling laws are empirical in nature (Kaplan et al. [a]): they describe how model performance changes with scaling factors through predictive functional forms fitted to observed losses — the foundations that existing scaling laws [a, b, c] and our work build on. However, our contributions go beyond an empirical study of scaling laws, as summarized below:
We’ve added these contributions to Section 1 (Introduction) in our revised paper.
References
[a] Kaplan et al. Scaling Laws for Neural Language Models. arXiv 2020.
[b] Hoffmann et al. Training Compute-Optimal Large Language Models. NeurIPS 2022.
[c] Ludziejewski et al. Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient. ICML 2025.
[d] Prairie et al. Parcae: Scaling Laws for Stable Looped Language Models. arXiv 2026.
[e] Schwethelm et al. How Much is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models. arXiv 2026.
[f] Lee et al. Sparse Layers Are Critical to Scaling Looped Language Models. arXiv 2026.
[g] Wang et al. SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers. arXiv 2026.