acceptodds
Under review as a conference paper at ICLR 2027

Scaling Laws for Looped Mixture of Experts

Abstract

Looped transformers and Mixture-of-Experts (MoE) offer complementary routes to efficient scaling: recurrence increases computational depth at fixed parameters, while MoE sparsity expands total capacity at fixed active compute. Yet existing scaling laws model recurrence or sparsity in isolation. In this work, we introduce *Loop Scaling Laws*, the first scaling law to jointly model recurrence and sparsity alongside model size and data. At its core is a bounded, sparsity-conditional recurrence mapping that characterizes the effective-parameter gain from looping and how sparsity raises this gain. The laws predict the held-out loss of looped models more accurately than prior alternatives, and recover the standard dense and MoE scaling laws as special cases. Beyond prediction, the fitted laws provide a principled foundation for designing looped MoE models under compute and memory constraints. Downstream evaluations further demonstrate the complementary benefits of the two axes: sparsity delivers active-parameter efficiency, recurrence yields total-parameter efficiency on reasoning, and joint scaling further advances the performance frontier. As a practical extension, we show these gains hold at trillion-token scale: at matched training compute, a looped MoE with law-derived recurrence matches a larger non-looped MoE on the reasoning benchmarks, while enabling test-time scaling through recurrence.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Looped Mixture of Experts, Mixture of Experts, Looped Transformers, Scaling Laws

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_9JrUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how recurrence and sparsity jointly affect the scaling of looped MoE models. It proposes a bounded effective-parameter mapping that captures diminishing returns from looping and how greater sparsity raises and sustains these gains. The fitted law improves held-out loss prediction and guides model design under compute and memory budgets. Downstream experiments show that looping can reduce the parameters needed for comparable reasoning performance, at the cost of higher inference compute.

Critical strengths
  1. The paper is clearly written and easy to follow. It helped me quickly understand the architecture of looped MoE models and their trade-offs.

  2. I appreciate work like this that provides useful empirical insights and guidance for future research.

Critical weaknesses
  1. The paper mainly contributes an empirical study of scaling laws.

  2. The predicted optimal configurations receive limited direct validation. The trillion-token experiment trains only one law-selected recurrence setting. Varying inference depth on that checkpoint does not establish that the selected recurrence is optimal compared with separately trained alternatives under the same training budget.

  3. The efficiency analysis relies mainly on approximate FLOPs and weight memory. Without actual latency and total memory measurements, including KV caches and runtime state, the practical benefits of looping remain unclear. The INT4 memory estimates also lack corresponding evaluations of quantized models.

  4. The following two points are mainly for discussion. Models such as LongCat-Flash dynamically allocate computation by varying the number of active FFN experts. Could looped MoE adopt a similar design during training, giving difficult tokens more recurrent passes and easy tokens fewer? How might this affect the proposed scaling law under a fixed average compute budget?

  5. Inspired by cross-layer DSA index reuse in models such as GLM-5.2, could looped MoE benefit from a regular hybrid structure—for example, interleaving recurrent MoE blocks with ordinary MoE layers that have separate weights? I would be interested in the authors’ thoughts on how such designs might change the trade-off between performance, compute, and memory.

Questions for authors

Any plan to train a bigger looped MoE? (3B, 8B or 30B) which can expand your scaling laws for looped MoE effectively.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 9JrU [Part 1/4]

Response to Reviewer 9JrU [Part 1/4]

We thank the reviewer for the thoughtful review and the interesting discussion points. We appreciate the recognition of our presentation (clearly written and easy to follow, helping readers quickly understand the architecture of looped MoE models and their trade-offs) and the value of our work (useful empirical insights and guidance for future research). Below, we address the questions and concerns, and we have revised our paper accordingly, with changes highlighted in blue.


Q1. Contribution beyond an empirical study of scaling laws

We thank the reviewer for this comment. Indeed, our work is grounded in empirically observed trends. In fact, neural scaling laws are empirical in nature (Kaplan et al. [a]): they describe how model performance changes with scaling factors through predictive functional forms fitted to observed losses — the foundations that existing scaling laws [a, b, c] and our work build on. However, our contributions go beyond an empirical study of scaling laws, as summarized below:

  1. A new scaling law formulation for looped MoE models. While prior scaling laws model looping recurrence with unbounded gains [d, e] or MoE sparsity at fixed recurrence [c, f, g], our loop scaling laws are the first to jointly model recurrence and sparsity alongside model size and data via a bounded, sparsity-conditional recurrence mapping, which recovers the dense and MoE scaling laws as special cases while providing more accurate fits than prior alternative formulations (Section 3.3).
  2. A systematic recipe for compute- and memory-optimal looped MoE models. We provide the first recipe for jointly deriving recurrence and sparsity under compute and memory budgets (Section 4.2), revealing that tight memory favors recurrence while larger memory favors sparsity, which provides principled guidance for designing looped MoE architectures.
  3. Comprehensive experiments on parameter-efficient scaling of looped MoE. Our scaling sweep experiments over four scaling axes (NN, DD, EE, RR), along with held-out evaluation (Sections 3.2–3.3) and evaluation on 14 downstream benchmarks (Section 4.3), provide empirical insights into the parameter efficiency of looped MoE models: ∼3× active-parameter efficiency overall by sparsity, and ∼2× total-parameter efficiency on reasoning by recurrence. Further trillion-token experiments and runtime benchmarks show the practical gains of looped MoE over a 2× larger non-looped MoE: it matches the reasoning performance and keeps latency at about 1.2× via speculative decoding with its built-in draft model, while using only about half of the runtime memory (Section 4.3).

We’ve added these contributions to Section 1 (Introduction) in our revised paper.\color{blue}{\text{We've added these contributions to Section 1 (Introduction) in our revised paper.}}


References

[a] Kaplan et al. Scaling Laws for Neural Language Models. arXiv 2020.

[b] Hoffmann et al. Training Compute-Optimal Large Language Models. NeurIPS 2022.

[c] Ludziejewski et al. Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient. ICML 2025.

[d] Prairie et al. Parcae: Scaling Laws for Stable Looped Language Models. arXiv 2026.

[e] Schwethelm et al. How Much is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models. arXiv 2026.

[f] Lee et al. Sparse Layers Are Critical to Scaling Looped Language Models. arXiv 2026.

[g] Wang et al. SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers. arXiv 2026.

AuthorsOpenReview ↗

Response to Reviewer 9JrU [Part 2/4]

Response to Reviewer 9JrU [Part 2/4]

Q2. Direct validation of the predicted optimal configurations

As suggested, we directly validate the law-predicted recurrence against separately trained models at matched training compute, using the available observed losses from our scaling sweep. Specifically, we compare the law-predicted losses of LoopMoE (E=8E{=}8) with separately trained models at varying recurrence R∈{1,2,3,4,6,8}R\in\lbrace 1,2,3,4,6,8\rbrace, each using its compute-matched token budget DR=Fˉtrain/[6Nunroll(R)]D_R=\bar F_{\text{train}}/[6N_{\text{unroll}}(R)], for A0.3B-1.3B LoopMoE at 5.5×10205.5{\times}10^{20} FLOPs and A1.0B-5.4B LoopMoE at 2.0×10212.0{\times}10^{21} FLOPs. As shown in Table I, the recurrence with the minimal law-predicted loss aligns with the recurrence with the minimal observed loss in both settings: R=2R{=}2 for A0.3B-1.3B LoopMoE and R=1R{=}1 for A1.0B-5.4B LoopMoE. Thus, the law identifies when looping pays off and when it does not. Moreover, Table II shows that on reasoning (mean of BBH and GSM8K), A0.3B-1.3B LoopMoE (with the recurrence of minimal predicted loss) nearly matches the 2×2\times larger A0.6B-2.9B MoE (23.323.3 vs. 23.623.6), similar to the findings in Table 2.

We’ve added these results to the Appendix D.3 in our revised paper.\color{blue}{\text{We've added these results to the Appendix D.3 in our revised paper.}}

Table I. Predicted vs. observed loss of LoopMoE (E=8E{=}8) at matched training compute. All looped models are trained separately with the same supervision. The minimal predicted and observed losses are shown in bold.

(a) A0.3B-1.3B LoopMoE at 5.5×10205.5{\times}10^{20} FLOPs (minimal predicted loss at R=2R{=}2)

│R=1R=2R=3R=4R=6R=8
Loss (predicted)│2.22792.22392.22732.23462.25422.2761
Loss (observed)│2.22922.22482.22722.23532.26252.2750

(b) A1.0B-5.4B LoopMoE at 2.0×10212.0{\times}10^{21} FLOPs (minimal predicted loss at R=1R{=}1)

│R=1R=2R=3R=4R=6R=8
Loss (predicted)│2.00802.01502.02612.03902.06682.0946
Loss (observed)│2.00562.01582.02822.03822.06702.0968

Table II. A0.6B-2.9B MoE vs. A0.3B-1.3B LoopMoE (E=8E{=}8) on downstream benchmarks at matched training compute. Both models are trained with the same supervision. Overall is the mean of 14 benchmarks.

Model│R│BBH³GSM8K⁸ARC‑C²⁵ARC‑EOBQAHSPIQASIQAWinoBoolQDROP³MMLU⁵NQ⁵TQA⁵Overall
A0.6B‑2.9B MoE│1│23.823.443.266.234.661.873.947.959.263.335.430.29.627.542.9
A0.3B‑1.3B LoopMoE│2│24.821.839.661.636.057.072.147.858.259.432.827.67.922.340.6
AuthorsOpenReview ↗

Response to Reviewer 9JrU [Part 3/4]

Response to Reviewer 9JrU [Part 3/4]

Q3. Practical efficiency: latency, total memory, and INT4 memory estimates

We thank the reviewer for this suggestion. We measured the practical runtime of the models in Table 2 on an H100 GPU: (1) latency: time per output token (TPOT, ms); and (2) runtime memory (GB): the peak memory measured by torch.cuda.max_memory_allocated, covering weights, KV cache, and runtime buffers.

Table III presents the practical runtime comparisons, where we compare the models in Table 2: non-looped MoE, LoopMoE, and LoopMoE with built-in speculative decoding [a] using the R=1R{=}1 exit as a built-in draft model. Compared with the 2×2\times larger A0.6B-2.9B MoE, A0.3B-1.3B LoopMoE at R=4R{=}4 matches its reasoning accuracy (36.536.5 vs. 36.436.4) with 0.5×0.5\times the peak memory (3.083.08 vs. 6.056.05 GB), at the cost of 2.1×2.1\times TPOT in latency (11.5111.51 vs. 5.525.52 ms). However, as LoopMoE gives valid outputs at every recurrent pass, its first pass (R=1R{=}1) can naturally serve as a built-in lossless draft model for speculative decoding [a] (where we set k=6k{=}6 drafted tokens per verification step), which reduces this latency gap to 1.2×1.2\times (6.686.68 vs. 5.525.52 ms). As looping reuses the same weights and only adds a modest KV cache, these efficiency gains hold across test-time recurrences: from R=2R{=}2 to R=5R{=}5, LoopMoE uses only 0.50.5–0.6×0.6\times the peak memory of the 2×2\times larger MoE, while speculative decoding keeps its latency within 0.90.9–1.3×1.3\times. At larger batch sizes or longer contexts, the KV cache takes a larger share of memory, which Section 4.2 notes as outside the scope of our weight-memory analysis.

Overall, these runtime results confirm the practical gains of looped MoE over a 2×2\times larger non-looped MoE: it matches the reasoning performance and keeps latency at about 1.2×1.2\times via speculative decoding with its built-in draft model, while using only about half of the runtime memory. We’ve added this new runtime efficiency analysis to Section 4.3 in our revised paper.\color{blue}{\text{We've added this new runtime efficiency analysis to Section 4.3 in our revised paper.}}

Table III. Runtime benchmarks of A0.6B-2.9B MoE and A0.3B-1.3B LoopMoE across test-time recurrence RR, measured on 1 H100 GPU with batch size 1 and the same inference stack (CUDA graphs and MoE kernels), with average over 3 runs on the same prompts at the trained context length. Inference FLOPs, TPOT, and peak memory ratios are normalized to the A0.6B-2.9B MoE baseline. TPOT: time per output token. Peak mem: peak GPU memory, including weights, KV cache, and runtime buffers.

Model│R│Setting│FLOPs ratio ↓TPOT ratio ↓Peak mem ratio ↓TPOT (ms)Peak mem (GB)│Reasoning (acc)Overall (acc)
A0.6B‑2.9B MoE│1│Non‑looped│1.0×1.0×1.0×5.526.05│36.450.2
A0.3B‑1.3B LoopMoE│1│Non‑looped│0.4×0.7×0.5×3.612.88│14.636.6
A0.3B‑1.3B LoopMoE│2│Looped│0.8×1.1×0.5×6.242.94│28.443.0
A0.3B‑1.3B LoopMoE│3│Looped│1.1×1.6×0.5×8.873.01│35.646.0
A0.3B‑1.3B LoopMoE│4│Looped│1.5×2.1×0.5×11.513.08│36.546.7
A0.3B‑1.3B LoopMoE│5│Looped│1.8×2.5×0.5×14.053.15│36.546.9
A0.3B‑1.3B LoopMoE│2│Looped + spec. decoding│0.8×0.9×0.5×5.003.17│28.443.0
A0.3B‑1.3B LoopMoE│3│Looped + spec. decoding│1.1×1.2×0.5×6.723.17│35.646.0
A0.3B‑1.3B LoopMoE│4│Looped + spec. decoding│1.5×1.2×0.5×6.683.30│36.546.7
A0.3B‑1.3B LoopMoE│5│Looped + spec. decoding│1.8×1.3×0.6×6.993.37│36.546.9

To address the INT4 concern, we now compute weight memory in bfloat16 precision, matching the training precision, and have recomputed the memory-constrained optima. We leave quantized-model optimization (which is beyond the scope of this work) to future work. We have updated Figure 6 (Section 4.2) and Table C.5(b) (Appendix C.4) to use bf16 precision accordingly in our revised paper.\color{blue}{\text{We have updated Figure 6 (Section 4.2) and Table C.5(b) (Appendix C.4) to use bf16 precision accordingly in our revised paper.}}


References

[a] Leviathan et al. Fast Inference from Transformers via Speculative Decoding. ICML 2023.

Reviewer_kyQeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Loop Scaling Laws, which jointly model model size, training data, recurrence, and MoE sparsity. The key idea is a bounded recurrence mapping where effective parameter gains diminish with additional loops, while greater sparsity helps sustain these gains. The proposed law improves held-out loss predictions over existing alternatives. The authors also explore compute- and memory-optimal configurations and demonstrate parameter-efficiency gains through downstream evaluations and training.

Critical strengths

Jointly modeling recurrence and MoE sparsity addresses a meaningful gap in existing scaling laws. The proposed formulation is intuitive and yields noticeably better held-out predictions than the baselines considered.

The paper includes a broad sweep, downstream evaluations, and a trillion-token experiment. The latter is particularly valuable in showing that combining recurrence and sparsity can improve parameter efficiency at a practical scale.

Critical weaknesses

One main concern i see is that the experiments mainly use one looping strategy and MoE configuration. It is unclear whether the proposed law generalizes to different looping mechanisms, routing strategies, or model architectures.

The compute- and memory-optimal choices in Figures 5-6 come primarily from the fitted law. Direct comparisons between independently trained configurations under matched budgets would make these claims more convincing.

The bounded recurrence mapping is reasonable, but its exponential form is largely an empirical choice. Evaluations only extend to R=16, leaving uncertainty about behavior at greater recurrence depths. This needs further validation.

The expert-path diversity analysis is interesting, but does not establish that increased routing diversity causes the observed gains. More controlled experiments would help support this interpretation.

The analysis focuses on theoretical FLOPs and weight memory, overlooking latency, routing overhead, and actual inference memory. The looped model also requires more inference compute to match the larger baseline, making the practical efficiency advantage less clear.

This is the most important point: the large-scale validation is somewhat limited. The trillion-token experiment is encouraging, but mainly considers one model comparison. The model scale is small.

Questions for authors

Please refer to the weaknesses above.

7 comments
AuthorsOpenReview ↗

Response to Reviewer kyQe [Part 1/5]

Response to Reviewer kyQe [Part 1/5]

We thank the reviewer for the positive assessment and constructive suggestions. We appreciate the recognition of our contribution (jointly modeling recurrence and sparsity addresses a meaningful gap in existing scaling laws), intuitive formulation (better held-out predictions), broad experiments (scaling sweep, downstream evaluations, and trillion-token experiment), and the value of showing the parameter efficiency of combining recurrence and sparsity at a practical scale. Below, we address the questions and concerns, and we have revised our paper accordingly, with changes highlighted in blue.


Q1. Generalization to different looping mechanisms, routing strategies, or model architectures

We thank the reviewer for the suggestion on validating generalization. Given the compute constraints, this work studies the standard middle-cycle looping, but provides a comprehensive sweep over four scaling axes (NN, DD, EE and RR) to model their relations more precisely. This scaling sweep already covers different model architectures: dense and MoE models (i.e., without routing vs. with routing), for which we evaluate the performance of our loop scaling laws in Section 3.2 (Figure 2) and Section 3.3 (Figure 3), respectively. To further strengthen our evaluation on a different looping mechanism, we evaluate our loop scaling laws on Saunshi et al. [a], which applies full-model looping to the entire network. Its experimental datapoints cover R∈{1,2,3,4,6}R\in\lbrace 1,2,3,4,6\rbrace and NN = 151M–906M (non-embedding parameters) at a single DD = 250B tokens (9 configurations from its Table 3). Since all runs share one DD, the data term BDβ+cB D^{\beta}+c of our law collapses to a single constant EE. We therefore fit the reduced form at fixed DD, L=ANeff(R)α+EL=A N_{\text{eff}}(R)^{\alpha}+E (where E=BDβ+cE=B D^{\beta}+c) on their datapoints, and compare the different recurrence mappings on held-out RR and NN. Table I shows that, compared with existing alternative formulations (linear and power-law), our bounded recurrence mapping gives lower RMSE scores, which is consistent with the findings in our paper (Figures 2 and 3) and shows that our loop scaling laws generalize to a different looping mechanism.

Table I. Loop scaling laws with different recurrence mappings on a different looping mechanism (full-model looping, Saunshi et al. [a]). The lowest RMSE is shown in bold.

Method│Datapoints and Protocols│LinearPowerNeff(R)N_{\text{eff}}(R)
Saunshi et al. [a]│seen: R∈{1,2,3,4}R\in\lbrace 1,2,3,4\rbrace, NN = 151M–906M, DD = 250B; held-out: R=6R{=}6│0.10000.00580.0045
Saunshi et al. [a]│seen: R∈{1,2,3,4,6}R\in\lbrace 1,2,3,4,6\rbrace, NN = 151M–453M, DD = 250B; held-out: NN = 906M│0.10710.01160.0035

Unlike different looping mechanisms, for which published datapoints are available (Saunshi et al. [a]), looped MoE remains an emerging area with few published results across routing strategies. We therefore leave the extension to different routing strategies (e.g., softmax vs. sigmoid gating) to future work, where our loop scaling laws can serve as a scaling-analysis tool to quantify their effective capacity (Section 4.1, Eq. 10).


References

[a] Saunshi et al. Reasoning with Latent Thoughts: On the Power of Looped Transformers. ICLR 2025.

AuthorsOpenReview ↗

Response to Reviewer kyQe [Part 2/5]

Response to Reviewer kyQe [Part 2/5]

Q2. Direct validation of compute- and memory-optimal choices

We thank the reviewer for this suggestion. Following the standard practice in scaling-law literature (e.g., Joint MoE Scaling Laws [a]), Figures 5–6 use the fitted law to compare configurations at resource budgets where not all configurations have observed losses, e.g., at matched training compute, each recurrence requires a different number of training tokens DR=Fˉtrain/[6Nunroll(R)]D_R=\bar F_{\text{train}}/[6N_{\text{unroll}}(R)]. As suggested, in the following, we directly validate the law-derived choices against independently trained configurations under matched budgets, using the available observed losses from our scaling sweep.

Recurrence selection at matched compute budgets. We compare the law-predicted losses of LoopMoE (E=8E{=}8) with separately trained models at varying recurrence R∈{1,2,3,4,6,8}R\in\lbrace 1,2,3,4,6,8\rbrace, using the directly observed losses at their compute-matched token budgets DRD_R. As shown in Table II, the recurrence with the minimal law-predicted loss aligns with the recurrence with the minimal observed loss in both settings: R=2R{=}2 for A0.3B-1.3B LoopMoE and R=1R{=}1 for A1.0B-5.4B LoopMoE, and the law closely predicts the directly observed losses. Thus, the law identifies when looping pays off and when it does not.

Table II. Predicted vs. observed loss of LoopMoE (E=8E{=}8) at matched training compute. All looped models are trained separately with the same supervision. The minimal predicted and observed losses are shown in bold.

(a) A0.3B-1.3B LoopMoE at 5.5×10205.5{\times}10^{20} FLOPs (minimal predicted loss at R=2R{=}2)

│R=1R=2R=3R=4R=6R=8
Loss (predicted)│2.22792.22392.22732.23462.25422.2761
Loss (observed)│2.22922.22482.22722.23532.26252.2750

(b) A1.0B-5.4B LoopMoE at 2.0×10212.0{\times}10^{21} FLOPs (minimal predicted loss at R=1R{=}1)

│R=1R=2R=3R=4R=6R=8
Loss (predicted)│2.00802.01502.02612.03902.06682.0946
Loss (observed)│2.00562.01582.02822.03822.06702.0968

Sparsity selection at matched memory budgets. We compare the law-derived E⋆E^{\star} with the directly observed losses of 1.0B models with RR fixed to 2 or 4, at the same 2.0×10212.0{\times}10^{21} training FLOPs as Table II(b), among the expert counts that fit within each weight-memory budget. Table III shows that the law-derived E⋆E^{\star} matches the expert count with the lowest observed loss under each memory budget, which supports using the fitted law for selecting sparsity under matched compute and memory constraints.

We’ve added these results to the Appendix D.3 in our revised paper.\color{blue}{\text{We've added these results to the Appendix D.3 in our revised paper.}}

Table III. Observed losses and law-derived expert count for 1.0B models with R=2 or R=4 at 2.0×10²¹ training FLOPs. Columns E=1–8 report the observed losses (bfloat16 weight memory in parentheses); the lowest feasible observed loss is shown in bold; †: configurations exceed the memory budget.

R│Memory budget (GB)│E=1 (2.0 GB)E=2 (3.3 GB)E=4 (5.8 GB)E=8 (10.8 GB)│Law‑derived E⋆
2│5.0│2.20222.13222.0695†2.0158†│2
2│10.0│2.20222.13222.06952.0158†│4
4│5.0│2.22282.15202.0911†2.0382†│2
4│10.0│2.22282.15202.09112.0382†│4

References

[a] Ludziejewski et al. Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient. ICML 2025.

AuthorsOpenReview ↗

Response to Reviewer kyQe [Part 3/5]

Response to Reviewer kyQe [Part 3/5]

Q3. Exponential form is an empirical choice, and behavior at greater recurrence depths

We thank the reviewer for this comment. Indeed, our proposed functional form is grounded in empirically observed trends. In fact, neural scaling laws are empirical in nature (Kaplan et al. [a]): they describe how model performance changes with scaling factors through predictive functional forms fitted to observed losses. Our bounded mappings model the diminishing returns of increasing recurrence seen in our experiments: the loss drops sharply over the first few passes and then approaches a plateau (R≤8R\le8, Section 1, Figure 1(b)). We further validate this trend at R=16R{=}16 (Section 3.2, Figure 2): when fitted on R≤8R\le8, the bounded mapping gives the lowest RMSE compared with the linear and power-law mappings; at R=16R{=}16, the loss already shows clear saturation: no further improvement despite using additional inference compute (Figure 2(b)). As stated in the boundary properties of the proposed functional forms (Eqs. 5 and 8), the bound is an asymptotic property of the mapping rather than a guarantee beyond the observed recurrence range. Moreover, larger recurrence depths are of limited practical value: inference compute grows linearly with RR, while the loss already shows saturation at R≈8R{\approx}8–1616. Therefore, the law is intended as a scaling-analysis tool for deriving practical looped MoE configurations within the empirically observed range, not as a universal law for unobserved recurrence depths of much greater value.


Q4. Interpretation of expert-path diversity

We use the expert-path diversity analysis in Appendix B to give an intuitive interpretation of how each token may access different experts across recurrent passes. We would like to clarify that this analysis is not intended as a proof that increased routing diversity directly causes the observed gains. Instead, it shows that higher MoE sparsity (i.e., a larger expert count) yields higher routing diversity (Table B.1), which is accompanied by better performance (Figure 3(b)).

Following the suggestion to provide more controlled experiments, we conduct an ablation on looped MoE models with normal and frozen routing at inference time (Table IV), comparing the loss with and without routing diversity across recurrent passes. With frozen routing, each token in each looped MoE layer reuses the same experts selected in the first recurrent pass for all later passes, rather than different experts across recurrent passes. Table IV shows that when freezing the router, the model performance consistently degrades with higher loss, with larger degradation at higher sparsity. This suggests that looped MoE models make use of routing diversity across recurrent passes to reduce the loss. As higher sparsity also gives higher routing diversity (Table B.1) and lower loss (Table IV), these results provide further supporting empirical evidence that routing diversity helps improve performance.

We’ve added these results to the Appendix B in our revised paper.\color{blue}{\text{We've added these results to the Appendix B in our revised paper.}}

Table IV. Validation loss with normal routing and frozen routing across expert counts EE at R=4R{=}4. ΔL=Lfrozen−Lnormal\Delta L = L_{\text{frozen}} - L_{\text{normal}}. E=1E{=}1 is the dense looped model without routing.

│E=1E=2E=4E=8E=16
LnormalL_{\text{normal}}│2.26062.19112.11112.05351.9974
LfrozenL_{\text{frozen}}│2.26062.26642.22682.20592.1903
ΔL\Delta L│0+0.0754+0.1157+0.1524+0.1929

References

[a] Kaplan et al. Scaling Laws for Neural Language Models. arXiv 2020.

Reviewer_1TQVOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a scaling law that jointly models model size, training data, recurrence, and mixture-of-experts sparsity. Its central contribution is a bounded effective-parameter mapping, where additional recurrent passes provide diminishing gains, and greater sparsity increases the fitted asymptotic gain and delays saturation. The formulation recovers dense, dense-looped, and non-looped MoE laws as special cases.

Critical strengths

S1: Jointly modeling recurrence and sparsity addresses a useful gap and connects naturally to existing scaling laws.

S2: Held-out predictions improve substantially, where recurrence RMSE falls from 0.0480 for the power-law baseline to 0.0100.

S3: The multidimensional sweep, 14 downstream benchmarks, and trillion-token experiment provide substantial empirical support.

Critical weaknesses

W1: Predicted architecture optima are not directly validated against neighboring configurations trained at equal compute. The large experiment also changes from final-pass to per-loop supervision.

W2: Observation subsampling establishes fit stability but does not establish calibrated uncertainty across training runs or extrapolation. Alternative saturation functions receive limited evaluation.

W3: Practical gains are conditional, where the smaller looped model uses 1.8× inference compute and trails overall performance. Memory optimization covers weights only.

W4: Exact data mixtures, learning-rate schedules, and the complete run/checkpoint manifest remain unclear.

Questions for authors

Q1: Do matched-compute training comparisons confirm the predicted recurrence optimum under the same supervision objective?

Q2: How stable are the conclusions under alternative functional forms and uncertainty estimates grouped by training run?

Q3: Can you provide complete reproduction details and measured latency and total-memory comparisons?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 1TQV [Part 1/4]

Response to Reviewer 1TQV [Part 1/4]

We thank the reviewer for the positive assessment and constructive suggestions. We appreciate the recognition of our contribution (jointly modeling recurrence and sparsity addresses a useful gap), predictive accuracy (reducing held-out recurrence RMSE), and substantial empirical support (the multidimensional sweep, 14 downstream benchmarks, and trillion-token experiment). Below, we address the questions and concerns, and we have revised our paper accordingly, with changes highlighted in blue.


Q1. Matched-compute validation of the predicted optima under the same supervision objective

We thank the reviewer for this suggestion. We would like to clarify that all scaling experiments (Section 4) are trained with final-pass supervision, under which Figure 7(d) shows that a smaller looped model can match a 2×2\times larger non-looped model on reasoning at matched training compute. Only Table 2 uses per-loop supervision as a practical extension that evaluates looped models at different test-time recurrences. Following the suggestion to provide direct matched-compute comparisons, in Table I below, we provide the law-predicted losses and observed losses on separately trained A0.3B-1.3B LoopMoE (E=8E{=}8) models of varying recurrence at matched compute, all trained with final-pass supervision. As shown, at the matched compute budget of 5.5×10205.5{\times}10^{20} FLOPs (where each model uses a token budget DR=Fˉtrain/[6Nunroll(R)]D_R=\bar F_{\text{train}}/[6N_{\text{unroll}}(R)]), LoopMoE with R=2R{=}2 attains the minimal law-predicted loss, which aligns with the minimal observed loss among the separately trained looped models. Moreover, as Table II shows, on downstream benchmarks, A0.3B-1.3B LoopMoE (with the recurrence of minimal predicted loss) nearly matches the 2×2\times larger A0.6B-2.9B MoE (E=8E{=}8, R=1R{=}1) on reasoning (mean of BBH and GSM8K; 23.323.3 vs. 23.623.6), similar to the findings in Table 2. Overall, these matched-compute training comparisons validate that the predicted recurrence optimum (i.e., the recurrence with the minimal law-predicted loss) aligns with the observed optimum under the same supervision objective.

We’ve added these results to the Appendix D.3 in our revised paper.\color{blue}{\text{We've added these results to the Appendix D.3 in our revised paper.}}

Table I. Predicted vs. observed loss of A0.3B-1.3B LoopMoE (E=8E{=}8) at matched training compute (5.5×10205.5{\times}10^{20} FLOPs). All looped models are trained separately with final-pass supervision. The minimal predicted and observed losses are shown in bold.

│R=1R=2R=3R=4R=6R=8
Loss (predicted)│2.22792.22392.22732.23462.25422.2761
Loss (observed)│2.22922.22482.22722.23532.26252.2750

Table II. A0.6B-2.9B MoE vs. A0.3B-1.3B LoopMoE (E=8E{=}8) on downstream benchmarks at matched training compute. Both models are trained with final-pass supervision. Overall is the mean of 14 benchmarks.

Model│R│BBH³GSM8K⁸ARC‑C²⁵ARC‑EOBQAHSPIQASIQAWinoBoolQDROP³MMLU⁵NQ⁵TQA⁵Overall
A0.6B‑2.9B MoE│1│23.823.443.266.234.661.873.947.959.263.335.430.29.627.542.9
A0.3B‑1.3B LoopMoE│2│24.821.839.661.636.057.072.147.858.259.432.827.67.922.340.6
AuthorsOpenReview ↗

Response to Reviewer 1TQV [Part 2/4]

Response to Reviewer 1TQV [Part 2/4]

Q2. Stability of conclusions under alternative functional forms and uncertainty estimates grouped by training run

We thank the reviewer for this suggestion. In Appendix C.4, we provided the bootstrap uncertainty of our proposed functional form by observation subsampling. As suggested, we further provide bootstrap uncertainty estimates to compare the alternative functional forms: linear, power-law, bounded Neff(R)N_{\text{eff}}(R), and sparsity-conditional bounded mapping Neff(R,m)N_{\text{eff}}(R,m), by resampling 90% of training runs without replacement 100 times to refit the loop scaling laws, and then compute the RMSE on held-out RR, EE, NN, and DD as in Figure 3(c). Table III provides the uncertainty estimates over 100 refits of the alternative functional forms. Among them, the proposed Neff(R,m)N_{\text{eff}}(R,m) attains the lowest RMSE estimate on all held-out sets, with a tighter confidence interval over the 100 subsampling runs. The ranking of functional forms based on their estimated RMSE on unseen recurrence, sparsity, model size, and data remains consistent with Figure 3(c) when subsampling across training runs to refit the laws with bootstrap uncertainty estimates. Thus, the conclusions remain stable when resampling training runs.

We’ve added these results to the Appendix C.5 in our revised paper.\color{blue}{\text{We've added these results to the Appendix C.5 in our revised paper.}}

Table III. Estimated held-out RMSE of different functional forms by subsampling 90% of training runs 100 times to refit the loop scaling laws. Held-out sets of RR, EE, NN, DD follow Figure 3(c). The lowest RMSE is shown in bold.

Evaluation│Linear│Power-law│Neff(R)N_{\text{eff}}(R)│Neff(R,m)N_{\text{eff}}(R,m)
│Estimate90% CI│Estimate90% CI│Estimate90% CI│Estimate90% CI
Held-out RR│0.2847[0.2670, 0.2910]│0.0480[0.0439, 0.0494]│0.0190[0.0174, 0.0199]│0.0100[0.0094, 0.0108]
Held-out EE│0.0896[0.0752, 0.0987]│0.0113[0.0096, 0.0129]│0.0075[0.0064, 0.0093]│0.0047[0.0045, 0.0071]
Held-out NN│0.0656[0.0619, 0.0982]│0.0077[0.0074, 0.0147]│0.0065[0.0060, 0.0134]│0.0050[0.0044, 0.0110]
Held-out DD│0.0824[0.0759, 0.0850]│0.0074[0.0071, 0.0077]│0.0058[0.0056, 0.0061]│0.0043[0.0042, 0.0046]
AuthorsOpenReview ↗

Response to Reviewer 1TQV [Part 3/4]

Response to Reviewer 1TQV [Part 3/4]

Q3. Practical gains of looped models: measured latency and total-memory comparison

We thank the reviewer for this comment. At matched training compute, looping trades additional inference compute for higher accuracy at a fixed parameter count and memory (Table 2). To address the concern on practical gains in inference latency and total memory with respect to overall performance, we provide a comprehensive evaluation of practical runtime for the models in Table 2 on an H100 GPU, measuring (1) latency: time per output token (TPOT, ms); and (2) runtime memory (GB): the peak memory measured by torch.cuda.max_memory_allocated, covering weights, KV cache, and runtime buffers.

Table IV presents the practical runtime comparisons, where we compare the models in Table 2: non-looped MoE, LoopMoE, and LoopMoE with built-in speculative decoding [a] using the R=1R{=}1 exit as a built-in draft model. Compared with the 2×2\times larger A0.6B-2.9B MoE, A0.3B-1.3B LoopMoE at R=4R{=}4 matches its reasoning accuracy (36.536.5 vs. 36.436.4) with 0.5×0.5\times the peak memory (3.083.08 vs. 6.056.05 GB), at the cost of 2.1×2.1\times TPOT in latency (11.5111.51 vs. 5.525.52 ms). However, as LoopMoE gives valid outputs at every recurrent pass, its first pass (R=1R{=}1) can naturally serve as a built-in lossless draft model for speculative decoding [a] (where we set k=6k{=}6 drafted tokens per verification step), which reduces this latency gap to 1.2×1.2\times (6.686.68 vs. 5.525.52 ms). As looping reuses the same weights and only adds a modest KV cache, these efficiency gains hold across test-time recurrences: from R=2R{=}2 to R=5R{=}5, LoopMoE uses only 0.50.5–0.6×0.6\times the peak memory of the 2×2\times larger MoE, while speculative decoding keeps its latency within 0.90.9–1.3×1.3\times.

Overall, these runtime results confirm the practical gains of looped MoE over a 2×2\times larger non-looped MoE: it matches the reasoning performance and keeps latency at about 1.2×1.2\times via speculative decoding with its built-in draft model, while using only about half of the runtime memory. We’ve added this new runtime efficiency analysis to Section 4.3 in our revised paper.\color{blue}{\text{We've added this new runtime efficiency analysis to Section 4.3 in our revised paper.}}

Table IV. Runtime benchmarks of A0.6B-2.9B MoE and A0.3B-1.3B LoopMoE across test-time recurrence RR, measured on 1 H100 GPU with batch size 1 and the same inference stack (CUDA graphs and MoE kernels), with average over 3 runs on the same prompts at the trained context length. Inference FLOPs, TPOT, and peak memory ratios are normalized to the A0.6B-2.9B MoE baseline. TPOT: time per output token. Peak mem: peak GPU memory, including weights, KV cache, and runtime buffers.

Model│R│Setting│FLOPs ratio ↓TPOT ratio ↓Peak mem ratio ↓TPOT (ms)Peak mem (GB)│Reasoning (acc)Overall (acc)
A0.6B‑2.9B MoE│1│Non‑looped│1.0×1.0×1.0×5.526.05│36.450.2
A0.3B‑1.3B LoopMoE│1│Non‑looped│0.4×0.7×0.5×3.612.88│14.636.6
A0.3B‑1.3B LoopMoE│2│Looped│0.8×1.1×0.5×6.242.94│28.443.0
A0.3B‑1.3B LoopMoE│3│Looped│1.1×1.6×0.5×8.873.01│35.646.0
A0.3B‑1.3B LoopMoE│4│Looped│1.5×2.1×0.5×11.513.08│36.546.7
A0.3B‑1.3B LoopMoE│5│Looped│1.8×2.5×0.5×14.053.15│36.546.9
A0.3B‑1.3B LoopMoE│2│Looped + spec. decoding│0.8×0.9×0.5×5.003.17│28.443.0
A0.3B‑1.3B LoopMoE│3│Looped + spec. decoding│1.1×1.2×0.5×6.723.17│35.646.0
A0.3B‑1.3B LoopMoE│4│Looped + spec. decoding│1.5×1.2×0.5×6.683.30│36.546.7
A0.3B‑1.3B LoopMoE│5│Looped + spec. decoding│1.8×1.3×0.6×6.993.37│36.546.9

References

[a] Leviathan et al. Fast Inference from Transformers via Speculative Decoding. ICML 2023.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their constructive feedback. We are encouraged that the reviewers recognized the contributions and value of our work, finding that it:

  • addresses a meaningful gap in existing scaling laws by jointly modeling looping recurrence and MoE sparsity (1TQV, kyQe);
  • provides an intuitive formulation that yields noticeably better held-out predictions than existing alternatives: linear and power-law mappings (1TQV, kyQe);
  • provides substantial empirical evaluation (multidimensional sweep, 14 downstream benchmarks, and trillion-token experiment), showing the parameter efficiency of combining recurrence and sparsity at a practical scale (1TQV, kyQe);
  • is clearly written, helping readers understand the architecture of looped MoE models and their trade-offs (9JrU);
  • provides useful empirical insights and guidance for future research (9JrU).

We have addressed all questions and concerns in our individual responses and revised the paper accordingly, with changes highlighted in blue\color{blue}{\text{blue}}. The main new results and analyses are:

  • Runtime benchmarks (Section 4.3, Table 3; Reviewers 1TQV, kyQe, 9JrU): Looped MoE matches the reasoning accuracy of a 2× larger MoE with ~0.5× the peak memory and ~1.2× the latency, using its first pass as a built-in draft model for speculative decoding. These results confirm the practical efficiency gains of looped MoE.
  • Direct validation at matched budgets (Appendix D.3; Reviewers 1TQV, kyQe, 9JrU): The law-predicted optimal recurrence (at matched compute) and expert count (at matched compute and memory) match the configurations with the minimal observed loss among separately trained models. These results show that the fitted law can guide the design of looped MoE models.
  • Routing-diversity controlled ablation (Appendix B, Table B.2; Reviewer kyQe): Freezing expert routing across recurrent passes consistently increases loss, with larger degradation at higher sparsity. These results provide evidence that routing diversity across recurrent passes contributes to the performance gains of looped MoE.
  • Robustness and generalization of our loop scaling laws (Reviewers 1TQV, kyQe, 9JrU): Compared to prior alternatives [a][b], our law attains the lowest held-out RMSE (i) under bootstrap resampling of training runs (Appendix C.5, Table C.6), (ii) on a different looping mechanism (Table I in our response to kyQe), and (iii) on larger looped MoE models (Response Part 4/4 to Reviewer 9JrU).

Additionally, we have clarified our contributions (Section 1; Reviewer 9JrU), added reproduction details on data and training (Appendix C.1, Tables C.1–C.2; Reviewer 1TQV), and updated the weight memory to the training precision bfloat16 rather than INT4 (Figure 6, Table C.5(b); Reviewer 9JrU).

We have uploaded the revised paper, which incorporates the above changes suggested by the reviewers to strengthen our work. Thank you again for your time and helpful feedback.

References

[a] Parcae: Scaling Laws for Stable Looped Language Models. COLM 2026.

[b] How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models. NeurIPS 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.