acceptodds
Under review as a conference paper at ICLR 2027

SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers

31 views

Abstract

Looped Transformers iterate a shared block of layers to gain effective depth without adding parameters, and prior work reports that this extra depth improves language modeling, multi-hop reasoning, and math. However, most of these evaluations compare models of the same size, where the looped model also spends more FLOPs per token, so its gains mix architectural advantage with extra compute. Meanwhile, work that compares Looped Transformers with unlooped ones at the same FLOPs often finds that the looped models perform worse, but there the unlooped models store more parameters. Therefore, to investigate whether looping itself has an advantage, we study looping on Mixture-of-Experts Transformers and compare each looped model with an unlooped Baseline that closely matches it in per-token FLOPs, total non-embedding parameters, and KV cache. Through three ablations, we arrive at an effective recipe we call SMELT (Sparse MoE Transformer, middle layers Loop Twice), which loops the middle half of layers twice and pays for the extra layer executions by narrowing the model. We scale SMELT and the Baseline up to 54B total non-embedding parameters and fit a separate Chinchilla-style scaling law to each architecture. SMELT's loss drops faster with compute, saving 14.7–18.0% of training compute on the compute-optimal frontier at FLOPs. Beyond validation loss, SMELT's advantage exceeds what its loss gap predicts on downstream benchmarks and grows with sample length and the number of in-context examples. Mechanistic analysis shows that, on their second pass, the looped layers put less attention on the attention sink and more on content-relevant tokens, an inductive bias that may underlie the observed performance gains. These results show that looping can improve Transformers even under matched budgets.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Looped Transformers, Mixture of Experts, Scaling Laws, Compute Efficiency

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 4.00 over 3 reviews, above 99% of the others.
Reviewer_1FWfOpenReview ↗
Rating
4(+2)
Confidence
4
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

The authors study whether looping itself provides an advantage in Transformers under fair budget matching. Looped Transformers iterate a shared block of layers to obtain greater effective depth without adding parameters, and prior work has reported that this extra depth improves language modeling, multi-hop reasoning, and math. However, most prior evaluations compare models of the same size, in which the looped model also uses more FLOPs per token, so the reported gains confound architectural benefits with extra compute. Through three ablation studies, the authors arrive at an effective design recipe called SMELT (Sparse MoE Transformer, middle layers Loop Twice), which loops the middle half of the layers twice and compensates for the extra layer executions by narrowing the model. They scale both SMELT and the baseline up to 54B total non-embedding parameters and fit a separate Chinchilla-style scaling law to each architecture. The paper reports that SMELT’s loss decreases faster with compute, yielding a 14.7–18.0% training compute saving on the compute-optimal frontier at 102110^{21} FLOPs.

Critical strengths
  1. The comparison against non-loop baselines is relatively fair and therefore convincing. I especially appreciate that the paper tries to match not only per-token FLOPs, but also total parameters and KV cache.

  2. The study is conducted on MoE models, which makes it more valuable for current large-scale model design.

  3. The NLP downstream evaluation is fairly comprehensive. The paper evaluates on a broad set of downstream tasks and also includes analyses on sample length and in-context learning.

Critical weaknesses
  1. The downstream evaluation does not include any math- or code-focused benchmarks. This is quite puzzling. Although the validation domains do include Math/STEM, the downstream tasks do not test these capabilities directly. Even basic and widely used benchmarks such as GSM8K or MATH500 are missing. It is therefore unclear whether the claimed advantages transfer to math and code tasks.

  2. The explanation for why loop=3 underperforms loop=2 is not convincing. The authors mainly attribute this to the fact that, under matched budgets, higher loop counts require narrower models. However, this does not seem sufficient. In the loop=2 analysis, the second visit already shows larger write magnitudes, and the cosine difference of the writes may also become significant (as discussed in works such as LoopCoder-v2: Only Loop Once for Efficient Test-Time Computation Scaling). It is therefore possible that increasing the number of loops leads to unstable writes and thus performance degradation. More analysis on loop=3 should be provided. Otherwise, it is difficult to rule out the possibility that the chosen loop design is itself suboptimal.

  3. The choice of looping the middle 50% of layers appears heuristic. This result may depend on the specific data or architecture used in this paper. As a result, I have concerns about the generality of the proposed recipe and whether the same choice would remain optimal for other models or training settings.

  4. The paper does not include any extrapolation experiments on loop count. This further increases my concern about the proposed loop architecture design. Without such experiments, it is hard to know whether the reported recipe is robust.

Questions for authors

NA

3 comments
AuthorsOpenReview ↗

Rebuttal 1/n

We thank Reviewer 1FWf for finding our budget-matched comparisons fair and convincing, for recognizing the practical value of studying MoE models, and for appreciating the breadth of our downstream evaluations.

Before addressing each point in detail, we would like to gently clarify the core scope of our claim regarding W2–W4. Our primary goal in this paper is to answer a foundational question: does looping itself provide a net advantage when per-token FLOPs, total parameters, and KV cache are all strictly matched? We present SMELT as a simple, effective recipe to test this hypothesis across scales, rather than as the globally optimal loop architecture.

At the same time, we fully agree that validating math/code transfer and probing the stability and generality of the recipe are essential, and we provide new SFT experiments and analyses for W1–W4 below.

W1: Math and Code Benchmarks

The models in the paper are pretrained base models, so we used domain-specific CE Gain on held-out validation sets as a pretraining proxy (Figure 16(a), Appendix C.1), where Code (20.4%) and Math/STEM (16.6%) show larger compute savings than general web data (14.8%).

To test downstream math and code directly, we fine-tuned our largest matched pair (1.6B active, S≈97S \approx 97\\%) with the same internal SFT data (≈\approx 120B tokens) and recipe, and evaluated both with the same harness. SMELT outperforms the Baseline on all four benchmarks.

BenchmarkBaselineSMELTΔ\Delta
GSM8K96.8997.57+0.68
MATH50084.2086.60+2.40
SWE-bench Multilingual [1]15.3317.67+2.34
SWE-bench Pro [2]19.1121.93+2.82

W2: Why Does r=3r=3 Underperform r=2r=2?

To separate the third pass from the narrowing that FLOP matching requires, we trained a control: r=3r=3 at SMELT's width and total parameters (1.33×1.33\times FLOPs and KV cache).

Model (200M, S≈85S \approx 85\\%)FLOPsTotal paramsVal. Loss ↓\downarrowDCLM Core ↑\uparrowWrite growth per passCross-pass cosine
Baseline (r=1r=1)1×1\times1×1\times1.944524.92——
SMELT (r=2r=2)1×1\times1×1\times1.925727.571.34×1.34\times0.360.36
r=3r=3, matched FLOPs (narrower)1×1\times1×1\times1.938527.151.31×1.31\times, 1.36×1.36\times0.500.50, 0.600.60
r=3r=3, SMELT's width1.33×1.33\times1×1\times1.897928.841.27×1.27\times, 1.33×1.33\times0.340.34, 0.450.45
  1. At the same width, the third pass helps. Validation loss drops from 1.92571.9257 to 1.89791.8979 (−0.0278-0.0278) and DCLM Core rises from 27.5727.57 to 28.8428.84.
  2. The regression at matched FLOPs comes from narrowing. The two r=3r=3 models differ only in the narrowing needed to match FLOPs, which costs +0.0406+0.0406 (1.8979→1.93851.8979 \to 1.9385), more than the third pass gains. So r=2r=2 remains the better use of a fixed FLOP budget, consistent with concurrent MoE work [4], where the per-loop gain is largest at r=2r=2.
  3. We see no unstable writes. Both r=3r=3 runs train without loss spikes, and their writes look alike (growth ≈1.3×\approx 1.3\times per pass, positive and rising cosine) even though one beats r=2r=2 and the other does not. The oscillating updates and r≥3r \ge 3 regression in LoopCoder-v2 [3] were observed in a different architecture: it is built on the Parallel Loop Transformer (PLT) [5], which SMELT does not use. To run loops in parallel, PLT feeds each token its left neighbor's state from the previous loop instead of its own, and LoopCoder-v2 attributes its regression to this position offset. SMELT loops sequentially, with each token continuing from its own state, so this cost does not exist in our model.
AuthorsOpenReview ↗

Rebuttal 2/n

W3: Generality of the 50% Span

We agree that the exact optimal span may vary across different datasets or architectures, and our goal is not to claim that 50% is universally optimal in every setting. Rather, we view 50% as a simple, effective baseline recipe:

  1. Selected empirically rather than heuristically, with strong performance nearby. We arrived at the 50% span via controlled sweeps at both S≈85S \approx 85\\% and S≈95S \approx 95\\% (Section 2.3, Figure 2(a), Table 9). Notably, all six nonzero spans (17% to 100%) beat the unlooped Baseline on validation loss, and neighboring spans perform nearly as well (at S≈95S \approx 95\\%, 33%, 50%, and 67% differ by less than 0.003 nats0.003\text{ nats}), showing that looping works well across a wide middle region rather than only at 50%.
  2. Transfers reliably across scales as a fixed recipe. Once selected at 200M (S≈85S \approx 85\\%), this simple 50% recipe transfers without any retuning across all 16 cells of our scaling grid (from 100M to 1.6B active parameters and up to 54B total parameters), outperforming the Baseline on validation loss in every cell. This is also consistent with prior studies that independently favor middle-layer looping across diverse setups [6–10].

W4: Test-Time Loop Extrapolation

SMELT is trained with a fixed loop count (r=2r=2) and is not designed for test-time loop extrapolation. On the 1.6B, S≈97S \approx 97\\% model, validation loss is 1.6561.656 for SMELT at r=2r=2 vs. 1.6711.671 for the Baseline. Running the same checkpoint with 3 and 4 loops at test time, with the residual scaling changed to 1/31/3 and 1/41/4 accordingly, raises validation loss to 1.7671.767 and 1.9231.923.

Enabling test-time extrapolation requires additional training techniques, such as randomized loop counts with input injection [8] and per-pass losses [11], and even with these, performance can still degrade beyond the trained depth [12]. LoopCoder-v2 [3] likewise keeps training and inference loop counts matched throughout. Our claim concerns this setting, where training and inference use the same loop count and SMELT consistently outperforms the Baseline.

AuthorsOpenReview ↗

Rebuttal 3/n

Revision. We will (1) add the post-SFT math and code results with the SFT protocol, (2) add the fixed-width r=3r=3 control and the per-pass write and cosine measurements for r=2r=2 and r=3r=3 to Section 2.5 and Appendix B.2, (3) state explicitly that SMELT is one effective loop design rather than the optimal one, and that the measured savings are a lower bound for looping, and (4) add the test-time extrapolation result to the limitations.

[1] Yang, J., Lieret, K., Jimenez, C. E., Wettig, A., Khandpur, K., Zhang, Y., Hui, B., Press, O., Schmidt, L., & Yang, D. (2025). SWE-smith: Scaling data for software engineering agents. Advances in Neural Information Processing Systems, 38. https://arxiv.org/abs/2504.21798

[2] Deng, X., Da, J., Pan, E., He, Y. Y., Ide, C., Garg, K., Lauffer, N., Park, A., Rane, C., Sampath, K., Krishnan, M., Kundurthy, S. R., Hendryx, S. M., Wang, Z., Zhang, C. B. C., Jacobson, N., Liu, B., & Kenstler, B. (2026). SWE-Bench Pro: Can AI agents solve long-horizon software engineering tasks? Proceedings of Machine Learning Research, 306, 23917–23934. https://proceedings.mlr.press/v306/deng26f.html

[3] Yang, J., Guo, S., Zhang, W., Zheng, T., Du, Y., Li, H.-S., Wu, J., Song, Y., Xing, Y., Cai, Q., Huang, Z., Hao, C., Tao, R., Liu, X., Zhao, W. X., Tang, M., Lv, W., Zhou, M., & Dai, B. (2026). LoopCoder-v2: Only loop once for efficient test-time computation scaling. arXiv preprint arXiv:2606.18023.

[4] Gao, Z., Chen, Y., Xiao, Y., Yang, X., Tao, R., Zhou, J., & Dai, B. (2026). Loop the Loopies! arXiv preprint arXiv:2607.16051.

[5] Wu, B., Chen, M., Luo, X., Yan, S., Yu, Q., Xia, F., Zhang, T., Zhan, H., Zhong, Z., Zhou, X., Qiao, S., & Bin, X. (2025). Parallel loop transformer for efficient test-time computation scaling. arXiv preprint arXiv:2510.24824.

[6] Saunshi, N., Dikkala, N., Li, Z., Kumar, S., & Reddi, S. J. (2025). Reasoning with latent thoughts: On the power of looped transformers. International Conference on Learning Representations (ICLR).

[7] Kapl, F., Angelis, E., Maile, K., von Oswald, J., & Bauer, S. (2026). From growing to looping: A unified view of iterative computation in LLMs. arXiv preprint arXiv:2602.16490.

[8] Geiping, J., McLeish, S., Jain, N., Kirchenbauer, J., Singh, S., Bartoldson, B. R., Kailkhura, B., Bhatele, A., & Goldstein, T. (2025). Scaling up test-time compute with latent reasoning: A recurrent depth approach. Advances in Neural Information Processing Systems (NeurIPS), 38.

[9] Prairie, H., Novack, Z., Berg-Kirkpatrick, T., & Fu, D. Y. (2026). Parcae: Scaling laws for stable looped language models. arXiv preprint arXiv:2604.12946.

[10] Chen, W., Li, T., Huang, W., Yin, Y., Shang, L., & Qin, C. (2026). LoopMoE: Unifying iterative computation with mixture-of-experts for language modeling. arXiv preprint arXiv:2606.04438.

[11] Zhu, R.-J., Wang, Z., Hua, K., Zhang, T., Li, Z., Que, H., Wei, B., Wen, Z., Yin, F., Xing, H., Li, L., Shi, J., Ma, K., Li, S., Kergan, T., Smith, A. K., Qu, X., Hui, M., Wu, B., ... Eshraghian, J. K. (2025). Scaling latent reasoning via looped language models. arXiv preprint arXiv:2510.25741.

[12] Yang, X.-W., Han, Z., Zhang, X.-H., Wei, W.-D., Shao, J.-J., Guo, L.-Z., & Li, Y.-F. (2026). Stabilizing recurrent dynamics for test-time scalable latent reasoning in looped language models. Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306.

Reviewer_zBr6OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies whether looping layers still helps a Transformer when the looped model gets no extra compute, parameters or KV cache. The authors loop the middle layers of an MoE Transformer and match it to an unlooped MoE Baseline on all three budgets, by narrowing the hidden size, adding experts, and using smaller attention heads with a bigger GQA ratio. Based on some ablations, they settle on looping the middle 50% of the layers twice. They call this recipe SMELT. Across multiple model sizes and sparsity levels, they reach a lower loss than the Baseline. They also separately fitted Chinchilla-style scaling laws, which translate this into a 14.7-18.0% compute saving at 1e21 FLOPs. The authors further measure the downstream gains and beat the baseline. Lastly, they prove that gains are larger on long documents and when in-context examples are given.

Critical strengths

(1) I like the experimental design of this paper. By matching per-token FLOPs, total parameters and KV cache at the same time, the authors remove the two confounds behind most earlier looping results (Table 1), and restoring the parameters through the expert count is a neat way to achieve this. The ablations also support the main claim in my opinion, since a deeper unlooped Baseline at matched budgets does not close the gap (Fig. 2b), and giving the Baseline more experts or more active parameters does not reproduce the long-document effect (Fig. 6a). This is the main reason why I believe that the gain comes from looping itself rather than from extra compute or capacity.

(2) SMELT is better than the Baseline for all four model sizes and all four sparsity levels. I like that each architecture gets its own scaling fit rather than a shared indicator. In addition, the residual analysis shows that the downstream gains are not simply a restatement of the loss gap. If it holds up, a ~15% saving at fixed parameter and KV-cache budgets would be of real practical relevance (this point weighs strongly in my assessment)

Critical weaknesses

(1) My main concern is that compute is only counted in FLOPs, and that the FLOP match holds at a single context length. SMELT executes 1.5x as many layers in sequence, at a narrower width and with more experts, which usually costs hardware utilization and latency. However, I was expecting throughput or wall-clock numbers, so the 14.7-18.0% should be read as a FLOP saving only. Moreover, the per-token FLOPs are measured at the mean document length. Since SMELT executes more layers and uses a bigger GQA ratio to keep the KV cache fixed, its attention FLOPs should grow faster with context length than those of the Baseline. In that case SMELT would get relatively more compute on long documents, which could explain part of the long-document and in-context gains in Sec. 4.2. Given that the compute saving is the main practical claim of the paper, I consider this a really important gap.

[action items] I would like to see the training throughput and inference latency for at least one matched pair (e.g., 1.6B at S≈97%), to show how much of the FLOP saving remains in wall-clock time.

(2) The claim that the advantage grows with compute rests on thin statistics. The paper trains each model only once, and its 96 matched pairs are six versions of each of the 16 run pairs, branched off at different training steps. Since these versions share the same training up to where they branch off, a win count like 83/96 makes the evidence look stronger than it is, as there are really only 16 independent comparisons. Moreover, the growing saving is not measured directly but comes from comparing two fitted curves, where SMELT's curve is only slightly steeper (exponent 0.250 vs. 0.237), and the paper gives no error bar for this difference. The error bars in Table 4 also show the problem, since at 1e22 FLOPs, which is beyond the trained range, two of the three include a saving of zero. Yet this claim is what carries the result beyond 1.6B.

[action items] I would like to see an error bar for the exponent difference, and a refit with the same irreducible loss E for both models, to check whether the steeper curve and the 1e21 saving survive. Also, I'd love some downstream win rates over the 16 independent comparisons and a second seed for one or two results, as it would also help to judge the noise.

[optional] If compute allows, even one matched pair at a larger scale (e.g., 8-32B active parameters) would show whether the saving really keeps growing beyond the trained range, as the fitted curves predict.

If these points are addressed, I am happy to raise my score.

Questions for authors

What is the SMELT/Baseline FLOP ratio at a few document lengths (e.g., 256, 1024 and 4096 tokens), and what are the attention head configurations of the two models? I would love to know whether the long-document gain in Fig. 6a still holds after accounting for any extra FLOPs, and whether it holds within a single domain

Do you have any explanation of why exactly 2 loops is the best? Will we have better gains if we scale to a huge number of loops (>20 loops)? I'm really curious about this.

6 comments
AuthorsOpenReview ↗

Rebuttal 1/n

We thank Reviewer zBr6 for the positive assessment of matching all three budgets at once, the ablations, and the separate scaling-law fits, and for the concrete action items, which we address below.

W1.1: Wall-Clock & Throughput

We benchmarked the 1.6B, S≈97S \approx 97% pair on NVIDIA H20 GPUs. Pretraining uses 128 H20 GPUs with FSDP and expert parallelism (EP=8\mathrm{EP}=8). Inference uses prefill–decode disaggregation (8 H20 GPUs for prefill and 8 for decode, EP=8\mathrm{EP}=8 on each side), with 4k-token inputs, 1k-token outputs, and batch size 32.

StageMetricBaselineSMELTChange
PretrainStep time (s / step)2.852.88+1.05%
PrefillThroughput (tok/s)95.28k89.36k−6.2%
DecodeThroughput (tok/s)4.51k3.57k−20.8%

Training: SMELT's step time is 1.05% longer, but in this pair its per-token FLOPs are also 0.85% higher (within the budget-matching tolerance). Its time per FLOP is therefore only 0.2% higher than the Baseline's. At this efficiency, the 14.7% compute saving at 102110^{21} FLOPs corresponds to about 14.5% less wall-clock training time.

Inference: Prefill throughput is 6.2% lower and decode throughput is 20.8% lower. The decode slowdown may reflect SMELT's greater sequential depth and additional expert-parallel all-to-all communication rounds. In the revision, we will restrict the claimed savings to training compute and training time and report these inference numbers as a limitation. That said, our SMELT implementation has no dedicated inference optimization yet; we expect that speculative decoding methods such as DFlash [1], which verify a block of draft tokens in one forward pass and shift decoding toward the compute-bound regime, could substantially narrow the gap between SMELT and the Baseline.

AuthorsOpenReview ↗

Rebuttal 2/n

W1.2 & Q1: Attention Cost Across Sequence Lengths

FLOP ratios & head configuration. We matched per-token FLOPs at the mean training document length (788 tokens); because SMELT executes more attention layers, its FLOP ratio over the Baseline grows slightly with length (table below). We apologize that due to company policy, we are unable to share the exact attention head counts and dimensions of our proprietary architecture (Appendix F.4); qualitatively, SMELT uses smaller attention heads with a higher GQA ratio to keep the KV cache matched (Table 2).

Compute-matched length gain. For each length bucket LL, we account for length-LL per-token FLOPs and interpolate the Baseline's loss at SMELT's cumulative FLOPs CSMELT(L)C_{\text{SMELT}}(L):

Matched Gain(L)=LBaseline,L(CSMELT(L))−LSMELT,L(CSMELT(L)).\text{Matched Gain}(L) = \mathcal{L}_{\text{Baseline}, L}\bigl(C_{\text{SMELT}}(L)\bigr) - \mathcal{L}_{\text{SMELT}, L}\bigl(C_{\text{SMELT}}(L)\bigr).

Averaged over the last 20k stable steps across the sparse pairs in Figure 6(a) (losses in mnat=10−3 nats\text{mnat} = 10^{-3}\text{ nats}):

Length bucket (upper bound)SMELT/Baseline FLOPs (average over pairs)Gain at equal tokens (mnat)Matched Gain (mnat)
321.0145.144.16
641.0148.247.64
1281.0158.818.34
2561.01710.7410.07
5121.02112.8011.71
10241.02712.2310.99
20481.04012.4910.89
40961.06112.9810.15
Mean, 32–256—8.237.55
Mean, 512–4096—12.6210.93
Long / short—1.53×1.45×

Charging SMELT for its extra attention FLOPs lowers the long-over-short difference from 4.394.39 to 3.38 mnat3.38\text{ mnat} (ratio 1.53×→1.45×1.53\times \to 1.45\times). Extra attention compute thus explains ~23% of the long-sample advantage, while 77% remains. We will add these FLOP ratios and the compute-matched analysis to Section 4.2.

Within single domains. Repeating Figure 6(a) within each validation domain confirms that the long/short ratio (512–4096 vs. 32–256 tokens) exceeds 1 everywhere, led by structured domains:

DomainCodeMath/STEMFinanceKnowledgeWeb
Long / short1.62×1.53×1.41×1.38×1.27×
AuthorsOpenReview ↗

Rebuttal 3/n

W2.1: Error Bars for the Exponent & Refit with Shared EE

In our original Table 3 (where each model fits its own EE), the exponent gap Δγ=γSMELT−γBaseline\Delta \gamma = \gamma_{\text{SMELT}} - \gamma_{\text{Baseline}} is +0.013+0.013, with a 95% bootstrap CI of [−0.0176,+0.0401][-0.0176, +0.0401] (2,000 paired-cell draws). This interval is wide mainly because EE and γ\gamma trade off strongly during fitting (correlation ≈0.97\approx 0.97 across draws)—a slightly higher EE shifts γ\gamma upward, and vice versa. Since both architectures are trained on the exact same corpus, we agree they should share the same irreducible loss EE.

Following the reviewer's suggestion, we jointly fit a single shared EE (1.44571.4457) and then refit (A,K,a,b,c)(A, K, a, b, c) for each architecture with EE held at this value:

ArchitectureEE (shared)AAKKaabbccγ=aca+c\gamma = \frac{ac}{a+c}RMSE
Baseline1.44571.44571.402×1031.402 \times 10^32.117×1062.117 \times 10^60.37160.37160.15320.15320.66240.66240.23810.23810.005530.00553
SMELT1.44571.44571.922×1031.922 \times 10^33.875×1063.875 \times 10^60.38790.38790.14730.14730.68790.68790.24800.24800.009520.00952

With a shared fixed EE, the estimated exponent gap is Δγ=+0.0100\Delta \gamma = +0.0100, with a 95% CI of [−0.0002,+0.0171][-0.0002, +0.0171] (Δγ>0\Delta \gamma > 0 in 97.2% of draws), and at S≈97S \approx 97\\% the CE Gain 95% CIs stay strictly above zero across all budgets, including 102210^{22} FLOPs:

Sparsity SSSMELT Budget CtgtC_{\text{tgt}}Independent-EE CE Gain [95% CI]Shared-EE CE Gain [95% CI]
≈85\approx 85\\%1020 FLOPs10^{20}\text{ FLOPs}10.010.0\\% [1.0,21.5][1.0, 21.5]9.09.0\\% [5.8,14.8][5.8, 14.8]
≈85\approx 85\\%1021 FLOPs10^{21}\text{ FLOPs}18.018.0\\% [8.2,28.0][8.2, 28.0]17.317.3\\% [11.8,24.6][11.8, 24.6]
≈85\approx 85\\%1022 FLOPs10^{22}\text{ FLOPs}23.523.5\\% [8.2,42.4][8.2, 42.4]24.924.9\\% [12.0,36.0][12.0, 36.0]
≈95\approx 95\\%1020 FLOPs10^{20}\text{ FLOPs}7.87.8\\% [3.4,14.9][3.4, 14.9]7.37.3\\% [4.8,13.4][4.8, 13.4]
≈95\approx 95\\%1021 FLOPs10^{21}\text{ FLOPs}15.815.8\\% [10.3,24.7][10.3, 24.7]15.815.8\\% [10.8,22.7][10.8, 22.7]
≈95\approx 95\\%1022 FLOPs10^{22}\text{ FLOPs}20.920.9\\% [0.0,48.1][0.0, 48.1]23.523.5\\% [11.7,34.5][11.7, 34.5]
≈97\approx 97\\%1020 FLOPs10^{20}\text{ FLOPs}6.86.8\\% [3.5,13.6][3.5, 13.6]6.56.5\\% [3.8,13.7][3.8, 13.7]
≈97\approx 97\\%1021 FLOPs10^{21}\text{ FLOPs}14.714.7\\% [8.1,25.0][8.1, 25.0]15.115.1\\% [10.1,21.9][10.1, 21.9]
≈97\approx 97\\%1022 FLOPs10^{22}\text{ FLOPs}19.619.6\\% [0.0,51.1][0.0, 51.1]22.922.9\\% [11.5,33.8][11.5, 33.8]
Reviewer_mr9zOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The authors conduct a systematic study evaluating a ladder of looped MoE transformers relative to a non-looped baseline. Unlike previous work, the authors actively control for per-token FLOPs, total parameters, and KV cache size. The authors' pre-training experiments and subsequent evaluation across a number of relevant benchmarks and task categories show that the looped MoE consistently outperforms the non-looped baseline. Crucially, the authors show a 14-18% CE gain, which is substantial.

Critical strengths
  1. The paper is very well structured, nice to read, and easy to understand.
  2. Apart from the weakness I highlight, the empirical study is very well constructed to avoid potential confounding factors when it comes to evaluating looping.
  3. The results spanning many important tasks reveal that the performance improvement of the method is commendable.
  4. The research direction of the paper is very relevant to practical pre-training workloads.
Critical weaknesses
  • Unfortunately, optimizer hyperparameters are very important in deep learning, especially in the context of pre-training. What effort was made to ensure that the looped MoE and the baseline were given an equal opportunity to find strong hyperparameters across the model scales evaluated? I am unable to fully understand the HP scaling strategy used from Appendix F1. This is quite important, as even a slight flaw in HP scaling can lead to substantial changes in measured CE at scale.
Questions for authors
  • How were optimization hyperparameters selected and scaled across the looped and baseline MoE.
3 comments

We thank Reviewer mr9z for finding the paper well structured, the experimental design well controlled against confounders, and the direction relevant to practical pretraining.

We agree that hyperparameters are critical: without a well-tuned Baseline, the gains of a new architecture could simply reflect hyperparameters that happen to favor it. Our protocol ensures that any hyperparameter advantage goes to the Baseline:

  1. The Baseline is tuned to be near-optimal across scales. The Baseline family is our team's tuned MoE scaling ladder. Its peak learning rate, independent weight decay (lr×wd\mathrm{lr} \times \mathrm{wd}), and initialization scheme were tuned by the team via grid sweep to be near-optimal at each model scale instead of using hyperparameter scaling rules (e.g., μ\muP), while the global batch size (256 sequences, ~1M tokens/step) and WSD schedule are shared across all runs (Appendix F.1).
  2. SMELT inherits the Baseline's hyperparameters without any search. At every scale and sparsity level, SMELT reuses the exact optimizer hyperparameters of its matched Baseline; the only change is the fixed residual scaling factor 1/r1/r inside the looped block [1]. Since these hyperparameters were tuned for the Baseline rather than SMELT, reusing them without adjustment keeps our CE Gain estimate conservative. In fact, given SMELT's narrower hidden dimension, we suspect from μ\muP intuition that it might prefer a larger learning rate, and we leave SMELT-specific tuning to future work.

Revision. Appendix F.1 will state the two points above and our hyperparameter protocol explicitly, complementing the existing note in Appendix F.4 that the Baseline configurations come from a tuned MoE family.

[1] Wang, S., Li, B., Zhang, G., Huang, W., Yan, S., & Li, J. (2026). On the residual scaling of looped transformers: Stability and transferability. arXiv preprint arXiv:2606.18524.

Thank you for the reply. I've updated my score accordingly.

Thank you for your thoughtful review and positive assessment of our work!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.