acceptodds
Under review as a conference paper at ICLR 2027

How to Loop MoE: Flatten the Experts, Untie the Attention

Abstract

Looped Transformers reuse one block of layers several times: by spending extra computation they push a model of fixed size further, and so use its parameters more fully; while sparse mixture-of-experts (MoE) models activate only a few of many experts for each token. Looped MoE bridges these two design philosophies and gives MoE models new potential for better expert usage, but it raises a question: how to loop a MoE? We answer it with Foil. With the expert parameters and the expert compute per token held fixed, Foil (1) flattens the experts, halving the expert layers, doubling the experts per layer and doubling the passes, so that every routing decision chooses from a larger pool, and (2) unties the attention, giving each pass its own attention parameters while the experts and routers stay shared. Experiments show that Foil clearly outperforms the unflattened looped baseline: at 20B tokens every Foil model has lower pretraining loss than the baseline; at 100B tokens the loss improves monotonically with the degree of flattening, the most flattened Foil ending 0.012 nat below the baseline at equal parameters and compute, with downstream accuracy on par or better; untying the attention also yields more balanced and more confident routing at equal shape. Our ablations analyse why Foil works and turn the findings into design guidance for looped MoE: the returns of looping and of widening the expert layers amplify each other, routing confidence tracks healthy expert use better than load balance, and a sparse looped MoE should therefore use more experts per layer and more passes. Code and configurations are available at https://anonymous.4open.science/r/how-to-loop-moe-iclr27-submission-014B.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mixture of experts, looped transformers, language model pretraining, language model architecture

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_T9yJOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Very interesting paper trying to claim that looping through more layers of moe will perform better nd there are more experts been selected. It's more like an experimental paper trying to argue that more experts are working better.

Critical strengths

The paper's idea is new and the equation representation is clear. The paper did experiments from small number of parameters all the way to 100B. And the paper came with the clear conclusion that their method can outperform base.

Critical weaknesses

I don't see the idea been apply to current production ready models. i.e. I don't know how much the results still hold after pretraining. I don't see scientific proof about the claim or at least some theory insights. The paper goes into an idea directly which I have no clue where it comes from.

Questions for authors

If the claim is more experts can work better why put them into more layers instead of increasing topk? e.g. if you used top8 what if you use top16, will it perform better?

Can we apply these ideas to production ready models e.g. qwen, deepseek or kimi and see how the idea holds?

6 comments
AuthorsOpenReview ↗

Rebuttal Part 1: Summary and Response to Weakness 1

Summary

We thank Reviewer T9yJ for the review, and for noting that the idea is new and that our method outperforms the baseline. We believe several of the concerns come from misunderstandings, which we clarify below. We provide the following clarifications, new experiments and revisions:

  1. Production-ready models (Weakness 1, Question 2): We explain why the idea does not yet appear in production models, and we apply the core designs of Qwen3, Qwen3.5, DeepSeek-V3 and Kimi to our models: Foil keeps its advantage, also on downstream tasks after pretraining (Table A1, Table A2).
  2. Scientific proof and theory (Weakness 2): We answer under two interpretations of "scientific proof" and show that our methodology, controlled experiments with formal derivation, is aligned with that of the technical reports of the models you mention.
  3. Where the idea comes from (Weakness 3): We explain where its novelty lies and list the prior research it builds on (Table C1).
  4. Top-k (Question 1): We clarify the setting: experts and compute are fixed, all models use top-2 routing, and flattening reduces the layers; a larger top-k buys its gain with compute (Table D1).

All corresponding changes have been added to the revised PDF and are marked in dark purple,\color{#5A2882}{\text{All corresponding changes have been added to the revised PDF and are marked in dark purple,}} and the changes that respond to several reviewers are marked in blue.\color{#0055CC}{\text{and the changes that respond to several reviewers are marked in blue.}}

We number the sentences under "Critical Weaknesses" as Weakness 1 to 3. Responses: Rebuttal Parts 1–3; tables: Rebuttal Parts 4–5.

Weakness 1, On production-ready models and on the results after pretraining

"I don't see the idea been apply to current production ready models. i.e. I don't know how much the results still hold after pretraining."

We believe it is entirely natural that this idea does not yet appear in current production models, and that this in fact speaks for our method:

  1. None of these models is a looped model. Foil changes how the experts are arranged inside the looped block and has to be applied from pretraining on.
  2. Our paper is a research paper. It studies an architectural innovation at the research frontier, not a reproduction or an analysis of existing models.
  3. This is exactly what shows the novelty of our method. These models have not yet adopted this approach, although our experiments show that it is effective, and no previous work has studied the same question.

Nevertheless, to address your concern, we added experiments. We applied the core designs of these models to ours: the attention of Qwen3 (grouped-query attention with normalised queries and keys), the hybrid attention of Qwen3.5 (gated full attention with gated delta-rule linear attention), the always-active shared expert of DeepSeek-V3 and Kimi, and the auxiliary-loss-free load balancing of DeepSeek-V3. With each, we pretrained Base and Foil-2, a representative flattened model: with the same design and the same number of parameters, Foil-2 has a lower loss than Base in every case (Table A1; the Qwen results are also in Appendix K, Table 30 of the revised PDF). On downstream tasks after pretraining, without further training, Foil still scores above the unflattened baseline (Table A2).

AuthorsOpenReview ↗

Rebuttal Part 2: Response to Weaknesses 2 and 3

Weakness 2, On scientific proof and theory

"I don't see scientific proof about the claim or at least some theory insights."

We respectfully disagree that our paper offers no scientific proof. We are not sure how you define "scientific proof", so we answer under the two interpretations we can think of.

  1. A pure and complete mathematical derivation, i.e. error bounds or convergence guarantees, as papers on optimal control prove stability. The question here is not whether we have done it, but:
    • Whether it can be done. To our knowledge, no existing theory can derive the loss of a trained language model from its architecture.
    • Whether it is necessary. In the field of LLMs, conclusions about architecture are not established by such derivations. Complete, standardised experimental validation, supported by formal derivation where it helps, is considered sufficient; the technical reports of today's leading open models, such as Qwen3, DeepSeek-V3 and Kimi K3, all proceed in this way.
  2. Experimental validation, as in the technical reports of the models you mention. There, architectural choices are likewise established by experiments; none of these reports proves by mathematical derivation that an architectural choice improves the model. The DeepSeek-V3 report, for example, states: "We investigate a Multi-Token Prediction (MTP) objective and prove it beneficial to model performance"; this proof is experimental. Our approach is the same, and we provide both:
    • Controlled experiments. Base and the Foil models have the same expert parameters, expert compute per token and effective depth (layers times passes), and differ only in how the experts are arranged, so a difference in loss can only come from the arrangement. Every Foil has a lower loss than Base in all twelve comparisons (554M and 1.4B parameters; 20B and 100B tokens); after 100B tokens, each difference is 6.7 to 49 times its standard error (Figure 3, Table 1 and Table 7 of the revised PDF).
    • Formal derivation. Section 2 and Appendix A state the looped block formally and derive what flattening holds fixed (the experts, the expert calls per token, the effective depth) and what it enlarges (the pool of every routing decision, the equivalent expert count), with lower and upper bounds on the distinct experts a token reaches. Appendix A also shows that, at the same shape, a model with shared attention is a special case of one with an attention per pass.

We therefore believe that we provide sufficient "scientific proof", and that our methodology is aligned with that of the technical reports of the leading open models.

Weakness 3, On where the idea comes from

"The paper goes into an idea directly which I have no clue where it comes from."

This idea has a clear lineage in prior research; it simply has no direct precedent, and this is exactly where its novelty lies.

First, we believe the question "where it comes from" does not necessarily apply to a new design:

  1. You also write in the strengths that "The paper's idea is new". A new architectural design usually has no ready-made source; this is what makes it new.
  2. For an architectural design, the more decisive question is whether it works under a controlled comparison. Our experiments show that it does (Weakness 2).

Second, our idea does build on lines of prior research (Table C1):

  1. Sparse MoE. Each layer holds many experts and every token activates a few of them (Switch Transformer, ST-MoE), in some designs with shared experts that are active for every token (DeepSeekMoE).
  2. Looped models and looped MoE. A looped model applies one block of layers several times (Universal Transformer, Huginn); MoEUT and LoopMoE apply looping to MoE models.

Building on these, our Introduction derives our question, and the first design choice, flattening the experts, from three observations (Figure 1, left). The second design choice, untying the attention, comes from our own experiments: with attention shared across passes, flattening discards the attention parameters of the layers it removes, and the gain stopped after the first step (Figure 4); giving every pass its own attention keeps these parameters at no extra compute, and the gain continues.

AuthorsOpenReview ↗

Rebuttal Part 3: Response to Questions 1 and 2

Question 1

"If the claim is more experts can work better why put them into more layers instead of increasing topk? e.g. if you used top8 what if you use top16, will it perform better?"

There is a misunderstanding here, and we are glad to clarify each point:

  1. Our claim is not that more experts work better. Adding experts adds parameters, against the premise of this paper that the parameters are held fixed, so it would not be a fair comparison. We only change the shape of the expert layers: all four models of our controlled comparison have the same 64 experts in the looped block.
  2. We do not put the experts into more layers. We do the opposite: fewer layers, each holding more experts.
  3. We do not use top-8. All models in this paper use top-2 routing.
  4. A larger top-k adds compute, so it is not a fair comparison. It activates more parameters per token, whereas our controlled experiments keep the compute and the parameters the same. Even if it improved the model, the gain would be bought with compute rather than come from the architecture design. A related experiment shows this: with one more active expert per token and layer, the loss falls, but the total compute of the model rises by 20% to 23% (the compute of the expert layers by 50%); at that higher compute, Foil-2 still has a lower loss than Base (Table D1).
  5. At fixed compute, letting every routing decision choose from more experts is exactly what flattening does: the pool grows from 8 experts in Base to 64 in Foil-1, while 2 are still selected each time.

Question 2

"Can we apply these ideas to production ready models e.g. qwen, deepseek or kimi and see how the idea holds?"

We are glad to answer this question:

  1. Our model's major components already derives from the architecture of a production-ready model. The MoE configuration of our Base model, 8 experts per layer with top-2 routing, is that of Mixtral (Jiang et al., 2024), a production model of Mistral AI.
  2. Our method is built on the generic Transformer architecture, so in principle it carries over. Its MLP experts and attention modules are those of the generic Transformer, the cornerstone of modern LLMs; it does not rely on any specific component such as gated delta-rule attention or multi-head latent attention.
  3. To address your concern, we added experiments. If your question refers to pretraining an open MoE model at the scale of Kimi or Qwen, we are afraid that this is not realistic for an academic study: the pretraining of DeepSeek-V3 alone took 2.66M GPU hours, according to its technical report. We nevertheless ran the closest experiments within our reach: we brought four core designs of these models into ours, namely (i) the attention of Qwen3, (ii) the hybrid attention of Qwen3.5, (iii) the always-active shared expert of DeepSeek-V3 and Kimi, and (iv) the auxiliary-loss-free load balancing of DeepSeek-V3, and pretrained Base and Foil-2 with each of them. Flattening remains effective every time: Foil-2 has a lower loss than Base, by 0.003 to 0.017 nat (Table A1).
Reviewer_88a6OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to design a looped sparse MoE model under fixed expert parameters and expert compute per token. The proposed method, Foil, combines two main ideas: (1) flattening the experts, i.e., reducing the number of expert layers while increasing the number of experts per layer and the number of loop passes proportionally, and (2) untying the attention across passes, while still sharing the experts and routers. The motivation is that looping allows each token to make multiple routing decisions and thus potentially reach a larger set of experts without increasing the stored expert parameters. Experimentally, the paper shows that Foil consistently outperforms an unflattened looped MoE baseline at equal parameters and compute. At 20B tokens, all Foil variants achieve lower pretraining loss than the baseline, and at 100B tokens the loss improves monotonically as the model becomes more flattened, with the most flattened variant outperforming the baseline by 0.012 nat. Downstream results are reported to be on par with or slightly better than the baseline.

Critical strengths
  1. The experimental settings are relatively comprehensive, and the conclusions are intuitive.
    The paper studies multiple settings, including different flattening levels, tied versus untied attention, different loop depths, and both 20B and 100B training budgets. The resulting trends are generally consistent and align well with intuition, which makes the empirical findings easier to trust.

  2. The paper addresses a meaningful design question for looped MoE models.
    The question of how to organize experts, layers, and loop passes in a looped MoE is clearly important. In this sense, the paper tackles a design problem with clear practical relevance, and the proposed study is meaningful for understanding and improving looped MoE architectures.

Critical weaknesses
  1. The paper lacks a more fine-grained analysis of cross-pass expert activation patterns.
    In particular, the paper does not analyze the similarity or overlap of activated experts across consecutive loop passes. Such an analysis could help clarify whether repeated passes are truly exploring different experts or simply revisiting similar routing patterns. Relatedly, I am curious whether using a separate router for each pass could lead to better performance, while likely introducing only a modest parameter increase.

  2. The downstream evaluation is somewhat limited.
    Although the paper includes several downstream tasks, the coverage is still relatively narrow. In particular, some common and basic evaluation datasets such as PIQA, ARC-C, and ARC-E are missing. Including these benchmarks would make the empirical evaluation more complete.

  3. The paper does not explore the use of shared experts.
    The current study focuses on standard sparse top-k routing, but does not consider hybrid designs with shared experts that are always activated for every token. Such a setting would be a meaningful implementation for current mainstream MoE.

  4. The paper does not evaluate auxiliary-loss-free MoE balancing strategies.
    All experiments rely on the standard auxiliary load-balancing loss and z-loss. It would strengthen the paper to test whether the reported conclusions remain valid under more recent auxiliary-loss-free balancing methods.

Questions for authors

NA

10 comments
AuthorsOpenReview ↗

Rebuttal Part 1: Summary and Response to Weakness 1

Summary

We thank Reviewer 88a6 for the careful review and the specific, constructive suggestions. In response, we provide the following new analyses and experiments:

  1. Analysis of cross-pass expert overlap (Weakness 1.1): on the final models of the paper, the routing of a token in consecutive passes is related, not random; the overlap of activated experts is 2.0–12.5 times that of random routing (Table A1, Table A2).
  2. A separate router for each pass (Weakness 1.2): we trained 5 new models at two sizes; separate routers do not help: the loss rises in four of the five models (Table A3, Table A4).
  3. PIQA, ARC-E and ARC-C (Weakness 2): we report the three tasks, zero-shot and five-shot, for the models of the paper and for four larger models with 1.4B parameters trained on 100B tokens; Foil is on par with Base at 554M parameters and above Base at 1.4B parameters (Table B1–Table B4).
  4. Shared experts that are always active (Weakness 3): we trained 5 new models; with shared experts Foil remains better than Base, and its advantage is larger (Table C1, Table C2).
  5. Auxiliary-loss-free balancing (Weakness 4): we trained 5 new models; Foil-2 remains better than Base at both sizes (Table D1).

All corresponding changes have been added to the revised PDF and are marked in dark green,\color{#005A32}{\text{All corresponding changes have been added to the revised PDF and are marked in dark green,}} and the changes that respond to several reviewers are marked in blue.\color{#0055CC}{\text{and the changes that respond to several reviewers are marked in blue.}}

Our responses are in Rebuttal Part 1 and Rebuttal Part 2; the tables are in Rebuttal Parts 3–8, lettered by weakness (A to D).

Weakness 1, On the analysis of cross-pass expert activation and a separate router for each pass

To address this weakness, we analyzed the overlap of the experts selected in consecutive passes on the final models of the paper, trained on 20B and on 100B tokens, and we trained 5 new models with a separate router for each pass.

Weakness 1.1, On fine-grained analysis of cross-pass expert activation patterns

"The paper lacks a more fine-grained analysis of cross-pass expert activation patterns. In particular, the paper does not analyze the similarity or overlap of activated experts across consecutive loop passes. Such an analysis could help clarify whether repeated passes are truly exploring different experts or simply revisiting similar routing patterns."

We find that the routing of a token in consecutive passes is related, not random.

We ran this analysis on the models trained on 20B tokens. With top-2 routing, the most common case (47–57% of the cases) is that a pass keeps one of the two experts selected in the previous pass and replaces the other (Table A1). The overlap, i.e. the number of experts that a token selects in both of two consecutive passes, is clearly above chance: it is 2.0–12.5 times the expected overlap under random routing (Table A2a). For example, in Foil-2 (32 experts per layer, 2 layers in the looped block, 8 passes) the overlap is 1.11, against an expected 0.125 under random routing. This shows that the routing of a token in one pass is related to its routing in the previous pass, rather than independent of it.

We also measured the models trained on 100B tokens, and they behave like the 20B models (Table A2b).

The related data and results are in Rebuttal Part 3.

Weakness 1.2, On a separate router for each pass

"Relatedly, I am curious whether using a separate router for each pass could lead to better performance, while likely introducing only a modest parameter increase."

We find that a separate router for each pass does not help: the loss rises in four of the five models.

We first ran this experiment at a small scale (250M-parameter models, 8B training tokens), with the routers untied across passes, i.e. each pass has its own router. The results did not improve, and the loss of the Foil models rose (Table A3a).

We then scaled the experiment up to the scale of the paper (554M-parameter models, 20B training tokens). There the loss rose for both the baseline and Foil (Table A3b).

We also found that, with a separate router for each pass, the overlap of activated experts across consecutive passes does drop, down to about the expected overlap under random routing (Table A4a and Table A4b).

The related data and results are in Rebuttal Part 4.

AuthorsOpenReview ↗

Rebuttal Part 2: Response to Weaknesses 2, 3 and 4

This comment continues Rebuttal Part 1 with our responses to Weaknesses 2, 3 and 4.

Weakness 2, On the downstream evaluation (PIQA, ARC-C and ARC-E)

"The downstream evaluation is somewhat limited. Although the paper includes several downstream tasks, the coverage is still relatively narrow. In particular, some common and basic evaluation datasets such as PIQA, ARC-C, and ARC-E are missing. Including these benchmarks would make the empirical evaluation more complete."

We evaluated Foil on these three tasks, and the results confirm the effectiveness of our method.

On the main models of the paper, which have 554M parameters, Foil performs on par with Base on the three tasks, with no clear difference (Table B1b, Table B2b). We therefore turn to larger models: Base and the Foil models with 1.4B parameters, against 554M parameters in the paper, pretrained on 100B tokens of FineWeb-Edu. At this scale, Foil shows an advantage over Base (Table B1a, Table B2a). We also put the questions of the three tasks together, and every 1.4B-parameter Foil model is above Base (Table B3).

We further find that the difference between Foil and Base on these downstream tasks becomes clearer as the scale grows, both in model parameters (Table B4a) and, for the larger models, in training tokens (Table B4b). We conjecture that this trend continues as the scale grows further.

The related data and results are in Rebuttal Part 5 and Rebuttal Part 6.

Weakness 3, On shared experts that are always active

"The paper does not explore the use of shared experts. The current study focuses on standard sparse top-k routing, but does not consider hybrid designs with shared experts that are always activated for every token. Such a setting would be a meaningful implementation for current mainstream MoE."

We tested shared experts, and the flattening idea of Foil remains effective when shared experts are used.

We applied shared experts to our models. In our setting, each layer has one shared expert that is always active, and the router selects two further experts (top-2), so three experts are active for every token. We trained models at two sizes: 250M parameters on 8B tokens, and 554M parameters, the size of the paper, on 20B tokens. The Foil models flatten Base step by step: written as (experts per layer, layers in the looped block, passes), Base is (8, 8, 2), Foil-3 is (16, 4, 4), Foil-2 is (32, 2, 8) and Foil-1 is (64, 1, 16). As the Foil model we mainly use Foil-2; its degree of flattening is moderate, between Foil-3 and Foil-1. The results show that, with shared experts, Foil is still better than Base (Table C1a, Table C1b), and its advantage is even larger (Table C2a, Table C2b).

The related data and results are in Rebuttal Part 7.

Weakness 4, On auxiliary-loss-free balancing

"The paper does not evaluate auxiliary-loss-free MoE balancing strategies. All experiments rely on the standard auxiliary load-balancing loss and z-loss. It would strengthen the paper to test whether the reported conclusions remain valid under more recent auxiliary-loss-free balancing methods."

We tested auxiliary-loss-free load balancing, and the flattening idea of Foil remains effective when it is used.

We applied an auxiliary-loss-free balancing strategy to our models, following the method of Wang et al. (2024). We trained models at two sizes: 250M parameters on 8B tokens, and 554M parameters, the size of the paper, on 20B tokens. The Foil models flatten Base step by step: written as (experts per layer, layers in the looped block, passes), Base is (8, 8, 2), Foil-3 is (16, 4, 4) and Foil-2 is (32, 2, 8). The results show that, with auxiliary-loss-free balancing, at the small size Foil-2 is better than Base and Foil-3 is on par with Base (Table D1a), and at the size of the paper Foil-2 is still clearly better than Base (Table D1b).

The related data and results are in Rebuttal Part 8.

AuthorsOpenReview ↗

Rebuttal Part 3: Experimental Results and Discussion for Weakness 1.1 (overlap of activated experts across consecutive passes)

This comment provides the experimental results and discussion for Rebuttal Part 1, Weakness 1.1.

Reviewer's point. The paper does not analyze the overlap of activated experts across consecutive loop passes, so it is unclear whether repeated passes explore different experts or revisit similar routing patterns.

What we did. We measured this overlap on the final models of the paper. No new training was needed.

  • Models: Base and the three Foil models of the paper, each trained on 20B and on 100B tokens (8 models, one training seed each). All four have the same expert parameters and the same expert compute per token; they differ in how the experts are arranged, written as (experts per layer, layers in the looped block, passes): Base (8, 8, 2), Foil-3 (16, 4, 4), Foil-2 (32, 2, 8), Foil-1 (64, 1, 16).
  • Data: 255,500 probe tokens from 500 documents, the same tokens used for the routing metrics of the paper.
  • Measurement: for every token and every layer of the looped block, we record the 2 experts selected in each pass and compare each pass with the next one.

What we found. The routing of a token in consecutive passes is related, not random. In the most common case a pass keeps one of the two experts selected in the previous pass and replaces the other (Table A1). The number of experts kept is 2.0–12.5 times what random routing would give (Table A2). The two tables below give the full numbers.



Table A1 — How many of its two experts a token keeps from one pass to the next

Each pass selects 2 experts for every token at every layer. The table gives the share of cases in which both, one, or none of these experts are selected again in the next pass at the same layer; one case is one token at one layer for one pair of consecutive passes. Models, trained on 20B tokens, written as (experts per layer, layers, passes): Base (8, 8, 2), Foil-3 (16, 4, 4), Foil-2 (32, 2, 8), Foil-1 (64, 1, 16).

ModelBoth experts keptOne expert keptNo expert kept
Base21%57%22%
Foil-328%54%18%
Foil-228%55%17%
Foil-116%47%37%

Takeaway: Keeping exactly one of the two experts is the most common casein all four models (47–57 percent of the cases).\boxed{\begin{array}{l}\text{Takeaway: Keeping exactly one of the two experts is the most common case} \cr \text{in all four models (47–57 percent of the cases).}\end{array}}



Table A2 — Overlap of activated experts across consecutive passes, compared with random routing

Overlap: the number of experts that a token selects in both of two consecutive passes at the same layer (at most 2; the table gives the mean over tokens). Expected overlap under random routing: 4/E, where E is the number of experts per layer. Ratio: overlap divided by expected overlap. Models, written as (experts per layer, layers, passes): Base (8, 8, 2), Foil-3 (16, 4, 4), Foil-2 (32, 2, 8), Foil-1 (64, 1, 16).

Table A2a — Models trained on 20B tokens

ModelExperts per layer (E)Overlap (max. 2)Expected overlap under random routing (4/E)Ratio
Base80.990.52.0
Foil-3161.090.254.4
Foil-2321.110.1258.9
Foil-1640.780.062512.5

Table A2b — Models trained on 100B tokens

ModelExperts per layer (E)Overlap (max. 2)Expected overlap under random routing (4/E)Ratio
Base81.040.52.1
Foil-3161.200.254.8
Foil-2321.130.1259.1
Foil-1640.660.062510.6

Takeaway: The overlap is 2.0–12.5 times the expected overlap under random routingat 20B tokens, and 2.1–10.6 times at 100B tokens.\boxed{\begin{array}{l}\text{Takeaway: The overlap is 2.0–12.5 times the expected overlap under random routing} \cr \text{at 20B tokens, and 2.1–10.6 times at 100B tokens.}\end{array}}

Reviewer_3dKmOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper explores how to introduce recurrence into MoE models. The approach that they take is to “flatten” the expert pool across neighboring layers. This holds the parameters constant while increasing the experts available at each layer. Each “layer” still gets its own attention weights. The presented results show that flattening the model more generally helps improve val loss compared to the base model. Additional ablation studies and analysis of the routing behavior are presented.

The authors find that sharing experts decreases the load balance and increases router confidence, indicating more specialization. Of course these things would likely also correlate with larger expert pools generally. The author reports that expanding the expert layers and looping are complementary. They also suggest that routing confidence may be a diagnostic to help guide the design of looped models.

Critical strengths

There is interest in both MoE models and increasing interest in the community in looped models as well. This work presents an empirical study of the nice idea to share experts across neighboring layers. This is a nice parameter- and compute-matched way to improve the expressivity of the model. Various aspects of the design such as tying/untying attention weights are similarly explored. Sharing experts across layers seems to lead to a minor, but consistent improvement in validation loss compared to the baseline.

The analysis of expert routing behavior is a nice addition here. The various metrics included are informative and the analysis is interesting.

Critical weaknesses

There is a number of recent work that has explored variants of expert sharing across layers. These are cited and mentioned briefly, but the novelty seems limited. The claimed distinction is “none has studied how the experts should be distributed over the looped block…when the expert parameters and the compute are fixed”. I suppose it is nice to have a more strictly controlled study, but the differences seem relatively thin here. Some of these seem to fall into the window of concurrent work. Qiu et al., 2026 in particular, seems to present a very similar idea but is quite recent. More broadly, the idea of sharing experts across layers has been shown to be useful before in the cited related work.

The expert use metric is set up as a core metric for analyzing the behavior, but then kind of disappears from the main paper. It shows up in Figure 1 and says “More expert per token” but under the compute and param-matched setting the paper claims, the trend is actually reversed (numbers in appendix). The figure shows a setup with increasing expert compute and therefore is arguably misleading. Increasing sharing while holding other things equal decreases the number of unique experts per token. I think this is a reasonable finding but somewhat contradicts the setup in the main paper.

The downstream accuracy results seem somewhat mixed and appear like they may be within noise. Things like error bars in Figure 3 would be helpful.

I find the presentation of the ablation studies and takeaways somewhat difficult to follow, both in terms of the writing and the presentation of the results. It is challenging to parse through all the different configurations in Table 1 especially given that the relevant loss differences are of less than 0.1. Some of the notation (k / E) and setups (half the passes, half the experts per layer) should’ve been clarified more in the text.

Ideally takeaway 1 could be supported by some nice visualization illustrating the point rather than having to parse through different configs to check the extent to which it’s true.

I’m really not sure that Takeaway 2 is well supported. There is a series of models and the presented metric sort of aligns with the performance. However, I’m not really sure why we should expect the MRR to align in such a way. It seems like a somewhat post-hoc association. It's presented as potentially being used in a prescriptive sense, but that isn't actually tested in the paper at all. Also if you're training models to observe that metric, you can just measure val loss. It seems like a more comprehensive quantitative comparison here could be done with all the models from Table 1.

Questions for authors

Can you expand on the relationship between this work and the prior cited work that has shared experts across layers?

What insight does the expert use metric provide? It’s presented in the main text, but then not really discussed.

Why would we expect the MMR saturation to relate to model quality?

10 comments
AuthorsOpenReview ↗

Rebuttal Part 1: Summary and Response to Weakness 1.1

Summary

We thank Reviewer 3dKm for the detailed review. We believe that several of the concerns come from misunderstandings, which we clarify below. In response, we provide the following clarifications, new analyses and revisions:

  1. Novelty (Weakness 1, Question 1): We explain what is unique to our work and how it differs from other MoE models, looped or not, in the question studied, in the model architecture and in the computation graph. We also provide a new experiment that supports our novelty (Table A1–Table A4).
  2. Number of distinct experts per token (Weakness 2, Question 2): We clarify that Figure 1 is about looping and not about our method, Foil, and that along the flattening sequence this number follows a lower bound set by the architecture. We have revised the paper so that the role of this metric is clear (Table B1, Table B2).
  3. Downstream accuracy (Weakness 3): We added error bars to Figure 3 as you suggested. We also report larger models, on which the advantage of Foil is clearer, and three further tasks (Table C1, Table C2).
  4. Presentation of the ablation studies (Weakness 4): We explain how Section 4 is organised and show that the loss differences in its table are far larger than their standard errors. We have rewritten the caption of this table and added a roadmap to Section 4 (Table D1, Table D2).
  5. Visualization for Takeaway 1 (Weakness 5): As you suggested, we added a figure that shows Takeaway 1 directly (Table E1).
  6. Takeaway 2 and MMR (Weakness 6, Question 3): We clarify that Takeaway 2 does not state that MMR corresponds to performance, and we have reworded it. Following your suggestion, we tested its statement on all our models, and the results are consistent with it (Table F1–Table F3).

All corresponding changes have been added to the revised PDF and are marked in dark cyan,\color{#008B8B}{\text{All corresponding changes have been added to the revised PDF and are marked in dark cyan,}} and the changes that respond to several reviewers are marked in blue.\color{#0055CC}{\text{and the changes that respond to several reviewers are marked in blue.}}

We number the paragraphs under "Critical Weaknesses" as Weakness 1 to 6 and the questions as Question 1 to 3. Our responses are in Rebuttal Parts 1–5; the tables are in Rebuttal Parts 6–9.

Weakness 1, On the novelty

We respectfully, but strongly, disagree that our contribution lacks novelty.

Our paper studies MoE and, more importantly, looping. Looping (applying one block of layers repeatedly) goes back to the Universal Transformer (Dehghani et al., 2019) and was recently scaled to large pretraining runs (Geiping et al., 2025; Zhu et al., 2025); it is an emerging, powerful tool with many open design questions.

Our contribution lies not in looped models alone or in MoE alone, but in the architecture of looped MoE models. In a looped MoE the two are coupled, so to get the best out of such a model they should not be treated separately. This is the value of our study: rather than changing only the passes or the looping mechanism, or only how the MoE shares its experts, we adjust the MoE and the loop together (e.g. Table E1).

To illustrate this, we compare our question and our design with each of the cited works (Table A1, Table A2), we set sharing across adjacent layers and looping side by side (Table A3), and we report a new experiment that moves our model towards the design of Qiu et al. by giving each pass its own router; the advantage of flattening then disappears (Table A4).

Weakness 1.1, On the novelty and the relation to prior work

"… the novelty seems limited. … I suppose it is nice to have a more strictly controlled study, but the differences seem relatively thin here."

The question we study has not been studied before.

We ask how the experts of a looped MoE should be arranged inside the looped block, and whether each pass should have its own attention, when the expert parameters, the expert compute per token and the effective depth (layers times passes) are all fixed. Our four main models are identical in these three respects (64 experts in the looped block, 32 expert calls per token, an effective depth of 16) and differ only in the arrangement; written as (experts per layer, layers in the looped block, passes), Base is (8, 8, 2), Foil-3 (16, 4, 4), Foil-2 (32, 2, 8) and Foil-1 (64, 1, 16) (Table A1).

Existing work falls into two groups, and neither addresses this question (Table A2):

  1. The five works that share experts across adjacent layers all study MoE models that are not looped.
  2. Two works study looped MoE models; the closest to ours is LoopMoE. It fixes the parameters and the compute, but its control is a MoE that is not looped: it studies whether looping helps. We take looping as given, additionally fix the number of experts, the expert calls per token and the effective depth, and study how the experts are distributed inside the looped block. The former conclusion does not imply the latter.

The related data and results are in Rebuttal Part 6.

AuthorsOpenReview ↗

Rebuttal Part 2: Response to Weakness 1.2

Weakness 1.2, On the relation to Qiu et al., 2026 and to earlier evidence that sharing experts is useful

"… Qiu et al., 2026 in particular, seems to present a very similar idea but is quite recent. More broadly, the idea of sharing experts across layers has been shown to be useful before …"

The expert sharing of Qiu et al. may look similar to our flattening, but the question we study is a different one. That sharing experts across layers helps is supported by external evidence, as our Limitations section states; our question starts where that evidence ends.

First, in model architecture, our model differs from that of Qiu et al.:

  1. The experts are visited in a different order. Once the loop is introduced, the order in which the expert pools are visited changes (example below; Table A3a). For Foil-1, whose looped block has a single layer, the difference lies in points 2 and 3.
  2. The first and last layers differ. Recent looped language models, such as Huginn (Geiping et al., 2025) and LoopMoE, have three parts: one or a few layers at the start and at the end are separate and run once, and only the block in the middle is looped. Our model follows this design, and we flatten only the looped block. Qiu et al. place every layer in a sharing group, so their first and last layers also share experts with their neighbours; their design is not of this looped kind (Table A3b).
  3. The router differs. We share one router across all passes; Qiu et al. use one router per layer (Table A3b).

Second, that sharing experts is useful does not imply that flattening a looped MoE is useful:

  1. The two computation graphs are different. Take 8 layers whose experts A,B,C,D,E,F,G,H, are merged pairwise into four pools, AB, CD, EF and GH, each used twice. Sharing across adjacent layers visits them as AB, AB, CD, CD, EF, EF, GH, GH; a loop visits them as AB, CD, EF, GH, AB, CD, EF, GH (Table A3a). Such a difference is very likely to lead to a different specialisation of the experts, and it grows with depth: for a model as deep as Kimi K3 (93 layers, 92 of them MoE layers), if every pool is shared twice, adjacent sharing reuses a pool in the very next layer, whereas a block looped twice has about 45 layers and reuses a pool about 45 layers later.
  2. The results are different too. In Qiu et al. (their Table 3), the size of the sharing group makes little difference (perplexity 18.69 to 18.75 for groups of 2 to 18 layers, against 18.83 without sharing). In their Table 4, only sharing between directly adjacent, aligned layers helps (18.71); sharing shifted by one layer gives 18.82 and random pairing 18.83, the level without sharing. In their design, whether sharing helps depends on which layers share, and the order of a loop is not their order. In our main models, by contrast, every flattening step lowers the loss: after 100B training tokens, the loss of the 554M-parameter models falls at each step from Base to Foil-1, by 0.0046, 0.0049 and 0.0024 nat (each at least 3.9 standard errors).
  3. When we move our model towards the design of Qiu et al., the gain of flattening disappears. To also examine the relation experimentally, we trained new models in which each pass has its own router, as each layer has in Qiu et al.; each pass then has its own router and its own attention, and only the experts are shared. With this change, the loss of the Foil models rises by 0.011 to 0.018 nat, more than their whole advantage over Base (Table A4). In our looped MoE models, the gain of flattening thus depends on the router shared across passes, which sharing across adjacent layers does not have. This is one more respect in which a looped MoE and the sharing of experts across the layers of a MoE are two different things.

We believe a misunderstanding may have arisen: our method may have been read as one more variant of sharing experts across the layers of a MoE. We would like to clarify that, once the loop is introduced, our method is no longer the same as existing methods that share experts across the layers of a MoE (Table A3). We therefore respectfully maintain that the contribution of this paper is novel.

Finally, as you note, the work of Qiu et al. is "quite recent" and falls "into the window of concurrent work".

The related data and results are in Rebuttal Part 6 (Table A3) and Rebuttal Part 7 (Table A4).

AuthorsOpenReview ↗

Rebuttal Part 3: Response to Weakness 2

Weakness 2, On the number of distinct experts per token and Figure 1

We understand your concern. Below we explain the role of the number of distinct experts per token (the "expert use metric") in the paper, clarify a misunderstanding about Figure 1, and show that the falling trend follows a lower bound set by the architecture. We have revised the paper and added explanations in the relevant parts.

Weakness 2.1, On the role of the expert use metric

"The expert use metric is set up as a core metric … but then kind of disappears from the main paper."

The analysis of the paper rests on the load balance B2B_2 and the routing confidence MMR; in the main text the number of distinct experts per token is used as an additional, descriptive metric, to motivate looping (Figure 1), and the appendix reports it for every model.

B2B_2 measures how evenly the experts are used, and MMR is the ratio of the highest router score to the median one. Section 2.3 of the revised PDF now presents the number of distinct experts per token after these two metrics, as an additional metric.

Weakness 2.2, On Figure 1

"It shows up in Figure 1 and says "More expert per token" but under the compute and param-matched setting … the trend is actually reversed … The figure … is arguably misleading."

We understand the point, but this is a misunderstanding: Figure 1 and the appendix describe two different things.

  1. Figure 1 does not answer "why flatten"; it answers "why loop a MoE". When a model of one shape is looped more often, the number of distinct experts a token reaches grows from 16 without looping to 36 with eight passes: looping has a natural advantage for a MoE. This is the motivation for studying Foil, not the method of Foil.
  2. The falling trend in the appendix is a different matter. In Figure 1 the shape of the model (the layers and the experts per layer) is fixed and only the passes grow, so the number of experts reached can only increase. The appendix reports this number when the shape itself changes, i.e. when a looped MoE is flattened. The two are different things and do not contradict each other.

Weakness 2.3, On flattening and the number of distinct experts per token

"Increasing sharing while holding other things equal decreases the number of unique experts per token. … somewhat contradicts the setup in the main paper."

The trend exists, and it follows the lower bound that each architecture sets. The reasons are (Base and the Foil models are defined in Rebuttal Part 1):

  1. The shape of the architecture gives these models the same upper bound but different lower bounds. The looped block of Base has 8 different layers, and every token selects 2 experts in each, so a token reaches at least 16 distinct experts; Foil-1 has a single layer, so its minimum is 2. The higher number of Base is guaranteed by its architecture, not by better routing (Table B1).
  2. The fall of the measured values comes almost entirely from the fall of the lower bound: the measured number falls from 24.1 to 9.7 while the lower bound falls from 16 to 2. Above the lower bound, the four models reach 8.1, 9.8, 8.5 and 7.7 further experts (Table B1).
  3. We claim that flattening gives every routing decision a larger pool of experts; we have never claimed that it lets a token use more experts.
  4. An additional experiment shows that reaching more distinct experts does not necessarily make a model better. When each pass has its own router, the number of distinct experts per token rises to 1.5 to 2.1 times its value, and the loss rises (Table B2).

We have added this explanation to Section 2.3 and to the table of routing metrics in the appendix.

The related data and results are in Rebuttal Part 7.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Thank You for the Response, and My Response to Your Comment (1/3): Novelty

Dear Reviewer 3dKm,

Thank you for your detailed reply; we are glad that the new figure, error bars and notation address those points.

1. The lineage

"Looping is also sharing experts across layers, so this work is also in that lineage."

Our work is related to this line, and we cite it. But if belonging to this lineage limited novelty, the same would hold for most looped models, including LoopMoE (EMNLP 2026). Would you see it that way?

2. The degree of difference

"I acknowledge it is a different sharing pattern than much of the prior work … and there are various architectural distinctions …"

Several accepted papers keep the looped, weight-shared Transformer and change or analyse one design choice in it. Would they count as novel by your standard?

  • Relaxed Recursive Transformers (ICLR 2025, arXiv:2410.20672): how strictly the weights are tied across passes.
  • Reasoning with Latent Thoughts (ICLR 2025, arXiv:2502.17416): a block looped several times against a non-looped model of the same effective depth.
  • Looped Transformers for Length Generalization (ICLR 2025, arXiv:2409.15647): an adaptive number of passes.
  • LoopFormer (ICLR 2026, arXiv:2602.11451): conditioning each pass on the step, with a consistency loss across numbers of passes.
  • Think-at-Hard (ICML 2026, arXiv:2511.08577): further passes only for the tokens predicted to be hard.
  • From Growing to Looping (ICML 2026, arXiv:2602.16490): the relation between looping and growing a model in depth.

Our increment is similar to theirs. These are real contributions, and ours is of the same kind: we keep the looped MoE and change how its experts are arranged and which parameters each pass owns, at fixed parameters and compute. Small changes can matter, as ResNet's skip connection did (He et al., 2016). When Foil's router is moved towards the expert-sharing design, the loss rises sharply, by 0.011 to 0.018 nat, more than Foil's whole advantage (Table A4).

3. Where the idea comes from matters

Our starting point is not sharing experts. Sharing lets the router of each layer borrow experts from adjacent layers. We instead ask how to arrange the experts of a looped block (more experts per layer, fewer layers, more passes) at fixed parameters and compute; no router borrows from another layer.

The origin of a method differs from the frameworks that can describe it. Deep neural networks came from connectionism, the imitation of biological neurons. They are now also described in statistical or Bayesian terms, as function approximators, or as dynamical systems; but these frameworks are general and describe many other models, and none of them gave rise to deep networks. Likewise, "sharing across depth" describes every looped or weight-tied model, and somewhat ours, but that is a description, not its origin.

4. The ICLR 2027 Reviewer Guide

"Novelty is of course subjective …"

The guide gives criteria:\color{#B31569}{\textbf{The guide gives criteria:}}

  • "Summarized in one sentence, a review aims to determine whether a submission will bring sufficient value to the community and contribute new knowledge."
    • New knowledge. In your words: "a nice parameter- and compute-matched way to improve the expressivity of the model"; "it is nice to have a more strictly controlled study". To our knowledge, no prior work studies how to arrange the experts of a looped MoE at fixed parameters and compute.
    • Sufficient value. Your review notes "consistent improvement"; your reply states: "I am aware of the scale of language modeling loss and that differences of <0.1 nats can be meaningful." At 1.4B parameters and 100B tokens, each Foil model has a lower loss than the baseline, by 0.012 to 0.021 nat (19 to 49 standard errors), and a higher mean accuracy over six downstream tasks, by 0.8 to 1.5 points, at the same parameters and compute.
  • "… Submissions bring value to the ICLR community when they convincingly demonstrate new, relevant, impactful knowledge (incl., empirical, theoretical, for practitioners, etc)." Ours is of this kind, empirical and for practitioners: how to build a looped MoE, from controlled comparisons at three model sizes with standard errors.
  • "Be open-minded: not all work needs to … work in an established theoretical framework … ask yourself whether the results are likely to be interesting or surprising to a broad audience …" Your review calls the idea "nice" and the analysis "informative" and "interesting".
  • The guide's own example of a review that leans to accept also raises a novelty question against a named earlier paper, and still concludes that the paper "could be a significant algorithmic contribution".

We understand that this paper may not feel novel enough to you personally. We hope that, as a reviewer rather than as a highly skilled reader, you could consider it under the ICLR 2027 criteria, under which it brings new knowledge and sufficient value.

AuthorsOpenReview ↗

Thank You for the Response, and My Response to Your Comment (2/3): Figure 1 and the Table of Absolute Values

Figure 1 and the number of distinct experts per token

"My critique here is that the presentation/motivation to increase the number of experts per token is misleading given that flattening in the controlled setting actually decreases the number of experts per token."

We see your point: the panel showed a setting that adds passes and thus compute, whereas in the controlled setting flattening lowers this number. We have removed the panel from Figure 1 and the corresponding observation from the introduction, and the metric no longer appears in the main text (this replaces what our first response said about Section 2.3). For the remaining material there are two solutions: (1) delete it; (2) keep it only as a discussion section in the appendix. The revised PDF implements (2): Appendix M, "Discussion on distinct experts per token" (with Table 31), is in grey to show that it can simply be deleted. We will follow your preference. The motivation does not depend on this metric: it rests on the gain of looping at equal parameters, and on the question of how to arrange the experts at fixed parameters and compute.

The table of absolute values

"My feedback is that the trends were not well communicated by a table of absolute values across many different configs …"

We agree that a table of absolute values does not show the trends well. The main text now shows them with Figure 5 only; the table is in the appendix, and its caption states that it is the numerical record behind Figure 5.

AuthorsOpenReview ↗

Thank You for the Response, and My Response to Your Comment (3/3): Finding 2 and MMR

1. What the statement is

"It sounds like "whether flattening once more or adding passes still brings a gain" does relate to model quality."

Our earlier sentence was too short. MMR does not measure how good a model is. The statement concerns the gain of a further step: once the MMR over the passes shows a peak, further flattening brings little or no gain. It is an association, not a sufficient or necessary condition, nor a causal relation.

2. How it is used

"The prescriptive performance is not convincing."

The peak is read from a trained model (one probe gives the MMR of every pass). Suppose a model flattened to 4 passes shows no peak: flattening further may be worth it. We flatten to 8 passes: the loss still falls, but a peak appears. Flattening to 16 passes is then no longer worth it: the gain will very likely be negative or very small. Our 1.4B models are such a case (Table G1). At 20B tokens, Foil-3 (4 passes) shows no peak; Foil-2 (8 passes) lowers the loss by a further 0.0084 nat and shows a peak; Foil-1 (16 passes) adds only 0.0009 nat, within the noise. At 100B tokens it is the same: +0.0086 nat with a peak, then −0.0001. At the step that produces the peak the loss may still improve, so the loss alone would suggest to keep flattening.

This cannot be proven mathematically and is an association only, but it has practical value: pretraining is expensive, and it helps to avoid runs that very likely bring nothing.

Over the Foil family, flattening lowers the loss clearly in all 7 steps whose result shows no peak (0.003 to 0.012 nat); after the first model with a peak, the next step brings +0.0009 and −0.0001 nat (1.4B) and −0.0015 nat (250M).

Table G1 — Flattening steps in the Foil family: decrease of the loss in each step (nat), and whether the model that the step produces shows a peak

Parameters, tokensBase → Foil-3 (4 passes)Foil-3 → Foil-2 (8 passes)Foil-2 → Foil-1 (16 passes)
554M, 20B+0.0032; Foil-3: no peak+0.0061; Foil-2: no peak−0.0020; Foil-1: first peak
554M, 100B+0.0046; Foil-3: no peak+0.0049; Foil-2: no peak+0.0024; Foil-1: first peak
1.4B, 20B+0.0094; Foil-3: no peak+0.0084; Foil-2: first peak+0.0009; Foil-1: peak (after the first)
1.4B, 100B+0.0124; Foil-3: no peak+0.0086; Foil-2: first peak−0.0001; Foil-1: peak (after the first)
250M, 8B+0.0068; Foil-3: first peak−0.0015; Foil-2: peak (after the first)Foil-1 not trained

3. The number of passes

"The forward looking test may be [confounded] by the simpler number of passes and the performance does not appear significant."

Three facts speak against the number of passes as the explanation:

  • Among the 44 models of Table F2, 3 of the 11 models with 8 passes show no peak, and 4 of the 13 with 4 passes show one.
  • In the same models, 15 of 17 with a peak show no clear gain in the flattening step into them or the next one, against 10 of 27 without. Among models with the same number of passes: 4 of 4 against 4 of 9 at 4 passes, and 7 of 8 against 1 of 3 at 8 passes.
  • For more passes at a fixed layer configuration (Table G2), a doubling of the passes lowers the loss by 0.053 to 0.070 nat while the resulting model shows no peak, and by 0.032 to 0.041 nat in the doubling after which a peak appears: the peak appears where the return to doubling the compute has halved. Here we state an association only, since no model beyond the peak was trained. If the paper is accepted, we will add these models (16 passes, three series) to the camera-ready version, to test whether the peak also anticipates the return of further passes.

Table G2 — More passes at a fixed layer configuration: decrease of the loss for each doubling of the passes (nat)

Experts per layer, layersLoss decrease while the resulting model shows no peakLoss decrease in the doubling after which the model shows a peak
8, 80.064 (1→2 passes); 0.060 (2→4 passes)0.033 (4→8 passes)
8, 40.068 (2→4 passes)0.032 (4→8 passes)
16, 40.070 (2→4 passes)0.041 (4→8 passes)
4, 80.053 (2→4 passes)no model with 8 passes

4. Its status

"It is presented as one of the two main takeaways of the paper, so it is critical that it is clear and well-supported for the paper to be accepted."

This is not a main conclusion of the paper. The main conclusion is that flattening a looped MoE and untying its attention improves the model at the same parameters and nearly the same compute. The two takeaways are findings of our analysis: we have renamed them Finding 1 and Finding 2, and Finding 2 is now worded as an empirical heuristic.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.