- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Very interesting paper trying to claim that looping through more layers of moe will perform better nd there are more experts been selected. It's more like an experimental paper trying to argue that more experts are working better.
Critical strengths
The paper's idea is new and the equation representation is clear. The paper did experiments from small number of parameters all the way to 100B. And the paper came with the clear conclusion that their method can outperform base.
Critical weaknesses
I don't see the idea been apply to current production ready models. i.e. I don't know how much the results still hold after pretraining. I don't see scientific proof about the claim or at least some theory insights. The paper goes into an idea directly which I have no clue where it comes from.
Questions for authors
If the claim is more experts can work better why put them into more layers instead of increasing topk? e.g. if you used top8 what if you use top16, will it perform better?
Can we apply these ideas to production ready models e.g. qwen, deepseek or kimi and see how the idea holds?
6 comments
Rebuttal Part 2: Response to Weaknesses 2 and 3
Weakness 2, On scientific proof and theory
"I don't see scientific proof about the claim or at least some theory insights."
We respectfully disagree that our paper offers no scientific proof. We are not sure how you define "scientific proof", so we answer under the two interpretations we can think of.
- A pure and complete mathematical derivation, i.e. error bounds or convergence guarantees, as papers on optimal control prove stability. The question here is not whether we have done it, but:
- Whether it can be done. To our knowledge, no existing theory can derive the loss of a trained language model from its architecture.
- Whether it is necessary. In the field of LLMs, conclusions about architecture are not established by such derivations. Complete, standardised experimental validation, supported by formal derivation where it helps, is considered sufficient; the technical reports of today's leading open models, such as Qwen3, DeepSeek-V3 and Kimi K3, all proceed in this way.
- Experimental validation, as in the technical reports of the models you mention. There, architectural choices are likewise established by experiments; none of these reports proves by mathematical derivation that an architectural choice improves the model. The DeepSeek-V3 report, for example, states: "We investigate a Multi-Token Prediction (MTP) objective and prove it beneficial to model performance"; this proof is experimental. Our approach is the same, and we provide both:
- Controlled experiments. Base and the Foil models have the same expert parameters, expert compute per token and effective depth (layers times passes), and differ only in how the experts are arranged, so a difference in loss can only come from the arrangement. Every Foil has a lower loss than Base in all twelve comparisons (554M and 1.4B parameters; 20B and 100B tokens); after 100B tokens, each difference is 6.7 to 49 times its standard error (Figure 3, Table 1 and Table 7 of the revised PDF).
- Formal derivation. Section 2 and Appendix A state the looped block formally and derive what flattening holds fixed (the experts, the expert calls per token, the effective depth) and what it enlarges (the pool of every routing decision, the equivalent expert count), with lower and upper bounds on the distinct experts a token reaches. Appendix A also shows that, at the same shape, a model with shared attention is a special case of one with an attention per pass.
We therefore believe that we provide sufficient "scientific proof", and that our methodology is aligned with that of the technical reports of the leading open models.
Weakness 3, On where the idea comes from
"The paper goes into an idea directly which I have no clue where it comes from."
This idea has a clear lineage in prior research; it simply has no direct precedent, and this is exactly where its novelty lies.
First, we believe the question "where it comes from" does not necessarily apply to a new design:
- You also write in the strengths that "The paper's idea is new". A new architectural design usually has no ready-made source; this is what makes it new.
- For an architectural design, the more decisive question is whether it works under a controlled comparison. Our experiments show that it does (
Weakness 2).
Second, our idea does build on lines of prior research (Table C1):
- Sparse MoE. Each layer holds many experts and every token activates a few of them (Switch Transformer, ST-MoE), in some designs with shared experts that are active for every token (DeepSeekMoE).
- Looped models and looped MoE. A looped model applies one block of layers several times (Universal Transformer, Huginn); MoEUT and LoopMoE apply looping to MoE models.
Building on these, our Introduction derives our question, and the first design choice, flattening the experts, from three observations (Figure 1, left). The second design choice, untying the attention, comes from our own experiments: with attention shared across passes, flattening discards the attention parameters of the layers it removes, and the gain stopped after the first step (Figure 4); giving every pass its own attention keeps these parameters at no extra compute, and the gain continues.
Rebuttal Part 3: Response to Questions 1 and 2
Question 1
"If the claim is more experts can work better why put them into more layers instead of increasing topk? e.g. if you used top8 what if you use top16, will it perform better?"
There is a misunderstanding here, and we are glad to clarify each point:
- Our claim is not that more experts work better. Adding experts adds parameters, against the premise of this paper that the parameters are held fixed, so it would not be a fair comparison. We only change the shape of the expert layers: all four models of our controlled comparison have the same 64 experts in the looped block.
- We do not put the experts into more layers. We do the opposite: fewer layers, each holding more experts.
- We do not use top-8. All models in this paper use top-2 routing.
- A larger top-k adds compute, so it is not a fair comparison. It activates more parameters per token, whereas our controlled experiments keep the compute and the parameters the same. Even if it improved the model, the gain would be bought with compute rather than come from the architecture design. A related experiment shows this: with one more active expert per token and layer, the loss falls, but the total compute of the model rises by 20% to 23% (the compute of the expert layers by 50%); at that higher compute, Foil-2 still has a lower loss than Base (
Table D1). - At fixed compute, letting every routing decision choose from more experts is exactly what flattening does: the pool grows from 8 experts in Base to 64 in Foil-1, while 2 are still selected each time.
Question 2
"Can we apply these ideas to production ready models e.g. qwen, deepseek or kimi and see how the idea holds?"
We are glad to answer this question:
- Our model's major components already derives from the architecture of a production-ready model. The MoE configuration of our Base model, 8 experts per layer with top-2 routing, is that of Mixtral (Jiang et al., 2024), a production model of Mistral AI.
- Our method is built on the generic Transformer architecture, so in principle it carries over. Its MLP experts and attention modules are those of the generic Transformer, the cornerstone of modern LLMs; it does not rely on any specific component such as gated delta-rule attention or multi-head latent attention.
- To address your concern, we added experiments. If your question refers to pretraining an open MoE model at the scale of Kimi or Qwen, we are afraid that this is not realistic for an academic study: the pretraining of DeepSeek-V3 alone took 2.66M GPU hours, according to its technical report. We nevertheless ran the closest experiments within our reach: we brought four core designs of these models into ours, namely (i) the attention of Qwen3, (ii) the hybrid attention of Qwen3.5, (iii) the always-active shared expert of DeepSeek-V3 and Kimi, and (iv) the auxiliary-loss-free load balancing of DeepSeek-V3, and pretrained Base and Foil-2 with each of them. Flattening remains effective every time: Foil-2 has a lower loss than Base, by 0.003 to 0.017 nat (
Table A1).
Rebuttal Part 1: Summary and Response to Weakness 1
Summary
We thank Reviewer T9yJ for the review, and for noting that the idea is new and that our method outperforms the baseline. We believe several of the concerns come from misunderstandings, which we clarify below. We provide the following clarifications, new experiments and revisions:
Table A1,Table A2).Table C1).Table D1).All corresponding changes have been added to the revised PDF and are marked in dark purple, and the changes that respond to several reviewers are marked in blue.
We number the sentences under "Critical Weaknesses" as Weakness 1 to 3. Responses:
Rebuttal Parts 1–3; tables:Rebuttal Parts 4–5.Weakness 1, On production-ready models and on the results after pretraining
We believe it is entirely natural that this idea does not yet appear in current production models, and that this in fact speaks for our method:
Nevertheless, to address your concern, we added experiments. We applied the core designs of these models to ours: the attention of Qwen3 (grouped-query attention with normalised queries and keys), the hybrid attention of Qwen3.5 (gated full attention with gated delta-rule linear attention), the always-active shared expert of DeepSeek-V3 and Kimi, and the auxiliary-loss-free load balancing of DeepSeek-V3. With each, we pretrained Base and Foil-2, a representative flattened model: with the same design and the same number of parameters, Foil-2 has a lower loss than Base in every case (
Table A1; the Qwen results are also in Appendix K, Table 30 of the revised PDF). On downstream tasks after pretraining, without further training, Foil still scores above the unflattened baseline (Table A2).