acceptodds
Under review as a conference paper at ICLR 2027

Mixture of Universal Experts: Scaling Virtual Width via Depth-Width Transformation

Abstract

Mixture-of-Experts (MoE) decouples model capacity from per-token computation, yet their scalability remains limited by the physical dimensions of depth and width. To overcome this, we propose Mixture of Universal Experts (MOUE), a MoE generalization introducing a novel scaling dimension: Virtual Width. In general, MoUE aims to reuse a universal layer-agnostic expert pool across layers, converting depth into virtual width under a fixed per-token activation budget. However, two challenges remain: a routing path explosion from recursive expert reuse, and a mismatch between the exposure induced by reuse and the conventional load-balancing objectives. We address these with three core components: a Staggered Rotational Topology for structured expert sharing, a Universal Expert Load Balance for depth-aware exposure correction, and a Universal Router with lightweight trajectory state for coherent multi-step routing. Empirically, MoUE consistently outperforms matched MoE baselines by up to 1.3% across scaling regimes, enables progressive conversion of existing MoE checkpoints with up to 4.2% gains, and reveals a new scaling dimension for MoE architectures.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture of Experts, Universal Transformer, Loop Transformer

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_PskfOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MoUE, a method for improving parameter utilization in MoE architectures. Specifically, it introduces a new scaling dimension, Virtual Width, which builds on MoE’s parameter efficiency through cross-layer expert sharing. The authors propose three novel components: Staggered Rotational Topology, Universal Expert Load Balance, and Universal Router. To further enhance scalability, they also describe how pretrained MoE models can be expanded along the virtual-width dimension. This offers a promising approach to continued training, as it can improve parameter utilization without increasing the physical parameter count.

Critical strengths
  1. The MoUE passage proposed gives another dimension to scaling the MoE Model.
  2. Without increasing total parameters or activated parameters, scaling virtual parameters increases model capability with no more devices or inference cost.
  3. The paper proposes a series of supporting facilities inclusive of Staggered Rotational Topology, Universal Expert Load Balance and Universal Router, making MoUE training stable and load balancing.
  4. The passage not only limit to train a MoUE from scrath, proposing curriculum routing warmup method to help trained MoE transfering to MoUE, which is quiet promising in this model explosion era.
Critical weaknesses
  1. The main text is not fully self-contained. For example, π in Eq. (1) and the Sim function used in the dual-pathway routing formulation in Section 4.4 are not clearly defined where they first appear. Although some details are provided in the appendix, these definitions should be included at their points of use in the main text. Additionally, the experimental section introduces depth expansion without sufficiently explaining how it is implemented.
  2. The improvement shown in the experiment is limited: only average +0.7 with double VP(MoUE 36A2/MoUE 18A2) in Table 1 and +1.1 after MoUE adaption in JetMoE. This result shows the risk of cost-inefficiency for MoUE training or adaption.
  3. The experiment is constraint in a small volume model and outdated model architecture.
  4. MoUE introduces cross-layer expert sharing. However, training and inference often employ pipeline parallelism. When shared experts reside on different GPUs or nodes, cross-layer access may incur additional inter-GPU or inter-node communication, potentially reducing training and inference efficiency
Questions for authors
  1. MoUE’s gains over MoE appear modest. In Table 1, approximately doubling the virtual parameter count yields only a 0.7-point improvement. Similarly, in the JetMoE conversion experiments, increasing VP from 8B to 14.6B—nearly twofold—results in only a 1.1-point improvement. Could the authors discuss prior methods that similarly leverage expert parameter reuse and compare their gains under a comparable degree of reuse, such as an approximately twofold increase in virtual parameters?
  2. Compared with MoE, MoUE requires cross-layer expert access, which may introduce additional inter-GPU or inter-node communication under pipeline parallelism. The MoUE architecture also appears to incur additional computational overhead. Could the authors conduct training and inference experiments with pipeline parallelism enabled and shared experts accessed across GPUs or nodes, and compare training and inference throughput against MoE under matched hardware and parallelism settings?
  3. Could the authors provide a more detailed explanation of how depth expansion is implemented?
  4. The MoUE conversion experiments primarily use JetMoE and OLMoE, both released in 2024, whose architectural designs may no longer reflect the state of the art. Could the authors also evaluate MoUE conversion on more recent MoE models, such as Qwen3.5? (I'm delight to increase score if this concern is solved.)
  5. If computational resources permit, could the authors also evaluate MoUE conversion on larger-scale MoE models? (Not Necessary)
Reviewer_HhcHOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes Mixture of Universal Experts (MoUE), an MoE architecture that allows experts to be shared across Transformer layers. The main idea is to increase the effective number of expert compositions, referred to as Virtual Width, without proportionally increasing the number of stored parameters. There are three components: staggered rotational topology for structured expert sharing, universal expert load balance for balancing expert utilization across layres, and a universal router that uses a dynamically updated routing state.

Critical strengths

The paper explores an interesting way to reuse experts across layers while retaining layer-specific specialization. The structured connectivity and load-balancing mechanisms are reasonable design choices, and the ablations suggest they contribute to performance.

The progressive warm start straregy seems to add value to the method overall. The practicality of the approach is even better with the ability of moue to adapt existing MoE model checkpoints.

Critical weaknesses

Section 4.4 updates the routing state using token representations aggregated across the current layer. It is unclear how this prevents information from future tokens from influencing earlier token predictions in subsequent layers. Unless the updates are implemented causally, this could violate autoregressive language modeling and affect the reported perplexity improvements.

Controlled experiments are of utmost importance. The depth-scaling experiments substantially increase activated parameters. For example, Table 1 shows an increase from 0.30B to 0.61B for the 700M backbone. Thus, the improvements cannot be attributed solely to better expert reuse. Comparisons against deeper conventional MoEs and simpler parameter-sharing baselines at matched FLOPs and training budgets are needed.

The motivation for Virtual Width is not fully convincing. The paper emphasizes exponentially many expert paths as evidence of increased capacity. However, conventional multilayer MoEs also have exponentially many possible expert combinations. Moreover, counting repeated access to shared parameters as virtual parameters does not necessarily reflect additional representational capacity. A stronger theoretical or empirical justification is needed.

On literature: Although MoEUT is discussed, there is no experimental comparison despite its relevance to cross-layer parameter sharing. On accuracy-compute trade-offs, it seems important to compare with existing work like MoSE (ICML 2026); this would help establish the practical benefits of Virtual Width.

The contribution of the proposed load balance is not sufficiently isolated. The normalization may partly act as a rescaling of the load-balancing coefficient; the paper should compare against standard and group-wise load balancing with properly tuned coefficients to establish the benefit of the normalization itself.

Efficiency evidence is limited; suppl. reports 9.6% higher training step time and substantially more serialized tensor entries in the large-scale experiment. These results make the claimed efficiency advantages less clear. Additional measurements of inference latency, throughput, GPU memory, and actual FLOPs would strengthen the evaluation.

Statistical evidence: some numbers are relatively small, and the uncertainty estimates mainly come from repeated evaluations of fixed checkpoints rather than independently trained models. More training seeds would benefit the paper by establishing the reliability of the results.

Questions for authors

Please refer to the weaknesses above.

3 comments

Thank you for the additional experiments, particularly the comparison at matched depth and compute and the expanded coefficient sweep. These address my main concerns, so I will raise my rating; I am waiting for the revised paper, which should clearly reflect these.

AuthorsOpenReview ↗

Manuscript Updates Following Your Helpful Feedback

Dear Reviewer HhcH,

Thank you for your follow-up and for recognizing the additional evidence. We have uploaded the revised manuscript, with the revisions marked in blue. Your main concerns are addressed in the following locations:

  1. Matched-depth and matched-compute comparison — Appendix G.1 (pp. 22–23), Table 13 (p. 23). The two L36 configurations share the same physical and activated parameter budgets, training-token budget, and total training FLOPs. Full MoUE improves validation PPL from 24.47 to 23.73 and nine-task average accuracy from 42.9 to 44.0 over the shared-FFN control with layer-wise LBL. Section 5.2 also highlights this comparison and points to the appendix.

  2. Expanded coefficient sweep — Appendix D (pp. 19–20), Table 10 (p. 20). We report all three objectives—layer-wise LBL, grouped balancing without connectivity normalization, and full UELB—at coefficients 5×10−45\times10^{-4}, 10−310^{-3}, and 5×10−35\times10^{-3}. Their best results are 26.37, 25.64, and 25.40 PPL, respectively. Section 4.3 and Appendix D clarify group aggregation, connectivity weighting, and the equal-connectivity case in which the normalization factor can be absorbed into the shared-branch coefficient.

  3. Router state and autoregressive causality — Section 4.4 (p. 5) and Appendix E (p. 20). We now explicitly index the state by sample, token position, and sharing pool, provide the token-wise update equations, and explain the execution order in prefill and decoding. The revised formulation makes clear that updates proceed across depth without reducing over samples or token positions.

We have also added an appendix contents page (p. 15) to make these additions easier to locate. We would be very grateful if you could update your rating after reviewing the revised manuscript. Thank you again for your careful review and constructive suggestions, which have helped us improve the manuscript.

Thanks; updated my score after reviewing the manuscript.

Reviewer_MuzBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Mixture of Universal Experts (MoUE), a generalization of Mixture-of-Experts (MoE) architectures that enables expert reuse across Transformer layers. The central idea is to introduce Virtual Width by allowing multiple layers to access a shared pool of universal experts, thereby increasing the space of possible expert compositions without proportionally increasing the number of physical parameters.

Critical strengths
  1. Interesting and practically relevant architectural design. The idea of enabling cross-layer expert reuse is intuitive and well motivated. The proposed framework offers a flexible way to explore parameter sharing in MoE architectures, while preserving compatibility with conventional MoE models.

  2. Comprehensive experimental evaluation. The authors evaluate MoUE under both width and depth expansion settings, covering multiple model configurations and downstream tasks. The width-expansion experiments are particularly encouraging, showing consistent improvements while keeping the physical and activated parameter budgets unchanged. The ablation studies also provide useful evidence for the effectiveness of the proposed topology and load-balancing strategy.

  3. Practical applicability to existing models. The progressive warm-start strategy is a valuable contribution, as it enables existing MoE checkpoints to benefit from cross-layer expert sharing without requiring training from scratch. The improvements observed on JetMoE and OLMoE further support the practical relevance of the proposed approach.

Critical weaknesses
  1. The interpretation of Virtual Width could be better justified. While the paper motivates Virtual Width through the combinatorial growth of possible expert paths, the relationship between this quantity and effective model capacity remains somewhat unclear. Standard MoE models also admit exponentially many expert-selection paths across layers. Moreover, the proposed virtual parameter count includes repeated accesses to shared parameters rather than independently learned parameters. Additional analysis connecting Virtual Width to actual functional diversity or representational capacity would strengthen this central claim.
  2. The efficiency comparisons could be more carefully controlled. Although the width-expansion experiments maintain fixed activated and total parameter budgets, the depth-expansion configurations increase the number of activated parameters. Consequently, part of the observed improvement may result from additional computation rather than cross-layer reuse alone. Comparisons with deeper MoE models using conventional parameter tying or other shared-layer architectures under matched computational budgets would help isolate the contribution of the proposed design.
  3. The practical computational overhead deserves further investigation. The proposed framework introduces additional routing mechanisms and potentially more complex expert-access patterns. Appendix I reports an approximately 9.6% increase in training step time in a large-scale setting. A more detailed comparison of training throughput, inference latency, and memory usage would help establish the practical efficiency advantages of MoUE. Additionally, reporting variation across independently trained models would make the relatively modest accuracy improvements more convincing.
Questions for authors
  1. Can the authors provide additional evidence that increasing Virtual Width leads to greater effective representational capacity, for example through measurements of expert-path diversity or inter-layer routing correlations?
  2. How does MoUE compare against simpler cross-layer expert-sharing strategies or existing recurrent/shared-layer architectures when matched for actual training FLOPs and activated parameters?
  3. For the Universal Router, could the authors clarify how the online routing state is updated and maintained under causal language modeling? Since the state update aggregates token representations within a layer, it would be helpful to explain how autoregressive causality is preserved.
Reviewer_HLPzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a parameter sharing scheme across layers. Adjacent layers can draw from the same expert pool in their mixture of experts layers. (Other topologies are also possible.)

Critical strengths

-- The method is pretrained on several models at reasonable sizes (160M and 700M with 32B on 140B tokens, respectively) and shows some gains over the baseline.

-- The overall idea of parameter sharing experts between layers is interesting and could be useful.

Critical weaknesses

-- This architecture is essentially a variant of a looped transformer, which would instead reuse the parameters when looping through depth. This is just a different topology of parameter sharing, so it is not completely novel.

-- The paper is misleading about number of tokens used to train, saying 3.4T tokens at some point but then not clearly specifying actual size of the subset of the pool used. From the the data-to-parameter ratio numbers of 200 tokens per parameter (which is fine), you can see that it is much smaller (about

-- The paper creates new words to define already simple concepts, which creates some confusion: for example "Universal Expert Load Balance" is used to define the clear load balancing equation that just normalizes appropriately by the number of experts possibly active. I find the terminology of "virtual parameters" to be especially confusing. The network does not add these as extra parameters, so it feels misleading to call them parameters. Why not call this by the standard name: parameter sharing across layers?

Questions for authors

-- Certain things are not defined, such as "Sim" on line 248.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Feedback, Rebuttal Responses and Evidence (1/2)

We thank all reviewers for their careful assessment of our work and their valuable, constructive suggestions. Their feedback has helped us clarify our contributions, improve the explanation of our mechanisms, and strengthen the experimental and efficiency analyses. Below, we summarize the recognized strengths, main suggestions, our rebuttal responses, and supporting evidence.

Strengths

  1. Architectural design and research value: Cross-layer expert reuse offers a valuable perspective on improving parameter utilization in MoE models. The structured sharing, load-balancing, and routing mechanisms are supported by ablation studies. (Reviewers HLPz, MuzB, HhcH, Pskf)
  2. Experimental coverage and performance gains: The experiments cover width expansion, depth expansion, and multiple downstream tasks, with consistent improvements particularly in width-expansion settings that preserve physical and activated parameter budgets. (Reviewers HLPz, MuzB, Pskf)
  3. Practical checkpoint conversion: The progressive warm-start strategy enables existing MoE checkpoints to benefit from cross-layer expert sharing, supporting the practical applicability of the approach. (Reviewers MuzB, HhcH, Pskf)

Questions and Suggestions

  1. Further clarify Virtual Width, the methodological contributions, notation, and implementation details. (Reviewers HLPz, MuzB, HhcH, Pskf)
  2. Add compute-matched comparisons for expert sharing and depth expansion. (Reviewers MuzB, HhcH, Pskf)
  3. Clarify the Universal Router’s state updates and autoregressive causality. (Reviewers MuzB, HhcH)
  4. Provide coefficient comparisons and normalization ablations for UELB. (Reviewers HLPz, HhcH)
  5. Extend the analysis of training, inference, memory usage, and parallel deployment efficiency. (Reviewers MuzB, HhcH, Pskf)
  6. Evaluate newer and larger models and provide independent training-seed results. (Reviewers MuzB, HhcH, Pskf)

Rebuttal Responses

  1. Methodological contributions and Virtual Width. (HLPz: Q1, Q4; MuzB: Q1; HhcH: Q3, Q4)
    MoUE introduces Virtual Width as an adjustable and measurable scaling dimension based on cross-layer expert accessibility, improving the utilization of existing parameters under fixed physical parameter and activated FFN computation budgets. To realize this opportunity, we combine structured sharing topology, cross-depth load balancing, and trajectory-aware routing to translate expert reuse into practical performance gains. MoUE retains layer-specific attention, normalization, and residual computation, while supporting progressive conversion of existing MoE checkpoints, connecting architectural research with practical applicability.

  2. Experimental budgets, sharing controls, and depth expansion. (HLPz: Q2; MuzB: Q2; Pskf: Q1, Q2; November 13 follow-up to HhcH: Section 3)
    We clarify the notation, training budgets, and implementation of depth expansion: 3.4T refers to the source corpus size, while the 160M and 700M baselines use approximately 32B and 140B training tokens, respectively. Their corresponding width variants preserve physical parameters and training-token budgets. Depth expansion allows additional layers to access existing FFN expert pools while keeping attention, normalization, and residual computation layer-specific. The additional L36 comparison matches training tokens, physical parameters, activated parameters, and total training FLOPs, demonstrating further benefits over basic FFN sharing.

  3. Universal Router causality. (MuzB: Q5; HhcH: Q1)
    UR maintains an independent state for each sample, token position, and sharing pool, updating it along the same token’s depth trajectory. Subsequent layers read the updated state for that token and pool. Vectorized prefill preserves this independence, and each newly decoded token starts from the prescribed initial state. With causal attention, these updates preserve autoregressive causality while using depth-wise routing history to coordinate expert selection.

  4. The role of UELB and coefficient comparisons. (HLPz: Q3; HhcH: Q5; November 13 follow-up to HhcH: Sections 1–2)
    UELB aggregates expert-selection frequencies and routing probabilities across the sharing group and corrects access opportunities according to the connectivity structure, supporting more effective training and utilization of shared experts. In the supplementary coefficient sweep, the best tested shared configuration using standard LBL achieves 26.37 PPL, compared with 25.40 for full UELB. Connectivity normalization derives weights directly from the topology, providing a practical load-balancing rule for different sharing structures.

AuthorsOpenReview ↗

Summary of Feedback, Rebuttal Responses and Evidence (2/2)

  1. Practical efficiency and parallel deployment. (MuzB: Q3; HhcH: Q6; Pskf: Q6)
    We provide additional measurements of training step time, throughput, memory usage, and autoregressive inference, illustrating the quality–efficiency trade-offs across sharing configurations. Small-model training step-time overhead is approximately 2.9% and 3.2%, while two-layer sharing, G2, reduces measured step time by 8.33% on the approximately 7B-A1B backbone. For deployment, we prioritize sharing within each PP partition and align it with EP expert placement to avoid additional cross-stage expert calls. These results highlight the value of jointly selecting sharing configurations and hardware layouts to control runtime costs.

  2. Model scale and result reliability. (Pskf: Q4, Q5; MuzB: Q4; HhcH: Q7)
    We extend the evaluation to 1.25B from-scratch pretraining and conversion of a contemporary hybrid-attention MoE with approximately 23B parameters. The 1.25B model achieves lower validation loss at multiple matched training-token checkpoints. Starting from the same pretrained checkpoint, the approximately 23B model improves training loss through progressive sharing while retaining the original attention architecture and Top-8 activation. Two independent MoUE training seeds also show consistent loss reductions and downstream improvements. Together with the JetMoE and OLMoE conversion results, these findings support the practical value of the method.

Additional Supporting Evidence

  1. Width-expansion gains under fixed budgets. With physical and activated parameters fixed within each series, average accuracy improves from 41.0 to 42.3 for the 160M series and from 43.8 to 44.7 for the 700M series.

  2. Sharing and routing ablations. Ordinary MoE, shared experts with standard LBL, and full MoUE achieve validation PPL of 26.25, 26.37, and 25.40, respectively. Removing the contextual pathway, fast-weight updates, or connectivity normalization yields 25.83, 25.59, and 25.75, respectively, supporting the effectiveness of these components.

  3. Same-depth, matched-budget comparison. At L36 with matched training budgets, full MoUE improves PPL from 24.47 to 23.73 and the nine-task average from 42.9 to 44.0 relative to the MoEUT-style FFN-sharing control.

  4. Diverse expert utilization. A routing probe records 373 distinct cross-layer expert pairs across 68,928 co-occurrence events, with the five most frequent pairs accounting for 19.26%. Together with expert preferences across Math, Code, and Commonsense and layer-wise usage statistics, these observations show differentiated expert utilization across domains and depths.

  5. Larger-scale training and checkpoint conversion. At 40B training tokens, the 1.25B experiment reduces validation loss from 2.428 to 2.409. JetMoE SFT, OLMoE SFT, and OLMoE CPT improve average scores by 1.1, 1.9, and 4.2 percentage points, respectively. On the approximately 23B hybrid-attention model, mean training loss over 406 common training steps decreases from 1.650948 to 1.642122, a 0.535% reduction.

  6. Training and inference efficiency. On the approximately 7B-A1B backbone with PP=1 and EP=8, G2 sharing reduces training step time from 2.062 s to 1.890 s, while G4 takes 2.394 s. In the supplementary inference measurements, TTFT p50 changes from 248 ms to 253 ms, and TPOT p50 from 95.9 ms/token to 98.5 ms/token.

  7. Independent training seeds. Both MoUE seeds achieve lower loss at every reported training checkpoint. At 30B tokens, the baseline and two MoUE seeds have losses of 3.271, 3.257, and 3.256, respectively, and supplementary evaluation-subset averages of 39.01, 39.89, and 39.92.

We sincerely thank all reviewers again for their constructive suggestions. We will incorporate these clarifications and additional results into the revised manuscript. Please feel free to raise any further questions—we welcome continued discussion and look forward to your responses!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.