- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies expert tying in Mixture-of-Experts (MoE) language models. MoE architectures scale Large Language Models efficiently by activating only a small fraction of experts per token, but the full parameter count—mostly coming from the experts—must still be stored in training and inference memory. To address this issue, the paper systematically investigates an architectural modification that shares expert parameters across consecutive Transformer layers while keeping routing and attention independent at each layer. The approach is evaluated on several common state-of-the-art MoE architectures, including OLMoE, Qwen3, and DeepSeek-style MoEs. On full-size OLMoE-1B-7B, expert tying reduces unique weight storage by half while keeping validation loss almost unchanged and maintaining similar downstream accuracy after 15.7B training tokens.
Critical strengths
-
The method is naturally motivated by recent loop-based approaches and is very intuitive. The idea of reusing expert parameters across depth is reasonable and straightforward.
-
The method is validated on several mainstream MoE architectures. The paper includes experiments on OLMoE, Qwen3-MoE, and DeepSeek-style MoE, which makes the empirical evidence more convincing.
-
The method may provide useful insights for looped Transformers in the MoE setting. Since expert tying can be viewed as a partial form of looping, the paper may be informative for future work on looped MoE Transformers.
Critical weaknesses
-
The paper does not analyze the difference in inference TPS / serving speed. Although the method brings around 50% memory savings, I am also curious whether it affects inference speed in practice. I would also like the authors to discuss whether this method may potentially affect compatibility with acceleration / serving frameworks such as vLLM or SGLang.
-
As the authors also acknowledge, the paper does not verify effectiveness on downstream tasks such as code and math.
-
(Minor) When evaluating DeepSeek-style MoE architectures, the paper does not adopt its aux-loss-free balancing strategy. It would be helpful to clarify why a standard auxiliary balancing loss is used instead, and whether the proposed method remains effective under a more DeepSeek-style training recipe.
Questions for authors
NA
Response by authors
Inference throughput and serving. On one H100, with a 512-token prompt, batch one and 32 cached decode steps, untied, g=4 and g=4/2× delivered 47.3, 46.7 and 42.6 decode tokens/s, respectively (21.15, 21.42 and 23.45 ms/token). We used native Transformers, float32 weights and BF16 autocast. On one H200 at the same sequence lengths, the largest tested batches that fit were 392, 440 and 424. The next increment of eight failed for each. Plain g=4 gave more batch headroom at similar observed single-stream latency. The expanded pool was slower. The capacity benchmark repeated one prompt across the batch.
For vLLM or SGLang, the loader and any weight-packing or sharding step would need to preserve one physical copy of each shared bank to realize the weight-memory savings. Attention remains layer-specific, so each layer still needs its own KV cache. We verified native cached decoding. Integration and throughput in either serving engine remain untested.
Code, math and expert balancing. Our six main tasks measure multiple-choice accuracy, and Appendix F of the paper also reports evaluations on factual recall. We did not evaluate code generation or mathematical reasoning, as those are harder to evaluate for models of this relatively small size trained from scratch, so the current results cannot establish transfer to either domain. For the DeepSeek-style models, we held the auxiliary balancing objective fixed across tied and untied runs and across families to isolate the sharing change under a common recipe. Per-layer routers could retain their own balancing biases while sharing the experts. Whether the same quality result holds under auxiliary-loss-free training remains open, but as recent quantile balancing methods are known to improve routing stability we are optimistic that those would also help additionally stabilize dynamics in the looping scenario.