acceptodds
Under review as a conference paper at ICLR 2027

Tying the Loop: Tied Expert Layers in Mixture-of-Experts Language Models

Abstract

Mixture-of-Experts (MoE) architectures efficiently scale Large Language Models (LLMs) by activating only a small fraction of their experts per token, yet the full parameter count—dominated by the expert parameters—must be held in training and inference memory. To address this, we systematically study expert tying, an architectural modification that shares expert parameters across consecutive transformer layers while preserving independent, layer-wise routing and attention. We evaluate this approach across common, state-of-the-art architectures, including OLMoE, Qwen3, and DeepSeek-style MoEs. On full-size OLMoE-1B-7B, tying experts halves unique weight storage with virtually unchanged validation loss and similar downstream accuracy after 15.7B training tokens. Controlled ablations identify the value of preserving layer-specific attention under expert sharing. Reinvesting saved parameters in larger shared expert pools lowers validation loss at the baseline parameter budget across all configurations, making expert tying useful for both reducing storage and improving parameter allocation, and providing a more favorable compute-to-memory trade-off.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Looped transformers, LLMs, Mixture of Experts model, MoE

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9DSTOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies expert tying in Mixture-of-Experts (MoE) language models. MoE architectures scale Large Language Models efficiently by activating only a small fraction of experts per token, but the full parameter count—mostly coming from the experts—must still be stored in training and inference memory. To address this issue, the paper systematically investigates an architectural modification that shares expert parameters across consecutive Transformer layers while keeping routing and attention independent at each layer. The approach is evaluated on several common state-of-the-art MoE architectures, including OLMoE, Qwen3, and DeepSeek-style MoEs. On full-size OLMoE-1B-7B, expert tying reduces unique weight storage by half while keeping validation loss almost unchanged and maintaining similar downstream accuracy after 15.7B training tokens.

Critical strengths
  1. The method is naturally motivated by recent loop-based approaches and is very intuitive. The idea of reusing expert parameters across depth is reasonable and straightforward.

  2. The method is validated on several mainstream MoE architectures. The paper includes experiments on OLMoE, Qwen3-MoE, and DeepSeek-style MoE, which makes the empirical evidence more convincing.

  3. The method may provide useful insights for looped Transformers in the MoE setting. Since expert tying can be viewed as a partial form of looping, the paper may be informative for future work on looped MoE Transformers.

Critical weaknesses
  1. The paper does not analyze the difference in inference TPS / serving speed. Although the method brings around 50% memory savings, I am also curious whether it affects inference speed in practice. I would also like the authors to discuss whether this method may potentially affect compatibility with acceleration / serving frameworks such as vLLM or SGLang.

  2. As the authors also acknowledge, the paper does not verify effectiveness on downstream tasks such as code and math.

  3. (Minor) When evaluating DeepSeek-style MoE architectures, the paper does not adopt its aux-loss-free balancing strategy. It would be helpful to clarify why a standard auxiliary balancing loss is used instead, and whether the proposed method remains effective under a more DeepSeek-style training recipe.

Questions for authors

NA

1 comment
AuthorsOpenReview ↗

Response by authors

Inference throughput and serving. On one H100, with a 512-token prompt, batch one and 32 cached decode steps, untied, g=4 and g=4/2× delivered 47.3, 46.7 and 42.6 decode tokens/s, respectively (21.15, 21.42 and 23.45 ms/token). We used native Transformers, float32 weights and BF16 autocast. On one H200 at the same sequence lengths, the largest tested batches that fit were 392, 440 and 424. The next increment of eight failed for each. Plain g=4 gave more batch headroom at similar observed single-stream latency. The expanded pool was slower. The capacity benchmark repeated one prompt across the batch.

For vLLM or SGLang, the loader and any weight-packing or sharding step would need to preserve one physical copy of each shared bank to realize the weight-memory savings. Attention remains layer-specific, so each layer still needs its own KV cache. We verified native cached decoding. Integration and throughput in either serving engine remain untested.

Code, math and expert balancing. Our six main tasks measure multiple-choice accuracy, and Appendix F of the paper also reports evaluations on factual recall. We did not evaluate code generation or mathematical reasoning, as those are harder to evaluate for models of this relatively small size trained from scratch, so the current results cannot establish transfer to either domain. For the DeepSeek-style models, we held the auxiliary balancing objective fixed across tied and untied runs and across families to isolate the sharing change under a common recipe. Per-layer routers could retain their own balancing biases while sharing the experts. Whether the same quality result holds under auxiliary-loss-free training remains open, but as recent quantile balancing methods are known to improve routing stability we are optimistic that those would also help additionally stabilize dynamics in the looping scenario.

Reviewer_AVjyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies parameter sharing on Mixture-of-Experts (MoE) architectures from three perspectives: sharing mode (sharing expert, attention, and router blocks), sharing topologies, and expert granularity. After initial experiments covering all three axes, they propose sharing the parameters of the expert layers across consecutive layers, which are grouped into two or four layers while leaving the first and last two expert layers untied, in addition to router-and attention-related parameters. They evaluate this proposal across several state-of-the-art MoE architectures such as OLMoE, Qwen3, and DeepSeek-style MoEs on a small scale. On a large scale, they evaluate only on the OLMoE family at a 7B total parameter scale with 1B active parameters. Tying expert parameters halves the number of unique parameters while not affecting the validation loss and downstream stream accuracy substantially at 7B/1B experiment. Moreover, they show that reusing one third of the saved parameters to increase the number of experts in the shared pools leads to a lower validation loss and higher downstream performance compared to the untied MoE baseline, outperforming it with using 4.71B parameters instead of 7.12B. The ablation experiments show that tying the attention-related parameters, in addition to the expert parameters, causes a similar performance hit but only saves a fraction of the parameters saved by expert tying, and the cost of tying routing parameters is inconclusive while potentially leading only to minor savings.

Critical strengths

Good ablations of the architecture design, which helps to understand the critical components of the design space
.

Some good baselines that serve as strong reference points to ground the analysis.

Critical weaknesses
  • The large-scale experiments are conducted on a single dataset with a single backbone family. This makes it unclear whether the improvements are robust across datasets, training recipes, longer training budgets, or post-training stages. The remaining experiments are at a small parameter count with a small sequence length limit of 512 tokens. The impact of sharing attention blocks might cost more on longer sequences.

  • The impact of g and expert pool width remains unclear. The small-scale experiments show that g=4, 2 x width achieves lower validation loss than g=2 across three architectures. This shows a pattern that gets better as g increases when supported by increasing the number of experts. But there is no experiment comparing g=4, 2x to g=7, 4x (or an equivalent version that matches the parameters to the g=4,2x version). On the other hand, g=2 seems to be performing better than g=4, 2 x width on downstream tasks while having a worse validation loss, but it is not evaluated on a large scale.

  • You cite MoRE but you don't have a comparison to it.

  • The similarity to the prior work decreases the impact of the findings.

Questions for authors
  • Could the authors extend the experiments involving key hyperparameters like g and expert pool width, which would help address the weaknesses mentioned above?

  • I find the topology notation a bit hard to understand initially, and it is not used consistently; the final g is not used in the initial notation. A small figure might help visualize the notation clearly.

  • Could the authors conduct a statistical test to determine whether differences compared to baseline are significant or not? This would help remove some of the vague claims, such as virtually unchanged etc.

  • Do you ablate the prelude/coda design vs. the MoRE design which does not use a prelude/coda? Im curious if you have any idea the significance of this design choice?

1 comment
AuthorsOpenReview ↗

Response by authors

Group size and pool width. At approximately 372M parameters in OLMoE, g=2 gives loss 3.136 and six-task accuracy 42.1% (see Table 4). Sharing across four layers and doubling the number of candidate experts gives 3.132 and 41.5%. DeepSeek-style and Qwen3-MoE show the same split in rank. The larger-group, wider-pool configurations had lower measured loss, while g=2 scored slightly higher downstream. We have no full-budget result for the larger matched-budget group you suggest or for g=2 at full scale, as it would require significantly more GPU hours.

Relation to MoRE. The concurrent MoRE study shows that adjacent layers can share an enlarged expert pool and improve perplexity at matched parameter and compute budgets. Our fixed-budget expansion results corroborate that finding. For compression, sharing the original-size pools across groups of four reduces OLMoE from 7.12B to 3.50B unique parameters. At 15.73B training tokens, our six-task benchmark rerun gives 47.56% for g=4 and 47.58% for untied. We also test which weights benefit from remaining private. In the Table 2 two-group ablation, untying routers with experts and attention shared reduces top-1 routing agreement from 73.3% to 16.0%. The displayed losses are 3.511 and 3.512, respectively. In our new 2,048-context ablation, untying attention with experts shared and routers private reduces loss by 0.0363. These comparisons identify what to preserve when reducing storage.

Topology. Each cell names the expert bank used at that depth.

Middle layer12345678
§3, two-group example12121212
§4, g=411112222

In §3, banks 1 and 2 alternate for 14 layers, then banks 3 and 4 alternate for 14. Both examples keep attention and routers private.

Do the prelude and coda help relative to sharing throughout the stack? We tested that design choice in two ways. In a depth-32 expert-sharing ablation at 186.1M unique parameters, final loss was 3.4538 with no private expert layers, 3.4632 with one at each end and 3.4689 with two at each end. The model sharing experts throughout had the lowest observed loss, by 0.0151 versus 2+2. Each design has one run. Matching storage changes how often the banks are reused (four, five or seven times in the shared groups), and the corresponding learning-rate divisors change too. This comparison gives us no basis to recommend 2+2 at this budget.

Table 7 in the appendix isolates placement by keeping four private expert layers, the reuse counts and the learning-rate treatment fixed across all five arms. Putting all four inside the stack gave 3.4734, putting two at each end gave 3.4689, and putting one at each end plus two inside gave 3.4667. The full spread was below 0.007, with one run per placement. Together these experiments address the boundary choice. We have no matched head-to-head against MoRE’s complete recipe, which also expands the pool and adds depth conditioning, or a production-scale boundary ablation.

Context length and scope. All Section 4 models trained at context 2,048. We also repeated the Section 3 attention comparison at 2,048, holding expert sharing and private routers fixed over 1.31B training tokens. On identical blocks from the Section 3 training-source evaluation stream, loss was 3.3694 with private attention and 3.4057 with shared attention. The additional sharing penalty was +0.0363 (95% paired evaluation-block interval +0.0340 to +0.0384, one training seed per arm). Full-scale evidence still covers one architecture and one training mixture. Longer contexts, other recipes and post-training transfer remain open.

External evaluation on text published after training. Thanks for this suggestion. To test performance on another corpus, we scored all three final checkpoints on the same 114,632 token positions from 56 US Federal Register documents published on 26–30 October 2026. Cross-entropy was 2.3306 for untied, 2.3198 for g=4 and 2.3066 for g=4 with twice the candidate experts. Relative to untied, the differences were −0.0108 (paired 95% interval [−0.0157, −0.0058]) and −0.0240 ([−0.0291, −0.0189]). The intervals come from resampling documents within this regulatory corpus. The documents postdate training, although they may contain older passages. Each configuration has one training run.

On the separate six-task, three-shot evaluation, the paired accuracy differences versus untied were −0.02 percentage points for g=4 (95% interval −0.87 to +0.84) and +0.56 for the doubled pool (−0.24 to +1.39), resampling examples within each task.

Reviewer_FEp4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper describes detailed study of shareing expert FFN weights across groups of consecutive tranformer layers, while preserving layer specific attention, routers, etc. The paper studies which compotents should be layer specific, how much quality is lost under different tying groups, and whether the saved parameter budget can be reused in a wider shared expert pool.

Some keys results:

  • Attention matters more than routers. With experts shared, keeping per-layer attention gives a clear loss gain; giving each layer its own router changes routing behavior dramatically but hardly affects loss.
  • Sharing experts across layers is nearly free. Groups of 2 cost almost no quality; groups of 4 roughly halve unique parameters for a small penalty.
  • Putting the freed parameters back as a larger shared expert pool beats the baseline at equal parameter count, with no extra expert compute.
Critical strengths
  • The paper presents some real practical value: the reduction in resident weights is direct and does not depend on unstructured sparsity or special compression formats.
  • Ablation is thoughtful. Attention, router, expert, topology, and granularity are separated more carefully than in most architecture papers. Multiple regimes are tested. Fine- and coarse-grained experts, reduced configurations, and one full 7B model all point in the same direction.
  • Authors are genuinely honeset and transparent about the limitations, confounds, and non-significant controls.
  • Systems claims are partly grounded. It reports resident weight memory, peak inference memory, DDP throughput, and a matched-microbatch result rather than only parameter counts.
Critical weaknesses

Do the tied and untied loss curves keep the same slope past 15.7B tokens, or does the gap reopen? Can you run a baseline where each layer routes over an equally large neighboring pool without sharing expert weights, to separate reuse from pool size?

Questions for authors

The iso-parameter comparison gives the tied model a 4x larger candidate pool, so the gain may be a pool-size effect rather than a sharing effect. Can you construct a baseline in which each layer routes over an equally large neighboring pool without sharing expert weights?

1 comment
AuthorsOpenReview ↗

Response by authors

Could the wider pool explain the gain? Yes. Suppose four adjacent layers each own N experts. If each router selects from the union of all four banks, every layer can call the same 4N expert weights. With identical groups and top-k routing, that is our expanded shared pool. A fully untied control with the same choice at every layer would store four independent banks of 4N experts, or 16N across the group. We did not train that control. Sliding or partially overlapping neighborhoods would also be distinct designs.

We did test a related control at the compressed budget of approximately 296M parameters. Each middle layer had 16 private experts, compared with 64 experts shared per four-layer group at the same top-k. Losses for private versus shared were 3.171 versus 3.171 in OLMoE and 3.183 versus 3.180 in DeepSeek-style MoE. Thus a narrower untied model also achieved similar loss at this budget. This control matches storage while allowing the number of candidates per layer to differ.

At approximately the original storage budget (submitted Table 4), the 4× pool lowered measured loss by 0.027–0.030 across the three reduced-scale families while keeping selected-expert FFN computation fixed. We will credit sharing with making the wider pool affordable. The experiment leaves open how much of the gain comes from wider choice versus joint training of experts across depths.

Could the gap reopen? Yes. We could not secure sufficient compute for a longer paired 7B run during rebuttal, as it would require over 1k GPU hours. At 15.73B training tokens, both tied models had lower loss than untied on 56 US Federal Register documents published after training (differences of −0.0108 for g=4 and −0.0240 for g=4/2×). Whether a gap persists beyond that endpoint would have to be studied on larger compute scale, but we note that our goal here is not to beat performance, but to keep performance in a good range while significantly reducing parameters. In our separate six-task benchmark rerun, g=4 scored 47.56% versus 47.58% for untied while reducing unique parameters from 7.12B to 3.50B.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.