acceptodds
Under review as a conference paper at ICLR 2027

Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE

Abstract

As model parameters scale up, sparsely activated Mixture-of-Experts (MoE) approaches have become essential for model scaling. However, mainstream models currently adopt hierarchical architectures, where experts in different layers, although topologically independent, exhibit substantial similarity and redundancy in their stored content. Compared to the hierarchically arranged Local Expert in existing mainstream models, we hypothesize that inter-layer shared Meta Expert hold greater potential. To this end, we conduct a multi-dimensional empirical comparison between Local and Meta Experts, and find that models incorporating Meta Experts exhibit lower parameter redundancy, together with lower cross-entropy loss and higher downstream task accuracy. With the activated quantity and the expert granularity held fixed, our model matches the performance of Vanilla MoE using only 70% of the experts, substantially improving the model's scaling potential. Beyond removing inter-layer redundancy among experts and improving information storage efficiency, our theoretical analysis further shows that, compared with Local Experts, Meta Experts provide hidden states with more flexible residual pathways, thereby improving information retrieval efficiency.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Mixture-of-Experts, Looped Transformer

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XxMAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies cross-layer Meta Expert sharing combined with layer-private Local Experts, varying sharing ratios, activation allocation, boundary layers, and shared sub-pools. It reports improved training loss across model scales and six-task downstream gains for MR75 + Loop2. Reduced-expert configurations match the baseline's training loss with approximately 70–80% of its routed-expert budget.

Critical strengths
  1. Systematic architectural exploration. The study separates sharing proportions, activation allocation, boundary-layer treatment, and sub-pool organization across four training scales, providing useful evidence about sharing topology.
  2. Positive downstream results. After 200B training tokens, MR75 + Loop2 improves all six reported tasks over Vanilla; MMLU rises from 40.87% to 41.75%, and C-Eval from 43.54% to 44.87% (Table 1).
Critical weaknesses
  1. Generalization and efficiency claims exceed the evaluation. The reported loss averages the final ten training steps, without held-out language-modeling loss or perplexity. Table 1 uses equal expert budgets and therefore does not validate downstream equivalence after expert reduction. Equal top-8 activation also does not establish equal total FLOPs when routing candidate pools differ. Missing benchmarks alone do not establish selective reporting.
  2. The mechanism analysis needs validation. Appendix A.2 computes covariance similarity rather than the stated linear CKA, undermining the claimed metric properties. Equations (7)–(8) describe training-update interactions, not direct evidence of improved retrieval efficiency; the SGD-based update interpretation also requires reconciliation with the Muon experiments.
Questions for authors
  1. Beyond the stated capability coverage, how was the six-task suite selected? Does Table 1 include all completed downstream evaluations? If additional tasks were evaluated, please report their complete results.
  2. Are held-out cross-entropy or perplexity and reduced-expert downstream results available? Please specify the evaluation data and protocol, or limit the 70–80% budget claim to training-loss equivalence.
  3. Which similarity formula generated the redundancy curves, and do the conclusions survive correction or justification? What evidence supports improved retrieval beyond shared-parameter training interactions, particularly under Muon?
  4. Do the FLOP estimates include router scoring and selection for each configuration? Please provide full accounting or qualify the comparison as matching executed-expert computation.
9 comments
AuthorsOpenReview ↗

1. The reported loss is effectively a held-out estimate and remains stable across random conditions

The reported loss averages the final ten training steps without held-out loss or perplexity. Are held-out cross-entropy/perplexity and reduced-expert downstream results available?

We clarify that our pretraining corpus is streamed without data repetition and contains abundant unique samples, so each token is seen exactly once during training. The loss over the final training steps is therefore computed on tokens the model has never fitted, making it effectively equivalent to a held-out estimate — a standard practice in LLM pretraining. We would add this clarification to the future revision if possible.

Regarding the stability of the comparison, we have supplemented multi-seed validation showing that the relative variance across seeds is an order of magnitude smaller than our improvement (see Appendix A.8 and the second point of our common response). The downstream results under reduced expert counts are provided in Appendix A.6.

AuthorsOpenReview ↗

2. Meta Expert's parameter savings preserve downstream performance

Table 1 uses equal expert budgets and therefore does not validate downstream equivalence after expert reduction.

We have supplemented long-training experiments with reduced expert counts: Meta Expert matches the baseline with 37.5% fewer experts in downstream evaluation, while the Vanilla MoE degrades significantly under the same reduction. Please refer to the second point of our common response to all reviewers, or to Appendix A.6 for details.

AuthorsOpenReview ↗

3. The redundancy curves come from a valid PSD kernel, the mislabeling does not affect the conclusion

Appendix A.2 computes covariance similarity rather than the stated linear CKA. Which similarity formula generated the redundancy curves, and do the conclusions survive correction?

We thank the reviewer for this careful observation. The kernel that generated all redundancy curves is the cosine similarity between vectorized centered second-moment matrices, Kij=cos(vec(Yic⊤Yic),vec(Yjc⊤Yjc)) , as defined in Eqs. (9)–(10).

We acknowledge that we incorrectly referred to this kernel as "linear CKA". The confusion arose from an equivalent formulation of linear CKA as the cosine similarity between vectorized Gram matrices: in that equivalence, the Gram matrices are the sample-space Grams YY⊤ , whereas our feature map uses the feature-space second moment Y⊤Y . Since both are second-moment statistics with identical vec-cosine forms, and since our kernel happens to share linear CKA's invariance to orthogonal transformations and isotropic scaling, we mistakenly identified it as linear CKA.

Importantly, this is purely a naming error rather than a computational one. The kernel is a valid positive semi-definite similarity measure, all reported curves and redundancy statistics are exactly the outputs of Eqs. (9)–(10), and therefore no result or conclusion is affected. In the revision, we have renamed the kernel accordingly and removed the incorrect CKA reference. We have additionally verified that recomputing the redundancy curves with standard linear CKA (Kornblith et al., 2019) yields the same qualitative conclusions.

Reviewer_F51TOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates cross-layer expert sharing in MoE models, combining layer-private Local Experts with shared Meta Experts. It studies sharing ratios and topologies, evaluates parameter efficiency across model scales, and reports downstream results for 6B-A1.6B models. The authors also provide a shared-gradient analysis to interpret Meta Experts as bidirectional residual pathways and explain their potential benefits for information storage and retrieval.

Critical strengths
  • The paper systematically explores practical sharing choices, including Meta/Local ratios, layer-specific access, and expert-pool partitioning. Controlling the expert budget and per-layer activation count makes these comparisons informative.

  • The parameter-reduction experiments suggest that shared experts can match the baseline’s training loss with fewer stored experts. This is a potentially useful direction for improving MoE parameter efficiency.

Critical weaknesses
  • The theoretical analysis describes how shared parameters couple layers, but does not sufficiently explain why this coupling should improve performance. Gradient accumulation and the first-order cross-layer influence establish a possible interaction, not that it is beneficial or improves retrieval. As Appendix A.4 acknowledges, shallow layers do not access current-pass deep activations. The “bidirectional residual pathways” interpretation therefore needs stronger empirical support beyond the consequences of parameter sharing itself.

  • The downstream gains are modest and their statistical reliability is unclear. For example, MR75 + Loop2 improves MMLU from 40.87 to 41.75 and TriviaQA from 19.24 to 19.32, while the combined configuration underperforms Vanilla on MMLU and C-Eval. Without multiple training seeds or uncertainty estimates, it is difficult to determine which improvements are robust rather than run-to-run variation.

  • The strongest parameter-saving claim is primarily supported by training loss, measured over the final ten training steps. The downstream comparison instead uses the same total expert budget. Consequently, the paper does not yet establish that the reported 20–30% reduction in experts preserves downstream capabilities, limiting the practical significance of this result.

Questions for authors
  • Can controlled interventions on cross-layer gradient contributions show that the proposed interaction actually causes the gains, rather than alternative effects such as regularization or differences in expert utilization?

  • Could you report multiple-seed results for Vanilla and the main Meta-Expert configurations, including uncertainty in downstream scores?

  • Do the reduced-expert models retain their advantages on held-out language-modeling loss and downstream tasks, rather than only matching training loss?

5 comments
AuthorsOpenReview ↗

1. Meta Experts are activated across many layers, forming Bidirectional Residual Pathways

The theoretical analysis shows that shared parameters couple layers, but not why this coupling should be beneficial; the "bidirectional residual pathways" interpretation needs stronger empirical support. Can controlled interventions on cross-layer gradient contributions show that this interaction actually causes the gains, rather than alternatives such as regularization or differences in expert utilization?

To provide the most direct evidence for the activation pattern and the bidirectional residual pathways, we have supplemented heatmaps of Meta Expert activations across layers, presented in the first point of our common response to all reviewers (see Appendix A.5).

As shown, the vast majority of Meta Experts are activated across many layers — an activation pattern markedly different from conventional architectures. While conventional architectures can only activate layer-private experts, our model can also activate experts that would otherwise be allocated to other layers. In other words, our model effectively enables hidden states near the output layers to activate experts near the input layers, which can be regarded as a form of reverse residual connection.

Meanwhile, since these experts are activated by multiple layers during the forward pass, they naturally receive gradients from the hidden states of different layers during backpropagation; this follows directly from the shared-parameter design.

Certainly, although the bidirectional residual pathways are evident, we do not rule out other effects that Meta Expert may introduce, such as the regularization mentioned by the reviewer. This paper offers the most direct explanation from the authors' perspective, and we welcome future work exploring alternative explanations for why Meta Expert is effective.

Notably, this interpretation has also been recognized by Reviewer uD3V, who commented that "explaining cross-layer expert sharing as a 'bidirectional residual pathway' gives a nice mathematical perspective on why sharing parameters across depth helps information retrieval." We believe this indicates that, although offered from one perspective, our explanation provides an intuitive and mathematically grounded view of how cross-layer parameter sharing facilitates information retrieval.

To be honest, it is difficult to completely explain the mechanism behind Meta Expert within a single paper, and we hope this perspective does not detract from the contributions of this work. But it is really a worth-research point, and we would like to leave it as one of our future works.

AuthorsOpenReview ↗

2. The gains of Meta Expert are stably reproducible across random seeds

The downstream gains are modest and their statistical reliability is unclear (e.g., MR75 + Loop2 improves MMLU from 40.87 to 41.75, while the combined configuration underperforms Vanilla on MMLU and C-Eval). Could you report multiple-seed results with uncertainty estimates?

We have supplemented multi-seed experiments, which show that the relative variance across seeds is an order of magnitude smaller than our improvement, confirming that our conclusions are stably reproducible (see Appendix A.8). The reduced-expert experiments further corroborate the robustness of our conclusions (see Appendix A.6). Please also refer to the second point of our common response to all reviewers for more details.

AuthorsOpenReview ↗

3. The parameter savings of Meta Expert hold on downstream tasks, not merely training loss

The parameter-saving claim is primarily supported by training loss over the final ten steps; it is unclear whether the reported 20–30% expert reduction preserves downstream capabilities. Do the reduced-expert models retain their advantages on held-out loss and downstream tasks?

We have supplemented long-training experiments with reduced expert counts to address exactly this concern (see Appendix A.6 and the second point of our common response). The results show that the reduced-expert models retain their advantages beyond training loss: Meta Expert matches the baseline with approximately 30% fewer experts in terms of loss, and with 37.5% fewer experts in downstream evaluation. In contrast, the Vanilla MoE degrades significantly under the same reduction. These results confirm that the parameter-saving advantage of Meta Expert extends to downstream capabilities.

Reviewer_uD3VOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes "Meta Expert," a new architecture for Mixture-of-Experts (MoE) models that shares experts across different layers instead of keeping them private to each layer (Local Experts) in Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE. The authors test different ratios of Meta and Local Experts, as well as different sharing structures (such as keeping only Local Experts in the first and last two layers, and grouping Meta Experts into two loops). Under the same compute budget (Iso-FLOPs), models with Meta Experts reduce parameter redundancy and achieve similar loss using only 70%–80% of the experts compared to standard MoE. The authors also provide a theoretical analysis showing that shared experts act like a two-way pathway between shallow and deep layers.

Critical strengths

The ablation studies show that the shallowest and deepest layers need layer-private experts (F2L2), while middle layers benefit from sharing. This matches our intuition about how Transformers process information.

Using linear CKA and the Gini coefficient to measure how much experts overlap is a clean and effective evaluation method.

Explaining cross-layer expert sharing as a "bidirectional residual pathway" gives a nice mathematical perspective on why sharing parameters across depth helps information retrieval.

Critical weaknesses

A globally shared expert pool is much larger than a single layer's pool. The paper does not report actual training throughput (tokens/sec), inference latency, or communication overhead (like All-to-All cost in Expert Parallelism).

In Section 4, the authors claim that shallow layers read information written by deep layers. However, there is no routing heatmap or statistic showing whether the same Meta Experts are actually activated across different layers, or if each expert just specializes in one specific layer during training.

Figure 3(a) shows two different ways to combine the Local router and Meta router, but the main text does not clearly explain which one is actually used in the final experiments. In Table 1, the best loss model (MR75 + F2L2 + Loop2) actually performs slightly worse than the Vanilla baseline on MMLU and C-Eval. Also, adding Loop2 lowers the loss but increases expert redundancy, which is not fully explained.

Questions for authors

Can you provide a heatmap or statistics of the routing distribution across layers? Are individual Meta Experts truly activated across many different depths, or do they mostly stick to specific layers?What is the actual training speed (tokens/second) and memory usage of MR75 compared to Vanilla MoE when using Expert Parallelism (EP=8EP=8)?In Figure 3(a), which routing design is used for the main experiments, and how do you balance the weights between Local and Meta experts?

5 comments
AuthorsOpenReview ↗

1. Clarification on Efficiency

The paper does not report training throughput or communication overhead (e.g., All-to-All cost in Expert Parallelism); what are the actual training speed and memory usage of MR75 compared to Vanilla MoE under EP=8?

We have supplemented speed benchmarks on D12–D24 (see Appendix A.7). Although introducing Meta Expert incurs additional communication overhead, the end-to-end token throughput is only marginally affected, remaining at approximately 85%–95% of the Vanilla model (i.e., a 5%–15% slowdown). We speculate this is because the computation of Meta Experts and Local Experts can be overlapped with the communication of both: beyond each expert type hiding its own communication latency, the computation and communication of Meta Experts can also be mutually overlapped with those of Local Experts. Memory usage measurements are also reported in Appendix A.7.

Regarding inference efficiency, we were unable to adapt our implementation to inference frameworks within the rebuttal period, as such adaptation involves substantial framework-specific engineering with many confounding factors (e.g., kernel support, batching and caching strategies, and serving configurations). We therefore report training throughput only. That said, these results partially reflect inference efficiency as well: the prefill stage of inference shares the same forward computation and EP communication pattern as training, so the mild overhead observed in training is expected to largely carry over to prefill. We leave a systematic inference benchmark, together with dedicated inference-side optimizations, to future work.

It is worth noting that this work primarily focuses on the effectiveness of Meta Expert, and we have not applied dedicated efficiency optimizations. With thorough infrastructure optimization in the future, the efficiency of this design can be further improved.

AuthorsOpenReview ↗

2. Meta Experts are repeatedly activated across layers

There is no routing heatmap or statistics showing whether the same Meta Experts are activated across different layers, or whether each expert specializes in a single layer.

Please refer to the first point of our common response to all reviewers; the experimental results are provided in Appendix A.5.

AuthorsOpenReview ↗

3. Clarification on Figure 3(a)

Figure 3(a) shows two ways to combine the Local router and Meta router, but which design is used in the main experiments, and how are the weights balanced?

We use separate routers for Local Experts and Meta Experts. Specifically, the Local Experts in each layer have a layer-private router, while each group of Meta Experts shares a global router. For each token, the final MoE output is computed by directly weighting the outputs of the selected experts with the routing weights from the two routers; we do not introduce any additional weighting coefficients between the two routers.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their careful and detailed suggestions. Below, we summarize the common concerns raised across reviews to facilitate easy reference for all reviewers, the AC, SAC, and PC.

Revision

We have uploaded the revised PDF, with all changes marked in blue

Meta Experts are repeatedly activated across different layers

The visualizations and discussion have been added to Appendix A.5 of the revised manuscript.

Reviewers uD3V, F51T, and XxMA all expressed interest in the activation behavior of Meta Experts, i.e., whether Meta Experts exhibit clear layer-wise activation patterns like Local Experts. This relates both to the actual function of these experts during the forward pass and to our theoretical analysis of the bidirectional residual connection.

To address this, we provide visualizations of the actual activation patterns (see Appendix A.5). The figures show that the vast majority of experts are sufficiently activated across many layers; Only a small number of experts are activated relatively sparsely, yet still in more than a single layer. This observation supports our analysis of the source of Meta Expert's gains.

Since Meta Expert provides the model with a more flexible information retrieval pathway (hidden states at both shallow and deep layers can access more experts, which in conventional layer-wise models would be scattered across layers), we have revised the description from "Retrieval-Efficient" to "Retrieval-Flexible" in the revision to characterize the role of Meta Expert more precisely.

Meta Expert's gains are stably reproducible and robust to expert reduction

These results and discussions have been added to Appendices A.8 and A.6 of the revised manuscript, respectively.

Reviewers F51T and XxMA both raised concerns about the downstream performance of Meta Expert: whether its loss advantage can be stably reproduced across multiple seeds, and whether it can maintain stable downstream performance after parameter reduction. First, we added multi-seed experiments (see Appendix A.8). The relative variance across seeds is an order of magnitude smaller than our improvement, confirming that the loss advantage of Meta Expert is stably reproducible.

Second, we supplemented long-training experiments with reduced expert counts (see Appendix A.6). In terms of loss, Meta Expert matches the baseline with approximately 30% fewer experts; In downstream evaluation, it still matches the Vanilla baseline with 37.5% fewer experts — The downstream performance is even more tolerant of expert reduction than the loss results suggest. The same phenomenon does not hold for Vanilla MoE: when its expert count is reduced by the same amount, its downstream performance degrades significantly.

It is worth noting that the parameter-reduction experiments provide stronger evidence than the multi-seed experiments. On one hand, different expert counts naturally introduce different sources of randomness. On the other hand, the fact that comparable downstream performance is reproduced even with fewer experts indicates that our conclusions are robust to the expert-pool size rather than artifacts of a particular configuration.

For your convenience, we also reproduce the results of our reduced-expert experiments (Tab. 6) directly in this comment:

100% of Experts62.5% of Experts
DatasetVanillaOurs│VanillaOurs
MMLU0.40870.4175│0.37790.4175
C-Eval0.43540.4487│0.41460.4421
CMMLU0.42030.4344│0.38620.4351
TriviaQA0.19240.1932│0.17410.1827
BBH0.29720.3030│0.28970.3009
OpenBookQA0.35000.3640│0.35000.3380
Hellaswag0.53850.5459│0.51730.5164
PIQA0.72360.7252│0.72030.7122
WinoGrand0.55410.5604│0.53280.5556
SimpleQA0.41240.4762│0.44480.4452
Average0.43330.4469│0.42080.4346

It is also worth noting that for pretraining, the loss is a more stable, lower-variance metric than downstream accuracy. With the training data fully controlled to be identical, the improvements we report are already highly significant. Moreover, improving downstream performance at the pretraining stage is inherently difficult when the recipe is fully controlled: as a reference, removing 37.5% of the experts from the Vanilla baseline only reduces its average score from 0.4333 to 0.4208 — notably, the average gain brought by Meta Expert (0.4333 → 0.4469) is even larger than the degradation caused by such a substantial parameter reduction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.