acceptodds
Withdrawn from ICLR 2027

Tiered Expert Embedding: Input-Side Expert Embeddings for Tiered Compression of Mixture-of-Experts LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Mixture-of-Experts (MoE) architectures scale model capacity without scaling per-token compute, but their massive expert parameters dominate memory and storage cost. Merging each layer’s experts is the direct way to shrink them: several original experts are merged into one and share its intermediate dimension. This remains viable because of a general property of gated FFNs: for any given input, only about half of an expert’s intermediate units pass the gating. The active subnetworks of merged members therefore rarely saturate the shared weights, and the residual merge error stays low-dimensional enough to be cancelled by a per-slot shift of the expert’s input. We propose Tiered Expert Embedding (TEE), a simple compression framework that (i) merges each layer’s experts with tiered, frequency-aware compression ratios, (ii) freezes the router and remaps original expert slots onto merged experts, and (iii) attaches a tiny per-slot expert embedding w0 on the input side to compensate for merging error. On Qwen3.6-35B-A3B, TEE compresses expert parameters by 5.33× and total parameters by 4.07× while leaving per-token activated parameters unchanged. Even with random expert pairing at a uniform 2:1 ratio, the pipeline loses only 0.8 points on MMLU-Pro: at 2× the capacity slack needs no pairing structure at all. At the full 5.33× ratio the compressed model is comparable to the purpose-built Qwen3.5-9B dense model on MMLU-Pro (69.3 vs. 69.2) while activating 2.6× fewer parameters per token, and outperforms MoLA under matched compression by +8.7 points.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mixture-of-experts, model compression, expert merging, expert embeddings, large language models, knowledge distillation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_UM8MOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Tiered Expert Embedding (TEE), a trained compression pipeline for fine-grained MoE LLMs. Experts are ranked by routing mass, assigned to tiers with different merge ratios, clustered by functional fingerprints, and merged by activation-weighted averaging. Each original router slot receives a learned input-side embedding before it is mapped to a smaller pool of physical experts. The router and non-expert backbone remain frozen, while embeddings and merged expert weights are optimized by KL distillation. The main experiment reports 5.33x expert compression and 4.07x full-model compression on Qwen3.6-35B-A3B, with an additional study on Nemotron-3.5-Lightning.

Critical strengths
  1. TEE targets a concrete and demanding deployment setting. Fitting a fine-grained MoE within 24 GB while keeping the original router interface is practically useful. The reported compression ratio is strong, and the transfer to a substantially different hybrid Mamba–MoE architecture is encouraging.

  2. The per-slot input embedding is simple and seems to matter. It adds little storage, while Table 2 shows a large drop when it is removed at the aggressive compression point. I also appreciate that the paper reports throughput and peak memory against a matched trainable-adapter baseline and openly discusses calibration dependence and the failure on Mixtral.

Critical weaknesses
  1. The experiments do not isolate the paper's central capacity-slack explanation. About 50% per-input SwiGLU sparsity does not by itself show that colliding experts use complementary active supports or that their merge residual is low-dimensional. The random 2:1 experiment still uses the full trained pipeline. The 48-expert no-embedding ablation shows that input embeddings help within TEE, but it does not validate the proposed co-active-subspace explanation or compare input-side compensation with alternatives under the same optimization budget. Also, under Equation (2), each slot embedding receives gradients from its own routed tokens and router weights, so colliding slots do not obviously receive identical gradients at zero initialization. I think the symmetry claim needs either a derivation or a direct control.

  2. It is still hard to tell which structural choices are doing the work and how novel the full method is. At the 48-expert target, there is no equal-budget control with random grouping inside fixed tiers or an approximately uniform allocation. This makes it hard to separate the gains from fingerprint clustering and tiering. The transfer evidence is also mixed: TEE transfers to Nemotron and nearly preserves GSM8K, but the default student drops 5.0 points on IFBench and 3.0/6.7 points on HumanEval pass@1/pass@5. ConMoE is already a close precedent for router-preserving slot-to-pool reassignment, so the new part seems to be the frequency-tiered trained merging and per-slot input embeddings. The comparison should also include the expert-channel-level structured pruning methods HEAPr (ICLR 2026) and CAMERA-P (AAAI 2026) so that the effectiveness and efficiency claims are easier to judge.

Questions for authors
  1. What evidence from the current runs shows that the merge residuals are low-dimensional or concentrated in a co-active subspace? How much performance is recovered after the embedding-only Phase A, before the merged expert weights are updated?
  2. Why should colliding slot embeddings receive identical gradients at zero initialization if they see different routed token sets and router weights? Is there a derivation or a zero-initialization control that supports this claim?
  3. At 48 experts per layer and with the same training budget, what happens with random grouping inside the fixed tiers or an approximately uniform 256-to-48 allocation? Do the gains from tiering and fingerprint clustering remain?
  4. Which part is new relative to ConMoE and other router-preserving shared-expert representations: the remapping, the frequency tiers, or the per-slot input embedding?
  5. How does TEE compare in effectiveness and efficiency with the expert-channel-level structured pruning methods HEAPr and CAMERA-P under comparable settings?
Reviewer_Q1EbOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The manuscript proposes a method to compress MOEs by merging experts into fewer experts. It uses expert activation frequencies to obtain compression ratios and knowledge distillation to recover performance. Experiments show memory savings, while some performance degradation remains.

Critical strengths

The paper discusses the problem of reducing MoE memory requirements which is useful. The method is simple and evaluated on different settings.

Critical weaknesses

1- The motivation of this work relies on the assumption that half of the intermediate units in a gated FFN are not active, allowing multiple experts to share the same capacity. However, to the best of my knowledge SwiGLU uses a smooth activation that does not produce exact zeros. So, it is not clear how the units can be inactive and how multiple experts can share the same capacity without losing information? Probably, this assumption is not valid. On the other hand, maybe the router parameters remain frozen, but expert merging can change hidden representations, which probably effect the routing decisions.

2- As I understand, the first-order error cancellation holds for the weighted average of expert parameters, which does not mean the behaviour of each expert can be preserved after merging. Also, as Eq.2 represent the model uses a vector to recover the differences between experts. I am not sure how this simple input shift can recover such differences, for example when we want to merge 10 experts into one?

Questions for authors

See the weaknesses.

Reviewer_eYoVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TEE compresses MoE models by merging experts while retaining the original router and logical expert slots. It ranks experts by routing mass within each layer and applies different merge ratios to three frequency tiers. Within each tier, it groups experts using output fingerprints and averages their weights. Each original slot receives a learned input shift to recover some of its behavior. The merged model is then trained by distillation. On the main model, TEE reduces the number of physical experts from 256 to 48 per layer while retaining the original top-8 routing.

Critical strengths
  • The method addresses a real deployment constraint: expert weights can exceed available GPU memory even when few experts are active per token. Keeping the original routing interface while sharing expert weights is a clear and useful design choice.

  • The paper includes ablations of input compensation and post-merge training. These results show that both are important at high compression ratios. The reported throughput and memory measurements also help connect the method to deployment.

Critical weaknesses
  • The practical value relative to other low-cost deployment methods is not established. The main compression setting lowers MMLU-Pro from 74.6 to 69.3 and IFBench from 63.7 to 58.7. These are substantial losses. TEE also retains the original number of active slots, so its main benefit is reduced weight storage and traffic rather than fewer expert computations. An important alternative is low-bit weight quantization combined with offloading or caching less frequently used experts. Such a system can retain separate expert weights, although it has its own quality, transfer, and memory costs. The paper should compare these trade-offs under a common GPU memory budget. The reported comparison with MoLA does not establish when merging is preferable to these alternatives. Without quality, latency, and memory results for a strong deployment baseline, it is unclear whether TEE offers a compelling operating point. This is central to the paper's practical importance.

  • The calibration evidence and knowledge evaluation are too limited to support broad use of the merge policy. The default calibration and distillation data come from only 70 MMLU-Pro development questions and related generated content. Repeated sampling adds tokens but does not necessarily add broad topic coverage. In a trained MoE, a narrow input distribution can naturally favor some experts. Low routing mass on that distribution does not establish that an expert is less important for the full model or safe to merge at a high ratio. The paper does not adequately test sensitivity to calibration size and domain. It includes an IFEval-derived calibration variant, but does not report a full cross-task comparison for that variant. This leaves possible domain trade-offs unresolved. The existing math and code tests are useful, but they do not provide broad evidence that expert merging preserves knowledge across subjects and languages. Results on SuperGPQA, MMLU, and MMMLU would be particularly useful.

Questions for authors
  • Under the same GPU memory budget, how does TEE compare with a strong low-bit quantization baseline and with quantization plus expert offloading or caching? Please report quality, decode throughput, prefill latency, and host memory use. What deployment conditions favor TEE after accounting for its adaptation cost?

  • Why are 70 MMLU-Pro development questions sufficient to determine expert tiers and merge groups? How stable are the rankings, groups, and final scores when calibration size, topic mix, and random seed change? Please separate the effect of changing merge statistics from the effect of changing distillation data.

  • Can you report teacher and compressed-model results on broader knowledge tests such as SuperGPQA, MMLU, and MMMLU, using the same default compressed model without task-specific recalibration? For alternative calibration sources, a full cross-task table would help show whether improvements in one domain come at the cost of another.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.