- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Tiered Expert Embedding (TEE), a trained compression pipeline for fine-grained MoE LLMs. Experts are ranked by routing mass, assigned to tiers with different merge ratios, clustered by functional fingerprints, and merged by activation-weighted averaging. Each original router slot receives a learned input-side embedding before it is mapped to a smaller pool of physical experts. The router and non-expert backbone remain frozen, while embeddings and merged expert weights are optimized by KL distillation. The main experiment reports 5.33x expert compression and 4.07x full-model compression on Qwen3.6-35B-A3B, with an additional study on Nemotron-3.5-Lightning.
Critical strengths
-
TEE targets a concrete and demanding deployment setting. Fitting a fine-grained MoE within 24 GB while keeping the original router interface is practically useful. The reported compression ratio is strong, and the transfer to a substantially different hybrid Mamba–MoE architecture is encouraging.
-
The per-slot input embedding is simple and seems to matter. It adds little storage, while Table 2 shows a large drop when it is removed at the aggressive compression point. I also appreciate that the paper reports throughput and peak memory against a matched trainable-adapter baseline and openly discusses calibration dependence and the failure on Mixtral.
Critical weaknesses
-
The experiments do not isolate the paper's central capacity-slack explanation. About 50% per-input SwiGLU sparsity does not by itself show that colliding experts use complementary active supports or that their merge residual is low-dimensional. The random 2:1 experiment still uses the full trained pipeline. The 48-expert no-embedding ablation shows that input embeddings help within TEE, but it does not validate the proposed co-active-subspace explanation or compare input-side compensation with alternatives under the same optimization budget. Also, under Equation (2), each slot embedding receives gradients from its own routed tokens and router weights, so colliding slots do not obviously receive identical gradients at zero initialization. I think the symmetry claim needs either a derivation or a direct control.
-
It is still hard to tell which structural choices are doing the work and how novel the full method is. At the 48-expert target, there is no equal-budget control with random grouping inside fixed tiers or an approximately uniform allocation. This makes it hard to separate the gains from fingerprint clustering and tiering. The transfer evidence is also mixed: TEE transfers to Nemotron and nearly preserves GSM8K, but the default student drops 5.0 points on IFBench and 3.0/6.7 points on HumanEval pass@1/pass@5. ConMoE is already a close precedent for router-preserving slot-to-pool reassignment, so the new part seems to be the frequency-tiered trained merging and per-slot input embeddings. The comparison should also include the expert-channel-level structured pruning methods HEAPr (ICLR 2026) and CAMERA-P (AAAI 2026) so that the effectiveness and efficiency claims are easier to judge.
Questions for authors
- What evidence from the current runs shows that the merge residuals are low-dimensional or concentrated in a co-active subspace? How much performance is recovered after the embedding-only Phase A, before the merged expert weights are updated?
- Why should colliding slot embeddings receive identical gradients at zero initialization if they see different routed token sets and router weights? Is there a derivation or a zero-initialization control that supports this claim?
- At 48 experts per layer and with the same training budget, what happens with random grouping inside the fixed tiers or an approximately uniform 256-to-48 allocation? Do the gains from tiering and fingerprint clustering remain?
- Which part is new relative to ConMoE and other router-preserving shared-expert representations: the remapping, the frequency tiers, or the per-slot input embedding?
- How does TEE compare in effectiveness and efficiency with the expert-channel-level structured pruning methods HEAPr and CAMERA-P under comparable settings?