- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
ConMoE splits MoE expert compression into two decoupled decisions: constructing a retained pool, and then, with pool, router and all weights frozen, assigning each removed expert's routed workload to a pool member by minimizing a router-conditioned functional reconstruction objective estimated on routed calibration states, accepted only when two disjoint calibration folds agree; fingerprint retrieval keeps candidate search cheap, and compact execution stores each retained expert once with integer maps while preserving route weights. Validation: three MoE architectures (Qwen3-30B-A3B, DeepSeek-MoE-16B, OLMoE-1B) at 50%/25% retention on six zero-shot benchmarks against pruning (Frequency, REAP), merging (M-SMoE, HC-SMoE) and substitution (MoNE) baselines, plus fixed-pool controlled comparisons with paired bootstrap CIs, held-out reconstruction diagnostics, assignment-rule and calibration-source controls, and peak-memory/throughput measurements on A100s.
Critical strengths
- The fixed-pool decoupling is a genuinely clean causal isolation, and it is selector-agnostic. Every nearest-vs-functional pair shares pool, weights, router and calibration data, so the only free variable is the integer identity→expert map; assignments are frozen on 64 construction prompts before evaluation on 64 disjoint prompts, paired bootstrap CIs exclude zero on all three models (+0.55 / +0.82 / +4.12, macro +1.83), and gains replicate across three calibration seeds.
- All three backbones are real pretrained MoE checkpoints rather than MoEs synthesized from dense models; every score is anchored to the uncompressed checkpoint reported in the same table, so no normalization hop is hidden; evaluation uses full splits via lm-eval-harness at two retention ratios rather than one favorable setting.
Critical weaknesses
- ConMoE removes 45.6–47.5% of unique parameters for a 29.6–48.4% peak-memory reduction, but activated FLOPs per token are unchanged (top-k fixed), so throughput improves only 1.11–1.24×. Weight-only INT8 on the full model reaches the same ~2× with all experts intact, and INT4 reaches ~4×; A memory-reduction method that never faces the dominant equal-bytes alternative has not established that dropping experts is the right way to spend the budget.
Questions for authors
- Why does random substitution beat nearest by 1.90 pp? Please report assignment load concentration (mean removed identities per retained expert, or assignment entropy) for nearest / random / functional.
- Full-model weight-only INT8 (≈ your 45–47% reduction) and INT4 on the same three models and six benchmarks, plus ConMoE ⊕ quantization. What evidence is there that expert dropping is the right way to spend a 2× memory budget?