RDO-MoE: Rate-Distortion Optimized Joint Rank-Sparsity Allocation for Mixture-of-Experts Compression

Sparse mixture-of-experts (MoE) language models store far more parameters than any one token activates; whether one runs on a commodity accelerator therefore hinges on how its expert bank is compressed, not on how sparsely it is routed. Existing pipelines settle that representation in stages, pruning before rating the survivors or scoring the base and the residual ranks under separate objectives, and can thereby miss the configurations favorable only when both are set together. We propose RDO-MoE (Rate–Distortion Optimized MoE compression), which prices shared-base sparsity and expert-residual rank together on one rate–distortion surface under a global serialized-byte budget. Joint per-layer surfaces turn each layer's weights and routing statistics into one finite table of nondominated pairs. Multiplier-indexed allocation dualizes that budget into one Lagrange multiplier and bisects it for a feasible integer allocation over the per-layer frontiers. A capability diagnostic outside the reconstruction objective reports signed per-task differences, not one score. The key insight is that the base and the rank index one surface rather than two budgets; pricing both against a single multiplier thus turns an ordered pair of searches into one query. On Qwen1.5-MoE-A2.7B at 40% compression it reaches 8.91 perplexity and a 55.7 aggregate against 9.16 and 55.4 for the best prior method; on Mixtral-8$\times$7B it reaches 5.49 against 5.88 at the same budget and the lowest perplexity at every budget. On the instance exact search can enumerate it stays within 0.0057 in relative perplexity at 19.6$\times$ the solver speed, while the 0.9- and 1.3-point WinoGrande and BoolQ regressions at 40% stay visible rather than being averaged away by the nine-task aggregate.