- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Directional Coverage (), a training-free, single-pass diagnostic for Mixture-of-Experts (MoE) layers. For a given token and layer , measures the clamped cosine similarity between the re-normalized top- expert mixture output and the counterfactual full-mixture output where all experts contribute under the router's full gate distribution. From this, the authors define as the minimum expert budget required by layer .
Applying across seven open-weight MoE architectures (OLMoE-1B-7B, Qwen3-30B-A3B, Qwen1.5-MoE-A2.7B, Phi-3.5-MoE, DeepSeek-MoE-16B, ERNIE-4.5-21B-A3B, and DeepSeek-V2-Lite) and five domains (WikiText, SQuAD v2, GSM8K, HumanEval, MBPP), the paper reports:
- A weight-space sanity check on OLMoE showing that flattened down_proj expert weights have near-unit normalized spectral effective rank ( ) and near-zero mean pairwise cosine, ruling out trivial weight copy collapse.
- Monotone saturation of with , paired with a U-shaped perplexity (and GSM8K accuracy) curve where activating experts beyond degrades performance—up to a 555x perplexity blow-up on Phi-3.5-MoE.
- Strong depth heterogeneity in , which is stable across domains for OLMoE, Phi-3.5-MoE, and ERNIE (–) but splits into a natural-language vs. structured (math/code) block pattern in higher-capacity fine-grained models such as Qwen3-30B-A3B and DeepSeek-MoE-16B ().
- A budget reallocation experiment (Section 4.7) showing that reallocating per-layer expert counts via a -guided rule at fixed total compute performs identically to a uniform per-layer budget on OLMoE and Qwen3-30B-A3B.
Critical strengths
- Broad empirical characterization of inference- sensitivity across seven production MoEs: Sweeping inference top- across seven diverse MoE families (16–128 experts, 16–48 layers, softmax vs. sparsemixer routing, with and without shared experts) provides a clear empirical demonstration of how severe over-provisioning degradation is on both language modeling perplexity and GSM8K exact match.
- Identification of domain-family block structure in fine-grained MoE layers: The five-domain rank correlation analysis (Table 1, Figure 14) reveals an interesting distinction between coarse/smaller MoEs—whose depth profiles remain domain-invariant (–)—and higher-capacity fine-grained MoEs (Qwen3-30B-A3B, DeepSeek-MoE-16B), where depth profiles shift systematically between natural language and structured math/code inputs ().
Critical weaknesses
- Conceptual flaw in treating the dense all-expert output as the ground-truth directional target: The diagnostic defines a layer as "under-provisioned" whenever fails to reach a 0.95 cosine similarity with the full all-expert mixture . Yet the paper's own experiments in Sections 4.2, 4.7, and 4.8 demonstrate that is heavily corrupted: because the model was trained with hard top- routing, activating the remaining unchosen experts injects residual-stream noise, causing perplexity to blow up by 2.5x–555x and crashing OLMoE's GSM8K accuracy from 40.0% at to 0.5% at . In early layers where router probability distributions are higher-entropy (less peaked), the sum of the low-probability tail experts pulls away from , inflating to 32 or 60. Measuring how many experts are needed to reconstruct untrained tail-expert noise does not measure how many experts a layer actually needs for accurate computation—which directly explains why reallocating compute according to in Section 4.7 yields zero improvement over uniform allocation.
- Direct internal contradiction between the text/diagrams and the data tables regarding "over-provisioned layers" and "bidirectional mismatch": A headline claim in the Abstract, Introduction, Figure 1, and Section 4.4 is that a single global causes a bidirectional mismatch within the same network by under-provisioning early layers () while simultaneously over-provisioning mid-to-late layers (). Specifically, Section 4.4 states: "For OLMoE at , layers are under-provisioned and layers are over-provisioned", and Figure 1 illustrates blue bars with labeled "over-provisioned ()". However, looking at the actual data in Figure 4, Figure 5, Table 4, and Table 13, OLMoE's on WikiText is 32 for layers 0–7, 16 for layers 8–9, 8 for layers 10–14, and 32 for layer 15. Thus, at , layers 10–14 are matched (, plotted as grey matched bars in Figure 5), and 0 layers are over-provisioned (zero green bars in Figure 5)! Section 4.7 even contradicts Section 4.4 in prose, noting: "no layer is truly over-provisioned relative to ". Even more concerning, Table 13 shows that across all seven models, five models have #over = 0 and the other two (DeepSeek-MoE-16B and DeepSeek-V2-Lite) have #under = 0—meaning not a single model in Table 13 has both under-provisioned and over-provisioned layers at its deployed .
Questions for authors
- Clarification on the contradiction between Section 4.4 / Figure 1 and Figure 5 / Table 13: How do you reconcile the claim in Section 4.4 and Figure 1 that OLMoE has over-provisioned layers () in layers 10–14 with Figure 4, Figure 5, Section 4.7, Table 4, and Table 13, which all show in layers 10–14 and #over = 0? Furthermore, since Table 13 reports either #over = 0 or #under = 0 for every single one of the seven models, how is the claim of a simultaneous "bidirectional per-layer mismatch" relative to supported by your data?
- Clarification on the -sweep grid for Qwen1.5-MoE: In Table 3 and Table 6, Qwen1.5-MoE-A2.7B () only takes values in across all layers and domains, whereas Table 12 shows that were evaluated for sequence-averaged (reaching at ). Were excluded from the candidate grid when computing per-layer in Table 6, or did every single layer of Qwen1.5-MoE fall below at ?
- Clarification on correlation with router entropy: Since expert outputs are nearly orthogonal, how closely does per-layer (and ) correlate with the simple cumulative router gate mass or router Shannon entropy?