acceptodds
Under review as a conference paper at ICLR 2027

DirCov: Diagnosing Expert Under-Allocation and Redundancy in Mixture-of-Experts via Directional Coverage

Abstract

Sparse Mixture-of-Experts (MoE) models activate a fixed top- subset of experts per token, yet whether this single, layer-agnostic budget matches the coverage each layer actually needs has not been directly measured. We introduce Directional Coverage (DirCov), a lightweight, training-free diagnostic that quantifies how well the top- expert combination reproduces the direction of the full mixture output, and need, the smallest budget a layer requires to reach coverage. DirCov is defined on the data-side geometry of expert outputs under the model's own router. A parameter-space sanity check finds that OLMoE expert weights are near full-rank with near-zero mean pairwise cosine, ruling out simple weight-copy collapse as an explanation for the observed layer profiles. Applying DirCov to seven production MoE families spanning to experts, to layers, top- from to , differing normalisation, and both softmax and sparsemixer routing, we find three consistent phenomena: (i) directional coverage saturates monotonically with ; (ii) perplexity is U-shaped in inference , so over-provisioning experts is actively harmful (up to a blow-up); and (iii) is strongly heterogeneous across layers (a depth-driven, universal effect); across five text/math/code domains the per-layer profile is highly stable for most models (–) but, in higher-capacity models, splits into a block structure along a natural-languagestructured boundary (within-group up to vs. across-group ). Because a single global top- cannot simultaneously satisfy under- and over-provisioned layers, we further show a bidirectional per-layer mismatch. The cross-architecture consistency indicates that fixed- mismatch is a structural property of current MoE design rather than an artefact of any one model.

Primary area:
interpretability and explainable AI
Keywords:
mixture-of-experts, sparse routing, expert allocation, model diagnostics, directional coverage

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_ULuSOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Directional Coverage (DirCov(k)\text{DirCov}(k)), a training-free, single-pass diagnostic for Mixture-of-Experts (MoE) layers. For a given token tt and layer ℓ\ell, DirCov(k)=Et[cos⁡+(yk(t),yfull(t))]\text{DirCov}(k) = \mathbb{E}_t[\cos^+(y_k(t), y_{\text{full}}(t))] measures the clamped cosine similarity between the re-normalized top-kk expert mixture output yk(t)y_k(t) and the counterfactual full-mixture output yfull(t)=∑e=1Eg~t,efe(xt)y_{\text{full}}(t) = \sum_{e=1}^E \tilde{g}_{t,e} f_e(x_t) where all EE experts contribute under the router's full gate distribution. From this, the authors define needk(ℓ)=min⁡{k:DirCovℓ(k)≥0.95}\text{need}_k(\ell) = \min\{k : \text{DirCov}_\ell(k) \ge 0.95\} as the minimum expert budget required by layer ℓ\ell.

Applying DirCov\text{DirCov} across seven open-weight MoE architectures (OLMoE-1B-7B, Qwen3-30B-A3B, Qwen1.5-MoE-A2.7B, Phi-3.5-MoE, DeepSeek-MoE-16B, ERNIE-4.5-21B-A3B, and DeepSeek-V2-Lite) and five domains (WikiText, SQuAD v2, GSM8K, HumanEval, MBPP), the paper reports:

  • A weight-space sanity check on OLMoE showing that flattened down_proj expert weights have near-unit normalized spectral effective rank ([0.999980,0.999984][0.999980, 0.999984] ) and near-zero mean pairwise cosine, ruling out trivial weight copy collapse.
  • Monotone saturation of DirCov(k)\text{DirCov}(k) with kk, paired with a U-shaped perplexity (and GSM8K accuracy) curve where activating experts beyond k0k_0 degrades performance—up to a 555x perplexity blow-up on Phi-3.5-MoE.
  • Strong depth heterogeneity in needk(ℓ)\text{need}_k(\ell), which is stable across domains for OLMoE, Phi-3.5-MoE, and ERNIE (ρ=0.83\rho = 0.83–0.950.95) but splits into a natural-language vs. structured (math/code) block pattern in higher-capacity fine-grained models such as Qwen3-30B-A3B and DeepSeek-MoE-16B (p=0.005p=0.005).
  • A budget reallocation experiment (Section 4.7) showing that reallocating per-layer expert counts via a needk\text{need}_k-guided rule at fixed total compute BB performs identically to a uniform per-layer budget on OLMoE and Qwen3-30B-A3B.
Critical strengths
  • Broad empirical characterization of inference-kk sensitivity across seven production MoEs: Sweeping inference top-kk across seven diverse MoE families (16–128 experts, 16–48 layers, softmax vs. sparsemixer routing, with and without shared experts) provides a clear empirical demonstration of how severe over-provisioning degradation is on both language modeling perplexity and GSM8K exact match.
  • Identification of domain-family block structure in fine-grained MoE layers: The five-domain rank correlation analysis (Table 1, Figure 14) reveals an interesting distinction between coarse/smaller MoEs—whose depth profiles remain domain-invariant (ρ=0.83\rho = 0.83–0.950.95)—and higher-capacity fine-grained MoEs (Qwen3-30B-A3B, DeepSeek-MoE-16B), where depth profiles shift systematically between natural language and structured math/code inputs (p=0.005p=0.005).
Critical weaknesses
  • Conceptual flaw in treating the dense all-expert output yfully_{\text{full}} as the ground-truth directional target: The diagnostic defines a layer as "under-provisioned" whenever yk(t)y_k(t) fails to reach a 0.95 cosine similarity with the full all-expert mixture yfull(t)=∑e=1Eg~t,efe(xt)y_{\text{full}}(t) = \sum_{e=1}^E \tilde{g}_{t,e} f_e(x_t). Yet the paper's own experiments in Sections 4.2, 4.7, and 4.8 demonstrate that yfull(t)y_{\text{full}}(t) is heavily corrupted: because the model was trained with hard top-k0k_0 routing, activating the remaining E−k0E - k_0 unchosen experts injects residual-stream noise, causing perplexity to blow up by 2.5x–555x and crashing OLMoE's GSM8K accuracy from 40.0% at k=8k=8 to 0.5% at k=64k=64. In early layers where router probability distributions g~t\tilde{g}_t are higher-entropy (less peaked), the sum of the E−k0E - k_0 low-probability tail experts pulls yfull(t)y_{\text{full}}(t) away from yk0(t)y_{k_0}(t), inflating needk\text{need}_k to 32 or 60. Measuring how many experts are needed to reconstruct untrained tail-expert noise does not measure how many experts a layer actually needs for accurate computation—which directly explains why reallocating compute according to needk\text{need}_k in Section 4.7 yields zero improvement over uniform allocation.
  • Direct internal contradiction between the text/diagrams and the data tables regarding "over-provisioned layers" and "bidirectional mismatch": A headline claim in the Abstract, Introduction, Figure 1, and Section 4.4 is that a single global k0k_0 causes a bidirectional mismatch within the same network by under-provisioning early layers (needk>k0\text{need}_k > k_0) while simultaneously over-provisioning mid-to-late layers (needk<k0\text{need}_k < k_0). Specifically, Section 4.4 states: "For OLMoE at k0=8k_0=8, layers {0,1,2,3,4,5}\{0,1,2,3,4,5\} are under-provisioned and layers {10,11,12,13,14}\{10,11,12,13,14\} are over-provisioned", and Figure 1 illustrates blue bars with needk<8\text{need}_k < 8 labeled "over-provisioned (need<k\text{need} < k)". However, looking at the actual data in Figure 4, Figure 5, Table 4, and Table 13, OLMoE's needk\text{need}_k on WikiText is 32 for layers 0–7, 16 for layers 8–9, 8 for layers 10–14, and 32 for layer 15. Thus, at k0=8k_0=8, layers 10–14 are matched (needk=8\text{need}_k = 8, plotted as grey matched bars in Figure 5), and 0 layers are over-provisioned (zero green bars in Figure 5)! Section 4.7 even contradicts Section 4.4 in prose, noting: "no layer is truly over-provisioned relative to k0k_0". Even more concerning, Table 13 shows that across all seven models, five models have #over = 0 and the other two (DeepSeek-MoE-16B and DeepSeek-V2-Lite) have #under = 0—meaning not a single model in Table 13 has both under-provisioned and over-provisioned layers at its deployed k0k_0.
Questions for authors
  • Clarification on the contradiction between Section 4.4 / Figure 1 and Figure 5 / Table 13: How do you reconcile the claim in Section 4.4 and Figure 1 that OLMoE has over-provisioned layers (needk<8\text{need}_k < 8) in layers 10–14 with Figure 4, Figure 5, Section 4.7, Table 4, and Table 13, which all show needk=8\text{need}_k = 8 in layers 10–14 and #over = 0? Furthermore, since Table 13 reports either #over = 0 or #under = 0 for every single one of the seven models, how is the claim of a simultaneous "bidirectional per-layer mismatch" relative to k0k_0 supported by your data?
  • Clarification on the kk-sweep grid for Qwen1.5-MoE: In Table 3 and Table 6, Qwen1.5-MoE-A2.7B (k0=4k_0=4) only takes values in {8,16,60}\{8, 16, 60\} across all layers and domains, whereas Table 12 shows that k=1,2,4k=1, 2, 4 were evaluated for sequence-averaged DirCov\text{DirCov} (reaching 0.9070.907 at k=4k=4). Were k∈{1,2,4}k \in \{1, 2, 4\} excluded from the candidate grid when computing per-layer needk\text{need}_k in Table 6, or did every single layer of Qwen1.5-MoE fall below 0.950.95 at k=4k=4?
  • Clarification on correlation with router entropy: Since expert outputs are nearly orthogonal, how closely does per-layer DirCov(k0)\text{DirCov}(k_0) (and needk\text{need}_k) correlate with the simple cumulative router gate mass ∑e∈Sk0(t)g~t,e\sum_{e \in S_{k_0}(t)} \tilde{g}_{t,e} or router Shannon entropy?
Reviewer_wS7GOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1(−1)
Soundness
1
Read review
Paper summary

This paper introduces DirCov and its derived need_k to measure per-layer expert coverage, and reports heterogeneous need_k profiles across seven MoE models and five domains. However, its claimed universal bidirectional mismatch is contradicted by its own results, in which most models have no over-provisioned layers at the deployed k_0.

The paper reports too many contradictory results and unreliable statistical measures. Please refer to the Weaknesses and Questions below.

Critical strengths

The paper asks a useful, underexplored output-side question.

Critical weaknesses
  1. The central 'bidirectional mismatch' claim is contradicted by Table 13. Five models have no over-provisioned layers, while the two DeepSeek models have no under-provisioned layers. None of the seven models exhibits both simultaneously.

  2. Table 4 gives needk=8=k0need_k=8=k_0 for layers 10–14, so they are matched, not over-provisioned. Table 13 reports 11/5/0 under/matched/over, while Section 4.7 explicitly states that no OLMoE layer is truly over-provisioned, contradicting Section 4.4 and Figures 5/13.

  3. Too many results are inconsistent. The claimed 2–3× full-activation degradation is only about 1.02–1.03× for Qwen1.5 and both DeepSeek models. Table 1 disagrees with Table 10, and Figure 10 disagrees with Table 3.

4/ The 0.95 threshold is arbitrary and lacks sensitivity analysis. The coarse k-grid also means reported (need_k) values are only interval estimates. For example, Qwen3’s value 128 may represent any true threshold from 33 to 128.

  1. Inference-only k-sweeps cannot establish claims about training-time redundancy or alternative per-layer budgets. They show that changing k in an already-trained model can be harmful

  2. There is a textbook case of pseudoreplication in section 4.3. The author fail to recognise the dependence among pairwise correlations and present a descriptive trend as a statistically significant finding. In Sec4.3 and Table 10, they compute (52)=10\binom{5}{2}=10 pairwise Spearman correlations from only five domains, split them into four within-group pairs and six across-group pairs, and then apply a one-sided Mann–Whitney U test, obtaining p=1/(104)=1/210≈0.005p=1/\binom{10}{4}=1/210\approx0.005. The reported p=0.005 is simply an artefact of applying the Mann–Whitney test to 10 correlations derived from five domain profiles. These correlations are mutually dependent, each domain appears in four of them, which violates the independence assumption of the test. Under a label-permutation (Mantel) test appropriate for this design, there are only (52)=10\binom{5}{2}=10 possible assignments of domains to the NL/structured groups, so the smallest attainable one-sided p-value is 0.1. The claimed significance of the block structure in Qwen3 and DeepSeek-MoE therefore does not hold as stated.

  3. Table 2 bootstraps over layers, although layers are fixed, depth-dependent, and non-independent. These intervals do not measure uncertainty over evaluation data; perplexity differences are reported without data-level uncertainty.

  4. The Qwen3 (k=128) GSM8K result uses only 125/200 completed examples without explaining the missing 75. More broadly, the paper provides few details necessary for reproduction.

  5. Fig13 and Fig5 are even identical. The presentation is very poor.

Questions for authors
  1. Table 13 shows that no evaluated model contains both under- and over-provisioned layers, and OLMoE is 11/5/0 rather than the bidirectional pattern shown in Figures 5/13. Which results are correct, and how can the paper’s central “universal bidirectional mismatch” claim be maintained?

  2. How exactly are y_k and y_{full} implemented for each architecture, particularly for shared experts, SparseMixer, correction-biased routing, and routers that do not renormalize selected weights? Please provide architecture-specific equations or code.

  3. Section 4.2 claims that going from the optimum to full activation multiplies perplexity by 2–3× even for the softmax-routed models. This is a misrepresentation of the paper's own data: in Table 2, the authors report an over-provisioning blow-up of "1.0×" for three of the six non-Phi models (and only 1.4× for ERNIE). Specifically, Table 11 shows that the ratio from best to full activation is only ~1.02× for Qwen1.5 (15.02 vs. 14.73), ~1.03× for DeepSeek-MoE (14.97 vs. 14.56), and ~1.02× for DeepSeek-V2-Lite (15.42 vs. 15.18). Even worse, for DeepSeek-V2-Lite, full activation (k=64, PPL 15.42) does not degrade performance relative to the deployed budget (k0=6, PPL 15.79) at all—it actually improves it! How can the authors justify making a sweeping "2–3× blow-up" claim in the main text when half of the evaluated models show virtually zero degradation (1.0× in their own Table 2) and one even improves over its deployed k0 at full activation?

Reviewer_Mk9LOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces DirCov, a measure of how closely the direction of the top-k expert output matches that of the full expert mixture. It defines needkneed_k as the smallest expert budget that reaches a coverage score of 0.95. The authors study seven MoE model families across five data domains to examine expert budgets, differences across layers, and task performance.

Critical strengths
  1. The analysis is easy to understand and covers a broad set of models. The paper studies expert budgets through their combined outputs, adding a useful view beyond routing frequency and load balance. Experiments across seven model families and five domains provide useful observations. The authors also report that changing layer budgets based on the measure does not consistently improve performance.
Critical weaknesses
  1. The link between coverage and actual expert needs is not established. The full expert mixture can itself have poor task performance, so matching its output does not always mean better predictions. The current experiments do not show that low or high coverage can reliably identify layers with too few or too many experts for the task.
  2. The main claim of mismatch in both directions conflicts with the figures and tables. Section 4.4 describes OLMoE layers 10-14 as having too many experts. However, Figure 5 and Table 13 show that these layers have needk=k0=8need_k=k_0=8, which means their budgets are matched. Table 13 reports 11 layers with too few experts, 5 matched layers, and no layers with too many experts. This does not support the claim that this model has both types of mismatch.
Questions for authors
  1. Please clarify the conflict between the text and Figure 5 and Table 13. Can experiments that change kk in one layer at a time show whether needkneed_k predicts the actual loss changes from adding or removing experts?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.