DirCov: Diagnosing Expert Under-Allocation and Redundancy in Mixture-of-Experts via Directional Coverage

Sparse Mixture-of-Experts (MoE) models activate a fixed top-$k$ subset of experts per token, yet whether this single, layer-agnostic budget matches the coverage each layer actually needs has not been directly measured. We introduce Directional Coverage (DirCov), a lightweight, training-free diagnostic that quantifies how well the top-$k$ expert combination reproduces the direction of the full mixture output, and need${}_k$, the smallest budget a layer requires to reach $95\%$ coverage. DirCov is defined on the data-side geometry of expert outputs under the model's own router. A parameter-space sanity check finds that OLMoE expert weights are near full-rank with near-zero mean pairwise cosine, ruling out simple weight-copy collapse as an explanation for the observed layer profiles. Applying DirCov to seven production MoE families spanning $16$ to $128$ experts, $16$ to $48$ layers, top-$k$ from $2$ to $8$, differing normalisation, and both softmax and sparsemixer routing, we find three consistent phenomena: (i) directional coverage saturates monotonically with $k$; (ii) perplexity is U-shaped in inference $k$, so over-provisioning experts is actively harmful (up to a $555\times$ blow-up); and (iii) $\text{needk}$ is strongly heterogeneous across layers (a depth-driven, universal effect); across five text/math/code domains the per-layer profile is highly stable for most models ($\rho =0.83$–$0.95$) but, in higher-capacity models, splits into a block structure along a natural-language$\leftrightarrow$structured boundary (within-group $\rho$ up to $0.94$ vs. across-group $\rho \approx 0.5$). Because a single global top-$k$ cannot simultaneously satisfy under- and over-provisioned layers, we further show a bidirectional per-layer mismatch. The cross-architecture consistency indicates that fixed-$k$ mismatch is a structural property of current MoE design rather than an artefact of any one model.