acceptodds
Under review as a conference paper at ICLR 2027

Coherent Overlap: Token-Specific Expert Committees in Sparse Mixture-of-Experts Routing

Abstract

Why do sparse mixture-of-experts (MoE) language models route a token to several experts? The standard explanation is geometric complementarity: co-selected experts cover different directions of the token representation. This view has shaped non-overlapping expert designs, diversity objectives, and similarity-based compression, but it has a sharper contextual implication: a selected expert should become more valuable with the experts selected beside it. We test this implication with a prefix-controlled 2×2 factorial that crosses the selected expert with its highest-scoring unselected rival and the actual co-selected context with load-matched alternatives. A positive interaction supports complementarity, whereas a negative interaction supports relevance-driven selection, where the router forms a token-specific committee of individually relevant experts from a shared geometric neighborhood. Across six open MoEs, routed-input subspaces share a large layer-wide core and actual routes fit tokens better than matched alternatives. The selected expert beats its rival in 100 of 102 geometric cells, yet the actual context narrows this advantage in all 102. In OLMoE, Mixtral, and DeepSeek-MoE, the negative interaction persists under order-free contexts, sensitivity analyses, and next-token loss, which fits no subspaces. The overlapping experts remain functionally valuable: their outputs separate more than their inputs, later experts beat unselected substitutes in next-token loss, Top-2 beats Top-1 at matched compute, and explicit complementarity regularization does not consistently improve validation loss. We call this pattern **coherent overlap**: MoE routers select useful expert committees from overlapping neighborhoods, so geometric similarity alone is a poor guide for analyzing, pruning, or designing experts. Code is available in the supplementary material.

Primary area:
interpretability and explainable AI
Keywords:
Mixture of Experts, Sparse Routing, Expert Specialization, Representation Geometry, Subspace Analysis, Mechanistic Interpretability, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_BKvqOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies why MoE models route each token to multiple experts. The authors challenge the common view of geometric complementarity and propose coherent overlap as an alternative explanation. They introduce a candidate-by-context factorial test and conduct experiments across six MoE models. The results show that selected experts have overlapping input subspaces and negative interaction effects, while still providing useful contributions to model predictions.

Critical strengths

(1) This paper provides an interesting and insightful perspective on understanding routing in MoE

(2) The authors have put considerable effort into conducting extensive experiments

Critical weaknesses

(1) Concerns regarding the geometric analysis. The router selects experts based on the input representations and its learned routing weights. The input subspace of each expert is then fitted using the tokens assigned to that expert by the same router. Finally, these router-conditioned subspaces are used to characterize geometric overlap among experts and explain the router's selection behavior. This creates a potentially self-referential relationship between how the expert subspaces are constructed and the routing patterns they are used to analyze.

In other words, The expert input subspaces are derived from router-selected tokens, so the observed geometric patterns may largely reflect the routing process itself rather than provide independent evidence of expert complementarity.

(2) The authors' arguments and supporting experiments is unconvincing to me.

Taking experiments in Section 6 as examples, I would argue that a model's dependence on its original computation path does not necessarily imply that each expert has a distinct function. While the router is not explicitly trained with optimal expert labels, its routing behavior is still shaped by the NLL objective. Therefore, many of the observed NLL changes under expert-patching interventions are largely expected and do not necessarily support the authors' proposed explanation.

Questions for authors

See weaknesses.

Reviewer_WvFJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies why sparse MoE language model route a token to multiple experts. A candidate-by-context factorial compares a selected expert with its highest-scoring eligible unselected rival, after either the actual selected prefix or load-matched alternative prefixes. Across six models, routed-input subspaces overlap and all 102 geometric interaction estimates are negative. Additional next-token NLL, output/Jacobian, matched-compute training, and expert-skipping experiments show that overlapping experts remains predictively useful. The authors interpret these findings as “coherent overlap”: selection of individually relevant experts from a shared neighborhood, rather than the particular contextual complementarity tested by the factorial.

Critical strengths
  1. Figure 3 shows that jointly replacing the candidate and its context can reverse the apparent value of later selected experts. Holding context fixed gives a more interpretable assessment of candidate quality. This supports acceptance because it provides a useful diagnostic principle beyond the individual models studied.

  2. Next-token loss interventions, output/Jacobian comparisons, compute-matched training, and leader-preserving skipping controls go beyond geometric similarity alone. The source-aware splits, paired bootstrap intervals, route replay checks, and explicit disclosure of fitted-basis bias further improve credibility. This empirical contribution would remain valuable under a narrower mechanistic interpretation.

Critical weaknesses
  1. Equation 5 measures how the selected expert’s advantage over a rival changes with context, rather than directly measuring synergy among co-selected experts. A negative interaction may also reflect generic diminishing returns. In Table 21, residual-matched donors cover only 13.4% of token-cells; on this subset, the matched alternative already yields Dnull=−0.0139D_{\mathrm{null}}=-0.0139, compared with D=−0.0196D=-0.0196 for the actual context. In the functional experiments, alternative experts inherit actual rank-specific weights, but the resulting contexts are not necessarily matched in predictive quality or output magnitude. The NLL factorial shows that fitted-subspace bias cannot be the sole explanation, but it does not distinguish the proposed mechanism from effects of context quality or loss curvature. Additional controls or a more qualified mechanistic interpretation would strengthen the argument.

  2. Appendix C.2 shows that fitted bases can reverse the interaction sign for genuinely complementary synthetic experts, making functional validation essential. Yet Table 30 reports only 44/63 negative functional estimates in the newer models, with 15 confidence intervals entirely below zero; both Qwen model-level intervals include zero. The 102/102 negative geometric estimates therefore do not establish equally consistent functional evidence across architectures. Moreover, the matched-compute comparison changes expert count and width simultaneously, the regularization study uses a small model, and the skipping experiments evaluate specific geometric heuristics. These findings support the conclusion that geometric overlap alone is an insufficient redundancy criterion, but do not establish coherent overlap as the cause of the architectural gains or justify broad conclusions about diversity-based methods.

Questions for authors
  1. Can the authors establish theoretical conditions under which complementarity necessarily implies (D>0), rather than treating this as an operational definition? Could a simple predictive model test whether generic diminishing returns can also produce (D<0)? Given the sign reversal demonstrated in Appendix C.2, can the authors further examine the sensitivity of the geometric interaction to subspace rank and estimation sample size?

  2. How much of the negative interaction are specific to actual co-selection rather than generic contextual saturation? Table 21 shows that residual-matched alternatives are available for only 13.4% of token-cells, and the matched-context null already exhibits a negative interaction. Can the authors report the null interaction, additional effect, and matching coverage across models and layers? For the functional experiments, can they further control for context quality, router scores, and activation magnitudes, or evaluate counterfactual routes with renormalized routing weights?

  3. Do the functional results consistently support the proposed mechanism across architectures? Although all 102 geometric interaction estimates are negative, Table 30 report only 44 of 63 negative functional interactions in the three newer models, with 15 confidence intervals entirely below zero. The model-level intervals for Qwen3 and Qwen3.6 also include zero. Can the authors provide higher-power evaluations or additional model-specific analyses to establish whether the negative functional interaction generalizes across architectures, rather than relying primarily on the pooled result?

  4. Do the downstream experiments isolate the benefits of coherent overlap from architectural and compression confounds? The matched-compute Top-1/Top-2 comparison in Figure 5C changes expert count and width simultaneously, making it difficult to attribute the performance difference specifically to multi-expert routing or coherent overlap. Can the authors provide controlled comparisons separating these factors? Additionally, can the expert-skipping experiments include stronger predictive-effect or output-based baselines under matched inference budgets, or clarify that their conclusions concern the tested geometric heuristics rather than optimized pruning and merging methods generally?

Reviewer_zvgTOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper asks whether experts selected together by sparse MoE routers are complementary because they cover different token-relevant representation directions. It proposes a prefix-controlled 2×2 factorial that crosses the selected expert and its highest-scoring unselected rival with the actual selected prefix and load-matched alternative prefixes. Under the paper's operational hypothesis, a positive difference-in-differences indicates that the actual context amplifies the selected expert's relative marginal value; a negative value indicates that it does not. Whether this operational prediction is necessary for complementarity is a central concern in this review.

Across six open MoEs, routed-input subspaces share a substantial layer-wide core, while actual routes fit tokens better than matched alternatives. The selected candidate beats its rival in nearly all geometric cells, but the candidate-by-context interaction is negative in all 102 geometric cells. The negative sign survives order-free variants and sensitivity analyses; a next-token-loss factorial, which does not use fitted subspaces, also gives a negative interaction in the principal models and a negative pooled direction in the newer models. Additional analyses report that overlapping input neighborhoods can produce more separated outputs, that later selected experts can improve next-token prediction, and that geometry-only novelty is not a reliable proxy for removable computation. A matched-compute small-model experiment favors Top-2 over Top-1 in three seeds, while an explicit complementarity objective does not consistently improve validation loss.

Critical strengths
  1. The factorial separates candidate identity from context. Crossing the candidate and context independently avoids conflating candidate quality with the context in route substitutions that change both. This is a useful analytical design, even though the paper does not establish that its predicted sign is a necessary test of complementarity.
  2. The paper examines several distinctions relevant to expert analysis. Its output-space, predictive-value, and skipping analyses illustrate that input overlap alone does not imply functional redundancy. These results provide useful caution for geometry-based pruning, although they do not resolve the validity of the central complementarity test.
Critical weaknesses
  1. The central complementarity conclusion is underdetermined. First, the paper states, “Complementarity is present only if a selected expert becomes more valuable because of the experts selected with it.” This premise is contestable: if the teammates already contribute relevant functions or directions, ss's remaining marginal value may be smaller. Thus a negative interaction does not establish that complementarity is absent. My primary concern is the validity of this inference: the paper does not establish that a positive candidate-by-context interaction is a necessary consequence of geometric complementarity. Replacing geometric coverage with next-token NLL does not resolve this conceptual issue, since both tests rely on the same unestablished implication. Because the distinction between complementarity and coherent overlap is central to the paper’s contribution, this is a fundamental limitation rather than a matter of presentation. Second, the geometric test fits expert subspaces on router-assigned tokens. Even with held-out evaluation tokens, this routing-conditioned fit can make an unselected expert's basis cover the token less well by construction, so the residual-coverage result cannot establish that ss is less valuable under SS than under WW. Finally, the NLL test compares absolute gains under different contexts without explicitly controlling for the available improvement headroom. The actual context may leave less room for further improvement than the alternative contexts, and the paper does not rule out this possibility as an explanation for the negative interaction. Thus neither experiment resolves whether complementarity is present.
  2. The geometric measurement is not faithful to native routing in at least one evaluated architecture, and some auxiliary evidence adds limited novelty. First, the geometric experiment's weight normalization is mismatched with the evaluated model's configuration: the analysis renormalizes each token's Top-k router weights, whereas DeepSeek-MoE-16B-Base specifies norm_topk_prob: false. This changes the token weights used to fit the expert subspaces and may alter the resulting bases and geometric measurements. Second, the matched-compute Top-1 versus Top-2 experiment is used to argue that multi-expert value need not come from separated input directions, but prior MoE work, including the DeepSeek-MoE technical report, already demonstrates benefits from splitting an FFN into experts. This auxiliary experiment adds limited novelty and is not necessary to establish the paper's central result.
Questions for authors
  1. What theoretical or empirical basis supports the claim that geometric complementarity must make the real co-selected context increase the selected expert's marginal value? How would the authors distinguish this prediction from the possibility that teammates already supply some useful function or directions, leaving a smaller residual contribution for the selected expert?
  2. Because each expert's basis is fitted on router-assigned tokens, what control shows that the negative geometric interaction is not caused by the router-conditioned fitting distributions of selected and unselected experts? Would the sign persist if the subspaces were fitted on a common, independently sampled token set or under a route-assignment control?
  3. Can the authors show that the negative NLL interaction persists after accounting for differences in available improvement headroom between the actual and alternative contexts?
  4. The geometric fitting renormalizes Top-k weights, whereas DeepSeek-MoE-16B-Base specifies norm_topk_prob: false. Can the authors recompute the geometric analysis with native weights and report whether the interaction's sign or magnitude changes?
  5. What hypothesis does the matched-compute Top-1 versus Top-2 experiment establish beyond prior demonstrations, including the DeepSeek-MoE technical report, that splitting an FFN across experts can be beneficial? How does that result support the paper's central complementarity claim?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.