acceptodds
Under review as a conference paper at ICLR 2027

On the Additivity of Expert Contributions in Mixture-of-Experts Models

Abstract

Sparse Mixture-of-Experts (MoE) language models activate a small subset of experts for each token, yet how strongly these experts interact in their effects on the loss remains less well understood. First, we evaluate six open-weight MoE language models on MMLU and compare the loss changes caused by removing experts individually and in pairs within the same layer. We find that removing two experts together changes the loss by approximately the sum of the changes caused by removing each separately, so the effects of experts on the loss are nearly additive. Second, we examine how deviations from this additive relation, which we call interaction, vary with domain overlap and layer depth. Pairs with greater domain overlap exhibit larger absolute interaction when their individual effects on the loss are comparable, while absolute interaction generally decreases in later layers. Finally, we study a single-layer MoE with squared loss, where interaction equals the inner product between expert contributions and vanishes when their outputs are orthogonal. We examine this connection on data generated by orthogonal linear teachers. Training from Gaussian initialization produces near-orthogonal expert outputs and reduces interaction relative to the loss changes caused by removing experts individually. We further prove that training started near the teacher solution recovers it at an exponential rate, yielding orthogonal outputs and vanishing interaction. These results establish this recovery as a sufficient mechanism for additivity in the controlled model.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture of Experts, Expert Interactions, Local Convergence

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kEJ3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how routed experts interact within the same MoE layer. The authors measure this by zeroing out the output of one or two experts. They keep the other routing weights fixed. They define interaction as the loss change from removing both experts minus the sum of individual loss changes.

The authors test every expert pair in six open weight MoE models. They use MMLU subjects as evaluation domains. The analysis yields three main findings.

First, the effects are nearly additive. Removing two experts costs about the sum of removing them individually. The regression slopes are between 0.97 and 1.00. Second, domain overlap increases interaction. Experts active in the same domains interact more. Generalist experts interact more than specialists. Third, interactions weaken in deeper layers. They are lowest in the final MoE layer.

The second part uses a simplified model. It has one MoE layer with softmax routing and linear experts. The inputs are Gaussian mixtures. The teachers are orthogonal and linear. The loss is squared error. Here, interaction is exactly the routing weighted inner product of expert outputs. SGD reduces both normalized interaction and output cosine similarity. The authors also prove local convergence to a teacher equivalent solution.

Critical strengths

S1. A large scale analysis with consistent results.

The empirical study is extensive. It covers all routed expert pairs across every MoE layer in six models. The knockout intervention is simple and clearly defined. The near additive pattern appears consistently across models. It also holds under two different loss definitions.

This is the main strength of the paper. The finding is highly relevant to expert pruning. Many pruning methods estimate expert importance individually. They assume the effects of removing multiple experts are additive. The results show this approximation works well at the aggregate level.

S2. Sound theoretical analysis with careful limitations.

The proofs for the main lemmas appear correct. This includes the case of rank deficient teachers. The authors are also very clear about what their results do not prove. They acknowledge that orthogonality does not imply additivity under cross entropy. They note the theorem does not cover the experimental setting. They also admit that absolute interaction increases during training.

These details make the claims easier to assess. The authors avoid overstating their theoretical results. The mathematical arguments are solid despite the gap between the simplified model and real LLMs.

Critical weaknesses

W1. The empirical results lack a proper baseline and overstate additivity.

This is my main concern. The large scale knockout analysis is the central contribution. However, it has three major issues.

First, the paper lacks a proper empirical baseline. Removing one expert from six is a very small perturbation. Single knockout effects are around 0.001 nats per token. Approximate additivity is expected for such small changes. A second order Taylor expansion predicts this. The key question is whether expert perturbations are more additive than other perturbations of the same size. The paper does not test this. I request a clear comparison against random directions with matched norms. You could also compare against groups of neurons in a dense network. Without this baseline, we do not know if the results are specific to MoE structure.

Second, the headline regression results hide the true pair level interactions. Figure 1 averages signed effects across domains. Positive and negative interactions cancel out. Figure 2 tells a different story. The binned median of absolute interaction is often 10 to 50 percent of the single knockout effect. This ratio is even higher for small effects. This contradicts the narrative of perfect additivity. Individual interactions are actually quite substantial. The paper needs to reconcile this framing with the data in Figure 2. Reporting normalized interaction for the LLMs would help clarify this.

Third, the domain overlap result is likely just coactivation. If two experts are never active in the same prompt, their interaction is exactly zero. Domain overlap just measures the chance for experts to interact. The current analysis does not prove that overlap explains anything beyond simple coactivation frequency.

W2. The simplified model does not explain the LLM results.

Section 4 tries to explain why additivity emerges. The connection to the empirical findings remains speculative. I see three main problems.

First, the theoretical result relies heavily on model assumptions. Interaction reduces to an inner product because of the squared loss. Orthogonality is just an assumption. The local convergence rate depends on unquantified variables. The math is fine, but it does not explain why real LLMs should behave this way.

Second, the proposed mechanism is never tested in the LLMs. The authors admit the theorem does not apply to deep networks. They do not check if expert output geometry relates to interaction in real models. Figure 5 offers weak support. The observed cosine similarities are only slightly above chance level. This does not prove the outputs are orthogonal enough to explain additivity.

Third, the toy experiment does not isolate output geometry. Normalized interaction is already very small at initialization. The model is nearly additive before training even starts. The routing factor is never measured. We do not know if routing changes or output geometry changes cause the interaction drop. The experiment fails to track convergence to the teacher.

Questions for authors
  1. What are the normalized interaction values for each LLM? Please break this down by layer and domain. Also, is the R squared in Figure 1 centered or uncentered? Figure 2 shows substantial absolute interactions. This might require qualifying the claim of near additivity.
  2. What numerical precision was used for the knockout experiments? How did you estimate the noise floor? Absolute interaction averages absolute values. Numerical noise can inflate this estimate for small effects. A noise floor analysis is necessary.
  3. How do the routing weights and distance to the teacher evolve during training in the toy experiment? Randomly placed centers would make routing nearly one hot. This alone could suppress interactions. Tracking these quantities would separate routing effects from output alignment effects.
  4. What is the random direction baseline for Figure 5 at each actual hidden size? Please also report the cosine similarity of expert pairs that are not co selected. This will show if the output geometry is truly unusual.
Reviewer_KF6NOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies experts interact in their effects on Sparse Mixture models. The paper found two points: (1) removing two experts together changes the loss by approximately the sum of the changes caused by removing each separately; (2) pairs with greater overlap in their activation profiles exhibit larger absolute interaction at comparable single-knockout effect sizes.

Critical strengths
  1. Understanding experts interaction in SMoE is an important question
  2. The finding of removing two experts together changes the loss by approximately the sum of the changes caused by removing each separately is interesting
Critical weaknesses
  1. Analyzing a pair of experts without considering TopK selected experts might lead to misleading results since dropping unselected experts might not lead to any losses.
  2. Why and when the phenomenon emerged is still unclear. Of the six analyzed MoE language models, there are pretraining models and post-training models (Qwen3-30B-A3B, Gemma-4-26B-A4B), which also include both language models and vision language models (Gemma-4-26B-A4B).
  3. If the metric is language-modeling loss, why does the paper not choose training from scratch model? If the paper chooses a downstream task (MMLU), why does the paper not choose downstream metrics (eg: accuracy)? What’s about the phenomenon if the paper uses pretraining LLMs datasets?
  4. Qwen2, Qwen3 and Gemma4 (Figure 1) seem not to follow a linear pattern, how to explain those models?
  5. Finding 3 “training can produce additive expert effects in a single-layer MoE” can not represent the MoE LLMs model due to model size and datasize are not aligned. Why does the paper not train real MoE-based LLMs from scratch to observe the phenomenon?
  6. The application of the findings is unclear. The author should relate the findings with real MoE-based LLMs applications such as pruning methods.
Questions for authors

Please refer to above weaknesses.

1 comment

Dear Reviewer KF6N,

Thank you for your review. The authors have posted their response. Please add a brief follow-up indicating whether your concerns have been addressed and noting any remaining questions.

The discussion phase closes on November 18 at 11:59 pm AoE.

Thank you for your attention and contribution.

Best regards,

Area Chair

Reviewer_aA8xOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines the influence of expert interaction on the loss. The authors conduct experiments on six MoE language models and find that experts’ effects on the loss add nearly as much as their outputs do. For a single-layer model with squared loss, the authors show that interaction equals the inner product of expert contributions. Experiments with orthogonal teachers show reduced normalized interaction after training, and a local convergence analysis establishes recovery of orthogonal teacher solutions.

Critical strengths

The paper examines an underexplored problem in the MoE domain and presents a substantial empirical study. The analysis includes six MoE language models and every routed-expert pair in every MoE layer, making the aggregate additivity finding informative. In addition, the authors explore the relationship between domain overlap and interaction within layers while controlling for single-knockout effect size. These controls help distinguish the association from differences in how strongly individual experts affect the loss.

Critical weaknesses
  1. Figure 1 and Table 3 in the manuscript average signed loss changes across domains, which allows positive and negative interactions to cancel. The absolute interaction analysis still averages within each domain first. These results support average additivity, but do not show whether it holds for individual questions.

  2. The experiments on six models are limited to MMLU. It remains unclear whether the near-additivity observation also holds for other tasks, such as math reasoning or coding.

  3. The interpretation of domain overlap could be clarified further. Similar activation frequencies across domains may not imply that two experts contribute to the same prompts. Comparing domain overlap with prompt-level co-activation may help determine whether the observed association reflects shared domain preferences.

Questions for authors
  1. Do deeper layers still show weaker interactions after accounting for differences in single-expert knockout effect sizes?

  2. In the synthetic experiment, how much of the decrease in normalized interaction comes from more orthogonal outputs?

1 comment

Dear Reviewer aA8x,

Thank you for your review. The authors have posted their response. Please add a brief follow-up indicating whether your concerns have been addressed and noting any remaining questions.

The discussion phase closes on November 18 at 11:59 pm AoE.

Thank you for your attention and contribution.

Best regards,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.