- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether expert co-activation in sparse MoE models reflects actual functional collaboration between experts. The authors introduce an interventional framework that compares expert coalitions under the same active-compute budget, includes randomly sampled experts in the candidate pool, and decomposes coalition utility into unary and interaction effects. Across three frozen MoE language models, they find that non-additive expert interactions are strongest in shallow layers and become very small in deeper layers, while a substantial part of the shallow interaction cannot be explained by pairwise effects alone. They further show that pairwise co-activation does not reliably predict signed interventional synergy, as pairwise synergy transfers poorly across documents and no co-activation–synergy correlation survives multiple-testing correction. Finally, they test whether the observed routing headroom can be exploited with simple learned predictors, but find no reliable gains at their experimental scale, while positive controls confirm that the evaluation pipeline can recover sufficiently strong signals.
Critical strengths
Strength 1: The paper clearly distinguishes expert co-activation from actual functional interaction through controlled interventions.
Strength 2: The experiments are carefully designed, with random expert baselines, equal-compute comparisons, and explicit numerical noise controls.
Strength 3: The analysis is thorough across multiple MoE models and reveals a clear pattern that non-additive effects are strongest in shallow layers.
Critical weaknesses
Weakness 1: The paper measures collaboration mainly through pairwise counterfactual synergy under a fixed compute budget. This does not cover context-dependent complementarity, interactions among more than two experts, or effects that emerge across layers. The results therefore show that co-activation is not a reliable indicator of the pairwise synergy measured in this paper, but they do not fully support the broader claim that co-activation does not reflect expert collaboration.
Weakness 2: The measured interaction is not necessarily generated by the experts at the intervened layer. For some of the studied models, expert outputs at that layer are combined additively, so the observed non-additive effect may instead arise from downstream nonlinearities or later routing changes. The experiments show that changing an expert coalition can produce a non-additive effect on final utility, but they do not isolate where this interaction is actually generated.
Weakness 3: Unstable average pairwise synergy across documents does not by itself show that co-activation is unrelated to collaboration. If collaboration depends on the hidden state or local context, the same expert pair may help in some cases and have little or even negative effect in others, making the average synergy unstable. The paper also finds that a substantial part of the shallow-layer non-additivity is not explained by pairwise effects. This leaves open the possibility that co-activation is related to context-dependent or higher-order collaboration.
Questions for authors
Question 1: Can the authors clarify how far the conclusion is intended to generalize beyond the pairwise counterfactual synergy studied here? In particular, do the results extend to context-dependent, higher-order, or cross-layer forms of expert collaboration?
Question 2: Can the authors separate interaction generated at the intervened layer from non-additive effects introduced later by downstream nonlinearities or subsequent routing changes?
Questino 3: Can the authors test whether co-activation becomes predictive after conditioning on the hidden state or finer-grained context, and whether group-level co-activation is related to higher-order synergy?