- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes GameMoE, a task-adaptation method that uses cooperative games to supervise MoE routing. Under a fixed Top-k budget, it evaluates individual and joint expert replacements through reductions in reference-token loss, using their utilities and pairwise interactions to train the final-layer router. Training combines this reward with cross-entropy and a centered routing anchor that constrains relative score changes. Experiments on OLMoE across three tasks show improved greedy-decoding performance while retaining native Top-8 inference.
Critical strengths
-
Interpretable interaction-aware routing formulation: The paper explicitly distinguishes individual replacement utilities from pairwise interactions. Equation (10) shows how each action’s update combines its single-action utility with an interaction term weighted by the other action’s selection probability, providing a clear link between coalition evaluation and router optimization.
-
Effective adaptation with native inference: GameMoE achieves the best greedy-decoding results among the evaluated methods across all three tasks, without test-time search. The parameter-matched CE-only comparison on SQuAD supports the benefit of the added training objective beyond adaptation capacity alone.
Critical weaknesses
-
The sensitivity to interaction upweighting needs stronger explanation: In Figure 2(a), SQuAD EM drops from 85.0 with the default interaction weight =2 to 75.4 with =1, which uses the original coalition utility. Both this variant and the no-interaction variant perform substantially worse than CE-only training (82.4). Equation (10) explains why upweighting can strengthen the interaction signal, but does not fully explain the large performance gap. Results across nearby interaction weights and multiple random seeds would help establish whether the improvement is stable.
-
The ablations do not isolate the contribution of coalition supervision from CRA: The CE-only baseline removes both coalition supervision and CRA, whereas the interaction-weight variants retain CRA. These comparisons support the combined training objective, but do not establish how much of the improvement comes from coalition supervision rather than the routing anchor. A CE + CRA baseline would help clarify this contribution.
Questions for authors
-
Why does using the original coalition utility (=1) perform substantially worse than CE-only training? Could you report results across multiple random seeds and nearby interaction weights to clarify the stability of this behavior?
-
How does CE + CRA perform without coalition supervision? Conversely, how does removing CRA affect the full method?
-
In the fixed-model diagnostic, additive selection recovers 98.9% of the loss reduction achieved by joint selection. How should we interpret this result alongside the downstream gains from interaction-aware training? Is there additional evidence connecting the measured pairwise interactions to these gains?
-
What is the additional training time and peak memory usage relative to the matched CE-only baseline? Although inference retains native routing, equal training epochs do not imply equal computational cost.