- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces COAX, a training-time method that addresses data imbalance in sparse Mixture-of-Experts (MoE) models by adding category-conditioned routing biases to designated expert subsets. This enables greater expert allocation to low-resource categories while preserving cross-category sharing and standard MoE inference without additional biasing. Experiments across multilingual, multimodal, and multi-domain settings on models up to 10.6B parameters demonstrate consistent improvements for minority languages and vision-related tasks, with limited impact on high-resource categories. Comparisons with existing methods, specialization analyses, and extensive ablations further support the effectiveness and efficiency of COAX.
Critical strengths
-
Simple and practical method. COAX introduces lightweight category-conditioned routing biases while preserving cross-category sharing, fixed per-token computation, and standard MoE inference.
-
Extensive empirical evaluation. Experiments span multilingual, multimodal, and multi-domain settings across three model scales, demonstrating consistent improvements for low-resource categories with limited impact on majority categories.
-
Thorough analyses and ablations. The paper provides detailed routing and specialization analyses, comparisons with alternative methods, and comprehensive ablations on expert allocation, bias strength, and training efficiency, offering strong empirical support for its design choices.
Critical weaknesses
-
The training-cost comparison could be explained more clearly. The paper measures efficiency using per-step time overhead and equivalent additional training tokens. Clarifying the assumptions used to relate these measures and how the average equivalent training amount is calculated would make this conclusion clearer.
-
The expert allocation strategy needs further clarification. The paper includes a sensitivity analysis of expert shares, but the reasons for assigning different shares to different categories could be stated more clearly. The paper also lacks details on how the preferred expert subsets () are constructed, including integer rounding and the assignment of specific expert IDs to each category. It would be useful to clarify whether changing these expert assignments affects performance.
-
The differences from related methods could be discussed more clearly. The paper compares COAX with hard routing and EDL, but could further explain how it differs from other category-aware routing methods in controlling expert allocation and in its use at inference time. This would help readers understand its contribution.
Questions for authors
-
How is the equivalent additional training cost calculated, and what assumptions support its comparison with the measured per-step training overhead?
-
What motivates the category-specific choices of relative expert share? How are the preferred expert subsets constructed, and would changing the assigned expert IDs affect performance?
-
What are the key methodological differences between COAX and existing category-aware routing methods, particularly regarding expert allocation and inference-time requirements?
3 comments
Dear Reviewer tdQr,
Thank you for your review. The authors have posted their response. Please add a brief follow-up indicating whether your concerns have been addressed and noting any remaining questions.
The discussion phase closes on November 18 at 11:59 pm AoE.
Thank you for your attention and contribution.
Best regards,
Area Chair
Comments to the response
Thank you for your response to my questions. I will keep my positive score.
We thank the reviewer for recognizing the simplicity and practicality of the method, and the empirical rigor and thorough analysis of our work. We address the concerns below:
Appendix G.1 describes how COAX's per-step overhead is measured. We measure iteration time after routing dynamics have stabilized, using the same hardware for both configurations. Since total training time is the number of steps × time per step, an overhead of x% per step increases total training time to (100+x)% of the baseline.
For token equivalence, we take the baseline and COAX checkpoints at the last step before cooldown (step 8000), both trained with the same seed and schedule. We then continue training the baseline from this checkpoint at a constant learning rate, validating every 100 steps, until its loss crosses the step 8000 COAX loss. This gives the additional steps the baseline needs to match COAX on each category. If the crossing happens between two checkpoints, we interpolate in log-tokens.
Every step processes the same number of tokens, so if this is y% of the original 8000 steps, the baseline requires (100+y)% of the original training compute to match COAX. Both x and y are fractions of the same training budget, making them directly comparable: COAX provides a favorable compute trade-off when y > x. For instance, at 10.6B, the coarser configuration requires only 1.3% more training time per step to achieve the same performance as the 2.7% equivalent additional baseline training (Table 29).
Our expert allocation choices follow a simple rule based on our ablations: allocate larger relative expert shares to smaller categories, without requiring precise tuning of the allocation. Appendix Table 19 shows that a relative expert share of 2× consistently outperforms proportional allocation (1×). Table 20 further shows relatively little sensitivity to the exact allocation between 2× and 8×, with larger shares (like 16×) sometimes degrading performance. Based on this, we use 2× relative expert share for relatively larger categories like Korean, vision, and audio (10–12.5%) and 4× for smaller ones like Urdu and Swahili (2.73% and 0.85%).
For constructing the preferred subsets, we first compute the desired number of experts as M_c ≈ N*p_c*r_c (eq. 11 in the main text). Our only placement-specific optimization is to distribute each category's experts equally across expert-parallel groups (two in our setup), so we round Mc to the nearest multiple of two. The subsets are mutually exclusive, with the majority category receiving the remaining experts. We do not expect performance to depend systematically on the specific expert IDs, since experts are exchangeable at initialization.
The main distinction is that COAX explicitly controls the preferred expert share of each category without enforcing exclusive routing or requiring category-aware inference.
EDL encourages specialization by maximizing the divergence between category-level routing distributions, but does not control their preferred expert shares and thus fails to improve low-resource performance (Tables 14–15). DEMix-style hard routing instead enforces dedicated expert subsets, preventing cross-category sharing during training and requiring the routing constraint to be retained at inference. Lingual-SMoE uses hierarchical language-guided routing and dynamically adjusts language-specific expert groups, requiring additional routing machinery. COAX instead uses a simple selection-only bias to control preferred expert shares while preserving cross-category sharing and standard top-K inference.