acceptodds
Under review as a conference paper at ICLR 2027

Overcoming Data Imbalances with Expert Selection Biases

16 views

Abstract

Mixture-of-Experts (MoE) models have become the standard architecture for frontier LLMs. The multiple parallel experts in each transformer block theoretically enable expert specialization according to categories such as domains, languages, or modalities during pretraining. However, when data is imbalanced across categories, minority categories can become concentrated in a small fraction of the expert pool or spread inefficiently across them, effectively limiting their available capacity. In this work, we demonstrate that category-conditioned expert-selection biases can preferentially route tokens from different categories toward specialized expert pools. By applying biases based on input category, we provide a simple way to induce soft category-aligned specialization. This intervention encourages categories toward distinct expert subsets while retaining cross-category sharing when useful. More importantly, it enables the over-allocation of expert capacity to categories that would otherwise be disadvantaged by data scarcity. We observe performance improvements across modalities, low-resource languages, and code in models up to 10.6B parameters. At our smallest and largest model scales, these gains are equivalent to allocating 18.5% and 2.7% additional baseline training tokens, respectively. Overall, our results show that expert biases can serve as a lightweight control for category-level expert allocation, making room for low-resource categories while retaining standard MoE inference.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-experts, Multimodal, Multilingual, Large Language Models, Low-resource

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tdQrOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces COAX, a training-time method that addresses data imbalance in sparse Mixture-of-Experts (MoE) models by adding category-conditioned routing biases to designated expert subsets. This enables greater expert allocation to low-resource categories while preserving cross-category sharing and standard MoE inference without additional biasing. Experiments across multilingual, multimodal, and multi-domain settings on models up to 10.6B parameters demonstrate consistent improvements for minority languages and vision-related tasks, with limited impact on high-resource categories. Comparisons with existing methods, specialization analyses, and extensive ablations further support the effectiveness and efficiency of COAX.

Critical strengths
  1. Simple and practical method. COAX introduces lightweight category-conditioned routing biases while preserving cross-category sharing, fixed per-token computation, and standard MoE inference.

  2. Extensive empirical evaluation. Experiments span multilingual, multimodal, and multi-domain settings across three model scales, demonstrating consistent improvements for low-resource categories with limited impact on majority categories.

  3. Thorough analyses and ablations. The paper provides detailed routing and specialization analyses, comparisons with alternative methods, and comprehensive ablations on expert allocation, bias strength, and training efficiency, offering strong empirical support for its design choices.

Critical weaknesses
  1. The training-cost comparison could be explained more clearly. The paper measures efficiency using per-step time overhead and equivalent additional training tokens. Clarifying the assumptions used to relate these measures and how the average equivalent training amount is calculated would make this conclusion clearer.

  2. The expert allocation strategy needs further clarification. The paper includes a sensitivity analysis of expert shares, but the reasons for assigning different shares to different categories could be stated more clearly. The paper also lacks details on how the preferred expert subsets (PcP_c) are constructed, including integer rounding and the assignment of specific expert IDs to each category. It would be useful to clarify whether changing these expert assignments affects performance.

  3. The differences from related methods could be discussed more clearly. The paper compares COAX with hard routing and EDL, but could further explain how it differs from other category-aware routing methods in controlling expert allocation and in its use at inference time. This would help readers understand its contribution.

Questions for authors
  1. How is the equivalent additional training cost calculated, and what assumptions support its comparison with the measured per-step training overhead?

  2. What motivates the category-specific choices of relative expert share? How are the preferred expert subsets constructed, and would changing the assigned expert IDs affect performance?

  3. What are the key methodological differences between COAX and existing category-aware routing methods, particularly regarding expert allocation and inference-time requirements?

3 comments

We thank the reviewer for recognizing the simplicity and practicality of the method, and the empirical rigor and thorough analysis of our work. We address the concerns below:

W1, Q1. training-cost comparison could be explained more clearly

Appendix G.1 describes how COAX's per-step overhead is measured. We measure iteration time after routing dynamics have stabilized, using the same hardware for both configurations. Since total training time is the number of steps × time per step, an overhead of x% per step increases total training time to (100+x)% of the baseline.

For token equivalence, we take the baseline and COAX checkpoints at the last step before cooldown (step 8000), both trained with the same seed and schedule. We then continue training the baseline from this checkpoint at a constant learning rate, validating every 100 steps, until its loss crosses the step 8000 COAX loss. This gives the additional steps the baseline needs to match COAX on each category. If the crossing happens between two checkpoints, we interpolate in log-tokens.

Every step processes the same number of tokens, so if this is y% of the original 8000 steps, the baseline requires (100+y)% of the original training compute to match COAX. Both x and y are fractions of the same training budget, making them directly comparable: COAX provides a favorable compute trade-off when y > x. For instance, at 10.6B, the coarser configuration requires only 1.3% more training time per step to achieve the same performance as the 2.7% equivalent additional baseline training (Table 29).

W2, Q2. expert allocation strategy needs further clarification

Our expert allocation choices follow a simple rule based on our ablations: allocate larger relative expert shares to smaller categories, without requiring precise tuning of the allocation. Appendix Table 19 shows that a relative expert share of 2× consistently outperforms proportional allocation (1×). Table 20 further shows relatively little sensitivity to the exact allocation between 2× and 8×, with larger shares (like 16×) sometimes degrading performance. Based on this, we use 2× relative expert share for relatively larger categories like Korean, vision, and audio (10–12.5%) and 4× for smaller ones like Urdu and Swahili (2.73% and 0.85%).

For constructing the preferred subsets, we first compute the desired number of experts as M_c ≈ N*p_c*r_c (eq. 11 in the main text). Our only placement-specific optimization is to distribute each category's experts equally across expert-parallel groups (two in our setup), so we round Mc to the nearest multiple of two. The subsets are mutually exclusive, with the majority category receiving the remaining experts. We do not expect performance to depend systematically on the specific expert IDs, since experts are exchangeable at initialization.

W3, Q3. The differences from related methods could be discussed more clearly

The main distinction is that COAX explicitly controls the preferred expert share of each category without enforcing exclusive routing or requiring category-aware inference.

MethodStandard MoE architectureExplicit expert shareAll experts accessible during trainingStandard inference
COAX✓✓✓✓
EDL✓✗✓✓
DEMix-style hard routing (our adaptation)✓✓✗✗
Lingual-SMoE✗✗✗✗

EDL encourages specialization by maximizing the divergence between category-level routing distributions, but does not control their preferred expert shares and thus fails to improve low-resource performance (Tables 14–15). DEMix-style hard routing instead enforces dedicated expert subsets, preventing cross-category sharing during training and requiring the routing constraint to be retained at inference. Lingual-SMoE uses hierarchical language-guided routing and dynamically adjusts language-specific expert groups, requiring additional routing machinery. COAX instead uses a simple selection-only bias to control preferred expert shares while preserving cross-category sharing and standard top-K inference.

Dear Reviewer tdQr,

Thank you for your review. The authors have posted their response. Please add a brief follow-up indicating whether your concerns have been addressed and noting any remaining questions.

The discussion phase closes on November 18 at 11:59 pm AoE.

Thank you for your attention and contribution.

Best regards,

Area Chair

Reviewer_tdQrOpenReview ↗

Comments to the response

Thank you for your response to my questions. I will keep my positive score.

Reviewer_fpiZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the problem of data imbalance across minority categories when pretraining MoE models. The author proposes COAX, which repurposes the per-expert router bias from auxiliary loss-free load balancing where it adds a fixed category-conditioned bias to designated subset of experts before top-k selection. This bias can be removed during inference. The size of the designated subset can exceed the category's training token share, thereby enables more allocation of expert capacity for the less-data categories. The experiments compare multilingual, multimodal, and domain mixtures at 345M-A147M, 3B-A654M, 10.6B-A1.2B models each trained on 10B tokens. The authors report the gains are equivalent to 18.5% and 2.7% additional training tokens for 345M and 10.6B models.

Critical strengths

Simplicity, as COAX changes only the pretraining expert-selection, and can have no additional inference-time overhead.

Extensive experimentation, the authors pretrained three models ranging from 345M-A147M and 10.6B-A1.2B across three multilingual, multimodal, and different domains, with 4-5 seeds for the main tables and report confidence intervals. The ablations are extensive too.

Statistically significant results for low-resource categories, Swahili in multilingual models and both vision categories in multimodal models at every model size.

Strong evidence that COAX induces functionally localized experts. By disabling the four Swahili designated experts increases Swahili loss by 107.1%, compared with only 9.3% when disabling the four experts most frequently used by Swahili in the unbiased baseline.

Critical weaknesses

The algorithmic novelty is limited. COAX repurposes the per-expert selection bias of auxiliary loss-free load balancing loss (Wang et al. 2024) but conditions the bias on category labels. Similar to DEMix and EDL, which also uses category labels for expert specialization, COAX provides the soft bias and it's based on the data share. This is useful but incremental.

The results are mixed.

  • For example, in more heterogeneous settings for multilingual models in table 2. The gains for English, Chinese, Korean, and Khmer are small and unclear if they are robust across different seeds.
  • For multimodal models, the 3B model actually regresses for audio and text. In general, the scale pattern isn't monotonic. The results are good at 345M model, mixed at 3B and small at 10.6B.
  • Also, in domain results, the results are not in favor of COAX with no inference bias.

The baselines are weak and don't support the data imbalance framing.

  • The comparison against DEMix-style baseline is better suited as hard-routing ablation. As both COAX and hard-routing use the same designated expert allocation, this will isolate soft vs hard bias and cross-category sharing.
  • With EDL, I am surprised that it makes every language worse including English.
  • The paper's main claim is about imbalance, so the relevant baselines would be imbalance methods not specialization methods as above. The comparison results against upsampling are in the appendix but not in the main text. Also simple upsampling does 14x better than COAX (-3.579% vs -0.26%) on the minority mean.
  • Multimodal and domain experiments have no method baseline.

Selecting experts for categories is not clear. For the bias it requires to know the subset of the experts for each those categories. Selecting expert indices Pc ⊆ {1, . . . ,N} and how many experts for each category, given there are many categories is not clear. Each category requires disjoint subset of experts, this might be bottlenecked for example, 64 experts in 345M model.

Evaluation on downstream benchmarks (Appendix B.1) is limited and inconclusive which makes it unclear whether the small validation loss gains translate into meaningful capabilities.

Questions for authors

Any guidance on how to select Pc ⊆ {1, . . . ,N} for each category? I see a limitation in case of training a multilingual multimodal model (many categories) which might exhaust the no. of experts available. Can the experts overlap? Is there any study on how categories would interact if they are allowed to overlap? - The English vs rest ablation (Table 5) is useful and remove the English bias to allow overlap is harmful (Appendix F.1), but the method still requires manually defining Pc and Mc.

Is it possible to provide evidence that the validation-loss gains for low-resource categories translate into improvement in downstream capability? The reported results in Appendix B.1 are inconclusive.

How the model developer selects the operating point between minority-capacity and majority performance? By increasing the capacity share of the minorities, the share of the dominant categories reduces. In domain settings, it regresses significantly on English (Table 4).

The main claim that the gains are equivalent to 18.5% and 2.7% additional training tokens for 345M and 10.6B models. In appendix G.2, the additional training experiment is done on multimodal model for 345M-A147M while mulitlingual model for 10.6B model. This doesn't provide fair evidence how COAX scales. The scaling experiment under same settings would be more fair.

Could you explain why EDL performs substantially worse in all the categories?

In Figure 2, expecting uniform-random rate for an unbiased baseline needs to be corrected. The unbiased baseline might steer certain experts for Swahili tokens more than random rate which is also shown in your table 11, appendix C.2 (4.58% vs 3.12%).

What is the interaction between the loss-free load balancing expert-bias with yours category-aligned expert bias?

5 comments
AuthorsOpenReview ↗

Response Part 1/3: Novelty and mixed results

We thank the reviewer for recognizing COAX's simplicity and the breadth of the experiments. We address the concerns below.

W1. The algorithmic novelty is limited… Similar to DEMix and EDL …

We view COAX's use of an existing expert-biasing mechanism as a practical strength rather than a weakness. By building on the expert-selection bias of Wang et al. (2024), COAX requires no new routing architecture and can be readily adopted in MoE training pipelines that already support expert biases. The contribution is in repurposing this mechanism to explicitly control the preferred expert share of each category independently of its token share. Compared to EDL, which encourages category-specific routing without directly controlling expert allocation, COAX allows developers to explicitly set the preferred expert share of each category, allowing low-resource categories to receive disproportionately more capacity. Unlike DEMix's hard routing during training or Lingual-SMoE's restricted language-specific expert pools, COAX preserves access to all experts and requires neither category labels nor specialized routing at inference.

W2. The results are mixed.

Multilingual and code gains are consistent across scales (Tables 1 and 4). Lower-resource Urdu and Swahili improve in every seed at every scale, with larger gains as data share decreases. Both vision categories also improve across scales, although the 3B text/audio results are mixed.

W2 point 1. The gains for English, Chinese, Korean, and Khmer are small and unclear if they are robust across different seeds.

Q4. the additional training experiment is done on multimodal model for 345M-A147M while mulitlingual model for 10.6B model.

We have run additional experiments to answer these questions/concerns. In a new three-seed 345M run on the eight-language mixture with English vs. rest grouping, we obtain the following mean changes in validation loss (%; negative is better):

Inference biasEnZhKoKmTaKaUrSw
Off−0.19−0.51−0.48−0.30−0.58−0.43−0.60−0.64
On−0.24−0.55−0.52−0.46−0.64−0.51−0.65−0.71

All eight languages improve even without inference bias. The baseline needs 10.5% more tokens to match the minority mean (11.4% with inference bias). This adds a same-mixture token-equivalence comparison at 345M alongside our 10.6B results.

Q3. in domain results, the results are not in favor of COAX with no inference bias.

In domains, a stronger α=0.3 annealed to zero improves all four minorities (−0.04% to −1.70%) at +0.75% English loss. This shows that COAX can also benefit lower-resource domains, but requires a stronger bias than our default multilingual and multimodal configurations.

AuthorsOpenReview ↗

Response Part 2/3: Baselines and expert allocation

W2 point 3. simple upsampling does 14x better than COAX

Upsampling Urdu/Swahili is stronger by itself (minority mean −3.579% vs. −0.260% for COAX), but combining the two improves further to −3.738% (Table 7), suggesting that COAX is complementary to data upsampling. COAX also incurs a much smaller English performance cost (+0.011% vs. +0.276%), meaning its improvements come with less of a majority-language trade-off. Table 6 shows that higher α values can shift this trade-off (e.g., α=0.3 with inference bias: +0.46% English loss, −1.65% minority loss). We have now included this comparison in the main paper (Ablations, Table 7).

the relevant baselines would be imbalance methods not specialization methods

Multimodal and domain experiments have no method baseline

We further evaluate whether other imbalance-aware methods can reproduce COAX's gains. We tested LIMoE's per-modality entropy losses at two coefficients (0.004/0.04): all six multimodal losses regress (+0.9% to +7.2%), while COAX improves all six (−0.4% to −0.9%). We also extended the comparisons with EDL and matched-allocation hard routing to multimodal and domain mixtures (relative validation-loss change, %; negative is better):

Multimodal (345M)TextT→VV→TA–TAudioASR
COAX−0.92−0.55−0.72−0.83−0.57−0.46
Hard routing−0.01−1.91−2.42−1.85−2.69−4.55
EDL−0.47−0.99−1.25−0.86−1.39−1.25
Domains (345M)EnglishCodeMathLegalMedical
COAX, bias off+0.26−0.19+0.10+0.04+0.07
COAX, bias on+0.14−0.35−0.05−0.15−0.11
Hard routing+2.50+1.30+6.49+7.26+4.91
EDL+1.78+2.45+2.18+2.76+2.80

COAX leads to less of a degradation for the majority classes than hard-routing and EDL in both sets of experiments. In the multimodal experiments, hard-routing and EDL benefit minority modalities more; COAX shows the best improvements for the majority modality while also helping the minority ones. In domains, EDL and hard routing regress everywhere, whereas COAX improves code without inference bias and all minority domains with bias retained. Together with Table 2, where only COAX improves over the multilingual baseline, these results suggest that COAX is more broadly beneficial across the tested mixtures, while EDL and hard routing can offer greater gains for low-resource modalities.

W4/Q1/Q3. Selecting experts for categories is not clear … Each category requires a disjoint subset of experts …

Our expert allocation choices follow a simple rule based on our ablations: allocate larger relative expert shares to smaller categories, without requiring precise tuning of the allocation. Appendix Table 19 shows that a relative expert share of 2× consistently outperforms proportional allocation (1×). Table 20 further shows relatively little sensitivity to the exact allocation between 2× and 8×, with larger shares (like 16×) sometimes degrading performance. Based on this, we use (and recommend) 2× relative expert share for relatively larger categories like Korean, vision, and audio (10–12.5%) and 4× for smaller ones like Urdu and Swahili (2.73% and 0.85%). We have also added this in the revised pdf (Lines 308-312),

We also tested overlapping subsets (345M four-language mixture, seed 31). Each minority kept its original exclusive experts and gained an equal number shared with English, doubling its preferred pool:

RoutingEnglishKoreanUrduSwahiliMinority mean
Overlap, bias off−0.08−0.05−0.42−0.49−0.32
Disjoint, bias off−0.20−0.43−0.70−0.62−0.58
Overlap, bias on−0.15−0.12−0.50−0.55−0.39
Disjoint, bias on−0.28−0.48−0.76−0.68−0.64

Relative validation-loss change (%) vs. aux-loss baseline; bias refers to inference-time bias.

Overlap underperforms the disjoint default here but still improves every language, suggesting that overlapping subsets are a viable option when the number of categories is large. A more informed overlap strategy could also exploit positive transfer between related categories, while keeping negatively interfering categories separate.

Q3. How the model developer selects the operating point between minority-capacity and majority performance

For the majority–minority trade-off, α=0.03 provides minority gains with little majority cost and standard inference; larger α can favor minorities further at an English cost (Table 6).

AuthorsOpenReview ↗

Response Part 3/3: Downstream capabilities, EDL, loss-free load balancing

W5/Q2 Evaluation on downstream benchmarks (Appendix B.1) is limited and inconclusive

We extended the 10.6B multilingual baseline and coarse COAX to 100B tokens. Both remain near chance on Belebele (26.4% vs. 26.5%) and SIB-200, and saturate MultiBLiMP (96.8% each), so these benchmarks cannot resolve capability differences. However, Table 10 shows gains in all 12 teacher-forced multimodal accuracy metrics, e.g., ASR top-5 word accuracy 47.50%→49.38% and captioning 78.52%→79.57%. Prior work has also established predictive relationships between pretraining loss and downstream performance [1, 2], supporting our use of validation loss as the primary metric when downstream benchmarks are inconclusive.

[1] Gadre et al., Language Models Scale Reliably with Over-Training and on Downstream Tasks, 2024. [2] Brandfonbrener et al., Loss-to-Loss Prediction: Scaling Laws for All Datasets, 2025.

Q5. why EDL performs substantially worse

EDL increases between-language router JSD from 0.0025 to 0.3300, confirming that its intended specialization objective is active, yet regresses the minority language mean by 1.016% (Tables 14–15). EDL does not allocate preferred shares, and routing divergence alone seems insufficient to improve the validation loss on minority languages (though response to W3 shows it works for minority modalities ). One possible explanation is that divergence exacerbates the concentration of lower-resource languages onto a few experts, a failure mode identified in [3, 4]. Such concentration could increase divergence if higher-resource languages avoid these experts, without actually improving the capacity available to minority categories.

[3] Mustafa et al., NeurIPS 2022: Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts. [4] Joseph et al., arXiv 2026: Mixture of Experts for Low-Resource LLMs.

Q6. expecting uniform-random rate for an unbiased baseline needs to be corrected

The dashed 3.12% in Figure 2 is the uniform-random reference, not the expected routing fraction of a learned baseline. The baseline routes ~2.95% of Swahili traffic to COAX's preassigned four experts; Table 12's 4.58% instead measures the baseline's four most-used experts, selected post hoc.

Q7. What is the interaction between the loss-free load balancing expert-bias with yours category-aligned expert bias

We also evaluated COAX (α=0.03, no inference bias) on matched loss-free-balancing baselines:

345M mixtureMajorityMinority range
Multimodal+0.36%−0.02% to +0.59%
Four languages+0.35%+0.17% to +0.76%
Domains+0.01%−0.51% to −0.26%

Loss-free balancing itself matches or improves our aux-loss baseline (up to −1.3%). COAX adds clear minority gains in domains, but less in the other two mixtures. A possible explanation is that global load balancing increases the bias of underused minority-preferred experts for all categories, drawing majority tokens into them and diluting category-specific allocation. Category-aware balancing targets or within-subset balancing could address this interaction. Importantly, functional specialization remains: removing Swahili's designated experts increases its loss by 11.5% (89.3% with annealed α=0.3). In comparison, for the loss-free baseline, removing the two most-used Swahili experts hurts Urdu more than Swahili (+8.9% vs. +6.9%) and Korean almost as much (+4.8%). These experts therefore appear to form a shared low-resource pathway rather than a Swahili-specific module. We view the joint design of load balancing and category-conditioned allocation as a natural direction for further study.

Reviewer_qwaKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses learning from imbalanced data mixtures in MoE models. COAX adds category-conditioned biases to the routing scores of designated expert subsets during pretraining, while retaining the original scores for weighting the selected experts. This encourages category-aligned specialization and allows low-resource categories to receive a preferred expert share larger than their token share, without increasing the number of active experts per token.Experiments across multilingual, multimodal, and domain mixtures report validation-loss improvements, particularly for low-resource languages and vision.

Critical strengths
  1. The intervention is simple and has supporting empirical evidence. The proposed method changes expert selection without increasing the number of active experts per token, while retaining soft sharing. The matched-seed multilingual and multimodal comparisons, together with the expert-removal analysis, support both performance gains in the tested settings and the emergence of category-localized functionality.
Critical weaknesses
  1. Manually defined categories may not capture the relevant long tail. Language-level grouping can hide rare knowledge within a common language, such as specialized mathematics, while separating related knowledge across languages. The grouping and expert-share ablations are useful, but do not establish that the chosen categories correspond to under-learning or gradient interference.

  2. The majority–minority trade-off lacks a clear selection criterion. The paper reports that stronger biasing can improve minority languages while degrading English, including after bias annealing. However, it does not explain how to choose expert shares and bias strength to balance these outcomes, limiting practical guidance on when the minority gains justify the majority cost.

Questions for authors
  1. See also weakness1. What evidence supports the chosen categories as a way to identify under-learned knowledge? For example, some previous works tried to define under-learned longtail knowledge from the model parameter/ data activations instead of defining manually.

[1] Dong et. al., NeurIPS 2024: Once Read is Enough: Domain-specific Pretraining-free Language Models with Cluster-guided Sparse Experts for Long-tail Domain Knowledge.

[2] Feldman et. al., NeurIPS 2020: What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation

  1. See also weakness 2. What validation criterion was used to select the default expert shares and bias strength, given the trade-off between minority gains and majority-language degradation?
3 comments

We thank the reviewer for recognizing the simplicity and the empirical rigor of our work. We address the concerns raised as follows:

W1, Q1. Language-level grouping can hide rare knowledge within a common language … the chosen categories correspond to under-learning or gradient interference.

Existing work establishes exactly these properties in the categories we consider. [1] quantifies and finds substantial negative transfer from text to vision, and weaker negative transfer to audio. Similarly, [2] shows that English has high negative interference coefficients towards languages like Urdu and Swahili. Complementarily, [3] shows that minority text tokens can collapse onto a single expert in multimodal MoEs, while [4] finds that underrepresented languages exhibit concentrated routing to a small subset of experts. These findings motivate our choice of modalities and languages under-represented in data as proxies for potentially under-learned categories.

Discovering under-learned knowledge from model parameters or data activations, as suggested by the reviewer, is a promising direction, but the methods mentioned require additional computation and methodological complexity, such as estimating example-level influence through repeated model training (Feldman and Zhang, 2020) or clustering learned representations during training (Dong et al., 2024). Given the existing evidence that the categories we consider are under-learned, and the availability of category labels in the training data, we argue that our simpler approach is a more practical choice in this setting, avoiding the additional overhead of identifying under-learned categories from model-dependent signals.

[1] Khandelwal et al., arXiv 2026: Scaling Laws for Multimodal Data Mixtures. [2] Longpre et al., ICLR 2026: ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilinguality. [3] Mustafa et al., NeurIPS 2022: Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts. [4] Joseph et al., arXiv 2026: Mixture of Experts for Low-Resource LLMs.

W2, Q2. What validation criterion was used to select the default expert shares and bias strength …

Our results support several practical recommendations for balancing minority gains against majority performance and inference costs, which we have now clarified in the revised paper.

For expert shares, we recommend allocating 2–8× the data share to lower-resource categories, with larger multipliers for smaller categories. Appendix Table 19 shows that 2× consistently outperforms proportional allocation (1×) in the multimodal mixture, while Table 20 shows relatively little sensitivity between 2× and 8× for Swahili, with slight degradation at 16×. Based on this, we use 2× for relatively larger categories like Korean, vision, and audio (10–12.5% data share), and 4× for smaller ones like Urdu and Swahili (2.73% and 0.85%). We recommend choosing the relative expert share based on the number of categories, their data proportions, and their relative importance to the practitioner. A larger number of categories would require smaller allocations closer to 2× or coarser grouping, as explored in Table 5.

For bias strength, we recommend α=0.03 as the default value, improving the minority mean by 0.42% with essentially no English degradation (−0.03%) and no inference-time bias required (Table 6). If stronger minority gains are desired, α=0.3 improves the minority mean by 1.65%, but degrades English by 0.46% and requires retaining the bias at inference. Annealing α=0.3 to zero avoids this inference requirement, retaining a 1.26% minority improvement at a 0.57% English cost. Thus, practitioners can select α based on acceptable majority degradation and inference overhead.

Finally, Appendix Figures 4 and 5 show that COAX's gains stabilize from approximately 5B tokens in the 10B-token run and persist through 100B tokens in the extended run, suggesting that smaller pilot runs can be informative for testing COAX configurations before committing to full training.

Thank you for the detailed response. Your explanation on expert allocation resolved my question, but my concern about the data grouping remains. I now think the paper as a well-developed engineering adaptation, but it falls somewhat short as a research contribution.

I am therefore keeping my rating at 2. On balance, I lean toward rejection, although I would not object to it if the AC recommends acceptance. This year’s four-point scale makes that distinction difficult to express, especially given reports that roughly two-thirds of submissions received a rating of 2. I would appreciate it if the AC took this nuance into account when weighing my review.

We are glad that our response resolved the reviewer’s concerns about expert allocation.

Regarding data grouping, we initially understood the concern as questioning whether our chosen categories are relevant in this context and responded with prior work showing that low-resource languages and modalities can experience negative interference during pretraining, which could lead to under-learning. However, we now understand the reviewer’s concern to be about the validity of defining low-resource categories a priori. While automatically discovering under-learned categories is an interesting question, it is not the one we pursue in this work.

We also note from a practical perspective that not every under-learned category is equally important to every LLM developer, and it is a perfectly valid developer choice to prioritize particular languages, modalities, or domains. COAX provides a simple, intuitive, and effective way to mitigate the effects of data imbalance for these categories.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.