acceptodds
Withdrawn from ICLR 2027

Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

17 views

Abstract

Sparse mixture-of-experts (MoE) large language models scale model capacity by routing each token to a small subset of experts. Their routers are regularized with load balancing terms and learn affinity scores through the language-model objective. However, these objectives do not provide direct alignment between routing affinities and token-level error. We introduce token-error supervision for sparse routing in two forms. The first form predicts an error score per expert. The affinity-weighted aggregate of these scores is aligned to the next-token cross-entropy loss, while the individual scores attenuate affinity before top- selection. The second directly aligns the router's affinities to the model's objective without requiring an additional head or inference-time modification. Both formulations use the Itakura–Saito divergence or an exponential negative log-likelihood for aligning affinities and token errors. Across two sparse MoE backbones and four multiple-choice question-answering benchmarks, we evaluate both supervision mechanisms. On Granite, our method improves accuracy by approximately 2.3 percentage points on average over a parameter-matched routing baseline. With stronger supervision, the gain on ARC-Challenge reaches 2.94 points. Both mechanisms preserve the native sparse execution budget and aggregation policy. Our code is available in the supplementary materials.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture of Experts, Large Language Models, Sparse Routing, Router Supervision, Token-Error Prediction, Affinity-Concentration Supervision

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 5 reviews, above 27% of the others.
Reviewer_aR4EOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

MoE routers does not have any direct loss signal to learn the token routing, it only learns indirectly from the overall language-modeling loss. The paper provides a post-training method to utilize the next-token loss (cross-entropy, CE) as a direct teaching signal for the router per token.

  1. The paper proposes to use two methods (TES which introduce an additional error head ) and ACS without any extra parameters addition, to tune the MoE router so that it gains direct signal from the per token CE loss realized at the end of the model layer.
  2. The paper mainly keeps the base model params frozen, except the affinity tuning case where base router is also updated, and trains Lora adapters to adapt the base model router.
  3. The experiments include two base models (Granite 3.1 and OLMoE) and they are trained on the train split of four tasks datasets (ARC-Challenge, OpenBookQA SciQ, MedMCQA) and are evaluated on their test split.
Critical strengths

None

Critical weaknesses
  1. The paper does not provide a broader scope of the proposed method in the lifecycle of an LLM models. In general, the LLM models are trained to perform well on multiple tasks in the post-training phase. But this work, fine-tunes the models on each task at a time and provides its results separately which is not a realistic approach. The effects should be observed when fine-tuned once on all combined dataset. Additionally, in case of TES, there is additional head added which changes the native topk selection. There is no ablations on the effect of this change on the base accuracy of tasks which are not included in this fine-tuning. Similarly, in case of ACS, even though there is no explicit head but it changes the routing behavior of the base model. But there is no comparison with the original accuracy of the base model before this fine-tuning was done to understand the effect of this approach for the tasks/basic capabilities not included as part of the fine-tuning.

  2. The results are very marginal (table 1). TES method: The Paper says “ACS-IS improves average accuracy by 2.1 points and TES-IS improves over Dual Affinity by 2.3 points on Granite and 0.5 on OLMoE, averaged across all four benchmarks”. But in table 1, Granite - TES-IS vs Dual Affinity (MedMCQA gain (+6.27) drives most of the TES average, and the baseline there is very noisy (37.40 ± 7.79 over 3 seeds). Excluding it, the average gain is about +0.9 which is very marginal. SciQ is near saturation and the TES-IS is either flat or negative there. On OLMoE, the TES-IS gain is very marginal as well (0.5). This results indicate that the method is not adding benefit universally on all models and the results are not significantly better where it’s adding value. ACS Method: - The paper claims +2.1 points but that is only for the case of Granite base model with a frozen router. With the router trainable case, the gains on Granite shrinks to +0.7, and it is concentrated in ARC-C. On OLMoE, ACS-IS is slightly worse on average in both settings. And On Granite, ENLL variation gives gains about as large as IS. And similarly on OLMoE, all three ENLL setups are flat or slightly negative on average.

  3. Overall, the models tested here are smaller in size, the eval tasks are multiple-choice only, and the dataset is tiny. Hence, the experimental set up is very narrow scoped.

  4. In Fig 2, the predicted error seems to be correlated with actual NLL. But eventually, that behavior doesn't turn into any significant accuracy gains on the tasks evaluated here. Hence, supervising the router with token loss while keeping top-K execution unchanged, is a decent direction to explore but the results are not promising.

Questions for authors

NA

Reviewer_eRiaOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces two token-level supervision mechanisms to align the routing behavior of sparse Mixture-of-Experts (MoE) large language models with the realized next-token cross-entropy (CE) loss. The first method, Token-Error Supervision (TES), employs a lightweight error prediction head to estimate expert-level error scores, which are then used to attenuate the native affinity logits before top-K selection. The second method, Affinity-Concentration Supervision (ACS), requires no additional parameters and directly aligns the squared concentration of the selected-expert affinities with the token loss using Itakura–Saito divergence or exponential negative log-likelihood. Evaluated on Granite 3.1 and OLMoE-1B across four multiple-choice question-answering (MCQA) benchmarks, both methods demonstrate notable accuracy improvements over parameter-matched baselines while preserving the native sparse execution budget.

Critical strengths
  • Lightweight and Practical for SFT: The proposed methods, particularly ACS, are highly lightweight and do not introduce any inference-time overhead or architectural modifications. They can be seamlessly integrated into the existing Supervised Fine-Tuning (SFT) stage, making them highly practical for post-training optimization.
  • Sound Theoretical Formulation: The use of scale-invariant measures like Itakura–Saito (IS) divergence and Exponential Negative Log-Likelihood (ENLL) to align routing statistics with token-level errors is theoretically well-motivated. The paper provides rigorous mathematical derivations regarding the gradient allocation and optimization optima.
Critical weaknesses
  • Outdated Backbones and Limited Scale: The experimental backbones (Granite 3.1 3B and OLMoE-1B) are relatively early-stage and small in scale. Given the rapid evolution of MoE architectures reaching tens or hundreds of billions of parameters, the lack of validation on larger, modern, and more complex fine-grained MoE models weakens the generalizability and persuasiveness of the claims.
  • Narrow Evaluation Scope (Lack of Generation Tasks): The evaluation is strictly limited to single-token prediction tasks (MCQA benchmarks). MoE models are predominantly utilized for their strong autoregressive generation capabilities. The paper completely fails to evaluate the impact of the proposed methods on open-ended generation tasks (e.g., summarization, coding, or complex reasoning), leaving it unclear whether aligning routing with token-level CE might inadvertently harm generation diversity, fluency, or long-context coherence.
  • Insufficient Analysis of Routing Behavior: While the paper focuses heavily on downstream accuracy, it lacks an in-depth mechanistic discussion on how the auxiliary losses alter the intrinsic routing behavior. For instance, since ACS explicitly pushes for higher affinity concentration, does this lead to "rich-get-richer" dynamics, causing expert collapse or dead experts? How does TES's error attenuation affect the semantic specialization and load balancing of the router? These critical mechanism-level insights are missing.
Questions for authors
  • Impact on Generation Capabilities: Have you evaluated the proposed methods on open-ended generation tasks? Since MCQA only tests single-token prediction, it is crucial to verify that the proposed routing supervision does not degrade the model's autoregressive generation quality or diversity.
  • Routing Behavior and Load Balancing: How do TES and ACS affect the intrinsic routing dynamics, such as expert utilization rates and load balancing? Specifically, does the concentration push in ACS lead to a higher variance in expert load (i.e., a few experts dominating while others become dead)? Please provide metrics like normalized activation entropy or Gini coefficients before and after applying your methods.
  • Scalability to Modern MoEs: The current backbones are relatively small. Can you provide insights, theoretical arguments, or preliminary results on how these methods scale to larger, modern fine-grained MoE architectures (e.g., models with hundreds of experts per layer)? Mechanistic Insights: Could you provide a deeper analysis of why the auxiliary loss helps at the mechanism level? For example, does the learned error score in TES actually make the router more "uncertainty-aware," and how does it correlate with the semantic specialization of the selected experts?
Reviewer_PMQNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes two mechanisms for supervising MoE routing with token-level cross-entropy loss. Token-Error Supervision (TES) learns expert-indexed error scores and uses them to adjust routing affinities, while Affinity-Concentration Supervision (ACS) directly regularizes the concentration of routing probabilities. Both approaches use Itakura–Saito divergence or exponential negative log-likelihood and preserve the original Top-K expert budget. Experiments on two MoE backbones and four multiple-choice benchmarks demonstrate improvements over matched baselines, particularly on Granite, although gains on OLMoE are less consistent.

Critical strengths
  1. Meaningful motivation and lightweight design. The paper introduces explicit token-error supervision into MoE routing without increasing the number of activated experts. The two complementary formulations provide practical alternatives with different inference-time overheads.

  2. Reasonable experimental controls and theoretical analysis. The paper includes parameter-matched baselines, three-seed evaluations, and analyses of the supervision objectives and error-prediction correlations, providing useful evidence for the proposed approach.

Critical weaknesses
  1. Insufficient evidence that expert-specific error predictions improve expert selection. TES supervises only an affinity-weighted aggregate error, which does not uniquely identify individual expert errors. The reported correlations demonstrate token-difficulty prediction but do not directly establish that the learned scores identify better experts or improve routing decisions beyond auxiliary regularization.

  2. Limited and inconsistent empirical improvements. Gains are concentrated on Granite, while ACS generally fails to improve OLMoE. The reported average TES improvement is also strongly influenced by MedMCQA, where the baseline exhibits high variance. Moreover, evaluation uses custom data partitions rather than standard official test splits, limiting external comparability and generalization claims.

Questions for authors

Just list questions in the weakness.

Reviewer_ccB4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a cross-entropy-guided routing framework for sparse Mixture-of-Experts (MoE) language models, aiming to align expert routing decisions with token-level prediction errors. This paper introduces two complementary mechanisms: Token-Error Supervision (TES) and Affinity-Concentration Supervision (ACS). TES employs an additional error prediction head to estimate expert-indexed error scores and adjusts routing affinities accordingly, while ACS directly supervises the concentration of selected-expert affinities without introducing additional parameters. Both mechanisms use Itakura-Saito divergence or exponential negative log-likelihood as auxiliary supervision objectives. Experiments on two MoE backbones and four multiple-choice question-answering benchmarks show improvements on the baselines, particularly on Granite.

Critical strengths
  1. The proposed TES and ACS mechanisms provide two complementary approaches to error-guided routing. TES explicitly incorporates predicted errors into expert selection, while ACS supervises routing concentration without additional inference-time parameters. Both mechanisms preserve the original top-K expert execution budget.
  2. This paper provides mathematical analyses of the proposed objectives, including their gradients, optimization behavior, and the identifiability limitations of expert-indexed error scores. The experiments also investigate supervision strength, routing transformations, error prediction, and supervision depth.
Critical weaknesses
  1. The effectiveness of expert-specific error estimation is not sufficiently demonstrated. TES predicts an error score for each expert, but its supervision is derived from the aggregate token-level loss of the complete routed model. As acknowledged in Appendix A.2, this aggregate objective does not uniquely identify the predictive error associated with an individual expert. Consequently, it remains unclear whether the learned expert-indexed scores accurately reflect differences in expert capabilities. Although the reported error-prediction correlations provide some supporting evidence, they primarily evaluate aggregated predictions rather than the quality of individual expert rankings.
  2. The improvements over simpler router fine-tuning baselines are not convincing. The paper primarily compares TES with the Dual Affinity baseline. However, Table 1 shows that TES-IS does not consistently outperform the simpler CE + affinity tuning baseline. For example, on Granite, TES-IS achieves 66.40%, 73.00%, and 88.33% on ARC-Challenge, OpenBookQA, and SciQ, respectively, whereas CE + affinity tuning achieves 67.53%, 74.00%, and 88.60%.
  3. The source of the improvements from ACS remains unclear. The proposed ACS objectives allow gradients to propagate through both affinity concentration and the observed token cross-entropy. Therefore, performance improvements may arise from additional loss-gradient modulation rather than improved routing decisions.
  4. The experimental improvements are inconsistent across models and datasets. The proposed approaches achieve encouraging results on Granite, but their advantages are considerably weaker on OLMoE. For example, frozen-affinity ACS-IS underperforms the CE baseline on ARC-Challenge, OpenBookQA, and MedMCQA. TES also produces mixed results across benchmarks. These observations suggest that the effectiveness of the proposed supervision mechanisms may depend strongly on the model architecture and task characteristics.
  5. The paper’s writing needs improvement. The main text of this paper does not fill 9 pages; important experimental results, methods, and theories from the appendix could be considered for inclusion in the main paper. The formatting of figures and tables could also be refined; Figure 2 and Table 3 should be placed on the same page as the accompanying text whenever possible.
Questions for authors
  1. Since the TES error head is supervised only through the aggregate token-level loss, how can we verify that its expert-indexed scores provide meaningful expert rankings?
  2. For ACS, could you provide ablations that separate gradients through affinity concentration from gradients through the observed token loss?
Reviewer_EZMAOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a cross-entropy-guided routing framework for sparse Mixture-of-Experts (MoE) language models, aiming to better align routing decisions with token-level prediction errors. The authors introduce two supervision mechanisms: Token-Error Supervision (TES), which predicts expert-specific error scores to adjust routing affinities, and Affinity-Concentration Supervision (ACS), which directly aligns router affinities with token-level cross-entropy without introducing additional parameters. Both methods employ Itakura–Saito divergence or exponential negative log-likelihood objectives while preserving the sparse expert execution budget. Experiments on two MoE backbones and four multiple-choice QA benchmarks demonstrate performance improvements, particularly on Granite.

Critical strengths

The paper addresses an important problem in MoE routing by explicitly incorporating token-level prediction errors into expert selection.The proposed TES and ACS methods are lightweight and compatible with existing sparse MoE architectures while preserving the active expert budget.Experiments on two MoE backbones and four QA benchmarks demonstrate the effectiveness of the proposed methods, with notable improvements on Granite.

Critical weaknesses

W1: The general motivation of aligning routing decisions with expert capabilities has already been explored. The main novelty lies in the particular cross-entropy-based supervision rather than identifying a fundamentally new problem.

W2: The aggregate token-level loss does not uniquely supervise individual expert errors, and the gains are less consistent across backbones, raising questions about the proposed mechanism and its robustness.

Questions for authors

1.Could the authors provide further empirical evidence that the predicted expert-specific errors reflect actual expert performance and lead to more effective routing decisions?

  1. Could the authors further clarify the key conceptual and methodological differences between the proposed cross-entropy-guided supervision and existing capability-aware or loss-aware routing approaches, particularly regarding the benefits of explicitly incorporating token-level prediction errors?

  2. Generalization and Robustness Across Models and Tasks. The proposed methods achieve noticeable improvements on Granite, while the gains on OLMoE are relatively limited or inconsistent. Could the authors explain these differences and provide additional evidence of effectiveness across different MoE architectures or more diverse tasks, such as open-ended generation and mathematical reasoning?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.