- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
MoE routers does not have any direct loss signal to learn the token routing, it only learns indirectly from the overall language-modeling loss. The paper provides a post-training method to utilize the next-token loss (cross-entropy, CE) as a direct teaching signal for the router per token.
- The paper proposes to use two methods (TES which introduce an additional error head ) and ACS without any extra parameters addition, to tune the MoE router so that it gains direct signal from the per token CE loss realized at the end of the model layer.
- The paper mainly keeps the base model params frozen, except the affinity tuning case where base router is also updated, and trains Lora adapters to adapt the base model router.
- The experiments include two base models (Granite 3.1 and OLMoE) and they are trained on the train split of four tasks datasets (ARC-Challenge, OpenBookQA SciQ, MedMCQA) and are evaluated on their test split.
Critical strengths
None
Critical weaknesses
-
The paper does not provide a broader scope of the proposed method in the lifecycle of an LLM models. In general, the LLM models are trained to perform well on multiple tasks in the post-training phase. But this work, fine-tunes the models on each task at a time and provides its results separately which is not a realistic approach. The effects should be observed when fine-tuned once on all combined dataset. Additionally, in case of TES, there is additional head added which changes the native topk selection. There is no ablations on the effect of this change on the base accuracy of tasks which are not included in this fine-tuning. Similarly, in case of ACS, even though there is no explicit head but it changes the routing behavior of the base model. But there is no comparison with the original accuracy of the base model before this fine-tuning was done to understand the effect of this approach for the tasks/basic capabilities not included as part of the fine-tuning.
-
The results are very marginal (table 1). TES method: The Paper says “ACS-IS improves average accuracy by 2.1 points and TES-IS improves over Dual Affinity by 2.3 points on Granite and 0.5 on OLMoE, averaged across all four benchmarks”. But in table 1, Granite - TES-IS vs Dual Affinity (MedMCQA gain (+6.27) drives most of the TES average, and the baseline there is very noisy (37.40 ± 7.79 over 3 seeds). Excluding it, the average gain is about +0.9 which is very marginal. SciQ is near saturation and the TES-IS is either flat or negative there. On OLMoE, the TES-IS gain is very marginal as well (0.5). This results indicate that the method is not adding benefit universally on all models and the results are not significantly better where it’s adding value. ACS Method: - The paper claims +2.1 points but that is only for the case of Granite base model with a frozen router. With the router trainable case, the gains on Granite shrinks to +0.7, and it is concentrated in ARC-C. On OLMoE, ACS-IS is slightly worse on average in both settings. And On Granite, ENLL variation gives gains about as large as IS. And similarly on OLMoE, all three ENLL setups are flat or slightly negative on average.
-
Overall, the models tested here are smaller in size, the eval tasks are multiple-choice only, and the dataset is tiny. Hence, the experimental set up is very narrow scoped.
-
In Fig 2, the predicted error seems to be correlated with actual NLL. But eventually, that behavior doesn't turn into any significant accuracy gains on the tasks evaluated here. Hence, supervising the router with token loss while keeping top-K execution unchanged, is a decent direction to explore but the results are not promising.
Questions for authors
NA