- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies how the router in a pretrained MoE changes during downstream fine-tuning. The authors argue that imbalanced expert usage during fine-tuning is not bad (as the fine-tuning data may just be suited to that expert). However, this imbalance can create its own problem caused by gradient updates. Experts that are selected more frequently receive more direct router-gradient updates, allowing them to have a disproportionate effect on routing boundaries and causing the pretrained routing structure to change. Standard load-balancing losses address the imbalance itself, but can interfere with useful specialization rather than addressing the effect that unequal exposure has on router updates.
The paper proposes Gradient Rebalancing (GR), which estimates how the gradient contributions received by each expert tend to expand or contract that expert’s routing region. It then dampens whichever direction dominates so that the expert’s own update has approximately zero net "tendency" to expand or contract its region. Importantly, this does not force expert utilization to become uniform or freeze the router, so routing can still adapt during fine-tuning. Across several pretrained MoEs and downstream tasks, GR generally improves downstream performance while producing substantially less routing drift than standard fine-tuning and other routing-preservation methods. The authors also show that the method adds relatively little training overhead and that simply scaling gradients according to expert exposure or update magnitude does not produce the same improvements.
Critical strengths
- They identify an interesting phenomenon: They show that imbalanced expert exposure during fine-tuning can itself drive routing changes, as frequently selected experts receive more direct gradient signals and therefore have a greater influence on how routing space changes.
- The validation shows that fixing this phenomenon actually helps. Rebalancing these gradient contributions reduces routing drift while improving downstream performance.:
Critical weaknesses
- GR depends on an approximate measure of routing-region change. The method uses a batch-level smooth approximation of the Top-k boundary and then reasons about its first-order change. More evidence that this quantity actually tracks the real routing changes during training would make the proposed mechanism more convincing.
Questions for authors
- When is GR actually necessary? The Commonsense-15K experiment suggests that its benefit depends on the degree of expert imbalance. Can the authors characterize the level of load imbalance or exposure asymmetry at which GR becomes beneficial, and could this be used to decide when to apply GR?
Not a question but the related work is missing relevant prior work on understanding router behavior during MoE fine-tuning and adaptation: Thérien, Benjamin, et al. "Continual Pre-training of MoEs: How robust is your router?." arXiv preprint arXiv:2503.05029 (2025)
2 comments
Thanks for clarifying on the my points. I will maintain my original score for this paper.
Dear Reviewer wpKQ, thank you for highlighting our analysis of exposure-induced routing drift and recognizing GR’s effectiveness. We appreciate your constructive comments and address your concerns below.
Weakness 1: Insufficient validation of the routing-region change estimate
Conclusion:
Our single-step experiment supports the estimate’s ability to predict hard Top-K routing changes, with 96.9% sign agreement under self-only updates and 89.1% order agreement under joint updates.
Explanation:
We evaluated 216 step–layer–batch combinations from a DeepSeek-V2-Lite FT w/o LB trajectory on Math14K. We estimated self-region tendency from training batches and replayed AdamW updates using fixed representations of 61,957 held-out tokens, isolating router updates from representation changes. Joint updates all router rows together; Self-only updates each row separately.
We have added and highlighted this experiment in the revised main text, alongside a comparison showing that GR reduces single-step load TV by 78.7% relative to norm-matched FT.
Question 1: When to apply GR based on the degree of exposure imbalance
Conclusion:
Our results suggest that GR is particularly beneficial under pronounced expert load imbalance, and future work could establish a quantitative criterion for when to apply it across downstream tasks.
Explanation:
We have not yet established a precise threshold for applying GR, given the diversity of downstream tasks. Nevertheless, our Commonsense-15K experiment shows that most fine-tuning methods perform Comparably in a relatively balanced setting. Given GR's low computational overhead, we encourage evaluating it on downstream tasks, especially those exhibiting exposure imbalance. We could examine how the benefits of GR vary with the degree of imbalance across tasks and datasets, as well as what other factors influence its effectiveness in our future work. We have added a discussion of our contributions and limitations to the main text.
Regarding the work you mentioned [1] , we have read the paper and found its analysis of layer-wise routing changes and their association with forgetting to be informative and relevant to our study. We have cited, discussed and highlighted this paper in the Related Work section of our revised manuscript.
We would appreciate your feedback on whether our responses have addressed your concerns. And we would be happy to clarify any remaining points.
[1] Continual Pre-training of MoEs: How robust is your router? TMLR, 2025.