- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Manifold Power Iteration (MPI) to improve MoE routing by aligning router weights with the principal singular directions of expert weights. MPI uses a "Power-then-Retract" approach to achieve this alignment efficiently while controlling router norms. Experiments on 1B–11B MoE models show improved convergence, downstream performance, and load balancing.
Critical strengths
-
MPI uses a single power iteration to avoid expensive SVD, followed by retraction for stable optimization. The method is simple and effective, achieving the intended router–expert alignment with negligible overhead.
-
The experiments cover different optimizers and model scales (1B–11B), showing consistent improvements in convergence and downstream performance. Ablation studies further support the effectiveness of MPI.
Critical weaknesses
-
The authors justify the principal singular direction by its optimal one-dimensional representation property. However, it remains unclear why this direction is particularly suitable for token routing. Further justification or comparison with alternative expert representations would strengthen the motivation.
-
In Section 5.2.1, the authors report training collapse when removing retraction under AdamW and Muon. However, no specific cases or training curves are provided in either the main paper or the appendix. Showing these results would better support the claimed importance of retraction for training stability.
Questions for authors
-
Could the authors further explain why the principal singular direction is particularly suitable for token routing, compared with other possible expert representations?
-
Could the authors provide the training curves or specific cases of training collapse mentioned in Section 5.2.1 to better demonstrate the importance of retraction?
3 comments
Summary of Responses and Clarifications
Dear Reviewer 9GC7,
Thank you for your time and effort during the rebuttal phase. As the discussion deadline approaches, we provide a brief summary of our responses for your quick reference.
We sincerely appreciate your constructive feedback and hope our clarifications address the questions raised.
- Further Explanation of the Principal Singular Direction for Token Routing
We further explain why the principal singular direction is suitable for token routing. Specifically, its alignment with an input provides a meaningful measure of input–expert affinity, as it captures the expert's strongest input-response mode. We hope this helps readers better understand our core assumption.
- Clarification on Training Collapse without Retraction
We clarify that training collapse was observed in our preliminary experiments without the retraction step. Since the collapse occurred at an early training stage, we terminated the runs and did not retain the corresponding experimental logs.
Thank you again for your thoughtful review and constructive feedback.
Sincerely,
Authors of Submission 12505
Thank you for the clarifications. The authors' responses have partially addressed my concerns, particularly regarding the motivation for using the principal singular direction. Although some concerns remain, I appreciate the additional explanations and will maintain my positive rating.
Rebuttal by Authors
We sincerely appreciate the reviewer's constructive comments and hope our clarifications have adequately addressed the concerns raised.
We thank the reviewer for this insightful question. As discussed in the paper, the principal singular direction captures the matrix’s dominant structure and underlies its optimal rank-one approximation, making it a natural choice for a compact expert representation.
Below, we further explain why this direction is suitable for token routing. We contend that alignment between an input and this direction provides a meaningful measure of the input’s affinity with an expert. Specifically, this direction maximizes the L2 norm of the expert’s pre-activation output for a unit-norm input, capturing its strongest input-response mode. We interpret a stronger response as suggesting that the input more effectively engages the knowledge encoded in the expert’s parameters, reflecting a closer token–expert match. This connection supports using this direction as an expert-specific reference for routing.
We hope this explanation further clarifies the rationale behind our choice. A more detailed discussion is provided in Appendix B.4 of the revised manuscript. Hope this can address your concerns.
We thank the reviewer for this suggestion. During our preliminary experiments exploring the proposed method, we observed training collapse when the retraction step was omitted and terminated the runs early. Since the collapse occurred at a early stage of training, we did not retain the corresponding experimental logs consequently.