acceptodds
Under review as a conference paper at ICLR 2027

Redesign Mixture-of-Experts Routers with Manifold Power Iteration

Abstract

The router is a cornerstone component of Mixture-of-Experts models. Each row of the router matrix represents an expert, computing its similarity to the MoE input to determine which subset of experts is activated. Ideally, each router row should condense the salient characteristics of its associated expert matrix into a representative vector, such that its dot product can better indicate whether the token should be routed to that expert. However, there is no explicit design principle in existing routers to enforce such a condensation. In this paper, we posit that each router row should be aligned with the principal singular direction of its associated expert matrix, which provides the most expressive one-dimensional representation of the expert matrix. Building on this insight, we propose a router redesign with Manifold Power Iteration (MPI). Specifically, it adopts a “Power-then-Retract” paradigm, in which the power iteration step steers the router weights toward the desired directions, while the retraction enforces a norm constraint to ensure efficient and stable optimization. Our theoretical analysis further confirms that the proposed router design indeed converges toward the target directions. We pretrain MoE models ranging from 1B to 11B parameters and conduct comprehensive experiments to demonstrate the effectiveness of our router design.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-Experts, MoE Router

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9GC7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Manifold Power Iteration (MPI) to improve MoE routing by aligning router weights with the principal singular directions of expert weights. MPI uses a "Power-then-Retract" approach to achieve this alignment efficiently while controlling router norms. Experiments on 1B–11B MoE models show improved convergence, downstream performance, and load balancing.

Critical strengths
  1. MPI uses a single power iteration to avoid expensive SVD, followed by retraction for stable optimization. The method is simple and effective, achieving the intended router–expert alignment with negligible overhead.

  2. The experiments cover different optimizers and model scales (1B–11B), showing consistent improvements in convergence and downstream performance. Ablation studies further support the effectiveness of MPI.

Critical weaknesses
  1. The authors justify the principal singular direction by its optimal one-dimensional representation property. However, it remains unclear why this direction is particularly suitable for token routing. Further justification or comparison with alternative expert representations would strengthen the motivation.

  2. In Section 5.2.1, the authors report training collapse when removing retraction under AdamW and Muon. However, no specific cases or training curves are provided in either the main paper or the appendix. Showing these results would better support the claimed importance of retraction for training stability.

Questions for authors
  1. Could the authors further explain why the principal singular direction is particularly suitable for token routing, compared with other possible expert representations?

  2. Could the authors provide the training curves or specific cases of training collapse mentioned in Section 5.2.1 to better demonstrate the importance of retraction?

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We sincerely appreciate the reviewer's constructive comments and hope our clarifications have adequately addressed the concerns raised.


W1 & Q1: Further Explanation of the Principal Singular Direction for Token Routing

We thank the reviewer for this insightful question. As discussed in the paper, the principal singular direction captures the matrix’s dominant structure and underlies its optimal rank-one approximation, making it a natural choice for a compact expert representation.

Below, we further explain why this direction is suitable for token routing. We contend that alignment between an input and this direction provides a meaningful measure of the input’s affinity with an expert. Specifically, this direction maximizes the L2 norm of the expert’s pre-activation output for a unit-norm input, capturing its strongest input-response mode. We interpret a stronger response as suggesting that the input more effectively engages the knowledge encoded in the expert’s parameters, reflecting a closer token–expert match. This connection supports using this direction as an expert-specific reference for routing.

We hope this explanation further clarifies the rationale behind our choice. A more detailed discussion is provided in Appendix B.4 of the revised manuscript. Hope this can address your concerns.


W2 & Q2: Additional Results of Training Collapse without Retraction:

We thank the reviewer for this suggestion. During our preliminary experiments exploring the proposed method, we observed training collapse when the retraction step was omitted and terminated the runs early. Since the collapse occurred at a early stage of training, we did not retain the corresponding experimental logs consequently.

AuthorsOpenReview ↗

Summary of Responses and Clarifications

Dear Reviewer 9GC7,

Thank you for your time and effort during the rebuttal phase. As the discussion deadline approaches, we provide a brief summary of our responses for your quick reference.

We sincerely appreciate your constructive feedback and hope our clarifications address the questions raised.


  • Further Explanation of the Principal Singular Direction for Token Routing

We further explain why the principal singular direction is suitable for token routing. Specifically, its alignment with an input provides a meaningful measure of input–expert affinity, as it captures the expert's strongest input-response mode. We hope this helps readers better understand our core assumption.

  • Clarification on Training Collapse without Retraction

We clarify that training collapse was observed in our preliminary experiments without the retraction step. Since the collapse occurred at an early training stage, we terminated the runs and did not retain the corresponding experimental logs.


Thank you again for your thoughtful review and constructive feedback.

Sincerely,

Authors of Submission 12505

Thank you for the clarifications. The authors' responses have partially addressed my concerns, particularly regarding the motivation for using the principal singular direction. Although some concerns remain, I appreciate the additional explanations and will maintain my positive rating.

Reviewer_pm6XOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Manifold Power Iteration (MPI), which aligns MoE router weights with the principal singular directions of their corresponding expert weight matrices through a single power iteration step followed by normalization. Experiments on MoE models ranging from 1B to 11B parameters demonstrate improvements in pretraining convergence, downstream performance, and load balancing.

Critical strengths
  • The proposed method is simple and efficient. MPI introduces a lightweight router modification without additional trainable parameters or inference overhead, with only a 0.2% training throughput reduction reported at the 11B scale.
  • Authors conduct pretraining experiments across multiple model scales and optimizers, demonstrating generally improved convergence and downstream performance.
Critical weaknesses
  • The central assumption that an expert's principal singular direction provides a suitable routing representation is not well justified. MoE routing can be viewed as token clustering [1, 2], but the direction of maximal response of an expert's gate projection matrix does not necessarily correspond to the token distribution or discriminative features associated with that expert. Furthermore, Section 5.1 reports that increasing power iterations from 1 to 10 degrades performance, although additional iterations should theoretically yield a more accurate approximation of the principal singular direction for fixed expert weights. This raises questions about the connection between spectral alignment and routing quality, which is central to the proposed method.
  • Table 4 demonstrates improved spectral alignment, but this is largely expected from the proposed operation and does not explain the downstream improvements. Moreover, an expert's behavior is jointly determined by Wg,Wp,WoW_g, W_p, W_o, and nonlinear activations, making it unclear why the principal singular direction of WgW_g alone should represent the expert. The empirical evidence also leaves some uncertainty. Improvements in Figure 2 are relatively modest, and the 11B downstream gains in Table 3 are substantially influenced by GSM8K. Additional analyses and standard zero-shot evaluations would help substantiate the proposed mechanism and the generality of its benefits.

[1] Zixiang Chen et al., “Towards Understanding the Mixture-of-Experts Layer in Deep Learning”, NeurIPS, 2022.

[2] Zewen Chi et al., “On the Representation Collapse of Sparse Mixture of Experts”, NeurIPS, 2022.

Questions for authors
  • Can the authors provide empirical evidence connecting the principal singular directions of expert weights to the representations or specialization of their routed tokens? In addition, how does the spectral alignment change when increasing power iterations from 1 to 10, given the reported performance degradation?
  • Why is WgW_g chosen to represent each expert rather than other expert weight matrices? Would alternative choices produce similar improvements?
  • Could the authors provide zero-shot evaluations on standard language modeling benchmarks (e.g., HellaSwag, PIQA, WinoGrande, and ARC) to complement the few-shot results in Table 3? This would help assess whether the improvements generalize across evaluation protocols, particularly given the large contribution of GSM8K to the 11B average improvement.
4 comments
AuthorsOpenReview ↗

Rebuttal By Authors (1)

We thank the reviewer for the constructive feedback and hope our responses address your concerns. If so, we would be grateful if you could reconsider your overall recommendation.


W1.1 & Q1.1: Further explanation for using the Principal Singular Direction for Routing

We thank the reviewer for this insightful question and would like to further clarify why the principal singular direction is a suitable representation for routing.

First, we want to clarify that we do not assume that the principal singular vector directly represents an expert's token distribution. Instead, we use it to characterize the affinity between an input token and an expert's parameters.

As the reviewer also pointed out, this direction maximizes the L2 norm of the expert's projected output under a unit L2-norm constraint on the input. From a neuron-level perspective, this corresponds to the strongest aggregate pre-activation response across the expert's neurons, which can lead to stronger activation of the parametric knowledge encoded in the expert. We argue that this provides a better indication of how well an input matches an expert, making it beneficial for routing.

We have added a more detailed discussion in Appendix B.4 of the revised manuscript. Thanks again for raising this insightful question.


W1.2 & Q1.2: Explanation of performance degradation with more Power Iterations

While more power iterations theoretically improve spectral alignment for fixed expert weights, this benefit must be balanced against the need for stable router updates. This is particularly important in MoE, where router updates affect both routing scores and expert assignments, thereby changing the active computation paths.

To investigate this trade-off, we reran the experiments with one and ten power iterations and logged the step-to-step changes in the effective routing directions. Specifically, we measure the Angular Update Size (AUS) between the effective routing weights folded with power iteration at consecutive training steps:

\mathrm{AUS}^{(t)} = \frac{1}{N}\sum_{i=1}^{N}\left\\|\mathbf{R}_{i}^{\prime(t)} - \mathbf{R}_{i}^{\prime(t-1)}\right\\|_2

We found that ten iterations resulted in larger AUS than one iteration (Figure 5 in revised manuscript), indicating greater directional changes that may alter token–expert assignments and destabilize routing. This provides a possible explanation for the performance degradation: the resulting instability may outweigh the benefits of improved spectral alignment.

We therefore adopt one power iteration per training step, balancing spectral alignment, routing stability, and computational cost. As discussed in Section 3.3, this trade-off is consistent with our principle-of-least-action formulation rather than contradicting its underlying premise.

Hope this can address your concerns.


W2.2 & Q1: Clarification of zero-shot evaluations and gains beyond GSM8K

We want to clarify that the zero-shot evaluations requested by the reviewer are already reported in Figure 2(b).

During pretraining, we follow the OLMoE evaluation protocol (Table 11 in [1]) to evaluate intermediate checkpoints using the completion formulation. All benchmarks except MMLU are evaluated zero-shot by scoring candidate answers based on their conditional likelihood and selecting the highest-scoring answer, while MMLU uses a 5-shot multiple-choice formulation. Our nine benchmarks include all four tasks mentioned by the reviewer: HellaSwag, PIQA, WinoGrande, and ARC. We have further clarified the evaluation setup in Appendix B.3.

Figure 2(b) reports the average performance across these nine benchmarks at every 10B training-token interval. Despite some fluctuations, our method achieves an approximately 1-point absolute gain (around 2% relative improvement). This demonstrates consistent benefits beyond the few-shot downstream evaluations in Table 3. We consider this a meaningful improvement, particularly as it is achieved under a standard and rigorous overtraining setup and accompanied by improved training loss.

We also report the aggregate results in Table 3 after excluding GSM8K. The improvements remain consistent, indicating that the overall gains are not solely driven by GSM8K.

ModelARC-CMMLUTriviaQANaturalQsBBHMBPPAVG.
MoE-3B55.9147.0145.7817.8729.5342.2539.73
MoE-3B w. \name58.9648.8346.5220.1330.9944.5441.66
MoE-11B61.5450.0055.4125.3031.1745.1244.76
MoE-11B w. \name62.2450.9356.8925.3631.4544.8745.29

We also note that GSM8K itself is a challenging benchmark for the base model, making the improvement on this benchmark non-trivial. Together with the zero-shot results in Figure 2(b), these results provide a more comprehensive assessment of our approach.

[1] OLMoE: Open Mixture-of-Experts Language Models

AuthorsOpenReview ↗

Rebuttal by Authors (2)


Q2: Choice of WgW_g for Expert Representation

We thank the reviewer for this insightful question. We choose WgW_g because it controls the expert's input-dependent nonlinear gating, potentially providing a more informative signal for token–expert matching.

Following the reviewer's suggestion, we compare the three candidate matrices WgW_g, WpW_p, and WoW_o by pretraining 1B MoE baselines for 50B tokens, using each for power iteration. We observe no substantial divergence in pretraining loss or downstream performance across these choices. In particular, using WpW_p and WoW_o leads to increases in pretraining loss of approximately 0.001 and 0.003, respectively These results suggest that alternative matrices also work, while WgW_g achieves the best training loss among the tested choices.

AuthorsOpenReview ↗

Summary of Responses and Clarifications

Dear Reviewer pm6X,

Thank you for your time and effort during the rebuttal phase. As the discussion deadline is less than 24 hours away, we provide a brief summary of our responses for your quick reference.

We would be happy to address any remaining questions and would appreciate your reconsidering your initial assessment in light of our responses.


  • Further Clarification on Choosing the Principal Singular Direction for Routing

We provide a complementary perspective on our original motivation to further clarify why spectral alignment is useful for routing. We argue that the projection of an MoE input onto an expert's principal singular direction can better capture its aggregate pre-activation response, thus providing an indication of input–expert affinity.

We hope this helps readers better understand our core assumption.

  • Explanation of Performance Degradation with more Power Iterations

We provide empirical evidence that more power iterations lead to larger changes in router updates, potentially destabilizing training. We therefore adopt one power iteration per training step to balance spectral alignment, routing stability, and computational cost.

  • Additional Evaluations on Zero-Shot and Downstream Performance

We clarify that the requested evaluations have already been reported. Together with the training-loss improvements under the overtraining setup, these results help demonstrate that the observed gains are non-trivial.

  • Motivation and Empirical Support for Choosing WgW_g

We explain the rationale for choosing WgW_g and provide additional empirical evidence to further support this choice.


Thank you for your time and thoughtful consideration of our submission.

Sincerely,

Authors of Submission 12505

Thank you for the detailed rebuttal and additional analyses. I appreciate the comparisons using different expert weight matrices, the clarification of the evaluation protocol, and the results excluding GSM8K. These address several of my original concerns.

My main concern remains the justification of the proposed routing principle. While the authors provide additional intuition based on pre-activation responses and input-expert affinity, I still find the connection between the principal singular direction and effective token-expert routing insufficiently supported. As noted in my original review, the improved spectral alignment in Table 4 is largely expected from the proposed operation, but does not directly establish its relationship to routing quality or expert specialization.

Regarding the performance degradation with ten power iterations, I appreciate the additional AUS analysis. However, larger step-to-step changes in effective router directions do not necessarily indicate instability in actual token-expert assignments. Moreover, the requested comparison of spectral alignment between one and ten iterations remains missing. Thus, the proposed alignment-stability trade-off remains plausible but insufficiently substantiated.

Although the improvements persist after excluding GSM8K, the 11B average gain decreases from 1.84 to 0.53 percentage points, indicating relatively modest improvements across the remaining benchmarks.

Overall, I recognize the simplicity, efficiency, and empirical improvements of MPI, and appreciate the authors' efforts in addressing my concerns. I find this paper borderline and have carefully considered the merits of its empirical contributions. However, given the paper's emphasis on a principled expert-router design, I remain insufficiently convinced by the justification and supporting analysis of its central routing principle. Therefore, I maintain my original rating of 2.

Reviewer_o7p1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper transforms each router row using the Gram matrix of its associated expert's gate projection, then normalizes it to a prescribed norm. This couples routing to expert weights through one power-iteration step. Experiments span several optimizers and models up to 11B parameters, with improvements in language-modeling loss, downstream accuracy, and expert balance.

Critical strengths

The transformation is simple and can be precomputed at inference, making it potentially practical. The empirical study covers multiple optimizers and substantial pretraining runs. Table 3 reports average gains at both 3B and 11B scales, rather than relying only on an alignment diagnostic. The normalization-only and additional-power-iteration ablations help distinguish spectral coupling from a generic router-norm constraint.

Critical weaknesses

The pseudocode in Figure 1(a) does not implement Equations (4)--(7). With the stated shapes R∈RN×DR\in\mathbb{R}^{N\times D} and Wg∈RN×D×dW_g\in\mathbb{R}^{N\times D\times d}, the product beginning with \texttt{R.unsqueeze(1) @ wg.transpose(1,2)} is incompatible when D≠dD\ne d. The code also uses exponentiation in \texttt{C_prime ** (N ** -0.5)} and \texttt{C ** (x @ R_prime.T)}, where the equations require multiplication. At the shown default C′=1C'=1, the latter gives constant logits. The actual implementation must be provided to establish which computation produced the results.

Questions for authors

Can the authors provide the actual router forward pass, including gradient flow through the transformation and any persistent power updates?

1 comment
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for the thoughtful feedback and hope our responses have addressed your concerns.


W1 & Q1: Clarification on the Pseudocode in Figure 1(a)

We thank the reviewer for carefully pointing out these presentation inaccuracies. We apologize for the slips in the pseudocode in Figure 1(a). These are confined to the presentation and do not reflect our actual implementation. Our implementation fully follows the operations described in Eqs. (4)--(7), and all reported results were obtained using this implementation.

In the revised manuscript, we have corrected Figure 1(a) to ensure consistency with the mathematical formulation. For transparency and reproducibility, we will make the source code of our implementation publicly available upon acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.