acceptodds
Under review as a conference paper at ICLR 2027

Concentration-Scaled Grassmannian Routing for Controllable Mixture-of-Experts

Abstract

A sparse mixture-of-experts model is useful only if its router can specialize experts without starving them and can adapt inference cost after training. Standard top- routers expose the first problem during training but provide only an indirect post-hoc compute interface. We propose Grassmannian MoE (GrMoE), which represents each expert by an orthonormal subspace and scores a normalized token by concentration-scaled projection energy. The resulting affinity is basis invariant and bounded, while a global concentration scale controls the sharpness of the routing distribution from one checkpoint. We distinguish these geometry-specific properties from generic consequences of softmax scaling and give a conditional load-balance bound under explicit mixture-balance and effective-margin assumptions. At 350M parameters, GrMoE improves perplexity and severe-concentration frequency over standard top- routing; after matching descriptor parameters, scoring MACs, diversity regularization, expert compute, and realized fan-out, it retains a smaller -PPL residual over the strongest Euclidean control. A fixed checkpoint reaches two active experts across with only PPL spread, and downstream, /, second-family, expert-parallel, and memory studies preserve the same scoped picture. We therefore present GrMoE as a geometric routing primitive for stable one-checkpoint compute control, not as a universal or frontier-scale routing claim.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mixture of experts, sparse routing, Grassmannian manifolds

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jxpwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a new way of routing,which represents each expert by an orthonormal subspace and scores a normalized token by concentration-scaled projection energy. The authors test it on a small scale 350M, not very sparse MoE, and observe somewhat better performance in load balancing and validation loss.

Critical strengths
  • The paper addresses a timely problem: Better MoE routing.

  • The paper has lots of theory (that I cannot understand), making the method look very theoretically grounded.

Critical weaknesses
  • The experiment is weak: The sparsity is too high, 150M activated out of 350M. Also the model size is too small.

  • The paper needs more baselines: e.g. https://openreview.net/forum?id=iydmH9boLb

  • The writing needs lots of improvement. I cannot understand most parts of the paper.

Questions for authors

Do the authors plan to release the code?

1 comment
AuthorsOpenReview ↗

Response to reviewer jxpw

Thank you for the review. The table summarizes our answers.

Your pointOur answerWhere in the revision
150M active out of 350M is not sparse enoughAttention and embeddings run for every token, so total-model sparsity understates expert sparsity. We also test pools of 64 and 128 experts with 6.3% of experts active.Sections 4.1 and 4.6; Tables 18–19
The model is too smallThe matched control now includes 1.3B and 3B models with 16 experts, where GrMoE keeps gains of 0.12 and 0.08 perplexity.Section 4.2, Table 2
Missing baseline: Guo et al., NeurIPS 2025Reproduced on DeepSeek-MoE-16B with the published recipe, combined with GrMoE in a 1.3B factorial, and added to the common-budget comparison.Sections 4.3 and 4.6; Tables 4, 36, 37
Will the code be released?Yes, upon acceptance.Reproducibility Statement

Sparsity and model scale

The primary model activates about 150M of its 350M parameters because attention, embeddings, and other shared components run for every token. Expert sparsity is a separate quantity, and we test it with larger expert pools.

ModelGrMoE active expertsSoftmax PPLAuxFree PPLGrMoE PPL
352M, 64 experts (Top-4 baselines)4.01 ± 0.0318.71 ± 0.2518.29 ± 0.1717.98 ± 0.11
160M, 128 experts (Top-8 baselines)8.03 ± 0.0523.84 ± 0.2923.31 ± 0.1722.97 ± 0.12

Both settings activate about 6.3% of the expert pool and use three seeds. At 128 experts, router scoring is 8.3% of counted forward MACs, so that setting tests routing quality rather than efficiency. For model size, the matched control at 1.3B and 3B is in our response to Reviewer 52ew (Table 2). The evidence covers models up to 3B parameters, and Section 6 states this limit.

Guo et al. as a baseline

Guo et al. add orthogonality and variance losses when post-training an existing MoE. GrMoE changes a different part of the router, the descriptor that scores an expert, and adds a compute dial after training. The two can therefore be combined. We test the relationship in three ways.

Native reproduction. We use DeepSeek-MoE-16B with its routing intact (https://proceedings.neurips.cc/paper_files/paper/2025/file/4598de7d243d528e38eb0c5d8155fb52-Paper-Conference.pdf): rank-32 LoRA in routers and experts, LoRA α = 128, dropout 0.1, learning rate 10⁻⁴, batch 32, and auxiliary, orthogonality, and variance coefficients of 0.001 each. Each task is fine-tuned separately on 6,000 examples for three epochs, with three post-training runs.

Native post-training of DeepSeek-MoE-16BGSM8K, %MATH500, %Relative GPU time
Auxiliary balancing loss only15.85.51.000
With Guo et al. orthogonality + variance losses34.710.51.028

The losses raise GSM8K from 15.8% to 34.7% and MATH500 from 5.5% to 10.5% at 2.8% more GPU time. This confirms the baseline under its intended recipe. It is not a direct comparison with GrMoE, because replacing the router of a pretrained checkpoint requires an adaptation step, and GrMoE is not a zero-shot retrofit.

Factorial at 1.3B. We pretrain low-rank and GrMoE routers separately, with three paired pretraining runs per router, and post-train each with and without the losses. All four cells share task examples and post-training budgets and are calibrated to two active experts. Each cell receives the same tuning allowance of four pilots, and a separate calibration set selects settings without test accuracy.

1.3B post-training cellGSM8K, %MATH500, %Active experts
Low-rank + span diversity, without the losses24.06.62.00
Low-rank + span diversity, with the losses26.27.42.00
GrMoE, without the losses25.37.02.00
GrMoE, with the losses26.57.52.00

The losses help both routers. GrMoE with the losses is 0.3 points above the low-rank router with the losses on GSM8K and 0.1 points above it on MATH500. The losses also lift the low-rank router above GrMoE without them. The two methods are therefore compatible, and the remaining GrMoE gain is small. These are point estimates from one post-training run per pretrained checkpoint, so they do not establish significance.

Common budget at 350M. Adapted to from-scratch Top-2 training, the losses give 17.98 perplexity, the closest baseline to GrMoE with its amortizer at 17.82 (Table 4).

Will the code be released?

Yes. The code will be released upon acceptance.

Please let us know whether these results answer your questions.

Reviewer_3DQbOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes GrMoE, a MoE routing method that represents experts as orthonormal subspaces and computes routing scores using concentration-scaled projection energies. It employs subspace diversity regularization and threshold-based routing with adjustable concentration to improve expert utilization and enable post-training compute control. Experiments report improvements in perplexity, load balancing, and inference efficiency.

Critical strengths
  • The paper addresses relevant problems in MoE routing, particularly expert starvation and flexible inference computation.

  • The experiments are relatively extensive.

Critical weaknesses
  • Poor presentation and writing quality. The manuscript is difficult to follow, with numerous awkward and unnecessarily convoluted expressions. More concerningly, obvious revision artifacts, such as "The revision is explicit..." and "the corrected N = 64 study," remain in the text. The writing gives the impression of being heavily LLM-assisted without sufficient human revision. Overall, the presentation lacks the clarity and care expected of a conference submission.

  • Insufficient motivation for the proposed method. The paper does not convincingly explain why MoE experts should be represented by orthonormal subspaces or why the specific properties of Grassmannian geometry would help address expert starvation and post-training compute control. The introduction largely assumes that subspace representations are more appropriate than routing vectors, without sufficient analysis or empirical observations to motivate this assumption. The necessity of the proposed geometric design remains unclear.

  • Questionable fairness of the main comparisons. Table 2 does not report the average number of activated experts or computational costs for each baseline. For reference, Table 6 reports 3.1 active experts for one GrMoE checkpoint at α = 1, compared with exactly two for Softmax Top-2. It is also unclear whether Softmax Top-2 uses a standard auxiliary load-balancing loss. These omissions make it difficult to attribute the reported improvements to better routing rather than differences in active computation or baseline configurations.

Questions for authors
  • What concrete observations motivate the Grassmannian parameterization?

  • What are the average activated expert counts and computational costs of all Table 2 baselines? Does Softmax Top-2 use an auxiliary load-balancing loss?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 3DQb – Part 1

Thank you for the careful review. You asked about the motivation, the active compute of the baselines, and the Top-2 balancing loss. The table summarizes our answers.

Your pointOur answerWhere in the revision
Revision artifacts in the textRemoved. The main text is rewritten.Sections 1–6
Motivation for orthonormal subspacesTwo measurements. One direction per expert is not enough, and extra router parameters do not explain the gain. Orthonormality keeps the score in [0, 1], so the dial acts on a fixed range.Section 1; Tables 1 and 34
Active experts and compute of the Table 2 baselinesYou are right that the submitted Table 2 (now Table 3) mixes compute budgets. We reran every router at a common budget and report calls per token, FFN work, MACs, and latency in every row.Section 4.3, Table 4
Auxiliary loss for Softmax Top-2Yes. The ten-seed runs and the common-budget rerun both use the standard load-balancing loss with coefficient 0.01.Table 3 caption; Section 4.3

What concrete observations motivate the design?

A single routing vector measures one signed projection of the token. An expert that serves tokens along several directions therefore needs more than one key. Two measurements from the 350M experiments show that this matters.

  1. One direction per expert is not enough, and more dimensions are not always better. Rank-1 routers reach 18.4 and 18.8 perplexity, rank-48 subspaces reach 17.8, and rank 96 worsens again to 18.3 (Table 34).
  2. Extra router parameters do not explain the gain. At equal descriptor parameters, scoring MACs, and diversity regularization, multi-key, unconstrained low-rank, and small-MLP routers reach 17.96 to 18.11 perplexity, while orthonormal subspaces reach 17.82 (Table 1).

The orthonormal constraint gives the score two properties that a free low-rank map lacks. The projection energy ae(x)=∥Ue⊤x∥2=x⊤UeUe⊤xa_e(x)=\Vert U_e^\top x\Vert^2=x^\top U_eU_e^\top x does not change when the basis is rotated inside the subspace. For a unit-norm token it also lies between 0 and 1, because repeated or rescaled basis vectors cannot inflate it. The dial α therefore always acts on the same bounded range. The overlap penalty matters as well. Removing it raises load CV from 0.075 to 0.168 and brings back severe concentration in 2 of 10 seeds (Table 6). A synthetic task with known subspaces tests the mechanism, but it does not show that language representations have this structure.

Please see the second part of our response for the compute of every baseline and the Top-2 balancing loss.

What are the active experts and compute of every Table 2 baseline?

The submitted Table 2, now Table 3, compares routers in their native configurations, so their compute differs. At α = 1, GrMoE with the amortizer activates 3.1 experts per token, the value you noted in the submitted Table 6 (now Table 13). Softmax Top-2 activates two. The 17.8 versus 18.6 gap in that table therefore mixes routing with compute, and its revised caption says so. Table 3 does not report the realized fan-out of every baseline. To answer your question, we reran all routers at a common expert-compute budget, and every row of Table 4 now reports its active compute. Work, MACs, and latency are relative to tuned Softmax Top-2.

RouterCalls per token (train / test)Expert FFN workForward MACsLatencyPPL
Softmax Top-2, tuned balancing loss2.00 / 2.001.0001.0001.00018.25
Switch Top-1, native expert width1.00 / 1.000.5000.7200.80018.62
Switch Top-1, doubled expert width1.00 / 1.001.0001.0001.01818.48
Expert Choice, matched slots2.00 / 2.00 average1.0001.0011.01218.20
Hash, two-expert dispatch2.00 / 2.001.0000.9990.99619.62
Soft MoE, matched physical slotsslot-based1.0001.0061.02818.08
DeepSeekMoE-style, routed + shared4 routed + 1 shared1.0001.0031.02418.02
AuxFree Top-22.00 / 2.001.0001.0000.99818.10
Top-2 + Guo et al. losses2.00 / 2.001.0001.0011.00217.98
GrMoE, two-active calibration2.02 / 2.001.0001.0041.00617.94
GrMoE + amortizer, two-active calibration2.02 / 2.001.0001.0051.00817.82
AuthorsOpenReview ↗

Response to Reviewer 3DQb – Part 2

All rows share the 350M backbone, tokenizer, 15B-token mixture, sequence length 1,024, global batch 256, AdamW schedule, and capacity factor 1.25. Values are point estimates over three training runs, and latency uses five timed launches. Dynamic routers are calibrated to two active experts on a separate 10M-token split without validation perplexity, and cumulative expert FFN MACs differ by at most 1%. Doubled-width Switch matches compute but has more expert parameters. The DeepSeekMoE-style row uses four quarter-width routed experts plus one full-width shared expert, which equals two reference FFNs. Latency covers the whole forward pass, including dispatch, at batch 16 and sequence length 1,024 on one A100.

GrMoE with its amortizer reaches 17.82. The closest other routers are Top-2 with the Guo et al. losses at 17.98, DeepSeekMoE-style routing at 18.02, and Soft MoE at 18.08, and most margins are smaller than in Table 3. Native Switch remains cheaper, at half the FFN work and 18.62 perplexity, so GrMoE does not dominate it on cost. The tuning searches also differ in cost, from 15B training tokens for fixed configurations to 23B when both searches run, so Table 4 does not compare total training cost. Appendix R gives the full protocol.

Does Softmax Top-2 use an auxiliary load-balancing loss?

Yes. Softmax Top-2 uses the standard auxiliary load-balancing loss with coefficient 0.01, both in the ten-seed runs of Table 3 and in the common-budget rerun of Table 4. The submission did not state this, and the Table 3 caption now does. For the rerun, a search over {0.001, 0.003, 0.01, 0.03}, with one 1B-token pilot per value followed by a fresh 15B-token run, selected the same coefficient. The Guo et al. losses receive a search of the same size over four weight pairs, Hash has no learned balancing loss, and AuxFree keeps its native bias update.

The two Top-2 results therefore share one loss setting. The ten-seed mean of 18.6 ± 0.4 includes the four severely concentrated seeds, which finish 1.01 perplexity worse than the other six, and the three rerun runs give 18.25. GrMoE with its amortizer is ahead in both tables, at 17.8 and 17.82.

Presentation

The revised paper defines the score, the compute dial, and the dispatch rule first (Section 2). The Bingham family now appears after the implemented router, as motivation for the concentration parameters only (Section 2.5). The assumptions of the balance theorem are stated before the theorem (Section 3), and the matched comparison is the first experimental result (Section 4.2).

Please let us know whether these results answer your questions.

I thank the authors for their detailed response. Some of my concerns have been addressed. However, I still find the paper difficult to follow, and I think the writing and presentation need significant improvement. I will therefore raise my rating to 2, since I do not feel that the paper currently meets the bar for a score of 3.

Reviewer_52ewOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes GrMoE, which represents each expert as a subspace with an orthonormal basis. Routing scores are computed from the projection energy of normalized inputs onto each subspace, with concentration parameters controlling the routing distribution. The paper analyzes load balancing under specific assumptions and evaluates subspace routing and its ability to adjust compute budgets. The evaluation includes language modeling experiments, matched comparisons, and measurements of inference overhead.

Critical strengths
  1. The matched comparisons are fairly detailed, accounting for router parameter count, scoring cost, expert compute budget, and diversity regularization. These controls aim to distinguish the benefits of the subspace structure from those of other factors.
  2. Adjusting the compute budget at inference time has practical value, and the paper provides concrete evidence for this capability. Using a fixed checkpoint calibrated to an average of two active experts, the PPL range across three threshold settings is only 0.03. This indicates that model quality after calibration is relatively stable within the tested range.
Critical weaknesses
  1. The comparison with the strongest matched baseline is limited to the 350M model with eight experts. The experiments on larger models and a second architecture do not include this matched control. It therefore remains unclear whether the gains from subspace routing persist in other settings after controlling for differences in parameter count, computation, and regularization.
  2. Some figure captions do not match the content of the figures. Figure 3 on page 21 contains three panels showing training loss, routing entropy, and concentration. Its caption describes two rows of entropy and load CV plots, but no CV panel is shown. Figure 5 on page 22 refers to a 350M model but shows 16 experts, whereas the main configuration uses eight.
Questions for authors
  1. Could the authors extend the matched comparison with the strongest low-rank Euclidean router to a larger model to assess whether the gains in Table 8 persist?
  2. Please check the captions of Figures 3 and 5, and shorten repeated qualifications of the conclusions and statements about manuscript revisions in the main text.
1 comment
AuthorsOpenReview ↗

Response to reviewer 52ew

Thank you for the review and for checking the appendix figures closely. The table summarizes our answers, and the direct answers follow.

Your pointOur answerWhere in the revision
The matched comparison exists only at 350M with 8 expertsThe same matched control now runs at 1.3B and 3B with 16 experts and in a LLaMA-style 350M decoder. GrMoE is better by 0.12, 0.08, and 0.07 perplexity.Section 4.2, Table 2
Figure 3 captionThe caption now names the three panels in order: training loss, routing entropy, and learned concentration.Figure 3
Figure 5 captionThe caption now describes a 16-expert model, which matches the plot.Figure 5
Repeated qualifications and revision statementsRemoved. Each limitation is stated once, in Section 6.Sections 1–6

Can the Table 8 matched comparison be extended to a larger model?

Yes. Table 2 of the revision repeats the matched control at 1.3B and 3B with 16 experts and in a 350M LLaMA-style decoder with RMSNorm, SwiGLU, and RoPE. The comparator is the strongest Euclidean router at 350M, unconstrained low-rank routing with span diversity. In each setting, both routers share the amortizer, diversity penalty, data order, optimizer, training fan-out controller, and frozen test fan-out. Descriptor parameters and scoring MACs match within 0.2%, and expert FLOPs are normalized to the low-rank router.

SettingRouterActive experts (train / test)Expert FLOPsPPLLoad CV
1.3B, 16 expertsLow-rank + span diversity2.02 / 2.001.00013.500.097
1.3B, 16 expertsGrMoE + amortizer2.02 / 2.001.00413.380.080
3B, 16 expertsLow-rank + span diversity2.01 / 2.001.00011.450.096
3B, 16 expertsGrMoE + amortizer2.02 / 2.001.00611.370.084
350M LLaMA-style, 8 expertsLow-rank + span diversity2.02 / 2.001.00017.480.087
350M LLaMA-style, 8 expertsGrMoE + amortizer2.02 / 2.001.00317.410.078

Runs use 30B tokens at 1.3B and 3B and 15B tokens for the LLaMA-style model, with three paired runs per setting. GrMoE improves perplexity by 0.12, 0.08, and 0.07 and has lower load CV in every setting. These are point estimates over three runs. They show the direction of the effect in each setting, but they do not establish statistical significance or a gain that grows with scale. At 350M the paired difference keeps its interval, −0.14 with 95% interval [−0.18, −0.10] (Table 1).

Can the figure captions be fixed?

Yes. Figure 3 now names its three panels in order: training loss of GrMoE and Softmax Top-2 over five seeds, routing entropy over the same seeds, and the learned concentration of each of the 16 experts in the 1.3B model. The caption no longer mentions a CV row. Figure 5 is now titled "Token-level routing in a 16-expert model", which matches the 16 experts in the plot.

Can the repeated qualifications and revision statements be cut?

Yes. The main text no longer describes the revision, and sentences such as "The revision is explicit about…" are gone. Each limitation now appears once, in Section 6, and the abstract states the scope once.

Please let us know whether these results answer your questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for detailed reviews. The main questions were whether the gain comes from the subspace geometry or from extra router capacity and compute, whether it holds beyond 350M parameters, how GrMoE compares with expert-specialization losses, and whether the paper is clearly written. The revised paper answers each with new results and a rewritten main text.

  • The matched control now covers four settings. After matching router parameters, scoring cost, diversity regularization, expert compute, and active experts, GrMoE improves perplexity over the strongest Euclidean router by 0.14 at 350M, with paired 95% interval [−0.18, −0.10]. It improves by 0.12 at 1.3B, by 0.08 at 3B, and by 0.07 in a LLaMA-style decoder (Tables 1 and 2).
  • Every baseline now shows its compute. We reran all routers at a common expert-compute budget and report calls per token, expert FFN work, forward MACs, and latency in every row. GrMoE with its amortizer reaches 17.82 perplexity, and the other routers reach 17.98 to 19.62. Native Switch stays cheaper, at half the FFN work and 18.62 perplexity (Table 4).
  • Specialization losses are reproduced and combined with GrMoE. We reproduce Guo et al. (NeurIPS 2025) on DeepSeek-MoE-16B with its published recipe and test their losses together with GrMoE in a 1.3B factorial. The losses help both routers, and GrMoE with the losses stays 0.3 points ahead of the low-rank router with the losses on GSM8K (Section 4.6, Tables 36 and 37).
  • The main text is rewritten. It states the problem and the motivation first, uses short sentences, states each limitation once, and contains no revision commentary. The Figure 3 and Figure 5 captions now match their panels.

Revisions at a glance

New result or changeRequested byWhere in the revision
Matched control at 1.3B and 3B with 16 experts and in a LLaMA-style decoder52ew, jxpwSection 4.2, Table 2
All routers at a common expert-compute budget, with calls per token, FFN work, MACs, and latency3DQbSection 4.3, Table 4; Appendix R
Balancing-loss setting of Softmax Top-23DQbTable 3 caption; Section 4.3
Concrete observations that motivate orthonormal subspaces3DQbSection 1
Active compute stated for the ten-seed table3DQbTable 3 caption
Guo et al. reproduced natively and combined with GrMoEjxpwSection 4.6; Appendix S, Tables 36–37
Figure 3 and Figure 5 captions corrected52ewFigures 3 and 5
Main text rewritten, revision commentary removed52ew, 3DQb, jxpwSections 1–6
Code releasejxpwReproducibility Statement

Table and section numbers in our replies refer to the revised paper. The submitted Table 2 is now Table 3. The submitted matched comparison, Table 4, is now Table 1, and its seed-level values remain in Table 8.

What the paper claims

GrMoE describes each expert by an orthonormal subspace and scores a normalized token by the energy of its projection onto that subspace. The score lies in [0, 1] and does not depend on the chosen basis, so one global scale α can set the compute budget of a trained checkpoint. The paper claims a small quality gain that survives matched controls, a stable compute dial, and a load-balance bound under stated margin assumptions. It does not claim that softmax scaling is new, that GrMoE is cheaper than every router, or results beyond 3B parameters and single-node serving.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.