acceptodds
Under review as a conference paper at ICLR 2027

Learning Token-Population Geometry for Sparse Mixture-of-Experts

Abstract

Routing in sparse mixture-of-experts (MoE) models determines both token-level computation and the populations from which experts learn. Load balancing controls how much traffic each expert receives, but does not specify how these populations are organized in representation space. We introduce the Geometry-Guided Learnable Router (GGLR), which uses expert input means maintained by an exponential moving average as an explicit population reference for a task-trained router. This reference defines population-relative coordinates for expert selection and target directions whose changes guide router updates, while selection-only biases regulate utilization. We analyze how these population-relative coordinates transform pairwise routing logits and how historical averaging affects the accuracy of the population reference. In matched 1.7-billion-parameter experiments on Nemotron-CC-v2, GGLR achieves the lowest observed endpoint loss and global load imbalance, while producing more separated directions of expert input means and more distinct learned router directions. Compared with Loss-Free Balancing with normalized output gains, GGLR reduces global maximum overload by 23.1%, while also lowering validation perplexity.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mixture-of-experts, routing, load balancing, population geometry

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8RpxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GGLR, a learned MoE router guided by historical token centroids. These centroids are used to center and scale routing inputs, while changes in their directions guide router updates. Language-modeling experiments show improved global load balance and slightly lower perplexity than the evaluated baselines.

Critical strengths
  1. The results show better global load balance while also achieving slightly lower perplexity than the evaluated baselines. This suggests that the method can improve expert utilization without hurting language-modeling quality.

  2. The authors examine both expert loads and the geometry of assigned tokens. The results show more separated expert-input means and router directions, providing evidence that the method changes how token populations are organized across experts.

Critical weaknesses
  1. The experiments do not establish the contribution of the proposed geometry-based router update. GGLR changes input normalization, router initialization, and the task-update magnitude while adding the geometry update. However, there are no ablations to separate these effects. The theoretical analysis describes the update but does not establish why it should improve routing. Therefore, the reported gains do not show whether this central component is useful. This affects my confidence in the main methodological contribution.

  2. The evidence for improved language-modeling quality is also limited. The perplexity gain over LFB is small and comes from a single training seed. Without repeated runs, it is unclear whether this improvement is consistent across random seeds, which can affect both training dynamics and the initial centroid estimates. Together with the missing ablations, this leaves insufficient evidence to attribute a reliable quality improvement to the proposed geometry-based update.

Questions for authors

1. Clarification: Why do the authors use changes in centroid directions to update router weights? Why was λ\lambda set to 0.5? Were other values tested?

2. Additional evidence: Could the authors remove the geometry update while keeping the same input normalization, router initialization, and reduced task update? This would help show whether the geometry update contributes to the improvement.

3. Additional evidence: Could the authors repeat the experiments with different training seeds and report the mean and variation? The perplexity gain over LFB is small, so it is important to check whether this improvement is consistent.

7 comments
AuthorsOpenReview ↗

Direction changes and mixing coefficient

Centroid directions provide a geometric reference for the router weights. Online K-means scores a token by its cosine similarity to each expert's centroid. If the token vectors are normalized to unit length and the rows of a linear router are the unit centroid directions, the router produces exactly these cosine scores (Appendix C.2). GGLR uses the directions of centered centroids as its reference and keeps the router weights trainable by the language-modeling loss.

As training changes the token representations, the centroid directions also move. GGLR adds the change in each reference direction to the corresponding router weight. Let ue,ue+u_e,u_e^+ denote the previous and updated unit reference directions, DeD_e the task-optimizer displacement, and λ\lambda the mixing coefficient between 0 and 1. GGLR updates the router weight as

we+=we+(1−λ)De+λ(ue+−ue).w_e^+=w_e+(1-\lambda)D_e+\lambda(u_e^+-u_e).

This update lets the router follow the movement of its input population while retaining the adjustments learned from the task. When a reference direction stays fixed, the geometric correction is zero. Repeatedly pulling a weight toward the current reference would also shrink the task-learned deviations from that direction, even after the reference stops moving.

Consider a 2D scoring example where the reference directions turn from horizontal to vertical. Initially, the two reference directions point right and left, and the router weights match them; the references then turn to point straight up and straight down, and we evaluate a new upward input:

w1=(1,0),w2=(−1,0),x=(0,1).w_1 = (1, 0), \quad w_2 = (-1, 0), \qquad x = (0, 1).

Set the task-optimizer displacement to zero, the mixing coefficient to 0.5, and the two selection biases equal. Applying the reference changes in the update rule gives the updated weights:

w1+=(1,0)+0.5((0,1)−(1,0))=(0.5,0.5),w2+=(−0.5,−0.5).w_1^+ = (1, 0) + 0.5\big((0, 1) - (1, 0)\big) = (0.5, 0.5), \qquad w_2^+ = (-0.5, -0.5).

For the upward input, the linear dot-product score is simply each weight vector's vertical component. The old weights are purely horizontal, so both vertical components are zero; after the update, Expert 1 has a positive vertical component and Expert 2 has a negative one:

Router stateExpert 1 scoreExpert 2 scoreScore gap
Before geometric update000
After geometric update0.5-0.51.0

In this example, the geometric update moves the router weights toward the new input directions and widens the score separation between the two experts, while a downward input simply reverses the two scores.

Regarding the choice of mixing coefficient, we use 0.5 as a balanced default that assigns equal scalar weights to the task-optimizer displacement and the geometric correction. To examine sensitivity to this choice, we additionally evaluated 0.25 and 0.75 at seed 42 and step 63,579:

CoefficientValidation lossWikiText-2 PPLGlobal MARDGlobal MaxVio65K p95 MARD65K p95 MaxVio
0.252.29800211.1362120.0153930.0401490.2090441.286163
0.52.29595711.0887020.0151730.0367560.2113811.296055
0.752.29966611.1420130.0211050.0551210.2490061.596512

Among the evaluated coefficients, 0.5 achieves the lowest validation loss, WikiText-2 perplexity, global MARD, and global MaxVio. The 0.25 setting has slightly lower local-window p95 MARD and MaxVio, while 0.75 has higher values on all six metrics than 0.5. Global metrics aggregate assignments over the validation sample within each layer and then average over 16 layers. Local metrics take the p95 over complete 65,536-token windows within each layer and then average over layers. For all six metrics, lower values indicate better performance.

AuthorsOpenReview ↗

Geometric-update ablation

To isolate the contribution of the geometric update, we removed the geometric correction while retaining the exact same input centering and radius scaling, router initialization, first-microbatch centroid bootstrapping, and 0.5 scaling of the task-optimizer update as in full GGLR. At seed 42 and step 63,579, the results are:

SettingValidation lossWikiText-2 PPLGlobal MARDGlobal MaxVio65K p95 MARD65K p95 MaxVio
GGLR w/o Geometric Update2.29967411.1188720.0231530.0605460.2128271.250683
GGLR2.29595711.0887020.0151730.0367560.2113811.296055

Compared with the ablated variant (GGLR w/o Geometric Update), full GGLR achieves lower validation loss (2.295957 vs. 2.299674), WikiText-2 perplexity (11.088702 vs. 11.118872), global MARD (0.015173 vs. 0.023153), and global MaxVio (0.036756 vs. 0.060546). The 65K local-window metrics are mixed, with a slightly lower p95 MARD (0.211381 vs. 0.212827) and a higher p95 MaxVio (1.296055 vs. 1.250683).

AuthorsOpenReview ↗

Repeated training runs

We selected Original LFB and LFB + gain normalization, the two most competitive baselines in our main comparison, for the multi-seed study alongside GGLR. Each method was trained with three seeds (42, 43, and 44) at the full 1.7B scale for 63,579 optimizer steps (100,001,120,256 tokens, approximately 100B per run). Including the additional runs at seeds 43 and 44, this comparison comprises nine full-budget training runs. We report validation loss for each seed and the three-seed mean and sample standard deviation:

MethodSeed 42Seed 43Seed 44MeanSample SD
LFB2.2974262.3154632.3056212.3061700.009031
LFB + gain normalization2.3037482.3102092.3099962.3079840.003670
GGLR2.2959572.3056102.3005422.3007030.004828

GGLR achieves lower validation loss than both LFB variants across all three matched seeds, with the lowest mean validation loss (2.300703, sample SD 0.004828) and a smaller sample standard deviation than standard LFB (2.306170, sample SD 0.009031). Across the three seeds, GGLR also achieves lower WikiText-2 test perplexity on every seed (mean 11.1609, SD 0.0724 vs. 11.2719, SD 0.1433 for LFB and 11.2601, SD 0.0706 for LFB + gain normalization) and the lowest mean global MARD (0.015232, SD 0.000079 vs. 0.101225, SD 0.007194 for LFB and 0.016926, SD 0.000501 for LFB + gain normalization) and global MaxVio (0.036622, SD 0.002246 vs. 0.571237, SD 0.169552 and 0.049613, SD 0.004396).

Reviewer_qwKaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors present the Geometry-Guided Learnable Router (GGLR) for sparse Mixture-of-Experts (MoE) models, aiming to incorporate the geometric structure of expert input populations into routing decisions beyond conventional load balancing. GGLR maintains exponential moving averages of expert input representations to construct population-relative routing coordinates and introduces a geometry-guided parameter update based on changes in expert population directions, while separately regulating expert utilization through selection biases.

The paper also provides analytical results characterizing the effects of population-relative coordinates on routing logits and the estimation properties of EMA-based population references. Experiments on a 1.7B-parameter MoE model trained on approximately 100B tokens demonstrate some improvements in validation loss, perplexity, global load balancing, and expert-population separation compared with several established routing baselines.

However, the reported gains are based on a single training seed, and the method shows greater local load imbalance than gain-normalized Loss-Free Balancing.

Critical strengths
  • The paper addresses an important limitation of conventional sparse Mixture-of-Experts routing by explicitly incorporating the geometric structure of expert input populations rather than focusing solely on load balancing.
  • The proposed GGLR framework offers an interesting integration of EMA-based population references, population-relative routing coordinates, and geometry-guided parameter updates while preserving task-driven expert selection.
  • The theoretical analysis provides useful insights into how population-relative transformations affect routing logits and how EMA estimation reduces reference noise under stationary assumptions.
  • The experimental results illustrate the enhancement in validation loss, perplexity, global load balancing, and expert-population separation compared with several established baselines.
  • Finally, the additional geometric analyses and transparent discussion of local load-balancing trade-offs further strengthen the empirical presentation.
Critical weaknesses

There are a couple of weaknesses I want clarification on from the authors:

  • Although GGLR achieves the best reported validation loss and perplexity, its improvements over the strongest language-modeling baseline are relatively small. Besides, all large-scale experiments use a single training seed (seed 42, Table 7), without independent repetitions or uncertainty estimates. Consequently, the robustness of these improvements to training randomness remains unclear, particularly given the substantial computational investment required for training 1.7B-parameter models.
  • The geometry-guided update in Equation 9 modifies router parameters through an additional displacement derived from changes in expert population directions. Although the paper provides useful algebraic characterizations of this update and analyzes population-reference estimation, it does not establish convergence or stability guarantees for the resulting optimization dynamics. Importantly, Equation 9 also scales the conventional optimizer displacement by a factor of (1-\lambda), independently of the geometric correction. Therefore, I believe we need theoretical justification and controlled comparisons to clarify this update rule's contribution and stability.
  • It is unclear whether the increased local routing imbalance causes transient expert congestion, reduced hardware utilization, or higher training latency.
  • Additional component-level ablations, including the effects of the EMA coefficient and geometric-update strength, are necessary to substantiate the claimed benefits of learning token-population geometry.
Questions for authors

There are two questions for the authors:

(1) Can the paper provide additional independent training runs or uncertainty estimates to establish whether the relatively small validation-loss improvement over Loss-Free Balancing is robust to training randomness?

(2) It would be great if the paper could provide an ablation comparing GGLR against an optimizer-only router update scaled by the same factor (1-\lambda). Additionally, how does the increased local load imbalance affect actual training throughput and expert utilization?

9 comments
AuthorsOpenReview ↗

Robustness across training seeds

To test whether the loss improvement is consistent across training seeds, we repeated GGLR and the two most competitive baselines in our main comparison: Original LFB and LFB + gain normalization. Each method uses seeds 42, 43, and 44, with a 1.7B model trained for the full 63,579 optimizer steps (100,001,120,256 tokens, approximately 100B per run). Seeds 43 and 44 add six complete training runs to the original three, giving nine full-budget runs in total.

MethodSeed 42Seed 43Seed 44MeanSample SD
LFB2.2974262.3154632.3056212.3061700.009031
LFB + gain normalization2.3037482.3102092.3099962.3079840.003670
GGLR2.2959572.3056102.3005422.3007030.004828

GGLR has lower validation loss than both baselines in all three matched seeds. Its mean loss is 2.300703 (sample SD 0.004828), compared with 2.306170 (0.009031) for Original LFB and 2.307984 (0.003670) for LFB + gain normalization. GGLR's sample SD is smaller than Original LFB's.

GGLR also has the lowest mean WikiText-2 perplexity and global imbalance. The three-seed global MARD is 0.015232 (SD 0.000079), compared with 0.101225 (0.007194) for Original LFB and 0.016926 (0.000501) for LFB + gain normalization. Global MaxVio is 0.036622 (0.002246), compared with 0.571237 (0.169552) and 0.049613 (0.004396), respectively.

The local-window results favor LFB + gain normalization: its 65K-window p95 MARD is 0.175753 (SD 0.000845) and p95 MaxVio is 1.065193 (0.017390), compared with 0.208600 (0.002436) and 1.273866 (0.022377) for GGLR. GGLR improves both local metrics over Original LFB, whose values are 0.272292 (0.008853) and 1.775953 (0.140411).

AuthorsOpenReview ↗

Isolating the geometric update

Following your suggestion, we trained an optimizer-only control with the same 0.5 task-update scaling as GGLR. This control removes only the geometric correction and keeps input centering and radius scaling, router initialization, and initial centroid bootstrapping unchanged. At seed 42 and step 63,579, we obtain:

SettingValidation lossWikiText-2 PPLGlobal MARDGlobal MaxVio65K p95 MARD65K p95 MaxVio
GGLR w/o Geometric Update2.29967411.1188720.0231530.0605460.2128271.250683
GGLR2.29595711.0887020.0151730.0367560.2113811.296055

Adding the geometric correction lowers validation loss, WikiText-2 perplexity, global MARD, and global MaxVio relative to this matched-scaling control. For the 65K-window metrics, it slightly lowers p95 MARD but increases p95 MaxVio. This comparison isolates the geometric correction from the reduction in task-update magnitude.

We also tested No Centroid, which retains the geometric router update but selects experts using unit-normalized inputs without population-relative centering and radius scaling. Its validation loss is 2.295992, WikiText-2 PPL is 11.081410, global MARD is 0.058147, global MaxVio is 0.153094, 65K p95 MARD is 0.218734, and 65K p95 MaxVio is 1.219704.

For the update dynamics, Appendix B.5 gives a decomposition of the router displacement. Let Dt=Wtopt−Wt−1D_t=W_t^{\mathrm{opt}}-W_{t-1} denote the task-optimizer displacement, and let UtU_t denote the matrix of centered unit reference directions. With a fixed mixing coefficient satisfying 0≤λ≤10\leq\lambda\leq1, the update is

Wt=Wt−1+(1−λ)Dt+λ(Ut−Ut−1).W_t=W_{t-1}+(1-\lambda)D_t+\lambda(U_t-U_{t-1}).

Across consecutive updates with matching reference endpoints, the intermediate reference terms cancel:

WT=W0+(1−λ)∑t=1TDt+λ(UT−U0).W_T=W_0+(1-\lambda)\sum_{t=1}^{T}D_t+\lambda(U_T-U_0).

Each expert's net geometric displacement therefore depends only on its initial and final reference directions and has norm at most 2λ2\lambda. When the reference direction stops moving, the geometric correction is zero, while the optimizer term continues to train the router.

AuthorsOpenReview ↗

Sensitivity to mixing and EMA coefficients

We tested the mixing coefficient and EMA retention coefficient separately at seed 42 and step 63,579. The mixing coefficient weights the geometric correction and gives the complementary weight to the task-optimizer displacement. The results for 0.25, 0.5, and 0.75 are:

CoefficientValidation lossWikiText-2 PPLGlobal MARDGlobal MaxVio65K p95 MARD65K p95 MaxVio
0.252.29800211.1362120.0153930.0401490.2090441.286163
0.52.29595711.0887020.0151730.0367560.2113811.296055
0.752.29966611.1420130.0211050.0551210.2490061.596512

Among these settings, 0.5 has the lowest validation loss, WikiText-2 perplexity, global MARD, and global MaxVio. The 0.25 setting has slightly lower 65K-window p95 MARD and MaxVio.

The EMA coefficient determines how much of the previous population mean is retained. A coefficient of 0 uses the current observation; 0.99 retains a longer history. We compared 0, 0.5, and 0.99:

EMA retentionValidation lossWikiText-2 PPLGlobal MARDGlobal MaxVio65K p95 MARD65K p95 MaxVio
02.29944111.1589710.0212270.0588750.2064141.203037
0.52.31361211.3068860.0279500.0735610.2485161.368877
0.992.29595711.0887020.0151730.0367560.2113811.296055

Retention 0.99 has the lowest validation loss, WikiText-2 perplexity, and global MARD and MaxVio in this comparison. Retention 0 has lower local-window p95 metrics. These results support the default settings for language-modeling quality and global balance, while showing the local-load differences across settings.

Reviewer_Ky5dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes the Geometry-Guided Learnable Router (GGLR), a sparse moe router that keeps expert preferences trained by the lm loss while using the geometry of past expert inputs to set the selection coordinates. It conducts experiments on 1.7B Nemotron-CC-v2 with 100B tokens and evaluates lm loss and load balancing performance (maxvio) comparing with other routing methods. Online K-means is a very scrict way to route with cosine similarity to running input centroids. While GGLR uses EMA as a reference, not the selector. The router of GGLR centers and scales each token with weighted center and EMA then conducts the scoring. The paper trains all baselines though they are rather simple. GGLR achieves good results on both lm quality and load balancing.

Critical strengths

The design of population reference using EMA is clear and a nice follow-up of k-means.

Critical weaknesses

Limit of technical contribution: It works on a real LB problem but a smaller scope of geometry; weak baselines and not significant eval improvements.

Questions for authors
  • With 100B tokens and a 1.7B model, are some evals on real benchmarks possible?
  • Most baselines are not targeting this issue, and the online K-means method (Ahrac et al., 2026) does not seem to be a well-recognized method. Is this an important issue that many people are working on?
5 comments
AuthorsOpenReview ↗

Token-Population Geometry and Related Baselines

Conventional load balancing controls expert assignment counts, while the same counts can correspond to different expert input populations. Recent work studies how routing relates to token-population geometry and expert specialization. Online K-means uses running centroids for selection (Ahrac, Hochwald, and Geva, "Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts", NeurIPS 2026; arXiv:2605.12476). AC (Adaptive Clustering) computes feature weights from expert-group dispersion to rescale routing inputs (Nielsen et al., ICLR 2025 proceedings). SimBal adds a soft orthogonality objective to push the learned router's Gram matrix toward the identity (Omi, Sen, and Farhadi, arXiv:2506.14038).

While Online K-means selects experts using centroids and SimBal regularizes router geometry, GGLR uses historical token-population statistics to guide a task-trained router. Specifically, historical expert-input means define its center and shared radius, target-direction changes guide router updates, and separate selection biases regulate utilization. In the paper’s five-method comparison (auxiliary-loss balancing, Original LFB, LFB + gain normalization, Online K-means, and GGLR), GGLR has the greatest separation of expert-input mean directions. Among the four learned selectors in that comparison, GGLR also has the lowest overlap between router directions; Online K-means has no learned router matrix. At the matched endpoint below, GGLR has the lowest validation NLL and global MARD and MaxVio among all eight methods.

Matched endpoint snapshot at step 63,579, training seed 42

MethodValidation NLLGlobal MARDGlobal MaxVio65K p95 MARD65K p95 MaxVio
Auxiliary-loss balancing2.3043850.2774631.4446990.3433301.711739
Global-batch LBL2.3098700.2895281.472027N/AN/A
Original LFB2.2974260.0957870.6084180.2621511.766937
LFB + gain normalization2.3037480.0167720.0478220.1755991.050526
Online K-means2.3462330.1765360.7238120.3208221.543625
Original LFB + SimBal2.3128450.1060910.4625130.2678871.608994
AC, previous-layer MAD2.3125680.2327711.1628200.3027471.493195
GGLR2.2959570.0151730.0367560.2113811.296055

Each 65K p95 value is the per-layer percentile over complete 65,536-token windows, then averaged over 16 layers. GGLR lowers validation NLL to 2.295957 compared to 2.297426 for Original LFB, while reducing global MARD from 0.095787 to 0.015173 and global MaxVio from 0.608418 to 0.036756. LFB + gain normalization retains the lowest 65K local-window imbalance. AC and SimBal record higher validation NLL and global imbalance than GGLR.

AuthorsOpenReview ↗

Downstream Benchmark Evaluation

We evaluated the compared methods at the matched step-63,579 endpoint after 100,001,120,256 training tokens on the 1.7B parameter model. The evaluation uses zero-shot configurations across five tasks: HellaSwag, PIQA, ARC-Easy, ARC-Challenge, and SST-2.

Raw accuracy, aggregated across available training seeds (%)

MethodHellaSwagPIQAARC-EasyARC-ChallengeSST-2
Auxiliary-loss balancing (1 training seed)35.47169.58754.20925.85349.083
Original LFB (3 training seeds)35.32569.65954.25124.60250.344
LFB + gain normalization (3 training seeds)35.40569.31454.19525.25649.350
Online K-means (1 training seed)35.02369.53253.87224.74451.032
Global-batch LBL (1 training seed)35.33269.20653.03024.65949.083
Original LFB + SimBal (1 training seed)35.05368.98853.83025.42749.083
AC, previous-layer MAD (1 training seed)26.64853.04729.25120.47848.968
GGLR (3 training seeds)35.64069.09754.92426.45151.147

Each training seed is evaluated once with each of three evaluation seeds (42, 43, and 44), and those three results are averaged. For methods with three training seeds, these three training-seed means are averaged equally. Methods with a single training seed use that seed's mean. In these downstream evaluations, GGLR improves upon the highest respective baseline accuracy by 0.169 percentage points on HellaSwag, 0.673 on ARC-Easy, 0.598 on ARC-Challenge, and 0.115 on SST-2.

We have added the five downstream benchmarks (Section 5.4, Table 1) and comparisons with Original LFB + SimBal and AC (Sections 5.1 and 5.2, Table 3) to the revised manuscript, with implementation details in Appendix C.3. We would appreciate your feedback on these additions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.