- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes GGLR, a learned MoE router guided by historical token centroids. These centroids are used to center and scale routing inputs, while changes in their directions guide router updates. Language-modeling experiments show improved global load balance and slightly lower perplexity than the evaluated baselines.
Critical strengths
-
The results show better global load balance while also achieving slightly lower perplexity than the evaluated baselines. This suggests that the method can improve expert utilization without hurting language-modeling quality.
-
The authors examine both expert loads and the geometry of assigned tokens. The results show more separated expert-input means and router directions, providing evidence that the method changes how token populations are organized across experts.
Critical weaknesses
-
The experiments do not establish the contribution of the proposed geometry-based router update. GGLR changes input normalization, router initialization, and the task-update magnitude while adding the geometry update. However, there are no ablations to separate these effects. The theoretical analysis describes the update but does not establish why it should improve routing. Therefore, the reported gains do not show whether this central component is useful. This affects my confidence in the main methodological contribution.
-
The evidence for improved language-modeling quality is also limited. The perplexity gain over LFB is small and comes from a single training seed. Without repeated runs, it is unclear whether this improvement is consistent across random seeds, which can affect both training dynamics and the initial centroid estimates. Together with the missing ablations, this leaves insufficient evidence to attribute a reliable quality improvement to the proposed geometry-based update.
Questions for authors
1. Clarification: Why do the authors use changes in centroid directions to update router weights? Why was set to 0.5? Were other values tested?
2. Additional evidence: Could the authors remove the geometry update while keeping the same input normalization, router initialization, and reduced task update? This would help show whether the geometry update contributes to the improvement.
3. Additional evidence: Could the authors repeat the experiments with different training seeds and report the mean and variation? The perplexity gain over LFB is small, so it is important to check whether this improvement is consistent.
7 comments
Geometric-update ablation
To isolate the contribution of the geometric update, we removed the geometric correction while retaining the exact same input centering and radius scaling, router initialization, first-microbatch centroid bootstrapping, and 0.5 scaling of the task-optimizer update as in full GGLR. At seed 42 and step 63,579, the results are:
| Setting | Validation loss | WikiText-2 PPL | Global MARD | Global MaxVio | 65K p95 MARD | 65K p95 MaxVio |
|---|---|---|---|---|---|---|
| GGLR w/o Geometric Update | 2.299674 | 11.118872 | 0.023153 | 0.060546 | 0.212827 | 1.250683 |
| GGLR | 2.295957 | 11.088702 | 0.015173 | 0.036756 | 0.211381 | 1.296055 |
Compared with the ablated variant (GGLR w/o Geometric Update), full GGLR achieves lower validation loss (2.295957 vs. 2.299674), WikiText-2 perplexity (11.088702 vs. 11.118872), global MARD (0.015173 vs. 0.023153), and global MaxVio (0.036756 vs. 0.060546). The 65K local-window metrics are mixed, with a slightly lower p95 MARD (0.211381 vs. 0.212827) and a higher p95 MaxVio (1.296055 vs. 1.250683).
Repeated training runs
We selected Original LFB and LFB + gain normalization, the two most competitive baselines in our main comparison, for the multi-seed study alongside GGLR. Each method was trained with three seeds (42, 43, and 44) at the full 1.7B scale for 63,579 optimizer steps (100,001,120,256 tokens, approximately 100B per run). Including the additional runs at seeds 43 and 44, this comparison comprises nine full-budget training runs. We report validation loss for each seed and the three-seed mean and sample standard deviation:
| Method | Seed 42 | Seed 43 | Seed 44 | Mean | Sample SD |
|---|---|---|---|---|---|
| LFB | 2.297426 | 2.315463 | 2.305621 | 2.306170 | 0.009031 |
| LFB + gain normalization | 2.303748 | 2.310209 | 2.309996 | 2.307984 | 0.003670 |
| GGLR | 2.295957 | 2.305610 | 2.300542 | 2.300703 | 0.004828 |
GGLR achieves lower validation loss than both LFB variants across all three matched seeds, with the lowest mean validation loss (2.300703, sample SD 0.004828) and a smaller sample standard deviation than standard LFB (2.306170, sample SD 0.009031). Across the three seeds, GGLR also achieves lower WikiText-2 test perplexity on every seed (mean 11.1609, SD 0.0724 vs. 11.2719, SD 0.1433 for LFB and 11.2601, SD 0.0706 for LFB + gain normalization) and the lowest mean global MARD (0.015232, SD 0.000079 vs. 0.101225, SD 0.007194 for LFB and 0.016926, SD 0.000501 for LFB + gain normalization) and global MaxVio (0.036622, SD 0.002246 vs. 0.571237, SD 0.169552 and 0.049613, SD 0.004396).
Direction changes and mixing coefficient
Centroid directions provide a geometric reference for the router weights. Online K-means scores a token by its cosine similarity to each expert's centroid. If the token vectors are normalized to unit length and the rows of a linear router are the unit centroid directions, the router produces exactly these cosine scores (Appendix C.2). GGLR uses the directions of centered centroids as its reference and keeps the router weights trainable by the language-modeling loss.
As training changes the token representations, the centroid directions also move. GGLR adds the change in each reference direction to the corresponding router weight. Let ue,ue+ denote the previous and updated unit reference directions, De the task-optimizer displacement, and λ the mixing coefficient between 0 and 1. GGLR updates the router weight as
we+=we+(1−λ)De+λ(ue+−ue).
This update lets the router follow the movement of its input population while retaining the adjustments learned from the task. When a reference direction stays fixed, the geometric correction is zero. Repeatedly pulling a weight toward the current reference would also shrink the task-learned deviations from that direction, even after the reference stops moving.
Consider a 2D scoring example where the reference directions turn from horizontal to vertical. Initially, the two reference directions point right and left, and the router weights match them; the references then turn to point straight up and straight down, and we evaluate a new upward input:
w1=(1,0),w2=(−1,0),x=(0,1).
Set the task-optimizer displacement to zero, the mixing coefficient to 0.5, and the two selection biases equal. Applying the reference changes in the update rule gives the updated weights:
w1+=(1,0)+0.5((0,1)−(1,0))=(0.5,0.5),w2+=(−0.5,−0.5).
For the upward input, the linear dot-product score is simply each weight vector's vertical component. The old weights are purely horizontal, so both vertical components are zero; after the update, Expert 1 has a positive vertical component and Expert 2 has a negative one:
In this example, the geometric update moves the router weights toward the new input directions and widens the score separation between the two experts, while a downward input simply reverses the two scores.
Regarding the choice of mixing coefficient, we use 0.5 as a balanced default that assigns equal scalar weights to the task-optimizer displacement and the geometric correction. To examine sensitivity to this choice, we additionally evaluated 0.25 and 0.75 at seed 42 and step 63,579:
Among the evaluated coefficients, 0.5 achieves the lowest validation loss, WikiText-2 perplexity, global MARD, and global MaxVio. The 0.25 setting has slightly lower local-window p95 MARD and MaxVio, while 0.75 has higher values on all six metrics than 0.5. Global metrics aggregate assignments over the validation sample within each layer and then average over 16 layers. Local metrics take the p95 over complete 65,536-token windows within each layer and then average over layers. For all six metrics, lower values indicate better performance.