- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes a new way of routing,which represents each expert by an orthonormal subspace and scores a normalized token by concentration-scaled projection energy. The authors test it on a small scale 350M, not very sparse MoE, and observe somewhat better performance in load balancing and validation loss.
Critical strengths
-
The paper addresses a timely problem: Better MoE routing.
-
The paper has lots of theory (that I cannot understand), making the method look very theoretically grounded.
Critical weaknesses
-
The experiment is weak: The sparsity is too high, 150M activated out of 350M. Also the model size is too small.
-
The paper needs more baselines: e.g. https://openreview.net/forum?id=iydmH9boLb
-
The writing needs lots of improvement. I cannot understand most parts of the paper.
Questions for authors
Do the authors plan to release the code?
Response to reviewer jxpw
Thank you for the review. The table summarizes our answers.
Sparsity and model scale
The primary model activates about 150M of its 350M parameters because attention, embeddings, and other shared components run for every token. Expert sparsity is a separate quantity, and we test it with larger expert pools.
Both settings activate about 6.3% of the expert pool and use three seeds. At 128 experts, router scoring is 8.3% of counted forward MACs, so that setting tests routing quality rather than efficiency. For model size, the matched control at 1.3B and 3B is in our response to Reviewer 52ew (Table 2). The evidence covers models up to 3B parameters, and Section 6 states this limit.
Guo et al. as a baseline
Guo et al. add orthogonality and variance losses when post-training an existing MoE. GrMoE changes a different part of the router, the descriptor that scores an expert, and adds a compute dial after training. The two can therefore be combined. We test the relationship in three ways.
Native reproduction. We use DeepSeek-MoE-16B with its routing intact (https://proceedings.neurips.cc/paper_files/paper/2025/file/4598de7d243d528e38eb0c5d8155fb52-Paper-Conference.pdf): rank-32 LoRA in routers and experts, LoRA α = 128, dropout 0.1, learning rate 10⁻⁴, batch 32, and auxiliary, orthogonality, and variance coefficients of 0.001 each. Each task is fine-tuned separately on 6,000 examples for three epochs, with three post-training runs.
The losses raise GSM8K from 15.8% to 34.7% and MATH500 from 5.5% to 10.5% at 2.8% more GPU time. This confirms the baseline under its intended recipe. It is not a direct comparison with GrMoE, because replacing the router of a pretrained checkpoint requires an adaptation step, and GrMoE is not a zero-shot retrofit.
Factorial at 1.3B. We pretrain low-rank and GrMoE routers separately, with three paired pretraining runs per router, and post-train each with and without the losses. All four cells share task examples and post-training budgets and are calibrated to two active experts. Each cell receives the same tuning allowance of four pilots, and a separate calibration set selects settings without test accuracy.
The losses help both routers. GrMoE with the losses is 0.3 points above the low-rank router with the losses on GSM8K and 0.1 points above it on MATH500. The losses also lift the low-rank router above GrMoE without them. The two methods are therefore compatible, and the remaining GrMoE gain is small. These are point estimates from one post-training run per pretrained checkpoint, so they do not establish significance.
Common budget at 350M. Adapted to from-scratch Top-2 training, the losses give 17.98 perplexity, the closest baseline to GrMoE with its amortizer at 17.82 (Table 4).
Will the code be released?
Yes. The code will be released upon acceptance.
Please let us know whether these results answer your questions.