- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes UniPool to reconsider the fixed expert-to-layer assignment in standard MoE models. During an initial training phase, all layers can access a global expert pool. The method uses pool-level load balancing, NormRouter, and a regularizer to encourage layer-specific expert preferences. Based on early routing statistics, experts are then assigned exclusively to individual layers, and the model continues training as a conventional layer-private MoE.
Experiments across five model scales (182M–1.5B) show consistent improvements in language modeling loss and downstream accuracy over vanilla MoE under matched expert-FFN budgets and routed expert FLOPs. The authors also provide ablations on expert allocation and report near-vanilla training throughput after the allocation is locked.
Critical strengths
-
A practical approach to learning expert allocation. The idea of using a short global-access phase to determine expert ownership is interesting. Unlike methods that require permanent cross-layer expert sharing, UniPool-lock returns to layer-private execution after the initial phase. This provides a practical way to explore more flexible expert allocation without maintaining the full cost of global sharing.
-
Consistent empirical improvements with useful controlled experiments. The method improves test loss across multiple model scales under matched expert-FFN budgets. The comparisons between random and learned allocations in Table 4 are particularly useful, as they show that the allocation selected after global-access training matters. The downstream evaluations also support the improvements observed in language modeling loss.
Critical weaknesses
1.The experiments do not sufficiently establish the paper's central explanation of why learned ownership helps. The paper argues that assigning experts to layers before training leads to redundancy, and that learning expert ownership can address this problem. However, the current evidence does not clearly establish this explanation.
In Table 4, control B locks a random allocation after global-access training, while control C locks an allocation specifically chosen to preserve the routing preferences formed during that training. Therefore, C is naturally more compatible with the existing expert weights and routers. The performance gap shows that preserving these preferences is useful, but it does not establish whether the model has discovered a meaningful and stable expert-to-layer specialization, or whether the benefit mainly comes from avoiding disruption when switching to layer-private training.
The routing randomization experiments in Tables 1 and 7 are also indirect evidence. Greater sensitivity to random routing does not necessarily mean that experts have become more specialized or less redundant.
This distinction matters because learning meaningful expert ownership is the paper's central motivation. The current experiments demonstrate the usefulness of the training procedure, but do not sufficiently establish the proposed explanation for its gains.
2.The claimed efficiency advantage is not sufficiently validated under comparable training costs and larger-scale settings. A main selling point of UniPool-lock is that it retains most of the benefits of global expert sharing at nearly the same cost as vanilla MoE. However, the efficiency evidence remains limited.
The full-pool phase is 22–36% slower per step, and Table 6 reports post-lock and end-to-end timing only for the 182M and 469M settings on eight GPUs. No corresponding timing results are reported for the 650M, 830M, or 1.5B models.
Moreover, the comparison matches expert-FFN FLOPs rather than total training compute. Even after locking, UniPool retains routers that score and normalize over the full global expert index, introducing additional computation that grows with the number of layers.
Without comparisons under matched total training compute or wall-clock budgets, it remains unclear how much of the reported quality improvement represents a better efficiency-quality trade-off. This is important because near-vanilla training cost is a central motivation for the proposed locking strategy.
Questions for authors
-
Distinguishing learned ownership from transition compatibility. In Table 4, how much of the gap between controls B and C can be explained by the disruption caused when the full-pool model is restricted to a layer-private allocation? Could the authors report the loss immediately before and after the locking transition, and compare allocations with similar retained routing mass? This would help determine whether the benefit reflects meaningful learned ownership beyond simply preserving the behavior established during the initial shared phase.
-
Evidence of stable expert specialization. How stable are the learned expert-to-layer assignments across different data subsets or nearby training checkpoints? Can the authors provide more direct evidence of reduced expert redundancy or increased functional specialization, beyond sensitivity to random routing? Such evidence would strengthen the paper's central explanation.
-
Efficiency under matched total compute. Can the authors report post-lock and end-to-end training costs for the larger model scales, particularly 1.5B, and compare UniPool-lock with vanilla MoE under the same total training compute or wall-clock budget? This would clarify whether the reported quality improvements translate into a practical efficiency advantage.
7 comments
Response to Reviewer kE3U (2/3): Stability and direct measures of redundancy
2. Stability and direct evidence of specialization
| Stability check | Result |
|---|---|
| Learned mask vs. dominant layer on separate test data, 1K checkpoint | 182M: 86/96 and 90/96 experts in two prefixes; 469M: 177/192 |
| Two reruns with the same seed and recipe, nondeterministic | 83/96 same owner (mean layer Jaccard 0.78); random allocations agree on 8.2/96 |
| Pre-mask dominant layer at 2K and 2.5K vs. 1K | 182M: 64/96; 469M: 118/192 and 126/192 |
Preferences at 1K are therefore reproducible across data and reruns (86–94% of experts, against 8.2/96 for random allocations). After 1K the attenuation schedule itself reshapes routing: the soft effective owner count per expert rises from 1.44 at 1K to 2.73 at 2.5K at 182M, and agreement with the 1K owner is 64/96 at 182M and 118–126/192 at 469M. So the allocation chosen at 1K is stable across data subsets and reruns.
Direct redundancy measures. We now measure redundancy directly on the final checkpoints and compare with vanilla MoE at the same scale. For each MoE layer, we run every expert of that layer on the layer's real inputs, using held-out sequences that are identical across models, and record three quantities:
- linear CKA between the outputs of each pair of experts;
- best-substitute error: on the tokens routed to an expert, the relative squared error when the closest other expert in the layer answers instead (1.0 means no better than a zero output);
- the LM-loss increase when every expert is replaced by its best substitute, divided by the increase when the layer is removed.
All three are medians over the first and second halves of the layers. CKA and substitute error are computed on expert outputs before gating, so the router's gate scale does not affect them.
| Model | CKA ↓ (shallow / deep) | Best-substitute error ↑ | Replace / remove ↑ |
|---|---|---|---|
| 182M vanilla, 3 seeds | 0.42–0.47 / 0.33–0.35 | 0.74–0.95 / 1.18–1.23 | 0.95–1.18 / 1.44–1.51 |
| 182M UniPool_lock, 4 runs (3 seeds) | 0.31–0.36 / 0.34–0.37 | 1.05–1.16 / 1.22–1.34 | 1.24–1.57 / 1.37–1.68 |
| 469M vanilla | 0.35 / 0.33 | 1.02 / 1.29 | 1.11 / 1.32 |
| 469M UniPool_lock | 0.29 / 0.34 | 1.31 / 1.27 | 1.44 / 1.38 |
In the shallow half of the network, UniPool_lock's experts are less redundant than vanilla's at both scales. At 182M, the ranges over three vanilla seeds and four UniPool_lock runs (three seeds, one of them run twice) do not overlap on any of the three shallow-layer measures, including when layer 0 is excluded. In vanilla, an expert's closest neighbour reproduces part of its output (squared error 0.74–0.95 of the output energy at 182M). In UniPool_lock, the closest other expert typically does no better than a zero output (median 1.05–1.31). Per layer, substitute error is higher in 67 of the 72 shallow-layer comparisons at 182M (6 layers × 4 UniPool_lock runs × 3 vanilla seeds) and in all 12 shallow layers at 469M. The deep half shows no difference.
The random-allocation controls have the same shallow-layer profile at 182M (control A: CKA 0.34, substitute error 1.06) and an intermediate one at 469M (control A: 0.33, 1.16; control B: 0.31, 1.12). At 469M, substitute error is above vanilla's in 12 of 12 shallow layers for UniPool_lock, 9 of 12 for control A, and 7 of 12 for control B; each is one run. So the lower redundancy is a property of UniPool's routing and balancing as a whole, and the learned allocation adds its own effect, visible in loss (Table 4) and in shallow-layer routing sensitivity: with each token's gate kept, randomizing routing within a layer's own experts costs UniPool_lock more than control A in the shallow half at both scales (+0.86 to +1.60 points for one- and two-layer windows, three tasks). In line with your point about Tables 1 and 7, the revision replaces Table 7's probe, which sets the gate to 1 and so inflates drops where learned gates are small, with this gate-preserving one. The revision adds these measurements and states the reduced redundancy for the shallow layers, where they show it.
We believe this resolves your request for direct evidence: measured on the experts themselves, redundancy is lower than in vanilla MoE in the shallow layers at both scales, and the allocation is stable across data and reruns.
Response to Reviewer kE3U (3/3): Efficiency at larger scales
3. Efficiency at larger scales
At EP=1, which both arms use up to 830M, post-lock steps stay within a few percent of vanilla (−0.9% to +3.8%), and end-to-end overhead is +0.4% to about +9%. Two settings need the closer look you ask for, and we now measure both: the full-pool phase, whose overhead grows with scale, and post-lock training under expert parallelism at 1.5B. The table uses median per-step wall time with matched micro-batch sizes; hosts differ between some pairs, so differences of a few percent are not resolved.
| Scale | Vanilla (s/it) | Before lock | After lock | End-to-end |
|---|---|---|---|---|
| 182M | 0.319 | +27% | +2.1% | +2.9% |
| 469M | 0.780 | +38% | −0.9% | +0.4% |
| 650M‡ | 1.080 | +81% | +3.8% | +6.4% |
| 830M, same node | 1.519 | +241% | +1.3% | about +9% (est.) |
| 1.5B as run, EP=8 vs. EP=1 | 5.74–5.94 | – | +53% to +62% | +56% (total runtime) |
| 1.5B, same node, both EP=8 | 4.94 | +106% | +88% | – |
| 1.5B, same node, with EP optimizations† | 4.94 | +106% | +4% | about +6% (est.) |
‡ Timing from per-step logs of an earlier 650M run whose candidate sets are K_l = 8–12; the K_l = 8 run has no per-step logs. The 830M row is a new same-node timing of the K_l = 8 recipe at EP=1 (steps 100–240 before the lock, 100–400 after), applied to the 60K-step schedule with the lock at 2K. † Timing run that locks at step 200; the after-lock level holds at about +4% from 2.6K to 4.3K steps after the lock. The end-to-end figure applies it to the 57K-step schedule with the lock at 1K.
Before the lock, each layer's grouped GEMM reads the weights of the whole pool rather than its own experts, so at our micro-batch sizes this phase is memory-bandwidth bound, which is consistent with its overhead growing with pool size. It is a systems cost of this phase, not of the locked model. The longer retained router costs 0.44% to 2.21% more active parameters per token from 182M to 830M.
At 1.5B we timed both arms on one node with matched parallelism (EP=8; vanilla at EP=1 is 17% slower): UniPool_lock is +88% per step after the lock. After the lock, the cause is mainly uneven load across expert-parallel devices, not extra computation (before the lock, it is routing and dispatch over the full pool):
- The learned allocation ignores device placement, so a layer's eight experts can share devices; the busiest device holds about 2.8 of them.
- Routing within a layer is less even than in vanilla, because UniPool balances load over the pool rather than per layer. In the 1.5B run, the first layer sends almost all tokens to one expert from step 3K on.
- With balanced placement and forced uniform routing, UniPool_lock is only 1.4% slower than vanilla under the same forced routing.
This is an engineering problem of expert parallelism rather than a cost of the method, and we addressed it with two EP-aware optimizations on top of the paper's compact post-lock dispatch: device-aware placement, where the allocation takes one expert from each device for every layer (an assignment problem solved exactly at the lock), and a per-layer balance loss after the lock, which evens out routing across each layer's devices. In timing runs that lock at step 200, overhead 200–400 steps after the lock is +99% without them and +18% with them, and it falls to +3.9% to +4.4% from 2.6K to 4.3K steps after the lock as routing evens out. The busiest device then carries 1.21 times the mean load (vanilla 1.11), and the first layer keeps a routing entropy of 2.6–2.9 bits (3 is uniform) through 4.3K steps, where the original run collapsed about 2K steps after the lock. At 182M (EP=1, a quality test only), the optimized recipe reaches 1.9020 against 1.9037 for a same-seed rerun of UniPool_lock, within run-to-run variation.
For matched wall-clock time: at 182M and 469M, vanilla trained for 25% and 15% more steps (75K and 69K instead of 60K) reaches 1.9065 and 1.7824, still behind the lock's 1.9040 and 1.7646. At 830M the estimate leaves most of the gain (about −0.026 of −0.0365). At 1.5B as run, it does not. With these optimizations and their +4% after the lock, the same estimate leaves about three quarters of the 1.5B gain (about −0.018 of −0.0247), assuming no quality cost from them, which the training run will confirm.
This answers your efficiency question with timing at every scale in the table, measured on the same footing up to 830M, where the lock keeps most of its gain at equal wall-clock time (measured at 182M and 469M, estimated at 830M). At 1.5B, these engineering optimizations cut the post-lock overhead from +88% to about +4%, and the training run will confirm the quality.
Response to Reviewer kE3U (1/3): Learned ownership versus transition compatibility
1. Separating learned ownership from transition compatibility
Your distinction is well taken, and the revision reframes Table 4 accordingly: it shows that the learned allocation matters, and the runs below test whether that is only transition compatibility. An allocation is defined relative to the expert weights at the lock, so C is more compatible at the lock by construction. The question we can test is whether that advantage is temporary or lasting. As you asked, we report the loss immediately before and after the lock and compare allocations with intermediate retained routing mass.
Loss around the lock (validation, 182M):
Frozen projection of a 1K checkpoint (held-out test data, hard mask, no further training):
As you note, the learned mask is chosen to retain routing mass, so this snapshot alone cannot separate the two effects; the training runs below do. A separate set of frozen checkpoints gives the same ordering at both scales. For learned, random, and owner-permuted masks respectively, ΔNLL is +0.41, +4.70 ± 0.53, and +5.39 ± 0.20 at 182M, and +0.55, +6.00 ± 0.28, and +6.39 ± 0.12 at 469M.
Over training (182M, original draw), B − C falls from 0.43 at the lock to below 0.03 by 10K, then holds near 0.013 from about step 30K to the end. Over five random allocations at 182M and three at 469M, every draw ends behind C on test (B − C of 0.0016–0.0132 and 0.0178–0.0277). So the disruption is real but temporary, while a difference persists to the end of training.
Allocations in between. The learned and random masks above retain about 86% and about 9% of routes, so we started runs in between at 469M. Each takes C's learned allocation at the lock and shuffles the owners among a random 25% or 50% of the experts, so every layer still has eight experts. These runs have not finished. For the three B draws, the validation gap to C over steps 30–41K was within 0.002 of the final test gap, so we compare over matched steps:
"Routing mass kept" is the share of the pre-lock routing counts, the ones used to choose the allocation, that the mask keeps. It is the allocation solver's coverage at 469M; the 86% above is the frozen projection at 182M. The residual grows with the share of experts reassigned, but not in proportion. Shuffling a quarter costs about a quarter of the average random-allocation gap (0.024 over steps 20–31K). Shuffling half already comes within 0.002 of the best of the three random draws. Each point is one run, and we will post final test values when the runs finish.
This answers your first question: C's advantage is not a temporary effect of the disruption. It persists long after the disruption has passed, and allocations that keep more of the learned ownership lose less. Following your suggestion, the revision presents it as this lasting benefit of the learned allocation, without attributing it to a specific specialization mechanism.