acceptodds
Under review as a conference paper at ICLR 2027

UniPool: Learning Expert-to-Layer Ownership from Brief Global Access

Abstract

Most Mixture-of-Experts (MoE) transformers preassign each expert to one layer for the whole of training. We study whether a brief global-access phase can learn a compatible expert-to-layer allocation that is then executed privately. UniPool-Lock gives every layer its own router over one global pool, balances aggregate pool usage, and scores experts with a scale-stable NormRouter. After the short ownership-learning phase, it assigns each expert to one layer, locks the disjoint allocation, and continues training as a layer-private MoE. With the same expert-FFN budget and routed expert FLOPs, this recipe lowers held-out loss relative to vanilla MoE by 0.024-0.037 across five dense-equivalent scales from 182M to 1.5B, including -0.0247 at 1.5B after 60B tokens. The ownership-learning phase lasts 2K steps, about 3.3% of training, and post-lock step time is within -0.7% to +2.8% of vanilla MoE in our throughput measurements. Controls locate the gain in full-pool training and in the allocation it produces: a random disjoint allocation fixed at initialization, trained with the same router and losses, matches vanilla, whereas locking the allocation learned during the full-pool phase recovers nearly all of the persistent full-pool gain, and substituting a random allocation at the lock forfeits about half of it. Keeping full-pool access throughout training (UniPool-Full) also improves over vanilla MoE at four scales and outperforms it with only 66.7% (182M) to 50% (469M and 650M) of its expert parameters.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture of Experts, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kE3UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UniPool to reconsider the fixed expert-to-layer assignment in standard MoE models. During an initial training phase, all layers can access a global expert pool. The method uses pool-level load balancing, NormRouter, and a regularizer to encourage layer-specific expert preferences. Based on early routing statistics, experts are then assigned exclusively to individual layers, and the model continues training as a conventional layer-private MoE.

Experiments across five model scales (182M–1.5B) show consistent improvements in language modeling loss and downstream accuracy over vanilla MoE under matched expert-FFN budgets and routed expert FLOPs. The authors also provide ablations on expert allocation and report near-vanilla training throughput after the allocation is locked.

Critical strengths
  1. A practical approach to learning expert allocation. The idea of using a short global-access phase to determine expert ownership is interesting. Unlike methods that require permanent cross-layer expert sharing, UniPool-lock returns to layer-private execution after the initial phase. This provides a practical way to explore more flexible expert allocation without maintaining the full cost of global sharing.

  2. Consistent empirical improvements with useful controlled experiments. The method improves test loss across multiple model scales under matched expert-FFN budgets. The comparisons between random and learned allocations in Table 4 are particularly useful, as they show that the allocation selected after global-access training matters. The downstream evaluations also support the improvements observed in language modeling loss.

Critical weaknesses

1.The experiments do not sufficiently establish the paper's central explanation of why learned ownership helps. The paper argues that assigning experts to layers before training leads to redundancy, and that learning expert ownership can address this problem. However, the current evidence does not clearly establish this explanation.

In Table 4, control B locks a random allocation after global-access training, while control C locks an allocation specifically chosen to preserve the routing preferences formed during that training. Therefore, C is naturally more compatible with the existing expert weights and routers. The performance gap shows that preserving these preferences is useful, but it does not establish whether the model has discovered a meaningful and stable expert-to-layer specialization, or whether the benefit mainly comes from avoiding disruption when switching to layer-private training.

The routing randomization experiments in Tables 1 and 7 are also indirect evidence. Greater sensitivity to random routing does not necessarily mean that experts have become more specialized or less redundant.

This distinction matters because learning meaningful expert ownership is the paper's central motivation. The current experiments demonstrate the usefulness of the training procedure, but do not sufficiently establish the proposed explanation for its gains.

2.The claimed efficiency advantage is not sufficiently validated under comparable training costs and larger-scale settings. A main selling point of UniPool-lock is that it retains most of the benefits of global expert sharing at nearly the same cost as vanilla MoE. However, the efficiency evidence remains limited.

The full-pool phase is 22–36% slower per step, and Table 6 reports post-lock and end-to-end timing only for the 182M and 469M settings on eight GPUs. No corresponding timing results are reported for the 650M, 830M, or 1.5B models.

Moreover, the comparison matches expert-FFN FLOPs rather than total training compute. Even after locking, UniPool retains routers that score and normalize over the full global expert index, introducing additional computation that grows with the number of layers.

Without comparisons under matched total training compute or wall-clock budgets, it remains unclear how much of the reported quality improvement represents a better efficiency-quality trade-off. This is important because near-vanilla training cost is a central motivation for the proposed locking strategy.

Questions for authors
  1. Distinguishing learned ownership from transition compatibility. In Table 4, how much of the gap between controls B and C can be explained by the disruption caused when the full-pool model is restricted to a layer-private allocation? Could the authors report the loss immediately before and after the locking transition, and compare allocations with similar retained routing mass? This would help determine whether the benefit reflects meaningful learned ownership beyond simply preserving the behavior established during the initial shared phase.

  2. Evidence of stable expert specialization. How stable are the learned expert-to-layer assignments across different data subsets or nearby training checkpoints? Can the authors provide more direct evidence of reduced expert redundancy or increased functional specialization, beyond sensitivity to random routing? Such evidence would strengthen the paper's central explanation.

  3. Efficiency under matched total compute. Can the authors report post-lock and end-to-end training costs for the larger model scales, particularly 1.5B, and compare UniPool-lock with vanilla MoE under the same total training compute or wall-clock budget? This would clarify whether the reported quality improvements translate into a practical efficiency advantage.

7 comments
AuthorsOpenReview ↗

Response to Reviewer kE3U (1/3): Learned ownership versus transition compatibility

1. Separating learned ownership from transition compatibility

Your distinction is well taken, and the revision reframes Table 4 accordingly: it shows that the learned allocation matters, and the runs below test whether that is only transition compatibility. An allocation is defined relative to the expert weights at the lock, so C is more compatible at the lock by construction. The question we can test is whether that advantage is temporary or lasting. As you asked, we report the loss immediately before and after the lock and compare allocations with intermediate retained routing mass.

Loss around the lock (validation, 182M):

StepVanillaABC
1K2.74772.72652.70132.6924
2K2.55042.61152.92752.4991
3K2.41382.41582.45432.3653

Frozen projection of a 1K checkpoint (held-out test data, hard mask, no further training):

AllocationΔNLL from full poolOriginal routes retained
Learned+0.3486.0%
Random (20 draws)+4.13 ± 0.468.7% ± 2.9%

As you note, the learned mask is chosen to retain routing mass, so this snapshot alone cannot separate the two effects; the training runs below do. A separate set of frozen checkpoints gives the same ordering at both scales. For learned, random, and owner-permuted masks respectively, ΔNLL is +0.41, +4.70 ± 0.53, and +5.39 ± 0.20 at 182M, and +0.55, +6.00 ± 0.28, and +6.39 ± 0.12 at 469M.

Over training (182M, original draw), B − C falls from 0.43 at the lock to below 0.03 by 10K, then holds near 0.013 from about step 30K to the end. Over five random allocations at 182M and three at 469M, every draw ends behind C on test (B − C of 0.0016–0.0132 and 0.0178–0.0277). So the disruption is real but temporary, while a difference persists to the end of training.

Allocations in between. The learned and random masks above retain about 86% and about 9% of routes, so we started runs in between at 469M. Each takes C's learned allocation at the lock and shuffles the owners among a random 25% or 50% of the experts, so every layer still has eight experts. These runs have not finished. For the three B draws, the validation gap to C over steps 30–41K was within 0.002 of the final test gap, so we compare over matched steps:

Allocation at the lockExperts with a new ownerRouting mass keptValidation loss − C
Learned (C)0 / 1920.74–0.75–
25% shuffled47 / 1920.61+0.006 (steps 20–31K)
50% shuffled95 / 1920.39+0.017 (steps 30–41K)
Random (B, three draws)about 184 / 192 expected–+0.018 to +0.030 (steps 30–41K)

"Routing mass kept" is the share of the pre-lock routing counts, the ones used to choose the allocation, that the mask keeps. It is the allocation solver's coverage at 469M; the 86% above is the frozen projection at 182M. The residual grows with the share of experts reassigned, but not in proportion. Shuffling a quarter costs about a quarter of the average random-allocation gap (0.024 over steps 20–31K). Shuffling half already comes within 0.002 of the best of the three random draws. Each point is one run, and we will post final test values when the runs finish.

This answers your first question: C's advantage is not a temporary effect of the disruption. It persists long after the disruption has passed, and allocations that keep more of the learned ownership lose less. Following your suggestion, the revision presents it as this lasting benefit of the learned allocation, without attributing it to a specific specialization mechanism.

AuthorsOpenReview ↗

Response to Reviewer kE3U (2/3): Stability and direct measures of redundancy

2. Stability and direct evidence of specialization

Stability checkResult
Learned mask vs. dominant layer on separate test data, 1K checkpoint182M: 86/96 and 90/96 experts in two prefixes; 469M: 177/192
Two reruns with the same seed and recipe, nondeterministic83/96 same owner (mean layer Jaccard 0.78); random allocations agree on 8.2/96
Pre-mask dominant layer at 2K and 2.5K vs. 1K182M: 64/96; 469M: 118/192 and 126/192

Preferences at 1K are therefore reproducible across data and reruns (86–94% of experts, against 8.2/96 for random allocations). After 1K the attenuation schedule itself reshapes routing: the soft effective owner count per expert rises from 1.44 at 1K to 2.73 at 2.5K at 182M, and agreement with the 1K owner is 64/96 at 182M and 118–126/192 at 469M. So the allocation chosen at 1K is stable across data subsets and reruns.

Direct redundancy measures. We now measure redundancy directly on the final checkpoints and compare with vanilla MoE at the same scale. For each MoE layer, we run every expert of that layer on the layer's real inputs, using held-out sequences that are identical across models, and record three quantities:

  • linear CKA between the outputs of each pair of experts;
  • best-substitute error: on the tokens routed to an expert, the relative squared error when the closest other expert in the layer answers instead (1.0 means no better than a zero output);
  • the LM-loss increase when every expert is replaced by its best substitute, divided by the increase when the layer is removed.

All three are medians over the first and second halves of the layers. CKA and substitute error are computed on expert outputs before gating, so the router's gate scale does not affect them.

ModelCKA ↓ (shallow / deep)Best-substitute error ↑Replace / remove ↑
182M vanilla, 3 seeds0.42–0.47 / 0.33–0.350.74–0.95 / 1.18–1.230.95–1.18 / 1.44–1.51
182M UniPool_lock, 4 runs (3 seeds)0.31–0.36 / 0.34–0.371.05–1.16 / 1.22–1.341.24–1.57 / 1.37–1.68
469M vanilla0.35 / 0.331.02 / 1.291.11 / 1.32
469M UniPool_lock0.29 / 0.341.31 / 1.271.44 / 1.38

In the shallow half of the network, UniPool_lock's experts are less redundant than vanilla's at both scales. At 182M, the ranges over three vanilla seeds and four UniPool_lock runs (three seeds, one of them run twice) do not overlap on any of the three shallow-layer measures, including when layer 0 is excluded. In vanilla, an expert's closest neighbour reproduces part of its output (squared error 0.74–0.95 of the output energy at 182M). In UniPool_lock, the closest other expert typically does no better than a zero output (median 1.05–1.31). Per layer, substitute error is higher in 67 of the 72 shallow-layer comparisons at 182M (6 layers × 4 UniPool_lock runs × 3 vanilla seeds) and in all 12 shallow layers at 469M. The deep half shows no difference.

The random-allocation controls have the same shallow-layer profile at 182M (control A: CKA 0.34, substitute error 1.06) and an intermediate one at 469M (control A: 0.33, 1.16; control B: 0.31, 1.12). At 469M, substitute error is above vanilla's in 12 of 12 shallow layers for UniPool_lock, 9 of 12 for control A, and 7 of 12 for control B; each is one run. So the lower redundancy is a property of UniPool's routing and balancing as a whole, and the learned allocation adds its own effect, visible in loss (Table 4) and in shallow-layer routing sensitivity: with each token's gate kept, randomizing routing within a layer's own experts costs UniPool_lock more than control A in the shallow half at both scales (+0.86 to +1.60 points for one- and two-layer windows, three tasks). In line with your point about Tables 1 and 7, the revision replaces Table 7's probe, which sets the gate to 1 and so inflates drops where learned gates are small, with this gate-preserving one. The revision adds these measurements and states the reduced redundancy for the shallow layers, where they show it.

We believe this resolves your request for direct evidence: measured on the experts themselves, redundancy is lower than in vanilla MoE in the shallow layers at both scales, and the allocation is stable across data and reruns.

AuthorsOpenReview ↗

Response to Reviewer kE3U (3/3): Efficiency at larger scales

3. Efficiency at larger scales

At EP=1, which both arms use up to 830M, post-lock steps stay within a few percent of vanilla (−0.9% to +3.8%), and end-to-end overhead is +0.4% to about +9%. Two settings need the closer look you ask for, and we now measure both: the full-pool phase, whose overhead grows with scale, and post-lock training under expert parallelism at 1.5B. The table uses median per-step wall time with matched micro-batch sizes; hosts differ between some pairs, so differences of a few percent are not resolved.

ScaleVanilla (s/it)Before lockAfter lockEnd-to-end
182M0.319+27%+2.1%+2.9%
469M0.780+38%−0.9%+0.4%
650M‡1.080+81%+3.8%+6.4%
830M, same node1.519+241%+1.3%about +9% (est.)
1.5B as run, EP=8 vs. EP=15.74–5.94–+53% to +62%+56% (total runtime)
1.5B, same node, both EP=84.94+106%+88%–
1.5B, same node, with EP optimizations†4.94+106%+4%about +6% (est.)

‡ Timing from per-step logs of an earlier 650M run whose candidate sets are K_l = 8–12; the K_l = 8 run has no per-step logs. The 830M row is a new same-node timing of the K_l = 8 recipe at EP=1 (steps 100–240 before the lock, 100–400 after), applied to the 60K-step schedule with the lock at 2K. † Timing run that locks at step 200; the after-lock level holds at about +4% from 2.6K to 4.3K steps after the lock. The end-to-end figure applies it to the 57K-step schedule with the lock at 1K.

Before the lock, each layer's grouped GEMM reads the weights of the whole pool rather than its own experts, so at our micro-batch sizes this phase is memory-bandwidth bound, which is consistent with its overhead growing with pool size. It is a systems cost of this phase, not of the locked model. The longer retained router costs 0.44% to 2.21% more active parameters per token from 182M to 830M.

At 1.5B we timed both arms on one node with matched parallelism (EP=8; vanilla at EP=1 is 17% slower): UniPool_lock is +88% per step after the lock. After the lock, the cause is mainly uneven load across expert-parallel devices, not extra computation (before the lock, it is routing and dispatch over the full pool):

  • The learned allocation ignores device placement, so a layer's eight experts can share devices; the busiest device holds about 2.8 of them.
  • Routing within a layer is less even than in vanilla, because UniPool balances load over the pool rather than per layer. In the 1.5B run, the first layer sends almost all tokens to one expert from step 3K on.
  • With balanced placement and forced uniform routing, UniPool_lock is only 1.4% slower than vanilla under the same forced routing.

This is an engineering problem of expert parallelism rather than a cost of the method, and we addressed it with two EP-aware optimizations on top of the paper's compact post-lock dispatch: device-aware placement, where the allocation takes one expert from each device for every layer (an assignment problem solved exactly at the lock), and a per-layer balance loss after the lock, which evens out routing across each layer's devices. In timing runs that lock at step 200, overhead 200–400 steps after the lock is +99% without them and +18% with them, and it falls to +3.9% to +4.4% from 2.6K to 4.3K steps after the lock as routing evens out. The busiest device then carries 1.21 times the mean load (vanilla 1.11), and the first layer keeps a routing entropy of 2.6–2.9 bits (3 is uniform) through 4.3K steps, where the original run collapsed about 2K steps after the lock. At 182M (EP=1, a quality test only), the optimized recipe reaches 1.9020 against 1.9037 for a same-seed rerun of UniPool_lock, within run-to-run variation.

For matched wall-clock time: at 182M and 469M, vanilla trained for 25% and 15% more steps (75K and 69K instead of 60K) reaches 1.9065 and 1.7824, still behind the lock's 1.9040 and 1.7646. At 830M the estimate leaves most of the gain (about −0.026 of −0.0365). At 1.5B as run, it does not. With these optimizations and their +4% after the lock, the same estimate leaves about three quarters of the 1.5B gain (about −0.018 of −0.0247), assuming no quality cost from them, which the training run will confirm.

This answers your efficiency question with timing at every scale in the table, measured on the same footing up to 830M, where the lock keeps most of its gain at equal wall-clock time (measured at 182M and 469M, estimated at 830M). At 1.5B, these engineering optimizations cut the post-lock overhead from +88% to about +4%, and the training run will confirm the quality.

Reviewer_WtoTOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces a global expert pool for MoE models, instead of assigning a fixed set of experts to each layer from the beginning of training. The authors argue that this approach increases model capacity by allowing more flexible expert-to-layer assignments during early training. Experimental results across different model scales are provided to support this claim.

Critical strengths
  1. Novel approach. The idea of learning expert-to-layer ownership through a global expert pool is novel and interesting. It provides a new perspective on improving MoE model capacity and expert diversity.

  2. Extensive experimental evaluation. The authors provide extensive experimental results across different model scales, along with several ablation studies. Collectively, these results provide a useful primary evaluation of the proposed method to verify the extent of its effectiveness.

Critical weaknesses
  1. Lack of clear evidence that performance gains are independent of additional training cost. The performance gain could primarily come from allowing each layer's router to explore a larger pool of experts during early training, rather than from the proposed expert-to-layer ownership mechanism itself. Moreover, the authors do not adequately separate the effect of additional training time from the performance improvement, both for UniPool-lock (up to 3.5% overhead) and UniPool-full (22–36% overhead). A comparison with vanilla MoE trained for the same wall-clock duration would help establish whether the method offers a genuine performance improvement beyond the additional training cost.

  2. Lack of clear evidence that the proposed method improves model capacity. Table 1 considers randomizing the router of only one layer at a time, whereas randomizing multiple layers simultaneously could lead to substantially larger performance degradation. A similar multi-layer analysis for UniPool would help better assess whether the proposed method actually increases model capacity and strengthens expert specialization.

Questions for authors

Please address the two weaknesses mentioned.

5 comments
AuthorsOpenReview ↗

Response to Reviewer WtoT (1/3): Update to our earlier response, training cost, and exploration versus ownership

This reply builds on our earlier response to you. The equal-wall-clock gains given there (0.0245 at 182M and 0.0298 at 469M) were read from the longer vanilla runs described below, at the point where their elapsed time matches UniPool_lock's. Below we add the final values of the same runs: even after the full 25% and 15% more steps, far beyond UniPool_lock's extra time, vanilla does not catch up. The reduced-pool UniPool_lock results in our earlier response (182M and 469M) will be reported in the revision; we will not be able to extend them to all scales within the discussion period. On "effective capacity": as in our earlier response, the evidence is Figure 2, whose reduced-pool results for UniPool_full measure effective capacity per stored expert parameter. We do not use routing sensitivity to argue for capacity (see part 3).

Training cost. All compared runs use the same number of steps and tokens, so the extra wall-clock time buys no additional updates. The remaining question is efficiency at equal wall-clock time, which two vanilla runs with the same recipe answer, trained for 25% and 15% more steps (75K and 69K instead of 60K, cosine schedule over the full run):

ScaleLock end-to-end overheadVanilla with extra stepsUniPool_lock
182M+2.9%+25% steps: 1.90651.9040
469M+0.4%+15% steps: 1.78241.7646

At these two scales, vanilla receives far more extra training than the lock's overhead and still does not beat it. This resolves the training-cost concern for UniPool_lock at these scales: the gain is not bought with extra training time. Those two long runs also give one estimate of how vanilla's loss improves with steps: −0.113 per unit of log steps at both scales. Below, we apply it to scales where no long vanilla run exists, so those figures are estimates rather than measurements.

Your concern applies to two cases, which the revision reports explicitly:

  • UniPool_full. At +27% (182M) and +38% (469M) during full-pool execution, its gain roughly matches what vanilla gains from the same extra time. At 182M its 1.9030 is within the tie threshold of vanilla at +25% steps. At 469M, extrapolating vanilla to +38% steps puts the two about equal (about 1.762 against 1.7636, within the tie threshold). Following your point, the revision presents it as a reference and a parameter-efficiency study rather than as a training recipe.
  • 1.5B as run. UniPool_lock took 161.8 h against vanilla's 104.0 h (+56%), and the estimate suggests vanilla would beat it at equal wall-clock time. After the lock, this overhead comes mainly from uneven load across expert-parallel devices rather than from extra computation: with balanced placement and forced uniform routing, UniPool_lock is 1.4% slower than vanilla under the same forced routing. Before the lock, the extra time is routing and dispatch over the full pool, as in our earlier response. The two arms used different parallelism; timed on one node with the same parallelism (EP=8), the unmodified recipe is +88% per step after the lock. Two changes address it: the allocation takes one expert from each device for every layer, and a per-layer balance loss is added after the lock. Timed on the same node as vanilla, they bring post-lock overhead down to about +4%. At that level the estimate leaves about three quarters of the 1.5B gain at equal wall-clock time (about −0.018 of −0.0247).

Exploration versus ownership. Control B tests this directly. The shared phase is one half of the method and the learned allocation is the other, and the learned allocation ends ahead of all eight random allocations we drew. At 182M we now have five random allocations, all with the same model seed. Full-pool access followed by a random lock recovers 53%, 80%, 94%, 63%, and 78% of the 0.0279 improvement of C over control A, 74% on average (B − C of 0.0132, 0.0056, 0.0016, 0.0102, and 0.0062). So at 182M the full-pool phase accounts for most of the gain, and the learned allocation adds about 0.007 on average (95% interval over draws 0.002–0.013), with a large spread across draws.

At 469M the split is different. Of the 0.0317 improvement of C over control A, three random allocations recover 31%, 44%, and 13% (B − C of 0.0218, 0.0178, and 0.0277; a second run of the first allocation gives 0.0227). There the learned allocation accounts for 56–87% of the gain, about 70% on average.

Following your question, the revision reports both scales instead of the single "about half" from one draw. This resolves whether the gain comes only from exploration: it does not, and the learned allocation's share is larger at the larger scale.

AuthorsOpenReview ↗

Response to Reviewer WtoT (2/3): Multi-layer randomization on production MoEs

Multi-layer randomization. You are right that the single-layer result is not enough on its own. We have now randomized contiguous windows of k layers together, both on production MoEs and on our own models.

Production MoEs. Each window is evaluated once, for k up to 4. Mean drop in the five-task average for deep-half windows (points):

Model (MoE layers)k=1k=2k=3k=4Actual drop / sum of single-layer drops
OLMoE-1B-7B (16)5.0610.3316.1920.661.02–1.18
Qwen1.5-MoE (24)1.633.134.896.760.96–1.03
DeepSeek-V2-Lite (26)1.152.423.764.901.07–1.20
Qwen3-30B-A3B (48)0.961.872.853.780.98–1.00

In the deep half the drop stays close to additive up to k=4: a window costs 0.96–1.20 times the sum of its single-layer drops, so a layer costs about the same, or up to about a fifth more, when randomized with its neighbours. The single-layer averages therefore do not hide a sudden collapse in this range. The earliest layers behave differently: layers 0–3 of Qwen3 together cost 33.8 points against 10.8 predicted, and layers 0–2 of Qwen1.5 cost 18.5 against 7.0.

Following your suggestion, the revision uses Table 1 only as motivation: it shows a small per-layer cost, which these results show adds up across layers without a sudden collapse up to k=4.

The multi-layer analysis you asked for is therefore done, on production MoEs here and on our own models in part 3.

AuthorsOpenReview ↗

Response to Reviewer WtoT (3/3): Multi-layer randomization on our models

Our models. We randomized windows of k = 1, 2, 3, 4, 6, and 12 layers at 182M, and k = 1 and 2 at 469M. Each scale has three models, one training run each: vanilla, control A (a random allocation fixed at initialization), and UniPool_lock. Control A shares UniPool_lock's router and losses, so the comparison between them isolates the learned allocation.

First, an improvement to the Table 7 probe. Its probe sets the selected expert's gate to 1, which inflates the drop wherever learned gates are small. In control A and UniPool_lock, NormRouter's gates in the first four layers are about 0.01–0.16, against 0.16–0.28 for vanilla. With gates set to 1, the mean single-layer drop at 182M is 2.34 points for vanilla, 4.30 for control A, and 4.82 for UniPool_lock. Control A has no learned allocation, yet it shows most of the gap that Table 7 attributes to UniPool_lock. The revision therefore replaces Table 7 with the gate-preserving comparison below. The UniPool_lock probe in Table 7 uses a disjoint checkpoint. For the comparison below, the probe instead draws uniformly among the layer's own experts and keeps each token's own gate.

The table below gives differences in drop between models over the shallow half of the network. Each entry is in points, averaged over windows, with a 95% CI from paired per-example differences. It uses HellaSwag, PIQA, and ARC-E, the three tasks above chance at these scales; ARC-C and WinoGrande sit at chance.

Shallow half182M, k=1182M, k=2469M, k=1469M, k=2
UniPool_lock − control A+0.97 [+0.29, +1.64]+1.60 [+0.72, +2.48]+0.86 [+0.39, +1.33]+1.39 [+0.76, +2.02]
UniPool_lock − vanilla+1.86 [+1.22, +2.50]+2.88 [+2.02, +3.74]+0.96 [+0.51, +1.42]+1.70 [+1.10, +2.30]
Control A − vanilla+0.89 [+0.28, +1.50]+1.28 [+0.47, +2.10]+0.10 [−0.34, +0.54]+0.32 [−0.25, +0.88]
  • Shallow half. UniPool_lock is more sensitive to randomized routing than both control A and vanilla, at both scales and both window sizes.
    • With all five tasks, the k=2 differences against control A remain significant: +1.25 [+0.26, +2.25] at 182M and +0.83 [+0.07, +1.59] at 469M.
    • The k=1 difference at 469M is positive but not significant on five tasks: +0.38 [−0.21, +0.97].
    • At 182M, a shuffle that keeps each expert's load gives the same direction (k=2, three tasks: +1.26 [+0.36, +2.15]).
  • Control A. It sits between the other two at 182M and matches vanilla at 469M. At 469M, therefore, the added sensitivity comes from the learned allocation.
  • Deep half. There is no such difference. At 469M with k=2, UniPool_lock is slightly less sensitive than control A: −0.87 [−1.50, −0.25].
  • Window size. Drops grow roughly additively. A two-layer window costs 0.87–1.07 times the sum of its two single-layer drops at 182M, and 0.99–1.07 times at 469M. At 182M we see no collapse up to k=6. From k=4 on, all three models approach the chance floor, which limits what larger windows can show at this scale.
  • Robustness. The direction holds across window sizes, perturbations, and two scales. The intervals reflect evaluation sampling, not training-seed variance, and are not corrected for the number of cells.

What this shows: under the learned allocation, which expert a shallow-layer token reaches matters more than under a random allocation or in vanilla MoE. Routing sensitivity alone does not establish specialization, so we also measured redundancy directly on the final checkpoints. In the shallow half, an expert's closest neighbour in UniPool_lock reproduces its output no better than a zero output (best-substitute error 1.05–1.31, against 0.74–1.02 for vanilla) at both scales. Random-allocation controls show a similar profile, so we attribute this to UniPool's routing and balancing as a whole. As in our earlier response, our evidence on capacity is Figure 2, effective capacity per stored expert parameter; we do not use routing sensitivity for it. What the paper supports is lower loss at a matched budget and this parameter efficiency.

With this, we believe your second concern is resolved in the form the evidence supports: the multi-layer analysis is done on four production MoEs and on our own models at two scales, and our claims are narrowed to what it shows.

Reviewer_T1NQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1(−1)
Presentation
2(−1)
Soundness
2
Read review
Paper summary

This paper studies whether learning the expert-to-layer assignment during training improves MoE models. UniPool initially lets every layer route over a shared expert pool using its own NormRouter. A pool-wide balancing loss keeps overall usage balanced, while a cardinality regularizer encourages each layer to concentrate on a small set of experts. The method chooses a disjoint assignment from held-out routing counts at step 1K and locks it at step 2K. Training then continues with private experts for each layer.

With matched expert parameters and routed compute, UniPool_lock gives lower test loss than vanilla MoE from 182M to 1.5B. Its post-lock step time is close to vanilla in the measured configurations. At 182M, a random allocation after the same shared phase retains about half of the improvement; the authors attribute the remaining gain to the learned allocation. The persistent shared-pool variant, UniPool_full, also improves parameter efficiency at several scales.

Critical strengths
  1. Learning expert-to-layer ownership through a short shared phase may be an interesting idea.
  2. The method shows some empirical gains in test loss and downstream performance, with step time close to vanilla configurations.
Critical weaknesses
  1. I am not sure how much of the B-C gap reflects layer-specific specialization. As the authors note, this comparison measures the allocation’s value after the shared phase. A random lock may assign layers to experts trained mainly on other layers’ tokens, introducing a transition cost. The current results do not separate these effects. This limits the interpretation of the learned allocation, although the main empirical result still holds.
  2. The sensitivity to the shared phase and cardinality regularizer is unclear. All lock runs allocate at 1K and lock at 2K, including the 1.5B run with twice the batch size. The same schedule working across scales is encouraging, but it remains unclear how short the phase can be or how much the regularizer determines the allocation. The authors acknowledge the missing schedule and regularizer ablations in App. B.
Questions for authors

I am mainly not yet convinced of the idea’s practical value. The reported gains are modest, and I have some concerns about the experimental evaluation.

Some more questions

  1. Appendix E shows training curves only for UniPool_full and the sharing-scope ablation. Could you show the B and C loss curves after the lock? Does the gap persist or shrink during training? This would help distinguish a lasting benefit from the transition cost of a mismatched allocation.
  2. How sensitive are the results to the phase length and to the cardinality regularizer?
5 comments
AuthorsOpenReview ↗

Response to Reviewer T1NQ (1/2): B and C after the lock

Do the B and C curves converge after the lock?

The curves answer this directly: the gap does not close. It shrinks quickly after the lock and then stops shrinking: B − C is 0.0134, 0.0126, and 0.0132 over the last three 10K-step windows, and 0.0132 on test. The revised PDF will add the curves. The table below is our original B run, which uses one random allocation. More allocation draws are summarized after it.

StepB − CA − B
1K (before B and C diverge)+0.0089+0.0252
2K (lock)+0.4285−0.3160
3K+0.0890−0.0386
5K+0.0461+0.0336
10K+0.0266+0.0255
20K+0.0200+0.0136
30–40K mean+0.0134+0.0135
40–50K mean+0.0126+0.0150
50–59K mean+0.0132+0.0148
Test at 60K+0.0132+0.0147

As you anticipated, a mismatched allocation has a transition cost, and it is large: B − C is 0.43 at the lock and falls below 0.03 by 10K. A smaller gap of about 0.013 then holds from roughly step 30K to the end of training.

The size of that residual depends on the draw. We have since trained B with four more random allocations, with the model seed unchanged. Their transition costs are similar (B − C of 0.08–0.14 at 3K and 0.015–0.023 at 10K), but at 60K they end 0.0016 to 0.0102 behind C, against 0.0132 for the original draw. Over the five draws, B − C on test averages 0.0074 (SD 0.0045), and every draw ends behind C. For scale, rerunning C itself with the same seed gives 1.9037 against 1.9040.

At 469M, A − C holds at about 0.031 from step 10K to step 59K, with 0.0317 on test, and the residual B − C is larger than at 182M. Three allocation draws end 0.0218, 0.0178, and 0.0277 behind C (mean 0.0224), all above the largest 182M value. A second run of the first allocation ends 0.0227 behind, so two runs of the same allocation differ by 0.0009. The 0.0277 draw had a much larger transition cost (B − C of 0.19 at 5K, against 0.03–0.04 for the other two) and stayed furthest behind from then on. Its gap was already the largest before its one loss spike after the lock, at step 11.2K, and did not widen across it.

A pure transition cost would keep decaying; this gap does not over the last 30K steps. Following your point, the revision presents this as a lasting benefit of the learned allocation, established across eight draws at two scales, and does not tie it to a single mechanism such as layer-specific specialization.

On the 1K row: four runs with the same configuration and seed reach 2.6924, 2.7013, 2.6873, and 2.7004 at step 1K (SD 0.0067). The B − C difference at that point is therefore within run-to-run GPU nondeterminism, and we found no configuration difference other than the allocation rule.

Allocations between learned and random. To separate the two effects you describe, we also started 469M runs that keep C's learned allocation but shuffle the owners of a random 25% or 50% of the experts. Over matched steps, their interim validation gaps to C are +0.006 (25%, steps 20–31K) and +0.017 (50%, steps 30–41K), against +0.018 to +0.030 for the three random allocations over steps 30–41K; for those three, this window came within 0.002 of the final test gap. The cost grows with the share of experts reassigned. We will post final values when these runs finish.

Together, these results resolve whether the B − C gap is only a temporary transition cost: it is not. Following your suggestion, the revision reframes the learned allocation's contribution as this lasting benefit, which grows with how much of the learned ownership is kept.

AuthorsOpenReview ↗

Response to Reviewer T1NQ (2/2): Phase length, the regularizer, and effect size

How sensitive are the results to the phase length and the regularizer?

Final test loss by lock step (the shorter phases are new runs, each with the allocation at half the lock step):

ScaleLock at 500Lock at 1KLock at 2KLock at 8K
182M1.90411.90221.90401.9070
469M1.7666–1.76461.7672

Each cell is one run. A same-seed rerun of the 2K run gives 1.9037, so run-to-run variation at this point is about 0.001. At 182M the lock can move from step 2K to step 500 (0.8% of training) with no measurable change. This also shortens the full-pool phase, which is the expensive part of training, by a factor of four. At 469M, locking at step 500 gives 1.7666 against 1.7646 (+0.0020), which keeps 94% of the lock's gain over a random allocation fixed at initialization (control A, 1.7963). A same-seed rerun at 469M, now training, will give the noise level there.

For the regularizer, we now have a complete 182M lock run without it (cardinality loss coefficient 0, everything else as the 2K run). Removing it lowers test loss by 0.005 (1.8988 against 1.9040; rerun 1.9037). That is one run, at the paper's 0.005 tie threshold, so quality at this scale does not depend on the regularizer. What it does affect is how evenly each layer routes, which matters for load under expert parallelism, and the revision will describe it that way. The same run at 469M is in progress: over steps 20K–27K of 60K, its validation loss is 0.001 below the run with the regularizer, so far the same. At 182M without it, routing before the lock is much broader (about 12 effective experts per layer at 1K versus about 9), and after the lock the first layer collapses to about one effective expert (about 1.5–3 with the regularizer). That has no visible quality cost at EP=1, but it concentrates that layer's tokens on one device under expert parallelism. For persistent full-pool training at 182M, adding the regularizer changes test loss from 1.9030 to 1.9061 with target 8 and to 1.8970 with target 16.

With these runs, the sensitivity question is answered: the lock step can move to 500 with no measurable change at 182M and +0.0020 at 469M, and the regularizer does not affect quality at 182M.

"The reported gains are modest." In perplexity terms, UniPool_lock gives −2.4% to −3.6% across the five scales. To judge practical value, we compare with vanilla given more training: two vanilla runs with the same recipe, trained for 25% more steps at 182M (75K instead of 60K) and 15% more at 469M (69K), with the cosine schedule spanning the longer run and the same batch size, so they also see 25% and 15% more tokens. At 182M the longer vanilla reaches 1.9065, still behind the lock's 1.9040, whose end-to-end cost is +2.9%. At 469M it reaches 1.7824, against the lock's 1.7646 at +0.4% cost. At these two scales the gain is therefore not only measurable but practical: it holds when vanilla gets far more extra training than the lock's extra time.

Thank you for the detailed response and the additional experiments. I have also read your responses to the other reviewers. The new B/C curves and the sensitivity runs answer my two specific questions, and I appreciate the longer vanilla runs at 182M and 469M. However, the responses also raise further concerns for me: the 1.5B run as reported took about 56% more wall-clock time than vanilla, the proposed fixes have not yet been validated by a full training run, and at 182M most of the gain comes from the brief full-pool phase rather than from the learned allocation, which changes a central claim of the submission. Overall, I am still not convinced of the practical value of the idea: the gains remain modest, and my concerns about the experimental evaluation remain. I therefore keep my score for now.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.