- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes BMoE that shares pools of latent expert cores across contiguous blocks of layers while retaining layer-specific input/output projections and routers. Eight configurations with 36 effective layers, ranging from 2.5B-A1.3B to 7.1B-A1.3B, are trained for approximately 208B tokens and compared with per-layer latent MoE (7.1B-A1.3B) and a whole-layer Loop baseline (3.8B-A1.3B). The reported four-layer-sharing configuration reduces the model from 7.1B-A1.3B to 4.0B-A1.3B (43.2% fewer total parameters) while improving the nine-task mean by 0.83 percentage points over the per-layer baseline.
Critical strengths
-
Fewer parameters with competitive model quality. Sharing experts across layers reduces the number of weights that need to be stored or moved in GPU. The reported experiments show that this can preserve or improve model quality, making it a promising way to reduce the training & inference requirements of MoE models.
-
Useful evidence about which components to share. The ablation shows that keeping each layer's input/output projections and normalization separate works better than sharing these components along with the experts. This provides a useful design insight for cross-layer expert sharing.
Critical weaknesses
-
My major concern is the constant learning-rate training setup. All models use a learning rate of 3.36e-4 after a 500-step warmup, with results reported at step 11,000 and no learning-rate decay. Recent MoE studies such as LatentMoE and Qwen3 include learning-rate decay in their pretraining schedules. The models may still be insufficiently converged at the chosen lr, and the observed gains could disappear after decay. It is thus important to check whether the quality ranking persists after a common learning-rate decay phase.
-
Missing balancing ablations and expert-load statistics. Block-wise balancing relaxes the balancing constraint on each individual layer, which could itself improve downstream performance. Please compare layer-wise and block-wise balancing with the BMoE architecture, expert pool size, and training setup fixed. The paper reports routing overlap and JS divergence, but not how evenly tokens are distributed across experts. Per-layer and block-level expert token counts would help verify the claimed balancing behavior and clarify its contribution to the gains.
Questions for authors
- Please address the training and balancing concerns detailed in Weaknesses 1 and 2.
- Table 1 matches the stored expert count between S4 and Loop, but S4 uses 256 candidate experts per layer whereas Loop uses 128. Could you add a Loop baseline with 256 candidates to assess the benefit of the larger pool, or more fine-grained expert architecture?
- What practical memory and speed benefits does BMoE provide over Per-Layer MoE and Loop? Please report peak training memory, training throughput, and inference latency/throughput under matched conditions, preferably separating prefill and decoding.
4 comments
This comment continues the previous one; Q2 is in the next comment. Results are in Appendices E and I of the revised PDF.
Notation. T = stored expert instances; N = candidate experts per layer or pool; Loop = the whole-layer repeated-call baseline; Sk as in Part 1.
W2, second half: Expert-load statistics
Main models, about 1M held-out tokens from five domains (per-layer counts sum to 8 × the number of tokens):
| Model | Unit | Units | CV mean [range] | Effective experts / N | Dead (max over units) |
|---|---|---|---|---|---|
| Per-Layer | layer | 36 | 0.105 [0.061, 0.233] | 127.2 / 128 | 0 |
| BMoE-S2 | pool | 18 | 0.092 [0.069, 0.193] | 254.8 / 256 | 0 |
| BMoE-S2 | layer | 36 | 0.337 [0.248, 0.506] | 241.0 / 256 | 3 |
| BMoE-S4 | pool | 9 | 0.080 [0.058, 0.124] | 255.2 / 256 | 0 |
| BMoE-S4 | layer | 36 | 0.587 [0.310, 0.909] | 214.5 / 256 | 43 |
| BMoE-S6 | pool | 6 | 0.077 [0.054, 0.114] | 255.2 / 256 | 0 |
| BMoE-S6 | layer | 36 | 0.784 [0.334, 1.336] | 190.8 / 256 | 66 |
- Pools are even. Mean pool CV is 0.077–0.092 with no dead experts and 254.8–255.2 effective experts out of 256, below the layer-level CV of Per-Layer (0.105).
- Within a pool, layers concentrate, more so as more layers share it (mean layer CV 0.34, 0.59, 0.78 for S2, S4, S6). A constraint applied to the pool allows this: in M-S4 a single layer leaves up to 43 experts unselected, yet no expert of the pool is dead.
- This does not cost quality. In the XS ablation (Part 1), layer-wise balancing removes the per-layer dead experts (34 to 0) and raises effective experts per layer from 162 to 242 with equal quality, so we read within-layer concentration as division of labor, not collapse.
Manuscript changes: Appendix E (load statistics table and discussion).
Q1: Loop-256 and finer-grained experts
Table 1 pairs models by stored expert instances T, while Loop has 128 candidates per layer and BMoE has 256. We added Loop-256 on XS (256 candidates per layer, the same T as S2):
| Model | T | Total params | Loss | Wiki PPL | 9-task mean |
|---|---|---|---|---|---|
| Loop-128 | 768 | 0.509B | 2.6717 | 36.526 | 39.99 |
| Loop-256 | 1,536 | 0.764B | 2.6143 | 34.895 | 40.86 |
| BMoE-S2 | 1,536 | 0.785B | 2.6053 | 33.231 | 40.81 |
(1) A wider candidate pool accounts for most of the gain, as Contribution 1 (duplicated storage turned into a wider per-layer pool) states. Loop-128 to Loop-256 doubles T (768 to 1,536) and candidates per layer (128 to 256) and lowers loss by 0.057, most of the 0.066 advantage of S2 over Loop-128. This supports the mechanism: the same storage buys more quality as a wider pool. The step doubles T and candidates together, so it does not separate the two. Table 1 pairs models by T because T sets the storage budget compared.
(2) With T and candidates both matched, the sharing boundary still gives a measurable gain. BMoE-S2 has 0.009 lower loss than Loop-256 (about 2 XS seed standard deviations) and 1.66 lower Wiki PPL (4.8% relative); the nine-task means are equal (40.81 vs 40.86; the XS seed standard deviation of this mean is 0.14–0.18pp). T, N, and activated computation match. Only the sharing boundary differs: the loop reuses the whole layer (attention, router, interfaces), whereas BMoE shares only the latent expert cores.
(3) Cost. S2 keeps six more sets of attention, router, and interfaces: 0.021B parameters (+2.7%).
Finer-grained experts (256 half-width experts, top-16, matching Loop-128 in storage and computation) were not completed in the discussion period; they need a new run with re-tuned top-K and balancing coefficients, and we leave them to future work.
Manuscript changes: Appendix I (XS variants table and discussion).
This comment completes our answer. Results are in Appendix F of the revised PDF.
Notation. Sk = BMoE with k adjacent layers sharing one pool; S4+SP = S4 with interfaces also shared; Loop = the whole-layer repeated-call baseline; N = candidate experts per layer; K = experts selected per token.
Q2: Memory, throughput, and inference
| Model | Total params | bf16 weights | Train step time (24 GPUs, same configuration) | Peak train memory / GPU |
|---|---|---|---|---|
| Per-Layer | 7.056B | 14.16 GB | 38.6 s | 58.6 GB |
| BMoE-S2 | 7.063B | 14.17 GB | 55.9 s | 60.5 GB |
| BMoE-S4 | 4.006B | 8.06 GB | 57.3 s | 57.7 GB |
| Loop | 3.762B | 7.55 GB | 38.7 s | 54.7 GB |
(1) Weights are the reliable gain. S4 stores 57% of the Per-Layer weights, which lowers deployment size, checkpoint transfer, and the number of shards.
(2) Peak training memory drops by only 0.9 GB per GPU. With EP=8 the expert weights are already partitioned, so the 6.10 GB weight difference becomes 6.10/8 ≈ 0.76 GB per GPU, consistent with the measured 0.9 GB. The other ~57 GB (activations, optimizer states, non-expert parameters) does not depend on expert sharing. Loop's peak is 3.9 GB lower than Per-Layer, more than this argument explains; we did not decompose it. The storage benefit therefore appears in global weights, not in per-GPU training memory.
(3) Step time is 45–48% higher, and the overhead follows candidates per layer, not sharing. All 256-candidate models (S2, S4, S4+SP) take 55.9–57.3 s and all 128-candidate models 38.6–38.7 s (Per-Layer, Loop). S4 stores 43% fewer experts than S2 at the same step time, so the cost is in routing and dispatch width. We did not write a fused dispatch kernel for 256 candidates.
(4) Prefill differs by about 1.5% of MoE-branch FLOPs. Prefill is dominated by the selected experts, whose computation is identical (same K, width, activated computation). Only the router term 2dN in Eq. 9 varies: 0.79 MFLOP for N=256 and 0.39 MFLOP for N=128, i.e. 2.9% and 1.5% of the MoE branch (4dℓ + 6Kℓh + 2dN). This is an analytic estimate from the paper's FLOP formula, not a measured prefill time.
(5) Decoding: no difference at bs=1, 1.17x expert reads at bs=64. Decoding is weight-bandwidth bound and reads only the K selected experts per token, so at bs=1 the bytes read do not depend on pool size. At bs=64, Per-Layer touches 110.7 of 128 experts (86%) and S4 touches 129.8 of 256 (51%): a ratio of 1.17x, below the 1.77x expected under uniform routing over a doubled pool. Only the expert-weight term grows, so the end-to-end read increase is below 17%.
(6) Why no wall-clock inference time. The only implementation is the HF reference, which loops over experts one by one. At bs=1 all four models take about 82 ms/token, so the measurement is dominated by interpreter overhead and the differences between models are below implementation noise. Items (4) and (5) are an analytic estimate and a measured count.
Manuscript changes: Appendix F (weights, step time, peak memory, prefill FLOP breakdown, decode expert reads).
Thank you for the additional experiments. The rebuttal response strengthen the submission. Two concerns still remain.
- The training-loss ranking is unchanged after cosine decay on both smaller backbones, XS (0.31B active parameters, 18B training tokens) and S (0.48B active parameters, 20B training tokens). But at the main scale, only S4 was continued through decay, so the relative ranking after a common cooldown remains uncertain.
- At XS, Loop-256 and BMoE-S2 match the number of candidate experts per layer (256), the number of stored expert instances (1,536), and the activated computation. BMoE-S2 has lower training loss (2.6053 vs. 2.6143) and WikiText perplexity (33.23 vs. 34.90), while the nine-task mean is nearly identical (40.81 vs. 40.86) and S2 uses 2.7% more total parameters. At the main scale, the Loop baseline uses 128 candidates, so the downstream advantage over a matched Loop baseline remains unclear.
I consider the paper borderline and would like to discuss these points with the Area Chair before finalizing my recommendation.
Thank you for four testable questions, and for restating the main result accurately (43.2% fewer parameters, +0.83pp nine-task mean). Below we add the evidence you identified as missing. This comment answers W1 (constant learning rate) and the first half of W2 (balancing ablation). The second half of W2 (load statistics) and Q1 (Loop-256) are in the next comment, and Q2 (memory and speed) in the one after.
Notation. Sk = BMoE with k adjacent layers sharing one pool of latent experts; M = the main 36-layer model; XS/S = the two smaller backbones.
W1: Constant learning rate
(1) At small scale, the ranking is unchanged after decay. XS and S are trained from scratch with a cosine decay over the last 10% of steps and identical settings otherwise:
(2) At the main scale, the decay phase does not expose under-convergence. We extended M-S4 beyond step 11,000 under the same constant learning rate, then decayed it to zero with a 2,000-step cosine schedule starting at step 14,000 (264B to 302B tokens):
Loss, Wiki PPL, and PTB PPL improve monotonically as the learning rate falls to 0. The nine-task mean rises from 52.17 to 52.93 and then stays flat (52.93 to 52.91, a 0.02pp difference within few-shot seed variation). There are no spikes and no degradation. This run only checks whether training was so far from convergence that decay could reverse the ranking. It is not a cross-model comparison: the other three main models were not continued, and the token budget (302B vs 208B) makes these rows incomparable with Table 2.
(3) What we did not do, and the claims that follow. Same-protocol continuation of the other three main models was not possible within the discussion period. At the main scale we therefore claim only that the decay phase behaves normally, not that the ranking is unchanged after decay. Decay lowers the M-S4 loss by about 0.056, so we state the main-scale claims at the strength the evidence supports. (i) The S2 advantage over Per-Layer is the main-scale claim: at both small scales its gap is preserved through decay (−0.0046 → −0.0045 at XS, −0.0055 → −0.0043 at S), and at M it is 14 times the XS seed noise (seed noise measured at XS only). (ii) For S4, the claim is parity with Per-Layer at 43.2% fewer parameters: loss and both perplexities are lower, and the nine-task mean is +0.83pp (95% interval lower end −0.25pp under a conservative evaluation standard error, App. D).
Manuscript changes: Appendix G (Tables on small-scale decay and the M-S4 continuation); Appendix H (backbone family).
W2, first half: Block-level vs. layer-wise balancing
On XS we fix the BMoE-S4 structure, pool size, N, data, and training settings, and change only block-level balancing to standard layer-wise balancing (same coefficient, 0.001):
Manuscript changes: Section 3.3 rewritten (S=1 reduces to the standard per-layer loss); Appendix I (balancing table); Appendix A (note on the auxiliary coefficients).