acceptodds
Desk rejected at ICLR 2027

BMoE: By segmenting experts to share and amplify single-layer routing candidates and reducing the total parameters

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Sparse mixture-of-experts (MoE) models activate few experts per token, yet must store the entire expert bank. Cross-depth sharing reduces this storage, but also asks shared experts to serve representations from different layers. We propose BMoE, a block-wise MoE that separates shared expert capacity from layer-specific transformations. Contiguous layers reuse a pool of latent expert cores, while learned input/output interfaces, routers, and attention remain layer-specific. Each depth can therefore access the same nonlinear cores through different learned maps. We evaluate eight 36-effective-layer configurations trained for approximately 208B scheduled token positions with matched selected-expert computation. At nearly equal total parameters (7.063B versus 7.056B), two-layer BMoE reduces WikiText-103 perplexity from 17.867 to 17.031 and raises the nine-task mean from 50.74 to 52.10 relative to a per-layer latent MoE. Four-layer BMoE improves all four aggregate metrics with 43.2% fewer parameters. It also outperforms whole-layer looping at matched expert storage. Tying the latent interfaces saves only 1.6% of parameters but degrades every aggregate metric; forward diagnostics show differentiated routing within shared pools. Together, these results show that the boundary of depth sharing matters: expert cores can be efficiently reused across depth while layer-specific transformations preserve flexible access to the shared capacity.

Primary area:
foundation or frontier models, including LLMs
Keywords:
pretraining, mixture of expert, parameter sharing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qcrEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes BMoE that shares pools of latent expert cores across contiguous blocks of layers while retaining layer-specific input/output projections and routers. Eight configurations with 36 effective layers, ranging from 2.5B-A1.3B to 7.1B-A1.3B, are trained for approximately 208B tokens and compared with per-layer latent MoE (7.1B-A1.3B) and a whole-layer Loop baseline (3.8B-A1.3B). The reported four-layer-sharing configuration reduces the model from 7.1B-A1.3B to 4.0B-A1.3B (43.2% fewer total parameters) while improving the nine-task mean by 0.83 percentage points over the per-layer baseline.

Critical strengths
  1. Fewer parameters with competitive model quality. Sharing experts across layers reduces the number of weights that need to be stored or moved in GPU. The reported experiments show that this can preserve or improve model quality, making it a promising way to reduce the training & inference requirements of MoE models.

  2. Useful evidence about which components to share. The ablation shows that keeping each layer's input/output projections and normalization separate works better than sharing these components along with the experts. This provides a useful design insight for cross-layer expert sharing.

Critical weaknesses
  1. My major concern is the constant learning-rate training setup. All models use a learning rate of 3.36e-4 after a 500-step warmup, with results reported at step 11,000 and no learning-rate decay. Recent MoE studies such as LatentMoE and Qwen3 include learning-rate decay in their pretraining schedules. The models may still be insufficiently converged at the chosen lr, and the observed gains could disappear after decay. It is thus important to check whether the quality ranking persists after a common learning-rate decay phase.

  2. Missing balancing ablations and expert-load statistics. Block-wise balancing relaxes the balancing constraint on each individual layer, which could itself improve downstream performance. Please compare layer-wise and block-wise balancing with the BMoE architecture, expert pool size, and training setup fixed. The paper reports routing overlap and JS divergence, but not how evenly tokens are distributed across experts. Per-layer and block-level expert token counts would help verify the claimed balancing behavior and clarify its contribution to the gains.

Questions for authors
  1. Please address the training and balancing concerns detailed in Weaknesses 1 and 2.
  2. Table 1 matches the stored expert count between S4 and Loop, but S4 uses 256 candidate experts per layer whereas Loop uses 128. Could you add a Loop baseline with 256 candidates to assess the benefit of the larger pool, or more fine-grained expert architecture?
  3. What practical memory and speed benefits does BMoE provide over Per-Layer MoE and Loop? Please report peak training memory, training throughput, and inference latency/throughput under matched conditions, preferably separating prefill and decoding.
4 comments

Thank you for four testable questions, and for restating the main result accurately (43.2% fewer parameters, +0.83pp nine-task mean). Below we add the evidence you identified as missing. This comment answers W1 (constant learning rate) and the first half of W2 (balancing ablation). The second half of W2 (load statistics) and Q1 (Loop-256) are in the next comment, and Q2 (memory and speed) in the one after.

Notation. Sk = BMoE with k adjacent layers sharing one pool of latent experts; M = the main 36-layer model; XS/S = the two smaller backbones.

W1: Constant learning rate

(1) At small scale, the ranking is unchanged after decay. XS and S are trained from scratch with a cosine decay over the last 10% of steps and identical settings otherwise:

ScaleModelBefore decayAfter decayGap to Per-Layer (before → after)
XSPer-Layer2.68622.6098—
XSBMoE-S22.68162.6053−0.0046 → −0.0045
XSBMoE-S42.73522.6589+0.0490 → +0.0491
SPer-Layer2.55042.4548—
SBMoE-S22.54492.4505−0.0055 → −0.0043
SBMoE-S42.57412.4903+0.0237 → +0.0355
  • At XS the three structures gain almost the same loss from decay (0.0764 / 0.0763 / 0.0763). Decay acts as an additive shift and does not reorder the structures.
  • At S the gains are 0.0956 / 0.0944 / 0.0838. S4 gains 0.012 less, so its gap to Per-Layer grows from 0.024 to 0.036. The relative positions of the three structures do not change.
  • Decay works normally for BMoE: no instability and no vanishing gain.

(2) At the main scale, the decay phase does not expose under-convergence. We extended M-S4 beyond step 11,000 under the same constant learning rate, then decayed it to zero with a 2,000-step cosine schedule starting at step 14,000 (264B to 302B tokens):

StepLRLossWiki PPLPTB PPL9-task mean
14,0003.36e-42.054516.58935.70952.17
15,6003.35e-52.000515.54933.19252.93
16,00001.998115.51233.02952.91

Loss, Wiki PPL, and PTB PPL improve monotonically as the learning rate falls to 0. The nine-task mean rises from 52.17 to 52.93 and then stays flat (52.93 to 52.91, a 0.02pp difference within few-shot seed variation). There are no spikes and no degradation. This run only checks whether training was so far from convergence that decay could reverse the ranking. It is not a cross-model comparison: the other three main models were not continued, and the token budget (302B vs 208B) makes these rows incomparable with Table 2.

(3) What we did not do, and the claims that follow. Same-protocol continuation of the other three main models was not possible within the discussion period. At the main scale we therefore claim only that the decay phase behaves normally, not that the ranking is unchanged after decay. Decay lowers the M-S4 loss by about 0.056, so we state the main-scale claims at the strength the evidence supports. (i) The S2 advantage over Per-Layer is the main-scale claim: at both small scales its gap is preserved through decay (−0.0046 → −0.0045 at XS, −0.0055 → −0.0043 at S), and at M it is 14 times the XS seed noise (seed noise measured at XS only). (ii) For S4, the claim is parity with Per-Layer at 43.2% fewer parameters: loss and both perplexities are lower, and the nine-task mean is +0.83pp (95% interval lower end −0.25pp under a conservative evaluation standard error, App. D).

Manuscript changes: Appendix G (Tables on small-scale decay and the M-S4 continuation); Appendix H (backbone family).

W2, first half: Block-level vs. layer-wise balancing

On XS we fix the BMoE-S4 structure, pool size, N, data, and training settings, and change only block-level balancing to standard layer-wise balancing (same coefficient, 0.001):

BalancingLossWiki PPL9-task meanEffective experts per layer / 256Max dead experts per layer
Block-level (default)2.658936.03040.81161.734
Layer-wise2.655135.64840.54242.10
  • The loss differs by 0.004, within the XS seed standard deviation (0.0039–0.0046). The downstream gap of 0.27pp is 1.5–1.9 XS training-seed standard deviations (0.14–0.18pp). The two have comparable quality.
  • Layer-wise balancing does make every layer use the whole pool (dead experts 34 to 0), but this does not buy quality.
  • The choice of balancing therefore does not change the quality of the shared structure at this scale. We keep block-level balancing because it matches the unit of ownership (the pool) and does not constrain the same pool repeatedly.

Manuscript changes: Section 3.3 rewritten (S=1 reduces to the standard per-layer loss); Appendix I (balancing table); Appendix A (note on the auxiliary coefficients).

This comment continues the previous one; Q2 is in the next comment. Results are in Appendices E and I of the revised PDF.

Notation. T = stored expert instances; N = candidate experts per layer or pool; Loop = the whole-layer repeated-call baseline; Sk as in Part 1.

W2, second half: Expert-load statistics

Main models, about 1M held-out tokens from five domains (per-layer counts sum to 8 × the number of tokens):

ModelUnitUnitsCV mean [range]Effective experts / NDead (max over units)
Per-Layerlayer360.105 [0.061, 0.233]127.2 / 1280
BMoE-S2pool180.092 [0.069, 0.193]254.8 / 2560
BMoE-S2layer360.337 [0.248, 0.506]241.0 / 2563
BMoE-S4pool90.080 [0.058, 0.124]255.2 / 2560
BMoE-S4layer360.587 [0.310, 0.909]214.5 / 25643
BMoE-S6pool60.077 [0.054, 0.114]255.2 / 2560
BMoE-S6layer360.784 [0.334, 1.336]190.8 / 25666
  • Pools are even. Mean pool CV is 0.077–0.092 with no dead experts and 254.8–255.2 effective experts out of 256, below the layer-level CV of Per-Layer (0.105).
  • Within a pool, layers concentrate, more so as more layers share it (mean layer CV 0.34, 0.59, 0.78 for S2, S4, S6). A constraint applied to the pool allows this: in M-S4 a single layer leaves up to 43 experts unselected, yet no expert of the pool is dead.
  • This does not cost quality. In the XS ablation (Part 1), layer-wise balancing removes the per-layer dead experts (34 to 0) and raises effective experts per layer from 162 to 242 with equal quality, so we read within-layer concentration as division of labor, not collapse.

Manuscript changes: Appendix E (load statistics table and discussion).

Q1: Loop-256 and finer-grained experts

Table 1 pairs models by stored expert instances T, while Loop has 128 candidates per layer and BMoE has 256. We added Loop-256 on XS (256 candidates per layer, the same T as S2):

ModelTTotal paramsLossWiki PPL9-task mean
Loop-1287680.509B2.671736.52639.99
Loop-2561,5360.764B2.614334.89540.86
BMoE-S21,5360.785B2.605333.23140.81

(1) A wider candidate pool accounts for most of the gain, as Contribution 1 (duplicated storage turned into a wider per-layer pool) states. Loop-128 to Loop-256 doubles T (768 to 1,536) and candidates per layer (128 to 256) and lowers loss by 0.057, most of the 0.066 advantage of S2 over Loop-128. This supports the mechanism: the same storage buys more quality as a wider pool. The step doubles T and candidates together, so it does not separate the two. Table 1 pairs models by T because T sets the storage budget compared.

(2) With T and candidates both matched, the sharing boundary still gives a measurable gain. BMoE-S2 has 0.009 lower loss than Loop-256 (about 2 XS seed standard deviations) and 1.66 lower Wiki PPL (4.8% relative); the nine-task means are equal (40.81 vs 40.86; the XS seed standard deviation of this mean is 0.14–0.18pp). T, N, and activated computation match. Only the sharing boundary differs: the loop reuses the whole layer (attention, router, interfaces), whereas BMoE shares only the latent expert cores.

(3) Cost. S2 keeps six more sets of attention, router, and interfaces: 0.021B parameters (+2.7%).

Finer-grained experts (256 half-width experts, top-16, matching Loop-128 in storage and computation) were not completed in the discussion period; they need a new run with re-tuned top-K and balancing coefficients, and we leave them to future work.

Manuscript changes: Appendix I (XS variants table and discussion).

This comment completes our answer. Results are in Appendix F of the revised PDF.

Notation. Sk = BMoE with k adjacent layers sharing one pool; S4+SP = S4 with interfaces also shared; Loop = the whole-layer repeated-call baseline; N = candidate experts per layer; K = experts selected per token.

Q2: Memory, throughput, and inference

ModelTotal paramsbf16 weightsTrain step time (24 GPUs, same configuration)Peak train memory / GPU
Per-Layer7.056B14.16 GB38.6 s58.6 GB
BMoE-S27.063B14.17 GB55.9 s60.5 GB
BMoE-S44.006B8.06 GB57.3 s57.7 GB
Loop3.762B7.55 GB38.7 s54.7 GB

(1) Weights are the reliable gain. S4 stores 57% of the Per-Layer weights, which lowers deployment size, checkpoint transfer, and the number of shards.

(2) Peak training memory drops by only 0.9 GB per GPU. With EP=8 the expert weights are already partitioned, so the 6.10 GB weight difference becomes 6.10/8 ≈ 0.76 GB per GPU, consistent with the measured 0.9 GB. The other ~57 GB (activations, optimizer states, non-expert parameters) does not depend on expert sharing. Loop's peak is 3.9 GB lower than Per-Layer, more than this argument explains; we did not decompose it. The storage benefit therefore appears in global weights, not in per-GPU training memory.

(3) Step time is 45–48% higher, and the overhead follows candidates per layer, not sharing. All 256-candidate models (S2, S4, S4+SP) take 55.9–57.3 s and all 128-candidate models 38.6–38.7 s (Per-Layer, Loop). S4 stores 43% fewer experts than S2 at the same step time, so the cost is in routing and dispatch width. We did not write a fused dispatch kernel for 256 candidates.

(4) Prefill differs by about 1.5% of MoE-branch FLOPs. Prefill is dominated by the selected experts, whose computation is identical (same K, width, activated computation). Only the router term 2dN in Eq. 9 varies: 0.79 MFLOP for N=256 and 0.39 MFLOP for N=128, i.e. 2.9% and 1.5% of the MoE branch (4dℓ + 6Kℓh + 2dN). This is an analytic estimate from the paper's FLOP formula, not a measured prefill time.

(5) Decoding: no difference at bs=1, 1.17x expert reads at bs=64. Decoding is weight-bandwidth bound and reads only the K selected experts per token, so at bs=1 the bytes read do not depend on pool size. At bs=64, Per-Layer touches 110.7 of 128 experts (86%) and S4 touches 129.8 of 256 (51%): a ratio of 1.17x, below the 1.77x expected under uniform routing over a doubled pool. Only the expert-weight term grows, so the end-to-end read increase is below 17%.

(6) Why no wall-clock inference time. The only implementation is the HF reference, which loops over experts one by one. At bs=1 all four models take about 82 ms/token, so the measurement is dominated by interpreter overhead and the differences between models are below implementation noise. Items (4) and (5) are an analytic estimate and a measured count.

Manuscript changes: Appendix F (weights, step time, peak memory, prefill FLOP breakdown, decode expert reads).

Thank you for the additional experiments. The rebuttal response strengthen the submission. Two concerns still remain.

  1. The training-loss ranking is unchanged after cosine decay on both smaller backbones, XS (0.31B active parameters, 18B training tokens) and S (0.48B active parameters, 20B training tokens). But at the main scale, only S4 was continued through decay, so the relative ranking after a common cooldown remains uncertain.
  2. At XS, Loop-256 and BMoE-S2 match the number of candidate experts per layer (256), the number of stored expert instances (1,536), and the activated computation. BMoE-S2 has lower training loss (2.6053 vs. 2.6143) and WikiText perplexity (33.23 vs. 34.90), while the nine-task mean is nearly identical (40.81 vs. 40.86) and S2 uses 2.7% more total parameters. At the main scale, the Loop baseline uses 128 candidates, so the downstream advantage over a matched Loop baseline remains unclear.

I consider the paper borderline and would like to discuss these points with the Area Chair before finalizing my recommendation.

Reviewer_M2CtOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BMoE, an architecture that shares a pool of latent expert cores across layers within each block while retaining layer-specific interfaces. The experimental results demonstrate the effectiveness of the design.

Critical strengths

The paper cleverly combines latent MoE with block-wise expert sharing, allowing layers to share expert capacity while retaining layer-specific transformations. The experiments support the effectiveness of the overall architecture, and the ablation study further demonstrates the value of the layer-specific interfaces. The paper is clearly written, with an intuitive central idea and an accessible presentation of both the design motivation and the experimental results.

Critical weaknesses

My main concern is that the current experiments do not sufficiently establish whether the benefits of this architecture persist as scale increases. Scalability is an important consideration in large-model architecture design, but the paper primarily compares configurations under a fixed backbone and training budget. While the paper includes sharing-span and expert-pool-size scans, it does not systematically examine whether its benefits persist across backbone scales or training budgets—for example, through a scaling-law analysis examining how its advantage over the baseline evolves. It is therefore difficult to assess how well the reported results would generalize to larger-scale training. Beyond direct experiments across scales, another valuable direction would be to examine how internal model metrics evolve as the model scales up. The authors provide some internal diagnostics, but these remain limited to the same backbone configuration and do not establish further evidence from a scalability perspective. Showing how internal behavior changes with scale would help explain why the architecture works and whether its benefits are likely to persist at larger scales.

Questions for authors

Please refer to the weaknesses above.

1 comment

Thank you for the positive assessment of the design and the writing, and for the scalability question. Below: what transfers, then scale, training budget, and internal metrics (App. H of the revised PDF).

Notation. Sk = k adjacent layers sharing one pool; N = candidate experts per pool; Loop = the whole-layer repeated-call baseline; XS/S/M = small, small-medium, and main 36-layer backbones.

1. The transferable conclusion: where the boundary is drawn, not how much is shared

Pool sharing itself is not our conclusion (it matches concurrent work). Our conclusion is that the sharing boundary should fall between capacity and access path:

  • The expert cores are expensive (they set storage); the interfaces (down/up projections plus post-aggregation RMSNorm) are cheap. Tying the interfaces saves only 1.6% of parameters but costs 1.27pp downstream.
  • Sharing does not collapse depths into one function: within-block adjacent-layer top-K overlap is 0.019, below the random level (8/256 = 0.031); whole-layer looping gives 0.816.
  • With layer-wise balancing at XS-S4, quality is unchanged, so balancing is not a quality lever there.
  • The S2/S4/S6 × N=128/192/256 sweep gives a monotonic storage–quality curve with diminishing returns from sharing, so a user can pick a point for a storage budget.

Each is measured at one scale (balancing at XS, the rest at M). The scale dependence we measured follows.

2. Scale: the parameter-matched comparison points the same way at three scales

We trained two smaller backbones at the same design ratios as the main model (latent width d/2, expert width 0.375d, top-8; Per-Layer 128 experts per layer, BMoE 256 experts per pool; same tokenizer, context length, and data order):

ScaleLayers / dNon-emb. activated paramsTokensPer-Layer lossS2 lossΔloss9-task mean (Per-Layer / S2)
XS12 / 7680.07B18B (≈240 per param)2.60982.6053−0.004541.22 / 40.81
S16 / 10240.17B20B (≈120 per param)2.45482.4505−0.004343.33 / 43.74
M36 / 15360.86B208B (≈240 per param)2.10652.0499−0.056650.74 / 52.10

(XS seed standard deviation: loss ≈0.004, nine-task mean 0.14–0.18pp. XS and S end with a 10% decay; M uses a constant learning rate.)

  • Same direction. At all three scales the S2 loss is not higher than Per-Layer. The gap is largest at M (0.057), with lower perplexity on both sets (−0.836 / −1.276).
  • The gap grows with scale: 0.004 at XS and S against 0.057 at M. At small scale the output head is a large share of the activated parameters (the 151,936-token vocabulary is kept), so the gain from a larger candidate pool is harder to see. At XS and S the gap equals the seed standard deviation, so we read them as "not worse" and put the conclusion at M.
  • Downstream results differ: S2 is 0.41pp lower at XS, 0.41pp higher at S, 1.36pp higher at M. Small-scale downstream evaluation has limited resolution (neighboring main-model checkpoints differ by about ±0.5pp), so cross-scale evidence is loss and perplexity, and downstream is judged at M (+1.36pp, 2.5–3.5x the standard error).
  • Three scales do not suffice to fit a scaling law; we do not extrapolate. The revised conclusion states the tested range: 0.07B–0.86B non-embedding activated parameters, up to 208B tokens.

For the storage-reducing S4, the loss relative to Per-Layer is +0.049 at XS, +0.035 at S, and −0.009 at M: the gap narrows with scale. This is a descriptive result of three points, and we read S4 at M as parity (App. D).

3. Training budget: no late-stage reversal within 208B tokens

On the main-model training curves (same data order, constant learning rate, loss smoothed over ±100 steps), S2 is not above Per-Layer at any point from 2k to 11k steps. The two small scales use different budgets (240 vs 120 tokens per parameter), and the sign of the S2 difference is the same under both.

4. Internal metrics across scales

Metric (BMoE-S4)XSSM
Within-block adjacent-layer top-K overlap (random level 8/256 = 0.031)0.0190.0200.019
Cross-block top-K overlap0.0270.0380.031
Effective experts per layer / 256162177214
Pool-level load CV0.300.220.08
Within-block routing JS divergence0.180.190.28
  • Differentiated routing does not degrade with scale. Within-block overlap is below the random level at all three scales; Loop's two calls of one layer overlap 0.68–0.82.
  • The pool is used more fully at larger scale. Effective experts per layer rise 162 → 177 → 214 and pool-level CV falls 0.30 → 0.22 → 0.08.
  • These are descriptive measurements; we do not claim to have isolated the mechanism.

Manuscript changes: Sections 1, 2, 3.1, and 7 (tested range); Appendix H.

Reviewer_czAfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a variant of the standard MoE architecture, where blocks are grouped and blocks within the same group share a pool of experts. This cross-depth sharing enables experts to be reused at each block within the group. The authors find that this enables them to reduce the total size of there MoE model without losing performance, and improve upon the performance when total parameters are matched.

P.S. There are broken references on pages 4 and 8, putting the note here since I didn't think it should fall in the category of critical weaknesses.

Critical strengths

[S1] Relatively large sweep over architecture configs ablating the method.

[S2] Ablation in table 3 shows how performance improvement from sharing has diminishing returns as sharing increases.

Critical weaknesses

[W1] This method is very similar to prior work from Qiu et al., which is not cited in the related work section or benchmarked against. At the very least some discussion of MoRE is necessary as its nearly the same method. It would also be helpful to have a comparison to MoRE.

Eric S. Qiu, Utku Umur Acikalin, Justin Lovelace, Christian Belardi, Arjun B. Mulchandani, Carla P Gomes, & Kilian Q Weinberger (2026). MoRE: Mixture of Reused Experts. In Third Conference on Language Modeling.

[W2] Subsection 3.3 block level balancing is very difficult to parse. I would argue the notation is hindering the explanation more than helping it.

[W3] A number of results are very small improvements in Loss, PPL, and Accuracy, it would strengthen to result to have error bars or some test for significance to show that the small margin is meaningful.

Questions for authors

[Q1] Have you ablated including the Prelude & Coda in the architecture design? That is, what happens if the sharing starts from the first block the same as MoRE?

3 comments

Thank you for the careful review, and for noting the configuration scan and the diminishing returns from sharing. Results are in Appendices D, H, and I of the revised PDF. This comment answers the P.S. and W1; W2, W3, Q1, and Presentation are in the next comment.

Notation. Sk = BMoE with k adjacent layers sharing one pool; S4+SP = S4 with interfaces also shared; T = stored expert instances; N = candidate experts per pool; K = experts selected per token; Loop = the whole-layer repeated-call baseline; XS = the smallest of our three backbones; Contribution 1 = turning duplicated cross-depth storage into a wider per-layer candidate pool.

P.S.: Broken references

Fixed. The diagnostics appendix had been excluded at build time; it is restored as Appendix C, all references resolve, and the affected §6.3 sentences are rewritten.

W1: Relation to MoRE

Timeline. MoRE (Qiu et al., 2026) appeared as an ICML 2026 workshop poster in July 2026 and on arXiv on 16 September 2026. We were not aware of it at submission. The revised Related Work cites MoRE, ReXMoE (Tan et al., 2025), and MoUE (Chen et al., 2026), and states where we differ.

What is the same. Both designs share one routed expert pool across adjacent layers and keep attention and routers private. Our S2 (candidates per layer doubled, total parameters unchanged) corresponds to MoRE's main setting R=2. We therefore do not claim "shared expert pool plus per-layer routers" as a contribution (Contribution 1 is rewritten accordingly).

What differs: where depth information enters the shared computation.

MoREBMoE
How shared experts know which layer calls themA learnable vector added to the layer input (additive depth embedding)Each layer wraps the shared latent expert cores with its own down-projection, up-projection, and post-aggregation RMSNorm (linear maps plus scale)
Settings studiedParameters unchanged, pool grows with sharing depthParameters unchanged (S2), and fixed pool size with fewer parameters (S4: 43% fewer, S6: 58% fewer)
Scale114M–1.15B total parameters, 6.55B tokens, context 1,0240.5B–7.1B total parameters, up to about 208B tokens, context 4,096

Direct comparison (XS, from scratch, matched computation). We replaced the layer-specific interfaces of BMoE-S4 with MoRE-style conditioning: interfaces shared within a block, plus a zero-initialized depth embedding per layer that feeds both the router and the down-projection.

ModelLossWiki PPL9-task mean
BMoE-S4 (layer-specific interfaces)2.658936.03040.81
MoRE-style (shared interfaces + depth embedding)2.672236.85340.53
Shared interfaces, no depth embedding (S4+SP)2.676737.13940.01

(XS training-seed standard deviation: loss about 0.0046, nine-task mean about 0.14–0.18pp.)

  • The depth embedding improves on shared interfaces alone by 0.0045 in loss and 0.286 in Wiki PPL, about 1x the seed standard deviation. This is the same size as the depth-embedding ablation reported by Qiu et al. (2026) (PPL 18.76 to 18.71), and is consistent with an additive depth vector providing limited layer conditioning.
  • Layer-specific interfaces beat the MoRE-style variant by 0.013 in loss (about 3x the standard deviation) and 0.823 in Wiki PPL. Small-scale downstream evaluation cannot resolve a difference of this size (0.28pp), so this conclusion rests on loss and perplexity.
  • At the main scale we have no MoRE-style variant, but the S4 vs. S4+SP interface ablation (same depth, block size, pool size, stored experts) is the closest pair: layer-specific interfaces improve loss by 0.0186, Wiki PPL by 0.720, PTB PPL by 2.482, and the nine-task mean by 1.27pp, for 0.064B parameters (1.6%).
  • Scope: this keeps our latent-expert backbone and replaces only the layer-conditioning mechanism. It is not a reproduction of MoRE, which uses full-width fine-grained experts, so absolute numbers should not share a table.

Manuscript changes: Section 2 (MoRE, ReXMoE, MoUE); Section 1, Contribution 1; Appendix I (XS variants table).

This comment continues the previous one. Notation: Sk = k adjacent layers sharing one pool; S4+SP = S4 with interfaces also shared; Loop = the whole-layer repeated-call baseline; M = the main 36-layer model; XS = the smallest backbone.

W2: Section 3.3 is hard to read

Rewritten in full. A pool is owned by a block of S layers, so their routing statistics are merged and the standard Switch/GShard loss is applied to the merged statistics; S=1 is exactly the standard per-layer loss. Two numbered equations follow (Eq. 6, statistics; Eq. 7, loss), with N/K and 1/G each explained in its own sentence.

Replacing block-level with layer-wise balancing leaves quality unchanged at XS-S4 (loss 2.6589 vs 2.6551, within seed noise).

Manuscript changes: Section 3.3; Appendix I.

W3: Error bars and significance

We agree this was a gap.

(1) Noise scale (XS, seeds 42/43/44; only the initialization changes, data order is fixed):

ModelLossWiki PPL9-task mean
Per-Layer2.6061 ± 0.003933.54 ± 0.3841.05 ± 0.18
BMoE-S42.6571 ± 0.004635.69 ± 0.5840.65 ± 0.14

S2 was not run with multiple seeds; the dispersion of these two models is the closest estimate we have.

(2) Main result: S2 vs. Per-Layer (M scale, equal parameters). Total parameters are 7.063B vs. 7.056B (+0.1%) with equal activated computation, so the quality gap is not bought with capacity:

MetricPer-LayerBMoE-S2ΔRelative
Training loss2.10652.0499−0.0566−2.7%
WikiText-103 PPL17.86717.031−0.836−4.7%
PTB PPL38.39537.119−1.276−3.3%
9-task mean50.7452.10+1.36pp—
  • All four metrics agree in sign. The loss gap is about 14 times the XS seed noise (an approximation, since seed noise is not measured at M), and +1.36pp is 3.5 times the standard error of one model's nine-task mean (0.39pp) or 2.5 times the unpaired upper bound for the SE of the difference (0.55pp).
  • This comparison carries the main-scale claim: parameters almost identical, all four metrics improve, both perplexities fall by 3%–5%.

(3) All four comparisons on the same standard (M). Combining per-task binomial standard errors over nine independent tasks gives SE ≈ 0.39pp for one model's nine-task mean. The two models are evaluated on the same items, so the SE of their difference is at most √2 × 0.39 ≈ 0.55pp.

ComparisonΔlossΔWiki PPLΔmean95% CI of Δmean (SE 0.55)Δmean / 0.39Δmean / 0.55
S2 vs. Per-Layer−0.0566−0.836+1.36pp[+0.28, +2.44]3.5x2.5x
S4 vs. Per-Layer (43.2% fewer parameters)−0.0093−0.554+0.83pp[−0.25, +1.91]2.1x1.5x
S4 vs. Loop (same T)−0.0094−0.357+0.94pp[−0.14, +2.02]2.4x1.7x
S4 vs. S4+SP (interface ablation)−0.0186−0.720+1.27pp[+0.19, +2.35]3.3x2.3x
  • Few-shot example seeds (M-S4, three seeds) vary the mean by 0.06–0.13pp, below the evaluation noise.
  • What each row supports. S2 vs. Per-Layer and S4 vs. S4+SP are differences: their intervals exclude zero even under the unpaired bound. S4 vs. Per-Layer and S4 vs. Loop are parity claims: the interval lower ends are −0.25pp and −0.14pp, while loss and both perplexities favor S4 (with 43.2% fewer parameters than Per-Layer). We state the S4 claim as parity at lower storage, not as superiority.

(4) Small scale. At XS the S2 gap (0.0045) equals the seed noise in (1), so this scale cannot resolve it. We place the main conclusion at M and read the small scales as "not worse".

Manuscript changes: Appendix D (noise scale and the four-comparison table); Appendix H (backbone family).

Q1: Prelude / coda

BMoE has no prelude or coda. The 36 layers are partitioned contiguously from the first layer with G = L/S; with S=4, nine blocks cover all 36 layers, so sharing starts at the first block, as in MoRE. The revised §3.1 states this in one sentence.

A variant with private first and last layers was not trained in the discussion period. At 36 layers and S=4, keeping the first two and last two layers private leaves 32 shared layers (G = 8) plus four layers with their own pools, so total parameters and the number of pools change together and the result cannot be read against the existing configurations. A meaningful ablation needs a separate sweep over S and N, which we leave to future work.

Manuscript changes: Section 3.1.

On Presentation

You rated Presentation 2. We read the cause as the broken references on pages 4 and 8 (a build error, now fixed as Appendix C) and the notation in §3.3 (rewritten, see W2). If other passages need expansion, please point them out and we will revise before the final version.

[W1] A note on the timeline, MoRE was published at COLM 2026 which released decisions on July 8th. Therefore it does not meet the criteria for contemporaneous work according to the ICLR 2027 reviewer guidelines which states anything published on or after July 17th is contemporaneous.

Thank you for clarifying the novelty. I agree that your alternative approach to capturing depth information, and the scale of the study are contributions. I also appreciate the direct comparison to a MoRE-style model at the XS scale. However its a single comparison using your backbone and not a direct comparison to MoRE, which makes it less than conclusive.

[W2] Thank you for addressing my presentation concerns.

[W3] I appreciate the effort put into providing error bars and significances. However the most significant result (S2 vs Per-Layer), is the configuration most similar to MoRE. Further the deviations from MoRE have limited support.

[Q1] Thank you for the clarification.

I feel that my original score of weak reject is appropriate, given the limited novelty and evidence supporting it. I see the primary contribution as the scaling of the approach which I don’t believe warrants a higher score without more interesting/surprising results/analysis.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.