acceptodds
Under review as a conference paper at ICLR 2027

Shaping Latent Mixture-of-Experts: From Expert Geometry to Layerwise Capacity Allocation

Abstract

Fine-grained mixture-of-experts (MoE) models commonly increase expert count by narrowing each expert, treating expert shape as a by-product of routing granularity. LatentMoE moves expert computation into a lower-dimensional space, exposing the input dimension and intermediate width of each expert as explicit design choices. We show that an expert's local Jacobian rank is jointly bounded by these two dimensions, and that balancing them maximizes the rank ceiling at fixed per-expert parameters. Controlled shape sweeps across model scales and routing granularities consistently favor near-square experts. We further use layerwise intrinsic dimension (ID) to guide capacity allocation across depth, co-varying the number and shape of experts while keeping per-layer parameter count, activated FLOPs, and communication volume constant. Compared with layer-homogeneous baselines, our ID-guided allocation achieves lower validation loss and stronger downstream performance. These results establish expert shape and layerwise allocation as practical design axes for MoEs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Pretraining, Mixture of Experts, Heterogeneous Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zWRqOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies expert shape in LatentMoE. A rank argument and cost-controlled sweeps show that near-square experts work best. It then uses layerwise intrinsic dimension to give shallow layers more experts with a smaller latent width and deep layers fewer, wider ones, keeping per-layer cost nearly constant. At two scales this lowers loss and improves downstream accuracy over uniform LatentMoE.

Critical strengths
  1. A sound theory guides the shape design of LatentMoE: a rank bound on each expert predicts that near-square experts work best, and cost-controlled sweeps at two scales confirm it.
  2. A good analysis of the role of MoE layers at different depths: intrinsic dimension rises through the shallow layers and peaks in the middle-to-deep layers, consistently across scales and training, which motivates giving different layers different expert shapes.
Critical weaknesses
  1. I am not fully convinced that the model is "nearly homogeneous from the system's perspective". Dispatch volume is matched, but message structure and expert GEMM sizes change with depth, which can unbalance pipeline stages. There are no pipeline-parallel results, and the throughput numbers come without hardware, parallel setup or variance.
  2. The evidence for the ID-guided allocation is not yet fully sufficient. The layerwise controls seem to be single runs; there are reversed-order controls but no random per-layer shuffle, and moving the boundary barely matters, so it is unclear what ID adds beyond "narrow early, wide late".
Questions for authors
  1. Under pipeline parallelism, how do per-stage times and bubbles compare with uniform LatentMoE, on what hardware and setup?
  2. How does the allocation compare with a random per-layer shuffle, with several seeds?
4 comments

Thank you for your careful reading and for recognizing the rank-based shape rationale and the depth-wise ID analysis. Below we address your concerns.

W1 / Q1. System homogeneity and pipeline parallelism.

We agree that the homogeneity holds for theoretical computation and communication, and cannot be perfectly realized because of implementation-level inefficiencies such as smaller grouped GEMMs. What the construction matches is per-layer parameter count, activated FLOPs and dispatch volume. Below we provide concrete pipeline-parallel measurements.

Setup: TP=1, PP=2, EP=8, DP=8, 16 micro-batches per iteration (micro-batch size 4, sequence length 4096), 16 NVIDIA H20 GPUs. The throughput in Appendix B.3 (Table 5) was measured on 64 GPUs; for stable and accurate per-stage timing, here we use two fixed 8-GPU nodes. For each scale we report two layer partitions, one where the last stage (holding the LM head) is the critical path and one where the first stage is the critical path. The uniform baseline is LatentMoE with alpha=2.

ScalePartition (layers S0+S1)ModelS0 computeS1 computeBubble S0 / S1Pipeline time
8B9+7 (S1 critical)Uniform LatentMoE8.66 s9.38 s17.4% / 10.5%10.48 ± 0.05 s
ID-guided8.84 s9.34 s15.5% / 10.6%10.45 ± 0.05 s
10+6 (S0 critical)Uniform LatentMoE9.62 s8.42 s7.8% / 19.3%10.43 ± 0.06 s
ID-guided9.79 s8.36 s7.6% / 21.0%10.59 ± 0.06 s (+1.5% vs. uniform)
3.4B7+5 (S1 critical)Uniform LatentMoE4.91 s5.77 s25.7% / 12.6%6.60 ± 0.06 s
ID-guided5.01 s5.70 s23.4% / 13.0%6.55 ± 0.06 s
8+4 (S0 critical)Uniform LatentMoE5.62 s5.03 s10.2% / 19.7%6.26 ± 0.04 s
ID-guided5.70 s4.97 s10.0% / 21.5%6.33 ± 0.04 s (+1.1% vs. uniform)

With our generic kernels, the narrower shallow configuration shows up as +1.5% (8B) and +1.1% (3.4B) when the first stage is critical, and is absorbed when the last stage is critical. We did not tune kernels for these shapes and will acknowledge this as a limitation.


W2 (a). Are the layerwise controls in Table 3 single runs? Because of compute limits, only the main results in Table 2 use three runs. As reported in Appendix B.1, the standard deviation of validation loss is 1.2e-3 to 1.5e-3, so for the variants in Table 3 we report single-run validation loss.


W2 (b) / Q2. Random per-layer shuffle.

We added a random-shuffle control at 3.4B, with two random-shuffle placements and two interleaved placements. ID-guided performs best.

PlacementNarrow MoE blocksVal. loss
ID-guided1, 2, 3, 42.2234
Shuffle A1, 2, 4, 92.2260
Interleave (every third block)1, 4, 7, 102.2331
Shuffle B2, 6, 9, 112.2335
Matched-count exchange8, 9, 10, 112.2433
Interleave (odd blocks, 6 narrow + 5 wide)1, 3, 5, 7, 9, 112.2352
Uniform narrow-2.2391
Uniform wide-2.2305

W2 (c). What does ID add beyond "narrow early, wide late"?

We agree that in the standard decoders studied here, the ID ordering and relative depth give the same split: about the first third of MoE blocks at both scales (4/11 and 5/15). ID at least indicates the direction and the approximate split point. First, the direction is not obvious. The reversed allocations are worse than both uniform endpoints at both scales (Table 3), so guessing the direction wrong costs more than not allocating at all. The ID profile supplies this direction from a measurable property of the representations. Second, the small effect of moving the boundary by one block only shows that the ID-based split tolerates small deviations. A larger shift does hurt: at 3.4B, moving the boundary from 4/11 to 8/11 raises validation loss from 2.2234 to 2.2367.

Thank you again for your careful review and constructive suggestions.

Thank you for the response.

  1. System homogeneity. Thank you for the pipeline-parallel measurements. They make the overall results more reliable. Could you also report the time of each layer? With per-layer times, one could estimate the stage balance for larger models or more pipeline stages without new runs.

  2. Uniform wide vs. uniform narrow. The two use the same parameters and compute, yet uniform wide is better at 3.4B. This seems different from mainstream designs such as DeepSeekMoE, which favor more and smaller experts. Why?

Thank you for the follow-up, and for the suggestion to report per-layer times. We report the per-layer times below and then use them to estimate PP=3 and PP=4.

Q1. Per-layer times

Component (forward + backward, per micro-batch)8B3.4B
Wide MoE layer (8B: 640×512, K/N = 16/512; 3.4B: 512×440, 14/448)62.0 ms44.8 ms
Uniform α = 2 MoE layer (8B: 640×512, 16/512; 3.4B: 512×384, 16/512)62.0 ms45.8 ms
Narrow MoE layer (8B: 464×512, 22/704; 3.4B: 352×448, 20/640)64.4 ms48.0 ms
Block 0 (dense)38.5 ms25.8 ms
LM head110.6 ms89.7 ms
Final norm and loss (last stage only)35.9 ms35.0 ms

Reconstructing a stage also requires two kinds of overhead that do not depend on expert shape. The first is charged to a particular layer according to its position:

Extra time charged to a layer by its position (per micro-batch)8B3.4B
First MoE layer of stage 0, extra forward+5.0 ms+3.9 ms
First layer of every later stage, extra forward+6.2 ms+6.9 ms
Last layer of a non-final stage, extra backward+0.7 ms+1.3 ms

The second is not attributable to any layer:

Schedule gap (per occurrence)8B3.4B
Between forward and backward on the last stage (loss post-processing)15.6 ms15.8 ms
Before each forward and each backward (host)2.4 ms2.1 ms
At each stage boundary (activation transfer and synchronisation)14.8 ms13.7 ms

Three observations may be useful.

First, summing the layer times and the position-dependent overheads according to the layer partition reproduces the per-stage compute times in our previous response to within 0.5%. For the 8B ID-guided model under the 9+7 partition, stage 0 is 38.5 + 5 × 64.4 + 3 × 62.0 + 5.0 + 0.7 = 552.2 ms per micro-batch, or 8.84 s per iteration, against the 8.84 s reported; stage 1 is 7 × 62.0 + 110.6 + 35.9 + 6.2 = 586.7 ms, or 9.39 s, against 9.34 s.

Second, at 8B our deep regime is the same configuration as the α = 2 baseline and measures identically, which is a useful internal check. At 3.4B the deep regime differs from the α = 2 baseline and is slightly faster, consistent with executing fewer and larger grouped GEMMs. The quantity that governs stage balance is the ratio between the two regimes: with our untuned kernels a narrow layer costs 3.9% more than a wide layer at 8B and 7.1% more at 3.4B.

Third, these numbers allow the pipeline time to be estimated for other configurations. For stage s, let c_s be the sum of its layer times and position-dependent overheads, with the LM head and loss added on the last stage, and let t_s be c_s plus two host gaps, plus the loss gap on the last stage. With p stages, m = 16 micro-batches and stage-boundary gap g, a 1F1B schedule gives

T = γ · [(m − 1) · max_s t_s + Σ_s t_s + 2(p − 1) · g].

The idealised formula ignores variation across micro-batches and occasional synchronisation waits, so we calibrate γ as the mean ratio of measured to predicted time for the uniform model at PP=2, giving γ = 1.016 at 8B and 1.031 at 3.4B. With these two constants the formula reproduces all eight measured PP=2 pipeline times, for both models and both partitions, to within 0.6%.

Applying it to PP=3 and PP=4 gives the following. The PP=2 rows are the measurements from our previous response; the PP=3 and PP=4 rows are estimates obtained from the per-layer times above, and every entry can be recomputed from the tables above. At each setting both models use the same layer partition, chosen as the best contiguous partition for the uniform model, so the comparison does not depend on re-tuning the partition for our model. At PP=4 we additionally report the best contiguous partition for the ID-guided model, which is what a deployment would use; at PP=3 the two coincide.

ScalePPModelPartition (layers per stage)Stage compute (s)Bubble per stagePipeline time
8B2uniform9+7 / 10+68.66 / 9.38; 9.62 / 8.4217.4% / 10.5%; 7.8% / 19.3%10.48 s; 10.43 s
ID-guided9+7 / 10+68.84 / 9.34; 9.79 / 8.3615.5% / 10.6%; 7.6% / 21.0%10.45 s; 10.59 s (+1.5%)
3uniform6+6+45.67 / 6.06 / 6.4126.0% / 20.9% / 16.3%7.66 s
ID-guided6+6+45.86 / 6.06 / 6.4123.5% / 20.9% / 16.3%7.66 s (+0.0%)
4uniform4+5+5+23.68 / 5.07 / 5.07 / 4.4340.5% / 18.1% / 18.1% / 28.4%6.19 s
ID-guided, uniform's partition4+5+5+23.80 / 5.15 / 5.07 / 4.4339.5% / 18.0% / 19.2% / 29.4%6.27 s (+1.4%)
ID-guided, own best partition5+4+5+24.83 / 4.12 / 5.07 / 4.4321.9% / 33.5% / 18.1% / 28.4%6.19 s (+0.0%)
3.4B2uniform7+5 / 8+44.91 / 5.77; 5.62 / 5.0325.7% / 12.6%; 10.2% / 19.7%6.60 s; 6.26 s
ID-guided7+5 / 8+45.01 / 5.70; 5.70 / 4.9723.4% / 13.0%; 10.0% / 21.5%6.55 s; 6.33 s (+1.1%)
3uniform5+5+23.43 / 3.80 / 3.5724.5% / 16.4% / 21.4%4.54 s
ID-guided5+5+23.57 / 3.72 / 3.5420.9% / 17.7% / 21.6%4.51 s (−0.6%)
4uniform3+4+4+11.96 / 3.06 / 3.06 / 2.8449.4% / 21.0% / 21.0% / 26.8%3.88 s
ID-guided, uniform's partition3+4+4+12.03 / 3.10 / 3.00 / 2.8247.7% / 20.2% / 22.9% / 27.4%3.89 s (+0.3%)
ID-guided, own best partition4+3+4+12.80 / 2.33 / 3.00 / 2.8227.5% / 39.6% / 22.4% / 26.9%3.86 s (−0.4%)

At PP=3 the best partition is the same for both models and the extra cost of the narrow layers falls on a stage that is not on the critical path, so the estimated pipeline time matches the uniform model at 8B and is slightly lower at 3.4B, where our deep configuration (512×440, K=14) is faster than the α = 2 layer (512×384, K=16). At PP=4 the uniform model's partition places narrow layers on the critical stage; holding that partition fixed, the estimated overhead is 1.4% at 8B and 0.3% at 3.4B, no larger than the 1.5% and 1.1% we measured at PP=2. Moving a single layer to rebalance the stages removes it, giving 6.19 s at 8B and 3.86 s at 3.4B. We report both so that the comparison can be read either way.

Q2. Uniform wide versus uniform narrow

The main difference from earlier MoE work is the latent structure: the d-dimensional hidden state is first compressed to ℓ before being passed to the experts, and this compression is lossy. A standard MoE does not pay this cost, because every expert still receives the full hidden state regardless of how fine the granularity is, and this is the setting in which the DeepSeekMoE result holds. Uniform narrow and uniform wide differ in exactly this respect: uniform narrow compresses the hidden state more (at 3.4B, ℓ = 352 against 512, from d = 1024) and obtains more experts in return (20/640 against 14/448). This creates a trade-off: the additional experts help, while the stronger compression hurts. In our experiments uniform wide is the better of the two, because for uniform narrow the loss from the stronger compression outweighs the gain from the additional experts.

This trade-off is also where our layerwise allocation comes from: we hypothesise that layers with lower ID tolerate a stronger compression ratio, so that spending it on more experts pays off in those layers even though it does not pay off when applied at every layer.

Thank you again for these questions. We believe the measurements and clarifications will help us improve the quality of the paper.

Reviewer_RUynOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies expert shape and layerwise allocation in LatentMoE. The authors use the local Jacobian rank ceiling, bounded by the smaller of the latent and intermediate widths, as a capacity proxy. At a fixed per-expert parameter count, this ceiling is largest for square experts. Shape sweeps at 3.4B and 8B favor a broad region near square shapes.

The authors then use layerwise intrinsic dimension (ID) to guide allocation. Shallow layers use a narrower latent and more experts, while deeper layers use a wider latent and fewer experts. The construction keeps per-layer compute, parameters and communication nearly constant. The resulting models have lower validation loss than the tested uniform and reversed allocations, and better average downstream performance than the uniform LatentMoE baselines. The paper interprets these results as evidence for matching per-expert capacity to layerwise ID.

Critical strengths
  1. The ablations on expert shape and allocation order may be of interest.
  2. The allocation shows some empirical gains with little additional overhead, keeping per-layer compute, parameters and communication nearly unchanged.
Critical weaknesses
  1. The gains over the best uniform shapes are small, about 0.005 and 0.007 in validation loss, and appear to come from single runs. The OLMES comparisons use the original LatentMoE settings rather than shape-tuned baselines. Repeating the key comparisons would make the gains more convincing.
  2. I am not convinced that the rank ceiling explains the gain. At 3.4B, uniform 448x440 already has the same deep-layer ceiling, q=440q = 440. ID-guided lowers shallow-layer qq while adding experts and activated width, so these effects are not separated. It is also unclear what ID adds beyond depth: narrowing roughly the first third of layers gives the same allocation in these experiments.
Questions for authors

My main concern is that the gains are very limited. I am also unsure what practical design guidance the proposed principles offer beyond what could be obtained through a standard scaling ladder.

1 comment

Thank you for your review. We would first like to clarify a possible misunderstanding about the source of the gain, and then answer your questions.

W2 (a). The source of the gain.

We appreciate your insightful analysis, but there may be a misunderstanding here. The gain of layerwise capacity allocation does not come from a larger rank ceiling. As explained at the beginning of Section 4, we consider the following trade-off in LatentMoE: compressing the per-expert parameter budget can hurt performance by constraining the capacity of each individual expert, while reinvesting the savings into more experts (or larger activated width) can improve performance. A uniform compression implicitly assumes that this trade-off is the same at every layer. We use q as a local-capacity proxy for each layer and hypothesize that lower-ID layers may tolerate a smaller q, so that the trade-off in these layers favors more, smaller experts. Our understanding of the source of the gain is therefore the same as yours. We will emphasize this analysis in the introduction for clarity.

W2 (b). What does ID add beyond depth?

In our method, ID and depth are not mutually exclusive: the ID analysis yields a simple depth-based capacity allocation. The role of ID is to indicate at which depths experts should be narrow and at which they should be wide.

Q1. The magnitude of the gains.

We agree that the loss gains look small in absolute terms. In terms of equivalent compute, however, at both 3.4B and 8B our ID-guided allocation saves 4%-6% of the FLOPs needed to reach the same loss as the best uniform shapes found by our sweeps (448x440 and 576x576). At matched parameters, computation, and communication volume, this is a meaningful margin for LLM pretraining. Gains of this size are used to make design decisions for production MoEs. Auxiliary-loss-free balancing was adopted in DeepSeek-V3, whose report compares balancing schemes using 0.005-nat differences (2.258 vs. 2.253) [1].

Q2. Practical design guidance.

A single scaling ladder can cost hundreds of thousands of GPU hours. Our principles can serve as a reference or prior that narrows the search range or reduces the number of ladder runs, saving considerable GPU hours.

W1. Baselines and significance.

According to the controlled shape sweeps in Section 3.2, the alpha=2 variant among the original LatentMoE settings in our main OLMES comparison already lies in the near-square performance plateau. As reported in Appendix B.1, the standard deviation of validation loss is 1.2e-3 to 1.5e-3, so our validation-loss improvements are significant.

We hope these clarifications and results address your concerns. Thank you again for your consideration.

References

[1] DeepSeek-AI. DeepSeek-V3 technical report. arXiv preprint arXiv:2412.19437, 2024.

Reviewer_K3QYOpenReview ↗
Rating
2(−1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1(−1)
Read review
Paper summary

This work studies the trade-off between the input dimension and intermediate width of individual experts in latent MoE architectures, treating expert shape as an architectural design variable. Building on this analysis, the authors investigate whether allocating expert shapes across model depth can improve performance while keeping the total parameter count and activated computation approximately unchanged.

For the first part, the authors find that, under a fixed per-expert parameter budget, experts with comparable input dimensions and intermediate widths—that is, near-square shapes—generally fall within a well-performing region.

The authors then vary the parameter budget of individual experts across layers while maintaining approximately square shapes and adjusting both the total number of experts and the number activated per token to satisfy the budget constraints. In the settings studied, their central observation is that deeper layers generally benefit from greater per-expert capacity than shallower layers.

Critical strengths
  1. The paper investigates an often-overlooked design question: how to choose the shapes of expert weight matrices in MoE models and allocate per-expert parameter budgets more effectively across depth. The experiments provide reasonably convincing evidence for the feasibility of these design choices.
Critical weaknesses
  1. Although the paper evaluates expert aspect ratios l/m at two model scales and two routing granularities, the experimental coverage remains limited in the range of factors considered. The results do not yet establish how broadly the favorable near-square region generalizes. For example, the best aspect ratio may change as models become wider and deeper or are required to handle longer contexts.

  2. The exploration of expert-shape and capacity allocation across depth also remains relatively limited. Layerwise allocation is validated primarily through two-segment configurations, and the current evidence is not yet sufficient to provide systematic guidance for broader architectural design.

Questions for authors
  1. The paper primarily studies expert-shape and capacity allocation under approximately equal parameter budgets across layers, motivated by the desire to maintain balanced computation and communication workloads. Have the authors considered allowing parameter budgets to vary across layers while keeping the overall budget fixed? Could this more flexible allocation yield performance gains large enough to justify dedicated systems optimizations for the resulting inter-layer workload imbalance?

  2. The proposed allocation primarily uses two segments. Have the authors explored other simple allocation schemes, such as interleaving different expert configurations across depth or gradually varying expert size and expert count from layer to layer?

  3. Beyond the fixed-budget comparisons reported in the paper, have the authors considered how architectural shape relates to the best achievable performance, as measured, for example, by validation loss? Could future work develop scaling laws that jointly account for model scale, expert shape, and layerwise capacity allocation, thereby providing more systematic and predictive guidance for architectural design?

3 comments

Thank you for your thoughtful review and for the constructive questions, which point to natural next steps beyond our design. Below we answer them with preliminary experiments and analysis.

Q1. Allowing per-layer budgets to vary under a fixed total budget.

This is an excellent question and a natural next step after our constant-cost design. Within the rebuttal period we could only test a simple variant at 3.4B. Starting from the ID-guided model, we keep both expert shapes and move routed budget from the shallow to the deep segment by changing only the number of experts:

ModelBlocks 1-4Blocks 5-11Per-layer routed budget vs. ID-guidedVal. loss
ID-guided352x448, K/N = 20/640512x440, K/N = 14/448-2.2234
Variable budget352x448, K/N = 15/480512x440, K/N = 16/512shallow -25%, deep +14.3%2.2247

This variant brings no significant gain, but this does not rule out the direction. Because of compute limits, we could not search finer-grained budget schedules or larger scales, where the trade-off may differ, and we will discuss this as future work.

Q2. Allocation schemes beyond two segments: interleaving and gradual variation.

Because of compute limits, we added two simple interleaved allocations at 3.4B. Both use exactly the narrow and wide configurations of the ID-guided model, so per-layer FLOPs, parameters and dispatch volume are unchanged and only the placement differs.

AllocationNarrow MoE blocksVal. loss
ID-guided (two segments)1-42.2234
Interleave, every third block (4 narrow + 7 wide, same as ID-guided)1, 4, 7, 102.2331
Interleave, odd blocks (6 narrow + 5 wide)1, 3, 5, 7, 9, 112.2352

Both interleaved allocations are clearly worse than the ID-guided allocation. The first interleave uses the same number of narrow and wide blocks as ID-guided. Together with the matched-count exchange in Table 3, this indicates that the gain comes from placing the low-capacity configuration in the low-ID prefix, not from mixing two configurations.

Because of compute limits and the large search space, we did not explore gradual variation. We see a calibrated per-layer schedule as promising future work.

Q3. Shape, best achievable performance, and scaling laws.

We agree this is a valuable direction and offer our current view. Existing MoE scaling laws describe the routed configuration either through the number of experts [1] or through granularity G = d_ff/m [2]. Both keep the expert input width tied to the model width d. LatentMoE decouples the latent width l from d, so the expert shape is exactly the dimension these laws do not yet cover. For dense models, shape-optimized scaling laws [3] show that such dimensions can be scaled predictably with compute.

  • Shape as an efficiency factor. At a fixed total and activated parameter budget, our sweeps suggest that expert shape acts as a multiplicative efficiency factor. A broad near-square plateau sits close to the best loss attainable at that budget, extreme shapes lose up to about 0.04 nats, and loss first improves with q and then saturates.
  • A possible functional form. One could add a shape term to an existing MoE scaling law, for example L = L_scale(N, D, G) + kappa(N, G) (log(l/m) - log r*)^2 + Delta_alloc, where G is granularity. Our sweeps are qualitatively consistent with a near-square optimum and with a penalty that steepens at finer granularity, but they are too coarse to pin down either r* or kappa: the plateau is broad enough that the empirically best shape need not be exactly square, and our two granularities were not swept over identical aspect-ratio ranges. Calibrating such a term would require a sweep grid designed for that purpose. Because the ID profile is consistent across scales when plotted against relative depth, the allocation term Delta_alloc could be parameterized by relative depth as a low-dimensional correction.

Thank you again for your constructive comments.

References

[1] Aidan Clark et al. Unified scaling laws for routed language models. In ICML, 2022.

[2] Jakub Krajewski et al. Scaling laws for fine-grained mixture of experts. In ICML, 2024.

[3] Ibrahim Alabdulmohsin et al. Getting ViT in shape: Scaling laws for compute-optimal model design. In NeurIPS, 2023.

Hello authors, thank you for your response to my review comments. Unfortunately, I feel that my concerns regarding the generalizability and reliability of the paper's results have not been addressed, and I also agree with the other reviewer, RUyn. In fact, my initial intended score was 2.5, but there is no such option available; out of my expectation for a positive response from the authors during the rebuttal phase, I gave a score of 3. I will now change it to 2.

Thank you for the follow-up. We are glad to offer the following additional results and discussion.

1. What we have added

In our first response we reported preliminary experiments and analysis for your three questions: a matched-total variant in which the per-layer routed budget varies (Q1), two interleaved allocations at 3.4B (Q2), and how expert shape might enter an MoE scaling law (Q3). The revised PDF carries these with additional experiments and a fuller discussion:

  • The placement experiments are in Appendix D (Table 10 on Page 18), extended with two random per-layer shuffles, which land at 2.2260 and 2.2335, between the ID-guided allocation at 2.2234 and the reversed control at 2.2433. The boundary sweep now includes a larger displacement of the boundary, and the variable-budget variant of Q1 is also in Appendix D (Table 9 and Page 18). Across eleven 3.4B placements under one matched budget, the contiguous low-ID prefix remains the best.
  • Appendix E.3 adds intrinsic-dimension profiles for four released MoE checkpoints (Figure 8 on Page 23 and Table 17 on Page 24). The depth-wise pattern is consistent across all six models, with ID rising through the shallow blocks, peaking in the middle-to-deep region, and then declining slowly and only mildly. The split point varies somewhat, though not greatly, between 0.25 and 0.41 of relative depth against 0.42 and 0.38 in ours, so the form of the allocation carries over while the boundary shifts modestly from model to model.
  • A new Appendix F discusses allocation schedules beyond two segments and how to transfer the rule to a new architecture (F.1), per-layer budget allocation (F.2), and a shape-aware scaling law with a candidate functional form and what calibrating it would require (F.3).

We are sorry that we cannot go further on three fronts, all for the same reason. A new scale would have to be far enough above 8B for extrapolation to be meaningful, and one aspect-ratio sweep there would cost several times the budget of all experiments in this paper. A shape-aware scaling law needs a grid, not a sweep: several scales, a shared set of aspect ratios at each, at least two granularities, several token budgets per point, and repeats to separate the shape term from run noise. Our existing checkpoints were designed to test an ordering, not to fit coefficients, so they cannot be reused for the fit. A gradual per-layer schedule costs one pretraining run per candidate over a large space. We would rather state this plainly than add underpowered evidence.

2. How broadly the near-square region generalizes

We cannot settle this with new runs at frontier scale. We do think, however, that current industry practice offers useful corroboration, and in exactly the regime your weakness names: systems that are much wider and deeper and that handle much longer contexts.

Several frontier systems released this year build on LatentMoE and report their expert dimensions. They are far beyond the range we can train: 0.8T to 2.8T total parameters, 62 to 102 layers, residual widths of 6656 to 8704, context windows up to 1M tokens, and in two cases native multimodality. Writing the expert shape as latent width l by expert intermediate width m, from most to least recent:

SystemDatelml/m
Muse Spark Open (Meta)Oct 2026409640961.00
MAI-Base-2 (Microsoft AI)Oct 2026409643520.94
Kimi K3 (Moonshot)Jul 2026358430721.17
MAI-Base-1 (Microsoft AI)Jun 2026307251200.60
  1. The most informative entry is not any single shape but a revision between two generations of the same model family. Microsoft AI's expert went from 3072 x 5120 (l/m = 0.60) in MAI-Base-1 to 4096 x 4352 (l/m = 0.94) in MAI-Base-2, while the model grew from 1T to 2.8T parameters and from 78 to 102 layers. MAI-Base-2 is rung L102 of the same published scaling ladder whose earlier rungs all fixed l/m = 2/3, so this is a deliberate revision of the shape rule as the model was scaled up, decided with a ladder and a compute budget far beyond ours. It moves in the direction our analysis points, and it is exactly the kind of evidence about behaviour under scaling that we cannot produce ourselves.
  2. The three most recent systems all sit close to square, within 0.17 of l = m, so they fall well inside the tolerance we state in Section 3.2 and not merely at its edge; Muse Spark Open uses exactly l = m = 4096. They come from three independent frontier efforts, at 85x to 350x our largest total parameter count, 5x to 7x our residual width, 4x to 6x our depth, and at context lengths up to 250x ours.

Thank you again for the follow-up. We hope the new results and the added discussion help address your concerns, and we are glad to discuss any of them further.

Comments on the paper

2 comments

Thank you for the follow-up. On the magnitude of the gains, although they are not large in absolute terms, we would note that the scale of loss noise is setting-dependent, varying with the training infrastructure, the data distribution, the validation set, the model architecture and other factors. In our well-built experimental environment the standard deviation of validation loss is 1.2e-3 to 1.5e-3, so our improvements are significant. In our production experience, an improvement above 5e-3 is already meaningful and one above 1e-2 would be excellent. Our gains come at matched parameters, activated FLOPs and communication volume, which is why our earlier response expressed them as 4-6% equivalent compute. We then answer your questions below.

1. Design recommendation

At a fixed per-layer routed budget, keep every expert near-square, and give the low-ID shallow prefix (about 1/3 of the layers) a smaller expert input width ℓ with proportionally more experts, while the remaining blocks use a larger ℓ with proportionally fewer.

To be safe, before committing to this, one can train one's own base model for about 2k steps and measure the ID profile to confirm that it matches the pattern we observe, with the first third of the blocks below the remaining two thirds.

2. Source of the gain

As we agreed in the previous round, the gain of the layerwise allocation comes from low-ID layers tolerating a smaller rank ceiling q, so that the same per-layer budget can be spent on more experts and more activated width. We do not attribute this gain to a larger rank ceiling; the larger ceiling is what motivates the near-square choice within a layer, not what explains the across-depth allocation.

On separating these variables: under our constant-cost construction they cannot be separated, because they are not independent variables. Let ℓ be the expert input width, m the expert intermediate width, N the number of routed experts in a layer and K the number activated per token. We hold three quantities fixed in every layer: the dispatch volume V = Kℓ, the activated expert compute C = Kℓm, and the total routed-expert parameters P = Nℓm. These three constraints immediately give

K = V/ℓ, N/K = P/C, m = C/V,

so changing ℓ forces a fixed matching value on K, N and m. Reducing ℓ by a factor r, for example, shrinks each expert's parameter count ℓm by the same factor r, while the activated expert count K, the pool size N and the total activated width Km = C/ℓ all rise by 1/r. In other words, whatever budget is saved inside each expert must reappear as more experts. This is a direct consequence of the budget constraint.

"A lower q", "more experts" and "a larger activated width" are therefore three descriptions of one move along ℓ. Varying them independently requires breaking the budget match, at which point the comparison is no longer between two ways of spending the same budget. The trade is also deliberate: as stated at the beginning of Section 4, we intend to give up per-expert capacity in the lower-ID layers in order to buy more experts and more activated width. Reducing q alone, while keeping the number of experts and width unchanged, would simply lower the parameter count of the layer and degrade performance directly. Our analysis in Section 5.3 shows that this trade is not profitable at every layer, and becomes profitable in the low-ID layers.

The revised PDF clarifies this in the main text: Section 1 (Page 2) now describes the trade-off explicitly.

3. ID versus depth

Yes. In our two models the recommendation is exactly what you state: smaller and proportionally more experts in roughly the first third of the MoE blocks.

To see how far this carries beyond our two models, we measured the same ID profile on four released MoE checkpoints, Qwen2-57B-A14B, Qwen3-30B-A3B-Base, DeepSeek-V2 and Qwen3-235B-A22B. The profiles are added to the revised PDF as Figure 8 on Page 23. The depth-wise pattern is consistent across all models: ID rises through the shallow blocks, peaks in the middle-to-deep region, and then declines slowly and only mildly toward the output. In the two deepest checkpoints, DeepSeek-V2 and Qwen3-235B-A22B, ID keeps rising until 0.7-0.8 of depth, but their normalized profiles are otherwise close in shape to ours.

The split point varies somewhat, though not greatly: 0.25 for Qwen2-57B-A14B and Qwen3-30B-A3B-Base, 0.40 for DeepSeek-V2, and 0.41 for Qwen3-235B-A22B. All of them therefore fall in the shallow part of the stack, so the recommendation keeps the same form, with the boundary shifting modestly from model to model.

What ID provides is the analysis that produced this depth-based split, and a way to verify it. Our experiments suggest the pattern is stable, but we would still recommend confirming it on the target model with a single 2k-step proxy run, which costs little. ID also supplies the direction, which depth alone does not.

Thank you again for the follow-up.

AuthorsOpenReview ↗

General Response

We sincerely thank all PCs, SACs, ACs, and Reviewers for their time and effort in evaluating our paper and for their constructive feedback.

We are grateful that the reviewers found the question worth asking: Reviewer K3QY called expert shape in MoEs "an often-overlooked design question", Reviewer zWRq found the rank-bound argument and the cost-controlled sweeps sound and the layerwise ID analysis informative, and Reviewers RUyn and zWRq noted that the allocation gains come "with little additional overhead, keeping per-layer compute, parameters and communication nearly unchanged".

The revision adds new experiments and discussion as follows.

Reviewer K3QY

  • Extending expert shaping across depth: A new discussion appendix in three parts, on allocation schedules beyond two segments and how to transfer the rule to a new architecture (F.1), per-layer budget allocation (F.2), and a shape-aware scaling law together with what calibrating it would require (F.3) (Appendix F, Pages 24-25).
  • Allocation schemes beyond two segments: Interleaved and randomly shuffled placements of the two expert configurations under a matched per-layer budget (Appendix D, Table 10 on Page 18).
  • Per-layer budget and boundary sensitivity: A matched-total variant that shifts routed budget between segments, and a larger displacement of the regime boundary (Appendix D, Page 18 and Table 9 on Page 18).

Reviewer RUyn

  • Source of the gain: The layerwise mechanism is now stated as a capacity-versus-granularity trade-off under a fixed per-layer budget (Section 1 on Page 2).
  • Magnitude of the gain: An equivalent-compute framing of the improvement, at matched parameters, activated FLOPs, and dispatch volume (Section 5.3 on Page 8).
  • ID versus depth: Intrinsic-dimension profiles of four released MoE checkpoints under our estimation protocol, showing where the ID-based split falls relative to depth (Appendix E.3, Figure 8 on Page 23 and Table 17 on Page 24).

Reviewer zWRq

  • Pipeline parallelism: Per-stage times, bubble fractions, and end-to-end pipeline time for the ID-guided and uniform models on a fixed allocation(Appendix B.4, Table 6 on Page 16).
  • Random per-layer shuffle: Included in the placement ablation listed above (Table 10 on Page 18).

All newly added manuscript content is highlighted in blue. Detailed responses to each reviewer's comments are provided in our individual replies.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.