- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies expert shape in LatentMoE. A rank argument and cost-controlled sweeps show that near-square experts work best. It then uses layerwise intrinsic dimension to give shallow layers more experts with a smaller latent width and deep layers fewer, wider ones, keeping per-layer cost nearly constant. At two scales this lowers loss and improves downstream accuracy over uniform LatentMoE.
Critical strengths
- A sound theory guides the shape design of LatentMoE: a rank bound on each expert predicts that near-square experts work best, and cost-controlled sweeps at two scales confirm it.
- A good analysis of the role of MoE layers at different depths: intrinsic dimension rises through the shallow layers and peaks in the middle-to-deep layers, consistently across scales and training, which motivates giving different layers different expert shapes.
Critical weaknesses
- I am not fully convinced that the model is "nearly homogeneous from the system's perspective". Dispatch volume is matched, but message structure and expert GEMM sizes change with depth, which can unbalance pipeline stages. There are no pipeline-parallel results, and the throughput numbers come without hardware, parallel setup or variance.
- The evidence for the ID-guided allocation is not yet fully sufficient. The layerwise controls seem to be single runs; there are reversed-order controls but no random per-layer shuffle, and moving the boundary barely matters, so it is unclear what ID adds beyond "narrow early, wide late".
Questions for authors
- Under pipeline parallelism, how do per-stage times and bubbles compare with uniform LatentMoE, on what hardware and setup?
- How does the allocation compare with a random per-layer shuffle, with several seeds?
4 comments
Thank you for the response.
-
System homogeneity. Thank you for the pipeline-parallel measurements. They make the overall results more reliable. Could you also report the time of each layer? With per-layer times, one could estimate the stage balance for larger models or more pipeline stages without new runs.
-
Uniform wide vs. uniform narrow. The two use the same parameters and compute, yet uniform wide is better at 3.4B. This seems different from mainstream designs such as DeepSeekMoE, which favor more and smaller experts. Why?
Thank you for the follow-up, and for the suggestion to report per-layer times. We report the per-layer times below and then use them to estimate PP=3 and PP=4.
Q1. Per-layer times
| Component (forward + backward, per micro-batch) | 8B | 3.4B |
|---|---|---|
| Wide MoE layer (8B: 640×512, K/N = 16/512; 3.4B: 512×440, 14/448) | 62.0 ms | 44.8 ms |
| Uniform α = 2 MoE layer (8B: 640×512, 16/512; 3.4B: 512×384, 16/512) | 62.0 ms | 45.8 ms |
| Narrow MoE layer (8B: 464×512, 22/704; 3.4B: 352×448, 20/640) | 64.4 ms | 48.0 ms |
| Block 0 (dense) | 38.5 ms | 25.8 ms |
| LM head | 110.6 ms | 89.7 ms |
| Final norm and loss (last stage only) | 35.9 ms | 35.0 ms |
Reconstructing a stage also requires two kinds of overhead that do not depend on expert shape. The first is charged to a particular layer according to its position:
| Extra time charged to a layer by its position (per micro-batch) | 8B | 3.4B |
|---|---|---|
| First MoE layer of stage 0, extra forward | +5.0 ms | +3.9 ms |
| First layer of every later stage, extra forward | +6.2 ms | +6.9 ms |
| Last layer of a non-final stage, extra backward | +0.7 ms | +1.3 ms |
The second is not attributable to any layer:
| Schedule gap (per occurrence) | 8B | 3.4B |
|---|---|---|
| Between forward and backward on the last stage (loss post-processing) | 15.6 ms | 15.8 ms |
| Before each forward and each backward (host) | 2.4 ms | 2.1 ms |
| At each stage boundary (activation transfer and synchronisation) | 14.8 ms | 13.7 ms |
Three observations may be useful.
First, summing the layer times and the position-dependent overheads according to the layer partition reproduces the per-stage compute times in our previous response to within 0.5%. For the 8B ID-guided model under the 9+7 partition, stage 0 is 38.5 + 5 × 64.4 + 3 × 62.0 + 5.0 + 0.7 = 552.2 ms per micro-batch, or 8.84 s per iteration, against the 8.84 s reported; stage 1 is 7 × 62.0 + 110.6 + 35.9 + 6.2 = 586.7 ms, or 9.39 s, against 9.34 s.
Second, at 8B our deep regime is the same configuration as the α = 2 baseline and measures identically, which is a useful internal check. At 3.4B the deep regime differs from the α = 2 baseline and is slightly faster, consistent with executing fewer and larger grouped GEMMs. The quantity that governs stage balance is the ratio between the two regimes: with our untuned kernels a narrow layer costs 3.9% more than a wide layer at 8B and 7.1% more at 3.4B.
Third, these numbers allow the pipeline time to be estimated for other configurations. For stage s, let c_s be the sum of its layer times and position-dependent overheads, with the LM head and loss added on the last stage, and let t_s be c_s plus two host gaps, plus the loss gap on the last stage. With p stages, m = 16 micro-batches and stage-boundary gap g, a 1F1B schedule gives
T = γ · [(m − 1) · max_s t_s + Σ_s t_s + 2(p − 1) · g].
The idealised formula ignores variation across micro-batches and occasional synchronisation waits, so we calibrate γ as the mean ratio of measured to predicted time for the uniform model at PP=2, giving γ = 1.016 at 8B and 1.031 at 3.4B. With these two constants the formula reproduces all eight measured PP=2 pipeline times, for both models and both partitions, to within 0.6%.
Applying it to PP=3 and PP=4 gives the following. The PP=2 rows are the measurements from our previous response; the PP=3 and PP=4 rows are estimates obtained from the per-layer times above, and every entry can be recomputed from the tables above. At each setting both models use the same layer partition, chosen as the best contiguous partition for the uniform model, so the comparison does not depend on re-tuning the partition for our model. At PP=4 we additionally report the best contiguous partition for the ID-guided model, which is what a deployment would use; at PP=3 the two coincide.
| Scale | PP | Model | Partition (layers per stage) | Stage compute (s) | Bubble per stage | Pipeline time |
|---|---|---|---|---|---|---|
| 8B | 2 | uniform | 9+7 / 10+6 | 8.66 / 9.38; 9.62 / 8.42 | 17.4% / 10.5%; 7.8% / 19.3% | 10.48 s; 10.43 s |
| ID-guided | 9+7 / 10+6 | 8.84 / 9.34; 9.79 / 8.36 | 15.5% / 10.6%; 7.6% / 21.0% | 10.45 s; 10.59 s (+1.5%) | ||
| 3 | uniform | 6+6+4 | 5.67 / 6.06 / 6.41 | 26.0% / 20.9% / 16.3% | 7.66 s | |
| ID-guided | 6+6+4 | 5.86 / 6.06 / 6.41 | 23.5% / 20.9% / 16.3% | 7.66 s (+0.0%) | ||
| 4 | uniform | 4+5+5+2 | 3.68 / 5.07 / 5.07 / 4.43 | 40.5% / 18.1% / 18.1% / 28.4% | 6.19 s | |
| ID-guided, uniform's partition | 4+5+5+2 | 3.80 / 5.15 / 5.07 / 4.43 | 39.5% / 18.0% / 19.2% / 29.4% | 6.27 s (+1.4%) | ||
| ID-guided, own best partition | 5+4+5+2 | 4.83 / 4.12 / 5.07 / 4.43 | 21.9% / 33.5% / 18.1% / 28.4% | 6.19 s (+0.0%) | ||
| 3.4B | 2 | uniform | 7+5 / 8+4 | 4.91 / 5.77; 5.62 / 5.03 | 25.7% / 12.6%; 10.2% / 19.7% | 6.60 s; 6.26 s |
| ID-guided | 7+5 / 8+4 | 5.01 / 5.70; 5.70 / 4.97 | 23.4% / 13.0%; 10.0% / 21.5% | 6.55 s; 6.33 s (+1.1%) | ||
| 3 | uniform | 5+5+2 | 3.43 / 3.80 / 3.57 | 24.5% / 16.4% / 21.4% | 4.54 s | |
| ID-guided | 5+5+2 | 3.57 / 3.72 / 3.54 | 20.9% / 17.7% / 21.6% | 4.51 s (−0.6%) | ||
| 4 | uniform | 3+4+4+1 | 1.96 / 3.06 / 3.06 / 2.84 | 49.4% / 21.0% / 21.0% / 26.8% | 3.88 s | |
| ID-guided, uniform's partition | 3+4+4+1 | 2.03 / 3.10 / 3.00 / 2.82 | 47.7% / 20.2% / 22.9% / 27.4% | 3.89 s (+0.3%) | ||
| ID-guided, own best partition | 4+3+4+1 | 2.80 / 2.33 / 3.00 / 2.82 | 27.5% / 39.6% / 22.4% / 26.9% | 3.86 s (−0.4%) |
At PP=3 the best partition is the same for both models and the extra cost of the narrow layers falls on a stage that is not on the critical path, so the estimated pipeline time matches the uniform model at 8B and is slightly lower at 3.4B, where our deep configuration (512×440, K=14) is faster than the α = 2 layer (512×384, K=16). At PP=4 the uniform model's partition places narrow layers on the critical stage; holding that partition fixed, the estimated overhead is 1.4% at 8B and 0.3% at 3.4B, no larger than the 1.5% and 1.1% we measured at PP=2. Moving a single layer to rebalance the stages removes it, giving 6.19 s at 8B and 3.86 s at 3.4B. We report both so that the comparison can be read either way.
Q2. Uniform wide versus uniform narrow
The main difference from earlier MoE work is the latent structure: the d-dimensional hidden state is first compressed to ℓ before being passed to the experts, and this compression is lossy. A standard MoE does not pay this cost, because every expert still receives the full hidden state regardless of how fine the granularity is, and this is the setting in which the DeepSeekMoE result holds. Uniform narrow and uniform wide differ in exactly this respect: uniform narrow compresses the hidden state more (at 3.4B, ℓ = 352 against 512, from d = 1024) and obtains more experts in return (20/640 against 14/448). This creates a trade-off: the additional experts help, while the stronger compression hurts. In our experiments uniform wide is the better of the two, because for uniform narrow the loss from the stronger compression outweighs the gain from the additional experts.
This trade-off is also where our layerwise allocation comes from: we hypothesise that layers with lower ID tolerate a stronger compression ratio, so that spending it on more experts pays off in those layers even though it does not pay off when applied at every layer.
Thank you again for these questions. We believe the measurements and clarifications will help us improve the quality of the paper.
Thank you for your careful reading and for recognizing the rank-based shape rationale and the depth-wise ID analysis. Below we address your concerns.
W1 / Q1. System homogeneity and pipeline parallelism.
We agree that the homogeneity holds for theoretical computation and communication, and cannot be perfectly realized because of implementation-level inefficiencies such as smaller grouped GEMMs. What the construction matches is per-layer parameter count, activated FLOPs and dispatch volume. Below we provide concrete pipeline-parallel measurements.
Setup: TP=1, PP=2, EP=8, DP=8, 16 micro-batches per iteration (micro-batch size 4, sequence length 4096), 16 NVIDIA H20 GPUs. The throughput in Appendix B.3 (Table 5) was measured on 64 GPUs; for stable and accurate per-stage timing, here we use two fixed 8-GPU nodes. For each scale we report two layer partitions, one where the last stage (holding the LM head) is the critical path and one where the first stage is the critical path. The uniform baseline is LatentMoE with alpha=2.
With our generic kernels, the narrower shallow configuration shows up as +1.5% (8B) and +1.1% (3.4B) when the first stage is critical, and is absorbed when the last stage is critical. We did not tune kernels for these shapes and will acknowledge this as a limitation.
W2 (a). Are the layerwise controls in Table 3 single runs? Because of compute limits, only the main results in Table 2 use three runs. As reported in Appendix B.1, the standard deviation of validation loss is 1.2e-3 to 1.5e-3, so for the variants in Table 3 we report single-run validation loss.
W2 (b) / Q2. Random per-layer shuffle.
We added a random-shuffle control at 3.4B, with two random-shuffle placements and two interleaved placements. ID-guided performs best.
W2 (c). What does ID add beyond "narrow early, wide late"?
We agree that in the standard decoders studied here, the ID ordering and relative depth give the same split: about the first third of MoE blocks at both scales (4/11 and 5/15). ID at least indicates the direction and the approximate split point. First, the direction is not obvious. The reversed allocations are worse than both uniform endpoints at both scales (Table 3), so guessing the direction wrong costs more than not allocating at all. The ID profile supplies this direction from a measurable property of the representations. Second, the small effect of moving the boundary by one block only shows that the ID-based split tolerates small deviations. A larger shift does hurt: at 3.4B, moving the boundary from 4/11 to 8/11 raises validation loss from 2.2234 to 2.2367.
Thank you again for your careful review and constructive suggestions.