- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes a training recipe to reduce (bound) the peak memory required for MoE training. This is done by combining 4 methods:
- chunked expert dispatch
- ring-based vocab projection
- checkpoint-boundary offload
- GPU-streamed offloaded AdamW
These methods are tested in isolation and together against weak baselines such as materialized fp32 logits and Adam with CPU offloading.
Even though the core contribution of this paper is a training recipe, no source code is provided.
Critical strengths
- each piece of the recipe reduces memory compared to the baseline while preserving numerics
- memory bound is known from the launch config, doesn't depend on dynamic routing behavior
- enables MoE training up to 1M context length on 120B–667B models (8–32x the context length of the FSDP2 baseline)
Critical weaknesses
The experimental setup seems contrived.
For example, Table 3 trains gpt-oss-20b (80GB in fp32) on 8xH200 GPUs (with 1,128GB memory) with activation offloading. Why is offloading necessary here? Is it because the token batch size is too large? In which case, wouldn't it be sufficient to use gradient accumulation, fused softmax, etc? The batch size is 556k tokens but the max context length of gpt-oss-20b is 128k so this must be at least 4 sequences. In which case we should be able to do gradient accumulation over the batch dimension.
Similarly, the isolated component tests seem weak. In Table 2, why is Ring-DTP compared against naive materialized FP32 logits rather than chunked / fused cross-entropy?
Finally, the FSDP2 baseline seems weak. For example, Megatron-Core already provides fused linear cross-entropy that never materializes the logits.
Questions for authors
- Have you tested and tried optimizing training in Megatron-Core?
- Table 2: Why was Ring-DTP only benchmarked against a naive materialized FP32 logit baseline?
- Table 3: Why is CPU activation offloading needed for a 20B model on 8xH200s (1,128 GB HBM)? What is the batch shape?
6 comments
Official Response from the Authors (Cont.)
A stronger FSDP2 baseline (fused linear CE, Megatron), and the CPU-Adam comparison
FSDP2 baseline: seems weak, e.g. Megatron-Core's fused linear cross-entropy never materializes the logits.
Megatron's fused linear CE. Megatron-LM's fused linear cross-entropy was introduced only for Blackwell GPUs (PR #2256). On our H200 nodes we use the same class of kernel, Liger fused linear CE, both in Table 2 and inside the end-to-end baseline.
A stronger FSDP2-best. 120B / 128K on 16 H200, full causal, layout FSDP 1 · EP 16 · CP 8 · TP 2. MFU uses the TorchTitan convention.
| Variant | Status | tok/s/GPU | MFU | Peak reserved (GiB) |
|---|---|---|---|---|
| As submitted (sweep) | PASS | 241 | 3.0% | 135.7 |
| + Liger fused linear CE | PASS | 189 | 2.3% | 133.8 |
| + gradient accumulation to 4.2M tokens per optimizer step | OOM at step 2, once fp32 Adam state is resident | — | — | — |
| + gradient accumulation + Liger fused linear CE | OOM at step 2, once fp32 Adam state is resident | — | — | — |
| Ours | PASS | 1,390 | 15.0% | 131.5 |
- Fused CE removes the logit allocation but not the peak. The peak is set by fp32 optimizer state (105–115 GiB, 75–82% of HBM before any activation).
- At 1M every layout fails on the fp32 logits, which Ring-DTP removes.
- Gradient accumulation cannot hold our 4.2M-token batch without offload, with or without fused CE
- End to end at 120B / 128K, swapping OffloadStreamAdamW for a multi-threaded host AdamW raises the optimizer’s per-step overhead from 20 s to 48 s (2.4×)
CPU optimizer offload, FP8 and pipeline parallelism. Each cell gives the best of three layouts (FSDP 1 · EP 16 with CP 8 · TP 2, CP 4 · TP 4, or CP 16), as tok/s/GPU and MFU under the TorchTitan convention.
| Configuration | 128K | 1M |
|---|---|---|
| FSDP2-best | 241, 3.0% | — |
| + CPU optimizer offload | 226, 2.8% | OOM on all 3 layouts (fp32 logits, 49 GiB allocation) |
| + FP8 + CPU offload | 223, 2.8% | — |
| PP = 2 + EP = 8 + CPU offload (1F1B / interleaved / DualPipeV) | OOM on all three | — |
| Ours | 1,390, 15.0% | 652, 43.4% |
We also ran Megatron-Core itself. At 1M its fused vocabulary-parallel cross-entropy fails on the 49 GiB fp32 logits; Transformer Engine's cross-entropy gets past that point but fails later on the expert-dispatch buffer
Q1: whether we tested and tried optimizing training in Megatron-Core.
Yes. We ran Megatron-Core during the discussion period and tuned it. It is a strong baseline, although our stack is the only system that trains 1M.
Setup.
- Software: Megatron-Core 0.16.1 with Transformer Engine 2.11.
- Starting point: Megatron's own Kimi-K2 model configuration (with shared-expert overlap and sequence parallelism), plus Megatron's standard performance options:
- grouped expert GEMM and fused token permutation;
- fused cross-entropy and the distributed optimizer.
- Model: the same as FSDP2-best, 120B (122.8B parameters, DeepSeek-V3-style attention, 384 experts, top-8 routing).
- Hardware and layout: 16 H200, TP 2 · CP 8 · EP 16, full-causal attention.
- Batch: our batch of 4.2M tokens per optimizer step.
- Data and initialisation: random initialisation; Megatron's built-in synthetic data.
- Our stack: runs its own 120B model, as in the paper (grouped-query attention, 384 experts, top-8). MFU uses each model's own FLOP count under the TorchTitan convention.
Tuning. We tuned Megatron-Core on one 8-GPU node, using a copy of the model with 184 experts. That gives about the same parameters and tokens per GPU as 120B on 16 GPUs. Rows are cumulative; tok/s/GPU at 64K:
| Megatron-Core setting | Result |
|---|---|
| Default: distributed optimizer, full recompute | OOM at the first optimizer step, while allocating Adam state (130.7 GiB allocated) |
| + optimizer CPU offload (100% of state) | 504 tok/s/GPU |
| + 4 micro-batches per optimizer step | 1,402 |
| + offload 50% of optimizer state instead | 1,800 |
| 100% offload at 32 micro-batches | exceeds the node's host memory |
Results in the next thread below -
Official Response from the Authors (Cont.)
Megatron-Core, code, and presentation
On 16 GPUs at 128K, the default data loader ran out of host memory. It builds a full 131,072 × 131,072 attention mask per sample on the CPU, which the fused attention kernel never reads. Disabling it (--no-create-attention-mask-in-dataloader) fixed the memory and removed seconds of data loading per micro-batch. At 1M we tried both of Megatron's cross-entropy implementations: its native fused vocabulary-parallel kernel and Transformer Engine's.
Results. 120B, 16 H200, 4.2M tokens per optimizer step. Each cell is tok/s/GPU · MFU.
| Context | Megatron-Core (tuned) | Our stack | FSDP2-best |
|---|---|---|---|
| 1M | fails: OOM on the fp32 logits (default cross-entropy); OOM in expert dispatch after 2 iterations (Transformer Engine cross-entropy) | trains: 652 · 43.4% (full run) | fails (OOM on the fp32 logits) |
Notes:
- 1M with the default cross-entropy: the failing allocation is 49.09 GiB, which is 131,072 tokens × 100,544 vocabulary entries × 4 bytes. Any 16-GPU layout without data parallelism gives each GPU the same logits (tokens × vocabulary / 16), so the TP/CP split cannot avoid it.
- 1M with Transformer Engine's cross-entropy: Megatron-Core completed 2 iterations at 247 tok/s/GPU (18.8% MFU). In the third, it failed allocating 13.62 GiB in the expert-dispatch buffer, whose size depends on how evenly tokens are routed.
What this shows.
- At 1M, dropless Megatron-Core does not train 120B on 16 H200. It fails on the two memory terms our mechanisms bound:
- the fp32 logits, which Ring-DTP removes;
- the routing-dependent expert-dispatch buffer, which PipelinedLLEP's token-chunked dispatch bounds.
- Our stack trains the same setting at 43% MFU.
- Our contribution is reach and per-GPU memory at long context: training at 1M, where FSDP2-best and dropless Megatron-Core run out of memory.
Source code: none is provided, although the core contribution is a training recipe.
We are committed to release the codes pending institutional approval. In the camera-ready version, we will attach a self-contained reference pseudo-code implementation of the four mechanisms : Ring-DTP, token-chunked expert dispatch, SCO and OffloadStreamAdamW, along with the full distributed training stack, with launch scripts, per-run configurations and the log parsers behind every table.
The final version also adds a reproducibility statement.
Presentation and soundness. The final version makes these changes:
- FLOP accounting. One convention for both systems, with matched-workload comparisons
- The above tested stronger baselines: fused linear CE, gradient accumulation, FP8 and pipeline parallelism, with reach restated as measured.
- Per-row tables:
- which mechanisms each row engages;
- the per-term memory budget measured on the allocator;
- the single-knob ablation;
- the full parallelism configuration of both systems.
- FSDP2-best defined in the main text, and the baseline architectures stated explicitly.
- Text fixes:
- the Table 2 caption will be corrected and the table extended;
- the experimental-setup description in the appendix will be clarified;
- reach statements are scoped to measured limits;
- New citations: Megatron-Core fine-grained activation offloading, the Megatron-Core MoE report, Deep Optimizer States, and DualPipe/DualPipeV.
We hope the stronger baselines, the new measurements and the code address the reviewer's concerns, and we would be grateful if the reviewer would reconsider their assessment in light of them.
Official Response from the Authors
Table 3 (why offload a 20B model; batch shape) and Table 2 (fused and chunked CE)
The reviewer is right that OpenAI’s gpt-oss-20b at its originally native context does not need activation offloading. But table 3 is a controlled measurement of one memory term at a long-context shape, and we use gpt-oss-20b only as a case-study architectural shape for infrastructure measurements, not with an intent to actually post-train this model for actual tasks.
Batch shape:
Gradient accumulation does not remove this term. At this shape, the 47 checkpoint boundaries hold about 17.6 GiB per rank (47 × 69,632 × 2,880 × 2 B), and SCO lowers the measured peak by 16.1 GiB. Because the step is a single sequence, there is no batch dimension to accumulate over. Gradient accumulation splits a batch across sequences; it cannot split one sequence.
Where SCO is needed in practice.
The final paper revision would state the condition under which SCO is needed.
We extended Table 2 with all of these methods: bf16, V = 201,088, P = 8, on 8 H200, on one harness. The harness first reproduces the submitted fp32 Table 2 (42.46 → 7.32 GiB at 16K). Each cell gives forward+backward peak allocated (GiB) / step time (ms).
The distinction is structural: