acceptodds
Under review as a conference paper at ICLR 2027

Flattening Every Memory Peak in Long-Context Mixture-of-Experts Training

Abstract

Training a Mixture-of-Experts (MoE) model at long context or large batch size fails as soon as any one component's peak allocation exceeds device memory, so the target is every peak at once, not the average footprint. Four are left unbounded by the parallelism plans in common use, and each grows differently: expert dispatch with the routing matrix, the vocabulary projection with tokens times vocabulary, gradient checkpoint boundaries with depth times sequence length, and optimizer state with parameter count. Which one runs out first changes with the model, the context length, and the device count, so lowering the largest only exposes the next. We bound all four with schedules whose GPU working set is fixed at launch: PipelinedLLEP extends least-loaded expert parallelism with a cap on the tokens each source contributes to a dispatch chunk, Ring-DTP circulates activations or weight shards around a ring at the vocabulary projection and folds each block of logits into an online log-sum-exp, Selective checkpoint offload (SCO) keeps the one long-lived tensor of each checkpoint boundary in CPU memory, and OffloadStreamAdamW turns the serial CPU Adam update of optimizer offload into a bucket pipeline. All four change only the order and granularity of computation and data movement, so the loss and gradients stay exact. In matched component tests, they cut the MoE dispatch peak by up to without losing throughput, the vocabulary projection peak by , and the offloaded optimizer step by faster. Composed on MoE models from 120B to 667B parameters, they train at 1M context length, – the reach of a tuned FSDP2 baseline, and up to its throughput.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
distributed-training, LMSYS, memory-efficiency, mixture-of-experts, long context

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bpfbOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a training recipe to reduce (bound) the peak memory required for MoE training. This is done by combining 4 methods:

  • chunked expert dispatch
  • ring-based vocab projection
  • checkpoint-boundary offload
  • GPU-streamed offloaded AdamW

These methods are tested in isolation and together against weak baselines such as materialized fp32 logits and Adam with CPU offloading.

Even though the core contribution of this paper is a training recipe, no source code is provided.

Critical strengths
  • each piece of the recipe reduces memory compared to the baseline while preserving numerics
  • memory bound is known from the launch config, doesn't depend on dynamic routing behavior
  • enables MoE training up to 1M context length on 120B–667B models (8–32x the context length of the FSDP2 baseline)
Critical weaknesses

The experimental setup seems contrived.

For example, Table 3 trains gpt-oss-20b (80GB in fp32) on 8xH200 GPUs (with 1,128GB memory) with activation offloading. Why is offloading necessary here? Is it because the token batch size is too large? In which case, wouldn't it be sufficient to use gradient accumulation, fused softmax, etc? The batch size is 556k tokens but the max context length of gpt-oss-20b is 128k so this must be at least 4 sequences. In which case we should be able to do gradient accumulation over the batch dimension.

Similarly, the isolated component tests seem weak. In Table 2, why is Ring-DTP compared against naive materialized FP32 logits rather than chunked / fused cross-entropy?

Finally, the FSDP2 baseline seems weak. For example, Megatron-Core already provides fused linear cross-entropy that never materializes the logits.

Questions for authors
  • Have you tested and tried optimizing training in Megatron-Core?
  • Table 2: Why was Ring-DTP only benchmarked against a naive materialized FP32 logit baseline?
  • Table 3: Why is CPU activation offloading needed for a 20B model on 8xH200s (1,128 GB HBM)? What is the batch shape?
6 comments
AuthorsOpenReview ↗

Official Response from the Authors

Table 3 (why offload a 20B model; batch shape) and Table 2 (fused and chunked CE)

Table 3 / Q3: why offload a 20B model on 8×H200, what the batch shape is, and whether gradient accumulation or fused softmax would suffice.

The reviewer is right that OpenAI’s gpt-oss-20b at its originally native context does not need activation offloading. But table 3 is a controlled measurement of one memory term at a long-context shape, and we use gpt-oss-20b only as a case-study architectural shape for infrastructure measurements, not with an intent to actually post-train this model for actual tasks.

Batch shape:

  • Each step is one sequence of up to 557,056 tokens (556,432 used), not several shorter ones.
  • It is sharded across the 8 ranks along the sequence dimension: sequence parallelism and expert parallelism both of degree 8, as in the table caption. That gives 69,632 tokens per rank.
  • This deliberately exceeds gpt-oss-20b's native 128K context: the table measures memory at a long-context shape, not model quality.
  • With SCO off, peak HBM is 139.79 GiB, at the device limit. With all 47 boundaries offloaded it falls to 123.73 GiB.
  • Throughput is unchanged (2,641 → 2,687 tok/s/GPU), and the largest batch that completes 10 steps rises from 557,056 to 655,360 tokens (+17.6%).
  • Node RAM rises from 402.5 to 593.4 GiB.

Gradient accumulation does not remove this term. At this shape, the 47 checkpoint boundaries hold about 17.6 GiB per rank (47 × 69,632 × 2,880 × 2 B), and SCO lowers the measured peak by 16.1 GiB. Because the step is a single sequence, there is no batch dimension to accumulate over. Gradient accumulation splits a batch across sequences; it cannot split one sequence.

Where SCO is needed in practice.

  • At 120B with 65,536 tokens per rank (128K and 1M context), SCO is required. With it, the run fits at 131.5 GiB (128K) and 139.5 GiB (1M).
  • At 241B and 667B, with 32,768 tokens per rank, the boundaries fit, and SCO is not used.

The final paper revision would state the condition under which SCO is needed.

Table 2 / Q2: why Ring-DTP is compared against materialized FP32 logits rather than chunked / fused cross-entropy.

We extended Table 2 with all of these methods: bf16, V = 201,088, P = 8, on 8 H200, on one harness. The harness first reproduces the submitted fp32 Table 2 (42.46 → 7.32 GiB at 16K). Each cell gives forward+backward peak allocated (GiB) / step time (ms).

Tokens per rankDenseRing-DTPCCELiger FLCEChunked CEGather-X + vocab-parallel CE
16,38421.4 / 2213.7 / 3395.8 / 87414.3 / 37315.1 / 29620.8 / 278
32,76840.0 / 4425.4 / 6596.2 / 1,74315.1 / 59115.5 / 59841.2 / 554
  • Ring-DTP has the lowest peak of all six methods and no method beats Ring-DTP on both memory and time at any size.

The distinction is structural:

  • Fused kernels remove the logits for a given set of tokens, but each rank still holds the full classifier weight and its gradient.
  • Ring-DTP shards the classifier across ranks that each hold their own tokens. Its per-strip computation can itself use a fused kernel, so the two compose.
AuthorsOpenReview ↗

Official Response from the Authors (Cont.)

A stronger FSDP2 baseline (fused linear CE, Megatron), and the CPU-Adam comparison

FSDP2 baseline: seems weak, e.g. Megatron-Core's fused linear cross-entropy never materializes the logits.

Megatron's fused linear CE. Megatron-LM's fused linear cross-entropy was introduced only for Blackwell GPUs (PR #2256). On our H200 nodes we use the same class of kernel, Liger fused linear CE, both in Table 2 and inside the end-to-end baseline.

A stronger FSDP2-best. 120B / 128K on 16 H200, full causal, layout FSDP 1 · EP 16 · CP 8 · TP 2. MFU uses the TorchTitan convention.

VariantStatustok/s/GPUMFUPeak reserved (GiB)
As submitted (sweep)PASS2413.0%135.7
+ Liger fused linear CEPASS1892.3%133.8
+ gradient accumulation to 4.2M tokens per optimizer stepOOM at step 2, once fp32 Adam state is resident———
+ gradient accumulation + Liger fused linear CEOOM at step 2, once fp32 Adam state is resident———
OursPASS1,39015.0%131.5
  • Fused CE removes the logit allocation but not the peak. The peak is set by fp32 optimizer state (105–115 GiB, 75–82% of HBM before any activation).
  • At 1M every layout fails on the fp32 logits, which Ring-DTP removes.
  • Gradient accumulation cannot hold our 4.2M-token batch without offload, with or without fused CE
  • End to end at 120B / 128K, swapping OffloadStreamAdamW for a multi-threaded host AdamW raises the optimizer’s per-step overhead from 20 s to 48 s (2.4×)

CPU optimizer offload, FP8 and pipeline parallelism. Each cell gives the best of three layouts (FSDP 1 · EP 16 with CP 8 · TP 2, CP 4 · TP 4, or CP 16), as tok/s/GPU and MFU under the TorchTitan convention.

Configuration128K1M
FSDP2-best241, 3.0%—
+ CPU optimizer offload226, 2.8%OOM on all 3 layouts (fp32 logits, 49 GiB allocation)
+ FP8 + CPU offload223, 2.8%—
PP = 2 + EP = 8 + CPU offload (1F1B / interleaved / DualPipeV)OOM on all three—
Ours1,390, 15.0%652, 43.4%

We also ran Megatron-Core itself. At 1M its fused vocabulary-parallel cross-entropy fails on the 49 GiB fp32 logits; Transformer Engine's cross-entropy gets past that point but fails later on the expert-dispatch buffer

Q1: whether we tested and tried optimizing training in Megatron-Core.

Yes. We ran Megatron-Core during the discussion period and tuned it. It is a strong baseline, although our stack is the only system that trains 1M.

Setup.

  • Software: Megatron-Core 0.16.1 with Transformer Engine 2.11.
  • Starting point: Megatron's own Kimi-K2 model configuration (with shared-expert overlap and sequence parallelism), plus Megatron's standard performance options:
    • grouped expert GEMM and fused token permutation;
    • fused cross-entropy and the distributed optimizer.
  • Model: the same as FSDP2-best, 120B (122.8B parameters, DeepSeek-V3-style attention, 384 experts, top-8 routing).
  • Hardware and layout: 16 H200, TP 2 · CP 8 · EP 16, full-causal attention.
  • Batch: our batch of 4.2M tokens per optimizer step.
  • Data and initialisation: random initialisation; Megatron's built-in synthetic data.
  • Our stack: runs its own 120B model, as in the paper (grouped-query attention, 384 experts, top-8). MFU uses each model's own FLOP count under the TorchTitan convention.

Tuning. We tuned Megatron-Core on one 8-GPU node, using a copy of the model with 184 experts. That gives about the same parameters and tokens per GPU as 120B on 16 GPUs. Rows are cumulative; tok/s/GPU at 64K:

Megatron-Core settingResult
Default: distributed optimizer, full recomputeOOM at the first optimizer step, while allocating Adam state (130.7 GiB allocated)
+ optimizer CPU offload (100% of state)504 tok/s/GPU
+ 4 micro-batches per optimizer step1,402
+ offload 50% of optimizer state instead1,800
100% offload at 32 micro-batchesexceeds the node's host memory

Results in the next thread below -

AuthorsOpenReview ↗

Official Response from the Authors (Cont.)

Megatron-Core, code, and presentation

On 16 GPUs at 128K, the default data loader ran out of host memory. It builds a full 131,072 × 131,072 attention mask per sample on the CPU, which the fused attention kernel never reads. Disabling it (--no-create-attention-mask-in-dataloader) fixed the memory and removed seconds of data loading per micro-batch. At 1M we tried both of Megatron's cross-entropy implementations: its native fused vocabulary-parallel kernel and Transformer Engine's.

Results. 120B, 16 H200, 4.2M tokens per optimizer step. Each cell is tok/s/GPU · MFU.

ContextMegatron-Core (tuned)Our stackFSDP2-best
1Mfails: OOM on the fp32 logits (default cross-entropy); OOM in expert dispatch after 2 iterations (Transformer Engine cross-entropy)trains: 652 · 43.4% (full run)fails (OOM on the fp32 logits)

Notes:

  • 1M with the default cross-entropy: the failing allocation is 49.09 GiB, which is 131,072 tokens × 100,544 vocabulary entries × 4 bytes. Any 16-GPU layout without data parallelism gives each GPU the same logits (tokens × vocabulary / 16), so the TP/CP split cannot avoid it.
  • 1M with Transformer Engine's cross-entropy: Megatron-Core completed 2 iterations at 247 tok/s/GPU (18.8% MFU). In the third, it failed allocating 13.62 GiB in the expert-dispatch buffer, whose size depends on how evenly tokens are routed.

What this shows.

  • At 1M, dropless Megatron-Core does not train 120B on 16 H200. It fails on the two memory terms our mechanisms bound:
    • the fp32 logits, which Ring-DTP removes;
    • the routing-dependent expert-dispatch buffer, which PipelinedLLEP's token-chunked dispatch bounds.
  • Our stack trains the same setting at 43% MFU.
  • Our contribution is reach and per-GPU memory at long context: training at 1M, where FSDP2-best and dropless Megatron-Core run out of memory.

Source code: none is provided, although the core contribution is a training recipe.

We are committed to release the codes pending institutional approval. In the camera-ready version, we will attach a self-contained reference pseudo-code implementation of the four mechanisms : Ring-DTP, token-chunked expert dispatch, SCO and OffloadStreamAdamW, along with the full distributed training stack, with launch scripts, per-run configurations and the log parsers behind every table.

The final version also adds a reproducibility statement.

Presentation and soundness. The final version makes these changes:

  • FLOP accounting. One convention for both systems, with matched-workload comparisons
  • The above tested stronger baselines: fused linear CE, gradient accumulation, FP8 and pipeline parallelism, with reach restated as measured.
  • Per-row tables:
    • which mechanisms each row engages;
    • the per-term memory budget measured on the allocator;
    • the single-knob ablation;
    • the full parallelism configuration of both systems.
  • FSDP2-best defined in the main text, and the baseline architectures stated explicitly.
  • Text fixes:
    • the Table 2 caption will be corrected and the table extended;
    • the experimental-setup description in the appendix will be clarified;
    • reach statements are scoped to measured limits;
  • New citations: Megatron-Core fine-grained activation offloading, the Megatron-Core MoE report, Deep Optimizer States, and DualPipe/DualPipeV.

We hope the stronger baselines, the new measurements and the code address the reviewer's concerns, and we would be grateful if the reviewer would reconsider their assessment in light of them.

Reviewer_VFptOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents techniques to address the memory peaks in long-context MoE training, targeting expert dispatch, vocabulary projection, gradient checkpoint boundaries, and optimizer state. The experiments evaluate the techniques at large model scale and long context length, showing promising improvements over an FSDP2 baseline.

Critical strengths
  1. Addressing each of the memory bottleneck in long-context MoE training is practically important, especially when these peaks interact with one another. It represents meaningful engineering work.
  2. The evaluated settings with MoE models from 120B to 667B parameters at 1M context on the stated hardware is quite substantial, whose improvements over the FSDP2 baseline are demonstrated.
Critical weaknesses

W1: The novelty of the two offloading techniques is inssuficiently distinguied from existing approaches, and the practical significance of their differences remains unclear.

  • The distinction between SCO and Megatron's fine-grained activation offloading is unclear. Megatron already supports selective input offloading, asynchronous prefetch, and composition with recomputation. What does SCO contribute beyond Megatron’s existing offloading, which limitations does it address, and how significant are those limitations in practice?
  • The distinction between OffloadStreamAdamW and Deep Optimizer States (2024, arXiv:2410.21316) is unclear. The latter already stages host-resident optimizer state on GPUs for updates and overlaps transfers. Which limitations of the prior approach does OffloadStreamAdamW address specifically?

W2: The baseline tuning appears incomplete, making it difficult to assess whether the reported 10.4x throughput improvemnt and the peak memory reduction holds against a strong MoE training baseline.

  • The described FSDP2 tuning sweep includes expert, context, and tensor parallelism, but does not mention pipeline Given the EP degree span across inter-node connect with 16/32/64 GPUs for the 120B/241B/667B models, could pipeline schedule like DualPipeV further improve communication overlap and thus the baseline performance? What memory trade-off would this involved?
  • Given the 120B–667B model scales evaluated, I found it quite surprising that FP8 wasn't used in the baseline evaluation. Would FP8 training improve the baseline throughput? Would the proposed techniques retain their memory and throughput benefits against an FP8-enabled baseline?
  • Please report the selected parallelism configurations and MFU for both systems under matched workloads and hardware, using consistent FLOP accounting, to help assess whether the baseline is sufficiently optimized.
Questions for authors

No additional questions beyond the requests raised in the weaknesses.

5 comments
AuthorsOpenReview ↗

Official Response from the Authors

W1, novelty of the two offloading techniques

W1: novelty of the two offloading techniques over existing approaches, and the practical significance of the differences.

Our contribution is not offloading itself. It is making each memory peak a launch-time constant, which the paper's peak bound depends on, and composing the four mechanisms in one stack. Below we state precisely what each offloading technique adds.

SCO vs Megatron's fine-grained activation offloading: what SCO adds, which limitations it addresses, and how significant they are in practice.

Megatron-Core's fine-grained offloading can move the same tensors SCO targets: the inputs of attn_norm and mlp_norm are a layer's two checkpoint boundaries, and the Megatron-Core MoE report describes offloading layer inputs alongside full recomputation. We now cite both. What SCO contributes is:

  • It is not tied to a model library. Megatron's offloading is implemented inside Megatron-Core's own transformer submodules. SCO works at the autograd level: it intercepts the tensors saved by any checkpointed region and identifies the boundary by where it is stored, not by module or shape. So it applies to stacks outside Megatron-Core, such as our DeepSpeed ZeRO-3 / Hugging Face stack, and it composes with the nested checkpointing that PipelinedLLEP uses. Both are active in our 120B runs.
  • A byte budget fixed at launch. SCO chooses which of the 2L − 1 boundaries to offload so that a host byte budget is never exceeded. That makes the GPU-resident boundary term a configuration constant. Megatron exposes the fraction of eligible groups to offload rather than a byte budget.
  • Practical significance. In our stack, SCO is what lets 120B fit 65,536 tokens per rank at 128K and 1M, at a host cost of 11.4 GiB per rank (11.0 GiB measured). In the single-knob ablation below, removing SCO raises peak allocated memory by 9.6 GiB, and the run fails at 65,536 tokens per rank but completes at 32,768.

Single-knob ablation. At 120B / 128K full causal on 16 H200, we removed one mechanism at a time from the full stack. When a variant no longer fits 65,536 tokens per rank, we halve its tokens per rank until it fits.

ConfigurationFits 65,536 tokens/rank?Where it failsLargest tokens/rank that fitsPeak allocated / reserved (GiB)
Full stackyes—65,53697.4 / 132.9
Without SCOnoexpert-dispatch backward, after the second optimizer update (a CUDA error inside NCCL, at 107.0 GiB peak allocated)32,76887.1 / 132.0
Without token chunking (K = 1)noexpert dispatch, after the first optimizer update (7.7 GiB request)32,76898.2 / 110.6
Without Ring-DTP (dense head)nologits, first sub-step (24.4 GiB request)32,768112.2 / 133.2

OffloadStreamAdamW vs Deep Optimizer States (arXiv:2410.21316): which limitations of the prior approach it addresses.

The final revision cites and discusses Deep Optimizer States (DOS; Middleware '24). Both stage host-resident state through the GPU and overlap transfers. The specific differences:

  • Where the arithmetic runs. DOS splits the update between processors: every k-th subgroup goes to the GPU (k = 2 is its empirical optimum) and the rest are updated on the CPU. OffloadStreamAdamW routes every subgroup through the GPU, so no Adam arithmetic remains on the host. This matters when host compute is scarce relative to parameters. Our ranks hold 7.5–10.4B parameters each, with 12 physical cores per GPU (2 × 48-core sockets, 8 GPUs per node), and NVIDIA's Megatron Bridge measurements identify CPU-side Adam compute, not transfers, as the bottleneck of offloaded Adam.
  • A bounded GPU working set. The GPU staging footprint is s slots × the sub-group size, fixed at launch. This is the optimizer term in our peak bound.
  • Scale. DOS is evaluated up to 20B on one 4×H100 node. We run 120B–667B across 2–8 nodes.

Measured significance. To make the comparison fair, we strengthened our host-update reference: NUMA-pinned and multi-threaded. All rows are gpt-oss-20b on 8×H200, averaged over 10 updates:

Update pathTime per update (s)
Host AdamW, reference used in the submission12.680
Host AdamW, NUMA-pinned, 1 / 4 / 8 threads8.369 / 3.952 / 3.467
OffloadStreamAdamW (2 slots)1.938–1.959
  • Per update: against the best host configuration, OffloadStreamAdamW is 1.8× faster, and the final version uses this number.
  • End to end: at 120B / 128K, swapping it for a multi-threaded host AdamW raises the optimizer's per-step overhead from 20 s to 48 s (2.4×).

The two designs are complementary. DOS's CPU/GPU split could use our streamed path for its GPU-side subgroups, and the final version would discuss this combination as future work.

AuthorsOpenReview ↗

Official Response from the Authors (Cont.)

W2, baseline strength (pipeline parallelism, FP8)

W2: baseline tuning appears incomplete, so it is unclear whether the 10.4x throughput gain and the peak-memory reduction hold against a strong MoE baseline.

We have now tested FSDP2-best along every axis the reviewer lists, on 16 H200 at 120B:

  • pipeline parallelism, including DualPipeV;
  • FP8;
  • CPU optimizer offload;
  • fused linear cross-entropy.

Pipeline parallelism: whether a pipeline schedule such as DualPipeV would improve the baseline's communication overlap, and at what memory trade-off.

What the sweep already showed. It included intra-node expert parallelism. At 120B, EP = 8 layouts pass up to 32K and run out of memory from 64K on, because each GPU then holds twice the expert parameters.

What we added. Pipeline parallelism with PP = 2 halves the layers per GPU and lets EP = 8 stay within a node: each pipeline stage is one node, so the expert all-to-all never crosses nodes. We ran it at 120B / 128K as PP = 2 × (CP 4 · TP 2) with EP = 8 under three schedules: 1F1B, interleaved 1F1B and DualPipeV. These runs need CPU optimizer offload, because without it parameter and optimizer state alone come to about 130 GiB per GPU.

All three schedules exceed HBM at 128K, failing on the fp32 logits (a 12.3 GiB allocation).

The memory trade-off. A pipeline needs several sequences in flight to keep its bubble small, and at 128K a single sequence's activations already dominate. The DualPipe README also lists PP + 1 in-flight activation sets per device (against PP for 1F1B) and 2× parameters. TorchTitan runs DualPipeV through PyTorch's ScheduleDualPipeV, without DualPipe's overlap kernels. Gains reported elsewhere are modest:

  • Piper (2026): TorchTitan's DualPipeV gains +3% over TorchTitan's 1F1B on a 1B model, and its 9B run went out of memory.
  • Megatron-Core MoE report: plain 1F1B is faster than interleaving at large virtual-pipeline sizes.

FP8: whether FP8 would improve baseline throughput, and whether our memory and throughput benefits hold against an FP8-enabled baseline.

We ran FSDP2-best with FP8 (rowwise) at 120B / 128K:

  • With CPU offload, FP8 gives no measurable gain: 223 vs 226 tok/s/GPU.
  • Without offload, the FP8 run exceeds HBM, as the bf16 run nearly does.

Why. On H200, TorchTitan's FP8 path converts the dense linears: attention projections, the dense FFN and the shared experts. Its FP8 grouped GEMM for routed experts requires CP = 1, and context parallelism is required at 128K, so the experts stay in bf16. More fundamentally, FP8 changes GEMM precision, while what binds the baseline is not a GEMM term:

  • fp32 master weights and Adam moments take 75–82% of HBM before any activation;
  • fp32 logits are where every long-context baseline run fails.

FP8 recipes keep the output layer and optimizer in high precision, as the Megatron-Core MoE report describes. Even with full coverage, that report attributes about 10–25% end-to-end to FP8. Our mechanisms therefore retain their memory benefit against an FP8 baseline, and they compose with FP8.

AuthorsOpenReview ↗

Official Response from the Authors (Cont.)

W2, matched workloads, consistent FLOP accounting, and the stronger baselines

Configurations and MFU: selected parallelism configurations and MFU for both systems under matched workloads and hardware, with consistent FLOP accounting.

Consistent FLOP accounting. Our points counted causal (half) attention, while the baseline points used TorchTitan's formula, which counts full attention. We have recomputed both systems under TorchTitan's convention, 6·N_active + 12·L·h·T, which is the baseline's own formula, and we report causal-only counting alongside. Peak is 989 TFLOP/s.

Matched workload. Both sides now run full-causal attention over the whole sequence, on the same H200 nodes. These numbers replace the submission's headline ratios here and in the final version, including the 10.4×, whose two sides had different attention workloads (packed short documents against full-causal sequences).

ModelContextSystemtok/s/GPUTFLOP/s (MFU), TorchTitan conventionTFLOP/s (MFU), causal-only
120B128KOurs1,390148.0 (15.0%)93.7 (9.5%)
120B128KFSDP2-best24129.6 (3.0%)18.7 (1.9%)
120B1MOurs652429.7 (43.4%)224.1 (22.7%)
241B128KOurs856175.4 (17.7%)108.3 (11.0%)
241B1MOurs†324424.6 (42.9%)220.1 (22.3%)
240B32K (longest passing)FSDP2-best28625.8 (2.6%)20.1 (2.0%)
667B128KOurs308170.9 (17.3%)103.7 (10.5%)
667B1MOurs125455.4 (46.1%)235.2 (23.8%)

Stronger baselines at 120B. Each cell gives the best of three layouts (FSDP 1 · EP 16 with CP 8 · TP 2, CP 4 · TP 4, or CP 16), as tok/s/GPU and MFU under the TorchTitan convention.

Configuration128K1M
FSDP2-best241, 3.0%—
+ Liger fused linear CE (CP 8 · TP 2 layout)189, 2.3%—
+ CPU optimizer offload226, 2.8%OOM on all 3 layouts (fp32 logits, 49 GiB allocation)
+ FP8 + CPU offload223, 2.8%—
PP = 2 + EP = 8 + CPU offload (1F1B / interleaved / DualPipeV)OOM on all three—
Ours1,390, 15.0%652, 43.4%

Summary.

  • At matched length (120B / 128K), our stack reaches 5.0× the MFU of the FSDP2-best configuration under either FLOP convention, and 6.2× the tok/s/GPU of the offloaded baseline.
  • 1M. Every baseline layout we tested fails on the fp32 logits, the term Ring-DTP removes, while our stack trains 1M at 43–46% MFU at all three scales.
  • 240B. The baseline's longest passing length is 32K; every layout fails at 64K.
  • 671B. The 671B baseline did not train stably in our TorchTitan setup: gradient norms were non-finite at every length. We no longer report ratios against it, and present our 667B results on their own.

We also ran a tuned Megatron-Core on the same model and layout as FSDP2-best

At 1M, dropless Megatron-Core runs out of memory on the fp32 logits and on the expert-dispatch buffer, while our stack trains at 43% MFU

Results. 120B, 16 H200, 4.2M tokens per optimizer step. Each cell is tok/s/GPU · MFU.

ContextMegatron-Core (tuned)Our stackFSDP2-best
1Mfails: OOM on the fp32 logits (default cross-entropy); OOM in expert dispatch after 2 iterations (Transformer Engine cross-entropy)trains: 652 · 43.4% (full run)fails (OOM on the fp32 logits)
Reviewer_5Sw6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies peak memory usage in long-context Mixture-of-Experts (MoE) training and identifies four major sources of memory pressure: expert dispatch, vocabulary projection, gradient checkpoint boundaries, and optimizer states. To address these bottlenecks, the authors propose four optimizations:

PipelinedLLEP: Partitions expert dispatch into bounded token chunks and overlaps communication with computation, while using checkpointing to control memory usage during backward; Ring-DTP: Computes distributed vocabulary projection and cross-entropy through ring-based communication and online log-sum-exp, avoiding the materialization of full logits; Selective Checkpoint Offload (SCO): Offloads selected checkpoint boundaries to CPU memory and asynchronously prefetches them during backward; OffloadStreamAdamW: Streams CPU-resident optimizer states through the GPU in buckets to accelerate AdamW updates.

The authors integrate the four optimizations into a distributed training system and evaluate it on MoE models ranging from 120B to 667B parameters. Compared with a tuned FSDP2 baseline, the proposed system supports context lengths of up to 1M tokens, achieving 8–32× longer context reach and up to 10.4× higher training throughput.

Critical strengths

The paper works on a practical problem with a systematic analysis. The authors examine multiple sources of peak memory usage across the training stack, rather than focusing on a single operator. The proposed optimizations target different memory bottlenecks and can be composed to improve the overall training capacity.

The paper provides careful discussions of implementation details, including autograd tensor lifetimes, nested checkpointing, communication-computation overlap, and optimizer streaming. The experiments cover end-to-end training at substantial model scales, with promising improvements in memory usage, context length, and throughput.

Critical weaknesses

It could be helpful to provide more breakdown of end-to-end performance improvements.

  • While the paper provides isolated benchmarks for the four proposed optimizations, the latency improvements of individual components appear relatively modest, with some introducing additional overhead. In contrast, the end-to-end experiments report throughput improvements of up to 10.4×. It would be helpful to better understand where these substantial gains come from and how much each optimization contributes to the overall improvement.
  • An end-to-end ablation showing the contribution of each component to memory savings and throughput, along with a breakdown of computation, communication, and optimizer time, would help clarify these results. MFU or related utilization metrics would also be useful.
Questions for authors
  • Could the authors provide more details about the parallelism configurations used by the FSDP2 baseline and the proposed system in the end-to-end experiments (e.g., tensor, context, and expert parallelism degrees)?
  • How does Ring-DTP compare with existing approaches such as Cut Cross-Entropy, Liger Kernel, and Megatron's vocabulary-parallel cross-entropy? I understand that these methods may not directly support the distinct-batch, sharded-vocabulary setting targeted by Ring-DTP, but it would be helpful to clarify their limitations in this setting and the additional benefits of Ring-DTP.
5 comments
AuthorsOpenReview ↗

Official Response from the Authors

Where the end-to-end gain comes from, and what each mechanism contributes

End-to-end gain: where the up-to-10.4× throughput gain comes from when the isolated component gains look modest, and how much each optimization contributes.

First, the size of the gain. The submission's 10.4× compared runs with different attention workloads: packed short documents on our side, full-causal sequences on the baseline's. We have recomputed both systems under one FLOP convention, at a matched full-causal workload. At 120B / 128K our stack reaches 5.8× the baseline's tok/s/GPU and 5.0× its MFU (15.0% vs 3.0%). The full table is in the post below table (d), and the final version uses these numbers.

The four mechanisms are primarily memory mechanisms, which allows fitting more tokens and weights into the same number of GPUs. This, along with leveraging computation-communication overlap, delivers higher speed.

120B, 128K, 16 H200FSDP2-bestOurs
LayoutFSDP 1 · EP 16 · CP 8 · TP 2ZeRO-3 + EP 16, sequence sub-group D = 2
Ranks each sequence is split across16 (CP 8 × TP 2)2
Tokens per GPU per step8,19265,536
Time per step (s)33.542.5 forward/backward; 57.6 with the optimizer update
tok/s/GPU2411,390
  • Each GPU does 8× the work per step in 1.3× the time.
  • The baseline cannot raise its tokens per GPU. fp32 optimizer state (115 GiB) and fp32 logits already fill its HBM, so it splits each sequence 16 ways and pays context- and tensor-parallel communication on every layer.
  • The gain is not optimizer amortisation. On the sub-steps that carry the optimizer update, we still run at 1,000–1,130 tok/s/GPU.

What each mechanism contributes: end-to-end ablation. At 120B / 128K full causal on 16 H200, we removed one mechanism at a time from the full stack. When a variant no longer fits 65,536 tokens per rank, we halve its tokens per rank until it fits.

ConfigurationFits 65,536 tokens/rank?Where it failsLargest tokens/rank that fitsPeak allocated / reserved (GiB)
Full stackyes—65,53697.4 / 132.9
Without SCOnoexpert-dispatch backward, after the second optimizer update (a CUDA error inside NCCL, at 107.0 GiB peak allocated)32,76887.1 / 132.0
Without token chunking (K = 1)noexpert dispatch, after the first optimizer update (7.7 GiB request)32,76898.2 / 110.6
Without Ring-DTP (dense head)nologits, first sub-step (24.4 GiB request)32,768112.2 / 133.2
  • Chunking and Ring-DTP. Without either one the run exceeds HBM, and the tokens per rank must halve. That is the memory each of them converts into per-GPU work.
  • OffloadStreamAdamW. It cuts the optimizer's per-step overhead from 48 s to 20 s (2.4×) against a multi-threaded host AdamW, at the cost of a bounded GPU staging buffer of about 21 GiB.
  • SCO. Without it, peak allocated memory rises by 9.6 GiB (97.4 → 107.0) and the run fails at 65,536 tokens per rank; it completes at 32,768. The boundaries SCO moves to the host are what let 120B hold 65,536 tokens per rank.
AuthorsOpenReview ↗

Official Response from the Authors (Cont.)

Memory budget, time breakdown and MFU

Ablation and breakdown: an end-to-end ablation of each component's memory and throughput contribution, a breakdown of computation, communication and optimizer time, and MFU.

(a) Which mechanisms each reported row engages. These are read from the run logs, and the final version adds them as a new table.

Row (full causal)Ring-DTP group PToken chunks KSCO boundaries offloadedOptimizer state on GPUPeak HBM (GiB)
120B, 128K81013 / 1325%131.5
120B, 1M81013 / 1325%139.5
241B, 128K / 1M810 / 270%107.0 / 104.8
667B, 128K / 1M810 / 770%119.3 / 120.8

At 241B and 667B the larger expert-parallel world halves the tokens per rank to 32,768. Boundaries and dispatch then fit in HBM, so those rows run with SCO off and K = 1. The final version states this explicitly.

(b) Per-term memory budget, measured on the allocator (120B / 128K, maximum over 16 ranks):

TermGiB
Static: weights, gradients, optimizer staging, GPU-resident optimizer share (predicted from configuration: 60.9)61.1
Transient working set: recomputation, dispatch chunks, logit strips, two in-flight boundaries36.3
Peak allocated97.4
Allocator cache (reserved − allocated)35.5
Peak reserved (≈ the nvidia-smi figure reported in the paper)132.9

The static terms match their configuration-level prediction to within 0.2 GiB. Most of the gap between the tensor peak (97.4 GiB) and the reported nvidia-smi peak is cached, unallocated memory in PyTorch's allocator.

For the baseline, the binding term is static fp32 state plus fp32 logits:

Baseline rowStatic fp32 state (GiB)Peak reserved (GiB)Outcome
120B, 64K115.0135.1the 32K layout fails on a 12.27 GiB fp32 log-softmax backward allocation
120B, 128K114.9135.7passes at the limit (allocator retries rise from 4 to 37)
240B, 64K≈105—every layout fails, twice on fp32 logits (24.55 GiB)

(c) Time breakdown. Our per-sub-step timers, with the optimizer update on every 4th sub-step:

RowForward/backward sub-step (s)Optimizer sub-step (s)Optimizer cost per step (s)Tokens per optimizer step
120B, 128K42.557.615.14.17M
120B, 1M96.5112.415.84.19M
241B, 128K32.255.323.14.17M
667B, 128K99.3125.626.38.34M
667B, 1M254.7281.426.78.38M

FSDP2-best, profiled (120B / 128K, one step, rank 0):

  • GPU window: 35.5 s.
  • Compute: 22.6 s (64%).
  • NCCL: 21.7 s busy, of which 10.8 s (30%) does not overlap compute. Most of it is ring-attention send/receive (13.6 s) and all-gather (5.6 s).
  • Optimizer: 0.06 s, since AdamW runs on the GPU.

Our own timers do not separate communication from compute. We will add a profiled breakdown of our stack in the final version.

(d) MFU. Under TorchTitan's FLOP convention for both systems, at a matched full-causal workload:

  • At 120B / 128K: 15.0% for ours, 3.0% for FSDP2-best.
  • At 1M: 43.4%, 42.9% and 46.1% at 120B, 241B and 667B.
  • Causal-only counting: 9.5% vs 1.9% at 128K, and 22–24% at 1M.

The full table is below.

ModelContextSystemtok/s/GPUTFLOP/s (MFU), TorchTitan conventionTFLOP/s (MFU), causal-only
120B128KOurs1,390148.0 (15.0%)93.7 (9.5%)
120B128KFSDP2-best24129.6 (3.0%)18.7 (1.9%)
120B1MOurs652429.7 (43.4%)224.1 (22.7%)
241B128KOurs856175.4 (17.7%)108.3 (11.0%)
241B1MOurs†324424.6 (42.9%)220.1 (22.3%)
240B32K (longest passing)FSDP2-best28625.8 (2.6%)20.1 (2.0%)
667B128KOurs308170.9 (17.3%)103.7 (10.5%)
667B1MOurs125455.4 (46.1%)235.2 (23.8%)

† Measured over the first six sub-steps.

AuthorsOpenReview ↗

Official Response from the Authors (Cont.)

Q1, parallelism configurations

Q1: parallelism configurations (tensor, context and expert parallelism degrees) of the FSDP2 baseline and our system in the end-to-end experiments.

The final version adds both tables in a new appendix.

Ours. Settings shared by every row:

  • ZeRO-3 over non-expert parameters, and expert parallelism over the full world (E_p = W).
  • No tensor parallelism in the transformer layers; Ring-DTP shards only the LM head. Each sequence is split over a sub-group of D ranks.
  • Full activation checkpointing.
  • Micro-batch 1, gradient accumulation 4, K_max = 10.
  • Optimizer state in pinned host memory, updated by OffloadStreamAdamW.
RowGPUs (nodes)DE_pPChunk budget c (K)Tokens per rankSCO budget (boundaries)Optimizer on GPUSub-group / slots
120B, 128K16 (2)21686,600 (10)65,53616 GiB (13/13)25%5e7 / 2
120B, 1M16 (2)161686,600 (10)65,53616 GiB (13/13)25%5e7 / 2
241B, 128K32 (4)432833,000 (1)32,768off (0/27)0%1e8 / 3
241B, 1M32 (4)3232833,000 (1)32,768off (0/27)0%1e8 / 3
667B, 128K64 (8)464833,000 (1)32,768off (0/77)0%1e8 / 3
667B, 1M64 (8)3264833,000 (1)32,768off (0/77)0%1e8 / 3

FSDP2-best (TorchTitan). Settings shared by every row:

  • Full activation checkpointing; no compile.
  • SDPA with a causal mask; PP = 1 and local batch 1 unless stated.
  • FSDP2-best is the highest-throughput passing layout from a sweep over every FSDP × CP × TP = 16, with TP ∈ {1, 2, 4}, CP ∈ {1 … 16} and EP ∈ {1, 2, 4, 8, 16}. That is 42 layouts at 120B; 240B used 3 layouts on 32 GPUs.
Model, contextFSDPEPCPTPPeak reserved (GiB)
120B, 32K41614136.6
120B, 64K11644135.1
120B, 128K11682135.7
240B, 32K (longest passing)8822130.5

Intra-node expert parallelism (EP = 8) passes at 120B up to 32K and runs out of memory from 64K on, so every passing layout at 64K and above uses EP = 16.

The 671B configuration did not train stably in our TorchTitan setup: gradient norms were non-finite at every length. We therefore report our 667B results on their own, not as ratios against it, here and in the final version.

We also evaluated stronger FSDP2-best variants: CPU optimizer offload (128K–1M), fused linear CE, gradient accumulation, FP8, and pipeline parallelism including DualPipeV. The results are in the table below. The final version adds their configurations in the new appendix.

Stronger baselines at 120B. Each cell gives the best of three layouts (FSDP 1 · EP 16 with CP 8 · TP 2, CP 4 · TP 4, or CP 16), as tok/s/GPU and MFU under the TorchTitan convention.

Configuration128K1M
FSDP2-best241, 3.0%—
+ Liger fused linear CE (CP 8 · TP 2 layout)189, 2.3%—
+ gradient accumulation to 4.2M tokens per optimizer step (CP 8 · TP 2 layout)OOM at step 2, once fp32 Adam state is resident—
+ CPU optimizer offload226, 2.8%OOM on all 3 layouts (fp32 logits, 49 GiB allocation)
+ FP8 + CPU offload223, 2.8%—
PP = 2 + EP = 8 + CPU offload (1F1B / interleaved / DualPipeV)OOM on all three—
Ours1,390, 15.0%652, 43.4%

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers and the AC

We thank all three reviewers for detailed and constructive reviews. In response we ran the experiments they asked for. Below we summarise them, together with the changes we will make in the final version; each point is answered in full in the individual responses.

  1. One FLOP convention, matched workload (VFpt, 5Sw6). In this response both systems are reported under TorchTitan's convention (6·N_active + 12·L·h·T, the baseline's own), with causal-only counting as a check, at the same full-causal attention workload. The final version replaces the submission's headline ratios, including the 10.4×, with these.
    • At 120B / 128K our stack reaches 5.0× the baseline's MFU (15.0% vs 3.0%) and 5.8× its tok/s/GPU.
    • At 1M context it trains 120B, 241B and 667B at 43–46% MFU.
  2. Stronger baselines (VFpt, bpfb). We re-ran FSDP2-best at 120B with every addition the reviewers listed and also Megatron-Core:
    • Megatron-Core (bpfb): tuned. At 1M it runs out of memory on the fp32 logits and the expert-dispatch buffer, while our stack trains at 43% MFU.
    • FP8: no measurable gain (223 vs 226 tok/s/GPU).
    • Fused linear cross-entropy (Liger): it removes the logit allocation but not the peak, which is set by fp32 optimizer state.
    • Pipeline parallelism with intra-node EP (1F1B, interleaved, DualPipeV): every schedule exceeds HBM at 128K.
    • At 1M every baseline layout fails on the fp32 logits that Ring-DTP removes. At 128K our stack remains 6.2× faster than the offloaded baseline.
    • Gradient accumulation cannot hold our 4.2M-token batch without offload, with or without fused CE: it runs out of memory at step 2, once fp32 Adam state is resident.
  3. Where the gain comes from, and an end-to-end ablation (5Sw6).
    • Mechanism. The gain comes from converting memory into per-GPU work: 8× more tokens per GPU per step, with each sequence split across 2 ranks instead of 16.
    • Ablation. Removing token chunking or Ring-DTP makes the 65,536-tokens-per-rank configuration exceed HBM, and OffloadStreamAdamW cuts the optimizer's per-step overhead 2.4×.
    • Memory. An allocator-level budget puts the tensor peak at 97.4 GiB, and the static terms match their prediction to within 0.2 GiB.
    • SCO. Without it, peak allocated memory rises by 9.6 GiB and the 65,536-tokens-per-rank run fails; it completes only at half the tokens per rank.
  4. Extended Table 2 (bpfb, 5Sw6): CCE, Liger, chunked CE and vocabulary-parallel CE, at 16K and 32K tokens per rank.
    • Ring-DTP has the lowest peak up to 32K tokens per rank, and no method beats it on both memory and time.
  5. Full configurations (5Sw6, VFpt). Per-row parallelism, offload and checkpointing settings for both systems.
  6. Positioning and related work (VFpt, bpfb).
    • The final version discusses Megatron-Core fine-grained activation offloading, Deep Optimizer States, the Megatron-Core MoE report and DualPipe/DualPipeV.
    • The optimizer is now compared against a strengthened, multi-threaded host AdamW (1.8× per update).
  7. Code (bpfb). We have uploaded the code for each of the four mechanisms (Ring-DTP, token-chunked expert dispatch, SCO and OffloadStreamAdamW) as supplementary material, each with a correctness test against a plain reference implementation; the release of the full training stack remains pending institutional approval.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.