- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes MeCeFO-v2, a fault-tolerant training framework for Mixture-of-Experts (MoE) models under hybrid data, pipeline, and expert parallelism. Following GPU failures, the framework distributes failed expert workloads across surviving expert-parallel workers rather than concentrating recovery on a single worker. It introduces router-aware low-rank expert-gradient approximation, and an adaptive controller for balancing gradient fidelity against memory and computation constraints.
Critical strengths
-The contribution is valid and the paper addresses an important challenge in large-scale MoE training through cooperative redistribution of failed expert workloads across expert parallel workers.
-The evaluation on a 128-GPU cluster with three MoE model and multiple failure scenarios demonstrates significant throughput improvements.
Critical weaknesses
-The theoretical analysis considers momentum SGD, whereas the implementation uses Adam. This mismatch weakens the claimed convergence guarantees.
-The convergence analysis relies on bounded gradient-error assumptions that are not rigorously established for the proposed attention-head selection and router-aware low-rank compression.
Questions for authors
-Can the authors derive explicit bounds relating gradient approximation error to the attention-head budget, expert compression ratio, and low-rank projection? How are the convergence guarantees applicable to the Adam optimizer used in practice rather than the momentum SGD considered theoretically?
-How do the reported throughput gains translate into actual wall-clock training time and total GPU-hours required to reach the same target perplexity? Can the authors demonstrate these gains under larger EP and correlated multi-node failures?
9 comments
W2 + Q1 (first part): Explicit bounds relating the error to the head budget, the compression ratio, and the projection rank
Relation to the known theory of Adam. Adam's convergence is delicate even with exact gradients:
- Reddi et al. (2018) give counterexamples for a fixed and propose AMSGrad.
- Chen et al. (2019) show convergence when the variation of the effective step size is controlled.
- Défossez et al. (2022) and Hong & Lin (2024) prove convergence for close to 1.
- Li, Rakhlin & Jadbabaie (2023) relax the smoothness assumption.
Theorem 2 has the same structure as the exact-gradient bound of Chen et al., and MeCeFO-v2 changes only its constants. MeCeFO-v2 therefore converges in every regime where these terms are controlled for exact Adam, and the worst cases are those of Adam itself.
Worst case versus practice. The theorems need only , and training is far from that limit. We measured the ratios along AdamW protection episodes of the 16B run (Table R2-4 in our response to Reviewer 2):
- the single-batch error is at most 0.161;
- the bias is at most 0.096 in the Euclidean norm and at most 0.103 in Adam's metric.
At , the leading constant of Theorem 1 is at most 1.4 times that of exact momentum SGD. The measured effect on the final model is +0.24% to +1.89% PPL.
W2 + Q1 (first part): Explicit bounds relating the error to the head budget, the compression ratio, and the projection rank
These bounds make Assumption 3 concrete: they show where the error can travel and how it depends on .
1. Error propagation under an exact forward pass (Theorem 3). MeCeFO-v2 keeps the forward pass exact, so all activations, the loss, and the gradient at the network output are exact. Only backward products on protected ranks change.
Setup.
- Write the network as blocks , namely attention or MoE sublayers including normalization and the residual connection.
- Let and be the exact Jacobians.
- The protected backward pass is and , with , and with on unprotected blocks.
- With , , , and ,
Proof. The adjoint error satisfies , starting from zero at the output. Unrolling this recursion and using gives the bound. ∎
Consequences.
- MoE blocks do not propagate error. On protected ranks, the Dgrad and router backward passes are exact, so . Expert compression changes only the weight gradients of the compressed experts.
- Attention errors travel only to earlier layers of the same replica. Blocks above the highest protected attention block receive exact adjoints, and the residual path is never approximated.
- DP averaging divides the error by . Replicas without protected ranks are exact.
2. Head budget (Proposition 4).
Setup. Consider a protected attention block with selection units (heads, or KV groups under GQA), upstream gradient , and retained set with . Let and be the input and weight Jacobian norms of unit . Then
Explicit Jacobian norms. The norms are explicit in weights and activations. The softmax Jacobian has norm at most 1/2, so RoPE leaves these norms unchanged, and normalization layers contribute their Jacobian norms as factors.
Dependence on . The bound is zero at and grows by one unit's contribution per discarded unit. If a unit's backward contribution scales with its value-output energy (the trend in Fig. 3a), the bound is proportional to . Our top- rule minimizes exactly this sum.
3. Compression budget and rank (Proposition 5).
Setup. For one expert linear sublayer and one sequence , the exact weight gradient is , with one rank-one term per token–expert assignment:
- is the cached input;
- is the gate-free output gradient;
- is the normalized gate, where is the top- mass of the router probabilities over experts.
Compressing a set of assignments replaces by , with . With and , because the top- of probabilities sum to at least .
Consequences.
- Selection. Our sequence-level rule (Eq. 1) compresses the assignments with the smallest scores, so it minimizes exactly.
- Budget. is zero at , nondecreasing, and convex in . This explains why moderate compression is almost free.
- Rank. is nonincreasing in and zero at full rank.
- Refresh. If comes from the reference gradient of the last refresh, then for the current gradient and every , . The first term is the spectral tail of the reference, and the second is the drift since the refresh, which controls.
Combined bound (Corollary 6). For a replica with protected MoE blocks and protected attention blocks , where is the amplification of a defect injected at block . DP averaging then divides the total by .
4. Why we also measure the coefficients. The constants are products of Jacobian norms across depth, and Kong et al. (2026) show that such coefficients can grow exponentially with the number of operators.
- What the bounds establish. They give the structure of Assumption 3, namely how the error depends on and where it can travel. They cannot certify a useful numerical for a 24-layer model in advance.
- How MeCeFO-v2 uses them. The bounds decide what to keep, since our selection rules minimize them. Direct measurement decides how much to drop: the controller measures the error on held-out microbatches, and Figs. 5–6 and Table R2-4 verify Assumption 3 along the actual runs.
- Why an exact forward pass matters. Kong et al. also show that forward-pass perturbations are the main obstacle to convergence, because they bias the gradient through the nonlinearities. MeCeFO-v2 avoids them by keeping the forward pass exact.
W1 + Q1 (second part): Convergence for the Adam optimizer used in practice
Thank you for the focused review. Both theory questions are now answered by new results. Theorem 2 proves convergence for Adam and AdamW, and Theorem 3 with Propositions 4–5 gives explicit error bounds in terms of the head budget, the compression ratio, and the projection rank. We also added the end-to-end and scaling experiments you asked for. Full proofs are in the new Appendix A.
Notation.
W1 + Q1 (second part): Convergence for the Adam optimizer used in practice
Why the paper analyzed momentum SGD.
We have now proved the Adam/AdamW case directly and also strengthened the momentum-SGD result.
Theorem 1 (revised; momentum SGD with any momentum). Let Cβ:=2(1−β1)1+β1. For every β1∈[0,1) and η≤2LCβ(1+ρ)21−ρ, T1∑t=0T−1E∥∇f(wt)∥2≤(1−ρ)ηT2Δ+(1−ρ)n2LCβ(1+ρ)2ησ2. With the best step size, this gives 1−ρ4(1+ρ)LCβΔσ2/(nT)+O(1/T). The theorem covers the β1=0.9 we train with.
Theorem 2 (new; Adam with bias correction, and AdamW). We write PyTorch's Adam as mt=β1mt−1+(1−β1)ht and xt+1=xt−Λtmt, with Λt=1−β1tαtdiag(1/(v^t+ϵ)). Here ht is the gradient fed to the optimizer (ht=g~t for MeCeFO-v2). Assume:
Then, with c:=1−β1β1 and Δ:=f(x1)−inff, ∑t=1TE∥∇f(xt)∥Λt−12≤1−ρ2[Δ+L(2c2AT+2AT′)+(1−β1)2Gc2VT], where:
Proof idea. With the virtual iterate zt=xt+c(xt−xt−1), one Adam step satisfies the exact identity zt+1−zt=−Λt−1ht+rt, with rt=−(Λt−Λt−1)(ht+cmt−1). Because Λt−1 is known before ht is drawn, the bias condition gives a descent of (1−ρ)∥∇f(xt)∥Λt−12. The remainder rt involves only the change of the preconditioner and produces VT.
Consequences.