acceptodds
Under review as a conference paper at ICLR 2027

MeCeFO-v2: Efficient Fault-Tolerant Optimization for MoE Training

Abstract

Sparse Mixture-of-Experts (MoE) models have become a central architecture for scaling large language models, but they also reshape the fault-tolerance problem during distributed training. Compared with dense models, MoE training couples hardware failures with expert parallelism, routing-dependent workload imbalance, and substantially enlarged training state, making conventional failover strategies inefficient. Existing MoE fault-tolerant methods primarily optimize checkpointing or elastic recovery, but leave open whether the training computation itself can be adapted online after failures with bounded memory and compute overhead. In this work, we propose MeCeFO-v2, a fault-tolerant optimization framework for MoE training under hybrid pipeline, expert, and data parallelism. Instead of assigning a failed worker's workload to a single pipeline neighbor, MeCeFO-v2 redistributes it cooperatively across expert-parallel peers, improving expert update balance and reducing the approximation burden on each surrogate worker. To keep failover training within resource budget, MeCeFO-v2 introduces two structure-aware approximations: norm-guided selective head retention for attention layers and router-aware sequence-level low-rank compression for MoE experts. We further develop an online controller that automatically selects the attention-head budget and expert compression ratio by minimizing profiled gradient error subject to memory and compute constraints. Theoretically, we establish the same convergence rate as standard distributed SGD under bounded gradient error. Experiments on large-scale MoE training demonstrate that MeCeFO-v2 maintains validation perplexity and downstream performance close to fault-free training while limiting throughput degradation to 0.72–6.15%, compared with 7.61–57.92% for prior MoE fault-tolerance baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-Experts (MoE), fault-tolerant training, distributed training, expert parallelism, adaptive failover, structure-aware approximation, low-rank compression, resource-aware optimization.

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_xDkTOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MeCeFO-v2, a fault-tolerant training framework for Mixture-of-Experts (MoE) models under hybrid data, pipeline, and expert parallelism. Following GPU failures, the framework distributes failed expert workloads across surviving expert-parallel workers rather than concentrating recovery on a single worker. It introduces router-aware low-rank expert-gradient approximation, and an adaptive controller for balancing gradient fidelity against memory and computation constraints.

Critical strengths

-The contribution is valid and the paper addresses an important challenge in large-scale MoE training through cooperative redistribution of failed expert workloads across expert parallel workers.

-The evaluation on a 128-GPU cluster with three MoE model and multiple failure scenarios demonstrates significant throughput improvements.

Critical weaknesses

-The theoretical analysis considers momentum SGD, whereas the implementation uses Adam. This mismatch weakens the claimed convergence guarantees.

-The convergence analysis relies on bounded gradient-error assumptions that are not rigorously established for the proposed attention-head selection and router-aware low-rank compression.

Questions for authors

-Can the authors derive explicit bounds relating gradient approximation error to the attention-head budget, expert compression ratio, and low-rank projection? How are the convergence guarantees applicable to the Adam optimizer used in practice rather than the momentum SGD considered theoretically?

-How do the reported throughput gains translate into actual wall-clock training time and total GPU-hours required to reach the same target perplexity? Can the authors demonstrate these gains under larger EP and correlated multi-node failures?

9 comments
AuthorsOpenReview ↗

W1 + Q1 (second part): Convergence for the Adam optimizer used in practice

Thank you for the focused review. Both theory questions are now answered by new results. Theorem 2 proves convergence for Adam and AdamW, and Theorem 3 with Propositions 4–5 gives explicit error bounds in terms of the head budget, the compression ratio, and the projection rank. We also added the end-to-end and scaling experiments you asked for. Full proofs are in the new Appendix A.

Notation.

  • gtg_t is the exact minibatch gradient, and g~t\tilde g_t is the gradient computed by MeCeFO-v2.
  • ρ:=1−δ‾\rho:=\sqrt{1-\underline\delta}, so Assumption 3 reads ∥g~t−gt∥≤ρ∥gt∥\|\tilde g_t-g_t\|\le\rho\|g_t\| and ∥Etg~t−∇f(wt)∥≤ρ∥∇f(wt)∥\|\mathbb E_t\tilde g_t-\nabla f(w_t)\|\le\rho\|\nabla f(w_t)\|.

W1 + Q1 (second part): Convergence for the Adam optimizer used in practice

Why the paper analyzed momentum SGD.

  • The role of the theory. The paper's main contribution is the failover system. The theory is there to show that the approximation does not change the convergence order of the optimizer.
  • Why momentum SGD. The approximation reaches the optimizer only through the gradient it receives, and momentum SGD isolates this effect cleanly.
  • A standard route. Treating Adam's preconditioner as a bounded coordinate-wise scaling of momentum SGD is common in the literature (Guo et al., 2021; Chen et al., 2019; Tang et al., 2021).

We have now proved the Adam/AdamW case directly and also strengthened the momentum-SGD result.

Theorem 1 (revised; momentum SGD with any momentum). Let Cβ:=1+β12(1−β1)C_\beta:=\frac{1+\beta_1}{2(1-\beta_1)}. For every β1∈[0,1)\beta_1\in[0,1) and η≤1−ρ2LCβ(1+ρ)2\eta\le\frac{1-\rho}{2LC_\beta(1+\rho)^2}, 1T∑t=0T−1E∥∇f(wt)∥2≤2Δ(1−ρ)ηT+2LCβ(1+ρ)2ησ2(1−ρ)n.\frac1T\sum_{t=0}^{T-1}\mathbb E\|\nabla f(w_t)\|^2\le\frac{2\Delta}{(1-\rho)\eta T}+\frac{2LC_\beta(1+\rho)^2\eta\sigma^2}{(1-\rho)n}. With the best step size, this gives 4(1+ρ)1−ρLCβΔσ2/(nT)+O(1/T)\frac{4(1+\rho)}{1-\rho}\sqrt{LC_\beta\Delta\sigma^2/(nT)}+O(1/T). The theorem covers the β1=0.9\beta_1=0.9 we train with.

Theorem 2 (new; Adam with bias correction, and AdamW). We write PyTorch's Adam as mt=β1mt−1+(1−β1)htm_t=\beta_1m_{t-1}+(1-\beta_1)h_t and xt+1=xt−Λtmtx_{t+1}=x_t-\Lambda_tm_t, with Λt=αt1−β1tdiag(1/(v^t+ϵ))\Lambda_t=\frac{\alpha_t}{1-\beta_1^t}\mathrm{diag}\big(1/(\sqrt{\hat v_t}+\epsilon)\big). Here hth_t is the gradient fed to the optimizer (ht=g~th_t=\tilde g_t for MeCeFO-v2). Assume:

  • LL-smoothness;
  • bounded fed gradients ∥ht∥≤Gc\|h_t\|\le G_c (clipping at 1.0 gives Gc=1G_c=1);
  • the bias part of Assumption 3 in Adam's metric, ∥Etht−∇f(xt)∥Λt−1≤ρ∥∇f(xt)∥Λt−1\|\mathbb E_th_t-\nabla f(x_t)\|_{\Lambda_{t-1}}\le\rho\|\nabla f(x_t)\|_{\Lambda_{t-1}}.

Then, with c:=β11−β1c:=\frac{\beta_1}{1-\beta_1} and Δ:=f(x1)−inf⁡f\Delta:=f(x_1)-\inf f, ∑t=1TE∥∇f(xt)∥Λt−12≤21−ρ[Δ+L(c22AT+2AT′)+Gc2(1−β1)2VT],\sum_{t=1}^{T}\mathbb E\|\nabla f(x_t)\|^2_{\Lambda_{t-1}}\le\frac{2}{1-\rho}\Big[\Delta+L\Big(\tfrac{c^2}{2}\mathcal A_T+2\mathcal A'_T\Big)+\frac{G_c^2}{(1-\beta_1)^2}\mathcal V_T\Big], where:

  • AT=E∑t∥xt−xt−1∥2\mathcal A_T=\mathbb E\sum_t\|x_t-x_{t-1}\|^2 and AT′=E∑t∥Λt−1ht∥2\mathcal A'_T=\mathbb E\sum_t\|\Lambda_{t-1}h_t\|^2 measure the size of the updates;
  • VT=E∑t(2L+12(1−ρ)λmin⁡(Λt−1))∥Λt−Λt−1∥2\mathcal V_T=\mathbb E\sum_t\big(2L+\frac{1}{2(1-\rho)\lambda_{\min}(\Lambda_{t-1})}\big)\|\Lambda_t-\Lambda_{t-1}\|^2 measures how fast the effective learning rates change.

Proof idea. With the virtual iterate zt=xt+c(xt−xt−1)z_t=x_t+c(x_t-x_{t-1}), one Adam step satisfies the exact identity zt+1−zt=−Λt−1ht+rtz_{t+1}-z_t=-\Lambda_{t-1}h_t+r_t, with rt=−(Λt−Λt−1)(ht+cmt−1)r_t=-(\Lambda_t-\Lambda_{t-1})(h_t+cm_{t-1}). Because Λt−1\Lambda_{t-1} is known before hth_t is drawn, the bias condition gives a descent of (1−ρ)∥∇f(xt)∥Λt−12(1-\rho)\|\nabla f(x_t)\|^2_{\Lambda_{t-1}}. The remainder rtr_t involves only the change of the preconditioner and produces VT\mathcal V_T.

Consequences.

  • Bounded effective step sizes and a constant base learning rate give O(1/nT)O(1/\sqrt{nT}) plus the average change of the preconditioner. That change is O(d/T)O(d/T) for AMSGrad-type second moments and O(log⁡T/T)O(\log T/T) for Adam with β2=1−1/T\beta_2=1-1/T.
  • With a stabilized second moment (Λt=αD\Lambda_t=\alpha D), Adam is exactly momentum SGD in the coordinates y=D−1/2xy=D^{-1/2}x, and Theorem 1 applies verbatim.
  • For AdamW, the decoupled decay depends only on the parameters, so the approximation does not change it. With bounded iterates, the decay adds only the usual O(λ2R2)O(\lambda^2R^2) term, which exact AdamW has as well.
AuthorsOpenReview ↗

W2 + Q1 (first part): Explicit bounds relating the error to the head budget, the compression ratio, and the projection rank

Relation to the known theory of Adam. Adam's convergence is delicate even with exact gradients:

  • Reddi et al. (2018) give counterexamples for a fixed β2\beta_2 and propose AMSGrad.
  • Chen et al. (2019) show convergence when the variation of the effective step size is controlled.
  • Défossez et al. (2022) and Hong & Lin (2024) prove convergence for β2\beta_2 close to 1.
  • Li, Rakhlin & Jadbabaie (2023) relax the smoothness assumption.

Theorem 2 has the same structure as the exact-gradient bound of Chen et al., and MeCeFO-v2 changes only its constants. MeCeFO-v2 therefore converges in every regime where these terms are controlled for exact Adam, and the worst cases are those of Adam itself.

Worst case versus practice. The theorems need only ρ<1\rho<1, and training is far from that limit. We measured the ratios along AdamW protection episodes of the 16B run (Table R2-4 in our response to Reviewer 2):

  • the single-batch error is at most 0.161;
  • the bias is at most 0.096 in the Euclidean norm and at most 0.103 in Adam's metric.

At ρ=0.161\rho=0.161, the leading constant of Theorem 1 is at most 1.4 times that of exact momentum SGD. The measured effect on the final model is +0.24% to +1.89% PPL.

W2 + Q1 (first part): Explicit bounds relating the error to the head budget, the compression ratio, and the projection rank

These bounds make Assumption 3 concrete: they show where the error can travel and how it depends on (κ,χ,r)(\kappa,\chi,r).

1. Error propagation under an exact forward pass (Theorem 3). MeCeFO-v2 keeps the forward pass exact, so all activations, the loss, and the gradient at the network output are exact. Only backward products on protected ranks change.

Setup.

  • Write the network as blocks xℓ=Fℓ(xℓ−1;wℓ)x_\ell=F_\ell(x_{\ell-1};w_\ell), namely attention or MoE sublayers including normalization and the residual connection.
  • Let Aℓ=∂Fℓ/∂xℓ−1A_\ell=\partial F_\ell/\partial x_{\ell-1} and Bℓ=∂Fℓ/∂wℓB_\ell=\partial F_\ell/\partial w_\ell be the exact Jacobians.
  • The protected backward pass is h~ℓ−1=(Aℓ+Eℓ)⊤h~ℓ\tilde h_{\ell-1}=(A_\ell+E_\ell)^\top\tilde h_\ell and g~ℓ=(Bℓ+Dℓ)⊤h~ℓ\tilde g_\ell=(B_\ell+D_\ell)^\top\tilde h_\ell, with h~L=hL\tilde h_L=h_L, and with Eℓ=Dℓ=0E_\ell=D_\ell=0 on unprotected blocks.
  • With aℓ=∥Aℓ∥a_\ell=\|A_\ell\|, bℓ=∥Bℓ∥b_\ell=\|B_\ell\|, cℓ=∥Eℓ⊤h~ℓ∥c_\ell=\|E_\ell^\top\tilde h_\ell\|, and dℓ=∥Dℓ⊤h~ℓ∥d_\ell=\|D_\ell^\top\tilde h_\ell\|, ∥h~ℓ−hℓ∥≤∑k>ℓ(∏j=ℓ+1k−1aj)ck,∥g~ℓ−gℓ∥≤dℓ+bℓ∥h~ℓ−hℓ∥.\|\tilde h_\ell-h_\ell\|\le\sum_{k>\ell}\Big(\prod_{j=\ell+1}^{k-1}a_j\Big)c_k,\qquad\|\tilde g_\ell-g_\ell\|\le d_\ell+b_\ell\|\tilde h_\ell-h_\ell\| .

Proof. The adjoint error satisfies h~ℓ−1−hℓ−1=Aℓ⊤(h~ℓ−hℓ)+Eℓ⊤h~ℓ\tilde h_{\ell-1}-h_{\ell-1}=A_\ell^\top(\tilde h_\ell-h_\ell)+E_\ell^\top\tilde h_\ell, starting from zero at the output. Unrolling this recursion and using g~ℓ−gℓ=Bℓ⊤(h~ℓ−hℓ)+Dℓ⊤h~ℓ\tilde g_\ell-g_\ell=B_\ell^\top(\tilde h_\ell-h_\ell)+D_\ell^\top\tilde h_\ell gives the bound. ∎

Consequences.

  • MoE blocks do not propagate error. On protected ranks, the Dgrad and router backward passes are exact, so cℓ=0c_\ell=0. Expert compression changes only the weight gradients of the compressed experts.
  • Attention errors travel only to earlier layers of the same replica. Blocks above the highest protected attention block receive exact adjoints, and the residual path is never approximated.
  • DP averaging divides the error by DD. Replicas without protected ranks are exact.

2. Head budget κ\kappa (Proposition 4).

Setup. Consider a protected attention block with GG selection units (heads, or KV groups under GQA), upstream gradient uu, and retained set SS with ∣S∣=κ|S|=\kappa. Let αq\alpha_q and γq\gamma_q be the input and weight Jacobian norms of unit qq. Then cℓ≤∥u∥∑q∉Sαq,dℓ≤∥u∥(∑q∉Sγq2)1/2.c_\ell\le\|u\|\sum_{q\notin S}\alpha_q,\qquad d_\ell\le\|u\|\Big(\sum_{q\notin S}\gamma_q^2\Big)^{1/2}.

Explicit Jacobian norms. The norms are explicit in weights and activations. The softmax Jacobian has norm at most 1/2, so αq≤∥WO,q∥[∥Sq∥∥WV,q∥+∥Vq∥2dk(∥Kq∥∥WQ,q∥+∥Qq∥∥WK,q∥)].\alpha_q\le\|W_{O,q}\|\Big[\|S_q\|\|W_{V,q}\|+\tfrac{\|V_q\|}{2\sqrt{d_k}}\big(\|K_q\|\|W_{Q,q}\|+\|Q_q\|\|W_{K,q}\|\big)\Big]. RoPE leaves these norms unchanged, and normalization layers contribute their Jacobian norms as factors.

Dependence on κ\kappa. The bound is zero at κ=G\kappa=G and grows by one unit's contribution per discarded unit. If a unit's backward contribution scales with its value-output energy sqs_q (the trend in Fig. 3a), the bound is proportional to ∑q∉Ssq\sum_{q\notin S}\sqrt{s_q}. Our top-κ\kappa rule minimizes exactly this sum.

3. Compression budget χ\chi and rank rr (Proposition 5).

Setup. For one expert linear sublayer and one sequence qq, the exact weight gradient is GW=∑(x,j)πj(x) δˉj(x)⊤aj(x)G_W=\sum_{(x,j)}\pi_j(x)\,\bar\delta_j(x)^\top a_j(x), with one rank-one term per token–expert assignment:

  • aj(x)a_j(x) is the cached input;
  • δˉj(x)\bar\delta_j(x) is the gate-free output gradient;
  • πj(x)=z(x)j/S(x)\pi_j(x)=z(x)_j/S(x) is the normalized gate, where S(x)S(x) is the top-KK mass of the router probabilities over MM experts.

Compressing a set C\mathcal C of assignments replaces aj(x)a_j(x) by aj(x)Pj,ra_j(x)P_{j,r}, with Pj,r=Vj,rVj,r⊤P_{j,r}=V_{j,r}V_{j,r}^\top. With Dˉ:=max⁡∥δˉj(x)∥\bar D:=\max\|\bar\delta_j(x)\| and Rˉr:=max⁡∥aj(x)(I−Pj,r)∥\bar R_r:=\max\|a_j(x)(I-P_{j,r})\|, ∥ΔGW∥F≤MK Dˉ Rˉr Zq(χ),Zq(χ):=∑(x,j)∈Cz(x)j,\|\Delta G_W\|_F\le\frac MK\,\bar D\,\bar R_r\,Z_q(\chi),\qquad Z_q(\chi):=\sum_{(x,j)\in\mathcal C}z(x)_j , because the top-KK of MM probabilities sum to at least K/MK/M.

Consequences.

  • Selection. Our sequence-level rule (Eq. 1) compresses the assignments with the smallest scores, so it minimizes Zq(χ)Z_q(\chi) exactly.
  • Budget. Zq(χ)Z_q(\chi) is zero at χ=0\chi=0, nondecreasing, and convex in χ\chi. This explains why moderate compression is almost free.
  • Rank. Rˉr\bar R_r is nonincreasing in rr and zero at full rank.
  • Refresh. If PrP_r comes from the reference gradient HH of the last refresh, then for the current gradient GG and every c>0c>0, ∥G(I−Pr)∥F≤c (∑k>rσk(H)2)1/2+∥G−cH∥F\|G(I-P_r)\|_F\le c\,(\sum_{k>r}\sigma_k(H)^2)^{1/2}+\|G-cH\|_F. The first term is the spectral tail of the reference, and the second is the drift since the refresh, which TSVDT_{\rm SVD} controls.

Combined bound (Corollary 6). For a replica with protected MoE blocks M\mathcal M and protected attention blocks A\mathcal A, ∥g~(d)−g(d)∥≤MK∑ℓ∈MDˉℓRˉℓ,rZℓ(χ)⏟expert compression: local+∑k∈A∥uk∥[(∑q∉Skγq2)1/2+Γk∑q∉Skαq]⏟head retention: propagates downward,\|\tilde g^{(d)}-g^{(d)}\|\le\underbrace{\frac MK\sum_{\ell\in\mathcal M}\bar D_\ell\bar R_{\ell,r}Z_\ell(\chi)}_{\text{expert compression: local}}+\underbrace{\sum_{k\in\mathcal A}\|u_k\|\Big[\Big(\sum_{q\notin S_k}\gamma_q^2\Big)^{1/2}+\Gamma_k\sum_{q\notin S_k}\alpha_q\Big]}_{\text{head retention: propagates downward}}, where Γk=∑ℓ<kbℓ∏j=ℓ+1k−1aj\Gamma_k=\sum_{\ell<k}b_\ell\prod_{j=\ell+1}^{k-1}a_j is the amplification of a defect injected at block kk. DP averaging then divides the total by DD.

4. Why we also measure the coefficients. The constants are products of Jacobian norms across depth, and Kong et al. (2026) show that such coefficients can grow exponentially with the number of operators.

  • What the bounds establish. They give the structure of Assumption 3, namely how the error depends on (κ,χ,r)(\kappa,\chi,r) and where it can travel. They cannot certify a useful numerical ρ\rho for a 24-layer model in advance.
  • How MeCeFO-v2 uses them. The bounds decide what to keep, since our selection rules minimize them. Direct measurement decides how much to drop: the controller measures the error on held-out microbatches, and Figs. 5–6 and Table R2-4 verify Assumption 3 along the actual runs.
  • Why an exact forward pass matters. Kong et al. also show that forward-pass perturbations are the main obstacle to convergence, because they bias the gradient through the nonlinearities. MeCeFO-v2 avoids them by keeping the forward pass exact.
Reviewer_14mXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents MeCeFO-v2, a fault-tolerant optimization framework for sparse Mixture-of-Experts training under hybrid data, pipeline, and expert parallelism. When a rank fails, the method obtains the missing parameters and optimizer states from a surviving data-parallel replica and distributes the failed experts across the remaining ranks in the affected expert-parallel group. To keep this additional workload within the original memory and compute budgets, protected ranks use two approximations: retaining backward activations only for attention heads selected by output norm, and applying low-rank weight-gradient approximation to low-router-score token-expert assignments. An online controller chooses the retained-head and compressed-assignment budgets based on profiled resource use and gradient error.

The paper provides a nonconvex convergence result for momentum SGD under a bounded relative gradient-error assumption. Experiments train Qwen3-MoE-style models of up to 16B total parameters on 128 H20 GPUs. Under the paper's simulated failure/recovery traces, MeCeFO-v2 reports validation perplexity and downstream accuracy close to fault-free training while limiting throughput degradation to 0.72--6.15%. Additional experiments cover physical node failures and ablate the two approximation mechanisms and the automatic controller.

Critical strengths
  1. The paper identifies a MoE-specific failure mode and proposes a coherent response.

A failed expert-parallel rank affects expert ownership, routing-dependent load, and state placement simultaneously. Redistributing the missing experts across all surviving EP peers is more balanced than assigning the entire failed workload to one neighboring rank, and it is well aligned with the structure of MoE training. This is the main technical reason I view the work as more than a straightforward extension of dense-model fault tolerance.

  1. The implementation is evaluated at meaningful scale and the approximation mechanisms receive useful ablations.

The 128-H20 experiments cover several model sizes and failure patterns, and the paper reports end-to-end throughput including detection, reconfiguration, state transfer, and warm-up. The memory ablation shows that both attention and MoE compression are needed to fit the larger microbatch configuration, while the validation and downstream results suggest that the resulting optimization noise is moderate in the tested setting.

Critical weaknesses
  1. The paper does not establish that the proposed complexity is necessary relative to modern asynchronous checkpointing with hot-spare recovery.

Asynchronous distributed checkpointing can overlap checkpoint writes with training, while local or in-memory checkpoints, replication across failure domains, automatic rendezvous, and reserved spare nodes can substantially reduce recovery time and lost work. Such a system still experiences a restart, so it is not identical to MeCeFO-v2; however, it is the most important production alternative to the paper's approach. The paper compares mainly against MoE checkpointing, reconstruction, and replication systems, but does not provide a competitive configuration that combines frequent asynchronous checkpointing with hot-spare replacement and fast local recovery. This omission matters because MeCeFO-v2 is evaluated with recovery durations of 2--8 hours. If a production cluster can replace a failed worker and reload recent state in seconds or minutes, the interval in which approximate failover is beneficial may be too short to justify dynamic expert migration, selective attention backward, low-rank expert gradients, SVD basis maintenance, online profiling, and their associated correctness risks. A recovery-latency sweep and a total-cost break-even analysis are needed to establish the regime in which the method is practically preferable.

  1. The reliability and optimization risks introduced by approximation are not characterized broadly enough.

MeCeFO-v2 prevents a hard interruption by modifying the training computation, creating the possibility of silent degradation rather than an observable restart. The convergence analysis assumes that relative gradient error remains bounded and proves a rate for momentum SGD, while the actual experiments use AdamW. The empirical error study and controller ablation are limited: the controller is compared with only three manually selected configurations, and the paper does not report variance across seeds or robustness to substantially different routing distributions and model architectures. Consequently, the experiments show that the method works in the tested traces, but do not yet demonstrate that the added machinery is safer or more reliable overall than a simpler recovery design.

Questions for authors
  1. Can the authors compare against a production-oriented baseline combining frequent asynchronous distributed checkpointing, node-local or in-memory checkpoint storage with cross-node replication, automatic rank reconfiguration, and a hot-spare node? Please report detection, rendezvous, load, and warm-up time separately, and sweep the end-to-end replacement/recovery latency from seconds or minutes to hours. At what recovery latency and failure rate does MeCeFO-v2 break even in goodput and GPU cost?

  2. How much of the reported advantage depends on assuming 2--8 hour recovery periods? If a replacement node can rejoin after, for example, 30 seconds, 2 minutes, or 10 minutes, would the system still enter protection mode, and would the benefit exceed the transition and profiling overhead?

  3. How does the controller behave under routing-distribution shift or when a failure occurs between profiling points? The main text describes periodic online profiling, while Appendix B.2 states that policy profiling is performed only twice, at iterations 1500 and 12500. Please clarify this apparent discrepancy and report sensitivity to stale profiles.

  4. Can the authors provide multi-seed results and a direct empirical check of the bounded-gradient-error condition throughout protection episodes when training with AdamW? An explanation of why the momentum-SGD analysis is informative for the actual AdamW configuration would also materially affect my assessment.

9 comments
AuthorsOpenReview ↗

W1 / Q1 / Q2: Comparison with asynchronous checkpointing and hot-spare recovery

Thank you for the thorough review. Your first point is the central practical question for this line of work, so we implemented the baseline you describe and measured it on our cluster. We also added the requested timing breakdown, latency sweep, and break-even analysis, measurements of the error condition under AdamW, and a convergence proof for Adam and AdamW.

W1 / Q1 / Q2: Comparison with asynchronous checkpointing and hot-spare recovery

We target clusters without spare GPUs, which we believe is the common case.

  • A hot spare is redundancy paid for during the whole job. In our 128-GPU job, one spare node is 6.25% of the GPUs. That is already more than MeCeFO-v2 loses under failures in any of the 15 settings of Table 3 (0.72–6.15%).
  • Failed nodes do not come back quickly. A failed node has to be drained, diagnosed, and repaired or replaced, and in a large job some node is almost always under repair. Meta projects a job-level mean time to failure of 1.8 h at 16k GPUs and 0.23 h at 131k GPUs (Kokolis et al., 2025). Llama 3 405B training had 419 unexpected interruptions in 54 days, 78% of them caused by hardware (Grattafiori et al., 2024). A spare pool therefore has to cover all the nodes that are under repair at the same time.
  • Prior work makes the same argument. ReCycle (Gandhi et al., 2024) estimates 280 spare GPUs, an 11% cost increase, for a 530B model. Like ReCycle, Bamboo, and Oobleck, MeCeFO-v2 relies on redundancy that training already has: the DP replicas hold every shard's state, and the EP peers run the same layers.

A restart is paid on every failure, however short the outage.

  • What a restart involves. In synchronous training, one failed GPU stops every DP replica. Even with in-memory checkpoints and a hot spare, the job has to detect the failure, re-form the process groups, load state onto the spare, warm up, and redo the iterations since the last checkpoint.
  • Measured cost in prior work. Gemini (Wang et al., 2023) checkpoints to CPU memory at high frequency and still reports about 7 minutes per failure with standby machines. At one failure per hour, each minute of restart and rollback costs 1.7% of training time.
  • MeCeFO-v2 does not restart. The surviving processes keep their GPU state, rebuild their communicators in place, and fetch only the failed shard from a DP replica.

The new baseline (Q1). We implemented the configuration you describe for the 16B model, on the same cluster and layout:

  • in-memory checkpoints every 10 iterations, replicated to a peer node in another failure domain;
  • persistent checkpoints every 100 iterations;
  • automatic rendezvous and one hot-spare node.

We set the in-memory checkpoint interval close to the Young/Daly optimum for the tested failure rate. Both systems use the same failure detector. Without failures, checkpointing costs the baseline 2.37% of its throughput (200.37k vs. 205.23k tokens/s).

Table R2-1. Cost per failure by phase (16B; seconds; mean over 20 injected failures).

SystemDetectionRendezvous / regroupState loadWarm-upLost workTotal
Async ckpt + hot spare337.219.645.260.8165.8
MeCeFO-v232.31.12.909.3

The tuned baseline loses 165.8 s per failure: 105.0 s for the restart and 60.8 s of repeated work. MeCeFO-v2 loses 9.3 s, about 18× less, and repeats nothing.

Recovery-latency sweep (Q1, Q2). Let LL be the time until the failed GPU, or a replacement, is available.

  • MeCeFO-v2 runs in protection mode for LL and then rejoins at the next iteration boundary, which takes 2.3 s.
  • The checkpoint baseline restarts on the spare. Without a spare, it also has to wait for LL.

We computed goodput at one failure per hour from the measured costs above (formulas in the new App. C.10).

  • The restart baseline: its throughput with checkpointing on (200.37k tokens/s), minus 165.8 s per failure, plus LL when there is no spare.
  • MeCeFO-v2: 11.6 s per failure (transition plus rejoin), plus its protected-mode overhead during the outage, calibrated on Table 3.
  • For LL = 4 h and 8 h, we report the measured Base-Fail and Slow-Recover runs, whose mean recovery times are 4 h and 8 h.

GPU-hours include the spare GPUs.

Table R2-2. Goodput and GPU cost versus recovery latency (16B, one failure per hour). The hot-spare column assumes that a spare is always available.

LLCkpt + hot spare (k tok/s)Ckpt, no spare (k tok/s)MeCeFO-v2 (k tok/s)GPU-h per 1B tokens, ckpt + hot spareGPU-h per 1B tokens, MeCeFO-v2
30 s191.5190.0203.0197.2175.1
2 min191.5185.6202.8197.2175.3
10 min191.5165.2201.8197.2176.2
1 h191.597.9197.7197.2179.9
4 h191.539.7194.0197.2183.2
8 h191.522.1191.3197.2185.8

What the sweep shows.

  • GPU cost. MeCeFO-v2 needs 5.8% (at 8 h) to 11.2% (at 30 s) fewer GPU-hours per token than the hot-spare baseline, even though the baseline is given a spare whenever it needs one.
  • Raw goodput. MeCeFO-v2 is ahead up to L=4L=4 h, and within 0.1% of the baseline at 8 h.
  • Against waiting for the repair. Without a spare, the restart baseline falls to 97.9k tokens/s at 1 h, about half of MeCeFO-v2.
  • With a single spare node. The previous failed node is still in repair when the next failure arrives with probability 1−e−λL1-e^{-\lambda L}, which is 63% at L=1L=1 h. The baseline's goodput then drops to 141.7k tokens/s at 1 h.

Short outages (Q2).

  • What MeCeFO-v2 does. It still enters protection mode and pays its transition, but it does not profile again. The controller reuses the latest surrogate and only re-checks the memory and compute constraints for the new failure set.
  • Accuracy cost. The approximation is active only during the outage, so its accuracy cost is negligible.
  • The baseline. It pays its full restart cost even for a 30-s outage.

The paper uses 2–8 h recovery windows because, without spares, an outage lasts as long as the hardware repair. Long windows are also the hard case for accuracy.

Break-even (Q1). We follow the effective-training-time model of Kokolis et al. (2025) and add the cost of spares. Relative to fault-free training, the goodput per GPU of the restart baseline is GHS=1−λ(τrs+Δck/2)1+wck/Δck⋅NN+S,G_{\rm HS}=\frac{1-\lambda(\tau_{\rm rs}+\Delta_{\rm ck}/2)}{1+w_{\rm ck}/\Delta_{\rm ck}}\cdot\frac{N}{N+S}, where:

  • λ\lambda is the failure rate and τrs\tau_{\rm rs} the restart time;
  • Δck\Delta_{\rm ck} is the checkpoint interval and wckw_{\rm ck} the stall per checkpoint;
  • NN is the number of training GPUs and SS the number of spare GPUs.

For MeCeFO-v2, it is GM=1−sMG_{\rm M}=1-s_{\rm M}, where sMs_{\rm M} is its measured throughput loss, which includes its transitions (Table 3).

At one failure per hour, GM>GHSG_{\rm M}>G_{\rm HS} for every restart time τrs≥0\tau_{\rm rs}\ge0, so there is no break-even point.

  • Why. Even with an instantaneous restart and free checkpoints, the spare factor N/(N+S)=0.941N/(N+S)=0.941 and the expected rollback of half a checkpoint interval leave the baseline at 0.928 of fault-free throughput per GPU. MeCeFO-v2 keeps 0.932 with 8-hour repairs and 0.945 with 4-hour repairs.
  • In terms of restart time. With our measured checkpoint overhead, the restart time at which the two would break even is negative: −155 s at 4 h and −104 s at 8 h. The measured restart time is 105.0 s. Since λ\lambda grows with job size and MeCeFO-v2's cost per failure is small, the gap widens at scale.

This comparison is in throughput. MeCeFO-v2's quality cost of +0.02–0.24 PPL is accounted for in time to target (Table R3-1, response to Reviewer 3).

AuthorsOpenReview ↗

W2 / Q4 (evidence): Reliability of the approximation, multi-seed results, and the error condition under AdamW

What the added machinery costs, and why it is needed.

  • The costs are already counted. The transition, the protected-mode overhead, the SVD refreshes, and profiling are all included in the end-to-end throughput of Table 3.
  • Why it is needed. The compression exists because the absorbed experts must fit into the survivors' memory and compute. Without it, protected ranks run out of memory at micro-batch size 32, and they lose 12–32% throughput at micro-batch size 16 (Table 14).
  • Training stays exact everywhere else. The approximation is confined to the backward pass on protected ranks. The forward pass, the optimizer states, and DP averaging are exact, and rejoining restores exact training (App. C.5).
  • Stability. We saw no loss spikes or divergence in any run.
  • Availability. Checkpointing remains the fallback for failures outside the support conditions, so the job is never less available than with checkpointing alone.

The two approaches are complementary. When spares exist, MeCeFO-v2 keeps training while the spare is brought up, and the spare joins through the same rejoin path we use for repaired GPUs (App. C.5). The restart then leaves the critical path.

These results are in a new Sec. 5.4, with implementation details in the new App. C.10.

W2 / Q4 (evidence): Reliability of the approximation, multi-seed results, and the error condition under AdamW

The accuracy cost is small and reproducible, and it is measured during training.

Existing runs.

  • Protection is almost always active. With a failure every 0.5–2 h and 2–8 h repairs, a 16B run sees 38–145 failures, and at least one GPU is down during 86.5–98.6% of the iterations.
  • The cost stays small. Across 3 models and 5 scenarios, PPL rises by 0.02–0.24 (at most 1.89%). Zero-shot accuracy stays within 0.45 points and SFT accuracy within 0.56 points.
  • The gap shrinks with scale. The worst-case gap is +0.24, +0.13, and +0.11 for the 1B, 8B, and 16B models.

Multi-seed results. The gap is the same whether we vary the training seed or the failure trace. The standard deviations are at most 0.00021, about 2% of the gap.

Table R2-3. Final validation loss of the 1B model under Base-Fail (mean ± std over 5 runs).

VariedNo faultMeCeFO-v2Gap
Training seed2.54043 ± 0.000122.55146 ± 0.00017+0.01103 (0.43%)
Failure trace2.54043 ± 0.000122.55132 ± 0.00021+0.01089 (0.43%)

Assumption 3 along AdamW protection episodes.

  • Setup. We resumed protected AdamW training from six saved checkpoints of the 16B run, with the actual failure set. Every 10 iterations, we compared the protected gradient with an exact shadow backward pass on the same microbatches.
  • Quantities. We report (a) the single-batch error and (b) the bias, estimating conditional expectations at fixed weights with 16 probe microbatches. We also report (c) the bias in Adam's metric diag(1/(v^t−1+ϵ))\mathrm{diag}(1/(\sqrt{\hat v_{t-1}}+\epsilon)), which is the condition of Theorem 2 below.

Table R2-4. Relative gradient error along AdamW protection episodes (16B).

QuantityMean99th pct.MaxNeeded by the theory
(a) single-batch error, Euclidean0.0720.1330.161< 1 (Theorem 1)
(b) bias, Euclidean0.0350.0820.096< 1 (Theorem 1)
(c) bias, Adam's metric0.0380.0880.103< 1 (Theorem 2)

What the measurements show.

  • All three conditions hold with a wide margin. The largest value of any ratio is 0.161, so Assumption 3 holds along the AdamW runs with ρ≤0.161\rho\le0.161.
  • Adam's metric changes little. Measured in Adam's metric, the bias is only slightly larger than in the Euclidean norm (0.103 vs. 0.096 at the maximum), so the condition of Theorem 2 holds with ρAdam≤0.103\rho_{\rm Adam}\le0.103.
  • The bias is about half the single-batch error. Much of the single-batch error therefore averages out over minibatches.
Reviewer_NScEOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper extends approximate online failover to MoE training with hybrid data, pipeline, and expert parallelism. It redistributes failed expert workloads across surviving expert-parallel peers and combines selective attention-head retention, router-aware low-rank expert-gradient compression, and a resource-constrained controller. The authors evaluate three MoE model scales under controlled failure scenarios, with additional node-failure experiments and a convergence analysis.

Critical strengths

S1. The research question is well framed: extending approximate failover to MoE training while retaining high throughput. This is executed by leveraging cooperative redistribution across expert-parallel peers, importance-based attention retention, router-aware compression, and the adaptive resource-constrained controller. The choices are well motivated, and their relationship to prior work is clearly explained in Section 2.

S2. The evaluation across three model sizes is a strength. The validation-loss curves in Figure 4 clearly show the quality comparison, and Table 3 reports strong throughput results across the tested failure scenarios.

Critical weaknesses

W1. The origin of the no-fault throughput differences in Table 3 is unclear. Each method uses its own reference throughput, so understanding those differences matters for interpreting the reported gains.

W2. The abstract and opening introduction make the contribution harder to assess than necessary. Phrases such as “reshape the fault-tolerance problem” and “substantially enlarged training state” need more concrete explanation; the assumptions, controlled failure regime, and numerical meaning of “close” performance should appear earlier.

Questions for authors

Q1. What causes the no-fault throughput differences between methods in Table 3, and how should we account for them when interpreting the comparisons?

Q2. What steps make the failure-dependent training runs reproducible? Appendix C.4 explains that sample order is preserved; please clarify how the failure traces are generated and whether the reported failure interval is per rank, group, or cluster.

Q3. How expensive are the SVD updates in amortized training time? Appendix B.2 specifies an update every 100 iterations, and Appendix C.6 includes end-to-end overhead, but the main text does not clarify the cost of these updates.

Q4. Could the abstract and introduction state the failure assumptions and measured regime explicitly, quantify the quality gaps, and explain the comparison underlying the enlarged-state claim? Section 5.1 and Table 1 describe the controlled failure modes, Appendix D.1 adds physical node failures, and Tables 2 and 4 already provide the quality results. Showing that context earlier, including best/worst PPL gaps in Table 2, would clarify the scope of the claims. Please also distinguish the MoE optimization opportunity from the centralized control-plane choice described in Appendix C.1.

6 comments
AuthorsOpenReview ↗

W1 / Q1: Where the no-fault throughput differences in Table 3 come from

Thank you for the careful review and the concrete suggestions. We rewrote the abstract and introduction along the lines you proposed and added the measurements you asked for.

W1 / Q1: Where the no-fault throughput differences in Table 3 come from

All methods use the same models, parallel layout (DP = 8, PP = 4, EP = 4), micro-batch size, and failure detector. For MoC-System, Lazarus, MoEvement, and MeCeFO-v2, which run on the same Megatron-based stack, the differences in the No Fault column come from the fault-tolerance work that each method does while nothing fails, mainly checkpointing:

  • MoC-System and MoEvement checkpoint frequently (partial-expert or sparse checkpoints) to limit rollback.
  • Lazarus also maintains expert replicas.
  • MeCeFO-v2 restores a failed shard from a healthy DP replica, so it does not need frequent checkpoints.

MeCeFO-v1 is earlier work, and its Table 3 runs use an older Megatron version. Its No Fault gap reflects this version difference, not fault-tolerance work: like MeCeFO-v2, it restores from a DP replica and needs no frequent checkpoints.

To make this explicit, we ran every method with its fault-tolerance machinery switched off (new App. C.8):

Table R1-1. No-fault throughput of Qwen3-MoE 16B-A2B (tokens/s).

MethodFault tolerance offNo fault (Table 3)Normal-operation overhead
MeCeFO-v1202.39k184.29k8.94%*
MoC-System204.31k152.46k25.38%
Lazarus200.12k143.98k28.05%
MoEvement203.71k161.37k20.78%
MeCeFO-v2205.23k203.73k0.73%

*Older Megatron version in Table 3, not fault-tolerance overhead.

On the same stack, the methods match. With fault tolerance off, all five methods run within 2.5% of each other. The gaps in Table 3 are therefore normal-operation overhead, apart from MeCeFO-v1's version difference: 20.78–28.05% for the checkpoint- and replication-based MoE systems, and 0.73% for MeCeFO-v2.

How to read Table 3. It reports two complementary measures.

  • The percentages measure robustness. Each method is compared with its own no-fault run.
  • The absolute throughput under failures is the end-to-end measure. It combines both effects, and MeCeFO-v2 is best on both. For example, on 16B under Frequent-Fail, MeCeFO-v2 is 2.12× faster than MoEvement (191.21k vs. 90.32k tokens/s): a factor of 1.26 comes from lower normal-operation overhead, and a factor of 1.68 from better retention under failures.

The caption of Table 3 now says this. Time and GPU-hours to a target perplexity, which also account for model quality, are in our response to Reviewer 3 (Table R3-1).

AuthorsOpenReview ↗

W2 / Q4: Making the abstract and introduction concrete

W2 / Q4: Making the abstract and introduction concrete

We agree and rewrote both. The revised abstract is at the end of this comment.

"Reshape the fault-tolerance problem" now names the mechanism.

  • Under expert parallelism, each GPU holds experts that no other GPU in its EP group holds, and any token of the group may be routed to them.
  • When one GPU fails, these experts are missing for every token of the group. The group's all-to-all stalls, and through synchronous data parallelism, so does the whole job.
  • Recovery therefore has to restore expert capacity inside the EP group, which dense models do not require.

"Substantially enlarged training state" now states the comparison.

  • An MoE model's parameters and optimizer states grow with its total size, while its compute per token grows only with its active size.
  • Qwen3-MoE 16B-A2B holds 8× more parameters than it activates per token. With mixed-precision Adam (16 bytes per parameter; Rajbhandari et al., 2020), its training state is about 256 GB, against about 32 GB for a dense model with the same compute per token.
  • Checkpointing, replicating, or migrating this state is correspondingly more expensive. This is visible in the no-fault overhead of the checkpoint- and replication-based baselines (Table R1-1).

The assumptions and the failure regime now appear in the abstract and the introduction.

  • MeCeFO-v2 requires (i) a healthy DP replica of the failed GPU's state and (ii) a surviving peer in every affected EP group. Other failures fall back to checkpoint recovery (App. C.7).
  • We evaluate cluster-level failure intervals of 0.5–2 h with 2–8 h repairs (Table 1), as well as physical 8-GPU node failures (App. D.1).

"Close" now has numbers. Validation PPL rises by 0.02–0.24 over fault-free training (0.24–1.89%):

ModelNo faultBest caseWorst case
Qwen3-MoE 1B-A0.4B12.6812.72 (+0.04, Sparse-Fail)12.92 (+0.24, Slow-Recover)
Qwen3-MoE 8B-A1.3B9.209.24 (+0.04, Fast-Recover)9.33 (+0.13, Frequent-Fail)
Qwen3-MoE 16B-A2B8.488.50 (+0.02, Fast-Recover)8.59 (+0.11, Frequent-Fail)

The worst-case gap shrinks as the model grows. On 16B, zero-shot accuracy stays within 0.45 points of fault-free training (Table 5), and SFT accuracy within 0.56 points (Table 13).

The MoE opportunity is in the data plane.

  • Expert parallelism provides E − 1 peers that run the same layer on the same tokens. The failed experts can therefore be spread over all of them, so each peer absorbs about 1/(E − 1) of the shard.
  • MoE models also provide cheap per-batch importance signals, namely routing scores and value-output norms, which show where approximation costs least.
  • The centralized controller of App. C.1 only agrees on the set of failed ranks. We adopted it from Lazarus, and a decentralized membership protocol such as SWIM or Rapid could replace it without changing the method. The introduction now states this separation.
AuthorsOpenReview ↗

The revised abstract

The revised abstract:

Sparse Mixture-of-Experts (MoE) models scale language models efficiently, but they make hardware failures during distributed training more costly. Under expert parallelism, each GPU hosts experts that no other GPU in its group holds, so a single failure stalls its expert-parallel group and, through synchronous data parallelism, the whole job; and because MoE parameter and optimizer state grows with total rather than active size (8× for a 16B-A2B model), checkpointing and replication are expensive. We propose MeCeFO-v2, which keeps MoE training running through failures without rollback or spare GPUs. As long as the failed GPU's state has a healthy data-parallel replica and its group keeps a surviving peer, MeCeFO-v2 restores the state from the replica and spreads the failed experts evenly over the surviving peers. To absorb the extra work within the original memory and compute budget, it keeps the forward pass exact and approximates only the backward pass on affected GPUs: it backpropagates through the attention heads with the largest value-output norms and uses low-rank weight gradients for the token–expert assignments with the lowest routing scores. A lightweight controller sets both budgets from profiled memory, compute, and gradient error. We prove that MeCeFO-v2 keeps the O(1/nT)O(1/\sqrt{nT}) rate of momentum SGD for any momentum, extend the analysis to Adam and AdamW, and bound the gradient error explicitly in terms of the approximation budgets. On Qwen3-MoE models with 1B–16B parameters trained on 128 GPUs, with a failure every 0.5–2 hours and 2–8-hour repairs, MeCeFO-v2 loses 0.72–6.15% throughput relative to fault-free training, compared with 7.61–57.92% for prior baselines, while validation perplexity rises by only 0.02–0.24.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.