acceptodds
Under review as a conference paper at ICLR 2027

FAILSAFE-MOE: TRAINING SPARSE MIXTURE OF EXPERTS FOR WORST CASE EXPERT OUTAGES

Abstract

Sparse Mixture-of-Experts (MoE) models route each token to a few of Eexperts, so an unavailable expert corrupts every token routed through it. We argue that worst-case expert-outage risk is an exposure phenomenon: it appears when router load is skewed, uniform expert dropout under-weights it, and weighting training outages by exposure — no search — reduces most of it at uniform-dropout cost. A sampled-max adversary drawing mof Ecandidates optimises a spectral risk measure with weight m/E on the worst outage, against 1/E for uniform dropout. In single-MoE-layer models, worst-case damage tracks router load skew: at an OLMoE-like 3.23×the exact adversary gains +1.48 pt of worst-case accuracy over conventional training, while uniform dropout is -2.35 pt worse than no outage training at all. An exposure-proportional sampler at uniform-dropout cost (1.13–1.21×a clean step) matches the exact adversary within ±1.0 pt (TOST, two benchmarks), but reduces the tail rather than removing it: 1.74 pt of worst-outage drop remains at 3.23×and 2.01 pt at 11.58×. In a 6-layer 99 M-parameter MoE language model on WikiText-103 the same sampler halves worst-outage loss damage (0.077 against 0.157 nats, 3 of 3 seeds, n=3) at equal steps and equal clean loss; given that compute as extra steps, a conventionally trained model is 0.27 nats better yet more fragile (0.444 nats of damage). The pre-registered first-order criticality diagnostic is refuted; router mass is the best cheap predictor overall.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Mixture-of-Experts, sparse MoE, expert outages, fault tolerance, worst-case robustness, expert dropout, minimax training, spectral risk measures, load balancing, router load skew

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_uvSGOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the robustness of sparse Mixture-of-Experts (MoE) language models to expert outages. It formulates worst-case expert-outage robustness as a minimax training objective and proposes FAILSAFE-MoE, a family of training strategies that expose the model to expert failures during training.

Critical strengths
  1. A simple training strategy. FAILSAFE-RM uses router mass to prioritize outage scenarios without exhaustively searching all experts. The reported results on the 99M-parameter language model are encouraging: the worst-outage NLL increase decreases from 0.157 to 0.077 nats compared with conventional random expert dropout, while clean performance remains comparable.

  2. A useful investigation of the limits of the proposed approach. The paper reports that the benefits depend on routing-load skew and become limited when expert utilization is relatively balanced. This helps characterize the conditions under which outage-aware training may be useful.

Critical weaknesses
  1. Limited practical motivation. The paper assumes that an individual expert becomes unavailable while the remaining experts continue operating normally. However, it does not sufficiently establish how common this scenario is in real-world MoE serving or why existing system-level fault-tolerance mechanisms are insufficient.

  2. Unclear incremental value over system-level solutions. FAILSAFE-MoE improves model quality under simulated expert outages, but does not address failure detection, recovery, or expert relocation. The paper does not clarify when outage-aware training provides meaningful benefits beyond existing system-level fault-tolerance mechanisms.

  3. Limited evaluation at realistic scales. Training experiments are conducted on a 99M-parameter MoE model, while the larger OLMoE evaluation is an inference-only outage audit. It remains unclear whether the reported gains generalize to large pretrained MoE models and realistic serving environments.

Questions for authors
  1. How does FAILSAFE-MoE perform when multiple experts become unavailable simultaneously? Does the proposed sampling strategy remain effective when the outage pattern involves correlated failures or multiple heavily loaded experts?

  2. How much of FAILSAFE-RM's improvement comes from exposure-based weighting, as opposed to simply excluding experts with no traffic in the current batch? Can the authors isolate these two effects more clearly?

  3. Beyond validation loss, can the authors evaluate whether outage-aware training affects clean-model performance on downstream tasks or generalization to unseen data, and quantify the trade-off between normal-operation quality and outage robustness?

2 comments

Thank you for the review and the direct questions on motivation and systems.

W1, how common the scenario is. These statistics come from training and HPC fleets, and they show how often a serving unit of the same hardware would lose a device. In expert-parallel serving a failed device takes its experts with it. In the decoding deployment of DeepSeek-V3 each GPU hosts one expert (DeepSeek-AI, 2024). On the Delta cluster the measured availability of a GPU node is about 99.4% for A100 and 99.3% for H100 (Cui et al., 2025). In a 54-day Llama 3 pre-training run on up to 16K GPUs, about 78% of 419 unexpected interruptions were attributed to hardware (Llama Team, 2024).

With the Delta availability and independent nodes, at least one node is down 4.7% to 5.5% of the time in a unit of 8 nodes. In a unit of 40 nodes, the size of the DeepSeek-V3 decoding unit, the share is 21.4% to 24.5%.

W2, what training adds to system-level recovery. Training complements detection, recovery and relocation. It sets what the model serves between a failure and the end of recovery, and wherever no copy of the lost experts exists. The recovery system of Sun et al. (2026) pauses in that window until every lost expert is hosted again, 11 s after one rank fails, against 348 s of unavailability for a full restart. They name the alternative, "continuing immediately over the reduced peer set", and set it aside without measuring it. We measure that option offline.

Token-level training lowers what that serving costs in quality. With 64 experts and top-8 routing in six MoE layers (E42, Appendix S.1), a node outage removes 8 of the 64 experts of every layer. The worst node then costs 0.0679 nats after token-level training against 0.2420 after conventional training. One-pass uniform dropout leaves 0.1703 nats. A randomly failed node costs 0.0543 nats, against 0.1475 and 0.1153. All three readings we fixed before the runs are met. There is no clean cost in any of the five seeds, and training needs one pass per step and no extra memory.

The benefit grows with the time a unit serves degraded. Without a spare, a unit serves degraded until repair. For a randomly failed node or rank in a 40-node unit, token-level training at ρ=0.15\rho=0.15 has lower expected NLL than conventional training (p at most 0.004). For node outages it is also lower than uniform dropout. With a dense first layer the clean cost is 0.0039 nats, and the break-even share for the worst node averages 5.46% [0.92%, 10.42%] of serving time. This is the share an outage must be active for the training to pay. With spares, recovery pauses take 0.012% to 1.2% of serving time, and token-level training improves what is served during each pause. In the 16-expert all-MoE model at ρ=0.15\rho=0.15 its expected NLL with spares is level with conventional training and uniform dropout (p at least 0.459), and at ρ=0.3\rho=0.3 it is higher than with uniform dropout by about its clean cost (general response, part 2).

Replication and standby copies. Training protects all experts at no memory cost, and copies add protection where memory for them exists. DeepSeek-V3 reserves 64 GPUs "for hosting redundant experts and shared experts". In OLMoE-1B-7B (E39, Section 7) a node outage costs up to 0.1735 nats on held-out web text with experts placed by index (0.191 under a random placement). Replicating the 8 most loaded experts per layer takes 12.5% extra expert memory and leaves 0.1472 nats. Int4 standby copies (25%) leave 0.0105 nats, and int8 copies (50%) remove the damage.

W3, scale. E42 gives our trained models the configuration of production MoEs, 64 experts with top-8 routing in six MoE layers (results under W2). The models we train from scratch stay at 99M parameters. In the 16-expert model at matched compute (E37, Section 6.4, Table 2) all three pre-specified hypotheses are confirmed. The worst single-expert outage costs 0.0112 nats after token-level training against 0.3424 after conventional training and 0.1699 after one-pass uniform dropout.

For the large pretrained model, a LoRA fine-tune of OLMoE with node outages (E40, Appendix O) removed 22.7% of the worst-node damage on web text for the placement used in training, from 0.1745 to 0.1349 nats. Our rule set 25% for a three-seed study, so this remains a one-seed trial. For an unseen placement it removed 6.5%, and on WikiText-103 5.5%.

Q1, several experts at once. Training on single outages also protects against the loss of a whole node, the natural correlated failure, and Bernoulli masks extend coverage to pairs of experts at a higher clean cost.

A node holds eight experts of every OLMoE layer. In OLMoE-1B-7B (E39) a rank outage costs at most 0.027 nats, against up to 0.1735 for a node outage. On WikiText-103 the 24 node outages cost on average 1.18 times the sum of their single-expert damages. In our trained models, the readings on node outages that we fixed before the runs are met (eight experts of every layer in E42, two in E43). In the 16-expert model (E37) the worst node, two experts of every layer, falls from 0.6012 nats under conventional training to 0.0907, and uniform dropout leaves 0.3360.

On pairs of single experts in the 16-expert model, token-level training leaves 0.2554 nats on the worst pair against 0.3822 after uniform dropout (p = 0.133). A Bernoulli mask at the same own-outage rate extends this coverage in a pre-specified test. At ρ=0.15\rho=0.15 its worst-pair damage is 0.2314 nats lower (one-sided p = 0.0018), while token-level sampling keeps the lower clean NLL. At ρ=0.3\rho=0.3 the mask reaches 0.0522 nats on the worst pair at twice the clean cost.

On the benchmark (E32, ten seeds, E=32E=32) the method leads under balanced load. There it beats uniform dropout, the exhaustive adversary, growing subsets and Bernoulli dropout at rate 0.25, with 62.47% on the worst pair against at most 61.67%. Under four outages it reaches 61.36% against at most 60.10%. Bernoulli dropout at rate 0.5 is level with it under balanced load. With a collapsed router, a two-expert variant of our method reaches 61.50% on the worst pair. The single-outage method reaches 51.58% there, against 57.12% for the exhaustive adversary and 60.33% for Bernoulli dropout at rate 0.5.

Q2, exposure weighting against excluding experts without traffic. The weighting carries most of the gain, and a new control isolates it.

The submission has one control, RandDrop-Live, which draws uniformly over the experts that carry traffic in the batch (submission Table 12, now Table 24). With a collapsed router it gains 2.76 pt of worst-outage accuracy over uniform dropout, against 7.06 pt for batch-level FailSafe-RM. Excluding experts without traffic thus explains about 39% of the gain. Under balanced load it gains −0.14-0.14 pt.

Our new control runs the same one-pass procedure as token-level FailSafe-RM but draws the removed expert uniformly from all EE. Its expected objective therefore equals that of uniform dropout at β=1\beta=1. The control against uniform dropout isolates the estimator, and the method against the control isolates the weight on each token's own experts. At E=32E=32 the method beats the control by 6.52 pt with a collapsed router and by 2.19 pt under balanced load (both p < 10−510^{-5}). The control itself is 0.54 and 1.77 pt above uniform dropout. The weight thus carries nearly all of the gain with a collapsed router, while weight and estimator contribute about equally under balanced load. The weight effect grows with the pool, from 1.85 pt at E=8E=8 to 8.70 pt at E=64E=64, as the identity of Section 4 leads us to expect. A uniform draw among the token's own experts performs the same as the gate-proportional draw, so the gain comes from the token's own experts.

The per-token draw also beats two exploratory variants of uniform dropout. A higher weight gains at most 2.02 pt over β=1\beta=1. Reweighting to the expected objective of the token-level draw trails it by 2.47, 9.55 and 16.08 pt at E=16E=16, 32 and 64, and is level at E=8E=8.

Q3, downstream tasks and unseen data. We now report both. Outages hurt downstream accuracy. At its worst node, a node outage of OLMoE (E39) lowers accuracy by up to 9.8 points on MMLU, 8.8 on ARC-Challenge and 3.9 on HellaSwag.

Outage-aware fine-tuning (E40, one seed) keeps clean downstream accuracy close to clean fine-tuning, at a clean NLL cost of 0.0039 nats. Clean MMLU accuracy is 47.2% against 47.0%, ARC-Challenge 55.8% against 57.0%, and HellaSwag 76.4% against 77.0%. Accuracy under a node outage shows no consistent change.

In the language model (E37) the protection carries over to another domain. On AG News the worst single outage costs 0.0100 nats after token-level training against 0.1816 after uniform dropout. On AG News and IMDB token-level training has no higher clean NLL than conventional training. In-domain its clean cost is 0.0150 nats, within our margin of 0.03, and part 1 (W2) gives the trade-off in expected NLL.

Reviewer_eCZ5OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies robustness to global expert outages in sparse MoE models under reroute-on-failure. It compares uniform single-expert dropout, sampled-max and exhaustive adversaries, and FAILSAFE-RM, which samples outages proportionally to current batch router mass. Exposure-weighted sampling approaches exhaustive adversarial training on small classification benchmarks with skewed routing. A 99M-parameter language model provides additional evidence of reduced worst-outage damage. The paper also applies an existing spectral-risk identity to sampled outages, reports a negative result for its criticality diagnostic, and audits single-expert outages in a pretrained OLMoE model.

Critical strengths

The authors evaluate every possible outage, match seeds, control for compute and training steps, select outages on held-out data, and add a RandDrop-Live baseline, which together rule out several alternative explanations I would otherwise have worried about. I also really appreciated that they separate exploratory analyses from pre-registered tests and report negative results. That made it much easier for me to judge exactly how far the evidence goes.

Critical weaknesses
  1. It is not clearly shown that the quality-robustness trade-off is worth it under realistic sparse execution. Most of the training evidence comes from simplified single-layer models. The 99M language-model study uses a single corpus and three seeds, and nearly all of the worst-case improvement shows up in layer 0. The lack of a clear advantage at k=8 in the SSB ablation makes me a bit worried about external validity.

  2. I’m not sure how to understand the reported 1.13–1.21x step ratios, because they’re measured against a dense clean baseline. At E=32, k=2, sparse execution already uses 16x less expert arithmetic than that baseline, and the proposed top-(k+1) implementation is never actually timed. In the LM experiment, each step costs roughly twice a clean step. At matched compute, RM makes clean NLL worse by about 0.268 nats while improving absolute worst-outage NLL by about 0.099 nats (4.160 vs. 4.259). To me this is a genuine trade-off, and neither method shows itself better than the other. It's better to have a measured sparse costs and comparisons against targeted layer-0 or structural alternatives, that way I could say when this trade-off is worth making.

Questions for authors
  1. I was confused by an apparent inconsistency: Section 6.3 says shared-expert and Exact training were not combined, but Table 21 reports fs_exact_s1 results. Could you clarify how these fit together?

  2. Could you report measured costs under actual sparse execution, and show how the trade-off between clean quality and worst-outage quality changes within different training budgets? Since the longer Clean run also sees more tokens, which is not quite fair.

  3. Are RM’s sampling probabilities detached from the computation graph? I’d like the paper to state exactly which adaptive surrogate is being optimized, and to make clear how it differs from the parameter-dependent expected objective in Eq. (1).

1 comment

Thank you for your review.

W1, external validity. Five seeds now replace three, and the protection carries to text the model never trained on. On AG News the worst single outage costs 0.0100 nats after token-level training against 0.1816 after uniform dropout. With 64 experts and top-8 routing, closer to production (E42), the three readings we fixed before the runs are met. The worst single outage costs 0.0087 nats with token-level training, against 0.0408 for one-pass uniform dropout and 0.0630 for conventional training. The worst node costs 0.0679 nats against 0.1703 and 0.2420, at 0.0055 nats lower clean NLL (p = 0.032). On the single-layer benchmark at 8 of 64 the gain over uniform dropout is 0.77 and 0.29 pt, while the language model at 8 of 64 shows the large gain above.

W2 and Q2, the trade-off at matched compute. At matched compute, token-level training now trades a small clean cost for a large gain under the worst outage. We retrained the language model on five new seeds with identical batches (E37, Table 2). All three pre-specified hypotheses are confirmed (one-sided p = 0.0002 and 0.0029), and the clean cost stays within the margin of 0.03.

Conventional training has a clean NLL of 4.0873 nats and 4.4297 under the worst single outage. One-pass uniform dropout has 4.0862 and 4.2561, and token-level training 4.1023 and 4.1135. Against conventional training, token-level training gives up 0.0150 nats of clean NLL and gains 0.3162 under the worst single outage. Against uniform dropout it gains 0.1426 under that outage.

The gap appears at every training budget. After 1,375 steps the worst single outage costs 0.0114 nats against 0.1015 for uniform dropout. The clean cost against conventional training is +0.0070, +0.0018 and +0.0144 nats after 1,375, 2,750 and 5,500 steps. Under sparse execution the median step takes 88.7 ms against 85.3 ms, within run-to-run noise. These measured step times replace the 1.13 to 1.21x ratios against a dense baseline.

When does it pay? With 64 experts at any outage probability, since in each seed token-level training has the lower clean NLL and lower damage on all seven measures. In the 16-expert all-MoE model at ρ=0.15\rho=0.15 the clean cost is 0.0030 nats. The break-even share divides the clean cost by the damage reduction per seed, then averages (two of five seeds show no clean cost). Against conventional training it is 1.58% for the worst single expert and 1.19% for the worst node, and 2.36% for a randomly failed node. At ρ=0.3\rho=0.3 the first two are 4.32% and 3.02%.

Without a spare device, token-level sampling has lower expected NLL than conventional training for a randomly failed node or rank in a 40-node unit (p at most 0.004). With spares at ρ=0.15\rho=0.15, or in an 8-node unit, the differences in expected NLL are within noise. The submitted two-pass variant needed outages active over 73% of the time, which is why the method now uses one forward and one backward pass.

Token-level training lowers damage outside the first layer too, and most of the gain sits in the first layer, where the worst single outage lies in every run (0.1587 nats, against 0.0205 outside it). E43 tests the structural alternative you ask for, a dense first layer as in DeepSeekMoE, with readings fixed before the runs. This layer removes most of the single-outage damage of conventional training (0.3424 to 0.0316 nats) and gives 0.0449 nats lower clean NLL. All-MoE token-level training leaves less damage (0.0193 nats), and the dense model has the lower NLL under the worst single outage (4.0741 against 4.1097).

Token-level training on top of the dense layer lowers the worst node from 0.1574 to 0.0324 nats and the worst single outage to 0.0059, at a clean cost of 0.0039 nats (p = 0.42). The break-even shares average 5.46% [0.92%, 10.42%] for the worst node and 24.77% [5.54%, 44.91%] for the worst single expert. An exploratory arm named before the runs compares it with uniform dropout with a dense first layer. It leaves 0.0059 against 0.0204 nats at the worst single outage and 0.0324 against 0.1017 at the worst node (both p < 0.001), at 0.0076 nats higher clean NLL (p = 0.086).

Q1, Section 6.3 and Table 21. We corrected the sentence in Section 6.3, and Table 21 (now Table 33) stands. There the exhaustive adversary with a shared expert reaches 63.60% worst-outage accuracy, against 59.33% for the shared expert alone.

Q3, detached sampling probabilities. Yes. The router mass (batch level) or the gates (token level) are computed without gradient, and the removed expert is a discrete draw. The update is therefore a semi-gradient, the gradient of ∑epeL(θ;{e})\sum_e p_e \mathcal{L}(\theta;\lbrace e\rbrace) with pp held fixed. The score-function term ∑eL(θ;{e})∇θpe\sum_e \mathcal{L}(\theta;\lbrace e\rbrace)\nabla_\theta p_e of the expectation in Eq. (1) is left out, and Section 5 states this surrogate.

Reviewer_w5KSOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper aims to check whether training can alleviate the performance drop when one of the experts is disabled in an MoE model. In addition, it assumes that the MoE model can reroute tokens to other experts. It then compares several ways to choose which expert to disable in training, including random selection, worst outage among a small subset of experts, worst outage among all experts, and sampling based on route mass.

Critical strengths
  • The problem is somehow meaningful: using training to prepare MoE models for expert failures.
  • Under the collapsed router, the proposed FAILSAFE-Exact and FAILSAFE-RM improve robustness. The latter method achieves similar performance to FAILSAFE-Exact at a lower cost.
Critical weaknesses
  • Readability and Presentation. I find it really difficult to follow: several concepts are introduced without demonstrations e.g., CVaR, CVaR mixtures, and tail-weighted training; Section 4.1 adds symbols and a bound that the author acknowledges as uninformative, so I am confused about how it should connect to the main method; Section 5 mixes the method, training procedure, implementation details, and computational limitations, making it difficult to understand what the key point is; the paper also has formatting problems, like Table 1's caption font size is not being aligned, headers are partly missing, and page numbers are only partially displayerd.

  • Experiment Results. In the OLMoE-like setting, RM improves worst-outage acc by 1.22, but clean acc also improves by 0.80. The reduction in outage-induced damage is only 0.42. Given that this is the main experiment, the improvement is not convincing.

Questions for authors
  • Which deployment scenarios match the assumed failure protocol, especially when experts become unavailable during test time? Also, how would the conclusions change when multiple experts become unavailable?

  • Please clarify Section 4.1 and its connection to the main method. How do Proposition 1 and its loose bound contribute to the paper?

1 comment

Thank you for pointing to the exact cell and to the formatting problems.

W1, readability and presentation. We rebuilt the paper around one claim in the introduction, one elementary identity with a short proof in Section 4, and one algorithm box in Section 5. The statement now numbered Proposition 1 is this identity, and the bound you asked about is in Appendix C. The other changes are these:

  • CVaR, CVaR mixtures and the spectral-risk view are now in Appendix B.
  • Section 4 is titled by its result (uniform dropout trains a token's own outages at rate k/Ek/E) and keeps a two-part corollary. The bound on the worst outage moved to Appendix B.
  • Section 5 has four parts: the method, what is optimised, the family of schemes, and cost.
  • Table 2 is the one table of the language models.
  • We corrected the font of the Table 1 caption and checked every page for running head and page number.

W2, the size of the improvement. The revised method removes most of the drop in the cell you quote. In that cell the submitted batch-level variant removed 0.42 pt of the 2.17 pt drop. This cell (E=32E=32, k=2k=2, αaux=0.01\alpha_{\text{aux}}=0.01) is now the cell of a pre-registered hypothesis on ten new seeds. Token-level FailSafe-RM cuts the worst-outage drop from 2.53 pt under conventional training to 0.70 pt. The paired difference of 1.83 pt clears the threshold of 1.0 pt (p = 1.0×10−61.0\times10^{-6}). The method also has 1.25 pt higher clean accuracy and is 0.95 pt above the exhaustive adversary.

The language models show the same effect. In the 16-expert model at matched compute (E37, Section 6.4, five new seeds), token-level training lowers the worst single-expert outage to 0.0112 nats, against 0.3424 under conventional training. One-pass uniform dropout leaves 0.1699 nats, and the clean cost of 0.0150 nats is within our margin of 0.03. With 64 experts and top-8 routing (E42, Appendix S.1) the worst single outage falls to 0.0087 nats, against 0.0630 under conventional training and 0.0408 under uniform dropout. The worst node falls to 0.0679 nats, against 0.2420 and 0.1703, with clean NLL 0.0055 nats below conventional training.

Q1, deployment and multiple outages. The protocol matches expert-parallel serving, where a lost device removes the experts it hosts. In the decoding deployment of DeepSeek-V3 each GPU hosts one expert (DeepSeek-AI, 2024). A lost GPU thus removes one expert of each MoE layer it serves (a rank outage), and a lost machine with several experts per layer removes several (a node outage). In two large research clusters the mean time to failure of a job on 1,024 GPUs was 7.9 hours (Kokolis et al., 2024). Sun et al. (2026) name the protocol we study, "continuing immediately over the reduced peer set", and set it aside without measuring it. Their system pauses until every expert is hosted again, 11 s after one rank fails, against 348 s of unavailability for a full restart. Training complements recovery. It sets the quality of serving during the recovery window and wherever no copy exists.

With several outages at once, the method protects against node outages, the correlated case. When a node fails, eight experts of every layer go together in E42 and two in E43, and the readings on node outages we fixed in advance are met (W2 gives the 64-expert numbers). For pairs of single experts, a two-expert variant of our method reaches 61.50% on the worst pair with a collapsed router, against 51.58% for the single-outage method and 57.12% for the exhaustive adversary. Bernoulli expert dropout at rate 0.5 matches our method on single outages and reaches 60.33% on that pair. Under balanced load the two are level. In the language model a reading fixed before the runs confirms that a Bernoulli mask at the same own-outage rate protects pairs better, by 0.2314 nats at ρ=0.15\rho=0.15 (one-sided p = 0.0018), at a higher clean cost.

Q2, Section 4.1 and Proposition 1. Section 4 now carries an elementary identity for one MoE layer that directly supports the method, and the old bound moved to Appendix C. The old bound motivated a pre-registered criticality diagnostic, and the method now uses router mass and the token's gates instead. The identity shows that an outage trains a token when it hits one of that token's own experts. Uniform dropout does so on a fraction k/Ek/E of its outage passes, while token-level FailSafe-RM does so at a constant rate. On the new seeds, the gain over drawing from all EE experts grows from 1.85 pt at E=8E=8 to 8.70 pt at E=64E=64, as the identity leads us to expect.

Reviewer_LyMKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FailSafe-MoE studies robustness to a single global expert outage in sparse MoE models, with token routing recomputed after an expert is removed. The paper compares uniform dropout, sampled worst-outage objectives, an exhaustive adversary, and FailSafe-RM, which selects the dropped expert in proportion to its batch router mass. The sampled-max analysis explains how increasing the candidate sample emphasizes the worst outage. In controlled single-layer settings, the exposure-weighted sampler improves worst-outage performance, particularly with skewed top-2 routing. In a 99M-parameter language model, it reduces worst-outage loss damage relative to uniform dropout over three paired seeds, although the paired test is not significant at conventional thresholds. An OLMoE audit and routing sweeps identify settings with limited headroom.

Critical strengths
  • The method is simple and has a clear motivation. Sampling outages by observed router mass is inexpensive and directly targets experts to which more tokens are exposed. The analysis clarifies why uniform dropout can under-emphasize the tail of the outage distribution.
  • The authors test several important alternatives and report limitations well: routing skew, top-k, load balancing, an always-on shared expert, matched compute, and the failed pre-registered criticality measure. These controls make it easier to see where the proposal works and where it may not.
Critical weaknesses
  • The practical operating regime remains uncertain. The strongest controlled gains appear in small single-layer models with top-2 routing and skewed load. Gains shrink under more balanced routing and larger top-k; the OLMoE top-8 audit also finds limited worst-single-expert damage. The 99M language-model study is a useful extension but has only three paired seeds, with a reported paired p-value of 0.099. These data are suggestive, not decisive evidence for a broader serving advantage. So hard to generalized.
  • The comparison with alternative resilience strategies is incomplete. Stronger load balancing and an always-on shared expert already recover much of the damage in the studied settings, but neither is combined with FailSafe-RM. A system designer would also compare training-time robustness with replication or remapping, accounting for memory, latency, and outage frequency. Finally, FailSafe-RM was introduced after the pre-registration lock, so its results should remain explicitly exploratory.
Questions for authors
  • Can you show results in a setting closer to production MoEs: fine-grained experts with top-k of 4 or more (for example 8 of 64), several MoE layers, and a larger model and token budget? Does FailSafe-RM keep any advantage there?
  • How does FailSafe-RM compare with a shared always-on expert and with stronger load balancing at matched clean accuracy, and do the combinations help? Combined gains would strengthen the practical case.
  • How does training for outages compare with replicating the r most loaded experts, as expert-parallel load balancers do, at modest memory cost? The worst outages are concentrated in the most loaded experts.
  • In the LM, the iso-compute CLEAN model has much better clean NLL but larger outage damage. What is the absolute NLL under the worst outage for each model at matched compute? At what outage probability would FailSafe-RM be preferable in expectation?
  • Please add a multi-expert dropout baseline (Bernoulli masking, or growing random subsets as in Chen et al., 2023). Does it close the gap to FailSafe-RM at q = 1 and q = 2?
  • Why is the worst outage in layer 0 of the LM in all 12 models? If first-layer routing is mostly by token identity, would a simpler fix aimed at that layer work? Minor: the abstract and introduction would read better with fewer qualifiers and a clearer split between confirmed and exploratory results.
2 comments

Thank you for your review.

W1, the operating regime. The language-model result now rests on five new seeds at matched compute, in place of the three seeds and p = 0.099 you cite (E37, Section 6.4, Table 2). All three pre-registered hypotheses hold. The worst single-expert outage costs 0.0112 nats after token-level training. It costs 0.1699 after one-pass uniform dropout (one-sided p = 0.0002) and 0.3424 after conventional training (one-sided p = 0.0029). The clean cost of 0.0150 nats lies inside the pre-registered margin of 0.03. At half the rate it falls to 0.0030 nats, with a worst single outage of 0.0193. The ordering holds for rank and node outages and on out-of-domain text.

With 64 experts and top-8 routing the method keeps its advantage (Q1). The worst single outage costs 0.0087 nats after token-level training, against 0.0408 after uniform dropout and 0.0630 after conventional training. The worst node costs 0.0679 nats against 0.2420 after conventional training. Finer experts shrink the damage of conventional training itself (0.0630 nats against 0.3424 with 16 experts), so the node becomes the costly unit.

On the single-layer benchmark, the pre-registered test gives 7.06 pt over uniform dropout at E=32E=32 with a collapsed router. Under balanced load the gain is 3.96 pt (descriptive), and the pre-registered worst-outage drop falls from 2.53 to 0.70 pt. As the identity of Section 4 predicts, the benchmark gain follows E/kE/k. At E=32E=32 it is 7.06, 2.19 and 0.76 pt for k=2k=2, 4 and 8, and at 8 of 64 it is 0.77 and 0.29 pt.

W2, alternatives, memory, latency and the label. Replication and outage training complement each other (Q3). In the language model, token-level training without copies leaves 0.0112 nats at the worst single outage. Conventional training with copies of the 4 most loaded experts leaves 0.1984. On the benchmark under balanced load, token-level training without copies also beats conventional training with copies of up to 8 of 32 experts (worst-outage drop 0.70 against 0.77 pt). Outage training adds no inference latency or memory, because the served model is unchanged. Its median training step is 2% to 4% longer. Replication and standby copies cost memory, and a recovery pause in the system of Sun et al. (2026) costs 11 s to 348 s of downtime. As you note, batch-level FailSafe-RM was introduced after the pre-registration lock, and the revision keeps that label on all of its results. Token-level FailSafe-RM has its own pre-registration with thresholds, fixed before its confirmation runs.

Q1, a production-like setting. Yes, on the three tests we fixed before the runs. E42 (Appendix S.1) uses 64 experts and top-8 routing in each of the six MoE layers, with the same parameters, compute, data and five seeds as the 16-expert confirmation. Token-level training at ρ=0.15\rho=0.15 has the lowest worst single outage (one-sided p = 0.0036 against uniform dropout) and the lowest worst node outage (one-sided p < 0.001). Its clean NLL is 0.0055 nats below conventional training (p = 0.032). The worst rank costs 0.0132 nats against 0.0498 after uniform dropout and 0.0729 after conventional training. The worst pair costs 0.0228 nats against 0.0883 and 0.1097. One-pass uniform dropout lowers the worst node outage below conventional training (p = 0.004), and token-level training lowers it further. These models keep 99M parameters, and the pretrained OLMoE-1B-7B gives the larger-scale view.

In OLMoE-1B-7B (E39, Section 7), a node outage costs up to 0.1735 nats and lowers MMLU accuracy by up to 9.8 points. A short LoRA fine-tune with node outages removed 22.7% of the worst-node damage for the trained placement (E40, Appendix O).

Q2, shared expert and load balancing (E33, ten seeds, E=32E=32). Token-level FailSafe-RM combines well with a shared expert. Together they reach 64.77% worst-outage accuracy under balanced load, the highest of all configurations at E=32E=32 and k=2k=2. The combination adds 1.19 pt of worst-outage accuracy and 0.60 pt of clean accuracy to the shared expert alone. A shared expert is also a strong baseline by itself. With conventional training it reaches the worst-outage accuracy of token-level FailSafe-RM without one (0.37 pt higher, p = 0.096), at 0.62 pt higher clean accuracy. Stronger load balancing (αaux=0.1\alpha_{\text{aux}}=0.1) is 1.61 pt lower on worst-outage accuracy and 1.55 pt lower on clean accuracy. Added to token-level FailSafe-RM it changes worst-outage accuracy by −0.11-0.11 pt, while the shared expert adds 1.56 pt. We corrected the sentence in Section 6.3 of the submission on the exhaustive adversary with a shared expert. Table 21 of the submission, now Table 33, has that arm at 63.60% against 59.33% for the shared expert alone.

Q3, replication. Training protects all experts at no memory cost, and replication adds protection for the experts that have copies. In OLMoE-1B-7B, replicating the 8 most loaded experts per layer costs 12.5% extra expert memory and leaves 0.1472 of the 0.1735 nats of the worst node outage.

Q4, matched compute and the break-even probability. In the 16-expert model at matched compute, clean NLL and NLL under the worst single outage are 4.0873 and 4.4297 nats after conventional training. After one-pass uniform dropout they are 4.0862 and 4.2561. After token-level training they are 4.1023 and 4.1135 at ρ=0.3\rho=0.3, and 4.0904 and 4.1097 at ρ=0.15\rho=0.15.

Token-level training pays once the outage is active for more than a break-even share of serving time. We take it per seed as clean cost over damage reduction, then average. Against conventional training at ρ=0.15\rho=0.15 it is 1.58% for the worst single expert and 1.19% for the worst node, and two of the five seeds show no clean cost. For a randomly failed single expert, rank and node they are 27.81%, 4.60% and 2.36%. At ρ=0.3\rho=0.3 the worst-unit shares are 4.32% and 3.02%. With 64 experts there is no clean cost, so training pays at any outage probability. With a dense first layer the share for the worst node averages 5.46% [0.92%, 10.42%].

Without spares, a 40-node unit has at least one node down 21.4% to 24.5% of the time. With spare devices, token-level training at ρ=0.15\rho=0.15 is level with both baselines in expected NLL (p at least 0.459), and at ρ=0.3\rho=0.3 its expected NLL exceeds that of uniform dropout by about its clean cost (p = 0.020 to 0.027). The submitted two-pass variant needed the worst outage active over 73% of the time, which is why the method now uses one pass.

Q5, multi-expert dropout (E32, ten seeds, E=32E=32). Token-level sampling keeps the lower clean cost, and Bernoulli masks extend coverage to simultaneous outages at a higher clean cost. Worst-outage accuracy (%) under 1, 2 and 4 simultaneous outages (4: greedy worst case):

trainingcollapsed: 1 / 2 / 4balanced: 1 / 2 / 4
uniform dropout55.15 / 48.13 / 37.4659.24 / 56.00 / 50.87
Bernoulli(0.25)61.12 / 56.30 / 49.4562.52 / 61.67 / 60.10
Bernoulli(0.5)62.17 / 60.33 / 55.7063.22 / 62.61 / 61.51
growing subsets59.02 / 53.99 / 47.9061.80 / 60.58 / 58.68
exhaustive adversary62.46 / 57.12 / 46.3262.26 / 60.72 / 57.55
token-level FailSafe-RM62.21 / 51.58 / 37.1663.21 / 62.47 / 61.36
two-expert variant63.16 / 61.50 / 47.9963.40 / 62.86 / 61.89

Under balanced load token-level FailSafe-RM is level with Bernoulli(0.5) and above the other baselines. With a collapsed router, Bernoulli(0.5) leads it by 8.74 pt on the worst pair and by 18.53 pt under four outages, and the two-expert variant reaches 61.50% on the worst pair. In the language model we pre-registered three readings at ρ=0.15\rho=0.15 (E41, Appendix N). Token-level sampling has a lower clean NLL than a Bernoulli mask at the same own-outage rate, by 0.0080 nats (one-sided p = 0.0070). The mask has the lower worst-pair damage, by 0.2314 nats (one-sided p = 0.0018). On the worst single outage the difference, 0.0045 nats, favours token-level sampling (90% interval 0.0022 to 0.0079). At equal clean cost, by interpolation, token-level sampling leaves 0.0193 nats at the worst single outage against 0.0473 for the mask.

Q6, layer 0. Yes, a dense first layer works, and token-level training on top lowers the damage further. With a dense first layer as in DeepSeekMoE (E43, Appendix S.2, fixed before the runs), conventional training loses 0.0316 nats at the worst single outage instead of 0.3424 (one-sided p = 0.0039). Clean NLL is also 0.0449 nats lower. All-MoE token-level training leaves less damage than the dense layer alone (0.0193 nats at ρ=0.15\rho=0.15 against 0.0316), and the dense model has the lower NLL under that outage (4.0741 against 4.1097).

Token-level training on top lowers the worst node from 0.1574 to 0.0324 nats (one-sided p = 0.0001), and the worst single outage from 0.0316 to 0.0059 (one-sided p = 0.0004), at a clean cost of 0.0039 nats (p = 0.42). An exploratory arm finds less damage than uniform dropout with a dense first layer, at the worst single outage and node (two-sided p < 0.001). Its clean NLL is 0.0076 nats higher (p = 0.086).

In all 12 submitted models the worst outage lies in layer 0 (Table 8 of the submission, now Table 9), and the checkpoints route layer 0 nearly by token identity in every arm (E36). Outside the first layer, token-level training lowers the worst single outage to 0.0039 nats, against 0.0245 after uniform dropout (p = 8.1×10−58.1\times10^{-5}).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.