- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the robustness of sparse Mixture-of-Experts (MoE) language models to expert outages. It formulates worst-case expert-outage robustness as a minimax training objective and proposes FAILSAFE-MoE, a family of training strategies that expose the model to expert failures during training.
Critical strengths
-
A simple training strategy. FAILSAFE-RM uses router mass to prioritize outage scenarios without exhaustively searching all experts. The reported results on the 99M-parameter language model are encouraging: the worst-outage NLL increase decreases from 0.157 to 0.077 nats compared with conventional random expert dropout, while clean performance remains comparable.
-
A useful investigation of the limits of the proposed approach. The paper reports that the benefits depend on routing-load skew and become limited when expert utilization is relatively balanced. This helps characterize the conditions under which outage-aware training may be useful.
Critical weaknesses
-
Limited practical motivation. The paper assumes that an individual expert becomes unavailable while the remaining experts continue operating normally. However, it does not sufficiently establish how common this scenario is in real-world MoE serving or why existing system-level fault-tolerance mechanisms are insufficient.
-
Unclear incremental value over system-level solutions. FAILSAFE-MoE improves model quality under simulated expert outages, but does not address failure detection, recovery, or expert relocation. The paper does not clarify when outage-aware training provides meaningful benefits beyond existing system-level fault-tolerance mechanisms.
-
Limited evaluation at realistic scales. Training experiments are conducted on a 99M-parameter MoE model, while the larger OLMoE evaluation is an inference-only outage audit. It remains unclear whether the reported gains generalize to large pretrained MoE models and realistic serving environments.
Questions for authors
-
How does FAILSAFE-MoE perform when multiple experts become unavailable simultaneously? Does the proposed sampling strategy remain effective when the outage pattern involves correlated failures or multiple heavily loaded experts?
-
How much of FAILSAFE-RM's improvement comes from exposure-based weighting, as opposed to simply excluding experts with no traffic in the current batch? Can the authors isolate these two effects more clearly?
-
Beyond validation loss, can the authors evaluate whether outage-aware training affects clean-model performance on downstream tasks or generalization to unseen data, and quantify the trade-off between normal-operation quality and outage robustness?
2 comments
Q1, several experts at once. Training on single outages also protects against the loss of a whole node, the natural correlated failure, and Bernoulli masks extend coverage to pairs of experts at a higher clean cost.
A node holds eight experts of every OLMoE layer. In OLMoE-1B-7B (E39) a rank outage costs at most 0.027 nats, against up to 0.1735 for a node outage. On WikiText-103 the 24 node outages cost on average 1.18 times the sum of their single-expert damages. In our trained models, the readings on node outages that we fixed before the runs are met (eight experts of every layer in E42, two in E43). In the 16-expert model (E37) the worst node, two experts of every layer, falls from 0.6012 nats under conventional training to 0.0907, and uniform dropout leaves 0.3360.
On pairs of single experts in the 16-expert model, token-level training leaves 0.2554 nats on the worst pair against 0.3822 after uniform dropout (p = 0.133). A Bernoulli mask at the same own-outage rate extends this coverage in a pre-specified test. At its worst-pair damage is 0.2314 nats lower (one-sided p = 0.0018), while token-level sampling keeps the lower clean NLL. At the mask reaches 0.0522 nats on the worst pair at twice the clean cost.
On the benchmark (E32, ten seeds, ) the method leads under balanced load. There it beats uniform dropout, the exhaustive adversary, growing subsets and Bernoulli dropout at rate 0.25, with 62.47% on the worst pair against at most 61.67%. Under four outages it reaches 61.36% against at most 60.10%. Bernoulli dropout at rate 0.5 is level with it under balanced load. With a collapsed router, a two-expert variant of our method reaches 61.50% on the worst pair. The single-outage method reaches 51.58% there, against 57.12% for the exhaustive adversary and 60.33% for Bernoulli dropout at rate 0.5.
Q2, exposure weighting against excluding experts without traffic. The weighting carries most of the gain, and a new control isolates it.
The submission has one control, RandDrop-Live, which draws uniformly over the experts that carry traffic in the batch (submission Table 12, now Table 24). With a collapsed router it gains 2.76 pt of worst-outage accuracy over uniform dropout, against 7.06 pt for batch-level FailSafe-RM. Excluding experts without traffic thus explains about 39% of the gain. Under balanced load it gains pt.
Our new control runs the same one-pass procedure as token-level FailSafe-RM but draws the removed expert uniformly from all . Its expected objective therefore equals that of uniform dropout at . The control against uniform dropout isolates the estimator, and the method against the control isolates the weight on each token's own experts. At the method beats the control by 6.52 pt with a collapsed router and by 2.19 pt under balanced load (both p < ). The control itself is 0.54 and 1.77 pt above uniform dropout. The weight thus carries nearly all of the gain with a collapsed router, while weight and estimator contribute about equally under balanced load. The weight effect grows with the pool, from 1.85 pt at to 8.70 pt at , as the identity of Section 4 leads us to expect. A uniform draw among the token's own experts performs the same as the gate-proportional draw, so the gain comes from the token's own experts.
The per-token draw also beats two exploratory variants of uniform dropout. A higher weight gains at most 2.02 pt over . Reweighting to the expected objective of the token-level draw trails it by 2.47, 9.55 and 16.08 pt at , 32 and 64, and is level at .
Q3, downstream tasks and unseen data. We now report both. Outages hurt downstream accuracy. At its worst node, a node outage of OLMoE (E39) lowers accuracy by up to 9.8 points on MMLU, 8.8 on ARC-Challenge and 3.9 on HellaSwag.
Outage-aware fine-tuning (E40, one seed) keeps clean downstream accuracy close to clean fine-tuning, at a clean NLL cost of 0.0039 nats. Clean MMLU accuracy is 47.2% against 47.0%, ARC-Challenge 55.8% against 57.0%, and HellaSwag 76.4% against 77.0%. Accuracy under a node outage shows no consistent change.
In the language model (E37) the protection carries over to another domain. On AG News the worst single outage costs 0.0100 nats after token-level training against 0.1816 after uniform dropout. On AG News and IMDB token-level training has no higher clean NLL than conventional training. In-domain its clean cost is 0.0150 nats, within our margin of 0.03, and part 1 (W2) gives the trade-off in expected NLL.
Thank you for the review and the direct questions on motivation and systems.
W1, how common the scenario is. These statistics come from training and HPC fleets, and they show how often a serving unit of the same hardware would lose a device. In expert-parallel serving a failed device takes its experts with it. In the decoding deployment of DeepSeek-V3 each GPU hosts one expert (DeepSeek-AI, 2024). On the Delta cluster the measured availability of a GPU node is about 99.4% for A100 and 99.3% for H100 (Cui et al., 2025). In a 54-day Llama 3 pre-training run on up to 16K GPUs, about 78% of 419 unexpected interruptions were attributed to hardware (Llama Team, 2024).
With the Delta availability and independent nodes, at least one node is down 4.7% to 5.5% of the time in a unit of 8 nodes. In a unit of 40 nodes, the size of the DeepSeek-V3 decoding unit, the share is 21.4% to 24.5%.
W2, what training adds to system-level recovery. Training complements detection, recovery and relocation. It sets what the model serves between a failure and the end of recovery, and wherever no copy of the lost experts exists. The recovery system of Sun et al. (2026) pauses in that window until every lost expert is hosted again, 11 s after one rank fails, against 348 s of unavailability for a full restart. They name the alternative, "continuing immediately over the reduced peer set", and set it aside without measuring it. We measure that option offline.
Token-level training lowers what that serving costs in quality. With 64 experts and top-8 routing in six MoE layers (E42, Appendix S.1), a node outage removes 8 of the 64 experts of every layer. The worst node then costs 0.0679 nats after token-level training against 0.2420 after conventional training. One-pass uniform dropout leaves 0.1703 nats. A randomly failed node costs 0.0543 nats, against 0.1475 and 0.1153. All three readings we fixed before the runs are met. There is no clean cost in any of the five seeds, and training needs one pass per step and no extra memory.
The benefit grows with the time a unit serves degraded. Without a spare, a unit serves degraded until repair. For a randomly failed node or rank in a 40-node unit, token-level training at ρ=0.15 has lower expected NLL than conventional training (p at most 0.004). For node outages it is also lower than uniform dropout. With a dense first layer the clean cost is 0.0039 nats, and the break-even share for the worst node averages 5.46% [0.92%, 10.42%] of serving time. This is the share an outage must be active for the training to pay. With spares, recovery pauses take 0.012% to 1.2% of serving time, and token-level training improves what is served during each pause. In the 16-expert all-MoE model at ρ=0.15 its expected NLL with spares is level with conventional training and uniform dropout (p at least 0.459), and at ρ=0.3 it is higher than with uniform dropout by about its clean cost (general response, part 2).
Replication and standby copies. Training protects all experts at no memory cost, and copies add protection where memory for them exists. DeepSeek-V3 reserves 64 GPUs "for hosting redundant experts and shared experts". In OLMoE-1B-7B (E39, Section 7) a node outage costs up to 0.1735 nats on held-out web text with experts placed by index (0.191 under a random placement). Replicating the 8 most loaded experts per layer takes 12.5% extra expert memory and leaves 0.1472 nats. Int4 standby copies (25%) leave 0.0105 nats, and int8 copies (50%) remove the damage.
W3, scale. E42 gives our trained models the configuration of production MoEs, 64 experts with top-8 routing in six MoE layers (results under W2). The models we train from scratch stay at 99M parameters. In the 16-expert model at matched compute (E37, Section 6.4, Table 2) all three pre-specified hypotheses are confirmed. The worst single-expert outage costs 0.0112 nats after token-level training against 0.3424 after conventional training and 0.1699 after one-pass uniform dropout.
For the large pretrained model, a LoRA fine-tune of OLMoE with node outages (E40, Appendix O) removed 22.7% of the worst-node damage on web text for the placement used in training, from 0.1745 to 0.1349 nats. Our rule set 25% for a three-seed study, so this remains a one-seed trial. For an unseen placement it removed 6.5%, and on WikiText-103 5.5%.