- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper targets MoE inference with experts offloaded to host memory. It argues that layer-wise prefetching such as Pre-gated MoE fails in multi-GPU, large-batch settings because PCIe transfer time far exceeds compute. LiteMoE has three components: an MLP predictor that prefetches experts for the next B = 4 layers with rollback on mispredictions, task-aware selective loading that skips experts for routing-insensitive tasks, and an SLO-aware scheduler that accounts for expert-loading latency. On five MoE models across A100 and L4 GPUs, it reports up to 2.18x throughput and 40-54% lower latency than Pre-gated MoE, at a 3-9 point cosine-similarity cost.
Critical strengths
-
Figures 3 and 26 show that well-defined tasks (CLSFY, COMP) tolerate routing errors far better than open-ended ones (CONV, SUM). Turning this into per-(task, layer) profiles (Section 3.2.1) gives a natural quality-efficiency knob for offloaded serving.
-
The study spans five MoE models (8-64 experts, top-2 to top-8 routing) and two GPU platforms. It reports system metrics alongside gold-reference quality, including ROUGE-L, MT-Bench, and perplexity (Table 1, Figure 16).
-
Appendix A.5 tests the classifier on paraphrased, cross-dataset, mixed, and out-of-distribution inputs (Table 2) and separates harmful from benign errors. It also bounds end-to-end impact under optimistic and pessimistic fallback models (Figures 19-21).
-
Rollback and the lossless LiteMoE/L variant let users trade speed for exactness. Figure 10 shows a control-plane overhead of about 2 ms, under 0.2% of request latency.
Critical weaknesses
- The main evaluation mostly does not require offloading. Per Figure 17 and Appendix A.4, at the GPU counts used, no model needs offloading on A100 and only Mixtral does on L4, contradicting Section 4. Figure 17(b) appears to show the all-resident deployment beating LiteMoE, yet it is absent from Figure 7, and the expert memory budget is never stated.
- The gains do not come from the claimed core mechanism. Coalescing transfers (Table 3) does not reduce the bytes moved over the bandwidth-bound link shown in Figure 1, and multi-layer prediction alone cuts A100 latency by only about 2-6% (Figure 24(b), Appendix A.6). Most gains come from lossy expert skipping and a generic scheduler that the baselines did not get.
- The quality metric and task taxonomy are weak. BERT cosine similarity stays above 0.9 for CLSFY even with every layer misrouted (Figure 3), while label agreement falls to roughly 0.5-0.8 (Figure 15). Perplexity rises 2-8% (Figure 16(e)), code, math, and reasoning tasks are absent, and no variance is reported.
- The predictor is not positioned against prior expert-prediction work. Cross-layer expert prediction has been explored before, for instance by applying the next layer's router to the current hidden state (Eliseev & Mazur, 2023) or by predicting later experts from intermediate hidden states (ProMoE; Song et al., 2024). Most notably, Libra (Yang et al., ICLR 2026) predicts future expert activations accurately with a router-based lookahead, yet it is not cited. I would ask the authors to cite Libra and to compare their MLP predictor against a B-layer extension of its lookahead.
Questions for authors
- What expert memory budget is used in Figures 7 and 13? Could you add the all-resident baseline and evaluate settings where offloading is truly necessary (e.g., Mixtral on 1-2 A100s), or show gains from the freed memory at higher request rates?
- How is a rollback "miss" defined, and how many experts per layer does the predictor emit? Please reconcile Figure 8 with Table 3, and clarify whether the router is bypassed or executed and checked.
- Could the authors compare the trained MLP predictor against a training-free baseline that applies the routers of layers N+1, ..., N+B to the hidden state of layer N, extending Libra's lookahead from one to B layers? Please report routing accuracy, rollback rate, and end-to-end latency for B = 1 to 4, with a measured rather than extrapolated B = 1 point (Figure 24(a)). Because this baseline needs no routing traces, it would also show whether the MLP's advantage holds beyond the datasets used to train it.
- Which configuration yields the 2.18x and 54% figures? Why do Appendix A.6 and Figure 22 imply smaller gains than Figures 7, 19, and 21?
- Was the scheduler applied to any baseline, and how was Pre-gated MoE's pre-gate obtained for these five models? Would Pre-gated MoE with rollback be a fairer exact baseline?