acceptodds
Under review as a conference paper at ICLR 2027

LiteMoE: Lightweight Multi-Layer Predictive Expert Preloading and Task-Aware MoE Inference

Abstract

Existing memory-efficient inference Mixture-of-Experts (MoE) systems offload experts to CPU memory and predict or preload experts one layer at a time. In multi-GPU and large-batch serving, these fine-grained transfers contend for PCIe bandwidth and expose expert-loading latency. We present LiteMoE, a memory-efficient MoE inference system built on two empirical properties of expert routing. First, expert activations across nearby layers are predictable enough to forecast the experts required by a group of future layers, allowing expert transfers to be coalesced and overlapped with computation. Second, tasks differ substantially in their sensitivity to routing errors, enabling selective expert loading when memory or transfer bandwidth is constrained. LiteMoE combines multi-layer expert prediction with task-aware expert selection based on routing sensitivity and predicted expert usage. Third, LiteMoE further uses task-aware scheduling to account for expert loading latency, reducing cross-request interference and SLO violations. Across multiple MoE models, workloads, batch sizes, and GPU configurations, LiteMoE improves throughput by up to 2.18 and reduces latency by up to 54% over state-of-the-art memory-efficient MoE inference systems while controlling output-quality degradation. When configured to preserve baseline accuracy, LiteMoE still achieves up to 1.85 higher throughput and 33% lower latency. These results show that exploiting cross-layer routing structure and task-dependent routing sensitivity can substantially reduce the communication cost of memory-efficient MoE inference.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Mixture-of-Experts (MoE), Expert Prediction, Expert Offloading

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_n8SXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper targets MoE inference with experts offloaded to host memory. It argues that layer-wise prefetching such as Pre-gated MoE fails in multi-GPU, large-batch settings because PCIe transfer time far exceeds compute. LiteMoE has three components: an MLP predictor that prefetches experts for the next B = 4 layers with rollback on mispredictions, task-aware selective loading that skips experts for routing-insensitive tasks, and an SLO-aware scheduler that accounts for expert-loading latency. On five MoE models across A100 and L4 GPUs, it reports up to 2.18x throughput and 40-54% lower latency than Pre-gated MoE, at a 3-9 point cosine-similarity cost.

Critical strengths
  • Figures 3 and 26 show that well-defined tasks (CLSFY, COMP) tolerate routing errors far better than open-ended ones (CONV, SUM). Turning this into per-(task, layer) profiles (Section 3.2.1) gives a natural quality-efficiency knob for offloaded serving.

  • The study spans five MoE models (8-64 experts, top-2 to top-8 routing) and two GPU platforms. It reports system metrics alongside gold-reference quality, including ROUGE-L, MT-Bench, and perplexity (Table 1, Figure 16).

  • Appendix A.5 tests the classifier on paraphrased, cross-dataset, mixed, and out-of-distribution inputs (Table 2) and separates harmful from benign errors. It also bounds end-to-end impact under optimistic and pessimistic fallback models (Figures 19-21).

  • Rollback and the lossless LiteMoE/L variant let users trade speed for exactness. Figure 10 shows a control-plane overhead of about 2 ms, under 0.2% of request latency.

Critical weaknesses
  • The main evaluation mostly does not require offloading. Per Figure 17 and Appendix A.4, at the GPU counts used, no model needs offloading on A100 and only Mixtral does on L4, contradicting Section 4. Figure 17(b) appears to show the all-resident deployment beating LiteMoE, yet it is absent from Figure 7, and the expert memory budget is never stated.
  • The gains do not come from the claimed core mechanism. Coalescing transfers (Table 3) does not reduce the bytes moved over the bandwidth-bound link shown in Figure 1, and multi-layer prediction alone cuts A100 latency by only about 2-6% (Figure 24(b), Appendix A.6). Most gains come from lossy expert skipping and a generic scheduler that the baselines did not get.
  • The quality metric and task taxonomy are weak. BERT cosine similarity stays above 0.9 for CLSFY even with every layer misrouted (Figure 3), while label agreement falls to roughly 0.5-0.8 (Figure 15). Perplexity rises 2-8% (Figure 16(e)), code, math, and reasoning tasks are absent, and no variance is reported.
  • The predictor is not positioned against prior expert-prediction work. Cross-layer expert prediction has been explored before, for instance by applying the next layer's router to the current hidden state (Eliseev & Mazur, 2023) or by predicting later experts from intermediate hidden states (ProMoE; Song et al., 2024). Most notably, Libra (Yang et al., ICLR 2026) predicts future expert activations accurately with a router-based lookahead, yet it is not cited. I would ask the authors to cite Libra and to compare their MLP predictor against a B-layer extension of its lookahead.
Questions for authors
  1. What expert memory budget is used in Figures 7 and 13? Could you add the all-resident baseline and evaluate settings where offloading is truly necessary (e.g., Mixtral on 1-2 A100s), or show gains from the freed memory at higher request rates?
  2. How is a rollback "miss" defined, and how many experts per layer does the predictor emit? Please reconcile Figure 8 with Table 3, and clarify whether the router is bypassed or executed and checked.
  3. Could the authors compare the trained MLP predictor against a training-free baseline that applies the routers of layers N+1, ..., N+B to the hidden state of layer N, extending Libra's lookahead from one to B layers? Please report routing accuracy, rollback rate, and end-to-end latency for B = 1 to 4, with a measured rather than extrapolated B = 1 point (Figure 24(a)). Because this baseline needs no routing traces, it would also show whether the MLP's advantage holds beyond the datasets used to train it.
  4. Which configuration yields the 2.18x and 54% figures? Why do Appendix A.6 and Figure 22 imply smaller gains than Figures 7, 19, and 21?
  5. Was the scheduler applied to any baseline, and how was Pre-gated MoE's pre-gate obtained for these five models? Would Pre-gated MoE with rollback be a fairer exact baseline?
Reviewer_gDTLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents LiteMoE, a memory-efficient MoE inference system that aims to reduce CPU–GPU expert-transfer overhead. The key idea is to predict the experts required by multiple future MoE layers and prefetch them together. LiteMoE further introduces task-aware selective expert loading based on offline routing-sensitivity profiling and predicted expert usage, as well as a task-aware request scheduler that accounts for expert-loading latency when making admission decisions.

Critical strengths

Reducing CPU–GPU expert-transfer overhead is important for memory-constrained MoE inference, particularly when expert loading becomes much more expensive than expert computation. The design of moving from layer-wise to multi-layer expert prediction can reduce transfer frequency and create more opportunities for communication–computation overlap. The paper explicitly studies the accuracy–efficiency trade-off, providing useful evidence about the cost of relaxing routing accuracy.

Critical weaknesses
  1. The multi-layer predictor is used not only to determine which experts to prefetch, but also to bypass the conventional router and directly use the predicted expert set as the activation decision. Thus, the system changes the model's routing behavior and is effectively performing approximate inference rather than lossless.
  2. The usage of the method is somehow limited. The benefit of task-aware expert selection and request scheduling comes from classifying requests into predefined task types and using an offline routing-sensitivity profile for each task, while real workloads are often heterogeneous, open-ended, and difficult to partition into a small number of stable task categories.
  3. The exact version of evaluated models are not specified in the paper (e.g. only ‘DeepSeek’ rather than DeepSeek-V2 or V3), and therefore the effectiveness under large-scale scenarios is unclear.
Questions for authors
  1. How well does the proposed task-aware optimization generalize to realistic heterogeneous workloads that do not fit into the predefined task categories? In particular, what happens when requests contain unseen, mixed, or continuously changing task types
  2. Could the authors specify the exact model versions and configurations used in the evaluation? Furthermore, how does LiteMoE scale to larger-scale MoE models with substantially more parameters and experts, such as newer generations of large MoE models?
Reviewer_kvX2OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LITEMoE, which involves multi-layer expert prediction, task-aware selective expert loading, and task-aware request scheduling to accelerate the MoE deployment on a CPU-GPU system.

Critical strengths
  1. CPU-GPU is a very important platform for MoE model deployment.
  2. The proposed methods are straightforward and rational.
Critical weaknesses
  1. The selected model size is small. Currently, models are grown to hundreds of GB. It is not known whether the proposed method can fit large MoE models.
  2. The selected batch size is small (b=4,8). In the server scenario, the batch size always grows to 100. In this case, the expert distribution may be more balanced.
  3. The proposed method includes expert prediction and expert loading are not new. For request scheduling, it should be evaluated on some real cases.
Questions for authors
  1. I think more theoretical analysis on prefetch is recommended. For example, how many expert updates are involved in each decoding step?
  2. Does this method also provide benefits in the prefill stage?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.