- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents MELINOE, a method for improving memory-constrained MoE inference by changing the model's routing behavior through finetuning. It fine-tunes the router with a cache-simulation loss that encourages reuse of a small, sequence-specific set of experts, together with a rank-matching loss that preserves the base model's expert preferences. A prompt-conditioned predictor is then trained to predict average expert usage and prefetch experts into GPU memory. Experiments on OLMoE, Phi-3.5-MoE, and Mixtral-8x7B show fewer CPU-GPU expert transfers and improved decoding throughput across several GPUs, while largely preserving downstream quality.
Critical strengths
-
The cache-simulation and rank-matching losses are well motivated. They directly target reshaping expert preferences while avoiding global router collapse, and the ablations show that the losses, rather than ordinary downstream fine-tuning, drive most of the throughput gain.
-
The throughput improvements are substantial and consistent across three MoE architectures and several hardware configurations.
-
The evaluation reports both throughput and transfer counts, and the component ablations help separate the effects of fine-tuning and prefetching. The fine-tuned checkpoints can also be combined with existing offloading methods.
Critical weaknesses
-
The method appears workload-specific and couples model training to a deployment configuration. The main results use separate checkpoints fine-tuned on each downstream dataset. Cross-task transfer is evaluated only between a small number of datasets and is weaker than in-domain tuning. In addition, the cache capacity and decay factor are part of the training loss even though cache size, quantization, and eviction policy are deployment choices. The paper shows some robustness to different VRAM budgets, but it does not establish that one model can be trained once and then used across diverse workloads and cache configurations. This makes the practical cost less clear, since each large-model run takes up to 8 hours on 2-4 H100s and the routing targets for the predictor must also be generated again.
-
The serving evaluation is limited to low batch sizes. Batch sizes only range from 1 to 4, and Figure 5 compares MELINOE only with the base model. The explanation that prior offloading systems do not support or test batched inference is not accurate for some systems such as Fiddler, which supports batched inference. At higher batch sizes, different sequences may activate a much larger union of experts, which is exactly where sequence-specific locality may become less useful. Without comparisons against offloading baselines or continuous-batching experiments, it is difficult to assess the benefit in throughput-oriented serving.
-
The additional value of the predictor is not fully established. Table 3 shows that most of the gain comes from fine-tuning, while prefetching provides a relatively small improvement despite requiring a separate embedding model, generated routing labels, and another trained MLP. The paper does not report Base Model + Prefetch performance. It is also unclear how a prediction made once from the initial prompt would work in agentic or multi-turn sessions, where tool outputs and changing context may shift the preferred experts over time.
Questions for authors
-
Can a single MELINOE checkpoint trained on a broad mixture be evaluated across Dolly15K, GSM8K, WikiText-103, and SciQ without task-specific fine-tuning?
-
How sensitive is one trained checkpoint to changing the physical cache capacity, quantization level, and eviction policy after deployment?
-
How does MELINOE compare with Fiddler and the other offloading baselines in batched scenarios?
-
What is the throughput of Base Model + Prefetch, and how much of the predictor's gain remains after including the embedding-model and prediction overhead?
Deployment Configuration Sensitivity and Larger Batch Experiment
Thank you for your constructive comments. We aim to address your feedback.
W1. Method Generality and Deployment Configuration Sensitivity
We have two key responses about MELINOE's generality to various downstream settings.
W2. Batch Size Analysis
We acknowledge that for larger batch sizes the active expert set per token across sequences would be larger. We have two responses.
W3. Prefetching Benefits
We agree that fine-tuning accounts for most of MELINOE’s throughput gain, and prefetching provides a marginal benefit based on our tested workloads. For example, in the settings we test in Section 4.3, prefetching can reduce end-to-end latency by up to 0.8 seconds, which can matter in latency critical settings. Long-horizon or agent workflows can still benefit from MELINOE fine-tuning objectives as gains seem to persist with increasing sequence length, but sequence-level prefetching would likely be less important. That being said, without enhanced sequence-level preferences, the prefetched experts are often evicted quickly by experts required by the first few tokens. The result is that cache transfers can actually marginally increase.