acceptodds
Under review as a conference paper at ICLR 2027

MELINOE: Fine-Tuning Enables Memory-Efficient Inference for Mixture-of-Experts Models

Abstract

Mixture-of-Experts (MoE) model architectures can significantly reduce the number of activated parameters per token, enabling computationally efficient training and inference. However, their large overall parameter counts have precluded their widespread usage in resource-constrained settings as all of the parameters must still be loaded into GPU memory. Prior works aim to address this memory bottleneck by offloading certain experts into CPU memory and porting them to GPU memory only when they are activated. In practice, these methods suffer from the significant I/O latency incurred by expert transfer. We present MELINOE, a method that fine-tunes an MoE model to more strongly prefer activating a smaller number of experts per sequence. Caching these preferred experts in GPU memory reduces expert churn and CPU-GPU transfer overhead. MELINOE increases throughput by – over efficient baselines and up to over transfer-heavy baselines while retaining or even improving the performance of the model on a downstream task, making it a consistently reliable method for improving MoE inference efficiency.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-Experts, Inference, Efficiency, On-Device

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_CKHcOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents MELINOE, a method for improving memory-constrained MoE inference by changing the model's routing behavior through finetuning. It fine-tunes the router with a cache-simulation loss that encourages reuse of a small, sequence-specific set of experts, together with a rank-matching loss that preserves the base model's expert preferences. A prompt-conditioned predictor is then trained to predict average expert usage and prefetch experts into GPU memory. Experiments on OLMoE, Phi-3.5-MoE, and Mixtral-8x7B show fewer CPU-GPU expert transfers and improved decoding throughput across several GPUs, while largely preserving downstream quality.

Critical strengths
  1. The cache-simulation and rank-matching losses are well motivated. They directly target reshaping expert preferences while avoiding global router collapse, and the ablations show that the losses, rather than ordinary downstream fine-tuning, drive most of the throughput gain.

  2. The throughput improvements are substantial and consistent across three MoE architectures and several hardware configurations.

  3. The evaluation reports both throughput and transfer counts, and the component ablations help separate the effects of fine-tuning and prefetching. The fine-tuned checkpoints can also be combined with existing offloading methods.

Critical weaknesses
  1. The method appears workload-specific and couples model training to a deployment configuration. The main results use separate checkpoints fine-tuned on each downstream dataset. Cross-task transfer is evaluated only between a small number of datasets and is weaker than in-domain tuning. In addition, the cache capacity and decay factor are part of the training loss even though cache size, quantization, and eviction policy are deployment choices. The paper shows some robustness to different VRAM budgets, but it does not establish that one model can be trained once and then used across diverse workloads and cache configurations. This makes the practical cost less clear, since each large-model run takes up to 8 hours on 2-4 H100s and the routing targets for the predictor must also be generated again.

  2. The serving evaluation is limited to low batch sizes. Batch sizes only range from 1 to 4, and Figure 5 compares MELINOE only with the base model. The explanation that prior offloading systems do not support or test batched inference is not accurate for some systems such as Fiddler, which supports batched inference. At higher batch sizes, different sequences may activate a much larger union of experts, which is exactly where sequence-specific locality may become less useful. Without comparisons against offloading baselines or continuous-batching experiments, it is difficult to assess the benefit in throughput-oriented serving.

  3. The additional value of the predictor is not fully established. Table 3 shows that most of the gain comes from fine-tuning, while prefetching provides a relatively small improvement despite requiring a separate embedding model, generated routing labels, and another trained MLP. The paper does not report Base Model + Prefetch performance. It is also unclear how a prediction made once from the initial prompt would work in agentic or multi-turn sessions, where tool outputs and changing context may shift the preferred experts over time.

Questions for authors
  1. Can a single MELINOE checkpoint trained on a broad mixture be evaluated across Dolly15K, GSM8K, WikiText-103, and SciQ without task-specific fine-tuning?

  2. How sensitive is one trained checkpoint to changing the physical cache capacity, quantization level, and eviction policy after deployment?

  3. How does MELINOE compare with Fiddler and the other offloading baselines in batched scenarios?

  4. What is the throughput of Base Model + Prefetch, and how much of the predictor's gain remains after including the embedding-model and prediction overhead?

1 comment
AuthorsOpenReview ↗

Deployment Configuration Sensitivity and Larger Batch Experiment

Thank you for your constructive comments. We aim to address your feedback.

W1. Method Generality and Deployment Configuration Sensitivity

We have two key responses about MELINOE's generality to various downstream settings.

  • First, in addition to our task-specific checkpoints, we fine-tuned a single OLMoE checkpoint on a broader mixture of data. We evaluated this checkpoint across several downstream tasks without further fine-tuning. The fine-tuning mixture contains FineWeb-Edu (30%), Tulu-3 SFT Mixture (25%), SODA (25%), and Wikipedia (20%). Our evaluation covers 6 disjoint downstream tasks for accuracy - WikiText-103 (PPL), the Commonsense Reasoning Suite (Acc Avg.), and Dolly15K (ROUGE-L). As with the more task-specific checkpoints in the main text, we measure efficiency during autoregressive decoding and report the average speedup and reduction in CPU–GPU expert transfers relative to base OLMoE across all tasks. The following accuracy and efficiency results compare base OLMoE to our fine-tuned checkpoint. With a single checkpoint fine-tuned on this broad data mixture, MELINOE achieves a 1.59× decoding speedup over base OLMoE with 57% fewer CPU–GPU expert transfers despite task diversity. While in some cases performance marginally decreases, we believe with a better data mixture and schedule design, MELINOE can close the performance gap while maintaining its throughput gains. Creating such a general-purpose, deployment-friendly MoE is an active area that we are pursuing as we scale up our experiments and diversify our training data.
WikiText-103CSR SuiteDolly15KCache Transfer ReductionRelative Speedup
Base OLMoE10.0469.30.185--
MELINOE8.1766.80.16257%1.59$\times$
  • Second, our ablations point to the fact that MELINOE is robust to various training configurations and downstream deployments. For example, the VRAM budget ablation in Figure 11 all use a single checkpoint for each of the three models across downstream memory constraints. This suggests that it is not necessary to fine-tune multiple models for different deployment configurations. Additionally, our ablations on the soft cache capacity (Appendix D.6) and decay factor (Appendix D.7) used in our MELINOE objective functions show that many choices of both parameters yield similarly effective ultimate results. Even in the worst case for both hyperparameters, the number of transfers is significantly reduced relative to the base model. Together, these results demonstrate that the method maintains meaningful throughput gains with various choices of hyperparameter configurations and downstream constraints.

W2. Batch Size Analysis

We acknowledge that for larger batch sizes the active expert set per token across sequences would be larger. We have two responses.

  • First, we believe that memory pressure from large batch sizes affects offloading techniques very broadly. In memory-constrained deployments, GPU VRAM is shared between non-expert model weights, the expert cache, the KV cache, and intermediate activations. The KV cache grows linearly with batch size, and with higher batch size the KV cache and activations alone would consume a substantial fraction of available memory, leaving less room for any expert cache. This is a primary reason why prior offloading methods mostly benchmark in small batch settings.
  • Second, we stress-test our method by using a batch size of 16 for the coarse-grained Mixtral-8x7B. We increase the VRAM budget to 32GB to accommodate the additional KV cache and activation overhead. We add Fiddler as a basline and benchmark on both GSM8K and Dolly15K. Even in this challenging regime, MELINOE achieves throughput gains over the base model. While this is reduced from the improvement observed at smaller batch sizes, the gains remain substantial and demonstrate that the optimizations we employ in MELINOE continue to provide meaningful efficiency benefits even as batch-level expert diversity increases.
Dolly15K (t/s)GSM8K (t/s)
Base Model4.755.01
Fiddler6.866.76
MELINOE21.1925.92

W3. Prefetching Benefits

We agree that fine-tuning accounts for most of MELINOE’s throughput gain, and prefetching provides a marginal benefit based on our tested workloads. For example, in the settings we test in Section 4.3, prefetching can reduce end-to-end latency by up to 0.8 seconds, which can matter in latency critical settings. Long-horizon or agent workflows can still benefit from MELINOE fine-tuning objectives as gains seem to persist with increasing sequence length, but sequence-level prefetching would likely be less important. That being said, without enhanced sequence-level preferences, the prefetched experts are often evicted quickly by experts required by the first few tokens. The result is that cache transfers can actually marginally increase.

OLMoE/Dolly15KOLMoE/GSM8K
Base Model730718
Base Model + Prefetch734740
Reviewer_WrRDOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a method that fine-tunes an MoE model to strongly prefer activating a smaller number of experts from total available experts per sequence, where caching these experts in GPU memory would reduce the CPU-GPU transfer overhead. The paper addresses a critical bottleneck in the wider adoption of MoE models for the resource-constrained settings - the models’ all parameters must be loaded into GPU memory in order to use them flexibly to process tokens. This work explores to combat a noted gap in the prior works, aiming to address this memory bottleneck by offloading certain experts into CPU memory and porting them to GPU memory only when they are activated for any token. In practice, these methods suffer from the significant I/O latency incurred by expert transfer too frequently from CPU to GPU memory.

  1. The paper introduces two additional losses to be added along with LLM loss when fine-tuning the router of a base MoE model. The method fine-tunes the Router, lora layers and a predictor head while keeping other model parameters frozen. the two losses added: Cache miss loss to simulate a list of cached experts and to penalize the router when it picks experts outside that cached list, with the goal of nudging the router to reuse same set of experts within a sequence. Order keeping loss to penalize the router when its experts order differ from the base model's experts preference order.
  2. The paper also trains a small predictor head to predict the experts which will be preferred by each layer based on the input which can be cached and preloaded in the GPU memory.
Critical strengths
  1. The use case tackled here is a very practical step towards making MoE models more device friendly and make them deployment ready.
  2. The Throughput (Fig. 3, from the text) shows decent gains compared to the baseline: OLMoE on H100: ~22 tok/s against ~14 for the best baseline (FLoE). OLMoE on RTX 4090: ~16 against ~8 (Fiddler). Phi-3.5-MoE on A100: ~14 against ~5. While the throughput numbers are great, the paper does not show the impact on model quality under similar task specific fine-tuning set up. Which should be something to add before the method can be fully claimed to improve throguhput while maintaining or improving the model quality than baseline.
  3. The paper includes good visualization (fig 3 and fig 2) which makes it easy to understand and it has good analysis on the throughput when differing batch settings (fig 5).
Critical weaknesses
  1. In table 2, the reported accuracy numbers for the base Model row is without fine-tuning the base model on the task while MELINOE is. The quality gain (for example Phi-3.5-MoE GSM8K 55.45 → 63.37) may come from task fine-tuning itself. A plain GSM8k NLL fine-tune on the base model would be a fair comparison. Hence, the claim that the proposed method either retains or improves the base model accuracy is not quite right. The fair comparison would be to also fine-tune the base model and the showcase the throughput benefit of MELINOE while matching the base model accuracy in exact similar set up.

  2. The base models are fine-tuned per task and then evaluated per task, ex- Dolly15K (instruction following) and GSM8K (math). This set up is not close to the practical set up because the model is expected to be tuned for a set of tasks and to work on all those tasks. Hence, the results shown here where accuracy is either matched or improved while improving the throughput is only applicable for a very specific case (when model is just built for one final task). In practice, when the model is tuned for a set of tasks, whether the tuning would cause accuracy degradation (as the router is being updated as per multiple tasks' experts preferences) for some tasks is unknown. Specifically for the case where the set of tasks include diversity: some tasks might have experts concentration per sequence while others might have spread out splits among experts.

Questions for authors

NA

1 comment
AuthorsOpenReview ↗

NLL-Only Control and Single Checkpoint Experiment

Thank you for your review and support of the paper's strengths. We address your remaining comments below.

W1. Fine-Tuned Baseline

We recreate Tables 2 and 3 but replace the base model in each baseline with a vanilla fine-tuned model (fine-tuned with NLL loss only, no auxiliary objectives) using the same training budget. While fine-tuning with NLL loss alone achieves the highest task performance, the performance gap between the vanilla fine-tuned model and MELINOE is consistently small. For example, for OLMoE on Dolly15K, the absolute difference in performance is 0.0038 in ROUGE-L score. In exchange for this marginal quality difference, MELINOE delivers the same throughput gains over prior methods as in the main text of the paper since fine-tuning with NLL alone has no demonstrable effect on throughput (Table 17).

OLMoE/Dolly15K (ROUGE-L)OLMoE/GSM8K (Acc)Phi-3.5-MoE/Dolly15K (ROUGE-L)Phi-3.5-MoE/GSM8K (Acc)
Fine-Tuned Model0.252482.520.234364.35
MELINOE0.248680.200.227063.37
Fiddler0.252482.520.234364.35
Mixtral-Offloading0.222164.360.183763.37
DeepSpeed-MoE0.252482.520.234364.35
FLoE0.234365.340.159160.78
MoE-Infinity0.252482.520.234364.35

W2. General-Purpose Checkpoint

In addition to our task-specific checkpoints, we fine-tuned a single OLMoE checkpoint on a broader mixture of data. We evaluated this checkpoint across several downstream tasks without further fine-tuning. The fine-tuning mixture contains FineWeb-Edu (30%), Tulu-3 SFT Mixture (25%), SODA (25%), and Wikipedia (20%). Our evaluation covers 6 disjoint downstream tasks for accuracy - WikiText-103 (PPL), the Commonsense Reasoning Suite (Acc Avg.), and Dolly15K (ROUGE-L). As with the more task-specific checkpoints in the main text, we measure efficiency during autoregressive decoding and report the average speedup and reduction in CPU–GPU expert transfers relative to base OLMoE across all tasks. The following accuracy and efficiency results compare base OLMoE to our fine-tuned checkpoint. With a single checkpoint fine-tuned on this broad data mixture, MELINOE achieves a 1.59× decoding speedup over base OLMoE with 57% fewer CPU–GPU expert transfers despite task diversity. While in some cases performance marginally decreases, we believe with a better data mixture and schedule design, MELINOE can close the performance gap while maintaining its throughput gains. Creating such a general-purpose, deployment-friendly MoE is an active area that we are pursuing as we scale up our experiments and diversify our training data.

WikiText-103CSR SuiteDolly15KCache Transfer ReductionRelative Speedup
Base OLMoE10.0469.30.185--
MELINOE8.1766.80.16257%1.59$\times$
Reviewer_t7McOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents MELINOE, a system for MoE inference under GPU limited memory constraints. In this setting, experts are offloaded to CPU memory but this decreases the throuhgput. MELINOE firsts fine-tunes an MoE model to more strongly prefer activating a smaller number of experts per sequence, such that the expert caching in the GPU becomes much more effective. So given a query, it is trained to exhibit stronger routing locality to favor a small number of experts. For this MELINOE modifies the loss function to include two auxiliary objectives. The first one is a cache simulation loss that penalizes CPU-to-GPU expert transfers in an memory offload setting. The second one is a rank matching loss that makes the fine-tuned router to maintain the preference ordering. It fine-tunes router weights, gate projection, and use LoRA-fine-tuning for experts, while all other weights are fozen. Combined with quantized expert weights, it exhibits higher throughput.

Critical strengths

The paper identifies that one can fine-tune MoE models to exhibit sparser, cache-friendly and more localized expert selection. This also allows the authors to predict which experts are activated with the query. The paper introduces several algorithmic ideas to enable it. In particular, it fine-tunes router weights, gate projection, and use LoRA-fine-tuning for experts, while all other weights are fozen.

The paper shows and I agree that this would be particularly useful in situations where GPU memory is limited. Because the predicted experts can be loaded in advance, the throughput is increased.

Critical weaknesses

There is a critical weakness in evaluation. Presumably, even if you use held-out dataset for validation, by fine-tuning with a specific dataset you are creating a model that is specialized for the type of queries contained in the dataset. For example, if you use only math questions (e.g, GSM8K) then the model is specialized to grade school math. So it is not surprising that the model accuracy goes up. This is an unfair evaluation because you did not report the same for the baseline. You should have reported the accuracy for all baselines using the same fine-tuned checkpoint as MELINOE.

Another point is that it is tightly coupled with a specific quantization method. So the evaluation doesn't highlight the benefit of routing locality induced by fine-tuning.

Questions for authors

What is the overhead of fine-tuning?

The evaluation shows that you can use targeted dataset to fine-tune the model, but how can you generalize this? Can you create a model that can be used for general purpose?

I have some issue with the evaluation. Presumably, even if you use held-out dataset for validation, by fine-tuning with a specific dataset you are creating a model that is specialized for the type of queries contained in the dataset. For example, if you use only math questions (e.g, GSM8K) then the model is specialized to grade school math. So it is not surprising that the model accuracy goes up. This is an unfair evaluation because you did not report the same for the baseline. You should have reported the accuracy for all baselines using the same fine-tuned checkpoint as MELINOE.

I also expect, when you fine-tune for certain task, such as math, other task (e.g., software engineering) may lose the accuracy.

Reporting throughput in Table 5 but omitting accuracy can be misleading.

Why does this have to be tied to HQQ INT4? If it isn't why is it coupled in the evaluation? In addition, in your evaluation, all other baselines use different quantization methods, and some even don't. So the comparison is not only unfair, but it is hard to isolate the effect of routing locality induced by fine-tuning.

1 comment
AuthorsOpenReview ↗

Method Generalization and Discussion of Quantized Experts

Thank you so much for your thoughtful review. We answer your questions below.

W1. Evaluation/Generalization

We agree and believe that demonstrating the method's generality is the key to making MELINOE broadly useful. We have three key responses that address your follow-up questions as well.

  • We recreate Tables 2 and 3 but replace the base model in each baseline with a vanilla fine-tuned model (fine-tuned with NLL loss only, no auxiliary objectives) using the same training budget. While fine-tuning with NLL loss alone achieves the highest task performance, the performance gap between the vanilla fine-tuned model and MELINOE is consistently small. For example, for OLMoE on Dolly15K, the absolute difference in performance is 0.0038 in ROUGE-L score. In exchange for this marginal quality difference, MELINOE delivers the same throughput gains over prior methods as in the main text of the paper since fine-tuning with NLL alone has no demonstrable effect on throughput (Table 17).
OLMoE/Dolly15K (ROUGE-L)OLMoE/GSM8K (Acc)Phi-3.5-MoE/Dolly15K (ROUGE-L)Phi-3.5-MoE/GSM8K (Acc)
Fine-Tuned Model0.252482.520.234364.35
MELINOE0.248680.200.227063.37
Fiddler0.252482.520.234364.35
Mixtral-Offloading0.222164.360.183763.37
DeepSpeed-MoE0.252482.520.234364.35
FLoE0.234365.340.159160.78
MoE-Infinity0.252482.520.234364.35
  • Tables 6 and 12 demonstrate two key factors about MELINOE’s behavior. First, throughput gains largely persist across tasks. Even on very different task distributions (e.g. applying a GSM8K fine-tuned model on Dolly15K), speedups over prior methods remain, suggesting that the shift in underlying routing behavior that makes the model more cache friendly can transfer across tasks. Second, task-specific fine-tuning largely preserves general language modeling performance. The fact that WikiText PPL remains close to the base model suggests that carefully balancing the MELINOE objectives can yield meaningful throughput gains without losing the model's generality.
  • Since our initial submission, we have been working towards a training run where we build a single OLMoE checkpoint fine-tuned on a broader variety of data and benchmarked on a mixture of different downstream tasks. The fine-tuning mixture contains FineWeb-Edu (30%), Tulu-3 SFT Mixture (25%), SODA (25%), and Wikipedia (20%). Our evaluation covers 6 disjoint downstream tasks for accuracy - WikiText-103 (PPL), the Commonsense Reasoning Suite (Acc Avg.), and Dolly15K (ROUGE-L). The following accuracy and efficiency results compare base OLMoE to our fine-tuned checkpoint. With a single checkpoint fine-tuned on this broad data mixture, MELINOE achieves a 1.59× decoding speedup over base OLMoE with 57% fewer CPU–GPU expert transfers. While in some cases performance marginally decreases, we believe with a better data mixture and schedule design, MELINOE can close the performance gap while maintaining its throughput gains.
WikiText-103CSR SuiteDolly15KCache Transfer ReductionRelative Speedup
Base OLMoE10.0469.30.185--
MELINOE8.1766.80.16257%1.59$\times$

W2. Effect of Quantization

We provide an ablation on quantized experts in Appendix D.3 (Table 15). Crucially, we find that fine-tuning, even without quantization, provides substantial throughput gains. The fine-tuned model with 8 resident experts has greater throughput than the quantized base model with 24 resident experts. The reason for quantizing is to increase the effective cache capacity within a limited VRAM budget. However, as the ablation shows, the method is not tied to a specific quantization scheme, and fine-tuning alone can provide demonstrable gains. Table 3 shows why fine-tuning helps even at a fixed cache capacity. With 16 resident experts per layer, fine-tuning OLMoE on Dolly15K reduces average transfers from 727 to 240 per layer over 64 generated tokens. In conjunction, these ideas demonstrate that the core contribution of the method is the fine-tuning procedure and not the application of quantization to the experts.

Q1. Fine-Tuning Overhead

The overhead of fine-tuning is provided in Appendix B.5. Please let us know if there are other details you would like provided.

Q2. Baseline Comparison

We have two responses to the fairness of our baseline comparison. First, we note that prior baselines benchmark similarly. Fiddler, MoE-Infinity, and FLoE all compare to prior methods without adapting their structure as the overall recipe for inference matters in demonstrating efficiency gains. Our paper attempts to align with these benchmarking practices. Second, our ablations on quantized experts (Appendix D.3), cache transfer reduction (Section 4.3), and cache eviction policy (Appendix D.8) provide controls for our method against other design choices, supporting our framework and the efficacy of fine-tuning for reducing the number of expert cache transfers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.