acceptodds
Under review as a conference paper at ICLR 2027

Cache-Aware Joint Router Adaptation for Memory-Efficient MoE Inference

Abstract

Mixture-of-Experts (MoE) models activate few experts per token, yet their full expert sets can exceed GPU memory and require repeated weight transfers during decoding. We formulate expert-cache management as a model-side algorithmic problem and propose cache-aware post-training that jointly adapts the MoE backbone and lightweight auxiliary routers while preserving the native inference-time Top-\(K\) rule. The update-only Temporal Router learns same-layer retention across tokens without proactive loading. The full Spatio-Temporal Router adds a Spatio Router that uses the causal predecessor's hidden state to refine the temporal cache before target-layer access. We evaluate both modes on Qwen3 and GPT-OSS across GSM8K, MATH, and CommonsenseQA. Temporal Router consistently improves hit rate and reduces expert-weight traffic over matched LM-only baselines. On Qwen3, the full mode improves adjusted hit rate by 1.15–18.03 points and reduces traffic by 4.6–53.3% relative to the strongest evaluated prefetching baseline; GPT-OSS results are competitive but task-dependent. Auxiliary-only training preserves baseline accuracy but yields modest coverage gains; joint post-training achieves substantially higher coverage. Sensitivity analyses distinguish the effects of cache capacity, refinement budget, and cache-loss weight on coverage, traffic, and quality.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Mixture-of-Experts, cache-aware post-training, expert caching and prefetching

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mTNnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

I find this paper addresses a very practical use case: optimization on how to deploy MoEs on edge devises where RAM is limited and the transfer cost form DRAM to the compute unit involves latency when each token involves switching experts independently. The proposed methods to add additional routers to encourage experts to be consistent for temporally close tokens is a practical approach.

  1. They introduced a post-training method to take a trained MoE model, add two auxiliary routers in addition to the native topk router. The auxiliary routers are post-trained, along with all parameters of the model, to promote consistent experts selection per adjacent tokens and increase cache hit rate for those experts.
  2. The paper validated the method using two post-trained checkpoints and on three task benchmarks.
  3. The results include two post-trained models and the two routers set up: Table 1, the full model wins against the other prefetchers, which pay a lot of traffic, however, It is not a clear win against plain LRU (Load is worse on 2 of 3 tasks, and accuracy drops 0.6–3.3 points). The Temporal Router is stronger against plain LRU: it has load 974 / 906 / 304 on the same three tasks, which is lower on all of them, with a small accuracy loss on MATH and CommonsenseQA only.
Critical strengths
  1. The use case tackled here is very critical use case to deploy MoE models on practical resource constrained edge devices where latency of transferring experts from GPU memory to the memory of compute unit matters a lot. And the MoE models, in their original form, put no constraints on the experts being selected for the temporally close tokens. As a result, in worst case, it is possible that adjacent tokens may require to swap in all new topk experts for each new token processing per layer. This poses a serious latency bottleneck, even through MoE brings compute efficiency compared to a dense counterpart.
  2. The proposed method seems intuitive (for the temporal router) to use current tokens hidden state at layer ll to predict which experts to maintain in the cache for this layer's next processing.
Critical weaknesses
  1. The proposed method is per-task fine-tuning method. The paper fine tunes the two post-trained models (Qwen and GPT OSS) on the training portion of GSM8k, Math and commonsenseQA task data and then evaluates the results on these respective benchmarks. This poses a constraint on the practical applicability of this method - The model needs to be fine-tuned for each and every tasks. The paper lacks any results for the tasks on which the original post-trained model was evaluated and which was not included in their fine-tuning: what is the impact on those kind of task if they are not included in the fine-tuning and does this method even bring any benefit in hit rate/cache usages for such tasks? The broader scope and intuition on why every task was included in its fine-tuning is lacking.

  2. The paper fine-tunes the models (where their base parameters are updated as well in addition to the auxiliary routers) on the specific tasks. But the paper does not evaluate the models' accuracy on the benchmarks where the original post-trained models were evaluated (Like knowledge/coding or other benchmarks) to validate if the experts caching adaptation brings any reduction to their original models' capabilities to assess the risk associated .This is highly relevant because the joint adaptation using temporal router, for example, is being trained using the next token's native router predictions as target, hence the native router of each layer is also being shifted to give more predictable/consisted distribution for this adaptation. This changes the original behavior of the MoE layers.

  3. There is no /very little details on the selected form of loss function used for each of the two auxiliary routers, including why the sigmoid was used or the value selected for τ\tau ? Also, did authors consider any other form of loss functions like classification loss/KL loss or is their any ablations showing the selected loss function choice works better?

  4. The results in table 1 is not very convincing: the full model wins against the other prefetchers, which pay a lot of traffic, however, It is not a clear win against plain LRU (Load is worse on 2 of 3 tasks, and accuracy drops 0.6–3.3 points). The Temporal Router is stronger against plain LRU: it has load 974 / 906 / 304 on the same three tasks, which is lower on all of them, with a small accuracy loss on MATH and CommonsenseQA only.

Questions for authors

There are many notations used to describe the temporal/spatio routers and the loss functions used to train them. It would greatly benefit the readers to include all notations clearly at one place in section3, before using them to describe the routers.

1 comment

Dear Reviewer mTNn,

As the discussion period is approaching its end, we would greatly appreciate any further feedback on our responses, especially regarding whether our clarifications have addressed your main concerns.

Thank you again for your time and consideration!

Reviewer_LxshOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper aims to address the expert weight loading latency during the decoding stage of the Mixture-of-Expert LLMs, when a small subset of the large model weights are required for decoding. Prior works in this fields typically do not involve training of the model. The authors formulate the exper-cache management as a model problem, and adapts the MoE backbone directly to try to improve the cache management and reduce weight loading caused by cache misses. The author designed temporal predictor, which predicts the same layer demand of next token, and the spatio predictor, which predicts the next layer demand of the same token. By jointly training these routers with the LLM, the authors demonstrate that they can improve the cache hit rate in theoretical simulations.

Critical strengths
  • The author provided a clean and modular formulation of the MoE cache prediction issue. While the readability of this paper has some issues (see below), the formulation itself is clean.

  • The author performed 5 seed experiments for multiple scenarios to ensure the results are statistically meaningful.

Critical weaknesses
  • This paper misses several major papers in this area that aimed to finetune the model to improve the cache hit rate. E.g. Pre-gated MoE (ISCA 2024), ReMoE (ICML 2026), MoE-Infinity. The author did not provide any references to these papers, let alone comparisons. This significantly hurts the credibility of the paper. Cited works such as STEP and ST-MoE are also ignored.

  • The authors have not performed any real hardware measurements. IIUC from L280-L289 te experiments are only counted based on a simulator. How would the proposed spatio-temporal router behave in real world? Will it bring up real efficiency gain?

  • From Table 1, the spatio-temporal router typically creates a greater memory load per token than the temporal router, and in most cases it's even worse than naive LRU with untrained model. So why do we need the spatio-temporal router? Furthermore, it seems to degrade the accuracy for Qwen3 for most scenarios, especially at the hand-picked setting on L257.

  • The paper lacks readability in general. While the authors admit the LLM assistance on manuscript editing, it makes me wonder whether the authors have read the paper themselves as stated, because many sentences on their own are not human readable. Internal logic steps are skipped, definitions of variables are removed, and certain code flags are leaked into the main text.

    • "Model-side algorithmic problem" (L013) introduces an abstract framing without explaining concretely what is being trained or changed. From my understanding this is just finetuning the LLM
    • "The causal predecessor's hidden state" does not immediately clarify that this means the preceding layer of the same token, or the previous token’s final layer.
    • "Stacks bounded pre-access refinement on the same retention stage" (L047) obscures the simple sequence: optionally prefetch experts before computation, then decide which loaded experts to retain afterward.
    • "It does not admit arbitrary high-scoring experts from the external pool: every insertion is already available from the completed access" (L154). This is difficult to comprehend. It takes me a while to understand that this can be simplified into a single sentence that "The Temporal Router only decides which already-loaded experts to retain, and it cannot fetch additional experts".
    • The cache-loss weight is mainly defined as λ\lambda on L145 and L179, but the author starts to use sw to refer to the weight. It turns out that this is the name of the flag in the Python code in attachments.
    • "Training uses bfloat16, TF32, and gradient checkpointing, without LoRA or weight quantization." I understand the convention but this is just unclear that which tensors are activated for BF16 and which is TF32.
Questions for authors

"Main settings (B, τ\tau, sw, R) are (20, 0.03, 0.1, 15) for Qwen3 and (8, 0.05, 0.01, 6) for GPT-OSS." Why do we need to use a 10x different sw for different model? Any explanations or this is just cherrypicking?

From Figure 3, it seems like the cache hit rate improvement is done by simply removing the effective experts, which is very likely to introduce generalizability problems. Is that the main source of the improvement of the cache hit rate?

1 comment

Dear Reviewer Lxsh,

As the discussion period is approaching its end, we would greatly appreciate any further feedback on our responses, especially regarding whether our clarifications have addressed your main concerns.

Thank you again for your time and consideration!

Reviewer_KBQQOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims at reducing weight transfers during decoding by exploring expert-cache management. A Spatio-Temporal Router is proposed, where the spatio router uses the predecessor’s hidden state to refine the expert cache, while the temporal router uses the expert selection of current token to update the cache for the next token. Qwen3-30B-A3B-Instruct-2507 and GPT-OSS-20B are used for evaluation.

Critical strengths

The proposed method combines Temporal Router and Spatio Router refinement, which are independently deployable.

Critical weaknesses

Limited novelty. The proposed cache update framework is conventional practice for cache reuse and replacement. The technical contributions of the proposed criteria for temporal router and spatio-temporal router are also weak.

The proposed expert-cache management method relies on retraining, which are impractical for large scale MoE models.

For performance evaluation, the preposed method is trained on the training set and evaluated on the test set for each specific benchmark. These evaluation procedure used for BERT-like models are inappropriate for LLMs, which are designed for general-purpose tasks.

The proposed method has accuracy drop compared with baseline methods.

Questions for authors

What is the post-training overhead?

How does the model perform without using a task-specific dataset?

1 comment

Dear Reviewer KBQQ,

As the discussion period is approaching its end, we would greatly appreciate any further feedback on our responses, especially regarding whether our clarifications have addressed your main concerns.

Thank you again for your time and consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.