acceptodds
Under review as a conference paper at ICLR 2027

Budget-Aware Expert Loading for Memory-Constrained MoE Inference

Abstract

Mixture-of-experts (MoE) models only activate few experts per token, yet CPU-offloaded inference can still incur substantial overhead from repeatedly loading missing experts to GPU when its memory is limited. In this paper, we propose , a training-free controller that explicitly coordinates expert loading under a fixed GPU cache capacity and a soft model-wide loading budget. The controller values each load by its marginal router-mass recovery over resident alternatives and compares this gain against a shared price updated by a request-local Lyapunov queue. Coupled with global LRU eviction, the controller coordinates loading across layers and decoding steps rather than loading every missing native-route expert. From a theoretical perspective, we prove an upper bound on the total number of loadings per request, thereby ensuring that our controller is time-efficient. We then conduct extensive experiments, involving three MoE models and four tasks spanning reasoning and code generation. The results demonstrate that substantially reduces expert-weight traffic and improves decode throughput, while achieving a comparable performance. For example, on MATH L5, using Lyapunov-LRU to run Qwen3-30B-A3B on L20 achieves a speedup of 1.38× and only incurs a tiny 4.4 percentage-point accuracy drop.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mixture of experts, memory-constrained inference, expert caching, Lyapunov optimization, adaptive expert admission

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_YZwjOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper focuses on MoE inference under memory constraints. It selectively loads experts, using cached experts to replace some missing ones. It reduces CPU-GPU weight transfer and maintains the generation quality as much as possible. This work also porposes a training-free Lyapunov-LRU, integrating routing probability gain, shared dynamic loading price, and global LRU cache management. The experiment is conducted on three MoE models, comparing the accuracy and loading capacity on GSM8K, MATH, and MBPP. The experiment on MATH L5 reduces the data transfer by about 97.8%, increases the decoding throughput by 1.38 times, and the accuracy decreases from 87.50% to 83.13%.

Critical strengths
  1. Training-free method for cached experts replacement. The controller works at inference time without retraining.
Critical weaknesses
  1. The paper only reports decoding throughput but no result on request completion time. Moreover, the accuracy drops significantly. In the main configuration reported (C=32, B=2), Qwen3-30B-A3B on the GSM8K dataset reduces the correct questions from 185 to 123 and the MBPP from 175 to 67 compared to the native routing.

  2. The method lacks a theoretical connection to actual quality. g(n) measures the recovery of routing probability quality, but doesn't explain how it constrains output errors caused by expert replacement. Proximity in probability doesn't imply proximity in expert outputs. It doesn't establish an optimality or bounded guarantee for the combined admission and eviction policy either.

Questions for authors
  1. It's unclear if the acceleration benefits mainly come from dynamic budget allocation or from loading benefit indicators. Compared to Uniform-LRU, Lyapunov-LRU simultaneiously changes the way of benefit judgement and budget allocation. Are there any ablation study to separately verify the contributions of the two designs?

  2. Why the throughput improves only by about 38% while the data transfer reduces by 98%? Could you provide an expert calculation, weight transfer, controller and synchronous operation time breakdown? What is the main bottleneck of the current system, and what percentage of the time does the controller itself consume?

5 comments
AuthorsOpenReview ↗

Response to Reviewer YZwj (Part 1/3): Request time and accuracy

Thank you for your comments.

1. Request completion time and accuracy (Weakness 1)

We have aggregated request generation times from the Physical Quality–Efficiency Trade-offs experiments in Section 7. At C=32C=32, B=2B=2 in the original physical experiments, mean request generation time is 600.52 seconds for our method versus 633.52 seconds for Demand-LRU on MATH L1, a 5.21% reduction. On MATH L5, the corresponding times are 1,376.56 versus 1,845.24 seconds, a 25.40% reduction. Each comparison uses the same 160 questions from the original physical runs. These measurements show lower observed average request generation time on both MATH tasks.

The GSM8K and MBPP accuracy drops in the paper's Table 9 occur at C=32C=32, B=2B=2, where both resources are tightly constrained. As a limiting example, consider C=8C=8, B=0B=0. A minimal resident expert pool with no sustained loading budget would be expected to perform poorly. Expecting high accuracy for every possible CC and BB is unrealistic. Our experiments identify useful configurations for the resources available to a user.

Table C1 compares C32/B2, C32/B4, and C48/B2 on all four tasks.

Table C1. Qwen3 accuracy and loading in the logical experiments.

TaskCCBBAccuracy (%)Loads per step
MATH L132290.0001.431
MATH L132498.1253.704
MATH L1482100.0002.005
MATH L532282.5001.846
MATH L532488.1253.972
MATH L548288.1251.998
GSM8K32261.5001.623
GSM8K32454.5003.210
GSM8K48289.0001.858
MBPP32233.5001.244
MBPP32439.0003.450
MBPP48274.0001.913

Increasing BB at C=32C=32 improves accuracy in most evaluated settings, while increasing CC to 48 with B=2B=2 yields clear improvements across all four datasets. These results support using a larger loading budget when the cache is small, and retaining a small budget when more cache is available.


AuthorsOpenReview ↗

Response to Reviewer YZwj (Part 2/3): Router benefit and benefit-score ablation

2. Router-based benefit and quality (Weakness 2)

The quality rationale is to recover the computation selected by the pretrained router under limited memory and loading budgets. The router provides the model's learned preference over experts for the current hidden state. For a fixed input and cache state, g(n)g(n) measures the additional router mass recovered by loading missing experts, so it rewards returning toward the model's native route. In our comparison of router top-KK selection with random expert selection, router-based selection gives substantially better task performance. We therefore regard the effectiveness of the native router as a reasonable assumption.

Corollary 1 in Appendix A.1 makes this connection explicit. At a fixed input with distinct router probabilities, every positive-gain addition replaces an expert outside the native route with a missing native-route expert. At zero loading price, the native top-KK route is fully recovered. The quality recovery under tight resource constraints in Table C1 and the benefit-score ablation in Table C2 support using this routing-recovery signal to retain task performance.

3. Benefit score and budget control (Question 1)

The benefit score helps decide which expert loads are worth making. The controller adjusts loading to the user's budget BB. We test these roles separately.

First, we keep the dynamic controller and B=2B=2 unchanged and compare two benefit scores. The alternative score uu uses the router scores of newly loaded experts that are actually selected for execution. Our gg also accounts for how much these experts improve on the experts already available in the cache. Both methods use the same candidate experts ranked by the router. Table C2 includes these dynamic-controller comparisons and the completed fixed-price variants.

Table C2. Benefit-score and pricing comparisons. Dynamic pricing uses B=2B=2.

Task / cacheScorePricing ruleAccuracy (%)Loads per step
GSM8K / C48ggDynamic89.0001.858
GSM8K / C48uuDynamic85.0001.947
GSM8K / C48ggFixed, α=0.150980\alpha=0.15098093.0000.941
GSM8K / C48uuFixed, α=0.200000\alpha=0.20000095.0001.053
MATH L5 / C32ggDynamic82.5001.846
MATH L5 / C32uuDynamic79.3751.801
MATH L5 / C32ggFixed, α=0.160000\alpha=0.16000087.5001.721
MATH L5 / C32uuFixed, α=0.205000\alpha=0.20500087.5001.776

Within the dynamic-controller pairs, gg improves accuracy by 4 percentage points on GSM8K and 3.125 percentage points on MATH L5 under the same B=2B=2 target. This supports evaluating a load by its improvement over the experts already available in the cache.


AuthorsOpenReview ↗

Response to Reviewer YZwj (Part 3/3): Dynamic budget control and decoding time

3. Benefit score and budget control (continued; Question 1)

Second, we test whether a fixed price continues to meet the requested budget when the cache becomes smaller. We choose prices using loading costs on separate development questions at C48 and then keep them unchanged at C32. Table C3 compares the fixed prices with our dynamic controller using the same B=2B=2 target.

Table C3. Budget control when fixed prices calibrated at C48 are reused at C32. Lyapunov-LRU uses B=2B=2.

TaskCCMethodLoads per step
GSM8K48Fixed price0.941
GSM8K48Lyapunov-LRU1.858
GSM8K32Fixed price2.921
GSM8K32Lyapunov-LRU1.623
MATH L548Fixed price1.987
MATH L548Lyapunov-LRU1.998
MATH L532Fixed price9.761
MATH L532Lyapunov-LRU1.846

The fixed prices meet the B=2B=2 target at C48 but exceed it when reused at C32. Our dynamic controller keeps the realized loading rate below B=2B=2 at both cache sizes. These experiments show the benefit score's role in preserving quality and the controller's role in adapting loading to the user's budget.

4. Transfer reduction and total decoding time (Question 2)

The 97.8% reduction measures transferred expert bytes, while throughput reflects the complete decoding process. Reducing expert transfers decreases one component of decoding time. The remaining computations and synchronization do not decrease in proportion to the transferred bytes, so total decoding time falls by a smaller percentage. In the paper's Table 2, MATH L5 throughput rises from 2.662 to 3.683 tokens/s. This corresponds to a decrease from 375.7 to 271.5 ms per token, or 27.7% less decoding time and 38.4% higher throughput. These measurements include expert loading, attention, expert computation, routing, synchronization, and controller execution.

Reviewer_WkdAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies CPU-offloaded MoE inference and asks which expert cache misses are worth loading. Lyapunov-LRU loads a missing expert only when its marginal router-mass recovery over resident experts exceeds a price set by a virtual queue shared across layers and steps, and it evicts with global LRU. The authors bound cumulative loads by BT+V/L+KBT + V/L + K. On Qwen3-30B-A3B offloaded to an L20, they report about 1.38× higher decode throughput on MATH at a 4.4–7.5 point accuracy cost.

Critical strengths
  1. The rule is simple and training-free. By Lemma 1, each decision reduces to thresholding at most KK sorted gains, and the shared queue lets all layers compete for one loading budget.
  2. With a moderate cache, the method works well. At C=48C=48, B=2B=2, Qwen3-30B-A3B nearly matches native routing on MATH L1/L5 and GSM8K with about 2 loads per step instead of 36–44, while Uniform-LRU loses far more accuracy (Tables 8–9).
  3. The full per-configuration results in Tables 4–9 make the claims easy to check.
Critical weaknesses
  1. The 1.38× speedup is measured in tokens/s, but approximate routing makes generations longer. From Tables 8–9 (C=32C=32, B=2B=2), decode steps per question grow by about 30% on MATH L1, 13% on L5, and 2× on GSM8K and MBPP. This would reduce the end-to-end gain to roughly 1.04× and 1.22× on MATH and could make GSM8K/MBPP slower, so per-request latency should be reported.
  2. The accuracy claims are overstated. At C=32C=32, Qwen3-30B-A3B drops from 92.5% to 61.5% on GSM8K and from 87.5% to 33.5% on MBPP (Table 9). All results are single greedy runs on 160–200 questions without confidence intervals, and some cells suggest considerable noise; for example, approximate routing beats native routing 87 to 63 on Qwen1.5 MATH L1 (Table 4).
  3. The comparisons are not load-matched, and an obvious alternative is missing. Because the queue starts at zero, Lyapunov-LRU can exceed BTBT by up to V/L+KV/L+K loads per request; on DeepSeek MBPP its realized rate is 2.421 vs. 1.371 for Uniform-LRU (Table 7). More importantly, there is no comparison with predictive prefetching, which hides transfers without changing the route. Libra (Yang et al., ICLR 2026), for instance, predicts next-layer experts by running the next gate on the current hidden states, with 86–92% accuracy on Qwen3-MoE at negligible cost. Since Demand-LRU needs only about two loads per layer per token (Table 8), such prefetching could plausibly recover much of the speedup with no accuracy loss. Also missing are Cache-Prior, a fixed-price threshold, a B=0B=0 baseline, and sensitivity to VV.
  4. Proposition 1 is the standard virtual-queue bound, and Uniform-LRU satisfies A≤BTA \le BT by construction, so the theory does not distinguish the method. In the physical runs, non-transfer overhead dominates (about 270 ms per token for Lyapunov-LRU), and the paper does not consider an FP8 checkpoint that would fit on the L20 without offloading.
Questions for authors
  1. Could you report per-request latency, generation length, and cap-hit rates for the physical runs?
  2. Could you compare methods at matched realized loads, including Demand-LRU with lookahead prefetching (e.g., as in Libra), a fixed-price rule, and B=0B=0?
  3. Can you add confidence intervals and explain why approximate routing beats native routing on Qwen1.5?
  4. How was V=3500V = 3500 chosen, and do you have physical results at C=48C=48?
6 comments
AuthorsOpenReview ↗

Response to Reviewer WkdA (Part 1/5): Generation length and request time

Thank you for your comments.

1. Generation length and request time (Weakness 1, Question 1)

The reported length increase occurs at C=32C=32, B=2B=2, where both the cache and loading budget are small. Restricting access to experts this strongly can weaken the model's ability to solve the task and to emit the end-of-sequence token at the appropriate point. This can delay termination and produce longer outputs.

We have aggregated generation lengths and measured times from the Physical Quality–Efficiency Trade-offs experiments in Section 7. The results below come from the same original physical experiments at C=32C=32, with B=2B=2, V=3500V=3500 for Lyapunov-LRU. Each task uses the same 160 questions for both methods.

Table B1. Request statistics from the physical experiments at C=32C=32, with B=2,V=3500B=2,V=3500 for Lyapunov-LRU.

TaskMethodMean time (s)Mean tokensAccuracy (%)
MATH L1Demand-LRU633.521,704.4799.375
MATH L1Lyapunov-LRU600.522,189.8091.875
MATH L5Demand-LRU1,845.244,892.8687.500
MATH L5Lyapunov-LRU1,376.565,043.2283.125

The fractions of requests reaching the generation limit are 0.625% and 11.25% for Demand-LRU and Lyapunov-LRU on MATH L1, and 17.50% and 24.375% on MATH L5.

Relative to Demand-LRU, generated tokens increase by 28.47% on MATH L1 and 3.07% on MATH L5, while the observed total generation time decreases by 5.21% and 25.40%, respectively. The measured times already include the longer outputs. Thus, the throughput advantage translates into lower observed mean request generation time on both tasks after accounting for the observed output lengths.

Table B2 compares mean generated tokens at C32/B2, C32/B4, and C48/B2 in the logical evaluations, using native routing as the reference.

Table B2. Mean generated tokens per question for Qwen3-30B-A3B in the logical evaluations.

TaskCCBBNative routing mean tokensLyapunov-LRU mean tokensLength ratio to native
MATH L13221,655.212,148.151.298×1.298\times
MATH L13241,655.211,713.521.035×1.035\times
MATH L14821,655.211,681.461.016×1.016\times
MATH L53224,648.575,250.181.129×1.129\times
MATH L53244,648.574,746.871.021×1.021\times
MATH L54824,648.574,766.721.025×1.025\times
GSM8K322868.601,813.182.087×2.087\times
GSM8K324868.601,485.281.710×1.710\times
GSM8K482868.60897.741.034×1.034\times
MBPP3222,469.414,956.922.007×2.007\times
MBPP3242,469.413,790.191.535×1.535\times
MBPP4822,469.413,009.741.219×1.219\times

Both C32/B4 and C48/B2 produce shorter outputs than C32/B2 on all four tasks. At C48/B2, mean output lengths are close to native routing on both MATH tasks and GSM8K. These results show that increasing the loading budget or cache capacity substantially alleviates length inflation in the tested configurations.


AuthorsOpenReview ↗

Response to Reviewer WkdA (Part 2/5): Accuracy under resource limits and native routing

2. Quality under tight resource limits (Weakness 2)

The GSM8K and MBPP accuracy drops in the paper's Table 9 occur at C=32C=32, B=2B=2, where both resources are tightly constrained. As a limiting example, consider C=8C=8, B=0B=0. A minimal resident expert pool with no sustained loading budget would be expected to perform poorly. Expecting high accuracy for every possible CC and BB is unrealistic. Our experiments identify useful configurations for the resources available to a user.

Table B3 compares C32/B2, C32/B4, and C48/B2 on all four tasks.

Table B3. Qwen3 accuracy and loading in the logical experiments.

TaskCCBBAccuracy (%)Loads per step
MATH L132290.0001.431
MATH L132498.1253.704
MATH L1482100.0002.005
MATH L532282.5001.846
MATH L532488.1253.972
MATH L548288.1251.998
GSM8K32261.5001.623
GSM8K32454.5003.210
GSM8K48289.0001.858
MBPP32233.5001.244
MBPP32439.0003.450
MBPP48274.0001.913

Increasing BB at C=32C=32 improves accuracy in most evaluated settings, while increasing CC to 48 with B=2B=2 yields clear improvements across all four datasets. These results support using a larger loading budget when the cache is small, and retaining a small budget when more cache is available.

3. Native routing and task accuracy (Weakness 2, Question 3)

Native routing does not guarantee the most accurate answer on every question. Changing the expert route can change token predictions and the subsequent generation trajectory, even with greedy decoding. This can turn an incorrect answer into a correct one on some questions, so approximate routing can outperform native routing on a particular evaluation set.


AuthorsOpenReview ↗

Response to Reviewer WkdA (Part 3/5): Loading budgets and additional baselines

4. Loading budgets and additional baselines (Weakness 3, Question 2)

BB is the user's requested average loading allowance. A policy need not exhaust it. Comparing policies at the same BB tests how effectively they use that allowance. Since realized loading depends on the generated sequence, we report both loads per step and total decoding loads. Our bound A≤BT+V/L+KA\le BT+V/L+K limits excess loading to a constant that does not grow with request length. The permitted excess in average loading therefore decreases as generations grow longer. In the vast majority of evaluated configurations, the realized loading rate does not substantially exceed the requested budget.

On DeepSeek MBPP at C32/B2, our rate is 2.421 versus Uniform-LRU's 1.371, but over the same 200 questions total decoding loads are 34,326 versus 39,250, with accuracy of 41.0% versus 25.5%. Thus, higher accuracy uses 12.55% fewer total loads. On Qwen1.5/GSM8K at C48/B4, accuracy is 36.5% versus 19.5%, with a lower rate, 3.465 versus 3.491, and fewer total loads, 194,648 versus 248,573. These gains are obtained without greater total loading cost.

We evaluated Cache-Prior using J=2J=2, following its paper's setting for granular MoEs, and η=0.5\eta=0.5, a value used in its reported experiments. At C32, the realized loading rates are 11.489 on GSM8K and 9.928 on MATH L5, well above the B=2B=2 loading target considered here. These fixed paper settings therefore do not meet this deployment's loading requirement, so their accuracies are not treated as a budget-matched comparison.

We test budget control by reducing the cache from C48 to C32. For the fixed-price baseline, we choose prices using loading costs on separate development questions at C48 and then keep them unchanged at C32. Table B4 compares the fixed prices with our dynamic controller using the same B=2B=2 target.

Table B4. Budget control when fixed prices calibrated at C48 are reused at C32. Lyapunov-LRU uses B=2B=2.

TaskCCMethodLoads per step
GSM8K48Fixed price0.941
GSM8K48Lyapunov-LRU1.858
GSM8K32Fixed price2.921
GSM8K32Lyapunov-LRU1.623
MATH L548Fixed price1.987
MATH L548Lyapunov-LRU1.998
MATH L532Fixed price9.761
MATH L532Lyapunov-LRU1.846

The fixed prices meet the B=2B=2 target at C48 but exceed it when reused at C32. Our dynamic controller keeps the realized loading rate below B=2B=2 at both cache sizes. These results show that the dynamic controller adapts to the change in cache capacity while retaining the same budget setting.


Reviewer_AR5xOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

MoE LLMs are known for its large memory footprint. Expert offloading, while reduces the memory demand, also introduces a huge latency due to the limited PCIe bandwidth. Under a fixed expert cache, rather than preseving the exact expert to fetch, the authors propose the Lyapunov-LRU method, a training-free method that decides whether an expert load is necessary. It evaluates the benefit of loading the top-n expert by estimating the ratio of probability recovered relative to the canonical scenario, and compare it against a budget that is shared across all layers. Empirically, the authors have recorded up to 1.38x higher throughput than the traditional LRU cache.

Critical strengths
  • Writing is clear, the manuscript is well structured and easy to follow. Figure 2 clearly explains the algorithm.

  • The algorithm itself is clean and simple. The training-free nature of this algorithm ensures that the algorithm can be applied easily without any training cost.

Critical weaknesses
  • The soundness of the experiments may have some major issues.

    • In Table 2 we have reported the physical efficiency on the per token throughput, which is good. However, it seems like the proposed algorithm may significantly increase the number of tokens generated for every question. In Table 8, if I understands correctly, T = A/\rho as denoted, and a simple calculation shows that the proposed method may create 343k+ tokens on Math L1, which is much greater than the 264k in baseline. Why? Taking this into the consideration, will the efficiency gain translates into the wall-clock savings in reality? It should also be noted that the accuracy is worse than proposed.
    • In Table 9, it seems like on GSM8k and MBPP, the accuracy of the proposed algorithm is significantly degraded. Besides, the number of tokens generated is ~2x higher?
    • The C=32 setting seems to be unnecessary on L20. Calculating the size of the expert cache, a C=32 expert cache only uses 13.5GB memory, and the non-expert weights are quite small (~3GB), which leaves plenty of spaces on the GPU. As a result, the throughput gain in table 2 does not look that useful because C can be much larger.
  • Limited comparison against prior works. L459 cites the Cache-Prior work but there's no empirical comparison at all. Multiple other works in the past has also investigated skipping/replacing/prefetching the experts and worth discussion, e.g. FasterMoE (PPoPP '22), Fiddler (ICLR 2025), Not All Experts are Equal (ACL 2024). None of them are discussed, let alone comparisons.

Questions for authors
  • See above for a plethora of questions regarding the soundness of this paper and comparison against other works.
  • Why is V=3500 selected for all models and tasks?
3 comments
AuthorsOpenReview ↗

Response to Reviewer AR5x (Part 1/3): Generation length and request time

Thank you for your comments.

1. Generation length and request time (Weakness 1, Question 1)

The reported length increase occurs at C=32C=32, B=2B=2, where both the cache and loading budget are small. Restricting access to experts this strongly can weaken the model's ability to solve the task and to emit the end-of-sequence token at the appropriate point. This can delay termination and produce longer outputs.

We have aggregated generation lengths and measured times from the Physical Quality–Efficiency Trade-offs experiments in Section 7. At C=32C=32, with B=2B=2, V=3500V=3500 for Lyapunov-LRU, each task uses the same 160 questions for both methods. Relative to Demand-LRU, generated tokens increase by 28.47% on MATH L1 and 3.07% on MATH L5, while the observed total generation time decreases by 5.21% and 25.40%, respectively. The measured times already include the longer outputs. Thus, the throughput advantage translates into lower observed mean request generation time on both tasks after accounting for the observed output lengths.

Table A1 compares mean generated tokens at C32/B2, C32/B4, and C48/B2 in the logical evaluations, using native routing as the reference.

Table A1. Mean generated tokens per question for Qwen3-30B-A3B in the logical evaluations.

TaskCCBBNative routing mean tokensLyapunov-LRU mean tokensLength ratio to native
MATH L13221,655.212,148.151.298×1.298\times
MATH L13241,655.211,713.521.035×1.035\times
MATH L14821,655.211,681.461.016×1.016\times
MATH L53224,648.575,250.181.129×1.129\times
MATH L53244,648.574,746.871.021×1.021\times
MATH L54824,648.574,766.721.025×1.025\times
GSM8K322868.601,813.182.087×2.087\times
GSM8K324868.601,485.281.710×1.710\times
GSM8K482868.60897.741.034×1.034\times
MBPP3222,469.414,956.922.007×2.007\times
MBPP3242,469.413,790.191.535×1.535\times
MBPP4822,469.413,009.741.219×1.219\times

Both C32/B4 and C48/B2 produce shorter outputs than C32/B2 on all four tasks. At C48/B2, mean output lengths are close to native routing on both MATH tasks and GSM8K. These results show that increasing the loading budget or cache capacity substantially alleviates length inflation in the tested configurations.


AuthorsOpenReview ↗

Response to Reviewer AR5x (Part 2/3): Quality under resource limits and cache capacity

2. Quality under tight resource limits (Weakness 1, Question 1)

The GSM8K and MBPP accuracy drops in the paper's Table 9 occur at C=32C=32, B=2B=2, where both resources are tightly constrained. As a limiting example, consider C=8C=8, B=0B=0. A minimal resident expert pool with no sustained loading budget would be expected to perform poorly. Expecting high accuracy for every possible CC and BB is unrealistic. Our experiments identify useful configurations for the resources available to a user.

Table A2 compares C32/B2, C32/B4, and C48/B2 on all four tasks.

Table A2. Qwen3 accuracy and loading in the logical experiments.

TaskCCBBAccuracy (%)Loads per step
MATH L132290.0001.431
MATH L132498.1253.704
MATH L1482100.0002.005
MATH L532282.5001.846
MATH L532488.1253.972
MATH L548288.1251.998
GSM8K32261.5001.623
GSM8K32454.5003.210
GSM8K48289.0001.858
MBPP32233.5001.244
MBPP32439.0003.450
MBPP48274.0001.913

Increasing BB at C=32C=32 improves accuracy in most evaluated settings, while increasing CC to 48 with B=2B=2 yields clear improvements across all four datasets. These results support using a larger loading budget when the cache is small, and retaining a small budget when more cache is available.

3. Cache capacity and physical evaluation (Weakness 1, Question 1)

The relevant constraint is the memory available to the user. For example, users with an RTX 4090 or RTX 3070 must choose a cache that fits their available memory. They cannot simply increase CC without regard to that limit. We use C=32C=32 as a controlled cache allocation in our physical experiments to study restricted-memory deployments. Users with more available memory can choose a larger CC for a better quality–loading trade-off.

We also conducted physical experiments with Qwen3-30B-A3B at C32, C48, C64, and C80, keeping B=2B=2 and V=3500V=3500 fixed. Each task uses the same 32 questions across cache sizes. Table A3 reports accuracy and the logical decoding loading rate ρ\rho.

Table A3. Physical experiments with Qwen3-30B-A3B and Lyapunov-LRU at B=2,V=3500B=2,V=3500.

TaskCCAccuracy (%)Decode loading rate ρ\rho
GSM8K3256.2501.5564
GSM8K4887.5002.0182
GSM8K6490.6251.9461
GSM8K8093.7501.5885
MATH L53275.0001.7537
MATH L54887.5001.9971
MATH L56490.6251.8600
MATH L58087.5001.3202

All three larger-cache configurations achieve higher accuracy than C32 on both tasks while maintaining low loading rates. More available cache memory therefore provides a practical way to recover task accuracy while keeping the requested loading budget unchanged. Users can choose the cache capacity according to their available memory.


AuthorsOpenReview ↗

Response to Reviewer AR5x (Part 3/3): Related work and sensitivity to V

4. Cache-Prior and related work (Weakness 2, Question 1)

We evaluated Cache-Prior using J=2J=2, following its paper's setting for granular MoEs, and η=0.5\eta=0.5, a value used in its reported experiments. At C32, the realized loading rates are 11.489 on GSM8K and 9.928 on MATH L5, well above the B=2B=2 loading target considered here. These fixed paper settings therefore do not meet this deployment's loading requirement, so their accuracies are not treated as a budget-matched comparison.

Fiddler is already discussed in Section 8. It preserves native routing and chooses between CPU execution and GPU weight transfer, whereas our method decides which expert loads to admit under a loading budget. We will add FasterMoE and Not All Experts are Equal to the related-work discussion. FasterMoE targets distributed MoE training and inter-GPU communication. Not All Experts are Equal reduces model size and computation through expert pruning and skipping. Because these methods address different execution settings or optimization decisions, we do not include direct system-level comparisons with them in the present evaluation.

5. Choice and sensitivity of VV (Question 2)

We use V=3500V=3500 as a common setting across models and tasks. VV balances router-mass recovery against accumulated loading pressure. At a fixed queue state ZZ, increasing VV lowers the loading price λ=LZ/V\lambda=LZ/V and increases the finite allowance V/L+KV/L+K. Table A4 reports the completed sensitivity sweeps at C48, with B=4B=4 on GSM8K and B=2B=2 on MATH L5. Within each task, CC and BB are fixed and only VV changes. The total request-load counts include 2,304 initialization loads, while the per-step rates count decoding loads.

Table A4. Effect of VV at C=48C=48, holding BB fixed within each task.

TaskBBVVAccuracy (%)Loads per stepMean total request loads
GSM8K487571.5003.7115,722.570
GSM8K43,50083.5003.9735,762.575
GSM8K414,00091.0004.0575,666.535
MATH L5287588.7501.93511,950.806
MATH L523,50088.1251.99811,823.531
MATH L5214,00088.7502.00911,913.981

Larger VV increases the realized loading rate in both tasks. GSM8K quality improves while MATH L5 quality remains stable. Total loads per request stay within a narrow range because generation length also changes. At V=14000V=14000, the aggregate rates exceed BB by only 1.42% and 0.44%, respectively. All 1,080 evaluated request endpoints satisfy A≤BT+V/L+KA\le BT+V/L+K. These results show how VV adjusts the quality–loading trade-off, with V=3500V=3500 providing a common setting that operates close to the requested loading target.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.