- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper focuses on MoE inference under memory constraints. It selectively loads experts, using cached experts to replace some missing ones. It reduces CPU-GPU weight transfer and maintains the generation quality as much as possible. This work also porposes a training-free Lyapunov-LRU, integrating routing probability gain, shared dynamic loading price, and global LRU cache management. The experiment is conducted on three MoE models, comparing the accuracy and loading capacity on GSM8K, MATH, and MBPP. The experiment on MATH L5 reduces the data transfer by about 97.8%, increases the decoding throughput by 1.38 times, and the accuracy decreases from 87.50% to 83.13%.
Critical strengths
- Training-free method for cached experts replacement. The controller works at inference time without retraining.
Critical weaknesses
-
The paper only reports decoding throughput but no result on request completion time. Moreover, the accuracy drops significantly. In the main configuration reported (C=32, B=2), Qwen3-30B-A3B on the GSM8K dataset reduces the correct questions from 185 to 123 and the MBPP from 175 to 67 compared to the native routing.
-
The method lacks a theoretical connection to actual quality. g(n) measures the recovery of routing probability quality, but doesn't explain how it constrains output errors caused by expert replacement. Proximity in probability doesn't imply proximity in expert outputs. It doesn't establish an optimality or bounded guarantee for the combined admission and eviction policy either.
Questions for authors
-
It's unclear if the acceleration benefits mainly come from dynamic budget allocation or from loading benefit indicators. Compared to Uniform-LRU, Lyapunov-LRU simultaneiously changes the way of benefit judgement and budget allocation. Are there any ablation study to separately verify the contributions of the two designs?
-
Why the throughput improves only by about 38% while the data transfer reduces by 98%? Could you provide an expert calculation, weight transfer, controller and synchronous operation time breakdown? What is the main bottleneck of the current system, and what percentage of the time does the controller itself consume?
5 comments
Response to Reviewer YZwj (Part 2/3): Router benefit and benefit-score ablation
2. Router-based benefit and quality (Weakness 2)
The quality rationale is to recover the computation selected by the pretrained router under limited memory and loading budgets. The router provides the model's learned preference over experts for the current hidden state. For a fixed input and cache state, measures the additional router mass recovered by loading missing experts, so it rewards returning toward the model's native route. In our comparison of router top- selection with random expert selection, router-based selection gives substantially better task performance. We therefore regard the effectiveness of the native router as a reasonable assumption.
Corollary 1 in Appendix A.1 makes this connection explicit. At a fixed input with distinct router probabilities, every positive-gain addition replaces an expert outside the native route with a missing native-route expert. At zero loading price, the native top- route is fully recovered. The quality recovery under tight resource constraints in Table C1 and the benefit-score ablation in Table C2 support using this routing-recovery signal to retain task performance.
3. Benefit score and budget control (Question 1)
The benefit score helps decide which expert loads are worth making. The controller adjusts loading to the user's budget . We test these roles separately.
First, we keep the dynamic controller and unchanged and compare two benefit scores. The alternative score uses the router scores of newly loaded experts that are actually selected for execution. Our also accounts for how much these experts improve on the experts already available in the cache. Both methods use the same candidate experts ranked by the router. Table C2 includes these dynamic-controller comparisons and the completed fixed-price variants.
Table C2. Benefit-score and pricing comparisons. Dynamic pricing uses .
| Task / cache | Score | Pricing rule | Accuracy (%) | Loads per step |
|---|---|---|---|---|
| GSM8K / C48 | Dynamic | 89.000 | 1.858 | |
| GSM8K / C48 | Dynamic | 85.000 | 1.947 | |
| GSM8K / C48 | Fixed, | 93.000 | 0.941 | |
| GSM8K / C48 | Fixed, | 95.000 | 1.053 | |
| MATH L5 / C32 | Dynamic | 82.500 | 1.846 | |
| MATH L5 / C32 | Dynamic | 79.375 | 1.801 | |
| MATH L5 / C32 | Fixed, | 87.500 | 1.721 | |
| MATH L5 / C32 | Fixed, | 87.500 | 1.776 |
Within the dynamic-controller pairs, improves accuracy by 4 percentage points on GSM8K and 3.125 percentage points on MATH L5 under the same target. This supports evaluating a load by its improvement over the experts already available in the cache.
Response to Reviewer YZwj (Part 3/3): Dynamic budget control and decoding time
3. Benefit score and budget control (continued; Question 1)
Second, we test whether a fixed price continues to meet the requested budget when the cache becomes smaller. We choose prices using loading costs on separate development questions at C48 and then keep them unchanged at C32. Table C3 compares the fixed prices with our dynamic controller using the same target.
Table C3. Budget control when fixed prices calibrated at C48 are reused at C32. Lyapunov-LRU uses .
| Task | Method | Loads per step | |
|---|---|---|---|
| GSM8K | 48 | Fixed price | 0.941 |
| GSM8K | 48 | Lyapunov-LRU | 1.858 |
| GSM8K | 32 | Fixed price | 2.921 |
| GSM8K | 32 | Lyapunov-LRU | 1.623 |
| MATH L5 | 48 | Fixed price | 1.987 |
| MATH L5 | 48 | Lyapunov-LRU | 1.998 |
| MATH L5 | 32 | Fixed price | 9.761 |
| MATH L5 | 32 | Lyapunov-LRU | 1.846 |
The fixed prices meet the target at C48 but exceed it when reused at C32. Our dynamic controller keeps the realized loading rate below at both cache sizes. These experiments show the benefit score's role in preserving quality and the controller's role in adapting loading to the user's budget.
4. Transfer reduction and total decoding time (Question 2)
The 97.8% reduction measures transferred expert bytes, while throughput reflects the complete decoding process. Reducing expert transfers decreases one component of decoding time. The remaining computations and synchronization do not decrease in proportion to the transferred bytes, so total decoding time falls by a smaller percentage. In the paper's Table 2, MATH L5 throughput rises from 2.662 to 3.683 tokens/s. This corresponds to a decrease from 375.7 to 271.5 ms per token, or 27.7% less decoding time and 38.4% higher throughput. These measurements include expert loading, attention, expert computation, routing, synchronization, and controller execution.
Response to Reviewer YZwj (Part 1/3): Request time and accuracy
Thank you for your comments.
1. Request completion time and accuracy (Weakness 1)
We have aggregated request generation times from the Physical Quality–Efficiency Trade-offs experiments in Section 7. At C=32, B=2 in the original physical experiments, mean request generation time is 600.52 seconds for our method versus 633.52 seconds for Demand-LRU on MATH L1, a 5.21% reduction. On MATH L5, the corresponding times are 1,376.56 versus 1,845.24 seconds, a 25.40% reduction. Each comparison uses the same 160 questions from the original physical runs. These measurements show lower observed average request generation time on both MATH tasks.
The GSM8K and MBPP accuracy drops in the paper's Table 9 occur at C=32, B=2, where both resources are tightly constrained. As a limiting example, consider C=8, B=0. A minimal resident expert pool with no sustained loading budget would be expected to perform poorly. Expecting high accuracy for every possible C and B is unrealistic. Our experiments identify useful configurations for the resources available to a user.
Table C1 compares C32/B2, C32/B4, and C48/B2 on all four tasks.
Table C1. Qwen3 accuracy and loading in the logical experiments.
Increasing B at C=32 improves accuracy in most evaluated settings, while increasing C to 48 with B=2 yields clear improvements across all four datasets. These results support using a larger loading budget when the cache is small, and retaining a small budget when more cache is available.