acceptodds
Withdrawn from ICLR 2027

The Price of a Miss: Serving-Aware Expert Routing for Edge MoE Inference

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Expert-cache misses connect routing decisions to the cost of serving Mixture-of-Experts models under a GPU memory budget. We present TollRoute, a training-free routing rule that replaces a missing expert with a resident expert only whenits router logit lies within a fixed margin of the best desired expert. The rule guar-antees conditional miss non-increase and retains at least an e−γ fraction of thedisplaced expert’s routing weight for each substitution. Its serving interface useslayer–expert cache keys, admits only experts transferred to the GPU, and cali-brates the transfer/CPU split with the target platform’s expert-processing paths.The evaluation pairs per-layer and global LRU caches at equal byte budgets onGranite-3.1-1B-A400M, with a single NVIDIA B300 and sequential teacher forc-ing. At γ = 0.5, the per-layer comparison combines a mean CE increase of atmost 2.0 mnats with relative miss reductions of 10.0–19.3%. A serving case studyconnects these routing statistics to transfer, CPU execution, and routing overhead,while retaining the integer allocation of missing experts. The resulting analysisidentifies when fewer misses reduce the exposed service cost and when eliminat-ing the final miss in a layer is decisive.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
mixture of experts, expert routing, edge inference, expert offloading, cache-aware routing, hardware-aware optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vGXbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes TollRoute, a training-free routing rule for MoE inference when only some experts fit in the GPU cache. TollRoute keeps the router’s selected experts when they are already in the GPU cache. Missing experts can be replaced by cached ones, but only if the replacement’s logit is within a fixed margin of the original top-1 score. With logits and cache contents held fixed, the authors show that the rule cannot increase misses and give a lower bound on the replacement’s routing weight. They also use a cost model to divide the remaining misses between GPU loading and CPU execution. The experiments use Granite-3.1-1B-A400M and three WikiText-2 chunks, reporting miss rates and cross-entropy alongside throughput estimates from the cost model.

Critical strengths

The policy is straightforward and requires no retraining. It uses the current GPU cache state to identify missing experts and limits substitutions through a fixed logit margin.

Critical weaknesses
  1. The value of the fixed margin is unclear The experiments do not show why the margin should be based on the original top-1 score. The main comparisons are with exact top-k routing and recency bias. The reported recency setting performs worse than exact routing in both miss rate and CE, and results for the other listed setting are missing. More informative comparisons would include giving cached experts a logit bonus or setting the margin relative to the expert being replaced. These policies should be tuned under the same cache protocol and compared at matched quality loss. The routing-coefficient bound also provides no guarantee that the replacement expert produces a similar output or preserves model quality.

  2. The results do not show a net speed benefit at small quality loss With the conservative margin, Table 1 reports relative miss reductions of 10.0–19.3%, but projected throughput is unchanged at the reported precision for all per-layer cache capacities. At C = 16, a margin of 1.5 raises projected throughput from 13.67 to 14.34 tok/s, about 4.9%, while CE increases by 0.7589 nats. These are projections that leave out additional routing and dispatch costs. The 217.2% result in Table 3 uses a different fill price and a continuous balance approximation. Without end-to-end timings at matched quality, I cannot tell whether the method saves time in practice.

  3. The experimental setup and calibration descriptions do not agree Section 4 names an NVIDIA B300 and a Xeon host. Appendix A, however, says that all experiments run on an Apple M4 Max with MPS, which is also listed in Table 5. The calibration descriptions raise a similar problem. Section 3.1 requires measurements of GPU transfers and CPU expert execution, and explicitly says that SSD reads and host-memory copies are not substitutes. Appendix B then uses SSD reads and memory copies to define its rates. Please clarify which platform each result comes from and how the 0.34 ms fill price and 66.1 ms GPU term were measured. The concurrent measurements supporting the service model should also be reported.

  4. The evaluation is small, and the replay results are missing The experiments use one model and only 3,072 teacher-forced tokens from one corpus. The gains should be checked during free generation and on models with different router logit scales.The result at C = 16 and γ = 0.5 also needs an explanation. Table 1 reports 0.27 substitutions/token, while the miss-rate reduction amounts to approximately 7.26 fewer misses/token, based on 192 expert selections/token. Changes in the cache or later routing decisions could explain this difference. Section 5.2 mentions replay and reuse analyses but does not report their results. Those numbers would help show how much of the reduction comes from direct substitutions and how much comes from subsequent cache and routing changes.The vertical axis labels in Figures 3 and 4 are also cut off and should be corrected.

Questions for authors
  1. How does the proposed margin compare with a logit bonus for cached experts or a margin based on the expert being replaced? Please tune these policies under the same protocol and compare them at matched quality.
  2. Which measurements and results come from B300, and which come from M4 Max? How do the GPU-transfer, CPU-kernel, and concurrent measurements relate to the SSD and memory-copy tests in Appendix B?
  3. Please report the fixed-route replay and live forward results for the 0.27 substitutions/token setting. How much of the miss reduction comes from direct substitutions, cache changes, and later routing decisions?
Reviewer_QfwmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

TollRoute changes the routing decisions of a frozen MoE using the current expert-cache state. It replaces missing experts with resident experts when their router scores satisfy a fixed margin rule. It combines this rule with explicit cache admission and eviction semantics. The paper studies cache misses and cross-entropy on three WikiText-2 chunks and projects serving throughput under different service-cost assumptions.

Critical strengths
  • The method connects routing to actual cache residency rather than relying only on recent expert use. The explicit cache rules and bounded replacement rule make the policy easier to reason about.

  • The paper reports quality changes alongside cache misses and clearly separates throughput projections from other measurements. This transparency helps reveal when fewer misses may or may not benefit serving.

Critical weaknesses
  • The results do not establish a compelling quality and speed trade-off. At cache capacity C = 16, exact routing projects to 13.67 tokens per second. With gamma = 0.5, TollRoute has a small mean CE increase of 0.0012 nats, but projected throughput remains 13.67. Increasing gamma to 1.0 raises projected throughput by only about 1.0%, to 13.80, while CE rises by 0.0667 nats. At gamma = 1.5, projected throughput improves by about 4.9%, to 14.34, but CE rises by 0.7589 nats. The latter corresponds to roughly a 2.14-fold geometric mean perplexity ratio across the paired chunks. At C = 24, the largest projected gain is about 3.7%, with a CE increase of 0.1663 nats. Thus, the configurations with clearer projected speed gains incur material quality loss, while the low-loss configurations show no visible gain at the reported precision in the base projection. Fewer cache misses alone do not establish useful acceleration. The paper needs to show an operating point where the speed gain justifies the quality cost.

  • The evidence is too limited to resolve this concern. The main throughput projections exclude additional policy overhead. The sensitivity analysis suggests larger gains under other service costs, including possible gains at a small margin, but those assumptions do not establish the net benefit on a deployed system. The quality study also covers only one model, one corpus, and three fixed chunks. This is not enough to show that the selected margin preserves broader model capabilities. A measured end-to-end quality and speed comparison on a relevant memory-limited platform could change my assessment. Without it, I find the demonstrated practical benefit insufficient.

Questions for authors
  • Can you demonstrate a setting with a meaningful measured end-to-end throughput gain and a small quality loss, using the same configuration for both measurements? Please include policy overhead, completed transfers, synchronization, and output combination.

  • Which real deployment regime supports the more favorable service-cost assumptions in Table 3? Can you validate the small-margin benefit on that platform rather than only through the analytical projection?

  • At the selected operating point, how do CE and downstream task performance change across more text samples and domains? Please choose the margin on separate calibration data and report the resulting speed and quality together.

Reviewer_dxAxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TollRoute, a training-free routing rule for MoE inference under a limited expert-cache budget. It replaces missing experts with resident alternatives whose router logits lie within a fixed margin of the highest desired expert logit. The authors establish two local properties: substitutions cannot increase misses at fixed logits and residency, and each replacement retains at least an e−γe^{-\gamma} fraction of the displaced expert’s normalized routing coefficient. Evaluation uses frozen Granite-3.1-1B-A400M on three WikiText-2 chunks, comparing per-layer and global LRU caches at matched byte budgets. At γ=0.5\gamma=0.5, the reported per-layer results show 10.0–19.3% relative miss reductions with mean cross-entropy increases of at most 2.0 millinats. Serving performance is assessed through throughput projections and analytical sensitivity analysis.

Critical strengths
  1. Relevant routing–serving interface with simple, verifiable local guarantees. TollRoute uses actual expert residency to guide substitution, addressing a meaningful connection between routing decisions and serving costs. Its theoretical properties are straightforward to verify: Proposition 1 follows from replacing missing experts with resident experts, while Proposition 2 combines the fixed logit-margin condition with the fact that replacements lie outside the original top-kk set, so the new softmax normalizer cannot exceed the original one (lines 187–202). These guarantees make the rule interpretable, although they do not establish model-level quality preservation or long-run miss reduction.

  2. Sound byte-matched cache comparison. Comparing per-layer and global caches at equal total expert-weight capacity is a well-motivated experimental design choice (lines 251–256). It controls for memory budget when comparing cache organizations. The explicit use of layer–expert identities also makes the intended residency granularity clear. These choices strengthen the evaluation framework by reducing confounding from unequal cache capacity.

Critical weaknesses
  1. The experimental configuration is inconsistent, and end-to-end serving benefits are not demonstrated. The main text states that evaluation uses NVIDIA B300 with an Intel Xeon Platinum 6776P host (lines 244–249), whereas Appendix A states that all experiments run on an Apple M4 Max with 48 GB unified memory and the MPS backend (lines 650–653). These descriptions leave unclear which platform produced the reported results and whether the evaluated cache and service paths were implemented or emulated. Furthermore, Tables 1 and 2 report projected throughput, and Table 3 is explicitly analytical. The projections exclude additional routing and dispatch overhead, but measured values for these costs and overall end-to-end performance are not provided (lines 412–417). At γ=0.5\gamma=0.5, Table 1 shows no improvement in displayed projected throughput despite fewer misses. Consequently, the paper does not yet establish that its low-degradation operating point yields a practical serving benefit after overhead.

  2. The routing rule does not directly optimize the measured miss price. The introduction frames the contribution as turning the engine’s latency model into the router’s objective (lines 54–60). However, Algorithm 1 uses logits, residency, and a fixed margin; it does not consult BPB_P, BHB_H, the allocation q(m)q(m), or the marginal service saving Tl(m)−Tl(m−r)T_l(m)-T_l(m-r). The paper acknowledges that hardware affects allocation and future residency while the gate remains unchanged at fixed logits and residency (lines 232–235). This supports a residency-aware routing interpretation, but the stronger claim of directly pricing routing decisions is not established. In particular, the rule can substitute experts when the immediate modeled latency saving is zero on the one-fill plateau. The authors should clarify the intended claim and provide evidence that the hardware-dependent feedback materially improves the quality–latency trade-off.

Questions for authors
  1. Experimental clarification: Which hardware and backend produced each result in Tables 1–3: B300/Xeon or Apple M4 Max/MPS? Please reconcile the main text and Appendix A, identify which components were physically executed versus emulated, and specify the source of the 66.1 ms GPU term and approximately 0.34 ms fill price.

  2. Evidence supporting the serving claim: Do you have measured end-to-end latency or throughput results that include routing, residency lookup, dispatch, transfers, CPU/GPU synchronization, and output merging? In particular, does γ=0.5\gamma=0.5 provide a net benefit over exact routing after these costs are included? Please provide the overhead breakdown and variability across repeated measurements.

  3. Methodological clarification: Is TollRoute intended to optimize measured service cost directly, or to use residency as a proxy while hardware affects subsequent cache evolution? If the latter, what evidence shows that this feedback improves realized serving performance, especially when substitutions leave the immediate service cost unchanged on the one-fill plateau?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.