- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes TollRoute, a training-free routing rule for MoE inference when only some experts fit in the GPU cache. TollRoute keeps the router’s selected experts when they are already in the GPU cache. Missing experts can be replaced by cached ones, but only if the replacement’s logit is within a fixed margin of the original top-1 score. With logits and cache contents held fixed, the authors show that the rule cannot increase misses and give a lower bound on the replacement’s routing weight. They also use a cost model to divide the remaining misses between GPU loading and CPU execution. The experiments use Granite-3.1-1B-A400M and three WikiText-2 chunks, reporting miss rates and cross-entropy alongside throughput estimates from the cost model.
Critical strengths
The policy is straightforward and requires no retraining. It uses the current GPU cache state to identify missing experts and limits substitutions through a fixed logit margin.
Critical weaknesses
-
The value of the fixed margin is unclear The experiments do not show why the margin should be based on the original top-1 score. The main comparisons are with exact top-k routing and recency bias. The reported recency setting performs worse than exact routing in both miss rate and CE, and results for the other listed setting are missing. More informative comparisons would include giving cached experts a logit bonus or setting the margin relative to the expert being replaced. These policies should be tuned under the same cache protocol and compared at matched quality loss. The routing-coefficient bound also provides no guarantee that the replacement expert produces a similar output or preserves model quality.
-
The results do not show a net speed benefit at small quality loss With the conservative margin, Table 1 reports relative miss reductions of 10.0–19.3%, but projected throughput is unchanged at the reported precision for all per-layer cache capacities. At C = 16, a margin of 1.5 raises projected throughput from 13.67 to 14.34 tok/s, about 4.9%, while CE increases by 0.7589 nats. These are projections that leave out additional routing and dispatch costs. The 217.2% result in Table 3 uses a different fill price and a continuous balance approximation. Without end-to-end timings at matched quality, I cannot tell whether the method saves time in practice.
-
The experimental setup and calibration descriptions do not agree Section 4 names an NVIDIA B300 and a Xeon host. Appendix A, however, says that all experiments run on an Apple M4 Max with MPS, which is also listed in Table 5. The calibration descriptions raise a similar problem. Section 3.1 requires measurements of GPU transfers and CPU expert execution, and explicitly says that SSD reads and host-memory copies are not substitutes. Appendix B then uses SSD reads and memory copies to define its rates. Please clarify which platform each result comes from and how the 0.34 ms fill price and 66.1 ms GPU term were measured. The concurrent measurements supporting the service model should also be reported.
-
The evaluation is small, and the replay results are missing The experiments use one model and only 3,072 teacher-forced tokens from one corpus. The gains should be checked during free generation and on models with different router logit scales.The result at C = 16 and γ = 0.5 also needs an explanation. Table 1 reports 0.27 substitutions/token, while the miss-rate reduction amounts to approximately 7.26 fewer misses/token, based on 192 expert selections/token. Changes in the cache or later routing decisions could explain this difference. Section 5.2 mentions replay and reuse analyses but does not report their results. Those numbers would help show how much of the reduction comes from direct substitutions and how much comes from subsequent cache and routing changes.The vertical axis labels in Figures 3 and 4 are also cut off and should be corrected.
Questions for authors
- How does the proposed margin compare with a logit bonus for cached experts or a margin based on the expert being replaced? Please tune these policies under the same protocol and compare them at matched quality.
- Which measurements and results come from B300, and which come from M4 Max? How do the GPU-transfer, CPU-kernel, and concurrent measurements relate to the SSD and memory-copy tests in Appendix B?
- Please report the fixed-route replay and live forward results for the 0.27 substitutions/token setting. How much of the miss reduction comes from direct substitutions, cache changes, and later routing decisions?