- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Temporal MoE, which allows at most one expert to change between consecutive tokens in each layer. When the number of resident experts equals the number of active experts (R=k), at most one new expert needs to be loaded per step. The method aims to overlap this loading with computation on the reused experts. Experiments show that Temporal MoE retains much of the quality advantage of standard MoE. The authors also evaluate hardware performance using a model with random weights and test the effects of routing constraints and distillation on released MoE models.
Critical strengths
When R=k, Temporal MoE guarantees at most one new expert load per token per layer. The model is trained with this constraint, making it suitable for memory-constrained inference. The authors compare it with dense and standard MoE models across different training budgets and expert granularities. They also run experiments with multiple random seeds and evaluate downstream tasks.
Critical weaknesses
- The hardware tests use a randomly initialized model rather than a trained Temporal MoE. Expert switching on the phone and laptop is also predefined. These results do not show that the same trained model can maintain quality, reduce memory use, and preserve inference speed.
- Overlapping expert loading with computation depends on the hardware and scheduling. The routing constraint alone cannot guarantee this overlap.
Questions for authors
- Can the authors evaluate a trained model on at least one device and report its quality, memory usage, and inference speed?
- Can the authors explain why the routing constraint would guarantee that expert loading overhead can be hidden? Does this depend on hardware bandwidth and scheduling?
2 comments
Thank you for your rebuttal, as well as for conducting the additional experiments. Your responses have satisfactorily addressed my concerns.
Thank you very much for taking the time to review, and for recognizing the memory constraints that motivate our temporal routing rule and how our experiments show its efficacy, especially in pretraining. We'd like to use this reply to clear up the weaknesses you raised, though we understand if you still hold some of them afterwards. In the revised PDF, Sections 2 and 5 now explain when the incoming load can be hidden, Section 5 now serves our trained 1.26B model before the 11B one, and Appendix D breaks down the timing on each device.
Weaknesses and questions
Random weights and predefined switching (Question 1). That is true. The serving runs use random weights and swap exactly one expert per layer on every token, which is the most the rule ever allows. So they test the hardware side, and they do not show trained quality. We have now run the on-device evaluation. It serves our pretrained 1.26B model on the A6000, the laptop and the phone with the trained router choosing the experts, and that router also swaps one expert per layer on every token. From 1k to 4k context, the A6000 averages 86% of its all-in-memory decode speed, against 79% for the 11B model (random weights). The laptop averages 95%, against 69%. The phone averages 76%, about the same as its 77% with the 11B model. Section 5 now leads with this model and treats the 11B model as the scale-up (Figure 6, Appendix D).
Whether the rule guarantees hidden loads (Question 2). You are right that the routing rule alone does not guarantee it, and that it depends on both bandwidth and scheduling. The method needs all of them together, and it is not one or the other. The rule guarantees that each layer loads at most one new expert per token. We then choose the expert size from the hardware's bandwidth ratio, so that this one load is small enough to fit behind the computation on the experts already in memory. The kernel schedules the load to run during that computation. Unifying the routing rule with the bandwidth and the scheduling is the contribution.