- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
I find this paper addresses a very practical use case: optimization on how to deploy MoEs on edge devises where RAM is limited and the transfer cost form DRAM to the compute unit involves latency when each token involves switching experts independently. The proposed methods to add additional routers to encourage experts to be consistent for temporally close tokens is a practical approach.
- They introduced a post-training method to take a trained MoE model, add two auxiliary routers in addition to the native topk router. The auxiliary routers are post-trained, along with all parameters of the model, to promote consistent experts selection per adjacent tokens and increase cache hit rate for those experts.
- The paper validated the method using two post-trained checkpoints and on three task benchmarks.
- The results include two post-trained models and the two routers set up: Table 1, the full model wins against the other prefetchers, which pay a lot of traffic, however, It is not a clear win against plain LRU (Load is worse on 2 of 3 tasks, and accuracy drops 0.6–3.3 points). The Temporal Router is stronger against plain LRU: it has load 974 / 906 / 304 on the same three tasks, which is lower on all of them, with a small accuracy loss on MATH and CommonsenseQA only.
Critical strengths
- The use case tackled here is very critical use case to deploy MoE models on practical resource constrained edge devices where latency of transferring experts from GPU memory to the memory of compute unit matters a lot. And the MoE models, in their original form, put no constraints on the experts being selected for the temporally close tokens. As a result, in worst case, it is possible that adjacent tokens may require to swap in all new topk experts for each new token processing per layer. This poses a serious latency bottleneck, even through MoE brings compute efficiency compared to a dense counterpart.
- The proposed method seems intuitive (for the temporal router) to use current tokens hidden state at layer to predict which experts to maintain in the cache for this layer's next processing.
Critical weaknesses
-
The proposed method is per-task fine-tuning method. The paper fine tunes the two post-trained models (Qwen and GPT OSS) on the training portion of GSM8k, Math and commonsenseQA task data and then evaluates the results on these respective benchmarks. This poses a constraint on the practical applicability of this method - The model needs to be fine-tuned for each and every tasks. The paper lacks any results for the tasks on which the original post-trained model was evaluated and which was not included in their fine-tuning: what is the impact on those kind of task if they are not included in the fine-tuning and does this method even bring any benefit in hit rate/cache usages for such tasks? The broader scope and intuition on why every task was included in its fine-tuning is lacking.
-
The paper fine-tunes the models (where their base parameters are updated as well in addition to the auxiliary routers) on the specific tasks. But the paper does not evaluate the models' accuracy on the benchmarks where the original post-trained models were evaluated (Like knowledge/coding or other benchmarks) to validate if the experts caching adaptation brings any reduction to their original models' capabilities to assess the risk associated .This is highly relevant because the joint adaptation using temporal router, for example, is being trained using the next token's native router predictions as target, hence the native router of each layer is also being shifted to give more predictable/consisted distribution for this adaptation. This changes the original behavior of the MoE layers.
-
There is no /very little details on the selected form of loss function used for each of the two auxiliary routers, including why the sigmoid was used or the value selected for ? Also, did authors consider any other form of loss functions like classification loss/KL loss or is their any ablations showing the selected loss function choice works better?
-
The results in table 1 is not very convincing: the full model wins against the other prefetchers, which pay a lot of traffic, however, It is not a clear win against plain LRU (Load is worse on 2 of 3 tasks, and accuracy drops 0.6–3.3 points). The Temporal Router is stronger against plain LRU: it has load 974 / 906 / 304 on the same three tasks, which is lower on all of them, with a small accuracy loss on MATH and CommonsenseQA only.
Questions for authors
There are many notations used to describe the temporal/spatio routers and the loss functions used to train them. It would greatly benefit the readers to include all notations clearly at one place in section3, before using them to describe the routers.
Dear Reviewer mTNn,
As the discussion period is approaching its end, we would greatly appreciate any further feedback on our responses, especially regarding whether our clarifications have addressed your main concerns.
Thank you again for your time and consideration!