- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors proposed CCQ, which compensates for the skipped experts with three components --- absorber, energy-guided selection, and residual sketch. The method is implemented in vLLM with customized CUDA kernels and achieves lower KL to the full model than seven baselines across three MoE models.
Critical strengths
- The error decomposition analysis is interesting and well motivates the method. The three components are designed to address distinct parts of the error, and the ablations in Table 2 show that each component contributes. The authors also implement customized CUDA kernels.
- The authors comprehensively evaluate their method across various MoE-based models.
- The paper is clearly written and easy to follow.
Critical weaknesses
- The novelty of this paper is somewhat limited.
- The speedup appears to come mostly from skipping itself rather than CCQ. CCQ is only 2-9% faster than plain top-K/2 skipping, owing to the custom kernels.
- In the downstream benchmarks, CCQ is mostly not significantly different from the absorber alone or the LoRA baseline. The LoRA baseline is merged into the weights and adds no inference cost. Therefore, it is hard to argue that the KL improvements from selection and sketching translates into performance gain.
- The experimental setting is limited.
- The method is evaluated only a fixed budget. Executing one more expert achieves much lower KL than CCQ (from Table 3), which raises the question of when a strict fixed budget is required in practice.
- The models are relatively small, and all measurements are on single GPU. It is not verified whether the method remains effective for larger models or multi-GPU deployments.
Questions for authors
Are results available for multiple training seed? If so, what are the mean and standard deviation of the KL and benchmark scores?