- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
MAP-MoE provides a mechanism for per-channel MoE expert pruning. MAP-MoE first pools experts's channels of the same group and find the best pruning ratio for different experts from different modality, subject to a global pruning rate, based on activation energy, then it chooses which experts to actually prune via RWE. MAP-MoE achieves good performance with existing baselines.
Critical strengths
The paper is well motivated, and comes with a clear package of motivation, solution, and it did a good job aiming for a complete validation cycle, grounding its contribution to actual end-to-end speedup during inference with an implementation co-design.
Critical weaknesses
- I find 4.1 and 4.2 a bit confusing to me - why do we use two different metrics for ranking the experts and for selection? Why couldn't we just use a single metric?
- The metrics used, while meaningful, does not have enough evidence to establish their connections to the final prediction. What is each terms contribution to the final prediction error? Maybe a KL term would help?
- The activation/stats are from the original unpruned model, and earlier pruning decisions would contribute to the current layers, but the scores wouldn't see this. How accurate would this be?
- The speedup appears to be deriving from two different software? One is HF Transformer, the other is on the custom kernel. To really show the benefit of the custom kernel, it would be great to separate the effect of FLOPs reduction (running with original Transformer lib) and improved implementation. If the naive is slower, I think it makes the new kernel shine.
Questions for authors
Please see weakness. Is there a way to blend in speedup into the search process? Is there any tradeoff we can make to incorporate speedup to equation (3)? Perhaps a small accuracy loss can bring in larger speedup at certain pruning region? Why does some of the experimental pruned results outperform unpruned dataset? Is there a clear explanation or we don't know yet?
Thank you for highlighting the distinction between budget and channel scores, the need to connect proxies to predictions, and the importance of separating arithmetic reduction from execution engineering. Your questions motivated direct shared-score controls, fixed-budget selector diagnostics and a recalibration comparison.
Question 1. Why use two scores, and can one score serve both stages?
For Question 1,We use activation energy to construct group-level loss curves that guide budget allocation. For channel selection, RWE additionally accounts for a channel’s contribution through the down projection and favors intermittently large activations through its concentration term. The two stages thus address group-level capacity allocation and fine-grained channel retention, respectively.
Following your suggestion, we compare our allocation/selection combination with alternatives that use the same score in both stages. The table reports accuracy averaged over all tasks for Qwen and InternVL:
The results show that our combination (Activation energy / RWE) achieves higher accuracy than both shared-score alternatives at P20 and P30. At P50, our method is subject to a capacity-protection constraint under which Activation energy / RWE (ours) and RWE / RWE effectively select the same channels; compared with Activation energy / activation energy, our method is nevertheless better at both P30 and P50.