- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a training-free method for pruning experts in MoE LLMs. It estimates the change in the MoE layer output when an expert is removed. The score considers the remaining experts and the replacement expert selected by the router. It evaluates on four MoE models at 25% and 50% expert removal. They report better downstream performance than several pruning baselines.
Critical strengths
-
The method requires no recovery training.
-
The paper examines output-distribution changes, different scoring components, and generation behavior, providing informative empirical analysis.
Critical weaknesses
-
The novelty is limited. The method remains a one-shot, per-expert ranking approach similar to REAP. More importantly, it does not address the interactions between experts when many experts are removed together, which is the actual optimization problem in MoE pruning.
-
The proposed importance score is not well aligned with the actual pruning objective. The final method removes 25–50% of experts simultaneously, why? The damage caused by removing a set of experts is not the sum of their individual deletion damages. Replacement experts may themselves be removed, surviving experts interact, and changes in earlier layers affect routing in later layers.
-
The paper has not established that its more detailed deletion counterfactual is the main reason for the improvements, rather than the aggregation choice or other effects of expert ranking.
Questions for authors
Why is conditional RMS the appropriate score for preserving the overall output distribution?
Gentle Follow-Up on RAZOR
Gentle Follow-Up on
RAZORDear Reviewer
1dhE,We sincerely appreciate your time and consideration!
As the discussion period is now less than 12 hours from closing, we would like to kindly follow up and invite you to revisit our responses to see whether your concerns have been adequately addressed. Your feedback at this stage would allow us sufficient time to undertake any potential additional work (e.g., experiments or further writing), and we really cherish your insightful advice.
Thank you again for your valuable time and kind consideration!
Best regards,
Authors of Paper
54898 (RAZOR)