- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper looks at channel-level pruning of routed experts in MoE LLMs, and argues the real question is where a fixed global channel budget should go (across layers and experts), not just which channels to keep. HARP uses one weight-only statistic, (motivated via the coefficient of variation of retained mass under random thinning), at three levels:
- Layer-SP drives a water-filling allocation of per-layer budgets.
- Expert-SP assigns each expert to one of three aligned widths , with exact closure to the global budget.
- Channel-SP picks the retained coupled gate/up/down channels.
The default pipeline is calibration-free. When calibration data is available, Wanda/ENP ranks can be added to Channel-SP for channel selection only, leaving the allocation untouched. Experiments cover Qwen3-30B-A3B, Qwen3.6-35B-A3B and DeepSeek-V2-Lite at 25%/50% pruning on an 8-task generative suite. There is also a width-aware vLLM backend that runs heterogeneous experts at their true widths on one A100.
Critical strengths
1. The framing is convincing, and the calibration-side results are genuinely useful. Figure 1 makes a nice case: depending on the corpus, 30–81 Qwen3 layer–expert slots see zero calibration tokens, and simply swapping the corpus moves Wanda/ENP results around. The fix is simple and practical: keep the allocation weight-derived and let calibration only refine channel choice. Under fixed widths, SP anchoring improves Wanda or ENP in 47 of 48 comparisons. I could see people adopting that part directly.
2. The allocation story is backed by a clean controlled progression. CSP HSP HARP isolates within-layer and then cross-layer redistribution, and the gains at 50% on the Qwen models are large (59.0 61.9 69.3 on Qwen3, 73.9 80.6 86.0 on Qwen3.6). It's also good to see an actual true-width vLLM backend with raw latency tables in the appendix, rather than FLOP counts only.
Critical weaknesses
(1) The SP score appears to be equivalent to AIMER's criterion, and the paper doesn't say so. AIMER scores an expert by over its gate/up/down weights and prunes the experts with larger scores. HARP's is exactly on the same parameter group. So unless I'm missing something, Expert-SP produces the same expert ordering as AIMER. The random-thinning derivation is a nice interpretation, but cancels out, and what remains is the same concentration statistic (equivalently of the magnitudes).
The related work describes AIMER only as "a calibration-free weight-concentration criterion," and the contributions list SP as something HARP derives. I don't think this sinks the paper: applying the statistic at layer and channel level, plus the hierarchical allocator, can still be a contribution. But the equivalence should be stated explicitly and the novelty claims adjusted.
A smaller attribution issue: the intro cites MoE-Pruner as a channel-pruning method that reduces expert widths, but it prunes individual weights, which the paper's own related-work section also says.
(2) The efficiency claims read more favorably than the raw numbers. In Tables 14–15, HARP is slower than equal-width CSP at every one of the 10 measured points, by roughly 0.2% to 14%. The "both reach 1.54× / 1.33×" statement picks the one batch per model where the gap is near zero. The 1.54× peak on Qwen3 also coincides with an odd dense measurement: the dense TPOT is 7.17 / 9.39 / 13.39 / 11.32 / 14.82 ms across batches 1–16, so batch 4 is slower than batch 8. With only 3 timed iterations, I'm not sure that point is stable. Speed is also measured on the checkpoints, while the accuracy headline uses .
None of this undermines the idea. I just think the honest framing is "HARP buys a lot of accuracy over CSP at a small latency cost." Showing that trade-off directly, with latency, more timing repetitions, and variance, would make the system section much more convincing.
Questions for authors
1. What do the realized tier counts actually look like? Appendix E.1 reports {3056, 32, 3056} low/mid/high experts for Qwen3 and {5112, 16, 5112} for Qwen3.6 (). With , the coverage floor asks for at least 20 (Qwen3) or 39 (Qwen3.6) experts per tier per layer "whenever feasible," and the selection rule is said to favor balanced populations. So I expected far more mid-width experts. Does this mean most layers hit the clipping bounds, making the floor infeasible and the allocation close to all-low / all-high per layer? Per-layer counts for the main checkpoints would help me understand how much Expert-SP is really doing versus Layer-SP.
2. Is the structured Wanda baseline on DeepSeek-V2-Lite working as intended? At 25% pruning it gets 15.0 MRP, versus 61.7 for Random, and 15% on WinoGrande, which is a binary task. That looks like output-format collapse rather than a weak criterion. Could the zero-token fallback or the shared-expert handling be involved? A sanity check would make the "consistently outperforms Wanda" claim easier to trust.
3. How were chosen, and how stable are the small margins? Table 2b and Table 11 both appear to tune on the same 8-task suite used for the main results, with single-seed greedy decoding. On DeepSeek-V2-Lite the gap to CSP is about 1 point (and HSP is below CSP at 50%). Was any held-out selection used? Do you have a sense of run-to-run or prompt-format variance at that scale?