acceptodds
Under review as a conference paper at ICLR 2027

HARP: Hierarchical Adaptive Redistribution for Pruning Mixture-of-Experts Models

Abstract

Channel-level pruning can reduce both the parameter footprint and activated computation of mixture-of-experts (MoE) models, but doing so effectively requires deciding how a fixed global channel budget should be distributed across layers and experts, as well as which channels should be retained. Existing approaches often rely on calibration activations, whose quality can vary substantially across routed experts and calibration distributions. We present HARP, a calibration-free framework for hierarchical adaptive redistribution of MoE channel capacity. HARP derives a model-intrinsic structural preservation (SP) prior from the relative fluctuation of retained absolute-weight mass under random thinning and applies the same functional at the layer, expert, and coupled-channel levels. A hierarchical allocator uses these priorities to construct hardware-aligned heterogeneous expert widths under an exact global budget, while Channel-SP realizes each width through coupled channel selection. When calibration data are available, empirical criteria can refine channel identities without changing the global allocation. Across Qwen3, Qwen3.6, and DeepSeek-V2-Lite at 25% and 50% channel pruning, HARP achieves the highest mean retained performance among the evaluated channel-pruning methods in all six settings, consistently outperforming calibration-based Wanda and ENP. SP anchoring improves calibration-assisted channel selection and reduces calibration sensitivity. Finally, a width-aware vLLM backend executes heterogeneous experts at their retained widths, achieving – end-to-end inference speedup on a single A100 GPU.

Primary area:
optimization
Keywords:
Mixture-of-Experts, Model Pruning, Structured Pruning, Channel Pruning, Model Compression, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vAAjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at channel-level pruning of routed experts in MoE LLMs, and argues the real question is where a fixed global channel budget should go (across layers and experts), not just which channels to keep. HARP uses one weight-only statistic, SP(Θ)=N∥Θ∥22/∥Θ∥12\mathrm{SP}(\Theta)=N\|\Theta\|_2^2/\|\Theta\|_1^2 (motivated via the coefficient of variation of retained mass under random thinning), at three levels:

  • Layer-SP drives a water-filling allocation of per-layer budgets.
  • Expert-SP assigns each expert to one of three aligned widths {Kmid−Δ,Kmid,Kmid+Δ}\{K_{\mathrm{mid}}-\Delta, K_{\mathrm{mid}}, K_{\mathrm{mid}}+\Delta\}, with exact closure to the global budget.
  • Channel-SP picks the retained coupled gate/up/down channels.

The default pipeline is calibration-free. When calibration data is available, Wanda/ENP ranks can be added to Channel-SP for channel selection only, leaving the allocation untouched. Experiments cover Qwen3-30B-A3B, Qwen3.6-35B-A3B and DeepSeek-V2-Lite at 25%/50% pruning on an 8-task generative suite. There is also a width-aware vLLM backend that runs heterogeneous experts at their true widths on one A100.

Critical strengths

1. The framing is convincing, and the calibration-side results are genuinely useful. Figure 1 makes a nice case: depending on the corpus, 30–81 Qwen3 layer–expert slots see zero calibration tokens, and simply swapping the corpus moves Wanda/ENP results around. The fix is simple and practical: keep the allocation weight-derived and let calibration only refine channel choice. Under fixed widths, SP anchoring improves Wanda or ENP in 47 of 48 comparisons. I could see people adopting that part directly.

2. The allocation story is backed by a clean controlled progression. CSP →\to HSP →\to HARP isolates within-layer and then cross-layer redistribution, and the gains at 50% on the Qwen models are large (59.0 →\to 61.9 →\to 69.3 on Qwen3, 73.9 →\to 80.6 →\to 86.0 on Qwen3.6). It's also good to see an actual true-width vLLM backend with raw latency tables in the appendix, rather than FLOP counts only.

Critical weaknesses

(1) The SP score appears to be equivalent to AIMER's criterion, and the paper doesn't say so. AIMER scores an expert by ∥w∥1/(N ∥w∥2)\|w\|_1/(\sqrt{N}\,\|w\|_2) over its gate/up/down weights and prunes the experts with larger scores. HARP's SP=N∥w∥22/∥w∥12\mathrm{SP} = N\|w\|_2^2/\|w\|_1^2 is exactly 1/AIMER21/\mathrm{AIMER}^2 on the same parameter group. So unless I'm missing something, Expert-SP produces the same expert ordering as AIMER. The random-thinning derivation is a nice interpretation, but qq cancels out, and what remains is the same concentration statistic (equivalently 1+CV21+\mathrm{CV}^2 of the magnitudes).

The related work describes AIMER only as "a calibration-free weight-concentration criterion," and the contributions list SP as something HARP derives. I don't think this sinks the paper: applying the statistic at layer and channel level, plus the hierarchical allocator, can still be a contribution. But the equivalence should be stated explicitly and the novelty claims adjusted.

A smaller attribution issue: the intro cites MoE-Pruner as a channel-pruning method that reduces expert widths, but it prunes individual weights, which the paper's own related-work section also says.

(2) The efficiency claims read more favorably than the raw numbers. In Tables 14–15, HARP is slower than equal-width CSP at every one of the 10 measured points, by roughly 0.2% to 14%. The "both reach 1.54× / 1.33×" statement picks the one batch per model where the gap is near zero. The 1.54× peak on Qwen3 also coincides with an odd dense measurement: the dense TPOT is 7.17 / 9.39 / 13.39 / 11.32 / 14.82 ms across batches 1–16, so batch 4 is slower than batch 8. With only 3 timed iterations, I'm not sure that point is stable. Speed is also measured on the Δ=64\Delta=64 checkpoints, while the accuracy headline uses Δ=128\Delta=128.

None of this undermines the idea. I just think the honest framing is "HARP buys a lot of accuracy over CSP at a small latency cost." Showing that trade-off directly, with Δ=128\Delta=128 latency, more timing repetitions, and variance, would make the system section much more convincing.

Questions for authors

1. What do the realized tier counts actually look like? Appendix E.1 reports {3056, 32, 3056} low/mid/high experts for Qwen3 and {5112, 16, 5112} for Qwen3.6 (Δ=64\Delta=64). With η=0.15\eta=0.15, the coverage floor asks for at least 20 (Qwen3) or 39 (Qwen3.6) experts per tier per layer "whenever feasible," and the selection rule is said to favor balanced populations. So I expected far more mid-width experts. Does this mean most layers hit the clipping bounds, making the floor infeasible and the allocation close to all-low / all-high per layer? Per-layer counts for the Δ=128\Delta=128 main checkpoints would help me understand how much Expert-SP is really doing versus Layer-SP.

2. Is the structured Wanda baseline on DeepSeek-V2-Lite working as intended? At 25% pruning it gets 15.0 MRP, versus 61.7 for Random, and 15% on WinoGrande, which is a binary task. That looks like output-format collapse rather than a weak criterion. Could the zero-token fallback or the shared-expert handling be involved? A sanity check would make the "consistently outperforms Wanda" claim easier to trust.

3. How were (γ,η,Δ)(\gamma,\eta,\Delta) chosen, and how stable are the small margins? Table 2b and Table 11 both appear to tune on the same 8-task suite used for the main results, with single-seed greedy decoding. On DeepSeek-V2-Lite the gap to CSP is about 1 point (and HSP is below CSP at 50%). Was any held-out selection used? Do you have a sense of run-to-run or prompt-format variance at that scale?

Reviewer_YPdmOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a hierarchical capacity allocation strategy to effectively allocate the budget for retaining the number of channels across layers and experts within a layer in a channel-pruned MoE model. Specifically, the paper uses a structural preservation metric (SP) to first identify the desired average number of retained channels for an MoE layer. Later, the identified desired number of channels is used to solve a combinatorial optimization problem to identify the desired number of channels for each expert inside the layer. Finally, the metric is further used to rank the channels inside an expert to be retained in the pruned model.

Critical strengths

The hierarchical capacity allocation strategy is carefully designed and outperforms other calibration-data-based and calibration-data-free channel-pruning methods.

Critical weaknesses
  1. I am doubtful about the theoretical soundness of the proposed structural preservation (SP) metric itself. In designing the metric, it has been assumed that all parameter groups have an equal probability of retention. However, not all groups are equally important. Some may be more useful than others, which is one of the general motivations for parameter pruning. Higher importance is given to the group that has greater sensitivity to 'random-thinning'. However, random-thinning is not an approach we take during pruning. On the other hand, the metric, SP, essentially evaluates the ratio of the L-2 and L-1 norm of the parameter group, i.e., a well-known measurement of parameter distribution. It's a well-known practice to use sparsity for parameter pruning in the model-pruning literature. From that perspective, the proposed metric is not a novel addition. Moreover, the paper does not provide any theoretical justification/proof for the non-triviality of the proposed metric. Also, it is not clear why the method is specific to MoE. Can it be applied to dense model, also?

  2. Despite its delicate design, the method does not outperform the expert-pruning baseline (AIMER) presented in this paper, which is much simpler and easy to implement in hardware. Also, the paper does not compare or at least discuss other potential design options (e.g., why using only three expert-width options instead of maintaining a continuous determination) in each stage of its hierarchical capacity allocation strategy. Therefore, it is not clear whether the design is delicate or generalizable.

Questions for authors

Please see weaknesses.

Reviewer_uaYBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes HARP, a calibration-free method for channel pruning of routed experts in MoE models. It uses a single weight-concentration score, motivated by the relative variance of retained weight mass under random thinning. The score is applied at three levels. It sets layer budgets, assigns experts to three aligned width tiers under an exact global budget, and selects coupled channels within each expert. Calibration-based rankings can optionally refine channel selection without changing the allocation. On Qwen3, Qwen3.6, and DeepSeek-V2-Lite at 25 and 50 percent pruning, HARP achieves the best mean retained performance among the channel-pruning baselines in all six settings. A width-aware vLLM backend gives 1.04 to 1.54 times end-to-end speedup over the dense model.

Critical strengths
  • Large, consistent gains from a cheap, data-free procedure. The CSP -> HSP -> HARP progression is a clean, controlled comparison that isolates the value of expert-level and layer-level redistribution. On Qwen3.6 at 50%, MRP goes 73.9 -> 80.6 -> 86.0, and the 47/48 improvement from SP anchoring of Wanda/ENP is notable.

  • The MoE channel pruning is treated as a global capacity-placement problem. It separates where capacity goes (structural prior) from which channels realize it (optionally calibration-refined). This framing is useful, and the uneven calibration coverage of experts (Fig. 1a) is a relevant observation for MoE pruning in general.

Critical weaknesses
  • Every calibration baseline uses uniform widths. The paper does not compare against other non-uniform allocation schemes, such as OWL-style layerwise ratios, TENP’s trapezoidal profile, or attribution-guided allocation, using the same channel scorer. So it is unclear whether the gains come from the proposed score or from heterogeneity in general.

  • Several baseline numbers look implausible. Wanda on DeepSeek-V2-Lite at 25 percent pruning retains only 15.0, far below random selection at 61.7. Magnitude-based channel selection in the ablation (44.8 and 12.1) also falls well below random (57.5). This suggests possible implementation or score-orientation issues, which would inflate the reported margins.

  • The theory does not explain why concentrated weights indicate importance. The random-thinning derivation only re-expresses a concentration statistic, and the appendix results are a standard exchange argument plus the fact that top-K maximizes a sum. Neither establishes the score as a meaningful importance surrogate.

  • Novelty relative to AIMER is unclear, since AIMER already uses a calibration-free weight-concentration criterion for ranking experts. The speedup is measured on the Delta 64 checkpoints, not the Delta 128 checkpoints behind the main accuracy results. The headline speedup is relative to the dense model, which any 50 percent pruned model would roughly match. Against equal-width CSP, HARP is never faster and is often slower, for example on Qwen3 at batch sizes 8 and 16.

  • The tier counts of the efficiency checkpoints have an almost empty middle tier: 32 of 6144 experts for Qwen3 and 16 of 10240 for Qwen3.6. This contradicts the stated minimum tier population of 15 percent per layer, so the measured models appear to differ from the described algorithm.

Questions for authors
  • see weakness

  • Why does structured Wanda on DeepSeek-V2-Lite at 25 percent pruning score far below random selection? Please verify the score orientation and coupled channel handling. If the baseline is corrected, how do the margins change?

  • Why does magnitude-based channel selection in the ablation perform much worse than random selection? Is the ranking direction correct?

  • How do the efficiency checkpoints end up with an almost empty middle tier, despite the stated minimum tier population of 15 percent per layer? Were they produced by the same algorithm as the main results?

  • Can authors report latency for the Delta 128 checkpoints that produce the main accuracy numbers? Can authors also explain why HARP is slower than equal-width CSP at several batch sizes?

  • Can authors compare against other non-uniform allocation scheme, such as OWL-style layer ratios or TENP’s trapezoidal profile, while keeping the same channel score for channel selection? This would show whether the layer and expert scores add value beyond heterogeneity itself.

  • How does the score differ from the weight-concentration criterion in AIMER, beyond being applied at more levels?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.