- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
HOPE formulates expert pruning by incoperating second order interactions across different experts.
Critical strengths
Introducing expert interactions into pruning seems to be a promising direction. The evaluations show positive results on large MoE models and challenging downstream tasks.
Critical weaknesses
-
The implemented objective is not equal to the objective presented in Theorem 2. The theorem uses the unconditional normalization , under which equals the expected upper-bound objective. In practice, the paper replaces it with the pair-dependent normalization . The resulting objective is another, looser upper bound with inverse-frequency reweighting. The paper does not compare these two objectives or analyze how this change affects the pruning decisions and final performance.
-
The proposed expert interaction is reduced to the product of the output norms of experts activated together. This ignores the directions of expert outputs and cannot distinguish reinforcing interactions from cancellation. Therefore, the proposed interaction appears closer to weighted co-activation than actual expert cooperation. In addition, although the method is described as higher-order pruning, it considers only second-order interactions and does not explore interactions beyond the pairwise level.
-
The theoretical error is defined only for local expert substitution and is not extended to the layer output, final model output, or downstream loss. The analysis therefore does not establish how minimizing the proposed objective affects the layer output or model output. Moreover, the bound is constructed using output norms and worst-case inequalities, but its tightness is not unclear.
-
HOPE does not consistently outperform the baselines at high pruning rates, despite this being the main setting in which the proposed objective is claimed to be most useful. The failure cases at high pruning rates require further analysis.
Questions for authors
See weakness.
3 comments
Response to review (part 2)
Why use norm products, which ignore the direction of expert outputs?
The reviewer is right that norm products ignore direction. This is deliberate, and is required by the theoretical bound (Appendix A). The squared substitution error contains the pruned experts' joint term , as well as terms and which involve the unknown replacement experts. The Cauchy–Schwarz step () is what removes the dependence on the replacements, and it is only valid with norms. A signed inner-product would capture exactly, but would lose the bound on and .
We also tested signed inner products empirically ( built from , 3 trials). Across both benchmarks, inner products never outperform norm products by more than 1 point, whereas norm products are better by up to 5.9 points (43.7 vs 37.8 on Tulu, Qwen3.5-122B at 50% pruning). Inner products are also less stable: one Qwen3.5-122B trial collapsed to 8% on LiveCodeBench. Given both the theoretical and empirical results, we believe norm products are a better formulation for HOPE.
| Model, rate | SWE-Pro: norm product (paper) | SWE-Pro: inner product | Tulu: norm product (paper) | Tulu: inner product |
|---|---|---|---|---|
| Qwen3.5-122B, 25% | 0.509 | 0.514 | 39.6 | 38.7 |
| Qwen3.5-122B, 50% | 0.425 | 0.434 | 43.7 | 37.8 |
| Qwen3.5-35B, 25% | 0.435 | 0.429 | 48.5 | 48.8 |
| Qwen3.5-35B, 50% | 0.405 | 0.387 | 42.9 | 42.7 |
How does the substitution error relate to the layer output and the model output, and how tight is the bound?
To measure this directly, we computed the exact errors on Qwen3.5-35B at 25% pruning, using Evol-CodeAlpaca calibration. We ran 500 calibration prompts through the model and, for every 32nd token (totaling 25k tokens), stored the outputs of all 256 experts and the router probabilities at 7 layers (0, 5, 10, 20, 30, 35 and 39). For each layer, we evaluated 321 candidate prune-sets of the same size: HOPE's optimized prune-set, 120 perturbations of it (20 each of 1, 2, 5, 10, 20 and 50 swaps), and 200 uniformly random prune-sets. For each candidate, we computed three quantities, averaged over tokens:
- , the HOPE objective (with the paper's conditional normalization)
- The substitution error bounded by Theorem 1 , where each token's pruned active experts are replaced by the next-highest-probability unpruned experts
- The total pruning error of the layer output , which includes both the substitution error and the renormalization error
We find that:
1) is strongly correlated with the substitution error. Across the 7 layers, the Spearman correlation between and the substitution error is 0.86–0.91 (pooled 0.993), and the Pearson correlation is 0.91–0.98 (pooled 0.961). The bound is conservative in absolute terms, but what matters for choosing a prune-set is that orders candidate prune-sets correctly, which these correlations show it does.
2) The substitution error almost perfectly tracks the total pruning error of the layer output. The Spearman correlation between the two is 0.996–0.9998, and the Pearson correlation is 0.998–0.9999. In other words, minimizing the substitution error is essentially the same as minimizing the error in the layer output.
3) The substitution error accounts for nearly all of the total pruning error. The substitution error is typically within 5% in magnitude of the total pruning error. That is, the renormalization of the retained experts' gates changes the error by only about 5% in the median (and by at most 12% for HOPE's prune-sets).
4) Lower-layer error carries through to the model output. Across all 36 pruning conditions (3 models × 2 calibration sets × 6 rates), HOPE gives lower WikiText-2 perplexity than REAP in 30 of 36 (GLM-4.5-Air: 12/12; Qwen3.5-122B: 11/12; Qwen3.5-35B: 7/12). Perplexity is the full model's next-token loss, so this shows that minimizing the per-layer objective carries through to the model's output distribution.
Because per-layer errors compound through the network, the most direct end-to-end evidence is still the downstream evaluation itself, and our experiments show that HOPE has the best average rank over all conditions.
Response to review (part 3)
Why only consider second-order interactions?
For the squared substitution error, second order is exact rather than a truncation. Expanding yields only pairwise terms, so this objective has no third- or higher-order terms to add. "Higher-order" in the title is relative to first-order (per-expert) scores, and we will clarify this in the manuscript. Alternative objectives with genuine higher-order terms would be far less tractable due to the combinatorial explosion of polynomial terms, and we've noted this challenge as future work.
Why does HOPE not consistently outperform baselines at high pruning rates?
No pruning method wins every condition, and we have tried to avoid claiming that HOPE does. However, HOPE is the most consistently strong method across conditions, and its advantage grows at higher pruning rates.
1) HOPE is the strongest method overall. Across all 54 conditions (model × calibration × rate × benchmark), HOPE has the best average rank (2.07, vs 2.35 for the next-best method), finishes first more often than any other method, is in the top two in two-thirds of conditions, and finishes last only once. It also outperforms every baseline in a majority of head-to-head comparisons. When HOPE is not the top method, it is typically close behind: it trails the winner by a median of only 0.56 points.
2) HOPE's advantage is largest at high pruning rates. At 40–50% pruning, HOPE's average rank improves to 1.83, and it outperforms every baseline in at least two-thirds of head-to-head comparisons (12 of 18 conditions against REAP, 14 of 18 against MAN and EAN, and 17 of 18 against frequency-based pruning). At 50% pruning, HOPE's average rank is 1.58 (vs 2.42 for REAP), and on SWE-Bench Pro it outperforms both REAP and MAN in all 6 model × calibration conditions (by 2.8 points on average over REAP, and by up to 6.1 points).
3) The closest competition is on Tulu at 40–50% pruning, where HOPE and REAP are tied (average rank 2.00 vs 2.00). Part of this is evaluation noise. In auditing the Tulu predictions, we found that PopQA and MMLU scores were uniformly biased across all methods. With those two tasks removed, HOPE's average rank over all 54 conditions improves from 2.07 to 1.94 (next best 2.26), and it has no last-place finishes at all.
Response to review (part 1)
We thank the reviewer for the thoughtful questions and concerns. We will address each point below.
How does conditionally-normalized HOPE compare to unconditionally-normalized HOPE?
Indeed, Theorem 2 holds exactly only for unconditional normalization, and conditional normalization changes the objective. We note that the manuscript does distinguish the two: S3.4 states that the unconditional formulation of F minimizes E[Z] exactly, and that "in practice" we use a conditional average instead, and Appendix A.7 describes conditional normalization as "the empirical choice which we found helps decouple interaction strength from co-selection frequency." Unconditional normalization is indeed what the theory suggests is optimal. However, our experiments have shown that conditional normalization yields better results overall. This observation led us to this small change in the objective, motivated by how it behaves in practice. We agree that this distinction should be more prominent, and we will make it explicit wherever the theoretical guarantees are stated. As the reviewer notes, this gap does not by itself undermine the empirical results, but here we will provide our experiments and reasoning for the change.
To compare the two objectives directly, we pruned Qwen3.5-35B-A3B and Qwen3.5-122B-A10B at 25% and 50% with unconditionally-normalized HOPE (HOPE-uncond), using the same Evol-CodeAlpaca calibration statistics (3 independent trials) as the paper. HOPE-uncond uses exactly the objective that Theorem 2 shows equals E[Z]. We used the same evaluation harness and settings as the paper.
We find empirically that conditional normalization is the better choice overall:
HOPE is the paper's method (conditional normalization). SWE-Pro is SWE-Bench Pro (mean reward); Tulu is the mean over 8 tasks. Bold marks the better of the two HOPE variants.
1) On Tulu, conditional HOPE outperforms unconditional HOPE in all 4 settings. On Qwen3.5-122B at 50%, conditional HOPE (paper) achieved 43.7 vs 41.0 with unconditional normalization. On Qwen3.5-35B at 50%, conditional HOPE achieved 42.9 vs 40.3 with unconditional normalization.
2) On SWE-Bench Pro, conditional HOPE matches or outperforms unconditional HOPE in all 4 settings (e.g. 0.435 vs 0.422 on Qwen3.5-35B at 25%, and 0.405 vs 0.401 at 50%). The only setting where unconditional HOPE is ahead is Qwen3.5-122B at 50%, by 1 point (0.435 vs 0.425).
Notably, both variants of HOPE still tend to outperform REAP.
Conditional normalization works well because it scores an expert pair by its contribution when active, rather than by how often it is active. This is what allows HOPE to keep experts which are rarely activated but contribute strongly when they are. At the same time, the theory-faithful unconditional objective is itself a strong pruning criterion, so HOPE's results do not depend on the departure from Theorem 2. We will update the manuscript, highlighting the theoretical connection with unconditional HOPE more clearly. Since our original variant of HOPE (conditional) is the strongest, we will propose it as a heuristic of the method that we show empirically works better for rarely co-occurring experts.