acceptodds
Under review as a conference paper at ICLR 2027

Higher-order pruning of experts in mixture-of-experts language models

Abstract

Mixture-of-Experts (MoE) language models suffer from large parameter counts, which create a significant memory bottleneck. Expert pruning is the most direct approach for reducing this parameter count, yet existing methods make pruning decisions for each expert independently, and assume experts' contributions are purely additive. In reality, expert usage in MoEs is inherently cooperative. We derive HOPE (Higher-Order Pruning of Experts), a second-order pruning objective which provably minimizes an upper bound on the error resulting from pruning. We show that REAP (a state-of-the-art first-order pruning method) is a special case of HOPE where interaction terms are ignored. Across three frontier MoE models (up to 122B parameters), two distinct calibration sets, and multiple benchmarks (including math, instruction following, coding, and an agentic suite), we demonstrate that HOPE produces better pruning decisions than existing methods, and its advantage is most pronounced at high pruning rates and on challenging agentic workloads. At 50% pruning, HOPE outperforms all baselines and achieves an average rank of 1.58 out of 5 methods (versus 2.42 for the next-best method, REAP), with gains of up to +6.1% on agentic coding. Over all conditions, HOPE again achieves the best average rank and surpasses every other method in the majority of head-to-head comparisons. By preserving cooperative expert structure that first-order methods ignore, HOPE enables aggressive compression with minimal degradation, particularly on complex tasks where diverse expert combinations are invoked over long sequences.

Primary area:
foundation or frontier models, including LLMs
Keywords:
llm, mixture-of-experts, pruning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_anFWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HOPE formulates expert pruning by incoperating second order interactions across different experts.

Critical strengths

Introducing expert interactions into pruning seems to be a promising direction. The evaluations show positive results on large MoE models and challenging downstream tasks.

Critical weaknesses
  1. The implemented objective is not equal to the objective presented in Theorem 2. The theorem uses the unconditional normalization 1/N1/N, under which p⊤Fp\mathbf{p}^{\top}F\mathbf{p} equals the expected upper-bound objective. In practice, the paper replaces it with the pair-dependent normalization 1/∣Xij∣1/|X_{ij}|. The resulting objective is another, looser upper bound with inverse-frequency reweighting. The paper does not compare these two objectives or analyze how this change affects the pruning decisions and final performance.

  2. The proposed expert interaction is reduced to the product of the output norms of experts activated together. This ignores the directions of expert outputs and cannot distinguish reinforcing interactions from cancellation. Therefore, the proposed interaction appears closer to weighted co-activation than actual expert cooperation. In addition, although the method is described as higher-order pruning, it considers only second-order interactions and does not explore interactions beyond the pairwise level.

  3. The theoretical error is defined only for local expert substitution and is not extended to the layer output, final model output, or downstream loss. The analysis therefore does not establish how minimizing the proposed objective affects the layer output or model output. Moreover, the bound is constructed using output norms and worst-case inequalities, but its tightness is not unclear.

  4. HOPE does not consistently outperform the baselines at high pruning rates, despite this being the main setting in which the proposed objective is claimed to be most useful. The failure cases at high pruning rates require further analysis.

Questions for authors

See weakness.

3 comments
AuthorsOpenReview ↗

Response to review (part 1)

We thank the reviewer for the thoughtful questions and concerns. We will address each point below.

How does conditionally-normalized HOPE compare to unconditionally-normalized HOPE?

Indeed, Theorem 2 holds exactly only for unconditional normalization, and conditional normalization changes the objective. We note that the manuscript does distinguish the two: S3.4 states that the unconditional formulation of FF minimizes E[Z]\mathbb{E}[Z] exactly, and that "in practice" we use a conditional average instead, and Appendix A.7 describes conditional normalization as "the empirical choice which we found helps decouple interaction strength from co-selection frequency." Unconditional normalization is indeed what the theory suggests is optimal. However, our experiments have shown that conditional normalization yields better results overall. This observation led us to this small change in the objective, motivated by how it behaves in practice. We agree that this distinction should be more prominent, and we will make it explicit wherever the theoretical guarantees are stated. As the reviewer notes, this gap does not by itself undermine the empirical results, but here we will provide our experiments and reasoning for the change.

To compare the two objectives directly, we pruned Qwen3.5-35B-A3B and Qwen3.5-122B-A10B at 25% and 50% with unconditionally-normalized HOPE (HOPE-uncond), using the same Evol-CodeAlpaca calibration statistics (3 independent trials) as the paper. HOPE-uncond uses exactly the objective that Theorem 2 shows equals E[Z]\mathbb{E}[Z]. We used the same evaluation harness and settings as the paper.

We find empirically that conditional normalization is the better choice overall:

Model, rateSWE-Pro: HOPESWE-Pro: HOPE-uncondSWE-Pro: REAPTulu: HOPETulu: HOPE-uncondTulu: REAP
Qwen3.5-122B, 25%0.5090.5050.46639.639.439.2
Qwen3.5-122B, 50%0.4250.4350.38743.741.040.2
Qwen3.5-35B, 25%0.4350.4220.42748.547.848.7
Qwen3.5-35B, 50%0.4050.4010.36242.940.343.0

HOPE is the paper's method (conditional normalization). SWE-Pro is SWE-Bench Pro (mean reward); Tulu is the mean over 8 tasks. Bold marks the better of the two HOPE variants.

1) On Tulu, conditional HOPE outperforms unconditional HOPE in all 4 settings. On Qwen3.5-122B at 50%, conditional HOPE (paper) achieved 43.7 vs 41.0 with unconditional normalization. On Qwen3.5-35B at 50%, conditional HOPE achieved 42.9 vs 40.3 with unconditional normalization.

2) On SWE-Bench Pro, conditional HOPE matches or outperforms unconditional HOPE in all 4 settings (e.g. 0.435 vs 0.422 on Qwen3.5-35B at 25%, and 0.405 vs 0.401 at 50%). The only setting where unconditional HOPE is ahead is Qwen3.5-122B at 50%, by 1 point (0.435 vs 0.425).

Notably, both variants of HOPE still tend to outperform REAP.

Conditional normalization works well because it scores an expert pair by its contribution when active, rather than by how often it is active. This is what allows HOPE to keep experts which are rarely activated but contribute strongly when they are. At the same time, the theory-faithful unconditional objective is itself a strong pruning criterion, so HOPE's results do not depend on the departure from Theorem 2. We will update the manuscript, highlighting the theoretical connection with unconditional HOPE more clearly. Since our original variant of HOPE (conditional) is the strongest, we will propose it as a heuristic of the method that we show empirically works better for rarely co-occurring experts.

AuthorsOpenReview ↗

Response to review (part 2)

Why use norm products, which ignore the direction of expert outputs?

The reviewer is right that norm products ignore direction. This is deliberate, and is required by the theoretical bound (Appendix A). The squared substitution error contains the pruned experts' joint term A=∥∑jgjfj∥2A = \|\sum_j g_j f_j\|^2, as well as terms BB and CC which involve the unknown replacement experts. The Cauchy–Schwarz step (∣B∣≤2AC|B| \leq 2\sqrt{A}\sqrt{C}) is what removes the dependence on the replacements, and it is only valid with norms. A signed inner-product FF would capture AA exactly, but would lose the bound on BB and CC.

We also tested signed inner products empirically (FF built from ∑⟨gifi,gjfj⟩\sum \langle g_i f_i, g_j f_j \rangle, 3 trials). Across both benchmarks, inner products never outperform norm products by more than 1 point, whereas norm products are better by up to 5.9 points (43.7 vs 37.8 on Tulu, Qwen3.5-122B at 50% pruning). Inner products are also less stable: one Qwen3.5-122B trial collapsed to 8% on LiveCodeBench. Given both the theoretical and empirical results, we believe norm products are a better formulation for HOPE.

Model, rateSWE-Pro: norm product (paper)SWE-Pro: inner productTulu: norm product (paper)Tulu: inner product
Qwen3.5-122B, 25%0.5090.51439.638.7
Qwen3.5-122B, 50%0.4250.43443.737.8
Qwen3.5-35B, 25%0.4350.42948.548.8
Qwen3.5-35B, 50%0.4050.38742.942.7

How does the substitution error relate to the layer output and the model output, and how tight is the bound?

To measure this directly, we computed the exact errors on Qwen3.5-35B at 25% pruning, using Evol-CodeAlpaca calibration. We ran 500 calibration prompts through the model and, for every 32nd token (totaling 25k tokens), stored the outputs of all 256 experts and the router probabilities at 7 layers (0, 5, 10, 20, 30, 35 and 39). For each layer, we evaluated 321 candidate prune-sets of the same size: HOPE's optimized prune-set, 120 perturbations of it (20 each of 1, 2, 5, 10, 20 and 50 swaps), and 200 uniformly random prune-sets. For each candidate, we computed three quantities, averaged over tokens:

  • ZZ, the HOPE objective (with the paper's conditional normalization)
  • The substitution error bounded by Theorem 1 ∥∑j∈Pgj(x)fj(x)−∑i∈Rgi′(x)fi(x)∥2\|\sum_{j \in P} g_j(x) f_j(x) - \sum_{i \in R} g'_i(x) f_i(x)\|^2, where each token's pruned active experts PP are replaced by the next-highest-probability unpruned experts RR
  • The total pruning error of the layer output ∥h(x)−h′(x)∥2\|h(x) - h'(x)\|^2, which includes both the substitution error and the renormalization error

We find that:

1) ZZ is strongly correlated with the substitution error. Across the 7 layers, the Spearman correlation between ZZ and the substitution error is 0.86–0.91 (pooled 0.993), and the Pearson correlation is 0.91–0.98 (pooled 0.961). The bound is conservative in absolute terms, but what matters for choosing a prune-set is that ZZ orders candidate prune-sets correctly, which these correlations show it does.

2) The substitution error almost perfectly tracks the total pruning error of the layer output. The Spearman correlation between the two is 0.996–0.9998, and the Pearson correlation is 0.998–0.9999. In other words, minimizing the substitution error is essentially the same as minimizing the error in the layer output.

3) The substitution error accounts for nearly all of the total pruning error. The substitution error is typically within 5% in magnitude of the total pruning error. That is, the renormalization of the retained experts' gates changes the error by only about 5% in the median (and by at most 12% for HOPE's prune-sets).

4) Lower-layer error carries through to the model output. Across all 36 pruning conditions (3 models × 2 calibration sets × 6 rates), HOPE gives lower WikiText-2 perplexity than REAP in 30 of 36 (GLM-4.5-Air: 12/12; Qwen3.5-122B: 11/12; Qwen3.5-35B: 7/12). Perplexity is the full model's next-token loss, so this shows that minimizing the per-layer objective carries through to the model's output distribution.

Because per-layer errors compound through the network, the most direct end-to-end evidence is still the downstream evaluation itself, and our experiments show that HOPE has the best average rank over all conditions.

AuthorsOpenReview ↗

Response to review (part 3)

Why only consider second-order interactions?

For the squared substitution error, second order is exact rather than a truncation. Expanding ∥∑jgjfj∥2=∑i,jgigj⟨fi,fj⟩\|\sum_j g_j f_j\|^2 = \sum_{i,j} g_i g_j \langle f_i, f_j \rangle yields only pairwise terms, so this objective has no third- or higher-order terms to add. "Higher-order" in the title is relative to first-order (per-expert) scores, and we will clarify this in the manuscript. Alternative objectives with genuine higher-order terms would be far less tractable due to the combinatorial explosion of polynomial terms, and we've noted this challenge as future work.

Why does HOPE not consistently outperform baselines at high pruning rates?

No pruning method wins every condition, and we have tried to avoid claiming that HOPE does. However, HOPE is the most consistently strong method across conditions, and its advantage grows at higher pruning rates.

1) HOPE is the strongest method overall. Across all 54 conditions (model × calibration × rate × benchmark), HOPE has the best average rank (2.07, vs 2.35 for the next-best method), finishes first more often than any other method, is in the top two in two-thirds of conditions, and finishes last only once. It also outperforms every baseline in a majority of head-to-head comparisons. When HOPE is not the top method, it is typically close behind: it trails the winner by a median of only 0.56 points.

2) HOPE's advantage is largest at high pruning rates. At 40–50% pruning, HOPE's average rank improves to 1.83, and it outperforms every baseline in at least two-thirds of head-to-head comparisons (12 of 18 conditions against REAP, 14 of 18 against MAN and EAN, and 17 of 18 against frequency-based pruning). At 50% pruning, HOPE's average rank is 1.58 (vs 2.42 for REAP), and on SWE-Bench Pro it outperforms both REAP and MAN in all 6 model × calibration conditions (by 2.8 points on average over REAP, and by up to 6.1 points).

3) The closest competition is on Tulu at 40–50% pruning, where HOPE and REAP are tied (average rank 2.00 vs 2.00). Part of this is evaluation noise. In auditing the Tulu predictions, we found that PopQA and MMLU scores were uniformly biased across all methods. With those two tasks removed, HOPE's average rank over all 54 conditions improves from 2.07 to 1.94 (next best 2.26), and it has no last-place finishes at all.

Reviewer_i8bgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HOPE selects experts for removal using a quadratic objective built from co-activated experts' gate-weighted output norms. The paper derives an upper bound on substitution error, then implements a conditionally normalized interaction matrix with continuous optimization and rounding. Experiments compare four pruning baselines across three MoE models, two calibration datasets, six pruning rates, general benchmarks, and SWE-Bench Pro. Supplementary analyses examine calibration stability, optimization overhead, and subsequent fine-tuning.

Critical strengths
  1. The interaction matrix can be accumulated during calibration without retraining. Table S7 reports approximately 7% additional calibration-and-selection time relative to REAP, making the approach attractive if its accuracy gains reproduce.

  2. Tables S1-S6 report three calibration trials across multiple models and calibration domains. At 50% pruning, HOPE exceeds REAP on all six SWE-Bench Pro configurations, including 42.33 versus 36.23 for Qwen3.5-35B with Evol-CodeAlpaca calibration. This supports practical value beyond isolated task improvements.

Critical weaknesses
  1. Theorem 2 establishes equality with the empirical expectation of Z using normalization by the total token count, whereas Sections 3.4 and A.7 implement pair-specific conditional normalization. This reweighting changes the objective and can destroy positive semidefiniteness. Consequently, the small rounding loss in Figure S5 does not certify global optimality of the relaxed problem solved by SLSQP, a local optimizer. Moreover, Theorem 1 bounds substitution error; Appendix A.4's assumption that removed and replacement gate masses are close does not establish that renormalization error is generally negligible. The claimed theoretical guarantee for the practical pruning procedure is therefore overstated, although the method remains a plausible heuristic.

  2. The interaction statistic multiplies norms and therefore cannot distinguish aligned from opposing expert outputs. Section 3.5 also attributes all gains over REAP to interactions, despite diagonal-only HOPE and REAP differing through second-moment versus squared-first-moment scoring. Figure S4's pooled rank correlation of 0.988 establishes neither identical within-layer prune sets nor matched downstream accuracy. A controlled downstream comparison with diagonal-only HOPE is needed to isolate the contribution of pairwise terms; the current analyses support correlated activity more directly than preservation of functional cooperation.

Questions for authors

What reasoning-mode, answer-extraction, and truncation settings produced the Tulu results? Tables S1-S2 report unpruned PopQA scores of 0.35 for both Qwen models and MMLU scores of 65.15 versus 26.68 for the 35B and 122B models, respectively. Please establish whether the 2,048-token generation budget in Appendix D.4 creates systematic scoring failures; this affects how much weight the aggregate rankings deserve.

3 comments
AuthorsOpenReview ↗

Response to review (part 1)

We thank the reviewer for the thoughtful questions and concerns. We will address each point below.

How tight is the HOPE optimization objective above the true pruning error?

We agree that Theorem 1 only bounds the substitution error, and that the argument in Appendix A.4 (that renormalization error is negligible) is a heuristic. To measure this directly, we computed the exact errors on Qwen3.5-35B at 25% pruning, using Evol-CodeAlpaca calibration. We ran 500 calibration prompts through the model and, for every 32nd token (totaling 25k tokens), stored the outputs of all 256 experts and the router probabilities at 7 layers (0, 5, 10, 20, 30, 35 and 39). For each layer, we evaluated 321 candidate prune-sets of the same size: HOPE's optimized prune-set, 120 perturbations of it (20 each of 1, 2, 5, 10, 20 and 50 swaps), and 200 uniformly random prune-sets. For each candidate, we computed three quantities, averaged over tokens:

  • ZZ, the HOPE objective (with the paper's conditional normalization)
  • The substitution error bounded by Theorem 1 ∥∑j∈Pgj(x)fj(x)−∑i∈Rgi′(x)fi(x)∥2\|\sum_{j \in P} g_j(x) f_j(x) - \sum_{i \in R} g'_i(x) f_i(x)\|^2, where each token's pruned active experts PP are replaced by the next-highest-probability unpruned experts RR
  • The total pruning error of the layer ∥h(x)−h′(x)∥2\|h(x) - h'(x)\|^2, which includes both the substitution error and the renormalization error

We find that:

1) ZZ is strongly correlated with the substitution error. Across the 7 layers, the Spearman correlation between ZZ and the substitution error is 0.86–0.91 (pooled 0.993), and the Pearson correlation is 0.91–0.98 (pooled 0.961).

2) The substitution error almost perfectly tracks the total pruning error. The Spearman correlation between the two is 0.996–0.9998, and the Pearson correlation is 0.998–0.9999. In other words, minimizing the substitution error is essentially the same as minimizing the total pruning error.

3) The substitution error accounts for nearly all of the total pruning error. The substitution error is typically within 5% in magnitude of the total pruning error. That is, the renormalization of the retained experts' gates changes the error by only about 5% in the median (and by at most 12% for HOPE's prune-sets).

We will include these empirical results in the manuscript.

AuthorsOpenReview ↗

Response to review (part 2)

How does conditionally-normalized HOPE compare to unconditionally-normalized HOPE?

Indeed, Theorem 2 holds exactly only for unconditional normalization, and conditional normalization changes the objective. We note that the manuscript does distinguish the two: S3.4 states that the unconditional formulation of FF minimizes E[Z]\mathbb{E}[Z] exactly, and that "in practice" we use a conditional average instead, and Appendix A.7 describes conditional normalization as "the empirical choice which we found helps decouple interaction strength from co-selection frequency." Unconditional normalization is indeed what the theory suggests is optimal. However, our experiments have shown that conditional normalization yields better results overall. This observation led us to this small change in the objective, motivated by how it behaves in practice. We agree that this distinction should be more prominent, and we will make it explicit wherever the theoretical guarantees are stated. As the reviewer notes, this gap does not by itself undermine the empirical results, but here we will provide our experiments and reasoning for the change.

To test the two objectives directly, we pruned Qwen3.5-35B-A3B and Qwen3.5-122B-A10B at 25% and 50% with unconditionally-normalized HOPE (HOPE-uncond), using the same Evol-CodeAlpaca calibration statistics (3 independent trials) as the paper. HOPE-uncond uses exactly the objective that Theorem 2 shows equals E[Z]\mathbb{E}[Z]. We used the same evaluation harness and settings as the paper.

We find empirically that conditional normalization is the better choice overall:

Model, rateSWE-Pro: HOPESWE-Pro: HOPE-uncondSWE-Pro: REAPTulu: HOPETulu: HOPE-uncondTulu: REAP
Qwen3.5-122B, 25%0.5090.5050.46639.639.439.2
Qwen3.5-122B, 50%0.4250.4350.38743.741.040.2
Qwen3.5-35B, 25%0.4350.4220.42748.547.848.7
Qwen3.5-35B, 50%0.4050.4010.36242.940.343.0

HOPE is the paper's method (conditional normalization). SWE-Pro is SWE-Bench Pro (mean reward); Tulu is the mean over 8 tasks. Bold marks the better of the two HOPE variants.

1) On Tulu, conditional HOPE outperforms unconditional HOPE in all 4 settings. On Qwen3.5-122B at 50%, conditional HOPE (paper) achieved 43.7 vs 41.0 with unconditional normalization. On Qwen3.5-35B at 50%, conditional HOPE achieved 42.9 vs 40.3 with unconditional normalization.

2) On SWE-Bench Pro, conditional HOPE usually matches or outperforms unconditional HOPE in all 4 settings (e.g. 0.435 vs 0.422 on Qwen3.5-35B at 25%, and 0.405 vs 0.401 at 50%). The only setting where unconditional HOPE is ahead is Qwen3.5-122B at 50%, by 1 point (0.435 vs 0.425).

Notably, both variants of HOPE still tend to outperform REAP.

Conditional normalization works well because it scores an expert pair by its contribution when active, rather than by how often it is active. This is what allows HOPE to keep experts which are rarely activated but contribute strongly when they are. At the same time, the theory-faithful unconditional objective is itself a strong pruning criterion, so HOPE's results do not depend on the departure from Theorem 2. We will update the manuscript, highlighting the theoretical connection with unconditional HOPE more clearly. Since our original variant of HOPE (conditional) is the strongest, we will propose it as a heuristic of the method that we show empirically works better for rarely co-occurring experts.

How close is the QP solution to optimality?

We agree that the small rounding loss in Fig. S5 shows that the relaxed solution is nearly integral, but does not by itself certify global optimality. We will reword Fig. S5 accordingly. To quantify how far HOPE's prune-sets could be from the global optimum, we ran two additional analyses over every layer of all three models (Evol-CodeAlpaca calibration), at 50% pruning. We find two lines of evidence that suggest the QP solution is close to the global optimum:

1) A certified lower bound on the global optimum. For any binary pp with ∑p=∣P∣\sum p = |P|, we have p⊤(F+σI)p=p⊤Fp+σ∣P∣p^\top (F + \sigma I) p = p^\top F p + \sigma |P|. Shifting FF by σ=−λmin⁡\sigma = -\lambda_{\min} (the minimum eigenvalue) therefore does not change the original optimal solution, but makes its relaxation convex. Solving this convex relaxation exactly gives a lower bound on the objective of every possible prune-set. HOPE's prune-set is provably within a median of 1.9–2.2% (depending on the model) of the global optimum.

2) Local discretized optimality. In 84% of layers, no single swap of a pruned expert for a retained expert improves the objective. Where a swap does help, the largest improvement (in objective value) in any layer was at most 1%.

AuthorsOpenReview ↗

Response to review (part 3)

How does diagonal-only HOPE compare to full-matrix HOPE and REAP?

Diagonal-only HOPE is indeed not the same as REAP. Diagonal-only HOPE and REAP agree closely in their overall ranking of experts (the pooled Spearman correlation of 0.988 in Fig. S4), but the formulation is slightly different. We will modify our claims that HOPE's gain over REAP comes from interactions, and that HOPE is a strict generalization of REAP.

To isolate the contribution of the pairwise terms, we ran the controlled comparison that was requested. Using the same setup as for HOPE-uncond above (Qwen3.5-35B-A3B and Qwen3.5-122B-A10B at 25% and 50%, Evol-CodeAlpaca calibration, 3 trials, the paper's evaluation settings), we pruned with diagonal-only HOPE (HOPE-diag): the same conditional normalization as the paper's HOPE, but with the off-diagonal terms of FF zeroed. We find that the off-diagonal terms generally provide additional performance gains.

Model, rateTulu: HOPETulu: HOPE-diagTulu: REAP
Qwen3.5-122B, 25%39.637.039.2
Qwen3.5-122B, 50%43.741.040.2
Qwen3.5-35B, 25%48.549.248.7
Qwen3.5-35B, 50%42.941.443.0

HOPE is the paper's method (full FF, conditional normalization). Tulu is the mean over 8 tasks. Bold marks the better of the two HOPE variants.

The interaction terms further improve performance. On Tulu, full HOPE outperforms HOPE-diag in 3 of 4 settings, by up to 2.7 points (43.7 vs 41.0 on Qwen3.5-122B at 50% pruning, and 39.6 vs 37.0 at 25%), and by 1.5 points on average across the four settings.

That is, the pairwise terms contribute to HOPE's advantage. We will include these results, and modify the conclusion in S4.3.

How do inner products compare to norm products?

Regarding the use of norm products, the reviewer is right that they ignore direction. This is deliberate, and is required by the theoretical bound (Appendix A). We also empirically tested a version of HOPE with inner products rather than norm products (i.e. FF built from ∑⟨gifi,gjfj⟩\sum \langle g_i f_i, g_j f_j \rangle, 3 trials). Across both benchmarks, inner products never outperform norm products by more than 1 point, whereas norm products are better by up to 5.9 points (43.7 vs 37.8 on Tulu, Qwen3.5-122B at 50% pruning). Inner products are also less stable: one Qwen3.5-122B trial collapsed to 8% on LiveCodeBench. Given both the theoretical and empirical results, we believe norm products are a better formulation for HOPE.

Model, rateSWE-Pro: norm product (paper)SWE-Pro: inner productTulu: norm product (paper)Tulu: inner product
Qwen3.5-122B, 25%0.5090.51439.638.7
Qwen3.5-122B, 50%0.4250.43443.737.8
Qwen3.5-35B, 25%0.4350.42948.548.8
Qwen3.5-35B, 50%0.4050.38742.942.7

What settings were used for producing the Tulu results?

Thank you for this question, which led us to audit our evaluation using the stored predictions of the unpruned models. All settings were identical across methods and conditions: the models' default reasoning (thinking) mode, Tulu's standard answer extraction and stop sequences, and a 2,048-token generation budget. We will state all of these explicitly in Appendix D.4.

The two anomalies that the reviewer identified are format artifacts. Because these settings are identical for every method, they cannot systematically favor one method, but they do add noise to the aggregate rankings. Below, we re-rank all 54 conditions (model × calibration × rate × benchmark) with PopQA and MMLU removed from the Tulu mean:

Paper (8 Tulu tasks)PopQA + MMLU removed
HOPE average rank2.071.94
MAN average rank2.352.26
REAP average rank2.672.56
HOPE top-1 rate39%46%
HOPE top-2 rate67%67%
HOPE last-place finishes10
REAP last-place finishes54
Freq last-place finishes4344
HOPE mean head-to-head win rate73%76%

HOPE's overall advantage is either unchanged or strengthened, and its single last-place finish disappears. We will add the per-task audit and the re-ranked results to the appendix, and remove the sentence in S4.2 that interprets PopQA results.

Reviewer_VYQvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a combinatorial expert scoring strategy for MoE expert pruning. By collecting expert importance statistics on a calibration dataset, it constructs an interaction matrix between experts (the F matrix), formulates the pruning objective as a quadratic program over the variables representing "which experts to retain," and proves that this objective minimizes an upper bound on the error induced by pruning. The key difference from first-order methods is that the objective retains the off-diagonal (pairwise interaction) terms, which biases the decision toward jointly retaining expert combinations with high "cooperation scores" rather than independently selecting the highest-scoring experts. Evaluation on three frontier MoE models (up to 122B parameters), two calibration sets, and multiple benchmark categories (math, instruction following, coding, and agentic suites) shows that HOPE achieves the best results among five methods at a 50% pruning rate, with particularly strong performance on long-horizon agentic tasks.

Critical strengths

This paper offers a novel perspective, using solid theory to demonstrate the important role of combinatorial experts across various tasks, and proposes the F matrix for computing the importance of expert combinations. By using the importance of expert combinations to guide which rarely co-selected experts should be pruned, it stabilizes model performance while maintaining the original pruning rate. This is to a certain extent a practical and insightful observation.

Critical weaknesses

The core assumption of this paper is that the expert interaction matrix F, estimated from a calibration dataset, can effectively characterize the relative contributions of expert combinations to model behavior, and can therefore guide the selection of expert subsets. However, modern MoE models typically undergo large-scale pretraining and multi-stage post-training, and their expert functions and routing behavior may involve broad task distributions and complex behavioral patterns. It therefore remains to be further justified whether interaction relationships estimated from limited calibration data can reliably reflect the true expert synergies of the model across different tasks and distributions.

Of particular concern is that the distribution of the calibration data may significantly affect the estimation of the F matrix, and thus the final pruning decisions. If the method is sensitive to the task composition, sample size, or sequence length of the calibration data, then its performance advantage on specific benchmarks may partly stem from distributional matching between the calibration data and the evaluation tasks, rather than entirely from combinatorial importance modeling itself.

Questions for authors

The problem this paper addresses is practically important, but the basic assumptions underlying expert pruning still need further clarification. Experts in MoE models are not independently trained modules; they are jointly optimized under a shared training objective, routing mechanism, and contextual dependencies. Therefore, even if certain experts have low activation frequency or low independent importance scores, they may still serve specific domains, long-tail inputs, or particular reasoning stages, and may have a non-negligible impact on the model's overall capability. In this case, how do the authors determine that a given expert or expert combination will not cause hard-to-recover capability loss after pruning? In particular, can the interaction relationships captured by the F matrix identify those experts that are activated infrequently but play a critical role for specific tasks? Can the authors provide concrete examples or quantitative analyses showing how HOPE identifies and retains such long-tail or complementary expert combinations?

1 comment
AuthorsOpenReview ↗

Response to review

We thank the reviewer for the thoughtful questions and concerns. We will address each point below.

Can F (which is estimated on finite calibration data) reflect true expert synergies, or is HOPE's advantage stemming from matching calibration and evaluation distributions?

Any calibration-based method (HOPE, REAP, MAN, etc.) inherits some dependence on its calibration data, but the practical question is whether HOPE's advantage is an artifact of distribution matching versus learning expert interactions. Several lines of evidence indicate that HOPE's advantage is not an artifact of distributional matching:

1) The evaluation is predominantly off-domain. The Evol-CodeAlpaca calibration set consists of single-turn code instructions, yet we evaluate on math, knowledge, instruction following, compositional reasoning, long-turn agentic tasks, etc. HOPE's largest gains over REAP at 50% pruning appear on SWE-Bench Pro, which shares neither format, length, nor tool use with the calibration prompts. Conversely, the SWE-Bench-verified calibration set consists of long agentic trajectories, and HOPE remains competitive on the short-form Tulu tasks.

2) HOPE is no more sensitive to calibration set than first-order methods. That is, HOPE's interaction terms do not rely on distributional differences any more than any existing first-order methods. Prune-sets are highly stable across random trials (Fig. 5a), and across the two very different calibration sets, prune-set overlap is essentially the same for HOPE and REAP (Jaccard 0.60 vs 0.61, Fig. 5b). If HOPE's pairwise statistics were overfit to the calibration distribution, its cross-calibration stability would be lower than REAP's, which we do not observe.

3) The pairwise statistics converge with modest data (Fig. S6).

That said, we agree that, like every calibration-based pruning method, HOPE's decisions depend on the calibration distribution, and we will state this explicitly in the Limitations.

Can HOPE identify experts that are rarely activated but critical for specific tasks?

We analyzed retention of rare experts directly across our three models (Qwen3.5-35B, Qwen3.5-122B and GLM-4.5-Air), using Evol-CodeAlpaca calibration, at 25% and 50% pruning. We find that rare-but-critical experts are largely protected.

We examined experts in the bottom 25% of activation frequency whose contribution when active Fk,kF_{k,k} value (i.e. diagonal of FF) is above the median. Below is the fraction of such experts which are retained:

Retention of rare-but-strong expertsHOPEREAPMANEANFrequency
Qwen3.5-35B, 50%0.880.910.900.070.00
Qwen3.5-122B, 50%0.830.840.870.130.00
GLM-4.5-Air, 50%0.940.940.930.370.00
Range over all three models, 25%0.99–1.000.95–1.000.98–1.000.47–0.840.00

This follows from HOPE's conditional normalization. An expert pair is scored by its contribution when active, not by how often it is active, which is precisely what allows it to save experts which are activated infrequently but are critical. Methods that do not normalize this way (EAN, frequency) lose most of these experts.

We were also able to find examples of experts that are putatively cooperating specialist pairs, which are retained by HOPE and pruned by REAP. In Qwen3.5-122B, upon 25% pruning (Evol-CodeAlpaca calibration), we find a pair of cooperating experts in Layer 45. Expert A is activated only on 0.40% of calibration tokens, but 1.20% of agentic SWE tokens. Its strongest interaction partner, Expert B, is co-selected with it far more than chance (PMI = 5.05 bits, about 33 times the rate that independent routing would predict). Their joint term is large: FA,B≈6FA,AF_{A,B} \approx 6F_{A,A}, which causes HOPE to retain both experts, whereas REAP prunes both.

Here, the pair's joint contribution (which only the off-diagonal terms of FF capture) keeps the specialist pair in the model despite each expert's low individual frequency on coding tokens. These experts are disproportionately used by agentic workloads.

How can pruning avoid causing hard-to-recover capability loss?

We agree that any pruning method risks losing capabilities that the calibration data does not represent, and this is a limitation shared by all pruning methods, not specific to HOPE. In comparison to other pruning methods, however, HOPE has competitive or superior performance, both in terms of breadth (our benchmarks span domains far from the calibration data, and HOPE has the best average rank over all conditions, S4.2), and recoverability (HOPE-pruned models remained better starting points for downstream SFT).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.