acceptodds
Under review as a conference paper at ICLR 2027

RAZOR: Pruning Replaceable Experts in LLMs

Abstract

Mixture-of-experts (MoE) models activate few experts per token but store the full expert pool. Pruning aims to reduce this burden at a fixed budget while preserving output distributions and, for reasoning models, reasoning ability. Deletion damage depends on functional replaceability by surviving computation, not usage or contribution magnitude alone. We introduce RAZOR, a training-free method that scores functional replaceability using consensus residuals, the deviations of expert outputs from the original weighted mixture. At a fixed layer input, an exact single-deletion identity accounts for survivor renormalization and router-selected refill. We aggregate these local scores over calibration tokens for budgeted pruning without gradients or recovery training. On GLM-4.7-Flash, Qwen3.6-35B-A3B, DeepSeek-V4-Flash-0731, and Hy3 at 25% and 50% expert removal, RAZOR achieves the highest nine-task macro average among the evaluated pruning methods in all eight settings. On the two backbones with matched REAP benchmark runs, it exceeds REAP by 2.12–5.59 points and wins all 36 paired task comparisons. It also lowers reverse KL relative to REAP in all four matched GLM-4.7-Flash and Qwen3.6-35B-A3B model–budget settings. Responses generated by Qwen3.6-35B-A3B nevertheless change in diversity, formatting, and termination, showing that task retention and predictive fidelity do not ensure generation stability.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, MoE, Model Pruning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_LVJmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a training-free method for pruning experts in MoE LLMs. It estimates the change in the MoE layer output when an expert is removed. The score considers the remaining experts and the replacement expert selected by the router. It evaluates on four MoE models at 25% and 50% expert removal. They report better downstream performance than several pruning baselines.

Critical strengths
  1. The method requires no recovery training.

  2. The paper examines output-distribution changes, different scoring components, and generation behavior, providing informative empirical analysis.

Critical weaknesses
  1. The novelty is limited. The method remains a one-shot, per-expert ranking approach similar to REAP. More importantly, it does not address the interactions between experts when many experts are removed together, which is the actual optimization problem in MoE pruning.

  2. The proposed importance score is not well aligned with the actual pruning objective. The final method removes 25–50% of experts simultaneously, why? The damage caused by removing a set of experts is not the sum of their individual deletion damages. Replacement experts may themselves be removed, surviving experts interact, and changes in earlier layers affect routing in later layers.

  3. The paper has not established that its more detailed deletion counterfactual is the main reason for the improvements, rather than the aggregation choice or other effects of expert ranking.

Questions for authors

Why is conditional RMS the appropriate score for preserving the overall output distribution?

Reviewer_1dhEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes RAZOR, a training-free method for compressing large MOE models by removing least useful experts. Instead of judging experts only by how often they are picked or how large their outputs are, the authors score each expert based on how well the remaining experts and router replacements can cover for its missing output. This approach achieves higher accuracy on problem-solving benchmarks compared to existing expert pruning techniques. The study also reveals that while model accuracy remains high, expert pruning can still alter text generation behaviors such as response length and formatting.

Critical strengths
  1. The paper introduces a training-free way to measure an expert's value by looking at how well surviving experts and router replacements can cover for its missing output.
  2. The math is clear. The method is evaluated on four different MoE models and nine benchmarks.
  3. It provides valuable measurements showing that high benchmark scores do not guarantee stable text generation. It uncovers issues like stray formatting tags and responses hitting the length limit.
  4. The scoring process requires only standard forward passes over calibration tokens, making it simple to apply.
Critical weaknesses
  1. The original models score below what their developers report.
  2. Comparisons with other pruning methods are limited.
  3. No error bars or significance tests are reported, and many of the gains are very small.
  4. The single-expert scoring formula does not account for interactions between multiple deleted experts, meaning two experts that look safe to drop individually might cause issues when removed together.
Questions for authors
  1. Since error builds up across layers, did you consider sequential pruning?
  2. Why does matching the model's overall output distribution using reverse KL divergence sometimes lead to worse perplexity on specific reference words?
  3. Can a quick round of light fine-tuning recover the performance lost when removing fifty percent of the experts?
1 comment
AuthorsOpenReview ↗

Gentle Follow-Up on RAZOR

Gentle Follow-Up on RAZOR

Dear Reviewer 1dhE ,

We sincerely appreciate your time and consideration!

As the discussion period is now less than 12 hours from closing, we would like to kindly follow up and invite you to revisit our responses to see whether your concerns have been adequately addressed. Your feedback at this stage would allow us sufficient time to undertake any potential additional work (e.g., experiments or further writing), and we really cherish your insightful advice.

Thank you again for your valuable time and kind consideration!


Best regards,

Authors of Paper 54898 (RAZOR)

Reviewer_GnP1OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies training-free whole-expert pruning of MoE reasoning models. It argues that magnitude scores such as REAP's routing-weighted output norm cannot identify the least harmful deletion. Instead it scores experts by "consensus residuals" ri=fi−cr_i = f_i - c, each expert's deviation from the routed mixture. These give exact single-deletion output changes at a fixed layer input, under survivor renormalization (RCS-LOO, Eq. 3) and under router refill by the rank-(k+1)(k{+}1) expert (RCS-Refill, Eq. 2). RAZOR is RCS-Refill with each expert's token-level damage aggregated by conditional RMS, keeping the top-BB experts per layer. Across GLM-4.7-Flash, Qwen3.6-35B-A3B, DeepSeek-V4-Flash-0731 and Hy3 at 25% and 50% removal, RAZOR has the best nine-task macro average among the evaluated methods. On the two backbones where REAP was benchmarked, it beats REAP by 2.12–5.59 points and wins all 36 task comparisons. The paper also reports reverse-KL/PPL fidelity, diversity, and changes in response form and termination.

Critical strengths
  1. The local analysis is clear. Prop. 1 gives a clean counterexample showing that gates and norms cannot identify the least harmful deletion. Props. 2–3 and Eq. 4 are correct. The constructed example in Fig. 1/App. A.1 checks out exactly

  2. There is consistent benchmark advantage across modern, large backbones. RAZOR has the top macro score in all eight settings. Against matched REAP it wins 36/36 tasks, and against RCS-LOO it wins 61/72. My recomputation from the tables confirms these counts. The external REAP⋆ checkpoint (41.7 vs. 42.4 for the in-house REAP) is a useful check on the reimplementation.

Critical weaknesses
  1. The baseline coverage is thin. On DeepSeek-V4 and Hy3 only the authors' own three variants are benchmarked, so half of the "best in all eight settings" claim is a self-comparison, even though REAP masks for these models exist. The paper stresses that deletions interact, yet there is no iterative re-scoring or reconstruction-based selection baseline. Merging methods or close prior method like MoNE (Zhang et al., ICLR 2026) are not cited or compared.

  2. The gain is not attributed to replaceability scoring. RAZOR differs from REAP in both the token quantity and the aggregation (mean→RMS). In the paper's own Table 9, REAP's quantity with RMS (REAP-RMS) matches or beats RAZOR on Qwen. On GLM, RAZOR beats REAP-RMS by only about 1.7%. Moving REAP from mean to RMS alone cuts KL by 4–15%, which is essentially all of RAZOR's KL advantage. In Table 6, plain RCS has the lowest KL on Qwen at both budgets. The direct test of deletion damage under deployed routing also shows no advantage over REAP. Yet REAP-RMS is never benchmarked, so the 2.1–5.6 point downstream gains cannot be credited to the consensus residual or refill.

Questions for authors
  1. Can you report downstream results for REAP-RMS (wi∥fi∥2w_i\|f_i\|_2 with conditional RMS), and ideally RCS with conditional mean, on GLM and Qwen at 25%/50%?

  2. How many experts per layer differ between the RAZOR and RCS-LOO checkpoints, and between RAZOR and REAP?

  3. What is the wall-clock calibration cost of RAZOR vs. REAP on the 284B/295B models?

1 comment
AuthorsOpenReview ↗

Response to Reviewer GnP1(2/4)

Q2. Separating the scoring rule from mean/RMS aggregation

A2. Thank you for identifying this essential control! We fully agree that the effects of scoring and aggregation should be distinguished. REAP scores routing-weighted output magnitude; RAZOR scores the local mixture perturbation remaining after deletion, renormalization, and replacement. It also changes aggregation from conditional Mean to RMS. We therefore compare both aggregation choices for each scoring rule.

Equal-token KL comparison. Your suggestion prompted us to revisit the KL evaluation and examine token balance more closely. Equal numbers of examples per domain do not guarantee equal numbers of scored tokens, because response lengths vary. We therefore match scored-token counts across all eight domains, while retaining the paper's equal weighting of domain means. Building on the earlier reverse-KL comparison, we additionally report forward KL to examine output distributions in both directions.

We compare all four scoring rules under both conditional Mean and RMS aggregation, including REAP-RMS; RAZOR corresponds to RCS-Refill-RMS. Within each model, all variants, pruning ratios, and KL directions use the same token positions. Lower is better; bold marks the minimum within each aggregation group.

Qwen3.6-35B-A3B

ScoreAggregationForward KL, 25%Forward KL, 50%Reverse KL, 25%Reverse KL, 50%
REAPMean0.0417080.2690830.0524900.414679
RCSMean0.0462250.2726330.0566700.405998
RCS-LOOMean0.0438820.2717970.0544480.398110
RCS-RefillMean0.0418960.2593050.0518400.389008
REAPRMS0.0299450.2188670.0348460.302848
RCSRMS0.0357800.2332210.0429760.318986
RCS-LOORMS0.0337720.2273190.0398730.305310
RCS-RefillRMS0.0253220.2070690.0296180.279137

GLM-4.7-Flash

ScoreAggregationForward KL, 25%Forward KL, 50%Reverse KL, 25%Reverse KL, 50%
REAPMean0.1726140.6901270.2187081.121620
RCSMean0.1841620.6555510.2268301.041235
RCS-LOOMean0.1740660.6572320.2189281.041335
RCS-RefillMean0.1690590.6589570.2127611.052076
REAPRMS0.1449830.5503110.1757430.855970
RCSRMS0.1596640.6430690.1931351.004211
RCS-LOORMS0.1407780.5587770.1714490.859803
RCS-RefillRMS0.1244440.5204920.1510280.790694

Key Takeaway: aggregation accounts for a substantial part of the measured KL improvement. With REAP's scoring rule fixed, changing Mean to RMS lowers KL by 16.0%–33.6% across both models, budgets, and directions. All three residual-based rules also benefit on these positions. This control confirms the reviewer's concern: comparisons against standard REAP reflect both scoring and aggregation changes; attribution to the scoring criterion requires holding aggregation fixed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.