- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the effects of post-training compression on Fairness, Bias, and Safety (FBS) in Mixture-of-Experts (MoE) language models. The authors compare Expert Compression (EC)—specifically expert merging (HC-SMoE, M-SMoE) and expert pruning (REAP, Frequency Pruning)—against fine-grained parameter-level compression (weight quantization: AWQ, GPTQ, Dynamic FP8; weight sparsity: WANDA, SparseGPT). Evaluated on three MoE architectures (Qwen3-30B-A3B, ERNIE-4.5-21B-A3B, Gemma-4-26B-A4B), the authors report that:
- Expert merging causes the most severe benchmark regressions on average, whereas expert pruning is less disruptive, and 8-bit quantization is the most stable.
- EC algorithms exhibit high sensitivity to the proportion of safety data present in the calibration set.
- Pruning as few as 3 of 6,144 experts in Qwen3-30B-A3B switches an AdvBench harmful prompt from refusal to compliance, whereas quantizing those same 3 experts to 2 bits maintains refusal.
- Apparent improvements on individual fairness metrics (e.g., lower BBQ ambiguous bias) often mask severe capability collapse.
Critical strengths
- This paper studies a timely and important topic. While compression effects on bias have been studied in dense models, the intersection of MoE routing/expert mechanics with fairness, bias, and safety is an important, emerging area.
- This paper reports multi-seed runs with error bars, which provide helpful insight into variance of post training compression.
- This paper discusses the insight that apparent "improvements" in bias benchmarks often coincide with catastrophic capability degradation.
Critical weaknesses
On benchmark selection
- This paper uses benchmarks that are slightly outdated. For example AdvBench consists of mostly simple direct prompts, which are trivial for modern LLMs as of 2026. Relying solely on AdvBench to make overarching claims about "safety costs" is inadequate. Modern safety research demands standardized, multi-faceted benchmarks that evaluate nuanced risks, multi-turn elicitation, and red-teaming. Some additional benchmarks for consideration: HarmBench, StrongREJECT, WildJailbreak.
- The evaluation is overreliant on substring/Regex matching, which the authors have also discussed in Appendix A, I wish the authors can provide more insights on evaluation metrics themselves.
On general capability retention benchmarks
- The authors only evaluate on IFEval, which is close to saturation and only evaluates a narrow notion of instruction following, i.e., strict format constraints. This does not reflect model's general capability IMO. Please refer to prior works such as XU et al., 2024 and Hong et al., 2024
On problem framing
- This sentence "Pruning just three of Qwen's 6,144 experts (0.049%) is sufficient to change a refusal into compliance... whereas 2-bit quantization does not." IMO is misleading and cheery picking according to appendix F.
- On comparison compression, the authors apply 2-bit quantization to only the 3 isolated experts and remaining 6k experts are uncompressed BF16, this IMO is not a reasonable way to evaluate low-bit quantization.
On experiment setup
- All compression methods are calibrated using only 128 samples with 2k tokens max, this setting is far from sufficient with the model sizes studied.
- The paper tests calibrating compression using 100% safety data (128 purely safety prompts). It is entirely expected that calibrating a compression algorithm exclusively on refusal prompts will overfit expert retention to safety and destroy general capabilities (IFEval collapses by >20–50%). Presenting this as an insightful "trade-off" is unconvincing to me.
On conclusion
- Several prior works have reported similar insights and those are missing from references, for example Xu et al., Findings of EMNLP 2024,
References
- Beyond Perplexity: Multi-dimensional Safety Evaluation of LLM Compression, XU et al., Findings of EMNLP 2024
- Decoding compressed trust: Scrutinizing the trustworthiness of efficient llms under compression. Hong et al., ICML 2024
- Pruning Weights but Not Truth: Safeguarding Truthfulness While Pruning LLMs. Fu et al., Findings of EMNLP 2025
Questions for authors
- Whole model low-bit quantization
- Generalization of refusal experts
- General capabilities evaluation
3 comments
Targeted expert interventions vs whole model quantization. We included whole model 8-bit quantization results in our initial submission to ensure a comparable compression ratio across all compression method families. Our targeted expert quantization interventions at 2 bits were not intended to replace our whole model quantization results nor be representative of an actual deployment scenario. Rather, we were attempting to causally link the absence of specific experts in the REAP pruned checkpoints to negative fairness and safety outcomes. By applying increasingly aggressive quantization and weight sparsity to these same experts, we identify how sensitive these experts are to increasing levels of compression.
To improve the robustness of these results and avoid any potential inadvertent cherry-picking, we repeated the targeted expert interventions over the original request and 20 paraphrases for each of the two cases (Appendix J). Our original 2-bit expert quantizer was symmetric round to nearest with absmax scales and no calibration data, so at 2 bits it keeps three levels and zeroes 84% (one scale per group of 128 weights) to 93% (one scale per output channel) of those experts' weights. Pruning the selected experts flips 16 of 40 refusals, while 2-bit quantization flips 6 with group scales and 9 with channel scales, 4-bit quantization only one, and 8-bit none. None of the ten random control sets per case, drawn from experts active in the refusal and matched by layer, flips any of these refusals. We agree the original sentence overstated the contrast. The revised text reports these counts in place of the original statement that 2-bit quantization preserves refusal. The abstract now states that 8-bit and group-wise 4-bit quantization of the same experts never change the refusal, and 2-bit changes it less often than pruning. We do not claim a general causal mechanism of refusal, but these results highlight that expert pruning appears more capable of removing entire functionalities than fine-grained compression at the same compression ratio.
For a more aggressive whole model quantization setting, we added AWQ W4A16 on all routed expert projections for Qwen and Gemma under all three data mixtures; it stays near the uncompressed level:
| Gemma, default mixture | Expert weight reduction | Disamb. BBQ acc. | IFEval |
|---|---|---|---|
| Uncompressed | 0% | 0.772 | 0.889 |
| AWQ W4A16 | 75% | 0.772 | 0.885 |
| REAP 50% | 50% | 0.681 | 0.651 |
| HC-SMoE 50% | 50% | 0.359 | 0.320 |
AWQ W4A16 is a single run; REAP and HC-SMoE average three seeds. Qwen W4A16 matches its baseline on HumanEval+ (0.902 vs 0.866) and HarmBench (0.112 vs 0.117). These 4-bit results also keep capability at a larger nominal reduction in expert weight storage than expert compression at 50%, reinforcing our finding that fine-grained compression is more robust than expert compression. The W4A16 results now appear in tables 6, 8, 9, and 11.
Calibration size (Appendix I). We use 128 calibration samples for all data mixtures to calibrate the compression methods, the number used by GPTQ and SparseGPT. To test whether expert compression needs more, we recompressed REAP and HC-SMoE at 50% with 1,024 samples for all three models, and with 256 and 512 for Gemma. REAP and HC-SMoE are insensitive to calibration size over this range. Results stay within seed noise across all three models, both methods and all nine metrics, and safety shows no trend with budget. The effects we report persist at eight times the calibration data. See Appendix I and Figure 6 for more details.
Safety-only calibration. We vary the safety data proportion to show that expert pruning and merging are more sensitive to calibration data than quantization and weight sparsity. At 50% compression, the spread of AdvBench refusal across the three mixtures reaches 0.89 for merging and 0.56 for pruning, but at most 0.13 for weight sparsity and 0.02 for quantization. The bias benchmarks show the same sensitivity, in opposite directions across methods: safety-only calibration lowers Qwen frequency pruning's biased rate on SocialStigmaQA from 0.53 to 0.09 but raises Qwen HC-SMoE's from 0.06 to 0.60, while AWQ W4A16 stays within 0.01. We agree that the capability loss under safety-only calibration follows from how REAP and HC-SMoE select experts: Qwen REAP calibrated only on safety data keeps refusal and GSM8K (0.931) but loses coding (HumanEval+ 0.079). A moderate share of safety data largely avoids this cost. The default data mixture, with 21 of 128 samples from safety data, raises AdvBench refusal over no safety data by 0.05 on average without lowering IFEval or BBQ accuracy, so we recommend it for expert compression and concretely state as much in the discussion and conclusion.
Related work. Thank you for pointing out these additional related works. We added Xu et al. (2024) and Fu et al. (2025) and sharpened the comparison with Hong et al. (2024). Those studies compress dense weights. Our study isolates structural expert removal and merging in MoEs, compares it with quantization and sparsity across three architectures, calibration mixtures and seeds, and tests individual experts causally.
Thank you for your detailed review.
Safety beyond AdvBench (Appendix F.1). HarmBench and StrongREJECT now cover all three models. We report HarmBench over its standard and contextual behaviours, and its copyright behaviours separately, since reproducing copyrighted text requires the model to have memorized it. At 50% with the default mixture:
The AdvBench findings therefore hold on held-out and harder benchmarks, and HarmBench also exposes harm that AdvBench misses. We did not use WildJailbreak because Tulu 3 WildJailbreak is one of our three safety calibration subsets, so it would not be held out.
We have not evaluated adaptive attacks or multi-turn attacks and agree these would be interesting extensions. Limitations and Future Work now include reference to both attacks.
Refusal matching. We rescored all 390 runs with three fine-tuned LLM judges. This exposes refusals the phrase list misses (uncompressed ERNIE refuses 0.771 under refusal phrase matching and 0.981 under the WildGuard classifier) and separates harmful compliance from degenerate output. Default ERNIE M-SMoE falls to 0.014 refusal under refusal phrase matching, but only 0.160 of its responses are harmful and 0.741 are degenerate (Appendix E). Refusal phrase matching counts every response without a refusal phrase as compliance, including degenerate output from a model that has lost capability. The LLM judges label harmful compliance directly, so we no longer mistake capability loss for harmful compliance.
General capability (Appendix F.2). GSM8K, HumanEval+ and MBPP+ show that expert compression leaves capabilities highly uneven. Under the default mixture, quantization retains at least 96% of the uncompressed score on every capability measure, and weight sparsity at least 76%, with its largest losses in coding under 2:4 masks. The weakest measure of an expert compression method, usually coding, retains 0 to 91%:
IFEval alone understated how uneven this loss is, which is why we highlight that every bias and safety result must be considered holistically with the capability benchmarks.