acceptodds
Under review as a conference paper at ICLR 2027

Lost in Pruning: Silent Multilingual Collapse and Training-Free Recovery for One-Shot MoE Pruning

Abstract

One-shot expert pruning shrinks the resident memory of sparse mixture-of-experts (MoE) models without retraining and is validated on benchmarks that resemble its calibration data. We show that this validation misses a silent multilingual collapse: a 25%-pruned Qwen3-30B-A3B keeps code near baseline and English perplexity within ×1.6, yet answers Korean prompts in Chinese. The damage is a loss of language access rather than of knowledge alone: on 2,800 parallel MMLU-ProX items the pruned model loses 56 points in Korean but 15 in English, and still answers 74% of its Korean losses correctly in English. Fluency, knowledge, answer extraction, and translation direction fail differently; on this model generative MMLU-ProX also falls to near zero in Japanese and French, generation into an uncovered language degrades far more than translation out of it, and the collapse recurs across six MoE architectures and four scoring criteria, implicating calibration coverage rather than any single scoring rule. Controlled expert exchanges at fixed compression, each designed before its checkpoints were built, probe this account directly: restoring 28% of the experts on which the default mask and a multi-view coverage mask disagree closes about 80% of the Korean NLL gap between the two masks and restores Korean-script responses, but only 11% of the KMMLU gap. This language-before-knowledge ordering recurs on a 256-expert model, where it was predicted before the run, and is absent in a model whose default pruning does not collapse Korean generation. Building on this analysis, Cross-Observation Expert Retention (COER) recomposes cached per-domain calibration views on CPU without raw text or retraining; at matched compression and observation budgets (three seeds on Qwen3) it restores target-language generation under all four criteria, generalizes to five languages without labeled calibration examples in those languages, and recovers 34–77% of lost HAERAE accuracy under gate weighting, with rank-union retaining code at or above the Qwen3 default.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mixture-of-experts, expert pruning, one-shot compression, multilingual robustness, silent failure, calibration coverage, language access vs knowledge, training-free recovery

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_eE3wOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper documents severe multilingual degradation after one-shot MoE expert pruning that appears relatively benign on code and English perplexity. It distinguishes response-language failures, answer-extraction failures, and multiple-choice accuracy losses using six model families, four saliency criteria, bilingual evaluation, and translation diagnostics. Fixed-compression expert exchanges show that selected restorations can recover Korean generation substantially earlier than KMMLU accuracy. COER combines cached calibration statistics through mixture synthesis or rank-union, enabling training-free selection with substantially improved multilingual performance under matched observation budgets.

Critical strengths
  1. The 2,800 paired English/Korean questions show that many Korean failures remain answerable in English, while teacher-forced translation and source ablations provide complementary evidence about the target-language deficit (Sections 4–5; Appendix C.5). Together with cross-family and cross-criterion tests, this makes the result much stronger than an isolated language-switching example.

  2. The fixed-compression exchanges separate restored-expert selection from removal choices, compare multiple restoration policies, and reproduce the main ordering across two Qwen models. Matched-budget and same-pool controls also distinguish observation diversity from the aggregation rule (Sections 6–7; Tables 2 and S19–S22). The resulting cached-statistics remedy is practical, and its limitations are reported transparently.

Critical weaknesses
  1. Generic English natural-language calibration already substantially repairs Korean fluency, and Korean-only calibration improves English accuracy (Section 6; Appendices C and C.5). The authors acknowledge the language/domain confound, but there is no matched code-plus-English-natural-language control against code-plus-Korean. Consequently, the evidence supports broader observation coverage more firmly than a specifically multilingual calibration prescription; this limits the precision of the practical recommendation.

  2. Exchanges are restricted to the disagreement set defined by the COER endpoint and the tested restoration policies (Section 7). The additional collapsing model, Qwen3.6, has only a 1.7-point endpoint KMMLU gap and one calibration seed, while the ERNIE ordering differs. These acknowledged qualifications make the result a compelling intervention case study, but provide weaker support for broader conclusions about how language and knowledge capabilities are distributed across experts.

Questions for authors

None

Reviewer_GFsxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper shows that one-shot MoE expert pruning calibrated on English code silently destroys access to uncalibrated languages. A 25%-pruned Qwen3-30B keeps code and English PPL but answers Korean in Chinese. On paired MMLU-ProX items it loses 56 points in Korean versus 15 in English. The effect is shown across 6 MoE families and 4 criteria. The authors trace the failure to target-side generation and propose COER, which composes cached per-domain calibration statistics (closed-form mixture or max-rank union). Pre-registered expert-exchange experiments show that language recovers before knowledge.

Critical strengths
  1. Important, well-evidenced failure mode. The paired EN/KO probe cleanly separates loss of language access from loss of knowledge. It also shows that calibration-aligned validation cannot certify pruned MoEs.
  2. Strong experimental hygiene. The paper uses multi-seed, budget-matched, same-pool and leave-language-out controls, pre-committed exchange designs, and honest negative results.
Critical weaknesses
  1. The method contribution is thin. The paper's own same-pool control shows the composition rule barely matters given a diverse pool. The closed-form synthesis is a trivial weighted-mean identity. The practical takeaway reduces to "add multilingual calibration data", which is expected. A single broad multilingual view (e.g., Aya-only) is not reported as a full baseline.
  2. Dense presentation and weaker mechanism claims. Numbers are hard to reconcile across sections: koPPL is ×14.4 vs. ×17.8, and Fig. S5 does not match Table 2. The Section 7 candidate set is defined through C, and routing disturbance is not excluded. The language-before-knowledge claim is therefore weaker than the abstract suggests.
Questions for authors
  1. What are the full-axis results for an Aya-only (or multilingual C4) view at a matched budget? What does composition add beyond data choice?
  2. Does the language-before-knowledge ordering hold when C is a mixture-derived mask?
  3. Knowledge recovers only under REAP. Does this undercut the "criterion-agnostic" claim?

Comments on the paper

2 comments

Thank you for your response and revisions. I have decided to maintain my original score of 2.

AuthorsOpenReview ↗

Thank you for your review and feedback

Thank you for taking the time to review our responses and the revised manuscript. We appreciate your careful review and the questions that helped us improve the experimental comparisons and clarify the paper's conclusions. We will continue to investigate multilingual robustness in expert pruning, building on this discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.