- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper documents severe multilingual degradation after one-shot MoE expert pruning that appears relatively benign on code and English perplexity. It distinguishes response-language failures, answer-extraction failures, and multiple-choice accuracy losses using six model families, four saliency criteria, bilingual evaluation, and translation diagnostics. Fixed-compression expert exchanges show that selected restorations can recover Korean generation substantially earlier than KMMLU accuracy. COER combines cached calibration statistics through mixture synthesis or rank-union, enabling training-free selection with substantially improved multilingual performance under matched observation budgets.
Critical strengths
-
The 2,800 paired English/Korean questions show that many Korean failures remain answerable in English, while teacher-forced translation and source ablations provide complementary evidence about the target-language deficit (Sections 4–5; Appendix C.5). Together with cross-family and cross-criterion tests, this makes the result much stronger than an isolated language-switching example.
-
The fixed-compression exchanges separate restored-expert selection from removal choices, compare multiple restoration policies, and reproduce the main ordering across two Qwen models. Matched-budget and same-pool controls also distinguish observation diversity from the aggregation rule (Sections 6–7; Tables 2 and S19–S22). The resulting cached-statistics remedy is practical, and its limitations are reported transparently.
Critical weaknesses
-
Generic English natural-language calibration already substantially repairs Korean fluency, and Korean-only calibration improves English accuracy (Section 6; Appendices C and C.5). The authors acknowledge the language/domain confound, but there is no matched code-plus-English-natural-language control against code-plus-Korean. Consequently, the evidence supports broader observation coverage more firmly than a specifically multilingual calibration prescription; this limits the precision of the practical recommendation.
-
Exchanges are restricted to the disagreement set defined by the COER endpoint and the tested restoration policies (Section 7). The additional collapsing model, Qwen3.6, has only a 1.7-point endpoint KMMLU gap and one calibration seed, while the ERNIE ordering differs. These acknowledged qualifications make the result a compelling intervention case study, but provide weaker support for broader conclusions about how language and knowledge capabilities are distributed across experts.
Questions for authors
None
Thank you for your response and revisions. I have decided to maintain my original score of 2.