Mitigating the Curse of Multilinguality in Many-to-Many Speech-to-Text Translation

Multimodal large language models (MLLMs) have achieved significant success in speech-to-text translation (S2TT). However, when processing multilingual speech inputs, a single speech encoder shared across all languages suffers from the curse of multilinguality: languages at different resource levels compete for limited representation capacity, leading to strong high-resource performance but substantial degradation on low-resource speech. To address this problem and improve multilingual consistency, we propose MSRT, a novel framework built around a resource-aware Mixture of Speech Encoders (MoSE). MoSE uses an explicit language router to assign each utterance to an appropriate expert encoder. A frozen expert preserves high-resource language capabilities, while a trainable expert adapts to and specializes in medium- and low-resource languages. We further introduce a five-stage curriculum learning strategy that substantially reduces data dependence, requiring under 10 hours of paired S2TT data per language on average for effective alignment. We conduct extensive experiments on 45 languages, systematically evaluating all $45\times 44$ translation directions. Our models, spanning 4B and 12B parameters, achieve state-of-the-art performance, outperforming substantially larger open-source models. Empirical analyses show that MoSE improves high-, medium-, and low-resource languages simultaneously, with the largest gains on low-resource speech, thereby mitigating the curse of multilinguality without compromising high-resource performance. To support future multilingual research, we will release our code and models.