acceptodds
Under review as a conference paper at ICLR 2027

Mitigating the Curse of Multilinguality in Many-to-Many Speech-to-Text Translation

Abstract

Multimodal large language models (MLLMs) have achieved significant success in speech-to-text translation (S2TT). However, when processing multilingual speech inputs, a single speech encoder shared across all languages suffers from the curse of multilinguality: languages at different resource levels compete for limited representation capacity, leading to strong high-resource performance but substantial degradation on low-resource speech. To address this problem and improve multilingual consistency, we propose MSRT, a novel framework built around a resource-aware Mixture of Speech Encoders (MoSE). MoSE uses an explicit language router to assign each utterance to an appropriate expert encoder. A frozen expert preserves high-resource language capabilities, while a trainable expert adapts to and specializes in medium- and low-resource languages. We further introduce a five-stage curriculum learning strategy that substantially reduces data dependence, requiring under 10 hours of paired S2TT data per language on average for effective alignment. We conduct extensive experiments on 45 languages, systematically evaluating all translation directions. Our models, spanning 4B and 12B parameters, achieve state-of-the-art performance, outperforming substantially larger open-source models. Empirical analyses show that MoSE improves high-, medium-, and low-resource languages simultaneously, with the largest gains on low-resource speech, thereby mitigating the curse of multilinguality without compromising high-resource performance. To support future multilingual research, we will release our code and models.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Speech-to-Text Translation, Multilinguality, Mixture of Experts

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qC5BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the “curse of multilinguality” in many-to-many speech-to-text translation (S2TT), where covering many languages degrades per-language quality, especially for low-resource languages. The authors propose MSRT, a multilingual speech LLM built around a resource-aware Mixture of Speech Encoders (MoSE). MoSE uses two homogeneous Whisper encoders: a frozen expert for high-resource languages and a trainable expert adapted to medium- and low-resource languages. Source language tokens route each utterance to one expert at inference. The proposed models MSRT-4B and MSRT-12B reach the highest average COMET scores and outperform the Gemini-3.5-Flash-Lite API and the much larger MCAT-27B. Evaluation covers all 1,980 directions across 45 languages.

Critical strengths

-The idea is simple (mixture of speech encoders with an input language token) and well motivated. The experiments show that it works and that it mitigates the curse of multilinguality described in the introduction.

-The evaluation is broad: 45 languages and all 1,980 directions, with competitive baselines.

-The ablation in Fig. 4 is informative. It compares a baseline with a single Whisper encoder to the 2-encoder and even a 3-encoder approach (testing finer separation into high / medium / low speech experts).

-Routing by language tokens keeps inference compute at that of a single encoder, which is practically attractive.

Critical weaknesses

-The five-stage curriculum is not described clearly enough. It is to me the key ingredient, yet I could only grasp it in spirit from Fig. 2 and Sec. 3.2. Missing are the exact data used at each stage for instance. I also noticed some inconsistencies in Table 1: =>L.211–212: Stages I and II are described as multilingual ASR (Stage I as English-only, then high-resource, then all languages), but the language token shown is always English. Is this a typo, or i missed something? =>L.213–214: the Stage IV and V rows are identical, although the text describes different stages. I understand that they differ in trainable modules (LLM LoRA is enabled in Stage V), so I suggest adding a “trainable modules” column to Table 1.

-No curriculum ablation: the five stages are presented as a key ingredient, but nothing tests them especially are all stages equally important ?

-A critical weakness: the LLM backbone is not identified. L.243 says “MSRT pairs a pretrained multilingual LLM (Shang et al., 2026) with MoSE and a speech adapter,” but the paper never says which LLM underlies the 4B and 12B models. Since the speech encoder is Whisper, the LLM is the main remaining variable. Without it I cannot judge whether the gains over baselines such as MCAT-27B come from MoSE or from a stronger backbone.

-Two encoders need more memory than one, even if compute per utterance is the same. Does “4B” and "12B" include both encoders? Maybe speech encoder parts are negligeable compared to LLM size but say it if so !

-Which Whisper version is used for the experts? None of this is stated.

-Routing relies on the source-language token, so the source language must be known at test time (acknowledged in the Limitations though)

-The single-encoder S2TT model (the one in the ablation of Fig. 4) should also be in the main table of results (table 3), otherwise it is a bit misleading. It scores 80.3 COMET, still above MCAT-27B (79.6) and Gemini (78.3). So part of the SOTA claim could come from the LLM backbone (which is unknown), not from MoSE.

Questions for authors

-Which Whisper version is used for the experts? None of this is stated.

-Which LLM backbones are you using ?

-Fig. 4: MoSE improves all resource groups over the single-encoder S2TT model (+3.0 low, +2.2 medium, +1.4 high). Low-resource gains being largest is expected, but why do high-resource languages improve at all (even English, +0.7), given that they go through the frozen, original Whisper encoder?

-Why does a third expert (one per resource group: high / medium / low) not pay off?

Minor remarks and typos

L.113: “End-to-end S2TT instead learns a direct mapping from source speech to target text.” Please add a reference here (the 1st e2e S2T appraoches to translation) L.191: a reference is missing for the Q-Former (“A Q-Former com-…”).

Reviewer_sGPYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces MSRT, a multilingual speech-to-text translation model designed to reduce the performance gap between high and low-resource languages. It introduces a Mixture of Speech Encoders (MoSE): a frozen encoder preserves high-resource language capabilities, while a trainable encoder specializes in medium and low-resource languages. The frozen encoder is based on Whisper. The second encoder is initialized from Whisper and further adapted. The proposed model is compared to gemini 3.5 flash lite, whisper + nllb cascade, and 3 e2e baselines (seamlessm4t, qwen omni, mcat) and is shown to outperforms those baselines (experiments are run on 45 * 44 directions and evaluated on FLEURS). Ablations include one expert vs 2, adding a third expert, speech translation data scaling, etc.

Critical strengths
  • the proposed idea is simple and effective: resource-aware specialization of speech encoders directly addresses competition between high and low-resource languages in a shared multilingual model.
  • strong empirical results: the proposed approach outperforms relatively strong baselines, some of them having substantially more parameters.
  • thorough evaluation: tests 45x44 directions
Critical weaknesses
  • the 2 encoder design seems adhoc. It would be good to investigate learning a gate or designing alternative groups (for example linguistically motivated, etc.) as in this work which seems very related: https://arxiv.org/abs/2607.24030 (roughly concurrent work, but should still be cited)
  • the baseline design is a bit unclear. For example, none of the baselines are comparable from a data perspective (arguably the first ablation is the comparable baseline). Why choose gemini 3.5 flash lite and not Gemini 3.5 Live Translate for example, etc. For the cascade, arguably there are stronger components than Whisper and NLLB.
Questions for authors
  • Could you evaluate on other sets than FLEURS, which would have non read speech (mtedx, etc.)
  • pretraining on common voice is sensible but common voice is mostly read speech, which may make this model not as robust to noise (and this is not tested because the test set is FLEURS which is also read speech only, see question above).
Reviewer_CLsSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a way to combine pre-trained Whisper encoders with an LLM for many-to-many speech-to-text translation (S2TT). The authors argue that existing models perform poorly on low-resource source languages. To address this, they use two Whisper-based encoders: a frozen one for high-resource languages, which preserves pre-trained knowledge, and a trainable one for medium- and low-resource languages. Each utterance is routed to either of the encoders according to a pre-defined list of high-resource languages. With the proposed curriculum training recipe, the 4B and 12B models outperform prior work, including SeamlessM4T-v2, Qwen3-Omni-30B and MCAT-27B, on all 45×44 FLEURS directions in both spBLEU and COMET. The experiments show the largest gains on low-resource languages and validate the effectiveness of using two separate encoders.

Critical strengths
  • Broad evaluation covering all 1,980 language directions.
  • The method is technically sound and the implementation is mostly clear and easy to follow.
Critical weaknesses

Limited novelty

The method is called a mixture of experts (MoE), but it differs substantially from a standard MoE. Typical MoE models stack many MoE layers, each with several trainable experts and a learned router that weights them by input. Here there is a single unit, routing is fixed by each language’s pre-defined resource level, and which encoder is trainable is also decided in advance. The authors acknowledge that no learned gate is used, but the MoE label still overstates the design. It is better described as a dual-encoder architecture, with one frozen encoder for high-resource languages and one trainable for low-resource languages. Since speech encoders are usually less well-optimized across languages than the LLM backbone, so it is not surprising that fine-tuning an encoder on low-resource speech improves translation. It is also unclear whether the shared-encoder baseline in Fig. 4 is trained or frozen. If it is frozen, the reported gain may come from fine-tuning alone rather than from splitting the encoders.

Key design choice not justified

The paper stresses several times that the high-resource encoder must stay frozen to preserve its capabilities, but no experiment isolates this. The expert-count ablation varies the number of encoders but does not test freezing. A simple check would be to make both encoders trainable and keep the rest of the recipe unchanged. If high-resource performance then drops, that would directly support the authors’ claim.

Missing details

The paper also does not explain how the 45 languages were chosen, which matters for interpreting the low-resource claims. The 44 non-English sources are almost all a subset of SeamlessM4T-v2’s 88 FLEURS languages. However, they exclude every Sub-Saharan African language and most Indic languages, and the “low-resource” group includes Swedish, Norwegian and Bulgarian. Specifying the backbone and the language-selection criteria, and ideally reporting results on harder languages, would make the results easier to reproduce and the claims more convincing.

In-domain training data

MSRT is trained on FLEURS S2TT training pairs and evaluated on the FLEURS test set, but several other baselines, such as Gemini-3.5-Flash-Lite and Qwen3-Omni, were presumably not trained on FLEURS. For example, Qwen3-Omni-30B officially supports speech input in only 19 languages, so 26 or more of the 45 source languages in this evaluation are outside its coverage -- this explains its low scores on some sources.

Questions for authors

See Weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.