- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the “curse of multilinguality” in many-to-many speech-to-text translation (S2TT), where covering many languages degrades per-language quality, especially for low-resource languages. The authors propose MSRT, a multilingual speech LLM built around a resource-aware Mixture of Speech Encoders (MoSE). MoSE uses two homogeneous Whisper encoders: a frozen expert for high-resource languages and a trainable expert adapted to medium- and low-resource languages. Source language tokens route each utterance to one expert at inference. The proposed models MSRT-4B and MSRT-12B reach the highest average COMET scores and outperform the Gemini-3.5-Flash-Lite API and the much larger MCAT-27B. Evaluation covers all 1,980 directions across 45 languages.
Critical strengths
-The idea is simple (mixture of speech encoders with an input language token) and well motivated. The experiments show that it works and that it mitigates the curse of multilinguality described in the introduction.
-The evaluation is broad: 45 languages and all 1,980 directions, with competitive baselines.
-The ablation in Fig. 4 is informative. It compares a baseline with a single Whisper encoder to the 2-encoder and even a 3-encoder approach (testing finer separation into high / medium / low speech experts).
-Routing by language tokens keeps inference compute at that of a single encoder, which is practically attractive.
Critical weaknesses
-The five-stage curriculum is not described clearly enough. It is to me the key ingredient, yet I could only grasp it in spirit from Fig. 2 and Sec. 3.2. Missing are the exact data used at each stage for instance. I also noticed some inconsistencies in Table 1: =>L.211–212: Stages I and II are described as multilingual ASR (Stage I as English-only, then high-resource, then all languages), but the language token shown is always English. Is this a typo, or i missed something? =>L.213–214: the Stage IV and V rows are identical, although the text describes different stages. I understand that they differ in trainable modules (LLM LoRA is enabled in Stage V), so I suggest adding a “trainable modules” column to Table 1.
-No curriculum ablation: the five stages are presented as a key ingredient, but nothing tests them especially are all stages equally important ?
-A critical weakness: the LLM backbone is not identified. L.243 says “MSRT pairs a pretrained multilingual LLM (Shang et al., 2026) with MoSE and a speech adapter,” but the paper never says which LLM underlies the 4B and 12B models. Since the speech encoder is Whisper, the LLM is the main remaining variable. Without it I cannot judge whether the gains over baselines such as MCAT-27B come from MoSE or from a stronger backbone.
-Two encoders need more memory than one, even if compute per utterance is the same. Does “4B” and "12B" include both encoders? Maybe speech encoder parts are negligeable compared to LLM size but say it if so !
-Which Whisper version is used for the experts? None of this is stated.
-Routing relies on the source-language token, so the source language must be known at test time (acknowledged in the Limitations though)
-The single-encoder S2TT model (the one in the ablation of Fig. 4) should also be in the main table of results (table 3), otherwise it is a bit misleading. It scores 80.3 COMET, still above MCAT-27B (79.6) and Gemini (78.3). So part of the SOTA claim could come from the LLM backbone (which is unknown), not from MoSE.
Questions for authors
-Which Whisper version is used for the experts? None of this is stated.
-Which LLM backbones are you using ?
-Fig. 4: MoSE improves all resource groups over the single-encoder S2TT model (+3.0 low, +2.2 medium, +1.4 high). Low-resource gains being largest is expected, but why do high-resource languages improve at all (even English, +0.7), given that they go through the frozen, original Whisper encoder?
-Why does a third expert (one per resource group: high / medium / low) not pay off?
Minor remarks and typos
L.113: “End-to-end S2TT instead learns a direct mapping from source speech to target text.” Please add a reference here (the 1st e2e S2T appraoches to translation) L.191: a reference is missing for the Q-Former (“A Q-Former com-…”).