- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on speech-to-speech translation, specifically translating full multi-speaker conversations while preserving each speaker's voice. To this end, the authors propose COMUS, which fine-tunes Qwen3-Omni to take an entire recording and predict a voice profile for each speaker in a single pass, followed by speaker-labeled speech tokens for each translated turn. To train this model, the authors construct SODA-SoulX, roughly 26K hours of synthetic English-Chinese conversation pairs plus 310 hours of real meeting recordings paired with synthesized translations. On real two-speaker meetings, COMUS outperforms zero-shot cascaded and hybrid baselines in translation quality and speaker attribution, and a small listening study rates it higher on speaker similarity and naturalness.
Critical strengths
- Most existing speech-to-speech translation handles one speaker at a time, so this paper targets a real gap in the literature. Translating a whole conversation while keeping each speaker's voice consistent across turns is an interesting next step, and doing it end to end avoids cascading errors across separate diarization, ASR, MT, and TTS stages.
Critical weaknesses
- COMUS is fine-tuned on in-domain meeting data (AMI, ICSI, RAMC, and AliMeeting), while all baselines are evaluated zero-shot. Table 13 suggests this fine-tuning accounts for much of the gain. Without it, COMUS scores below Qwen3-Omni S2TT (11.1 vs. 20.8 speaker-agnostic BLEU on AliMeeting), and when AliMeeting is held out of fine-tuning, COMUS still only matches Qwen3-Omni. A cascade built on the same backbone and given the same in-domain adaptation would show how much of the gain comes from the framework itself.
- The evaluation relies heavily on LLM-generated reference translations. Since no human translations exist for the evaluation sets, all BLEU references come from GPT-5.5, while the training translations are produced by Claude. The authors use different LLMs to avoid "style leakage", but BLEU against one LLM's output still rewards translations that resemble it. The only human evaluation (10 listeners rating 10 conversations) is just too small.
Questions for authors
- The reproducibility statement says the derived speech data can't be released because of redistribution restrictions and voice-privacy concerns. Is it possible to at least release the translated transcripts and construction code, so that others can regenerate the dataset?
5 comments
Response to Reviewer AnNW - Part 2/3
Performance on longer conversations. To understand the remaining gap, we grouped conversations into source-duration quartiles, with 50 conversations per direction in each group. The table reports SAtASR-BLEU, which evaluates generated speech through ASR. Differences are Qwen3-Omni-FT S2TT+TTS minus COMUS.
| Source duration (EN→ZH | ZH→EN) | COMUS | Qwen3-Omni-FT S2TT+TTS | Difference |
|---|---|---|---|
| Q1: 10–14 | 10–18 s | 31.5 | 23.1 | 31.6 | 23.4 | +0.2 | +0.3 |
| Q2: 14–23 | 18–27 s | 32.4 | 20.8 | 36.8 | 22.5 | +4.4 | +1.7 |
| Q3: 23–34 | 28–44 s | 33.8 | 18.7 | 35.6 | 21.7 | +1.8 | +3.0 |
| Q4: 34–219 | 46–111 s | 30.0 | 19.0 | 39.6 | 24.0 | +9.6 | +4.9 |
For the shortest conversations (Q1), COMUS and the adapted cascade achieve similar SAtASR-BLEU scores: 31.5 | 23.1 versus 31.6 | 23.4 for EN→ZH | ZH→EN. We did not detect a significant difference in this group (95% confidence intervals for the difference: [−5.1, +5.1] | [−2.8, +3.3]). For the longest conversations (Q4), COMUS obtains 30.0 | 19.0, compared with 39.6 | 24.0 for the cascade. The cascade’s advantages of 9.6 | 4.9 points are the largest observed across the quartiles, and the corresponding confidence intervals exclude zero ([+3.7, +16.2] | [+2.9, +7.1]).
Longer conversations require more generated speech tokens: the average rises from 537 | 819 in Q1 to 2,011 | 3,188 in Q4, with a maximum of 6,843. The percentage of conversations with at least one translated turn lacking corresponding speech rises from 0% in Q1 to 22% | 14% in Q4. This is consistent with a length-related speech-generation limitation, although further analysis is needed to determine how often the decoding limit is responsible.
Speech tokens represent approximately 75% of target tokens overall and 80–85% for conversations longer than 80 seconds. We hypothesize that this imbalance may affect text learning, but we have not tested that explanation directly. We will report long-conversation performance and incomplete speech generation as limitations, and investigate more compact speech representations in future work.
On the automatic speech metrics, COMUS obtains higher predicted naturalness than the fully adapted cascade (UTMOS 3.38 | 3.79 versus 2.85 | 3.48) and higher ZH→EN speaker similarity (0.89 versus 0.86).
Response to Reviewer AnNW - Part 3/3
2. Human validation of the GPT-5.5 references. Five bilingual annotators with translation experience independently assessed 100 short excerpts from 69 conversations for semantic adequacy and naturalness (1–5), as well as major semantic errors. They did not use external LLMs or translation software.
For our SODA, AMI, and AliMeeting data, the sampled references received average scores of 4.73 for adequacy and 4.59 for naturalness, with major errors marked in 1% of annotator judgments.
On XDailyDialog, the GPT-5.5 translations received average adequacy and naturalness scores of 4.90 and 4.89, compared with 4.72 and 4.72 for the published references. Major errors were marked in 0% versus 5% of judgments. Because the two reference types come from different source dialogues, these results are descriptive and do not establish that one is better than the other.
Overall, the assessment supports the quality of the sampled references, although possible single-reference stylistic bias and errors in longer conversations remain. We also report reference-free, LLM-based conv-MQM and an expanded human listening study of system outputs separately.
3. Expanded human listening study. To complement the automatic translation metrics, we expanded the listening study from 10 to 20 conversations (10 per direction) and from 10 to 18 listeners. The selected conversations are 10–35 seconds long, a range that includes approximately 70% of conversations in the objective evaluation set, whose median duration is 25 seconds. Each listener rated all 20 conversations for all four systems, yielding 360 listener–conversation ratings per system and dimension. We also added a prosody-similarity dimension.
| System | Spk.-Turn ↑ | Spk. Sim. ↑ | Pros. Sim. ↑ | Conv. Nat. ↑ | Trans. Qual. ↑ | Overall ↑ |
|---|---|---|---|---|---|---|
| Conv-Cascade-C1 | 0.44* | 2.89* | 2.67* | 2.78* | 2.84* | 2.63* |
| SD+UniSS | 0.45* | 3.04* | 2.70* | 2.59* | 2.56* | 2.41* |
| Qwen3-Omni-FT S2TT+TTS | 0.91 | 3.42 | 3.37 | 3.55 | 3.73 | 3.55 |
| COMUS | 0.92 | 3.49 | 3.45 | 3.50 | 3.72 | 3.52 |
* COMUS is significantly better under the reported analysis: Holm-corrected p < 0.05 and a 95% confidence interval excluding zero.
Translation quality. Listeners rated COMUS higher than Conv-Cascade-C1 and SD+UniSS (3.72 versus 2.84 and 2.56). Against the adapted Qwen3-Omni-FT S2TT+TTS system, the ratings were 3.72 versus 3.73; we did not detect a statistically significant difference.
Other dimensions. Under the reported analysis, COMUS outperforms Conv-Cascade-C1 and SD+UniSS on all six dimensions. Averaged over the five 1–5 scales, the differences are +0.77 [0.50, 1.04] and +0.88 [0.61, 1.13], respectively. Against the adapted S2TT+TTS system, we did not detect a significant difference on any dimension; the five-scale mean difference is +0.01 [−0.19, +0.20] (p = 0.524).
These human ratings complement the automatic metrics, but the study remains limited to 20 conversations of 10–35 seconds. It does not cover the longest conversations where the largest objective performance gap appears. We address the quality of the LLM-generated references separately through human validation.
4. Data release. We have confirmed with our legal team that we will release the code used to construct the SODA-SoulX text data, along with the translated text for the synthetic evaluation set and the AMI and ICSI meeting evaluation sets. We cannot release the remaining data due to our institution's legal restrictions.
5. Updated demo: https://demopage-1.github.io/s2st-rebuttal-samples/
Summary. The new experiments show that COMUS outperforms the strongest evaluated conventional cascade on SAtBLEU, while stronger adapted pipelines using joint SD+ASR or S2TT achieve higher automatic translation scores. Human assessment supports the quality of the sampled GPT-5.5 references, and the expanded listening study provides a complementary evaluation of the generated speech. We will revise the paper to clarify the role of in-domain adaptation and the limitations on longer conversations.
Response to Reviewer AnNW - Part 1/3
We thank the reviewer for highlighting the importance of matched-data baselines and reliable evaluation references. To address these concerns, we adapted stronger Qwen3-Omni baselines using the same real-meeting training data used to adapt COMUS, expanded the listening study, and obtained bilingual human assessments of sampled reference translations.
1. In-domain adaptation and stronger baselines. We agree that our submitted comparisons did not adequately separate the benefit of real-meeting adaptation from other system differences. As the reviewer notes, Table 13 shows that adaptation is important: on AliMeeting, COMUS achieves 11.1 SAgBLEU without final real-data fine-tuning, compared with 20.8 for Qwen3-Omni S2TT. COMUS reaches 25.5 after fine-tuning on all real-meeting training data and 20.9 when fine-tuned on RAMC with AliMeeting held out. The last result shows cross-dataset transfer, but does not establish an advantage over an equally adapted baseline.
To address this concern, we built stronger baselines using Qwen3-Omni, the same backbone as COMUS, and adapted their translation and joint diarization–ASR stages using the same real-meeting training data used to adapt COMUS. The new cascades translate the whole conversation, retain one voice reference per speaker, and use CosyVoice3 for speech generation. Results are reported on real-eval as EN→ZH | ZH→EN.
What do the new comparisons show? Replacing per-turn NLLB with whole-conversation Qwen3-Omni translation raises SAtBLEU from 13.3 | 6.5 to 20.7 | 16.2 with pyannote 3.1 fixed. This comparison changes both the translation model and its context, so it does not measure the effect of context alone. With Community-1 and the known speaker count fixed at two, fine-tuning the translation stage raises SAtBLEU from 21.7 | 16.9 to 27.9 | 22.6.
Replacing Community-1 with the fine-tuned joint diarization–ASR front end further raises SAtBLEU to 37.6 | 27.5, while reducing DER from 27.0 | 18.5 to 7.1 | 4.1. The jointly fine-tuned S2TT+TTS system also performs strongly, reaching 38.3 | 27.1 SAtBLEU. These results demonstrate the value of strong speaker-aware models combined with in-domain adaptation, although changing both the front end and its adaptation does not isolate the effect of joint modeling alone.
The original pyannote 3.1 baseline is also a pretrained meeting-domain system: its segmentation model was trained using the AMI and AliMeeting training sets. This prior exposure does not, however, make its adaptation identical to that of COMUS or the new fine-tuned baselines.
COMUS versus the stronger baselines. COMUS outperforms the strongest evaluated conventional cascade, which uses Community-1 and fine-tuned translation, on SAtBLEU (34.9 | 24.3 versus 27.9 | 22.6). The stronger pipelines with joint SD+ASR or speaker-attributed S2TT achieve higher automatic translation scores than COMUS. They use joint models for speaker-aware understanding followed by separate translation and/or speech-generation stages, while COMUS generates speaker profiles and translated speech tokens in one autoregressive sequence.
Thus, the new results support a system-level advantage over the evaluated conventional cascades, while showing that a strong adapted pipeline with joint speaker-aware components can outperform COMUS on automatic translation metrics. The systems still differ in components, objectives, and training, so the comparison does not isolate architecture alone.