acceptodds
Under review as a conference paper at ICLR 2027

Towards End-to-End Conversational Multi-Talker Speech-to-Speech Translation

Abstract

We present COMUS, an end-to-end conversational multi-talker S2ST framework that leverages full contextual information to jointly model semantic meaning, speaker attribution, and prosodic characteristics for end-to-end multi-talker S2ST. Built on Qwen3-Omni, COMUS adapts the Thinker to autoregressively predict a single stream of text, speaker-label, and BiCodec speech tokens, replacing the original multi-token speech-generation module. Specifically, given the entire recording, COMUS predicts speaker profile tokens for each speaker alongside semantic tokens tagged with speaker labels, indicating speaker turns within a single autoregressive generation process. To provide the required supervision, we construct SODA-SoulX, comprising approximately 26K hours of paired speaker-consistent conversational speech across English and Chinese. Experiments on real conversational datasets, including AMI and AliMeeting, demonstrate that COMUS achieves speaker-attributed ASR-BLEU of 31.9 and 20.4 for English-to-Chinese and Chinese-to-English translation, respectively, outperforming all baseline methods. Further evaluations show translation advantages under noise and overlapping speech. Subjective listening tests show consistent advantages of COMUS in speaker similarity, translation accuracy and conversational naturalness over baseline methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Speech-to-speech translation, Conversational speech translation, Multi-talker speech processing, End-to-end speech generation, Speaker-aware modeling, Multimodal large language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_AnNWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on speech-to-speech translation, specifically translating full multi-speaker conversations while preserving each speaker's voice. To this end, the authors propose COMUS, which fine-tunes Qwen3-Omni to take an entire recording and predict a voice profile for each speaker in a single pass, followed by speaker-labeled speech tokens for each translated turn. To train this model, the authors construct SODA-SoulX, roughly 26K hours of synthetic English-Chinese conversation pairs plus 310 hours of real meeting recordings paired with synthesized translations. On real two-speaker meetings, COMUS outperforms zero-shot cascaded and hybrid baselines in translation quality and speaker attribution, and a small listening study rates it higher on speaker similarity and naturalness.

Critical strengths
  • Most existing speech-to-speech translation handles one speaker at a time, so this paper targets a real gap in the literature. Translating a whole conversation while keeping each speaker's voice consistent across turns is an interesting next step, and doing it end to end avoids cascading errors across separate diarization, ASR, MT, and TTS stages.
Critical weaknesses
  • COMUS is fine-tuned on in-domain meeting data (AMI, ICSI, RAMC, and AliMeeting), while all baselines are evaluated zero-shot. Table 13 suggests this fine-tuning accounts for much of the gain. Without it, COMUS scores below Qwen3-Omni S2TT (11.1 vs. 20.8 speaker-agnostic BLEU on AliMeeting), and when AliMeeting is held out of fine-tuning, COMUS still only matches Qwen3-Omni. A cascade built on the same backbone and given the same in-domain adaptation would show how much of the gain comes from the framework itself.
  • The evaluation relies heavily on LLM-generated reference translations. Since no human translations exist for the evaluation sets, all BLEU references come from GPT-5.5, while the training translations are produced by Claude. The authors use different LLMs to avoid "style leakage", but BLEU against one LLM's output still rewards translations that resemble it. The only human evaluation (10 listeners rating 10 conversations) is just too small.
Questions for authors
  • The reproducibility statement says the derived speech data can't be released because of redistribution restrictions and voice-privacy concerns. Is it possible to at least release the translated transcripts and construction code, so that others can regenerate the dataset?
5 comments
AuthorsOpenReview ↗

Response to Reviewer AnNW - Part 1/3

We thank the reviewer for highlighting the importance of matched-data baselines and reliable evaluation references. To address these concerns, we adapted stronger Qwen3-Omni baselines using the same real-meeting training data used to adapt COMUS, expanded the listening study, and obtained bilingual human assessments of sampled reference translations.

1. In-domain adaptation and stronger baselines. We agree that our submitted comparisons did not adequately separate the benefit of real-meeting adaptation from other system differences. As the reviewer notes, Table 13 shows that adaptation is important: on AliMeeting, COMUS achieves 11.1 SAgBLEU without final real-data fine-tuning, compared with 20.8 for Qwen3-Omni S2TT. COMUS reaches 25.5 after fine-tuning on all real-meeting training data and 20.9 when fine-tuned on RAMC with AliMeeting held out. The last result shows cross-dataset transfer, but does not establish an advantage over an equally adapted baseline.

To address this concern, we built stronger baselines using Qwen3-Omni, the same backbone as COMUS, and adapted their translation and joint diarization–ASR stages using the same real-meeting training data used to adapt COMUS. The new cascades translate the whole conversation, retain one voice reference per speaker, and use CosyVoice3 for speech generation. Results are reported on real-eval as EN→ZH | ZH→EN.

SystemDiarizationTranslationSAtBLEU↑SAtASR-BLEU↑conv-MQM↑DER↓
Pure Cascade (submitted)pyannote 3.1NLLB, per turn13.3 | 6.514.7 | 6.5−17.4 | −37.728.1 | 19.2
SoulX-Transcriber (submitted)SoulX-Trans, ownNLLB, per turn16.5 | 8.316.3 | 7.9−17.5 | −29.319.7 | 10.5
Conv-Cascade (new)pyannote 3.1Qwen3-Omni, whole conv.20.7 | 16.222.1 | 14.8−10.5 | −10.728.1 | 19.2
Conv-Cascade (new)Community-1, 2 spkQwen3-Omni, whole conv.21.7 | 16.922.5 | 15.1−11.3 | −11.827.0 | 18.5
+ fine-tuned MT (new)Community-1, 2 spkQwen3-Omni FT, whole conv.27.9 | 22.628.4 | 20.3−10.5 | −9.827.0 | 18.5
+ fine-tuned SD+ASR (new)Qwen3-Omni FTQwen3-Omni FT, whole conv.37.6 | 27.535.4 | 23.0−4.3 | −3.77.1 | 4.1
Qwen3-Omni-FT S2TT+TTS (new)joint, one passjoint38.3 | 27.135.9 | 22.9−4.1 | −3.78.3 | 4.8
COMUS (submitted)joint, one passjoint34.9 | 24.331.9 | 20.4−7.6 | −6.89.3 | 4.1

What do the new comparisons show? Replacing per-turn NLLB with whole-conversation Qwen3-Omni translation raises SAtBLEU from 13.3 | 6.5 to 20.7 | 16.2 with pyannote 3.1 fixed. This comparison changes both the translation model and its context, so it does not measure the effect of context alone. With Community-1 and the known speaker count fixed at two, fine-tuning the translation stage raises SAtBLEU from 21.7 | 16.9 to 27.9 | 22.6.

Replacing Community-1 with the fine-tuned joint diarization–ASR front end further raises SAtBLEU to 37.6 | 27.5, while reducing DER from 27.0 | 18.5 to 7.1 | 4.1. The jointly fine-tuned S2TT+TTS system also performs strongly, reaching 38.3 | 27.1 SAtBLEU. These results demonstrate the value of strong speaker-aware models combined with in-domain adaptation, although changing both the front end and its adaptation does not isolate the effect of joint modeling alone.

The original pyannote 3.1 baseline is also a pretrained meeting-domain system: its segmentation model was trained using the AMI and AliMeeting training sets. This prior exposure does not, however, make its adaptation identical to that of COMUS or the new fine-tuned baselines.

COMUS versus the stronger baselines. COMUS outperforms the strongest evaluated conventional cascade, which uses Community-1 and fine-tuned translation, on SAtBLEU (34.9 | 24.3 versus 27.9 | 22.6). The stronger pipelines with joint SD+ASR or speaker-attributed S2TT achieve higher automatic translation scores than COMUS. They use joint models for speaker-aware understanding followed by separate translation and/or speech-generation stages, while COMUS generates speaker profiles and translated speech tokens in one autoregressive sequence.

Thus, the new results support a system-level advantage over the evaluated conventional cascades, while showing that a strong adapted pipeline with joint speaker-aware components can outperform COMUS on automatic translation metrics. The systems still differ in components, objectives, and training, so the comparison does not isolate architecture alone.

AuthorsOpenReview ↗

Response to Reviewer AnNW - Part 2/3

Performance on longer conversations. To understand the remaining gap, we grouped conversations into source-duration quartiles, with 50 conversations per direction in each group. The table reports SAtASR-BLEU, which evaluates generated speech through ASR. Differences are Qwen3-Omni-FT S2TT+TTS minus COMUS.

Source duration (EN→ZH | ZH→EN)   COMUS   Qwen3-Omni-FT S2TT+TTSDifference
Q1: 10–14 | 10–18 s31.5 | 23.131.6 | 23.4+0.2 | +0.3
Q2: 14–23 | 18–27 s32.4 | 20.836.8 | 22.5+4.4 | +1.7
Q3: 23–34 | 28–44 s33.8 | 18.735.6 | 21.7+1.8 | +3.0
Q4: 34–219 | 46–111 s30.0 | 19.039.6 | 24.0+9.6 | +4.9

For the shortest conversations (Q1), COMUS and the adapted cascade achieve similar SAtASR-BLEU scores: 31.5 | 23.1 versus 31.6 | 23.4 for EN→ZH | ZH→EN. We did not detect a significant difference in this group (95% confidence intervals for the difference: [−5.1, +5.1] | [−2.8, +3.3]). For the longest conversations (Q4), COMUS obtains 30.0 | 19.0, compared with 39.6 | 24.0 for the cascade. The cascade’s advantages of 9.6 | 4.9 points are the largest observed across the quartiles, and the corresponding confidence intervals exclude zero ([+3.7, +16.2] | [+2.9, +7.1]).

Longer conversations require more generated speech tokens: the average rises from 537 | 819 in Q1 to 2,011 | 3,188 in Q4, with a maximum of 6,843. The percentage of conversations with at least one translated turn lacking corresponding speech rises from 0% in Q1 to 22% | 14% in Q4. This is consistent with a length-related speech-generation limitation, although further analysis is needed to determine how often the decoding limit is responsible.

Speech tokens represent approximately 75% of target tokens overall and 80–85% for conversations longer than 80 seconds. We hypothesize that this imbalance may affect text learning, but we have not tested that explanation directly. We will report long-conversation performance and incomplete speech generation as limitations, and investigate more compact speech representations in future work.

On the automatic speech metrics, COMUS obtains higher predicted naturalness than the fully adapted cascade (UTMOS 3.38 | 3.79 versus 2.85 | 3.48) and higher ZH→EN speaker similarity (0.89 versus 0.86).

AuthorsOpenReview ↗

Response to Reviewer AnNW - Part 3/3

2. Human validation of the GPT-5.5 references. Five bilingual annotators with translation experience independently assessed 100 short excerpts from 69 conversations for semantic adequacy and naturalness (1–5), as well as major semantic errors. They did not use external LLMs or translation software.

For our SODA, AMI, and AliMeeting data, the sampled references received average scores of 4.73 for adequacy and 4.59 for naturalness, with major errors marked in 1% of annotator judgments.

On XDailyDialog, the GPT-5.5 translations received average adequacy and naturalness scores of 4.90 and 4.89, compared with 4.72 and 4.72 for the published references. Major errors were marked in 0% versus 5% of judgments. Because the two reference types come from different source dialogues, these results are descriptive and do not establish that one is better than the other.

Overall, the assessment supports the quality of the sampled references, although possible single-reference stylistic bias and errors in longer conversations remain. We also report reference-free, LLM-based conv-MQM and an expanded human listening study of system outputs separately.

3. Expanded human listening study. To complement the automatic translation metrics, we expanded the listening study from 10 to 20 conversations (10 per direction) and from 10 to 18 listeners. The selected conversations are 10–35 seconds long, a range that includes approximately 70% of conversations in the objective evaluation set, whose median duration is 25 seconds. Each listener rated all 20 conversations for all four systems, yielding 360 listener–conversation ratings per system and dimension. We also added a prosody-similarity dimension.

SystemSpk.-Turn ↑Spk. Sim. ↑Pros. Sim. ↑Conv. Nat. ↑Trans. Qual. ↑Overall ↑
Conv-Cascade-C10.44*2.89*2.67*2.78*2.84*2.63*
SD+UniSS0.45*3.04*2.70*2.59*2.56*2.41*
Qwen3-Omni-FT S2TT+TTS0.913.423.373.553.733.55
COMUS0.923.493.453.503.723.52

* COMUS is significantly better under the reported analysis: Holm-corrected p < 0.05 and a 95% confidence interval excluding zero.

Translation quality. Listeners rated COMUS higher than Conv-Cascade-C1 and SD+UniSS (3.72 versus 2.84 and 2.56). Against the adapted Qwen3-Omni-FT S2TT+TTS system, the ratings were 3.72 versus 3.73; we did not detect a statistically significant difference.

Other dimensions. Under the reported analysis, COMUS outperforms Conv-Cascade-C1 and SD+UniSS on all six dimensions. Averaged over the five 1–5 scales, the differences are +0.77 [0.50, 1.04] and +0.88 [0.61, 1.13], respectively. Against the adapted S2TT+TTS system, we did not detect a significant difference on any dimension; the five-scale mean difference is +0.01 [−0.19, +0.20] (p = 0.524).

These human ratings complement the automatic metrics, but the study remains limited to 20 conversations of 10–35 seconds. It does not cover the longest conversations where the largest objective performance gap appears. We address the quality of the LLM-generated references separately through human validation.

4. Data release. We have confirmed with our legal team that we will release the code used to construct the SODA-SoulX text data, along with the translated text for the synthetic evaluation set and the AMI and ICSI meeting evaluation sets. We cannot release the remaining data due to our institution's legal restrictions.

5. Updated demo: https://demopage-1.github.io/s2st-rebuttal-samples/

Summary. The new experiments show that COMUS outperforms the strongest evaluated conventional cascade on SAtBLEU, while stronger adapted pipelines using joint SD+ASR or S2TT achieve higher automatic translation scores. Human assessment supports the quality of the sampled GPT-5.5 references, and the expanded listening study provides a complementary evaluation of the generated speech. We will revise the paper to clarify the role of in-domain adaptation and the limitations on longer conversations.

Reviewer_SFRoOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

COMUS is a multi-speaker speech-to-speech translation system built on the Qwen3-Omni Thinker. It processes a full recording without external speaker labels, transcripts, or voice references. It generates text, speaker IDs, and BiCodec tokens, reusing one global voice profile per speaker. Its best CoM mode produces timestamped source transcripts and speaker-attributed translations before speech. The authors build SODA-SoulX with about 1.17 million synthetic bilingual conversation pairs and 310 hours of real source conversations paired with synthesized target speech. Staged multitask training and adaptation improve translation and source diarization on mostly two-speaker English–Chinese tests, with additional overlap, noise, speaker-count, cross-dataset, and listening studies. The system and data work are useful, but the architecture's advantage over a similarly capable cascade remains unclear.

Critical strengths
  1. A sensible design for keeping speaker identity across turns. Reusing one indexed voice profile per speaker gives the model a clear way to link translated turns. Combining speaker discovery, full-conversation translation, and voice-conditioned generation is useful engineering work. CoM's intermediate text does not undermine this integration.
  2. Substantial data and experiments, with useful system gains. On real-eval, source DER is 9.3%/4.1%, versus pyannote 3.1's 28.1%/19.2% and SoulX-Transcriber's 19.7%/10.5%. SAtASR-BLEU is 31.9/20.4, versus SD+UniSS's 12.1/9.1 (EN→ZH/ZH→EN). Tests include real recordings, overlap, noise and competing speakers, separate meetings for adaptation/evaluation, and positive listening results. The evidence goes beyond clean audio or one baseline.
Critical weaknesses
  1. The value of end-to-end modeling is unclear. A cascade can also use full-conversation context and persistent speaker profiles. What limitation does COMUS overcome: speed, expressive fidelity, speaker assignment, or quality for the cost? Combining stages alone does not establish a benefit that justifies the joint model and training effort.

  2. The comparison does not isolate the architecture or establish an advantage over a similarly capable cascade. The modular baselines use NLLB-3.3B, segment-level translation, and segment-level voice references; COMUS uses Qwen3-Omni, full context, and extensive task training. Gains could come from components, context, or data. The Qwen3-Omni text baseline lacks speech output, and CoM/Direct use different training setups. There is no matched runtime comparison; the 75% token reduction is relative to the original MTP path. A strong cascade with comparable translation, persistent voice references, and current diarization/TTS is needed to clarify the architectural benefit.

  3. The evaluation and demo do not support the broader claims strongly enough. The demo shows only UniSS + diarization, whose samples sound noticeably weaker to me than a well-configured current dubbing pipeline should. KrillinAI/OpenCreator is a useful modular workflow reference, not a measured winner on this benchmark. The paper tests more systems numerically, but still lacks a similarly capable cascade and broader comparative audio. Source DER does not verify audible output identity, and UTMOS/SIM do not verify emotion/prosody preservation. Tests are mostly two-speaker; four-speaker DER reaches 33.8%/21.1% under different adaptation settings across systems. The positive listening study covers only ten conversations, and 96% refers to conversation-level judgments. An updated diarization baseline such as Community-1 would also strengthen the pyannote 3.1 comparison.

Questions for authors
  1. What measurable benefit does end-to-end modeling bring over a cascade with comparable translation, persistent speaker references, and stated adaptation settings? How much comes from architecture rather than data/components? A matched shared-profile versus per-turn-profile comparison would help.

  2. Why is the demo baseline representative of a strong current cascade? Please provide stronger modular results and audio, with diarization, translation context, voice-reference, and synthesis settings. KrillinAI is a practical reference, not a substitute for a matched test.

  3. Can you test newer diarization on the same subset and scoring setup, with equal speaker-count information? Does output attribution check audible identity or use generated IDs? Reference-transcript/fixed-target-text controls, SIM and conv-MQM settings, human error checks, and paired conversation-level uncertainty would help locate and verify the gains.

  4. Which results directly support efficiency or emotion/prosody preservation? What do the three/four-speaker tests establish despite different adaptation settings? Please provide evidence or narrow the claims to offline, mostly two-speaker translation.

7 comments
AuthorsOpenReview ↗

Response to Reviewer SFRo - Part 1/3

We thank the reviewer for a detailed and constructive review.

We agree that the submitted comparisons did not fully isolate the contribution of COMUS's architecture. In particular, the original cascades used turn-by-turn NLLB-3.3B translation and single-turn voice references. We have therefore built stronger conversation-level baselines, evaluated in-domain adaptation of the translation and diarization–ASR stages, and expanded the listening study. All results are reported on real-eval as EN→ZH | ZH→EN.

1. Stronger, adapted baselines. The new cascades use Qwen3-Omni, the same backbone as COMUS, for whole-conversation translation, retain one voice reference per speaker, and use CosyVoice3 for speech generation. We first fine-tune the translation stage, then evaluate a fine-tuned joint diarization–ASR front end, using the same real-meeting training data used to adapt COMUS. These systems provide a stronger comparison and help assess the contributions of model components and in-domain adaptation.

SystemDiarizationTranslationSAtBLEU↑SAtASR-BLEU↑conv-MQM↑DER↓
Pure Cascade (submitted)pyannote 3.1NLLB, per turn13.3 | 6.514.7 | 6.5−17.4 | −37.728.1 | 19.2
SoulX-Transcriber (submitted)SoulX-Trans, ownNLLB, per turn16.5 | 8.316.3 | 7.9−17.5 | −29.319.7 | 10.5
Conv-Cascade (new)pyannote 3.1Qwen3-Omni, whole conv.20.7 | 16.222.1 | 14.8−10.5 | −10.728.1 | 19.2
Conv-Cascade (new)Community-1, 2 spkQwen3-Omni, whole conv.21.7 | 16.922.5 | 15.1−11.3 | −11.827.0 | 18.5
+ fine-tuned MT (new)Community-1, 2 spkQwen3-Omni FT, whole conv.27.9 | 22.628.4 | 20.3−10.5 | −9.827.0 | 18.5
+ fine-tuned SD+ASR (new)Qwen3-Omni FTQwen3-Omni FT, whole conv.37.6 | 27.535.4 | 23.0−4.3 | −3.77.1 | 4.1
Qwen3-Omni$_{FT}$ S2TT+TTS (new)joint, one passjoint38.3 | 27.135.9 | 22.9−4.1 | −3.78.3 | 4.8
COMUS (submitted)joint, one passjoint34.9 | 24.331.9 | 20.4−7.6 | −6.89.3 | 4.1

How much gain comes from architecture, data, and components? Our expanded comparisons show that stronger conversational translation, in-domain adaptation, and joint speaker-aware modeling all lead to substantial improvements in the evaluated systems.

Conversational translation and adaptation. With pyannote 3.1 fixed, replacing per-turn NLLB with whole-conversation Qwen3-Omni translation raises SAtBLEU from 13.3 | 6.5 to 20.7 | 16.2. Replacing pyannote 3.1 with Community-1, while providing the known speaker count of two, further improves SAtBLEU to 21.7 | 16.9 and reduces DER to 27.0 | 18.5. Fine-tuning Qwen3-Omni for speaker-aware translation on the same in-domain meeting data then raises SAtBLEU to 27.9 | 22.6.

Joint speaker-aware modeling. The jointly trained SoulX-Transcriber obtains a lower DER than Community-1 (19.7 | 10.5 versus 27.0 | 18.5). Fine-tuning Qwen3-Omni jointly for ASR and diarization reduces DER further to 7.1 | 4.1. When this joint front end replaces Community-1 in the adapted cascade, SAtBLEU rises from 27.9 | 22.6 to 37.6 | 27.5. The fine-tuned joint S2TT system achieves similarly strong results: 38.3 | 27.1 SAtBLEU and 8.3 | 4.8 DER. Together, these results demonstrate the practical value of combining joint speaker-aware modeling with in-domain adaptation.

The pyannote 3.1 baseline is also a strong pretrained system: its segmentation model was trained using the AMI and AliMeeting training sets. Thus, the joint models are being compared against a diarization baseline with prior exposure to meeting data.

COMUS versus the stronger baselines. COMUS outperforms the strongest evaluated conventional cascade on SAtBLEU (34.9 | 24.3 versus 27.9 | 22.6), while generating speaker-attributed translated speech within a unified autoregressive framework. The stronger systems above use joint SD+ASR or speaker-attributed S2TT followed by separate translation and/or speech-generation stages; they achieve higher automatic translation scores than COMUS.

These comparisons establish substantial system-level gains, although they do not fully isolate the individual effects of model architecture, context, and training data. To understand the remaining gap, we analyzed performance by conversation duration, separately examining text translation and generated speech.

AuthorsOpenReview ↗

Response to Reviewer SFRo - Part 2/3

Performance on longer conversations. We grouped conversations into source-duration quartiles, with 50 conversations per direction in each group. The table reports SAtASR-BLEU, which evaluates the generated speech through ASR. Differences are Qwen3-Omni-FT S2TT+TTS minus COMUS.

Source duration (EN→ZH | ZH→EN)   COMUS   Qwen3-Omni-FT S2TT+TTSDifference
Q1: 10–14 | 10–18 s31.5 | 23.131.6 | 23.4+0.2 | +0.3
Q2: 14–23 | 18–27 s32.4 | 20.836.8 | 22.5+4.4 | +1.7
Q3: 23–34 | 28–44 s33.8 | 18.735.6 | 21.7+1.8 | +3.0
Q4: 34–219 | 46–111 s30.0 | 19.039.6 | 24.0+9.6 | +4.9

For the shortest conversations (Q1), COMUS and the adapted cascade achieve similar SAtASR-BLEU scores: 31.5 | 23.1 versus 31.6 | 23.4 for EN→ZH | ZH→EN. We did not detect a significant difference in this group (95% confidence intervals for the difference: [−5.1, +5.1] | [−2.8, +3.3]). For the longest conversations (Q4), COMUS obtains 30.0 | 19.0, compared with 39.6 | 24.0 for the cascade. The cascade’s advantages of 9.6 | 4.9 points are the largest observed across the quartiles, and the corresponding confidence intervals exclude zero ([+3.7, +16.2] | [+2.9, +7.1]).

Longer conversations also require substantially more speech tokens. The average rises from 537 | 819 in Q1 to 2,011 | 3,188 in Q4, with a maximum of 6,843. The percentage of conversations with at least one translated turn lacking corresponding speech rises from 0% in Q1 to 22% | 14% in Q4. This is consistent with a length-related speech-generation limitation, although further analysis is needed to determine how often the decoding limit is responsible.

Speech tokens represent approximately 75% of target tokens overall and 80–85% for conversations longer than 80 seconds. We hypothesize that this imbalance may affect text learning, but we have not tested that explanation directly. We will report long-conversation performance and incomplete speech generation as limitations, and investigate more compact speech representations in future work.

On the automatic speech metrics, COMUS obtains higher predicted naturalness than the cascade with all components fine-tuned (UTMOS 3.38 | 3.79 vs. 2.85 | 3.48) and higher ZH→EN speaker similarity (0.89 vs. 0.86).

AuthorsOpenReview ↗

Response to Reviewer SFRo - Part 3/3

2. Expanded listening test. We evaluated 20 conversations (10 per direction) with 18 listeners. The conversations were 10–35 seconds long, a range that includes approximately 70% of conversations in the objective evaluation set, whose median duration is 25 seconds. Each listener rated all 20 conversations for all four systems, yielding 360 listener–conversation ratings per system and dimension. We also added a prosody-similarity dimension. The test does not include the longest conversations, where the largest objective performance gap appears.

SystemSpk.-Turn ↑Spk. Sim. ↑Pros. Sim. ↑Conv. Nat. ↑Trans. Qual. ↑Overall ↑
Conv-Cascade-C10.44*2.89*2.67*2.78*2.84*2.63*
SD+UniSS0.45*3.04*2.70*2.59*2.56*2.41*
Qwen3-Omni-FT S2TT+TTS0.913.423.373.553.733.55
COMUS0.923.493.453.503.723.52

* COMUS is significantly better under the reported analysis: Holm-corrected p < 0.05 and a 95% confidence interval excluding zero.

COMUS outperforms Conv-Cascade-C1 and SD+UniSS on all six dimensions. Averaged over the five 1–5 scales, the reported differences are +0.77 [0.50, 1.04] and +0.88 [0.61, 1.13], respectively. For the new prosody-similarity dimension, COMUS scores 3.45, compared with 2.67 and 2.70.

Against Qwen3-Omni-FT S2TT+TTS, we did not detect a statistically significant difference on any dimension. The five-scale mean difference is +0.01 [−0.19, +0.20] (p = 0.524). These results should be interpreted within the 10–35-second duration range of the listening test.

3. What we claim after these experiments.

  • COMUS. COMUS predicts speaker profiles and translated speech tokens for a conversation in one autoregressive pass, including when the input contains overlapping speakers. It outperforms the strongest evaluated conventional cascade on SAtBLEU (34.9 | 24.3 versus 27.9 | 22.6), even though the cascade is given the known number of speakers. COMUS also obtains higher UTMOS than the reported adapted cascade. In the listening test, we found no significant difference in translation quality versus adapted S2TT+TTS.

  • SODA-SoulX. We introduce 26K hours of synthetic bilingual speech and 310 hours of real source meetings paired with synthesized target speech. We will release the text-construction code and selected evaluation translations.

  • Task and evaluation. We study multi-speaker, voice-preserving conversational S2ST, with evaluation covering translation, speaker attribution, overlapping input, and human validation of sampled references.

  • Comparison with stronger joint systems. Pipelines using joint SD+ASR or speaker-attributed S2TT achieve higher text scores than COMUS, while relying on separate translation and/or speech generation stages. These results demonstrate the strength of joint speaker-aware modeling and provide demanding baselines for end-to-end conversational S2ST.

  • Long-conversation limitation. COMUS has a larger performance gap and more incomplete speech outputs on longer conversations. A speech-token budget limitation is a plausible explanation that requires further testing.

We will clarify in the paper that our evaluation focuses on offline, primarily two-speaker translation, describe the three- and four-speaker results as exploratory, and remove unsupported runtime-efficiency claims.

4.New Demo Page: https://demopage-1.github.io/s2st-rebuttal-samples/

Reviewer_CHRxOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes COMUS, an end-to-end multi-speaker speech-to-speech translation system built on Qwen3-Omni. It jointly predicts speaker identities, translations, and speech tokens while processing entire conversations. The authors also introduce SODA-SoulX, a 26K-hour synthetic English–Chinese conversational dataset. Experiments on real meeting recordings show improvements in translation quality, speaker attribution, and naturalness over cascaded baselines

Critical strengths
  1. The paper addresses an important and challenging problem: translating multi-speaker conversations while preserving speaker identity and conversational context.
  2. Strong experimental results across translation, diarization, noise robustness, and subjective evaluation. The improvements over existing cascaded systems are significant.
Critical weaknesses
  • The comparison with cascaded baselines is not fully controlled. COMUS has access to the entire conversation, while the baselines translate segments independently. This makes it difficult to attribute the gains to the proposed end-to-end architecture rather than simply having access to conversational context. A stronger baseline that performs translation jointly over the full conversation would help isolate this contribution.
  • The subjective evaluation is limited. Only 10 conversations (five per direction) are evaluated by 10 listeners, without statistical significance testing.
Questions for authors
  • How much of the reported improvement would remain if the cascaded baseline also used a strong LLM to translate the entire conversation jointly, rather than translating individual segments?
  • Can the authors report statistical significance tests or confidence intervals for the subjective results in Table 4?
  • Since the evaluation references are generated using GPT-5.5, has their translation quality been validated by human annotators?
2 comments
AuthorsOpenReview ↗

Response to Reviewer CHRx - Part 1/2

We thank the reviewer for the helpful comments and the three focused questions. In response, we evaluated stronger whole-conversation baselines, expanded the listening study with statistical analysis, and obtained human assessments of the GPT-5.5 evaluation references.

1. Comparison with whole-conversation baselines. We built new cascades that use Qwen3-Omni, the same backbone as COMUS, to translate the whole conversation. They retain one voice reference per speaker and use CosyVoice3 for speech generation. We also evaluate in-domain adaptation of the translation stage and a joint diarization–ASR front end using the real-meeting training data used to adapt COMUS. Results are on real-eval and reported as EN→ZH | ZH→EN.

SystemDiarizationTranslationSAtBLEU ↑SAtASR-BLEU ↑conv-MQM ↑DER ↓
Pure Cascade (submitted)pyannote 3.1NLLB, per turn13.3 | 6.514.7 | 6.5−17.4 | −37.728.1 | 19.2
SoulX-Transcriber (submitted)SoulX-Trans, ownNLLB, per turn16.5 | 8.316.3 | 7.9−17.5 | −29.319.7 | 10.5
Conv-Cascade (new)pyannote 3.1Qwen3-Omni, whole conv.20.7 | 16.222.1 | 14.8−10.5 | −10.728.1 | 19.2
Conv-Cascade (new)Community-1, 2 spkQwen3-Omni, whole conv.21.7 | 16.922.5 | 15.1−11.3 | −11.827.0 | 18.5
+ fine-tuned MT (new)Community-1, 2 spkQwen3-Omni FT, whole conv.27.9 | 22.628.4 | 20.3−10.5 | −9.827.0 | 18.5
+ fine-tuned SD+ASR (new)Qwen3-Omni FTQwen3-Omni FT, whole conv.37.6 | 27.535.4 | 23.0−4.3 | −3.77.1 | 4.1
Qwen3-Omni-FT S2TT+TTS (new)joint S2TTjoint S2TT38.3 | 27.135.9 | 22.9−4.1 | −3.78.3 | 4.8
COMUS (submitted)jointjoint34.9 | 24.331.9 | 20.4−7.6 | −6.89.3 | 4.1

Using whole-conversation Qwen3-Omni translation increases the pyannote-based cascade's SAtBLEU from 13.3 | 6.5 to 20.7 | 16.2. COMUS remains higher at 34.9 | 24.3, a difference of 14.2 | 8.1 points. This comparison changes both the translation model and its context, so it does not isolate the effect of context alone.

Against the stronger conventional cascade using Community-1 and fine-tuned translation, COMUS achieves 34.9 | 24.3 versus 27.9 | 22.6 SAtBLEU. Community-1 is given the known speaker count of two.

The strong performance of the adapted joint SD+ASR and speaker-attributed S2TT systems highlights the potential of end-to-end speaker-aware modeling with in-domain adaptation. COMUS extends this joint modeling to speaker-attributed speech generation by predicting speaker profiles and translated speech tokens in a single autoregressive sequence.

2. Expanded listening study and statistical significance. We expanded the study from 10 to 20 conversations (10 per direction) and from 10 to 18 listeners. The conversations are 10–35 seconds long, a range covering approximately 70% of the objective evaluation conversations; the evaluation set's median duration is 25 seconds. Each listener rated every conversation for all four systems, giving 360 listener–conversation ratings per system and dimension. We also added a prosody-similarity dimension.

SystemSpk.-Turn ↑Spk. Sim. ↑Pros. Sim. ↑Conv. Nat. ↑Trans. Qual. ↑Overall ↑
Conv-Cascade-C10.44*2.89*2.67*2.78*2.84*2.63*
SD+UniSS0.45*3.04*2.70*2.59*2.56*2.41*
Qwen3-Omni-FT S2TT+TTS0.913.423.373.553.733.55
COMUS0.923.493.453.503.723.52

* COMUS is significantly better under the reported analysis: Holm-corrected p < 0.05 and a 95% confidence interval excluding zero.

On the mean of the five 1–5 scales, COMUS improves by +0.77 [0.50, 1.04] over Conv-Cascade-C1 and +0.88 [0.61, 1.13] over SD+UniSS. Against the adapted S2TT+TTS system, we did not detect a significant difference on any dimension; the five-scale mean difference is +0.01 [−0.19, +0.20] (p = 0.524). For translation quality, the ratings are 3.72 versus 3.73. The study does not include the longest conversations, where the largest objective performance gap appears.

AuthorsOpenReview ↗

Response to Reviewer CHRx - Part 2/2

3. Human validation of the GPT-5.5 references. Five bilingual annotators with translation experience independently assessed 100 short excerpts from 69 conversations for semantic adequacy, naturalness (1–5), and major semantic errors, without using external LLMs or translation software.

For SODA, AMI, and AliMeeting, the sampled references received average scores of 4.73 for adequacy and 4.59 for naturalness, with major errors marked in 1% of annotator judgments.

On XDailyDialog, the GPT-5.5 translations received adequacy and naturalness scores of 4.90 and 4.89, compared with 4.72 and 4.72 for the published references. Major errors were marked in 0% versus 5% of judgments. Because the two reference types come from different source dialogues, this comparison is descriptive and does not establish that one is better.

These checks support the quality of the sampled references, although single-reference stylistic bias and errors in longer conversations remain possible. We also report reference-free, LLM-based conv-MQM and the human listening results above as complementary evaluations.

4. Updated demo page https://demopage-1.github.io/s2st-rebuttal-samples/

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.