acceptodds
Under review as a conference paper at ICLR 2027

Chorus: Extending Single-Speaker TTS to Multi-Speaker Conversation Generation

Abstract

Multi-speaker conversational speech generation synthesizes a conversation among several speakers from its transcript. Most existing systems generate the conversation as a single-channel mixture, which entangles the speakers. This makes the generated speech difficult to use for downstream tasks that require a separate channel for each speaker. It also complicates per-speaker evaluation, which requires diarization to attribute each speech segment to the corresponding speaker. Systems that generate one channel per speaker address these issues but are limited to two speakers and one form of timing control. We introduce Chorus, which assigns a weight-shared single-speaker text-to-speech (TTS) model to each speaker and connects the models through a communication block whose computation does not depend on the number of speakers. Chorus therefore supports an arbitrary number of speakers, chosen at inference time. Chorus retains the infilling objective of its base model, F5-TTS. Each model receives the audio of its channel, a mask over the frames to generate, and a text input in which special tokens mark the turns of that speaker. Within this formulation, four tasks are obtained by choosing which frames are masked and how the transcript is structured: generation of all channels from a transcript, generation of one channel conditioned on the recorded audio of the others, chunk generation of long conversations conditioned on the previous chunk, and generation with turn timing specified either by explicit start and end times or by the turn order alone. On the two-speaker ZipVoice-Dialog test set, Chorus is competitive with the baselines on intelligibility and speaker similarity, although it is fine-tuned on less conversational speech. From the turn order alone, it produces backchannels and interruptions whose timing a turn-taking judge scores above the ground truth. The same model generates two- to four-speaker conversations on AMI with a rise of one point in per-channel word error rate, making it, to our knowledge, the first system that generates one channel per speaker for more than two speakers.

Primary area:
generative models
Keywords:
multi-speaker conversational speech generation, speech synthesis, turn-taking, generative model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_nok2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Chorus extends a pretrained single-speaker TTS model to multi-speaker conversational speech synthesis. It uses weight-shared F5-TTS branches, one per speaker, with Transform–Average–Concatenate (TAC) blocks exchanging information across branches. The model generates separate, synchronized speaker channels and supports turn-order conditioning, timestamp conditioning, and response generation conditioned on other speakers’ complete audio. Longer conversations are generated in chunks, with the final five seconds of the preceding chunk provided as context. Experiments evaluate speech quality, intelligibility, speaker similarity, timing, and conversational event statistics, including generation with two to four speakers.

Critical strengths

● A flexible architecture for separate-channel conversational synthesis. Sharing a pretrained TTS model across speakers and connecting its branches through communication blocks provides a practical extension to multiple speakers without speaker-count-specific parameterization. Producing separate channels also makes speaker-specific timing and overlapping speech directly representable. ● Several useful synthesis settings within one framework. The same model supports turn-order and timestamp conditioning, generation conditioned on recorded speaker channels, and chunked long-form synthesis. Experiments demonstrate these capabilities and evaluate scaling from two to four speakers, providing evidence of practical versatility beyond conventional single-speaker TTS.

Critical weaknesses

● Limited evidence of long-form conversational coherence. Generation uses chunks of at most 45 seconds with only five seconds of preceding audio as context. Table 3 demonstrates improved intelligibility, but does not establish long-range prosodic or emotional consistency or smooth chunk boundaries. ● Mixed evidence for TAC’s coordination benefits. Chorus underproduces overlap compared with real conversations (1.33 vs. 3.29 s/min). In Table 10, adding cross-channel communication further suppresses overlap without improving per-channel intelligibility or quality. Although response-generation results are more encouraging, stronger evidence is needed to show that TAC improves natural coordination rather than merely reducing simultaneous speech.

Questions for authors
  1. Long-form evaluation: What are the duration distribution and maximum duration of the conversations evaluated in Table 3? Do you have evidence specifically assessing chunk-boundary continuity and longer-range prosodic or emotional consistency, beyond aggregate intelligibility and speaker-similarity metrics?
  2. Interpretation of the TAC ablation: In Table 10, cross-channel averaging reduces overlap substantially below the ground-truth level without improving per-channel intelligibility or quality. What evidence shows that it produces more appropriate interruptions, backchannels, or turn transitions in all-speaker generation? Human judgments or event-specific analyses could materially change my assessment.
  3. Effect of training stage: The communication ablations use only first-stage training. Does the same comparison after the complete training procedure support a stronger benefit from TAC, particularly for conversations with more than two speakers?
3 comments
AuthorsOpenReview ↗

Response to Reviewer nok2 (Part 1/3)

We thank the reviewer for the constructive review. We address every point below. In short, we add F0, Loudness and Timbre on the long-form set, which show no acoustic discontinuity at chunk boundaries (W1, Q1). We add event-level overlap and gap statistics on ZipVoice-Dialog and AMI, in which gaps and overlaps longer than 3 s are more frequent without the communication block than with it (W2, Q2). We compare the first-stage models with and without the communication block on 2, 3 and 4 speakers, and the first-stage and the two-stage models (Q3). We also report a human evaluation of Chorus against ZipVoice-Dialog-Stereo and the recordings (Q2). Table, figure and section numbers refer to the submitted version. Appendix letters for new material refer to the revised version.

W1 & Q1: Long-form evaluation

Durations. The average duration of the long-form generation samples is 83 seconds, with a maximum of 167 seconds. We will add these statistics to Section 5.1.

MinMedianMeanMax
55 s77 s83 s167 s

Chunk boundaries. For every pair of consecutive speech units of the same speaker, we measure the difference in loudness, pitch and timbre. We separate the pairs into two types: (1) both units inside the same chunk, and (2) the last unit of one chunk and the first unit of the next chunk. If the chunked inference produced a discontinuity, the pairs of type (2) should show larger jumps. The table shows that the jumps across a chunk boundary are as small as those inside a chunk for loudness, F0 and timbre. This suggests that the chunked inference produces continuous speech across chunk boundaries. We will report this result in Section 5.1 and Appendix G.

Pair TypeΔLoudness (log-RMS)ΔF0 (semitones)Timbre Distance
Pair in the same chunk0.1051.550.080
Pair at boundary0.0861.390.072

Consistency over the dialogue. To test for drift over a long dialogue, we additionally report the scores of each chunk by its position in the dialogue. From the first to the third chunk, UTMOS-Ch decreases from 3.96 to 3.89 and SIM-Ch from 0.93 to 0.91, and WER-Ch shows no trend (3.4, 4.6 and 2.5). The fourth chunk exists for 10 dialogues only, so its scores are less reliable. We did not evaluate long-range prosodic or emotional consistency. Each chunk is conditioned only on the last 5 seconds of the previous chunk, and we state this as a limitation in Appendix J. We will add this result to Section 5.1 and Appendix G.

Chunk Position1st2nd3rd4th
WER-Ch3.44.62.55.0
UTMOS-Ch3.963.943.893.79
SIM-Ch0.930.920.910.89

W2 & Q2: Clarification on Communication Block Ablation

While the communication block shows a clear benefit in response generation, the results in all-speaker generation are mixed. To examine this setting more closely, we compare the first-stage models with and without TAC on the ZipVoice-Dialog test set and on AMI with 2, 3, and 4 speakers. All results below are from all-speaker generation in turn-order mode, where the text gives only the order of the turns and not their timing. Table 10 reports only the overlap duration per minute. This hides a second failure case besides prolonged overlaps: prolonged gaps between speakers (dead air). Therefore, we analyze both overlaps and gaps. For each, we report the number of events per minute (# events / length of the conversation), the mean duration per event (total duration of the events / # events), and the frequency of long events (# events longer than 3 s / length of the conversation). Without TAC, gaps longer than 3 s occur more often than in the recordings on both datasets and at every speaker count, which indicates unnatural dead air. With TAC, such gaps are less frequent than without TAC in every condition. Overlaps longer than 3 s are also less frequent with TAC than without TAC in every condition. The effect of the block is not limited to a reduction of simultaneous speech. We will add these statistics for both datasets to Appendix E.

AuthorsOpenReview ↗

Response to Reviewer nok2 (Part 2/3)

W2 & Q2: Clarification on Communication Block Ablation, continued

ZipVoice-Dialog, first-stage models all-speaker generation setting

Overlaps: Although the recordings and the model without TAC have similar overlap durations per minute, their overlap patterns differ when examined at the event level. Specifically, without TAC, each overlap event lasts longer on average than in the recordings (0.76 s against 0.60 s). In the worst case, an overlap lasts 7.6 seconds, substantially longer than the maximum observed in the recordings (2.6 seconds), and overlaps longer than 3 s occur more often than in the recordings. This suggests that without TAC the timing of the two channels is not coordinated, and prolonged overlaps occur. With TAC, the overlap is lower than in the recordings in both the number of events (2.6 against 5.5 per minute) and the duration per event (0.43 s against 0.60 s). We acknowledge this as a limitation of the current model.

Table A: Overlap events, ZipVoice-Dialog test set.

Recordingswith TACwithout TAC
# Overlap / Minute5.52.64.5
Overlap (s) / Minute3.31.23.5
Overlap (s) / Event0.600.430.76
# Overlap > 3 s / Minute0.000.000.12

Dead air: A similar pattern emerges for gaps too. If we consider only the gap duration per minute, all systems exhibit similar values. However, event-level statistics show that without TAC, the model produces fewer gap events, but each gap lasts longer on average. This results in prolonged periods of silence between speakers. In the worst case, the longest gap before another speaker begins speaking is 8.6 seconds, compared to 4.2 seconds in the recordings, and gaps longer than 3 s occur more often than in the recordings. In contrast, with TAC the gap duration per minute matches the recordings and the mean gap is slightly shorter. Gaps longer than 3 s remain more frequent than in the recordings, but they occur less often than without TAC. These results further suggest that without TAC the speaker turns are not coordinated when only the text is given.

Table B: Gap events, ZipVoice-Dialog test set.

Recordingswith TACwithout TAC
# Gap / Minute4.75.53.3
Gap (s) / Minute3.93.94.1
Gap (s) / Event0.840.701.27
# Gap > 3 s / Minute0.030.100.33

AMI, first-stage models with and without the communication block, 2 / 3 / 4 speakers

Overlaps: Unlike on the ZipVoice-Dialog test set, the overlap statistics on AMI do not separate as clearly. In the two-speaker setting, the model without TAC is close to the recordings in the number of overlaps, the duration per minute, and the duration per event. With TAC, the number of overlaps rises with the number of speakers as in the recordings and is close to them with three and four speakers, while without TAC it grows more slowly. However, each overlap with TAC is about 40% shorter than in the recordings, and overlaps longer than 3 s are less frequent. We acknowledge this as a limitation of the current model.

Table C: Overlap events, AMI, 2 / 3 / 4 speakers.

(2 / 3 / 4 speakers)Recordingswith TACwithout TAC
# Overlap / Minute4.9 / 8.6 / 12.13.6 / 7.9 / 11.24.3 / 6.9 / 8.9
Overlap (s) / Minute5.1 / 10.1 / 15.82.1 / 6.0 / 8.75.2 / 7.5 / 9.1
Overlap (s) / Event1.04 / 1.18 / 1.310.60 / 0.75 / 0.781.21 / 1.08 / 1.02
# Overlap > 3 s / Minute0.20 / 0.64 / 1.180.00 / 0.15 / 0.260.38 / 0.47 / 0.50

Dead air: Without TAC, gaps over 3 s occur more often than in the recordings at 2, 3, and 4 speakers, and they grow with the number of speakers. These results again suggest that without TAC the timing of the turns is not coordinated. With TAC, the model produces more gaps than the recordings. However, each of these gaps is shorter than in the recordings on average, and gaps over 3 s are less frequent than in the recordings with two and three speakers and as frequent with four speakers. The gap duration per minute with TAC is above that of the recordings at three and four speakers (5.5 and 8.1 s against 4.7 and 4.6 s), and below that of the model without TAC at every speaker count.

Table D: Gap events, AMI, 2 / 3 / 4 speakers.

(2 / 3 / 4 speakers)Recordingswith TACwithout TAC
# Gap / Minute2.4 / 4.1 / 4.84.4 / 6.8 / 9.32.4 / 5.1 / 6.9
Gap (s) / Minute3.2 / 4.7 / 4.63.0 / 5.5 / 8.14.3 / 7.6 / 10.2
Gap (s) / Event1.31 / 1.13 / 0.960.69 / 0.81 / 0.871.83 / 1.49 / 1.47
# Gap > 3 s / Minute0.23 / 0.29 / 0.260.08 / 0.10 / 0.270.45 / 0.63 / 0.82
AuthorsOpenReview ↗

Response to Reviewer nok2 (Part 3/3)

Q2: Human Evaluation

We conducted a listening test on Prolific comparing the recordings, ZipVoice-Dialog-Stereo and Chorus on 20 conversations of the ZipVoice-Dialog test set. In the naturalness test, raters heard a two-speaker conversation and rated how natural and expressive it is on a 1-5 scale. In the speaker similarity test, raters heard a reference clip and one speaker's channel of the conversation, and rated on a 1-5 scale how likely the two come from the same speaker. Each clip was rated by 10 raters. The table reports the mean ± the half-width of the 95% bootstrap confidence interval.

Table E: Listening test, ZipVoice-Dialog test set.

SystemNaturalnessSpeaker similarity
Recordings3.54 ± 0.203.09 ± 0.32
ZipVoice-Dialog-Stereo3.03 ± 0.173.10 ± 0.37
Chorus3.38 ± 0.103.74 ± 0.23

Chorus is rated more natural than ZipVoice-Dialog-Stereo (p = 0.004, paired permutation test). The recordings remain the most natural, and their difference to Chorus is not significant (p = 0.18). Chorus also obtains a higher speaker similarity than ZipVoice-Dialog-Stereo (p = 0.006, paired permutation test). Chorus also has a higher speaker similarity than the recordings (p = 0.017). E2 TTS (Eskimez et al., 2024) reports the same pattern on audiobooks. There, the ground truth receives a lower similarity score than every tested system, which the authors attribute to speakers who vary their voice for different characters. Our first hypothesis is that spontaneous conversation has the same effect. A speaker's voice changes with what they are doing in the conversation, while a zero-shot system takes its voice from the reference. Our second hypothesis is that backchannels contribute to the drop. The recordings are rated lowest where the speaker only gives short replies such as "Mm-hmm" or "Okay". We will add the protocol and the results to Section 5.1 and Appendix I.

The listening test compares complete systems and does not isolate the effect of TAC. Our evidence on TAC in all-speaker generation is the event-level analysis above (Tables A to D).

Q3: Effect of Training Stage

We could not complete the second training stage for the model without TAC within the rebuttal period, so we cannot answer this question directly. Instead, we provide two analyses. First, we compare the first-stage models with and without TAC on AMI with 2, 3 and 4 speakers (W2 & Q2 above). Without TAC, gaps longer than 3 s become more frequent as the number of speakers grows. Second, we compare the model with TAC after the first stage and after both stages below.

We evaluate the first-stage model on the AMI test set with 2, 3 and 4 speakers. For this model, 3 and 4 speakers are unseen counts, as the first stage trains only on conversations with 1 or 2 speakers. The second stage primarily improves the WER. The first-stage model already generalizes to more than two speakers. Its WER-Ch, SIM-Ch and UTMOS-Ch remain stable across the three speaker counts. Its overlap and gap durations per minute are higher than those of the two-stage model at 3 and 4 speakers (overlap 6.0 and 8.7 s against 4.4 and 5.7 s, gap 5.5 and 8.1 s against 4.4 and 6.1 s), but the individual events remain short. These results suggest that the first-stage model preserves its turn-taking behavior at speaker counts unseen during training. We do not report the SIM-Ch of the recordings, since the evaluation uses prompt voices that differ from the recorded speakers. We will add these results to Appendix F.

Table F: First-stage and two-stage models, AMI, 2 / 3 / 4 speakers.

(2 / 3 / 4 speakers)RecordingsChorus (two stages)First-stage only
WER-Ch14.0 / 17.0 / 19.08.9 / 9.5 / 9.915.8 / 14.5 / 14.8
UTMOS-Ch3.03 / 2.90 / 2.723.78 / 3.73 / 3.693.75 / 3.69 / 3.67
SIM-Ch-0.91 / 0.91 / 0.900.92 / 0.91 / 0.91
Overlap (s) / Minute5.1 / 10.1 / 15.82.4 / 4.4 / 5.72.1 / 6.0 / 8.7
Overlap (s) / Event1.04 / 1.18 / 1.310.60 / 0.65 / 0.640.60 / 0.75 / 0.78
Gap (s) / Minute3.2 / 4.7 / 4.62.6 / 4.4 / 6.13.0 / 5.5 / 8.1
Gap (s) / Event1.31 / 1.13 / 0.960.66 / 0.75 / 0.830.69 / 0.81 / 0.87
Reviewer_3t6gOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Chorus extends the single-speaker TTS model F5-TTS to multi-speaker conversation. It runs one weight-shared copy of F5-TTS per speaker, each producing its own audio channel, and links the copies after every DiT block with a TAC block that averages features across channels, so the number of speakers is set at inference time. Special tokens encode either turn order or frame-level timestamps, and the infilling mask selects the task: all-speaker, response or long-form generation. On the two-speaker ZipVoice-Dialog test set, Chorus is close to ZipVoice-Dialog-Stereo (WER-Ch 6.61 vs. 6.16, SIM-Ch 0.863 vs. 0.856). On AMI it generates 2- to 4-speaker conversations, and timestamp mode reaches a timing F1 of 0.93.

Critical strengths
  1. A simple recipe that covers several tasks with one model. Weight sharing, a speaker-count-invariant block and speaker-relative tokens make the number of speakers an inference-time choice, and the same model handles all-speaker, response and long-form generation with either turn-order or timestamp control. It starts from a public checkpoint, adds 92M parameters, uses less conversational data than ZipVoice-Dialog-Stereo, and keeps its single-speaker ability. Per-speaker channel data is scarce, so a cheap way to generate it is useful.

  2. Several of the claimed capabilities work as stated. Timestamp mode raises timing F1 from 0.62 to 0.93 on scripted and from 0.67 to 0.95 on spontaneous conversations, and chunking cuts long-form WER-Ch from 19.01 to 4.40. In response generation, the communication block brings overlap with the given channel close to the real value. The authors are also open about the limits of their metrics.

Critical weaknesses
  1. The communication block, the central component, is not shown to help in the main task, and the multi-speaker result has no reference point. In all-speaker generation, the variant without the block matches it on WER-Ch, UTMOS-Ch and SIM-Ch, and its overlap is closer to the real value (3.53 vs. 1.18 s/min, ground truth 3.29; Table 10). The AMI results for 2–4 speakers have no ablation, no per-turn TTS baseline and no ground-truth overlap, and per-channel metrics score each channel alone, so they cannot show that speakers coordinate. Stage 2 also trains on meetings with 4–8 speakers, so speaker-count invariance is a property of the architecture rather than a tested result. This matters because one channel per speaker for more than two speakers is the main novelty claim.

  2. Interaction quality rests on automatic proxies, and there is only one comparison system. There is no listening test, although the paper itself notes that UTMOS favors read speech and that diarization degrades on overlap. The judge scores above ground truth in the abstract show that Chorus's turn events are easy for the judge to predict, not that they are natural: Chorus produces only 40% of the real overlap, and the variant without communication also scores above ground truth. Table 1 has no Chorus row in the mixture panel, so Chorus is compared only with ZipVoice-Dialog-Stereo, where the results are mixed. Since the stated goal is natural conversational data, these measurements are not enough.

Questions for authors
  1. For the AMI subsets in Table 2, can you report the ground-truth overlap, the variant without communication, a per-turn F5-TTS baseline, and the stage-1 model (trained on at most two speakers) on the 3- and 4-speaker subsets? This would show whether the block coordinates speakers and whether invariance holds beyond the speaker counts seen in training.
  2. Could you add a small listening test against ZipVoice-Dialog-Stereo and ground truth, and report mixture metrics for Chorus so that it can be compared with the single-channel systems in Table 1?
  3. In response generation, was the given channel cleaned of cross-talk? If not, it may reveal when the target speaker spoke. A run with the given channel time-shifted would show whether the generated turns actually follow the given audio.
5 comments
AuthorsOpenReview ↗

Response to Reviewer 3t6g (Part 1/5)

We thank the reviewer for the constructive review. We address every point below. In short, we add event-level overlap and gap statistics, in which gaps and overlaps longer than 3 s are more frequent without the communication block than with it (W1, Q1). We add new results to Table 2: the recordings and a concatenation baseline (Q1). We add the mixture metrics of Chorus and ZipVoice-Dialog-Stereo to Table 1 (W2, Q2). We conduct a human evaluation comparing the recordings, ZipVoice-Dialog-Stereo and Chorus (W2, Q2). We reword the claim on the turn-taking judge in the abstract (W2). We remove the cross-talk from the given channel in response generation and find that WER, UTMOS and the turn-taking score do not degrade, and we delay the given channel by 2 seconds and find that the generated speech moves with it in most responses (Q3). Table, figure and section numbers refer to the submitted version. Appendix letters for new material refer to the revised version.

W1 & Q1: Communication block ablation and the multi-speaker evaluation

Our answer has three parts: (a) the communication block ablation, (b) the generalization of the first stage to more than two speakers, and (c) the reference points for Table 2.

(a) Clarification on Communication Block Ablation (W1 & Q1)

While the communication block shows a clear benefit in response generation, the results in all-speaker generation are mixed. To examine this setting more closely, we compare the first-stage models with and without TAC on the ZipVoice-Dialog test set and on AMI with 2, 3, and 4 speakers. All results below are from all-speaker generation in turn-order mode, where the text gives only the order of the turns and not their timing. Table 10 reports only the overlap duration per minute. This hides a second failure case besides prolonged overlaps: prolonged gaps between speakers (dead air). Therefore, we analyze both overlaps and gaps. For each, we report the number of events per minute (# events / length of the conversation), the mean duration per event (total duration of the events / # events), and the frequency of long events (# events longer than 3 s / length of the conversation). Without TAC, gaps longer than 3 s occur more often than in the recordings on both datasets and at every speaker count, which indicates unnatural pauses. With TAC, such gaps are less frequent than without TAC in every condition. Overlaps longer than 3 s are also less frequent with TAC than without TAC in every condition. We will add these statistics for both datasets to Appendix E.

ZipVoice-Dialog, first-stage models all-speaker generation setting

Overlaps: Although the recordings and the model without TAC have similar overlap durations per minute, their overlap patterns differ when examined at the event level. Specifically, without TAC, each overlap event lasts longer on average than in the recordings (0.76 s against 0.60 s). In the worst case, an overlap lasts 7.6 seconds, substantially longer than the maximum observed in the recordings (2.6 seconds), and overlaps longer than 3 s occur more often than in the recordings. This suggests that without TAC the timing of the two channels is not coordinated, and prolonged overlaps occur. With TAC, the overlap is lower than in the recordings in both the number of events (2.6 against 5.5 per minute) and the duration per event (0.43 s against 0.60 s). We acknowledge this as a limitation of the current model.

Table A: Overlap events, ZipVoice-Dialog test set.

Recordingswith TACwithout TAC
# Overlap / Minute5.52.64.5
Overlap (s) / Minute3.31.23.5
Overlap (s) / Event0.600.430.76
# Overlap > 3 s / Minute0.000.000.12
AuthorsOpenReview ↗

Response to Reviewer 3t6g (Part 2/5)

W1 & Q1: Communication block ablation and the multi-speaker evaluation, continued

Dead air: A similar pattern emerges for gaps too. If we consider only the gap duration per minute, all systems exhibit similar values. However, event-level statistics show that without TAC, the model produces fewer gap events, but each gap lasts longer on average. This results in prolonged periods of silence between speakers. In the worst case, the longest gap before another speaker begins speaking is 8.6 seconds, compared to 4.2 seconds in the recordings, and gaps longer than 3 s occur more often than in the recordings. In contrast, with TAC the gap duration per minute matches the recordings and the mean gap is slightly shorter. Gaps longer than 3 s remain more frequent than in the recordings, but they occur less often than without TAC. These results further suggest that without TAC the speaker turns are not coordinated when only the text is given.

Table B: Gap events, ZipVoice-Dialog test set.

Recordingswith TACwithout TAC
# Gap / Minute4.75.53.3
Gap (s) / Minute3.93.94.1
Gap (s) / Event0.840.701.27
# Gap > 3 s / Minute0.030.100.33

AMI, first-stage models with and without the communication block, 2 / 3 / 4 speakers

Overlaps: Unlike on the ZipVoice-Dialog test set, the overlap statistics on AMI do not separate as clearly. In the two-speaker setting, the model without TAC is close to the recordings in the number of overlaps, the duration per minute, and the duration per event. With TAC, the number of overlaps rises with the number of speakers as in the recordings and is close to them with three and four speakers, while without TAC it grows more slowly. However, each overlap with TAC is about 40% shorter than in the recordings, and overlaps longer than 3 s are less frequent. We acknowledge this as a limitation of the current model.

Table C: Overlap events, AMI, 2 / 3 / 4 speakers.

(2 / 3 / 4 speakers)Recordingswith TACwithout TAC
# Overlap / Minute4.9 / 8.6 / 12.13.6 / 7.9 / 11.24.3 / 6.9 / 8.9
Overlap (s) / Minute5.1 / 10.1 / 15.82.1 / 6.0 / 8.75.2 / 7.5 / 9.1
Overlap (s) / Event1.04 / 1.18 / 1.310.60 / 0.75 / 0.781.21 / 1.08 / 1.02
# Overlap > 3 s / Minute0.20 / 0.64 / 1.180.00 / 0.15 / 0.260.38 / 0.47 / 0.50

Dead air: Without TAC, gaps over 3 s occur more often than in the recordings at 2, 3, and 4 speakers, and they grow with the number of speakers. These results again suggest that without TAC the timing of the turns is not coordinated. With TAC, the model produces more gaps than the recordings. However, each of these gaps is shorter than in the recordings on average, and gaps over 3 s are less frequent than in the recordings with two and three speakers and as frequent with four speakers. The gap duration per minute with TAC is above that of the recordings at three and four speakers (5.5 and 8.1 s against 4.7 and 4.6 s), and below that of the model without TAC at every speaker count.

Table D: Gap events, AMI, 2 / 3 / 4 speakers.

(2 / 3 / 4 speakers)Recordingswith TACwithout TAC
# Gap / Minute2.4 / 4.1 / 4.84.4 / 6.8 / 9.32.4 / 5.1 / 6.9
Gap (s) / Minute3.2 / 4.7 / 4.63.0 / 5.5 / 8.14.3 / 7.6 / 10.2
Gap (s) / Event1.31 / 1.13 / 0.960.69 / 0.81 / 0.871.83 / 1.49 / 1.47
# Gap > 3 s / Minute0.23 / 0.29 / 0.260.08 / 0.10 / 0.270.45 / 0.63 / 0.82
AuthorsOpenReview ↗

Response to Reviewer 3t6g (Part 3/5)

W1 & Q1: Communication block ablation and the multi-speaker evaluation, continued

(b) Generalization of the first stage to more than two speakers (W1)

We evaluate the first-stage model on the AMI test set with 2, 3 and 4 speakers. For this model, 3 and 4 speakers are unseen counts, as the first stage trains only on conversations with 1 or 2 speakers. The second stage primarily improves the WER. The first-stage model already generalizes to more than two speakers. Its WER-Ch, SIM-Ch and UTMOS-Ch remain stable across the three speaker counts. Its overlap and gap durations per minute are higher than those of the two-stage model at 3 and 4 speakers (overlap 6.0 and 8.7 s against 4.4 and 5.7 s, gap 5.5 and 8.1 s against 4.4 and 6.1 s), but the individual events remain short. These results suggest that the first-stage model preserves its turn-taking behavior at speaker counts unseen during training. We do not report the SIM-Ch of the recordings, since the evaluation uses prompt voices that differ from the recorded speakers. We will add these results to Appendix F.

Table E: First-stage and two-stage models, AMI, 2 / 3 / 4 speakers.

(2 / 3 / 4 speakers)RecordingsChorus (two stages)First-stage only
WER-Ch14.0 / 17.0 / 19.08.9 / 9.5 / 9.915.8 / 14.5 / 14.8
UTMOS-Ch3.03 / 2.90 / 2.723.78 / 3.73 / 3.693.75 / 3.69 / 3.67
SIM-Ch-0.91 / 0.91 / 0.900.92 / 0.91 / 0.91
Overlap (s) / Minute5.1 / 10.1 / 15.82.4 / 4.4 / 5.72.1 / 6.0 / 8.7
Overlap (s) / Event1.04 / 1.18 / 1.310.60 / 0.65 / 0.640.60 / 0.75 / 0.78
Gap (s) / Minute3.2 / 4.7 / 4.62.6 / 4.4 / 6.13.0 / 5.5 / 8.1
Gap (s) / Event1.31 / 1.13 / 0.960.66 / 0.75 / 0.830.69 / 0.81 / 0.87

(c) Reference points for the multi-speaker table (Table 2) (W1 & Q1)

Recordings Reference: The overlap produced by Chorus scales with the number of speakers, but it stays below that of the recordings, similar to the two-speaker setting shown in Figure 3. Chorus produces 74 to 80% of the number of overlaps in the recordings, and each overlap is about half as long. Without timestamps, long simultaneous speech is less frequent in the output of Chorus than in the recordings. Note that we can still explicitly control the overlap through timestamp mode (Table 4), and controlling this behavior when no timestamps are given is left as future work. We will note this behavior in Section 5.1 and discuss the controllability of the turn-order mode in future work.

Table F: Overlap events of Chorus (two stages), AMI, 2 / 3 / 4 speakers.

(2 / 3 / 4 speakers)RecordingsChorus
# Overlap / Minute4.9 / 8.6 / 12.13.9 / 6.8 / 9.0
Overlap (s) / Minute5.1 / 10.1 / 15.82.4 / 4.4 / 5.7
Overlap (s) / Event1.04 / 1.18 / 1.310.60 / 0.65 / 0.64

Concatenation baseline: We synthesize each turn alone with the single-speaker F5-TTS, using the same sampling settings, and concatenate the turns. The UTMOS-Ch of Chorus is within 0.03 of this baseline. Its WER-Ch is 1.5 points lower at 2 speakers and 0.6 and 1.2 points higher at 3 and 4 speakers, and its SIM-Ch is 0.02 to 0.03 lower. The difference is in the interaction. By construction, the concatenation baseline has no overlap and no gap, while Chorus generates overlaps and gaps between speakers within the same model. We will add both rows to Table 2.

Table G: Chorus and the concatenation baseline, AMI, 2 / 3 / 4 speakers.

(2 / 3 / 4 speakers)WER-ChUTMOS-ChSIM-Ch
Chorus8.9 / 9.5 / 9.93.78 / 3.73 / 3.690.91 / 0.91 / 0.90
Concatenate F5-TTS10.4 / 8.9 / 8.73.81 / 3.76 / 3.690.93 / 0.93 / 0.93

First-stage models with and without TAC: WER-Ch, UTMOS-Ch and SIM-Ch of the two models are close at every speaker count. As on the ZipVoice-Dialog test set (Table 10), TAC does not improve the per-channel metrics. The two models differ in the timing between speakers, as the event-level statistics above show (Tables C and D). We agree with the reviewer that the per-channel metrics score each channel alone. The evidence for coordination between speakers is in the event-level statistics, which are computed across channels. We will add both rows to Appendix E.

Table H: First-stage models with and without TAC, AMI, 2 / 3 / 4 speakers.

(2 / 3 / 4 speakers)WER-ChUTMOS-ChSIM-Ch
First stage, with TAC15.8 / 14.5 / 14.83.75 / 3.69 / 3.670.92 / 0.91 / 0.91
First stage, without TAC16.9 / 14.1 / 13.23.75 / 3.72 / 3.690.92 / 0.92 / 0.92

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.