- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Chorus extends a pretrained single-speaker TTS model to multi-speaker conversational speech synthesis. It uses weight-shared F5-TTS branches, one per speaker, with Transform–Average–Concatenate (TAC) blocks exchanging information across branches. The model generates separate, synchronized speaker channels and supports turn-order conditioning, timestamp conditioning, and response generation conditioned on other speakers’ complete audio. Longer conversations are generated in chunks, with the final five seconds of the preceding chunk provided as context. Experiments evaluate speech quality, intelligibility, speaker similarity, timing, and conversational event statistics, including generation with two to four speakers.
Critical strengths
● A flexible architecture for separate-channel conversational synthesis. Sharing a pretrained TTS model across speakers and connecting its branches through communication blocks provides a practical extension to multiple speakers without speaker-count-specific parameterization. Producing separate channels also makes speaker-specific timing and overlapping speech directly representable. ● Several useful synthesis settings within one framework. The same model supports turn-order and timestamp conditioning, generation conditioned on recorded speaker channels, and chunked long-form synthesis. Experiments demonstrate these capabilities and evaluate scaling from two to four speakers, providing evidence of practical versatility beyond conventional single-speaker TTS.
Critical weaknesses
● Limited evidence of long-form conversational coherence. Generation uses chunks of at most 45 seconds with only five seconds of preceding audio as context. Table 3 demonstrates improved intelligibility, but does not establish long-range prosodic or emotional consistency or smooth chunk boundaries. ● Mixed evidence for TAC’s coordination benefits. Chorus underproduces overlap compared with real conversations (1.33 vs. 3.29 s/min). In Table 10, adding cross-channel communication further suppresses overlap without improving per-channel intelligibility or quality. Although response-generation results are more encouraging, stronger evidence is needed to show that TAC improves natural coordination rather than merely reducing simultaneous speech.
Questions for authors
- Long-form evaluation: What are the duration distribution and maximum duration of the conversations evaluated in Table 3? Do you have evidence specifically assessing chunk-boundary continuity and longer-range prosodic or emotional consistency, beyond aggregate intelligibility and speaker-similarity metrics?
- Interpretation of the TAC ablation: In Table 10, cross-channel averaging reduces overlap substantially below the ground-truth level without improving per-channel intelligibility or quality. What evidence shows that it produces more appropriate interruptions, backchannels, or turn transitions in all-speaker generation? Human judgments or event-specific analyses could materially change my assessment.
- Effect of training stage: The communication ablations use only first-stage training. Does the same comparison after the complete training procedure support a stronger benefit from TAC, particularly for conversations with more than two speakers?
3 comments
Response to Reviewer nok2 (Part 2/3)
W2 & Q2: Clarification on Communication Block Ablation, continued
ZipVoice-Dialog, first-stage models all-speaker generation setting
Overlaps: Although the recordings and the model without TAC have similar overlap durations per minute, their overlap patterns differ when examined at the event level. Specifically, without TAC, each overlap event lasts longer on average than in the recordings (0.76 s against 0.60 s). In the worst case, an overlap lasts 7.6 seconds, substantially longer than the maximum observed in the recordings (2.6 seconds), and overlaps longer than 3 s occur more often than in the recordings. This suggests that without TAC the timing of the two channels is not coordinated, and prolonged overlaps occur. With TAC, the overlap is lower than in the recordings in both the number of events (2.6 against 5.5 per minute) and the duration per event (0.43 s against 0.60 s). We acknowledge this as a limitation of the current model.
Table A: Overlap events, ZipVoice-Dialog test set.
| Recordings | with TAC | without TAC | |
|---|---|---|---|
| # Overlap / Minute | 5.5 | 2.6 | 4.5 |
| Overlap (s) / Minute | 3.3 | 1.2 | 3.5 |
| Overlap (s) / Event | 0.60 | 0.43 | 0.76 |
| # Overlap > 3 s / Minute | 0.00 | 0.00 | 0.12 |
Dead air: A similar pattern emerges for gaps too. If we consider only the gap duration per minute, all systems exhibit similar values. However, event-level statistics show that without TAC, the model produces fewer gap events, but each gap lasts longer on average. This results in prolonged periods of silence between speakers. In the worst case, the longest gap before another speaker begins speaking is 8.6 seconds, compared to 4.2 seconds in the recordings, and gaps longer than 3 s occur more often than in the recordings. In contrast, with TAC the gap duration per minute matches the recordings and the mean gap is slightly shorter. Gaps longer than 3 s remain more frequent than in the recordings, but they occur less often than without TAC. These results further suggest that without TAC the speaker turns are not coordinated when only the text is given.
Table B: Gap events, ZipVoice-Dialog test set.
| Recordings | with TAC | without TAC | |
|---|---|---|---|
| # Gap / Minute | 4.7 | 5.5 | 3.3 |
| Gap (s) / Minute | 3.9 | 3.9 | 4.1 |
| Gap (s) / Event | 0.84 | 0.70 | 1.27 |
| # Gap > 3 s / Minute | 0.03 | 0.10 | 0.33 |
AMI, first-stage models with and without the communication block, 2 / 3 / 4 speakers
Overlaps: Unlike on the ZipVoice-Dialog test set, the overlap statistics on AMI do not separate as clearly. In the two-speaker setting, the model without TAC is close to the recordings in the number of overlaps, the duration per minute, and the duration per event. With TAC, the number of overlaps rises with the number of speakers as in the recordings and is close to them with three and four speakers, while without TAC it grows more slowly. However, each overlap with TAC is about 40% shorter than in the recordings, and overlaps longer than 3 s are less frequent. We acknowledge this as a limitation of the current model.
Table C: Overlap events, AMI, 2 / 3 / 4 speakers.
| (2 / 3 / 4 speakers) | Recordings | with TAC | without TAC |
|---|---|---|---|
| # Overlap / Minute | 4.9 / 8.6 / 12.1 | 3.6 / 7.9 / 11.2 | 4.3 / 6.9 / 8.9 |
| Overlap (s) / Minute | 5.1 / 10.1 / 15.8 | 2.1 / 6.0 / 8.7 | 5.2 / 7.5 / 9.1 |
| Overlap (s) / Event | 1.04 / 1.18 / 1.31 | 0.60 / 0.75 / 0.78 | 1.21 / 1.08 / 1.02 |
| # Overlap > 3 s / Minute | 0.20 / 0.64 / 1.18 | 0.00 / 0.15 / 0.26 | 0.38 / 0.47 / 0.50 |
Dead air: Without TAC, gaps over 3 s occur more often than in the recordings at 2, 3, and 4 speakers, and they grow with the number of speakers. These results again suggest that without TAC the timing of the turns is not coordinated. With TAC, the model produces more gaps than the recordings. However, each of these gaps is shorter than in the recordings on average, and gaps over 3 s are less frequent than in the recordings with two and three speakers and as frequent with four speakers. The gap duration per minute with TAC is above that of the recordings at three and four speakers (5.5 and 8.1 s against 4.7 and 4.6 s), and below that of the model without TAC at every speaker count.
Table D: Gap events, AMI, 2 / 3 / 4 speakers.
| (2 / 3 / 4 speakers) | Recordings | with TAC | without TAC |
|---|---|---|---|
| # Gap / Minute | 2.4 / 4.1 / 4.8 | 4.4 / 6.8 / 9.3 | 2.4 / 5.1 / 6.9 |
| Gap (s) / Minute | 3.2 / 4.7 / 4.6 | 3.0 / 5.5 / 8.1 | 4.3 / 7.6 / 10.2 |
| Gap (s) / Event | 1.31 / 1.13 / 0.96 | 0.69 / 0.81 / 0.87 | 1.83 / 1.49 / 1.47 |
| # Gap > 3 s / Minute | 0.23 / 0.29 / 0.26 | 0.08 / 0.10 / 0.27 | 0.45 / 0.63 / 0.82 |
Response to Reviewer nok2 (Part 3/3)
Q2: Human Evaluation
We conducted a listening test on Prolific comparing the recordings, ZipVoice-Dialog-Stereo and Chorus on 20 conversations of the ZipVoice-Dialog test set. In the naturalness test, raters heard a two-speaker conversation and rated how natural and expressive it is on a 1-5 scale. In the speaker similarity test, raters heard a reference clip and one speaker's channel of the conversation, and rated on a 1-5 scale how likely the two come from the same speaker. Each clip was rated by 10 raters. The table reports the mean ± the half-width of the 95% bootstrap confidence interval.
Table E: Listening test, ZipVoice-Dialog test set.
| System | Naturalness | Speaker similarity |
|---|---|---|
| Recordings | 3.54 ± 0.20 | 3.09 ± 0.32 |
| ZipVoice-Dialog-Stereo | 3.03 ± 0.17 | 3.10 ± 0.37 |
| Chorus | 3.38 ± 0.10 | 3.74 ± 0.23 |
Chorus is rated more natural than ZipVoice-Dialog-Stereo (p = 0.004, paired permutation test). The recordings remain the most natural, and their difference to Chorus is not significant (p = 0.18). Chorus also obtains a higher speaker similarity than ZipVoice-Dialog-Stereo (p = 0.006, paired permutation test). Chorus also has a higher speaker similarity than the recordings (p = 0.017). E2 TTS (Eskimez et al., 2024) reports the same pattern on audiobooks. There, the ground truth receives a lower similarity score than every tested system, which the authors attribute to speakers who vary their voice for different characters. Our first hypothesis is that spontaneous conversation has the same effect. A speaker's voice changes with what they are doing in the conversation, while a zero-shot system takes its voice from the reference. Our second hypothesis is that backchannels contribute to the drop. The recordings are rated lowest where the speaker only gives short replies such as "Mm-hmm" or "Okay". We will add the protocol and the results to Section 5.1 and Appendix I.
The listening test compares complete systems and does not isolate the effect of TAC. Our evidence on TAC in all-speaker generation is the event-level analysis above (Tables A to D).
Q3: Effect of Training Stage
We could not complete the second training stage for the model without TAC within the rebuttal period, so we cannot answer this question directly. Instead, we provide two analyses. First, we compare the first-stage models with and without TAC on AMI with 2, 3 and 4 speakers (W2 & Q2 above). Without TAC, gaps longer than 3 s become more frequent as the number of speakers grows. Second, we compare the model with TAC after the first stage and after both stages below.
We evaluate the first-stage model on the AMI test set with 2, 3 and 4 speakers. For this model, 3 and 4 speakers are unseen counts, as the first stage trains only on conversations with 1 or 2 speakers. The second stage primarily improves the WER. The first-stage model already generalizes to more than two speakers. Its WER-Ch, SIM-Ch and UTMOS-Ch remain stable across the three speaker counts. Its overlap and gap durations per minute are higher than those of the two-stage model at 3 and 4 speakers (overlap 6.0 and 8.7 s against 4.4 and 5.7 s, gap 5.5 and 8.1 s against 4.4 and 6.1 s), but the individual events remain short. These results suggest that the first-stage model preserves its turn-taking behavior at speaker counts unseen during training. We do not report the SIM-Ch of the recordings, since the evaluation uses prompt voices that differ from the recorded speakers. We will add these results to Appendix F.
Table F: First-stage and two-stage models, AMI, 2 / 3 / 4 speakers.
| (2 / 3 / 4 speakers) | Recordings | Chorus (two stages) | First-stage only |
|---|---|---|---|
| WER-Ch | 14.0 / 17.0 / 19.0 | 8.9 / 9.5 / 9.9 | 15.8 / 14.5 / 14.8 |
| UTMOS-Ch | 3.03 / 2.90 / 2.72 | 3.78 / 3.73 / 3.69 | 3.75 / 3.69 / 3.67 |
| SIM-Ch | - | 0.91 / 0.91 / 0.90 | 0.92 / 0.91 / 0.91 |
| Overlap (s) / Minute | 5.1 / 10.1 / 15.8 | 2.4 / 4.4 / 5.7 | 2.1 / 6.0 / 8.7 |
| Overlap (s) / Event | 1.04 / 1.18 / 1.31 | 0.60 / 0.65 / 0.64 | 0.60 / 0.75 / 0.78 |
| Gap (s) / Minute | 3.2 / 4.7 / 4.6 | 2.6 / 4.4 / 6.1 | 3.0 / 5.5 / 8.1 |
| Gap (s) / Event | 1.31 / 1.13 / 0.96 | 0.66 / 0.75 / 0.83 | 0.69 / 0.81 / 0.87 |
Response to Reviewer nok2 (Part 1/3)
We thank the reviewer for the constructive review. We address every point below. In short, we add F0, Loudness and Timbre on the long-form set, which show no acoustic discontinuity at chunk boundaries (W1, Q1). We add event-level overlap and gap statistics on ZipVoice-Dialog and AMI, in which gaps and overlaps longer than 3 s are more frequent without the communication block than with it (W2, Q2). We compare the first-stage models with and without the communication block on 2, 3 and 4 speakers, and the first-stage and the two-stage models (Q3). We also report a human evaluation of Chorus against ZipVoice-Dialog-Stereo and the recordings (Q2). Table, figure and section numbers refer to the submitted version. Appendix letters for new material refer to the revised version.
W1 & Q1: Long-form evaluation
Durations. The average duration of the long-form generation samples is 83 seconds, with a maximum of 167 seconds. We will add these statistics to Section 5.1.
Chunk boundaries. For every pair of consecutive speech units of the same speaker, we measure the difference in loudness, pitch and timbre. We separate the pairs into two types: (1) both units inside the same chunk, and (2) the last unit of one chunk and the first unit of the next chunk. If the chunked inference produced a discontinuity, the pairs of type (2) should show larger jumps. The table shows that the jumps across a chunk boundary are as small as those inside a chunk for loudness, F0 and timbre. This suggests that the chunked inference produces continuous speech across chunk boundaries. We will report this result in Section 5.1 and Appendix G.
Consistency over the dialogue. To test for drift over a long dialogue, we additionally report the scores of each chunk by its position in the dialogue. From the first to the third chunk, UTMOS-Ch decreases from 3.96 to 3.89 and SIM-Ch from 0.93 to 0.91, and WER-Ch shows no trend (3.4, 4.6 and 2.5). The fourth chunk exists for 10 dialogues only, so its scores are less reliable. We did not evaluate long-range prosodic or emotional consistency. Each chunk is conditioned only on the last 5 seconds of the previous chunk, and we state this as a limitation in Appendix J. We will add this result to Section 5.1 and Appendix G.
W2 & Q2: Clarification on Communication Block Ablation
While the communication block shows a clear benefit in response generation, the results in all-speaker generation are mixed. To examine this setting more closely, we compare the first-stage models with and without TAC on the ZipVoice-Dialog test set and on AMI with 2, 3, and 4 speakers. All results below are from all-speaker generation in turn-order mode, where the text gives only the order of the turns and not their timing. Table 10 reports only the overlap duration per minute. This hides a second failure case besides prolonged overlaps: prolonged gaps between speakers (dead air). Therefore, we analyze both overlaps and gaps. For each, we report the number of events per minute (# events / length of the conversation), the mean duration per event (total duration of the events / # events), and the frequency of long events (# events longer than 3 s / length of the conversation). Without TAC, gaps longer than 3 s occur more often than in the recordings on both datasets and at every speaker count, which indicates unnatural dead air. With TAC, such gaps are less frequent than without TAC in every condition. Overlaps longer than 3 s are also less frequent with TAC than without TAC in every condition. The effect of the block is not limited to a reduction of simultaneous speech. We will add these statistics for both datasets to Appendix E.