acceptodds
Under review as a conference paper at ICLR 2027

Continuo: A Large-Scale Dataset for Controllable Long-Form Speech Generation

Abstract

Generating long-form speech requires models to keep content, speaker identity, turn-taking, and expressive style consistent over time. Yet most open training corpora contain examples lasting only tens of seconds and open long-form benchmarks cover at most two speakers per recording. We introduce Continuo, a 229K-hour multilingual dataset for controllable long-form speech generation. From in-the-wild data, Continuo constructs short utterances, single-speaker long-form speech, and multi-speaker dialogues, preserving transcripts, speaker identities, pauses, and acoustic transitions. Examples longer than five minutes account for 50.0% of single-speaker long-form hours and 66.8% of dialogue hours. To support expressive control, Continuo provides natural language instructions for 4.77M examples and nonverbal-vocalization tags for 933K examples. We also introduce Continuo-testset, 23 hours of bilingual recordings of up to 30 minutes and four speakers, each speaker paired with reference audio. We train autoregressive (AR) and non-autoregressive (NAR) models on Continuo. Our 0.6B-backbone NAR model achieves the lowest Chinese CER and English WER on TTSD-Eval (5.27%/9.93%) among compared systems, while our AR model is on par with Qwen3-TTS-12Hz-VD on InstructTTSEval (zh/en: 76.2/78.3 vs. 77.1/77.9). These results validate Continuo as a comprehensive data foundation for long-form speech generation and minute-scale expressive control. Demo: https://continuo6.github.io/continuo-demo/.

Primary area:
datasets and benchmarks
Keywords:
speech datasets, speech synthesis, text-to-speech, long-form speech generation, controllable speech generation, multi-speaker dialogue generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9P4XOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Continuo is a multilingual resource for controllable long-form speech generation, built from 229,388 hours of source-level-deduplicated audio across 41 languages. Its pipeline creates overlapping short-utterance, single-speaker long-form, and multi-speaker dialogue views while retaining continuous source audio, pauses, and speaker-turn structure. Subsets contain 4.77 million instruction-annotated examples and 933 thousand examples with nonverbal-vocalization tags. The paper also introduces a human-verified Chinese/English testset with 118 recordings totaling 23.3 hours, durations up to 30 minutes, and reference audio for up to four speakers. Autoregressive and masked non-autoregressive models with 0.6B sequence backbones demonstrate utility across long-form synthesis, instruction following, and nonverbal generation. Expressive-control evaluations use shorter-than-five-minute subsets, separately from the full-duration synthesis tests.

Critical strengths
  1. A substantial and useful data contribution with a well-motivated long-form representation. Preserving continuous audio, original pauses, and speaker transitions addresses information that independent short utterances lose. The corpus combines substantial long-duration coverage with multilingual, dialogue, instruction, and nonverbal supervision. The paper clearly distinguishes the 229K unique source hours from the 398K hours summed across overlapping views, and does not imply that all examples have expressive annotations. The accompanying 118-case benchmark adds a useful combination of long recordings, up to four speakers, and per-speaker references. Its source/duplicate-group isolation and human verification are explicitly described. The linked testset is publicly accessible with audio and structured fields, and the processing repository is available. These are tangible resources; the full training-manifest release is still planned for acceptance (Appendix H).

  2. Broad downstream validation supports the resource's practical value. Evaluating both AR and NAR architectures is more convincing than demonstrating only one training recipe. Continuo-NAR achieves 5.27% Chinese CER and 9.93% English WER on TTSD-Eval, while Continuo-AR obtains competitive short-form instruction-following results. The paper also reports weaknesses, including NAR degradation at longer durations, rather than suggesting universal superiority. The whole-sequence versus chunked comparison is particularly informative: content error and perceived naturalness can move in opposite directions (Appendix G.1/Table 20). This helps explain why long-form evaluation needs perceptual continuity checks alongside transcription and speaker metrics.

Critical weaknesses
  1. End-to-end annotation quality on the retained training corpus remains insufficiently quantified. Appendix B provides useful component benchmarks, a language-routing audit, and confidence-gating diagnostics, but these do not directly estimate the final corpus's joint transcription, speaker-assignment, temporal-style, and NV annotation errors. For example, the gated emotion recognizer still has WF1 of 0.66-0.77 on the reported validation sets, and captions for recordings longer than 30 seconds are generated from extracted attributes without listening to the waveform. These are reasonable engineering choices, but their downstream annotation fidelity needs a representative audit of the actual retained examples, including longer dialogues and less-represented languages. Human verification of the 118 evaluation cases does not establish the quality of the training annotations. This uncertainty limits how confidently others can use the expressive labels and interpret the dataset's coverage claims.

  2. The proposed benchmark's speaker scores need explicit coverage accounting. Appendix F.2 obtains speaker-scored segments through ASR/script matching, requiring at least 50% exact anchors for forced alignment and excluding segments shorter than 0.5 seconds. Equation (4) then averages SIM and ACC over the resulting segments. As written, this can make speaker performance conditional on successful content alignment: omitted, badly transcribed, or otherwise unalignable portions may not contribute to the speaker score. Evaluating every case does not by itself ensure that comparable portions of each system's output are scored. The paper should report the scored fraction of audio/script/turns for every system and explain how empty or failed alignments are handled. This is a clarification and validation concern, not evidence that the reported rankings are incorrect.

On balance, I lean toward weak acceptance. The scale, preservation of long-form structure, released evaluation resource, and validation under two modeling paradigms constitute a useful contribution. The concerns above constrain the strength of the quality and speaker-consistency conclusions, but appear addressable with targeted audits and fuller evaluation reporting.

Post-rebuttal update (19 Nov 2026):

The annotation audit, speaker-score coverage tables and manual-alignment check substantially address my main concerns, and the added training schedule improves reproducibility. I am keeping my rating at 3 (weak acceptance). The final-data audit is limited to Chinese and English, and the expressive labels still contain noticeable errors; these limitations should remain explicit. The remaining reporting details do not change my overall assessment. This is my final recommendation after the rebuttal, and I have no further questions.

Questions for authors
  1. Final-data quality: Can you report a stratified manual audit of the retained training examples, covering transcription accuracy, speaker attribution, temporal instruction fidelity, and NV event type/placement? Please separate the final-data results from component benchmarks, and include longer dialogues and languages beyond English/Chinese where the corresponding annotations are available.

  2. Speaker-score coverage: What proportions of generated duration, script tokens, and speaker turns enter SIM/ACC for each system? How are slices below the alignment threshold and cases with no valid segments handled? Does the comparison remain similar when coverage is reported alongside the scores and checked on a manually aligned subset?

  3. Training reproducibility: Please specify the actual optimizer steps or epochs and effective audio/token exposure at each curriculum stage, together with the sampling ratios across overlapping views and annotated subsets. Appendix E gives learning rates and duration/token limits, but these additional quantities are needed to reproduce the data exposure and assess how the three views are used. Please identify the exact versioned manifests/configurations corresponding to the reported runs.

6 comments
Reviewer_9P4XOpenReview ↗

Note on AI tool use during my review

Following the PCs' request, I'm adding a note about how AI tools were used during my reviewing process. I discussed this paper with GPT-6-Astra while working on my review. I used it as a sounding board: I shared my own reading notes — mainly the gap between the component-level benchmarks in Appendix B and the end-to-end annotation quality of the retained training corpus, and the speaker-score coverage accounting around Eq. (4) / Appendix F.2 — and asked it to check whether my criticisms were technically sound and whether I had overlooked obvious counterarguments. It also helped me polish the wording of the review text. My assessment of the paper, the rating, and all specific claims in the review are my own: I verified every cited detail (the corpus statistics, the WF1 ranges, the alignment thresholds, the released testset and repository links, etc.) directly against the manuscript. The AI discussion helped me articulate the coverage-accounting concern more precisely, but it did not change my overall assessment. My questions for the authors in the review stand as written, and I look forward to their responses.

AuthorsOpenReview ↗

Response to Reviewer 9P4X (Part 1/3)

Dear Reviewer 9P4X,

Thank you for the positive assessment and the precise, actionable questions. We ran each analysis you proposed and uploaded a revised manuscript, with all revisions marked in blue.

(1) Final-data quality audit (Response to Weakness 1 & Question 1)

“Question 1: Can you report a stratified manual audit of the retained training examples, covering transcription accuracy, speaker attribution, temporal instruction fidelity, and NV event type/placement? Please separate the final-data results from component benchmarks …”

Please refer to Table 4 (Line 369–384, Pages 7–8) and Appendix B.6, Tables 25–26 (Line 1268–1292, Page 24).

We agree that component benchmarks do not measure label quality on Continuo itself, and add a human audit:

  • Transcription accuracy and speaker attribution. The testset pool (50.2 h) was drawn from the collected corpus, labeled by the same pipeline as the training data, and then held out. We compare the unedited pipeline labels with the annotators' corrected labels over the full pool, so the comparison estimates training-label quality. In Chinese / English, it holds 262 / 175 short segments, 17 / 24 long-form recordings and 90 / 58 dialogues (Table R1).
  • Style instructions. We randomly sampled 300 training instructions (150 Chinese, 150 English audio). Ten annotators each checked 30 of them. An instruction receives credit only if every stated attribute, including emotion and each time interval, matches the audio. One wrong interval in a minutes-long description therefore fails the whole instruction, so the proportions in Table R2 are a conservative lower bound.
  • Nonverbal vocalization (NV) tags. We audited 300 training items (150 Chinese, 150 English audio). Ten annotators each checked 30 of them. We compute NV accuracy for each item and average over items (Table R3).

Table R1. Transcripts and speaker labels of 50.2h annotation pools.

Chinese CER ↓English WER ↓Chinese speaker acc. ↑English speaker acc. ↑
Short0.99%1.48%97.30%97.30%
Long-form1.12%0.94%100%100%
Dialogue1.52%1.11%94.40%93.40%

Table R2. Human audit of 300 randomly sampled training instructions.

Chinese style acc. ↑English style acc. ↑
Short89.60%90.40%
Long-form83.60%81.20%
Dialogue78.80%78.20%

Table R3. Mean per-item NV label accuracy in a human audit of 300 training items.

Training-data subsetChinese NV accuracy ↑English NV accuracy ↑
Short-form85.30%86.70%
Long-form80.60%82.70%
Dialogue79.70%74.30%

(2) Languages beyond English and Chinese (Response to Question 1)

“… and include longer dialogues and languages beyond English/Chinese where the corresponding annotations are available.”

Please refer to Appendix B.2, Table 17 (Line 1120–1133, Page 21) and Appendix I (Line 2042–2045, Page 38).

Style and NV annotations exist only for Chinese and English, so those two audits cover every annotated language. For transcripts, we lack qualified annotators in other languages and cannot offer a final-data audit. We state this in Limitations.

As a component benchmark, Table R4 gives the error rates of our transcription model, Qwen3-ASR-1.7B, on the four largest of these languages, which together hold 72.1% of non-Chinese/English hours. It is on par with Whisper-large-v3 on FLEURS, each being better on two languages, and clearly better on Common Voice for Japanese and Korean. Both benchmarks are read speech, so these numbers do not measure accuracy on in-the-wild audio.

Table R4. ASR error rates (%) for four additional languages.

BenchmarkSystemdejakoes
FLEURSQwen3-ASR-1.7B (ours)3.925.202.573.36
FLEURSWhisper-large-v34.304.802.603.12
Common VoiceQwen3-ASR-1.7B (ours)5.8511.645.884.65
Common VoiceWhisper-large-v3–15.2011.90–
AuthorsOpenReview ↗

Response to Reviewer 9P4X (Part 2/3)

(3) Speaker-score coverage (Response to Weakness 2 & Question 2)

“Question 2: What proportions of generated duration, script tokens, and speaker turns enter SIM/ACC for each system? How are slices below the alignment threshold and cases with no valid segments handled? Does the comparison remain similar when coverage is reported alongside the scores and checked on a manually aligned subset?”

Please refer to Appendix F.2, Tables 32–34 (Line 1708–1762, Pages 32–33).

(i) Coverage. Table R5 gives the coverage on CONTINUO-testset, now also in Appendix F.2.

Table R5. Scoring coverage of the SIM/ACC protocol on CONTINUO-testset (Chinese / English).

SystemDuration (%)Script tokens (%)Turns (%)ACC ↑
GT82.0 / 77.091.3 / 88.283.2 / 82.999.33 / 98.38
Higgs Audio v221.6 / 34.63.4 / 3.94.7 / 5.075.00 / 72.84
VibeVoice80.1 / 78.587.7 / 85.783.1 / 81.493.78 / 95.98
MOSS-TTSD73.2 / 69.991.9 / 82.783.8 / 72.099.28 / 99.22
Continuo-AR70.5 / 74.692.3 / 90.784.7 / 84.098.57 / 95.05
Continuo-NAR71.3 / 72.578.1 / 80.282.1 / 79.697.38 / 96.46
  • The protocol has a ceiling. Even ground-truth recordings reach only 77–82% of duration, 88–91% of script tokens and 83% of turns, so coverage should be read against GT.
  • Most strong systems are scored on comparable portions. VibeVoice, MOSS-TTSD and Continuo-AR are within 4 points of GT on tokens and turns, with one exception: in English, MOSS-TTSD covers 82.7% of tokens and 72.0% of turns. Continuo-NAR covers fewer tokens (78–80%) at near-GT turn coverage, consistent with its higher error rates on this set.
  • Very low coverage marks failed generation. Higgs Audio v2 has only 3–5% of tokens and turns scored, matching its 75–81% CER/WER, and it ranks last under either accounting.

(ii) Unscored portions. We do not impute unscored outputs; they are counted as uncovered. Excluding Higgs Audio v2, this happens for three reasons: chunks with under 50% exact ASR matches are not force-aligned (0.0–7.7% of script tokens), segments shorter than 0.5 s are excluded from speaker scoring (3.4–9.3%), and some tokens receive no time alignment (2.0–8.7%). A case with no valid segments at all is excluded from the SIM and ACC averages.

(iii) Manual alignment. We randomly selected 10 Chinese and 10 English cases, aligned every system's output manually and recomputed SIM and ACC. Each cell gives automatic / manual alignment.

Table R6. SIM and ACC under automatic / manual alignment on CONTINUO-testset.

SystemChinese SIM ↑Chinese ACC ↑English SIM ↑English ACC ↑
GT0.865 / 0.86698.9 / 98.40.850 / 0.85099.4 / 99.5
Higgs Audio v20.736 / 0.72869.6 / 70.00.668 / 0.65270.8 / 65.5
VibeVoice0.796 / 0.79391.8 / 91.50.736 / 0.73489.6 / 89.5
MOSS-TTSD0.861 / 0.85999.1 / 99.20.818 / 0.81799.8 / 99.6
Continuo-AR0.830 / 0.83098.6 / 98.70.735 / 0.73293.0 / 91.8
Continuo-NAR0.831 / 0.83196.8 / 96.30.783 / 0.78292.3 / 92.5

Except for Higgs Audio v2, which has few scored segments, manual alignment changes SIM by at most 0.003 and ACC by at most 1.2 points. Among the synthesis systems, both Chinese rankings and the English SIM ranking are identical under the two alignments. In English, only Continuo-AR and Continuo-NAR swap ACC order, with a gap below 1 point. Automatic alignment therefore gives the same comparison as manual alignment.

Reviewer_fu1YOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces CONTINUO, a multilingual corpus for controllable long-form speech generation containing approximately 229K hours of source-level deduplicated audio across 41 languages. Its processing pipeline constructs overlapping short-utterance, single-speaker long-form, and multi-speaker dialogue views while preserving temporal continuity and speaker assignments. Selected subsets contain natural-language style instructions and nonverbal-vocalization annotations. The paper also introduces CONTINUO-testset, comprising 118 human-verified cases totaling approximately 23 hours, with recordings up to 30 minutes and four speakers, each accompanied by reference audio. Validation uses autoregressive and masked non-autoregressive models with 0.6B backbones across long-form synthesis, instruction-following, and nonverbal-vocalization benchmarks.

Critical strengths

● The corpus addresses a consequential data limitation in long-form speech generation. Its construction preserves continuous audio, pauses, speaker transitions, and links between short and long examples. Combining these structures with expressive annotations provides a useful training resource for capabilities that isolated text–speech pairs do not directly capture. The experiments across two modeling paradigms provide evidence of practical utility, including competitive results on external dialogue, instruction-following, and NV benchmarks. ● The benchmark and analyses improve the evaluation of long-form generation. CONTINUO-testset combines extended recordings, up to four speakers, per-speaker reference audio, human verification, and source-level train–test separation. Stratification by duration and speaker count reveals failures obscured by aggregate scores. The comparison between chunked and whole-sequence generation is particularly informative: lower content error can coincide with worse perceived naturalness, motivating evaluation of continuity over complete recordings.

Critical weaknesses

The dialogue construction pipeline filters heavily overlapping speaker regions and discards windows where simultaneous speech exceeds 5% of the duration. While this is a reasonable choice for obtaining cleaner supervision, it biases the corpus toward conversations with largely alternating speaker turns. Natural interactions also involve interruptions, backchannels, and simultaneous speech, whose coverage and modeling remain insufficiently established. Furthermore, the paper does not clearly explain how residual overlapping speech is transcribed, aligned, and represented in the training targets, or how removing overlapping speaker regions interacts with preserving continuous audio. Without an overlap-specific analysis, the demonstrated multi-speaker capability should be interpreted primarily as turn-taking dialogue synthesis, with limited evidence for more complex conversational interactions.

Questions for authors

What proportion of candidate dialogue data is excluded by the overlap filters, and how is remaining overlapping speech handled in the transcripts, speaker labels, and training targets? Does CONTINUO-testset contain and explicitly evaluate simultaneous speech, or is its intended scope primarily alternating-turn dialogue?

3 comments
AuthorsOpenReview ↗

Response to Reviewer fu1Y (Part 1/2)

Dear Reviewer fu1Y,

Thank you for the focused review and for recognizing continuous audio, pauses and speaker transitions as a core strength of Continuo. We have quantified the overlap filter, stated the turn-taking scope throughout the paper and uploaded a revised manuscript, with all revisions marked in blue.

(1) Scope: turn-taking dialogue (Response to Weakness)

“While this is a reasonable choice for obtaining cleaner supervision, it biases the corpus toward conversations with largely alternating speaker turns. … the demonstrated multi-speaker capability should be interpreted primarily as turn-taking dialogue synthesis.”

Please refer to the Abstract (Line 017), Section 1 (Line 074), Section 3.1 (Line 200), Section 5 (Line 294) and Appendix I (Line 2040–2042, Page 38).

We apologize for the lack of clarity regarding CONTINUO's scope in the original submission. CONTINUO is specifically designed for multilingual turn-taking dialogues, aiming to address data scarcity in long-form dialogue TTS while preserving natural flow. We have added "turn-taking" to the abstract, introduction, Sec. 3.1 and Sec. 5 to state this scope explicitly.

CONTINUO does not model overlapping speech and intentionally filters it out to ensure clean supervision, in line with prior works like MOSS-TTSD and VibeVoice. We balance this filtering strategy to avoid aggressive cuts that might disrupt conversational transitions, though minor overlaps may occasionally persist due to imperfect automatic detection.

Accurately modeling overlap presents orthogonal challenges: it requires much finer-grained timing, and existing approaches rely heavily on multi-channel telephone data (e.g., Fisher in dGSLM and Moshi) or synthetic simulation rather than in-the-wild audio. As this work centers on long-sequence turn-taking modeling, we leave overlap modeling to future research.

AuthorsOpenReview ↗

Response to Reviewer fu1Y (Part 2/2)

(2) How much candidate dialogue data the overlap filters exclude (Response to Question)

“What proportion of candidate dialogue data is excluded by the overlap filters …?”

Please refer to Appendix A.1, Tables 11–12 (Line 855–897, Pages 16–17).

Our overlap-handling rules (Appendix A.1) are applied sequentially to all intersecting segment pairs and overall dialogue windows. We audited them on 1,000 h of candidate dialogue audio.

Table R1. Overlap cases by rule (1,000 h; all percentages are of the 243,834 cases).

RuleActionCasesRemovedKept
R1: overlap > 2 sRemove both segments8,575 (3.5%)8,575 (3.5%)0
R2: ≤ 2 s, > 40% of both segmentsRemove both segments27,029 (11.1%)27,029 (11.1%)0
R2: ≤ 2 s, > 40% of one segmentRemove that segment157,400 (64.6%)157,400 (64.6%)0
R3: ≤ 2 s, ≤ 40% of both segmentsKeep both segments50,830 (20.8%)14,433 (5.9%)36,397 (14.9%)
R4: window with > 5% simultaneous speechExclude window–359 (0.1%)−359 (0.1%)
Total243,834 (100%)207,796 (85.2%)36,038 (14.8%)

Table R2. Overlap cases by type.

Overlap typeCasesRemovedKept
Fully contained: one segment lies entirely inside the other (includes backchannels)151,870 (62.3%)151,870 (62.3%)0
Partial: the segments intersect, but neither contains the other91,964 (37.7%)55,926 (22.9%)36,038 (14.8%)
Total243,834 (100%)207,796 (85.2%)36,038 (14.8%)
  • Filtering effect. The filters reduce total overlap duration from 33.91 h to 2.52 h (a 92.6% reduction) and remove 85.2% of overlap cases, including every fully contained case. This yields predominantly single-speaker training signals for turn-taking modeling, at the cost of backchannel coverage.
  • Residual overlaps. The remaining 14.8% of cases consist almost entirely of brief boundary collisions at speaker handoffs, with a median duration of only 0.14 s.

(3) Handling of residual overlap (Response to Question)

“… and how is remaining overlapping speech handled in the transcripts, speaker labels, and training targets?”

Please refer to Appendix A.1, Multi-speaker dialogue (Line 855–861, Page 16).

Since the remaining overlaps are short (median 0.14 s), overlapping segments are treated as sequential turns:

  • Transcripts & speaker labels. Each speech segment is transcribed independently and assigned a single speaker identity. Turns are serialized strictly by their onset times (e.g., [S1] ... [S2] ...) with turn-level timestamps, without introducing specialized simultaneity tokens.
  • Audio & training targets. The source waveforms are preserved without any editing. Each training window consists of a contiguous, continuous span of the original recording.

This serialization matches our target models and prior dialogue systems such as MOSS-TTSD and VibeVoice.

(4) Simultaneous speech in CONTINUO-testset (Response to Question)

“Does CONTINUO-testset contain and explicitly evaluate simultaneous speech, or is its intended scope primarily alternating-turn dialogue?”

No. Simultaneous speech is 0.041% of its duration (at most 0.24% in any window), and no metric scores overlap. Its intended scope is turn-taking dialogue.

We hope this resolves your concern and would be grateful if you would consider raising your score.

AuthorsOpenReview ↗

Response to Reviewer fu1Y

Dear Reviewer fu1Y,

Thank you again for your great efforts and the valuable comments. We have quantified the overlap filtering and clarified the turn-taking scope in the revision. We would greatly appreciate it if you could share any further feedback or remaining questions. We are happy to provide additional clarifications if needed.

Best regards,

Authors

Reviewer_FB3MOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Continuo, a large-scale speech dataset (229k hours, 41 languages) built from in-the-wild speech such as podcasts and livestreams. It covers different speech lengths and number of speakers: short utterances, single-speaker long-form passages and dialogues with 2 to 6 speakers. The long-form versions keep the original full audio, while the short utterances use trimmed versions of the original long audios. A subset of speech clips that are less than 5 minutes are also synthetically annotated with style instructions (by an existing model, Qwen3-Omni-Captioner) and Chinese/English nonverbal tags if they exist. The authors also release a 23 hr test set in English and Chinese consisting of 118 recordings (<= 30 min) with up to 4 speakers in a given recording. The paper trains two models on the data, one autoregressive (AR) and one non-autoregressive (NAR), each with a 0.6B backbone. The NAR model has the lowest error rates on the TTSD-Eval dialogue benchmark and the best scores on the NV-Bench nonverbal benchmark. The AR model follows style instructions about as well as Qwen3-TTS-VD on InstructTTSEval. However, both models are behind the larger 8B MOSS-TTSD on every metric of the paper's own test set, and have higher error rates than it on Long-TTS-Eval.

Critical strengths
  1. Continuo is designed for long-form, multi-speaker generation as well as style labels, which fills a gap since existing open datasets do not support all 3 of these well. Instead of splitting recordings into segments of <= 30 s like Emilia, it keeps different views of the same long source audio: short utterances, single-speaker passages and dialogues with 2 to 6 speakers, simply using time interval metadata, which is clever. The dialogue view (96k hr) is much larger than the 6.8k hr OpenDialog. While existing style label datasets like ParaSpeechCaps or CapSpeech only describe short clips, Continuo has per-speaker style labels (covering multiple speakers in a given dialogue) and also has descriptions of how emotion, speaking rate and loudness change over time for longer segments. The paper also has a human-verified test set of utterances up to 30 minutes and 4 speakers per utterance.
  2. Small models (both NAR and AR versions) trained on Continuo are competitive with much larger systems. With 0.6B param backbones, Continuo-NAR has the lowest TTSD-Eval error rates, Continuo-AR has the highest Long-TTS-Eval speaker similarity in both languages, and Continuo-AR is within 1 point of the 1.7B Qwen3-TTS-VD on InstructTTSEval. So the data can train good long-form, dialogue and instruction-following models, even though they do not beat MOSS-TTSD everywhere.
Critical weaknesses
  1. The paper does not actually show that Continuo is better than existing data, the main thing to show! Its only comparison with other datasets is Table 1's checklist. The paper should have, at minimum, a baseline trained on the best mix of existing open datasets. Then, comparing that baseline with training on Continuo alone or added to the mix, would show what it actually adds. The trained models in the paper only show the data is usable, not that it is better that a simple mix of existing datasets.
    1. Table 1 and Section 2 also miss mentioning some datasets that should be cited: YODAS (https://arxiv.org/abs/2406.00899), whose YODASv2 release has unsegmented video-level audio in 100+ languages, Libriheavy (50k hrs of audiobooks, https://arxiv.org/abs/2309.08105), SpeechCraft (https://arxiv.org/abs/2408.13608) and Parler-TTS' style labelled dataset of 45k hr (https://arxiv.org/abs/2402.01912).
  2. Appendix B shows that the pipeline's tools, which are mostly off-the-shelf models, do well on public benchmarks. But apart from a small language-ID check, the paper does not show how well any tools worked on Continuo, so it is hard to gauge the quality of the dataset.
    1. The evals/checks run on the pipeline's components/tools do not show how well they work on noisy in-the-wild audio, and some tools are not checked at all. The ASR model is validated only on LibriSpeech test-clean, and some numbers come from the tools' own papers (Tables 8 and 10). The diarization model and the in-house synthetic-speech classifier have no reported accuracies. The classifier also covers only Chinese and English, so 36% of long-form hours and 44% of dialogue hours are never filtered, presumably (Table 16).
    2. The style labels (or a subset) of the training dataset are not checked by humans, and the quality filter for long-form examples is looser. Concerningly, for segments over 30 s the captioner writes instructions without hearing the audio, but only uses the extracted attribute list, which is weaker as it cannot add to or fix that list. Long-form examples have a looser filter of an avg DNSMOS of 2.4, while short utterances need 3.0 per segment plus transcript checks (Appendix A.1). The labels are also narrow, e.g. 68% of NV tags are breathing or "Uhm" and 93% of emotion labels are neutral or happy (Figure 6).
  3. The model experiments do not fully support the paper's claims and are selectively reported in the abstract.
    1. MOSS-TTSD beats both Continuo models on all six metrics of the paper's own test set and on Long-TTS-Eval error rates, but the abstract only mentions the TTSD-Eval wins. Strong open baselines are also missing. MGM-Omni introduced Long-TTS-Eval but is not included in the table, Fish Audio S2 is cited only for its tokenizer although it also generates multi-speaker dialogue, and FireRedTTS-2 (https://arxiv.org/abs/2509.02020), an open dialogue model for up to 4 speakers, is not cited.
    2. The evidence for expressive control is weak. On NV-Bench, NV accuracy is scored by Multilingual-NVASR which is the same model the authors used to label NVs in their training data, which is a big issue; Continuo-NAR’s large gains could come from this. The long-form control evals only compare the two Continuo models, with no baselines, so there is no way to tell whether their scores are good. SoulX-Podcast (https://arxiv.org/abs/2510.23541, not cited) would be a natural open baseline, since it supports laughter, sigh and breathing tags in long dialogues.
Questions for authors
  1. Is train-test splitting also done at the speaker/show level? Other episodes of the same podcast could put test speakers into training.
  2. How accurate is the synthetic-speech classifier, and how much audio did it catch as synthetic?

Here are some experiment requests that would help:

  1. The existing-corpora baseline from Critical Weakness 1, at the same training budget. A smaller-scale version is fine.
  2. Some kind of direct evaluation of training-data quality, e.g. how often annotators corrected the pipeline's transcripts and speaker labels on the 118 test cases, plus a human audit of a random sample of long-form and dialogue training examples.
  3. The missing baselines (MGM-Omni on Long-TTS-Eval, FireRedTTS-2 and Fish Audio S2 on the dialogue sets, SoulX-Podcast for long-form NV control), and an NV evaluation by human raters rather than Multilingual-NVASR, as it was already used for creating the training data.
8 comments
AuthorsOpenReview ↗

Response to Reviewer FB3M (Part 1/5)

Dear Reviewer FB3M,

Thank you for the careful review and specific experiment requests. We ran additional experiments and uploaded a revised manuscript, with all revisions marked in blue. Tables R1–R14 below refer to results in this response.

(1) Comparison with existing open corpora (Response to Weakness 1 & Experiment Request 1)

“The paper does not actually show that Continuo is better than existing data, the main thing to show! … The trained models in the paper only show the data is usable, not that it is better that a simple mix of existing datasets.”

Please refer to Table 5 (Line 387–406, Page 8) and Appendix G.4–G.5 (Line 1924–1973, Pages 36–37).

We compare CONTINUO with existing open corpora in two settings. For single-speaker zero-shot synthesis, we compare Emilia and CONTINUO-SHORT on Seed-TTS-Eval (Table R1). Within each pair, we hold the architecture, codec, recipe, batch size, and training budget of 150,000 steps fixed. The Continuo-NAR pair uses the Fish Audio S2 Pro codec, and the Continuo-AR pair uses the CosyVoice3 tokenizer. All runs use AdamW, 9,000 warmup steps, and a cosine schedule configured for 300,000 steps.

Table R1. Seed-TTS-Eval results.

ModelChinese CER ↓Chinese SIM ↑English WER ↓English SIM ↑
Emilia NAR1.50.7372.10.637
CONTINUO-SHORT-NAR1.130.7371.980.639
Emilia AR1.270.6962.210.605
CONTINUO-SHORT-AR1.130.6911.690.604

With the training setup fixed within each pair, switching from Emilia to CONTINUO-SHORT reduces CER or WER by 6–25% relative across the two architectures, with comparable speaker similarity.

For dialogue synthesis, we compare OpenDialog and CONTINUO-DIALOGUE on the ZipVoice-Dialog testsets (Table R2). OpenDialog uses segments of at most 30 seconds, so we cannot evaluate on CONTINUO-testset. Both runs start from the same Continuo-NAR checkpoint after training on CONTINUO-LONG. We finetuned one on OpenDialog for 60,000 steps, following ZipVoice-Dialog, comparing with the stage S3 checkpoint trained on CONTINUO-DIALOGUE.

Table R2. ZipVoice-Dialog testsets results for Continuo-NAR finetuned on OpenDialog or CONTINUO-DIALOGUE from the same checkpoint.

Model (dialogue data)Chinese cpSIM ↑Chinese CER ↓Chinese UTMOS ↑English cpSIM ↑English WER ↓English UTMOS ↑
Ours (OpenDialog)0.5624.922.610.4858.773.10
Ours, CONTINUO-DIALOGUE-NAR0.6062.012.510.5312.553.26

The CONTINUO-DIALOGUE checkpoint achieves lower CER and WER and higher cpSIM. These comparisons show lower synthesis error rates against one existing open corpus in each setting. The incremental effect of adding CONTINUO to a mixture of existing open corpora remains to be measured.

(2) Missing dataset citations (Response to Weakness 1.a)

“Table 1 and Section 2 also miss mentioning some datasets that should be cited: YODAS, … Libriheavy, … SpeechCraft … and Parler-TTS' style labelled dataset of 45k hr.”

Please refer to Table 1 and Section 2 (Line 055–107, Page 2).

Thank you for pointing this out. We have added the missing citations for YODAS, Libriheavy, SpeechCraft, and the 45,000-hour style-annotated corpus described by Lyth and King.

AuthorsOpenReview ↗

Response to Reviewer FB3M (Part 2/5)

(3) ASR robustness on noisy speech (Response to Weakness 2.a)

“The ASR model is validated only on LibriSpeech test-clean, and some numbers come from the tools' own papers.”

Please refer to Appendix B.2, Table 16 (Line 1100–1118, Page 21).

Appendix B.2 now reports results for our deployed Qwen3-ASR-1.7B model on public benchmarks that include noisy speech (Table R3).

Table R3. ASR results (WER) for the deployed Qwen3-ASR-1.7B model and baselines.

TestsetOurs (Qwen3-ASR-1.7B)Whisper-large-v3GPT-4o-TranscribeGemini-2.5-Pro
LibriSpeech test-clean2.351.511.392.89
LibriSpeech test-other3.383.973.753.56
WenetSpeech test_net5.019.8615.314.43
WenetSpeech test_meeting5.8219.1132.2713.47
GigaSpeech test8.79.7625.59.37

Qwen3-ASR-1.7B achieves the lowest error rates across nearly all evaluated benchmarks, with the sole exception of the clean read-speech set. Its largest gains are on in-the-wild datasets such as WenetSpeech and GigaSpeech, which supports its use for annotating in-the-wild audio.

(4) Diarization accuracy (Response to Weakness 2.a)

“The diarization model and the in-house synthetic-speech classifier have no reported accuracies.”

Please refer to Appendix B.3, Table 18 (Line 1134–1148, Page 22).

Table R4 reports diarization error rate (DER, %; lower is better) for DiariZen-Large-s80-v2, the diarizer in our pipeline, against pyannote 3.1. DiariZen is better on all four benchmarks, with the largest gains on RAMC and MSDWild, the two closest to CONTINUO's source material.

Table R4. DER (%, ↓) for DiariZen and baselines on four public benchmarks.

TestsetDiariZen-Large-s80-v2 (ours)pyannote 3.1
MSDWild15.825.3
VoxConverse9.111.3
DIHARD314.521.7
RAMC1122.2

(5) Synthetic-speech detector: accuracy and removal rate (Response to Weakness 2.a & Question 2)

“Question 2: How accurate is the synthetic-speech classifier, and how much audio did it catch as synthetic?”

Please refer to Appendix B.5, Tables 23–24 (Line 1233–1259, Pages 23–24).

We trained the detector exclusively on public anti-spoofing datasets. Table R5 reports equal error rates (EERs).

Table R5. EER for the CONTINUO synthetic-speech detector and Nes2NetX on three evaluation sets.

TestsetNes2NetX EER (%) ↓Ours EER (%) ↓
In-the-Wild7.81.3
SONAR31.50.3
ADD 2023 Round 218.51.8

Our detector achieves lower EERs than Nes2NetX on all three sets, which include unseen in-the-wild deepfakes and audio from recent TTS systems.

Table R6. Estimated share of Chinese and English candidate audio removed after being flagged as likely synthetic in 1000 hours audio data.

LanguageShortLongDialogue
Chinese19.90%17.30%9.50%
English11.20%9.90%4.30%

As shown in Table R6, up to 20% of Chinese and English candidate audio samples were flagged as synthetic and removed.

(6) Language coverage of the detector (Response to Weakness 2.a)

“The classifier also covers only Chinese and English, so 36% of long-form hours and 44% of dialogue hours are never filtered, presumably (Table 16).”

Please refer to Appendix B.5 (Line 1262–1266, Page 24) and Appendix I (Line 2042–2045, Page 38).

It is correct that the screening process currently focuses on Chinese and English. This was a deliberate choice. We only apply the detector where its accuracy can be verified on in-domain audio. For other languages, the available benchmarks do not support such verification: MLAAD consists largely of clean, synthetic read speech, and ML-ITW offers about two hours per language. Neither tells us how reliable a detector is on noisy, in-the-wild audio in other languages. Filtering with an unvalidated detector would risk discarding genuine speech at an unknown rate while giving a false sense of assurance, which we consider worse than transparently marking the data as unscreened.

AuthorsOpenReview ↗

Response to Reviewer FB3M (Part 3/5)

(7) Human audit of the training labels (Response to Weakness 2.b & Experiment Request 2)

“The style labels (or a subset) of the training dataset are not checked by humans …”

“Experiment request 2: Some kind of direct evaluation of training-data quality, e.g. how often annotators corrected the pipeline's transcripts and speaker labels on the 118 test cases, plus a human audit of a random sample of long-form and dialogue training examples.”

Please refer to Table 4 (Line 369–384, Pages 7–8) and Appendix B.6, Tables 25–26 (Line 1268–1292, Page 24).

We agree that component benchmarks on public test sets do not measure label quality on CONTINUO itself. We have added a three-part human audit:

  • Transcripts and speaker labels. This is the in-the-wild evaluation you suggested. CONTINUO-testset was drawn from the collected corpus, processed by the same pipeline, and then removed from training. We compare the pipeline labels with the annotators' corrected labels over the full annotated pool of 50.2 h. In Chinese / English, it holds 262 / 175 short segments, 17 / 24 long-form recordings and 90 / 58 dialogues (Table R7).
  • Style instructions. We randomly sampled 300 training instructions (150 Chinese, 150 English audio). Ten annotators each checked 30 of them. An instruction receives credit only if every stated attribute, including emotion and each time interval, matches the audio. One wrong interval in a minutes-long description therefore fails the whole instruction, so the proportions in Table R8 are a conservative lower bound.
  • Nonverbal vocalization (NV) tags. We audited 300 training items (150 Chinese, 150 English audio). Ten annotators each checked 30 of them. We compute NV accuracy for each item and average over items (Table R9).

Table R7. Transcripts and speaker labels of 50.2h annotation pools.

Chinese CER ↓English WER ↓Chinese speaker acc. ↑English speaker acc. ↑
Short0.99%1.48%97.30%97.30%
Long-form1.12%0.94%100%100%
Dialogue1.52%1.11%94.40%93.40%

Table R8. Human audit of 300 randomly sampled training instructions.

Chinese style acc. ↑English style acc. ↑
Short89.60%90.40%
Long-form83.60%81.20%
Dialogue78.80%78.20%

Table R9. Mean per-item NV label accuracy in a human audit of 300 training items.

Training-data subsetChinese NV accuracy ↑English NV accuracy ↑
Short-form85.30%86.70%
Long-form80.60%82.70%
Dialogue79.70%74.30%

(8) Captions for segments longer than 30 s (Response to Weakness 2.b)

“Concerningly, for segments over 30 s the captioner writes instructions without hearing the audio, but only uses the extracted attribute list, which is weaker as it cannot add to or fix that list.”

Please refer to Section 3.2 (Line 210–215, Page 4) and Appendix B.6, Table 25 (Line 1268–1282, Page 24).

For audio longer than 30 seconds, the captioner receives timestamped attributes extracted from the recording. This accommodates the model's 30s audio input limit, and the resulting caption depends on the extracted attributes. Across the two languages, as shown in Table R8, full-instruction accuracy was 81.2–83.6% for long-form audio and 78.2–78.8% for dialogue, compared with 89.6–90.4% for short audio.

(9) DNSMOS threshold for long-form examples (Response to Weakness 2.b)

“Long-form examples have a looser filter of an avg DNSMOS of 2.4, while short utterances need 3.0 per segment plus transcript checks (Appendix A.1).”

Please refer to Appendix A.1, Table 10 (Line 819–836, Page 16).

Following Emilia's filtering pipeline, we use a duration-weighted mean DNSMOS threshold of 2.4 for long-form and dialogue audio, so that a brief noisy passage does not discard an otherwise clean recording. Long-form examples need no separate transcript check. They are built by extending across adjacent same-speaker regions (Section 3.1), so their transcripts are concatenations of timestamped segment-level transcripts and inherit the segments' text–audio alignment. Human-corrected labels confirm this: long-form CER/WER is 1.12%/0.94% and dialogue CER/WER is 1.52%/1.11% (Table R7).

Table R10. Share of sampled audio hours from recordings with a mean DNSMOS above 3.0.

LanguageLong-formDialogue
Chinese69.60%50.80%
English80.90%77.40%
All languages76.00%62.80%

As shown in Table R10, despite the looser 2.4 threshold, most of our long-form and dialogue audio exceeds the stricter 3.0 threshold (76.0% for long-form and 62.8% for dialogue).

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

Dear Reviewers and ACs,

We thank all three reviewers for their constructive and detailed reviews. We are glad that the reviewers found that CONTINUO fills a gap by combining long-form, multi-speaker and expressive supervision (FB3M, fu1Y, 9P4X), that preserving continuous audio, pauses and speaker transitions is valuable (fu1Y, 9P4X), that CONTINUO-testset with up to four speakers and per-speaker references is a useful benchmark (FB3M, fu1Y, 9P4X), and that validation under both AR and NAR paradigms and the chunked vs. whole-sequence analysis are informative (fu1Y, 9P4X).

We have uploaded a revised PDF with all changes marked in blue. The main additions are:

  1. Comparison with existing open corpora (FB3M). With the training setup fixed, replacing Emilia with CONTINUO-SHORT reduces Seed-TTS-Eval CER/WER by 6–25% relative, and replacing OpenDialog with CONTINUO-DIALOGUE reduces ZipVoice-Dialog CER/WER from 4.92%/8.77% to 2.01%/2.55% (Sec. 5.4, App. G.4–G.5).
  2. Human audit of the training labels (FB3M, 9P4X). Pipeline transcripts have 0.94–1.52% CER/WER and 93.4–100% speaker accuracy against human-corrected labels; whole-instruction style accuracy is 78.2–90.4% and NV tag accuracy is 74.3–86.7% (Table 4, App. B.6).
  3. Validation of pipeline components (FB3M). ASR on noisy and in-the-wild sets, diarization error rates, and the synthetic-speech detector's EERs and removal rates (App. B.2, B.3, B.5).
  4. Additional baselines (FB3M). MGM-Omni on Long-TTS-Eval, FireRedTTS-2 and Fish Audio S2 on CONTINUO-testset, and SoulX-Podcast for long-form NV control (App. G.1–G.2).
  5. Human evaluation on NV-Bench (FB3M). Human ratings agree in direction with the Multilingual-NVASR scores (App. F.4).
  6. Overlap audit and turn-taking scope (fu1Y). The overlap filters remove 85.2% of overlap cases and reduce overlapped duration from 33.91 h to 2.52 h; the turn-taking scope is now stated throughout (App. A.1, App. I).
  7. Speaker-score coverage (9P4X). Scored fractions of duration, script tokens and turns for every system, plus a manual-alignment check that changes SIM by at most 0.003 (App. F.2).
  8. Training schedule (9P4X). Steps and audio exposure for each curriculum stage (App. E.2).
  9. Release safeguards (FB3M, ethics). The CONTINUO-testset repository is now gated, and the training set will be released only as URL manifests with annotations (Ethics Statement, App. H).
  10. Abstract and citations (FB3M). The abstract now states that the 8B MOSS-TTSD remains stronger on CONTINUO-testset and Long-TTS-Eval, and we cite YODAS, Libriheavy, SpeechCraft, Parler-TTS, FireRedTTS-2 and SoulX-Podcast.
AuthorsOpenReview ↗

Thank You to the ICLR Reviewing Committee

We sincerely thank all three reviewers for their careful reading of our submission and their constructive feedback. We are also grateful to the Area Chairs, Senior Area Chairs, and Program Chairs.

During the discussion period, we received responses from 2 of the 3 reviewers. Both acknowledged that our revisions and additional experiments addressed most of their concerns. Here is a summary of their replies and score changes:

  • Reviewer FB3M increased their overall rating from 2 (Weak rejection) to 3 (Weak acceptance), commenting on the substantial additional work in our response and noting that most of their concerns have been addressed.
  • Reviewer 9P4X maintained their overall rating of 3 (Weak acceptance), stating that the training-label audit, speaker-score coverage tables, and manual-alignment check substantially address their main concerns, and that the added training schedule improves reproducibility.

We once again thank all the reviewers for their valuable suggestions, which have helped us substantially improve our work. We also extend our gratitude to the Area Chairs, Senior Area Chairs, and Program Chairs for their hard work during this review cycle.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.