acceptodds
Under review as a conference paper at ICLR 2027

VibeVoice-ASR-Streaming: Who Spoke What in Real Time

Abstract

Real-time recognition of multi-speaker conversations is becoming increasingly important. Such systems need to transcribe speech, identify the speaker of each segment, and produce speaker-attributed transcripts with low delay. Existing end-to-end speaker-attributed ASR models jointly perform transcription and speaker attribution. However, they generally process complete recordings. Meanwhile, streaming LLM-based ASR has mainly focused on single-speaker transcription and generally lacks speaker-attribution capability. To address this gap, we present VibeVoice-ASR-Streaming, an end-to-end framework for streaming speaker-attributed ASR. It interleaves fixed-duration audio chunks with previously generated speaker labels and text in a single autoregressive context, which preserves speaker identities across chunks and enables speaker-attributed transcription in a single process without a separate diarization stage. On five recognition benchmarks, our 7B model achieves the lowest mean recognition error among the compared streaming systems. Across 13 speaker-attributed settings, it achieves the best or tied-best cpWER/cpCER in 12 settings. The expected latency is 2.00s. We we will release the 1.5B and 7B model weights together with the inference code.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
streaming automatic speech recognition, speaker-attributed speech recognition, multi-speaker speech recognition

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_6TKKOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces VibeVoice-ASR-Streaming, an end-to-end LLM-based streaming speaker-attributed automatic speech recognition (ASR) model. The model processes incoming audio in fixed-duration chunks (with a lookahead chunk) and interleaves these latent audio representations with previously generated speaker labels and transcripts inside a single autoregressive LLM context. The model is built on Qwen2.5 1.5B and 7B backbones with frozen VibeVoice acoustic/semantic encoders. The model is trained via a three-stage curriculum: offline pre-training, streaming pre-training on 420k hours of speech aligned via Qwen3-ForcedAligner-0.6B, and streaming fine-tuning on public datasets plus 4.5k hours of synthetic multi-speaker meeting data. Across multiple benchmarks, the 7B model achieves state-of-the-art streaming speaker-attributed performance (cpWER/cpCER) with an algorithmic latency of ~2.00 s, outperforming commercial cloud services and matching or beating streaming recognition-only APIs

Critical strengths
  1. The proposed framework is straightforward. The empirical results look strong.
  2. The ablation study provides some empirical evidence about streaming conversion vs. offline initialization, speaker label position, etc., which is informative.
  3. The authors promised to release the model weights. This could be a nice contribution to the open-source community.
Critical weaknesses
  1. The technical novelty is a bit limited. The authors follow the established LLM-based streaming ASR formulation with interleaved audio and text chunks.
  2. The proposed system can support up to 8-min sessions. It is unclear whether this also applies to the compared baselines.
  3. Although in section 3.1, the authors mentioned that hotwords can be prepended as optional context, no experiments are conducted regarding this setting.
Questions for authors

N/A

Reviewer_vSdVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the author proposed a streaming speaker-attributed ASR model that performs both transcription and speaker attribution in real-time. The author combines the VibeVoice and Qwen2.5 to produce 7.5 latent frames per second for real-time speech recognition and speaker attribution. The author conducts experiments and validates recognition-only streaming ASR results and streaming speaker-attributed ASR results on the four meeting benchmarks and MLCChallenge.

Critical strengths

The author proposed a strong system that shows effective performance in real-time ASR and speaker attribution. The model architecture is simple, but the overall performance is reasonable, and the author conducts a wide range of experiments to validate the performance of the model in different datasets and languages.

Critical weaknesses

I think there are several weaknesses I would like to point out:

  1. Since the author proposed speaker attribution, I am wondering why the author did not compare the system with some cascaded systems, like having an offline speaker diarization model and then an ASR model. That might be giving a strong baseline to compare against the current streaming system.

  2. One limitation nowadays for these pre-trained speech models is their pre-trained knowledge, and I think the author should consider acknowledging the possibility of the test data overlapping between the pre-trained dataset.

  3. There is not strong exploration of the model performance in different speaker setups or different noisy setups, it is challenging to estimate how robust the system will be in such conditions.

Questions for authors
  1. Since the author proposed speaker attribution, I am wondering why the author did not compare the system with some cascaded systems, like having an offline speaker diarization model and then an ASR model. That might be giving a strong baseline to compare against the current streaming system. Would the author be showing more baselines in this kind instead of commercial baselines?

  2. One limitation nowadays for these pre-trained speech models is their pre-trained knowledge, and I think the author should consider acknowledging the possibility of the test data overlapping between the pre-trained dataset. Would the author be providing some insights?

  3. There is not strong exploration of the model performance in different speaker setups or different noisy setups, it is challenging to estimate how robust the system will be in such conditions. Would the author be providing some details on the system performance in different recording and speaker presences, particularly overlapped speech, conditions?

Reviewer_NvfWOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents VibeVoice-ASR-Streaming, an end-to-end LLM-based framework for streaming speaker-attributed automatic speech recognition. The model processes audio in fixed-duration chunks with a small lookahead and interleaves acoustic representations with generated speaker labels and transcriptions in a shared autoregressive context. This allows the model to maintain speaker identities across chunks while jointly performing transcription and speaker attribution, without requiring a separate diarization or speaker-tracking module. The system builds on pretrained VibeVoice acoustic and semantic encoders and a Qwen2.5 LLM backbone. Training follows three stages: non-streaming training, streaming pre-training, and streaming fine-tuning. The authors provide 1.5B and 7B model variants and evaluate them across multilingual meeting transcription, speaker-attributed recognition, and conventional single-speaker ASR.

The results are good, with the 7B model achieving the lowest average recognition error among the compared streaming systems and the best or tied-best speaker-attributed error in 12 of 13 evaluation settings. The paper further provides extensive ablations examining chunk size, lookahead, model scale, and speaker-label placement, as well as comparisons with the original non-streaming model.

Critical strengths

The paper presents one of the first open-source, end-to-end LLM-based systems for streaming speaker-attributed ASR, addressing an important practical problem that requires both low-latency transcription and consistent speaker attribution. The proposed system achieves competitive or superior performance against several deployed streaming ASR and speaker-attribution services across multilingual meeting benchmarks. The release of model weights and inference code further increases the practical value of the contribution and its potential utility to the research community.

The paper provides empirical evaluation across multiple datasets, languages, and recognition settings. The ablations investigate the effects of chunk size, lookahead, model scale, and speaker-label placement, while the comparison with the non-streaming model quantifies the performance degradation introduced by streaming constraints. These experiments show how the model performs under different settings, although the findings are largely expected.

Critical weaknesses

The proposed approach largely combines established techniques, including pretrained acoustic and semantic encoders, an LLM backbone, chunk-wise speech–text interleaving, and autoregressive speaker-label generation. In particular, the interleaving formulation closely follows previous streaming Speech-LLM approaches, with speaker attribution incorporated into the generated text. While the resulting system is practically valuable and performs well, the paper introduces little fundamentally new methodology or modeling insight. Consequently, the contribution is predominantly an engineering and empirical achievement, and the paper reads more like a comprehensive technical report than a methodological research contribution.

Despite the comprehensive evaluation, important details of the training pipeline are missing. The paper does not specify the total size or composition of the Stage 1 training corpus. Although Stage 2 reportedly uses approximately 420,000 hours of speech, the constituent datasets and their mixing proportions are not disclosed. Similarly, the exact dataset mixture used for Stage 3 is unclear. Furthermore, while the authors describe generating approximately 4,520 hours of synthetic multi-speaker speech using domain-specific vocabulary, overlap augmentation, and room impulse responses, they do not provide sufficient details about the synthetic data generation procedure, including the speech synthesis models, construction of conversations, speaker selection, or generation parameters. These are distinct but equally important reproducibility concerns. Given that the primary contribution is empirical rather than methodological, these omissions are particularly consequential: they limit reproducibility and make it difficult to assess the relative contributions of the model design, training scale, and data composition to the reported performance.

Questions for authors

1. Could the authors provide the total number of training hours and the constituent datasets for Stage 1, along with the dataset composition and mixing proportions for Stages 2 and 3? It would also be helpful to distinguish publicly available and proprietary data sources.

2. Could the authors describe the synthetic multi-speaker data generation pipeline in greater detail? Specifically, which speech synthesis models were used, how were the meeting-style conversations and speaker turns constructed, and how were speaker overlap, acoustic conditions, and domain-specific vocabulary controlled? Additionally, could the authors quantify the contribution of synthetic training data through an ablation with and without it, particularly for recognition and speaker-attribution performance? Are the synthetic recordings or generation scripts planned for release?

3. Could the authors evaluate how speaker-attribution accuracy changes when a speaker returns after a long gap or several other speakers have spoken? This would help show whether the model can maintain speaker identities over long intervals.

4. Could the authors report the measured end-to-end speaker-attribution latency of the proposed system using the same wall-clock definition applied to the cloud-based baselines? Currently, the reported 2.00-second expected algorithmic latency is compared against measured wall-clock latencies, making the comparison less direct.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

Dear AC and Reviewers,

Thank you for your constructive comments and valuable suggestions. We are pleased that our responses have addressed most of the reviewers' concerns. The additional experiments and clarifications provided in our responses have been incorporated into the revised manuscript and are summarized below:

  1. [Updated] Appendix A — Training data. We provide the data scales, source categories, and sampling proportions for all three training stages, distinguish public and proprietary data, and clarify the separation between training and evaluation data.

  2. [Updated] Appendix B — Latency and cascaded baselines. In B.1, we report latency estimates based on wall-clock measurements of the 7B 22-frame model using the streaming demo. The average estimate is 2.48 s, including WebSocket network transmission and runtime overhead. In B.3, we add two cascaded SA-ASR baselines combining Nemotron 3 Diarization with Qwen3-ASR-1.7B and Whisper-large-v3-turbo, respectively, and compare both WER/CER and cpWER/cpCER using the same input audio and scoring protocol.

  3. [Added] Appendix C — Synthetic data generation. We detail dialogue construction, hotword prompt generation, VibeVoice-based speech synthesis, speaker selection, and acoustic augmentation. We also provide representative transcription examples.

  4. [Added] Appendix D — Robustness analysis. We add analyses grouped by the longest speaker-return gap within each session, overlap ratio, and number of speakers, together with a comparison of the recording conditions in AMI-IHM and AMI-SDM.

  5. [Added] Appendix E — Hotword evaluation. We evaluate the released 7B model with and without hotword context on the 24 Chinese and English vertical-domain subsets of GigaSpeechBench. Hotword context reduces Chinese B-CER from 28.82% to 9.23% and English B-WER from 16.02% to 7.76%, with improvements across all 24 subsets. We provide the vocabulary construction procedure and evaluation settings.

  6. [Updated] Main text — Sections 4 and 5. We add pointers to these supplementary experiments and analyses. We also add the corresponding references.

We sincerely appreciate the reviewers' feedback, which has helped us improve the clarity and empirical evaluation of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.