- Rating
- 3
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces VibeVoice-ASR-Streaming, an end-to-end LLM-based streaming speaker-attributed automatic speech recognition (ASR) model. The model processes incoming audio in fixed-duration chunks (with a lookahead chunk) and interleaves these latent audio representations with previously generated speaker labels and transcripts inside a single autoregressive LLM context. The model is built on Qwen2.5 1.5B and 7B backbones with frozen VibeVoice acoustic/semantic encoders. The model is trained via a three-stage curriculum: offline pre-training, streaming pre-training on 420k hours of speech aligned via Qwen3-ForcedAligner-0.6B, and streaming fine-tuning on public datasets plus 4.5k hours of synthetic multi-speaker meeting data. Across multiple benchmarks, the 7B model achieves state-of-the-art streaming speaker-attributed performance (cpWER/cpCER) with an algorithmic latency of ~2.00 s, outperforming commercial cloud services and matching or beating streaming recognition-only APIs
Critical strengths
- The proposed framework is straightforward. The empirical results look strong.
- The ablation study provides some empirical evidence about streaming conversion vs. offline initialization, speaker label position, etc., which is informative.
- The authors promised to release the model weights. This could be a nice contribution to the open-source community.
Critical weaknesses
- The technical novelty is a bit limited. The authors follow the established LLM-based streaming ASR formulation with interleaved audio and text chunks.
- The proposed system can support up to 8-min sessions. It is unclear whether this also applies to the compared baselines.
- Although in section 3.1, the authors mentioned that hotwords can be prepended as optional context, no experiments are conducted regarding this setting.
Questions for authors
N/A
Global Response
Dear AC and Reviewers,
Thank you for your constructive comments and valuable suggestions. We are pleased that our responses have addressed most of the reviewers' concerns. The additional experiments and clarifications provided in our responses have been incorporated into the revised manuscript and are summarized below:
[Updated] Appendix A — Training data. We provide the data scales, source categories, and sampling proportions for all three training stages, distinguish public and proprietary data, and clarify the separation between training and evaluation data.
[Updated] Appendix B — Latency and cascaded baselines. In B.1, we report latency estimates based on wall-clock measurements of the 7B 22-frame model using the streaming demo. The average estimate is 2.48 s, including WebSocket network transmission and runtime overhead. In B.3, we add two cascaded SA-ASR baselines combining Nemotron 3 Diarization with Qwen3-ASR-1.7B and Whisper-large-v3-turbo, respectively, and compare both WER/CER and cpWER/cpCER using the same input audio and scoring protocol.
[Added] Appendix C — Synthetic data generation. We detail dialogue construction, hotword prompt generation, VibeVoice-based speech synthesis, speaker selection, and acoustic augmentation. We also provide representative transcription examples.
[Added] Appendix D — Robustness analysis. We add analyses grouped by the longest speaker-return gap within each session, overlap ratio, and number of speakers, together with a comparison of the recording conditions in AMI-IHM and AMI-SDM.
[Added] Appendix E — Hotword evaluation. We evaluate the released 7B model with and without hotword context on the 24 Chinese and English vertical-domain subsets of GigaSpeechBench. Hotword context reduces Chinese B-CER from 28.82% to 9.23% and English B-WER from 16.02% to 7.76%, with improvements across all 24 subsets. We provide the vocabulary construction procedure and evaluation settings.
[Updated] Main text — Sections 4 and 5. We add pointers to these supplementary experiments and analyses. We also add the corresponding references.
We sincerely appreciate the reviewers' feedback, which has helped us improve the clarity and empirical evaluation of our work.