- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes Search-ASR, a method designed to address accuracy limitations of conventional speech recognition models when transcribing entities such as new product names and technical terms. Built upon Qwen3-Omni, the approach leverages supervised fine-tuning and reinforcement learning to enable the model to autonomously decide whether to perform web searches. Retrieved information is then fused with the original audio to produce transcriptions. Meanwhile, a multi-agent framework automatically generates training data to further boost the model’s entity recognition performance. Experimental results show that compared with vanilla Qwen3-Omni, the method reduces the word error rate from 4.58% to 2.15% on English speech recognition tasks and from 4.91% to 3.66% on Chinese. The current limitations are that the retrieval results carry the risk of missing correct entities, and the retrieval process increases inference latency.
Critical strengths
-
The paper introduces web search into ASR, enabling the model to acquire information about newly emerging entities. Compared with methods that rely on pre-built hotword lists, this approach is expected to reduce the cost of manually maintaining and updating entity lexicons.
-
The paper presents a complete multi-stage SFT and RL training scheme and incorporates multi-agent data synthesis, effectively improving the model's entity recognition and search decision-making capabilities.
Critical weaknesses
-
Inference latency matters a lot for ASR systems, especially for real-time and streaming speech recognition. Adding web search here will inevitably create extra overhead. The experiments only report how often the model searches and how many tokens it generates. There’s little analysis on actual runtime, so it’s hard to tell how well this approach would work in practice.
-
The entity recognition test set they built is mostly limited to AI-related content. They haven’t tested it on other specialist areas. We still don’t know if it can generalize well to fields like medicine or law.
Questions for authors
- This work mainly tests its approach on an ASR benchmark with AI entities. Would it be possible for the authors to run extra entity recognition experiments on other domains, for example medicine and law, to check how well Search-ASR generalizes across different fields?
- Figure 4 shows search frequency and average generated token counts over different training stages, but these do not map directly to actual inference time. I wonder if the authors could add latency comparisons like average end-to-end runtime to Figure 4, and break down the time cost between model inference and web search, to give a fuller picture of the method’s efficiency.
- VibeVoice-ASR and Qwen3-ASR, two of the open-source models compared here, can already boost entity recognition using hotwords or contextual prompts. I’m curious whether the authors turned on these features when evaluating them. If they were disabled, it would be helpful to include hotword-RAG ASR baselines, so we can more fairly see how Search-ASR stacks up against existing entity-enhancement approaches.
3 comments
The authors have addressed most of my concerns.
Dear Reviewer zBGX,
Thank you for letting us know that our response addressed most of your concerns. We appreciate your feedback. If you have any further questions, please feel free to raise them at any time.
Response to Reviewer zBGX
We thank the reviewer for the constructive suggestions and for recognizing the value of on-demand Web search and our multi-stage training framework.
1. Generalization to medicine and law
We added Chinese medical and legal entity evaluations using the paper's normalization, entity-recall, and Judge scoring procedures. The medical set contains 460 Web recordings; the legal set contains 100 samples synthesized with Qwen-Audio-TTS 3.0 and augmented with noise.
Entity recall and Judge scores improve in both domains. Legal recall increases from 81% to 89%, and the Judge score from 0.79 to 0.84. These results provide evidence of entity-recognition gains beyond AI.
2. Measured latency and its breakdown
We evaluated the full English and Chinese entity sets (498 and 395 samples), plus 500 randomly selected samples from each of GigaSpeech TEST and WenetSpeech TEST_MEETING. These subsets were fixed across models, giving 1,893 utterances in total. All local models run on one NVIDIA H100 using vLLM, TP=1, and CUDA Graph optimization. Measurements exclude loading, compilation, and warm-up, but retain failed samples.
Search-ASR runs with thinking enabled. Times are averaged over all samples; samples without Web search have zero search time. Model inference time excludes Web search. End-to-end latency also includes orchestration overhead. Added latency is measured relative to direct ASR with Qwen3-Omni.
Overhead is task-dependent. On English/Chinese entities, mean end-to-end latency is 1.67/1.88 s, adding 1.43/1.66 s; Web search accounts for 43.60%/48.36% of total latency. On GigaSpeech/TEST_MEETING, search rates fall to 4.20%/0.60%, mean search time to 0.08/0.01 s, and additional end-to-end time to 0.34/0.22 s. Infrequent search limits retrieval overhead on general ASR, but model inference remains slower than direct ASR.
Future work will explore shorter latent CoT [1] and Shanks-like interleaving of listening, reasoning, and tool use [2] to reduce reasoning and search latency.
3. Hotword settings and retrieval baselines
The original open-source baselines used direct ASR without hotwords or contextual retrieval. We added two-pass experiments for VibeVoice-ASR and Qwen3-ASR on the original 498 English and 395 Chinese entity samples. Each model's initial transcript is used to retrieve candidates. These are supplied through its context interface for a second recognition pass over the original audio. All models use the same RAG tool and configuration. Nested 1k/10k lexicons contain all target entities, with other entries fixed random samples from our internal lexicon.
EN/ZH denote English/Chinese.
Retrieval improves entity recall for both baselines. With 10k entries, Qwen3-ASR reaches 78.11%/54.94% English/Chinese recall. Search-ASR with RAG alone achieves 77.91%/62.03%, with similar English recall and higher Chinese recall than Qwen3-ASR. Adding Web search further improves Search-ASR's entity recall, outperforming Search-ASR with either hotword RAG or Web search alone. Thus, both the baselines and Search-ASR benefit from hotword retrieval, while Search-ASR can further combine RAG and Web evidence.
[1] Hao et al. "Training large language models to reason in a continuous latent space." arXiv:2412.06769, 2024.
[2] Chiang et al. "Shanks: Simultaneous hearing and thinking for spoken language models." ACL, Vol. 1: Long Papers, 2026.