acceptodds
Under review as a conference paper at ICLR 2027

Search-ASR: Training Speech Agents to Search for Entity-Aware Speech Recognition

Abstract

Large language models have advanced automatic speech recognition (ASR), improving multilingual and dialectal recognition and contextual understanding. However, despite accurately transcribing most of an utterance, ASR systems still struggle with emerging and long-tail entities owing to static parametric knowledge and limited coverage of prebuilt hotword retrieval resources. This paper introduces Search-Augmented Agentic ASR (Search-ASR), which trains speech agents to autonomously query Web search engines using acoustic cues and utterance context, retrieve candidate names and semantic information, and combine this evidence with the audio to produce transcriptions. We further develop a multi-stage post-training framework and a multi-agent data synthesis system that establish search capabilities through supervised finetuning and reinforcement learning while reducing redundant searches and unnecessary rewriting. Experiments on English and Chinese entity-focused ASR benchmarks show that Search-ASR outperforms all evaluated open-source baselines and remains competitive with evaluated closed-source systems. Compared with its initialization model, Qwen3-Omni, Search-ASR reduces English word error rate and Chinese character error rate by 53.1% and 25.5%, while improving entity recall in both languages.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Automatic Speech Recognition, Large Audio Language Models, Agentic Search, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zBGXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Search-ASR, a method designed to address accuracy limitations of conventional speech recognition models when transcribing entities such as new product names and technical terms. Built upon Qwen3-Omni, the approach leverages supervised fine-tuning and reinforcement learning to enable the model to autonomously decide whether to perform web searches. Retrieved information is then fused with the original audio to produce transcriptions. Meanwhile, a multi-agent framework automatically generates training data to further boost the model’s entity recognition performance. Experimental results show that compared with vanilla Qwen3-Omni, the method reduces the word error rate from 4.58% to 2.15% on English speech recognition tasks and from 4.91% to 3.66% on Chinese. The current limitations are that the retrieval results carry the risk of missing correct entities, and the retrieval process increases inference latency.

Critical strengths
  1. The paper introduces web search into ASR, enabling the model to acquire information about newly emerging entities. Compared with methods that rely on pre-built hotword lists, this approach is expected to reduce the cost of manually maintaining and updating entity lexicons.

  2. The paper presents a complete multi-stage SFT and RL training scheme and incorporates multi-agent data synthesis, effectively improving the model's entity recognition and search decision-making capabilities.

Critical weaknesses
  1. Inference latency matters a lot for ASR systems, especially for real-time and streaming speech recognition. Adding web search here will inevitably create extra overhead. The experiments only report how often the model searches and how many tokens it generates. There’s little analysis on actual runtime, so it’s hard to tell how well this approach would work in practice.

  2. The entity recognition test set they built is mostly limited to AI-related content. They haven’t tested it on other specialist areas. We still don’t know if it can generalize well to fields like medicine or law.

Questions for authors
  1. This work mainly tests its approach on an ASR benchmark with AI entities. Would it be possible for the authors to run extra entity recognition experiments on other domains, for example medicine and law, to check how well Search-ASR generalizes across different fields?
  2. Figure 4 shows search frequency and average generated token counts over different training stages, but these do not map directly to actual inference time. I wonder if the authors could add latency comparisons like average end-to-end runtime to Figure 4, and break down the time cost between model inference and web search, to give a fuller picture of the method’s efficiency.
  3. VibeVoice-ASR and Qwen3-ASR, two of the open-source models compared here, can already boost entity recognition using hotwords or contextual prompts. I’m curious whether the authors turned on these features when evaluating them. If they were disabled, it would be helpful to include hotword-RAG ASR baselines, so we can more fairly see how Search-ASR stacks up against existing entity-enhancement approaches.
3 comments
AuthorsOpenReview ↗

Response to Reviewer zBGX

We thank the reviewer for the constructive suggestions and for recognizing the value of on-demand Web search and our multi-stage training framework.

1. Generalization to medicine and law

We added Chinese medical and legal entity evaluations using the paper's normalization, entity-recall, and Judge scoring procedures. The medical set contains 460 Web recordings; the legal set contains 100 samples synthesized with Qwen-Audio-TTS 3.0 and augmented with noise.

DomainModelCER ↓Entity Recall ↑Judge ↑
MedicalQwen3-Omni, direct ASR2.02%86.96%0.62
MedicalSearch-ASR1.97%90.87%0.66
LegalQwen3-Omni, direct ASR0.89%81.00%0.79
LegalSearch-ASR1.07%89.00%0.84

Entity recall and Judge scores improve in both domains. Legal recall increases from 81% to 89%, and the Judge score from 0.79 to 0.84. These results provide evidence of entity-recognition gains beyond AI.

2. Measured latency and its breakdown

We evaluated the full English and Chinese entity sets (498 and 395 samples), plus 500 randomly selected samples from each of GigaSpeech TEST and WenetSpeech TEST_MEETING. These subsets were fixed across models, giving 1,893 utterances in total. All local models run on one NVIDIA H100 using vLLM, TP=1, and CUDA Graph optimization. Measurements exclude loading, compilation, and warm-up, but retain failed samples.

Search-ASR runs with thinking enabled. Times are averaged over all samples; samples without Web search have zero search time. Model inference time excludes Web search. End-to-end latency also includes orchestration overhead. Added latency is measured relative to direct ASR with Qwen3-Omni.

Test setSamplesQwen3-Omni direct ASR (s)Search-ASR inference (s)Search-ASR Web search (s)Search-ASR end-to-end (s)Added latency (s)Search-ASR Web search rate
English entities4980.240.930.731.67+1.4344.98%
Chinese entities3950.220.950.911.88+1.6658.99%
GigaSpeech TEST5000.210.470.080.55+0.344.20%
WenetSpeech TEST_MEETING5000.170.370.010.40+0.220.60%

Overhead is task-dependent. On English/Chinese entities, mean end-to-end latency is 1.67/1.88 s, adding 1.43/1.66 s; Web search accounts for 43.60%/48.36% of total latency. On GigaSpeech/TEST_MEETING, search rates fall to 4.20%/0.60%, mean search time to 0.08/0.01 s, and additional end-to-end time to 0.34/0.22 s. Infrequent search limits retrieval overhead on general ASR, but model inference remains slower than direct ASR.

Future work will explore shorter latent CoT [1] and Shanks-like interleaving of listening, reasoning, and tool use [2] to reduce reasoning and search latency.

3. Hotword settings and retrieval baselines

The original open-source baselines used direct ASR without hotwords or contextual retrieval. We added two-pass experiments for VibeVoice-ASR and Qwen3-ASR on the original 498 English and 395 Chinese entity samples. Each model's initial transcript is used to retrieve candidates. These are supplied through its context interface for a second recognition pass over the original audio. All models use the same RAG tool and configuration. Nested 1k/10k lexicons contain all target entities, with other entries fixed random samples from our internal lexicon.

EN/ZH denote English/Chinese.

ModelSettingLexiconEN WER (%) ↓EN Recall (%) ↑EN Judge ↑ZH CER (%) ↓ZH Recall (%) ↑ZH Judge ↑
VibeVoice-ASRDirect-3.5269.080.5417.1434.180.15
VibeVoice-ASRTwo-pass RAG1k2.7577.710.6719.1243.800.21
VibeVoice-ASRTwo-pass RAG10k2.7477.710.6717.4743.540.22
Qwen3-ASR-1.7BDirect-3.1274.100.605.5240.760.33
Qwen3-ASR-1.7BTwo-pass RAG1k2.7678.110.664.8154.180.44
Qwen3-ASR-1.7BTwo-pass RAG10k2.7578.110.664.6854.940.44
Search-ASRDirect-3.0970.480.624.3652.910.45
Search-ASRAgent RAG1k2.4577.910.674.0162.030.53
Search-ASRAgent RAG10k2.4577.910.674.0062.030.53
Search-ASRAgent RAG+Web10k2.2080.320.703.7465.570.57

Retrieval improves entity recall for both baselines. With 10k entries, Qwen3-ASR reaches 78.11%/54.94% English/Chinese recall. Search-ASR with RAG alone achieves 77.91%/62.03%, with similar English recall and higher Chinese recall than Qwen3-ASR. Adding Web search further improves Search-ASR's entity recall, outperforming Search-ASR with either hotword RAG or Web search alone. Thus, both the baselines and Search-ASR benefit from hotword retrieval, while Search-ASR can further combine RAG and Web evidence.

[1] Hao et al. "Training large language models to reason in a continuous latent space." arXiv:2412.06769, 2024.

[2] Chiang et al. "Shanks: Simultaneous hearing and thinking for spoken language models." ACL, Vol. 1: Long Papers, 2026.

The authors have addressed most of my concerns.

Dear Reviewer zBGX,

Thank you for letting us know that our response addressed most of your concerns. We appreciate your feedback. If you have any further questions, please feel free to raise them at any time.

Reviewer_mV3WOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposed Search-ASR, a ASR system that can directly determine whether and when to search, mainly to improve the entity recognition. They uses simulated data to establish search capability into the base model, with multi-staged training conducted. The results show that the error rate is reduced, and the search capability is learned and become more efficient after RL post-training.

Critical strengths

The paper is easy to follow

The training for search capability seems to be useful (from Table 2)

Critical weaknesses

Concern regarding the baselines

Although the authors have mentioned some retrieval based methods, they are not included in the baseline. Although the focuses might be different, it's still better to know the performance differences between a retreival based method an the proposed one.

The generalizability is not tested

The proposed framework is evaluated on only a single large, LLM-based model. It remains unclear whether the search capability can be effectively extended to smaller models, which may lack the capacity to learn such behavior.

Concern regarding time efficiency

Given the model's large size and the additional latency introduced by search, the overall framework is already non-streamable. In this case, how much does search latency really matter? More importantly, given the strong performance of cascaded approaches, what motivates building and training a search-augmented ASR system instead?

Concern regarding the selected evaluation datasets

Search-ASR is evaluated on some easy and clean dataset. Can you further evaluate on more difficult and realistic corpora to demonstrate that the robustness is not damaged throughout the SFT and RL training?

Questions for authors

Please directly see and address the above weaknesses and questions.

2 comments
AuthorsOpenReview ↗

Response to Reviewer mV3W

We thank the reviewer for the constructive suggestions and for recognizing the paper's clarity and the effectiveness of search capability training.

1. Retrieval baselines

We added two-pass baselines on the English and Chinese entity-focused ASR benchmarks. Each model's initial transcript is used to retrieve candidates from a shared lexicon. Candidates are supplied through the model's context interface for a second pass over the original audio. We evaluate this second output. All models use the same RAG tool and settings. Nested 1k/10k lexicons contain all target entities, with other entries fixed random samples from our internal lexicon.

EN/ZH denote English/Chinese.

ModelSettingLexiconEN WER (%) ↓EN Recall (%) ↑EN Judge ↑ZH CER (%) ↓ZH Recall (%) ↑ZH Judge ↑
VibeVoice-ASRDirect-3.5269.080.5417.1434.180.15
VibeVoice-ASRTwo-pass RAG1k2.7577.710.6719.1243.800.21
VibeVoice-ASRTwo-pass RAG10k2.7477.710.6717.4743.540.22
Qwen3-ASR-1.7BDirect-3.1274.100.605.5240.760.33
Qwen3-ASR-1.7BTwo-pass RAG1k2.7678.110.664.8154.180.44
Qwen3-ASR-1.7BTwo-pass RAG10k2.7578.110.664.6854.940.44
Search-ASRAgent RAG1k2.4577.910.674.0162.030.53
Search-ASRAgent RAG10k2.4577.910.674.0062.030.53
Search-ASRAgent RAG+Web1k2.2080.320.703.6965.820.56
Search-ASRAgent RAG+Web10k2.2080.320.703.7465.570.57

Retrieval improves both baselines. With 10k entries, Search-ASR with RAG alone has similar English recall to Qwen3-ASR and higher Chinese recall. Adding Web improves Search-ASR's recall in both languages, exceeding both baselines and Search-ASR with either RAG or Web alone.

2. Smaller models

Starting from Qwen2.5-Omni-3B, we trained for one epoch using all SFT data from both stages and evaluated on the same English and Chinese test sets.

Qwen2.5-Omni-3B settingEN WER (%) ↓EN Recall (%) ↑EN Judge ↑ZH CER (%) ↓ZH Recall (%) ↑ZH Judge ↑
Original, direct ASR4.3068.880.489.5336.710.22
SFT, Agent+Web3.0874.300.618.0850.630.31

Error rates decrease in both languages; recall improves by 5.42 points in English and 13.92 points in Chinese. This suggests that a 3B speech model can learn search-augmented transcription.

3. Cascades and latency

We tested Qwen3-30B-A3B and Qwen3.5-35B-A3B cascades as alternatives to dedicated post-training using text models of comparable size. Under the original protocol, they receive only initial transcripts and can access web search.

Each cascade combines Qwen3-Omni, Web search, and the text correction model named below. Search-ASR also uses Web search.

SystemEN WER (%) ↓EN Recall (%) ↑EN Judge ↑ZH CER (%) ↓ZH Recall (%) ↑ZH Judge ↑
Qwen3-30B-A3B3.6275.100.665.1657.720.47
Qwen3.5-35B-A3B8.1179.920.7022.9154.680.43
Qwen3.7-Max2.8379.520.746.1166.580.56
Search-ASR2.1580.120.703.6664.810.55

Search-ASR has lower error rates and higher recall than both the 30B and 35B cascades. Without task-specific post-training, the 35B model sometimes repeats its reasoning until reaching the 8,192-token limit. The Qwen3.7-Max cascade has higher Chinese recall and Judge scores in both languages, but higher error rates.

Mean end-to-end latency, output token counts, and truncation rates:

SystemEN latency (s)ZH latency (s)EN tokensZH tokensEN truncation (%)ZH truncation (%)
Qwen3-Omni, direct0.240.2232.4229.050.000.00
Qwen3-30B-A3B cascade6.356.391033.621026.330.000.25
Qwen3.5-35B-A3B cascade4.038.15456.711326.682.4112.91
Qwen3.7-Max API cascade37.1934.66--0.200.51
Search-ASR1.671.88168.48172.780.000.00

Search-ASR is faster than all cascades. Long outputs add overhead to the 30B cascade; the 35B cascade also has a pronounced latency tail. These results support a favorable accuracy-latency trade-off in this setting. Users must wait for the full transcript even in non-streaming ASR. Future work will explore shorter latent CoT [1] and Shanks-like interleaving of listening, reasoning, and tool use [2] to reduce inference latency.

4. Harder speech benchmarks

Test setQwen3-OmniSearch-ASRWeb search rate
GigaSpeech TEST (WER ↓)11.70%10.51%3.55%
WenetSpeech TEST_MEETING (CER ↓)5.89%6.41%0.30%

Search-ASR outperforms its base model on GigaSpeech but performs worse on TEST_MEETING. These modest changes suggest that its ability to generalize to general ASR tasks is largely preserved after post-training.

[1] Hao et al. "Training large language models to reason in a continuous latent space." arXiv:2412.06769, 2024.

[2] Chiang et al. "Shanks: Simultaneous hearing and thinking for spoken language models." ACL, Vol. 1: Long Papers, 2026.

Dear Reviewer mV3W,

Thank you again for your suggestions. We have run additional experiments in response and hope the results address your concerns. If so, would you consider a higher rating in light of the new evidence? We would be happy to discuss any remaining questions.

Reviewer_hLueOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Search-ASR, which trains a speech model to use Web search to resolve uncertain entity names. It combines four post-training steps with a pipeline that generates text-based correction tasks from four model failures. Experiments show consistent improvements in English and Chinese entity recognition.

Critical strengths
  1. The paper addresses a practical problem: recognizing emerging and uncommon entity names without updating a fixed vocab.
  2. The experimental improvements are substantial in both languages.
  3. The ablation and failure analysis are useful.
Critical weaknesses
  1. The entity-focused evaluation is limited in scale and diversity.
  2. The synthetic data ablation shows that additional tasks help, but does not establish the benefit of the proposed generation and difficulty-calibration process over a simpler approach with the same data budget.
Questions for authors
  1. Do the entitiy-recognition gains hold on naturally recorded English speech and entities outside tech-related domains?
  2. How does the proposed synthesis pipeline compare with simpler task generation using the same number of training examples? How much does difficulty calibration contribute?
2 comments
AuthorsOpenReview ↗

Response to Reviewer hLue

We thank the reviewer for the constructive suggestions and for recognizing the practical value of our work, the bilingual entity-recognition gains, and the ablation and failure analyses.

1. Evaluation diversity and generalization

We added evaluations on naturally recorded English speech and Chinese medical and legal speech, using the paper's normalization, entity recall, and Judge scoring procedures. The medical set contains 460 Web recordings; the legal set contains 100 samples synthesized with Qwen-Audio-TTS 3.0 and augmented with noise. For English, we sampled 200 texts from the paper's English entity test set. Two speakers recorded these texts, with 100 recordings per speaker.

Naturally recorded English

ModelWER ↓Entity Recall ↑Judge ↑
Qwen3-Omni6.10%51.00%0.37
Qwen3-ASR-1.7B5.81%57.00%0.36
VibeVoice-ASR12.18%54.50%0.28
Search-ASR3.83%72.00%0.53

Search-ASR achieves lower WER than all three baselines. Relative to Qwen3-Omni, entity recall improves by 21 percentage points to 72.00%, and the Judge score increases from 0.37 to 0.53. Thus, the gains extend beyond TTS to naturally recorded speech.

Medical and legal entities

DomainModelCER ↓Entity Recall ↑Judge ↑
MedicalQwen3-Omni2.02%86.96%0.62
MedicalSearch-ASR1.97%90.87%0.66
LegalQwen3-Omni0.89%81.00%0.79
LegalSearch-ASR1.07%89.00%0.84

Both entity recall and Judge score improve on the full set of 460 medical recordings. On the legal diagnostic set, recall increases from 81% to 89%, and the Judge score rises from 0.79 to 0.84. These results provide evidence of entity-recognition gains beyond technology-related domains.

2. Difficulty calibration under an equal data budget

We added an equal-data ablation comparing the full pipeline with Search-ASR w/o difficulty calibration. This variant retains capability-gap-driven task generation and task-validity checks but removes difficulty-based filtering and adjustment. The two variants use equal numbers of synthetic examples at each RL stage. SFT data and all other training and evaluation settings are unchanged.

EN/ZH denote English/Chinese.

ModelEN WER (%) ↓EN Recall (%) ↑EN Judge ↑ZH CER (%) ↓ZH Recall (%) ↑ZH Judge ↑
Search-ASR w/o difficulty calibration2.2179.120.673.7163.040.51
Search-ASR2.1580.120.703.6664.810.55

Calibration reduces English WER from 2.21% to 2.15% and Chinese CER from 3.71% to 3.66%. Mean Judge scores increase from 0.67 to 0.70 and from 0.51 to 0.55, respectively, while entity recall improves in both languages. These consistent gains support the incremental contribution of difficulty calibration when the amount of training data is held constant.

Dear Reviewer hLue,

Thank you again for your suggestions. We have included additional experiments in our response. Do the results address your concerns? We would be happy to discuss any further questions.

Comments on the paper

2 comments
Area_Chair_sxU6OpenReview ↗

Reviewers, please respond to the authors while the discussion is still active

Dear reviewers,

The authors have responded to your reviews. Please check their responses and reply to the authors while the discussion is still open.

Thanks,

AC

AuthorsOpenReview ↗

Summary of Rebuttal and Additional Results

We thank all reviewers for their constructive feedback. Our rebuttal adds the following experiments and analyses to address the main concerns.

  • Generalization: Additional experiments support Search-ASR's generalization to human recordings, other specialized domains, and smaller models. We added evaluations on 200 human-recorded English utterances, 460 Chinese medical recordings from the Web, and 100 synthetic legal utterances with added noise. Search-ASR improves entity recall over Qwen3-Omni on all three test sets. Search training also improves entity recognition in Qwen2.5-Omni-3B, showing that this capability can be developed in smaller speech agents. [Reviewers: hLue, mV3W, zBGX]

  • Training contribution: Difficulty calibration provides additional gains with the same amount of training data. We used equal numbers of synthetic training examples at each of the two RL stages, while keeping the SFT data and all other training and evaluation settings unchanged. Difficulty calibration improves error rates, entity recall, and Judge scores in both languages, supporting its incremental contribution independently of increases in data volume. [Reviewer: hLue]

  • Baseline comparisons: Additional comparisons support Search-ASR's advantages in entity recognition and the accuracy–latency trade-off. We added hotword-RAG baselines using the same retrieval tools and lexicons, along with search-enabled cascaded systems. Search-ASR and the other ASR baselines perform similarly when using only hotword lexicons and RAG retrieval, while adding Web search further improves Search-ASR's entity recall. Search-ASR outperforms cascades using comparably sized 30B or 35B models in both accuracy and latency, while achieving search-based ASR correction performance comparable to the larger Qwen3.7-Max cascade. [Reviewers: mV3W, zBGX]

  • Practical efficiency: Runtime measurements clarify Search-ASR's response latency and additional overhead. We measured end-to-end latency, model inference time, and Web search time across 1,893 utterances. Search-ASR's mean latency on the English and Chinese entity sets is lower than that of all evaluated cascades. We also explicitly report its additional overhead relative to direct ASR and identify further inference latency reduction as a main direction for future optimization. [Reviewers: mV3W, zBGX]

Reviewer zBGX stated that our response addressed most of their concerns. We believe these additional results further support the value of learning effective search behavior in speech agents for entity-aware speech recognition, and we welcome further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.