acceptodds
Under review as a conference paper at ICLR 2027

FINE-GRAINED DEVICE-DIRECTED SPEECH RECOGNITION: FROM OFFLINE MODELING TO STREAMING INFERENCE

Abstract

A smart speaker in a shared space must select device requests from mixed conversation, adapt to changing functions, and preserve information beyond the transcript as interaction unfolds. We introduce fine-grained device-directed speech recognition (DDSR), which jointly transcribes semantic units and identifies their intents. An offline realization establishes this recognition and selection interface, with contextual verification for unresolved intents. Because selection depends on the device's supported functions and users' expressions, dialogue-based intent management updates the candidate inventory and registers difficult expressions without changing model weights. To retain information that transcription alone omits, the interface associates each semantic unit with speaker, emotion, and acoustic-event information. We then extend the interface to continuous input through fine-grained streaming device-directed speech recognition (FSDR). Intent-delimited unit completion enables incremental selection, while parallel attribute heads provide rich outputs without inserting additional attribute tokens into autoregressive transcription. On English and Chinese evaluations, offline DDSR achieves recording-level equal error rates of 1.25% and 5.46%, compared with 2.74% and 6.26% for a reproduced SELMA DDSD baseline; FSDR achieves 4.60% and 9.82%. Further evaluations assess dynamic intent management, acoustic-event classification, emotion recognition, and speaker attribution, connecting fine-grained content selection with configurable, informative, and incremental outputs for voice assistants.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Device-directed speech recognition, Streaming speech recognition, Spoken language understanding, Speech-language models, Dynamic intent management

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Q43vOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper tackles the problem of a smart device assistant in a room which must adapt to real user dialogues, where it might have to distinguish requests adressed to itself versus part of user conversations.

Critical strengths
  • The proposed method is evaluated on three axes: New intent recognition, Management parsing and long-tail expresisons
Critical weaknesses
  • I found the paper poorly written: lots of jargon, little context, paragraph with little transition and no clear conclusions/insights etc. Overall the writing felt very disconnected and hard to extract signal from

  • The benchmarks and evaluations are not clear. It is not clear whether the benchmarks are well establsished or newly proposed in the paper. The fact that there is also a single baseline makes it hard to place the paper in the literature.

Questions for authors
  • What was the goal of reproducing the baseline Selma ? And if it was reproduced, why not reproduce it with the same backbone as the proposed model ? (Voxtral vs Qwen)
Reviewer_d6FWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed a new device-directed speech recognition, that addresses a series problems of contextual selection.

Critical strengths

1. The research question proposed, and the figure 1 as an example, helped audience to catch the research problem clearly.

Critical weaknesses

1. The abstract can be better written. After reading a few sentences in the first paragraph of the introduction section, I can understand the problem proposed by the author, but the abstract is not as clear.

2. The sequence that DDSR is predicting, it is better to give a little bit more details (Section 3.1) on how each is measured and what metrics are used, so audience can keep it in mind then go through your experiments.

3. From the Methodology section:

How is the intent inventory originally constructed?

What exactly is the speaker label?

What is the candidate label (tokenized by Qwen2)?

Why the word 'Rich' is used? Because it contains more than one label (speaker, emotion, transcript, etc)?

For what datasets (including its split, if applicable) have been used, or combined for training, testing, and what (and why) synthetic speech was used for augmentation, can the authors describe them all together more clearly? From reading the experiments part, a few datasets, with original different purposes (ASR, emotion recognition, etc) are used and even reconstructed, so I hope they all can be better expressed to make the pipeline and overall presentation better.

4. If more space is allowed, I hope the authors can elaborate on the metrics used a bit more.

Questions for authors

See weaknesses listed.

Reviewer_ZWYeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents the well motivated problem of detecting intents meant for a device in possibly a stream of speech. There is a need to determine whether commands found in the speech are meant for the device, or possibly to another person. The author's present this as Device-Directed Speech Recognition (DDSR) as a follow-on to previous work called Device-Directed Speech Detection (DDSD). The author's also present a fine-grained streaming device-directed speech recognition framework (FSDR). Training is done on Librispeech en, AISHELL-1 and SELMA training data. Evals are done on synthetic data it appears and MELD and some eval set that are not well described "New intents" "Management" "Registered Expressions".

Critical strengths

The paper is addressing an important area of human-device interaction where disambiguating speech directed at a device and speech directed at other people/devices is important. They build on reasonable systems including SELMA and Qwen Audio.

Critical weaknesses

The paper is very difficult to read and thus understand what the evaluation tasks are. How is the DDSR system trained beyond what DDSD does? How does FDSD now become streaming? It does not seem possible at all to reproduce what is being presented.

Questions for authors

How is the DDSR system trained beyond what DDSD does?

How does FDSD build on this?

Are all the results on synthetic data? How would the approaches do on real data? How would QWEN-Audio do with fine-tuning on the task?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.