acceptodds
Withdrawn from ICLR 2027

Unified Target-Speaker ASR with Text and Enrollment Speech Cues

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Target-speaker automatic speech recognition (TS-ASR) aims to recognize speech from a designated speaker while suppressing interfering speakers in multi-talker environments. Conventional TS-ASR systems typically rely on an enrollment utterance to specify the target speaker, requiring additional speech from the same speaker at inference time. Text-guided approaches provide an alternative by exploiting known lexical content, such as a wake word, to locate the target speaker directly from the observed speech. However, these two forms of target guidance are typically studied independently, despite providing complementary information about the target speaker. In this work, we propose a Unified Dual-Cue TS-ASR framework that accommodates text cues, enrollment speech, or their combination within a single model. The text cue interacts with the mixture representation to extract target-speaker information conditioned on known lexical content, while an independent enrollment utterance provides complementary speaker information from a different utterance of the same speaker. Cross-attention cue-conditioning modules are embedded inside the shared Conformer blocks to condition ASR on the designated speaker and suppress interfering speech. The shared conditioning interface supports inference with text cues, enrollment speech, or their combination. During dual-cue training, both modalities are supplied and negative-cue sampling provides cue-validity supervision. Experiments on 30,000 two-speaker evaluation mixtures cover five recording and domain conditions and four oracle text-cue lengths. With five-character text cues, the proposed concatenated dual-cue method achieves an overall character error rate (CER) of 8.80%, compared with 17.32% and 29.06% under text-only and enrollment-speech-only inference, respectively. It also outperforms parallel dual-cue fusion, which obtains 9.49% CER, and yields lower dual-cue CER across all five evaluation subsets. These results demonstrate the advantage of jointly exploiting complementary lexical and speaker information for target-speaker ASR.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
target-speaker ASR, text cue, enrollment speech, unified speech recognition, multi-speaker speech

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_5AhGOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a target-speaker ASR framework that takes text cues, enrollment speech, or both through one shared conditioning interface. The contribution is architectural: a single model that supports different ways of identifying the target speaker, trained to distinguish valid cues from invalid ones.

Critical strengths
  • The paper addresses a real limitation of target-speaker recognition. Applications differ in what information is available to identify the desired speaker, and a common interface for lexical and acoustic guidance could make separate systems for each setting unnecessary. The individual mechanisms are established, but combining them this way is a useful contribution.

  • Processing enrollment speech without a separate speaker encoder also removes a model dependency. More broadly, the paper treats target specification as a flexible part of recognition, which gives a common framework for studying how different cues interact.

Critical weaknesses
  • SE-DiCoW already incorporates enrollment-sequence attention into ASR [1], LLM-TSE combines textual and enrollment guidance for speech extraction [2], and CALM integrates speaker conditioning with lexical biasing for recognition [3]. Their formulations differ from this submission, however, the authors should state what it adds in joint target specification and validity supervision, separately from the broader ideas of sequential enrollment conditioning and acoustic linguistic integration, which are already established.

  • Section 2.3 motivates negative-cue training as a way for the model to reject incorrect information and fall back on a valid alternative, yet the evaluation never systematically tests recognition when cues are incorrect or contradictory. Training on negative examples does not by itself show that the model behaves reliably in those conditions. Since resolving uncertain target information is a central reason for combining the modalities, this gap weakens the paper's main argument.

  • The paper acknowledges this tradeoff, but training always supplies both modalities, so it is unclear whether the degradation comes from the shared architecture or from the mismatch between training and inference. For applications where cue availability varies, that distinction decides whether one unified model can replace specialist systems.

References

[1] Alexander Polok, Dominik Klement, Samuele Cornell, Matthew Wiesner, Jan  ernock˝, Sanjeev Khudanpur, and Luk·a Burget. "SE-DiCoW: Self-Enrolled Diarization-Conditioned Whisper." ICASSP, pp. 16712 16716, 2026. (https://arxiv.org/abs/2601.19194).

[2] Xiang Hao, Jibin Wu, Jianwei Yu, Chenglin Xu, and Kay Chen Tan. "Typing to Listen at the Cocktail Party: Text-Guided Target Speaker Extraction." IEEE Transactions on Cognitive and Developmental Systems, 18(2):361 372, 2026; preprint first published 2023. (https://arxiv.org/abs/2310.07284).

[3] Muhammad Shakeel, Yosuke Fukumoto, Chikara Maeda, Chyi-Jiunn Lin, and Shinji Watanabe. "CALM: Joint Contextual Acoustic-Linguistic Modeling for Personalization of Multi-Speaker ASR." ICASSP, 2026. DOI: 10.1109/ICASSP55912.2026.11463102. (https://arxiv.org/abs/2601.22792).

Questions for authors

Q1. How does recognition behave when one cue is incorrect, or when the cues point to different speakers?

Q2. Does training with missing modalities fix the single-cue degradation while keeping the benefit of joint conditioning?

Q3. Beyond differences in inputs and task formulation, what is the specific methodological advance over the closest prior architectures?

Reviewer_URTaOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper does target-speaker ASR on two-speaker mixtures. A single model can be told who the target is in three ways: a short text cue which consisits 2–5 characters taken from the target's transcript, an enrollment utterance, or both. Cues go into each Conformer block through cross-attention, either Concat (share the same attention) or Parallel (not share).
  • The model is trained with both cues always present. When a cue doesn't match, it outputs a reject label instead of a transcript.
  • All data comes from AISHELL-1/2. With both cues, CER drops to 8.80, versus about 15.5 for the single-cue expert models.
  • The authors say one checkpoint covers all three modes.
Critical strengths
  • The problem is real. Sometimes a wake word is given, sometimes an enrollment is given, sometimes both.
  • Every cue setting is evaluated on the same mixtures, so the comparisons inside the paper are fair.
  • The dual-cue gain is large. Negative-cue ablations are clear.
  • Code, models and data are released.
Critical weaknesses
  • I'm not fully convinced by the novalty of the paper, the individual components and the multi-cue idea itself have prior work (e.g., anchored/keyword-guided ASR), so the contribution rests on the unified model actually working in all modes, and it doesn't. Cross-attention has been widely used for TS-ASR, also, feeding the enrollment in as a sequence rather than an embedding already exists in target speech extraction.
  • The "unified" term doesn't really work. This is my main concern. Authors claimed that the proposed approach is a unified solution for TS-ASR (either target's transcript, or an enrollment utterance, or both), however, it's not the case. When one cue is missing, performance dropped a lot. So in practice you'd still need the expert models. Modality dropout is the obvious fix, and it's left to future work.
  • There're no external baselines from other works. Target-speaker ASR on two-speaker mixtures has been widely investigated in the past decade, however, the experiments are only performed within the setting of two speakers, no noise and Mandarin only.
Questions for authors
  • Have you tried modality dropout? Does it bring single-cue performance back up to the experts?
  • Also, the concerns in my Critical Weaknesses need to be addressed.
Reviewer_MbVZOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a target-speaker ASR model conditioned on a text cue, enrollment speech, or both. Text and sequential enrollment features condition a Conformer through cross-attention, using either concatenated cue sequences or separate attention stages. Training supplies both modalities and uses negative-cue sampling with structured CTC targets. Evaluation on 30,000 simulated Mandarin mixtures shows substantial improvements when both cues are available. The paper also evaluates cue length, recording-device variation, and transcription outside the supplied text span.

Critical strengths
  1. The benefit of combining lexical and speaker information is substantial and intuitive.

  2. The evaluation includes useful controlled comparisons. Cue-length conditions reuse the same mixtures, and the AISHELL-2 device conditions use matched source content and mixing conditions.

Critical weaknesses

My main concern is that the methodological novelty of the unified formulation is not sufficiently established. Lexical cues and enrollment speech can naturally be treated as conditioning inputs, and concatenating their representations for cross-attention appears to be a fairly standard design. The results demonstrate that the two cues are complementary, but do not clearly isolate the benefit of the proposed architecture over straightforward dual-cue conditioning in LLM based ASR model. A compact model is a practical advantage, although another plausible route would be to prompt a large audio-language model with the same cues and distill its behavior into a small, cue-conditioned recognizer. This alternative is untested, but the paper does not explain what distinctive advantage its formulation offers beyond combining the available information.

Questions for authors
  1. What is the main technical contribution beyond conditioning an ASR model on both lexical and speaker information? Could you compare against a straightforward text-plus-speaker-embedding baseline using the same backbone and training data?

  2. How would your approach compare with prompting a strong audio-language model with both cues and distilling its predictions into a compact, cue-conditioned student? I am not assuming that this alternative would perform better, but would like to understand whether the proposed method offers a specific advantage in accuracy, inference cost, training requirements, or handling missing cues.

Reviewer_ab7NOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a Unified Dual-Cue Target-Speaker ASR framework that enables a single model to flexibly support text-only, enrollment-speech-only, and combined dual-cue inference modes. By incorporating negative-cue sampling during training, the model also acquires the ability to assess cue validity, rejecting mismatched cues while relying on valid target information. Evaluated on the newly constructed Unified-TS-ASR-Eval benchmark, the proposed concatenated dual-cue approach achieves an overall CER of 8.80%, significantly outperforming text-only inference (17.32%), enrollment-only inference (29.06%), and the parallel fusion variant (9.49%).

Critical strengths

Effective and Parameter-Efficient Unified Architecture: The proposed framework cleanly unifies lexical (text cue) and acoustic (sequential enrollment speech) target specifications into a shared cross-attention conditioning interface within Conformer blocks. By adopting an embedding-free enrollment representation with low-frame-rate (LFR) compression and a shared acoustic frontend, the model eliminates the need for an external pretrained speaker encoder (e.g., CAM++) while achieving competitive single-cue performance with fewer parameters. Furthermore, the concatenated dual-cue design (Concat) achieves a substantial reduction in overall CER without introducing any additional parameter overhead compared to the text-only system.

Thorough Analysis of Component Mechanisms and Operational Boundaries: Through comprehensive ablations, the paper excels at dissecting architectural mechanisms and operational boundaries: (1) characterizing the trade-off between unified (Concat) and modality-specific (Parallel) attention projections; (2) verifying via Outside-cue CER and attention visualizations that lexical cues actively enhance subsequent speaker tracking rather than merely copying tokens; and (3) demonstrating across five device subsets how lexical cues effectively compensate for acoustic cue degradation under recording-device mismatch.

Controlled Benchmark and Reproducibility: The construction and release of the 30,000-mixture Unified-TS-ASR-Eval benchmark—along with open-sourced code, checkpoints, and deterministic outside-cue scoring scripts—provide a clean, well-controlled testbed for evaluating multi-modal cue interactions across matched content and varying recording devices.

Critical weaknesses

The empirical evaluation relies entirely on internal ablations within the authors' own 9-layer Conformer-CTC codebase. While matching the acoustic backbone ensures a controlled architectural comparison, it leaves significant gaps in positioning this work against the broader literature:

Absence of Published SOTA Single-Cue Baselines: Although Section 4 cites recent keyword-guided TS-ASR methods, embedding-free target extraction approaches, and Whisper-based TS-ASR models, none of these external systems are included in the experimental comparison. For enrollment conditioning, the authors only compare their approach against a basic internal FiLM-based baseline. Because experiments are conducted on the newly proposed Unified-TS-ASR-Eval without benchmarking against established open-source TS-ASR or cascaded TSE+ASR systems, the true effectiveness of the proposed model remains unverified. Including strong, published baselines is essential to anchor the evaluation and contextualize the actual performance gains of the proposed architecture.

Lack of Naive Dual-Cue Baselines: For dual-cue evaluation (Text + Enroll), the proposed Concat (U6) and Parallel (U3) models are only compared against each other and against single-cue inputs. Since utilizing two complementary cues naturally provides more information than a single cue, a critical missing baseline is a naive combination of the two specialist models—such as posterior/logit interpolation between the best enrollment-only model (E1) and text-only model (T3) during CTC decoding, or a cascaded pipeline. Without this, it is difficult to quantify the true gain of in-block sequence concatenation over simple post-hoc or cascaded integration of the two modalities.

Questions for authors
  1. To anchor the performance on the newly constructed Unified-TS-ASR-Eval benchmark, could the authors provide evaluation results for at least one representative published external baseline—such as an open-source cascaded TSE+ASR pipeline, a recent keyword-guided TS-ASR method, or a Whisper-based TS-ASR model?
  2. To isolate the benefit of in-block cross-attention fusion from the natural advantage of simply having access to two cues, how do Concat (U6) and Parallel (U3) compare against a naive post-hoc combination of the existing single-cue specialist models—for instance, frame-level CTC logit/posterior interpolation between the already trained E1 (or E3) and T3 models during decoding?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.