acceptodds
Under review as a conference paper at ICLR 2027

CE-TSE: Cross-Emotion and Physical Acoustics for Robust Target Speaker Extraction

Abstract

Extracting the target speaker's voice from noisy mixtures environment is a core challenge for intelligent auditory systems. Target Speaker Extraction (TSE) aims to mimic the human "Cocktail Party Effect". When humans extract target speech in noisy environments, they rely on not only speaker identity, but also spatial acoustic cues and emotional information. However, existing TSE benchmark datasets lack both realistic acoustic conditions such as room reverberation and spatial geometry, and complete emotional annotations. Therefore, we manually annotate emotional labels for over 2,300 utterances from the REAL‑TSE dataset, consuming approximately 58 person‑hours of annotation effort. Then we incorporate real room impulse responses and noise to perform physical spatial simulation to build the Cross-Emotion Target Speaker Extraction dataset (CE-TSE Dataset). Building upon this dataset, we propose the Cross-Emotion Target Speaker Extraction Network (CETSE-Net). CETSE-Net is equipped with an emotion disentanglement module (CE), which decouples emotional features from speaker identity representations. Experimental results show that CETSE-Net achieves consistent improvements across all evaluation metrics: +1.844 dB on SI-SNR, +1.736 dB on SDR and +0.42 on PESQ.

Primary area:
datasets and benchmarks
Keywords:
Cross-Emotion Target Speaker Extraction Dataset, Emotion Disentanglement, Physical Acoustic Augmentation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_KDe3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that target speaker extraction (TSE) suffers when the enrollment and target utterances differ in emotion. It manually labels emotions for 2,332 enrollment utterances of REAL-TSE and builds the CE-TSE dataset by mixing emotional speech corpora and REAL-TSE with room impulse responses and noise. It proposes an emotion-disentanglement module with shared encoder, speaker and emotion latents, reconstruction, classification and adversarial losses, while only the speaker latent is used at inference.

Critical strengths

Targets a real and under-studied factor and contributes manual emotion annotations for a real-recording TSE corpus, and Tte module is backbone-agnostic and yields consistent gains in the reported setting.

Critical weaknesses

The dataset and experiments are under-specified since no number or hours of mixtures, splits, speaker disjointness, or fraction and construction of cross-emotion pairs. There is no breakdown of matched vs mismatched emotion, so the motivating problem and the benefit of the module are never directly demonstrated. The evaluation is even only on the authors' own synthetic data. Moreover, the gain is not attributed to emotion disentanglement because L_spk and L_emo share one weight, with no speaker-only multitask control, and some claims are overstated.

Questions for authors

Which is the training/evaluation target, dry or reverberant? How are references selected, and how many annotators labeled the data? Why is CETSE-Net the same performance as USEF-TSE + CE? why the SepFormer is missing in tab 2

Reviewer_wapCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on the target speaker extraction problem, and highlights the fact that existing datasets do not provide both emotion labels and realistic acoustic simulation. The paper makes two contributions:

  1. CE-TSE dataset - a dataset that has labeled emotions and realistic acoustics with a target speaker, interferer speakers and noise
  2. CETSE-Net: a model with a dedicated emotion-identity disentanglement module, which separates emotional information from the enrollment speech.

The proposed CETSE-Net achieves benefits in terms of SI-SNR, SDR and PESQ compared to state of hte art.

Critical strengths

The idea of trying to disentangle emotion and keep only identity to assist with target speaker extraction is sensible

The evaluation shows tangible benefits in terms of the three metrics applied

The dataset created is valuable for the research community as it combines realistic acoustic conditions with emotion information.

Critical weaknesses
  • Has not evaluated against more recent papers such as i) USEF-TSE (Universal Speaker Embedding-Free TSE, 2024); ii) EvoTSE (Evolving Enrollment for TSE, 2026): iii) MTSE (Multi-Target Speaker Extraction, 2025):; iv) Target Speaker Extraction through Comparing Noisy Positive and Negative Audio Enrollments (2025):

  • Would this approach of disentangling emotion work with noisy audio enrollments? Or can it be adapted in the future to work with noisy audio enrollments?

  • Does it generalise with out of distribution emotions not used in the training phase?

Questions for authors
  • It may be difficult to get samples of a speaker with different emotions at the training stage. How is the technique degraded as one uses less emotion labels, or fewer types of emotions in the training dataset than in the testing. - Does it generalise with out of distribution emotions not used in the training phase?

  • what if in the training dataset we have a subset of speakers with emotion labels and a subset without? Or hybrid datasets some with emotion ground truth and some without? How would training work in this case?

  • Would this approach of disentangling emotion work with noisy audio enrollments? Or can it be adapted in the future to work with noisy audio enrollments?

  • There are cultural/background/and other differences in describing emotions - What happens when the distinction is not easy to make?

Reviewer_byNpOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates target speaker extraction (TSE) under conditions where the reference and target speech exhibit emotion mismatches. The authors annotate emotion labels for 2,332 clips derived from the REAL-TSE Dataset, apply room impulse responses (RIRs) and noise simulation to construct a synthetic dataset (CE-TSE), and propose CETSE-Net, featuring an emotion disentanglement (CE) module.

Critical strengths
  1. Addressing emotion mismatch between enrollment and target speech targets a relevant and under-explored robustness issue in real-world conversational audio.
  2. Manual emotional annotation of real-scene audio (58 person-hours) can be a contribution to the community.
Critical weaknesses
  1. A critical flaw in the evaluation framework lies in the handling of the dataset. After checking the REAL-TSE paper, I found that Real-TSE is a genuine real-world recorded dataset that inherently lacks ground truth (GT) clean signals, precluding the calculation of signal-level metrics such as SI-SNR in evaluation. To circumvent this, the authors resort to full physical simulation and generate simulated mixture data, it conflates synthetic simulation performance with true in-the-wild robustness, failing to evaluate the system on native, uncorrupted real recordings via standard robust metrics (such as Word Error Rate or Speaker Similarity).
  2. Important details regarding the dataset pipeline are omitted:
  • Final split sizes, speaker-disjointness guarantees, component corpus distributions, and exact emotion-pair sampling strategies remain unspecified.
  • Inter-annotator agreement metrics for the manual emotion labels are entirely missing, making it difficult to assess the quality and reliability of the newly introduced annotations.
  1. Applying an identical RIR to both target and interfering sources fails to simulate distinct spatial positions for multiple speakers.
  2. Table 1 incorrectly claims that existing benchmarks lack spatial simulation, overlooking datasets like WHAMR! which implement room simulation and source-specific RIR convolutions
  3. Table 5: Sepformer or Spex+?
Questions for authors
  1. Given that the original REAL-TSE Dataset is a real-world recording collection devoid of clean ground-truth references (relying instead on WER-based evaluations), please clarify how signal-level metrics like SI-SNR were computed. If these metrics were evaluated exclusively on a fully synthetic simulation subset, please correctly describe the data pipeline to avoid misrepresenting real-world evaluation.
  2. Are both source signals processed through the exact same RIR filter? If so, how does this accurately model distinct source geometries in a physical acoustic space?

And refer to the weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.