- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes FocusMe, a speech enhancement framework that combines air-conduction and bone-conduction signals to recover target speech under environmental noise and competing speech. The method introduces two components: (1) Cross-Attention Feature Fusion (CAFF), which adaptively combines AC and BC spectral features, and (2) a Frame-Level Speaker Embedding (FSE) extractor, which derives speaker-discriminative representations from synchronized BC speech without requiring a separate enrollment utterance. The model is jointly trained using speech reconstruction and speaker classification objectives.
Critical strengths
-
The paper addresses an important problem: speech enhancement in challenging environments containing both environmental noise and competing speakers. Using synchronized BC signals for target speech extraction without separate speaker enrollment is appealing. The proposed framework incorporates BC information for both acoustic reconstruction and speaker conditioning. Explicitly supervising speaker-discriminative representations alongside reconstruction to improve target speech preservation.
-
FocusMe achieves the best average PESQ, STOI, ESTOI, DNSMOS, and CER among the compared methods. Improvements are also observed under severe environmental noise and competing speech.
Critical weaknesses
-
The FSE extractor is intended to provide speaker conditioning, but the embeddings are derived from synchronized BC speech, which contains both speaker characteristics and phonetic information aligned with the target AC signal. So, it is unclear whether the improvements arise primarily from better speaker identification or from additional synchronized speech information. Although removing the speaker classification objective degrades performance, this does not fully explain the learned representations.
-
The experiments introduce environmental noise and competing speech into AC recordings while retaining the corresponding BC signals without additional corruption. However, BC sensors in wearable devices can experience motion artifacts, contact variations, and mechanical vibrations from activities such as walking or head movement. Since FocusMe relies on BC signals for both acoustic fusion and speaker conditioning, such distortions could affect its performance. Although the authors acknowledge this limitation, the current evaluation does not establish how robust the proposed method is under realistic wearable conditions.
-
The evaluation uses synthetically constructed AC mixtures from synchronized ABCS and ESMB recordings. Although the speaker-disjoint splits are good, the experiments do not evaluate recordings captured in genuinely noisy, reverberant environments with simultaneous AC and BC interference. Furthermore, differences in BC sensor placement and contact characteristics across wearable devices could affect the learned representations. These limitations question the method's application to real-world settings.
Questions for authors
-
Can the authors provide additional insight into whether the FSE representations primarily encode speaker identity or synchronized phonetic information? For example, comparing embeddings from synchronized BC speech with embeddings extracted from a different utterance by the same speaker could help distinguish these contributions.
-
How sensitive is FocusMe to BC signal degradation, such as motion artifacts, contact variations, or temporary loss of sensor contact? Even a limited sensitivity analysis using synthetic BC perturbations would help assess the reliability of the proposed dual-use BC conditioning mechanism.
-
It would be valuable to evaluate FocusMe on real-world noisy AC–BC recordings and across different BC sensor configurations to establish whether the reported improvements generalize beyond the controlled setups.
Q1: Can the authors provide additional insight into whether the FSE representations primarily encode speaker identity or synchronized phonetic information?
A1: We thank the reviewers for pointing this out. We would like to clarify that Section 5.4 analyzes frame-level FSE embeddings from three speakers excluded from training, with 100 utterances per speaker. PCA projections show that embeddings from the same speaker tend to form compact clusters, and K-means assignments largely align with speaker labels. These findings provide qualitative evidence that FSE retains speaker-discriminative information across utterances. The ablation results further support FSE’s contribution to FocusMe’s enhancement performance.
Q2: How sensitive is FocusMe to BC signal degradation, such as motion artifacts, contact variations, or temporary loss of sensor contact?
A2: Following the reviewer’s suggestion, we recorded speech under six conditions to assess FocusMe’s sensitivity to practical variations in headset use: (a) normal wearing, (b) head movement, (c) walking, (d) touching the headset by hand, (e) walking while only the interfering speaker was speaking, and (f) loose wearing. We have added this study to the appendix of the revised manuscript. The table below reports the results for conditions (a)–(d) (Table 9, Appendix E.3). ΔWER denotes the change relative to condition (a), measured in percentage points (pp).
The results indicate that FocusMe tolerates the tested movement and contact perturbations. The qualitative example for condition (e) also illustrates effective suppression of interfering speech while the wearer is walking (Figure 10, Appendix E.3). However, a loose headset fit remains a failure case: FocusMe and all evaluated baselines perform poorly under this condition (Figure 11, Appendix E.3). Improving robustness to variations in headset contact is therefore an important direction for future work.
Q3: It would be valuable to evaluate FocusMe on real-world noisy AC–BC recordings and across different BC sensor configurations to establish whether the reported improvements generalize beyond the controlled setups.
A3: To address this suggestion, we recorded paired AC–BC signals directly from a headset in a subway and evaluated FocusMe and the baseline methods on the same recorded segment. We have added this study to the appendix of the revised manuscript. Appendix E.1 describes the recording setup, and Appendix E.2 presents the output spectrograms. For this recording, the spectrograms qualitatively suggest that FocusMe suppresses noise more effectively than the baselines while preserving more speech detail. Although EBEN [1], a BC-only system, reconstructs more mid- and high-frequency components, informal listening suggested distortions in speaker timbre and possible changes in the perceived speech content. Due to equipment constraints, we have not yet evaluated FocusMe’s generalization across different BC sensor configurations. This evaluation will be an important focus of our future work.
Reference
[1] Julien Hauret, Thomas Joubaud, Veronique Zimpfer, and ´´Eric Bavu. EBEN: Extreme bandwidth extension network applied to speech signals captured with noise-resilient body-conduction microphones. In IEEE International Conference on Acoustics, Speech and Signal Processing, pp.1–5, 2023.