acceptodds
Under review as a conference paper at ICLR 2027

BAT: Lightweight Causal Bone-Conduction-Guided Speech Separation with Cross-Modal Conditioning and Adaptive Mask Bounds

Abstract

We study the recovery of wearer and interlocutor speech from a noisy air-conduction (AC) mixture and a synchronized bone-conduction (BC) reference available only from the wearer. BC provides a speaker-specific cue, but its high-frequency attenuation and locally varying reliability complicate its use for separation. We introduce BAT, a compact frame-causal framework that adapts BC conditioning throughout feature fusion and spectral reconstruction. Its Cross-Band Harmonic Reliability Fusion (CHRF) module aggregates cross-band BC context and conditions AC features according to local cross-sensor reliability. The Confidence-Adaptive Mask (CAM) decoder combines separator features with cross-sensor agreement and BC activity to adapt the allowable complex-mask ranges. We also introduce LS-Bone, English AC–BC pairs obtained by controlled re-recording of LibriSpeech, and VibroMix, which combines these pairs with Chinese ABCS speech and environmental noise. On VibroMix-2Mix, BAT achieves 17.3 dB mean SI-SNRi, exceeding the two-output Robust Fusion adaptation by 2.6 dB. Mean SDRi and PESQ improve by 2.5 dB and 0.2, respectively. Under the common causal profiling configuration, BAT uses approximately 46.2% fewer parameters and 34.6% fewer multiply-accumulate operations than the original single-target Robust Fusion model. Code and audio demos are available at https://anonymous.4open.science/w/BAT-Demo-3603/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
speech separation, bone conduction, multimodal learning, causal modeling, cross-modal conditioning, adaptive mask bounds, lightweight neural networks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iYA5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BAT, a speech separation architecture that separates a noisy AC-mixture speech signal between the wearer and an interlocutor using a synchronized bone-conduction (BC) reference from the wearer. The AC and BC input are encoded by a shared audio encoder. Then, the cross-band fusion module uses BC features and local cross-sensor cues to condition the separator. The decoder adapts complex-mask bounds during reconstruction.

A new dataset named LS-Bone is also introduced. It consists of a set of paired English AC–BC recordings, and construct the two-speaker VibroMix benchmark using the LS-Bone, a Mandarin AC–BC speech corpus, and environmental noise from WHAM! and DNS. It also creates an evaluation set VibroMix-2Mix, on which BAT reports 17.3 dB mean SI-SNR improvement, 2.6 dB above a two-output adaptation of Robust Fusion. Ablation studies are performed, supporting the contributions of the proposed fusion and decoder components.

Critical strengths
  1. Strong empirical results on the VibroMix-2Mix. BAT improves mean SI-SNRi by 2.6 dB over the strongest reported two-output BC-guided baseline, with good recovery of both the guided and unguided speakers.

  2. Ablation studies are performed on the fusion operators and mask bounds. The results show the effectiveness of the proposed choices.

Critical weaknesses
  1. Missing information regarding the dataset construction. For example, the split of the underlying recordings is unspecified. The paper gives mixture-level train, development, and test counts but does not clearly state whether the original paired AC–BC recordings are disjoint across splits. Reusing a recording in different mixtures could make the test results unreliable.

  2. Some baseline comparisons need qualification. Blind separators receive no BC cue identifying the wearer, yet their outputs are scored in fixed guided–unguided order without permutation matching. Additionally, the implementation of Robust Fusion adaptation (the main baseline) was not described in enough detail.

Questions for authors
  1. Were the original AC–BC recording pairs disjoint across training, development, and test, before mixtures were generated? Were speakers disjoint?

  2. According to Appendix C.2, the evaluation assumes that the guided speaker comes first. How do the blind-separator results change with permutation-invariant matching?

  3. What architectural, loss, and training changes were made to adapt Robust Fusion for two-output?

3 comments
AuthorsOpenReview ↗

Response to Reviewer iYA5

We sincerely thank the reviewer for recognizing BAT's strong empirical results on VibroMix-2Mix, its recovery of both guided and unguided speech, and the ablation evidence supporting the proposed fusion and decoder components. We respond below.

[W1] [Q1] The recording-level dataset split is unspecified, raising concerns about recording reuse across splits. Were the original AC–BC recording pairs and speakers disjoint across training, development, and test before mixture generation?

Response. The training, validation, and test sets are fully speaker-disjoint. Each speaker’s original paired AC–BC recordings are confined to that speaker’s split, so the same recording pair is not reused across splits when constructing mixtures. We have clarified this in the “Dataset splits” paragraph of Appendix B.2, with the additions marked in blue.

[W2] [Q2] Blind separators are scored in a fixed guided–unguided order without BC wearer-identification cues or permutation matching. How do their results change under permutation-invariant matching?

Response. The blind baselines use permutation-invariant training (PIT), and the reported scores already use permutation-invariant output–reference matching during evaluation. Their values therefore remain unchanged. The fixed guided–unguided order specifies the clean references for source-wise statistics; predictions are matched to these references before aggregation. We have revised this passage in Appendix C.2 to explicitly state that permutation-invariant matching is used for metric computation, with the revisions marked in blue.

[W2] [Q3] The two-output adaptation of Robust Fusion is insufficiently described. What architectural, loss-function, and training changes were made?

Response. For the two-output comparison in Table 1, Robust Fusion was adapted by modifying only its output layer to estimate both wearer and interlocutor speech, matching BAT’s two-output setting. The remaining network architecture was retained. All models in this comparison used the same training framework and negative SNR loss, averaged over the two outputs. These adaptation and training details are documented in Appendix C.1, with the additions marked in blue.

I thank the authors for the rebuttal. The clarification regarding the speaker-disjoint and recording-disjoint dataset splits resolves my concern about potential data leakage. I am also satisfied with the explanation of the permutation-invariant evaluation protocol for blind separators and the implementation and training details of the two-output adaptation of Robust Fusion.

I appreciate that these clarifications have been incorporated into the revised manuscript, improving its clarity and reproducibility. My original concerns have been addressed, and I have no further questions.

Dear Reviewer,

Thank you very much for your careful review and for taking the time to assess our rebuttal. We also appreciate your recognition of our work’s contributions. We sincerely thank you for confirming that our revisions have addressed your concerns. Your constructive feedback has helped us improve the clarity and reproducibility of our paper.

Kind regards,

The Authors

Reviewer_aq5wOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work presents a method for separating user speech from other speech in the environment, such as on a smart glasses or earbud platform. The method leverages a bone-conduction microphone as a separate channel to aid in the separation process. The method extracts cross-frequency band features from the bone conduction channel, which it then uses to condition the over-the-air microphone features to enable more robust separation. A decoder is proposed to dynamically predict masks for speech separation. Evaluation across several baselines and datasets show promising results. The bone-conduction part of the dataset was generated by playing existing datasets through an artificial mouth and vibration platform.

Critical strengths
  1. The design of the method makes sense and is coherent. It makes sense to condition the over-the-air path with the bone-conduction path, given it can capture relatively clean voice. The follow on design to reconstruct speech signals based on these features is also fine.

  2. The performance of the method, along with computational efficiency seems promising. There are also ablations for most parts of the architecture for both task performance and different aspects of system efficiency (e.g., MACs, CPU, etc.), which is nice.

Critical weaknesses
  1. Novelty is incremental. This work seems to be mostly an amalgamation of existing techniques, as most of them have been employed in previous literature. For example, affine feature conditioning (e.g., FiLM), bone-conduction and over-the-air microphone feature fusion (e.g., BAF-Net), complex ratio masking and combining temporal + full-band frequency features with recurrent and attention-based architectures (e.g., TF-GridNet). The performance improvements seem to mostly stem from the fact that this work is much more heavily engineered and as more bells and whistles than other works. This seems to be evidenced by the fact that many components that are added only increase performance by a small amount (e.g., adding the CHRF module and CAM improves by 0.5 dB or less over the baselines). The efficiency improvements are also not consistent; while the proposed method has the lowest number of parameters, it's MACs are significantly higher than some of the baselines.

  2. The experimental rigor can be improved. There are several dimensions of the experiments that are weak. First, when evaluating against blind methods, it is unclear whether those methods are permutation invariant. Blind source separation methods typically require assigning separated sources to the correct speaker, and if this is not done well, then performance drops. However, the proposed method does not suffer from this problem since you have the correct permutation based on who is captured by the bone-conduction microphone. The second issue is that all of the bone-conduction evaluation is done through an artificial mouth; I would expect at least some experiments on real people. Finally, it is unclear through the experiments about the generalizability and robustness of the approach (e.g., unseen speakers, placement and/or capture quality of the bone-conduction microphone).

Questions for authors
  1. What is the core novelty of the approach?

  2. Please clarify the evaluation against blind protocols. If they weren't evaluated with the correct wearer designation, please justify why this is a fair comparison.

  3. How well does the approach generalize (e.g., on unseen users, microphone placements)

  4. How well does the approach work in realistic settings, beyond the artificial mouth.

4 comments
AuthorsOpenReview ↗

Response to Reviewer aq5w (1/2): Novelty and Efficiency

We sincerely thank the reviewer for recognizing the coherence of our method design, its promising separation performance and computational efficiency, and the ablation studies covering the main architectural components. We respond below.

[W1] [Q1] The approach appears to combine established techniques, including FiLM, AC–BC fusion (BAF-Net), complex ratio masking, and recurrent/attention-based spectrotemporal modeling. The small gains attributed to CHRF and CAM (0.5 dB or less) suggest that improvements may mainly come from additional engineering. What is the core novelty?

Response. Our motivation is to prevent mid- and high-frequency information loss when low-frequency BC cues guide broadband AC speech separation. CHRF combines cross-band BC context with reliability-gated conditioning and a direct AC path to preserve broadband AC features, outperforming FiLM under the same backbone (Table 3). CAM uses separator features and the agreement and BC-activity cues from CHRF to adapt complex-mask bounds, linking feature fusion to spectral reconstruction. BAF-Net [4] instead fuses independently enhanced AC and BC spectra. BAT achieves substantially better separation than the two-output BAF-Net adaptation (Table 1).

RTFS-Net [1] (ICLR 2024) and Dolphin [2] (ICLR 2026) are application-focused multimodal speech separation studies; RTFS-Net’s core contributions include Cross-dimensional Attention Fusion, while its backbone uses established recurrent units and time–frequency self-attention. Olsen et al. [3] (ICLR 2026) likewise build on existing separation components to introduce probabilistic early exits for audio-only speech separation. Similarly, BAT uses established recurrent and attention operations for time–frequency streaming, while its task-specific contribution lies in coupling reliability-aware fusion with adaptive reconstruction in a lightweight architecture.

CHRF and CAM are closely coupled because CAM uses CHRF’s cross-sensor agreement and BC-activity cues to guide adaptive mask reconstruction. Jointly replacing CHRF with cross-attention and CAM with a tanh mask, the strongest alternative fusion operator by SI-SNRi and the strongest standard mask baseline (Tables 3–4), reduces SI-SNRi from 17.3 to 16.1 dB and SDRi from 17.5 to 16.3 dB.

FusionMaskSI-SNRi (dB)SDRi (dB)
CHRF (ours)CAM (ours)17.317.5
Cross-attentiontanh16.116.3

The 1.2 dB drop in each metric exceeds the 0.9 dB sum of the individual drops. These results are reported in Appendix H, Table 8. We also evaluated BAT and the joint variant on real AC–BC recordings captured using a Timekettle W4 headset in an open-plan office. The wearer walked around while speaking with a fixed interlocutor, amid ambient office noise and background conversations among other office workers. The output spectrograms show better preservation of mid- and high-frequency speech structure by BAT than the joint variant (Appendix H, Figure 9). These additions are included in the revised manuscript and marked in blue.

[W1] Efficiency improvements are inconsistent across metrics: despite the low parameter count, MACs are significantly higher than those of some baselines.

Response. BAT’s MACs mainly come from GRU and self-attention operations over time–frequency sequences. GRUs reuse the same weights across sequence positions, so repeated computation increases MACs without adding learned parameters.

Using parameter count as a proxy for model capacity, BAT is smaller than most baselines: it uses 0.7M parameters, compared with 1.3M for Robust Fusion, 3.0M for FlowDense, and 15.8M for BAF-Net in their original single-target configurations (Table 2). With this compact parameterization, BAT achieves 17.3 dB SI-SNRi and 17.5 dB SDRi, ranking first across all five evaluation metrics in Table 1. MACs alone do not determine execution time. Under the same profiling protocol, BAT processes one second of audio in 9.2 ms on a GPU, the fastest among the compared methods in Table 2.

References

[1] Samuel Pegg et al. RTFS-Net: Recurrent Time-Frequency Modelling for Efficient Audio-Visual Speech Separation. ICLR, 2024.

[2] Kai Li et al. Efficient Audio-Visual Speech Separation with Discrete Lip Semantics and Multi-Scale Global-Local Attention. ICLR, 2026.

[3] Kenny Falkær Olsen et al. Knowing When to Quit: Probabilistic Early Exits for Speech Separation Networks. ICLR, 2026.

[4] Yunsik Kim and Yoonyoung Chung. Modality-Specific Speech Enhancement and Noise-Adaptive Fusion for Acoustic and Body-Conduction Microphone Framework. Interspeech, 2025, pp. 3833–3837.

AuthorsOpenReview ↗

Response to Reviewer aq5w (2/2): Evaluation and Generalization

[W2] [Q2] It is unclear whether blind baselines are evaluated with permutation-invariant matching and correct wearer assignment. Please clarify the evaluation protocol and justify the comparison's fairness if the wearer is not correctly identified.

Response. The blind baselines use permutation-invariant training (PIT) and permutation-invariant output–reference matching during evaluation. Predictions are matched to the clean wearer and interlocutor references, which are stored in a fixed guided–unguided order for source-wise statistics. Thus, the fixed reference order does not constrain predicted output order or penalize source permutations. We have revised this passage in Appendix C.2 to explicitly state that permutation-invariant matching is used for metric computation, with the revisions marked in blue.

[W2] [Q3] [Q4] How well does the approach generalize to unseen users and realistic settings beyond the artificial mouth, and remain robust to changes in BC microphone placement or capture quality?

Response. The training, validation, and test sets are speaker-disjoint, so all test speakers are unseen during training. This split is now explicit in the “Dataset splits” paragraph of Appendix B.2, with the additions marked in blue.

We additionally evaluated BAT and the comparison methods on real AC–BC recordings captured using a Timekettle W4 headset in an open-plan office. The wearer walked around while speaking with a fixed interlocutor, amid ambient office noise and background conversations among other office workers. Such movement can introduce variations in sensor contact position and capture quality during normal use. The separated audio and corresponding spectrograms provide qualitative evidence of generalization to these real recording conditions. In these examples, BAT provides cleaner separated speech and addresses the loss or attenuation of mid- and high-frequency components observed in the compared outputs. These comparisons are included in Appendix F, Figure 8, and Appendix H, Figure 9, with the additions marked in blue.

Thank you for the clarifications and additional experiments. I'm still confused about the permutation invariance because in the previous version it explicitly states that no permutation was done in C.2, but now everything does use permutation matching (even the tables that remained the same, like table 1)?

Ablations are nice, and the explanations somewhat clarify the approach, though the paradigm the approach takes is still very similar to others, and the gains are fairly low.

The office environment evaluation is interesting, but there are still issues like robustness across wearers, placements, etc.

Given these, I will retain my score. Even if the answer to the permutation matching question is answered, I'll still retain this score.

We understand your concern about whether PIT was used for the blind separation baselines, and we appreciate you raising this concern, which gives us an opportunity to explain the protocol more clearly.

In the original implementation, the blind separators were trained with PIT. During evaluation, the two possible output–reference assignments were compared, and the better assignment was selected before computing the reported metrics. Our earlier description was intended to explain the fixed guided–unguided order used to present and interpret the table results after matching. However, the wording “without permutation matching” could obscure this preceding matching step. The original results already incorporated permutation-invariant matching and therefore remain unchanged.

As discussed in our previous response, the ablation spectrograms show improved mid- and high-frequency speech reconstruction and noise suppression. We would appreciate your consideration of these qualitative improvements alongside the numerical results when assessing the gains.

Regarding robustness across wearers and placements, the Timekettle W4 bone-conduction earphones used in our recordings have a defined ear-worn configuration that constrains the BC sensor to a relatively consistent position around the ear. This structure helps limit differences in sensor placement across wearers and placement fluctuations during normal use.

We hope this explanation clarifies the potentially confusing wording in the previous version and the intended wearing configuration. We respectfully ask you to take these clarifications and the supporting evidence into account when reconsidering the contributions of our work.

Reviewer_2rk9OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The authors present BAT, an approach for more efficiently leveraging a bone-conducting (BC) microphone to isolate the wearer's speech from a noisy air microphone (AC). BAT slightly improves upon Robust Fusion (RF) and other existing approaches, while requiring less parameters (up to 46% less than RF) and flops (up to 34% less than RF), as depicted in Table 1 and 2. An ablation of fusion operators show that their approach (CHRF) slightly outperforms other approaches (Table 3), and similarly their mask parameterization narrowly improves over more standard options (Table 4).

In addition, two new BC-augmented datasets, LS-Bone (34 hours) and VibroMix-2 (96 hours), are constructed from LibriSpeech (English) and VibroMix (Chinese) to characterize the performance of their approach relative to existing methods.

Critical strengths
  • The parameter and flop reductions over their baseline Robust Fusion without degradation (up to 46/34% resp.) are solid.
  • The new datasets constitute significant contributions to research on BC methods, insofar as they reliably mimic real conditions.
Critical weaknesses
  • The paper is squarely a "systems paper" and appears to be low in novelty.
  • Their fusion and masking approaches only narrowly outperform standard methods, and the gains are likely not statistically significant.
  • Overall my current assessment is that while the paper makes contributions to engineering BC-augmented processing, it is of low novelty and significance in terms of research on learning representations, making it a poor match for ICLR.
Questions for authors
  • Can elaborate on the novel aspects of the work that pertain to representation learning? Thank you.
1 comment
AuthorsOpenReview ↗

Response to Reviewer 2rk9

We sincerely thank the reviewer for recognizing BAT's reductions in parameter count and computation relative to Robust Fusion without performance degradation, and for highlighting the potential research value of the new BC datasets. We respond below.

[W1] [W3] [Q1] The work appears primarily system-oriented, with limited novelty and significance for representation learning. What are its specific representation-learning contributions and their relevance to ICLR?

Response. Our submission is in the primary area “applications to computer vision, audio, language, and other modalities,” focusing on efficient multimodal speech separation rather than a general-purpose representation-learning framework. RTFS-Net [1] (ICLR 2024) and Dolphin [2] (ICLR 2026) are examples of application-focused multimodal speech separation. RTFS-Net’s core contributions include Cross-dimensional Attention Fusion, while its backbone uses established recurrent units and time–frequency self-attention. For audio-only speech separation, Olsen et al. [3] (ICLR 2026) build on existing separation components to introduce probabilistic early exits under variable computational budgets. BAT similarly develops a task-specific design: CHRF incorporates low-frequency BC cues through cross-band context and reliability-gated conditioning while preserving broadband AC features; CAM combines separator features with cross-sensor cues to adapt complex-mask bounds. Their coupling addresses mid- and high-frequency information loss in BC-guided separation within a lightweight streaming model that recovers both speakers using BC guidance from only the wearer. This task-specific contribution is consistent with ICLR’s applications area.

[W2] Their fusion and masking approaches only narrowly outperform standard methods, and the gains are likely not statistically significant.

Response. In the single-module ablations, CHRF improves SI-SNRi by 0.3 dB and SDRi by 0.4 dB over cross-attention, the strongest alternative fusion operator by SI-SNRi in Table 3. CAM improves the same metrics by 0.6 dB and 0.5 dB over tanh, the strongest standard mask baseline in Table 4. Adding these individual gains gives 0.9 dB in each metric. However, CHRF and CAM are coupled through cross-sensor cues used in feature fusion and adaptive mask reconstruction, so their combined contribution needs to be evaluated jointly. We therefore replaced both modules with cross-attention and a tanh mask. This joint variant achieves 16.1 dB SI-SNRi and 16.3 dB SDRi, compared with 17.3 dB and 17.5 dB for full BAT.

FusionMaskSI-SNRi (dB)SDRi (dB)
CHRF (ours)CAM (ours)17.317.5
Cross-attentiontanh16.116.3

The measured gain is thus 1.2 dB in both metrics, exceeding the 0.9 dB obtained by simply adding the individual gains. These results are reported in Appendix H, Table 8, with the additions marked in blue.

We also ran BAT and this cross-attention + tanh-mask variant on real AC–BC recordings captured using a Timekettle W4 headset in an open-plan office. The wearer walked around while speaking with a fixed interlocutor, amid ambient office noise and background conversations among other office workers. The output spectrograms show that BAT better reconstructs mid- and high-frequency speech components in these recordings. These comparisons are included in Appendix H, Figure 9, with the additions marked in blue.

References

[1] Samuel Pegg et al. RTFS-Net: Recurrent Time-Frequency Modelling for Efficient Audio-Visual Speech Separation. ICLR, 2024.

[2] Kai Li et al. Efficient Audio-Visual Speech Separation with Discrete Lip Semantics and Multi-Scale Global-Local Attention. ICLR, 2026.

[3] Kenny Falkær Olsen et al. Knowing When to Quit: Probabilistic Early Exits for Speech Separation Networks. ICLR, 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Hope the reviewers will take note of our response

Dear Reviewers,

Following your initial review comments, we have revised our paper, added experiments, and provided responses addressing your main concerns. The revisions are marked in blue.

We hope to interact with you during the discussion and further improve the quality of our paper.

Thank you very much for your time and consideration.

Kind regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.