acceptodds
Under review as a conference paper at ICLR 2027

CoReSep: Input Composition for Unifying Speech Separation and Target Speaker Extraction with a Pretrained Speech Foundation Model

Abstract

Speech separation (SS) and target speaker extraction (TSE) share the same underlying goal of disentangling overlapping speakers, providing a natural basis for unified modeling. However, existing unified systems either extend acoustically driven separation backbones with speaker-selection mechanisms, tying source attribution primarily to acoustic evidence, or adopt single-stream language-model generation that requires repeated inference for multi-speaker SS without explicit cross-stream coordination. To address these limitations, we introduce CoReSep, a unified SS and TSE system built around two key ideas. First, separation is performed by a pretrained speech foundation model (SFM) itself, whose contextual prior provides high-level constraints on source attribution beyond acoustic evidence alone. Second, a novel input composition mechanism enables joint multi-stream modeling by specifying each task entirely through the input: replicating the mixture representation instantiates multiple jointly decoded streams for SS, whereas prefixing an enrollment representation anchors the single output stream to the target speaker for TSE. In this way, a standard single-stream SFM serves as the unified separator without architectural modification. A conditional flow-matching module then restores acoustic detail by conditioning on both the separated representation and the mixture Mel spectrogram, after which a vocoder reconstructs the waveform. Experimental results demonstrate that CoReSep outperforms previous task-specific and unified systems in linguistic integrity on both SS and TSE, while maintaining competitive perceptual quality. On Libri2Mix, it achieves differential word error rates (dWERs) of 0.85%/3.65% for SS and 1.22%/3.93% for TSE on the Mix-Clean/Mix-Both subsets, respectively. It also maintains its advantage over a strong discriminative SS baseline on three- and four-speaker mixtures and outperforms it on all metrics on EchoSet, whose reverberation and random overlap ratios are closer to real-world conditions. Audio samples are available online.samehttps://anonymous.4open.science/w/CoReSep-Demo-4E0B/

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
speech separation, target speaker extraction, speech foundation models, unified model, flow matching

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_KEnkOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a generative SS/TSE system consisting of a fine-tuned WavLM as separator, then use flow matching, and a vocoder. WavLM predicts clean-source representations, which condition Mel-spectrogram reconstruction together with the mixture Mel. The proposed “input composition” concatenates copies of mixture features for SS and prepends enrollment features for TSE.

Critical strengths
  1. The system achieves consistently lower dWER than the evaluated baselines on both SS and TSE, providing evidence of improved speech-content preservation, particularly under noise.
Critical weaknesses
  1. The technical novelty is very limited. Clean-representation regression with WavLM is inherited from PASE, and the flow-matching/vocoder reconstruction pipeline is pretty standard and already present in StuPASE. The main addition here is straightforward input concatenation and replication. I feel the extensive terminology and presentation overcomplicate this simple design without establishing a substantial methodological contribution.

  2. The “phonological prior” is insufficiently defined and its claimed role is not demonstrated. Comparing pretrained initialization against random initialization establishes the benefit of pretraining, not a specifically phonological mechanism. The detailed claims in lines 446–452 about lower-layer speaker discrimination and higher-layer linguistic constraints are unsupported by the reported experiments.

  3. The ablations are insufficient to establish the claimed advantages of input composition. Zero-padding does not isolate the contribution of cross-stream interaction, while the output-head baseline differs in both training setup and computational budget. A direct flow-matching/vocoder baseline under the same SS/TSE setup is also missing, leaving the contribution of the WavLM separation stage inadequately established.

Questions for authors
  1. The computational cost seems to be high: the full system requires approximately 285.5/578.2 GMAC/s for two-/four-speaker SS (Appendix A). This limits its practical appeal despite the single-pass separator.

  2. The motivation for using dWER instead of standard WER is unclear. Its interpretation as an indicator of source-attribution errors also needs clearer justification. (line 314–318)

  3. The main experiments assume a known speaker count. Although Appendix A.4 relaxes this assumption, the system still requires a predefined upper bound on the number of output streams, which seems to be impractical.

  4. What is the substantive advantage of removing selection modules, category prompts, or task tokens beyond simplifying the architecture? (lines 160–161)

  5. The distinction between “phonological prior,” “phonetic information,” and “linguistic integrity” needs clearer explanation. In particular, does linguistic integrity refer to preserving spoken content or semantic meaning?

5 comments
AuthorsOpenReview ↗

Response to Reviewer KEnk (1/4): W1, W2

We sincerely thank you for recognizing that CoReSep achieves consistently lower dWER than the evaluated baselines on both SS and TSE, particularly under noise. We respond to your comments as follows:

W1. Technical Novelty:

We agree that clean-representation regression and the flow-matching/vocoder reconstruction pipeline are inherited from PASE and StuPASE, respectively, and we do not claim them as methodological contributions. Our key contribution is input composition, which adapts a pretrained single-stream speech foundation model (SFM) to jointly perform multi-speaker separation and enrollment-conditioned TSE within a shared backbone, by letting each composed segment represent a single speaker while all streams are jointly modeled by the pretrained Transformer.

Input composition is indeed lightweight to implement, which we view as a strength rather than a limitation:

Highly effective and readily transferable. Input composition achieves the lowest dWER across all main comparisons on both SS and TSE, and clearly outperforms all ablated variants on SS. Moreover, it introduces no architectural modification, selection module, or new embedding unseen during pretraining. It can thus be directly applied to existing pretrained SFMs while fully retaining the benefits of pretrained initialization, and a single model handles both SS and TSE across two to four speakers.

Extensible to other tasks. PASE-style speech enhancement corresponds to the special case of a single mixture copy without enrollment, and other conditional speech tasks, such as acoustic echo cancellation and speech editing, could potentially be formulated by composing the corresponding inputs. Input composition thus offers a promising path toward unifying SS and TSE with a broader range of speech tasks within one pretrained backbone.

Regarding presentation, our terminology is intended to provide a concise, formal description of how SS and TSE share the same backbone and parameters while differing only in their input configurations. Terms such as the phonological prior are introduced to interpret why pretrained initialization benefits these tasks. We believe the current presentation balances formal rigor with brevity. Nevertheless, we will streamline it where possible and more clearly distinguish the central contribution from the inherited components.

W2. Phonological Prior:

Following PASE [1], we use “phonological prior” to refer to the contextual constraints on phonetic sequences that WavLM learns during pretraining. PASE used this concept to interpret the benefit of WavLM pretraining based on its analyses of speech enhancement. We adopted the term because CoReSep uses the same pretrained WavLM backbone.

Our initialization ablation shows that WavLM pretraining improves performance on SS and TSE in our setting. We agree that this comparison leaves open which aspects of pretraining account for the improvement.

PASE’s interpretation focuses on how phonetic context can guide the recovery of degraded speech. However, in SS and TSE, recovering speech content also requires determining which speaker produced it. Prior analyses of WavLM [2, 3] suggest that the same backbone provides cues for both purposes. Acoustic and speaker-related information is more prominent in lower and intermediate layers, while phonetic and word-level information is more prominent in higher layers. We therefore hypothesize that these cues play complementary roles in CoReSep, with lower and intermediate layers supporting speaker discrimination and higher layers helping maintain linguistic coherence within each output stream. These proposed layer-specific roles remain untested in our experiments. We will define the phonological prior in Section 4.3 and revise lines 446–452 to present our interpretation explicitly as a hypothesis.

References

[1] Rong, X., Hu, Q., Yesilbursa, M., Wojcicki, K., & Lu, J. (2026). PASE: Leveraging the Phonological Prior of WavLM for Low-Hallucination Generative Speech Enhancement. Proceedings of the AAAI Conference on Artificial Intelligence, 40(39), 32826–32834. https://doi.org/10.1609/aaai.v40i39.40562

[2] S. Chen et al., "WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing," in IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505-1518, Oct. 2022, doi: 10.1109/JSTSP.2022.3188113.

[3] A. Pasad, B. Shi and K. Livescu, "Comparative Layer-Wise Analysis of Self-Supervised Speech Models," ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece, 2023, pp. 1-5, doi: 10.1109/ICASSP49357.2023.10096149.

AuthorsOpenReview ↗

Response to Reviewer KEnk (2/4): W3

W3. Ablations of Input Composition and the Separation Stage:

Cross-stream interaction. We have added Stream-Isolated Composition, which keeps all mixture copies but masks attention across them (Table R3). Isolating the streams has no consistent effect on TSE, whose input contains a single stream, but degrades SS increasingly with more speakers: dWER rises from 3.56% to 4.38% on Libri2Mix and from 10.52% to 13.81% on Libri3Mix, and reaches 51.78% on Libri4Mix (vs. 28.27%), where the interior copies become indistinguishable. Nevertheless, this variant still outperforms Zero-Padded Composition on Libri2Mix and Libri3Mix. These results indicate that both the per-stream copies and cross-stream interaction contribute to performance, with cross-stream interaction becoming particularly important for large speaker counts in our architecture.

Table R3: Ablation on input composition.

ArchitectureMulti-Stream ImplementationLibri2Mix TSELibri2Mix SSLibri3Mix TSELibri3Mix SSLibri4Mix TSELibri4Mix SS
WavLMInput Composition5.553.5611.6310.5223.5328.27
Zero-Padded Composition5.686.2116.4918.9335.9342.95
Stream-Isolated Composition4.394.3811.6913.8124.9451.78
WavLM+HeadsLightweight LSTM Heads–5.87–21.47–39.81
Large BS-RNN Heads–4.83–18.57–36.99

Output-head baseline. The SUPERB-style baseline shares the same WavLM backbone, pretrained initialization, training data, and training configuration as Input Composition; the only difference is the multi-stream implementation under ablation, i.e., reading out the speaker streams through multiple output heads rather than composing the input. Because its number of heads is fixed, it requires a separate model per speaker count, following SUPERB, each trained on the corresponding subset of the same data.

Regarding computation, the difference in computational cost stems from the mechanism being ablated: with an identical backbone, Input Composition processes one mixture copy per stream, whereas the SUPERB-style baseline encodes the mixture once and uses a separate output head for each stream. We agree that the original ablation did not control for computational cost. To make the comparison fairer, we added a SUPERB-style baseline with larger BS-RNN heads whose computational cost is comparable to that of Input Composition (Table R3). The larger heads reduce SS dWER only slightly, from 5.87%/21.47%/39.81% to 4.83%/18.57%/36.99% on Libri2Mix, Libri3Mix, and Libri4Mix, respectively. Input Composition still achieves substantially lower dWER across all three datasets (3.56%/10.52%/28.27%), supporting its advantage at a comparable computational cost. We will include this additional baseline and the corresponding analysis in Section 4.3.

Flow-matching baseline. We clarify that the conditional flow matching (CFM) in CoReSep is a reconstruction module, not a separation model. Source attribution is performed entirely by the WavLM separator: PIT supervision for SS and enrollment conditioning for TSE are both applied at this stage. The CFM is trained subsequently with the separator frozen and only reconstructs each already-separated representation into its corresponding source waveform. Thus, applying flow matching directly to SS would require a separate multi-source generative separation formulation, rather than a direct replacement of the reconstruction module.

Furthermore, to isolate the contribution of the WavLM separator, Table 3 evaluates its outputs directly using the L24 vocoder, without the CFM reconstruction module. The separator already achieves dWERs of 1.51%/3.92% for TSE and 1.29%/3.58% for SS on Mix-Clean/Mix-Both, respectively. These results demonstrate that effective source attribution and linguistic content preservation are already achieved at the separation stage, while the CFM serves to restore fine-grained acoustic details.

AuthorsOpenReview ↗

Response to Reviewer KEnk (3/4): Q1–Q3

Q1. Computational Cost and Future Efficiency:

As shown in Table 6, the cost is dominated by reconstruction rather than separation. For two-speaker SS, the separator accounts for about 38 GMACs/s (13% of the total), whereas the CFM accounts for about 236 GMACs/s (83%), because it requires 8 sampling steps with two forward passes each under classifier-free guidance, per stream. The same holds for four speakers (separator about 14%, CFM about 82%). This reflects the design priorities of this work, which focuses on introducing and evaluating a unified approach to SS and TSE, with an emphasis on separation performance rather than computational efficiency. The separator itself, which performs separation in a single pass, is therefore not the bottleneck. The reconstruction module is also largely independent of the separator and can be replaced by more efficient designs. For example, a one-step module such as MeanFlow would reduce the reconstruction cost by a factor of 16. Improving computational efficiency while maintaining separation performance is therefore an important direction for future work.

Q2. Motivation and Limitations of dWER:

We use dWER because the original LibriSpeech transcripts are not directly applicable to our test references. Specifically, under the "min" setting of LibriMix, each mixture is truncated to the duration of its shortest source, making the original transcripts inconsistent with the resulting clean reference segments. We therefore transcribe both the clean references and separated outputs using the same ASR model and compute the WER between their transcriptions. This paired comparison also helps reduce the ASR model's intrinsic recognition errors on clean speech, allowing dWER to better reflect changes in recognized content introduced by separation and reconstruction. Nevertheless, we acknowledge that dWER remains susceptible to ASR errors caused by reconstruction artifacts. It measures transcript-level content fidelity and provides only indirect evidence of source-attribution errors. We will clarify why we use dWER and discuss its limitations in Section 4.1.

Q3. Practicality of Speaker Count Assumptions:

We understand the reviewer's concern. However, we would first note that the main comparison is conducted under the same assumption for all systems: all SS baselines also assume a known speaker count, so the known-count setting does not favor CoReSep. Moreover, requiring an upper bound is also common practice for separation with an unknown speaker count. Attractor-based methods, e.g., [4], also require a predefined maximum, and fixed-size outputs trained to emit silence for absent sources [5] are bounded by construction. We will clarify this in the revision.

References

[4] Y. Lee, S. Choi, B.-Y. Kim, Z.-Q. Wang and S. Watanabe, "Boosting Unknown-Number Speaker Separation with Transformer Decoder-Based Attractor," ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul, Republic of Korea, 2024, pp. 446-450, doi: 10.1109/ICASSP48485.2024.10446032.

[5] S. Wisdom et al., "What's all the Fuss about Free Universal Sound Separation Data?," ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada, 2021, pp. 186-190, doi: 10.1109/ICASSP39728.2021.9414774.

Reviewer_K7dtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CoReSep unifies monaural speech separation (SS) and target speaker extraction (TSE) using a pretrained WavLM model as the separator. SS is specified by concatenating multiple copies of the mixture representation, while TSE is specified by prepending an enrollment representation. The resulting output segments are supervised with clean-source WavLM representations, using permutation-invariant training for SS and silence targets for surplus streams. A conditional flow-matching module and Mel vocoder reconstruct the waveforms. The paper reports improvements in linguistic-content metrics on LibriMix and EchoSet, together with analyses of pretraining, input composition, unknown speaker counts, and reconstruction limitations.

Critical strengths

The input composition of CoReSep enables joint multistream separation and enrollment-conditioned extraction without adding task-specific modules to the WavLM separator. The contribution is a useful extension of existing representation-distillation and reconstruction approaches to unified SS/TSE. Table 4 provides a meaningful control: replacing the additional mixture copies with zeros preserves sequence length, parameter count, and compute, but increases Mix-Both SS dWER from 3.56/10.52/28.27 to 6.21/18.93/42.95 on Libri2Mix/3Mix/4Mix. This supports the value of giving each output segment its own mixture representation.

On Libri2Mix Mix-Both, CoReSep achieves 3.65% SS dWER versus 4.99% for the retrained TF-GridNet, and 3.93% TSE dWER versus 5.47% for MeanFlow-TSE. Improvements also hold in the evaluated three- and four-speaker SS settings. On EchoSet, after both models are trained with reverberant mixtures, CoReSep improves all six reported metrics over TF-GridNet. The gains therefore extend beyond comparisons with the released unified-model checkpoints. The controlled ablations and the reconstruction analysis in Appendix A.6 make the empirical contribution more convincing. Taken together, the modeling simplicity and these consistent improvements support my recommendation of weak acceptance.

Critical weaknesses

The source-attribution interpretation is less directly supported than the performance gains. Section 4.1 uses dWER as the primary indicator of attribution errors. However, ASR-transcription agreement does not disentangle leakage, omissions, and within-utterance speaker switches from recognition or resynthesis effects, and can miss identity errors that preserve the transcript. LPS, SBS, and SpkSim provide complementary evidence. Appendix A.6 explicitly discusses ASR sensitivity and reconstruction effects, but there is no separate breakdown of attribution failures. Likewise, Section 4.3 acknowledges both acoustic/speaker cues and phonetic/contextual information. Table 3 supports the overall benefit of pretraining without isolating their contributions. The remaining concern is therefore the specificity of the source-attribution and phonological explanations, rather than a failure to acknowledge alternative factors. This limits mechanistic insight without invalidating the reported gains.

Unknown-count content fidelity is not evaluated on the complete threshold-selected output set. Appendix A.4 reports strong counting accuracies and separate over- and under-estimation rates. However, dWER is computed on the ground-truth number of reference-matched streams and is independent of the activity threshold. Counting errors are therefore measured separately, but their consequences are not included in this content score. Moreover, Table 5 evaluates outputs decoded by the L24 vocoder rather than the complete CFM-based reconstruction system. The results establish useful fixed-capacity separation and counting capabilities, while leaving content fidelity after threshold selection and full reconstruction unmeasured. This limits the evaluation of an important extension, rather than undermining the main known-count results.

Questions for authors

Evidence for source attribution: Building on the discussion in Section 4.3 and Appendix A.6, is there a targeted analysis separating attribution failures from recognition or reconstruction errors that supports the interpretation of the dWER gains? Existing diagnostics, or a small manually checked subset, would be useful. Please distinguish experimentally established benefits of pretraining from hypotheses about the particular contributions of phonological context and acoustic/speaker cues.

Unknown-count outputs: Could you evaluate the actual set of streams retained by the activity threshold using the complete reconstruction back-end, and specify how missing and spurious outputs are accounted for? Permutation alignment is appropriate, but the evaluated output set should be determined by the threshold rather than by selecting exactly the ground-truth number of streams. Results on a representative subset would help connect the counting analysis to end-to-end separation quality.

Vocoder training data: Appendix A.3 states that the vocoders use the same data as Rong et al. (2026a), excluding LibriSpeech. However, StuPASE v2, Section 3.1 states that LibriSpeech is added for flow-matching and Mel-vocoder training. Was a modified recipe used here? Appendix A.1 already describes train–test separation for the mixture-generation pipeline; please additionally clarify the corpora and splits used for both vocoders and their checks for overlap with evaluation material. This is a reproducibility clarification, not an allegation of leakage.

2 comments
AuthorsOpenReview ↗

Response to Reviewer K7dt (1/2): W1/Q1

We sincerely thank you for recognizing the value of input composition, our controlled ablations and reconstruction analysis, and the consistent improvements across settings. We respond to your comments as follows:

W1/Q1. Attribution Errors and the Limitations of dWER, and the Isolated Contributions of the Acoustic and Phonetic Information:

We agree that ASR-transcription agreement does not disentangle leakage, omissions, and within-utterance speaker switches from recognition or resynthesis effects, and can miss identity errors that preserve the transcript.

Recognition effects. We compute dWER by applying the same ASR model to the separated outputs and the clean references. This reduces the impact of shared recognition biases.

Resynthesis effects. As shown in Appendix A.6, reconstructing speech from oracle representations yields dWERs of 0.48% on Mix-Clean and 0.60% on Mix-Both. These low values suggest that the reconstruction backend introduces little transcript disagreement when given oracle representations.

Speaker identity. We report SpkSim as complementary evidence of speaker identity preservation. Our competitive SpkSim scores help address the concern that reconstructed speech may have a correct transcription but an incorrect speaker identity.

Breakdown of source-attribution errors. We agree that a breakdown by error type would strengthen the source-attribution interpretation. To the best of our knowledge, no established metric can reliably determine which discrepancies captured by dWER are attributable to interference leakage, target speech omissions, or within-utterance speaker switches. We therefore complemented dWER with a small-scale listening assessment. For each mixture we examined, we listened to the CoReSep and baseline outputs alongside the clean source references to identify these errors directly. All three error types recurred in the baseline outputs and were less frequent in CoReSep's outputs, suggesting that improved source attribution contributed to its lower dWER on the mixtures we examined.

We provide ten selected cases on the demo page (https://anonymous.4open.science/w/CoReSep-Demo-4E0B/), including five cases of within-utterance speaker switching, three of interference leakage, and two of target speech omission. The examples span SS and TSE (five cases each) and cover both Mix-Clean and Mix-Both. In each case, the baseline exhibited the corresponding attribution error, while CoReSep avoided that error and achieved a lower dWER. A systematic evaluation of source-attribution errors remains an important direction for future work. We will clarify in Section 4.1 that dWER provides indirect evidence of source attribution and discuss its limitations.

Contributions of acoustic and phonological cues. We agree that Table 3 demonstrates the overall benefit of initializing with pretrained WavLM without isolating the contributions of acoustic/speaker cues and phonological/contextual information. These cues may play complementary roles. Acoustic and speaker cues may help the model distinguish speakers and maintain speaker identity within each output stream, while phonological and contextual information may help maintain a coherent linguistic trajectory within each output stream. We will revise Section 4.3 to present these roles as hypotheses and clarify that Table 3 supports only the overall benefit of pretraining.

AuthorsOpenReview ↗

Response to Reviewer K7dt (2/2): W2/Q2, Q3

W2/Q2. End-to-End Evaluation of Unknown-Count Outputs:

We have evaluated the threshold-selected outputs end-to-end on the complete Mix-Both test sets (3,000 mixtures each), reconstructing all N^\hat{N} selected streams with the complete backend. Each missing reference counts all of its words as deletions, and each spurious output counts all words of its transcription as insertions, with the same denominator as in the known-NN setting. The results are shown in Table R1. With unknown speaker counts, the overall dWERs are 5.18%, 11.61%, and 29.55% on Libri2Mix, Libri3Mix, and Libri4Mix, respectively, compared with 3.44%, 10.76%, and 29.28% in the known-NN setting.

Table R1: SS with unknown speaker count on the Mix-Both test sets.

Libri2MixLibri3MixLibri4Mix
Counting accuracy (%)98.4096.4399.47
dWER (%), known NN (K=NK = N)3.4410.7629.28
dWER (%), unknown NN (K=4K = 4)5.1811.6129.55

Table R2: dWER on the Mix-Both test sets, broken down by the counting outcome of the unknown-NN setting.

Test setCounting outcome#Mixtures (%)Known NN dWER (%)Unknown NN dWER (%)
Libri2MixCorrect2,952 (98.40%)3.403.95
Over-estimated48 (1.60%)5.3056.48
Libri3MixCorrect2,893 (96.43%)10.3610.30
Over-estimated105 (3.50%)23.4554.67
Under-estimated2 (0.07%)50.0037.50
Libri4MixCorrect2,984 (99.47%)29.2329.53
Under-estimated16 (0.53%)41.4633.99

As shown in Table R2, the two settings yield similar dWER on correctly counted mixtures (3.40% vs. 3.95%, 10.36% vs. 10.30%, and 29.23% vs. 29.53%, respectively). Counting errors are rare (1.60%, 3.57%, and 0.53% of mixtures, respectively), with over-estimation accounting for nearly all errors on Libri2Mix and Libri3Mix. We will revise Appendix A.4 to provide the full evaluation protocol, a breakdown by counting outcome, and an analysis of missing and spurious outputs.

Q3. Vocoder Training Data:

We apologize for the inaccurate description in Appendix A.3. The two vocoders in fact follow different recipes. The L24 vocoder is trained on exactly the same data as PASE [1], which does not include LibriSpeech, whereas the Mel vocoder is trained on exactly the same data as StuPASE [2], which includes LibriSpeech with the same data filtering. For the Mel vocoder, LibriSpeech is used only from its training subsets, and no test-set utterance is included. Since all our LibriMix evaluation sets, including the extended Libri4Mix, are built from test-clean, whose speakers are disjoint from the training subsets, neither vocoder has been exposed to any evaluation utterance or speaker. We will correct Appendix A.3 to specify the corpora and splits used for both vocoders.

References

[1] Rong, X., Hu, Q., Yesilbursa, M., Wojcicki, K., & Lu, J. (2026). PASE: Leveraging the Phonological Prior of WavLM for Low-Hallucination Generative Speech Enhancement. Proceedings of the AAAI Conference on Artificial Intelligence, 40(39), 32826–32834. https://doi.org/10.1609/aaai.v40i39.40562

[2] Rong, X., Gao, J., Wang, Z., Yesilbursa, M., Wojcicki, K., & Lu, J. (2026). StuPASE: Towards Low-Hallucination Studio-Quality Generative Speech Enhancement. Proc. Interspeech 2026, 1795–1800. https://doi.org/10.21437/Interspeech.2026-837

Reviewer_mbvUOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

In this paper the authors propose CoReSep, a method able to perform both speech separation (SS) and target speech extraction (TSE), not requiring iterative application as previous techniques (LLaSE-G1, UniSE) and being able to use a self-supervised speech foundation model (SFE) as a pre-trained backbone. The pipeline comprises of a CNN encoder mapping to a latent space, a transformer backbone (the SFE model) that outputs the separated / extracted semantic tokens, and a reconstruction step that maps back to time domain (chaining conditional flow matching and a mel vocoder). The novelty in the paper is related to the way the backbone handles the inputs. Naturally, the authors repeat the input mixture K times if K sources have to be separated and prefix the CNN encoding of the enrolment for extraction. The separated predictions are naturally aligned with the inputs and are matched to the latent targets (computed with a frozen version of the SFE model) via PIT. The reconstruction part is trained as a second step. The authors therefore compare their method with standalone SS and TSE models together with unified models showcasing strong results.

Critical strengths
  • The mechanism introduced by the authors is very natural and can adapt a pretrained speech LM for the joint separation / TSE task in an easy way.
  • The experimental section is solid, showcasing results on different numbers of sources, comparing both with standalone state-of-the-art TSE / SS models and unified models. CoReSep showcases better results on multiple metrics (e.g., DNSMOS, dWER) and the ablation study is particularly interesting when the authors show that zero-padding performance is poorer.
  • The paper is very well written, without grammatical errors and a very good flow. Quite a good surprise, having seen so many AI slop papers this year.
Critical weaknesses
  • My main concern is why authors have evaluated only against TF-GridNet on the #sources > 2 setting? To show that the model scales well with more sources they should also include other baselines, especially the unified ones (the category of CoReSep).
  • The GMSDI method can perform both joint-source generation and single-target extraction in a non-recursive way and being a generative model. Please discuss the differences with this method in the related work section.

References

  • [1] Postolache, E., Mariani, G., Cosmo, L., Benetos, E., & Rodolà, E. (2024, April). Generalized multi-source inference for text conditioned music diffusion models. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 6980-6984). IEEE.
Questions for authors
  • As mentioned in previous section, I would like to know why the authors have evaluated only on TF-GridNet in the #sources > 2 setting.
2 comments
AuthorsOpenReview ↗

Response to Reviewer mbvU

We sincerely thank you for recognizing the naturalness of our input composition mechanism, the thoroughness of our experiments and ablations, and the clarity of our writing. We respond to your comments as follows:

W1/Q1. Multi-Speaker Baseline Comparisons:

Very few SS/TSE systems release checkpoints for mixtures with more than two speakers, and the available checkpoints were trained on different datasets and under different conditions, making a controlled comparison difficult. We therefore retrained the baselines using our own data pipeline. Given the cost of retraining every system, we selected TF-GridNet, the SS baseline with the lowest dWER on Libri2Mix (Table 1), and trained two separate versions for the three- and four-source settings. We agree that our original evaluation did not include a TSE baseline in these settings. To address this, we also retrained USEF-TFGridNet and report its results in Table 1. USEF-TFGridNet consistently underperforms CoReSep across the evaluated conditions. On Libri3Mix, USEF-TFGridNet achieves dWERs of 12.62% and 24.28% under Mix-Clean and Mix-Both, respectively, compared with 5.50%/11.83% for CoReSep. On Libri4Mix, the corresponding dWERs are 26.52%/40.53% for USEF-TFGridNet and 14.93%/24.68% for CoReSep.

As for the unified systems, LLaSE-G1 and UniSE both realize SS by recursive extraction in which each pass takes a single reference speaker, and neither defines a procedure beyond two speakers. Moreover, on Libri2Mix, their SS dWERs are already more than an order of magnitude higher than those of CoReSep (UniSE: 15.48%/44.86%; LLaSE-G1: 39.79%/68.61%; CoReSep: 0.85%/3.65% on Mix-Clean/Mix-Both, respectively). We therefore follow their original evaluations and report them on Libri2Mix only.

W2. Comparison with GMSDI:

Thank you for pointing us to GMSDI. Like CoReSep, GMSDI supports non-recursive joint multi-source separation and single-source extraction, but the two differ in two key respects.

(1) Source specification. GMSDI specifies individual sources through source-specific text embeddings, e.g., instrument labels or semantic descriptions. In contrast, all sources in speech separation belong to the same broad category and thus cannot be distinguished by category labels, so CoReSep performs blind SS without source labels and specifies the target speaker in TSE using an enrollment utterance.

(2) Separation framework. GMSDI is an inference procedure built on a text-conditioned diffusion model, which is trained via score matching on mixtures with associated text embeddings, without requiring isolated source targets. Separation and extraction are performed at inference through iterative score-based posterior sampling under mixture-consistency constraints. In contrast, CoReSep trains the separator directly for separation: a pretrained speech foundation model, adapted via input composition, regresses clean-source representations under permutation-invariant training for SS and produces all streams in a single forward pass. Although CoReSep also employs a generative flow-matching module, it is used only for subsequent acoustic reconstruction, not for source separation.

We will add this discussion to the Related Work section.

I thank the authors for their in-depth answer on the related baselines and appreciate they are adding a discussion on the Related Work section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.