- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents a generative SS/TSE system consisting of a fine-tuned WavLM as separator, then use flow matching, and a vocoder. WavLM predicts clean-source representations, which condition Mel-spectrogram reconstruction together with the mixture Mel. The proposed “input composition” concatenates copies of mixture features for SS and prepends enrollment features for TSE.
Critical strengths
- The system achieves consistently lower dWER than the evaluated baselines on both SS and TSE, providing evidence of improved speech-content preservation, particularly under noise.
Critical weaknesses
-
The technical novelty is very limited. Clean-representation regression with WavLM is inherited from PASE, and the flow-matching/vocoder reconstruction pipeline is pretty standard and already present in StuPASE. The main addition here is straightforward input concatenation and replication. I feel the extensive terminology and presentation overcomplicate this simple design without establishing a substantial methodological contribution.
-
The “phonological prior” is insufficiently defined and its claimed role is not demonstrated. Comparing pretrained initialization against random initialization establishes the benefit of pretraining, not a specifically phonological mechanism. The detailed claims in lines 446–452 about lower-layer speaker discrimination and higher-layer linguistic constraints are unsupported by the reported experiments.
-
The ablations are insufficient to establish the claimed advantages of input composition. Zero-padding does not isolate the contribution of cross-stream interaction, while the output-head baseline differs in both training setup and computational budget. A direct flow-matching/vocoder baseline under the same SS/TSE setup is also missing, leaving the contribution of the WavLM separation stage inadequately established.
Questions for authors
-
The computational cost seems to be high: the full system requires approximately 285.5/578.2 GMAC/s for two-/four-speaker SS (Appendix A). This limits its practical appeal despite the single-pass separator.
-
The motivation for using dWER instead of standard WER is unclear. Its interpretation as an indicator of source-attribution errors also needs clearer justification. (line 314–318)
-
The main experiments assume a known speaker count. Although Appendix A.4 relaxes this assumption, the system still requires a predefined upper bound on the number of output streams, which seems to be impractical.
-
What is the substantive advantage of removing selection modules, category prompts, or task tokens beyond simplifying the architecture? (lines 160–161)
-
The distinction between “phonological prior,” “phonetic information,” and “linguistic integrity” needs clearer explanation. In particular, does linguistic integrity refer to preserving spoken content or semantic meaning?
5 comments
Response to Reviewer KEnk (2/4): W3
W3. Ablations of Input Composition and the Separation Stage:
Cross-stream interaction. We have added Stream-Isolated Composition, which keeps all mixture copies but masks attention across them (Table R3). Isolating the streams has no consistent effect on TSE, whose input contains a single stream, but degrades SS increasingly with more speakers: dWER rises from 3.56% to 4.38% on Libri2Mix and from 10.52% to 13.81% on Libri3Mix, and reaches 51.78% on Libri4Mix (vs. 28.27%), where the interior copies become indistinguishable. Nevertheless, this variant still outperforms Zero-Padded Composition on Libri2Mix and Libri3Mix. These results indicate that both the per-stream copies and cross-stream interaction contribute to performance, with cross-stream interaction becoming particularly important for large speaker counts in our architecture.
Table R3: Ablation on input composition.
| Architecture | Multi-Stream Implementation | Libri2Mix TSE | Libri2Mix SS | Libri3Mix TSE | Libri3Mix SS | Libri4Mix TSE | Libri4Mix SS |
|---|---|---|---|---|---|---|---|
| WavLM | Input Composition | 5.55 | 3.56 | 11.63 | 10.52 | 23.53 | 28.27 |
| Zero-Padded Composition | 5.68 | 6.21 | 16.49 | 18.93 | 35.93 | 42.95 | |
| Stream-Isolated Composition | 4.39 | 4.38 | 11.69 | 13.81 | 24.94 | 51.78 | |
| WavLM+Heads | Lightweight LSTM Heads | – | 5.87 | – | 21.47 | – | 39.81 |
| Large BS-RNN Heads | – | 4.83 | – | 18.57 | – | 36.99 |
Output-head baseline. The SUPERB-style baseline shares the same WavLM backbone, pretrained initialization, training data, and training configuration as Input Composition; the only difference is the multi-stream implementation under ablation, i.e., reading out the speaker streams through multiple output heads rather than composing the input. Because its number of heads is fixed, it requires a separate model per speaker count, following SUPERB, each trained on the corresponding subset of the same data.
Regarding computation, the difference in computational cost stems from the mechanism being ablated: with an identical backbone, Input Composition processes one mixture copy per stream, whereas the SUPERB-style baseline encodes the mixture once and uses a separate output head for each stream. We agree that the original ablation did not control for computational cost. To make the comparison fairer, we added a SUPERB-style baseline with larger BS-RNN heads whose computational cost is comparable to that of Input Composition (Table R3). The larger heads reduce SS dWER only slightly, from 5.87%/21.47%/39.81% to 4.83%/18.57%/36.99% on Libri2Mix, Libri3Mix, and Libri4Mix, respectively. Input Composition still achieves substantially lower dWER across all three datasets (3.56%/10.52%/28.27%), supporting its advantage at a comparable computational cost. We will include this additional baseline and the corresponding analysis in Section 4.3.
Flow-matching baseline. We clarify that the conditional flow matching (CFM) in CoReSep is a reconstruction module, not a separation model. Source attribution is performed entirely by the WavLM separator: PIT supervision for SS and enrollment conditioning for TSE are both applied at this stage. The CFM is trained subsequently with the separator frozen and only reconstructs each already-separated representation into its corresponding source waveform. Thus, applying flow matching directly to SS would require a separate multi-source generative separation formulation, rather than a direct replacement of the reconstruction module.
Furthermore, to isolate the contribution of the WavLM separator, Table 3 evaluates its outputs directly using the L24 vocoder, without the CFM reconstruction module. The separator already achieves dWERs of 1.51%/3.92% for TSE and 1.29%/3.58% for SS on Mix-Clean/Mix-Both, respectively. These results demonstrate that effective source attribution and linguistic content preservation are already achieved at the separation stage, while the CFM serves to restore fine-grained acoustic details.
Response to Reviewer KEnk (3/4): Q1–Q3
Q1. Computational Cost and Future Efficiency:
As shown in Table 6, the cost is dominated by reconstruction rather than separation. For two-speaker SS, the separator accounts for about 38 GMACs/s (13% of the total), whereas the CFM accounts for about 236 GMACs/s (83%), because it requires 8 sampling steps with two forward passes each under classifier-free guidance, per stream. The same holds for four speakers (separator about 14%, CFM about 82%). This reflects the design priorities of this work, which focuses on introducing and evaluating a unified approach to SS and TSE, with an emphasis on separation performance rather than computational efficiency. The separator itself, which performs separation in a single pass, is therefore not the bottleneck. The reconstruction module is also largely independent of the separator and can be replaced by more efficient designs. For example, a one-step module such as MeanFlow would reduce the reconstruction cost by a factor of 16. Improving computational efficiency while maintaining separation performance is therefore an important direction for future work.
Q2. Motivation and Limitations of dWER:
We use dWER because the original LibriSpeech transcripts are not directly applicable to our test references. Specifically, under the "min" setting of LibriMix, each mixture is truncated to the duration of its shortest source, making the original transcripts inconsistent with the resulting clean reference segments. We therefore transcribe both the clean references and separated outputs using the same ASR model and compute the WER between their transcriptions. This paired comparison also helps reduce the ASR model's intrinsic recognition errors on clean speech, allowing dWER to better reflect changes in recognized content introduced by separation and reconstruction. Nevertheless, we acknowledge that dWER remains susceptible to ASR errors caused by reconstruction artifacts. It measures transcript-level content fidelity and provides only indirect evidence of source-attribution errors. We will clarify why we use dWER and discuss its limitations in Section 4.1.
Q3. Practicality of Speaker Count Assumptions:
We understand the reviewer's concern. However, we would first note that the main comparison is conducted under the same assumption for all systems: all SS baselines also assume a known speaker count, so the known-count setting does not favor CoReSep. Moreover, requiring an upper bound is also common practice for separation with an unknown speaker count. Attractor-based methods, e.g., [4], also require a predefined maximum, and fixed-size outputs trained to emit silence for absent sources [5] are bounded by construction. We will clarify this in the revision.
References
[4] Y. Lee, S. Choi, B.-Y. Kim, Z.-Q. Wang and S. Watanabe, "Boosting Unknown-Number Speaker Separation with Transformer Decoder-Based Attractor," ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul, Republic of Korea, 2024, pp. 446-450, doi: 10.1109/ICASSP48485.2024.10446032.
[5] S. Wisdom et al., "What's all the Fuss about Free Universal Sound Separation Data?," ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada, 2021, pp. 186-190, doi: 10.1109/ICASSP39728.2021.9414774.
Response to Reviewer KEnk (1/4): W1, W2
We sincerely thank you for recognizing that CoReSep achieves consistently lower dWER than the evaluated baselines on both SS and TSE, particularly under noise. We respond to your comments as follows:
W1. Technical Novelty:
We agree that clean-representation regression and the flow-matching/vocoder reconstruction pipeline are inherited from PASE and StuPASE, respectively, and we do not claim them as methodological contributions. Our key contribution is input composition, which adapts a pretrained single-stream speech foundation model (SFM) to jointly perform multi-speaker separation and enrollment-conditioned TSE within a shared backbone, by letting each composed segment represent a single speaker while all streams are jointly modeled by the pretrained Transformer.
Input composition is indeed lightweight to implement, which we view as a strength rather than a limitation:
Highly effective and readily transferable. Input composition achieves the lowest dWER across all main comparisons on both SS and TSE, and clearly outperforms all ablated variants on SS. Moreover, it introduces no architectural modification, selection module, or new embedding unseen during pretraining. It can thus be directly applied to existing pretrained SFMs while fully retaining the benefits of pretrained initialization, and a single model handles both SS and TSE across two to four speakers.
Extensible to other tasks. PASE-style speech enhancement corresponds to the special case of a single mixture copy without enrollment, and other conditional speech tasks, such as acoustic echo cancellation and speech editing, could potentially be formulated by composing the corresponding inputs. Input composition thus offers a promising path toward unifying SS and TSE with a broader range of speech tasks within one pretrained backbone.
Regarding presentation, our terminology is intended to provide a concise, formal description of how SS and TSE share the same backbone and parameters while differing only in their input configurations. Terms such as the phonological prior are introduced to interpret why pretrained initialization benefits these tasks. We believe the current presentation balances formal rigor with brevity. Nevertheless, we will streamline it where possible and more clearly distinguish the central contribution from the inherited components.
W2. Phonological Prior:
Following PASE [1], we use “phonological prior” to refer to the contextual constraints on phonetic sequences that WavLM learns during pretraining. PASE used this concept to interpret the benefit of WavLM pretraining based on its analyses of speech enhancement. We adopted the term because CoReSep uses the same pretrained WavLM backbone.
Our initialization ablation shows that WavLM pretraining improves performance on SS and TSE in our setting. We agree that this comparison leaves open which aspects of pretraining account for the improvement.
PASE’s interpretation focuses on how phonetic context can guide the recovery of degraded speech. However, in SS and TSE, recovering speech content also requires determining which speaker produced it. Prior analyses of WavLM [2, 3] suggest that the same backbone provides cues for both purposes. Acoustic and speaker-related information is more prominent in lower and intermediate layers, while phonetic and word-level information is more prominent in higher layers. We therefore hypothesize that these cues play complementary roles in CoReSep, with lower and intermediate layers supporting speaker discrimination and higher layers helping maintain linguistic coherence within each output stream. These proposed layer-specific roles remain untested in our experiments. We will define the phonological prior in Section 4.3 and revise lines 446–452 to present our interpretation explicitly as a hypothesis.
References
[1] Rong, X., Hu, Q., Yesilbursa, M., Wojcicki, K., & Lu, J. (2026). PASE: Leveraging the Phonological Prior of WavLM for Low-Hallucination Generative Speech Enhancement. Proceedings of the AAAI Conference on Artificial Intelligence, 40(39), 32826–32834. https://doi.org/10.1609/aaai.v40i39.40562
[2] S. Chen et al., "WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing," in IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505-1518, Oct. 2022, doi: 10.1109/JSTSP.2022.3188113.
[3] A. Pasad, B. Shi and K. Livescu, "Comparative Layer-Wise Analysis of Self-Supervised Speech Models," ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece, 2023, pp. 1-5, doi: 10.1109/ICASSP49357.2023.10096149.