acceptodds
Under review as a conference paper at ICLR 2027

Soloist: A Generative Model for Blind Voice Separation in the Wild

Abstract

Real-world voice separation must recover individual voices from overlapping speech, environmental noise, and music, yet isolated source recordings are rarely available for supervision. We introduce Soloist, a generative model for blind voice separation that autoregressively generates individual voice streams as audio-codec sequences conditioned on the mixture, without requiring a target-speaker query or a predefined voice count. We propose a two-stage training paradigm to learn from synthetic mixtures and adapt to real recordings. The first stage combines supervised separation on synthetic mixtures with isolated source targets and an auxiliary task that predicts speaker timelines from real recordings, learning who speaks when from natural acoustics. The second stage applies reinforcement learning to real recordings, with rewards supplied by a model judge that compares candidate separations based on voice purity and content preservation without isolated source targets. Across diverse scenarios, Soloist recovers clean speech from multi-speaker overlap and noisy recordings, and extracts singing vocals from music accompaniment while maintaining relatively high pitch accuracy and intelligibility. On real-world recordings with unknown voice counts, Soloist also separates voices well. Finally, we integrate Soloist into a full-duplex conversation processing pipeline that assembles separated outputs into continuous speaker tracks, demonstrating its practical utility in real conversations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
blind source separation, generative audio modeling, reinforcement learning, speech separation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4UGpOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Soloist, a 5.39B parameter generative model based on Qwen3-4B and discrete audio codec tokens for blind voice separation. The model is trained in two stages: supervised training on constructed speech mixtures, teacher-separated vocals, and speaker timelines, followed by reinforcement learning (RL) on real recordings using rewards from an LLM judge.

Critical strengths

The two-stage RL judge design for adapting a separator on real mixtures without ground-truth stems is a thoughtful idea.

Critical weaknesses
  1. The paper fine-tunes a massive 5.3 billion parameter LLM for a relatively straightforward voice isolation task, yet fails to compare against state-of-the-art separation backbones or standard unsupervised adaptation methods. Although the paper is written as if it solves general blind separation, it only evaluates on a tiny amount of 1-speaker or 2-speaker separation with limited synthetic datasets (and 46 WildTV clips) that do not support broad claims. The exact same RL adaptation could be applied to a state-of-the-art source separation model such as TF-Locoformer [A] without spending massive compute to fine-tune an LLM. Even more importantly, the authors do not compare against state-of-the-art unsupervised or self-supervised adaptation methods such as MixIT [B] for source separation or RemixIT [C] for speech enhancement, nor do they compare against baselines in terms of FLOPs, peak inference memory, and execution time.

  2. The evaluation does not use proper audio source separation fidelity metrics such as SI-SNR/SI-SDR [D] to measure waveform reconstruction fidelity, relying instead on ASR WER and DNSMOS. Furthermore, the evaluation setup against baselines is questionable: the authors compare against pretrained models trained on slightly different tasks, and in Table 3 they compute the WER of the best output speaker slot even though fixed output slots should not be evaluated in a permutation-invariant manner during inference.

[A] Saijo, K., Wichern, G., Cornell, S., Germain, F.G., Khurana, S. and Le Roux, J., 2024, September. TF-Locoformer: Transformer with local modeling by convolution for speech separation and enhancement. In 2024 18th International Workshop on Acoustic Signal Enhancement (IWAENC) (pp. 205-209). IEEE.

[B] Wisdom, S., Tzinis, E., Erdogan, H., Weiss, R.J., Wilson, K. and Hershey, J.R., 2020. Unsupervised sound separation using mixture invariant training. In Advances in Neural Information Processing Systems (NeurIPS) (Vol. 33, pp. 3846-3857).

[C] Tzinis, E., Adi, Y., Ithapu, V.K., Xu, B., Smaragdis, P. and Kumar, A., 2022. RemixIT: Continual self-training of speech enhancement models via bootstrapped remixing. IEEE Journal of Selected Topics in Signal Processing, 16(6), pp. 1329-1341.

[D] Le Roux, J., Wisdom, S., Erdogan, H. and Hershey, J.R., 2019, May. SDR-half-baked or well done?. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 626-630). IEEE.

Questions for authors

How does Soloist compare against a state-of-the-art discriminative separator (such as TF-Locoformer [A], TF-GridNet, or MossFormer) when that separator is adapted on the same unlabeled real recordings using unsupervised/self-training methods like MixIT [B] or RemixIT [C] (or fine-tuned with your LLM judge reward), alongside a full comparison of parameter count, FLOPs/MACs, peak GPU memory, and inference real-time factor (RTF)?

1 comment
Reviewer_4UGpOpenReview ↗

Response to authors

Thank you for the response and the new experiments.

  1. Thank you for clarifying how the streams are picked in Table 3. Table 25 confirms that SepFormer, MossFormer and ConvTasNet in Tables 3 and 4 are two speaker speech separation checkpoints, also for singing, so they are trained for a different task.

  2. TF-Locoformer gets a lower PI-WER than Soloist on LibriMix (0.044 against 0.060 on clean, 0.152 and 0.120 against 0.192 on noisy) with 15M parameters. Soloist needs about 500 times the parameters and an RTF of 0.23 against 0.011. This is not "a few times a specialist's compute". These results should be in Table 2 and not only in the appendix.

  3. On WildTV the baselines are evaluated without adaptation, while Soloist is trained on real film, TV and podcast recordings. This comparison does not show that a 5.39B model is needed. A standard separator adapted on the same real recordings with MixIT [B], RemixIT [C] or your judge is still missing. That these methods use a different training signal is the reason to compare against them. Your judge only ranks candidate outputs, as in Appendix C.2 where they come from different checkpoints, so it does not need an LLM that samples.

  4. In Table 22 Soloist gets an SI-SDR of -3.2 and -5.0 dB, which is below the unprocessed mixture (0.0 and -2.5 dB), where TF-Locoformer gets 15.2 and 8.2 dB. TF-Locoformer is also better on PESQ and ESTOI. The model resynthesizes the voices and does not recover the source signals, and these numbers should be in the main tables.

  5. The evaluation above two speakers is 10 WildTV clips, and nothing above three voices is evaluated.

Overall the scores do not look good against a much smaller separator and the evaluation remains marginal, so I keep my score unchanged.

Reviewer_3TPNOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper addresses blind voice separation, which separates the voices of different speakers into individual tracks without a target-speaker query or a given voice count. It proposes Soloist, a generative separator built on a Qwen3-4B backbone with AuT features and codec tokens. It first predicts the voice count and then generates one codec stream per voice via a local transformer. Supervised training combines constructed mixtures with known voice sources, songs with pseudo-vocal targets, and real recordings annotated only with speaker timelines. The model is then adapted with reinforcement learning on real recordings, where the reward is produced by two models: Qwen3-Omni describes the mixture and each separated track independently, and Qwen3.6-27B ranks the candidates against the unprocessed mixture as an anonymous anchor, with an order-consistency check. Soloist infers voice counts accurately on real multi-speaker recordings and achieves the lowest WER among the compared systems on real recordings, while specialist separators trained on matched synthetic data lead on LibriMix.

Critical strengths
  1. The paper combines several training signals in a single generative separator in a new way. One decoder predicts the voice count and separates a variable number of voice streams from mixtures with noise or music, without a target query. Speaker timelines, which carry no waveform targets, are cast as a text-only auxiliary task and effectively improve count inference. Reinforcement learning then compares complete candidate separations of the same real recording, extending LLM-style post-training to multi-stream separation without isolated sources.
  2. The model-judge reward is carefully designed. Describing each track independently before ranking in text, using the unprocessed mixture as an anonymous anchor, and filtering groups by order consistency address known weaknesses of model judges.
  3. The paper is thoroughly documented. The appendix specifies the data construction, output format, reward rules, and exact judge prompts, which supports reproducibility.
Critical weaknesses

Post-rebuttal update. The authors’ new experiments address both weaknesses. For the judge contribution, RL improves PI-WER over SFT at the true count (0.060 / 0.192 vs. 0.128 / 0.332), and among count-correct candidates the judge selects better separations than random selection or the count and structure gates (Table 18). For speaker identity, the new SpkSim results and the low rate of speaker swaps are encouraging; a dedicated speaker-verification embedding would make this evidence more conclusive, since WavLM layer-14 features also encode content. I have raised my score accordingly.


  1. Speaker identity preservation is not evaluated for speech. Soloist resynthesizes each voice through a codec rather than masking the mixture, so an output can contain the correct words while altering the voice or assigning it to the wrong speaker. The speech evaluations rely on WER-based metrics and the non-intrusive DNSMOS, which do not measure whether each stream preserves the identity of the corresponding speaker. Vocal similarity is reported only for singing. Since LibriMix provides isolated references, it would be helpful to report speaker similarity between each output stream and its matched reference.
  2. The contribution of the model-judge reward is not isolated. After RL, PI-WER drops substantially, while count accuracy rises to 1.000 on all sets. Because PI-WER counts all tokens of missing or extra streams as errors, and the signal checks directly penalize count mismatches against the annotated count, part of this gain may come from correcting the voice count rather than from better separation. As a result, the benefit of the describe-then-rank judge, which is the second contribution, is not directly shown.
Questions for authors
  1. The Mel-Band RoFormer teacher used for song pseudo-vocals is not included as a baseline, and SongYT uses its outputs as references. On MUSDB18-HQ, which has isolated vocals, Soloist has lower RPA (0.438) and SingerSim (0.819) than SAM Audio (0.643/0.909) and MossFormer (0.563/0.845). How does Soloist compare with the teacher and other dedicated vocal separators on MIR-1K and MUSDB18-HQ?
  2. On ViW, all separators increase WER over the unprocessed input (0.261 vs. 0.302 for Soloist). Since DNSMOS is non-intrusive (reference-free) and may favor resynthesized speech, how does Soloist compare with dedicated speech enhancement models on this set?
Reviewer_gEV7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Soloist separates an unknown number of voices by predicting a voice count and then generating a codec sequence for each output stream. Continuous and discrete representations of the mixture condition an autoregressive language model. Training combines synthetic mixtures, teacher-separated singing targets, and speaker timelines from natural recordings. Reinforcement learning adds structural checks and semantic rankings obtained from audio descriptions. Experiments cover speech, singing, and noise suppression, with the strongest reported results in transcription accuracy and voice counting.

Critical strengths
  • Combining source supervision with natural speaker timelines addresses the limited availability of isolated targets for real recordings. Soloist can return a variable number of streams without a speaker query. Table 6 tests the contributions of timeline supervision and reinforcement learning, so these design choices have evidence beyond the final system comparison.
  • On LibriMix with the source count supplied, Soloist achieves PI-WER of 0.060 for clean mixtures and 0.192 for noisy mixtures. Blind evaluation on WildTV gives PI-WER of 0.150. Accounting for unmatched streams makes the latter evaluation relevant to deployment, where predicting too many or too few voices is part of the problem.
  • Reward construction is documented through the mixture anchor, order-consistency filtering, count penalties, and per-stream scores. Appendix C.2 checks the semantic ranking procedure against transcription errors under controlled conditions. Reporting the less favorable MUSDB18-HQ results also gives a concrete boundary on the demonstrated performance.
Critical weaknesses
  • Faithful voice recovery remains less established than transcription accuracy. WER and PI-WER do not measure whether generated codec sequences preserve speaker identity, vocal detail, or timing. Singing metrics address part of this concern, although Table 4 reports RPA of 0.438 and singer similarity of 0.819 on MUSDB18-HQ, versus 0.643 and 0.909 for SAM Audio. A plausible, intelligible reconstruction can still differ from the source. Since recovery of voices in natural recordings is a central claim, I would like a focused speech evaluation of identity preservation and residual interference, using source references or independent listening judgments.
  • Table 6 combines better counting with changes in within-stream quality, so it does not isolate the semantic reward's contribution. Reinforcement learning raises count accuracy from 0.926, 0.735, and 0.761 to 1.000 across the three datasets, and PI-WER penalizes missing or extra streams. Appendix C.2 reports 88.9% pairwise agreement only after selecting retained groups, sufficiently large WER differences, and non-tied rankings. Its ranking temperature and filtering also differ from those used during reinforcement learning. Existing fixed-count results would separate part of this effect, and a count-only reward control in a revision would test the independent contribution of semantic ranking.
  • WildTV contains 46 manually checked clips, which limits the breadth of the natural-recording evidence without uncertainty estimates. Because training also uses natural media, documenting source-level separation between training, reward development, and WildTV would resolve the remaining data-overlap ambiguity, alongside uncertainty across evaluation clips.
Questions for authors
  1. Does the existing evaluation include fixed-count results for both SFT and RL, or a subset where both predict the correct count? Such results would distinguish count correction from improved transcription within matched streams.
  2. What source-level checks exclude WildTV recordings from training and reward development? Were clips from the same underlying recording treated as a single source during this check?
  3. How do the authors interpret the conditional agreement in Appendix C.2 under the actual reward settings? In particular, what motivates the different ranking temperature and filtering choices in validation and training?

Comments on the paper

1 comment
Area_Chair_9Xd2OpenReview ↗

Reminder: Discussion Phase Ends November 18 (11:59 PM AoE)

Hi reviewers, a quick reminder that the discussion phase ends on November 18 at 11:59 PM AoE. Please take some time to read the author responses, engage in further discussions if you have any remaining questions or concerns, and update your reviews and scores as appropriate. Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.