- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Soloist, a 5.39B parameter generative model based on Qwen3-4B and discrete audio codec tokens for blind voice separation. The model is trained in two stages: supervised training on constructed speech mixtures, teacher-separated vocals, and speaker timelines, followed by reinforcement learning (RL) on real recordings using rewards from an LLM judge.
Critical strengths
The two-stage RL judge design for adapting a separator on real mixtures without ground-truth stems is a thoughtful idea.
Critical weaknesses
-
The paper fine-tunes a massive 5.3 billion parameter LLM for a relatively straightforward voice isolation task, yet fails to compare against state-of-the-art separation backbones or standard unsupervised adaptation methods. Although the paper is written as if it solves general blind separation, it only evaluates on a tiny amount of 1-speaker or 2-speaker separation with limited synthetic datasets (and 46 WildTV clips) that do not support broad claims. The exact same RL adaptation could be applied to a state-of-the-art source separation model such as TF-Locoformer [A] without spending massive compute to fine-tune an LLM. Even more importantly, the authors do not compare against state-of-the-art unsupervised or self-supervised adaptation methods such as MixIT [B] for source separation or RemixIT [C] for speech enhancement, nor do they compare against baselines in terms of FLOPs, peak inference memory, and execution time.
-
The evaluation does not use proper audio source separation fidelity metrics such as SI-SNR/SI-SDR [D] to measure waveform reconstruction fidelity, relying instead on ASR WER and DNSMOS. Furthermore, the evaluation setup against baselines is questionable: the authors compare against pretrained models trained on slightly different tasks, and in Table 3 they compute the WER of the best output speaker slot even though fixed output slots should not be evaluated in a permutation-invariant manner during inference.
[A] Saijo, K., Wichern, G., Cornell, S., Germain, F.G., Khurana, S. and Le Roux, J., 2024, September. TF-Locoformer: Transformer with local modeling by convolution for speech separation and enhancement. In 2024 18th International Workshop on Acoustic Signal Enhancement (IWAENC) (pp. 205-209). IEEE.
[B] Wisdom, S., Tzinis, E., Erdogan, H., Weiss, R.J., Wilson, K. and Hershey, J.R., 2020. Unsupervised sound separation using mixture invariant training. In Advances in Neural Information Processing Systems (NeurIPS) (Vol. 33, pp. 3846-3857).
[C] Tzinis, E., Adi, Y., Ithapu, V.K., Xu, B., Smaragdis, P. and Kumar, A., 2022. RemixIT: Continual self-training of speech enhancement models via bootstrapped remixing. IEEE Journal of Selected Topics in Signal Processing, 16(6), pp. 1329-1341.
[D] Le Roux, J., Wisdom, S., Erdogan, H. and Hershey, J.R., 2019, May. SDR-half-baked or well done?. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 626-630). IEEE.
Questions for authors
How does Soloist compare against a state-of-the-art discriminative separator (such as TF-Locoformer [A], TF-GridNet, or MossFormer) when that separator is adapted on the same unlabeled real recordings using unsupervised/self-training methods like MixIT [B] or RemixIT [C] (or fine-tuned with your LLM judge reward), alongside a full comparison of parameter count, FLOPs/MACs, peak GPU memory, and inference real-time factor (RTF)?
Response to authors
Thank you for the response and the new experiments.
Thank you for clarifying how the streams are picked in Table 3. Table 25 confirms that SepFormer, MossFormer and ConvTasNet in Tables 3 and 4 are two speaker speech separation checkpoints, also for singing, so they are trained for a different task.
TF-Locoformer gets a lower PI-WER than Soloist on LibriMix (0.044 against 0.060 on clean, 0.152 and 0.120 against 0.192 on noisy) with 15M parameters. Soloist needs about 500 times the parameters and an RTF of 0.23 against 0.011. This is not "a few times a specialist's compute". These results should be in Table 2 and not only in the appendix.
On WildTV the baselines are evaluated without adaptation, while Soloist is trained on real film, TV and podcast recordings. This comparison does not show that a 5.39B model is needed. A standard separator adapted on the same real recordings with MixIT [B], RemixIT [C] or your judge is still missing. That these methods use a different training signal is the reason to compare against them. Your judge only ranks candidate outputs, as in Appendix C.2 where they come from different checkpoints, so it does not need an LLM that samples.
In Table 22 Soloist gets an SI-SDR of -3.2 and -5.0 dB, which is below the unprocessed mixture (0.0 and -2.5 dB), where TF-Locoformer gets 15.2 and 8.2 dB. TF-Locoformer is also better on PESQ and ESTOI. The model resynthesizes the voices and does not recover the source signals, and these numbers should be in the main tables.
The evaluation above two speakers is 10 WildTV clips, and nothing above three voices is evaluated.
Overall the scores do not look good against a much smaller separator and the evaluation remains marginal, so I keep my score unchanged.