- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies how well frozen audio encoders (CLAP, AST, and AudioMAE) retain information about a weaker audio source when probed on synthetic mixtures of sources from ESC-50, UrbanSound8K, and FSD50K. The authors point out two shortcuts in mixture probing benchmarks (sampling distinct classes without replacement and pairing sources by natural co-occurrence), and show that after closing these shortcuts, linear and shallow probes on frozen clip embeddings degrade as the number of sources increases. Using an oracle Ideal Binary Mask (IBM) corrupted by random bin flipping, they argue that this degradation is driven by the lack of masker suppression in the frozen encoder, and show that pooling over frame tokens recovers part of the lost accuracy compared to clip-level pooling.
Critical strengths
None
Critical weaknesses
-
The main finding of the paper (that audio classifiers fail to capture the weakest source in a mixture as interfering sources are added, and that source separation resolves this) is an already known property of audio classifiers that has been shown in the literature, specifically by Tzinis et al. [A]. Evaluating this known property on newer encoders like CLAP does not make the core claim novel, and the authors completely neglect to compare against existing audio source separation methods used as a front-end before CLAP or the other encoders [A] to show what the actual capability of these models is when paired with real source separation.
-
Several claims in the paper are incomplete and reflect a weak grounding in the audio separation literature. First, what actually determines whether a classifier captures multiple sources is not the raw number of sources in the mixture, but the actual overlap of those sources in the time-frequency (Fourier) domain, as was also shown in [A] since models can easily capture multiple sources when they do not overlap heavily in the Fourier domain. Second, the authors only use an Ideal Binary Mask (IBM) rather than standard separation masks such as the Ideal Ratio Mask (IRM) or Ideal Phase-Sensitive Mask (PSM) [B]. Third, the paper is poorly written (for example, putting hyper-specific details like AUC per with 95% CIs and 63 cells directly in the abstract) and relies on trivial synthetic experiments and filler math like Proposition 2, which assumes uniform energy retention in expectation across frequencies even though audio representations are sparse in the time-frequency domain.
[A] Tzinis, E., Wisdom, S., Hershey, J.R., Jansen, A. and Ellis, D.P., 2020, May. Improving universal sound separation using sound classification. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 96-100). IEEE.
[B] Erdogan, H., Hershey, J.R., Watanabe, S. and Le Roux, J., 2015, April. Phase-sensitive and recognition-boosted speech separation using deep recurrent neural networks. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 708-712). IEEE.
Questions for authors
Can you provide results where an actual learned audio source separation model is used as a front-end before CLAP, AST, and AudioMAE, as well as oracle evaluations using ideal ratio mask or phase-aware masks (IRM or PSM [B]) instead of random bit-flipping of a hard IBM?
[B] Erdogan, H., Hershey, J.R., Watanabe, S. and Le Roux, J., 2015, April. Phase-sensitive and recognition-boosted speech separation using deep recurrent neural networks. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 708-712). IEEE.