- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This work studies the representational geometry in 19 pretrained audio models (encoders) trained with different training regimes (supervised, SSL etc.) with different data domains (speech, music, sounds, birds etc.). Study reports several results which among which the most important are that more uniform embeddings work better without training and that discriminative models tend to have them. On the other hand when finetuning, we can gain more from clustered embeddings although those are also models with the weaker initial performance.
Critical strengths
- A very interesting, fresh, and potentially practically important topic
- The general idea for the experiments is insightful and when properly performed this could be an important study. I appreciate nevertheless the statistical evaluations performed to compare the results
Critical weaknesses
- The writing of the paper is unnecessarily convoluted, stretched, quite far from the proper academic style - see Introduction where the same information (e.g. evaluation on downstream tasks only and not geometry) is repeated several times. The core question the paper asks is finally defined in the second page (around line 68)
- The clarity of main contributions list can be improved. It’s hard to extract the true core contributions from those convoluted descriptions
- Given the recent advances with follow-up works related to The Platonic Representation Hypothesis, limiting the experiments to just a single domain is a bit disappointing. I’m curious how the main claims of this work would transfer even to the image domain. Nevertheless I don’t consider this as a significant limitation
- “To our knowledge, no prior work has systematically measured convergence across a broad, paradigmdiverse set of audio foundation models spanning multiple training paradigms, architectures, and domains.” - I recently came across “What Do Language Models Hear? Probing for Auditory Representations in Language Models” by Ngo & KIM ACL 2024 - probably worth taking a look, might be relevant, although authors consider rather correspondence between audio and language models
- Figure 1 is of the poor quality - some texts are barely readable
- The results presented in Figure 2 are not very convincing, with quite small differences, and the r=0 value is within the confidence intervals for almost all evaluated setups
- How does “Finding 4.2 showed that embedding uniformity governs whether a model’s frozen distances are already usable, without any downstream training.” - I think this is an overstatement or am I missing something? We can see from the results that kNN accuracy is higher for discriminative models, what is rather expected? It’s hard to clearly see those results, I’d suggest plotting them as e.g. a scatter plot of uniformity vs. frozen kNN accuracy, with points colored by domain and per-domain correlations.
- The main claim of Tab 2. is that models with more clustered embeddings have higher gains, but they also have lower initial scores, what is not taken into account
Small:
- There is an issue with hyperrref
Questions for authors
See weaknesses