- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This work introduces a new SSL framework for audio representation learning (NAPE). NAPEs pretraining does not use masked spectrogram/image reconstruction or student-teacher distillation but only uses a causal transformer to predict the next patch embedding from previous ones.
Critical strengths
Simple and novel new SSL method in audio. The proposed objective itself is much simpler than previous methods in audio and potentially removes some of the complicated or cumbersome objectives such as student-teacher distillation or reconstruction with extra decoders in pretraining. It only predicts the next patch embedding based on prior ones, an elegant alternative and potentially easy framework. It also seems to achieve competitive performance.
Exhaustive ablations. One of the most important aspects of SSL papers or new methods are exhaustive ablations. This work provides a pretty comprehensive investigation of the different architectural components and beyond.
Critical weaknesses
Incomplete SOTA, overstated positioning and missing baslines. Several recent SOTA methods are missing from Table 6. This includes the AudioMosaic paper as well as the Better Audio Transformer (both from this year's ICML). They would change performance rankings in the main benchmark and question the SOTA claims the authors make in the introduction. The linear probing comparison is also misleading and provides an incomplete picture. The paper references work in related work (which should be in the main paper to better position this work btw!) about more advanced probing methods (e.g., the unmute or the better audio transformer paper) and the change of model rankings without finetuning or the embedding positions for probing. However, no results for more advanced probing methods and/or other models are shown in this work. To make more credible SOTA claims and a better positioning of this model also without finetuning, some probes of the baselines (not offered in all the original papers but easy to do for a better evaluation within the same setup as this model) and/or advanced probing methods would be very important. The baseline results are all taken from previously published papers without the same training conditions.
Efficiency claims and runtime. The paper emphasizes simplicity and scalability compared to others (which is partly true), there is no runtime or computational comparison to other models. NAPE's does not provide real performance gains to other models. For instance, related base models like SSLAM (and other missing SOTA models) are even better than the large NAPE model on AS20k with much less parameters. So it's not clear if NAPE also offers a better training efficieny for scaling and computational cost/performance per parameter.
Audio-level causality. The paper's linearization/scanning strategy does not fully capture the chronological ordering across spectrogram frequencies. So the causlity is not really the audio but only the patch causality. Changes to the transformer architecture itself (to the patches to make full timeßband patches) would be a natural solution and caputre true audio-level causality.
Questions for authors
The questions are the weaknesses that can be addressed. I like the method and the paper and I'm open to lean toward accept for this paper if the weaknesses are properly addressed/answered.