- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces a new method for few-shot multi-label audio tagging that addresses systematic over-prediction calibration failures. It presents a small benchmark with given splits to evaluate generalization to unseen classes. The work frames few-shot multi-label audio tagging around decision quality rather than ranking performance alone.
Critical strengths
- efficient and elegant method translated from vision to audio.
- the topic is relevant to the field
- a small scale benchmark is introduced for better future evaluation
Critical weaknesses
Missing citations. Especially the introduction contains many claims that are not backed up by citations at all. This creates a false impression of novelty. For example, "rare annotations" or "inherently multilabel" is mentioned but no related work that backs this claim up. Bioacoustic monitoring is mentioned as an example but e.g. no birdset dataset/benchmark that underlines this argument. Additionally, AudioSet or FSD50k are introduced as standard datasets without citing them properly or citing papers that use these as the standard setting. This makes the field appear completely novel. All citations are not properly formatted with brackets.
Order of definitions. For example, the paper suddenly introduces terms like prototype. There are a lot of different prototype definitions and the explanation came pretty late after using the term in this work. Same for "pretrain-then-meta-train recipe", this should be introduced earlier so that it's understandable.
Incorrect statements and incomplete related work. There are also statements like: "prior multilabel fewshot tagging centers on singlelabel" without any citations. Moreover, this is not true, especially in bioacoustics that is mentioned as a use case. For example these works do few-shot multilabel classification:
- "Simulated soundscapes and transfer learning boost the performance of acoustic classifiers under data scarcity" (Weldy et al.)
- "Can Masked Autoencoders Also Listen to Birds?" (Rauch et al.)
- "Domain-Invariant Representation Learning of Bird Sounds" (Moummad et al.)
- this and a lot of others are missing in realted work as well
There are also SSL pretraining methods missing in related work (very random and barebones). Current SOTA models in audio such as Audio-MAE, EAT, SSLAM, BAT, AudioMosaic are all missing (and none of them is used in the benchmark).
Limited data and model scope. The FSD50k/AudioSet benchmark for example does not contain any bioacoustic datasets even though this is mentioned a few times as a motivation. There are a lot of bioacoustic datasets available that could, at least partially such as Birdset, be used for this. Also there is no benchmark engagement with current SOTA SSL models even though they dominate current SOTA on AudioSet (see the models mentioned above). A more complete picture would include modern SSL approaches as well. Also the margins of the results are pretty small and only one seed was used, questioning the statistical significance of the results.
Lack of method ablations. While there are ablations on shot count and computational cost, there are no ablations about what makes the method work (only the GAT in the results table as an additional ablation).
Questions for authors
Please address the weaknesses mentioned above.