- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Motivated by the fact that we now have strong joint audio-language embedding models and captioning models, this paper presents a method for few-shot audio classification aims to leverage support informatino from these two modalities as needed. They contribute two modules: Prior-Centric Semantic Scaffold (PriorCore) and Support-Aware Transductive Refinement (SATR). The first aims to fuse semantic (class text) and support information (caption text + audio) at both temporally fine and coarse grains, this is then converted into a reliability signal that informs how reliable audio and caption support is over the target classes. In parallel audo and caption features are input into a label propagation network (SATR), the outputs of which are merged with that of the direct support evidence, modulated by the support reliability signal. They compare this approach with existing few-shot methods on ESC-50, US8k, and FSDKaggle2018, along with ablation to help understand the impact of their contributions.
Critical strengths
- The paper identifies an interesting problem that has arisen in the past few years but hasn't been adequately addressed, i.e., "how do we leverage both audio-language models and few-shot support for low-data regime audio classification?"
- The solution appears to be sound and performs well over competing methods and baselines.
Critical weaknesses
- Clarity. First of all, the introduction was clear. I had no problem with that. The methods were mostly clear but should be improved. Section 3 should do a better job communicating how the different modules relate to the overall motivation of the paper. For me, the initial confusion was between the caption information and the class text information. I suggest clearly showing that the text class information is only used in PriorCore to relate the support information to the classes, and that the "support" is inclusive of both the audio feature and the audio caption text features. The experiments were not clear. In Section 4.2, it's unclear from the paper how classes are handled in the experiments, e.g., how the novel classes are held out. Rather than point to MGCL, please state it in the paper, as this is important for understanding the validity of the experiments. Also, while MGCL paper contains ESC-50 and FSDKaggle2018, it doesn't include US8k. So, when the paper states that US8k follows the protocol where twelve-way vocabulary repeats two of the classes, this is confusing since it is not in MGCL paper. Furthermore, US8k only has 10 classes. So, what is the training data for US8k in a 12-way setup? Maybe the whole model is trained on the aggregation of all the datasets? If so, that isn't communicated in the paper and needs to be. Related, the statement "The rows other than PCSA in Table 1 are as reported under the MGCL protocol (Zhao et al., 2024)" - what aspects of the MGCL paper is this referring to? Please clarify.
- The experiments don't adequately isolate the different proposed modules, which lowers the informativeness of the paper. Ablations results are muddied by the addition of the cache module that wasn't even introduced in this paper (and in general it seems the cache hurts performance). At the moment, it is unclear where the real gains come from, especially as it relates to the motivation, e.g.,
- Caption (i.e., text-mediated) features vs audio features
- SATR vs PriorCore (Or even a full analysis of PiorCore vs SATR + PriorCore, rather than just a few episodes)
- Fine vs Coarse
- The experiments and analysis don't adequately answer the motivation of this paper, which lowers the informativeness of the paper. I would think that with audio examples you would gain specificity while leveraging class prior information, e.g., you want to detect certain instances of a class, rather than all instances of a class, like a specific type of engine, power tool, drum, or even an individual one, like in a few-shot individual id scenario or even few-shot audio-moment retrieval. But the current evaluation setup does not investigate this. In its current form, it is unclear what advantage there is by combining these two support types and if both are necessary or not. This requires more than just new ablations, but a new evaluation setup as well. At the very least the experiments should dive deeper into when one source of information is needed more than the other. Section 4.6 notes that the benefit of of SATR for instance varies a lot by episode, but it doesn't investigate the charcteristics of those episodes, i.e., why was it helpful in certain episodes but not others.
Questions for authors
- Were the models trained on each dataset separately or in aggregate?
- How were classes held out for evaluation? In particular, for US8k, which only has 10 classes.
- How should we interpret the US8k results when two of the classes are repeated?
- What aspects of the MGCL protocol does the paper adopt
- Why was CLAP-Base not included for US8k?
- What is the performance comparison of caption (i.e., text-mediated) features vs audio features?
- What is the performance comparison SATR vs PriorCore?
- What is the performance comparison of fine vs coarse?
- How would you augment/modify the experimental design or validation tasks to more clearly demonstrate the benefit of the proposed method?
- In what scenarios does the model benefit from semantic prior vs. support information?