- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes CoSA to align audio with a frozen LLM using multiple textual descriptions. It trains a projector using responses generated from these descriptions, weights samples by response consistency, and adds an auxiliary probe to preserve audio attributes.
Critical strengths
- Motivation: Building LLMs that understand audio is an important research topic (as covered in Audio LLM, speech-aware LLM literature). Many methods use captions to align audio with pre-trained LLMs, but a single caption may miss important details. Like the authors try to motivate in the introduction, a caption describing a door closing may not mention footsteps before it or someone speaking at the same time. I think using multiple descriptions covering different "views" is a reasonable direction.
Critical weaknesses
-
Writing. I found the paper difficult to follow. For example, the introduction should explain more clearly what a "surrogate" is and how it is used in training. A clearer overview of the training process and the role of each component before introducing the equations would help readers understand the method.
-
Missing details and experiments. The paper does not clearly explain how the different views are generated. What's the extra cost of CoSA compared to other standard methods? The paper also needs to cover more backbones and more comprehensive experiments including quantitative studies.
Questions for authors
None