- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces MEmoS2S-Bench, which evaluates 4 multimodal abilities in a single interaction. The authors evaluate 12 speechLLMs in three groups; General speechLLMs, Speech-LLMs with reasoning, and emotion-focused speechLLMs. For statelevel, no model leads every stage. The paper presents 3 gaps (hearing, credit, and saying) that were not exposed in stage-wise evaluation. They also release 2 datasets; MELD-ST-Reason and MDET-Reason.
Critical strengths
The paper shows that in order to understand an interaction, one should look at the holistic picture and not just a single aspect. The authors made great effort in identifying benchmarks, collecting data and running the experiment.
Critical weaknesses
There is no human validation of gold rationales. All quality is judged by LLMs only. Although this speeds up the research process, it is dangerous to rely on the LLM labels without any verification from human coders
Questions for authors
Is there a reason for no human validation?