acceptodds
Under review as a conference paper at ICLR 2027

MEmoS2S-Bench: An Open-Ended Benchmark for Emotional Understanding, Reasoning, and Empathetic Spoken Response Generation in Multilingual Speech-LLMs

Abstract

Speech large language models (SpeechLLMs) are deployed as spoken conversational partners, a role that requires hearing how something was said, reasoning about why the speaker feels that way, and answering aloud. Existing benchmarks test these abilities in isolation, so they cannot show whether the stages work together inside one interaction. We present MEmoS2S-Bench, which elicits an emotion, a rationale, a text reply and a spoken reply within a single interaction, intervenes on each intermediate stage, and pairs every intervention with controls. It comprises MELD-ST-Reason (3180 utterances in English, German and Japanese) and MDET-Reason (11951 utterances in English and Chinese); we evaluate twelve SpeechLLMs over 3720 scored cells. The controls expose three gaps that stage-wise evaluation cannot see. A hearing gap: from speech alone the models recover less emotion than a text-only LLM reading the words (macro-F1 0.358 against 0.512 on English), and every model improves with the transcript. A credit gap: a judge shown the gold emotion credits gold intermediates with gains that shrink several-fold under a label-blind judge, and a rationale helps only through the emotion it carries: one borrowed from another item with the same emotion works as well as the item's own. With delivery held fixed, the correctness of a model's own emotion makes no reliable difference to a blind judge. A saying gap: several models write the expected language but their speech is identified as another; for Japanese, five models' speech is identified as Chinese with high confidence. MEmoS2S-Bench makes these gaps measurable and attributable to a stage.

Primary area:
datasets and benchmarks
Keywords:
speech language models, emotional reasoning, empathetic response generation, multilingual benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CrxoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces MEmoS2S-Bench, which evaluates 4 multimodal abilities in a single interaction. The authors evaluate 12 speechLLMs in three groups; General speechLLMs, Speech-LLMs with reasoning, and emotion-focused speechLLMs. For statelevel, no model leads every stage. The paper presents 3 gaps (hearing, credit, and saying) that were not exposed in stage-wise evaluation. They also release 2 datasets; MELD-ST-Reason and MDET-Reason.

Critical strengths

The paper shows that in order to understand an interaction, one should look at the holistic picture and not just a single aspect. The authors made great effort in identifying benchmarks, collecting data and running the experiment.

Critical weaknesses

There is no human validation of gold rationales. All quality is judged by LLMs only. Although this speeds up the research process, it is dangerous to rely on the LLM labels without any verification from human coders

Questions for authors

Is there a reason for no human validation?

Reviewer_Yh8jOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces MEmoS2S-Bench, a multilingual benchmark for evaluating emotion recognition, emotion-grounded reasoning, and empathetic text and speech generation in SpeechLLMs. It combines existing speech datasets with automatically generated emotion rationales and intervention-based evaluations. Experiments on twelve SpeechLLMs reveal limitations in speech emotion recognition, discrepancies between label-visible and label-blind LLM evaluations, and inconsistencies between generated text and speech across languages.

Critical strengths

Systematic intervention-based evaluation

The benchmark examines how intermediate emotion predictions and rationales affect downstream responses rather than evaluating each capability independently. In particular, the comparison between label-visible and label-blind judges reveals substantial evaluation bias.

Multilingual text–speech consistency analysis

The paper identifies cases where models generate text in the expected language but fail to produce corresponding speech. This exposes limitations that text-only evaluations would miss.

Critical weaknesses

Questionable validity of the emotion reasoning ground truth

The gold rationales are generated using ground-truth emotion labels, and the generation process explicitly asks the model to justify the gold emotion when its initial prediction disagrees. This raises concerns that the rationales reflect post-hoc label justification rather than independently grounded acoustic reasoning. Although the authors provide acoustic consistency checks and audio-aware validation, these do not fully address the label-conditioning bias. The paper also shows that gold emotions can be recovered from rationales with over 91% accuracy and that rationales borrowed from different utterances with the same emotion have comparable downstream effects. These findings suggest that rationale-based evaluation may primarily reward emotion-label alignment rather than meaningful, utterance-specific reasoning. Independent human validation of the reference rationales remains necessary.

Limited validation of empathetic speech generation and potential protocol bias

The primary response evaluation relies on text-based LLM judges. Although an audio-aware judge is included, it evaluates responses using the input speech rather than directly assessing the emotional delivery of generated speech. Without human listening evaluations, the benchmark does not establish whether generated speech conveys appropriate empathy. Furthermore, the evaluation imposes structured intermediate outputs and text-before-speech generation without demonstrating equivalence to the models' native S2S inference. The prompts also encourage generic responses when transcripts are unavailable, while the evaluation rubric penalizes generic replies. These limitations make it difficult to distinguish genuine weaknesses in empathetic spoken interaction from effects introduced by the evaluation protocol.

Questions for authors
  1. How do the authors distinguish genuine acoustic reasoning from post-hoc label justification in gold rationales generated using ground-truth emotions? Can they validate the reasoning metrics against independently annotated human rationales?

  2. Can the authors provide human listening evaluations to establish whether the automatic and LLM-based metrics reflect perceived empathy and appropriateness of the generated speech?

  3. How does the structured evaluation protocol compare with each model's native S2S inference? Does requiring explicit intermediate outputs affect response quality or model rankings?

Reviewer_5kRfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a benchmark focus on speech-to-speech emotional interaction, namely MEmoS2S-bench. Ten prompting conditions withhold, elicit, or insert gold versions of the emotion and rationale, to support a various testing scenarios. The authors release two datasets: MELD-ST-Reason and MDET-Reason. The first one takes three languges and the second include bilingual setup. The report includes 12 speech llms and three main findings are shared, including:

  • Hearing gap: speech-only emotion recognition falls below a text-only GPT-4.1 reading the transcript.
  • Credit gap: the reply-quality gains from gold intermediates mostly disappear when the LLM judge is not told the gold emotion, and a borrowed rationale with the same emotion helps as much as the item’s own.
  • Saying gap: several models write the correct language but speak another; for example, Japanese replies are identified as Chinese.
Critical strengths
  • The main design is useful for speech LLM designs. It evaluates all stages on the same interaction and keeps the text and spoken replies separate so that their agreement can be measured.
  • The control experiments are solid (notably, table 4, 5, 6, and 17), including a variety of ablations.
  • The text-speech mismatch is evaluated in details. The authors also check the Whisper language identifier on human input speech, which show that the issue is not an LID artifact.
  • The paper is frank on its own defects (the duplicated tags in the translated files, the substring false positive in the leakage check, and the single-row cell in the earlier reasoning-language grid)
Critical weaknesses
  • No human evaluation is a main weakness. The central credit gap claim depends on it -> and emotion transfer is important to test with human annotations. A human rating study on a reasonable subset would make the central claim defensible. It should include raters who hear the audio and raters who are and are not shown the label.
  • MELD-ST labels may not fit the benchmark, especially for dubbed languages. Basically, label transfer to dubs and text carries most of the label signal.
  • The gold rationales in the paper do not support a reasoning stage score due to its origin (produced by audio reasoner with gold labels)
  • The “saying gap” partly reflects documented language support, and some numbers need checking. To my knowledge, several of the evaluated models (e.g., VITA, Qwen, Kimi) are all documented with Chinese and English for speech generation. Japanese read as Chinese may be due to that.
Questions for authors
  • Can you add a human evaluation (audio-hearing raters, with and without the label) on a subset to establish which judge tracks human judgment?
  • What are the documented speech-output languages of each model? How do the saying-gap results look when restricted to supported languages?
  • Can you report the full-pipeline results on MDET-Reason, not only recognition?
  • How do the German and Japanese results change on items where the dubbed audio was re-annotated, or where emotion2vec+agrees with the label?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.